jiebi/CodeConvo huggingface dataset:
Code change:
nodeInfoCopy := potentialNodes[(int(offset)+i)%len(potentialNodes)].Clone() stateCopy := state.Clone() pods, numPDBViolations, status := selectVictimsOnNode(ctx, fh, stateCopy, pod, nodeInfoCopy, pdbs) <del> if status.IsSuccess() { </del> <ins> if status.IsSuccess() && len(pods) != 0 { </ins> victims := extenderv1.Victims{ Pods:             pods, NumPDBViolations: int64(numPDBViolations),

Related discussion:
text: If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.
Code change:
nodeInfoCopy := potentialNodes[(int(offset)+i)%len(potentialNodes)].Clone() stateCopy := state.Clone() pods, numPDBViolations, status := selectVictimsOnNode(ctx, fh, stateCopy, pod, nodeInfoCopy, pdbs) <del> if status.IsSuccess() { </del> <ins> if status.IsSuccess() && len(pods) != 0 { </ins> victims := extenderv1.Victims{ Pods:             pods, NumPDBViolations: int64(numPDBViolations),

Related discussion:
text: Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.
Code change:
nodeInfoCopy := potentialNodes[(int(offset)+i)%len(potentialNodes)].Clone() stateCopy := state.Clone() pods, numPDBViolations, status := selectVictimsOnNode(ctx, fh, stateCopy, pod, nodeInfoCopy, pdbs) <del> if status.IsSuccess() { </del> <ins> if status.IsSuccess() && len(pods) != 0 { </ins> victims := extenderv1.Victims{ Pods:             pods, NumPDBViolations: int64(numPDBViolations),

Related discussion:
text: Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.
Code change:
type DynamicPluginProber interface { Init() error <del> // If an error occurs, events are undefined. </del> <ins> // aggregates events for successful drivers and errors for failed drivers </ins> Probe() (events []ProbeEvent, err error) }

Related discussion:
text: When kubelet is running if you create two volume plugin directories here: /usr/libexec/kubernetes/kubelet-plugins/volume/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/cifs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds You will see these expected logs: Now if you copy over the cifs plugin file to cifs directory and then copy the uds plugin the errors go away. The pods using the uds volume will function but any pods attempting to use the cifs volume will fail with the following error from kubelet: desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pv" err=no volume plugin matched Restarting kubelet fixes this problem as then both plugins are present and properly . A functional volume plugin should be to kubelet even if another volume plugin is missing or invalid. Fixing the invalid volume plugin currently only adds the one that was invalid and fixed and other valid plugins that were previously present are skipped. Start kubelet service on a node. Create two directories for two flex volume plugins in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory. Eg. nodeagent~uds and cifs Kubelet logs will now start complaining that the plugins are missing. Add the files to the plugin folders and then start a pod using the plugin. Notice that only the last plugin file copied is available to be used by pods. The first plugin will generate this error if used by a volume "err=no volume plugin matched" Performing a touch on the folder of the missing plugin will cause kubelet to properly index it. <details</details<detailsLocal kube cluster </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig k8s-infra
/sig storage /remove-sig k8s-infra
Code change:
// Check with the cloud provider to see if the node still exists. If it // doesn't, delete the node immediately. <del> exists, err := ensureNodeExistsByProviderIDOrInstanceID(instances, node) </del> <ins> exists, err := ensureNodeExistsByProviderID(instances, node) </ins> if err != nil { glog.Errorf("Error getting data for node %s from cloud: %v", node.Name, err) continue

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The change here removes exiternal id, but the call to instance id was not made used at all, it will always return false. At this time no node was able to register What you expected to happen: ProviderId was used in check How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
similar to are you using --cloud-provider in your controller manager? are you also setting it on your kubelet?
I'm using acs-engine to deploy the cluster. Config is as following: I've proposed a fix. PTAL
/sig cluster-lifecycle
Code change:
setup run_federation_apiserver run_federation_controller_manager <del> # TODO: Fix for secrets, replicasets and deployments. SUPPORTED_RESOURCES=("configmaps" "daemonsets" "events" "ingress" "namespaces" "services") </del> <ins> # TODO: Fix for replicasets and deployments. SUPPORTED_RESOURCES=("configmaps" "daemonsets" "events" "ingress" "namespaces" "services" "secrets") # Set wait for deletion to true for federation apiserver since resources are # deleted asynchronously. # This is a temporary workaround until https://github.com/kubernetes/kubernetes/issues/42594 is fixed. WAIT_FOR_DELETION="true" </ins> # WARNING: Do not wrap this call in a subshell to capture output, e.g. output=$(runTests) # Doing so will suppress errexit behavior inside runTests runTests

Related discussion:
text: disabled them since was flaky. We need to find and fix the issue and then reenable these tests. cc Related issues: and
Probably should be higher priority so it doesn't get dropped
The issue is that we dont wait for the resource to be deleted unless we are using a reaper: This breaks for federation since we use finalizers. kubectl should wait for the resource to be deleted. We should also set OrphanDependents = false. This will have no effect in kubernetes but will enable cascading deletion in federation. Ref cc Will send a fix.
Sent
Updated to remove the wait (GET) at the end so it no longer fixes this issue. Sent to update the test to include a wait which fixes this issue.
Code change:
import ( "context" "fmt" <ins> "reflect" </ins> "time" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/client-go/util/workqueue" "k8s.io/klog" "k8s.io/kubernetes/pkg/scheduler/apis/config"

Related discussion:
text: There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help
Code change:
"testing" "github.com/stretchr/testify/require" <ins> auditapi "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit" </ins> "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/testutil" )

Related discussion:
text: go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.
Code change:
"testing" "github.com/stretchr/testify/require" <ins> auditapi "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit" </ins> "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/testutil" )

Related discussion:
text: adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:
Code change:
return fmt.Sprintf(`{"spec":{"template":{"spec":{"containers":[{"name":"%s","image":"%s"}]}}}}`, containerName, containerImage) } <del> // deleteDaemonSetAndOrphan deletes the given DaemonSet and orphans all its dependents. // It also checks that all dependents are orphaned, and the DaemonSet is deleted. func deleteDaemonSetAndOrphan(c clientset.Interface, ds *apps.DaemonSet) { trueVar := true deleteOptions := &metav1.DeleteOptions{OrphanDependents: &trueVar} deleteOptions.Preconditions = metav1.NewUIDPreconditions(string(ds.UID)) err := c.AppsV1().DaemonSets(ds.Namespace).Delete(ds.Name, deleteOptions) Expect(err).NotTo(HaveOccurred()) err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkDaemonSetPodsOrphaned(c, ds.Namespace, ds.Spec.Template.Labels)) Expect(err).NotTo(HaveOccurred(), "error waiting for DaemonSet pods to be orphaned") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkDaemonSetHistoryOrphaned(c, ds.Namespace, ds.Spec.Template.Labels)) Expect(err).NotTo(HaveOccurred(), "error waiting for DaemonSet history to be orphaned") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkDaemonSetDeleted(c, ds.Namespace, ds.Name)) Expect(err).NotTo(HaveOccurred(), "error waiting for DaemonSet to be deleted") } </del> func newDaemonSet(dsName, image string, label map[string]string) *apps.DaemonSet { return &apps.DaemonSet{ ObjectMeta: metav1.ObjectMeta{

Related discussion:
text: Please see : https://k8s- fails just after this line: Failure was introduced in: But this was not noticed until it started running in [Conformance] tests. You can see the same problem in other dashboards too: https://k8s- https://k8s-
cc /sig apps
/priority critical-urgent
/kind bug /priority failing-test
/milestone v1.10
Is this just an oversight of not removing the template generation when the API update was done?
/cc
triaged in progress
i have never touched this test before ... so i just threw in a WIP, feel free to use/discard it. there seems to be lot of methods in there that is no longer used, may be we can rip those out too. (see )
xref
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action Required: This issue has not been updated since Mar 5. Please provide an update. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Code change:
&o.config.Conntrack.TCPCloseWaitTimeout.Duration, "conntrack-tcp-timeout-close-wait", o.config.Conntrack.TCPCloseWaitTimeout.Duration, "NAT timeout for TCP connections in the CLOSE_WAIT state") <ins> fs.BoolVar(&o.config.Conntrack.TCPBeLiberal, "conntrack-tcp-be-liberal", o.config.Conntrack.TCPBeLiberal, "Enable liberal mode for tracking TCP packets by setting nf_conntrack_tcp_be_liberal to 1") </ins> fs.DurationVar(&o.config.Conntrack.UDPTimeout.Duration, "conntrack-udp-timeout", o.config.Conntrack.UDPTimeout.Duration, "Idle timeout for UNREPLIED UDP connections (0 to leave as-is)") fs.DurationVar(&o.config.Conntrack.UDPStreamTimeout.Duration, "conntrack-udp-timeout-stream", o.config.Conntrack.UDPStreamTimeout.Duration, "Idle timeout for ASSURED UDP connections (0 to leave as-is)")

Related discussion:
text: What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.