jiebi/CodeConvo huggingface dataset:
changed the default behavior of scheduler framework's - from stopping at the first failed filter to always running all filters. This change breaks Cluster Autoscaler at scale. Cluster Autoscaler imports the scheduler framework to simulate filters on artificial nodes. In a (pretty common) one-pod-per-node scenario, the number of calls is ~quadratic wrt. the number of nodes needed for unschedulable pods - it can be a lot of calls. Before , in such scenario the vast majority of calls would stop at failing Fit or NodeAffinity filters, and not run the rest of them (e.g. PV-related filters). After the change, every call to runs all filters, which completely breaks Cluster Autoscaler at scale (an example scenario of 300 1-pod-per-node pods and 200 node groups takes 10min in 1.24, and ~50s in <1.22). Can we get the old behavior back? Or re-introduce the config option so that Cluster Autoscaler can set it accordingly? Ideally, we'd also like to get this backported as far as possible (the change was introduced in 1.24).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
This seems to be a regression. I would be interested in adding a toggle flag that allows one to toggle on that behavior if this was done on purpose.
That PR wasn't supposed to change the behavior actually. Scheduler is always defaulted to stop upon the first Filter failure, and it's just we offered an option to continue go through other Filter plugins upon failures, and that option was removed recently. Let me check the code to see if there is a regression.
When removed the following changes: It should have kept . I will submit a fix along with test, and backport to 1.24, 1.25, 1.26. Thanks for reporting!
cc /assign /kind bug /kind regression /priority critical-urgent /triage/accepted
Thanks Kuba, this is indeed a regression on the scheduler end.
/open for patches
/re-open /reopen
Reopened this issue. <detailsIn response to : /re-open /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the reporting, and thanks for the fix.
Really sorry about this regression, and hope not to happen again. I think there's no e2e tests between kubernetes scheduler and cluster-autoscaler, so can we also have testcases of pods unschedulable in scheduling perf-test, maybe this can also help to prevent this happening again. UT benchmark test also makes sense, but it's not graphical.
Thanks for the quick fix! Unfortunately I'm afraid this kind of issue would only be caught in a large-scale test scenario, which is not cheap to run continuously.
For this particular issue, UT ( in the PR) is able to catch it. And hence doesn't need to be that complicated to test with cluster-autoscaler. However, standing from a broader angle, why we didn't catch the regression in perf test is b/c the triggering actor is unschedulable Pods, instead of schedulable Pods (which are the happy path most perf tests covered). So going forward, I'd like to see more perf tests to mixin unschedulable Pods, and hence more likely to surface this kind of issue. (we have preemptio preemption test suite but doesn't seem adequate)
This is not for the particular issue, but keeping an eye on whether there's any performance degradation between two kubernetes components, like we do for scheduling performance. but it's actually out of the scope of scheduler. I'll add it as one of the goals of
Actually, the schedulerperf has a test case . And, the schedulerperf collects the metric (), which has the label in it (). But, the problem is currently we don't distinguish the metric values by label on each data; we just collect/aggregate it regardless the label.
(We can distinguish the metric values by any labels in the schedulerperf as we do in the metric ()) I'd say just distinguishing the happy path and unhappy path with label in metric should be enough to observe the particular performance degradation in the unhappy path.
could you help share the perf-dash link where this metric () is exposed?
Hm, cannot see any scheduler metrics result from the current perf-dash I see some configuration that seems to be to parse the schedulerperf result file though. https://perf- TBH, I haven't visited the perf-dash so much, so not sure whether it's not working for a long time, or got broken recently. Also, another thing, I noticed the perf-dash doesn't show all test cases. All test cases after haven't been executed.
So... probably, something wrong happens in  job (probably in the test case) and the schedulerperf is failing. The schedulerperf doesn't make the result file when fails, so I'm guessing that's why we cannot see any scheduler metrics results from the perf-dash. Okay, I found it. Let me open another issue for this.
cc is this actually a regression, and is it resolved yet?
It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. /close
Closing this issue. <detailsIn response to : It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved /assign
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent
Kubernetes e2e suite [It] [sig-network] Firewall rule should have correct firewall rules for e2e cluster See also a similar cluster with the same test failing: /kind failing-test /kind flake /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes
e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
Are you the correct assignee for this? If so, can you please give an update. If not, can you please reassign it to the correct person. CC
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
Yes, I guess so. I'm looking into it.
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
We need to get this fixed soon and picked up in release-1.5 since it is consistently red in
ack i have a PR currently running
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
If you found the root cause, we will close my pending PR that printing out some debug info. What is the PR # of the commit you mentioned above?
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
: here it is
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
We timeout waiting for pod sometimes. Is it reasonable to use a longer timeout? Use for in that test.
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
: sure thing, do you want to do that in a fresh PR?
I updated my pending PR .
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
master-blocking: ci-kubernetes-unit It starts flaking today 30-05-2024 The test fail is This failure may be caused by the error message's order sensitivity. The error message's order of Resource Classes versus is different. N/A /sig node /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig testing /sig scheduling (sig testing doesn't own all of the unit tests, just test frameworks, tooling, CI infra, etc) This appears to be a flake in scheduler unit tests which would be sig scheduling ( )
.. and it appears to be DRA specific subset of scheduling so /sig node as well
FYI - looks like this is happening every 3-6 hours give or take. Yesterday it was every six hours and today it seems to be every 3-4 hours. Looking into it a bit more, it looks like it may just be an issue with a connection via the test while it's trying to hit the server. I don't think the test itself is failing, I think the connection to the server is failing. ! CC
Hey All - hope you're well. Michael here from the Release Signal team. Just wanted to touch base and make sure we're good to go for the cut on the 4th. CC /sig-node
Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.
Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In order to migrate to a new version of kubernetes on GKE. I cordoned(Kubectl cordon <node) all nodes in the old node pool(1.8.9) in order to drain the node pool(pods on the node) with an intention to cause the pods to run on a new version of the node pool(1.10.2). As soon as I cordoned all the nodes in the old node pool, it caused the downtime of ingress(nginx). The nginx controller pods where running fine on the cordoned node pool. But the cordon caused the nodes to be removed from the target pool of ingress load balancer. Below are the documents that suggest the above same method that I followed as best practice for zero downtime upgrade. 1). 2). What you expected to happen: I expected "kubectl cordon <node" not to have the node/s removed from the target pool of the loadbalancer because the nginx controller pods where running absolutely fine on these nodes. How to reproduce it (as minimally and precisely as possible): 1).Cordon all the nodes in an old node pool that runs nginx controller. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container OS Kernel (e.g. ): 4.14.22+ Install tools: Others: Cordon causes to remove the nodes from the target pool of the ingress load balancer. The health checks fail on the nodes of the target pool, which should not be the case. Because the nginx controller pods run fine on these nodes.
Technically, this isn't a Kubernetes bug as this is working as intended; however, it does beg the question why we remove unschedulable nodes if they're marked as ready since we perform the to determine LB backend nodes. is it scheduable? is it not a master? is it not marked for exclusion? is it marked as ready?