jiebi/CodeConvo huggingface dataset:
Code change:
st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

Related discussion:
text: If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.
Code change:
st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

Related discussion:
text: Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.
Code change:
st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

Related discussion:
text: Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.
Code change:
By("Confirming that stateful set scale up will halt with unhealthy stateful pod") sst.BreakProbe(ss, testProbe) sst.WaitForRunningAndNotReady(*ss.Spec.Replicas, ss) <ins> sst.WaitForStatus(ss, 0) </ins> sst.UpdateReplicas(ss, 3) sst.ConfirmStatefulPodCount(1, ss, 10*time.Second)

Related discussion:
text: https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM
Code change:
expected := `# HELP kube_pod_resource_limit [ALPHA] Resources limit for workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any. # TYPE kube_pod_resource_limit gauge kube_pod_resource_limit{namespace="test",node="node-one",pod="foo",priority="",resource="custom",scheduler="",unit=""} 6 <del> kube_pod_resource_limit{namespace="test",node="node-one",pod="foo",priority="",resource="memory",scheduler="",unit="bytes"} 2e+09 </del> <ins> kube_pod_resource_limit{namespace="test",node="node-one",pod="foo",priority="",resource="memory",scheduler="",unit="bytes"} 2.68435456e+09 </ins> # HELP kube_pod_resource_request [ALPHA] Resources requested by workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any. # TYPE kube_pod_resource_request gauge kube_pod_resource_request{namespace="test",node="node-one",pod="foo",priority="",resource="cpu",scheduler="",unit="cores"} 2

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted
Code change:
"client_config.go", "config.go", "doc.go", <ins> "flag.go", </ins> "helpers.go", "loader.go", "merged_client_builder.go",

Related discussion:
text: Because always return names with a prefix like , it cannot be directly used to iterate over all namespaces to execute other commands. I've compiled from source: Reviewing some scripts, I came across a usage of like this: That line fetches namespace names to later run in a loop: That seemed to be a perfect use case for the flag, but it can't be used without extra hacks to get rid of the prefix (FWIW, is invalid). Changing the current output of might not be possible or desirable, for this could impact existing users that rely on the output format. If I'm wrong about the above limitation, than it would make sense to output just the name when and there is a single resource type, while using the ideas introduced in to show the resource type only when there is more than one type in the output, or when is explicitly set. Before: After:
could you help me ping the appropriate team to triage this? Thanks!
Please re-post your question to . We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases. We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on github is making it difficult for us to use issues to identify real bugs. The Kubernetes team scans stackoverflow on a regular basis, and will try to ensure your questions don't go unanswered. Before posting a new question, please search stackoverflow for answers to similar questions, and also familiarize yourself with: Again, thanks for using Kubernetes. The Kubernetes Team
perhaps I didn't do a good job at explaining the problem, but this is not a question of how to do something, but a report on a problem with the CLI design.
Sorry about that :-/. I'll try to route to the proper team.
update: console examples to the description.
the -o format is designed for scripting, the use case is just different from yours. You should be able to paste the output of into . Does do what you want? If this is a common use case we could create another -o flag value.
Code change:
"client_config.go", "config.go", "doc.go", <ins> "flag.go", </ins> "helpers.go", "loader.go", "merged_client_builder.go",

Related discussion:
text: the use cases I've observed doing code reviews: over all namespaces to get certain resources from that namespace: over all namespaces, then iterate of over resource types that produce logs and fetch logs one by one: Above we can use for listing pods because later we'll pass it to that accept the syntax. We cannot do the same for namespaces, because doesn't like the format.
Alternatively we could be kubectl recognize as valid
I'll take a look on it if it's not taken by anyone yet. CC
Have you started this?
Only investigated due to being out of office since Thursday last week. I can work on it in upcoming week, but I'll understand if it's piority issue and you'll reassign it.
Ready :-)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
// Add entries to the map for both the current and persistent portals // pointing to the SCSI host for those connections <del> portal := strings.TrimSpace(string(addr)) + ":" + strings.TrimSpace(string(port)) </del> <ins> // JoinHostPort will add `[]` around IPv6 addresses. portal := net.JoinHostPort(strings.TrimSpace(string(addr)), strings.TrimSpace(string(port))) </ins> portalHostMap[portal] = hostNumber <del> persistentPortal := strings.TrimSpace(string(persistentAddr)) + ":" + strings.TrimSpace(string(persistentPort)) </del> <ins> persistentPortal := net.JoinHostPort(strings.TrimSpace(string(persistentAddr)), strings.TrimSpace(string(persistentPort))) </ins> portalHostMap[persistentPortal] = hostNumber } }

Related discussion:
text: Kubelet was not able to attach + mount an iSCSI PV when it has IPv6 server address. Volume is mounted + attached. Set up iSCSI target that's reachable over IPv6 and create a PV for it. For example: Kubelet logs: Notice around the IP address. Various iSCSI files in /sys and /dev/disk/by-path do not have these square brackets and kubelet is then not able to find the iSCSI host in /sys. That's where comes from. No response <detailsA recent master: console NAME="Fedora Linux" VERSION="36 (Workstation Edition)" </details<detailshack/local-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon
/assign
Code change:
var err error if useRealProxier { <ins> nodeIP := utilnode.GetNodeIP(client, nodeName) if nodeIP == nil { klog.V(0).Infof("can't determine this node's IP, assuming 127.0.0.1") nodeIP = net.ParseIP("127.0.0.1") } </ins> // Real proxier with fake iptables, sysctl, etc underneath it. //var err error proxier, err = iptables.NewProxier(

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.
Code change:
var err error if useRealProxier { <ins> nodeIP := utilnode.GetNodeIP(client, nodeName) if nodeIP == nil { klog.V(0).Infof("can't determine this node's IP, assuming 127.0.0.1") nodeIP = net.ParseIP("127.0.0.1") } </ins> // Real proxier with fake iptables, sysctl, etc underneath it. //var err error proxier, err = iptables.NewProxier(

Related discussion:
text: ): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL
Code change:
containerDetails map[string]*docker.Container expectedRemoved  []string }{ <ins> // Remove oldest containers. </ins> { containers: []docker.APIContainers{ {

Related discussion:
text: Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)
Code change:
job := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ <del> Labels:          labels, Annotations:     annotations, Name:            name, OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </del> <ins> Labels:            labels, Annotations:       annotations, Name:              name, CreationTimestamp: metav1.Time{Time: scheduledTime}, OwnerReferences:   []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </ins> }, } cj.Spec.JobTemplate.Spec.DeepCopyInto(&job.Spec)

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : Cronjob should include the last time when a job was successfully completed. Why is this needed: Currently, a Cronjob's only includes the last time when a job was scheduled. This makes monitoring cronjobs much more complicated, because instead of simply comparing to the current timestamp, the monitoring system now has to hunt for the last job related to the cronjob and persist the state somewhere.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd be interesting in picking this up if the apps team thinks this is a good ticket - it seems like it's fairly beginner friendly.
Hi how is this going on? I could take it over if you don't have the time for this issue. :)
Yes please do! I meant to update that I actually wasn't able to pick this up, but it slipped my mind
/assign
Hi do you have any thoughts on this feature?
/unassign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Any updates?
Reiterating the mentions to trigger a notification: <detailsIn response to : Any updates? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Trying to understand the usage of lastSuccessfulTime. In a normally working case, lastSuccessfulTime lastScheduledTime =last job completed lastSuccessfulTime < lastScheduledTime =last job completed, but a newer one is running Is that the expectation ?
We wanted to use it in monitoring (e.g. fire an alert if is more than X ago). could also mean that the job has been failing for some time, so I don't know what would you get out of that. I guess you could also use as a proxy for "job has failed at least once"?