Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state? If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases. Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same. Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.

Discussion:
nodeInfoCopy := potentialNodes[(int(offset)+i)%len(potentialNodes)].Clone() stateCopy := state.Clone() pods, numPDBViolations, status := selectVictimsOnNode(ctx, fh, stateCopy, pod, nodeInfoCopy, pdbs) <del> if status.IsSuccess() { </del> <ins> if status.IsSuccess() && len(pods) != 0 { </ins> victims := extenderv1.Victims{ Pods:             pods, NumPDBViolations: int64(numPDBViolations),

if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

} return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {