Change:
What happened: When a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage, the scheduler doesn't consider that pod for preemption even there are pods can be preempted on another node. What you expected to happen: When the pod's nominated node is considered as 'UnschedulableAndUnresolvable ' (For example, the taint being by those conditions, ), the scheduler should try to preemption on other nodes. How to reproduce it (as minimally and precisely as possible): Create low priority pod, which schedule to nodeA Create mid priority pod, which schedule to nodeB Create high priority pod, which preempt the low priority pod on nodeA, and set high priority pod's nominated node as nodeA The nodeA's kubelet is down, the node life cycle controller taint the nodeA with '' (or taint with disk ) before low priority pod finishes the termination. The high priority pod is keeping as pending status. The scheduler doesn't try to preempt the mid-priority pod on nodeB Anything else we need to know?: Environment: Kubernetes version (use ): 18.0 Cloud provider or hardware configuration: baremetal OS (e.g: ): ubuntu1804 Kernel (e.g. ): 0-46-generic Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
/assign
This seems to be a duplicate of .
actually, I found this case after review . I file this issue since I feel it should be another case. The want to cleanup the nomnated node for cluster autoscaler. This is case need to allow the pod to preempt on other nodes again even it has a nomanted node. But I may understand incorrectly, please mark it duplicated if I'm wrong. Thanks for help anyway. :)
If the nominatedNodeName can be cleared properly in any case, this issue will no longer exist. From this point of view, I'd say this issue is a sub-case of .
BTW: in your description, if "The nodeA is down", how it comes out "a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage"? nodeA should NOT have existed in the result of Filter(), right? (if the nodeA is just being shutdown, we still treat it as a normal node)
If the NodeA is down, the node life cycle controller will add taint '' to the node, then the TaintToleration filter will return 'UnschedulableAndUnresolvable '. It also means that it isn't only about nodeA is down, there are other conditions in this ' to return 'UnschedulableAndUnresolvable'. Beyond the TaintToleration filter, other filters may return 'UnschedulableAndUnresolvable' also, as my understanding, that means that node won't be available for that pod anymore, then we should try other nodes for preemption.
then we should better describe it as "being applied with a Taint", which is more general. When you say "node is down", it usually refers to the node is deleted physically.
I got it, thanks again! let me correct it
Also it's different from . Let's not mention "node is down" in the description to avoid confusion.
thanks, done :)

Discussion:
return "", err } <del> if !podEligibleToPreemptOthers(pod, fh.SnapshotSharedLister().NodeInfos()) { </del> <ins> if !podEligibleToPreemptOthers(pod, fh.SnapshotSharedLister().NodeInfos(), m[pod.Status.NominatedNodeName]) { </ins> klog.V(5).Infof("Pod %v/%v is not eligible for more preemption.", pod.Namespace, pod.Name) return "", nil }

// considered for preemption. // We look at the node that is nominated for this pod and as long as there are // terminating pods on the node, we don't consider this for preempting more pods. <del> func podEligibleToPreemptOthers(pod *v1.Pod, nodeInfos framework.NodeInfoLister) bool { </del> <ins> func podEligibleToPreemptOthers(pod *v1.Pod, nodeInfos framework.NodeInfoLister, nominatedNodeStatus *framework.Status) bool { </ins> if pod.Spec.PreemptionPolicy != nil && *pod.Spec.PreemptionPolicy == v1.PreemptNever { klog.V(5).Infof("Pod %v/%v is not eligible for preemption because it has a preemptionPolicy of %v", pod.Namespace, pod.Name, v1.PreemptNever) return false } nomNodeName := pod.Status.NominatedNodeName if len(nomNodeName) > 0 { <ins> // If the pod's nominated node is considered as UnschedulableAndUnresolvable by the filters, // then the pod should be considered for preempting again. if nominatedNodeStatus.Code() == framework.UnschedulableAndUnresolvable { return true } </ins> if nodeInfo, _ := nodeInfos.Get(nomNodeName); nodeInfo != nil { podPriority := podutil.GetPodPriority(pod) for _, p := range nodeInfo.Pods {

} } <ins> func TestPodEligibleToPreemptOthers(t *testing.T) { tests := []struct { name                string pod                 *v1.Pod pods                []*v1.Pod nodes               []string nominatedNodeStatus *framework.Status expected            bool }{ { name:                "Pod with nominated node", pod:                 st.MakePod().Name("p_with_nominated_node").UID("p").Priority(highPriority).NominatedNodeName("node1").Obj(), pods:                []*v1.Pod{st.MakePod().Name("p1").UID("p1").Priority(lowPriority).Node("node1").Terminating().Obj()}, nodes:               []string{"node1"}, nominatedNodeStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, tainttoleration.ErrReasonNotMatch), expected:            true, }, { name:                "Pod with nominated node, but without nominated node status", pod:                 st.MakePod().Name("p_without_status").UID("p").Priority(highPriority).NominatedNodeName("node1").Obj(), pods:                []*v1.Pod{st.MakePod().Name("p1").UID("p1").Priority(lowPriority).Node("node1").Terminating().Obj()}, nodes:               []string{"node1"}, nominatedNodeStatus: nil, expected:            false, }, { name:                "Pod without nominated node", pod:                 st.MakePod().Name("p_without_nominated_node").UID("p").Priority(highPriority).Obj(), pods:                []*v1.Pod{}, nodes:               []string{}, nominatedNodeStatus: nil, expected:            true, }, { name:                "Pod with 'PreemptNever' preemption policy", pod:                 st.MakePod().Name("p_with_preempt_never_policy").UID("p").Priority(highPriority).PreemptionPolicy(v1.PreemptNever).Obj(), pods:                []*v1.Pod{}, nodes:               []string{}, nominatedNodeStatus: nil, expected:            false, }, } for _, test := range tests { var nodes []*v1.Node for _, n := range test.nodes { nodes = append(nodes, st.MakeNode().Name(n).Obj()) } snapshot := internalcache.NewSnapshot(test.pods, nodes) if got := podEligibleToPreemptOthers(test.pod, snapshot.NodeInfos(), test.nominatedNodeStatus); got != test.expected { t.Errorf("expected %t, got %t for pod: %s", test.expected, got, test.pod.Name) } } } </ins> func TestNodesWherePreemptionMightHelp(t *testing.T) { // Prepare 4 nodes names. nodeNames := []string{"node1", "node2", "node3", "node4"}

return p } <ins> // NominatedNodeName sets `n` as the .Status.NominatedNodeName of the inner pod. func (p *PodWrapper) NominatedNodeName(n string) *PodWrapper { p.Status.NominatedNodeName = n return p } </ins> // PodAffinityKind represents different kinds of PodAffinity. type PodAffinityKind int