Change:
What happened: Scheduler decided that it wants to schedule Pod P on Node N after preempting some pods from N. It set the P.nominatedNodeName to N. After a while (before scheduling actually happened) node N was removed from the cluster. Scheduler did not reset P.nominatedNodeName to empty value. What you expected to happen: I would expect scheduler to unset P.nominatedNodeName when the node is no longer valid. The trivial reason for node not being valid any more is it is removed from the cluster. But there are more cases to cover. Out of the top of my head I may think of: extra Pod was scheduled to node and there is not enough resources any more either scheduled by scheduler or directly as static pod/daemonset node became unschedulable How to reproduce it (as minimally and precisely as possible): We did not try to reproduce issue manually. Yet following setup should work. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources. add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster Anything else we need to know?: Current behavior does not play well with Cluster Autoscaler which assumes that Pod with nominatedNodeName will be scheduled in a while. Therefore such pods do not trigger scale-up. If we can have P.nominatedNodeName set incorrectly for unbounded period of time CA will never provision node for P to run. Environment: Kubernetes version (use ): We observed the behavior during presubmit tests which run k8s built from HEAD of master on 27.11.2019. Versions logged in test log: I1125 18:55:11.691] Client Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} I1125 18:55:11.692] Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE /sig scheduling
/assign
/cc
Another interesting case would be creating a new pod with pod antiaffinity blocking the pod with nominated node name from scheduling: up a cluster with pods, including a long-terminating low-priority pod A. high-priority pod B that has the same requests as A. pod C with negligible resource requests and zonal topology pod antiaffinity matching pod B (and no other pods in the cluster). What I'd expect: Pod C keeps pending, nominatedNodeName is cleared. Pod A is restarted on the same node it was running before. I haven't actually tested this scenario, so I'm not sure what will actually happen. If report is correct, this may be a more convenient way of reproducing the issue (no need to add / remove nodes, it's not important which pod will go to which node).
I'm not sure if we need to clear nominatedNodeName on Pod when node is no longer valid immediately. And we have other logic to clear nominatedNodeName. Reuse your example. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources. add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster After this, The H will be scheduled again. Obviously it's going to fail. H will enter the preemption process and the is empty. It will return itself as preemption process will clean nominatedNodeName on the Pod H.
when I have some time I will check if reproduction steps I describe actually work. But we surely observed pods with lingering nominatedNodeName which pointed to inexistent node, during our E2E tests. Quick thought. The method has many return points. If it returned at the place you linked it seems should be cleared. But there are many return points below, which set the list of pods to be cleared to nil. Is it possible that for a pod to consistently exit from with one of those return points?
I think you are right. Actually there are many return points to return nil. I think maybe we need to clear the of the pod after the filter failed again and before the new preemption process to avoid potential risks. WDYT?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I believe it's a bug as in old logic, e.g., v1.18.3, if the Pod (with nominatedNodeName set) cannot be scheduled and also preemption doesn't help ( nil potentialNodes is a corner case, it's more common the pod can't fit on all potentialNodes), it won't hit L413: then in old recordingFailure logic, nominatedNodeName remains as is. In the latest codebase, introduced a fix to combine setting nominatedNodeName as well update the pod's unschedulable condition. However, this doesn't help in this issue either as it ignores the case when the pod's nominatedNodeName is nil:
/reopen as regresses - we need to come up with a complete fix.
Reopened this issue. <detailsIn response to : /reopen as regresses - we need to come up with a complete fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
what are the implications of not clearing the nominated node name?
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
podCopy := pod.DeepCopy() // NominatedNodeName is updated only if we are trying to set it, and the value is // different from the existing one. <del> if !podutil.UpdatePodCondition(&podCopy.Status, condition) && (len(nominatedNode) == 0 || pod.Status.NominatedNodeName == nominatedNode) { </del> <ins> if !podutil.UpdatePodCondition(&podCopy.Status, condition) && pod.Status.NominatedNodeName == nominatedNode { </ins> return nil } <del> if nominatedNode != "" { podCopy.Status.NominatedNodeName = nominatedNode } </del> <ins> podCopy.Status.NominatedNodeName = nominatedNode </ins> return patchPod(client, pod, podCopy) }

expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","message":"newMessage","reason":"newReason","type":"currentType"}]}}`, }, { <del> name: "Should not make patch request if pod condition already exists and is identical and nominated node name is not set", </del> <ins> name: "Should make patch request if pod condition already exists and is identical but nominated node name is different", </ins> currentPodConditions: []v1.PodCondition{ { Type:               "currentType",

Message:            "currentMessage", }, currentNominatedNodeName: "node1", <del> expectedPatchRequests:    0, </del> <ins> expectedPatchRequests:    1, </ins> }, { name: "Should make patch request if pod condition already exists and is identical but nominated node name is set and different",

v1 "k8s.io/api/core/v1" policy "k8s.io/api/policy/v1beta1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

} } <ins> func TestNominatedNodeCleanUpUponNodeDeletion(t *testing.T) { // Initialize scheduler. testCtx := initTest(t, "preemption") defer testutils.CleanupTest(t, testCtx) cs := testCtx.ClientSet defer cleanupPodsInNamespace(cs, t, testCtx.NS.Name) // Create a node with some resources and a label. nodeRes := &v1.ResourceList{ v1.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), v1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(100, resource.DecimalSI), } nodeNames := []string{"node1", "node2"} for _, nodeName := range nodeNames { _, err := createNode(testCtx.ClientSet, nodeName, nodeRes) if err != nil { t.Fatalf("Error creating nodes: %v", err) } } // Fill the cluster with one high-priority and one low-priority Pod. highPriPod, err := createPausePod(cs, mkPriorityPodWithGrace(testCtx, "high-pod", highPriority, 60)) if err != nil { t.Fatalf("Error creating high-priority pod: %v", err) } // Make sure the pod is scheduled. if err := testutils.WaitForPodToSchedule(cs, highPriPod); err != nil { t.Fatalf("Pod %v/%v didn't get scheduled: %v", highPriPod.Namespace, highPriPod.Name, err) } lowPriPod, err := createPausePod(cs, mkPriorityPodWithGrace(testCtx, "low-pod", lowPriority, 60)) if err != nil { t.Fatalf("Error creating low-priority pod: %v", err) } // Make sure the pod is scheduled. if err := testutils.WaitForPodToSchedule(cs, lowPriPod); err != nil { t.Fatalf("Pod %v/%v didn't get scheduled: %v", lowPriPod.Namespace, lowPriPod.Name, err) } // Create a medium-priority Pod. medPriPod, err := createPausePod(cs, mkPriorityPodWithGrace(testCtx, "med-pod", mediumPriority, 60)) if err != nil { t.Fatalf("Error creating med-priority pod: %v", err) } // Check its nominatedNodeName field is set properly. if err := waitForNominatedNodeName(cs, medPriPod); err != nil { t.Fatalf("NominatedNodeName was not set for pod %v/%v: %v", medPriPod.Namespace, medPriPod.Name, err) } // Get the live version of low and med pods. lowPriPod, _ = getPod(cs, lowPriPod.Name, lowPriPod.Namespace) medPriPod, _ = getPod(cs, medPriPod.Name, medPriPod.Namespace) want, got := lowPriPod.Spec.NodeName, medPriPod.Status.NominatedNodeName if want != got { t.Fatalf("Expect med-priority's nominatedNodeName to be %v, but got %v.", want, got) } // Delete the node where med-priority pod is nominated to. cs.CoreV1().Nodes().Delete(context.TODO(), got, metav1.DeleteOptions{}) if err := wait.Poll(200*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { _, err := cs.CoreV1().Nodes().Get(context.TODO(), got, metav1.GetOptions{}) if apierrors.IsNotFound(err) { return true, nil } return false, err }); err != nil { t.Fatalf("Node %v cannot be deleted: %v.", got, err) } // Finally verify if med-priority pod's nominatedNodeName gets cleared. if err := wait.Poll(200*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { pod, err := cs.CoreV1().Pods(medPriPod.Namespace).Get(context.TODO(), medPriPod.Name, metav1.GetOptions{}) if err != nil { t.Errorf("Error getting the medium priority pod info: %v", err) } if len(pod.Status.NominatedNodeName) == 0 { return true, nil } return false, err }); err != nil { t.Errorf("The nominated node name of the medium priority pod was not cleared: %v", err) } } </ins> func mkMinAvailablePDB(name, namespace string, uid types.UID, minAvailable int, matchLabels map[string]string) *policy.PodDisruptionBudget { intMinAvailable := intstr.FromInt(minAvailable) return &policy.PodDisruptionBudget{

// cleanupPodsInNamespace deletes the pods in the given namespace and waits for them to // be actually deleted. func cleanupPodsInNamespace(cs clientset.Interface, t *testing.T, ns string) { <del> if err := cs.CoreV1().Pods(ns).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{}); err != nil { </del> <ins> if err := cs.CoreV1().Pods(ns).DeleteCollection(context.TODO(), *metav1.NewDeleteOptions(0), metav1.ListOptions{}); err != nil { </ins> t.Errorf("error while listing pod in namespace %v: %v", ns, err) return }