Change:
On Kubernetes v1.30.0 (and v1.30.1), can crash with the following panic if a pod is defined in a certain way: The crash happens because the is in certain cases. should not crash, on v1.29.4 it doesn't happen. On v1.29.4 the is just printing these error logs, but doesn't crash: Create a pod like this: The important part is that the affinity doesn't match a real/valid node. Once this pod is being processed by it will crash and continue to do so until the pod is deleted. This issue was triggered during the rotation of control-plane nodes. In our setup we update the control plane by scaling from 1 to 2 instances. Once both a ready, the old is being terminated via EC2 API. At this stage the kube-controller-manager sometimes_ manages to create a replacement daemonset pod once the old node is being deleted. This results in a pod targeting a no longer existing node via as illustrated in the example above. For some reason, when the is crashing the doesn't delete the extra/invalid daemonset pod. Not sure if this is another issue or it has always happened in our setup, but only v1.30 makes crash which causes an actual issue. <details</details<detailsWe run a custom Kubernetes setup on AWS EC2 (not EKS). </details<details</details<detailsCustom tooling: </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc /priority important-soon
can you take a look? I believe all the latest patch releases are affected too, because of
cc
/priority critical-urgent
This crash happens when preFilter plugins filtered out all nodes ... If preFilter filtered out some nodes, here len(nodes) will be a subset of allNodes. Why are we using this subset at first? I think should works fine?(seems logical to me)
I think so. In general, we just want to try a different set of nodes. For the case of Daemonsets, it doesn't really matter, as we will just test one node.
/assign Successfully catched this with a unit test. I'll file a small pr with fix and a newly UT.
I am having the same issue with 1.28.10 on a clean new cluster created with kops 1.28.5: The pod that it's trying to create is pending with the following messages: (just for the sake of being indexed for those who will search for the same error to find this github issue)
Fix is on the way: Just waiting for reviews from approver. After this pr got merged, I'll do cherry-picks for v1.28(1.27?)~1.30
cherry-picks:
Thanks for fixing this so fast. How do I pull this new code? I see that your PR was merged, but I don't see a new release to pull? looking for the 1.30
This change will be included in the upcoming patch releases, scheduled for 2024-06-11. You can check out the for more details.
I'm seeing the issue after upgrade v1.26.11-v1.27.14 Should I search for a broken pod? Or what else can be causing this?
Yes, a broken pod could be causing this. You can also try downgrading to v1.27.13
Phew, manually changing the scheduler version to 1.27.13 fixed the issue and showed the broken pod in logs. Thanks!

Discussion:
// always try to update the sched.nextStartNodeIndex regardless of whether an error has occurred // this is helpful to make sure that all the nodes have a chance to be searched processedNodes := len(feasibleNodes) + len(diagnosis.NodeToStatusMap) <del> sched.nextStartNodeIndex = (sched.nextStartNodeIndex + processedNodes) % len(nodes) </del> <ins> sched.nextStartNodeIndex = (sched.nextStartNodeIndex + processedNodes) % len(allNodes) </ins> if err != nil { return nil, diagnosis, err }

// since this case has no score plugin, we'll only try to find one node in Filter stage wantEvaluatedNodes: ptr.To[int32](1), }, <ins> { name: "test prefilter plugin returned an invalid node", registerPlugins: []tf.RegisterPluginFunc{ tf.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), tf.RegisterPreFilterPlugin( "FakePreFilter", tf.NewFakePreFilterPlugin("FakePreFilter", &framework.PreFilterResult{ NodeNames: sets.New("invalid-node"), }, nil), ), tf.RegisterFilterPlugin("TrueFilter", tf.NewTrueFilterPlugin), tf.RegisterBindPlugin(defaultbinder.Name, defaultbinder.New), }, nodes:     []string{"1", "2"}, pod:       st.MakePod().Name("test-prefilter").UID("test-prefilter").Obj(), wantNodes: nil, wErr: &framework.FitError{ Pod:         st.MakePod().Name("test-prefilter").UID("test-prefilter").Obj(), NumAllNodes: 2, Diagnosis: framework.Diagnosis{ NodeToStatusMap: framework.NodeToStatusMap{ "1": framework.NewStatus(framework.UnschedulableAndUnresolvable, "node is filtered out by the prefilter result"), "2": framework.NewStatus(framework.UnschedulableAndUnresolvable, "node is filtered out by the prefilter result"), }, }, }, }, </ins> } for _, test := range tests { t.Run(test.name, func(t *testing.T) {

}), preemptedPodIndexes: map[int]struct{}{2: {}}, }, <ins> { name:         "The pod cannot be scheduled when nodeAffinity specifies a non-existent node.", nodes:        []*nodeConfig{{name: "node-1", res: defaultNodeRes}}, existingPods: []*v1.Pod{}, pod: testutils.InitPausePod(&testutils.PausePodConfig{ Name:      "unschedulable-pod", Namespace: testCtx.NS.Name, Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"invalid-node"}, }, }, }, }, }, }, }, Resources: defaultPodRes, }), }, </ins> } for _, test := range tests {