Change:
Since v1.27.0, the for appear to have extra validation on whether the values exist, despite using the operator. This works as expected on 1.26.14, and no longer works on 1.27.0 onwards (tested up to 1.29.2). I am unsure whether this is an intentional breaking change in 1.27.0 or something wrong in our manifests and we've just so happened to get away with it for a few years. The only thing that stands out to me in the changelog for 1.27.0 is some changes to the schedulers plugin but this may well be unrelated. When using the operator, I'd expect the expression to only apply to nodes where the match is truey. Note, using Kind for easy reproduction Since exists in the hostname values of this cluster, the pod schedules as expected. node does not exist in this environment, and so is ignored. Same expectations as 1.26.14, but in this case, the pod stays stuck in pending state due to the following; I have tested this all the way up to 1.29.2. No response v1.27.0 through to v1.29.2 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
In a similar vein, we have noticed the same issue in K8s v1.27 when the label doesn't match the node name. We see , despite (I think) the nodeinfo util supposed to be called using a node name.
/sig scheduling
This is because the non-existent node is obtained from the cache, which has been fixed by before , get node will return error: after it's fixed since version 1.30, it can run normally PTAL, need to cherry-pick for the previous version?
Looks like my PR accidentally fixed this issue. is the trigger of this issue, introduced in v1.27; volumebinding plugin just returns node names in PreFilterResult based on nodeAffinity in PV, and thus PreFilterResult could contain node names actually doesn't exist. One Line Summary: If nodeaffinity in PV has non-existing node name in with , the scheduling always fails. And, this bug is already coincidentally fixed by . Questions: In the first place, is it a bug worth cherry-picking? If Yes, should we cherry-pick , which actually isn't the PR motivated for this issue? Or should we go another way
If it's a regression, we should consider cherry-pick. looks big, but the actual code changes are about 10 lines (the rest are comments and tests), so I think it would be ok to cherry-pick. Can you explain why your PR fixes the problem?
Before my PR, we take NodeInfo of nodes in preFilterResult via (). Consequently, we could get not found errors from it when preFilterResult has non-existing node name(s). And, after my PR, we are not using anymore.
The pitfall is that when PreFilterResult was introduced, scheduler directly fetch the nodeNames returned by PreFilter() in its node snapshot, so technically speaking, when a plugin (in-tree or out-of-tree) return non-existent/illegal nodes, the pod scheduling flow will abort immediately. And the PR happened to stomp on this pitfall.
do you mean that you would like to see a smaller version of for cherry-pick? Note that it is only about 10 lines of actual change. The rest is comments and tests. Since this is already released code, I would suggest cherry-picking as-is. Also, worth testing E2E whether the issue reproduces in v1.30. do you think you can give it a try?
yup, i'm in favor of picking the minimum fix, along with a minimum test : b/c in addition to the "fix" part, actually introduced a slight behavior change. do you think the above ^^ minimum fix + makes sense?
can guard it.
Could you let me try it?picking the minimum fix and add e2e test
A simplified version of the sample is used for testing, also get this failedScheduling error, kubectl describe pod pause: !
That small fix should work. I don't have a strong opinion on whether we should have a minimum fix vs cherry-pick as it is. Defer to you both on that point.
Let's move ahead with the small fix, since we all agree on it.
/reopen Cherry-pick PR(s) aren't done
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI on a related crash
Are there any news on the backporting the solution to older versions (v1.27+)?
This is already backported and released. But the fix for the crash reported in was not released yet. Let's close this issue for now, as it is completed.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The issue here stems from that volume-binding's prefilter assumes that it can equate the matchConstraint hostname labels with node names. This doesn't have to be the case. See here. Prefiltering will always fail in this situation. AFAICT this issue persists with the current head of the master branch.

Discussion:
e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2eruntimeclass "k8s.io/kubernetes/test/e2e/framework/node/runtimeclass" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <ins> e2epv "k8s.io/kubernetes/test/e2e/framework/pv" </ins> e2erc "k8s.io/kubernetes/test/e2e/framework/rc" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutils "k8s.io/kubernetes/test/utils"

ginkgo.By("Expect all pods are scheduled and running") framework.ExpectNoError(e2epod.WaitForPodsRunning(ctx, cs, ns, replicas, time.Minute)) }) <ins> // Regression test for an extended scenario for https://issues.k8s.io/123465 ginkgo.It("when PVC has node-affinity to non-existent/illegal nodes, the pod should be scheduled normally if suitable nodes exist", func(ctx context.Context) { nodeName := GetNodeThatCanRunPod(ctx, f) nonExistentNodeName1 := string(uuid.NewUUID()) nonExistentNodeName2 := string(uuid.NewUUID()) hostLabel := "kubernetes.io/hostname" localPath := "/tmp" podName := "bind-pv-with-non-existent-nodes" pvcName := "pvc-" + string(uuid.NewUUID()) _, pvc, err := e2epv.CreatePVPVC(ctx, cs, f.Timeouts, e2epv.PersistentVolumeConfig{ PVSource: v1.PersistentVolumeSource{ Local: &v1.LocalVolumeSource{ Path: localPath, }, }, Prebind: &v1.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{Name: pvcName, Namespace: ns}, }, NodeAffinity: &v1.VolumeNodeAffinity{ Required: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      hostLabel, Operator: v1.NodeSelectorOpIn, // add non-existent nodes to the list Values: []string{nodeName, nonExistentNodeName1, nonExistentNodeName2}, }, }, }, }, }, }, }, e2epv.PersistentVolumeClaimConfig{ Name: pvcName, }, ns, true) framework.ExpectNoError(err) bindPvPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "pause", Image: imageutils.GetE2EImage(imageutils.Pause), VolumeMounts: []v1.VolumeMount{ { Name:      "data", MountPath: "/tmp", }, }, }, }, Volumes: []v1.Volume{ { Name: "data", VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: pvc.Name, }, }, }, }, }, } _, err = f.ClientSet.CoreV1().Pods(ns).Create(ctx, bindPvPod, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitForPodNotPending(ctx, f.ClientSet, ns, podName)) }) </ins> }) func patchPod(cs clientset.Interface, old, new *v1.Pod) (*v1.Pod, error) {