Repository: kubernetes/autoscaler
Pull request #8097: Prevent nil dereference of preFilterStatus
Author: login: norbertcyran
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: area/cluster-autoscaler
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: kind/bug
description: Categorizes issue or PR as related to a bug.
name: ok-to-test
description: Indicates a non-member PR verified by an org member that is safe to test.
name: kind/regression
description: Categorizes issue or PR as related to a regression from a prior release.
Created: 2025-05-06T09:45:12Z | Merged: 2025-05-07T10:17:16Z

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->
/kind bug
/kind regression

#### What this PR does / why we need it:
When a pod has node affinity set for a non-existing node, `RunFiltersOnNode` causes nil dereference panic, as it tries to dereference `preFilterStatus`, which will be nil in that situation. With this change, `preFilterStatus` will not be dereferenced in the affected branch, and instead, we will pass to the error message the names of plugins that filtered out some nodes.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
b/415163100

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
NONE
```




Diff:
diff --git a/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go b/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go
index e38e63bcbb21..60d7f1468cbf 100644
--- a/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go
+++ b/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go
@@ -111,12 +111,14 @@ func (p *SchedulerPluginRunner) RunFiltersOnNode(pod *apiv1.Pod, nodeName string
 
 	state := schedulerframework.NewCycleState()
 	// Run the PreFilter phase of the framework for the Pod and check the results. See the corresponding comments in RunFiltersUntilPassingNode() for more info.
-	preFilterResult, preFilterStatus, _ := p.fwHandle.Framework.RunPreFilterPlugins(context.TODO(), state, pod)
+	preFilterResult, preFilterStatus, nodeFilteringPlugins := p.fwHandle.Framework.RunPreFilterPlugins(context.TODO(), state, pod)
 	if !preFilterStatus.IsSuccess() {
+		// nil check on preFilterStatus not required, as IsSuccess returns true for nil
 		return nil, nil, clustersnapshot.NewFailingPredicateError(pod, preFilterStatus.Plugin(), preFilterStatus.Reasons(), "PreFilter failed", "")
 	}
 	if !preFilterResult.AllNodes() && !preFilterResult.NodeNames.Has(nodeInfo.Node().Name) {
-		return nil, nil, clustersnapshot.NewFailingPredicateError(pod, preFilterStatus.Plugin(), preFilterStatus.Reasons(), "PreFilter filtered the Node out", "")
+		// in this scope, preFilterStatus is most likely nil
+		return nil, nil, clustersnapshot.NewFailingPredicateError(pod, strings.Join(nodeFilteringPlugins.UnsortedList(), ", "), nil, "PreFilter filtered the Node out", "")
 	}
 
 	// Run the Filter phase of the framework for the Pod and the Node and check the results. See the corresponding comments in RunFiltersUntilPassingNode() for more info.
diff --git a/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner_test.go b/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner_test.go
index abebaa3aa7dc..b4281ca915c7 100644
--- a/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner_test.go
+++ b/cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner_test.go
@@ -44,6 +44,8 @@ func TestRunFiltersOnNode(t *testing.T) {
 	p600 := BuildTestPod("p600", 600, 500000)
 	p8000 := BuildTestPod("p8000", 8000, 0)
 	p500 := BuildTestPod("p500", 500, 500000)
+	podWithAffinity := BuildTestPod("pod_with_affinity", 500, 500, WithNodeNamesAffinity("n1000"))
+	podWithInvalidAffinity := BuildTestPod("pod_with_affinity", 500, 500, WithNodeNamesAffinity("non-existing-node"))
 
 	n1000 := BuildTestNode("n1000", 1000, 2000000)
 	SetNodeReadyState(n1000, true, time.Time{})
@@ -67,20 +69,26 @@ func TestRunFiltersOnNode(t *testing.T) {
 	assert.NoError(t, err)
 
 	tests := []struct {
-		name          string
-		customConfig  *config.KubeSchedulerConfiguration
-		node          *apiv1.Node
-		scheduledPods []*apiv1.Pod
-		testPod       *apiv1.Pod
-		expectError   bool
+		name                        string
+		customConfig                *config.KubeSchedulerConfiguration
+		node                        *apiv1.Node
+		scheduledPods               []*apiv1.Pod
+		testPod                     *apiv1.Pod
+		expectError                 bool
+		wantFailingPredicateName    string
+		wantFailingPredicateReasons []string
+		wantErrorSubstrings         []string
 	}{
 		// default predicate checker test cases
 		{
-			name:          "default - other pod - insuficient cpu",
-			node:          n1000,
-			scheduledPods: []*apiv1.Pod{p450},
-			testPod:       p600,
-			expectError:   true,
+			name:                        "default - other pod - insuficient cpu",
+			node:                        n1000,
+			scheduledPods:               []*apiv1.Pod{p450},
+			testPod:                     p600,
+			expectError:                 true,
+			wantFailingPredicateName:    "NodeResourcesFit",
+			wantFailingPredicateReasons: []string{"Insufficient cpu"},
+			wantErrorSubstrings:         []string{"NodeResourcesFit", "Insufficient cpu"},
 		},
 		{
 			name:          "default - other pod - ok",
@@ -90,19 +98,41 @@ func TestRunFiltersOnNode(t *testing.T) {
 			expectError:   false,
 		},
 		{
-			name:          "default - empty - insuficient cpu",
-			node:          n1000,
-			scheduledPods: []*apiv1.Pod{},
-			testPod:       p8000,
-			expectError:   true,
+			name:                        "default - empty - insuficient cpu",
+			node:                        n1000,
+			scheduledPods:               []*apiv1.Pod{},
+			testPod:                     p8000,
+			expectError:                 true,
+			wantFailingPredicateName:    "NodeResourcesFit",
+			wantFailingPredicateReasons: []string{"Insufficient cpu"},
+			wantErrorSubstrings:         []string{"NodeResourcesFit", "Insufficient cpu"},
+		},
+		{
+			name:                        "default - empty - ok",
+			node:                        n1000,
+			scheduledPods:               []*apiv1.Pod{},
+			testPod:                     p600,
+			expectError:                 false,
+			wantFailingPredicateName:    "NodeResourcesFit",
+			wantFailingPredicateReasons: []string{"Insufficient cpu"},
 		},
 		{
-			name:          "default - empty - ok",
+			name:          "default - affinity on existing node - ok",
 			node:          n1000,
 			scheduledPods: []*apiv1.Pod{},
-			testPod:       p600,
+			testPod:       podWithAffinity,
 			expectError:   false,
 		},
+		{
+			name:                     "default - affinity on non-existing node - error",
+			node:                     n1000,
+			scheduledPods:            []*apiv1.Pod{},
+			testPod:                  podWithInvalidAffinity,
+			expectError:              true,
+			customConfig:             customConfig,
+			wantFailingPredicateName: "NodeAffinity",
+			wantErrorSubstrings:      []string{"PreFilter filtered the Node out"},
+		},
 		// custom predicate checker test cases
 		{
 			name:          "custom - other pod - ok",
@@ -150,10 +180,11 @@ func TestRunFiltersOnNode(t *testing.T) {
 				assert.Nil(t, state)
 				assert.NotNil(t, predicateError)
 				assert.Equal(t, clustersnapshot.FailingPredicateError, predicateError.Type())
-				assert.Equal(t, "NodeResourcesFit", predicateError.FailingPredicateName())
-				assert.Equal(t, []string{"Insufficient cpu"}, predicateError.FailingPredicateReasons())
-				assert.Contains(t, predicateError.Error(), "NodeResourcesFit")
-				assert.Contains(t, predicateError.Error(), "Insufficient cpu")
+				assert.Equal(t, tt.wantFailingPredicateName, predicateError.FailingPredicateName())
+				assert.Equal(t, tt.wantFailingPredicateReasons, predicateError.FailingPredicateReasons())
+				for _, wantErrorSubstring := range tt.wantErrorSubstrings {
+					assert.Contains(t, predicateError.Error(), wantErrorSubstring)
+				}
 			} else {
 				assert.Nil(t, predicateError)
 				assert.NotNil(t, state)
diff --git a/cluster-autoscaler/utils/test/test_utils.go b/cluster-autoscaler/utils/test/test_utils.go
index 0b9bfa7aafc7..026a4e41266e 100644
--- a/cluster-autoscaler/utils/test/test_utils.go
+++ b/cluster-autoscaler/utils/test/test_utils.go
@@ -194,6 +194,29 @@ func WithDeletionTimestamp(deletionTimestamp time.Time) func(*apiv1.Pod) {
 	}
 }
 
+// WithNodeNamesAffinity sets pod's affinity for specific nodes.
+func WithNodeNamesAffinity(nodeNames ...string) func(*apiv1.Pod) {
+	return func(pod *apiv1.Pod) {
+		pod.Spec.Affinity = &apiv1.Affinity{
+			NodeAffinity: &apiv1.NodeAffinity{
+				RequiredDuringSchedulingIgnoredDuringExecution: &apiv1.NodeSelector{
+					NodeSelectorTerms: []apiv1.NodeSelectorTerm{
+						{
+							MatchFields: []apiv1.NodeSelectorRequirement{
+								{
+									Key:      metav1.ObjectNameField,
+									Operator: apiv1.NodeSelectorOpIn,
+									Values:   nodeNames,
+								},
+							},
+						},
+					},
+				},
+			},
+		}
+	}
+}
+
 // BuildTestPodWithEphemeralStorage creates a pod with cpu, memory and ephemeral storage resources.
 func BuildTestPodWithEphemeralStorage(name string, cpu, mem, ephemeralStorage int64) *apiv1.Pod {
 	startTime := metav1.Unix(0, 0)


Reviews:
user: login: towca
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T17:35:47Z
user: login: norbertcyran
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T08:38:55Z

Review comments:
diff_hunk: @@ -111,12 +111,14 @@ func (p *SchedulerPluginRunner) RunFiltersOnNode(pod *apiv1.Pod, nodeName string
 
 	state := schedulerframework.NewCycleState()
 	// Run the PreFilter phase of the framework for the Pod and check the results. See the corresponding comments in RunFiltersUntilPassingNode() for more info.
-	preFilterResult, preFilterStatus, _ := p.fwHandle.Framework.RunPreFilterPlugins(context.TODO(), state, pod)
+	preFilterResult, preFilterStatus, pluginsWithNodes := p.fwHandle.Framework.RunPreFilterPlugins(context.TODO(), state, pod)
path: cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go
user: login: towca
body: nit: I'd prefer a more accurate name here, `pluginsThatFilteredNodesOut` is too verbose, but maybe something like `filteringPlugins`?
created_at: 2025-05-06T17:33:21Z
author_association: COLLABORATOR
diff_hunk: @@ -111,12 +111,14 @@ func (p *SchedulerPluginRunner) RunFiltersOnNode(pod *apiv1.Pod, nodeName string
 
 	state := schedulerframework.NewCycleState()
 	// Run the PreFilter phase of the framework for the Pod and check the results. See the corresponding comments in RunFiltersUntilPassingNode() for more info.
-	preFilterResult, preFilterStatus, _ := p.fwHandle.Framework.RunPreFilterPlugins(context.TODO(), state, pod)
+	preFilterResult, preFilterStatus, pluginsWithNodes := p.fwHandle.Framework.RunPreFilterPlugins(context.TODO(), state, pod)
path: cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go
user: login: norbertcyran
body: sounds good, renamed to `nodeFilteringPlugins`
created_at: 2025-05-07T08:38:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: k8s-ci-robot
created_at: 2025-05-06T09:45:22Z
body: Hi @norbertcyran. Thanks for your PR.

I'm waiting for a [kubernetes](https://github.com/orgs/kubernetes/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fautoscaler).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

author_association: CONTRIBUTOR
user: login: towca
created_at: 2025-05-06T17:35:33Z
body: Good find, thanks for the fix! Feel free to unhold if you don't want to fix the nit.

/lgtm
/approve
/hold
author_association: COLLABORATOR
user: login: k8s-ci-robot
created_at: 2025-05-06T17:35:46Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/autoscaler/pull/8097#" title="Author self-approved">norbertcyran</a>*, *<a href="https://github.com/kubernetes/autoscaler/pull/8097#issuecomment-2855401162" title="Approved">towca</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fautoscaler).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[cluster-autoscaler/OWNERS](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/OWNERS)~~ [towca]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: towca
created_at: 2025-05-06T17:36:30Z
body: /ok-to-test
author_association: COLLABORATOR
user: login: towca
created_at: 2025-05-07T09:25:01Z
body: /lgtm
author_association: COLLABORATOR
user: login: towca
created_at: 2025-05-07T10:16:10Z
body: /unhold
author_association: COLLABORATOR
user: login: aaronfern
created_at: 2025-08-04T09:33:00Z
body: We found that this bug exists on v1.32
Can this fix please be backported to v1.32 as well?
author_association: NONE
user: login: jackfrancis
created_at: 2025-08-04T21:42:18Z
body: /cherry-pick cluster-autoscaler-release-1.32
author_association: CONTRIBUTOR
user: login: k8s-infra-cherrypick-robot
created_at: 2025-08-04T21:42:55Z
body: @jackfrancis: new pull request created: #8405

<details>

In response to [this](https://github.com/kubernetes/autoscaler/pull/8097#issuecomment-3152484917):

>/cherry-pick cluster-autoscaler-release-1.32


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: jackfrancis
created_at: 2025-08-04T21:45:07Z
body: /cherry-pick cluster-autoscaler-release-1.31
author_association: CONTRIBUTOR
user: login: k8s-infra-cherrypick-robot
created_at: 2025-08-04T21:45:46Z
body: @jackfrancis: #8097 failed to apply on top of branch "cluster-autoscaler-release-1.31":
```
Applying: Prevent nil dereference of preFilterStatus
Using index info to reconstruct a base tree...
A	cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go
A	cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner_test.go
M	cluster-autoscaler/utils/test/test_utils.go
Falling back to patching base and 3-way merge...
Auto-merging cluster-autoscaler/utils/test/test_utils.go
CONFLICT (modify/delete): cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner_test.go deleted in HEAD and modified in Prevent nil dereference of preFilterStatus. Version Prevent nil dereference of preFilterStatus of cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner_test.go left in tree.
CONFLICT (modify/delete): cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go deleted in HEAD and modified in Prevent nil dereference of preFilterStatus. Version Prevent nil dereference of preFilterStatus of cluster-autoscaler/simulator/clustersnapshot/predicate/plugin_runner.go left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 Prevent nil dereference of preFilterStatus

```

<details>

In response to [this](https://github.com/kubernetes/autoscaler/pull/8097#issuecomment-3152490151):

>/cherry-pick cluster-autoscaler-release-1.31


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: jackfrancis
created_at: 2025-08-04T21:48:33Z
body: @norbertcyran if you're willing to manually patch the `cluster-autoscaler-release-1.31` branch we could include this in a future 1.31.x release (I'll be happy to lgtm/approve)
author_association: CONTRIBUTOR
user: login: norbertcyran
created_at: 2025-08-08T10:05:12Z
body: @jackfrancis IIRC, that was a regression introduced in 1.32
author_association: CONTRIBUTOR