Repository: kubernetes/autoscaler (Go)

Issue:
Cluster Autoscaler tries to schedule evicted pods on drained nodes
**Which component are you using?**:
cluster-autoscaler

**What version of the component are you using?**:
1.26.3

**What k8s version are you using (`kubectl version`)?**:
1.26

**What environment is this in?**:
EKS

**What did you expect to happen?**:
cluster autoscaler shouldn't try to simulate the scheduling behavior for pods that are evicted (api-server return 201 for the /evcition request) on the drained nodes.

**What happened instead?**:

during the first cluster-autocaler reconcilation, cluster-autoscaler decided to scale down a underutilized node (our threashold is 70)
```
I0414 10:21:40.188957 hinting_simulator.go:77] Pod pod-hbhqf can be moved to ip-10-160-48-58.us-west-2.compute.internal
I0414 10:21:40.188962 hinting_simulator.go:77] Pod pod-lx6fg can be moved to ip-10-160-52-115.us-west-2.compute.internal
I0414 10:21:40.293272       1 actuator.go:212] Scale-down: removing node ip-10-160-50-114.us-west-2.compute.internal, utilization: {0.6940581542351454 0.6892291641475079 0 cpu 0.6940581542351454}, pods to reschedule: pod-hbhqf,pod-lx6fg
```
the node drain started, and we have PDB maxUnavailable to set 1, in the audit log, we observed `pod-hbhqf ` eviction call succeeded, the new pod `pod-r9t79` coming up and got scheduled (but not ready yet), and `pod-lx6fg` eviction call failed. 
```
- 03:21:45.446 AM
   PDB status disruptionsAllowed: 0
   Kubelet killing pod-hbhqf container app
- 03:21:45.474 AM
   /api/v1/namespaces/ns/pods/pod-hbhqf/eviction returned 201
- 03:21:45.697 AM
   pod-r9t79 pod creation returned 201
- 03:21:45.698 AM
  Bind pod-r9t79 to node ip-10-160-52-115.us-west-2.compute.internal succeeded
- 03:21:45.975 AM
  /api/v1/namespaces/ns/pods/pod-lx6fg/eviction returned 429 (PDB violation)
```
during the second cluster-autoscaler reconcilation, it tried to bring up a new node in the AWS Autoscale scaling group:
```
I0414 10:21:50.361899 hinting_simulator.go:77] Pod pod-hbhqf can be moved to ip-10-160-48-58.us-west-2.compute.internal
I0414 10:21:50.362342 hinting_simulator.go:116] failed to find place for pod-lx6fg: cannot put pod-lx6fg on any node
I0414 10:21:50.485239       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ns", Name:"pod-lx6fg", APIVersion:"v1", FieldPath:""}): type: 'Normal' reason: 'TriggeredScaleUp' pod triggered scale-up: [{asg-name 9->10 (max: 10)}]
```

**Analysis**
We think that the current code puts all pods on the draining nodes to the unscheduable pod list, then runs the scheduling simulation. But actually some of the pods are the draining nodes are already in the eviction process and the deletionTimestamp have been set.
Should we ignore the pods that the eviction are allowed?
https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/#how-api-initiated-eviction-works



Discussion:

Patch:
diff --git a/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes.go b/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes.go
index 495aae1c08f8..58459da50f17 100644
--- a/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes.go
+++ b/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes.go
@@ -51,6 +51,11 @@ func currentlyDrainedPods(context *context.AutoscalingContext) []*apiv1.Pod {
 			continue
 		}
 		for _, podInfo := range nodeInfo.Pods {
+			// Filter out pods that has deletion timestamp set
+			if podInfo.Pod.DeletionTimestamp != nil {
+				klog.Infof("Pod %v has deletion timestamp set, skipping injection to unschedulable pods list", podInfo.Pod.Name)
+				continue
+			}
 			pods = append(pods, podInfo.Pod)
 		}
 	}


Test patch:
diff --git a/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes_test.go b/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes_test.go
index 678afbd05709..0aebcbd4d671 100644
--- a/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes_test.go
+++ b/cluster-autoscaler/core/podlistprocessor/currently_drained_nodes_test.go
@@ -69,6 +69,20 @@ func TestCurrentlyDrainedNodesPodListProcessor(t *testing.T) {
 				BuildTestPod("p2", 200, 1),
 			},
 		},
+		{
+			name:         "single node undergoing deletion, pods with deletion timestamp set",
+			drainedNodes: []string{"n"},
+			nodes: []*apiv1.Node{
+				BuildTestNode("n", 1000, 10),
+			},
+			pods: []*apiv1.Pod{
+				BuildScheduledTestPod("p1", 100, 1, "n"),
+				BuildTestPod("p2", 200, 1, WithNodeName("n"), WithDeletionTimestamp(time.Now())),
+			},
+			wantPods: []*apiv1.Pod{
+				BuildTestPod("p1", 100, 1),
+			},
+		},
 		{
 			name:         "single empty node undergoing deletion",
 			drainedNodes: []string{"n"},
diff --git a/cluster-autoscaler/utils/test/test_utils.go b/cluster-autoscaler/utils/test/test_utils.go
index 81c939cbfc43..de868d1ef9c9 100644
--- a/cluster-autoscaler/utils/test/test_utils.go
+++ b/cluster-autoscaler/utils/test/test_utils.go
@@ -150,6 +150,13 @@ func WithMaxSkew(maxSkew int32, topologySpreadingKey string) func(*apiv1.Pod) {
 	}
 }
 
+// WithDeletionTimestamp sets deletion timestamp to the pod.
+func WithDeletionTimestamp(deletionTimestamp time.Time) func(*apiv1.Pod) {
+	return func(pod *apiv1.Pod) {
+		pod.DeletionTimestamp = &metav1.Time{Time: deletionTimestamp}
+	}
+}
+
 // BuildTestPodWithEphemeralStorage creates a pod with cpu, memory and ephemeral storage resources.
 func BuildTestPodWithEphemeralStorage(name string, cpu, mem, ephemeralStorage int64) *apiv1.Pod {
 	startTime := metav1.Unix(0, 0)