Repository: DataDog/watermarkpodautoscaler
Pull request #278: Fix Evicted Pods Skewing Available Replicas Calculation
Author: login: justin-lesko
Labels: name: bug
description: Something isn't working
name: qa/skip-qa
name: mergequeue-status: done
Created: 2025-03-10T21:03:32Z | Merged: 2025-03-12T16:49:34Z

### What does this PR do?

The PR fixes a bug related to the `minAvailableReplicaPercentage` option and evicted pods.

> **minAvailableReplicaPercentage:** Indicates the minimum percentage of replicas that need to be available in order for the controller to autoscale the target. For instance, if set at 50 and less than half of the pods behind the target are in an Available state, the target will not be scaled by the controller.

#### Background
According to the [Kubernetes Documention](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/)
> During a node-pressure eviction, the kubelet sets the [phase](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) for the selected pods to Failed, and terminates the Pod. 

Because evicted pods remain in the `Terminated` state until garbage collected, they were skewing the `minAvailableReplicaPercentage` calculation because the WPA considers any pod that's been Ready at least once to be part of its target. 

#### Example
```
NAME                                                    READY   STATUS    RESTARTS   AGE
nginx-7b87cc957f-2ws2p                                  0/2     Error     1          37m
nginx-7b87cc957f-4cxmq                                  2/2     Running   0          57s
```
Given the above scenario, previously, the available replica percentage would have been `50%`. Now, with the `Failed` pod phase filter in place, it would be `100%`.
  
### Motivation

A bug report.

### Additional Notes

Anything else we should know when reviewing?

### Describe your test plan
I first tried simulating evictions by calling the eviction API, but the evicted pods were not persisting as expected. Upon closer inspection of the documentation I learned:
> Node-pressure eviction is not the same as [API-initiated eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/).

From there, I moved on to simulating a `Node-pressure` eviction.

#### Step 1: Deploy Cluster With Limited Resources
I deployed a `k3s` cluster with the following command:
```
k3d cluster create \
  --agents 2 \
  --agents-memory 3g \
  --k3s-arg "--disable=traefik@server:0" \
  --k3s-arg "--kubelet-arg=eviction-hard=memory.available<1Gi@agent:0" \
  --k3s-arg "--kubelet-arg=eviction-hard=memory.available<1Gi@agent:1" \
  --k3s-node-label "role=worker@agent:0,1" 
```
The included flags translate to:
* Deploy 2 "agent" (i.e. worker) nodes
* Agent nodes have 3Gi of memory, overridden from using all of the memory allocated to docker on my local machine
* Both agent nodes' kubelets enforce a hard 1Gi of available memory, and will evict pods that infringe on that
* Agent nodes are tagged with `role=worker`

#### Step 2: Force Memory Consumption
With a low bar for memory eviction in place, I needed a way for my deployed pods to consume enough memory to trigger an eviction. I looked into using `stress-ng` but I wasn't able to find a way to gradually increase memory usage. Ultimately I settled on a small Go program that allocates memory in a loop below.

<details>
<summary>go.mod</summary>
<br>

```
module memory-eater

go 1.23
```
</details>

<details>
<summary>main.go</summary>
<br>

```
package main

    import (
      "fmt"
      "runtime"
      "runtime/debug"
      "time"
    )

    func main() {
      // turn off garbage collection
      debug.SetGCPercent(-1)

      var memUsage []byte

      // allocate 5MB chunks
      allocSize := 5 * 1024 * 1024 

      fmt.Println("Starting memory consumption...")

      for i := 1; ; i++ {
        block := make([]byte, allocSize)
        for j := range block {
          block[j] = 1 // Actively write to memory to keep the memory in use
        }
        memUsage = append(memUsage, block...)

        memStats := &runtime.MemStats{}
        runtime.ReadMemStats(memStats)
        fmt.Printf("Iteration: %d | Allocated Memory: %.2f MB\n", i, float64(memStats.Alloc)/1024/1024)

        time.Sleep(1 * time.Second)
      }
    }

```
</details>

#### Step 3: Deploy a WPA Scalable Deployment
I deployed a basic Nginx deployment with a few key modifications
1. A taint toleration, so pods can still be scheduled after the node experiences memory pressure
    ```
    tolerations:
      - key: "node.kubernetes.io/memory-pressure"
        operator: "Exists"
        effect: "NoSchedule"
    ```
2. A node selector to target the agent nodes
    ```
    nodeSelector:
        role: worker
    ```
3. A sidecar container with the memory-eater application mounted
    ```
    - name: memory-eater
      image: golang:1.23
      imagePullPolicy: Always
      workingDir: /usr/src/memory-eater
      command: ["/bin/sh", "-c", "sleep infinity"]
      volumeMounts:
        - name: go-program
          mountPath: /usr/src/memory-eater
    ```

#### Step 4: Deploy/Configure the WPA
Deploy the WPA with the `minAvailableReplicaPercentage` option 

```
apiVersion: datadoghq.com/v1alpha1
kind: WatermarkPodAutoscaler
metadata:
  name: nginx
spec:
  algorithm: average
  minReplicas: 3
  maxReplicas: 30
  tolerance: "0.0"
  minAvailableReplicaPercentage: 50 # <--This is key to the test
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  metrics:
  - type: External
    external:
      highWatermark: 9
      lowWatermark: 8
      metricName: datadogmetric@<NAMESPACE>:nginx
      metricSelector: {}
```

#### Step 5: Trigger evictions
Trigger enough evictions so that evicted pods outnumber ready pods by running the following command N times:
```
# Issue an exec command without in the background, without tailing the logs
kubectl exec <POD> -c memory-eater -n <NAMESPACE> -- nohup go run . >/dev/null 2>&1 &
```

#### Step 6: Generate Load
I port-forwarded to my Nginx service

```
kubectl port-forward service/nginx -n workload-nginx 8080:80
```

Then I used k6 to generate a constant amount of load to trigger a desired scale-up

```
import http from 'k6/http';
import { check } from 'k6';

export let options = {
  scenarios: {
    fixed_rps: {
      executor: 'constant-arrival-rate',
      rate: 50, 
      timeUnit: '1s',
      duration: '30m',
      preAllocatedVUs: 100,
      maxVUs: 200,
    }
  }
};

export default function () {
  const res = http.get('http://localhost:8080');
  check(res, { 'status was 200': (r) => r.status == 200 }); // probably unnecessary
}
```

#### Step 7: Observe The Difference Between WPA Versions
##### Old Behavior- stuck, unable to scale while evicted pods skew available replica calculation
![WPA Failure Behavior](https://github.com/user-attachments/assets/f69d7438-e639-4d83-855a-7f1bfd3d0814)

##### New Behavior- able to scale despite the evicted pods persisting
![WPA Fix Behavior](https://github.com/user-attachments/assets/7a69d8aa-c527-4a01-802e-2e317fa20258)




Diff:
diff --git a/controllers/datadoghq/replica_calculator.go b/controllers/datadoghq/replica_calculator.go
index 0e294491..f106e18a 100644
--- a/controllers/datadoghq/replica_calculator.go
+++ b/controllers/datadoghq/replica_calculator.go
@@ -516,6 +516,22 @@ func (c *ReplicaCalculator) getReadyPodsCount(log logr.Logger, targetName string
 			incorrectTargetPodsCount++
 			continue
 		}
+
+		// PodFailed
+		//During a node-pressure eviction, the kubelet sets the phase for the selected pods to Failed, and terminates
+		// the Pod. These pods should be ignored because they may not be garbage collected for a long time.
+		// https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
+
+		// PodSucceeded
+		// A Deployment’s Pod should never be in PodSucceeded. If it is, it usually means:
+		// - The Pod is running a one-shot script instead of a service.
+		// - The restartPolicy is misconfigured.
+		// - A Job-like process was accidentally set up in a Deployment.
+		if pod.Status.Phase == corev1.PodFailed || pod.Status.Phase == corev1.PodSucceeded {
+			incorrectTargetPodsCount++
+			continue
+		}
+
 		_, condition := getPodCondition(&pod.Status, corev1.PodReady)
 		// We can't distinguish pods that are past the Readiness in the lifecycle but have not reached it
 		// and pods that are still Unschedulable but we don't need this level of granularity.
diff --git a/controllers/datadoghq/replica_calculator_test.go b/controllers/datadoghq/replica_calculator_test.go
index c946f8a9..835c6c91 100644
--- a/controllers/datadoghq/replica_calculator_test.go
+++ b/controllers/datadoghq/replica_calculator_test.go
@@ -1964,6 +1964,50 @@ func TestTooManyUnreadyPods(t *testing.T) {
 	tc.runTest(t)
 }
 
+// Pods that have either [PodFailed, PodSucceeded] statuses should not be counted when evaluating the
+// MinAverageReplicaPercentage as they both indicate pods that will never be ready again.
+
+// This test makes sure that these pod statuses are ignored when calculating the ready replicas.
+func TestMinAverageReplicaPercentageIgnoresFailedAndSucceededPods(t *testing.T) {
+	logf.SetLogger(zap.New())
+	metric1 := v1alpha1.MetricSpec{
+		Type: v1alpha1.ExternalMetricSourceType,
+		External: &v1alpha1.ExternalMetricSource{
+			MetricName:     "loadbalancer.request.per.seconds",
+			MetricSelector: &metav1.LabelSelector{MatchLabels: map[string]string{"foo": "bar"}},
+			HighWatermark:  resource.NewMilliQuantity(10000, resource.DecimalSI),
+			LowWatermark:   resource.NewMilliQuantity(7000, resource.DecimalSI),
+		},
+	}
+
+	tc := replicaCalcTestCase{
+		expectedReplicas: 3,
+		readyReplicas:    1,
+		pos: metricPosition{
+			isAbove: true,
+			isBelow: false,
+		},
+		scale: makeScale(testDeploymentName, 3, map[string]string{"name": "test-pod"}),
+		wpa: &v1alpha1.WatermarkPodAutoscaler{
+			Spec: v1alpha1.WatermarkPodAutoscalerSpec{
+				Algorithm:                     "average",
+				MinAvailableReplicaPercentage: 34,
+				Tolerance:                     *resource.NewMilliQuantity(20, resource.DecimalSI),
+				Metrics:                       []v1alpha1.MetricSpec{metric1},
+				ReplicaScalingAbsoluteModulo:  v1alpha1.NewInt32(1),
+			},
+		},
+		// simulate a pod eviction and a pod
+		podPhase: []corev1.PodPhase{corev1.PodRunning, corev1.PodFailed, corev1.PodSucceeded},
+		metric: &metricInfo{
+			spec:                metric1,
+			levels:              []int64{30000},
+			expectedUtilization: 30000, // only 1 ready replica so it's 100% utilized
+		},
+	}
+	tc.runTest(t)
+}
+
 // We have pods that are pending and one is within an acceptable window.
 func TestPendingNotExpiredScale(t *testing.T) {
 	logf.SetLogger(zap.New())


Reviews:
user: login: vboulineau
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T17:38:06Z
user: login: clamoriniere
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-12T12:02:22Z
user: login: justin-lesko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T14:28:09Z
user: login: clamoriniere
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-12T16:32:53Z

Review comments:
diff_hunk: @@ -516,6 +516,15 @@ func (c *ReplicaCalculator) getReadyPodsCount(log logr.Logger, targetName string
 			incorrectTargetPodsCount++
 			continue
 		}
+
+		// During a node-pressure eviction, the kubelet sets the phase for the selected pods to Failed, and terminates the Pod.
+		// https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
+		// These pods should be ignored because they may not be garbage collected for a long time
+		if pod.Status.Phase == corev1.PodFailed {
path: controllers/datadoghq/replica_calculator.go
user: login: clamoriniere
body: I think I saw also `corev1.PodSucceeded` occurence in the path. that was not cleanup fast enough.
From what I understand `PodSucceeded` should only happen with Job's pod. However with Deployment miss configuration it can happen. and I feel I already saw it internally. 

for a quick search in kubernetes docs:
```
A Deployment’s Pod should never be in PodSucceeded. If it is, it usually means:
	•	The Pod is running a one-shot script instead of a service.
	•	The restartPolicy is misconfigured.
	•	A Job-like process was accidentally set up in a Deployment.
```

can we also ignore them too ? 
created_at: 2025-03-12T12:02:22Z
author_association: COLLABORATOR
diff_hunk: @@ -516,6 +516,15 @@ func (c *ReplicaCalculator) getReadyPodsCount(log logr.Logger, targetName string
 			incorrectTargetPodsCount++
 			continue
 		}
+
+		// During a node-pressure eviction, the kubelet sets the phase for the selected pods to Failed, and terminates the Pod.
+		// https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
+		// These pods should be ignored because they may not be garbage collected for a long time
+		if pod.Status.Phase == corev1.PodFailed {
path: controllers/datadoghq/replica_calculator.go
user: login: justin-lesko
body: That sounds like a good improvement, thanks!
created_at: 2025-03-12T14:28:09Z
author_association: CONTRIBUTOR

PR comments:
user: login: justin-lesko
created_at: 2025-03-12T16:33:47Z
body: /merge
author_association: CONTRIBUTOR
user: login: dd-devflow[bot]
created_at: 2025-03-12T16:33:52Z
body: <!-- "FeedbackWorkflowId":"74d71342-2cb7-4552-bfd5-d92a6b7f7d68_36" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/74d71342-2cb7-4552-bfd5-d92a6b7f7d68_36)
`2025-03-12 16:33:52 UTC` :information_source: Start processing command `/merge`

---
`2025-03-12 16:33:58 UTC` :information_source: **MergeQueue:** pull request added to the queue

The expected merge time in `main` is approximately 0s (p90).

---
`2025-03-12 16:49:40 UTC` :information_source: **MergeQueue:** This merge request was merged

author_association: NONE