Repository: kubernetes-sigs/descheduler

Issue #1634: HighNodeUtilization will evict nodes that do not have resource types specified in the configuration profile thresholds.
Created: 2025-02-20T13:31:22Z
What version of descheduler are you using?
descheduler version: v0.28.0
Does this issue reproduce with the latest release?
Yes
Which descheduler CLI options are you using?
Please provide a copy of your descheduler policy config file
  - name: HighNodeUtilization
     args:
       thresholds:
         "cloudml.gpu/v100-32g": 40

What k8s version are you using (kubectl version)?
kubectl version Output$ kubectl version

What did you do?

What did you expect to see?
If the node does not possess the resources specified in the configuration profile thresholds, then it will not be considered as a underutilized node and thus will not be subject to eviction.
In a cluster where multiple types of GPU nodes exist, if a certain GPU type is not included in the threshold settings of the configuration file, the node hosting that GPU will be deemed as underutilized node and will be evicted. This occurs even if the GPU allocation rate on that node is at 100%, which is evidently unreasonable.
What did you see instead?
In the cluster, a node that did not contain  "cloudml.gpu/v100-32g" resources was mistakenly identified as a underutilized node and was consequently evicted.
2025-02-20T12:06:51.894357974Z I0220 12:06:51.894295       1 nodeutilization.go:198] "Node is underutilized" node="test", usage=map[cloudml.gpu/v100-32g:0 cpu:13777m memory:19106058Ki pods:33] usagePercentage=map[cpu:10.76328125 memory:1.8098670918577757 pods:12.992125984251969]

Duplicate issue #1635: fix: HighNodeUtilization plugin incorrectly evicts all pods when only GPU resources are configured
Created: 2025-02-24T06:58:11Z
What version of descheduler are you using?
descheduler version: main branch (latest development version)
Does this issue reproduce with the latest release?
Yes, this issue can be reproduced with the latest release.
Which descheduler CLI options are you using?
Using descheduler as a Kubernetes deployment with a ConfigMap-based policy configuration.
Please provide a copy of your descheduler policy config file
apiVersion: descheduler/v1alpha2
kind: DeschedulerConfiguration
kind: DeschedulerConfiguration
deschedulerPolicy:
  strategies:
    HighNodeUtilization:
      enabled: true
      params:
        thresholds:
          "cloudml.gpu/v100-32g": 40
          "cloudml.gpu/h20-96g": 40
        numberOfNodes: 0
  nodeSelector:
    node.miks.io/type: "gpu"
What k8s version are you using (kubectl version)?
Client version: v1.24.0
Server version: v1.21
What did you do?

Deployed descheduler with the above configuration which only includes GPU resource thresholds
Created pods with and without GPU resource requests on nodes
Observed the behavior when node GPU utilization falls below the threshold (40%)

Steps to reproduce:
# 1. Deploy a pod with GPU request
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: gpu-container
    image: nvidia/cuda
    resources:
      requests:
        "cloudml.gpu/v100-32g": "1"
      limits:
        "cloudml.gpu/v100-32g": "1"

# 2. Deploy a pod without GPU request
apiVersion: v1
kind: Pod
metadata:
  name: non-gpu-pod
spec:
  containers:
  - name: nginx
    image: nginx
What did you expect to see?
When a node's GPU utilization falls below the threshold:

Only pods that have requested the specified GPU resources (cloudml.gpu/v100-32g or cloudml.gpu/h20-96g) should be considered for eviction
Pods that don't use these GPU resources should not be affected

What did you see instead?
When a node's GPU utilization falls below the threshold:

The HighNodeUtilization plugin evicts all evictable pods from the node, regardless of whether they use the specified GPU resources or not
This includes pods that don't request any GPU resources, which should not be affected by the GPU utilization threshold

This behavior is problematic because:

It unnecessarily disrupts pods that don't contribute to the GPU utilization
It may cause service disruption for non-GPU workloads
It doesn't align with the principle of only balancing the specifically configured resources

The root cause appears to be in the pod filtering logic of the HighNodeUtilization plugin, where it doesn't properly check whether pods are using the configured GPU resources before selecting them for eviction.

Closing comment (2025-02-24T11:36:01Z): same as: #1634