Project: Volcano (Orchestration & Management / Scheduling & Orchestration)
File: usage-based-scheduling.md

@william-wang Feb 16 2022 Currently the pod is scheduled based on the resource request and node allocatable resource other than the node usage. This leads to the unbalanced resource usage of compute nodes. Pod is scheduled to node with higher usage and lower allocation rate. This is not what users expect. Users expect the usage of each node to be balanced. Support node usaged based scheduling. Filter nodes whose usage is higher than usage threshold that user defined. Prioritize node with node usage and scheduling pod to node with low usage. The resource oversubscription is not considered in this project. Node GPU resource usage is out of scope. A separated goroutine is created in scheduler cache to talk with Metrics source(like prometheus, elasticsearch) which is used to collect and aggregate node usage metrics. The node usage data in cache is consumed by usage based scheduling plugin and other plugins like rescheduling plugin. The struct is as below. ``` type NodeUsage struct { MetricsTime time.Time cpuUsageAvg map[string]float64 memUsageAvg map[string]float64 } type NodeInfo struct { ResourceUsage NodeUsage } ``` PredictFn()Filter nodes whose usage is higher than usage threshold that user defined NodeOrder()Prioritize node with node real-time usage Preemptable()Pod whose node with lower usage is able to preempt pod whose nodes with higher usage ``` actions: "enqueue, allocate, backfill" tiers: plugins: name: priority name: gang name: conformance name: usage # usage based scheduling plugin enablePredicate: false # If the value is false, new pod scheduling is not disabled when the node load reaches the threshold. If the value is true or left blank, new pod scheduling is disabled. arguments: usage.weight: 5 cpu.weight: 1 memory.weight: 1 thresholds: cpu: 80 # The actual CPU load of a node reaches 80%, and the node cannot schedule new pods. mem: 70 # The actual Memory load of a node reaches 70%, and the node cannot schedule new pods. plugins: name: overcommit name: drf name: predicates name: proportion name: nodeorder name: binpack metrics: # metrics server related configuration type: prometheus # Optional, The metrics source type, prometheus by default, support "prometheus", "prometheus_adapt" and "elasticsearch" address: http://192.168.0.10:9090 # Mandatory, The metrics source address interval: 30s # Optional, The scheduler pull metrics from Prometheus with this interval, 30s by default tls: # Optional, The tls configuration insecureSkipVerify: "false" # Optional, Skip the certificate verification, false by default elasticsearch: # Optional, The elasticsearch configuration index: "custom-index-name" # Optional, The elasticsearch index name, "metricbeat-*" by default username: "" # Optional, The elasticsearch username password: "" # Optional, The elasticsearch password hostnameFieldName: "host.hostname" # Optional, The elasticsearch hostname field name, "host.hostname" by default ``` The plugins allow user to configure the cpu and memory average threshold within 5m. Any node whose usage is higher than the value of `CpuUsageAvg.5m` or `MemUsageAvg.5m` is filtered. If no threshold is configured, the node gets into priority

5m average usage is a typical value, more threshold can be added in the future if needed. The key format `CpuUsageAvg.<period>` such as `CpuUsageAvg.1h` . There are several factors need to consider while evaluating which node is the best to allocate pod firstly. The first factor is the node average usage in a period of time such as 5m. The node with the lowest usage gets the highest score with this factor. The second factor is the node usage fluctuation curve in a period of time. Suppose there are two nodes with similar usage, The usage of one node fluctuates over a wide range and the other one fluctuates over a narrow range like the `node1` in below tables. The `node1` has higher possibility to get a higher score than `node2`. This is useful to avoid the risk that node get overloaded in peak hours. The third factor identified is the resource dimension. Take the below table as example. if there is pending pod which is a compute sensitive pod, it is more suitable to schedule it to `node2` with higher mem weight. DRF might be suitable to handle the case to calculate the cpu, mem, gpu share for pod and each node then make the best match. Finally, there should a model to balance multiple factors with weight and calculate the final score for nodes. Only the cpu usage factor will be considered in the alpha version. | factors | node1 | node2 | | - | - | | | usage | cpu 80% | cpu 78% | | usage fluctuation curve | 5 | 40 | | resource dimension | cpu 80%, mem 20%| cpu 20%, mem 80% | | ... | ... | ... | | | | | The monitoring data of Volcano usage can be obtained from "Prometheus", "Custom Metrics API" and "Eleasticsearch", where the corresponding type of "Custom Metrics Api" is "prometheus_adapt". It is recommended to use the Custom Metrics API mode, and the monitoring indicators come from Prometheus Adapt. Ensure that Prometheus Adaptor is properly installed in the cluster and the custom metrics API is available. Set the user-defined indicator information. The rules to be added are as follows. For details, see ``` rules: seriesQuery: '{name=~"nodecpuseconds_total"}' resources: overrides: instance: resource: node name: matches: "nodecpuseconds_total" as: "nodecpuusage_avg" metricsQuery: avgovertime((1 - avg (irate(<<.Series>>{mode="idle"}[5m])) by (instance))[10m:30s]) seriesQuery: '{name=~"nodememoryMemTotal_bytes"}' resources: overrides: instance: resource: node name: matches: "nodememoryMemTotal_bytes" as: "nodememoryusage_avg" metricsQuery: avgovertime(((1-nodememoryMemAvailable_bytes/<<.Series>>))[10m:30s]) ``` Scheduler Configuration: ``` actions: "enqueue, allocate, backfill" tiers: plugins: name: priority name: gang name: conformance name: usage # usage based scheduling plugin enablePredicate: false # If the value is false, new pod scheduling is not disabled when the node load reaches the threshold. If the value is true or left blank, new pod scheduling is disabled. arguments: usage.weight: 5 cpu.weight: 1

1 thresholds: cpu: 80 # The actual CPU load of a node reaches 80%, and the node cannot schedule new pods. mem: 70 # The actual Memory load of a node reaches 70%, and the node cannot schedule new pods. plugins: name: overcommit name: drf name: predicates name: proportion name: nodeorder name: binpack metrics: # metrics server related configuration type: prometheusadaptor # Optional, The metrics source type, prometheus by default, support "prometheus", "prometheusadaptor" and "elasticsearch" interval: 30s # Optional, The scheduler pull metrics from Prometheus with this interval, 30s by default ``` Scheduler Configuration: ``` actions: "enqueue, allocate, backfill" tiers: plugins: name: priority name: gang name: conformance name: usage # usage based scheduling plugin enablePredicate: false # If the value is false, new pod scheduling is not disabled when the node load reaches the threshold. If the value is true or left blank, new pod scheduling is disabled. arguments: usage.weight: 5 cpu.weight: 1 memory.weight: 1 thresholds: cpu: 80 # The actual CPU load of a node reaches 80%, and the node cannot schedule new pods. mem: 70 # The actual Memory load of a node reaches 70%, and the node cannot schedule new pods. plugins: name: overcommit name: drf name: predicates name: proportion name: nodeorder name: binpack metrics: # metrics server related configuration type: prometheus # Optional, The metrics source type, prometheus by default, support "prometheus", "prometheus_adaptor" and "elasticsearch" address: http://192.168.0.10:9090 # Mandatory, The metrics source address interval: 30s # Optional, The scheduler pull metrics from Prometheus with this interval, 30s by default ``` Scheduler Configuration ``` actions: "enqueue, allocate, backfill" tiers: plugins: name: priority name: gang name: conformance name: usage # usage based scheduling plugin enablePredicate: false # If the value is false, new pod scheduling is not disabled when the node load reaches the threshold. If the value is true or left blank, new pod scheduling is disabled. arguments: usage.weight: 5 cpu.weight: 1 memory.weight: 1 thresholds: cpu: 80 # The actual CPU load of a node reaches 80%, and the node cannot schedule new pods. mem: 70 # The actual Memory load of a node reaches 70%, and the node cannot schedule new pods. plugins: name: overcommit name: drf name: predicates name: proportion name: nodeorder name: binpack metrics: # metrics server related configuration type: elasticsearch # Optional, The metrics source type, prometheus by default, support "prometheus", "prometheus_adaptor" and "elasticsearch" address: http://192.168.0.10:9090 # Mandatory, The metrics source address interval: 30s # Optional, The scheduler pull metrics from Prometheus with this interval, 30s by default tls: # Optional, The tls configuration insecureSkipVerify: "false" # Optional, Skip the certificate verification, false by default elasticsearch: # Optional, The elasticsearch configuration index: "custom-index-name" # Optional, The elasticsearch index name, "metricbeat-*" by default username: "" # Optional, The elasticsearch username password: "" # Optional, The elasticsearch password hostnameFieldName: "host.hostname" # Optional, The elasticsearch hostname field name, "host.hostname" by default ```