Project: gocrane (Observability and Analysis / Continuous Optimization)
File: scheduling-pods-based-on-actual-node-load.md

title: "Crane-scheduler" description: "Scheduling pods based on actual node load" weight: 15 Crane-scheduler is a collection of scheduler plugins based on , including: Make sure your kubernetes cluster has Prometheus installed. If not, please refer to . Configure the rules of Prometheus to get expected aggregated data: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: example-record spec: groups: name: cpumemusage_active interval: 30s rules: record: cpuusageactive expr: 100 - (avg by (instance) (irate(nodecpuseconds_total{mode="idle"}[30s])) * 100) record: memusageactive expr: 100*(1-nodememoryMemAvailablebytes/nodememoryMemTotalbytes) name: cpu-usage-5m interval: 5m rules: record: cpuusagemaxavg1h expr: maxovertime(cpuusageavg_5m[1h]) record: cpuusagemaxavg1d expr: maxovertime(cpuusageavg_5m[1d]) name: cpu-usage-1m interval: 1m rules: record: cpuusageavg_5m expr: avgovertime(cpuusageactive[5m]) name: mem-usage-5m interval: 5m rules: record: memusagemaxavg1h expr: maxovertime(memusageavg_5m[1h]) record: memusagemaxavg1d expr: maxovertime(memusageavg_5m[1d]) name: mem-usage-1m interval: 1m rules: record: memusageavg_5m expr: avgovertime(memusageactive[5m]) ``` !!! warning "Troubleshooting" The sampling interval of Prometheus must be less than 30 seconds, otherwise the above rules(such as cpuusageactive) may not take effect. There are two options: Install Crane-scheduler as a second scheduler Replace native Kube-scheduler with Crane-scheduler === "Main" ```bash helm repo add crane https://gocrane.github.io/helm-charts helm install scheduler -n crane-system --create-namespace --set global.prometheusAddr="REPLACEMEWITHPROMETHEUSADDR" crane/scheduler ``` === "Mirror" ```bash helm repo add crane https://finops-helm.pkg.coding.net/gocrane/gocrane helm install scheduler -n crane-system --create-namespace --set global.prometheusAddr="REPLACEMEWITHPROMETHEUSADDR" crane/scheduler ``` Backup `/etc/kubernetes/manifests/kube-scheduler.yaml` ```bash cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/ ``` Modify configfile of kube-scheduler(`scheduler-config.yaml`) to enable Dynamic scheduler plugin and configure plugin args: ```yaml title="scheduler-config.yaml" apiVersion: kubescheduler.config.k8s.io/v1beta2 kind: KubeSchedulerConfiguration ... profiles: schedulerName: default-scheduler plugins: filter: enabled: name: Dynamic score: enabled: name: Dynamic weight: 3 pluginConfig: name: Dynamic args: policyConfigPath: /etc/kubernetes/policy.yaml ... ``` Create `/etc/kubernetes/policy.yaml`, using as scheduler policy of Dynamic plugin: ```yaml title="/etc/kubernetes/policy.yaml" apiVersion: scheduler.policy.crane.io/v1alpha1 kind: DynamicSchedulerPolicy spec: syncPolicy: name: cpuusageavg_5m period: 3m name: cpuusagemaxavg1h period: 15m name: cpuusagemaxavg1d period: 3h name: memusageavg_5m period: 3m name: memusagemaxavg1h period: 15m name: memusagemaxavg1d period: 3h predicate: name: cpuusageavg_5m maxLimitPecent: 0.65 name: cpuusagemaxavg1h maxLimitPecent: 0.75 name: memusageavg_5m maxLimitPecent: 0.65 name: memusagemaxavg1h maxLimitPecent: 0.75 priority: name: cpuusageavg_5m weight: 0.2 name: cpuusagemaxavg1h weight: 0.3 name: cpuusagemaxavg1d weight: 0.5 name: memusageavg_5m weight: 0.2 name: memusagemaxavg1h weight: 0.3 name: memusagemaxavg1d weight: 0.5 hotValue: timeRange: 5m count: 5 timeRange: 1m count: 2 ``` Modify `kube-scheduler.yaml` and replace kube-scheduler image with Crane-scheduler ```yaml title="kube-scheduler.yaml" ... image: docker.io/gocrane/crane-scheduler:0.0.23 ... ``` Install : === "Main" ```bash kubectl apply -f https://raw.githubusercontent.com/gocrane/crane-scheduler/main/deploy/controller/rbac.yaml kubectl apply -f https://raw.githubusercontent.com/gocrane/crane-scheduler/main/deploy/controller/deployment.yaml ``` === "Mirror" ```bash kubectl apply -f https://gitee.com/finops/crane-scheduler/raw/main/deploy/controller/rbac.yaml kubectl apply -f https://gitee.com/finops/crane-scheduler/raw/main/deploy/controller/deployment.yaml ``` Test Crane-scheduler with following example: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: cpu-stress spec: selector: matchLabels: app: cpu-stress replicas: 1 template: metadata: labels: app: cpu-stress spec: schedulerName: crane-scheduler hostNetwork: true tolerations: key: node.kubernetes.io/network-unavailable operator: Exists effect: NoSchedule containers: name: stress image: docker.io/gocrane/stress:latest command: ["stress", "-c", "1"] resources: requests: memory: "1Gi" cpu: "1" limits: memory: "1Gi" cpu: "1" ``` !!! Note Change `crane-scheduler` to `default-scheduler` if `crane-scheduler` is used as default. There will be the following event if the test pod is successfully scheduled: ```bash Type Reason Age From Message - - - Normal Scheduled 28s crane-scheduler Successfully assigned default/cpu-stress-7669499b57-zmrgb to vm-162-247-ubuntu ```