Project: Volcano (Orchestration & Management / Scheduling & Orchestration)
File: multi-scheduler.md

In a cluster, there are Multiple Schedulers for different workload, e.g. default scheduler for system daemon, and volcano scheduler for biz workload. when the schedulers work in parallel, it is easy to occur the resource conflicts. Classify Pods to different resource group and each resource group specifies its own scheduler. There are no resource conflicts when Multiple schedulers work in parallel. To devide the cluster resource, user need to set the taints and labels on cluster nodes and set the corresponding tolerations and nodeSelector on the pods. It will need a lot of manpower to modify the yaml files. For less user operation, volcano supports the action about the pod parts automatically and adds a new MutatingAdmissionWebhook in volcano admission deployment to do it. The configmap defines some resource groups, each resource group contains The identification information of the pod object, volcano support two fields: Namespace field Annotation field The pod's data volcano needs to patch, volcano support to patch the fields and the fields are optional and not mandatory. User can set them according the application scenario. Tolerations Affinity NodeSelector SchedulerName If the object field is not setted, it is filled with a default as the following: ```` resourceGroup: XXX object: key: annotation # set the field and the value to be matched value: "volcano.sh/resource-group: XXX" # XXX is the value in resourceGroup field ```` For example ```` apiVersion: v1 kind: ConfigMap metadata: name: volcano-admission-configmap namespace: volcano-system data: volcano-admission.conf: | resourceGroups: resourceGroup: management # set the resource group name object: key: namespace # set the field and the value to be matched value: mng-ns-1 schedulerName: default-scheduler # set the scheduler for patching tolerations: # set the tolerations for patching effect: NoSchedule key: taint operator: Exists labels: volcano.sh/nodetype: management # set the nodeSelector for patching resourceGroup: cpu object: key: annotation value: "volcano.sh/resource-group: cpu" schedulerName: volcano labels: volcano.sh/nodetype: cpu resourceGroup: gpu # if the object is unsetted, default is: the key is annotation, schedulerName: volcano # the annotation key is fixed and is "volcano.sh/resource-group", The corresponding value is the resourceGroup field labels: volcano.sh/nodetype: gpu resourceGroup: fixed # if the object is unsetted, default is: the key is annotation, schedulerName: volcano # the annotation key is fixed and is "volcano.sh/resource-group", The corresponding value is the resourceGroup field object: key: annotation value: "volcano.sh/resource-group-job-role: master" affinity: "{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"volcano.sh/nodetype\",\"operator\":\"In\",\"values\":[\"fixed\"]}]}]}}}" ```` If the pod matches serval resource groups, volcano will match the resource group by sort. Default scheduler for system daemon, and volcano scheduler for biz workload Here is a cluster as the following: |node| label | taint| |-|--|--| |node1|

management| management-taint:NoSchedule| |node2| none| none| |pod | kind | namespace| |-|-|-| |deployment-A|deployment| mng-ns-1| |volcano-job-B|volcano job | default| Edit volcano-admission-configmap ```` apiVersion: v1 kind: ConfigMap metadata: name: volcano-admission-configmap namespace: volcano-system data: volcano-admission.conf: | resourceGroups: resourceGroup: management # set the resource group name object: key: namespace # set the field and the value to be matched value: mng-ns-1 schedulerName: default-scheduler # set the scheduler for patching tolerations: # set the tolerations for patching effect: NoSchedule key: management-taint operator: Exists labels: volcano.sh/nodetype: management # set the nodeSelector for patching ```` Submit deployment-A and volcano-job-B Check the deployment Pod ```` deployment-A: .... nodeSelector: volcano.sh/nodetype: management ... schedulerName: default-scheduler ... tolerations: effect: NoSchedule key: management-taint operator: Exists volcano-job-B: .... <none> .... ```` Check the result of the pod's scheduling ```` The pod in deployment-A is scheduled to node1. the pod in volcano-job-B is scheduled to node2. ```` Here is a cluster as the following: |node|label| |-|--| |node1| volcano.sh/nodetype: cpu| |node2| volcano.sh/nodetype: gpu| |volcano job | annotation| |-|-| |job-A|volcano.sh/resource-group: cpu| |job-B|volcano.sh/resource-group: gpu| Edit volcano-admission-configmap ```` apiVersion: v1 kind: ConfigMap metadata: name: volcano-admission-configmap namespace: volcano-system data: volcano-admission.conf: | resourceGroups: resourceGroup: cpu object: key: annotation value: "volcano.sh/resource-group: cpu" schedulerName: volcano labels: volcano.sh/nodetype: cpu resourceGroup: gpu schedulerName: volcano labels: volcano.sh/nodetype: gpu ```` Submit job-A and job-B Check the Pod information ```` job-A: .... nodeSelector: volcano.sh/nodetype: cpu ... schedulerName: volcano .... job-B: .... nodeSelector: volcano.sh/nodetype: gpu ... schedulerName: volcano .... ```` Check the result of the pod's scheduling ```` The pod in job-A is scheduled to node1. The pod in job-B job is scheduled to node2. ```` Here is a cluster as the following: |node|label| |-|--| |node1| volcano.sh/nodetype: fixed| |node2| none| none| |volcano job | annotation| |-|-| |job-A|volcano.sh/resource-group-job-role: master| |job-B|none| Edit volcano-admission-configmap ```` apiVersion: v1 kind: ConfigMap metadata: name: volcano-admission-configmap namespace: volcano-system data: volcano-admission.conf: | resourceGroups: resourceGroup: fixed # if the object is unsetted, default is: the key is annotation, schedulerName: volcano # the annotation key is fixed and is "volcano.sh/resource-group", The corresponding value is the resourceGroup field object: key: annotation value: "volcano.sh/resource-group-job-role: master" affinity: "{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"volcano.sh/nodetype",\"operator\":\"In\",\"values\":[\"fixed\"]}]}]}}}" ```` Submit job-A and job-B Check the Pod information ```` job-A: .... affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: matchExpressions: key: volcano.sh/nodetype operator: In values: fixed ... schedulerName: volcano .... job-B: .... ... schedulerName: volcano .... ```` Check the result of the pod's scheduling ```` The pod in job-A is scheduled to node1. The pod in job-B job is scheduled to node1/node2. ```` Enable this feature may modify pod information and affect resource utilization. The feature will divide the cluster resource and may decrease the resource usage. The feature will add some additional information to the pods, such as tolerations and nodeSelector data.