Kubernetes is a popular open source orchestration solution for container-based workloads. With Kubernetes, these workloads can be effectively managed in ways similar to traditional HPC clustering methods, though Kuberenetes alone does not offer all of the scheduling capabilities of Slurm - such as batch scheduling and gang scheduling - which we will discuss later in this post.
Kubernetes is based on clusters of nodes (either physical or virtual machines) that are controlled by a master. Each node hosts a group of pods (containers). These pods share resources in the node and exist in a local network. This network enables pods to communicate with each other while still containing isolated workloads or applications.
Below is a diagram of how Kubernetes Master relates to nodes and some of the components contained in each.
kube-scheduler
The default scheduler for Kubernetes deployments is kube-scheduler. This scheduler is run as part of the control plane. When you use Kubernetes, pods are frequently created and destroyed. When you create a new, the scheduler is needed to assign that pod to a node. Based on specified resource requirements and available resources, kube-scheduler locates a suitable node.
Locating a suitable node requires filtering nodes according to scheduling requirements. Any nodes that meet the requirements (called feasible nodes) are surfaced and scored to determine the best match. Scoring is based on multiple factors, including hardware, software, and policy restraints, data locality, affinity and anti-affinity definitions, resource requirements, and inter-workload interference.
Once a match is found, the pod is scheduled and a notification is sent to the API server. This notification makes the node accessible. If no nodes are available, the pod remains queued with the scheduler until a node is available.
For more information, check out the documentation:
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/. Understanding Slurm GPU Management