Project: Volcano (Orchestration & Management / Scheduling & Orchestration)
File: volcano-introduction.pdf

Volcano: Bring Batch Capability 
Into Kubernetes
Da Ma (@k82cn)
Huawei Expert
Da Ma Software Architect
Kubernetes SIG -Scheduling co -Leader
Volcano & kube -batch creator
Expert at Huawei (now)
Ex-IBM Spectrum CE/L3 Team/Tech Lead
Jilin University master’s degree, 
majoring in grid computing and distributed 
system
Gaps
Job/Queue Management
•Queue status/configuration
•Hierarchical queue
•Job with multiple pod template
•Lifecycle management of Job, e.g. restart, 
suspend/resume
•Error Handling, e.g. restart job if pod failed (MPI, TFJob)
•Indexed Job
•Task dependency, e.g. Spark (executor/driver)
•Delay Pod Creation
•…
Runtime
•Singularity
•…Scheduler
•Coscheduling
•Faire -share
•Queue
•Preemption/Reclaim
•Reserve/Backfill
•Topology (network, accelerator)
•…
Others
•Throughput
•Round -trip
•Data locality (Data Aware Scheduling)
•…Volcano: A Kubernetes Native Batch System
Website: https://volcano.sh
Github: http://github.com/volcano -sh/volcano
Twitter: https://twitter.com/volcano_sh
Slack: http://volcano -sh.slack.com
Email: volcano -sh@googlegroups.comOverall Architecture
Domain frameworks:
•Deployment/Installation of framework in k8s
•Map framework’s terms/concepts into common concept, e.g. Job, 
Queue
•Enable related features for frameworks, e.g. gang -scheduling for 
TensorFlow training
Common Service for high performance workload:
•Batch scheduling, e.g. fair -share, gang -scheduling
•Enhanced job management, e.g. multiple pod template, error handling
•Accelerator, e.g. GPU, FPGA
•kubectl plugins, e.g. show Job/Queue informationOverall Architecture
The policy in vk-scheduler is pluggable, e.g. DRF, Priority, Gang
vk-controllers includes JobExController , QueueController
Volcano handles high performance workloadKubectl creates a JobEx object in 
apiserver if all admission passed
JobExController create Pods based 
on its replicas and templates
vk-scheduler get the notification 
of Pod from apiserver
vk-scheduler chooses one host for 
the Pod of JobEx based on its 
policy
kubelet gets the notification of Pod 
from apiserver; and then start the 
containerScenarios: MPI
mpirunworker_1
worker_3worker_2
•Multiple Pod Template
•Lifecycle Policy
•Gang-scheduling•ssh or kubectl
•Complete job when mpirun completed
•Headless serviceScenarios: MPI
•The workers are deleted by job controller 
because of sshd
•The pod of mpiexec/mpirun will not be 
deleted for output
•The pod of mpiexec/mpirun may restart few 
times because of network setup delayScenarios: Faire Share
Gang-scheduling: Job Execution Time
http://status.openlabtesting.org/builds?project=theopenlab%2Fvolcano
•Case 1: 1 job with 2ps + 4workers
•Case 2: 2 jobs with 2ps + 4workers
•Case 3: 5 jobs with 2ps + 4workers
•No enough resource for 2 Jobs to run 
concurrently; one of them wasting resources 
without Gang -Scheduling !
•2 of 5 jobs was finished because of deadlock 
(+20 hours)Task-Topology + Binpack
•The execution time of 3 jobs in total; 2ps + 
4workers for each job
•The execution time is unstable when tested by 
default scheduler
•The improvement dependent on data 
exchanges between pods
•Task -topology within a Job also improved 
scheduler’s performance
•Open Source at volcano -sh/volcano#272
Reference: “Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters”
Integrations
Framework Status API
MPI Done Volcano Job
Horovod Done Volcano Job
Kubeflow/tf -operator Done PodGroup
Kubeflow/arena Done Volcano Job
Spark-Operator On-going PodGroup
Cromwell On-going Volcano Job
PaddlePaddle On-going Volcano Job
… On-going Volcano Job / PodGroupPipeline
●GPU Share/Topology
●Job Management
●Queue Management
●Hierarchical Queue
●Preemption/Reclaim
●……Call for 
Contribution