Project: Docker Swarm (Orchestration & Management / Scheduling & Orchestration)
File: generic_resources.md

* * * This document describes the a solution to managing accountable node level resources unknown to docker swarm. Each node is different in its own way, some nodes might have access to accelerators, some nodes might have access to network devices and others might support AVX while others only support SSE. Swarmkit needs some simple way to account for these resources without having to implement them each time a new kind of resource comes into existence. While it is true that some resources can be advertised with labels, many resources have a shareable capacity and cant be represented well as a label. The implementation we chose is to reuse a proven solution used by industry projects (mesos and kubernetes) which lead us to implement two kinds of generic resources: Discrete (int64) Set Other types of resource like scalar can be extended Discrete resources are for use cases where only an unsigned is needed to account for the resource (see Linux Realtime). A set would mostly be used for every resource which would need an exclusive access to it. Future work might require new mechanisms to be made to allow generic resources to be cluster wide in order to satisfy other use cases (e.g: pool of licenses) Future work might require to add filters at the resource level Future work might require to share resources Exclusive access to discrete accelerators: GPU devices FPGA devices MICs (Many-Integrated Core, such as Xeon Phi) ... Support for tracking additional cgroup quotas like cpurtruntime. * PersistentDisks in GCE Counting slots allowed access to a shared parallel file system. * * * * Associate multiple generic resources with a node Request some portion of available generic resources in the service during service creation Enable users to define and schedule generic resources in a vanilla swarmkit cluster Solve how generic resources allocations are to be enforced or

Solve how generic resources are discovered Solve how to filter at the resources level Solve how cluster-level generic resources should be advertised The services may only ask for generic resources as integers as the solution for asking for specific resources can be solved in many different ways (filters, multiple kinds of resources, ...) and should not be addressed in this PR. ``` $ # Single resource $ swarmctl service create --name nginx --image nginx:latest --generic-resources "banana=2" $ # Multiple resources $ swarmctl service create --name nginx --image nginx:latest --generic-resources "banana=2,apple=3" ``` A node may advertise either an discrete number of resources or a set of resources. It is the scheduler's job to decide which resource to assign and keep track of which task owns which resource. ``` $ swarmd -d $DIR --join-addr $IP --join-token $TOKEN --generic-node-resources "banana=blue,banana=red,banana=green,apple=8" ``` As swarmkit is not responsible for exposing the resources to the container (or acquiring them), it needs a way to communicate how many generic resources were assigned (in the case of discrete resources) or / and what resources were selected (in the case of sets). The reference implementation of the executor exposes the resource value to software running in containers through environment variables. The exposed environment variable is prefixed with `DOCKERRESOURCE` and it's key uppercased. See example in the next section. If we run `swarmctl inspect` we can see: ```bash $ swarmctl node inspect node-with-generic-resources ID : 9toi8u8zo1qbkiw1d1nrsevdd Hostname : node-with-generic-resources Status: State : READY Availability : ACTIVE Address : 127.0.0.1 Platform: Operating System : linux Architecture : x86_64 Resources: CPUs : 12 Memory : 31 GiB apple : 3 banana : red, blue, green Plugins: Network : [bridge host macvlan null overlay] Volume : [local nvidia-docker] Engine Version : 1.13.1 $ swarmctl service create --name nginx --image nginx:latest --generic-resources "banana=2,apple=2" $ swarmctl service inspect nginx ID : abxelhl822d8zyjqam3m3szb0 Name : nginx Replicas : 1/1 Template Container Image : nginx:latest Resources Reservations: banana : 2 apple : 2 Task ID Service Slot Image Desired State Last State Node - - -- - - - 6pbwd5qj7i0nsxlyi803qpf2x nginx 1 nginx:latest RUNNING RUNNING 12 seconds ago node-with-generic-resources $ # ssh to the node $ docker inspect $CONTAINER_ID --format '{{.Config.Env}}' | tr -s ' ' '\n' [DOCKERRESOURCEBANANA=red,blue DOCKERRESOURCEAPPLE=2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin NGINX_VERSION=1.13.0-1~stretch NJS_VERSION=1.13.0.0.1.10-1~stretch] ```