Project: Apache Hadoop (App Definition and Development / Database)
File: UsingGpus.md

<! Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> As of now, only Nvidia GPUs are supported by YARN YARN node managers have to be pre-installed with Nvidia drivers. When Docker is used as container runtime context, nvidia-docker 1.0 needs to be installed (Current supported version in YARN for nvidia-docker). In `resource-types.xml` Add following properties ``` <configuration> <property> <name>yarn.resource-types</name> <value>yarn.io/gpu</value> </property> </configuration> ``` In `yarn-site.xml` `DominantResourceCalculator` MUST be configured to enable GPU scheduling/isolation. For `Capacity Scheduler`, use following property to configure `DominantResourceCalculator` (In `capacity-scheduler.xml`): | Property | Default value | | | | | yarn.scheduler.capacity.resource-calculator | org.apache.hadoop.yarn.util.resource.DominantResourceCalculator | ``` <property> <name>yarn.nodemanager.resource-plugins</name> <value>yarn.io/gpu</value> </property> ``` This is to enable GPU isolation module on NodeManager side. By default, YARN will automatically detect and config GPUs when above config is set. Following configs need to be set in `yarn-site.xml` only if admin has specialized requirements. 1) Allowed GPU Devices | Property | Default value | | | | | yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices | auto | Specify GPU devices which can be managed by YARN NodeManager (split by comma). Number of GPU devices will be reported to RM to make scheduling decisions. Set to auto (default) let YARN automatically discover GPU resource from system. Manually specify GPU devices if auto detect GPU device failed or admin only want subset of GPU devices managed by YARN. GPU device is identified by their minor device number and index. A common approach to get minor device number of GPUs is using `nvidia-smi -q` and search `Minor Number` output. When minor numbers are specified manually, admin needs to include indices of GPUs as well, format is `index:minornumber[,index:minornumber...]`. An example of manual specification is `0:0,1:1,2:2,3:4"`to allow YARN NodeManager to manage GPU devices with indices `0/1/2/3` and minor number `0/1/2/4`. numbers . 2) Executable to discover GPUs | Property | value | | | | | yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables | /absolute/path/to/nvidia-smi | When `yarn.nodemanager.resource.gpu.allowed-gpu-devices=auto` specified, YARN NodeManager needs to run GPU discovery binary (now only support `nvidia-smi`) to get GPU-related information. When value is empty (default), YARN NodeManager will try to locate discovery executable itself. An example of the config value is: `/usr/local/bin/nvidia-smi` 3) Docker Plugin Related Configs Following configs can be customized when user needs to run GPU applications inside Docker

They're not required if admin follows default installation/configuration of `nvidia-docker`. | Property | Default value | | | | | yarn.nodemanager.resource-plugins.gpu.docker-plugin | nvidia-docker-v1 | Specify docker command plugin for GPU. By default uses Nvidia docker V1.0, `nvidia-docker-v2` is available for V2.x. | Property | Default value | | | | | yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint | http://localhost:3476/v1.0/docker/cli | Specify end point of `nvidia-docker-plugin`. Please find documentation: https://github.com/NVIDIA/nvidia-docker/wiki For more details. 4) CGroups mount GPU isolation uses CGroup to do per-GPU device isolation. Following configs should be added to `yarn-site.xml` to automatically mount CGroup sub devices, otherwise admin has to manually create devices subfolder in order to use this feature. | Property | Default value | | | | | yarn.nodemanager.linux-container-executor.cgroups.mount | true | In general, following config needs to be added to `container-executor.cfg` ``` [gpu] module.enabled=true ``` When user needs to run GPU applications under non-Docker environment: ``` [cgroups] root=/sys/fs/cgroup yarn-hierarchy=yarn ``` When user needs to run GPU applications under Docker environment: 1) Add GPU related devices to docker section: Values separated by comma, you can get this by running `ls /dev/nvidia*` ``` [docker] docker.allowed.devices=/dev/nvidiactl,/dev/nvidia-uvm,/dev/nvidia-uvm-tools,/dev/nvidia1,/dev/nvidia0 ``` 2) Add `nvidia-docker` to volume-driver whitelist. ``` [docker] ... docker.allowed.volume-drivers ``` 3) Add `nvidia_driver_<version>` to readonly mounts whitelist. ``` [docker] ... docker.allowed.ro-mounts=nvidiadriver375.66 ``` 4) If use `nvidia-docker-v2` as gpu docker plugin, add `nvidia` to runtimes whitelist. ``` [docker] ... docker.allowed.runtimes=nvidia ``` Distributed shell currently support specify additional resource types other than memory and vcores. Run distributed shell without using docker container (Asks 2 tasks, each task has 3GB memory, 1 vcore, 2 GPU device resource): ``` yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \ -jar <path/to/hadoop-yarn-applications-distributedshell.jar> \ -shell_command /usr/local/nvidia/bin/nvidia-smi \ -container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \ -num_containers 2 ``` You should be able to see output like ``` Tue Dec 5 22:21:47 2017 +--+ | NVIDIA-SMI 375.66 Driver Version: 375.66 | |-+-+-+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla P100-PCIE... Off | 0000:04:00.0 Off | 0 | | N/A 30C P0 24W / 250W | 0MiB / 12193MiB | 0% Default | +-+-+-+ | 1 Tesla P100-PCIE... Off | 0000:82:00.0 Off | 0 | | N/A 34C P0 25W / 250W | 0MiB / 12193MiB | 0% Default | +-+-+-+ +--+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +--+ ``` For launched container task. You can also run distributed shell with Docker container. `YARNCONTAINERRUNTIMETYPE`/`YARNCONTAINERRUNTIMEDOCKER_IMAGE` must be specified to use docker container. ``` yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \ -jar <path/to/hadoop-yarn-applications-distributedshell.jar> \ -shellenv YARNCONTAINERRUNTIMETYPE=docker \ -shellenv YARNCONTAINERRUNTIMEDOCKER_IMAGE=<docker-image-name> \ -shell_command nvidia-smi \ -container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \ -num_containers 2 ```