Change:
This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: doesn't have our disclaimer for 3rd party content. Proposed Solution: Add a third party content disclaimer. See for an example. Page to Update: <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--Additional Information: /language en /help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : This is a Bug Report <!-- See for guidance on writing an actionable issue description. --Problem: doesn't have our disclaimer for 3rd party content. Add a third party content disclaimer. See for an example. <!--Kubernetes Version:--/language en /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Separately, it would be good if didn't include advice about installing the plugins. Instead, we can hyperlink to the vendors' own docs (where those exist).

Discussion:
{{< feature-state state="beta" for_k8s_version="v1.10" >}} <del> Kubernetes includes **experimental** support for managing AMD and NVIDIA GPUs </del> <ins> Kubernetes includes **experimental** support for managing GPUs </ins> (graphical processing units) across several nodes. <del> This page describes how users can consume GPUs across different Kubernetes versions and the current limitations. </del> <ins> This page describes how users can consume GPUs, and outlines some of the limitations in the implementation. </ins> <!-- body --> ## Using device plugins <del> Kubernetes implements {{< glossary_tooltip text="Device Plugins" term_id="device-plugin" >}} </del> <ins> Kubernetes implements {{< glossary_tooltip text="device plugins" term_id="device-plugin" >}} </ins> to let Pods access specialized hardware features such as GPUs. <ins> {{% thirdparty-content %}} </ins> As an administrator, you have to install GPU drivers from the corresponding hardware vendor on the nodes and run the corresponding device plugin from the <del> GPU vendor: </del> <ins> GPU vendor. Here are some links to vendors' instructions: </ins> <del> * [AMD](#deploying-amd-gpu-device-plugin) * [NVIDIA](#deploying-nvidia-gpu-device-plugin) </del> <ins> * [AMD](https://github.com/RadeonOpenCompute/k8s-device-plugin#deployment) * [NVIDIA](https://github.com/NVIDIA/k8s-device-plugin#quick-start) </ins> <del> When the above conditions are true, Kubernetes will expose `amd.com/gpu` or `nvidia.com/gpu` as a schedulable resource. </del> <ins> Once you have installed the plugin, your cluster exposes a custom schedulable resource such as `amd.com/gpu` or `nvidia.com/gpu`. </ins> You can consume these GPUs from your containers by requesting <del> `<vendor>.com/gpu` the same way you request `cpu` or `memory`. However, there are some limitations in how you specify the resource requirements when using GPUs: - GPUs are only supposed to be specified in the `limits` section, which means: * You can specify GPU `limits` without specifying `requests` because Kubernetes will use the limit as the request value by default. * You can specify GPU in both `limits` and `requests` but these two values must be equal. * You cannot specify GPU `requests` without specifying `limits`. - Containers (and Pods) do not share GPUs. There's no overcommitting of GPUs. - Each container can request one or more GPUs. It is not possible to request a fraction of a GPU. Here's an example: </del> <ins> the custom GPU resource, the same way you request `cpu` or `memory`. However, there are some limitations in how you specify the resource requirements for custom devices. GPUs are only supposed to be specified in the `limits` section, which means: * You can specify GPU `limits` without specifying `requests`, because Kubernetes will use the limit as the request value by default. * You can specify GPU in both `limits` and `requests` but these two values must be equal. * You cannot specify GPU `requests` without specifying `limits`. Here's an example manifest for a Pod that requests a GPU: </ins> ```yaml apiVersion: v1 kind: Pod metadata: <del> name: cuda-vector-add </del> <ins> name: example-vector-add </ins> spec: restartPolicy: OnFailure containers: <del> - name: cuda-vector-add # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile image: "registry.k8s.io/cuda-vector-add:v0.1" </del> <ins> - name: example-vector-add image: "registry.example/example-vector-add:v42" </ins> resources: limits: <del> nvidia.com/gpu: 1 # requesting 1 GPU ``` ### Deploying AMD GPU device plugin The [official AMD GPU device plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin) has the following requirements: - Kubernetes nodes have to be pre-installed with AMD GPU Linux driver. To deploy the AMD device plugin once your cluster is running and the above requirements are satisfied: ```shell kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/v1.10/k8s-ds-amdgpu-dp.yaml ``` You can report issues with this third-party device plugin by logging an issue in [RadeonOpenCompute/k8s-device-plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin). ### Deploying NVIDIA GPU device plugin There are currently two device plugin implementations for NVIDIA GPUs: #### Official NVIDIA GPU device plugin The [official NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin) has the following requirements: - Kubernetes nodes have to be pre-installed with NVIDIA drivers. - Kubernetes nodes have to be pre-installed with [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker) - Kubelet must use Docker as its container runtime - `nvidia-container-runtime` must be configured as the [default runtime](https://github.com/NVIDIA/k8s-device-plugin#preparing-your-gpu-nodes) for Docker, instead of runc. - The version of the NVIDIA drivers must match the constraint ~= 384.81. To deploy the NVIDIA device plugin once your cluster is running and the above requirements are satisfied: ```shell kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta4/nvidia-device-plugin.yml ``` You can report issues with this third-party device plugin by logging an issue in [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin). #### NVIDIA GPU device plugin used by GCE The [NVIDIA GPU device plugin used by GCE](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu) doesn't require using nvidia-docker and should work with any container runtime that is compatible with the Kubernetes Container Runtime Interface (CRI). It's tested on [Container-Optimized OS](https://cloud.google.com/container-optimized-os/) and has experimental code for Ubuntu from 1.9 onwards. You can use the following commands to install the NVIDIA drivers and device plugin: ```shell # Install NVIDIA drivers on Container-Optimized OS: kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/stable/daemonset.yaml # Install NVIDIA drivers on Ubuntu (experimental): kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/stable/nvidia-driver-installer/ubuntu/daemonset.yaml # Install the device plugin: kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.14/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml </del> <ins> gpu-vendor.example/example-gpu: 1 # requesting 1 GPU </ins> ``` <del> You can report issues with using or deploying this third-party device plugin by logging an issue in [GoogleCloudPlatform/container-engine-accelerators](https://github.com/GoogleCloudPlatform/container-engine-accelerators). Google publishes its own [instructions](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus) for using NVIDIA GPUs on GKE . </del> ## Clusters containing different types of GPUs If different nodes in your cluster have different types of GPUs, then you

```shell # Label your nodes with the accelerator type they have. <del> kubectl label nodes <node-with-k80> accelerator=nvidia-tesla-k80 kubectl label nodes <node-with-p100> accelerator=nvidia-tesla-p100 </del> <ins> kubectl label nodes node1 accelerator=example-gpu-x100 kubectl label nodes node2 accelerator=other-gpu-k915 </ins> ``` <ins> That label key `accelerator` is just an example; you can use a different label key if you prefer. </ins> ## Automatic node labelling {#node-labeller} If you're using AMD GPU devices, you can deploy

``` ``` <del> Name:               cluster-node-23 Roles:              <none> Labels:             beta.amd.com/gpu.cu-count.64=1 beta.amd.com/gpu.device-id.6860=1 beta.amd.com/gpu.family.AI=1 beta.amd.com/gpu.simd-count.256=1 beta.amd.com/gpu.vram.16G=1 beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/hostname=cluster-node-23 Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 … </del> <ins> Name:               cluster-node-23 Roles:              <none> Labels:             beta.amd.com/gpu.cu-count.64=1 beta.amd.com/gpu.device-id.6860=1 beta.amd.com/gpu.family.AI=1 beta.amd.com/gpu.simd-count.256=1 beta.amd.com/gpu.vram.16G=1 kubernetes.io/arch=amd64 kubernetes.io/os=linux kubernetes.io/hostname=cluster-node-23 Annotations:        node.alpha.kubernetes.io/ttl: 0 … </ins> ``` With the Node Labeller in use, you can specify the GPU type in the Pod spec:

resources: limits: nvidia.com/gpu: 1 <del> nodeSelector: accelerator: nvidia-tesla-p100 # or nvidia-tesla-k80 etc. </del> <ins> affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: – matchExpressions: – key: beta.amd.com/gpu.family.AI # Arctic Islands GPU family operator: Exist </ins> ``` <del> This will ensure that the Pod will be scheduled to a node that has the GPU type </del> <ins> This ensures that the Pod will be scheduled to a node that has the GPU type </ins> you specified.