# Schedule GPUs

Schedule GPUsConfigure and schedule GPUs for use as a resource by nodes in a cluster.FEATURE STATE: Kubernetes v1.26 [stable]Kubernetes includes stable support for managing AMD and NVIDIA GPUs (graphical processing units) across different nodes in your cluster, using device plugins.This page describes how users can consume GPUs, and outlines some of the limitations in the implementation.Using device pluginsKubernetes implements device plugins to let Pods access specialized hardware features such as GPUs.Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.As an administrator, you have to install GPU drivers from the corresponding hardware vendor on the nodes and run the corresponding device plugin from the GPU vendor. Here are some links to vendors' instructions:AMDIntelNVIDIAOnce you have installed the plugin, your cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu.You can consume these GPUs from your containers by requesting the custom GPU resource, the same way you request cpu or memory. However, there are some limitations in how you specify the resource requirements for custom devices.GPUs are only supposed to be specified in the limits section, which means:You can specify GPU limits without specifying requests, because Kubernetes will use the limit as the request value by default.You can specify GPU in both limits and requests but these two values must be equal.You cannot specify GPU requests without specifying limits.Here's an example manifest for a Pod that requests a GPU:apiVersion: v1 kind: Pod metadata: name: example-vector-add spec: restartPolicy: OnFailure containers: - name: example-vector-add image: "registry.example/example-vector-add:v42" resources: limits: gpu-vendor.example/example-gpu: 1 # requesting 1 GPU Manage clusters with different types of GPUsIf different nodes in your cluster have different types of GPUs, then you can use Node Labels and Node Selectors to schedule pods to appropriate nodes.For example:# Label your nodes with the accelerator type they have. kubectl label nodes node1 accelerator=example-gpu-x100 kubectl label nodes node2 accelerator=other-gpu-k915 That label key accelerator is just an example; you can use a different label key if you prefer.Automatic node labellingAs an administrator, you can automatically discover and label all your GPU enabled nodes by deploying Kubernetes Node Feature Discovery (NFD). NFD detects the hardware features that are available on each node in a Kubernetes cluster. Typically, NFD is configured to advertise those features as node labels, but NFD can also add extended resources, annotations, and node taints. NFD is compatible with all supported versions of Kubernetes. By default NFD create the feature labels for the detected features. Administrators can leverage NFD to also taint nodes with specific features, so that only pods that request those features can be scheduled on those nodes.You also need a plugin for NFD that adds appropriate labels to your nodes; these might be generic labels or they could be vendor specific. Your GPU vendor may provide a third party plugin for NFD; check their documentation for more details.apiVersion: v1 kind: Pod metadata: name: example-vector-add spec: restartPolicy: OnFailure # You can use Kubernetes node affinity to schedule this Pod onto a node # that provides the kind of GPU that its container needs in order to work affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "gpu.gpu-vendor.example/installed-memory" operator: Gt # (greater than) values: ["40535"] - key: "feature.node.kubernetes.io/pci-10.present" # NFD Feature label values: ["true"] # (optional) only schedule on nodes with PCI device 10 containers: - name: example-vector-add image: "registry.example/example-vector-add:v42" resources: limits: gpu-vendor.example/example-gpu: 1 # requesting 1 GPUGPU vendor implementationsIntelNVIDIAItems on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.