Chapter 9. Configuring virtual GPUs for instances To support GPU-based rendering on your instances, you can define and manage virtual GPU (vGPU) resources according to your available physical GPU devices and your hypervisor type. You can use this configuration to effectively spread the rendering workloads between all your physical GPU devices , and to control the scheduling of your vGPU-enabled instances. To enable vGPU in the Compute service (nova), you must perform the following tasks: Identify the nodes on which you want to configure vGPUs. Retrieve the PCI address for each physical GPU on each Compute node, or for each SR-IOV virtual function (VF) if the GPU supports SR-IOV. Configure the GPU profiles on each Compute node. Each instance hosted on the configured Compute nodes can support GPU workloads with virtual GPU devices that correspond to the physical GPU devices. The Compute service tracks the number of vGPU devices that are available for each GPU profile you define on each host. The Compute service schedules instances to these hosts, attaches the devices, and monitors the vGPU usage. When an instance is deleted, the Compute service adds the vGPU devices back to the available pool. Important Red Hat enables the use of NVIDIA vGPU in RHOSO without the requirement for support exceptions. However, Red Hat does not provide technical support for the NVIDIA vGPU drivers. The NVIDIA vGPU drivers are shipped and supported by NVIDIA. You require an NVIDIA Certified Support Services subscription to obtain NVIDIA Enterprise Support for NVIDIA vGPU software. For issues that result from the use of NVIDIA vGPUs where you are unable to reproduce the issue on a supported component, the following support policies apply: When Red Hat does not suspect that the third-party component is involved in the issue, the normal Scope of Support and Red Hat SLA apply. When Red Hat suspects that the third-party component is involved in the issue, the customer will be directed to NVIDIA in line with the Red Hat third party support and certification policies . For more information, see the Knowledge Base article Obtaining Support from NVIDIA . 9.1. Supported configurations and limitations Supported GPU cards For a list of supported NVIDIA GPU cards, see Virtual GPU Software Supported Products on the NVIDIA website. Limitations when using vGPU devices Each instance can use only one vGPU resource. Live migration of vGPU instances between hosts is not supported. Evacuation of vGPU instances is not supported. If you need to reboot the Compute node that hosts the vGPU instances, the vGPUs are not automatically reassigned to the recreated instances. You must either cold migrate the instances before you reboot the Compute node, or manually allocate each vGPU to the correct instance after reboot. To manually allocate each vGPU, you must retrieve the mdev UUID from the instance XML for each vGPU instance that runs on the Compute node before you reboot. You can use the following command to discover the mdev UUID for each instance: Replace <instance_name> with the libvirt instance name, OS-EXT-SRV-ATTR:instance_name , returned in a /servers request to the Compute API. By default, vGPU types on Compute hosts are not exposed to API users. To expose the vGPU types on Compute hosts to API users, you must configure resource provider traits and create flavors that require the traits. Alternatively, if you only have one vGPU type, you can grant access by adding the hosts to a host aggregate. For more information, see Creating and managing host aggregates . If you use NVIDIA accelerator hardware, you must comply with the NVIDIA licensing requirements. For example, NVIDIA vGPU GRID requires a licensing server. For more information about the NVIDIA licensing requirements, see NVIDIA License Server Release Notes on the NVIDIA website. 9.2. Preparing to configure the Compute service for vGPU Before you configure the Compute service for vGPU, you must prepare the data plane nodes that you want to use for vGPU and you must download and install the NVIDIA device driver. Procedure Access the remote shell for openstackclient : Identify a node that you want to use for vGPU: Retrieve the IP address of the Compute node that you want to use for vGPU: Use SSH to connect to the data plane node: Create the file /etc/modprobe.d/blacklist-nouveau.conf . Disable the nouveau driver by adding the following configuration to blacklist-nouveau.conf : Regenerate the initramfs : Download and install the NVIDIA driver from the NVIDIA portal. For more information, see NVIDIA DOCS HUB . Reboot the node: Repeat this procedure for all nodes that you want to allocate for vGPU instances. 9.3. Configuring the Compute service for vGPU You need to retrieve and assign the vGPU type that corresponds to the physical GPU device in your environment and configure a vGPU type. Note You can configure only whole node sets. Reconfiguring a subset of the nodes within a node set is not supported. If you need to reconfigure a subset of nodes within a node set, you must scale the node set down, and create a new node set from the previously removed nodes. Prerequisites The oc command line tool is installed on your workstation. You are logged in to Red Hat OpenStack Services on OpenShift (RHOSO) as a user with cluster-admin privileges. You have selected the OpenStackDataPlaneNodeSet CR that defines the nodes that you can configure vGPU on. For more information about creating an OpenStackDataPlaneNodeSet CR, see Creating an OpenStackDataPlaneNodeSet CR with pre-provisioned nodes in the Deploying Red Hat OpenStack Services on OpenShift guide. Procedure Virtual GPUs are mediated devices. Retrieve the PCI address for each device that can create mediated devices on each Compute node: Note The PCI address of the GPU - or the GPU SR-IOV virtual function (VF) that can create vGPUs - is used as the device driver directory name, for example, 0000:84:00.0. In this procedure, the vGPU-capable resource is called an mdev device. Note Recent generations of NVIDIA cards now support SR-IOV. Refer to the NVIDIA documentation to discover if your GPU is SR-IOV-capable. Review the supported mdev types for each available pGPU device on each Compute node to discover the available vGPU types: Replace <mdev_device> with the PCI address for the mdev device, for example, 0000:84:00.0. For example, the following Compute node has 4 pGPUs, and each pGPU supports the same 11 vGPU types: Create or update the ConfigMap CR named nova-extra-config.yaml and set the values of the parameters under [devices] : For more information about creating ConfigMap objects, see Creating and using config maps . Optional: To configure more than one vGPU type, map the supported vGPU types to the pGPUs: The nvidia-35 vGPU type is supported by the pGPUs that are in the PCI addresses 0000:84:00.0 and 0000:85:00.0. The nvidia-36 vGPU type is supported only by the pGPUs that are in the PCI address 0000:86:00.0. Create a new OpenStackDataPlaneDeployment CR to configure the services on the data plane nodes and deploy the data plane, and save it to a file named compute_vgpu_deploy.yaml on your workstation: In the compute_vgpu_deploy.yaml CR, specify nodeSets to include all the OpenStackDataPlaneNodeSet CRs you want to deploy. Ensure that you include the OpenStackDataPlaneNodeSet CR that you selected as a prerequisite. That OpenStackDataPlaneNodeSet CR defines the nodes that you want to want to use for vGPU. Warning If your deployment has more than one node set, changes to the nova-extra-config.yaml ConfigMap might directly affect more than one node set, depending on how the node sets and the DataPlaneServices are configured. To check if a node set uses the nova-extra-config ConfigMap and therefore will be affected by the reconfiguration, complete the following steps: Check the services list of the node set and find the name of the DataPlaneService that points to nova. Ensure that the value of the edpmServiceType field of the DataPlaneService is set to nova . If the dataSources list of the DataPlaneService contains a configMapRef named nova-extra-config , then this node set uses this ConfigMap and therefore will be affected by the configuration changes in this ConfigMap . If some of the node sets that are affected should not be reconfigured, you must create a new DataPlaneService pointing to a separate ConfigMap for these node sets. Replace <nodeSet_name> with the names of the `OpenStackDataPlaneNodeSet`CRs that you want to include in your data plane deployment. Save the compute_vgpu_deploy.yaml deployment file. Deploy the data plane: Verify that the data plane is deployed: Tip Append the -w option to the end of the get command to track deployment progress. Access the remote shell for openstackclient and verify that the deployed Compute nodes are visible on the control plane: Optional: Enable SR-IOV VFs of the GPUs. For more information, see Preparing virtual function for SRIOV vGPU on the NVIDIA DOCS HUB.

Commands:
virsh dumpxml <instance_name> | grep mdev
oc rsh openstackclient
openstack hypervisor list
ssh <node_ipaddress>
blacklist nouveau options nouveau modeset=0
dracut --force grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline
sudo reboot
ls /sys/class/mdev_bus/
ls /sys/class/mdev_bus/<mdev_device>/mdev_supported_types
ls /sys/class/mdev_bus/0000:84:00.0/mdev_supported_types: NVIDIA-35 NVIDIA-36 NVIDIA-37 NVIDIA-38 NVIDIA-39 NVIDIA-40 NVIDIA-41 NVIDIA-42 NVIDIA-43 NVIDIA-44 NVIDIA-45 ls /sys/class/mdev_bus/0000:85:00.0/mdev_supported_types: NVIDIA-35 NVIDIA-36 NVIDIA-37 NVIDIA-38 NVIDIA-39 NVIDIA-40 NVIDIA-41 NVIDIA-42 NVIDIA-43 NVIDIA-44 NVIDIA-45 ls /sys/class/mdev_bus/0000:86:00.0/mdev_supported_types: NVIDIA-35 NVIDIA-36 NVIDIA-37 NVIDIA-38 NVIDIA-39 NVIDIA-40 NVIDIA-41 NVIDIA-42 NVIDIA-43 NVIDIA-44 NVIDIA-45 ls /sys/class/mdev_bus/0000:87:00.0/mdev_supported_types: NVIDIA-35 NVIDIA-36 NVIDIA-37 NVIDIA-38 NVIDIA-39 NVIDIA-40 NVIDIA-41 NVIDIA-42 NVIDIA-43 NVIDIA-44 NVIDIA-45
apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 34-nova-vgpu.conf: | [devices] enabled_mdev_types = nvidia-35, nvidia-36
[devices] enabled_mdev_types = nvidia-35, nvidia-36 [mdev_nvidia-35] device_addresses = 0000:84:00.0,0000:85:00.0 [vgpu_nvidia-36] device_addresses = 0000:86:00.0
apiVersion: core.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: compute-vgpu
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: compute-vgpu spec: nodeSets: - openstack-edpm - compute-vgpu - - <nodeSet_name>
oc create -f compute_vgpu_deploy.yaml
oc get openstackdataplanenodeset NAME STATUS MESSAGE compute-vgpu True Deployed
oc rsh -n openstack openstackclient openstack hypervisor list