OpenShift AI - GPU Node


#### OpenShift AI

Red Hat® OpenShift® AI とは、柔軟でスケーラブルな人工知能 (AI) および機械学習 (ML) プラットフォームです。このプラットフォームにより、企業はハイブリッドクラウド環境全体で AI 対応アプリケーションを大規模に作成および提供できるようになります。

OpenShift AI はオープンソース・テクノロジーを使用して構築されており、実験、モデル提供、革新的なアプリケーションの提供のための、信頼性と一貫性に優れた運用機能を提供します。

https://www.redhat.com/ja/technologies/cloud-computing/openshift/openshift-ai

#### GPU Node
AI 学習等で使用する GPU Node を設定します。
ここでは、Red Hat OpenShift on IBM Cloud (ROKS) 4.14 の環境に、以下のドキュメントを参考にして OpenShift AI 導入後に GPU Node を設定してみます。

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.13/html-single/installing_and_uninstalling_openshift_ai_self-managed/index#enabling-nvidia-gpus_install

上記に記載のある通り、以下の NVIDIA のドキュメントを参照してみます。

https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/index.html

##### Installing the Node Feature Discovery (NFD) Operator
Operator Hub から `Node Feature Discovery (NFD) Operator` をデフォルト設定で導入します。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/7f6c1e83-279a-6d75-11cb-777e0c8c12f6.png)

完了すると `openshift-nfd` Project に以下のようなリソースが配置されます。
```shell
$ oc get all -n openshift-nfd
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                                          READY   STATUS    RESTARTS         AGE
pod/nfd-controller-manager-585b4785d8-dsr2t   2/2     Running   0                22h
pod/nfd-master-66c886fb64-bkt5m               1/1     Running   0                19d
pod/nfd-worker-4bljq                          1/1     Running   9 (19d ago)      55d
pod/nfd-worker-7nnwd                          1/1     Running   9 (19d ago)      55d
pod/nfd-worker-clntr                          1/1     Running   9 (19d ago)      55d
pod/nfd-worker-dr4fr                          1/1     Running   9 (19d ago)      55d
pod/nfd-worker-p7qxd                          1/1     Running   15 (7d19h ago)   55d
pod/nfd-worker-ph9kq                          1/1     Running   9 (19d ago)      55d
pod/nfd-worker-q8qqv                          1/1     Running   9 (19d ago)      55d
pod/nfd-worker-slf59                          1/1     Running   2 (6d1h ago)     6d1h
pod/nfd-worker-vg6jl                          1/1     Running   10 (10d ago)     55d
pod/nfd-worker-w49gp                          1/1     Running   12 (19d ago)     55d
pod/nfd-worker-w8tgt                          1/1     Running   9 (19d ago)      55d

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/nfd-controller-manager-metrics-service   ClusterIP   192.16.206.100   <none>        8443/TCP    55d
service/nfd-master                               ClusterIP   192.16.104.236   <none>        12000/TCP   55d

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nfd-worker   11        11        11      11           11          <none>          55d

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nfd-controller-manager   1/1     1            1           55d
deployment.apps/nfd-master               1/1     1            1           55d

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/nfd-controller-manager-585b4785d8   1         1         1       22h
replicaset.apps/nfd-controller-manager-64b8dfcb6b   0         0         0       55d
replicaset.apps/nfd-master-66c886fb64               1         1         1       55d
```

NFD Operator の NodeFeatureDiscovery Tab でインスタンスを、デフォルト設定で作成します。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/bb5271a3-87cf-924a-ba72-3782390f0e58.png)

GPU Node で `feature.node.kubernetes.io/pci-10de.present` Label が `true` に設定されていることを確認します。
今回の環境では `INSTANCE-TYPE (=Flavor)` が `gx3.24x120.l40s` と `gx3.16x80.l4` の Node に設定されている事が分かります。
```shell
$ oc get node -L "feature.node.kubernetes.io/pci-10de.present" -L "beta.kubernetes.io/instance-type" -L "nvidia.com/gpu.product" -L "nvidia.com/mig.capable"
NAME           STATUS   ROLES           AGE    VERSION            PCI-10DE.PRESENT   INSTANCE-TYPE   GPU.PRODUCT   MIG.CAPABLE
99.888.0.16    Ready    master,worker   91d    v1.27.15+6147456                      cx2.16x32
99.888.0.17    Ready    master,worker   91d    v1.27.15+6147456                      cx2.16x32
99.888.0.4     Ready    master,worker   140d   v1.27.13+048520e                      mx2.8x64
99.888.0.5     Ready    master,worker   140d   v1.27.13+048520e                      mx2.8x64
99.888.0.6     Ready    master,worker   140d   v1.27.13+048520e                      mx2.8x64
99.888.128.4   Ready    master,worker   9d     v1.27.16+03a907c   true               gx3.16x80.l4    NVIDIA-L4     false
99.888.128.5   Ready    master,worker   91d    v1.27.15+6147456                      cx2.16x32
99.888.128.6   Ready    master,worker   91d    v1.27.15+6147456                      cx2.16x32
99.888.64.4    Ready    master,worker   91d    v1.27.15+6147456                      cx2.16x32
99.888.64.5    Ready    master,worker   91d    v1.27.15+6147456                      cx2.16x32
```
GPU Node で NVIDIA GPU を確認してみます。
```shell
sh-4.4# lspci | grep -i nvidia
04:01.0 3D controller: NVIDIA Corporation AD104GL [L4] (rev a1)
```
##### Installing the NVIDIA GPU Operator
Operator Hub から `NVIDIA GPU Operator` をデフォルト設定で導入します。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/116eb1b7-3353-8a8e-90d6-56bc7797c382.png)

完了すると `nvidia-gpu-operator` Project に以下のようなリソースが配置されます。
```shell
$ oc get all -n nvidia-gpu-operator
NAME                                           READY   STATUS      RESTARTS   AGE
pod/gpu-feature-discovery-vpm77                1/1     Running     0          7d19h
pod/gpu-feature-discovery-x6sl9                1/1     Running     0          6d2h
pod/gpu-operator-789c7877fc-7n5vj              1/1     Running     0          19d
pod/nvidia-container-toolkit-daemonset-c9ds5   1/1     Running     0          6d2h
pod/nvidia-container-toolkit-daemonset-gmnpn   1/1     Running     0          7d19h
pod/nvidia-cuda-validator-8jhkk                0/1     Completed   0          6d2h
pod/nvidia-cuda-validator-ltwhz                0/1     Completed   0          7d19h
pod/nvidia-dcgm-exporter-4dndg                 1/1     Running     0          6d2h
pod/nvidia-dcgm-exporter-pd58s                 1/1     Running     0          7d19h
pod/nvidia-dcgm-p4bmn                          1/1     Running     0          6d2h
pod/nvidia-dcgm-rsk5g                          1/1     Running     0          7d19h
pod/nvidia-device-plugin-daemonset-r58dx       1/1     Running     0          6d2h
pod/nvidia-device-plugin-daemonset-sxqct       1/1     Running     0          7d19h
pod/nvidia-driver-daemonset-5hv4v              1/1     Running     0          6d2h
pod/nvidia-driver-daemonset-9qq6x              1/1     Running     1          55d
pod/nvidia-node-status-exporter-fs6v5          1/1     Running     0          6d2h
pod/nvidia-node-status-exporter-sjc9n          1/1     Running     1          55d
pod/nvidia-operator-validator-f5pr9            1/1     Running     0          7d19h
pod/nvidia-operator-validator-pcdvf            1/1     Running     0          6d2h

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/gpu-operator                  ClusterIP   192.16.247.242   <none>        8080/TCP   55d
service/nvidia-dcgm                   ClusterIP   192.16.157.103   <none>        5555/TCP   55d
service/nvidia-dcgm-exporter          ClusterIP   192.16.182.25    <none>        9400/TCP   55d
service/nvidia-node-status-exporter   ClusterIP   192.16.173.104   <none>        8000/TCP   55d

NAME                                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                          AGE
daemonset.apps/gpu-feature-discovery                     2         2         2       2            2           nvidia.com/gpu.deploy.gpu-feature-discovery=true                       55d
daemonset.apps/nvidia-container-toolkit-daemonset        2         2         2       2            2           nvidia.com/gpu.deploy.container-toolkit=true                           55d
daemonset.apps/nvidia-dcgm                               2         2         2       2            2           nvidia.com/gpu.deploy.dcgm=true                                        55d
daemonset.apps/nvidia-dcgm-exporter                      2         2         2       2            2           nvidia.com/gpu.deploy.dcgm-exporter=true                               55d
daemonset.apps/nvidia-device-plugin-daemonset            2         2         2       2            2           nvidia.com/gpu.deploy.device-plugin=true                               55d
daemonset.apps/nvidia-device-plugin-mps-control-daemon   0         0         0       0            0           nvidia.com/gpu.deploy.device-plugin=true,nvidia.com/mps.capable=true   55d
daemonset.apps/nvidia-driver-daemonset                   2         2         2       2            2           nvidia.com/gpu.deploy.driver=true                                      55d
daemonset.apps/nvidia-mig-manager                        0         0         0       0            0           nvidia.com/gpu.deploy.mig-manager=true                                 55d
daemonset.apps/nvidia-node-status-exporter               2         2         2       2            2           nvidia.com/gpu.deploy.node-status-exporter=true                        55d
daemonset.apps/nvidia-operator-validator                 2         2         2       2            2           nvidia.com/gpu.deploy.operator-validator=true                          55d

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gpu-operator   1/1     1            1           55d

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/gpu-operator-789c7877fc   1         1         1       55d
```

例えば `nvidia-driver-daemonset-5hv4v` Pod を確認してみると、GPU Node の `/run/nvidia` を Pod の `/run/nvidia` に Mount していることが分かります。

On Pod
```shell
$ oc exec -ti nvidia-driver-daemonset-5hv4v -- ls -la /run/nvidia
total 20
drwxr-xr-x. 6 root root  140 Oct 15 04:56 .
drwxr-xr-x. 1 root root 4096 Oct 15 04:59 ..
dr-xr-xr-x. 1 root root 4096 Oct 15 04:55 driver
drwxr-xr-x. 3 root root   60 Oct 15 05:00 mps
-rw-r--r--. 1 root root    6 Oct 15 04:56 nvidia-driver.pid
drwxr-xr-x. 2 root root   60 Oct 21 11:58 toolkit
drwxr-xr-x. 2 root root  140 Oct 21 11:59 validations
```
On GPU Node
```shell
sh-4.4# ls -la /run/nvidia
total 12
drwxr-xr-x.  6 root root  140 Oct 14 23:56 .
drwxr-xr-x. 49 root root 1300 Oct 21 04:55 ..
dr-xr-xr-x.  1 root root 4096 Oct 14 23:55 driver
drwxr-xr-x.  3 root root   60 Oct 15 00:00 mps
-rw-r--r--.  1 root root    6 Oct 14 23:56 nvidia-driver.pid
drwxr-xr-x.  2 root root   60 Oct 21 06:58 toolkit
drwxr-xr-x.  2 root root  140 Oct 21 06:59 validations
```
Pod 上では NVIDIA Driver に関する各種 Process が実行されていることが分かります。
```shell
[root@nvidia-driver-daemonset-5hv4v drivers]# ps -ef | grep $(cat /run/nvidia/nvidia-driver.pid)
root      26209  26197  0 Oct15 ?        00:00:00 /bin/bash -x /usr/local/bin/nvidia-driver init
root      53023  26209  0 Oct15 ?        00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep infinity
root     153810 151857  0 02:41 pts/0    00:00:00 grep --color=auto 26209

[root@nvidia-driver-daemonset-5hv4v drivers]# ps -ef | grep 26197
root      26197      1  0 Oct15 ?        00:00:00 /usr/bin/conmon -b /var/data/crioruntimestorage/overlay-containers/
root      26209  26197  0 Oct15 ?        00:00:00 /bin/bash -x /usr/local/bin/nvidia-driver init
root      52962  26197  0 Oct15 ?        00:00:01 nvidia-persistenced --persistence-mode
root     154453 151857  0 02:42 pts/0    00:00:00 grep --color=auto 26197

[root@nvidia-driver-daemonset-5hv4v drivers]# head -n 50 /usr/local/bin/nvidia-driver
#! /bin/bash -x
# Copyright (c) 2018-2020, NVIDIA CORPORATION. All rights reserved.

set -eu

RUN_DIR=/run/nvidia
PID_FILE=${RUN_DIR}/${0##*/}.pid
DRIVER_VERSION=${DRIVER_VERSION:?"Missing DRIVER_VERSION env"}
KERNEL_UPDATE_HOOK=/run/kernel/postinst.d/update-nvidia-driver
NUM_VGPU_DEVICES=0
NVIDIA_MODULE_PARAMS=()
NVIDIA_UVM_MODULE_PARAMS=()
NVIDIA_MODESET_MODULE_PARAMS=()
NVIDIA_PEERMEM_MODULE_PARAMS=()
TARGETARCH=${TARGETARCH:?"Missing TARGETARCH env"}
USE_HOST_MOFED="${USE_HOST_MOFED:-false}"
DNF_RELEASEVER=${DNF_RELEASEVER:-""}
RHEL_VERSION=${RHEL_VERSION:-""}
RHEL_MAJOR_VERSION=8

OPEN_KERNEL_MODULES_ENABLED=${OPEN_KERNEL_MODULES_ENABLED:-false}
[[ "${OPEN_KERNEL_MODULES_ENABLED}" == "true" ]] && KERNEL_TYPE=kernel-open || KERNEL_TYPE=kernel

DRIVER_ARCH=${TARGETARCH/amd64/x86_64} && DRIVER_ARCH=${DRIVER_ARCH/arm64/aarch64}
echo "DRIVER_ARCH is $DRIVER_ARCH"

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
source $SCRIPT_DIR/common.sh

_update_package_cache() {
    if [ "${PACKAGE_TAG:-}" != "builtin" ]; then
        echo "Updating the package cache..."
        if ! yum -q makecache; then
            echo "FATAL: failed to reach RHEL package repositories. "\
                 "Ensure that the cluster can access the proper networks."
            exit 1
        fi
    fi
}

_cleanup_package_cache() {
    if [ "${PACKAGE_TAG:-}" != "builtin" ]; then
        echo "Cleaning up the package cache..."
        rm -rf /var/cache/yum/*
    fi
}

_get_rhel_version_from_kernel() {
    local rhel_version_underscore rhel_version_arr
    rhel_version_underscore=$(echo "${KERNEL_VERSION}" | sed 's/.*el\([0-9]\+_[0-9]\+\).*/\1/g')

[root@nvidia-driver-daemonset-5hv4v drivers]# which nvidia-persistenced
/usr/bin/nvidia-persistenced
[root@nvidia-driver-daemonset-5hv4v drivers]# nvidia-persistenced --help

nvidia-persistenced:  version 550.90.07

  The NVIDIA Persistence Daemon.

  A tool for maintaining persistent driver state, specifically for use by the NVIDIA Linux driver.

  Copyright (C) 2013-2018 NVIDIA Corporation.


nvidia-persistenced [options]

  -v, --version
      Print the utility version and exit.

  -h, --help
      Print usage information for the command line options and exit.

  -V, --verbose
      Controls how much information is printed. By default, nvidia-persistenced will only print errors and warnings to syslog for unexpected events, as well as startup and shutdown notices. Specifying this
      flag will cause nvidia-persistenced to also print notices to syslog on state transitions, such as when persistence mode is enabled or disabled, and informational messages on startup and exit.

  -u USERNAME, --user=USERNAME
      Runs nvidia-persistenced with the user permissions of the user specified by the USERNAME argument. This user must have write access to the /var/run/nvidia-persistenced directory. If this directory does
      not exist, nvidia-persistenced will attempt to create it prior to changing the process user and group IDs. If this option is not given, nvidia-persistenced will not attempt to change the process user
      ID.

  -g GROUPNAME, --group=GROUPNAME
      Runs nvidia-persistenced with the group permissions of the group specified by the GROUPNAME argument. If both this option and the --user option are given, this option will take precedence when
      determining the group ID to use. If this option is not given, nvidia-persistenced will use the primary group ID of the user specified by the --user option argument. If the --user option is also not
      given, nvidia-persistenced will not attempt to change the process group ID.

  --persistence-mode, --no-persistence-mode
      By default, nvidia-persistenced starts with persistence mode enabled for all devices. Use '--no-persistence-mode' to force persistence mode off for all devices on startup.

  --uvm-persistence-mode, --no-uvm-persistence-mode
      UVM persistence mode is only supported on the single GPU confidential computing configuration. By default, nvidia-persistenced starts with UVM persistence mode disabled for all devices. Use
      '--uvm-persistence-mode' to force UVM persistence mode on for supported devices on startup.

  --nvidia-cfg-path=PATH
      The nvidia-cfg library is used to communicate with the NVIDIA kernel module to query and manage GPUs in the system. This library is required by nvidia-persistenced. This option tells nvidia-persistenced
      where to look for this library (in case it cannot find it on its own). This option should normally not be needed.

For more detailed usage information, please see the nvidia-persistenced manpage and the "Using the nvidia-persistenced Utility" section of the NVIDIA Linux Graphics Driver README.
```
別の `nvidia-container-toolkit-daemonset-c9ds5` Pod を確認してみると、NVIDIA Driver の Validation Check と `nvidia-toolkit` の処理を行っていることが分かります。

```shell
$ oc exec -ti nvidia-container-toolkit-daemonset-c9ds5 -- cat -n /usr/bin/entrypoint.sh
Defaulted container "nvidia-container-toolkit-ctr" out of: nvidia-container-toolkit-ctr, driver-validation (init)
     1  #!/bin/bash
     2
     3  until [[ -f /run/nvidia/validations/driver-ready ]]
     4  do
     5    echo "waiting for the driver validations to be ready..."
     6    sleep 5
     7  done
     8
     9  set -o allexport
    10  cat /run/nvidia/validations/driver-ready
    11  . /run/nvidia/validations/driver-ready
    12
    13  #
    14  # The below delay is a workaround for an issue affecting some versions
    15  # of containerd starting with 1.6.9. Staring with containerd 1.6.9 we
    16  # started seeing the toolkit container enter a crashloop whereby it
    17  # would recieve a SIGTERM shortly after restarting containerd.
    18  #
    19  # Refer to the commit message where this workaround was implemented
    20  # for additional details:
    21  #   https://github.com/NVIDIA/gpu-operator/commit/963b8dc87ed54632a7345c1fcfe842f4b7449565
    22  #
    23  sleep 5
    24
    25  exec nvidia-toolkit
```
NVIDIA GPU Operator の ClusterPolicy Tab でインスタンスを、デフォルト設定で作成します。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/c5c2fc66-2dbb-777e-cbfd-b2a74d2af0b0.png)

以上で、基本的な GPU Node の設定は完了です。

##### Node Labels
今回導入した2つの Operator は非常に多くの Node Label を設定します。これらを使用する事で詳細な確認が可能です。

```shell
feature.node.kubernetes.io/cpu-cpuid.ADX
feature.node.kubernetes.io/cpu-cpuid.AESNI
feature.node.kubernetes.io/cpu-cpuid.AMXBF16
feature.node.kubernetes.io/cpu-cpuid.AMXINT8
feature.node.kubernetes.io/cpu-cpuid.AMXTILE
feature.node.kubernetes.io/cpu-cpuid.AVX
feature.node.kubernetes.io/cpu-cpuid.AVX2
feature.node.kubernetes.io/cpu-cpuid.AVX512BF16
feature.node.kubernetes.io/cpu-cpuid.AVX512BITALG
feature.node.kubernetes.io/cpu-cpuid.AVX512BW
feature.node.kubernetes.io/cpu-cpuid.AVX512CD
feature.node.kubernetes.io/cpu-cpuid.AVX512DQ
feature.node.kubernetes.io/cpu-cpuid.AVX512F
feature.node.kubernetes.io/cpu-cpuid.AVX512FP16
feature.node.kubernetes.io/cpu-cpuid.AVX512IFMA
feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI
feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI2
feature.node.kubernetes.io/cpu-cpuid.AVX512VL
feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI
feature.node.kubernetes.io/cpu-cpuid.AVX512VPOPCNTDQ
feature.node.kubernetes.io/cpu-cpuid.AVXVNNI
feature.node.kubernetes.io/cpu-cpuid.CLDEMOTE
feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8
feature.node.kubernetes.io/cpu-cpuid.FMA3
feature.node.kubernetes.io/cpu-cpuid.FSRM
feature.node.kubernetes.io/cpu-cpuid.FXSR
feature.node.kubernetes.io/cpu-cpuid.FXSROPT
feature.node.kubernetes.io/cpu-cpuid.GFNI
feature.node.kubernetes.io/cpu-cpuid.HYPERVISOR
feature.node.kubernetes.io/cpu-cpuid.IA32_ARCH_CAP
feature.node.kubernetes.io/cpu-cpuid.IBPB
feature.node.kubernetes.io/cpu-cpuid.IBRS
feature.node.kubernetes.io/cpu-cpuid.LAHF
feature.node.kubernetes.io/cpu-cpuid.MD_CLEAR
feature.node.kubernetes.io/cpu-cpuid.MOVBE
feature.node.kubernetes.io/cpu-cpuid.MOVDIR64B
feature.node.kubernetes.io/cpu-cpuid.MOVDIRI
feature.node.kubernetes.io/cpu-cpuid.OSXSAVE
feature.node.kubernetes.io/cpu-cpuid.SERIALIZE
feature.node.kubernetes.io/cpu-cpuid.SHA
feature.node.kubernetes.io/cpu-cpuid.SPEC_CTRL_SSBD
feature.node.kubernetes.io/cpu-cpuid.STIBP
feature.node.kubernetes.io/cpu-cpuid.SYSCALL
feature.node.kubernetes.io/cpu-cpuid.SYSEE
feature.node.kubernetes.io/cpu-cpuid.TSXLDTRK
feature.node.kubernetes.io/cpu-cpuid.VAES
feature.node.kubernetes.io/cpu-cpuid.VMX
feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ
feature.node.kubernetes.io/cpu-cpuid.WAITPKG
feature.node.kubernetes.io/cpu-cpuid.WBNOINVD
feature.node.kubernetes.io/cpu-cpuid.X87
feature.node.kubernetes.io/cpu-cpuid.XGETBV1
feature.node.kubernetes.io/cpu-cpuid.XSAVE
feature.node.kubernetes.io/cpu-cpuid.XSAVEC
feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT
feature.node.kubernetes.io/cpu-cpuid.XSAVES
feature.node.kubernetes.io/cpu-hardware_multithreading
feature.node.kubernetes.io/cpu-model.family
feature.node.kubernetes.io/cpu-model.id
feature.node.kubernetes.io/cpu-model.vendor_id
feature.node.kubernetes.io/kernel-config.NO_HZ
feature.node.kubernetes.io/kernel-config.NO_HZ_FULL
feature.node.kubernetes.io/kernel-selinux.enabled
feature.node.kubernetes.io/kernel-version.full
feature.node.kubernetes.io/kernel-version.major
feature.node.kubernetes.io/kernel-version.minor
feature.node.kubernetes.io/kernel-version.revision
feature.node.kubernetes.io/memory-numa
feature.node.kubernetes.io/pci-1013.present
feature.node.kubernetes.io/pci-10de.present
feature.node.kubernetes.io/pci-1af4.present
feature.node.kubernetes.io/system-os_release.ID
feature.node.kubernetes.io/system-os_release.VERSION_ID
feature.node.kubernetes.io/system-os_release.VERSION_ID.major
feature.node.kubernetes.io/system-os_release.VERSION_ID.minor
nvidia.com/cuda.driver-version.full
nvidia.com/cuda.driver-version.major
nvidia.com/cuda.driver-version.minor
nvidia.com/cuda.driver-version.revision
nvidia.com/cuda.driver.major
nvidia.com/cuda.driver.minor
nvidia.com/cuda.driver.rev
nvidia.com/cuda.runtime-version.full
nvidia.com/cuda.runtime-version.major
nvidia.com/cuda.runtime-version.minor
nvidia.com/cuda.runtime.major
nvidia.com/cuda.runtime.minor
nvidia.com/gfd.timestamp
nvidia.com/gpu-driver-upgrade-state
nvidia.com/gpu.compute.major
nvidia.com/gpu.compute.minor
nvidia.com/gpu.count
nvidia.com/gpu.deploy.container-toolkit
nvidia.com/gpu.deploy.dcgm
nvidia.com/gpu.deploy.dcgm-exporter
nvidia.com/gpu.deploy.device-plugin
nvidia.com/gpu.deploy.driver
nvidia.com/gpu.deploy.gpu-feature-discovery
nvidia.com/gpu.deploy.node-status-exporter
nvidia.com/gpu.deploy.nvsm
nvidia.com/gpu.deploy.operator-validator
nvidia.com/gpu.family
nvidia.com/gpu.machine
nvidia.com/gpu.memory
nvidia.com/gpu.mode
nvidia.com/gpu.present
nvidia.com/gpu.product
nvidia.com/gpu.replicas
nvidia.com/gpu.sharing-strategy
nvidia.com/mig.capable
nvidia.com/mig.strategy
nvidia.com/mps.capable
nvidia.com/vgpu.present
```