Change:
runcmd: - modprobe configs <del> # Setup the installation target at make it executable - mkdir -p /home/kubernetes/bin/nvidia - mount --bind /home/kubernetes/bin/nvidia /home/kubernetes/bin/nvidia - mount -o remount,exec /home/kubernetes/bin/nvidia # Compile and install the nvidia driver (precompiled driver installation currently fails) - docker run --net=host --pid=host -v /dev:/dev -v /:/root -v /home/kubernetes/bin/nvidia:/usr/local/nvidia -e NVIDIA_INSTALL_DIR_HOST=/home/kubernetes/bin/nvidia -e NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia -e NVIDIA_DRIVER_VERSION=460.91.03 --privileged gcr.io/cos-cloud/cos-gpu-installer:latest # Run the installer again, as on the first try it doesn't detect the libnvidia-ml.so # on the second attempt we detect it and update the ld cache. - docker run --net=host --pid=host -v /dev:/dev -v /:/root -v /home/kubernetes/bin/nvidia:/usr/local/nvidia -e NVIDIA_INSTALL_DIR_HOST=/home/kubernetes/bin/nvidia -e NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia -e NVIDIA_DRIVER_VERSION=460.91.03 --privileged gcr.io/cos-cloud/cos-gpu-installer:latest </del> <ins> # Install GPU drivers - https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus - cos-extensions install gpu - mount --bind /var/lib/nvidia /var/lib/nvidia - mount -o remount,exec /var/lib/nvidia /var/lib/nvidia # Run nvidia-smi to verify installation - /var/lib/nvidia/bin/nvidia-smi </ins> # Remove build containers. They're very large. - docker rm -f $(docker ps -aq) # Standard installation proceeds

Discussion:
I can't find a single instance of this test passing in testgrid. Run via and potentially other suites. /sig node /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The test fails because there is no GPUs reported by kubelet [1]. Looking at the node info right before the failure, the node does not have any gpu capacity [2]. The artifacts only have kubelet log, but not device plugin or driver installer log. So I am not sure if there's any error on them. Is there any way to see the device plugin or driver installer log? [1] [2]
/assign I have a partial fix here - The actual feature seems broken too tho - hidden by the fact that it's been failing to provision properly for a really long time :(
Since merged this test has been relatively stable. Going to close this issue now (kept it around while I was on PTO to take a look when I got back). Seems like there are some infrequent flakes that we should follow up on after we remove DKK. /close
Closing this issue. <detailsIn response to : Since merged this test has been relatively stable. Going to close this issue now (kept it around while I was on PTO to take a look when I got back). Seems like there are some infrequent flakes that we should follow up on after we remove DKK. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This is failing again, I'm not sure what regressed but it was 2wks ago.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I keep seeing errors like But our serial suite doesn't have GPUs attached.

I'd expect this to be skipped: do you have any idea why sometimes these tests pass, sometimes they get skipped, and sometimes they time out? It makes no sense to me:
Wrote to try to dump what's matching and not causing these tests to get skipped
The tests get skipped when they run on nodes without GPUs attached, which is 2 of 3 runs every time, but cos-stable2 has 2 GPUs (). The failures happen because something in the GPU setup process broke again (I assume a bad image bump bc we don't pin), and so the test fails waiting for the GPUs to show up on the node after installing the plugin, but they are attached to the node.
I think I have a fix here -