Troubleshooting GCP + CUDA/NVIDIA + Docker and Keeping it Running! | by Thushan Ganegedara | Towards Data Science

I had a Google Cloud Platform (GCP) instance which was all setup well and running fine a day ago, which was set up following my previous tutorial.
medium.com
Let me tell you a bit about the machine I had,
Ubuntu: 16.04
GPU: 1 x P100
CUDA: 9.1
NVIDIA: 387.xx
But, something really weird happened to me when I started my GCP instance and tried running the docker container yesterday, using,
sudo docker start <container_name>sudo docker attach <container_name>
Nope, it didn’t work as I wanted it to and took me inside the container. Instead, it gave me the following error
Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device “/dev/nvidiactl”: no such file or directoryError: failed to start containers: x
This is likely to have happened because my GCP instance bravely decided that it would be totally fine to go ahead and update everything itself and things will magically work out so much better! Well, I got news! doesn’t happen that way. So it would be much appreciated if GCP gives us a way to turn off automatic updates during initial setup. In fact there has been many reports about NVIDIA drivers going bananas (or missing) with other updates (evidence 1, 2, 3).
Alright, before getting into the details of troubleshooting, let me summarize what I’m going to do step by step.
Check if my GPU is (physically) recognized by the machine
Check if NVIDIA can see the GPU
If NVIDIA cannot see the GPU, see if you have the CUDA/NVIDIA packages installed and check if NVIDIA drivers are loaded properly
If CUDA/NVIDIA packages are properly installed, try fixing PATH variables
If fixing PATH variables didn’t fix, uninstall any existing CUDA/NVIDIA packages and try reinstalling compatible CUDA and NVIDIA packages
Turn the automatic updates off, so I will update things manually without breaking everything.
First and foremost, before checking if libraries are properly installed, see of your machine physically sees the GPU by typing in,
lspci | grep 3D
This should give something like,
00:04.0 3D controller: NVIDIA Corporation Device 15f8 (rev a1)
If not, this could be due to GPU being unplugged or just nudging out from the socket because you moved the machine or something.
The first thing to do is, NOT jump to conclusions and rigorously start typing sudo apt-get install <this-and-that> hoping for the best! In fact the best thing in such a situation (and mostly ignored by many) is to identify what is the issue. First let us see what we’ve got. Go ahead and type,
nvidia-smi
You should ideally get something like,
If you get something like,
NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
This could be due to two reasons,
You are not logged in as root so NVIDIA cannot communicate (Solution: type in sudo -s to login as the root and then try nvidia-smi again)
You actually do not have compatible NVIDIA drivers installed (or the config is broken) (Solution needs more work. And this is the type of the issue I discuss here)
So for NVIDIA-SMI to work properly, you need few things properly setup. They are,
CUDA (Includes cuda-<version>, cuda-blas-<version>, cuda-nvcc-<version>, cuda-toolkit-<version>, etc.)
NVIDIA Libraries (Includes nvidia-<version>, nvidia-docker, nvidia-modprobe, nvidia-settings, etc.)
Let’s check if these are installed, try typing
dpkg -l | grep nvidia
You should get
Next try,
dpkg -l | grep cuda
this should give
Note that, actual list is much longer. But if get quite a few hits for CUDA, things should be alright. But don’t get too comfy! I had encounters things didn’t work even with these installed.
Let’s do another check to see if NVIDIA kernel modules are properly loaded by,
dmesg | grep NVIDIA
Ideally you should see,
[ 2.261511] nvidia: module license ‘NVIDIA’ taints kernel.[ 2.316304] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 384.111 Tue Dec 19 23:51:45 PST 2017 (using threaded interrupts)[ 2.319524] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 384.111 Tue Dec 19 22:56:18 PST 2017
If you don’t see anything, that is problematic! This means that NVIDIA drivers haven’t been loaded properly.
If you get both these outputs as they are shown, but not the correct dmesg message, you have the things you need all in the machine. So it could be due to some simple mis-configuration of PATH variables. So open up the .bashrc file and append the following two lines to it.
PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}LD_LIBRARY_PATH=/usr/local/cuda-9.1/lib64\                          ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Then exit the text editor and run,
source ~/.bashrc
Reboot the machine and see if things are working fine by trying out nvidia-smi again. (PS: Remember to try with sudo privileges)
If you are reading this part, you weren’t lucky as some of the folks out there. Alright, let’s go ahead and toil the soil! In my opinion, it is not worthwhile anymore to dig deeper and try to find the single grain of mistake in a bowl full of libraries. In fact, it would be much easier, if we remove the current corrupted libraries and install things correctly from the scratch.
First we need to figure out which goes along with which. By this I mean we need to make sure we download specific (and correct) versions of CUDA and NVIDIA drivers correct for the Graphic card you have. So let’s go ahead and find out what is the latest.
Go to,
www.nvidia.com
And type in the details about,
Graphic card: I have a NVIDIA Tesla P100
CUDA: Let’s go with 9.x
Here’s the results I got,
Now we’ll make sure we stick to these particular versions when doing the installation, to avoid any discrepancies.
First let us remove any existing CUDA/NVIDIA libraries with,
sudo apt-get remove --purge cuda*sudo apt-get remove --purge nvidia*
First get the CUDA toolkit .deb with,
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.1.85-1_amd64.debsudo dpkg -i cuda-repo-*sudo apt-get updatesudo apt-get install cuda -y
Reboot the system and try,
sudo dpkg -l | grep cudasudo dpkg -l | grep nvidia
and you should see the correct output as shown in figure 2 and figure 3.
I just realized the NVIDIA driver 387.xx which is shipped with CUDA 9.1 didn’t work for my NVIDIA Tesla P100. So I had to first uninstall that and install NVIDIA 384.xx. This could be different for you depending on the GPU card you have in your instance.
You know what is weird! I was totally fine working with CUDA 9.1 and NVIDIA 387.xx as seen in figure 1. But now, NVIDIA 387.xx wasn’t compatible with CUDA 9.1 any longer. I’m not sure why but hopefully can get to the bottom of that!
Let’s do this by,
sudo apt-get remove --purge nvidia-*
Now, let us install NVIDIA 384.xx driver manually by,
sudo apt-get install nvidia-384sudo apt-get install nvidia-modprobe
You will need nvidia-modprobe for nvidia-docker. Now do a quick check on path variables to see if they are properly set
echo $PATH : You should have /usr/local/cuda-<version>/bin in this variableecho $LD_LIBRARY_PATH : You should have/usr/local/cuda-<version>/lib64 in this variable
Now try nvidia-smi and you should see something similar to figure 1, meaning things are back to normal (hopefully!).
One more thing, don’t forget what started this whole ordeal in the first place. It was the automatic updates. Updates are important to keep your machine secure from outside threats and everything, but if it is going to break my machine every 5 seconds I update, NO THANK YOU! I’ll manually update it myself. So to do this, open the following file in a text editor,
/etc/apt/apt.conf.d/10periodic
and set
APT::Periodic::Update-Package-Lists “0”;
This should stop those “pesky” (however, important) automatic updates off. But remember to update your OS consistently because, you don’t want someone to hack into your machine.
So in this post, we discussed how to troubleshoot the GCP instance if you ever get into trouble of broken configuration, missing drivers, etc. And the process I recommend is,
Check if my GPU is (physically) recognized by the machine
Check if NVIDIA can see the GPU
If NVIDIA cannot see the GPU, see if you have the CUDA/NVIDIA packages installed and check if NVIDIA drivers are loaded properly
If CUDA/NVIDIA packages are properly installed, try fixing PATH variables
If fixing PATH variables didn’t fix, uninstall any existing CUDA/NVIDIA packages and try reinstalling compatible CUDA and NVIDIA packages
Turn the automatic updates off, so I will update things manually without breaking everything.