Cuda.is_available() return False on Docker image pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel

shaolinkhoa: <p>Hi,<br>
I build my docker image from PyTorch image: <code>pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel</code></p>
<p>My server:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.04    Driver Version: 455.23.04    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:3B:00.0 Off |                  N/A |
|  0%   32C    P0   101W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 3090    Off  | 00000000:AF:00.0 Off |                  N/A |
| 30%   32C    P0    67W / 350W |      0MiB / 24268MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>I’ve already installed <a href="https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian" rel="noopener nofollow ugc">NVIDIA Container Toolkit</a> and restart the Docker<br>
<code>sudo systemctl restart docker</code></p>
<p>I can run <code>nvidia-smi</code> inside Docker container but when I try</p>
<p><code>sudo docker run --rm --gpus all khoa/pytorch:1.7 python -c 'import torch as t; print(t.cuda.is_available()); print(t.backends.cudnn.enabled)'</code></p>
<p><code>cuda.is_available()</code> return <strong>False</strong><br>
while <code>backends.cudnn.enabled</code> return <strong>True</strong></p>
<blockquote>
<p>/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py:52: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729096996/work/c10/cuda/CUDAFunctions.cpp:100.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False<br>
True</p>
</blockquote>
<p>Can anyone help me?<br>
I can’t run my code because of this.</p>

ptrblck: <p>The container is working for me so I guess your <code>docker</code> setup isn’t working properly.<br>
Are you able to run any other container shipped with CUDA applications?</p>
<pre><code class="lang-auto">nvidia-docker run -it --ipc=host pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel
root@d11e05a20388:/workspace# python -c "import torch; print(torch.cuda.is_available())"
True
</code></pre>

shaolinkhoa: <p><code>pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel</code> also doesn’t work.</p>
<blockquote>
<p>sudo docker run --rm --gpus all pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel python -c ‘import torch as t; print(t.cuda.is_available())’</p>
</blockquote>
<blockquote>
<p>False<br>
/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py:52: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729096996/work/c10/cuda/CUDAFunctions.cpp:100.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0</p>
</blockquote>
<p>So weird, my other server (server B) has the same configuration  that can run my image.</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.04    Driver Version: 455.23.04    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:3B:00.0 Off |                  N/A |
|  0%   38C    P8    34W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 3090    Off  | 00000000:AF:00.0 Off |                  N/A |
|  0%   32C    P8    34W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>Meanwhile, server A has this problem after I rebooted it.<br>
I also think it because of Docker. But I’ve tried<br>
<code>sudo systemctl restart docker</code><br>
but still doesn’t work.<br>
The different between <code>sudo docker info</code><br>
<strong>Server A</strong>:</p>
<blockquote>
<p>Runtimes: runc nvidia<br>
Kernel Version: 5.4.0-53-generic</p>
</blockquote>
<p><strong>Server B</strong>:</p>
<blockquote>
<p>Runtimes: runc<br>
Kernel Version: 5.4.0-52-generic</p>
</blockquote>

shaolinkhoa: <p>From this <a href="https://github.com/NVIDIA/nvidia-container-runtime#docker-engine-setup" rel="noopener nofollow ugc">guide</a> and I’m using <strong><code>nvidia-docker2</code></strong> so I removed <code>runc nvidia</code> by deleting <code>/etc/docker/daemon.json</code> and reboot the server A.<br>
But it still doesn’t work.</p>
<p>This problem happens because:<br>
while training, there was a bug made the function divided 0, so my server was frozen/crashed.<br>
So I <code>ctrl+C</code> the task and rebooted the server.<br>
Now, <code>cuda.is_available()</code> always return False</p>
<p>Please help me.</p>

ptrblck: <p>My best guess would be that (unwanted) updates might have been executed, which wiped the NVIDIA driver on your system after the restart. If that’s the case, you would have to reinstall them and recheck your container.</p>

shaolinkhoa: <p>Thank you.<br>
I reinstall the Nvidia driver and it works.<br>
I should have done it from the beginning.</p>