Unable to detect CUDA (CUDA unknown error)

erwawa: <p>The error message is as follows,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.is_available()
/home/yanjie/anaconda3/envs/sarah/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729047590/work/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>This error is really weird. I was able to use the CUDA in the morning. But somehow after rebooting, the error showed up. I created a new environment with conda but still cannot solve the problem. Based on the error message, I added <code>export CUDA_VISIBLE_DEVICES=1</code> in the bashrc file but it didn’t help. I’ve also rebooted for infinity times.</p>
<p>############################################################################</p>
<p>The output of nvidia-smi is<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 450.119.03   Driver Version: 450.119.03   CUDA Version: 11.0     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  GeForce RTX 207…  Off  | 00000000:08:00.0  On |                  N/A |<br>
|  0%   36C    P8    26W / 215W |    276MiB /  7959MiB |      1%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|    0   N/A  N/A       986      G   /usr/lib/xorg/Xorg                 53MiB |<br>
|    0   N/A  N/A      1637      G   /usr/lib/xorg/Xorg                 80MiB |<br>
|    0   N/A  N/A      1772      G   /usr/bin/gnome-shell              129MiB |<br>
±----------------------------------------------------------------------------+</p>
<p>############################################################################</p>
<p>The output of nvcc -V is<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2020 NVIDIA Corporation<br>
Built on Thu_Jun_11_22:26:38_PDT_2020<br>
Cuda compilation tools, release 11.0, V11.0.194<br>
Build cuda_11.0_bu.TC445_37.28540450_0</p>
<p>############################################################################</p>
<p>For more other information, I run the collect_env.py and the output is as follows,</p>
<p>Collecting environment information…<br>
/home/yanjie/anaconda3/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py:80: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /opt/conda/conda-bld/pytorch_1634272126608/work/c10/cuda/CUDAFunctions.cpp:112.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
PyTorch version: 1.10.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.2<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04 LTS (x86_64)<br>
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.17</p>
<p>Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-5.11.0-38-generic-x86_64-with-debian-bullseye-sid<br>
Is CUDA available: False<br>
CUDA runtime version: 11.0.194<br>
GPU models and configuration: GPU 0: GeForce RTX 2070 SUPER<br>
Nvidia driver version: 450.119.03<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] mypy-extensions==0.4.3<br>
[pip3] numpy==1.20.3<br>
[pip3] numpydoc==1.1.0<br>
[pip3] torch==1.10.0<br>
[pip3] torchaudio==0.10.0<br>
[pip3] torchvision==0.11.1<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.2.89              hfd86e86_1<br>
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch<br>
[conda] mkl                       2021.3.0           h06a4308_520<br>
[conda] mkl-service               2.4.0            py37h7f8727e_0<br>
[conda] mkl_fft                   1.3.1            py37hd3c417c_0<br>
[conda] mkl_random                1.2.2            py37h51133e4_0<br>
[conda] mypy_extensions           0.4.3                    py37_0<br>
[conda] numpy                     1.19.1                   pypi_0    pypi<br>
[conda] numpy-base                1.20.3           py37h74d4b33_0<br>
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1<br>
[conda] pytorch                   1.10.0          py3.7_cuda10.2_cudnn7.6.5_0    pytorch<br>
[conda] pytorch-mutex             1.0                        cuda    pytorch<br>
[conda] torch                     1.5.0                    pypi_0    pypi<br>
[conda] torchaudio                0.10.0               py37_cu102    pytorch<br>
[conda] torchvision               0.11.1               py37_cu102    pytorch</p>
<p>############################################################################</p>
<p>PS: In the morning, when everything is still workable, the runtime CUDA should be 10.1, the cudatoolkit should be 10.2 and the CUDA version shown in the nvidia-smi should be 11.0. I totally have no idea why a simple reboot can be so disastrous.</p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="erwawa" data-post="1" data-topic="135093">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/48db29/48.png" class="avatar"> erwawa:</div>
<blockquote>
<p>I totally have no idea why a simple reboot can be so disastrous.</p>
</blockquote>
</aside>
<p>A simple reboot shouldn’t cause any issues. However, by default your system might try to update all packages in each reboot and I guess that Ubuntu might have tried to update your NVIDIA drivers or CUDA and left them in a broken state.</p>

erwawa: <p>So should I uninstall the nvidia drivers and reinstall it? Thanks.</p>

ptrblck: <p>Yes, I would reinstall the drivers and disable automatic updates, if not already done.</p>

erwawa: <p>Really amazing. I uninstall the nvidia driver and reinstall a 470.74 version as recommended. Then, everything is back. Really appreciate your help!</p>

Atharva_Kshirsagar: <p>I am getting something similar, except nvcc is not working for me</p>

mabl3: <p>I just want to leave this here because I had the same issue but with a different setup, and kept getting this thread as a search result for my problem. Maybe someone else finds this helpful!</p>
<p>So I got the same error as OP, but while trying to run a program using Pytorch in a Docker container that’s based on an NGC pytorch:&lt;xx.yy&gt;-py3 image. (It was version 22.12, but 25.05 also failed so I don’t think it has much to do with the versions. I also tried different GPU driver versions. Host OS is Ubuntu 24.04, GPU is Quadro RTX 6000.)</p>
<p>Inside the container, <code>nvidia-smi</code> showed the host driver and cuda version as expected, but in python, <code>torch.cuda.is_available()</code> was false and trying to run the application resulted in the “CUDA unknown error”.</p>
<p>The solution was to add the <code>--privileged</code> flag to the docker run command, i.e.: <code>docker run --privileged --rm --gpus all --it ...</code>. Without that flag, some mmap operation fails, idk. It leads to Pytorch not being able to use the GPU, anyway.</p>
<p>Btw I finally figured that out when I tried to run only the base NGC pytorch container and got way more helpful error messages that way,</p>