UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero

coballe: <p>This issue has suddenly arisen whenever I run torch.cuda.is_available.</p>
<blockquote>
<p>UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729009598/work/c10/cuda/CUDAFunctions.cpp:100.)`</p>
</blockquote>
<p>Output of collect_env.py</p>
<blockquote>
<p>Collecting environment information…<br>
PyTorch version: 1.7.0<br>
Is debug build: True<br>
CUDA used to build PyTorch: 10.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.2 LTS (x86_64)<br>
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27)  [GCC 9.3.0] (64-bit runtime)<br>
Python platform: Linux-5.11.0-25-generic-x86_64-with-glibc2.10<br>
Is CUDA available: False<br>
CUDA runtime version: 10.1.243<br>
GPU models and configuration: GPU 0: GeForce GTX 1080 Ti<br>
Nvidia driver version: 450.119.03<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] geotorch==0.2.0<br>
[pip3] numpy==1.19.2<br>
[pip3] numpydoc==1.1.0<br>
[pip3] torch==1.7.0<br>
[pip3] torch-cluster==1.5.8<br>
[pip3] torch-geometric==1.6.3<br>
[pip3] torch-geometric-temporal==0.0.11<br>
[pip3] torch-scatter==2.0.5<br>
[pip3] torch-sparse==0.6.8<br>
[pip3] torch-spline-conv==1.2.0<br>
[pip3] torchaudio==0.7.0a0+ac17b64<br>
[pip3] torchcontrib==0.0.2<br>
[pip3] torchdiffeq==0.2.1<br>
[pip3] torchvision==0.8.1<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.1.243             h6bb024c_0<br>
[conda] geotorch                  0.2.0                    pypi_0    pypi<br>
[conda] mkl                       2020.4             h726a3e6_304    conda-forge<br>
[conda] mkl-service               2.3.0            py38h1e0a361_2    conda-forge<br>
[conda] mkl_fft                   1.3.0            py38h5c078b8_1    conda-forge<br>
[conda] mkl_random                1.2.0            py38hc5bc63f_1    conda-forge<br>
[conda] numpy                     1.19.2           py38h54aff64_0<br>
[conda] numpy-base                1.19.2           py38hfa32c7d_0<br>
[conda] numpydoc                  1.1.0                      py_1    conda-forge<br>
[conda] pytorch                   1.7.0           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch<br>
[conda] torch-cluster             1.5.8                    pypi_0    pypi<br>
[conda] torch-geometric           1.6.3                    pypi_0    pypi<br>
[conda] torch-geometric-temporal  0.0.11                   pypi_0    pypi<br>
[conda] torch-scatter             2.0.5                    pypi_0    pypi<br>
[conda] torch-sparse              0.6.8                    pypi_0    pypi<br>
[conda] torch-spline-conv         1.2.0                    pypi_0    pypi<br>
[conda] torchaudio                0.7.0                      py38    pytorch<br>
[conda] torchdiffeq               0.2.1                    pypi_0    pypi<br>
[conda] torchvision               0.8.1                py38_cu101    pytorch</p>
</blockquote>
<p>Output of nvcc --version</p>
<blockquote>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Sun_Jul_28_19:07:16_PDT_2019<br>
Cuda compilation tools, release 10.1, V10.1.243</p>
</blockquote>
<p>Finally, output of nvidia-smi</p>
<blockquote>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 450.119.03   Driver Version: 450.119.03   CUDA Version: 11.0     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  GeForce GTX 108…  Off  | 00000000:04:00.0  On |                  N/A |<br>
| 14%   51C    P5    12W / 250W |    255MiB / 11177MiB |      1%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|    0   N/A  N/A       933      G   /usr/lib/xorg/Xorg                 35MiB |<br>
|    0   N/A  N/A      1506      G   /usr/lib/xorg/Xorg                 78MiB |<br>
|    0   N/A  N/A      1632      G   /usr/bin/gnome-shell              126MiB |<br>
|    0   N/A  N/A      3021      G   /usr/lib/firefox/firefox            2MiB |<br>
±----------------------------------------------------------------------------+</p>
</blockquote>
<p>Any help would be appreciated.</p>

ptrblck: <p>This error is raised e.g. if your system cannot communicate with the GPU, which might be caused e.g. by a driver update without a restart or any other setup issue.<br>
On my personal workstation I see this issue after waking the system from its “suspend” status, as this still does seem to cause such issues (after restarting it, it works again).</p>

coballe: <p>Thanks for the reply. Unfortunately, restarting my machine doesn’t resolve the issue.</p>

Xin.Z: <p>Hello, this issue also happened when I wake the Ubuntu 22.04 and run torch.cuda.is_available().<br>
If I reboot it, it will work again. How can I fix it without rebooting the system?<br>
My GPU is RTX3090 with the newest driver 515.43.<br>
Thank you!</p>

ptrblck: <p>You could try to execute:</p>
<pre><code class="lang-auto">sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm
</code></pre>
<p>which helps on my Ubuntu system after it was suspended.</p>

Xin.Z: <p>Thank you for your reply!<br>
I tried the two commands but they did not work.<br>
If I run torch.available, CUDA will report the same problem. Maybe it is a bug about power management of NVIDIA Driver?</p>

ptrblck: <p>Yeah, I think it’s a known issue in the interaction of the “Suspend” mode and the driver.<br>
When I have IDEs open, I get sometimes the error: <code>rmmod: ERROR: Module nvidia_uvm is in use</code> and cannot reset the GPU(s). In that case I have to reboot unfortunately, but ~9/10 times these two commands do the job and I can properly use the GPU again.</p>

Oguzhan: <p>I got error after millions of trying these 2 command and still torch.cuda_is_avaliable returns cpu <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

Lukas_Rois: <p>Anyone still who still has this issue try:</p>
<pre><code class="lang-auto">sudo apt-get install nvidia-modprobe
</code></pre>
<p>worked for me!<br>
Source: <a href="https://github.com/pytorch/pytorch/issues/49081" class="inline-onebox" rel="noopener nofollow ugc">RuntimeError: CUDA unknown error · Issue #49081 · pytorch/pytorch · GitHub</a></p>

Jason_Li: <p>seemed to work for me too!</p>

ThreeStones1029: <p>Thank you，I ran into this problem when the program was still running, but the system went to sleep and was then interrupted. After I woke up from sleep, torch.cuda.is_available() ran into this issue. After running these two commands, it  works.</p>

mtrantalainen: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="129335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>ERROR: Module nvidia_uvm is in use</p>
</blockquote>
</aside>
<p>I, too, have the problem that the kernel module nvidia_uvm cannot be removed because of error <code>ERROR: Module nvidia_uvm is in use</code>. Do you know if there’s a way to figure out who is using the module? If it’s a process, I could probably kill the offending process.</p>
<p>And the error was triggered by having GPU computation active while putting the system to S3 sleep so this is definitely related to sleep states.</p>

ptrblck: <p>You could try running:</p>
<pre><code class="lang-auto">lsmod | grep nvidia
lsof | grep nvidia
</code></pre>
<p>which should give you information about the processes.</p>

Dongwei_Zhang: <p>I have the same problem in vscode just now, and I just restart the conda environment kernel and it works</p>

csesaswati: <p>I have same issue with the original post. I used this command and restarted the machine but it did not solve the problem.</p>

turbotimon: <aside class="quote no-group" data-username="mtrantalainen" data-post="12" data-topic="129335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mtrantalainen/48/67899_2.png" class="avatar"> mtrantalainen:</div>
<blockquote>
<p>Do you know if there’s a way to figure out who is using the module? If it’s a process, I could probably kill the offending process.</p>
</blockquote>
</aside>
<p>If I’m not wrong, you can see the processes and their PID by running <code>nvidia-smi</code>?</p>
<p>(Btw, <code>sudo rmmod nvidia_uvm &amp;&amp; sudo modprobe nvidia_uvm</code> helped me. System restart also. It is clearly related to hibernate in my case on nixOS)</p>

DenisLevchenko: <p>Worked for me, thank you</p>

mtrantalainen: <aside class="quote no-group" data-username="turbotimon" data-post="16" data-topic="129335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/turbotimon/48/69503_2.png" class="avatar"> turbotimon:</div>
<blockquote>
<p>If I’m not wrong, you can see the processes and their PID by running <code>nvidia-smi</code>?</p>
</blockquote>
</aside>
<p>Yeah, I would have assumed so, too, but it later turned out that the culprit was <code>nvtop</code> which was running in a single terminal window. It causes <code>ERROR: Module nvidia_uvm is in use</code> even when <code>nvidia-smi</code> doesn’t show it at all.</p>

turbotimon: <p>Note for lightning: This exception even occurs if you want to <strong>train explicitly on the CPU</strong> with <code>Trainer(accelerator=“cpu”)</code>. Because <code>torch/cuda/__init__.py</code> is still loaded during <code>trainer.fit</code>. The solution is:</p>
<pre><code class="lang-auto">import os
# Disable GPU visibility. Make sure its BEFORE importing torch (or any other module that uses torch)
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import torch
import lightning
...
trainer=Trainer(accelerator=“cpu”)
</code></pre>

Varat7v2: <p>Thanks this worked for me on Ubuntu 24.04.</p>