No process using GPU, but `CUDA error: all CUDA-capable devices are busy or unavailable`

username: nkla
cooked: <p>See latest post for <code>CUDA error: all CUDA-capable devices are busy or unavailable</code></p>
<h1>Resolved issue</h1>
<p>I was running Pytorch without issues using GTX 1080 Ti. I recently obtained a RTX3090, and had to make appropriate updates on nvidia drivers for Ampere architecture support. However, I started getting errors when trying to put variables into GPU with .cuda(), and <code>torch.cuda.is_available()</code> returns <code>False</code>. See below.</p>
<p>The same error also occurs in a separate (new) machine with Quadro RTX 5000, leading me to speculate this could be a setup error. However, I do not know the commonalities between the two machines</p>
<p>Machines experiecing the same errors</p>
<ul>
<li>RTX3090
<ul>
<li>Debian Testing</li>
<li>nvidia-driver: 455.38, from Debian experimental</li>
<li>nvidia-cuda-toolkit: 11.0.3-2, from Debian testing</li>
</ul>
</li>
<li>Quadro RTX5000
<ul>
<li>Debian Testing (VM, vfio passthrough)</li>
<li>nvidia-driver: 450.80, from Debian testing</li>
<li>nvidia-cuda-toolkit: 11.0.3-2, from Debian testing</li>
</ul>
</li>
</ul>
<p>Please let me know if you have any suggestions on troubleshooting this issue.</p>
<p>Thanks</p>
<p><strong>Following results are from the RTX3090 machine</strong></p>
<h1>Miniconda env</h1>
<pre><code class="lang-auto">$ python3 -c 'import torch; print(torch.cuda.is_available())'
/home/user/dev/miniconda3/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729096996/work/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<pre><code class="lang-auto">$ python3 -c 'import torch; torch.rand(3).cuda()'
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/home/user/dev/miniconda3/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.
</code></pre>
<h2>Miniconda installation</h2>
<pre><code class="lang-auto">$ conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
</code></pre>
<h1>Pip env</h1>
<pre><code class="lang-auto">$ python3 -c 'import torch; print(torch.cuda.is_available())'
/home/user/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<pre><code class="lang-auto">$ python3 -c 'import torch; torch.rand(3).cuda()'
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/home/user/.local/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.
</code></pre>
<h2>Pip installation</h2>
<ul>
<li>Attempted to use torch nightly 1.8, with same error</li>
</ul>
<pre><code class="lang-auto">$ python3 -m pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<h1>System specs</h1>
<pre><code class="lang-auto">$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	testing
Codename:	bullseye
</code></pre>
<pre><code class="lang-auto">$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    On   | 00000000:01:00.0  On |                  N/A |
|  0%   37C    P8    33W / 350W |    282MiB / 24245MiB |     16%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
</code></pre>
<pre><code class="lang-auto">$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
</code></pre>
<pre><code class="lang-auto">$ apt list --installed | grep nvidia-*
glx-alternative-nvidia/testing,unstable,now 1.2.0 amd64 [installed,automatic]
libegl-nvidia0/experimental,now 455.38-1 amd64 [installed,automatic]
libegl-nvidia0/experimental,now 455.38-1 i386 [installed,automatic]
libgl1-nvidia-glvnd-glx/experimental,now 455.38-1 amd64 [installed,automatic]
libgl1-nvidia-glvnd-glx/experimental,now 455.38-1 i386 [installed,automatic]
libgles-nvidia1/experimental,now 455.38-1 amd64 [installed,automatic]
libgles-nvidia1/experimental,now 455.38-1 i386 [installed,automatic]
libgles-nvidia2/experimental,now 455.38-1 amd64 [installed,automatic]
libgles-nvidia2/experimental,now 455.38-1 i386 [installed,automatic]
libglx-nvidia0/experimental,now 455.38-1 amd64 [installed,automatic]
libglx-nvidia0/experimental,now 455.38-1 i386 [installed,automatic]
libnvidia-cfg1/experimental,now 455.38-1 amd64 [installed,automatic]
libnvidia-compiler/experimental,now 455.38-1 amd64 [installed,automatic]
libnvidia-eglcore/experimental,now 455.38-1 amd64 [installed,automatic]
libnvidia-eglcore/experimental,now 455.38-1 i386 [installed,automatic]
libnvidia-glcore/experimental,now 455.38-1 amd64 [installed,automatic]
libnvidia-glcore/experimental,now 455.38-1 i386 [installed,automatic]
libnvidia-glvkspirv/experimental,now 455.38-1 amd64 [installed,automatic]
libnvidia-glvkspirv/experimental,now 455.38-1 i386 [installed,automatic]
libnvidia-ml-dev/testing,unstable,now 11.0.3-2 amd64 [installed,automatic]
libnvidia-ml1/experimental,now 455.38-1 amd64 [installed,automatic]
libnvidia-ptxjitcompiler1/experimental,now 455.38-1 amd64 [installed,automatic]
libnvidia-ptxjitcompiler1/experimental,now 455.38-1 i386 [installed,automatic]
nvidia-alternative/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-cuda-dev/testing,unstable,now 11.0.3-2 amd64 [installed]
nvidia-cuda-gdb/testing,unstable,now 11.0.3-2 amd64 [installed]
nvidia-cuda-toolkit-doc/testing,testing,unstable,unstable,now 11.0.3-2 all [installed,automatic]
nvidia-cuda-toolkit/testing,unstable,now 11.0.3-2 amd64 [installed]
nvidia-driver-bin/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-driver-libs/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-driver-libs/experimental,now 455.38-1 i386 [installed,automatic]
nvidia-driver/experimental,now 455.38-1 amd64 [installed]
nvidia-egl-common/now 455.23.04-1 amd64 [installed,local]
nvidia-egl-icd/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-egl-icd/experimental,now 455.38-1 i386 [installed,automatic]
nvidia-installer-cleanup/testing,unstable,now 20151021+12 amd64 [installed]
nvidia-kernel-common/testing,unstable,now 20151021+12 amd64 [installed]
nvidia-kernel-dkms/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-kernel-support/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-legacy-check/now 455.23.04-1 amd64 [installed,local]
nvidia-modprobe/experimental,now 455.23.04-1 amd64 [installed,automatic]
nvidia-opencl-common/now 455.23.04-1 amd64 [installed,local]
nvidia-opencl-dev/testing,unstable,now 11.0.3-2 amd64 [installed]
nvidia-opencl-icd/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-openjdk-8-jre/testing,unstable,now 9.+8u252-b09-1~deb9u1~11.0.3-2 amd64 [installed,automatic]
nvidia-persistenced/testing,unstable,now 450.57-1 amd64 [installed]
nvidia-profiler/testing,unstable,now 11.0.3-2 amd64 [installed,automatic]
nvidia-settings/testing,unstable,now 450.80.02-1 amd64 [installed]
nvidia-smi/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-support/testing,unstable,now 20151021+12 amd64 [installed]
nvidia-vdpau-driver/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-visual-profiler/testing,unstable,now 11.0.3-2 amd64 [installed,automatic]
nvidia-vulkan-common/now 455.23.04-1 amd64 [installed,local]
nvidia-vulkan-icd/experimental,now 455.38-1 amd64 [installed,automatic]
nvidia-vulkan-icd/experimental,now 455.38-1 i386 [installed,automatic]
nvidia-xconfig/testing,unstable,now 450.66-1 amd64 [installed]
xserver-xorg-video-nvidia/experimental,now 455.38-1 amd64 [installed]
</code></pre>
<h1>Testing nvcc</h1>
<p>I’m not an expert in CUDA, but I copied a helloworld code and ran without errors</p>
<pre><code class="lang-auto">//hello.cu
// This is the REAL "hello world" for CUDA!
// It takes the string "Hello ", prints it, then passes it to CUDA with an array
// of offsets. Then the offsets are added in parallel to produce the string "World!"
// By Ingemar Ragnemalm 2010
 
#include &lt;stdio.h&gt;
 
const int N = 16; 
const int blocksize = 16; 
 
__global__ 
void hello(char *a, int *b) 
{
	a[threadIdx.x] += b[threadIdx.x];
}
 
int main()
{
	char a[N] = "Hello \0\0\0\0\0\0";
	int b[N] = {15, 10, 6, 0, -11, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
 
	char *ad;
	int *bd;
	const int csize = N*sizeof(char);
	const int isize = N*sizeof(int);
 
	printf("%s", a);
 
	cudaMalloc( (void**)&amp;ad, csize ); 
	cudaMalloc( (void**)&amp;bd, isize ); 
	cudaMemcpy( ad, a, csize, cudaMemcpyHostToDevice ); 
	cudaMemcpy( bd, b, isize, cudaMemcpyHostToDevice ); 
	
	dim3 dimBlock( blocksize, 1 );
	dim3 dimGrid( 1, 1 );
	hello&lt;&lt;&lt;dimGrid, dimBlock&gt;&gt;&gt;(ad, bd);
	cudaMemcpy( a, ad, csize, cudaMemcpyDeviceToHost ); 
	cudaFree( ad );
	cudaFree( bd );
	
	printf("%s\n", a);
	return EXIT_SUCCESS;
}
</code></pre>
<pre><code class="lang-auto"># nvcc hello.cu -o hello
# ./hello
Hello Hello
</code></pre>
<h1>Setting envs</h1>
<p>Executing the following prior to importing torch do not resolve the errors</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
</code></pre>

username: ptrblck
cooked: <p>The error</p>
<pre><code class="lang-auto">CUDA initialization: CUDA unknown error 
</code></pre>
<p>is unfortunately not very helpful.<br>
Could you check <code>dmesg</code> for any XID error codes and post them here?<br>
Also, could you check, if docker containers with CUDA11 and PyTorch work fine on your machine?</p>

username: granth_jain
cooked: <p>Hi,</p>
<p>I had an issue on RTX2060 where cuda was not available.</p>
<p>I reinstalled it and worked fine. Mine was dependency issue with tensorflow as tensorflow-gpu runs on cuda11.</p>
<p>Please make sure once that you have installed cuda correctly and also check with tensorflow-gpu if cuda is running fine.<br>
Hope it helps.<br>
Thanks</p>

username: nkla
cooked: <p>Thanks ptrblck, granth_jain.</p>
<p>When I investigate <code>dmesg</code>,</p>
<pre><code class="lang-auto"># dmesg|grep "NVRM"
[    9.976755] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  455.38  Thu Oct 22 06:06:59 UTC 2020
</code></pre>
<p>I noticed if I run the torch methods which error out, I get the following</p>
<pre><code class="lang-auto">[43613.854296] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
[43613.854575] nvidia_uvm: Unknown symbol radix_tree_preloads (err -2)
</code></pre>
<p><a href="https://forums.developer.nvidia.com/t/455-38-and-kernel-5-9-6/159039" rel="noopener nofollow ugc">This was caused by Nvidia incompatibility with Kernel 5.9</a>. I <strong>downgraded from 5.9 to 5.8</strong>, and the errors are resolved.</p>
<p>I applied the fixes to both computers and the errors are resolved.</p>
<p>However, my Quardo RTX 5000 machine is  encountering another error, where</p>
<pre><code class="lang-auto">$ python3 -c 'import torch; torch.randn(1).to(0)'
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable
</code></pre>
<p>I verified that no process is using the GPU,</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 5000     On   | 00000000:04:00.0 Off |                  Off |
| 33%   26C    P8     6W / 230W |      1MiB / 16125MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>and the compute mode is in default, not exclusive,</p>
<pre><code class="lang-auto">$ nvidia-smi -a | grep Compute
    Compute Mode                          : Default
</code></pre>
<p>This is running inside a kvm hypervisor with vfio passthrough, and I verified that <code>nvidia</code> driver is attached to the GPU</p>
<pre><code class="lang-auto">04:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1)
        Subsystem: Dell TU104GL [Quadro RTX 5000]
        Kernel driver in use: nvidia
        Kernel modules: nvidia
05:00.0 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
        Subsystem: Dell TU104 HD Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
06:00.0 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
        Subsystem: Dell TU104 USB 3.1 Host Controller
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
07:00.0 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
        Subsystem: Dell TU104 USB Type-C UCSI Controller
</code></pre>
<p>I attempted to</p>
<ul>
<li>Remove all <code>nvidia-*</code> packages and reinstall <code>nvidia-driver</code> (tried both 450.80 and 455.38) and <code>nvidia-cuda-toolkit</code> (11.0)</li>
<li>Reinstall Pytorch for Cuda 11.0 using miniconda and pip.</li>
</ul>
<p>The server is headless, and no desktop environment was installed. Thus, there should be no graphics-based processes using the gpu.</p>
<p>Do you know what is causing this issue? Can this be caused by VFIO, although everything seem to be in order? Thanks!</p>
<p><strong>More tests</strong><br>
I downloaded and compiled the <a href="https://gist.github.com/f0k/0d6431e3faa60bffc788f8b4daa029b1" rel="noopener nofollow ugc">script to test cuda functionality</a>. The output shows error code 201 for <code>cMemGetInfo</code>.</p>
<pre><code class="lang-auto">$ ./cuda_check
Found 1 device(s).
Device: 0
  Name: Quadro RTX 5000
  Compute Capability: 7.5
  Multiprocessors: 48
  CUDA Cores: 3072
  Concurrent threads: 49152
  GPU clock: 1815 MHz
  Memory clock: 7001 MHz
  cMemGetInfo failed with error code 201: invalid device context
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="nkla" data-post="4" data-topic="102623">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/58956e/48.png" class="avatar"> nkla:</div>
<blockquote>
<p>Do you know what is causing this issue? Can this be caused by VFIO, although everything seem to be in order? Thanks!</p>
</blockquote>
</aside>
<p>I don’t know, if VFIO could cause this issue. Could you try to run a CUDA sample on this node without VFIO e.g. in a docker container or on the bare metal?</p>

username: nkla
cooked: <p>I gave up with VFIO. The original error was me not passing through other 3 components of the GPU (audio, usb, scsi). Now, pytorch works sometimes only if the GPU was originally attached by nouveau, then bind to vfio. Eg, if GPU was originally only used by vfio-pci, pytorch will not work in guest. Instead, python3 binary will be frozen and unkillable, requiring a reset of the guest.</p>
<p>Seems like vfio is not ready for deep learning. I wonder how do colabs run the services? Will be running bare metal now, thanks.</p>

username: chumingqian
cooked: <p>Hi,<br>
I  have  same  question with you , my  machine  is  RTX 2060 Notebook series,   and  i  install  cuda 11.0.3,    cudnn   for  11.0;</p>
<h4>UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)return torch._C._cuda_getDeviceCount() &gt; 0</h4>
<p>Now i am trying to  uninstall  cuda11  and cudnn;<br>
I  was  wondering   what 's  cuda  version  of  your    reinstall  and  before you reintall  did you  unintall the  cuda11  and  cudnn  both?</p>

username: nkla
cooked: <p>Hi, are you running kernel&gt;=5.9? You’ll need to downgrade to &lt;=5.8 since nvidia does not support 5.9 yet.</p>

username: granth_jain
cooked: <p>Hi,</p>
<p>I don’t exactly remember my previous cuda version,  I installed cuda version 11.</p>
<p>installing tensorflow-gpu then installing pytorch with same cuda version as tensorflow-gpu cuda did the trick for me.</p>
<p>Thanks</p>

username: chumingqian
cooked: <p>Hi nkla,<br>
Thanks for  your reply,  mine  kernel  is  5.4.0-52-generic;<br>
And  i  according to  userWarning:</p>
<h4><a name="this-may-be-due-to-an-incorrectly-set-up-environment-eg-changing-env-variable-cuda_visible_devices-after-program-start-setting-the-available-devices-to-be-zero-1" class="anchor" href="#this-may-be-due-to-an-incorrectly-set-up-environment-eg-changing-env-variable-cuda_visible_devices-after-program-start-setting-the-available-devices-to-be-zero-1"></a>this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero</h4>
<p>So,  i  adding the   " export CUDA_VISIBLE_DEVICES=0   "   via the  source gedit ~/.bashrc;<br>
Now ,   it 's  works  fine;</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.cuda.is_available()<br>
True</p>
</blockquote>
</blockquote>
</blockquote>
<p>Thanks again</p>