CUDA error: all CUDA-capable devices are busy or unavailable

safakkbilici: <p>PyTorch throws this “File “”, line 1, in  CUDA error: all CUDA-capable devices are busy or unavailable” error when I transfer any tensor/data into cuda device, even though torch.cuda.is_available() returns True. I have came across same error issues, but none of them worked for me.</p>
<p>I downloaded PyTorch v1.1.0  cudatoolkit=9.0 (9.0.176) with conda.</p>
<p>And I have GeForce GTX 1060 with Max-Q Design with driver Version: 460.32.03, the output of nvcc --version is V9.1.85 (on Ubuntu 18.04). I run the code torch.cuda.get_device_properties(device) and returns</p>
<p>_CudaDeviceProperties(name=‘GeForce GTX 1060 with Max-Q Design’, major=6, minor=1, total_memory=6078MB, multi_processor_count=10).</p>
<p>I compiled a cuda hello word code if there is error with cuda itself, but it compiled successfully. Also I checked if my GPU in exclusive process mode; although not, I run the script nvidia-smi -i 0 -c 0 to switch to default mode. Still the same error… Is there anything I missed?</p>

ptrblck: <p>Could you update to the latest stable release or the nightly and rerun your script, please?</p>

safakkbilici: <p>I updated it to latest stable versions, it gives the same error. By the way, doesn’t the version need to be specified when downloading? (cudatoolkit=9.0 via conda installer). Since my CUDA version is 9.1.85, the latest version is specified as cudatoolkit=11.1 at <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a>.</p>

safakkbilici: <p>I did exactly nothing and today, my CUDA and PyTorch work perfectly.</p>

wmpauli: <p>I’m running into a related issue today. I first test whether cuda <code>is_available</code>. but then when I try to use it, i still get this error message. Weirdly, nothing is using the gpu at all at this point.</p>
<pre><code class="lang-auto">Tue Feb 14 01:31:48 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00007F44:00:00.0 Off |                    0 |
| N/A   41C    P0    57W / 149W |      3MiB / 11441MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>torch version: 1.12.1</p>

wmpauli: <p>Turns out that upgrading cuda drivers and torch fixed the issue.</p>