DistributedDataParallel: RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

username: Jimmy2027
cooked: <p>Hello, I get the error <code>RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable</code> when I try to run my code on one server A with 2 GPUs while the code runs fine on another server B. What could be causing this issue? Could this be related to the CUDA Version? On the server A where the code fails the cuda version is 10.1 while the server B where the code runs has cuda version 11.</p>
<p>The full error (the process freezes after this output):</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/cluster/home/klugh/software/anaconda/envs/temp/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/cluster/home/klugh/software/anaconda/envs/temp/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/cluster/home/klugh/software/anaconda/envs/temp/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 109, in rebuild_cuda_tensor
    storage = storage_cls._new_shared_cuda(
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable
</code></pre>

username: rvarm1
cooked: <p>Are you using a shared cluster/machine by any chance? The GPU may not be available if another application/user has taken control of it. You can check current GPU usage using the <code>nvidia-smi</code> command.</p>

username: Jimmy2027
cooked: <p>Hi <a class="mention" href="/u/rvarm1">@rvarm1</a>, thanks for the answer! I am indeed using a shared cluster, but when I run <code>nvidia-smi</code> the GPUs seem to be free:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39       Driver Version: 418.39       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:89:00.0 Off |                    0 |
| N/A   37C    P0    44W / 300W |      0MiB / 32480MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:8A:00.0 Off |                    0 |
| N/A   35C    P0    43W / 300W |      0MiB / 32480MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/ngimel">@ngimel</a> <a class="mention" href="/u/ptrblck">@ptrblck</a>, have you seen errors like this before?</p>

username: ptrblck
cooked: <p>Based on the output of <code>nvidia-smi</code>, it seems the GPUs are in <code>EXCLUSIVE Process</code> mode, which would allow only a single context.</p>
<pre><code class="lang-auto">nvidia-smi -i 0 -c 0
nvidia-smi -i 1 -c 0
# or for both directly
nvidia-smi -c 0 
</code></pre>
<p>should reset both GPUs to the default mode again.</p>

username: blmt
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you expand on what exclusive process vs shared mode entails? As far as I understand it’s a common practice in compute clusters to have the GPU set up in “exclusive process” mode and that’s not changeable by a user.<br>
How does pytorch work when doing distributed work as opposed to the regular case?</p>

username: ptrblck
cooked: <p>The exclusive mode might be the right choice for your compute cluster and you can stick to it, if it’s working.<br>
However, I would not recommend it as the default mode, if you are unsure about its limitations (single context creation) and are using your local workstation.</p>
<aside class="quote no-group" data-username="blmt" data-post="6" data-topic="102763">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blmt/48/31727_2.png" class="avatar"> blmt:</div>
<blockquote>
<p>How does pytorch work when doing distributed work as opposed to the regular case?</p>
</blockquote>
</aside>
<p>The recommended approach is to use <code>DistributedDataParallel</code>, with a single process per GPU.<br>
Each device would thus create an own context.</p>