SLURM: RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable

dmpribak: <p>I am trying to train EG3D on a slurm cluster using multiple gpus. But am getting the following error:</p>
<pre><code class="lang-auto">File "/home/dmpribak/ondemand/data/sys/myjobs/projects/default/4/train.py", line 395, in main
    launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run)
  File "/home/dmpribak/ondemand/data/sys/myjobs/projects/default/4/train.py", line 105, in launch_training
    torch.multiprocessing.spawn(fn=subprocess_fn, args=(c, temp_dir), nprocs=c.num_gpus)
  File "/home/dmpribak/.conda/envs/eg3d3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 246, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dmpribak/.conda/envs/eg3d3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/home/dmpribak/.conda/envs/eg3d3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 163, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/dmpribak/.conda/envs/eg3d3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
    fn(i, *args)
  File "/home/dmpribak/ondemand/data/sys/myjobs/projects/default/4/train.py", line 54, in subprocess_fn
    training_loop.training_loop(rank=rank, **c)
  File "/home/dmpribak/ondemand/data/sys/myjobs/projects/default/4/training/training_loop.py", line 196, in training_loop
    torch.distributed.broadcast(param, src=0)
  File "/home/dmpribak/.conda/envs/eg3d3/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/dmpribak/.conda/envs/eg3d3/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1906, in broadcast
    work = default_pg.broadcast([tensor], opts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>I have the latest version of pytorch installed with cuda 12.1. torch.cuda.is_available() returns true and I am able to see both gpus printed with<br>
<code>for i in range(torch.cuda.device_count()):         print(torch.cuda.get_device_properties(i).name)</code></p>
<p>Here is my nvidia-smi output:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/858139af79438f99af8c358971b9f893eb4c96de.png" data-download-href="https://discuss.pytorch.org/uploads/default/858139af79438f99af8c358971b9f893eb4c96de" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/858139af79438f99af8c358971b9f893eb4c96de.png" alt="image" data-base62-sha1="j32mMWkfbkeBZMDgw6lqw0d5C2y" width="690" height="394" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">715×409 12.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please let me know if any further information would be helpful in pinning down the cause of this</p>

dmpribak: <p>I’d also like to add that I am able to run training completely fine when I only use a single gpu</p>

ptrblck: <p>Try disabling the “exclusive process” mode your GPUs are using right now and return the script.</p>

dmpribak: <p>I have tried this, but I am running this on a university cluster and don’t have the permission to do so</p>

ptrblck: <p>In this case you might need to ask an admin to change it to check if this is blocking the multiprocessing workload.</p>

dmpribak: <p>I can ask. Is there anything else that might cause this issue that I could try in the meantime?</p>

Howardyangyixuan: <p>I also have this problem. In my case, I work with two A100 Driver Version: 535.129.03  CUDA Version: 12.1, Pytorch 2.1.0.dev20230621+cu117.</p>
<pre><code class="lang-auto"># simplest sample from Pytorch Get Started
Traceback (most recent call last):
  File "test.py", line 44, in &lt;module&gt;
    x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype)
RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>before using Pytorch dev version I got this error</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: operation not supported
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>

kazu: <p>I had the same error message as dmpribak’s when I ran a sample program of Distributed Data Parallel.<br>
Disabling the exclusive process by nvidia-smi -c=0 has solved the problem!<br>
Thanx.</p>

raghavm1: <p>I’m still facing this issue. All my 4 GPUs where I’m running DDP are in default mode and not exclusive mode.<br>
My code throws this error after 1 day of execution, and I’m currently resuming from existing checkpoints.</p>
<p>The command <code>nvidia-smi</code> also seems to hang for me once this error occurs.</p>

ptrblck: <p>Check for any <code>Xid</code>s in <code>dmesg</code> as the error could be related to your setup (e.g. an overheating machine) and might be unrelated to PyTorch.</p>

raghavm1: <p>Hmm I just checked but nothing there in <code>dmesg</code><br>
any other logs or places you could suggest I check?</p>
<p>[Edit]: When I try to kill the job, it hangs and then I get these log messages</p>
<pre><code class="lang-auto">Message from syslogd@localhost at Sep 13 20:02:47 ...
 kernel:[82362.359551] watchdog: BUG: soft lockup - CPU#44 stuck for 74s! [nvidia-smi:1881056]
</code></pre>
<p>For context, I’m also doing multiprocessing operations for each batch before I pass the output to a GPU.</p>

johann-petrak: <p>Getting this intermittently again and again too and there is no indication what could be going on as the GPU is not actually used that much: only 1 of 4G are used according to nvidia-smi<br>
This is extremely annoying and I have found quite a few reports on the internet where people have the same problem but no real reproducable solution or even an explanation of what is going on.</p>