SLURM cluster CUDA error: all CUDA-capable devices are busy or unavailable

username: import-antigravity
cooked: <p>I’m using a pre-trained Inception network to get some GAN metrics, and I have the following code to move the model to a GPU for evaluation:</p>
<pre><code class="lang-auto">def __init__(self, ...):
    ...
    self._model = InceptionV3(...)
    if cuda.is_available():
        self._model.to('cuda')
</code></pre>
<p>When I run this code on an HPC, I get the following error:<br>
<code>RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable</code></p>
<p>Is there something wrong with how I’m moving the model to a GPU? This code should work even if I have an arbitrary number of GPUs right?</p>

username: import-antigravity
cooked: <p>Update, this error only happens when I request a node with more than 1 GPU</p>

username: ptrblck
cooked: <p>I guess your multi-GPU slurm setup isn’t working correctly and might be masking all available devices, so that PyTorch isn’t able to use any.<br>
If you are trying to use e.g. 2 GPUs, you could run this small test:</p>
<pre><code class="lang-python">for d in range(torch.cuda.device_count()):
    print(torch.cuda.get_device_name(d))
</code></pre>
<p>If this doesn’t print anything, then your container might not be able to “see” any devices.</p>

username: import-antigravity
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="114622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">for d in range(torch.cuda.device_count()):
    print(torch.cuda.get_device_name(d))
</code></pre>
</blockquote>
</aside>
<pre><code class="lang-auto">&gt;&gt;&gt; for d in range(torch.cuda.device_count()):
...     print(torch.cuda.get_device_name(d))
...
GeForce RTX 2080 Ti
GeForce RTX 2080 Ti
</code></pre>

username: import-antigravity
cooked: <p>To reiterate, this only happens when I have more than one GPU.</p>

username: ptrblck
cooked: <p>I don’t know what causes this issue, as the Python script can apparently access both GPUs, while you are still getting the SLURM error, so I guess that this error is raised by your custom script?</p>

username: import-antigravity
cooked: <p>Well, this happens whether I use an <code>sbatch</code> script or do an interactive session with <code>srun</code>. Both times I request 1 or 2 GPUs with <code>--gres=gpu:n</code>. There isn’t really room for something to go wrong with the script so I find it hard to imagine that could be the problem. It’s possible it’s an issue with pytorch-lightning but the problem only happens with my pre-trained Inception model which is just a <code>nn.Module</code>.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="import-antigravity" data-post="7" data-topic="114622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/import-antigravity/48/30128_2.png" class="avatar"> import-antigravity:</div>
<blockquote>
<p>Well, this happens whether I use an <code>sbatch</code> script or do an interactive session with <code>srun</code>. Both times I request 1 or 2 GPUs with <code>--gres=gpu:n</code>. There isn’t really room for something to go wrong with the script so I find it hard to imagine that could be the problem.</p>
</blockquote>
</aside>
<p>Are both workflows failing (<code>sbatch</code> and <code>srun</code>)? If so, how did you get the valid GPU output?</p>
<aside class="quote no-group" data-username="import-antigravity" data-post="7" data-topic="114622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/import-antigravity/48/30128_2.png" class="avatar"> import-antigravity:</div>
<blockquote>
<p>It’s possible it’s an issue with pytorch-lightning but the problem only happens with my pre-trained Inception model which is just a <code>nn.Module</code>.</p>
</blockquote>
</aside>
<p>Could you remove Lightning, if you think this might be the issue and run a “pure” PyTorch script?</p>

username: import-antigravity
cooked: <p>That was from a python shell in an interactive <code>srun</code> session.</p>

username: ptrblck
cooked: <p>Thanks for the update.<br>
Unfortunately, I don’t understand when this error is triggered, since you were able to use both GPUs while running via <code>srun</code>, but nevertheless are also hitting the error when running through <code>sbatch</code> or <code>srun</code>.</p>
<p>If that’s the case, I would assume that your actual script is not working correctly or, as you suggested, Lightning or any other 3rd party library might cause problems.<br>
Were you able to run a pure PyTorch script?</p>

username: import-antigravity
cooked: <p>I’ll try and set aside some time today to try making a vanilla pytorch script. Thanks for helping me with this!</p>

username: import-antigravity
cooked: <p>I haven’t been able to make a pure pytorch script reproducing the error since it would be non-trivial because the code involved with the error extends a pytorch-lightning class (which extends <code>nn.Module</code>), but someone in another forum has suggested that the problem is that Pytorch does not support <code>self.to('cuda')</code> in multi-GPU scenarios, and you have to specify with <code>'cuda:n'</code> where n is the device index. Do you know if this is true?</p>

username: ptrblck
cooked: <p><code>to('cuda')</code> calls are supported in single-GPU and multi-GPU runs and will push the tensor or module to the default device.<br>
If your distributed setup masks all devices and sets only one GPU to be visible, the <code>to('cuda')</code> operation will use this visible device and <code>to('cuda:n')</code> would fail.<br>
On the other hand, if all devices are visible, you might recreate multiple CUDA contexts as also seems to be the issue in <a href="https://discuss.pytorch.org/t/strange-number-of-processes-per-gpu/116927/2">this topic</a>.</p>

username: import-antigravity
cooked: <p>I think I resolved it. I registered the Inception network as a part of my main model and then they were able to be on the same device.</p>