Net.cuda() cannot access MIG instances

Tianyu9748: <pre><code class="lang-auto">GPU 0: NVIDIA A100-PCIE-40GB (UUID: GPU-744fbd6b-59e0-0daa-223e-19ac51400ee0)
  MIG 4g.20gb     Device  0: (UUID: MIG-a917c543-57fe-51ea-8bca-6ec8c68518c5)
  MIG 2g.10gb     Device  1: (UUID: MIG-c170d227-b149-5a1e-85b4-706e305dff4b)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc94c7b88abe4d8d3074a5ecf5c8ac0c49192188.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc94c7b88abe4d8d3074a5ecf5c8ac0c49192188" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc94c7b88abe4d8d3074a5ecf5c8ac0c49192188.png" alt="image" data-base62-sha1="tbOaaofp5HWog7RW9fPaveWmbmo" width="690" height="254" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">792×292 6.08 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To run on MIG 4g.20gb:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=MIG-a917c543-57fe-51ea-8bca-6ec8c68518c5 python vittrain.py
</code></pre>
<p>To run on MIG 2g.10gb:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=MIG-c170d227-b149-5a1e-85b4-706e305dff4b python vittrain.py
</code></pre>
<p>However, the second command cannot execute, error returned is <code>RuntimeError: No CUDA GPUs are available</code></p>
<p>and <code>torch.cuda.device_count()</code> returns 1, which should be 2.</p>
<p>Is this a bug?</p>
<p>Best<br>
Max</p>

ptrblck: <aside class="quote no-group" data-username="Tianyu9748" data-post="1" data-topic="164379">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tianyu9748/48/53979_2.png" class="avatar"> Tianyu9748:</div>
<blockquote>
<p>and <code>torch.cuda.device_count()</code> returns 1, which should be 2.</p>
</blockquote>
</aside>
<p>You won’t be able to use multiple MIG devices in a single script, so that’s expected.<br>
Could you post the exact MIG commands you’ve used which would reproduce that the second device cannot be used?</p>

Tianyu9748: <p>Hi ptrblck,<br>
Command is:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=MIG-c170d227-b149-5a1e-85b4-706e305dff4b python vgginfer.py
</code></pre>
<p>Every py file I can run it with <code>CUDA_VISIBLE_DEVICES=$UUID python model.py</code> on the first MIG instance.<br>
However, I cannot run it on the second instance.</p>
<p>I also try add <code>CUDA_VISIBLE_DEVICES=$UUID</code> before <code>import torch</code> in the py file, it also doesn’t work.</p>
<p>Best<br>
Max</p>

ptrblck: <p>The posted commands don’t show how you’ve created the MIG setup and how to reproduce the issue, so could you post a minimal, executable code snippet which reproduces the issue, please?</p>

Tianyu9748: <p>I have found the reason, MPS.<br>
MPS was ran on the Top of the device, instead of on each instance.<br>
In this case, MPS places data and launches kernels on the first instance only, since it has no idea about other instances.</p>
<p>After I shut down MPS, it works for me.</p>
<p>Best<br>
Max</p>