`torch.cuda.current_device()` always return 0

oasjd7: <p><code>torch.cuda.current_device()</code> always return 0<br>
How can I print real using device?</p>

albanD: <p>The used device is the one returned by this function.<br>
What makes you think it uses another one? How do you change the device to use?</p>

oasjd7: <p><a class="mention" href="/u/alband">@albanD</a><br>
I have 4 GPUs. and  I allocate one training process to each one of the GPUs.<br>
So I use <code>os.environ['CUDA_VISIBLE_DEVICES']</code><br>
and I want to print the current running GPU number.</p>

SimonW: <p><code>CUDA_VISIBLE_DEVICES</code> is the mask used by CUDA to determine what devices it exposes to the user program, which is pytorch in this case. There is no way pytorch can know about that reliably.</p>

albanD: <p>Hi,</p>
<p>Within you application, gpu numbers will always start at 0 and grow up from there.<br>
When you use <code>CUDA_VISIBLE_DEVICES</code>, you hide some devices so they won’t be numbered.<br>
If you have 4 gpus: 0, 1, 2, 3.<br>
And run CUDA_VISIBLE_DEVICES=1,2 python some_code.py. Then the device that you will see within python are device 0, 1. Using device 0 in your code will use device 1 from global numering. Using device 1 in your code will use 2 outside.<br>
So in your case if you always set <code>CUDA_VISIBLE_DEVICES</code> to a single device, in your code, the device id will always be 0, that is expected. Unfortunately, there is no way to know what is the global numbering.</p>

lamour_hey: <p>Hi,</p>
<p>I also encounter same problems. Could you help me?</p>
<p>I have 8 cards but pytorch only detect the first one card.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae6c3faa74633b7fb1231153d7d88715f09fb8d0" title="50"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0_2_689x417.png" alt="50" data-base62-sha1="oT106dlxnsgmeR3QCxSpuwc0Qy4" width="689" height="417" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0_2_689x417.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0_2_1033x625.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0.png 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">50</span><span class="informations">1106×670 95.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Deepali: <p>Do you see all 8 devices in the output for <code>nivida-smi</code>?</p>
<p>Is CUDA_VISIBLE_DEVICES set?</p>

lamour_hey: <p>Yes. I see all the 8 devices. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>

Deepali: <p>CUDA_VISIBLE_DEVICES is not set?</p>

lamour_hey: <p>In the .bashrc? Not set</p>

Deepali: <p>just <code>echo $CUDA_VISIBLE_DEVICES</code> and verify.</p>

lamour_hey: <p>Hi Deepali,</p>
<p>I tried it but it still failed. I also run deviceQuery. It only detect one devices. Is it possible due to driver version?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3a11cd1bac0401709791f4842040bcb225e17a7d.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a11cd1bac0401709791f4842040bcb225e17a7d" title="25"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3a11cd1bac0401709791f4842040bcb225e17a7d_2_690x497.png" alt="25" data-base62-sha1="8hHPAVQMG9mpNe8rhTbzeYYTtPD" width="690" height="497" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3a11cd1bac0401709791f4842040bcb225e17a7d_2_690x497.png, https://discuss.pytorch.org/uploads/default/original/2X/3/3a11cd1bac0401709791f4842040bcb225e17a7d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3a11cd1bac0401709791f4842040bcb225e17a7d.png 2x" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">25</span><span class="informations">962×693 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/db0cf5c7c7ff4a106282edd0094c2528c7abecb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/db0cf5c7c7ff4a106282edd0094c2528c7abecb9" title="55"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/db0cf5c7c7ff4a106282edd0094c2528c7abecb9_2_538x500.png" alt="55" data-base62-sha1="vfOkDytnK008jWxjOHeDGleMuHn" width="538" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/db0cf5c7c7ff4a106282edd0094c2528c7abecb9_2_538x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/db0cf5c7c7ff4a106282edd0094c2528c7abecb9_2_807x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/db0cf5c7c7ff4a106282edd0094c2528c7abecb9.png 2x" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">55</span><span class="informations">809×751 91 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Deepali: <p>Yes, I was just about to suggest you to run deviceQuery and check. So looks like issue on the system.</p>

lamour_hey: <p>Shall I contact my admin directly?</p>

Deepali: <p>Yes, that will be helpful.</p>

RIVERS: <p>Try to use os.environ[‘CUDA_VISIBLE_DEVICES’] before u import torch</p>