Why is my gpu device specification weird?

lingvisa: <p>I have 4 gpu and 3 are being used, but the 3rd one is empty, so I want to use it, as shown below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/349bccd1b862edae52c9b765e4d6979ea0941116.png" data-download-href="https://discuss.pytorch.org/uploads/default/349bccd1b862edae52c9b765e4d6979ea0941116" title="Screen Shot 2021-10-21 at 10.39.09 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349bccd1b862edae52c9b765e4d6979ea0941116_2_690x323.png" alt="Screen Shot 2021-10-21 at 10.39.09 PM" data-base62-sha1="7voCYhlewCaXA59gAJzxGeqCGYS" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349bccd1b862edae52c9b765e4d6979ea0941116_2_690x323.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349bccd1b862edae52c9b765e4d6979ea0941116_2_1035x484.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/4/349bccd1b862edae52c9b765e4d6979ea0941116.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-10-21 at 10.39.09 PM</span><span class="informations">1168×548 158 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then in my script, I have:<br>
<code>device = torch.device("cuda:3" if torch.cuda.is_available() else "cpu")</code></p>
<p>However, this gives me a weird error message:<br>
<code>RuntimeError: CUDA error: invalid device ordinal</code></p>
<p>Then I try gpu cuda:0, 1, or 2, it correctly used the GPU but got an out of memory error. This makes sense because other programs are using them. What I don’t understand is that, why cuda:3 produce that error?</p>
<p>Then I changed to:<br>
evice = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)</p>
<p>And at command line i use:<br>
<code>CUDA_VISIBLE_DEVICES=3 python my_script.py</code></p>
<p>This works fine, but this doesn’t allow me to use nohup:<br>
<code>nohup CUDA_VISIBLE_DEVICES=3 python my_script.py &amp;</code></p>
<p>Anyway, why. can’t I specify cuda:3 in the script without using the command-line option?</p>

albanD: <p>Are you sure you don’t have a <code>CUDA_VISIBLE_DEVICE=0,1,2</code> by default?</p>

lingvisa: <p>Oh, yes, it is in my bash file.</p>
<p><code>export CUDA_VISIBLE_DEVICES=0,1,2</code></p>
<p>Does this overwrite the device=torch.device(‘cuda:3’) setting in the script?</p>

albanD: <p>Well, it hides the 4th  device. So you cannot use it.</p>

lingvisa: <p>A further question on this:</p>
<p>I want to use multiple gpu, so I do:</p>
<pre><code class="lang-auto">device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
def get_bert():
    from transformers import BertTokenizer, BertModel
    tokenizer = BertTokenizer.from_pretrained("hfl/chinese-roberta-wwm-ext", model_max_length=128)
    model = BertModel.from_pretrained("hfl/chinese-roberta-wwm-ext")
    if torch.cuda.device_count() &gt; 1:
        print("Let's use", torch.cuda.device_count(), "GPUs!")
        model = torch.nn.DataParallel(model)
        #model = torch.nn.DataParallel(model, device_ids=[0, 1])
    model = model.to(device)

    return tokenizer, model
</code></pre>
<p>From nvidia-smi screenshot, you can see that there are 4 GPUs, however, the torch.cuda.device_count()=3, and my program only uses 3 GPUs(0,1,2), while GPU 3 is idle. If I specify:<br>
<code>device_ids=[0,1,2,3]</code><br>
It says ‘AssertionError: Invalid device id’.</p>
<p>Another question is when using multl-GPUs, is this specification of device right?<br>
<code>device = torch.device("cuda" if torch.cuda.is_available() else "cpu")</code></p>
<p>This seems to only get ‘cuda:0’, but the code above does use 3 GPUs, which I don’t quite understand.</p>

albanD: <blockquote>
<p>From nvidia-smi screenshot, you can see that there are 4 GPUs, however, the torch.cuda.device_count()=3,</p>
</blockquote>
<p>If you still have the CUDA_VISIBLE_DEVICES that hides the 4th GPU that is expected. Keep in mind that nvidia-smi does NOT respect this env variable and will always show you all the devices.</p>
<blockquote>
<p>This seems to only get ‘cuda:0’, but the code above does use 3 GPUs, which I don’t quite understand.</p>
</blockquote>
<p>It does get you device 0 by default.<br>
3 GPUs are used because of your DataParallel call where you don’t specify the device_ids and so it uses all visible GPUs.</p>

lingvisa: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="134841">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>If you still have the CUDA_VISIBLE_DEVICES that hides the 4th GPU that is expected.</p>
</blockquote>
</aside>
<p>My fault:</p>
<p>‘If you still have the CUDA_VISIBLE_DEVICES that hides the 4th GPU that is expected.’: I thought I had disabled the setting in the bash, but it’s still on.</p>