Only first gpu is allocated (eventhough I make other gpus visible)

username: Yun_Lee
cooked: <p>The problem is that<br>
eventhough I specified certain gpus that can be shown with os.environ[“CUDA_VISIBLE_DEVICES”],<br>
the program keeps using <strong>only first</strong> gpu.</p>
<p>(But other program works fine and other specified gpus are allocated well.<br>
because of that, I think it is not nvidia or system problem.<br>
nvidia-smi shows all gpus well and there’s no problem.<br>
I didn’t have problem with allocating gpus with below codes before (except when the system is not working)<br>
it works fine when I use</p>
<pre><code class="lang-auto">torch.device("cuda:other_gpu_id" if torch.cuda.is_available() else "cpu")
</code></pre>
<p>)</p>
<pre><code class="lang-auto">os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBILE_DEVICES"] = str(args.gpu)
</code></pre>
<p>I wrote that before running main function.<br>
and it works fine for other programs in same system (pytorch 1.8.1 with cuda11.1).</p>
<p>I printed that args.gpu variable, and could see that the value is not “0”.</p>
<p>and I tried below</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=4,5 python script.py
</code></pre>
<p>but still only first gpu is allocated.</p>
<p>when I asked the question in stackoverflow,<br>
I could get the answer that those “os.envrion~” work in tensorflow and keras based code and not for pytorch code</p>
<p>As I didn’t have problem running pytorch code with those specifying-gpu codes before,<br>
I want to know what makes it work and not work.</p>

username: ptrblck
cooked: <p>I’m unsure what the actual issue is. Are you trying to mask specific GPUs and this is not working correctly using the <code>os.environ</code> call? If so, set <code>CUDA_VISIBLE_DEVICES</code> in the terminal (either as part of the actual command or <code>export</code> it).</p>
<p>Or are you making all GPUs visible and cannot use any other device?<br>
In that case, what does <code>x = torch.randn(1, device='cuda:1')</code> return and what does <code>nvidia-smi</code> show?</p>

username: Yun_Lee
cooked: <p>Thank you for answering to the question.</p>
<p>yes the actual issue is first one.<br>
and this is not working correctly using os.environ call.</p>
<p>the program worked well with second case. If I specify available gpus with torch.device(), it worked well and can see it is allocated well when I see nvidia-smi.</p>
<p>I am curious why sometimes os.environ call do not work in pytorch program</p>
<p>I am sorry for changing the word.<br>
when I tried<br>
CUDA_VISIBLE_DEVICES=4 python script.py again<br>
I can see certain gpu is allocated well…<br>
(I saw… setting CUDA_VISIBLE_DEVICES in terminal didn’t work …and couldn’t find the answer here and that’s why I made new question<br>
but now maybe I need to change the question or delete it)</p>
<p>can I ask you why os.environ does not work in some cases?<br>
I put this on top of the code (after importing) before main function and until now it worked well.<br>
after I declared visible devices with os.environ.<br>
[object].cuda() was allocated well with visible devices.</p>
<p>but for this program, it keeps assigning to only first gpu when I use [object].cuda()</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Yun_Lee" data-post="3" data-topic="120088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yun_lee/48/13932_2.png" class="avatar"> Yun_Lee:</div>
<blockquote>
<p>I am curious why sometimes os.environ call do not work in pytorch program</p>
</blockquote>
</aside>
<p>If I understand you correctly, setting <code>CUDA_VISIBLE_DEVICES</code> in the terminal works correctly, but doesn’t work if you try to set it inside the script via <code>os.environ</code>.<br>
This is a commonly met issue, if you are not making sure that the env var is set before importing any CUDA related libraries, which is why I’m recommending to set it in the terminal.</p>