CUDA_VISIBLE_DEVICE is of no use

Liang: <p>I have a 4-titan XP GPU server. When i use os.environ[“CUDA_VISIBLE_DEVICES”] =“0,1” to allocate GPUs for a task in python, I find that only GPU 0 is used. And there is out of memroy problems even GPU 1 is free.<br>
Should I allocate memory to different GPUs myself?</p>

colesbury: <p>Use <code>CUDA_VISIBLE_DEVICES</code> (not “DEVICE”). You have to set it before you launch the program – you can’t do it from within the program.</p>

Liang: <p>My bad, there is a typo in my post. But in my code, when i use<br>
os.environ[“CUDA_VISIBLE_DEVICES”] =“1,2”<br>
, only GPU 1 is used. At least, such a line in Python has its own effect. It can control the use of GPUs.<br>
However, It is supposed to make GPU 1 and 2 available for the task, but the result is that only GPU 1 is available. Even when GPU 1 is out of memory, GPU 2 is not used. Is there any other switches controlling parallel computing between two GPUs?<br>
BTW, another question: Does Pytorch tend to use GPUs one by one or allocate equal memories to GPUs</p>

ptrblck: <p>You can push your data to a specific GPU using <code>.cuda(gpu_id)</code>. E.g. you can load a generator network on one GPU and the discriminator to the other.<br>
Another option is to use the <a href="http://pytorch.org/docs/master/nn.html#dataparallel-layers-multi-gpu-distributed" rel="nofollow noopener">DataParallel</a> module.</p>

Liang: <p>Do you mean that if I want to use two GPUs at the same time, i have to change my source codes and add parallel-related codes?<br>
Or it would just use one GPU at most if not specified.</p>

ptrblck: <p>Basically it is just one line to use DataParallel:</p>
<pre><code>net = torch.nn.DataParallel(model, device_ids=[0, 1, 2])
output = net(input_var)
</code></pre>
<p>Just wrap your model with <code>DataParallel</code> and call the returned net on your data.<br>
The <code>device_ids</code> parameter specifies the used GPUs.</p>

Liang: <p>It doesn’t work for me. Still, only the first GPU is used. Is there any more tricks?</p>

Liang: <p>Does data parallel only support more than batch=1? Actually, I only use batch=1.</p>

SimonW: <p>You need to use &gt;=#gpu batch set to apply data parallel. As its name suggests, data parallel just pushes computation for different data in a batch to different gpus.</p>

MrTuo: <p>Hi! I’m adding  <code>os.environ['CUDA_VISIBLE_DEVICES'] = "2"</code>  in my code does not work, the code always select first GPU, however  <code>CUDA_VISIBLE_DEVICES=2 python train.py</code>  works.<br>
I find that <code>os.environ['CUDA_VISIBLE_DEVICES']</code> in this <a href="https://github.com/victoresque/pytorch-template/blob/master/train.py#L68" rel="nofollow noopener">code</a> is work. Do you know why?</p>

pjavia: <p><a class="mention" href="/u/mrtuo">@MrTuo</a> This is how pytorch 0.4.1 convention works. If you say CUDA_VISIBLE_DEVICES=2, 3. Then for pytorch GPU - 2 is cuda:0 and GPU - 3 is cuda:1. Just check your code is consistent with this convention or not?</p>

mbanani: <p>I had this same issue where setting <code>CUDA_VISIBLE_DEVICES=2 python train.py</code> works but setting  <code>os.environ['CUDA_VISIBLE_DEVICES'] = "2"</code> didn’t. The cause of the issue for me was importing the torch packages before setting <code>os.environ['CUDA_VISIBLE_DEVICES']</code>, moving it to the top of the file before importing torch solved it. Hope this helps.</p>

RIVERS: <p>That’s helpful for me, thanks 3000 times</p>

cheng_xie: <p>thank you, it works.</p>

MattBC: <p>Hey, I have the opposite problem: code is using both of my GPUs by default, no matter what I do.  These are different GPU models and I DO NOT want to use them for parallel processing.  I’ve tried setting GPU <span class="hashtag">#0</span> with cuda_visible_devices, tried setting it with torch, moved it to beginning of code, nothing is working.</p>
<p>Just for the record, I am doing deep learning object detection importing arcgis and torch.  Everything else seems to work fine now, until I try to test learning rate and it tells me my GPUs are imbalanced and that I should exclude GPU <span class="hashtag">#1</span>.  I never wanted GPU <span class="hashtag">#1</span> to be utilized in the first place.</p>
<p>EDIT: Nevermind, it appears to be working now after I did move it towards beginning of code.  I guess I reset the kernel somehow, which made it work.  I’m just a rookie <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>

Lame: <p>When I tried this solution (I have two gpu), it shows an error<br>
<code>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!</code></p>

ptrblck: <p>I’m not sure which solution you are referring to, but the error could be raised, if you manually specify a device inside the model.<br>
Could you post an executable code snippet, which would reproduce the issue, so that we could debug it, please?</p>

saurbhc: <p>Saved 30 hours of my life. Thanks a ton.</p>

Arthur_Conmy: <p>It’s also possible to run into this with bad conda environments.</p>
<p>For me</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_VISIBLE_DEVICES"] = "2" # just use one GPU on big machine
import torch
assert torch.cuda.device_count() == 1
</code></pre>
<p>Failed, but it was because my environment was problematic, and only</p>
<pre><code class="lang-auto">import torch 
print(torch.cuda.current_device())
</code></pre>
<p>actually raised an error.</p>