CUDA_VISIBLE_DEVICE not working

username: avalon1511
cooked: <p>I am trying to run my code on GPU 6  of a 10 GPU machine. I have tried the following:</p>
<ol>
<li>In the terminal: <code>CUDA_VISIBLE_DEVICES=6 python myscript.py</code>
</li>
<li>In the code : <code>CUDA_VISIBLE_DEVICES=6</code> at the top of my script before I import anything else</li>
<li>In the terminal:</li>
</ol>
<pre><code class="lang-auto">export CUDA_VISIBLE_DEVICES=6
python myscript.py
</code></pre>
<ol start="4">
<li>In the code before I import torch:</li>
</ol>
<pre><code class="lang-auto">import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="6"
</code></pre>
<p>None of these work, my code just defaults to GPU 0.</p>
<p>Really frustrated and confused, please help.</p>

username: ptrblck
cooked: <p>How did you verify that it’s not working? PyTorch will use <code>cuda:0</code> for the masked GPU in case you were expecting <code>cuda:6</code> to work.<br>
Check <code>nvidia-smi</code> and you should see that <code>GPU6</code> is the one being used via <code>cuda:0</code> in PyTorch after masking it with the env variable.</p>

username: avalon1511
cooked: <p>When I run <code>nvidia-smi</code> I see as most of my GPU memories being free which should not be the case if the code was running on GPU 6.</p>

username: avalon1511
cooked: <p>Found an interesting error. I ran</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.device_count()
&gt;&gt;&gt;1
&gt;&gt;&gt; torch.cuda.current_device()
0
</code></pre>
<p>so only GPU 0 is being recognized for some reason even though other people are running their code on all other GPUs.</p>
<p>What am I doing wrong?</p>

username: ptrblck
cooked: <p>As mentioned before, <code>CUDA_VISIBLE_DEVICES=</code> will make the specified GPUs available to the system while PyTorch will still name the available GPUs starting from device id 0.<br>
In your use case this would mean that <code>CUDA_VISIBLE_DEVICES=6</code> will use the GPU with id 6 and PyTorch will see and use it as <code>cuda:0</code>. It will thus be mapped from <code>GPU6</code> in the system → <code>cuda:0</code> in PyTorch.<br>
The same applies for any other usage of <code>CUDA_VISIBLE_DEVICES</code> and you can also change the order of GPUs.</p>

username: avalon1511
cooked: <p>That makes sense! The particular error I was facing was because the GPU server needed a restart. it is fixed now</p>
<p>Although I have a new error when wrapping my module in <code>nn.DataParallel</code></p>
<pre><code class="lang-auto">module must have its parameters and buffers on device 
cuda:6 (device_ids[0]) but found one of them on device: cuda:0
</code></pre>
<p>Not sure why this is happening. My setup is as follows:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
&lt;model description&gt;

model = MyModel()
model = torch.nn.DataParallel(model, device_ids=[6,7]).cuda()

x = torch.randn(4).cuda()
out = model(x)
</code></pre>

username: avalon1511
cooked: <pre><code class="lang-auto">model = torch.nn.DataParallel(model, device_ids=[6,7]).cuda()
</code></pre>
<p>to</p>
<p><code>model = torch.nn.DataParallel(model, device_ids=[0,1]).cuda()</code></p>