Running on specific GPU device

username: CCL
cooked: <p>I’m trying to specify specify which single GPU to run code on within Python code, by setting the GPU index visible to PyTorch. Here’s what I’ve tried:</p>
<pre><code class="lang-auto">for i in range(8): #8 gpus
    os.environ["CUDA_AVAILABLE_DEVICES"] = str(i)
    print(torch.cuda.device_count())
    # this line always outputs 8 (all 8 devices) instead of 1...
    ...
</code></pre>
<p>I’m using PyTorch 1.0.0. How do I specify which GPU machine (by index or otherwise) to run code on (without using <code>.to(device)</code>) within Python code?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/ccl">@CCL</a>, you will need to set the <code>CUDA_AVAILABLE_DEVICES</code> env var before launching the process. Sth like:</p>
<pre><code class="lang-auto">$ CUDA_AVAILABLE_DEVICES=0 python main.py
</code></pre>
<p>If you just want to set the default device, you can use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.set_device" rel="nofollow noopener"><code>set_device</code></a></p>
<h3>Update</h3>
<p>Please ignore the code above. I miss-read the variable name. It should be <code>CUDA_VISIBLE_DEVICES</code></p>

username: CCL
cooked: <p>Hi thanks for the reply, is it possible to set it with python instead of in shell?</p>

username: mrshenli
cooked: <blockquote>
<p>Hi thanks for the reply, is it possible to set it with python instead of in shell?</p>
</blockquote>
<p>Yes, but you need to make sure it is set before initializing the CUDA context. See the code below:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1" 
torch.cuda.device_count()  # print 1
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"  
torch.cuda.device_count()  # still print 1
</code></pre>

username: CCL
cooked: <p>Do you mean that once it’s set, it cannot be changed? For my case, I’m hoping to make GPU 0 visible on the 1st iteration, GPU 1 visible on the 2nd, etc till GPU 7 and iter 8. Is there a way to do this from Python? Thanks a lot!</p>

username: mrshenli
cooked: <blockquote>
<p>Do you mean that once it’s set, it cannot be changed?</p>
</blockquote>
<p>I believe so.</p>
<blockquote>
<p>For my case, I’m hoping to make GPU 0 visible on the 1st iteration, GPU 1 visible on the 2nd, etc till GPU 7 and iter 8. Is there a way to do this from Python?</p>
</blockquote>
<p>Can this be done by explicitly passing <code>torch.cuda.device(i)</code> to tensors/modules or use <code>torch.cuda.set_device(i)</code>? Is there any reason that you would like to change the visible devices?</p>

username: CCL
cooked: <p>I’m trying to do some parallelism but can’t figure out how to initialise processes with different ranks, each process on 1 different GPU. I am modifying some distributed computing code, but instead of having numerous nodes, I only have 1 8 GPU machine to work with.</p>
<p>(Pls bear with me, I’m a beginner in distributed computing!) So far I’ve worked out that the line <code>dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url, world_size=1, rank=args.rank)</code> intialises the same process on all 8 GPUs, but this causes problems later in my code, where I need to get the specific GPU machine index. I tried to do this with <code>torch.cuda.current_device()</code> but it also returns 0 despite <code>nvidia-smi</code> showing that all 8 GPUs have been used.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="CCL" data-post="7" data-topic="89841">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/dbc845/48.png" class="avatar"> CCL:</div>
<blockquote>
<p>So far I’ve worked out that the line <code>dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url, world_size=1, rank=args.rank)</code> intialises the same process on all 8 GPUs</p>
</blockquote>
</aside>
<p>The <code>init_process_group</code> API only sets up the process where this function is invoked. And, as the <code>world_size</code> is set to 1, It only expects one process in the distributed training gang. If you would like to use multiple processes, please see this <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">example</a>.</p>
<blockquote>
<p>but this causes problems later in my code, where I need to get the specific GPU machine index.</p>
</blockquote>
<p>To get machine index, will it work if you use <code>args.rank</code>?</p>
<blockquote>
<p>I tried to do this with  <code>torch.cuda.current_device()</code>  but it also returns 0 despite  <code>nvidia-smi</code>  showing that all 8 GPUs have been used.</p>
</blockquote>
<p>The <code>torch.cuda.current_device()</code>  returns the current device. By default, it is the first GPU, which is indeed indexed by 0.</p>

username: CCL
cooked: <p>I just tried using args.rank, but it seems like they all return rank 0. I’m really quite lost on how to do parallelism.</p>

username: mrshenli
cooked: <p>How did you launch those 8 processes? Did you launch it using similar code in the <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">example</a> or the <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md" rel="nofollow noopener">launching script</a>?</p>
<p>And how did you set <code>args.rank</code>? I presume it’s command line args + argparse?</p>

username: mrshenli
cooked: <p>It will be helpful to have a self-contained min repro code. So that we can help debug.</p>

username: Jonathan
cooked: <p>To use a different gpu in the system, isn’t when you declare the device<br>
mydevice=torch.device(“cuda:2”)<br>
or<br>
mydevice=torch.device(“cuda”, 2)</p>
<p>the point is you have to pass the ordinal for the gpu you want to use.<br>
See torch.device at <a href="https://pytorch.org/docs/stable/tensor_attributes.html#torch.torch.device" class="inline-onebox" rel="noopener nofollow ugc">Tensor Attributes — PyTorch 1.8.1 documentation</a></p>