Difference between torch.device("cuda") and torch.device("cuda:0")

fermat97: <p>Hi, I am using a computation server with multiple nodes each of which has 4 GPUs and they are managed with <code>SLURM</code>. I want my code to send the data and model to one or multiple GPUs. I assumed if I use <code>torch.device("cuda")</code> it makes the device to be a GPU without particularly specifying the device name (0,1,2,3).  I would like to make sure if I understand the difference between these two command correctly.</p>
<pre><code class="lang-auto">torch.device("cuda") # without specifying the cuda device number 
torch.device("cuda:0") # use cuda device 0
</code></pre>
<p>Is that correct?</p>

LeviViana: <p>If you run <code>.device("cuda")</code>, your tensor will be routed to the CUDA current device, which by default is the <code>0</code> device.</p>

fermat97: <p>Is there any way to choose automatically what ever <code>GPU index</code> is available?</p>

LeviViana: <p>What do you mean by available ? Do you mean in terms of GPU utilization ? Memory ? Or simply existence ?</p>

fermat97: <p>Existence. I launch a job with <code>SLURM</code>, so I cannot define <code>CUDA_VISIBLE_DEVICES</code>, it simply sends my code to a node but does not define which GPU to use, and as you said it goes automatically to index <code>0</code>, in other words I can never use other GPUs on the node.</p>

LeviViana: <p>You could try <code>torch.cuda.device_count()</code> to get the number of GPUs available, and maybe <code>torch.cuda.get_device_name(device_id)</code> to get the name of the used device.</p>

bing: <p>What will be the <code>device_id</code> in <code>torch.cuda.get_device_name(device_id)</code> then?</p>

albanD: <p>Any number in <code>range(torch.cuda.device_count())</code>.</p>

bing: <p>okay, so I am trying to run my model on 2 gpu’s I have. Following is my code and it is giving me<br>
<code>Type Error: '&lt;' not supported between instances of 'range' and 'int' </code></p>
<pre><code class="lang-auto">device_id = torch.cuda.device_count()
device = torch.cuda.get_device_name(range(device_id))

if torch.cuda.device_count()&gt;1:
   model = nn.DataParallel(model)
   model = model.to(device)
elif train_on_gpu:
   model = model.to(device)
</code></pre>

albanD: <p><code>torch.cuda.device_count()</code> will give you the number of available devices, not a device number<br>
<code>range(n)</code> will give you all the integers between 0 and n-1 (included). Which are all the valid device numbers.</p>

bing: <p>Yes, I am doing the same -</p>
<pre><code class="lang-auto">device_id = torch.cuda.device_count()
device = torch.cuda.get_device_name(range(device_id))
</code></pre>
<p>but it is throwing an error<br>
<code>Type Error: '&lt;' not supported between instances of 'range' and 'int'</code></p>

albanD: <p><code>get_device_name</code> expects the number corresponding to the device.<br>
<code>range()</code> returns a list of numbers.</p>

bing: <p>Ohh, sorry I am stupid, I forgot to include it in a loop</p>

bing: <p>okay, another doubt, after I stored name of gpu’s in the device<br>
<code>device = torch.cuda.get_device_name(range(device_id))</code>  then how to use it in model like in the following calling -</p>
<pre><code class="lang-auto">if torch.cuda.device_count()&gt;1:
   model = nn.DataParallel(model)
   model = model.to(device)
elif train_on_gpu:
   model = model.to(device)
</code></pre>

albanD: <p>You can just use the device id, you don’t need the name.<br>
You can do <code>.cuda(deviceid)</code> or <code>.to("cuda:{}".format(deviceid))</code>.</p>

bing: <p>I am getting<br>
<code>RuntimeError: CUDA error: invalid device ordinal</code><br>
while running both ways.</p>

albanD: <p>If you have <code>torch.cuda.device_count() == 1</code> Then you can use only <code>0</code> as a device id.<br>
If you have <code>torch.cuda.device_count() == 2</code> Then you can use <code>0</code> and <code>1</code> as valid device ids. etc.</p>

bing: <p>This is how I am doing -</p>
<pre><code class="lang-auto">device_id = torch.cuda.device_count()
if torch.cuda.device._count()&gt;1:
   model = nn.DataParallel(model)

model = model.to("cuda:{}".format(device_id))
</code></pre>

albanD: <p><code>device_id = torch.cuda.device_count()</code> this does not give you a valid device id but the number of available devices !</p>

bing: <p>okay, I am going to cry now.<br>
Then how to include multiple gpu’s in this code, like in the following way-</p>
<pre><code class="lang-auto">device_id = torch.cuda.device_count()
if torch.cuda.device_count() == 1:
   model = model.to("cuda:{}".format(0))
elif torch.cuda.device_count ==2:
   model = nn.DataParallel(model)
   model = model.to("cuda:{}".format(1))

</code></pre>
<p>But it isn’t too much hard-coded way to do things?</p>