Is there any difference between x.to('cuda') vs x.cuda()? Which one should I use?

username: yxchng
cooked: <p>Is there any difference between x.to(‘cuda’) vs x.cuda()? Which one should I use? Documentation seems to suggest to use x.to(‘cuda’).</p>

username: neighthan
cooked: <p>I’m quite new to PyTorch, so there may be more to it than this, but I think that one advantage of using <code>x.to(device)</code> is that you can do something like this:</p>
<pre><code class="lang-python">device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
x = x.to(device)
</code></pre>
<p>Then if you’re running your code on a different machine that doesn’t have a GPU, you won’t need to make any changes. If you explicitly do <code>x = x.cuda()</code> or even <code>x = x.to('cuda')</code> then you’ll have to make changes for CPU-only machines.</p>

username: notnami
cooked: <p>.cuda()/.cpu() is the old, pre-0.4 way. As of 0.4, it is recommended to use .to(device) because it is more flexible, as neighthan showed above.</p>

username: Rui_Zhang
cooked: <p>Hi,</p>
<p>I am seeing a big timing difference in the following piece of code:</p>
<pre><code class="lang-auto">import torch
from datetime import datetime
a = torch.rand(20000,20000)
a = a.cuda()

#a = torch.rand(20000,20000)
#a.to('cuda')

i=0
t1 = datetime.now()
while i&lt; 500:
    a += 1
    a -= 1
    i+=1
t2 = datetime.now()
print('cuda', t2-t1)
</code></pre>
<p>This code will take 1 min.</p>
<p>Using</p>
<pre><code class="lang-auto">a = torch.rand(20000,20000)
a.to('cuda')
</code></pre>
<p>instead takes only ~1 second.</p>
<p>Am I getting something wrong?</p>
<p>I am using torch 1.0+</p>

username: ptrblck
cooked: <p>CUDA operations are asynchronous, so you would have to synchronize all CUDA ops before starting and stopping the timer:</p>
<pre><code class="lang-python">a = torch.rand(20000,20000)
a = a.cuda()

i=0
torch.cuda.synchronize()
t1 = time.time()
while i&lt; 500:
    a += 1
    a -= 1
    i+=1
torch.cuda.synchronize()
t2 = time.time()
print('cuda', t2-t1)

a = torch.rand(20000,20000)
a = a.to('cuda')


i=0
torch.cuda.synchronize()
t1 = time.time()
while i&lt; 500:
    a += 1
    a -= 1
    i+=1
torch.cuda.synchronize()
t2 = time.time()
print('cuda string', t2-t1)

&gt; cuda 5.500105619430542
&gt; cuda string 5.479088306427002
</code></pre>
<p>Also, it seems you’ve forgotten to reassign <code>a.to('cuda')</code> to <code>a</code>, so that this code will run on the CPU.</p>

username: Rui_Zhang
cooked: <p>Thank you very much for the prompt reply and the good catch to the problem.</p>

username: Brando_Miranda
cooked: <p>is this true? what if you are using multiple GPUs? would pytorch allocate the right GPUs automatically without have to specify them if one uses <code>.cuda()</code>? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>PS: asked similar question <a href="https://discuss.pytorch.org/t/run-pytorch-on-multiple-gpus/20932/65" class="inline-onebox">Run Pytorch on Multiple GPUs</a> but didn’t get an answer to that (if <code>.cuda()</code> allocates to the right gpu automatically or if it has to be manually done all the time).</p>

username: albanD
cooked: <p>They have the same behavior: <code>to("cuda")</code> and <code>.cuda()</code> when you don’t specify the device, will use device <code>0</code>.<br>
You can specify the device for both with <code>to("cuda:0")</code> and <code>.cuda(0)</code>.</p>

username: Brando_Miranda
cooked: <p>so we <strong>have</strong> to specify the device? always if we use multiple gpus?</p>
<p>(I am trying to have to modify my code as little as possible but I keep getting issues that tensors are not in the right GPUS…I am testing it with pytorch’s resnet18 )</p>

username: albanD
cooked: <p>No you don’t have to specify the device.<br>
As mentioned in my first sentence above, if you don’t specify the device, device 0 will be used.</p>

username: Brando_Miranda
cooked: <p>hmmm then I don’t understand why my model doesn’t work with multiple GPUs.</p>

username: albanD
cooked: <p>Do you try to use a GPU that is not the 0th somewhere?<br>
Note that you can use <code>CUDA_VISIBLE_DEVICES=0</code> to hide all but the 0th GPU and avoid this issue altogether <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>