What is the difference between doing `net.cuda()` vs `net.to(device)`?

username: pinocchio
cooked: <p>I was going through this post (<a href="https://discuss.pytorch.org/t/solved-make-sure-that-pytorch-using-gpu-to-compute/4870/29" class="inline-onebox">[SOLVED] Make Sure That Pytorch Using GPU To Compute</a>) and I had the question, what is the difference between these two pieces of code?</p>
<pre><code class="lang-auto">import torch.nn as nn
net = nn.Sequential(OrderedDict( [ ('fc1',nn.Linear(3,1)) ]) )
net.cuda()
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

use_cuda = torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")

net = nn.Sequential( OrderedDict([ ('fc1', nn.Linear(3,1)) ]) )
net.to(device)
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

use_cuda = torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")

net = nn.Sequential( OrderedDict([ ('fc1', nn.Linear(3,1)) ]) )
net = net.to(device)
</code></pre>
<p>which one is the recommended one? Which one is the one that is hardware agnostic (i.e. no matter type of gpu or even cpu).</p>
<p>is there some sort of internal flag I can check to see if things are properly placed in GPU?</p>

username: nairbv
cooked: <p><code>cuda()</code> and <code>to('cuda')</code> are going to do the same thing, but the later is more flexible. As you can see in your example code, you can specify a device that might be ‘cpu’ if cuda is unavailable.</p>
<p>If you attempt to call <code>cuda()</code> on a system that doesn’t have a GPU, you’ll get:<br>
<code>AssertionError: Torch not compiled with CUDA enabled</code>.</p>
<p>With the explicit call, you can also use multiple cuda devices – e.g. <code>to('cuda:0')</code> is different from <code>to('cuda:1')</code>. The simpler <code>cuda()</code> call will just use the default cuda device.</p>

username: pinocchio
cooked: <p>what about:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

use_cuda = torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")

net = nn.Sequential( OrderedDict([ ('fc1', nn.Linear(3,1)) ]) )
net = net.to(device)
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

use_cuda = torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")

net = nn.Sequential( OrderedDict([ ('fc1', nn.Linear(3,1)) ]) )
net.to(device)
</code></pre>
<p>?</p>
<p>Whats the difference? Which one do I choose?</p>

username: nairbv
cooked: <aside class="quote no-group" data-username="pinocchio" data-post="3" data-topic="69278">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p>net.to(device)</p>
</blockquote>
</aside>
<p>Your second variant is copying the net to the device, but not assigning the copy to anything. <code>to(device)</code> is not an in-place operation, so this effectively doesn’t do anything.</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="nairbv" data-post="4" data-topic="69278">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nairbv/48/13143_2.png" class="avatar"> nairbv:</div>
<blockquote>
<p>Your second variant is copying the net to the device, but not assigning the copy to anything. <code>to(device)</code> is not an in-place operation, so this effectively doesn’t do anything.</p>
</blockquote>
</aside>
<p>Not sure if thats correct. It seems that there are no errors thrown (if I have data not in cuda/gpu the type of the tensor won’t match the net so an error would be thrown). So it seems that <code>net.to(device)</code> mutates my net and must put it in gpu…otherwise I would have seen an error thrown.</p>
<p>I wonder if this a unintended bug in pytorch and where they meant <code>net = net.to(device)</code> to be the only thing to work.</p>

username: nairbv
cooked: <p>ah, you’re correct. I’m used to tensor <code>to</code> which is out of place. It might be better habit to re-assign, as I have seen a number of mistakes where a <code>tensor.to(device)</code> was assumed to be in-place.</p>
<pre><code class="lang-auto">In [14]: net=torch.nn.Linear(3,4)
In [15]: net.weight.device
Out[15]: device(type='cpu')
In [16]: net.to('cuda')
Out[16]: Linear(in_features=3, out_features=4, bias=True)
In [17]: net.weight.device
Out[17]: device(type='cuda', index=0)
</code></pre>

username: pinocchio
cooked: <aside class="quote no-group" data-username="nairbv" data-post="6" data-topic="69278">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nairbv/48/13143_2.png" class="avatar"> nairbv:</div>
<blockquote>
<p>It might be better habit to re-assign, as I have seen a number of mistakes where a <code>tensor.to(device)</code> was assumed to be in-place.</p>
</blockquote>
</aside>
<p>what does the in-placeness of <code>.to(device)</code> have to do with this?</p>
<p>I’m confused.</p>

username: ptrblck
cooked: <p>If you call <code>.to()</code> on a tensor, the operation will not be performed in-place, thus you need to assign a new variable to the it.<br>
On the module however, all parameters will be pushed internally to the specified device, so you don’t need to assign the model back to the call.</p>
<p>However, as <a class="mention" href="/u/nairbv">@nairbv</a> mentioned, it might be a good habit just to use the assignment by default to avoid possible errors. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="69278">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>However, as <a class="mention" href="/u/nairbv">@nairbv</a> mentioned, it might be a good habit just to use the assignment by default to avoid possible errors. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>This is what I’m trying to understand. I always expected <code>.to(device)</code> to work the way it does. I don’t understand why I need to re-assign. I always expected <code>.to(device)</code> to mutate things.</p>

username: ptrblck
cooked: <p>It does not work in-place on tensors, so you have to reassign the result:</p>
<pre><code class="lang-python">x = torch.randn(1, device='cpu')
print(x.device)
&gt; cpu

x.to('cuda')
print(x.device)
&gt; cpu

x = x.to('cuda')
print(x.device)
&gt; cuda:0
</code></pre>

username: Ved
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> You have been so patient in explaining this <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:">  <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=10" title=":joy:" class="emoji" alt=":joy:"></p>

username: Ardeal
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="69278">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If you call <code>.to()</code> on a tensor, the operation will not be performed in-place, thus you need to assign a new variable to the it.<br>
On the module however, all parameters will be pushed internally to the specified device, so you don’t need to assign the model back to the call.</p>
<p>However, as <a class="mention" href="/u/nairbv">@nairbv</a> mentioned, it might be a good habit just to use the assignment by default to avoid possible errors.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Do you mean that <code>tensor.to()</code> is not in-place, and <code>tensor.cuda()</code> is in-place?<br>
that is the major difference with <code>to()</code> and <code>cuda()</code>?</p>

username: ptrblck
cooked: <p>No, neither are in-place on a tensor.<br>
Calling <code>to()</code> or <code>cuda()</code> on an <code>nn.Module</code> object will internally move all parameters and buffers to the device, dtype, or memory format so you wouldn’t need to reassign a model:</p>
<pre><code class="lang-python">model.to('cuda') # works
tensor.to('cuda') # tensor is still on the original device afterwards, as the CUDATensor wasn't assigned
</code></pre>

username: Ardeal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>there is no difference between <code>to()</code> and <code>cuda()</code>.<br>
there is difference when we use <code>to()</code> and <code>cuda()</code> between Module and tensor:<br>
on Module(i.e. network), Module will be moved to destination device,<br>
on tensor, it will still be on original device. the returned tensor will be move to destination device</p>
<p>right?</p>

username: ptrblck
cooked: <p>Yes, almost. The first point is right, in case you are only concerned about the device, but <code>to()</code> can also change the dtype, memory-layout etc. so has more functionality that <code>cuda()/cpu()</code>.</p>

username: Ardeal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>if we only consider device, there is no difference between <code>to()</code> and <code>cuda()</code>.<br>
if we consider other functionalities, <code>to()</code> has much more functionalities than <code>cuda()</code></p>

username: Ardeal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>One more questions:</p>
<pre><code class="lang-python"># code 1:
device=torch.device('cuda')
net.to(device)

# code 2:
device=torch.device('cuda:0') # 'cuda:1'
net.to(device)
</code></pre>
<p>what is difference between ‘cuda’ and ‘cuda:0’ or ‘cuda:1’? what case should I use them in?</p>

username: ptrblck
cooked: <p><code>torch.device('cuda')</code> (or just the <code>'cuda'</code> string) will use the default device, while <code>torch.device('cuda:1')</code> (or the <code>cuda:1</code> string) will explicitly use GPU1.<br>
The <a href="https://pytorch.org/docs/stable/notes/cuda.html">CUDA semantics docs</a> explain this behavior with some examples:</p>
<pre><code class="lang-python">cuda = torch.device('cuda')     # Default CUDA device
cuda0 = torch.device('cuda:0')
cuda2 = torch.device('cuda:2')  # GPU 2 (these are 0-indexed)

x = torch.tensor([1., 2.], device=cuda0)
# x.device is device(type='cuda', index=0)
y = torch.tensor([1., 2.]).cuda()
# y.device is device(type='cuda', index=0)

with torch.cuda.device(1):
    # allocates a tensor on GPU 1
    a = torch.tensor([1., 2.], device=cuda)

    # transfers a tensor from CPU to GPU 1
    b = torch.tensor([1., 2.]).cuda()
    # a.device and b.device are device(type='cuda', index=1)

    # You can also use ``Tensor.to`` to transfer a tensor:
    b2 = torch.tensor([1., 2.]).to(device=cuda)
    # b.device and b2.device are device(type='cuda', index=1)

    c = a + b
    # c.device is device(type='cuda', index=1)

    z = x + y
    # z.device is device(type='cuda', index=0)

    # even within a context, you can specify the device
    # (or give a GPU index to the .cuda call)
    d = torch.randn(2, device=cuda2)
    e = torch.randn(2).to(cuda2)
    f = torch.randn(2).cuda(cuda2)
    # d.device, e.device, and f.device are all device(type='cuda', index=2)
</code></pre>