Is the .cuda() operation performed in place?

nicolas-gervais: <p>Is <code>model.cuda()</code> or <code>model.to(device)</code> performed in place? Or do I need to reassign the model as such?</p>
<pre><code class="lang-auto">if torch.cuda.is_available():
    model = model.cuda()
</code></pre>
<p>or is this ok?</p>
<pre><code class="lang-auto">if torch.cuda.is_available():
    model.cuda()
</code></pre>
<p>Thank you in advance.</p>

Nikronic: <p>Hi,</p>
<p>No, <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.cuda" rel="nofollow noopener">.cuda</a> or its reverse <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.cpu" rel="nofollow noopener">.cpu</a> operations are not inplace. They create a copy on device side if it does not exist.</p>
<p>Bests</p>

ptrblck: <p>While <a class="mention" href="/u/nikronic">@Nikronic</a>’s post is true for tensors (and parameters), the <code>to()</code>, <code>cuda()</code>, and <code>cpu()</code> calls will be executed recursively on all parameters and buffers inside an <code>nn.Module</code>, so you don’t have to re-assign your <code>model</code>:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
print(model.weight.device)
&gt; cpu
model.cuda()
print(model.weight.device)
&gt; cuda:0

x = torch.randn(1)
print(x.device)
&gt; cpu
x.cuda() # ERROR!!!
print(x.device)
&gt; cpu
</code></pre>

nicolas-gervais: <p>hi, thank you very much for answering.</p>
<p>the reason i was asking this is that I answered a torch question on stack overflow a few months ago and it gained a lot of traction. however, i don’t code much in torch anymore, so i wanted to make sure everything i said was right. turns out, a few thousands people saw my post where i re-assigned the neural net as such:</p>
<pre><code class="lang-auto">if torch.cuda.is_available():
    model = model.cuda()
</code></pre>
<p>which is incorrect (or suboptimal) if i understand you correctly. either way, this is the <a href="https://stackoverflow.com/a/59013131/10908375" rel="nofollow noopener">post</a> i’m referring to.</p>

ptrblck: <p>I wouldn’t say the reassignment is wrong or bad in any sense and would also make sure you are not forgetting to reassign a tensor using the <code>to()</code> call. Also, while this might add a slight overhead of reassigning an object, the “cost” should be noise compared to the actual data transfer.</p>

gzguevara: <p>How is memory allocation handled in this example? Model.to(‘cuda’) would copy all parameters and buffers to the gpu but the same parameters and buffers would keep residing in the cpu too?</p>
<p>And regarding the first comment. “.cuda() &amp; .cpu() create a copy on the device side if the model does not exist on that device”.<br>
What are the conditions for that? Does the model on the device side has to point to a variable with the same name as the model on the host side?</p>

ptrblck: <aside class="quote no-group" data-username="gzguevara" data-post="6" data-topic="84961">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gzguevara/48/66083_2.png" class="avatar"> gzguevara:</div>
<blockquote>
<p>Model.to(‘cuda’) would copy all parameters and buffers to the gpu</p>
</blockquote>
</aside>
<p>Yes</p>
<aside class="quote no-group" data-username="gzguevara" data-post="6" data-topic="84961">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gzguevara/48/66083_2.png" class="avatar"> gzguevara:</div>
<blockquote>
<p>but the same parameters and buffers would keep residing in the cpu too?</p>
</blockquote>
</aside>
<p>Not if you are overriding the same variable. I.e. <code>model = model.to(device)</code> will free the resources on the previously used device. <code>model_new = model.to(device)</code> will keep it.</p>
<aside class="quote no-group" data-username="gzguevara" data-post="6" data-topic="84961">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gzguevara/48/66083_2.png" class="avatar"> gzguevara:</div>
<blockquote>
<p>What are the conditions for that?</p>
</blockquote>
</aside>
<p>Unsure, but are you asking what the conditions for not existing are?</p>