Can we use .cpu() to save memory on GPU?

TsingZ0: <p>I try to save the memory on GPU by .cpu() using the code like the following, but failed.</p>
<pre><code class="lang-auto">import ...

model = MyModel(...)
loss = Myloss(...)

def train():
   model.to('cuda')
   loss.to('cuda')

   model.train()

   # optimize parameters

   model.cpu()
   loss.cpu()
</code></pre>
<p>The memory cost does not decrease. Why does <code>.cpu()</code> fail to move the parameters from the GPU<br>
memory to the memory of CPU?</p>

RaLo4: <blockquote>
<p><a href="https://discuss.pytorch.org/t/can-we-use-cpu-to-save-memory-on-gpu/101470">Can we use .cpu() to save memory on GPU?</a></p>
</blockquote>
<p>Since I don’t know what you are trying to do here I can’t really answer that question.<br>
As for this question:</p>
<aside class="quote no-group" data-username="TsingZ0" data-post="1" data-topic="101470">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tsingz0/48/30641_2.png" class="avatar"> TsingZ0:</div>
<blockquote>
<p>Why does <code>.cpu()</code> fail to move the parameters from the GPU<br>
memory to the memory of CPU?</p>
</blockquote>
</aside>
<p><code>.cpu()</code> is not inplace for a tensor, so assuming <code>loss</code> is a tensor you need to write it this way:</p>
<pre><code class="lang-auto">loss = loss.cpu()
</code></pre>
<p>Also! One reason why a lot of people are running out of vram is because they are trying to keep their total loss without detaching the graph or moving the tensor to cpu. Like so:</p>
<pre><code class="lang-auto">running_loss += loss     #this is wrong! don't use this!
</code></pre>
<p>If keeping your loss is what you are trying to do, but don’t want it to be on your gpu, then you can do this instead:</p>
<pre><code class="lang-auto">running_loss += loss.item()
</code></pre>
<p>This will extract the loss as a float. Completely detached and moved to cpu.</p>

5had3z: <p>You may also need to use torch.cuda.empty_cache()</p>

TsingZ0: <aside class="quote no-group" data-username="RaLo4" data-post="2" data-topic="101470">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/65b543/48.png" class="avatar"> RaLo4:</div>
<blockquote>
<p><code>running_loss += loss.item()</code></p>
</blockquote>
</aside>
<p>It works! Thank you so much!</p>

TsingZ0: <aside class="quote no-group" data-username="5had3z" data-post="3" data-topic="101470">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/5had3z/48/28066_2.png" class="avatar"> 5had3z:</div>
<blockquote>
<p>torch.cuda.empty_cache()</p>
</blockquote>
</aside>
<p>It works and <code>.cpu()</code> is really able to save memory. However, the maximum memory cost is still high. With RaLo4’s help, I know that using</p>
<aside class="quote no-group" data-username="RaLo4" data-post="2" data-topic="101470">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/65b543/48.png" class="avatar"> RaLo4:</div>
<blockquote>
<p><code>running_loss += loss.item()</code></p>
</blockquote>
</aside>
<p>saves a lot of memory.</p>