Link between require_grad and moving tensors between CPU and GPU memory

username: marcelwa
cooked: <p>Hi!</p>
<p>I am moving tensors between the CPU and GPU memory with <code>.to(device)</code> and <code>.cpu()</code>.<br>
I found out that all tensor that get in or out of the <code>nn.Linear</code> layer are locked in GPU memory.<br>
That is why I created my own <code>Linear</code> layer and I found out that if <code>require_grad=False</code> I get the expected  use of memory in the GPU.<br>
If <code>require_grad=True</code>, then I cannot swap the tensors to CPU memory.<br>
<code>.cpu()</code> or deleting the reference with <code>del tensor_var</code> have no impact if <code>require_grad=True</code>.</p>
<p>Can someone explain the background to this behaviour and maybe suggest a workaround to be able to use Pytorch’s autograd?</p>
<p>A huge thank you in advance for every answer!</p>

username: ptrblck
cooked: <p>Could you explain what “locked in GPU memory” means and how you are measuring it?<br>
Note that PyTorch caches the GPU memory, so after deleting a tensor, the memory will not be immediately released to avoid synchronizing (and thus expensive) memory reallocations.</p>

username: marcelwa
cooked: <p>To measure the GPU memory usage, I use</p>
<pre><code class="lang-auto">stats = torch.cuda.memory_stats()
current_active_byte =  stats["active_bytes.all.current"]
</code></pre>
<p>and calculate the difference before and after the statement of interest.<br>
The value of <code>current_active_byte</code> does not decrease with <code>tensor_var.cpu()</code> or <code>del tensor_var</code> if  <code>require_grad=True</code>.</p>

username: ptrblck
cooked: <p>I cannot reproduce this behavior using this code snippet:</p>
<pre><code class="lang-python">def print_active_bytes():
    stats = torch.cuda.memory_stats()
    current_active_byte =  stats["active_bytes.all.current"]
    print(current_active_byte)


# initial usage
print_active_bytes()
&gt; 0

# vanilla tensor
x = torch.randn(1024, device='cuda')
print_active_bytes()
&gt; 4096

del x
print_active_bytes()
&gt; 0

# requires_grad=True
x = torch.randn(1024, device='cuda', requires_grad=True)
print_active_bytes()
&gt; 4096

del x
print_active_bytes()
&gt; 0
</code></pre>

username: marcelwa
cooked: <p>I can reproduce it with the following code</p>
<pre><code class="lang-auto">import torch

def active_bytes():
    stats = torch.cuda.memory_stats()
    current_active_byte =  stats['active_bytes.all.current']
    return current_active_byte


# initial usage
print("Init usage {}". format(active_bytes()))

# vanilla tensor
x = torch.randn((256, 128), device='cuda')
w = torch.randn((128, 512), device='cuda')
l = torch.matmul(x, w)
print("Vanilla tensor {}". format(active_bytes()))

del x
print("Vanilla tensor: del x {}". format(active_bytes()))
del w
print("Vanilla tensor: del w {}". format(active_bytes()))
l = l.cpu()
print("Vanilla tensor: l = l.cpu() {}". format(active_bytes()))

# requires_grad=True
x = torch.randn((256, 128), device='cuda', requires_grad=True)
w = torch.randn((128, 512), device='cuda', requires_grad=True)
l = torch.matmul(x, w)
print("requires_grad=True {}". format(active_bytes()))

del x
print("requires_grad=True: del x {}". format(active_bytes()))
del w
print("requires_grad=True: del w {}". format(active_bytes()))
l = l.cpu()
print("requires_grad=True: l = l.cpu() {}". format(active_bytes()))
</code></pre>
<p>The output I get is</p>
<pre><code class="lang-auto">Init usage 0
Vanilla tensor 917504
Vanilla tensor: del x 786432
Vanilla tensor: del w 524288
Vanilla tensor: l = l.cpu() 0
requires_grad=True 917504
requires_grad=True: del x 917504
requires_grad=True: del w 917504
requires_grad=True: l = l.cpu() 393216
</code></pre>
<p>In my “production” code <code>.cpu()</code> does not free up memory as it does in this example, but I did not come up with a code snippet to show that behaviour yet.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="marcelwa" data-post="5" data-topic="97504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/48.png" class="avatar"> marcelwa:</div>
<blockquote>
<p>In my “production” code <code>.cpu()</code> does not free up memory as it does in this example</p>
</blockquote>
</aside>
<p>This is expected, as moving <code>l</code> to the CPU won’t detach it from the computation graph.</p>
<p>Add</p>
<pre data-code-wrap="python"><code class="lang-python">l = l.detach()
print("requires_grad=True: l = l.cpu() {}". format(active_bytes()))
</code></pre>
<p>after the <code>cpu()</code> operation, the graph should be freed, and you should get 0 active bytes again.</p>
<p>Note that <code>cuda()</code>, <code>cpu()</code> and <code>to()</code> operations are differentiable, so that the computation graph will be kept alive and you can backpropagate through these operations.</p>

username: marcelwa
cooked: <p>If I understood it correctly, when I would detach the tensor, then <code>backward()</code> does not work anymore and then there would no point of using Pytorch anymore.</p>
<p>Additionally, changing the second part to</p>
<pre><code class="lang-auto"># requires_grad=True
x = torch.randn((256, 128), device='cuda', requires_grad=True)
w = torch.randn((128, 512), device='cuda', requires_grad=True)
l = torch.matmul(x, w)
print("requires_grad=True {}". format(active_bytes()))

x = x.cpu()
print("requires_grad=True: x = x.cpu() {}". format(active_bytes()))
w = w.cpu()
print("requires_grad=True: w = w.cpu() {}". format(active_bytes()))
l = l.cpu()
print("requires_grad=True: l = l.cpu() {}". format(active_bytes()))
</code></pre>
<p>with the output</p>
<pre><code class="lang-auto">requires_grad=True 917504
requires_grad=True: x = x.cpu() 917504
requires_grad=True: w = w.cpu() 917504
requires_grad=True: l = l.cpu() 393216
</code></pre>
<p>shows that the inputs to the matrix multiplication are not freed.</p>
<p>The tensors should stay part of the computation graph so that backpropagation works.<br>
The question is why does Pytorch not free the memory in the GPU for tensors that are part of the computation graph?<br>
The tensors are not lost, they are just stored somewhere else.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="marcelwa" data-post="7" data-topic="97504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/48.png" class="avatar"> marcelwa:</div>
<blockquote>
<p>The tensors should stay part of the computation graph so that backpropagation works.<br>
The question is why does Pytorch not free the memory in the GPU for tensors that are part of the computation graph?</p>
</blockquote>
</aside>
<p>PyTorch doesn’t free these tensors because they are needed for the backward pass to work properly, since they are part of the computation graph.<br>
In your example you are creating new tensors <code>x</code>, <code>w</code>, and <code>l</code>, which doesn’t delete the computation graph. I.e. <code>l.mean().backward()</code> would still work.<br>
You won’t be able to access the <code>.grad</code> attribute from <code>x</code> and <code>w</code> anymore, but the gradients will still be calculated. If you change the CPU tensors to e.g. <code>x1</code> and <code>w1</code>, you would be able to print <code>x.grad</code> and <code>w.grad</code> after the backward pass.</p>
<p>You cannot move the computation graph to the CPU after its creation by moving leaf variables to the CPU (at least I’m not aware of a method to do so).</p>

username: marcelwa
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="97504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You cannot move the computation graph to the CPU after its creation by moving leaf variables to the CPU (at least I’m not aware of a method to do so).</p>
</blockquote>
</aside>
<p>Alright, that’s not what I have hoped for <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> but thank you a lot for your help!</p>