Why Tensor object is not deleted immediately?

username: eee4447b4b8a2cbdfbbd
cooked: <p>I have followed some codes about CUDA caching allocator and got a question.</p>
<p>When I execute the code below, the storage object created at i = 0 is not deleted immediately when first for loop iteration ends, but deleted when second one ends.<br>
So, the input tensor at i = 1 can’t reuse the input tensor at i = 0.</p>
<p>What makes this happen?</p>
<pre><code class="lang-auto">import torch                                                                  
from torch import nn                                                          
                                                                              
torch.manual_seed(0)                                                          
                                                                              
device, dtype = torch.device('cuda'), torch.float32                           
                                                                              
m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2)).to(device, dtype)
                                                                              
for i in range(0, 20):                                                        
  print("iter %d start" % (i))                                                
  input = torch.randn(16, 16, 128, 128).to(device)                            
  output = m(input)                                                           
  print("iter %d complete" % (i))                                             
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>python makes this happen.<br>
Because when you do <code>out = f(inp)</code>, it first gets the <code>inp</code>, then evaluates <code>f(inp)</code>, then delete the content of <code>out</code> then associate the result to <code>out</code>.<br>
As you can see, the deletion here happens after the new value has been computed.</p>
<p>A simple way around this is to move your loop content inside a function so that the whole body of the function goes out of scope when you exit the function and before you exit the current iteration of the loop.</p>

username: eee4447b4b8a2cbdfbbd
cooked: <p><a class="mention" href="/u/alband">@albanD</a><br>
Oh Thanks! That makes sense.<br>
But according to my log, input tensor at i = 0 is deallocated after the convolution at i =1.<br>
As you said, input tensor at i = 0 should be deallocated before the convolution.</p>
<p>What could have happened?</p>

username: albanD
cooked: <p>For the <code>input</code> in your code sample, on top of being references by the python variable called “input”, the Tensor is already referenced in the computational graph (because it is needed for the backward computation). And that computational graph is kept alive by <code>output</code> that references it.<br>
So the Tensor cannot be freed until <code>output</code> goes out of scope.</p>
<p>Note that if you call backward() on that graph (with retain_graph=False, the default), then it will free up the graph and that reference to input will go away. And so <code>input</code> should get deallocated, on the next iteration, as soon as the “input” python variable is overwritten.</p>

username: eee4447b4b8a2cbdfbbd
cooked: <p>Now everything’s clear to me.</p>
<p>Thanks for explanation!</p>