Memory leak when appending tensors to a list

username: Siamak_Shakeri
cooked: <p>Here is the snippet:</p>
<pre><code>    for ii,line in enumerate(lines):

        if line['evidence'][-1][-1][-1]  != None:
            #feats, labels = tf_idf_claim(line)
            feats, ev_sent, _ = fasttext_claim(line,ft_model,db,params)
            labels = ind2indicator(ev_sent,feats.shape[-1])
            all_labels.append(labels.numpy())

            pred_labels, scores = model(feats.unsqueeze_(0).transpose_(1,2).cuda(device_id))
            all_scores[ii] = scores
</code></pre>
<p>I see that the gpu errors out OOM after a while. This is running on Tesla K80.<br>
If I change the last line to all_scores[ii] = scores.detach().cpu().numpy(), it gets fixed.<br>
This is only process running on the gpu core.</p>

username: ptrblck
cooked: <p>I’m not sure how <code>scores</code> was calculated, but it could still hold a reference to the computation graph.<br>
If that’s the case, you are storing the whole computation graph in your list in each iteration, which eventually fills up your GPU memory.<br>
Detaching the tensor should fix this issue as you’ve already mentioned.</p>

username: Siamak_Shakeri
cooked: <p>scores come from softmax outputs. The scores are Tensors of shape [1,10,100]. Does it mean any tensor calculated by the model holds a reference to the whole graph?<br>
I am new to pytorch, so my question might be dumb. Thanks for your help.</p>

username: ptrblck
cooked: <p>In this case the softmax output needs the computation graph to be able to calculate the gradients in the backward pass.<br>
You can check it by printing the <code>grad_fn</code>:</p>
<pre><code class="lang-python">print(scores.grad_fn)
# Should return something like
&lt;SoftmaxBackward at 0x7f371721a668&gt;
</code></pre>
<p>If you store something from your model (for debugging purpose) and don’t need to calculate gradients with it anymore, I would recommend to call <code>detach</code> on it as it won’t have any effects if the tensor is already detached.</p>

username: Siamak_Shakeri
cooked: <p>That makes sense to me now. Thanks for the detailed answer.</p>

username: Yanru_Huang
cooked: <p>Hi, I have an question on this sentence:</p>
<blockquote>
<p>you are storing the whole computation graph</p>
</blockquote>
<p>Interpreting this sentence seems not so intuitive,  simply by looking at code, the list<br>
only stores the variable <code>scores</code>,  ? Does it turns out the information(grad_fn and some other stuff need for autograd) of every tensor that is in the computation would be store in the <code>all_scores[ii]</code> ?<br>
Thanks !</p>

username: ptrblck
cooked: <p>Yes, that’s correct. <code>scores</code> is attached to a computation graph via its <code>grad_fn</code> and if you store the <code>scores</code> tensor in e.g. a <code>list</code> the entire graph with all intermediate tensors needed to compute the gradients in this computation graph will be stored with it.</p>

username: Yanru_Huang
cooked: <p>So does it mean if I have gpu device that holds 12 GB and my model takes 6.1G of gpu memory during training, If I stores the output of model to a list, in this case <code>score</code> , Do I get OOM immediately since the total memory consumption is 6.1 * 2 = 12.2 GB ?</p>

username: ptrblck
cooked: <p>No, since the memory is not only used by the last computation graph, but also the model parameters and potentially other data on the GPU.</p>

username: Yanru_Huang
cooked: <p>Ok, Is there any way to get or calculate the memory usage of computation graph ?<br>
By the way does torch destroy all computation(free memory taken by graph) when <code>backward()</code> is called ,and rebuild the graph when next <code>forward()</code> is called ?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Yanru_Huang" data-post="12" data-topic="25937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yanru_huang/48/54404_2.png" class="avatar"> Yanru_Huang:</div>
<blockquote>
<p>Ok, Is there any way to get or calculate the memory usage of computation graph ?</p>
</blockquote>
</aside>
<p>An easy way would be to check the allocated memory while the output of the forward pass (and thus the computation graph) is alive, delete the output, and re-check the allocated memory. The delta would then correspond to the the memory usage of the computation graph.</p>
<aside class="quote no-group" data-username="Yanru_Huang" data-post="12" data-topic="25937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yanru_huang/48/54404_2.png" class="avatar"> Yanru_Huang:</div>
<blockquote>
<p>By the way does torch destroy all computation(free memory taken by graph) when <code>backward()</code> is called ,and rebuild the graph when next <code>forward()</code> is called ?</p>
</blockquote>
</aside>
<p>Yes, that’s the case in the default setup. PyTorch will not be able to free the computation graph and the intermediate tensors after the <code>backward</code> call if you are storing references to the computation graph as already described. Also, using <code>retain_graph=True</code> in <code>backward</code> will not free the graph since you are explicitly keeping it alive.</p>

username: Yanru_Huang
cooked: <blockquote>
<p>An easy way would be to check the allocated memory while the output of the forward pass (and thus the computation graph) is alive, delete the output, and re-check the allocated memory. The delta would then correspond to the the memory usage of the computation graph.</p>
</blockquote>
<p>This is a great idea ! I would try it if I need it in the future.</p>
<blockquote>
<p>PyTorch will not be able to free the computation graph and the intermediate tensors after the <code>backward</code> call if you are storing references to the computation graph as already described<br>
Let’s say if I have a graph looks like this :</p>
</blockquote>
<pre><code class="lang-auto">while 1 : 
    c = a  + b
    e = c + d 
    g = e + f
    g.backward()
</code></pre>
<p>Assume that <code>a, b, c, d, e, f, g</code> all requires grad. So in the normal situation, Inside while loop<br>
the <code>backward()</code> is called and every thing in the graph is destroyed as describe above. But now if I some how make <code>c</code> by setting <code>retain_graph=True</code>, I ave two question:</p>
<ol>
<li>During the backward is <code>c</code> is an only tensor would be kept in the graph, or there is any other tensor would stay alive in graph due to c is not erased ?</li>
<li>To my understanding, if the loop goes on, since every time  the graph is not cleaned complete , OOM would occur at some point ?</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>None of the mentioned tensors will be deleted in the <code>backward</code> call, since you have explicitly created references to them by creating the variables. The <code>backward</code> call will only free tensors in the computation graph if no object holds a reference to them anymore.</p>
</li>
<li>
<p>Also no, since you are overriding all tensors explicitly and there are no intermediates which wouldn’t be freed, but you can also run your example and check the actual memory usage to verify it.</p>
</li>
</ol>

username: Yanru_Huang
cooked: <p>I don’t completely understand this sentence  “holds a reference to”, To my knowledge, if I write something like this :</p>
<pre><code class="lang-auto">a = 0
a = 1
</code></pre>
<p>In this case,Can I say <code>a</code> does not hold a reference to <code>object 0</code> and now holding the reference to <code>object 1</code>, to be easier to understand,can I say a  now points to <code>object 1</code> ? Correct me if I am wrong.</p>
<p>Another case confuses me as well, by looking at the comment from <a href="https://github.com/Megvii-BaseDetection/YOLOX/issues/103#issuecomment-888050727" rel="noopener nofollow ugc">Memory leak issue from yolox </a> , they claimed that they solved memory leak issue by using <code>detach()</code></p>
<pre><code class="lang-auto">def update(self, values=None, **kwargs):
       if values is None:
           values = {}
       values.update(kwargs)
       for k, v in values.items():
           if isinstance(v, torch.Tensor):
               v = v.detach()
           self[k].update(v)
</code></pre>
<p>In this case why <code>detach()</code> should be used, if the keys in dictionary are fixed, even I don’t use <code>detach()</code>, every iteration or some period time I store the value into <code>self[k]</code> from the tensor that is still in the computation graph, I think this would increase the memory usage for sure, however doesn’t  the "amount of extra memory usage " should be zero ?</p>
<pre><code class="lang-auto">self["loss"] = loss_t1
self["loss"] = loss_t2 #the loss_t1 is destroyed
....

</code></pre>
<p>In this case, why would memory  usage is increasing  ? Thanks</p>

username: Omroth
cooked: <pre><code class="lang-auto">class A:
   def __init__(self, my_pony):
      self.my_pony = my_pony

b = something large
a = A(b)
b = None
</code></pre>
<p>“something large” won’t be garbage collected as a “holds a reference to it”</p>

username: Yanru_Huang
cooked: <p>Sorry for interrupting you, I would like to know if it’s possible for you to reply <a href="https://discuss.pytorch.org/t/memory-leak-when-appending-tensors-to-a-list/25937/17">the question that I post above</a> to help me clear my confusion. Thanks !</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/omroth">@Omroth</a> already explained the reference holding via a small example, but maybe this code would also help you as you could play around with it to understand a bit more about the memory allocation:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

print(torch.cuda.memory_allocated()/1024**2)
# 0.0

device = 'cuda'
x = torch.randn(1024, 1024, device=device)
# expected: 4MB = 1024 * 1024 * 4 / 1024**2
print(torch.cuda.memory_allocated()/1024**2)
# 4.0

model = nn.Sequential(
    nn.Linear(1024, 1024, bias=False),
    nn.Linear(1024, 1024, bias=False),
    nn.Linear(1024, 1024, bias=False),
    nn.Linear(1024, 1024, bias=False)
).to(device)
# expected: 20MB = 4MB (previous) + 16MB via 1024 * 1024 * 4 * 4 / 1024**2
print(torch.cuda.memory_allocated()/1024**2)
# 20.0

for _ in range(3):
    out = model(x)
    print(torch.cuda.memory_allocated()/1024**2)

# expected: 36MB = 20MB (previous) + 16MB for intermediates
# 36.0
# 36.0
# 36.0

del out
# expected: 20MB (as previously)
print(torch.cuda.memory_allocated()/1024**2)
# 20.0

for _ in range(3):
    out = model(x)
    print(torch.cuda.memory_allocated()/1024**2)
    out.mean().backward()
    print(torch.cuda.memory_allocated()/1024**2)

# 36.0 # 1st forward: 20MB (base) + 16 (intermediates)
# 40.0 # 1st backward: intermediates deleted, but 20MB (base) + 16MB for gradients + 4MB for `out`
# 52.0 # 2nd forward: 20MB (base) + 16 (grad) + 16MB (intermediates)
# 40.0 # 2nd backward: same as 1st backward
# 52.0 # 3rd forward: same as 2nd forward
# 40.0 # 3rd backward: same as 1st backward

del out
print(torch.cuda.memory_allocated()/1024**2)
# 36.0 # 20mb (base) + 16MB (grad)

model.zero_grad(set_to_none=True)
print(torch.cuda.memory_allocated()/1024**2)
# 20.0 # 20MB (base)

l = 0
for _ in range(3):
    out = model(x)
    l += out.mean()
    print(torch.cuda.memory_allocated()/1024**2)

# 36.00048828125
# 48.00048828125
# 60.00048828125

l = l.backward()
print(torch.cuda.memory_allocated()/1024**2)
# 40.00048828125

model.zero_grad(set_to_none=True)
print(torch.cuda.memory_allocated()/1024**2)
# 24.00048828125

del out
print(torch.cuda.memory_allocated()/1024**2)
# 20.00048828125
</code></pre>
<p>Also note that all these issues are strictly speaking not memory leaks even though the increase in memory usage is often called a leak by users. As you can see in my example, the increase in memory usage is expected and deleting the right tensors will also free the memory and make it reusable, which means that your script it not leaking memory (this memory would be lost and you won’t be able to recover it).</p>

username: Yanru_Huang
cooked: <p>Wow, this is really awesome and informative. Thanks !<br>
I have an question on this part of code,</p>
<pre><code class="lang-auto">for _ in range(3):
    print("Before forwarding: {} MB.".format(torch.cuda.memory_allocated()/1024**2))
    out = model(x)
    print("After forwarding: {} MB".format(torch.cuda.memory_allocated()/1024**2))
    print("The id of out: {}".format(id(out)))
    del out
</code></pre>
<p>The output is :</p>
<pre><code class="lang-auto">Before forwarding: 20.0 MB.
After forwarding: 36.0 MB
The id of out: 140380622503616
Before forwarding: 20.0 MB.
After forwarding: 36.0 MB
The id of out: 140380622499328
Before forwarding: 20.0 MB.
After forwarding: 36.0 MB
The id of out: 140380622502848
</code></pre>
<p>In this case when <code>del out</code>  is performed, the memory is free as expected, but without <code>del out</code>, the memory allocation would stay 36.0 MB after first iteration, I would like to know why this happen ? Wouldn’t the <code>out</code> gets collected at end of iteration and there’s no more object hold the reference to computation graph and the memory taken by graph should be free? Thanks !</p>

username: ptrblck
cooked: <p>No, <code>del</code> would still be alive until it is overwritten with the new output of the model or when do you think it would be deleted?<br>
Simply because the loop ends doesn’t mean Python frees variables:</p>
<pre><code class="lang-python">for i in range(2):
    print("iter {}".format(i))
    if 'out' in locals() or 'out' in globals():
        print("out is alive in iter {}".format(i))
    out = 1
    
# iter 0
# iter 1
# out is alive in iter 1
</code></pre>
<p>Based on your assumption <code>"out is alive in iter 1"</code> should not have been printed since somehow <code>out</code> was automatically deleted?</p>