Autograd + garbage collection

srishti-git1110: <p>I am trying to understand how pytorch’s autograd engine frees up memory.<br>
Referred to <a href="https://discuss.pytorch.org/t/understanding-garbage-collection-of-computation-graphs-in-pytorch/1934">this</a> but I still have questions.</p>
<p>What does freeing of graph exactly mean?</p>
<p>According to my understanding from what I researched, freeing of backward graphs implies that the references to saved tensors are freed from the memory but the underlying graph remains.</p>
<p>The following code shows that even after a backward call on k, it had its backward graph remaining which was in turn passed to tensor m as well.</p>
<pre><code class="lang-auto">import torch
m = torch.tensor([0.0], requires_grad=True)
for i in range(1):
  x = torch.tensor([3.0], requires_grad=True)
  k = x*2
  k.backward()
  m = k

print(m.requires_grad)
</code></pre>
<p><code>True</code></p>
<pre><code class="lang-auto">print(m)
</code></pre>
<p><code>tensor([6.], grad_fn=&lt;MulBackward0&gt;)</code></p>
<br>
<p>This also forms one use of detach() - when we need to use a variable’s value outside of the training loop, we detach it before doing so; as the backward call only frees the references to saved tensors but the graph hangs around in the memory.<br>
While if we use the detached tensor outside the train loop, the backward graph of the original one gets garbage collected. See -</p>
<pre><code class="lang-auto">import torch
m = torch.tensor([0.0], requires_grad=True)
for i in range(1):
  x = torch.tensor([3.0], requires_grad=True)
  k = x*2
  k.backward()
  m = k.detach()

print(m.requires_grad)
print(m)
</code></pre>
<p><code>False</code><br>
<code>tensor([6.])</code><br>
<br><br>
Could you please elaborate why PyTorch chose to keep those backward graphs in memory when the saved tensors are anyway gone?<br>
What’s the utility of keeping those while also considering <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#computational-graph" rel="noopener nofollow ugc">the fact</a> that graphs are created from scratch every time after a backward call?</p>

evrimozmermer: <p>Firstly, PyTorch works with dynamic graphs. Everytime you put a tensor to a operation, if it is not included in a graph, a new graph is created so that you can do all the operations that requires a graph. Computation graphs are not only used for backprop but also for inference. If you create the graph of a model, you can export that graph and use it in inference, which will be more efficient than loading the model with its class and checkpoint and making inference. By using the graph, you can make load your model for inference without model file and checkpoint.<br>
Most probably there are other reasons to keep the graph, but this is what comes to my mind now.<br>
You can imagine computation graph as the compiled version of a code. You can either reload the code and compile and run it, or you can directly run the compiled version of the code.</p>

srishti-git1110: <p>That’s a great answer. I did not know we could export model graphs, too.<br>
I have in past exported model checkpoints in .pt files and those always (for reasons unknown to me) give errors while loading in another notebook/colab.<br>
Thanks for the answer.</p>

srishti-git1110: <p>FOR ANYONE READING THIS THREAD:<br>
Please note (in the second block of code) the backward graph of the original tensor <code>k</code> getting garbage collected implies that the detached tensor <code>m</code> no longer points to the computation graph and since the scope of <code>k</code> is limited to the for loop, the graph <code>k</code> points to gets garbage collected when GC hits automatically.</p>

elliotwaite: <p>Hi, I’m also curious why the backward graph is not automatically garbage collected when <code>retain_graph=False</code>. This is also confusing because <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html#torch-tensor-backward" rel="noopener nofollow ugc">the documentation</a> for the <code>retain_graph</code> parameter seems to suggest otherwise:</p>
<blockquote>
<p><strong>retain_graph</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a><em>,</em> <em>optional</em> ) – If <code>False</code> , the graph used to compute the grads will be freed.</p>
</blockquote>
<p>Is that documentation correct?</p>
<p><a class="mention" href="/u/evrimozmermer">@evrimozmermer</a>, you mentioned that there are other uses for graphs other than backprop, such as exporting them for later use. I have heard of exporting forward graphs for later use, such as for inference, but I have not thought about the potential of also exporting backward graphs. Is it possible to export a backward graph for later reuse, even if the saved tensors that were referenced by that backward graph have already been freed?</p>

evrimozmermer: <p>One example for use of backward graph can be the gradient update of the teacher model in bring your own latent (byol). You can check the paper in arxiv. Teacher model is basically being updated by the moving average of the gradients of the student model.</p>

l11: <p>i am curious about how the ref count work in the tensor of the dynamic graph in Pytorch, which paper talk about that?</p>

soulitzer: <p>Not sure if there’s a paper on these types of things, but you may be interested in an epsidode on the PyTorch podcast covering the technical details of a particular part of this, <a href="https://open.spotify.com/episode/637yesGKqPE9XGeCmNRjnb?si=23607bb5d1664bfb" class="inline-onebox" rel="noopener nofollow ugc">Spotify</a></p>