How to free the graph after create_graph=True

kolorado: <p>Hi,</p>
<p>I use autograd.grad function with create_graph=True. After I finish, I want to release the GPU memory the created backward graph. How can I do it?</p>

dejanbatanjac: <p>Easy, just check out the official <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.backward" rel="nofollow noopener">docs</a>:</p>
<ul>
<li>
<strong>retain_graph</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="nofollow noopener"> <em>bool</em> </a> <em>,</em> <em>optional</em> ) – If <code>False</code> , the graph used to compute the grad will be freed. Note that in nearly all cases setting this option to <code>True</code> is not needed and often can be worked around in a much more efficient way. Defaults to the value of <code>create_graph</code> .</li>
<li>
<strong>create_graph</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="nofollow noopener"> <em>bool</em> </a> <em>,</em> <em>optional</em> ) – If <code>True</code> , graph of the derivative will be constructed, allowing to compute higher order derivative products. Defaults to <code>False</code> .</li>
</ul>
<p>So if you set <code>retain_graph=False</code> this looks promising.</p>

kolorado: <p>Do you mean to cal autograd.grad function again this time with retain_graph=False??? This seems nasty!</p>

dejanbatanjac: <p>If you check the docs, when you do <code>create_graph=True</code> this means <code>retain_graph</code> will take this True, since it defaults to the value of <code>create_graph</code>.</p>
<p>If you do <code>create_graph=True</code>, you should set also <code>retain_graph=False</code> to release the memory.</p>

albanD: <p>Hi,</p>
<p>I don’t think this is what he wants.<br>
<code>retain_graph</code> can be used if you want to call <code>backward()</code> multiple times on the same graph.<br>
<code>create_graph</code> is used if you want to <code>backward()</code> through the <code>backward()</code>. But since this second <code>backward()</code> will go through the current graph, then <code>retain_graph</code> has to be set as well. If you set it to <code>False</code>, you will get an error.</p>
<p><a class="mention" href="/u/kolorado">@kolorado</a> the graph is deleted as soon as all references to it are gone. This would be the Tensor that contains the loss for example.<br>
Note that pytorch uses a custom gpu memory allocator. So when objects are freed, they are not returned to the OS directly. And so you won’t see in <code>nvidia-smi</code> that objects have been freed. This memory is available to be reused by pytorch though. So the next forward will actually not allocate more memory on your GPU. You can find the functions <a href="https://pytorch.org/docs/stable/cuda.html#memory-management" rel="nofollow noopener">here in the doc</a> to be able to see exactly how much memory is used and how much is cached to be re-used later.</p>

dejanbatanjac: <aside class="quote no-group" data-username="albanD" data-post="5" data-topic="58476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>I don’t think this is what he wants.</p>
</blockquote>
</aside>
<p>True but if in some case he doesn’t plan to <code>backward()</code> multiple times on the same graph, which I think is most likely the case it would be great to do:</p>
<pre><code class="lang-auto">backward(create_graph=True, retain_graph=False, ...)
</code></pre>
<p>to free the graph.</p>
<p>As per this <a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795/2">post</a>.</p>

albanD: <p>Hi,</p>
<p>But if you create the graph, you’re planning on calling backward on it no?<br>
Basically the existing graph will be a subset of the new graph created by <code>create_graph=True</code>. So if you ever want to use this newly created graph, you must pass <code>retain_graph=True</code> when you create it.</p>
<p>Or am I misunderstanding your point?</p>