Why autograd graph is not freed?

akt_09: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3dfeeb27ec811f8533e89c478fcb53188a656daa.png" data-download-href="https://discuss.pytorch.org/uploads/default/3dfeeb27ec811f8533e89c478fcb53188a656daa" title="torch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3dfeeb27ec811f8533e89c478fcb53188a656daa.png" alt="torch" data-base62-sha1="8QrhKh2H1wxQbhhKpIG8o1JLeC6" width="441" height="500" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch</span><span class="informations">458×519 14.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why the autograd graph is not freed after w.backward()?<br>
and torch.autograd.grad(w,x)?<br>
It should have been freed, no retrain_graph=True.</p>

ptrblck: <p>The original <code>w</code> tensor is replaced by the <code>x+i</code> operation, which doesn’t need to store any intermediate activations and thus allows you to call <code>w.backward()</code> multiple times.</p>

akt_09: <p>Thank you for your reply. But does that mean no looping network is created if replacing can be done?</p>

ptrblck: <p>There won’t be a “looping network”, since you are not creating a sequence of operations, but are rerunning <code>x + i</code>.<br>
A graph would be created e.g. if you would assign the result back to <code>x</code>.</p>

akt_09: <pre><code class="lang-auto">x=torch.tensor(10.).requires_grad_()

w=torch.tensor([0.]).requires_grad_()
z=torch.empty(1)
for i in range(5):
    #w.require_grad=True
    w=x*2+i
    

w.backward() #works
w.backward() # grads freed, does not work

</code></pre>
<p>why this does not work?</p>
<pre><code class="lang-auto">x=torch.tensor(10.).requires_grad_()

w=torch.tensor([0.]).requires_grad_()
z=torch.empty(1)
for i in range(5):
    #w.require_grad=True
    w=(x-1)-i

w.backward() #works
w.backward() # works
</code></pre>
<p>Why the above works?</p>
<pre><code class="lang-auto">x=torch.tensor(10.).requires_grad_()

w=torch.tensor([0.]).requires_grad_()
z=torch.empty(1)
for i in range(5):
    #w.require_grad=True
    w=(x/1)-i

w.backward() #works
w.backward() # grads freed, does not work
</code></pre>
<p>Why above one does not work?</p>
<p>Does pytorch treat +/- differently than * or / because of how derivatives are calculated for them? As for +/- upward gradient is just distributed, but  for  * and / there are actually different ways to calculate derivative?</p>
<p>So, pytorch create graphs to determine gradients, simple as that. If network only distributes gradient, then there are no intermediate operations to be freed?</p>

akt_09: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48cd1952cabc2f6fc26c16f64cc2b3ee01c9f9b9.png" alt="pytorch ques2" data-base62-sha1="ao1NChldYsoMMMy68PqrU4xsq8N" width="544" height="252"></p>
<p>w.backward() is working but if <code>x=x*1 +1</code> then <code>w.backward()</code> can not be called after calling it earlier. Why?</p>
<p>I think intermediate activations, when created and when removed is the reason.</p>

tom: <aside class="quote no-group" data-username="akt_09" data-post="5" data-topic="112447">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akt_09/48/33323_2.png" class="avatar"> akt_09:</div>
<blockquote>
<p>Does pytorch treat +/- differently than * or / because of how derivatives are calculated for them? As for +/- upward gradient is just distributed, but for * and / there are actually different ways to calculate derivative?</p>
</blockquote>
</aside>
<p>Yes, the key difference is that +/- doesn’t need any saved tensors (just shapes for broadcasting) while * and / do need the inputs.<br>
This is also a reason why you can inplace-modify sometimes and sometimes hit the “a value needed for backward has been inplace modified” (or whatever, from the top of my head) error.</p>
<p>Best regards</p>
<p>Thomas</p>