Why is this compuation graph failling?

samuel_beaussant: <p>I am currently playing with the autograd engine to understand its implementation better. I have made up a very simple computation graph as shown in the code below :</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F



A = torch.arange(16, requires_grad = True, dtype=torch.float32)
B = torch.arange(32, requires_grad = True, dtype=torch.float32)
C = torch.arange(4, requires_grad = True, dtype=torch.float32)
D = torch.arange(16, requires_grad = True, dtype=torch.float32)
 
A_ = A.reshape(4,4)+1
B_ = B.reshape(2,4,4)
C_ = C.reshape(4,1)
D_ = D.reshape(4,4)

E = A_ + B_
F = E@C_
G = F * C_
T = G.swapdims(1,2)
R = T.sum()
print(R)

H = B_ + D_
I = H / A_
R_ = I.mean()
Z = R_ * G
Res = Z.sum()
print(Res)

G.retain_grad()
E.retain_grad()
F.retain_grad()
A_.retain_grad()
B_.retain_grad()
C_.retain_grad()

R.backward()
#H.retain_grad()
Res.backward()

print("== leaf tensor ==")
print(A.grad)
print(B.grad)
print(C.grad)
print(D.grad)
</code></pre>
<p>This code throws an error “RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.”</p>
<p>I believe this is due to the tensor G being reused in the second computation graph.<br>
My first question is why ? Does autograd clear the entire computation graph or only the nodes used during backprop ? Since G is part of another graph that was not used during the first backward call, I expected the G tensor and all tensors used to compute it, to not be freed after the first backward call.</p>

ptrblck: <aside class="quote no-group" data-username="samuel_beaussant" data-post="1" data-topic="197726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_beaussant/48/62803_2.png" class="avatar"> samuel_beaussant:</div>
<blockquote>
<p>Since G is part of another graph that was not used during the first backward call, I expected the G tensor and all tensors used to compute it, to not be freed after the first backward call.</p>
</blockquote>
</aside>
<p>Based on your screenshot it seems <code>G</code> was indeed used in the fist <code>backward</code> via:</p>
<pre data-code-wrap="python"><code class="lang-python">T = G.swapdims(1, 2)
R = T.sum()
...
R.backward()
</code></pre>
<p>Afterwards you are calling <code>Res.backward()</code> which was created from <code>G</code> again via:</p>
<pre data-code-wrap="python"><code class="lang-python">Z = R_ * G
Res = Z.sum()
</code></pre>
<p>PS: It’s always better to post code snippets by wrapping them into three backticks ```, as it would allow us to copy/paste the code to reproduce the issue and the search engine could index it.</p>

samuel_beaussant: <p>Thank you, I edited my question appropriately. So autograd, by default, will free Tensor after backprop even if they are still required by another part of the computation graph that was not backprop yet ?</p>
<p>Also I have a follow up question : replacing G by C_ in the second computation as follow</p>
<pre><code class="lang-auto">Z = R_ * C_
</code></pre>
<p>will not raise a RunTimeError. C_ is not a leaf node (result of a reshape op) so by default it should freed after the first backward call doesn’t it ? Why is this still working ?</p>

ptrblck: <aside class="quote no-group" data-username="samuel_beaussant" data-post="3" data-topic="197726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_beaussant/48/62803_2.png" class="avatar"> samuel_beaussant:</div>
<blockquote>
<p>So autograd, by default, will free Tensor after backprop even if they are still required by another part of the computation graph that was not backprop yet ?</p>
</blockquote>
</aside>
<p>Yes, since the default eager mode isn’t aware of future calls and dependencies. You would thus need to use <code>retain_graph=True</code> in the first <code>backward</code> call to allow the second one to reuse the same intermediate activations and to free them afterwards.</p>
<aside class="quote no-group" data-username="samuel_beaussant" data-post="3" data-topic="197726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_beaussant/48/62803_2.png" class="avatar"> samuel_beaussant:</div>
<blockquote>
<p>C_ is not a leaf node (result of a reshape op) so by default it should freed after the first backward call doesn’t it ?</p>
</blockquote>
</aside>
<p>No, it won’t be freed since you’ve explicitly created a reference to it. You can still access and <code>print</code> <code>C_</code> after the <code>backward</code> calls.</p>

samuel_beaussant: <p>Thank you for the calrifications. Can you elaborate a bit more on how I’ve explicitely created a reference to C_ ? I don’t get the difference between how G is created and how C_ is created.</p>

ptrblck: <aside class="quote no-group" data-username="samuel_beaussant" data-post="5" data-topic="197726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_beaussant/48/62803_2.png" class="avatar"> samuel_beaussant:</div>
<blockquote>
<p>Can you elaborate a bit more on how I’ve explicitely created a reference to C_ ?</p>
</blockquote>
</aside>
<p>By creating the assignment:</p>
<pre data-code-wrap="python"><code class="lang-python">C_ = C.reshape(4,1)
</code></pre>
<aside class="quote no-group" data-username="samuel_beaussant" data-post="5" data-topic="197726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_beaussant/48/62803_2.png" class="avatar"> samuel_beaussant:</div>
<blockquote>
<p>I don’t get the difference between how G is created and how C_ is created.</p>
</blockquote>
</aside>
<p>The previous issue is not caused by <code>G</code> being deleted, but <em>its computation graph</em> including the intermediate activations needed for the gradient computation.</p>

samuel_beaussant: <p>Ok i get it, thank you for clarifying the matter !</p>