Call .backward() to clear graph

Yoo: <p>I was reading this article in which it says the graph will be cleaned in the step loss.backward():</p><aside class="quote quote-modified" data-post="20" data-topic="4903">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">Why do we need to set the gradients manually to zero in pytorch?</a> 
  </div>
  <blockquote>
    Here are three equivalent code, with different runtime/memory comsumption. 
Assume that you want to run sgd with a batch size of 100. 
(I didn’t run the code below there might be some typos, sorry in advance) 
1: single batch of 100 (least runtime, more memory) 
# some code
# Initialize dataset with batch size 100
for input, target in dataset:
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    opt.zero_grad()
    loss.backward()
    # graph is cleared here
  …
  </blockquote>
</aside>

<p>Is there any way that I could get access to the underlying graph to make sure it is freed?<br>
I tried to check loss.grad_fn and found it stayed the same after backward(), although the id changed</p>
<pre><code class="lang-auto">x = torch.tensor([0.5, 0.75], requires_grad=True)
y = torch.tensor([0.1, 0.90], requires_grad=True)
z = torch.exp(x * y).sum()
print(z.grad_fn)
z.backward() 
print(z.grad_fn)
</code></pre>

srishti-git1110: <aside class="quote no-group" data-username="Yoo" data-post="1" data-topic="164294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yoo/48/53396_2.png" class="avatar"> Yoo:</div>
<blockquote>
<p>the graph will be cleaned in the step loss.backward()</p>
</blockquote>
</aside>
<p>What this strictly means is the the references to the saved tensors are lost but the underlying graphs still hangs on in the memory. Are you asking something along the lines?<br>
This might be the reason why the grad_fn remains intact as you checked.</p>

Yoo: <aside class="quote no-group" data-username="srishti-git1110" data-post="2" data-topic="164294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<p>. Are you asking something along the lines?</p>
</blockquote>
</aside>
<p>Hi thank you. I was asking if I could check the references were indeed lost after I call backward(), like I detach a tensor and I could clearly see the difference</p>
<p>And how this step will save the memory if the graph still hangs there?</p>

srishti-git1110: <aside class="quote no-group" data-username="Yoo" data-post="3" data-topic="164294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yoo/48/53396_2.png" class="avatar"> Yoo:</div>
<blockquote>
<p>I was asking if I could check the references were indeed lost after I call backward()</p>
</blockquote>
</aside>
<p>Yes. The references to the saved tensors are definitely lost after a backward call unless you specify <code>retain_graph=True</code> as an argument to the backward method which you shouldn’t unless necessary.</p>
<p>To see this, try a second backward call right after the first, and the error message shall convey the same.</p>
<aside class="quote no-group" data-username="Yoo" data-post="3" data-topic="164294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yoo/48/53396_2.png" class="avatar"> Yoo:</div>
<blockquote>
<p>how this step will save the memory</p>
</blockquote>
</aside>
<p>As the references to the saved tensors are lost, memory is aggressively freed.</p>

Yoo: <p>What does “references” explicitly mean? Does it mean saved intermediate values of the graph?</p>

srishti-git1110: <p>How things work internally (what data structures are used etc.) is something I’m not completely sure about.<br>
But yes, “references” to the tensors that’ll be required for gradient computation on a backward call are saved as the graph builds in the forward pass.</p>
<p>You might also like to read more on how PyTorch uses dynamic computational graphs.</p>