Which is freed, which is not?

username: AnzCol
cooked: <p>Here is a minimum nontrivial example that make me confused:</p>
<p>Example1 : passed</p>
<pre><code>import torch

x = torch.nn.Linear(3,3)
a = torch.autograd.Variable(torch.randn(2, 3))

for i in range(100):
    y = torch.sum(x(a))
    y.backward()
</code></pre>
<p>Example2 : failed. RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
<pre><code>import torch

x = torch.nn.Linear(3,3)
a = torch.autograd.Variable(torch.randn(2, 3))
y = torch.sum(x(a))

for i in range(100):
    y.backward()
</code></pre>
<p>What confused me is that in principle, the ‘graph’ will be freed after calling y.backward() in the first example and obviously x,a are contained in the graph. Then x, a will be freed too. So I did not expect the first example will pass since during the second y.backward() call we are not able to locate x and a. The second example is even more confusing to me. Just moving y=torch.sum() outside the loop will create an error.</p>
<p>So my question is: in both cases when the ‘graph’ is created? which autograd.Variables/nn.Module are contained in the graph? If the graph is freed after y.backward() is called, why the first example can pass? Why the second one can’t?</p>

username: albanD
cooked: <p>Here is your code with some comment on what is in the graph and what is not, let me know if it helps:</p>
<pre><code class="lang-auto">import torch

x = torch.nn.Linear(3,3)
# x is an nn.Module
# It contains two "leaf" Variables which mean a Variable that requires gradients

a = torch.autograd.Variable(torch.randn(2, 3))
# a is a Variable that does not require gradient, has no graph associated to
# it since no operation is done.

for i in range(100):
    x_out = x(a)
    # x_out is a `Variable` with the associated graphs of all computations
    # corresponding to the forward function of x
    # Note that all the objects of the graph (Functions and intermediary Variables)
    # are only accessible from the python object x_out 
    y = torch.sum(x_out)
    # y has a graph containing the sum operation and all the graph of x_out
    y.backward()
    # Go through the whole graph associated with y and compute the gradient for
    # all the leafs Variable
    # To reduce memory usage, all the intermediary Variables are freed.
</code></pre>
<pre><code class="lang-auto">import torch

x = torch.nn.Linear(3,3)
# nn.Module with two leafs Variables
a = torch.autograd.Variable(torch.randn(2, 3))
# a Variable that does not require grads
y = torch.sum(x(a))
# y contains a Variable with associated to it the graph
# corresponding to the forward function of x and 
# the sum Function

for i in range(100):
    y.backward()
    # First iteration:
    # We call backward and free the intermediary Variables of the graph
    # Second iteration:
    # You try to go through the graph associated with y but it has
    # already been cleared
    
</code></pre>

username: AnzCol
cooked: <p>Thanks that’s an excellent explanation! As you explained, in the second example when y.backward() was called the second time the variable ‘a’ was freed and the graph was not built again. That’s the reason why we can not let the gradient flow through. But I’m curious about, was ‘x’ also been freed? x is nn.Module so it is a collection of two leaf variables so in principle it is also an ‘intermediate variable’.</p>
<p>Besides, thanks to your comment I found another question that is not obvious to me. Consider the following code:</p>
<pre><code>import torch

x = torch.nn.Linear(3,3)
a = torch.autograd.Variable(torch.randn(2, 3))
x_out = x(a)

for i in range(100):
    y = torch.sum(x_out)
    y.backward()
</code></pre>
<p>This code will fail, which is unexpected. I think in the second iteration the graph is recreated because we call <code>y=torch.sum(x_out)</code> , which recreates all the Variables associated with y again. The problem disappears if I move <code>x_out</code> in the loop:</p>
<pre><code>import torch

x = torch.nn.Linear(3,3)
a = torch.autograd.Variable(torch.randn(2, 3))

for i in range(100):
    x_out = x(a)
    y = torch.sum(x_out)
    y.backward()
</code></pre>
<p>So my guess is, when you apply an operation like <code>torch.sum</code> it won’t recreate a graph. But <code>x_out = x(a)</code> will recreate a graph. Actually it is just a <code>forward()</code> call of the <code>nn.Module</code> object <code>x</code>. After all these experiences my crude conclusion is: the <code>forward()</code> call of <code>nn.Module</code> will recreate a graph associate with its return value. But this is insufficient…</p>

username: smth
cooked: <p>in your example where you call <code>y.backward()</code> in the for loop, when <code>y.backward()</code> is called, the gradient flows all the way to <code>x</code>. And then the input buffers in <code>x</code> are freed, When you call <code>y = torch.sum(x_out)</code> and then <code>y.backward()</code> the second time, the gradients again try to flow all the way to <code>x</code>, but because <code>x</code> needs it’s input to compute correct gradients (and since input was freed in the first <code>backward()</code> call), it will error out. You can declare to <code>y.backward</code> to not free the graph by saying <code>y.backward(retain_graph=True)</code></p>

username: jdhao
cooked: <p>I think the graph associated with the computation is something like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fa968f3ff341897929abc6951137cac2355ebe53.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/fa968f3ff341897929abc6951137cac2355ebe53" title="微信图片_20171108210335"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fa968f3ff341897929abc6951137cac2355ebe53_2_666x500.jpg" alt="微信图片_20171108210335" data-base62-sha1="zKNWhmSs1ZAaeODP2Tk3TEHgKhd" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fa968f3ff341897929abc6951137cac2355ebe53_2_666x500.jpg, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fa968f3ff341897929abc6951137cac2355ebe53_2_999x750.jpg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fa968f3ff341897929abc6951137cac2355ebe53_2_1332x1000.jpg 2x" data-dominant-color="C1BAB4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20171108210335</span><span class="informations">1440×1080 36.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The input to <code>x</code> is <code>a</code>. You say that</p>
<blockquote>
<p>then the input buffers in x are freed</p>
</blockquote>
<p>But if we inspect <code>a</code> after <code>y.backward()</code>, <code>a</code> still exists. Also upon looking at <a href="http://pytorch.org/docs/master/_modules/torch/nn/modules/linear.html#Linear" rel="noopener nofollow ugc">source of nn.Linear</a>, <code>input</code> is stored as an attribute of the module. What does it mean that “input buffers in x are freed”?</p>

username: albanD
cooked: <p>Any python object can be referenced from many places.<br>
When we say “freed” here, it means that the graph will not reference this object anymore. Of course if the user keeps a reference to the object, it won’t be destroyed as it can be used by someone else. But as soon as you remove your other reference to the object, it will be destroyed.</p>

username: jdhao
cooked: <p>In your first code snippet, the computational graph is created only once, so when you use for loops and try to backward on the graph for more than once without using <code>retain_graph=True</code>, the error will occur.</p>
<p>In you second code example, each time the for loop is executed, a fresh new graph is created and you can <code>backward()</code> through it once. If you try to backward in the loop for a second time, it will also fail:</p>
<pre><code class="lang-auto">import torch

x = torch.nn.Linear(3,3)
a = torch.autograd.Variable(torch.randn(2, 3))

for i in range(100):
    x_out = x(a)
    y = torch.sum(x_out)
    y.backward() # fine
    y.backward() # fail cause you are trying to backprop on the same net for a second time
</code></pre>

username: two_Two
cooked: <p>Hi,I have couple questions to ask , before showing my question I write an simple code first:</p>
<pre><code class="lang-auto">a = torch.tensor(2.0,requires_grad=True)
c = a*5 
c.backward()  #After back prop,the buffer would be deleted 
</code></pre>
<p>And if we do it once again,</p>
<pre><code class="lang-auto">c.backward()
</code></pre>
<p>Apparently, it would give :</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>So the questions are :</p>
<p>1.What does deleting(free) buffer or temporary variable mean ? Does it mean we can’t use those tensors or functions anymore?</p>
<p>2.Can you please show a simple example to show or check if the buffers and temporary variable are deleted after backward?</p>
<p>3.Based on :</p>
<blockquote>
<p>if the user keeps a reference to the object, it won’t be destroyed as it can be used by someone else</p>
</blockquote>
<p>How do we prevent the buffer or temporary variable being freed after backward?</p>
<p>I really appreciate all your help!</p>

username: albanD
cooked: <p>Hi,</p>
<p>First a bit more context.<br>
In python, the same object can be used at multiple places and be referenced.</p>
<pre><code class="lang-python">t = torch.rand(10) # Here we create a Tensor
l = [t] # Here the list contains the same Tensor
print(t) # t is accessible
del t # We delete t
print(l[0]) # The other reference to t is still valid and t was not freed.
</code></pre>
<p>Same thing the other way</p>
<pre><code class="lang-python">t = torch.rand(10) # Here we create a Tensor
l = [t] # Here the list contains the same Tensor
print(t) # t is accessible
del l # We delete the list that contains the reference to t
print(t) # t still exists
</code></pre>
<p>The buffers we save in the computational graph are like in the example above when I save a reference to <code>t</code> in a list.</p>
<ol>
<li>Freeing the buffers mean deleting the reference held by the computational graph to these buffers. You cannot access them <strong>from</strong> the computational graph. But other references are still valid.</li>
<li>We don’t have an api to check this. But the backward <strong>will</strong> delete all the saved buffers.</li>
<li>If you already hold onto another reference they won’t be freed. If you want to keep the reference stored in the graph, you need to pass the <code>keep_graph=True</code> argument to the backward function.</li>
</ol>
<p>Let me know if you want more details.</p>

username: two_Two
cooked: <p>Firstly,Thanks for the clear explanations!<br>
For summary(if wrong,please do correct my statement):<br>
If the object is referenced by more than just one ,after backward ,the only reference would be destroyed is computational graph one,the others that reference this object won’t be freed,If the object is only referenced by computational graph ,after backward,this reference would be freed and not be able to access it from  the computational graph.</p>
<p>And one more question is I would like to know what would be freed after back prop :</p>
<pre><code class="lang-auto">a = torch.tensor([2.0,3.0],requires_grad = True).view(2,1)
l = nn.Linear(1,3)
y  = l(a)
z = y*5 
o = torch.sum(z)
o.backward() 
</code></pre>
<p>During backward,the intermediate variables would be deleted,The word “intermediate” got me thinking that : Does it mean that the all the variables and functions related to operations “between” a and o would be freed or “include” a and o would be freed?</p>
<p>Thanks in advance!</p>

username: albanD
cooked: <p>Yes you got it right.</p>
<p>Now what is “intermediate” is a whole other can of worms <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Here again, for memory optimization, we only keep what is strictly necessary. And so every Function is responsible to save only what it needs.<br>
This means that there is not general rule of what is an intermediate value, it is dependent on the Function itself.</p>
<p>If you take the multiplication as an example, the definition of it’s derivative is given <a href="https://github.com/pytorch/pytorch/blob/829499e626362e6a1badeba4e28c0f7a7a120fde/tools/autograd/derivatives.yaml#L578-L580">here</a> as:</p>
<pre><code class="lang-auto">- name: mul(Tensor self, Tensor other) -&gt; Tensor
  self: grad * other
  other: grad * self
</code></pre>
<p>This means that the gradient for <code>self</code> is given by <code>grad * other</code> and the gradient for <code>other</code> is given by <code>grad * self</code>.<br>
As you can see, if the gradient for <code>self</code> is needed, only <code>other</code> is required for the backward and only this will be saved.</p>
<p>A hacky way (works in this example, might not work all the time) to see if something is needed for backward is to modify it inplace and see if the backward fails:</p>
<pre><code class="lang-python">import torch

self = torch.rand(10, requires_grad=True).clone() # This one requires gradients
# We do a clone to be allowed to do inplace ops on it later
other = torch.rand(10) # This one does NOT requires gradients

# Backward works
output = self * other
output.backward(torch.ones_like(output))

# Here we only ask for gradients for self. So self should not be saved:
output = self * other
self += 1
output.backward(torch.ones_like(output)) # Works !

# But we need the value of other:
output = self * other
other += 1
output.backward(torch.ones_like(output)) # Fails !
</code></pre>

username: two_Two
cooked: <p>Man!this is really really great! Thanks!</p>