What does doing .backward twice do?

pinocchio: <p>I was curious to call backward twice in series, where I had a sequential computation (graph). I have a loss function in the middle and another at the end. I call backward on both of them. That does not seem to be equivalent to getting the second derivative (cuz I did it manually with Sympy). I wasn’t trying to do the second derivative (though it would be nice to figure that out too), but was curious what that is doing.</p>
<p>So what is it doing?</p>
<hr>
<p>Experiment results:</p>
<pre><code class="lang-auto">---- test results ----
experiment_type = grad_constant_but_use_backward_on_loss
-- Pytorch results
g = 12.0
dJ_dw = 16838.0
-- Sympy results
g_SYMPY = 2*x*(w*x - y)
dJ_dw_SYMPY = 197258.000000000
---- test results ----
experiment_type = grad_analytic_only_backward_on_J
-- Pytorch results
g = tensor([12.], grad_fn=&lt;MulBackward0&gt;)
dJ_dw = 197258.0
-- Sympy results
g_SYMPY = 2*x*(w*x - y)
dJ_dw_SYMPY = 197258.000000000
</code></pre>
<hr>
<p>Code:</p>
<pre><code class="lang-auto">    ## variable declaration
    w = torch.tensor([2.0], requires_grad=True)

    x = torch.tensor([3.0], requires_grad=False)
    y = torch.tensor([4.0], requires_grad=False)

    x2 = torch.tensor([5.0], requires_grad=False)
    y2 = torch.tensor([6.0], requires_grad=False)
    if True:
        ## computes backard pass on J (i.e. dJ_dw) but g is was backwarded passed already
        # compute g
        loss = (w*x-y)**2
        loss.backward()
        g = w.grad.item() # dl_dw
        # compute w_new
        w_new = w - (g+w**2) * g
        # compute final loss J
        J = (w_new + x2 + y2)**2
        # computes derivative of J
        J.backward()
        #dw_new_dw = w_new.grad.item()
        dJ_dw = w.grad.item()

    print('---- test results ----')
    print(f'experiment_type = {experiment_type}')
    print('-- Pytorch results')
    print(f'g = {g}')
    #print(f'dw_new_dw = {dw_new_dw}')
    print(f'dJ_dw = {dJ_dw}')
    print('-- Sympy results')
    g, dw_new_dw, dJ_dw = symbolic_test(experiment_type)
    print(f'g_SYMPY = {g}')
    #print(f'dw_new_dw_SYMPY = {dw_new_dw}')
    print(f'dJ_dw_SYMPY = {dJ_dw}')
</code></pre>
<p>and sympy code:</p>
<pre><code class="lang-auto">def symbolic_test(experiment_type):
    '''
    '''
    w, x, y, x2, y2, g = symbols('w x y x2 y2 g')

    loss = (w*x - y)**2
    grad = diff(loss,w)
    if experiment_type == 'grad_constant':
        ## compute g
        eval_grad = grad.evalf(subs={w:2,x:3,y:4})
        g = eval_grad
        ## compute w_new
        w_new = w - (g+w**2) * g
        ## compute final loss J
        J = (w_new + x2 + y2)**2
        ##
        dw_new_dw = diff(w_new,w).evalf(subs={w:2,x:3,y:4})
        dJ_dw = diff(J,w).evalf(subs={w:2,x:3,y:4,x2:5,y2:6})
    else:
        ## include grad as a symbolic variable into next expressions
        g = grad
        ## compute w_new
        w_new = w - (g+w**2) * g
        ## compute final loss J
        J = (w_new + x2 + y2)**2
        ##
        dw_new_dw = diff(w_new,w).evalf(subs={w:2,x:3,y:4})
        dJ_dw = diff(J,w).evalf(subs={w:2,x:3,y:4,x2:5,y2:6})
    return g, dw_new_dw, dJ_dw
</code></pre>
<hr>
<p>interesting! It seems it doesn’t matter which of these two I used:</p>
<pre><code class="lang-auto">        g = w.grad.item() # dl_dw
        #g = w.grad # dl_dw
</code></pre>
<hr>
<p>This still does seem to make a difference!</p>
<pre><code class="lang-auto">        g = w.grad # dl_dw
        g.requires_grad = True
</code></pre>
<hr>
<p>cross posted:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/what-does-doing-backward-twice-do/58057" class="inline-onebox">What does doing .backward twice do?</a></li>
<li><a href="https://stackoverflow.com/questions/58359316/what-does-doing-backward-twice-do-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/58359316/what-does-doing-backward-twice-do-in-pytorch</a></li>
<li><a href="https://www.quora.com/unanswered/What-does-doing-backward-twice-do-in-Pytorch" rel="nofollow noopener">https://www.quora.com/unanswered/What-does-doing-backward-twice-do-in-Pytorch</a></li>
</ul>

albanD: <p>Hi,</p>
<p>A few points:</p>
<ul>
<li>Is that expected that the experiment types you use do not match the string tested in the if statement in the sympy code?</li>
<li>Could you provide the code for the two different pytorch runs you do?</li>
<li>I would avoid using <code>.item()</code> in pytorch as it unpacks the content into a regular python number and thus it breaks gradient computation. If you want to have a new Tensor such that no gradients will flow back, you should use <code>.detach()</code>.</li>
<li>Note that when you call <code>.backward()</code>, it accumulates into the <code>.grad</code> field. So here you accumulate both the gradients for loss and J into <code>w.grad</code> in the code sample you provided.</li>
</ul>

pinocchio: <p>need to read:</p>
<aside class="quote quote-modified" data-post="4" data-topic="58059">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-one-obtain-gradients-as-data-efficiently/58059/4">How does one obtain gradients as data efficiently?</a> 
  </div>
  <blockquote>
    the whole code just for reference: 
'''
This script is written to test:
 1)what happens if we differentiate through pytorch's backward pass and to see
    if its consistent with the doing a second derivative
'''
import torch
import torch.nn as nn

from grad_test_sympy import symbolic_test

import sys

from pdb import set_trace as st

def test_does_backward_backward_match_second_derivative(experiment_type,grad_computation_type='dl_dw_zero_grad'):
    '''
     1)what happens if we differentiate th…
  </blockquote>
</aside>

<p>and</p>
<aside class="quote quote-modified" data-post="2" data-topic="16861">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/2">Clone and detach in v0.4.0</a> 
  </div>
  <blockquote>
    We’ll provide a migration guide when 0.4.0 is officially released. Here are the answers to your questions: 


tensor.detach() creates a tensor that shares storage with tensor that does not require grad. tensor.clone()creates a copy of tensor that imitates the original tensor's requires_grad field. 
You should use detach() when attempting to remove a tensor from a computation graph, and clone as a way to copy the tensor while still keeping the copy as a part of the computation graph it came from. …
  </blockquote>
</aside>


pinocchio: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>I would avoid using <code>.item()</code> in pytorch as it unpacks the content into a regular python number and thus it breaks gradient computation. If you want to have a new Tensor such that no gradients will flow back, you should use <code>.detach()</code> .</p>
</blockquote>
</aside>
<p>Hi AlbanD,</p>
<p>I was puzzled about your comment on using <code>.item()</code>. Why would it break gradient computations? Do you have a high-level explanation of why it does that?</p>
<p>Perhaps more importantly, doesn’t that mean that using python floats in torch might cause problems? (since that’s what <code>.item()</code> extracts) Does that mean I should wrap <strong>everything</strong> in torch variables?</p>
<p>Thanks in advance!</p>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Note that when you call <code>.backward()</code> , it accumulates into the <code>.grad</code> field. So here you accumulate both the gradients for loss and J into <code>w.grad</code> in the code sample you provided.</p>
</blockquote>
</aside>
<p>I don’t want to accumulate (which is what happens when I call backwards multiple time in the same computation graph). I want to extract the gradients for params wrt to <code>loss</code> and use those gradients (call them <code>dl_dw</code>) for later computations of <code>J</code> and <code>dJ_dw</code>. The way I’m doing it is not right because I can’t even call backward on <code>J</code> because when I try to zero out the gradients before calling backward on <code>J</code> pytorch halts on me due to illegal in-place operations.</p>
<p>However, I noticed that pytorch doesn’t allow me to do because it warns me about (not really (illegal)) in-place operations in my case. Is there a way to extract intermediate gradients without pytorch halting without my permission?</p>
<p>In case you need it, see new example code I cooked up for this</p>
<pre><code class="lang-auto">import torch
from torchviz import make_dot

x = torch.ones(10, requires_grad=True)
weights = {'x':x}

y = x**2
z = x**3

l = (x-2).sum()
l.backward()
g_x = x.grad

#g_x.requires_grad = True ## Adds it to the computation graph!
print(f'g_x = x.grad = {x.grad}\n')

x.zero_()

#weights['g_x'] = g_x
#print(f'weights = {weights}\n')

J = (y + z + g_x).sum()
J.backward()
print(f'g_x = x.grad = {x.grad}\n')

make_dot(J,params=weights)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-27-c4086df8154e&gt; in &lt;module&gt;
     20 print(f'g_x = x.grad = {x.grad}\n')
     21 
---&gt; 22 x.zero_()
     23 
     24 #weights['g_x'] = g_x

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>

albanD: <aside class="quote no-group" data-username="pinocchio" data-post="4" data-topic="58057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p>I was puzzled about your comment on using <code>.item()</code> . Why would it break gradient computations? Do you have a high-level explanation of why it does that?</p>
</blockquote>
</aside>
<p>Because it returns a python number and we cannot track gradients for python numbers, only torch Tensors.</p>
<blockquote>
<p>Does that mean I should wrap <strong>everything</strong> in torch variables?</p>
</blockquote>
<p>It will be done automatically for you whenever needed. The Tensors you create that require gradients are Tensor already.<br>
So you don’t need to wrap anything. But you should not unwrap things either.</p>
<blockquote>
<p>x.zero_()</p>
</blockquote>
<p>This operation on <code>x</code> is not valid. You cannot change its value in a differentiable manner as it is a leaf…<br>
If you just want to zero-out the content of <code>x</code> without the autograd knowing, you should do:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
  x.zero_()
</code></pre>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="58057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>This operation on <code>x</code> is not valid. You cannot change its value in a differentiable manner as it is a leaf…<br>
If you just want to zero-out the content of <code>x</code> without the autograd knowing, you should do:</p>
<pre><code class="lang-auto">with torch.no_grad():
  x.zero_()
</code></pre>
</blockquote>
</aside>
<p>I was reading your post on what <code>torch.no_grad()</code> does and it says I won’t be able to do backward passes on my computation graph if I use that (though with some speed/memory benefits).</p>
<p>That is not what I am trying to do. I want things to be differentiable but I want to be able to clear out gradients of sub-graphs whenever I want.</p>
<p>More clearly explained: I create a forward pass and in a earlier part of the graph I want to collect gradients to later use those gradients to create the whole computation graph. But the gradients are part of the graph and I want them zeroed out since they should be treated as constants part of the computation graph. Thus I will call backward twice and zero out the intermediate gradients collected (but I will extract to use them later).</p>
<p>Does that make sense?</p>

albanD: <p>You meant to do <code>x.grad.zero_()</code> ?</p>

pinocchio: <p>Oh I see. <code>.zero_()</code> makes all the content of a tensor zero (not just gradients). But I was trying to make the contents of a tensor that is part of a computation graph zero (with an in-place operation that is not tracked). So Pytorch was trying to protect me from it. But Pytorch was ok with <code>x.grad</code> being zeroed because it is not part of the computation graph so far, so it’s fine to call <code>zero_()</code> on it.</p>
<p>The (wrong) assumption I made was that <code>.zero_()</code> was built to make gradients zero, not tensors. So naturally, I was very confused as of why pytorch was complaining.</p>
<p>Thanks AlbanD! Your such a boss in this forum. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/muscle.png?v=9" title=":muscle:" class="emoji" alt=":muscle:"></p>