In the grad_fn ，I find a next_functions ,But I don't understand the meaning of the attribute

wuyongyu: <pre><code class="lang-auto">a=Variable(torch.randn(1),requires_grad=True)
p=a*a
p_tmp = p.expand_as(p)
grad_acc = p_tmp.grad_fn.next_functions[0][0]
                 
</code></pre>
<p>I dont  know the meaning of ‘next_functions’,and I couldn’t find the define of it.</p>

tom: <p>First, don’t use <code>Variable</code> anymore! Be modern!</p>
<p>Regarding your question, the next_functions will allow you to traverse the recorded calculation graph (“backward graph”).<br>
The backward graph will end in AccumulateGrad nodes for the leaves (they have a <code>.variable</code> attribute pointing to the leaf tensor) - and yours does pretty quickly as you only have one operation. Let’s have a slightly more elaborate one:</p>
<pre><code class="lang-python">a = torch.randn(1, requires_grad=True)
b = a*(a+2)
print (b.grad_fn.next_functions)
print (b.grad_fn.next_functions[1][0].next_functions)
print (b.grad_fn.next_functions[0][0].variable is a)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">((&lt;AccumulateGrad object at 0x7fbe7aa96780&gt;, 0), (&lt;AddBackward0 object at 0x7fbe7aa96748&gt;, 0))
((&lt;AccumulateGrad object at 0x7fbe7aa96780&gt;, 0), (None, 0))
True
</code></pre>
<p>So in ‘x*(x+2)’ you have one branch for <code>x</code> and one for <code>x+2</code> and the latter has a <code>x</code> branch and an uninteresting <code>2</code> branch.<br>
Except at the leaves, you cannot, in general access the variables of the calculation from the graph.</p>
<p>Best regards</p>
<p>Thomas</p>

wuyongyu: <p><a class="mention" href="/u/tom">@tom</a> thanks very much ! I remember that and be modern. <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=6" title=":grin:" class="emoji" alt=":grin:"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=6" title=":grin:" class="emoji" alt=":grin:"></p>

saluto: <p>Thanks for your explanation. Can you explain what the second element in each of these inner tuples stands for? They are all <code>0</code> and I cannot create an example causing a different value. Can they be e.g. <code>1</code>?</p>

tom: <p>The number is the input number to the next backward function, so can only be non-zero when a function has multiple differentiable outputs (there aren’t that many, but e.g. the RNN functions typically do).<br>
A minimal example that doesn’t serve much purpose except showing you a <code>1</code> is:</p>
<pre><code class="lang-python">a, b = torch.randn(2, requires_grad=True).unbind()
c = a+b
print(c.grad_fn.next_functions)
</code></pre>
<p>Unbind is not that well-known, it is the “opposite” of stack, splitting a tensor along one (the first by default) dimension into a list.</p>
<p>Best regards</p>
<p>Thomas</p>

saluto: <p>Thanks, again. Now I understand.</p>

saluto: <p>Sorry for bothering you again. Off topic, but probably not worth a new thread: Do you have an explanation for the following behaviour regarding <code>unbind()</code>?</p>
<pre><code class="lang-python">a = torch.randn(2, requires_grad=True)
a0, a1 = a.unbind()
# a0, a1 = a  # but this works
a0.backward()
</code></pre>
<p>Causes this error:</p>
<pre><code class="lang-auto">RuntimeError: Expected a Tensor of type Variable but found an undefined Tensor at position #1 for iterable argument #0 'tensors'
</code></pre>
<p>Best regards,<br>
Saluto</p>

tom: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/9977" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/fritzo" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/648532?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/9977" target="_blank" rel="nofollow noopener">Issue: torch.stack() gradient errors in 0.4.1</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/fritzo" target="_blank" rel="nofollow noopener">fritzo</a>
	on <a href="https://github.com/pytorch/pytorch/issues/9977" target="_blank" rel="nofollow noopener">2018-07-28</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/facebook-github-bot" target="_blank" rel="nofollow noopener">facebook-github-bot</a>
	on <a href="https://github.com/pytorch/pytorch/issues/9977" target="_blank" rel="nofollow noopener">2018-08-09</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">Issue description
We're seeing Pyro code breakage due to interaction of torch.stack and binary_cross_entropy_with_logits as used inside torch.Bernoulli.log_prob().
Possibly related: #4274
Code examples
The following...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">nightly-announce</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>fixed in master:</p>
<aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/9995" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
      <a href="https://github.com/t-vi" target="_blank" rel="nofollow noopener">
    <img alt="t-vi" src="https://avatars2.githubusercontent.com/u/20787943?v=4" class="thumbnail onebox-avatar" width="90" height="90">
  </a>

<h4>
  <a href="https://github.com/pytorch/pytorch/pull/9995" target="_blank" rel="nofollow noopener">Deal with undefined tensors in unbind backward</a>
</h4>

<div class="date">
  by <a href="https://github.com/t-vi" target="_blank" rel="nofollow noopener">t-vi</a>
  on <a href="https://github.com/pytorch/pytorch/pull/9995" target="_blank" rel="nofollow noopener">08:43PM - 29 Jul 18</a>
</div>

<div class="github-commit-stats">
  <strong>2 commits</strong>
  changed <strong>3 files</strong>
  with <strong>25 additions</strong>
  and <strong>5 deletions</strong>.
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Best regards</p>
<p>Thomas</p>

saluto: <p>Really great, thanks!</p>

guihao-liang: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="24463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<pre><code class="lang-auto">((&lt;AccumulateGrad object at 0x7fbe7aa96780&gt;, 0), (&lt;AddBackward0 object at 0x7fbe7aa96748&gt;, 0))
((&lt;AccumulateGrad object at 0x7fbe7aa96780&gt;, 0), (None, 0))
True
</code></pre>
</blockquote>
</aside>
<p>Hi Tom,</p>
<p>what about the tuple <code>(None, 0)</code>?</p>
<hr>
<p>it seems to be the backprop function for the constant <code>2</code>.</p>

tom: <aside class="quote no-group" data-username="guihao-liang" data-post="10" data-topic="24463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/guihao-liang/48/49948_2.png" class="avatar"> guihao-liang:</div>
<blockquote>
<p>it seems to be the backprop function for the constant <code>2</code>.</p>
</blockquote>
</aside>
<p>Indeed. Internally, PyTorch loves to connect all relevant inputs to the graph, and if those don’t require gradients, you get None.</p>

BruceDai003: <aside class="quote no-group" data-username="tom" data-post="11" data-topic="24463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>Internally, PyTorch loves to connect all relevant inputs to the graph, and if those don’t require gradients, you get None.</p>
</blockquote>
</aside>
<p>Thanks for the explanation, I’m actually running some test_autograd.py unit test. And there is typically a gradgradcheck for a test function. Some function failed gradgradcheck, although they passed gradcheck. So as I’m using VScode python debugger to debug this case. I really need to understand how some function, e.g. addcdiv do the backward pass. Now I see how we can use grad.fn.next_functions to recover a lot of information. And also from the derivatives.yaml file, the backward calculation formula for all the functions are already there. But my question is how can we mannully get the gradients for the inputs if we only have output, or output.grad_fn.<br>
As you mentioned, for example, <code>y=x*2</code>, we can backtrace to x from AccummulateGrad object’s variable attribute, but dy/dx = 2, this is the gradient, where is this information stored?</p>

tom: <aside class="quote no-group" data-username="BruceDai003" data-post="12" data-topic="24463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brucedai003/48/41490_2.png" class="avatar"> BruceDai003:</div>
<blockquote>
<p>But my question is how can we mannully get the gradients for the inputs if we only have output, or output.grad_fn.</p>
</blockquote>
</aside>
<p>To the best of my knowledge, not all “backward-functions” are exposed. In particular those in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/FunctionsManual.cpp" rel="noopener nofollow ugc">FunctionsManual.cpp</a>, the backward computations which typically are composed themselves of PyTorch functions, are not themselves exported.<br>
(At least that was the state last I looked. Personally, I think that they should be available programmatically, but it didn’t catch on.)</p>
<p>Best regards</p>
<p>Thomas</p>

BruceDai003: <p>Thanks, in the case of <code>addcdiv</code>, when I do gradgradcheck, that is to calculate the second order of derivatives. I just successfully manually used grad_fn(torch.ones(1, device=‘cuda:0’)) to get the grad to the inputs of this grad_fn. And by looking at the next_functions, and then pass the grad of the inputs from last step as the input to this next_functions’ input, I can manually get the gradients to the leafs eventually. However, for some of the scalars, I have to do a manual <code>tensor.sum()</code> to get the scalar gradients in the end. So I think the grad_fn class does contain the information including what shapes are accepted, what the constants to mul/div are.(e.g. in y = 2 * x, the 2 is a constant)<br>
So my manual calculations using grad_fn does coninside with what should be. But on my machine, the gradgrad results for <code>addcdiv</code> is not all correct. So I’m stuck at how to debug this. In the <code>addcdiv</code> example, the backward-functions are all very simple, like SumBackward, MulBackward, NegBackward, DivBackward, AccumulateGrad. And I already got the graph of how they are chained together. But I don’t know where can I set a breakpoint in the c++/cuda source code in order to debug this. For example, if DivBackward for cuda tensor has some bug, where should I put a breakpoint for this? Much appreciated.</p>

tom: <p>Right, calling the grad_fn works these days.<br>
So there are three parts:</p>
<ol>
<li>part of the interface is generated at build-time in torch/csrc/autograd/generated . These include the code for the autograd Node classes like DivBackward.</li>
<li>the generated wrapping code ultimately calls the function specified in tools/autograd/derivatives.yaml, which you can find in the source code (either in FunctionsManual.cpp or in aten/src/ATen/native/native_functions.yaml and implemented in ATen).</li>
<li>the graph recording for the double backward is done by treating the PyTorch calls in the backward as regular PyTorch function calls and they again generate autograd nodes, so you are again in the situation of 1./2.</li>
</ol>
<p>If you track down the functions to step 2 of the double backward, you could set breakpoints there.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>By the way: Do you have a (smallish) reproducing example of the failing gradgradcheck? It is very likely that people would be keen to help debug it if they can reproduce the error.</p>

amsword: <p>I have a question about addition operation between a scaler and a tensor. As follows, it crashes with shape mismatched error. It seems like y (a scaler) is implicitly expanded to the same size as x (a tensor). But in this case, how the gradient of y is calculated through next_functions?</p>
<pre><code class="lang-python">      x = torch.randn(4,4, requires_grad=True)
      y = torch.tensor(2., requires_grad=True)
      z = x * y
      l = z.sum()
      dl = torch.tensor(1.)
      back_sum = l.grad_fn
      dz = back_sum(dl)
      back_mul = back_sum.next_functions[0][0]
      dx, dy = back_mul(dz)
      back_x = back_mul.next_functions[0][0]
      back_x(dx)

      back_y = back_mul.next_functions[1][0]
      back_y(dy)
</code></pre>
<pre><code class="lang-auto">RuntimeError: output with shape [] doesn't match the broadcast shape [4, 4]
  12207     back_y = back_mul.next_functions[1][0]
&gt; 12208     back_y(dy)
</code></pre>

tom: <p>So the backward of <code>y = x.expand(y_size)</code> is <code>dl_dx = dl_dy.sum_to_size(x.size())</code>.</p>
<p>Just as broadcasting is inserting implicit expands, the autograd engine will insert implicit <a href="https://github.com/pytorch/pytorch/blob/8535418a06d75025541370cc656a8b6a0330ca0d/torch/csrc/autograd/engine.cpp#L694" rel="noopener nofollow ugc">“expand backwards” in the form of sumtosize</a>. If you do the backwards manually, you have to do the <code>sum_to_size</code> yourself.</p>
<p>Best regards</p>
<p>Thomas</p>