[Newb] Is there a way to step into Variable._execution_engine.run_backward()

Maziar: <p>Hi</p>
<p>I am trying to understand/instrument autograd in PyTorch. I’ve put a pdb.set_trace() before backward and traced the code until Variable._execution_engine.run_backward() is called which prevents pdb to step into. I presume this is where the code calls C++ extensions.</p>
<p>If so, is there a way to continue debugging and step into this function?</p>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<p>This function and the whole autograd engine is implemented in cpp.<br>
You will need a cpp debugger if you really want to step into it. It might not be very helpful though depending on what you’re trying to achieve?</p>

Maziar: <p>You are right. Following yours and others advice in <a href="https://discuss.pytorch.org/t/how-to-use-pdb-or-gdb-debug-from-python-into-c-c-code/8152">this thread</a>, I cloned PyTorch’s source, built it with debugging flags, and set a pdb_trace point before backward. However I don’t actually know what happens after Variable._execution_engine.run_backward(), therefore I can’t put a breakpoint using gdb on called C++ function, so that it would stop when it reaches the underlying code.</p>
<p>As for my purpose, while I know autograd’s high level functionality,  I am trying to study/tweak/understand how it is implemented.</p>

albanD: <p>The cpp engine is based on <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/function.h" rel="nofollow noopener">Function</a> which are similar to the python ones. They are the elementary operations that are considered.<br>
The forward pass attaches a grad_fn to the Tensors during the forward pass. Then you have a graph of Functions stored in cpp. Accessed from the next_functions field.</p>
<p>When computing a backward, the engine traverses this graph. The entry point in cpp is <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.cpp#L536" rel="nofollow noopener">here</a> and you can look in the same file for all the code.</p>

yukino: <p>Where can I find the implementaion for add-edge, sub-edge, multiple-edge and so on ??</p>

albanD: <p>Hi,</p>
<p>Where did you see these functions?</p>

yukino: <p>I didnt see these function, but I know pytorch can do any differential operator, so it might be some files that implement these differentiation</p>

albanD: <p>I’m not sure what you expect these functions to be / do?<br>
There are no functions to manipulate the graph explicitly. It is built when performing operations but cannot be changed.</p>

yukino: <p>The <a>file</a> you told me is actually what I expected to find, but in cpp files.</p>
<p>what is like …<br>
y=x**4<br>
and we can get gradient of y wrt x as 4x^3<br>
so the code must return something like nx^(n-1)</p>

yukino: <p>import torch</p>
<p>x = torch.autograd.Variable(torch.Tensor([2]),requires_grad=True)<br>
y = 5*x**2<br>
y.backward()<br>
x.grad</p>
<p>we get 20 because 5x^2 -&gt; 10x = 10 * 2 = 20<br>
And I would like to know the code inside backward() ,  there might be some function return 10 * input , right?</p>

albanD: <p>Hi,</p>
<p>This is actually never built explicitly.<br>
What happens is that when the functions of the forward are seen, they corresponding backward ops are recorded.<br>
And so if you say <code>y = f( g(x) )</code> where g is the square function and f is time 5, noting <code>z = g(x)</code>, what the backward computes is <code>dy/dx</code> as <code>dy/dz * dz/dx = df(z)/dz * dg(x)/dx = 5 * 2x</code>. Note that the <code>2x</code> is computed first by the backward of g then 5 times that result for the backward of f.</p>
<p>Basically pytorch is just doing backprop, each elementary functions one at a time. Nothing more fancy.</p>

yukino: <p>g(x)=x^2 ,but where pytorch compute dg(x)/dx as 2x or any those elementary function <strong>backward</strong>.<br>
This is what I want to find out.<br>
Also, thanks you for answer my question!!</p>

albanD: <p>You can find <a href="https://github.com/pytorch/pytorch/blob/d71501259b759066e39024e0b2c6a86d975a0bb5/tools/autograd/derivatives.yaml#L664-L665" rel="nofollow noopener">here</a> the lines that define tha backward of the <code>pow</code> functions as the <code>pow_backward</code> functions.<br>
If you look for this function, you will find it <a href="https://github.com/pytorch/pytorch/blob/6615797837917ab7f0cd956779ad702994c0479c/tools/autograd/templates/Functions.cpp#L136-L143" rel="nofollow noopener">here</a> with a slightly more general formula for any power.<br>
Also remember that these always compute the backward pass. So if we have <code>o = f(i)</code>and then <code>o</code> is used to compute some loss L. it computes <code>dL/di = dL/do * do/di</code>. <code>dL/do</code> is given in the <code>grad</code> argument of the function.</p>

yukino: <p>This -&gt;  <strong>templates/Functions.cpp</strong> is what I want !!!<br>
THX A LOT <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>