Traversing the recorded graph manually

yewang: <p>I’d like to explore the computation graph recorded by autograd manually. I’ve been playing around with <code>next_functions</code>, which seems to be relevant to my needs:</p>
<pre><code class="lang-auto">x = torch.ones(4, requires_grad=True)
loss = x.pow(2)
print(loss.grad_fn)
print(loss.grad_fn.next_functions[0])
</code></pre>
<p>But I’m not sure I completely understand what’s going on. Each element of <code>next_functions</code> is a tuple of a function object and a number. I don’t know what those numbers are. Also, I haven’t found a way to access the input tensor <code>x</code> just by accessing attributes or calling methods on <code>loss</code>. How can I do it?</p>

albanD: <p>Hi,</p>
<ul>
<li>The numbers correspond to the index of the argument for this input: For a function with <code>forward(cts, in1, in2)</code>, the edge that gave <code>in1</code> will have number 0 and the edge that gave <code>in2</code> will have number 1.</li>
<li>In the computational graph, inputs do not really exist. It is just a graph of <code>Function</code>s. Each <code>Function</code> is responsible for saving whatever state it will need for its backward.</li>
<li>For the particular case of <code>Function</code>s created in python, you have the <code>saved_tensors</code> field to get things that were saved during the forward. For <code>Function</code>s created in cpp, there is usually no api to access the saved elements from python.</li>
</ul>

yewang: <p>If I create a <code>CustomFunction</code>, what is shown in <code>output_tensor.grad_fn</code> is <code>CustomFunctionBackward</code>, I couldn’t find a way to access <code>CustomFunction</code> from the output tensor, is it possible / useful? Despite that, using this backward function that appeared out of nowhere I can access <code>saved_tensors</code> if I saved them in <code>CustomFunction.forward</code>, which is the kind of thing I wanted to achieve, but like you said I couldn’t access the tensors of built-in functions, unfortunately.</p>
<p>Is there any documentation about how / why this backward class (<code>CustomFunctionBackward</code>) is created? Also, is it reasonable to think of the first argument <code>ctx</code> in <code>CustomFunction.forward</code> as an instance of <code>CustomFunction</code>? I believe in an old version of pytorch it used to be <code>self</code>, but I’m not sure how it works now. Is there some special context class that is passed at some point during the forward / backward pass?</p>

albanD: <p><code>CustomFunctionBackward</code> is created to represent the state of your <code>Function</code>. An instance of it is created during the <code>.apply()</code> call and is passed as the ctx (though the fact that the context is of this class is not important).<br>
The relevant code is the <code>FunctionMeta</code> in <code>torch/autograd/function.py</code> and <code>THPFunction_apply()</code> in <code>torch/csrc/autograd/python_function.cpp</code>.</p>

qiuyang: <p><a class="mention" href="/u/alband">@albanD</a> Hi,I have a question.torch.nn.Parallel.Scatter also has apply function,Why  after pytorch call Scatter.apply function,the source code will get into Scatter.forward function rather than Scatter.backward in forward process?</p>

albanD: <p>Hi,</p>
<p>When you execute a Function, it runs it’s “forward” and then optionally creates a node that contains the “backward” version of that Function.<br>
For that newly created Function, it’s “forward” is actually Scatter.backward.</p>

qiuyang: <p>Hi,I am sorry I can’t understand.How to explain that BackwardCFunction implements apply function like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d002d6af674d93c730d54e70fd2ece733cf2eb27.png" data-download-href="https://discuss.pytorch.org/uploads/default/d002d6af674d93c730d54e70fd2ece733cf2eb27" title="46"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d002d6af674d93c730d54e70fd2ece733cf2eb27_2_690x118.png" alt="46" data-base62-sha1="tG9oww0pu5R6wTXOiqNzr3DqZE3" width="690" height="118" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d002d6af674d93c730d54e70fd2ece733cf2eb27_2_690x118.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d002d6af674d93c730d54e70fd2ece733cf2eb27_2_1035x177.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d002d6af674d93c730d54e70fd2ece733cf2eb27_2_1380x236.png 2x" data-dominant-color="303227"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">46</span><span class="informations">1532×264 35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It call backward function?I really want to know about it.Does the apply function be called in Variable._execution_engine.run_backward function?</p>
<p>And I have another question.I know that pytorch use DistributedDataParallel to get distributed train.I read the source code about Recucer.It has pending variable to synchronize multiple GPU cards in one node.But how to synchronize differnent nodes?For example,There are three node. The first node finish one step gradients calculation.How does the first node know whether the other nodes finish gradients calculation or not?<br>
Thank you very much!These questions has been bothering me for days.</p>

albanD: <p>Hi,</p>
<p>The question about the apply is answered here: <a href="https://discuss.pytorch.org/t/how-does-pytorch-implements-backward-process/53443/2" class="inline-onebox">How does pytorch implements backward process?</a></p>
<p>I am not very familiar with DistributedDataParallel unfortunately, but it is most likely just uses the autograd logic that makes sure that multiple use of a single Tensor will accumulate all the gradients because performing the next step. Basically, the op before the Reducer knows that 3 copies exists and so will wait 3 completion before executing.</p>

theevann: <p>Hello,</p>
<p>For vizualisation purposes, I would really like to be able to access the saved tensors (ie. access the <code>ctx</code> argument in input of <code>forward</code> and <code>backward</code> of a <code>Function</code> if my understanding is correct).</p>
<p>Is there really no way (even a little hacky) to access the saved elements in python for <code>Function</code> created in cpp (which is the case of most functions) ?</p>
<p>Thank you much,</p>

albanD: <p>There is no current way I’m afraid.<br>
This might be possible to add this but that would be a lot of work with the code that automatically generates these cpp Functions.</p>

william_hero: <p>I think the thing you need is variable:</p>
<pre><code class="lang-python">a = torch.randn(1, requires_grad=True)
b = a*(a+2)
print (b.grad_fn.next_functions)
print (b.grad_fn.next_functions[1][0].next_functions)
print (b.grad_fn.next_functions[0][0].variable)
</code></pre>

vishal-purohit: <p>This does not work. I got error that specific function has no attribute called variable</p>

vishal-purohit: <p>Hello,</p>
<p>Is there method to access and manipulate saved tensors for python??</p>

albanD: <p>Hi,</p>
<p>No there is none.</p>
<p>Also the function has a <code>variable</code> attribute only if it is an <code>AccumulateGrad</code> Node.</p>

vishal-purohit: <p>Hi,<br>
Thanks for your reply I have posted a detailed question in this post please can you look at this discussion and tell me if there is any way to accomplish the task.</p><aside class="quote" data-post="1" data-topic="112521">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishal-purohit/48/34921_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/forward-prop-and-backprop-on-two-physically-different-devices/112521">Forward Prop and Backprop on two physically different devices</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi, 
I want to train a network in such a way that forward pass happens on one physical device and backprop happens on a physically different device only loss from forward prop will be transmitted to different device to carry out backprop. Both devices will have same architecture. How can I accomplish this funtionality?
  </blockquote>
</aside>

<p>Thank you.</p>