How does pytorch implements backward process?

username: qiuyang
cooked: <p>I know we can write loss.backward() to finish grad calculation.But when I read source code about backward,I am confused.<br>
I add some print information in pytorch python source code to trace the backward process.<br>
Basically loss.backward() will call torch.autograd.<strong>init</strong>.py backward function and get into Variable._execution_engine.run_backward function.This is a cpp function.When we use DataParallel,the source code finally will call torch.nn.parallel.Gather.backward function and torch.nn.parallel.Broadcast.backward function.How are these two class called?<br>
In addition,Gather.backward funtion will call Scatter.apply function, and Scatter.apply function will call Scatter.forward function.How does it woks?<br>
The Scatter class inherits from torch.autograd.Function class.Function class inherits from a python metaclass.The metaclass bulid a class that the class inherits from BackwardCFunction class.<br>
So in my opinion,if we call Scatter.apply function,the Scatter.backward function will be called.But why not?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/089ee047305bac8d2f4f1c44ff164253e98ae79f.png" data-download-href="https://discuss.pytorch.org/uploads/default/089ee047305bac8d2f4f1c44ff164253e98ae79f" title="41"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/089ee047305bac8d2f4f1c44ff164253e98ae79f_2_690x125.png" alt="41" data-base62-sha1="1egd22zUvYpVZ6grSwBeDNJXtiT" width="690" height="125" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/089ee047305bac8d2f4f1c44ff164253e98ae79f_2_690x125.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/089ee047305bac8d2f4f1c44ff164253e98ae79f_2_1035x187.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/089ee047305bac8d2f4f1c44ff164253e98ae79f_2_1380x250.png 2x" data-dominant-color="2F3127"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">41</span><span class="informations">1562×284 35.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Because there is a static <code>apply</code> function that is called when you do <code>Scatter.apply()</code> and a non-static one that is called on an instance of scatter: <code>Scatter().apply()</code>.</p>

username: qiuyang
cooked: <p>Is the static apply function from _C._FunctionBase class?<br>
I mean that it is from THPFunction_apply?Thanks for your reply!</p>

username: albanD
cooked: <p>Yes, and it’s defined <a href="https://github.com/pytorch/pytorch/blob/59094c409ea552001d0f5356df9fdb0086ccee12/torch/csrc/autograd/python_function.cpp#L1056" rel="nofollow noopener">here</a> as a static method.</p>

username: qiuyang
cooked: <p>Interesting!And I have a final question.Could you tell me where does the non static apply function called?</p>

username: albanD
cooked: <p>Sure.<br>
In the engine, when a new task can be executed <a href="https://github.com/pytorch/pytorch/blob/59094c409ea552001d0f5356df9fdb0086ccee12/torch/csrc/autograd/engine.cpp#L330" rel="nofollow noopener">here</a>.<br>
It first does some checking and input/output preparation <a href="https://github.com/pytorch/pytorch/blob/59094c409ea552001d0f5356df9fdb0086ccee12/torch/csrc/autograd/engine.cpp#L540" rel="nofollow noopener">here</a>.<br>
It then does all the required hooks, the use the <code>operator()</code> function on the actual Node (previously called Function) <a href="https://github.com/pytorch/pytorch/blob/59094c409ea552001d0f5356df9fdb0086ccee12/torch/csrc/autograd/engine.cpp#L505" rel="nofollow noopener">here</a>.<br>
This is routed to the main Node class <a href="https://github.com/pytorch/pytorch/blob/59094c409ea552001d0f5356df9fdb0086ccee12/torch/csrc/autograd/function.h#L112-L119" rel="nofollow noopener">here</a>.<br>
Which calls the <code>apply</code> function of <code>Node</code>, which is a <code>PyNode</code> in your case as it is implemented in python.<br>
And the <code>PyNode</code> gets and calls the <code>.apply</code> attribute from the class instance <a href="https://github.com/pytorch/pytorch/blob/59094c409ea552001d0f5356df9fdb0086ccee12/torch/csrc/autograd/python_function.cpp#L120-L123" rel="nofollow noopener">here</a>.</p>
<p>Looking at it this way, there are a few levels of indirection indeed <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: qiuyang
cooked: <p>I understood!Thanks a lot!</p>