How can I call backward functions?!

kolorado: <p>Hi,</p>
<p>I want to call some backward functions like:<br>
AddBackward<br>
MulBackward<br>
FftWithSizeBackward<br>
etc…</p>
<p>Where are they and how can I call them with my input?</p>
<p>Thanx</p>

kolorado: <p>Would you help me? <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@alband</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

judepark: <p>It depends on what you used the loss function.</p>

kolorado: <p>It does not depends on loss function, I just want to call the function AddBackward for example. This call is individual and has nothing to do with neural network! <a class="mention" href="/u/fmassa">@fmassa</a> <a class="mention" href="/u/jekbradbury">@jekbradbury</a></p>

judepark: <p>I mean, for example, whatever you used the loss function like nn.CrossEntropyLoss, it can be NllLossBackward …</p>
<p>what you want is just call the backward function manually, check this documentation. (<a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html</a>).</p>

kolorado: <p>As I said please forget about loss! For each operation we have a forward function which we can call as easy as calling for ex: torch.add(x,y). And we have backward function which I dont know how to call manually for example something like torch. AddBackward(ctx,x,y)! I know that autograd call them one by one somehow but I dont know how to call it my self!</p>

albanD: <p>Hi,</p>
<p>Can you please avoid pinging everyone? It creates a lot of noise for us and we do read all the posts.</p>
<p>It depends on which functions you want? There is no common api to access the backward functions without doing the forward.</p>

kolorado: <aside class="quote no-group" data-username="kolorado" data-post="1" data-topic="56524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/9f8e36/48.png" class="avatar"> kolorado:</div>
<blockquote>
<p>AddBackward<br>
MulBackward<br>
FftWithSizeBackward</p>
</blockquote>
</aside>
<p>Thanks for your reply. Sure I wont do it in future.</p>
<p>I want to call these for example:<br>
AddBackward<br>
MulBackward<br>
FftWithSizeBackward</p>
<p>I don’t mind calling them in c++ or python. I just want to see the definition and call them later on.</p>

albanD: <p>For the forward function doing <code>o = x + y</code>, the backward is <code>gx = go</code> and <code>gy = go</code>.<br>
For the forward function doing <code>o = x * y</code>, the backward is <code>gx = y * go</code> and <code>gy = x * go</code>.<br>
For the fft, it depends on which forward function you use. If you use a regular <code>o = fft(x)</code>, I think the gradient is just <code>gx = ifft(go)</code>. If you want to be sure, you can find <a href="https://github.com/pytorch/pytorch/blob/71af7c46bb831330807b87776edfb770999d2aae/aten/src/ATen/native/SpectralOps.cpp#L149-L173" rel="nofollow noopener">here</a> how the top level fft functions are linked to the low level one called <code>_fft_with_size</code>. <a href="https://github.com/pytorch/pytorch/blob/e72b0be2e1c6f63d410cb176fe2380b9990c8f2b/tools/autograd/derivatives.yaml#L1502-L1504" rel="nofollow noopener">Here</a> you can find how the backward of that function is defined for each of it’s arguments. And the <code>fft_backward()</code> can be found <a href="https://github.com/pytorch/pytorch/blob/9b7011c5c2a47575ef5cd70185d2c66f39b198b9/tools/autograd/templates/Functions.cpp#L2006-L2106" rel="nofollow noopener">here</a>. Unfortunately thiss function is not accessible directly as the arguments are quite far from the regular python api. But you can easily find the corresponding <code>torch.[]fft</code> call that correspond to your forward.</p>

kolorado: <p>Thnx <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Another follow up question, what is grad[0] and grad[1] and .defined()?</p>
<pre><code class="lang-auto">Tensor var_std_mean_backward(const variable_list&amp; grads, const Tensor &amp; self, const Tensor &amp; r1, const Tensor &amp; r2, bool unbiased, bool is_std) {
  Tensor grad;
  if (grads[0].defined()) {
    grad = is_std ? std_backward(r1, grads[0], self, unbiased) : var_backward(grads[0], self, unbiased);
  }
  if (grads[1].defined()) {
    Tensor mean_grad = mean_backward(grads[1], self.sizes(), self.numel());
    grad = grads[0].defined() ? grad + mean_grad : mean_grad;
  }
  return grad;
}
</code></pre>
<p>And where are these grad_fn coming from</p>
<blockquote>
<p>&lt;AddBackward0 at 0x1185d8048&gt;</p>
</blockquote>
<p>?</p>
<pre><code class="lang-auto">a=torch.rand(2,3).requires_grad_()                                                                                                                    
b=torch.rand(2,3)                                                                                                                                     
z=(a+b).requires_grad_() 
f=z.sum()                                                                                                                                     
f.backward()                                                                                                                                         
z.grad_fn                                                                                                                                            
Out[17]: &lt;AddBackward0 at 0x1185d8048&gt;
</code></pre>
<p>surpassingly if you search whole repository for the AddBackward you dont find anything. So where are these string coming from?<br>
<a href="https://github.com/pytorch/pytorch/search?q=AddBackward&amp;unscoped_q=AddBackward" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/search?q=AddBackward&amp;unscoped_q=AddBackward</a></p>

albanD: <p><code>grad[0]</code> and <code>grad[1]</code> are the gradient wrt the 0th and 1st output of the forward function. What I called <code>go</code> for my add and backward examples above.</p>
<p>You can’t find these functions because they are automatically generated.<br>
The gradients for most ops are defined in the <a href="https://github.com/pytorch/pytorch/blob/e72b0be2e1c6f63d410cb176fe2380b9990c8f2b/tools/autograd/derivatives.yaml" rel="nofollow noopener">derivatives.yaml</a> file. And the corresponding cpp binding is generated automatically.</p>