Defining Autograd functions tutorial - typo?

Sam_Lerman: <p>Hi, I just read through this clear and straightforward tutorial on defining custom Autograd functions in Pytorch:<br>
<a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a></p>
<p>The code is very clear, but one thing I’d like to clarify on is the comment in the code that says “In the backward pass we receive a Tensor containing the gradient of the loss<br>
with respect to the output, and we need to compute the gradient of the loss<br>
with respect to the input.”</p>
<p>Shouldn’t it be “we need to compute the gradient of the <em>output</em><br>
with respect to the input”?</p>
<p>I’m confused how we could define the gradient of the loss with respect to the input, since typically we think of the gradient of a function as the gradient of that function’s output with respect to its input.</p>
<p>Forgive me if I misunderstood, but I’m also somewhat confused about what exactly is referred to by “gradient of the loss with respect to the output”. In the formulation of backprop that I’m familiar with, the function’s gradient is simply multiplied to the recursive sum of forward gradients (to put it into very loose English). I could give a very short and hopefully simple formal summary of that formulation if needed; it would definitely help me to understand how Pytorch’s autograd fits into my existing understanding.</p>
<p>Thanks so much for any further clarifications!</p>

harsha_g: <p>Your loss <code>L</code> is a function of <code>y</code> (target) and <code>y_hat</code>(output). The first step in backpropagation is <code>dL/dy_hat</code> (gradient of the loss with respect to the output) and that’s what you are reading in the tutorial.</p>

Sam_Lerman: <p>In the general case, for a function not belonging to the output layer, would the correct interpretation be “we need to compute the gradient of the  <em>output</em> with respect to the input”?</p>

harsha_g: <p>Sure, that’s a reasonable interpretation.</p>

Sam_Lerman: <p>Let me clarify. The ReLU function is  <code>f(x)=max(0,x)</code> . It means if  <code>x&lt;=0</code>  then  <code>f(x)=0</code> , else  <code>f(x)=x</code> . In the first case, when  <code>x&lt;0</code> , the derivative of  <code>f(x)</code>  with respect to  <code>x</code>  is  <code>f'(x)=0</code> . So, we perform  <code>grad_input[input &lt; 0] = 0</code> . In other words, we "compute the gradient of the   <em>output</em>  with respect to the input” and not "compute the gradient of the loss with respect to the input”.</p>

Sam_Lerman: <p>Oh okay, sorry, didn’t see your reply before posting. So my understanding in that regard is correct?</p>

harsha_g: <p>Not in the original post but yes in the next post <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> .</p>

Sam_Lerman: <p>Not sure which posts you’re referring to, I don’t think my understanding has changed. But I’m going to assume we’re on the same page about the meaning of the second half of the comment <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  as for the first half of the comment, “gradient of the loss with respect to the output”, is there a general-case interpretation for that as well? I need to use a custom Autograd function that belongs to arbitrary hidden layers, not just output layers. Would it help if I gave some quick backprop formulas so that we could synchronize our language in a formal way?</p>

Sam_Lerman: <p>Basically, I’m trying to clarify how the variable <code>grad_output</code> in that tutorial is computed. Is there an explicit formula written anywhere? If not, I can quickly derive backprop in simple terms and maybe we can identify where <code>grad_output</code> fits into that derivation. It really wouldn’t be as complicated as it sounds. I just need to figure out whether I can use this for my research.</p>

harsha_g: <p><a class="mention" href="/u/sam_lerman">@Sam_Lerman</a> I can assure you my knowledge of Autograd is atrocious <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:">. However, just in case, did you take a look at this post <img src="https://discuss.pytorch.org/images/emoji/apple/point_right.png?v=9" title=":point_right:" class="emoji" alt=":point_right:"> <a href="https://discuss.pytorch.org/t/exact-meaning-of-grad-input-and-grad-output/14186/9" class="inline-onebox">Exact meaning of grad_input and grad_output</a>?</p>
<p>On a side note, the two posts I was referring to were on either side of my first reply. Your original post and your subsequent post to my reply. In my minimal understanding of backpropagation and automatic differentiation, I would not necessarily think the docstring you were referring to is either wrong or misleading. It’s best left to the individual to interpret it in a way that could aid them in generalizing their intuition (which could be wrong too <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>