What is 2 in "grad_y_pred = 2*(y_pred - y)" in gradient calculation

Sanjayvarma11: <p>Hello guys, I am new to PyTorch.I am going through an example where I found this code</p>
<pre><code class="lang-auto">    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2*(y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h &lt; 0] = 0
    grad_w1 = x.t().mm(grad_h)

</code></pre>
<p>I didn’t understand at least one line of it. So please can anyone help me by explaining clearly. I tried some reference books where everyone said that this code is out of context.<br>
My questions are:<br>
1)why we used 2 in the first line of code i.e" grad_y_pred = 2*(y_pred - y)"<br>
2)In the code above, what does <code>grad_h[h &lt; 0] = 0</code> signify?<br>
3)what is the use of <code>grad_h = grad_h_relu.clone()</code>??<br>
4)Entire code with significance of each line in one point please  .<br>
Thanks in advance</p>

albanD: <p>Hi,</p>
<p>This code is a bit out of context. Where did you get it from?<br>
So I am going to guess that you try to do the backward pass by hand for a network whole loss function is <code>loss = ||y_pred - y||^2</code> with weights <code>w1</code> and <code>w2</code>. And <code>y_pred = w2 * h_relu</code> and <code>h_relu = relu(w1 * x)</code>.</p>
<p>If this is true, then the code above computes the backward pass using the chain rule by hand.<br>
With each <code>grad_xxx</code> being the derivative of the loss wrt xxx. You can write the gradient of each formula above and you will get these expressions.</p>
<p>The <code>.clone()</code> and <code>[h&lt;0]</code> are here because <code>grad_h[h &lt; 0] = 0</code> changes <code>grad_h</code> inplace to set all the values where the inputs were negative to 0 (the backward of the relu). But because they don’t want to change <code>grad_h_relu</code> inplace (that contains the derivative of the output of the relu). They <code>.clone()</code> that before.</p>

Sanjayvarma11: <p>In this book i found that the above code is out of context<br>
<a href="https://books.google.co.in/books?id=zlvIDwAAQBAJ&amp;pg=PA132&amp;lpg=PA132&amp;dq=what+does+this++++grad_y_pred+%3D+2*(y_pred+-+y)+mean&amp;source=bl&amp;ots=2iNI8yD04P&amp;sig=ACfU3U0kgDY0jXCFxF-SigM9TZf65ninuA&amp;hl=en&amp;sa=X&amp;ved=2ahUKEwi-xde_mbXnAhV3zjgGHcWZAVUQ6AEwAHoECAoQAQ#v=onepage&amp;q=what%20does%20this%20%20%20%20grad_y_pred%20%3D%202*(y_pred%20-%20y)%20mean&amp;f=false" class="onebox" target="_blank" rel="nofollow noopener">https://books.google.co.in/books?id=zlvIDwAAQBAJ&amp;pg=PA132&amp;lpg=PA132&amp;dq=what+does+this++++grad_y_pred+%3D+2*(y_pred+-+y)+mean&amp;source=bl&amp;ots=2iNI8yD04P&amp;sig=ACfU3U0kgDY0jXCFxF-SigM9TZf65ninuA&amp;hl=en&amp;sa=X&amp;ved=2ahUKEwi-xde_mbXnAhV3zjgGHcWZAVUQ6AEwAHoECAoQAQ#v=onepage&amp;q=what%20does%20this%20%20%20%20grad_y_pred%20%3D%202*(y_pred%20-%20y)%20mean&amp;f=false</a></p>

Sanjayvarma11: <p>Thank you so much sir  for answering question.<br>
I joined a course where my instructor provided this code .There i found this code.<br>
So here every line calculates  gradients of different variables involved in forward propagation.Is it right sir??</p>

albanD: <p>Yes it’s right. And it reuses the gradients from the previous variables to do so. That is the backpropagation algorithm.</p>