Updates during Backpropagation

Hmrishav_Bandyopadhy: <p>This question might seem to be a bit theoretical and not suited as it is not directly linked to pytorch, but please bear with me as I have been asking this about and haven’t got a reply from anywhere <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>
<p>We know that during backpropagation, the weight update of the last layer is calculated by calculating the derivative of the loss with respect to the last layer’s weights and plugging in the differential values of the activation function and the inputs to the last layers. Now, when the (last-1) layer’s derivatives are calculated with the help of the derivatives of the inputs to the last layer ( and the output of the (last-1) layer), then are previous weight values of the last layer used to fill in the equation or the updated weight values used? In other words, are the weights of the last layer updated as soon as the gradients are calculated, or does it wait till calculating all the gradients of all the layers and then updates them simultaneously when we call optimizer.step() ?</p>
<p>For a more detailed view of the problem , please see–<br>
<a href="https://drive.google.com/file/d/11RPvQCbMm6yLmQwEcZ_5uOg37F8mdOvr/view?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://drive.google.com/file/d/11RPvQCbMm6yLmQwEcZ_5uOg37F8mdOvr/view?usp=sharing</a></p>

albanD: <p>Hi,</p>
<p>In pytorch, the weights are only updated when you do <code>optimizer.step()</code> which happens after the <code>.backward()</code> pass is completed. So no.<br>
In general, if you rewrite all your formulas to add a <code>t</code> parameter to your weights to differentiate the version before the update and the one after. You will see that the gradient only ever depends on the original value, and never on the value after update (t+1). So if you compute the gradients, it should never happen in general, even if you fold the optimizer step into the backward pass.</p>

Hmrishav_Bandyopadhy: <p>Thanks a lot! That cleared things up <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Hmrishav_Bandyopadhy: <p>Another small query if you will… in my Equation(2), I have the term as this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0fd6e258dbb3211003145daa661f583e5ca9373.png" alt="Screenshot from 2020-07-20 22-45-28" data-base62-sha1="mYbmL8pEn4QRZ5j4TkZrWDOeA4r" width="327" height="73"><br>
However, In “Efficient Backprop” by Yann Le Cun, (<a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="noopener nofollow ugc">http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf</a> - Page 4 Equation 9), he has a transpose on the Wn term…and I can’t seem to understand why a transpose of the weights is being done…</p>

albanD: <p>This is most likely due to the differentiation rules depending on the convention you use.<br>
You can check this by making sure the different matrix dimensions match.</p>

Hmrishav_Bandyopadhy: <p>Oh… Okay thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Hmrishav_Bandyopadhy: <p>Hi,<br>
There is something else in this context which has been bothering me–<br>
Suppose we have an input layer X0, weight W1, and output X1, then</p>
<ol>
<li>
<p>X1.gradient is a <em>scalar formed by summing up the gradients of all units of X1</em></p>
</li>
<li>
<p>W1 is a weight matrix containing internal weights w(i,j) which are updated based on :<br>
* The <em>scalar gradient of X1</em><br>
* And the <em>vector X0</em></p>
</li>
</ol>
<p>This however, brings me up to the conclusion that for each w(i,j) to be updated individually, we should have the scalar X1 and each unit of the X0 matrix available for the specific i and j – Which is not plausible as the X0 matrix has a different shape as compared to the W1 matrix. How exactly are individual weights of the W1 matrix computed?</p>

albanD: <p>Hi,</p>
<p>Where does “X1.gradient is a *scalar formed by summing up the gradients of all units of X1” comes from?<br>
The gradient for W is the outer product between the gradient of X1 (which is the same size as X1, not a scalar) and X0 which is is input.</p>

Hmrishav_Bandyopadhy: <p>X1 is the output vector–and here’s what i found from that efficient backprop paper:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f501450b327b24e22c76e44672c828f1328dd8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/3f501450b327b24e22c76e44672c828f1328dd8a" title="Screenshot from 2020-07-27 19-32-03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f501450b327b24e22c76e44672c828f1328dd8a_2_690x279.png" alt="Screenshot from 2020-07-27 19-32-03" data-base62-sha1="925Ei5KWzXRVfjN3hdiOaOx2P22" width="690" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f501450b327b24e22c76e44672c828f1328dd8a_2_690x279.png, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f501450b327b24e22c76e44672c828f1328dd8a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f501450b327b24e22c76e44672c828f1328dd8a.png 2x" data-dominant-color="E7E79B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-07-27 19-32-03</span><span class="informations">941×381 54.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here The scalar error at the node is not the gradient of the error with respect to X1?</p>
<p>Page 8 — 9th line of <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="noopener nofollow ugc">this</a></p>

albanD: <p>I think here, the “node” is a single element of X1. And so its gradient is just a single number.</p>

Hmrishav_Bandyopadhy: <p>Ah, but then if ẟ is not a single number, that brings down the whole concept of ‘Zigzagging’, right?–As the weight vectors are actually not changing together in ‘fixed directions’. Their directions are unique as ẟ is a vector. But then, can you give me a pointer to an article or a paper which explains why normalizing is necessary without using this ‘Zigzagging’ reference  as most of what i have come across uses this. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>I don’t think a proof exist for neural nets in general.<br>
But this considers a simpler case with a single weight. And I guess they claim that it extends similarly when you have more layers/weights?</p>