Why dloss_dw.sum() and dloss_db.sum() in "Deep Learning with PyTorch"?

oat: <p>I’m reading “Deep Learning with PyTorch” by Eli Stevens, Luca Antiga, and Thomas Viehmann. As shown on page 116, the gradients for w and b are calculated as <code>dloss_dw.sum()</code> and <code>dloss_db.sum()</code>.</p>
<p>Although the authors offered explanations, can you help to understand:</p>
<ol>
<li>Why the <strong>element-wise multiplication</strong> between <code>dloss_dtp</code> and <code>dmodel_dw</code>, and that between <code>dloss_dtp</code> and <code>dmodel_db</code>?</li>
<li>Why the <strong>summation</strong> for <code>dloss_dw</code> and <code>dloss_db</code>, when returning the gradient vector?</li>
</ol>
<p>I created a diagram as shown below to help myself to understand the vector calculations under the hood. However, I’m still not sure why the summation is applied in the last step. (x is t_u, y_hat is t_p, y is t_c in the book)</p>
<p>Thanks.</p>
<p>Excerpt from page 116 in “Deep Learning with PyTorch”:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/067ed3e0f96c2884565e0f36b7f36a3191a7e4fa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/067ed3e0f96c2884565e0f36b7f36a3191a7e4fa" title="Screenshot 2021-01-10 at 13.54.58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/067ed3e0f96c2884565e0f36b7f36a3191a7e4fa_2_690x390.jpeg" alt="Screenshot 2021-01-10 at 13.54.58" data-base62-sha1="VsAGVC6q9K6TG5QyqXZ2FroDHA" width="690" height="390" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/067ed3e0f96c2884565e0f36b7f36a3191a7e4fa_2_690x390.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/067ed3e0f96c2884565e0f36b7f36a3191a7e4fa_2_1035x585.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/067ed3e0f96c2884565e0f36b7f36a3191a7e4fa_2_1380x780.jpeg 2x" data-dominant-color="F7F7EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-01-10 at 13.54.58</span><span class="informations">2632×1488 348 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My diagram on gradient calculation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/325b4f7f198f0b928ad5a6d9247b7240b6ef9554.png" data-download-href="https://discuss.pytorch.org/uploads/default/325b4f7f198f0b928ad5a6d9247b7240b6ef9554" title="Screenshot 2021-01-10 at 14.14.05"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325b4f7f198f0b928ad5a6d9247b7240b6ef9554_2_690x405.png" alt="Screenshot 2021-01-10 at 14.14.05" data-base62-sha1="7btvkTTJg20fYK0AGF8nsSIPll2" width="690" height="405" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325b4f7f198f0b928ad5a6d9247b7240b6ef9554_2_690x405.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325b4f7f198f0b928ad5a6d9247b7240b6ef9554_2_1035x607.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/325b4f7f198f0b928ad5a6d9247b7240b6ef9554_2_1380x810.png 2x" data-dominant-color="F3F0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-01-10 at 14.14.05</span><span class="informations">1824×1072 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <ol>
<li>The elementwise multiplication is a radical simplification of the compute involved in the the chain rule. You could probably do something like <code>dloss_dw = torch.diagonal_embedding(dmodel_dw(...)) @ dloss_dtp</code> to be closer to the chain rule but less computationally efficient.</li>
<li>The summation is from using the same term multiple times, here from broadcasting. You can try this yourself on paper by spelling out the gradients per element (for two elements). Or you could have a simpler version with just some scalar <code>g(x,y)</code> and try to compute the derivative of <code>f(x) = g(x,x)</code> to see the same abstract principle.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>