vector-Jacobian product calculation

anlytix: <p>In the <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">autograd tutorial</a> in the 60 minute blitz, the vector jacobian product calculation seems to miss a constant factor (m) on the right hand side (for example, in the first product, there are m equal terms of dl/dx1, that add to m.dl/dx1 - not just dl.dx1:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbfb66f33bfc028f5fc154401a2e532e7958cd23.png" data-download-href="https://discuss.pytorch.org/uploads/default/cbfb66f33bfc028f5fc154401a2e532e7958cd23" title="Screen Shot 2020-05-03 at 9.34.45 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbfb66f33bfc028f5fc154401a2e532e7958cd23.png" alt="Screen Shot 2020-05-03 at 9.34.45 PM" data-base62-sha1="t6vyicgvZLb3P0Joax56F0Wn8IP" width="690" height="231" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-05-03 at 9.34.45 PM</span><span class="informations">884×296 9.79 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there some convention or calculation I am missing?</p>
<p>Thanks,<br>
Anand</p>

albanD: <p>Hi,</p>
<p>Not it wouldn’t be m*, it is that: <code>dl/dx1 = sum_i (dl/dyi * dyi/dx1)</code>. As you need to accumulate the gradient contribution from all the outputs to get the full gradient.</p>

anlytix: <p>Thanks, <a class="mention" href="/u/alband">@albanD</a>.</p>
<p>I don’t understand - sum_i (dl/dyi * dyi/dx1) = sum_i (dl/dx1) - but the sum is over m terms, and all terms are dl/dx1 - so, why isn’t that m times dl/dx1?</p>

albanD: <p>Because you cannot do that simplification. And in particular, you don’t know <code>dl/dx1</code>, that’s what you’re trying to compute.</p>

anlytix: <p>We are trying to compute the matrix-vector product - and I am claiming that the first term computes to m.dl/dx1, and not dl/dx1 as shown. Just look at the Jacobian transposed as shown, and multiplying terms of first row of jacobian transposed and v, and add them up.  The canceling of dy_i is exactly permitted by the chain rule, because, for each respective product term, it is dy1, dy2, etc. respectively, in both the numerator and denominator. So, for the first term of the output vector, the sum is <code>(dy1/dx1 * dl/dy1) + (dy2/dx1 * dl/dy2) + ... </code>, and in each term, the dyi term cancels out, by the chain rule.</p>
<p>I totally might be missing something, but I don’t see it.</p>

albanD: <p>All we’re doing here is doing an abuse of notation. You should say that <code>dl/dyi * dyi/dx = dl/dx_i</code> where the <code>_i</code> signifies that this only considers the contribution from y_i.</p>
<p>If we write things down: You have a function <code>Y = f(X)</code>.<br>
And we are given for every entry in Y, a gradient wrt a loss <code>dl/dyi</code>.<br>
Now if you want to compute the gradient for a given entry in X, let’s say x0, you want <code>dl/dx0</code>. And to compute this, you use the chain rule to consider all the elements in Y that contribute to <code>l</code> which were made from <code>x0</code>. And so <code>dl/dx0 = sum_i (dl/dyi * dyi/dx0)</code> where i allows you to consider all the outputs one by one.</p>

anlytix: <p>Ok, got it. Thanks a lot for explaining. My partial diffferential calculus is very rusty, and I was confusing between partial derivative chain rule (where it does not make sense to cancel out the dx terms in each partial derivative product, since they are different <em>partial</em> derivative terms being multiplied) with the univariate chain rule, where we can cancel out such terms.</p>
<p>For others who land here with similar confusion, <a href="http://tutorial.math.lamar.edu/Classes/CalcIII/ChainRule.aspx" rel="nofollow noopener">This article</a> about chain rule for partial derivative (see case 2) also helped clarify.</p>
<p>Thanks again for clarifying - this was very baffling - now it is clear.<br>
Anand</p>