Accessing Local Gradient information

username: sawal86
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f921a8685b0ef67a0c70fcab846f8a56dfea06e1.png" alt="10" data-base62-sha1="zxV06FQseoTyuMrK4HZvvTYydrj" width="634" height="123"></p>

username: albanD
cooked: <p>Hi,</p>
<p>These values are not needed as is, only their product with <code>dL/dA</code> is needed. So these values are not always computed actually. It depends on the function f and sometimes the product is computed without ever creating this matrix explicitly.</p>

username: Yaroslav_Bulatov
cooked: <p>These values are not needed for reverse AD which is what PyTorch uses. It instead computes values like dloss/dA2 as intermediates, which you can access using <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.register_hook" rel="noopener nofollow ugc">tensor hook</a> on A2.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/267e44a7f208090642948944e80d80e71a3c5408.png" data-download-href="https://discuss.pytorch.org/uploads/default/267e44a7f208090642948944e80d80e71a3c5408" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267e44a7f208090642948944e80d80e71a3c5408_2_690x417.png" alt="image" data-base62-sha1="5uwFG1bGrZ1tWsWPt5PALJYaPvG" width="690" height="417" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267e44a7f208090642948944e80d80e71a3c5408_2_690x417.png, https://discuss.pytorch.org/uploads/default/original/3X/2/6/267e44a7f208090642948944e80d80e71a3c5408.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/6/267e44a7f208090642948944e80d80e71a3c5408.png 2x" data-dominant-color="CECECE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">884×535 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: sawal86
cooked: <p>Thank you for your reply. Please correct me if I am wrong. Based on my understanding of backprop, local gradients are needed to compute the gradient that is used for performing the updates. For example<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e5f87b861c2ead002fbcce256c53c0b91b72e85.png" alt="22" data-base62-sha1="239m8k54q8eMHLiv2u4LrivPh3L" width="635" height="194"></p>

username: sawal86
cooked: <p>Thanks for your reply. However, as far as I know, local gradients are necessary to compute the gradient needed for performing the updates.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e5f87b861c2ead002fbcce256c53c0b91b72e85.png" alt="22" data-base62-sha1="239m8k54q8eMHLiv2u4LrivPh3L" width="635" height="194"></p>

username: albanD
cooked: <p>As <a class="mention" href="/u/yaroslav_bulatov">@Yaroslav_Bulatov</a> mentioned, the backward function computes, given a <code>dl/dA</code>, <code>dl/dW</code>.<br>
The formula tells you that <code>dl/dW = dl/dA * dA/dW</code> but sometimes <code>dA/dW</code> has a structure such that it is not needed to actually compute it.</p>
<p>For the sum operation for example, if <code>A = W.sum()</code> (assuming W being 1D), then <code>dA/dW = 1</code> (a 1D vector full of ones). In such case, <code>dl/dW</code> can be computed by simply expanding <code>dl/dA</code> to the size of <code>W</code>. And <code>dA/dW</code> is never needed as a matrix to compute gradients.</p>

username: sawal86
cooked: <p>But what about the cases where dA/dW is needed to be computed explicitly?</p>

username: albanD
cooked: <p>In these cases it is computed as a temporary variable and deleted before exiting the function to reduce memory consumption.</p>

username: sawal86
cooked: <p>I see. Is there anyway to access that temporary variable?</p>

username: albanD
cooked: <p>You can call <code>.retain_grad()</code> on it before the backward call. That way, its .grad field will be populated.</p>

username: llseek
cooked: <p>looks like .retain_grad() retains dl/dW instead of dA/dW ?</p>