Track .grad gradient graph

username: Meraki
cooked: <p>Maybe the title is confusing. Here is the small example:</p>
<pre><code class="lang-auto">w = torch.tensor([3.2], requires_grad=True)
p = torch.tensor([2.0, 1, 7], requires_grad=True)
g = torch.sum(p ** 2)
e = w * g
e.backward(retain_graph=True)
f = p.grad

l = e + 0.5 * f.mean()
l.backward()
</code></pre>
<p>here f is the funciton of w, however we loss gradient graph of f ( only tensor), we know f=2pw actually. when we do backward() for l, we want f contributes to gradient of w (df/dw = 2p). How to make it here?</p>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You want to do <code>e.backward(create_graph=True)</code> that will make sure the backward pass run in a differentiable manner.</p>
<p>Also creating a .grad that requires grad is usually not recommended because when you do <code>l.backward()</code> here, you will accumulate in the same <code>p.grad</code> which can easily become confusing. You can do the following to make it simpler:</p>
<pre><code class="lang-auto">w = torch.tensor([3.2], requires_grad=True)
p = torch.tensor([2.0, 1, 7], requires_grad=True)
g = torch.sum(p ** 2)
e = w * g
f = autograd.grad(e, p, create_graph=True)[0]

l = e + 0.5 * f.mean()
l.backward()
</code></pre>

username: Meraki
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d6034f7c2f3d5d34e65c175705424cb6263d10c.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d6034f7c2f3d5d34e65c175705424cb6263d10c" title="Screenshot 2020-06-25 at 22.41.49"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d6034f7c2f3d5d34e65c175705424cb6263d10c_2_406x500.png" alt="Screenshot 2020-06-25 at 22.41.49" data-base62-sha1="8KXfntkh2nU5D6SBtAMHNCi6x2s" width="406" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d6034f7c2f3d5d34e65c175705424cb6263d10c_2_406x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d6034f7c2f3d5d34e65c175705424cb6263d10c_2_609x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d6034f7c2f3d5d34e65c175705424cb6263d10c_2_812x1000.png 2x" data-dominant-color="EFF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-06-25 at 22.41.49</span><span class="informations">1426×1752 186 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi, albanD! Thanks for your help. However I am so confused about the second derivative of f with respect to w ( the most right part of the graph). What is expandbackward here? what about the label 0 and 1 after backward? Are Mulbackward0 the same for both path?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes these graphs are not very easy to read usually.<br>
These come from the graph created inside the backward. The 0/1 values here are used to differentiate the different overloads of a function (mult(t1, t1) vs mul(t1, 2) for example).</p>

username: Meraki
cooked: <p>Yep. Could you please help me explain the right most backward flow step by step for this example? like MeanBackward0 corresponding to f.mean(). I just fell into the trouble to find 1 to 1 correspondence.</p>

username: albanD
cooked: <p>Well the problem is that the 1-1 correspondence is wrt to the ops executed during the backward pass. So not the ops you wrote.</p>
<p>For example, the ExpandBackward comes from the backward formula of sum that uses expand.<br>
MulBackward1 and the PowBackward on the right side come from the backward pass of the power function that depends on its input.</p>

username: Meraki
cooked: <p>I see. Some backward are not intuitive but reasonable in pytorch records. Thanks for your help.</p>

username: BruceDai003
cooked: <p>How did you generate this graph?</p>

username: albanD
cooked: <p>You can find this tool here: <a href="https://github.com/szagoruyko/pytorchviz">https://github.com/szagoruyko/pytorchviz</a></p>

username: BruceDai003
cooked: <p>Save my day by looking into the source code. Much appreciated.</p>