How to get the gradients for both the input and intermediate variables?

oat: <p>Assuming the following forward pass: x → y → z, in which<br>
x is a scalar<br>
y = x * x<br>
z = 2 * y</p>
<p>I want to track the following derivatives/gradients, including the gradient for the intermediate function y:</p>
<ul>
<li>dz/dy, gradient of z with respect to y, which should be “2”</li>
<li>dy/dx, gradient of y with respect to x, which should be “2x”</li>
<li>dz/dx, gradient of z with respect to x, which should be “4x”</li>
</ul>
<p>So, I initiated both x and y with the <strong><code>requires_grad=True</code></strong> argument. However, I can only get <strong><code>y.grad</code></strong>, which is “2”, and <strong><code>x.grad</code></strong> returned “None”, as shown below.</p>
<p>May I ask:</p>
<ul>
<li>Why I’m unable to get x.grad (dz/dx) in this case?</li>
<li>How to get the gradients for both the input and intermediate variables via .backward()?</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dcb130dc2b59c5a59496077ec12a009d73e1482.png" data-download-href="https://discuss.pytorch.org/uploads/default/9dcb130dc2b59c5a59496077ec12a009d73e1482" title="Screenshot 2020-12-10 at 13.58.04"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dcb130dc2b59c5a59496077ec12a009d73e1482_2_344x499.png" alt="Screenshot 2020-12-10 at 13.58.04" data-base62-sha1="mvU2XOeBJ7XBGygH6r0QY9Ec8i6" width="344" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dcb130dc2b59c5a59496077ec12a009d73e1482_2_344x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dcb130dc2b59c5a59496077ec12a009d73e1482_2_516x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dcb130dc2b59c5a59496077ec12a009d73e1482.png 2x" data-dominant-color="071B29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-12-10 at 13.58.04</span><span class="informations">669×970 49.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="oat" data-post="1" data-topic="105730">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oat/48/34409_2.png" class="avatar"> oat:</div>
<blockquote>
<p>Why I’m unable to get x.grad (dz/dx) in this case?</p>
</blockquote>
</aside>
<p>You are detaching the computation graph by recreating a tensor in:</p>
<pre data-code-wrap="python"><code class="lang-python"> y= torch.tensor(x * x)
</code></pre>
<p>use <code>y = x*x</code> instead.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

oat: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>However, it seems <strong><code>y.grad</code></strong> is not tracked using <code>y = x * x</code>:</p>
<pre><code class="lang-auto">x = torch.tensor(0.3, requires_grad=True)
print(x)
# [output] tensor(0.3000, requires_grad=True)

y = x * x
print(y)
# [output] tensor(0.0900, grad_fn=&lt;MulBackward0&gt;)

z = 2 * y
print(z)
# [output] tensor(0.1800, grad_fn=&lt;MulBackward0&gt;)

z.backward()

print(y.grad)
# [output] None

print(x.grad)
# [output] tensor(1.2000)
</code></pre>
<p>Anyway, I found <a href="https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94">this post</a> that’s relevant to my question, and I’ll digest it first.</p>

ptrblck: <p>Your code should raise a warning and explain the reason and workaround for it:</p>
<pre><code class="lang-python">UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  print(y.grad)
</code></pre>
<p>After adding <code>y.retain_grad()</code> you’ll get the gradient value for <code>y</code>.</p>

oat: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>I’m doing the test in a Jupyter notebook via vscode. I’m not sure why I don’t get the warning message you pointed out.</p>
<p>Anyway, adding <strong><code>y.retain_grad()</code></strong> does work in this case as you suggested:</p>
<pre><code class="lang-auto">x = torch.tensor(0.3, requires_grad=True)
print(x)
# [output] tensor(0.3000, requires_grad=True)

y = x * x
print(y)
# [output] tensor(0.0900, grad_fn=&lt;MulBackward0&gt;)

y.retain_grad()

z = 2 * y
print(z)
# [output] tensor(0.1800, grad_fn=&lt;MulBackward0&gt;)

z.backward()

print(y.grad)
# [output] tensor(2.)

print(x.grad)
# [output] tensor(1.2000)
</code></pre>