Confusion about https://pytorch.org/docs/stable/autograd.html

milan_kalkenings: <p>I think I found a mistake on <a href="https://pytorch.org/docs/stable/autograd.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.1 documentation</a>.<br>
The explanations of <code>torch.Tensor.is_leaf</code> &amp; <code>torch.Tensor.detach_</code> imply that a <code>Tensor</code> with <code>requires_grad = False</code> is a leaf, but exactly the opposite is the case, since the gradient is calculated w.r.t. all leaves. Am I wrong?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64ad33784eab9a673838bbe66effd207862e7b3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/64ad33784eab9a673838bbe66effd207862e7b3c" title="err"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64ad33784eab9a673838bbe66effd207862e7b3c.png" alt="err" data-base62-sha1="emCPn6QRhFkRBsfVsIu4l6sF5Ok" width="690" height="312" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">err</span><span class="informations">889×403 20.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Kind Regards</p>

srishti-git1110: <p>Hi Milan,<br>
A simple explanation -</p>
<p>A leaf tensor is any tensor on which no operations <em>(that autograd can track)</em> have been performed.<br>
When a tensor is first initialized, it becomes a leaf node/tensor.<br>
Basically, all inputs and weights of a neural network are leaf tensors and hence leaf nodes in the computational graph.<br>
Whenever <em>any</em> operation (that autograd can track - this is important) is performed on a leaf tensor, it does not remain a leaf anymore.</p>
<p>See:</p>
<pre><code class="lang-auto">x = torch.tensor([1.0, 2.0])
print(x.is_leaf) # True
print(x.requires_grad) # False

y = (x*2).sum()
print(y.is_leaf) # True
print(y.requires_grad) # False
</code></pre>
<p>Although y is the result of some operations on x, y is still a leaf as <strong>autograd isn’t tracking anything.</strong></p>
<aside class="quote no-group" data-username="milan_kalkenings" data-post="1" data-topic="162841">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/milan_kalkenings/48/42748_2.png" class="avatar"> milan_kalkenings:</div>
<blockquote>
<p>a <code>Tensor</code> with <code>requires_grad = False</code> is a leaf, but exactly the opposite is the case,</p>
</blockquote>
</aside>
<p>Not necessarily. It has less to do with the requires_grad attribute and more to do with what autograd tracks.<br>
As you saw above, <em>exactly the opposite</em> is not always the case. A tensor with <code>requires_grad = False</code> can be a leaf.</p>
<p><code>detach</code> simple detaches a tensor from its current computation graph and returns a new tensor which is a leaf and has its <code>requires_grad = False</code>.</p>
<p><code>detach_</code> is just that but does it in-place without returning a new tensor.</p>
<p>Does this answer your question?</p>

milan_kalkenings: <p>Thank you for clarification!<br>
I have a follow-up question:<br>
why is it, that</p>
<pre><code class="lang-auto">a = torch.tensor([1., 2.], requires_grad=True)
b = 2 * a
c = a.sum()
d = (2*a).sum()
print(a.is_leaf)  # True
print(b.is_leaf)  # False
print(c.is_leaf)  # False
print(d.is_leaf)  # False

# ,but
a = torch.tensor([1., 2.], requires_grad=False)
b = 2 * a
c = a.sum()
d = (2*a).sum()
print(a.is_leaf)  # True
print(b.is_leaf)  # True
print(c.is_leaf)  # True
print(d.is_leaf)  # True
</code></pre>

srishti-git1110: <p>In the first snippet, since <code>a</code> has <code>requires_grad=True</code>, the autograd engine is now tracking every operation involving <code>a</code>.</p>
<p>Hence, all calculations involving <code>b, c, d</code> are being tracked which is why they aren’t leaves any more. See:</p>
<aside class="quote no-group" data-username="srishti-git1110" data-post="2" data-topic="162841">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<p>Whenever <em>any</em> operation (that autograd can track - this is important) is performed on a leaf tensor, it does not remain a leaf anymore.</p>
</blockquote>
</aside>
<p>For the second code, a’s <code>requires_grad=False</code> and autograd isn’t tracking, so all <code>b, c, d</code> are leaves - this is conceptually exactly what the code sample in my previous reply is.</p>
<p>Let me know if you have further questions,<br>
Srishti</p>

milan_kalkenings: <p>Thanks, you helped me a lot!</p>