What is the purpose of `is_leaf`?

Redcxx: <blockquote>
<p>All Tensors that have <a href="https://pytorch.org/docs/master/autograd.html#torch.Tensor.requires_grad" rel="noopener nofollow ugc"> <code>requires_grad</code> </a> which is  <code>False</code>  will be leaf Tensors by convention.</p>
</blockquote>
<blockquote>
<p>For Tensors that have <a href="https://pytorch.org/docs/master/autograd.html#torch.Tensor.requires_grad" rel="noopener nofollow ugc"> <code>requires_grad</code> </a> which is  <code>True</code> , they will be leaf Tensors if they were created by the user. This means that they are not the result of an operation and so  <code>grad_fn</code>  is None.</p>
</blockquote>
<blockquote>
<p>Only leaf Tensors will have their <a href="https://pytorch.org/docs/master/autograd.html#torch.Tensor.grad" rel="noopener nofollow ugc"> <code>grad</code> </a> populated during a call to <a href="https://pytorch.org/docs/master/autograd.html#torch.Tensor.backward" rel="noopener nofollow ugc"> <code>backward()</code> </a>. To get <a href="https://pytorch.org/docs/master/autograd.html#torch.Tensor.grad" rel="noopener nofollow ugc"> <code>grad</code> </a> populated for non-leaf Tensors, you can use <a href="https://pytorch.org/docs/master/autograd.html#torch.Tensor.retain_grad" rel="noopener nofollow ugc"> <code>retain_grad()</code> </a>.</p>
</blockquote>
<p>Hi I just started learning about autograd and I have some confusion about the PyTorch docs above, it says that if <code>requires_grad</code> is <code>False</code> then it is a leaf tensor and only leaf tensor will have their <code>grad</code> populated during the call to <code>backward</code>, but why a tensor’s <code>grad</code> is populated when it does not require gradient? What is the purpose of the <code>is_leaf</code> attribute?</p>

Nikronic: <p>Hi,</p>
<p>Actually, I prefer to link a blog that explained this idea very interesting rather than explaining myself as it will be based on this.<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://blog.paperspace.com/favicon.png" class="site-icon" width="16" height="16">
      <a href="https://blog.paperspace.com/pytorch-101-understanding-graphs-and-automatic-differentiation/" target="_blank" rel="nofollow noopener" title="08:03PM - 29 May 2019">Paperspace Blog – 29 May 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:116/74;"><img src="https://blog.paperspace.com/content/images/2019/05/full_graph.png" class="thumbnail" width="116" height="74"></div>

<h3><a href="https://blog.paperspace.com/pytorch-101-understanding-graphs-and-automatic-differentiation/" target="_blank" rel="nofollow noopener">PyTorch Basics: Understanding Autograd and Computation Graphs</a></h3>

<p>In this article, we dive into how PyTorch's Autograd engine performs automatic differentiation.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Bests</p>

Redcxx: <p>Hi,<br>
Thanks for the great link to the blog, I have read it and gained lots of insights about how autograd works but the article only explains the definition of leaf node but does not explain why it is needed, perhaps the author think it should be intuitive <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">. It also does not answer some questions I had, such as:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; b = torch.rand(10, requires_grad=True).cuda()
&gt;&gt;&gt; b.is_leaf
False
# b was created by the operation that cast a cpu Tensor into a cuda Tensor
&gt;&gt;&gt; f = torch.rand(10, requires_grad=True, device="cuda")
&gt;&gt;&gt; f.is_leaf
True
# f requires grad, has no operation creating it
</code></pre>
<p>Does the above mean that if we call <code>cuda()</code> on a tensor then we have to use <code>retain_grad=True</code> so that the <code>grad</code> attribute will be populated?</p>

Nikronic: <p>We need graph leaves to be able to compute gradients of final tensor w.r.t. them. Leaf nodes are not functions or simply put, have not been obtained from mathematical operations. For instance, in a nn.Linear(in, out) module, weight and bias are leaf nodes so when you call <code>.backward</code> on a loss function that uses this linear layer, gradient of loss function will be calculated w.r.t. the weights and bias. In the other words, all parameters of layers are leaf nodes.</p>
<aside class="quote no-group" data-username="Redcxx" data-post="3" data-topic="87000">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/redcxx/48/19316_2.png" class="avatar"> Redcxx:</div>
<blockquote>
<p><code>&gt;&gt;&gt; b = torch.rand(10, requires_grad=True).cuda()</code></p>
</blockquote>
</aside>
<p>Because <code>.cuda()</code> is an operation, then yes, you have use <code>retain_grad=True</code>.</p>
<p>Here is an example that might help:</p>
<pre><code class="lang-auto">conv = nn.Conv2d(1, 3, 2)  # its params requires grad, we created it so it is a leaf
print(conv.weight.requires_grad)  # true but its grad is None as it is a leaf
print(conv.weight.is_leaf)  # true
z= conv(torch.randn(1, 1, 10, 10)).sum()  # output of sum math operation so is not leaf but requires grad
print(z.is_leaf)  # false  
z.backward()
print(conv.weight.grad_fn)  # none
print(z.grad_fn)  # &lt;SumBackward0 object&gt;
</code></pre>
<p>Although, after reading few posts, some questions raised for me. I think we need to view some presentations about it (let’s google it!)</p>
<p>References:</p>
<ol>
<li><a href="https://discuss.pytorch.org/t/how-do-i-calculate-the-gradients-of-a-non-leaf-variable-w-r-t-to-a-loss-function/5112" class="inline-onebox">How do I calculate the gradients of a non-leaf variable w.r.t to a loss function?</a></li>
<li><a href="https://discuss.pytorch.org/t/what-are-means-of-leaf-variable-and-accumulated-gradient/12874" class="inline-onebox">What are means of leaf variable and accumulated gradient?</a></li>
</ol>

Redcxx: <p>Thanks for the explanation! They are pretty clear but I still have some questions about it because the result is kind of weird to me:</p>
<pre><code class="lang-auto"># scenario 1
&gt;&gt;&gt; a = torch.rand(3,2,requires_grad=True)
&gt;&gt;&gt; b = torch.rand(2,3)
&gt;&gt;&gt; loss = 10 - (a @ b).sum()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; a.grad  
tensor([[-0.9418, -1.4957],
        [-0.9418, -1.4957],
        [-0.9418, -1.4957]])  # expected, has gradient
&gt;&gt;&gt; b = torch.rand(2,3)   # expected, no gradient
&gt;&gt;&gt;
 
...
# scenario 2
&gt;&gt;&gt; a = torch.rand(3,2,requires_grad=True).cuda()
&gt;&gt;&gt; b = torch.rand(2,3).cuda()
&gt;&gt;&gt; loss = 10 - (a @ b).sum()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; a.grad   # expected since operation `cuda` makes it a non-leaf, but why?
&gt;&gt;&gt; b.grad  # expected, no gradient
&gt;&gt;&gt;

...
# scenario 3
&gt;&gt;&gt; ll = nn.Linear(3,3).cuda()
&gt;&gt;&gt; inp = torch.rand(3).cuda()
&gt;&gt;&gt; loss = (10 - ll(inp)).sum()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; ll.weight.grad
tensor([[-0.8497, -0.1128, -0.8081],
        [-0.8497, -0.1128, -0.8081],
        [-0.8497, -0.1128, -0.8081]], device='cuda:0')  # why it has gradient?

</code></pre>
<p>As you see in scenario 2, by moving the tensor cuda I was not able to get the gradient without using <code>a.retain_grad()</code> or <code>device='cuda'</code>. In scenario 3 moving the layer to cuda the gradient of the weight is still there. May I know why it is designed this way? I know that result of an operation like multiplication does not requires its gradient to be calculated since intermidate values’s gradient usually not useful, but why does operation like <code>cuda</code> also cause the tensor to be non-leaf but stay as leaf for weight in layer?</p>

Nikronic: <p>About scenario 1, we are ok right?<br>
But about case 2, I don’t know why it does not accumulate gradients even it computes for it. In docs, literally says <em>The fact that gradients need to be computed for a Tensor do not mean that the <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.grad" rel="nofollow noopener"> <code>grad</code> </a> attribute will be populated, see <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.is_leaf" rel="nofollow noopener"> <code>is_leaf</code> </a> for more details.</em> <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.requires_grad" rel="nofollow noopener">(source)</a></p>
<p>But about case 3, logically, if we want to update weight and bias, we need to corresponding gradients, but about code, I am not sure because we can trick scenario 2 to be similar to scenario 3 using defining tensors directly into GPU instead of copying from CPU, in this way, case 2 will be a leaf and will have grad. Also, weight and bias n nn.Linear are parameters a specific case of tensors, that is, maybe that is the reason that enables this case.</p>
<pre><code class="lang-auto">a = torch.rand(3,2,requires_grad=True, device='cuda:0')
b = torch.rand(2,3).cuda()
loss = 10 - (a @ b).sum()
loss.backward()
print(a.is_leaf)
a.grad

# output
True
tensor([[-1.4735, -1.9717],
        [-1.4735, -1.9717],
        [-1.4735, -1.9717]], device='cuda:0')
</code></pre>
<p>I really need to study a few things, sorry for my lack of knowledge!</p>

Redcxx: <p>Thank you for the explanation! You really helped me alot!</p>

Nikronic: <p>Great, actually, after this I doubted about many things! I need to study few things to get a clear idea of what is happening.<br>
Good luck</p>

Ursula_Iguaran: <p>The docs don’t really indicate what is going on with is_leaf. In particular, I think the sentence that says “Only leaf Tensors will have their grad populated…” is misleading. From what I can guess, leaves don’t really have to do with populating grad; requires_grad is what governs that.</p>
<p>I think the is_leaf property is really about the reverse-graph. When x.backward() is called, all of the action happens on the “reverse-differentation mode” graph at x. x is the root, and the graph runs up along (against arrows in) the forward graph from x. While only tensors with requires_grad = True appear in the reverse-graph, the graph_fn of every tensor visited (even those with requires_grad = False) is used to produce the maps corresponding to the reverse-graph’s arrows. When the process hits a non-leaf, it knows it can keep mapping along to more nodes. On the other hand, when the process hits a leaf, it knows to stop; leaves have no graph_fn.</p>
<p>If this is right, it makes it more clear why weights are “leaves with requires_grad = True”, and inputs are “leaves with requires_grad = False.” You could even take this as a definition of “weights” and “inputs”.</p>

Carlos_Segura: <p>Based on PyTorch’s design philosophy, <code>is_leaf</code> is not explained because it’s not expected to be used by the user unless you have a specific problem that requires knowing if a variable (when using autograd) was created by the user or not.</p>
<blockquote>
<p><em>“If there’s a single input to an operation that requires gradient, its output will also require gradient. Conversely, only if all inputs don’t require gradient, the output also won’t require it. Backward computation is never performed in the subgraphs, where all Tensors didn’t require gradients.”</em> – <a href="https://pytorch.org/docs/stable/notes/autograd.html#:~:text=Every%20Tensor%20has%20a%20flag,computation%20and%20can%20increase%20efficiency." rel="noopener nofollow ugc">Autograd mechanics — PyTorch 1.8.1 documentation</a></p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">a = torch.randn(2, 4)
b = torch.randn(4, 2)
c = a.mm(b)
tuple(map(lambda t: t.is_leaf, (a, b, c)))
...
#  (True, True, True)
</code></pre>
<p>Here <em><strong>c</strong></em> is <em>false</em> (not a leaf) because it’s a tensor not “created” directly by the user (meaning it’s the result of <em><strong>a</strong></em> and <em><strong>b</strong></em>) <strong>and</strong> <code>requires_grad</code> is <em>true</em>.</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.randn(2, 4).requires_grad_()
b = torch.randn(4, 2)
c = a.mm(b)
tuple(map(lambda t: t.is_leaf, (a, b, c)))
...
#  (True, True, False)
</code></pre>

Akshay_tiwari1: <p>I think all your questions have been answered by others except for -<br>
<code>" but why a tensor’s grad is populated when it does not require gradient?"</code></p>
<p>I assume that you mean<br>
<code>why would a tensor’s </code>grad<code> be populated when it's requires_grad() is False?</code></p>
<p>The answer is that a tensor’s <code>grad</code> is not populated when it’s requires_grad() is False.</p>
<blockquote>
<p>x = torch.tensor(1.0, requires_grad = True)<br>
y = torch.tensor(2.0)<br>
z = x * y</p>
<p>w = torch.tensor(3.0).requires_grad_(False)<br>
o = z*w</p>
<p>o.backward()</p>
<p>for i, name in zip([x, y, z, w], “xyzw”):<br>
print(f"{name}\ndata: {i.data}\nrequires_grad: {i.requires_grad}\ngrad: {i.grad}\ngrad_fn: {i.grad_fn}\nis_leaf: {i.is_leaf}\n")</p>
</blockquote>
<p>In the above example you would see that w has <code>is_leaf True</code> but it’s <code>requires_grad is False</code> as a result its <code>grad is None</code> after applying the <code>.backward()</code>.<br>
Which means that requires_grad is more powerful than is_leaf and the gradients are not calculated w.r.t w even if <code>is_leaf is True</code>, because <code>requires_grad is False.</code></p>

offchan: <p>This is how I intuitively understand leaf tensors:<br>
<strong>Leaf tensors are tensors that stop the flow of gradients on the backward pass. They can store gradients themselves. But they will not allow gradients to flow back through them to other tensors.</strong><br>
They are usually parameters and input tensors of the network. But they can also be tensors that are made from operations as long as they do not allow gradients to flow through them.</p>
<p>For example, if you have tensor A and B that are made from torch.ones(), they will be leaf tensors because they cannot let gradients to flow through them even if you set requires_grad to True. That’s because there are no other predecessor tensors to flow into. There are only A and B.</p>
<p>If you do C=A*B, then C will be leaf tensor if it’s not allowing gradients to flow to A and B.<br>
C will not be leaf tensor if it’s allowing gradients to flow to A or B.<br>
How can C allow gradients to flow to A or B? Just set either A or B to requires_grad=True.<br>
If you set A.requires_grad = True then C will have to flow the gradients to A, making C not a leaf tensor anymore.</p>
<p>What’s the implication or use case of this knowledge?<br>
When you see a leaf tensor, you will know that gradients will not flow through it. Backward pass stops right at this tensor. So GPU usage will be low.</p>