Aren't weights of models non_leaves. So how do they have their grad values?

username: SiddharthSingi
cooked: <p>I know its obvious why weights need to have gradient values, but I have a particular question about leaves and non leaves. Please read on, this question is not as long as it seems</p>
<p>So I understand that the grad is not available for any intermediate node, only available for leaf nodes (However the reasoning is still not perfectly clear to me like <a href="https://discuss.pytorch.org/t/what-is-the-purpose-of-is-leaf/87000/6">here</a>)</p>
<p>Example of no grad value for intermediate node: (feel free to skip directly to the question below)</p>
<pre><code class="lang-auto">a = torch.rand(10, requires_grad=True)
b = torch.rand(10)
c  = a + b

d = torch.tensor(5.0)
e = c*d

e.mean().backward()
print(a.grad)
print(c.grad)
</code></pre>
<p>Here a.grad should give me output since it is a leaf node, however c.grad is an intermediate node and shouldnt give me any output, which is exactly what we get:</p>
<p>Output:</p>
<pre><code class="lang-auto">tensor([0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000,
        0.5000])
C:\Anaconda\envs\robot\lib\site-packages\torch\tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by 
mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "
None
</code></pre>
<p>.</p>
<p><strong>MY QUESTION</strong> is: In any model that we create, with multiple multiple layers, arent the weights of each layers non_leaves, hence how come they have grad values but other non_leaves do not.</p>
<p>For example:</p>
<pre><code class="lang-auto">class ActorNet(nn.Module):
    def __init__(self, obs_space, action_space):
        super().__init__()
        self.fc1 = nn.Linear(obs_space, 32)
        self.fc2 = nn.Linear(32, action_space)        

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x
</code></pre>
<p>Here isnt the layer fc2 a non leaf. However its weights will still definitely have a grad value.</p>

username: KFrank
cooked: <p>Hi Siddharth!</p>
<aside class="quote no-group" data-username="SiddharthSingi" data-post="1" data-topic="100410" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddharthsingi/48/26161_2.png" class="avatar"> SiddharthSingi:</div>
<blockquote>
<pre><code class="lang-auto">    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x
</code></pre>
<p>Here isnt the layer fc2 a non leaf.</p>
</blockquote>
</aside>
<p>On the contrary, <code>fc2.weight</code> (and <code>fc2.bias</code>) are leaf tensors.</p>
<p>In the forward pass the values of <code>fc2.weight</code> do not depend on<br>
the values of <code>x</code> (nor on the values in <code>fc1</code>).  They only change (in<br>
the conventional use case) when you call <code>optimizer.step()</code>.</p>
<p>Are you imagining that only the first layer is a leaf, that is that in:</p>
<pre data-code-wrap="python"><code class="lang-python">    def forward(self, x):
        # x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x
</code></pre>
<p><code>fc2</code> becomes a leaf <em>because</em> we commented out <code>fc1</code>?</p>
<p>To reiterate, the values of <code>x</code> of course depend upon whether <code>fc1</code><br>
was called, but the values of <code>fc2.weight</code> do not.</p>
<p>Best.</p>
<p>K. Frank</p>

username: SiddharthSingi
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Thanks for the reply. It’s now such an obvious reason as to why the weights are leaf tensors and not non leaves. I was imagining the graph completely wrong in my head. Thanks for helping out.</p>