Requires_grad propagation

username: Luigi
cooked: <p>Hi guys, as mentioned in <a href="https://discuss.pytorch.org/t/batch-gradient-descent-with-custom-model-tensor-operations/92289">this post</a> I have some tensor operations and a vector among them contains  the parameters I want to optimise.<br>
That said, I’m having troubles in propagating the <code>requires_grad</code> through these simple operations.<br>
Every time the <code>forward</code> is called <code>v</code> is computed as:</p>
<p><code> v = self.sym @ self.v_short + self.fixed</code></p>
<p>where <code>self.v_short</code> is a <code>torch.nn.Parameter()</code> vector and it’s exactly what I want to train, while <code>self.sym</code> and <code>self.fixed</code> are  <code>torch.tensor()</code>, respectively a constant matrix and a constant vector (hence, no gradient and training needed). As I was expecting, <code>self.sym</code> and <code>self.fixed</code> have <code>requires_grad=False</code>, while  <code>self.v_short</code> has <code>requires_grad=True</code>.<br>
However, for some reasons that I cannot figure out, <code>v</code> has <code>requires_grad=False</code>.</p>
<p>In the trying of figuring out what the problem was, I came across multiple posts -like <a href="https://stackoverflow.com/questions/44913720/requires-grad-relation-to-leaf-nodes" rel="noopener nofollow ugc">this one</a>- where was said that:</p>
<blockquote>
<p>if a leaf node  <code>requires_grad</code> , all subsequent nodes computed from it will automatically also  <code>require_grad</code></p>
</blockquote>
<p>which makes sense! And I’m wondering if you could explain why this is not happening for me?<br>
Thx in advance for helping!</p>

username: JuanFMontesinos
cooked: <p>Because a nn.Parameters is not a leaf node. The other ones are constants (buffers inside a nn.Module). In fact you don’t reall need to use a nn.Module. You can optimize a tensor directly.</p>
<pre><code class="lang-auto">import torch

cte1 = torch.rand(5, 3).requires_grad_(False)
cte2 = torch.rand(5, 5).requires_grad_(False)
tensor = torch.ones(3, 5).requires_grad_()

optim = torch.optim.SGD([tensor], lr=1)

print(f'Initial tensor'
      f'{tensor}')
for i in range(5):
    optim.zero_grad()
    print(f'Iteration {i}')
    output = cte1 @ tensor + cte2
    print(f'Requires grad? {output.requires_grad}')
    output.sum().backward()
    print(f'Tensor gradients \n'
              f' {tensor.grad}')
    optim.step()
    print(f'Tensors \n'
              f' {tensor}')
</code></pre>
<pre><code class="lang-auto">Initial tensortensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]], requires_grad=True)
Iteration 0
Requires grad? True
Tensor gradients 
 tensor([[1.4718, 1.4718, 1.4718, 1.4718, 1.4718],
        [1.7690, 1.7690, 1.7690, 1.7690, 1.7690],
        [2.1010, 2.1010, 2.1010, 2.1010, 2.1010]])
Tensors 
 tensor([[-0.4718, -0.4718, -0.4718, -0.4718, -0.4718],
        [-0.7690, -0.7690, -0.7690, -0.7690, -0.7690],
        [-1.1010, -1.1010, -1.1010, -1.1010, -1.1010]], requires_grad=True)
Iteration 1
Requires grad? True
Tensor gradients 
 tensor([[1.4718, 1.4718, 1.4718, 1.4718, 1.4718],
        [1.7690, 1.7690, 1.7690, 1.7690, 1.7690],
        [2.1010, 2.1010, 2.1010, 2.1010, 2.1010]])
Tensors 
 tensor([[-1.9435, -1.9435, -1.9435, -1.9435, -1.9435],
        [-2.5379, -2.5379, -2.5379, -2.5379, -2.5379],
        [-3.2019, -3.2019, -3.2019, -3.2019, -3.2019]], requires_grad=True)
Iteration 2
Requires grad? True
Tensor gradients 
 tensor([[1.4718, 1.4718, 1.4718, 1.4718, 1.4718],
        [1.7690, 1.7690, 1.7690, 1.7690, 1.7690],
        [2.1010, 2.1010, 2.1010, 2.1010, 2.1010]])
Tensors 
 tensor([[-3.4153, -3.4153, -3.4153, -3.4153, -3.4153],
        [-4.3069, -4.3069, -4.3069, -4.3069, -4.3069],
        [-5.3029, -5.3029, -5.3029, -5.3029, -5.3029]], requires_grad=True)
Iteration 3
Requires grad? True
Tensor gradients 
 tensor([[1.4718, 1.4718, 1.4718, 1.4718, 1.4718],
        [1.7690, 1.7690, 1.7690, 1.7690, 1.7690],
        [2.1010, 2.1010, 2.1010, 2.1010, 2.1010]])
Tensors 
 tensor([[-4.8871, -4.8871, -4.8871, -4.8871, -4.8871],
        [-6.0759, -6.0759, -6.0759, -6.0759, -6.0759],
        [-7.4039, -7.4039, -7.4039, -7.4039, -7.4039]], requires_grad=True)
Iteration 4
Requires grad? True
Tensor gradients 
 tensor([[1.4718, 1.4718, 1.4718, 1.4718, 1.4718],
        [1.7690, 1.7690, 1.7690, 1.7690, 1.7690],
        [2.1010, 2.1010, 2.1010, 2.1010, 2.1010]])
Tensors 
 tensor([[-6.3588, -6.3588, -6.3588, -6.3588, -6.3588],
        [-7.8448, -7.8448, -7.8448, -7.8448, -7.8448],
        [-9.5048, -9.5048, -9.5048, -9.5048, -9.5048]], requires_grad=True)

Process finished with exit code 0

</code></pre>

username: Luigi
cooked: <p>Hey Juan, thx for replying!<br>
You gave me a very good insight,  I just checked whether <code>self.v_short</code> was a leaf node and it is! <img src="https://discuss.pytorch.org/images/emoji/apple/grimacing.png?v=12" title=":grimacing:" class="emoji" alt=":grimacing:" loading="lazy" width="20" height="20"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb248d4a552017dd4bfe47231e93fb81d6d377ae.png" alt="image" data-base62-sha1="xyaw5W21bVALACAUwHOMns5bxAW" width="243" height="35"></p>
<p>However, <code>requires_grad</code> for <code>v</code> is still <code>False</code>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0fee07bdbcd9b1f376004c01b35ce0819b7b909.png" alt="image" data-base62-sha1="w6p1wyAly8tKt8V8LUPsbnppzIJ" width="318" height="72"></p>
<p>Unfortunately it has to be a <code>nn.Module</code> because it’s part of a bigger framework that I’m modifying.</p>
<p>So, I’m stuck again and I don’t know what is wrong! <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>