A question about F.conv2d

BarCodeReader: <p>Below is a simple reproducible code snippet:<br>
I know this might be a simple question, but I never thought of it before and just want to confirm here.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.conv = nn.Conv2d(1,1,3, padding=1, bias=False)
        self.weight1 = self.conv.weight * 10
        
    def forward(self, x):

        out = self.conv(x)
        out2 = F.conv2d(out, self.weight1, padding=1)

        return out2
</code></pre>
<p>then during back-propagation, I noticed that <code>self.weight1</code> will not be updated.</p>
<pre><code class="lang-auto">net = Net()
# -&gt; checkpoint 1: print(net.conv1.weight, net.weight1)
optimizer1 = optim.SGD(net.parameters(), lr=0.001)
Loss1 = nn.L1Loss()

k = torch.ones((1,1,5,5))
gt = torch.zeros((1,1,5,5))

# back propagation
optimizer1.zero_grad()

pred = net(k)
loss1 = Loss1(gt, pred)

loss1.backward()
optimizer1.step()
# -&gt; checkpoint 2: print(net.conv1.weight, net.weight1)
</code></pre>
<p>for the above 2 checkpoints, the outputs are below:</p>
<pre><code class="lang-auto"># checkpoint 1: print(net.conv1.weight, net.weight1)
Parameter containing:
tensor([[[[ 0.1280,  0.1039, -0.1972],
          [-0.2293,  0.1194, -0.1370],
          [-0.3182, -0.1437,  0.0395]]]], requires_grad=True) 
tensor([[[[ 1.2796,  1.0391, -1.9720],
          [-2.2928,  1.1938, -1.3697],
          [-3.1816, -1.4372,  0.3954]]]], grad_fn=&lt;MulBackward0&gt;)

# checkpoint 2: print(net.conv1.weight, net.weight1)
Parameter containing:
tensor([[[[ 0.1337,  0.1111, -0.1900],
          [-0.2223,  0.1278, -0.1290],
          [-0.3123, -0.1371,  0.0456]]]], requires_grad=True) 
tensor([[[[ 1.2796,  1.0391, -1.9720],
          [-2.2928,  1.1938, -1.3697],
          [-3.1816, -1.4372,  0.3954]]]], grad_fn=&lt;MulBackward0&gt;)
</code></pre>
<p>you can see <code>self.weight1</code> in network does not get updated. but <code>self.weight1</code> is a <code>torch.tensor</code> and it has a gradient, i.e. <code>net.weight1.requires_grad</code> is True. <strong>This is because</strong> <code>self.weight1</code> <strong>is a non-leaf tensor.</strong></p>
<p>This is not my confusion, my question is, since <code>self.weight1</code> is not get updated, does that mean, the gradients from loss directly propagate to <code>self.conv,</code> because the layer <code>F.conv</code> is not learning(i.e. because <code>F.conv</code> is not part of the graph)?</p>
<p>Really appreciate if anyone can give me some hints.</p>

albanD: <p>Hi,</p>
<p>To make sure that a Module attribute is recognized as a learnable parameter, it needs to be wrapped in the <code>nn.Parameter</code>.<br>
As you code is today, <code>weight1</code> won’t actually appear in <code>net.parameters()</code> and so the optimizer won’t know about it.</p>
<p>Also for the <code>.grad</code> field of a Tensor to be populated, you want to make sure that it is a leaf Tensor (it has no history). Right now, because <code>weight1</code> is computed based on <code>conv.weight</code>, it is not a leaf (you can check .is_leaf()).<br>
You can detach it from its history to force it to be a leaf. So the updated code is:<br>
<code>self.weight1 = nn.Parameter((self.conv.weight * 10).detach())</code></p>

BarCodeReader: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Thanks for your reply! Its really clear and helpful.<br>
Yes I understand that we need to use <code>nn.Parameter</code> to wrap the weight.<br>
<em><strong>But here I want to know a bit more about what will happen to gradient flow if we intentionally let it be a non-leaf component.</strong></em></p>
<p>So for the code I shared above, does it mean that the gradient from Loss will directly flow to <code>self.conv1</code> and <code>F.conv2d</code> there is just like an operator outside the network(since its non-leaf)?</p>
<p>And in general, can I assert like below?<br>
<strong>for connection in series:</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66000c40f737c01dd17434974b86ebf2430a405c.png" alt="st1" data-base62-sha1="eykNSE8IEixlxDWcs5s2GebB93S" width="372" height="308"><br>
<strong>for connection in parallel(torch.cat()):</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86c56505ea14444d9a865c67da07201eec3df023.png" alt="st2" data-base62-sha1="jeeTCk0u0fKBOOAbwXiK2puub6P" width="341" height="362"></p>
<p>Really thanks for your time and help!</p>

albanD: <p>Hi,</p>
<p>If you leave it as your first example, when you backprop the <code>F.conv2d</code>, no gradient get accumulated since no input is a leaf.<br>
But then you backprop through the computation of <code>self.weight1</code> and that will add gradients to <code>self.conv.weight</code>.<br>
In parallel, you backprop through <code>self.conv</code> which will also accumulate gradients into <code>self.conv.weight</code>.<br>
So in the end, the gradients in <code>self.conv.weight</code> is just the sum of the gradients of all the places where it is used.</p>

BarCodeReader: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Thanks for the answer! Appreciate.</p>