Learnable scalar weight in PyTorch and guarantee the sum of scalars is 1

sudonto: <p>Hello, maybe I miss already-post discussion regarding this title.<br>
So, i have this code:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    
    def __init__(self, channel, reduction=16, n_segment=8):
        super(MyModule, self).__init__()
        self.channel = channel
        self.reduction = reduction
        self.n_segment = n_segment
        
        self.conv1 = nn.Conv2d(in_channels=self.channel, out_channels=self.channel//self.reduction, kernel_size=1, bias=False)
        self.conv2 = nn.Conv2d(in_channels=self.channel, out_channels=self.channel//self.reduction, kernel_size=1, bias=False)
        self.conv3 = nn.Conv2d(in_channels=self.channel, out_channels=self.channel//self.reduction, kernel_size=1, bias=False)
        #whatever

        # learnable weight
        self.W_1 = nn.Parameter(torch.randn(1), requires_grad=True)
        self.W_2 = nn.Parameter(torch.randn(1), requires_grad=True)
        self.W_3 = nn.Parameter(torch.randn(1), requires_grad=True)

    def forward(self, x):
        
        # whatever
        
        ## branch1                
        bottleneck_1 = self.conv1(x)
        
        ## branch2
        bottleneck_2 = self.conv2(x)
        
        ## branch3                
        bottleneck_3 = self.conv3(x)
        
        ## summation
        output = self.avg_pool(self.W_1*bottleneck_1 + 
                          self.W_2*bottleneck_2 + 
                          self.W_3*bottleneck_3) 
        
        return output
</code></pre>
<p>As you see, 3 learnable scalars (W_1, W_2, and W_3) are used for weighting purpose. But, this approach will not guarantee that the sum of those scalars is 1. How to make the summation of my learnable scalars equals to 1? Thanks</p>

InnovArul: <p>I could think of the following way:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self, channel, reduction=16, n_segment=8):
        ...
        self.W = nn.Parameter(torch.randn(3))
        ...

    def forward(self, x):
        # whatever
        ...

        w = self.W.softmax(-1)
        output = self.avg_pool(w[0]*bottleneck_1 + 
                               w[1]**bottleneck_2 + 
                               w[2]*bottleneck_3) 
        ...

</code></pre>

sudonto: <p>Hei <a class="mention" href="/u/innovarul">@InnovArul</a> , that’s genius approach,many thanks!</p>

sudonto: <p>Hei again <a class="mention" href="/u/innovarul">@InnovArul</a> , when I <code>print(w)</code>, I saw repeating value of <code>w</code>? Why is <code>w</code> not updating? I declare <code>W</code> like this: <code>self.W = nn.Parameter(torch.randn(3), requires_grad=True)</code></p>

InnovArul: <p>Maybe it is learning equal weightage? What do you think?<br>
I am not sure of your exact scenario.</p>
<p>A small correction. When you are using <code>nn.Parameter</code>, it automatically assigns <code>requires_grad=True</code>. So you don’t need to specify it manually.</p>

sudonto: <p><a class="mention" href="/u/innovarul">@InnovArul</a> , I want to implement like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3826625dd7816351b0781fe7316e65daef2dfbd.png" alt="image" data-base62-sha1="pC0QriIcEUxT52bKmBPwbyEYAt7" width="357" height="244"></p>

InnovArul: <p>What are the values getting printed btw? Are they add to 1? Maybe you have to train for longer time?</p>
<aside class="quote no-group" data-username="sudonto" data-post="1" data-topic="149700">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sudonto/48/23727_2.png" class="avatar"> sudonto:</div>
<blockquote>
<pre><code class="lang-auto">        self.conv1 = nn.Conv2d(in_channels=self.channel, out_channels=self.channel//self.reduction, kernel_size=1, bias=False)
        self.conv2 = nn.Conv2d(in_channels=self.channel, out_channels=self.channel//self.reduction, kernel_size=1, bias=False)
        self.conv3 = nn.Conv2d(in_channels=self.channel, out_channels=self.channel//self.reduction, kernel_size=1, bias=False)
</code></pre>
</blockquote>
</aside>
<p>Also they seem to have shared the parameters between all 3 convs. Your code doesn’t seem to do that. I do not know if that’s a key point to consider.</p>

sudonto: <p>What are the values getting printed btw?</p>
<blockquote>
<p><code>print(w)</code> after <code>self.avg_pool</code> gives me like this:<br>
tensor([0.1944, 0.0687, 0.7370], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1981, 0.1346, 0.6673], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2181, 0.4030, 0.3789], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6972, 0.2325, 0.0704], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0573, 0.0222, 0.9205], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0967, 0.6594, 0.2438], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0540, 0.3833, 0.5628], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1071, 0.8304, 0.0625], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1369, 0.1455, 0.7176], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1141, 0.6166, 0.2694], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5121, 0.3077, 0.1802], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6380, 0.2686, 0.0934], device=‘cuda:0’, grad_fn=)<br>
tensor([0.3382, 0.1036, 0.5582], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1021, 0.3302, 0.5676], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0386, 0.6035, 0.3578], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1217, 0.5602, 0.3181], device=‘cuda:0’, grad_fn=)<br>
Epoch: [0][0/43008], lr: 0.00500	Time 3.649 (3.649)	Data 0.115 (0.115)	Loss 5.1573 (5.1573)	Prec@1 0.000 (0.000)	Prec@5 0.000 (0.000)<br>
tensor([0.1944, 0.0687, 0.7370], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1981, 0.1346, 0.6673], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2181, 0.4030, 0.3789], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6972, 0.2325, 0.0704], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0573, 0.0222, 0.9205], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0967, 0.6594, 0.2438], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0540, 0.3833, 0.5628], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1071, 0.8304, 0.0625], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1369, 0.1455, 0.7176], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1141, 0.6166, 0.2694], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5121, 0.3077, 0.1802], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6380, 0.2686, 0.0934], device=‘cuda:0’, grad_fn=)<br>
tensor([0.3382, 0.1036, 0.5582], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1021, 0.3302, 0.5676], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0386, 0.6035, 0.3578], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1217, 0.5602, 0.3181], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1944, 0.0687, 0.7370], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1981, 0.1346, 0.6673], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2181, 0.4030, 0.3789], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6972, 0.2325, 0.0704], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0573, 0.0222, 0.9205], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0967, 0.6594, 0.2438], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0540, 0.3833, 0.5628], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1071, 0.8304, 0.0625], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1369, 0.1455, 0.7176], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1141, 0.6166, 0.2694], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5121, 0.3077, 0.1802], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6380, 0.2686, 0.0934], device=‘cuda:0’, grad_fn=)<br>
tensor([0.3382, 0.1036, 0.5582], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1021, 0.3302, 0.5676], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0386, 0.6035, 0.3578], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1217, 0.5602, 0.3181], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1944, 0.0687, 0.7370], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1981, 0.1346, 0.6673], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2181, 0.4030, 0.3789], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6972, 0.2325, 0.0704], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0573, 0.0222, 0.9205], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0967, 0.6594, 0.2438], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0540, 0.3833, 0.5628], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1071, 0.8304, 0.0625], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1369, 0.1455, 0.7176], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1141, 0.6166, 0.2694], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5121, 0.3077, 0.1802], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6380, 0.2686, 0.0934], device=‘cuda:0’, grad_fn=)<br>
tensor([0.3382, 0.1036, 0.5582], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1021, 0.3302, 0.5676], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0386, 0.6035, 0.3578], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1217, 0.5602, 0.3181], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1944, 0.0687, 0.7370], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1981, 0.1346, 0.6673], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2181, 0.4030, 0.3789], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6972, 0.2325, 0.0704], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0573, 0.0222, 0.9205], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0967, 0.6594, 0.2438], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0540, 0.3833, 0.5628], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1071, 0.8304, 0.0625], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1369, 0.1455, 0.7176], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1141, 0.6166, 0.2694], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5121, 0.3077, 0.1802], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6380, 0.2686, 0.0934], device=‘cuda:0’, grad_fn=)<br>
tensor([0.3382, 0.1036, 0.5582], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1021, 0.3302, 0.5676], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0386, 0.6035, 0.3578], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1217, 0.5602, 0.3181], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1944, 0.0687, 0.7370], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1981, 0.1346, 0.6673], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2181, 0.4030, 0.3789], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6972, 0.2325, 0.0704], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0573, 0.0222, 0.9205], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0967, 0.6594, 0.2438], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0540, 0.3833, 0.5628], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1071, 0.8304, 0.0625], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1369, 0.1455, 0.7176], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1141, 0.6166, 0.2694], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5121, 0.3077, 0.1802], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6380, 0.2686, 0.0934], device=‘cuda:0’, grad_fn=)<br>
tensor([0.3382, 0.1036, 0.5582], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1021, 0.3302, 0.5676], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0386, 0.6035, 0.3578], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1217, 0.5602, 0.3181], device=‘cuda:0’, grad_fn=)<br>
…</p>
</blockquote>
<p>Are they add to 1?</p>
<blockquote>
<p>Yes</p>
</blockquote>
<p>Maybe you have to train for longer time?</p>
<blockquote>
<p>I will try, I only run 3sec before</p>
</blockquote>
<p>Also they seem to have shared the parameters between all 3 convs</p>
<blockquote>
<p>My mistake to write my code in this discussion, I actually have only 1 conv, not 3 conv as in the question</p>
</blockquote>

sudonto: <p><a class="mention" href="/u/innovarul">@InnovArul</a> , still, it prints repeating value after running 15minutes</p>

InnovArul: <p>Can you paste those values here?</p>

sudonto: <p>tensor([0.5125, 0.2359, 0.2516], device=‘cuda:0’, grad_fn=)<br>
tensor([0.7157, 0.2214, 0.0629], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2182, 0.4743, 0.3075], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5767, 0.1281, 0.2952], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0621, 0.7963, 0.1416], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1327, 0.1156, 0.7517], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2615, 0.5469, 0.1917], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0467, 0.4114, 0.5419], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6102, 0.0481, 0.3417], device=‘cuda:0’, grad_fn=)<br>
tensor([0.4151, 0.4901, 0.0948], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1035, 0.1519, 0.7446], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2012, 0.6926, 0.1062], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6431, 0.0715, 0.2854], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2183, 0.6838, 0.0978], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1041, 0.5261, 0.3698], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2220, 0.1611, 0.6169], device=‘cuda:0’, grad_fn=)<br>
Epoch: [0][0/86017], lr: 0.00500	Time 3.243 (3.243)	Data 0.078 (0.078)	Loss 5.1598 (5.1598)	Prec@1 0.000 (0.000)	Prec@5 0.000 (0.000)<br>
tensor([0.5125, 0.2359, 0.2516], device=‘cuda:0’, grad_fn=)<br>
tensor([0.7157, 0.2214, 0.0629], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2182, 0.4743, 0.3075], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5767, 0.1281, 0.2952], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0621, 0.7963, 0.1416], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1327, 0.1156, 0.7517], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2615, 0.5469, 0.1917], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0467, 0.4114, 0.5419], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6102, 0.0481, 0.3417], device=‘cuda:0’, grad_fn=)<br>
tensor([0.4151, 0.4901, 0.0948], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1035, 0.1519, 0.7446], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2012, 0.6926, 0.1062], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6431, 0.0715, 0.2854], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2183, 0.6838, 0.0978], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1041, 0.5261, 0.3698], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2220, 0.1611, 0.6169], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5125, 0.2359, 0.2516], device=‘cuda:0’, grad_fn=)<br>
tensor([0.7157, 0.2214, 0.0629], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2182, 0.4743, 0.3075], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5767, 0.1281, 0.2952], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0621, 0.7963, 0.1416], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1327, 0.1156, 0.7517], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2615, 0.5469, 0.1917], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0467, 0.4114, 0.5419], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6102, 0.0481, 0.3417], device=‘cuda:0’, grad_fn=)<br>
tensor([0.4151, 0.4901, 0.0948], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1035, 0.1519, 0.7446], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2012, 0.6926, 0.1062], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6431, 0.0715, 0.2854], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2183, 0.6838, 0.0978], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1041, 0.5261, 0.3698], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2220, 0.1611, 0.6169], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5125, 0.2359, 0.2516], device=‘cuda:0’, grad_fn=)<br>
tensor([0.7157, 0.2214, 0.0629], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2182, 0.4743, 0.3075], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5767, 0.1281, 0.2952], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0621, 0.7963, 0.1416], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1327, 0.1156, 0.7517], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2615, 0.5469, 0.1917], device=‘cuda:0’, grad_fn=)<br>
tensor([0.0467, 0.4114, 0.5419], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6102, 0.0481, 0.3417], device=‘cuda:0’, grad_fn=)<br>
tensor([0.4151, 0.4901, 0.0948], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1035, 0.1519, 0.7446], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2012, 0.6926, 0.1062], device=‘cuda:0’, grad_fn=)<br>
tensor([0.6431, 0.0715, 0.2854], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2183, 0.6838, 0.0978], device=‘cuda:0’, grad_fn=)<br>
tensor([0.1041, 0.5261, 0.3698], device=‘cuda:0’, grad_fn=)<br>
tensor([0.2220, 0.1611, 0.6169], device=‘cuda:0’, grad_fn=)<br>
tensor([0.5125, 0.2359, 0.2516], device=‘cuda:0’, grad_fn=)<br>
tensor([0.7157, 0.2214, 0.0629], device=‘cuda:0’, grad_fn=)</p>

sudonto: <p>There should be grad_fn=SoftmaxBackward, but when I paste here, grad_fn is empty</p>

InnovArul: <aside class="quote no-group" data-username="sudonto" data-post="9" data-topic="149700" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sudonto/48/23727_2.png" class="avatar"> sudonto:</div>
<blockquote>
<p><a class="mention" href="/u/innovarul">@InnovArul</a> , still, it prints repeating value after running 15minutes</p>
</blockquote>
</aside>
<p>Can you point out the repeating values?</p>

sudonto: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c685dc9e3acbd56c5ea76722803d3daee7b0285.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c685dc9e3acbd56c5ea76722803d3daee7b0285" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c685dc9e3acbd56c5ea76722803d3daee7b0285.png" alt="image" data-base62-sha1="8CofDaFrlshNvYyZPXtBUtFgGJ7" width="501" height="500" data-dominant-color="E0DFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">694×692 16.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

sudonto: <p><a class="mention" href="/u/innovarul">@InnovArul</a> , I also print <code>model.module.base_model.layer1[0].mve.W.data</code> after <code>optimizer.step()</code> and the tensor values dont change at all after several iteration. Really need your help</p>

InnovArul: <p>I am not sure how they are repeating exactly when you train with shuffled training data.<br>
Also, Try to not regularize this <code>self.W</code> to see its effect.</p>
<p>Atleast this part of the code looks fine to me. I do not know about your other parts of the code.</p>

sudonto: <p>By the way, when I print modules using <code>for m in self.modules(): blablabla</code> the <code>self.W</code> is not listed in the list. What a strange.</p>
<p>UPDATE:<br>
<a class="mention" href="/u/innovarul">@InnovArul</a> , Turns out, I need to wrap my self.W in nn.ParameterList() to be recognized in model.modules()</p>
<p>UPDATE2:<br>
It’s updating NOW, thxxxxxxx!!!</p>

InnovArul: <p>Do you mean, using <code>nn.ParameterList()</code> works, but <code>nn.Parameter()</code> does not work?<br>
In my understanding, there is no difference between these two in this scenario.</p>
<p>Maybe if you mention how you solved it, it will be helpful to others.</p>

sudonto: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, sorry for replying late.<br>
So, in my code, I have a function to assign different learning rate and weight decay to layers so that all layers dont have same LR and WD. For this, I call <code>model.modules</code> to iterate all modules/layers inside my model. Unfortunately, <code>self.W</code> cannot be seen. Searching on the Internet, several people used <code>nn.ParameterList</code> or <code>nn.ParameterDict</code> so that <code>nn.Parameter</code> declared in <code>class MyModule</code> can be seen when using <code>model.modules </code> .</p>

InnovArul: <p>I understand. With <code>self.W</code> being <code>nn.Parameter()</code>, you can access it by <code>model.named_parameters()</code> or <code>model.parameters()</code>.</p>