Linear layer params not updating

username: Stevellen
cooked: <p>Hello,<br>
I’ve been implementing a data fusion model, and it mostly works, but in one of the modules I am having trouble with the parameters of a linear layer not updating. I’m guessing it’s something in my intermediate calculations, but I can’t seem to figure out what.<br>
Here is my init and forward implementation:</p>
<pre><code class="lang-auto">class WeightedCombinationModule(nn.Module):
    def __init__(self, in_size, pool_size, hidden_size=64, 
                    rnn_layers=2, dropout=0.7, a=9.0, b=0.01, c=10.0):
        super(WeightedCombinationModule, self).__init__()
        
        # create class variables
        self.__dict__.update(locals())
        # pooling layer
        self.Pool = nn.MaxPool2d((1, self.pool_size), self.pool_size//2)
        
        # quality weight parameters
        wc_h = 1
        wc_w = (self.in_size[-1] - (self.pool_size-1)-1) // (self.pool_size//2) + 1

        self.lin = nn.Linear(wc_h*wc_w*in_size[1], 1)

        # RNN 
        self.gru = nn.GRU(wc_w, self.hidden_size, self.rnn_layers, batch_first=True)

    def forward(self, Sn):     
        Fn = torch.stack([self.Pool(s.squeeze(1)) for s in Sn.split(1,1)], 1)

        # encoding vector
        un = torch.flatten(Fn, 2, 4)
        # quality weights 
        inter = self.lin(un).squeeze(2)
        en = torch.div(inter, un.size(2))
        a_tildas = self.a / (1+torch.exp(-en/self.b)) + self.c

        a_sum = torch.sum(a_tildas, 1)
        alphas = a_tildas / a_sum.unsqueeze(1)
        a_temp = alphas.view(*alphas.size(), 1, 1, 1)
        C = a_temp * Fn
        C = torch.sum(C, 1).squeeze(2)
        
        # rnn
        output, _ = self.gru(C) # (batch, seq-len, hidden-size)
        r_wc = output.sum(2)  # (batch, seq-len)
        return r_wc
</code></pre>
<p>Edit: For clarity, here is a sample of one step’s gradients. You can see that the params of the linear layer have no gradient for some reason.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a18a7fca17401901d0536cfd5bba659627d0dc86.png" data-download-href="https://discuss.pytorch.org/uploads/default/a18a7fca17401901d0536cfd5bba659627d0dc86" title="grads_(-0.001, 0.15)_15"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a18a7fca17401901d0536cfd5bba659627d0dc86_2_690x459.png" alt="grads_(-0.001, 0.15)_15" data-base62-sha1="n33ByVJmez9PkuITIPlPHiKwohE" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a18a7fca17401901d0536cfd5bba659627d0dc86_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a18a7fca17401901d0536cfd5bba659627d0dc86_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a18a7fca17401901d0536cfd5bba659627d0dc86_2_1380x918.png 2x" data-dominant-color="FBFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grads_(-0.001, 0.15)_15</span><span class="informations">2160×1440 60.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you print the gradients directly from the <code>self.lin</code> layer?<br>
If the computation graph is detached somewhere (which I cannot see in your code, as it looks fine), you shouldn’t be able to see any values in <code>print(model.lin.weight.grad)</code> but it should instead print a <code>None</code> value.</p>

username: Stevellen
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for the response. It gave me the idea to include an additional check on the gradient plotting function that does not include gradients if they are None. After doing so the plot still contains the “wc_module.lin.weight” gradient, and printing it explicitly showed me that there are values there, they are just incredibly small. My mistake for not checking more thoroughly, it’s just odd to me that those parameters have such small updates compared to the rest of them.</p>

username: ptrblck
cooked: <p>I’m not sure, how all variables are defined, e.g. <code>self.b</code>, but you could try to remove certain parts of the code, which involve the linear layer for its calculation, and try to isolate which operation might decrease the gradient magnitude.<br>
If I understand the code correctly, you are applying a manual sigmoid, which might create small gradients, if the input is saturated (very small or very large).</p>