Bias are not updating in decoder model

Anuj_Daga: <pre><code class="lang-auto">class Framework(nn.Module):
    def __init__(self):
        super(Framework, self).__init__()
        self.fc1 = nn.Linear(input_shape, 512)
        self.fc21 = nn.Linear(512, 128)
        self.fc22 = nn.Linear(512, 128)
        self.fc3 = nn.Linear(128, 512)
        self.fc4 = nn.Linear(512, input_shape)
        for m in self.modules():
            if isinstance(m, nn.Linear):
                torch.nn.init.xavier_uniform_(m.weight)
                torch.nn.init.zeros_(m.bias)

    def encoder(self, x):
        h1 = F.relu(self.fc1(x)+self.fc1.bias)
        return self.fc21(h1)+self.fc21.bias, F.elu(self.fc22(h1)+self.fc22.bias)+1
        

    def sampling(self, mu, var):
        eps= torch.nn.init.uniform_(var, a=0.0, b=1.0)
        return mu + eps*var

    def decoder(self, z):
        h3 = F.relu(self.fc3(z)+self.fc3.bias)
        #print(self.fc3.bias)
        return torch.sigmoid(self.fc4(h3)+self.fc4.bias)

    def forward(self, x):
        mu, var = self.encoder(x.view(-1, input_shape))
        z = self.sampling(mu, var)
        return self.decoder(z), mu, var

model1 = Framework().to(device)
model2 = Framework().to(device)
model3 = Framework().to(device)

import itertools
#deine optimizers
f_params=model1.parameters()
s_params=model2.parameters()
t_params=model3.parameters()
dvne_params=itertools.chain(f_params,s_params,t_params)
optimizer = optim.RMSprop(dvne_params, lr=learning_rate)


</code></pre>

Anuj_Daga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please help me with this?</p>

zarkopafilis: <p>I understand your intuition, but pytorch linear layers use bias by default in all initialization, forward and backward pass phases. You don’t need to explicitly specify it.</p>
<p>Please see:<br>
<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L74" rel="nofollow noopener">pytorch/linear.py</a></p>

Anuj_Daga: <p>you why is the bias not updating in the fc3 and fc4 layers?<br>
Do you think there is a conceptual problem or a coding problem here.</p>

zarkopafilis: <aside class="quote no-group" data-username="Anuj_Daga" data-post="1" data-topic="80897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anuj_daga/48/23174_2.png" class="avatar"> Anuj_Daga:</div>
<blockquote>
<p>+self.fc4.bias</p>
</blockquote>
</aside>
<p>Have you tried removing the <code>+self.fc*.bias</code> parts from the above snippet, or plotting the bias value every epoch?</p>

Anuj_Daga: <p>yes,I have tried getting bias values after epoch hence fc1,fc2 are updating nicely.but fc3 and fc4 biases are not updating</p>

ptrblck: <p>The code looks generally alright.<br>
Could you check the gradients of all parameters? Maybe you are dealing with vanishing gradients?</p>
<p>This dummy code snippet prints valid gradients for all parameters:</p>
<pre><code class="lang-python">input_shape = 1
model = Framework()

x = torch.randn(1, 1)
out = model(x)
out[0].mean().backward()

for name, param in model.named_parameters():
    print(name, param.grad.abs().sum())
&gt; fc1.weight tensor(1.2002)
fc1.bias tensor(5.8217)
fc21.weight tensor(8.7014)
fc21.bias tensor(2.8945)
fc22.weight tensor(8.8635)
fc22.bias tensor(2.9484)
fc3.weight tensor(177.9702)
fc3.bias tensor(7.6872)
fc4.weight tensor(15.0663)
fc4.bias tensor(0.4998)
</code></pre>

Anuj_Daga: <p>Sorry,I could not get how to solve the issue.<br>
for me snippet looks something like this:<br>
<code>fc1.weight tensor(5.9704) fc1.bias tensor(4.6652) fc21.weight tensor(191.8801) fc21.bias tensor(1.5342) fc22.weight tensor(382.4296) fc22.bias tensor(3.0577) fc3.weight tensor(455.5275) fc3.bias tensor(3.5586) fc4.weight tensor(39.6243) fc4.bias tensor(0.2452)</code></p>

ptrblck: <p>Your output also shows, that <code>fc3.bias</code> and <code>fc4.bias</code> have valid gradients.<br>
As long as you pass these parameters to the optimizer (e.g. via <code>optim.SGD(model.parameters(), ...)</code>), they will be updated.</p>
<p>How are you checking, if they are updated or not?</p>

Anuj_Daga: <p>I was checking by printing the model1.state_dict. but they did n’t updated.<br>
Thanks again <a class="mention" href="/u/ptrblck">@ptrblck</a>, I got the Solution just now.<br>
I was missing Variable() in the above line:</p>
<pre><code class="lang-auto">eps = Variable(torch.randn_like(std))
</code></pre>
<p>I know the bias are updating now,but If possible,can you check if I am conceptually correct?</p>

ptrblck: <p><code>Variables</code> are deprecated since PyTorch <code>0.4</code> so you should use tensors now.<br>
It seems <code>eps</code> shouldn’t be updated, so you should be able to create it as a tensor without setting <code>requires_grad=True</code>.</p>