Lstm.weight_hh_l0 doesn't update after gradient descent (but the other parameters do update!)

suzil: <p>I was trying to write some tests for my model using the concept from <a href="https://medium.com/@keeper6928/how-to-unit-test-machine-learning-code-57cf6fd81765" rel="nofollow noopener">this blog post</a> that checks if parameters change after performing a gradient descent step. However, when I tried doing this, I found that some parameters wouldn’t change.</p>
<p>I have reproduced a simple example below where <code>lstm.weight_ih_l0</code>, <code>lstm.bias_ih_l0</code>, <code>lstm.bias_hh_l0</code>, <code>fc.weight</code>, <code>fc.bias</code> change after performing gradient descent, while <code>lstm.weight_hh_l0</code> does not change.</p>
<p>Shouldn’t all of the trainable parameters be updated after every gradient descent step? Is there something wrong with the example code?</p>
<pre><code class="lang-auto">#!/usr/bin/env python3
# torch==0.4.0
# numpy==1.15.0

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np


class MyLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(MyLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(
            input_size,
            hidden_size,
            num_layers,
            batch_first=True,
        )
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        # Set initial states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)

        # Forward propagate LSTM
        out, _ = self.lstm(x, (h0, c0))

        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :])

        return out


model = MyLSTM(input_size=5, hidden_size=8, num_layers=1, num_classes=1)
optimizer = optim.Adagrad(model.parameters(), lr=0.01)
criterion = nn.MSELoss()

params = list(model.state_dict().keys())
assert len(params) == len(list(model.parameters()))
# ['lstm.weight_ih_l0', 'lstm.weight_hh_l0', 'lstm.bias_ih_l0',
# 'lstm.bias_hh_l0', 'fc.weight', 'fc.bias']

# Create some dummy data
inputs = torch.FloatTensor([
    [0.2, 0.3, 0.4, 0.5, 0.6],
    [0.2, 0.3, 0.4, 0.5, 0.6],
    [0.2, 0.3, 0.4, 0.5, 0.6],
]).unsqueeze(1)
target = torch.FloatTensor([1, 0, 1]).unsqueeze(1)

# Save parameter values before gradient descent
before = [model.state_dict()[param].clone() for param in params]

output = model(inputs)
loss = criterion(output, target)
loss.backward()
optimizer.step()

# Save parameter values after gradient descent
after = [model.state_dict()[param].clone() for param in params]

for i, (b, a) in enumerate(zip(before, after)):
    # lstm.weight_hh_l0 doesn't change, but all the other ones do
    # (including lstm.bias_hh_l0!)
    if np.allclose(b, a):
        print('Unchanged:', params[i])
</code></pre>

John_Smith: <p>If you check the gradient of the parameters:</p>
<pre><code class="lang-auto">for p in model.parameters():
    print(p.grad)
</code></pre>
<p>You can see the gradients for one of the parameters are all 0 which means that variable will not get updated during back pass.</p>
<p>You need to check why the gradients are 0s for that variable.</p>

suriyadeepan: <p>I have a similar LSTM model for imdb sentiment classification. Noticed that one of the LSTM parameters doesn’t change at all.</p>
<pre><code class="lang-auto"># trainable parameters
['embedding.weight', 'lstm.weight_ih_l0', 'lstm.weight_hh_l0', 'lstm.bias_ih_l0', 'lstm.bias_hh_l0', 'linear.weight', 'linear.bias']
# unchanged parameter
lstm.weight_ih_l0 
</code></pre>
<p><strong>Edit</strong> : I’ve checked the gradients. They are very small for <code>lstm.weight_ih_l0</code> but not zero, in the range of <code>e^-42</code>.</p>

abhinavkashyap92: <p>Any update on this ?</p>

suzil: <p>Gradients do change, they are just very small.</p>
<aside class="quote" data-post="3" data-topic="22359">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/suriyadeepan/48/2250_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/lstm-weight-hh-l0-doesnt-update-after-gradient-descent-but-the-other-parameters-do-update/22359/3">Lstm.weight_hh_l0 doesn't update after gradient descent (but the other parameters do update!)</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    I have a similar LSTM model for imdb sentiment classification. Noticed that one of the LSTM parameters doesn’t change at all. 
# trainable parameters
['embedding.weight', 'lstm.weight_ih_l0', 'lstm.weight_hh_l0', 'lstm.bias_ih_l0', 'lstm.bias_hh_l0', 'linear.weight', 'linear.bias']
# unchanged parameter
lstm.weight_ih_l0 

Edit : I’ve checked the gradients. They are very small for lstm.weight_ih_l0 but not zero, in the range of e^-42.
  </blockquote>
</aside>


nilinykh: <p>I have the same issue. Gradients for weight_hh_l0 in my 1 layer LSTM network are just zeros. I have checked if they are really zeros, and yes, they are simply 0s. Throughout the backward pass, they never get updated. What could be the reason for such behaviour?</p>