LSTM model not learning and updating weights

ekrx: <p>I’m working on a time series forecasting project and encountering issues with my model learning. Based on the weights’ progression in the screenshot (each entry represents an epoch), the weights seem to be stagnating.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef44483cdb5bda7fe971332777a179afa897b7d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef44483cdb5bda7fe971332777a179afa897b7d3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef44483cdb5bda7fe971332777a179afa897b7d3_2_465x500.png" alt="image" data-base62-sha1="y8EpnsRJiLqed4F2udRcM0Sk5kD" width="465" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef44483cdb5bda7fe971332777a179afa897b7d3_2_465x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef44483cdb5bda7fe971332777a179afa897b7d3_2_697x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef44483cdb5bda7fe971332777a179afa897b7d3_2_930x1000.png 2x" data-dominant-color="FCFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1086×1167 50.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is my model:</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(LSTMModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers)
        self.fc_1 = nn.Linear(hidden_size, 128)
        self.relu = nn.ReLU()
        self.fc = nn.Linear(128, 5)

    def forward(self, x):
        out, (hidden, cell) = self.lstm(x)
        out = out[:, -1, :]
        out = self.fc_1(out)
        out = self.relu(out)
        return self.fc(out)

model = LSTMModel(6, 64, 25)
loss_function = nn.CrossEntropyLoss(weight=class_weights)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

model.train()
for x_batch, y_batch in train_data_loader:
        x_batch, y_batch = x_batch.to(device), y_batch.to(device)
        output = model(x_batch)
        loss = loss_function(output, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>Is there any feedback I can get onto why the Model seems to not be learning and weights not being updated?</p>

vdw: <p>The line</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM(input_size, hidden_size, num_layers)
</code></pre>
<p>implies that you use the default value of <code>batch_first=False</code> so the expected input shape is <code>(seq_len, batch_size, input_size)</code> and the shape of <code>out</code> after</p>
<pre><code class="lang-auto">out, (hidden, cell) = self.lstm(x)
</code></pre>
<p>will be <code>(seq_len, batch_size, hidden_size)</code> (since you done use a bidirectional LSTM).</p>
<p>This now means the line</p>
<pre><code class="lang-auto">out = out[:, -1, :]
</code></pre>
<p>will yield an <code>out</code> with shape <code>(seq_len, hidden_size)</code>. What you probably want is a shape of <code>(batch_size, hidden_size)</code>. Therefore you need</p>
<pre><code class="lang-auto">out = out[-1]
</code></pre>
<p>I assume that your shape of <code>x_batch</code> is actually already of shape <code>(batch_size, seq_len, input_size)</code> – i.e., not the expected shape given that you initialized the <code>nn.LSTM</code> with <code>batch_first=False</code> (default). If that is indeed the case, the simplest solution should be to change the code to</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
</code></pre>
<p>Note that in this case the line has to remain</p>
<pre><code class="lang-auto">out = out[:, -1, :]
</code></pre>

ekrx: <p>Thanks <a class="mention" href="/u/vdw">@vdw</a></p>
<p>I did the changes you suggested and I still see the weights pretty much stagnant.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b988d154a91282cbdc31a7115cb91f4270b1ebc6.png" data-download-href="https://discuss.pytorch.org/uploads/default/b988d154a91282cbdc31a7115cb91f4270b1ebc6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b988d154a91282cbdc31a7115cb91f4270b1ebc6_2_548x499.png" alt="image" data-base62-sha1="qtjsGemlKWulLQLGb2pjK5Xq8ia" width="548" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b988d154a91282cbdc31a7115cb91f4270b1ebc6_2_548x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b988d154a91282cbdc31a7115cb91f4270b1ebc6_2_822x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b988d154a91282cbdc31a7115cb91f4270b1ebc6.png 2x" data-dominant-color="F6F1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1071×976 82.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would expect to see more variation however the weights seem to remain in 0.</p>
<p><strong>UPDATE:</strong><br>
I added a print statement:</p>
<pre><code class="lang-auto">        loss.backward()
        print(model.lstm.weight_ih_l0.grad[0][0])
</code></pre>
<p>and seems the weights become very small that vanish to zero and after that they stay there:</p>
<pre><code class="lang-auto">tensor(-2.2441e-27, device='cuda:0')
tensor(2.1074e-29, device='cuda:0')
tensor(6.9026e-28, device='cuda:0')
tensor(-4.6863e-29, device='cuda:0')
tensor(-3.3049e-28, device='cuda:0')
tensor(-1.1580e-29, device='cuda:0')
tensor(3.9923e-30, device='cuda:0')
tensor(1.4204e-27, device='cuda:0')
tensor(2.1197e-30, device='cuda:0')
tensor(-2.6489e-29, device='cuda:0')
tensor(-3.7362e-30, device='cuda:0')
tensor(1.1758e-30, device='cuda:0')
tensor(6.1142e-29, device='cuda:0')
tensor(-1.5144e-30, device='cuda:0')
tensor(2.3485e-31, device='cuda:0')
tensor(4.2870e-33, device='cuda:0')
tensor(-5.3474e-33, device='cuda:0')
tensor(-2.6197e-29, device='cuda:0')
tensor(-9.1210e-32, device='cuda:0')
tensor(5.1670e-28, device='cuda:0')
tensor(6.1556e-31, device='cuda:0')
tensor(-8.9021e-29, device='cuda:0')
tensor(-2.2017e-28, device='cuda:0')
tensor(-6.5199e-33, device='cuda:0')
tensor(4.5335e-31, device='cuda:0')
tensor(-4.3888e-32, device='cuda:0')
tensor(-1.5471e-32, device='cuda:0')
tensor(4.2088e-30, device='cuda:0')
tensor(-1.1161e-31, device='cuda:0')
tensor(9.0165e-31, device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
</code></pre>