Training loss is not changing at all while training LSTM

username: KoushikSahu
cooked: <p>I am trying to solve a text classification problem. My training data has input as a sequence of 80 numbers in which each represent a word and target value is just a number between 1 and 3.<br>
I pass it through this model:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, tokenize_vocab_count):
        super().__init__()
        self.embd = nn.Embedding(tokenize_vocab_count+1, 300)
        self.embd_dropout = nn.Dropout(0.3)
        self.LSTM = nn.LSTM(input_size=300, hidden_size=100, dropout=0.3, batch_first=True)
        self.lin1 = nn.Linear(100, 1024)
        self.lin2 = nn.Linear(1024, 512)
        self.lin_dropout = nn.Dropout(0.8)
        self.lin3 = nn.Linear(512, 3)
    
    def forward(self, inp):
        inp = self.embd_dropout(self.embd(inp))
        inp, (h_t, h_o) = self.LSTM(inp)
        h_t = F.relu(self.lin_dropout(self.lin1(h_t)))
        h_t = F.relu(self.lin_dropout(self.lin2(h_t)))
        out = F.softmax(self.lin3(h_t))
        return out
</code></pre>
<p>My training loop is as follows:</p>
<pre><code class="lang-auto">model = Model(tokenizer_obj.count+1).to('cuda')

optimizer = optim.AdamW(model.parameters(), lr=1e-2)
loss_fn = nn.CrossEntropyLoss()

EPOCH = 10

for epoch in range(0, EPOCH):
     for feature, target in tqdm(author_dataloader):
         train_loss = loss_fn(model(feature.to('cuda')).view(-1,  3), target.to('cuda'))
         optimizer.zero_grad()
         train_loss.backward()
         optimizer.step()
      print(f"epoch: {epoch + 1}\tTrain Loss : {train_loss}")
</code></pre>
<p>I printed out the feature and target dimension and it is as follows:</p>
<pre><code class="lang-auto">torch.Size([64, 80]) torch.Size([64])
</code></pre>
<p>Here 64 is the batch_size.<br>
I am not doing any validation as of now.<br>
When I train I am getting a constant loss value and no change</p>
<pre><code class="lang-auto">/home/koushik/Software/miniconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/rnn.py:50: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.3 and num_layers=1
  "num_layers={}".format(dropout, num_layers))
  0%|                                                                                                                                                 | 0/306 [00:00&lt;?, ?it/s]/media/koushik/Backup Plus/Code/Machine Deep Learning/NLP/src/Deep Learning/model.py:20: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  out = F.softmax(self.lin3(h_t))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 89.36it/s]
epoch: 1        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 89.97it/s]
epoch: 2        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 89.35it/s]
epoch: 3        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 89.17it/s]
epoch: 4        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 88.72it/s]
epoch: 5        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 87.75it/s]
epoch: 6        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 85.67it/s]
epoch: 7        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 85.40it/s]
epoch: 8        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 84.49it/s]
epoch: 9        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03&lt;00:00, 84.21it/s]
epoch: 10       Train Loss : 1.0986120700836182
</code></pre>
<p>Can anyone please help</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> uses <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally, so it expects raw logits as the model outputs.<br>
Remove the <code>F.softmax</code> at the end of your model and pass the output of <code>self.lin(3)</code> directly to the criterion.<br>
Let me know, if that helps. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KoushikSahu
cooked: <p>Yes, model training now. Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: KoushikSahu
cooked: <p>I am having another problem now. I am training the model but the loss is going up and down repeatedly. It’s not decreasing or converging. I tried to decrease the learning rate but it didn’t work.<br>
Here’s the loss plot at lr = 1e-3 for 30 epochs:-<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbddef5d3f8f57af97e4d8bf9a1cde1f18b7088b.png" alt="Figure_1" data-base62-sha1="zW7kZOPSmndXKHUTelzHW5ENwYH" width="640" height="480"><br>
Here’s the loss plot at lr = 1e-6 for 30 epochs:-<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b59e8327b44e9a6dbfd438711c105b48af56c226.png" alt="Figure_2" data-base62-sha1="pUG23M7X6Bx87yfo1EaDX9owPvU" width="640" height="480"><br>
Here’s the loss plot at lr = 1e-9 for 30 epochs:-<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29431e70423a224e63e4e5675c6d38310893db9b.png" alt="Figure_3" data-base62-sha1="5T1nqwQVJQlkaWHZ9zDDLX995dV" width="640" height="480"></p>