Train loss doesn't decrease

Odin_NI: <p>Hello,</p>
<p>I solve problem of classification. I have a lot of keywords in text and must detected their existence in text. I used torchtext classification with embeddingbag and embedding. Accuracy was equal to 90%. But, I hope that model prediction will be 95% +.<br>
Now, I use embedding + LSTM and have global problem with loss. Loss is not change. I was changing lr from 1000 to 0.0001, but loss change from 0.7 to 0.69.<br>
Help me please <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>
<pre><code class="lang-auto">class TextClassificationModel_vec(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_size):
        super(TextClassificationModel_vec, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0, sparse=True)
        self.lstm = nn.LSTM(input_size=embed_dim, hidden_size=hidden_size, num_layers=1, batch_first=True)
        self.fc = nn.Linear(hidden_size, 1)
        self.dropout = nn.Dropout(p=0.6)
        
    def forward(self, x):
        x = self.embedding(x)
        x = self.dropout(x)
        x, __ = self.lstm(x)
        x = self.fc(x[:, -1, :])
        return x
</code></pre>
<pre><code class="lang-auto">vocab = Vocab(c, min_freq=1)
vocab_size = len(vocab) # 11038 
emsize = 64
hidden_size = 128
model = TextClassificationModel_vec(vocab_size, emsize, hidden_size).to(device)

EPOCHS = 15000
BATCH_SIZE = 50

criterion = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
</code></pre>

Flock1: <p>Have you tried any other optimizer? Like Adam or Adagrad?</p>

Odin_NI: <p>Yes, I used Adam at first, but then i switched to sgd.</p>

AlphaBetaGamma96: <p>Have you tried printing out the gradient values for <code>model.parameters()</code> after calling <code>loss.backward()</code>? Maybe your initial gradient is near 0, so the model won’t update at all?</p>

Odin_NI: <p>I have just checked. Yes indeed gradients first 10e-5 then drops to 10e-45. Can’t fix it?</p>

AlphaBetaGamma96: <p>Ok, so that’s the issue. It has nothing to do with the optimizer. Do you use any in-place operations? That would kill the gradients.</p>

Odin_NI: <p>In train function:</p>
<pre><code class="lang-auto">total_loss += loss.item()
total_acc += (torch.round(torch.sigmoid(predicted)) == label).sum().item()
</code></pre>
<p>In collate.</p>
<pre><code class="lang-auto">def collate_batch(batch):
    label_list, text_list = [], []
    for _label, _text in batch:
        label_list.append(torch.FloatTensor([_label]))
        text = torch.tensor(_text, dtype=torch.int64)
        text_list.append(text)

    label_list = torch.stack(label_list)
    text_list = torch.stack(text_list)
    
    return label_list.to(device), text_list.to(device)
</code></pre>
<p>I am sorry, I am newbie to programming. <img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=9" title=":cold_sweat:" class="emoji" alt=":cold_sweat:"></p>

AlphaBetaGamma96: <p>No need to apologize! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>When you calculate your gradient what variable are you back-propagating? So, are you calculating your gradients by doing <code>total_loss.backward()</code> ?</p>

Odin_NI: <p>I think it will be more convenient this way <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
<pre><code class="lang-auto">def train(dataloader):
    model.train()
    
    total_acc, total_count = 0, 0
    total_loss = 0
    for idx, (label, text) in enumerate(dataloader):
        optimizer.zero_grad()
        predicted = model(text)
        loss = criterion(predicted, label)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        total_acc += (torch.round(torch.sigmoid(predicted)) == label).sum().item()
        total_count += label.size(0)
    return total_acc, total_loss, total_count 
</code></pre>

AlphaBetaGamma96: <p>I was reading through this <a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html" rel="noopener nofollow ugc">tutorial</a> on Text Classification and I can’t really see what’s wrong with your model.</p>
<p>One thing that could be a problem is with the cost function <code>BCEWithLogitsLoss</code> , after looking at the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html?highlight=bcewithlogitsloss#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">docs</a> they pass a <code>pos_weight</code> variable. (which defaults to <code>None</code>). Perhaps this could be the issue? Maybe you’re multipling all your losses by <code>None</code> which would give a near 0 gradient?</p>

Odin_NI: <p>Did not help <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

AlphaBetaGamma96: <p>Ah, no! <img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=9" title=":cold_sweat:" class="emoji" alt=":cold_sweat:"></p>
<p>Then I don’t know what else to recommend. Perhaps there’s something of use in the tutorial I shared above? Perhaps someone else with more experience might be able to help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Sorry!</p>

Odin_NI: <p>This is my first experience with LSTM. Unfortunately, not yet successful. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
Thank you for your help</p>

Odin_NI: <p>I have change LSTM on GRU and it work. But, accuracy is lower than when I used only Embedding.<br>
Embedding + Linear: acc = 0.95<br>
Embedding + GRU + Linear: acc = 0.9<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji only-emoji" alt=":expressionless:"></p>

AlphaBetaGamma96: <p>That’s pretty strange behaviour, perhaps mention this to a developer? They might have a better idea as to why it works with a GRU rather than an LSTM! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  But glad to hear it’s working!</p>