LSTM + FC training stagnation

username: Paulo_Mann
cooked: <p>Hello,</p>
<p>I’m trying to train an LSTM network with a fully connected layer on top of it. But I am facing some issues because I’m not so sure if my model is correctly written, or my training procedure is wrong.</p>
<p>The task is a binary classification with some sequential data of variable length, the batch is a tensor of size <code>torch.Size([32, 58735, 49])</code>, for example, where 32 is the batch size, 58735 is the length of the biggest sequence, and 49 is the number of features of time <code>t</code> of the sequence. I also pass a list of <code>lengths</code> to the model with size <code>torch.Size([32])</code>, equal to the number of examples (i.e., batch size) to use the <code>pack_padded_sequence</code> function, because of the variable length of the sequences. My model is written as:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):

    def __init__(self, input_size, hidden_size):
        super(LSTM, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(self.input_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(self.hidden_size, 1)

    def forward(self, x, lengths):
        bs = x.size(0)
        self.ho = torch.zeros((1, bs, self.hidden_size), device=x.device)
        self.co = torch.zeros((1, bs, self.hidden_size), device=x.device)
        input_lengths, perm_idx = lengths.sort(0, descending=True)
        x = x[perm_idx][:, :input_lengths.max()]
        x = pack_padded_sequence(x, input_lengths, batch_first=True)
        lstm_out, (self.ho, self.co) = self.lstm(x, (self.ho, self.co))
        lstm_out, lengths = pad_packed_sequence(lstm_out)
        x = self.fc(self.ho.squeeze())
        return x.squeeze()
</code></pre>
<p>And the training procedure can be seen <a href="https://gist.github.com/paulomann/6e42258550a50ac2ab1cfdad409981e0" rel="nofollow noopener">here with this gist</a></p>
<p>The problem is that when training with exactly this procedure and this LSTM model, the training accuracy and val accuracy stagnates since the beginning to the end with 0.7730 and 0.2699 accuracies respectively. Since this is the first time I make use of nn.LSTM and pack_padded_sequence, I’m not so sure where the problem could lie in.</p>

username: shubhvachher
cooked: <p>check the outputs of the LSTM… See what it is predicting to see why the accuracies don’t change.</p>
<p>I’m facing something similar here <a href="https://discuss.pytorch.org/t/mqrnn-seq2seq-lstm-mlp-model-gives-constant-predictions/49196" class="inline-onebox">MQRNN (Seq2Seq LSTM + MLP) model gives constant predictions</a></p>

username: Paulo_Mann
cooked: <p>Ok, I analyzed the output predictions from epoch 0, and it starts predicting class 1 for some examples… But 1 or 2 epochs later it stops predicting class 1 and starts predicting almost all examples as class 0 from epoch 1 and beyond. Also, investigating my dataset, both training and validation are very unbalanced for both classes, with 77% of examples in class 0 and 23% in class 1. Maybe the LSTM is overfitting and predicting only class 0 ?</p>
<p>I gonna try the pos_weight of the <code>BCEWithLogitsLoss</code> and see what happens.</p>