LSTM not working properly on validation

username: Amasu
cooked: <h2>Hi, I am working on sentiment analysis. I converted my text to word2vec embeddings using genism. Now, I am passing these encoded vectors to lstm as inputs and using lstm to classify sentiment of the text. But, the isssue that I am facing is that for training my loss decreases throughout and accuracy increases to 95% almost. But for my validation loss decreases for few epochs but then it start increasing might be because of over-fitting. The validation accuracy is stuck and fluctuates between 49 and 51. I am not sure what the problem is. This is my first time working on NLP. Kindly, help me out.</h2>
<h2>Sentiments 0 and 1<br>
Activation = Sigmoid<br>
Learning rate = 5e-3<br>
loss = BCEloss<br>
optimizer = SGD</h2>
<p><strong>Code:</strong></p>
<p>class LSTMSentimentNet(nn.Module):</p>
<pre><code>def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, word_embeddings, drop_prob=0.64):

    super(LSTMSentimentNet, self).__init__()

    self.output_size = output_size

    self.n_layers = n_layers

    self.hidden_dim = hidden_dim

    

    self.embedding = nn.Embedding(vocab_size, embedding_dim)

    self.embedding.weight = nn.Parameter(word_embeddings, requires_grad=False)

    self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=drop_prob, batch_first=True, bidirectional=True)

    self.dropout = nn.Dropout(drop_prob)

    self.fc = nn.Linear(hidden_dim, output_size)

    self.sigmoid = nn.Sigmoid()

    

def forward (self, input_words):

    input_words = input_words.long()           

    embedded_words = self.embedding(input_words)  

    lstm_out, h = self.lstm(embedded_words)        
    lstm_out = self.dropout(lstm_out)

    lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim) 
    fc_out = self.fc(lstm_out)                    
    sigmoid_out = self.sigmoid(fc_out)          

    sigmoid_out = sigmoid_out.view(batch_size, -1)

    sigmoid_last = sigmoid_out[:, -1]            

    return sigmoid_last, h

def init_hidden(self, batch_size):

    weight = next(self.parameters()).data

    if (is_cuda):

        hidden = (weight.new(self.n_layers*2, batch_size, self.hidden_dim).zero_().cuda(),

                     weight.new(self.n_layers*2, batch_size, self.hidden_dim).zero_().cuda())

    else:

        hidden = (weight.new(self.n_layers*2, batch_size, self.hidden_dim).zero_(),

                      weight.new(self.n_layers*2, batch_size, self.hidden_dim).zero_())

    return hidden
</code></pre>
<hr>
<p>Training</p>
<p>for i in range(epochs):</p>
<pre><code>h = model.init_hidden(batch_size)

num_correct = 0

val_correct = 0

model.train()

print('Epoch: {}/{}...'.format(i+1, epochs))

for inputs, labels in train_loader:

    h = tuple([e.data for e in h])

    inputs, labels = inputs.to(device), labels.to(device)

    model.zero_grad()

    output, h = model(inputs)

    loss = criterion(output.unsqueeze(1), labels.float())

    loss.backward()

    nn.utils.clip_grad_norm_(model.parameters(), clip)

    optimizer.step()

    pred = torch.round(output.squeeze())  # Rounds the output to 0/1

    correct_tensor = pred.eq(labels.float().view_as(pred))

    correct = np.squeeze(correct_tensor.cpu().numpy())

    num_correct += np.sum(correct)

train_acc = num_correct/len(train_loader.dataset)

print("\tTrain Loss: {:.6f}".format(loss.item())+", Train accuracy: {:.3f}%".format(train_acc*100))

            

with torch.no_grad():

    model.eval()

    val_h = model.init_hidden(batch_size)

    val_losses = []

    for inp, target in val_loader:

        val_h = tuple([each.data for each in val_h])

        inp, target = inp.to(device), target.to(device)

        out, val_h = model(inp)

        val_loss = criterion(out.unsqueeze(1), target.float())

        val_losses.append(val_loss.item())

        val_pred = torch.round(out.squeeze())  # Rounds the output to 0/1

        val_correct_tensor = pred.eq(target.float().view_as(val_pred))

        corrects = np.squeeze(val_correct_tensor.cpu().numpy())

        val_correct += np.sum(corrects)

    val_acc = val_correct/len(val_loader.dataset)

    print("\tValidation Loss: {:.6f}".format(val_loss)+", Validation accuracy: {:.3f}%".format(val_acc*100))</code></pre>

username: vdw
cooked: <p>I’m (reasonably) sure that your <code>forward()</code> method is off. There are too many <code>view()</code> calls that shouldn’t be needed, but you have them to hammer the dimensions of the tensors to the required shape. While the network won’t complain, it won’t learn anything either. See one problem with <code>view()</code> <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">here</a>.</p>
<p><code>lstm_out</code> contains ALL the hidden states of the last LSTM layer and in both directions. I’m pretty sure that</p>
<pre><code>lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
</code></pre>
<p>messes up your tensor. What’s the shape of <code>lstm_out</code> after that command. If the first dimension is not your batch size – which I doubt it is – <code>lstm_out</code> is messed up.</p>
<p>While you can use <code>lstm_out</code>, I would first go with the hidden state <code>h</code> to make it a bit simpler. You might want to try the following changes to your <code>forward()</code> method:</p>
<pre><code>forward(...):
    ...
    lstm_out, h = self.lstm(embedded_words)
    # Get the the last hidden state of the last LSTM layer (see LSTM docs)
    # Note the h[0] since h is a tuple for LSTMs
    last_h = h[0].view(self.n_layers, 2, batch_size, self.hidden_dim)
    # Combine the hidden states of forward and backward LSTM
    final_h = last_h[0] + last_h[1]
    # final_h should have the shape (batch_size, hidden_dim)
    fc_out = self.fc(final_h)
    # sigmoid_out = sigmoid_out.view(batch_size, -1) # Should not be needed anymore
    ...</code></pre>

username: Amasu
cooked: <p>Hi, I tired what you suggested. I got this error.</p>
<p>ERR: Target and input must have the same number of elements. target nelement (100) != input nelement (200)</p>
<p>I also printed the shapes I am getting. My batch size is 100, and number of layers 3.</p>
<p>last_h torch.Size([3, 2, 100, 64])<br>
final_h torch.Size([2, 100, 64])<br>
fc_out torch.Size([2, 100, 1])</p>

username: vdw
cooked: <p>Argh, so sorry, my bad! The line</p>
<pre><code>last_h = h[0].view(self.n_layers, 2, batch_size, self.hidden_dim)
</code></pre>
<p>was supposed to be</p>
<pre><code>last_h = h[0].view(self.n_layers, 2, batch_size, self.hidden_dim)[-1]
</code></pre>
<p>The <code>[-1]</code> gets the last of your 3 layers. Then the shape of <code>l_last</code> will be (2, batch_size, hidden_dim), with 2 reflecting the two directions. Then <code>last_h[0]</code> and <code>last_h[1]</code> will also makes sense with both having the shape (batch_size, hidden_dim), as will have <code>final_h</code>.</p>

username: Amasu
cooked: <p>Thanks man. It worked. I wrote forward function carefully this time without using views.</p>