LSTM: Validation accuracy not improving

vince120: <p>Hi,</p>
<p>I know this problem have been addressed many times but I cannot find any answers so I’m trying again. I’m building a LSTM classifier to predict a class based on a text. The issue is that my validation accuracy stagnate around 35%.</p>
<p>I’m wondering if it’s my model or my data prepation which is not working. Does my model looks correct for you or do I miss something? Thanks !</p>
<pre><code class="lang-auto">class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, label_size, padding_idx, batch_size):
        super(LSTMClassifier, self).__init__()
        self.vocab_size = vocab_size
        self.embedding_dim = embedding_dim
        self.hidden_dim = hidden_dim
        self.label_size = label_size
        self.num_layers = 1
        self.batch_size = batch_size

        self.emb = nn.Embedding(self.vocab_size, self.embedding_dim, padding_idx=padding_idx)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)#,batch_first=True)
        self.fc = nn.Linear(hidden_dim, label_size)

    def zero_state(self, batch_size): 
        # implement the function, which returns an initial hidden state.
        h0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda()
        c0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda()
        return h0, c0
    
    def forward(self, x):
        x = self.emb(x)
        batch_size = x.size(1)
        h0, c0 = self.zero_state(batch_size)
        out, (hn, cn) = self.lstm(x, (h0, c0))
        out = self.fc(hn[-1, :, :])
        return out
</code></pre>

theairbend3r: <p>Pro tip: You don’t have to intialize the hidden state to 0s in LSTMs. PyTorch does that automatically.</p>
<p>That network looks fine imo. Maybe try changing the embedding size, stacked layers, and input_size.</p>
<p>You’re passing the hidden layer from the last rnn output. Instead you can using the output value from the last time step. Or maybe average out all the hidden/output values.</p>
<p>What’s your train accuracy?</p>

vdw: <p>From what I can see from having a quick look at your code:</p>
<ul>
<li>The shape of <code>x</code> after <code>self.emb(x)</code> should be <code>(batch_size, seq_len, embed_dim)</code>
</li>
<li>So <code>batch_size = x.size(1)</code> gives you the wrong value</li>
<li>In your code you NOT use <code>batch_first=True</code>, but the batch is in fact the first dimension <code>x</code> (this is probably the reason why the network throws no error despite the wrong <code>batch_size</code>)</li>
</ul>
<p>There are two approaches to do this</p>
<p>(1) <code>batch_first=False</code></p>
<pre><code> def forward(self, x):
    batch_size = x.size(0)
    x = self.embed(x) # shape = (batch_size, seq_len, embed_dim)
    x = x.transpose(0, 1) # shape = (seq_len, batch_size, embed_dim)
    h0, c0 = self.zero_state(batch_size)
    out, (hn, cn) = self.lstm(x, (h0, c0))
    out = self.fc(hn[-1, :, :])
    return out
</code></pre>
<p>(2) <code>batch_first=True</code></p>
<pre><code> def forward(self, x):
    batch_size = x.size(0)
    x = self.embed(x) # shape = (batch_size, seq_len, embed_dim)
    h0, c0 = self.zero_state(batch_size)
    out, (hn, cn) = self.lstm(x, (h0, c0))
    out = self.fc(hn[-1, :, :])
    return out</code></pre>

vince120: <p><a class="mention" href="/u/theairbend3r">@theairbend3r</a></p>
<p>Thanks for the tips and the recommendations,  my training error is also not increasing. Basically, around 20 epochs, i have the following performances and it will never go beyond:</p>
<blockquote>
<p>Epoch 0 | Train loss 1.587 | Train acc 0.263 |  Valid loss 1.575 | Valid acc 0.280<br>
Epoch 27 | Train loss 1.073 | Train acc 0.553 |  Valid loss 1.596 | Valid acc 0.357</p>
</blockquote>
<p>I tried to change the HP but nothing help.</p>
<p><a class="mention" href="/u/vdw">@vdw</a> thanks for the solutions, I have done the changes but performances stay the same. I’ll dig into that, maybe my data are in the wrong shape.</p>