Why my model does not train on LSTM?

username: Vatoz.Z
cooked: <p>I am currently working on a regression classification model. But my scores always keep steady, doesn’t decrease.  Is there any problem with my model?</p>
<pre><code class="lang-auto">
class LSTM(nn.Module): 
    def __init__(self):
        super(LSTM, self).__init__()

        embedding_dimension = 40
        pre_trained_embedding = torch.FloatTensor(TEXT.vocab.vectors)
        self.target_dimension = 1 # label: 1 or 0 
        self.embedding_layer = nn.Embedding.from_pretrained(pre_trained_embedding)
        self.hidden_dimension = 200 
        self.lstm = nn.LSTM(embedding_dimension, self.hidden_dimension, num_layers=1, batch_first=True)
        n_lstm_out = 200
        self.dense_layer = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(n_lstm_out, 100),
            nn.Dropout(0.5),
            nn.Linear(100, self.target_dimension, ),
            nn.Sigmoid(),
        )
        
        self.optimizer = optim.SGD(self.parameters(), lr=0.000001)
        self.loss = nn.MSELoss()


    def learn(self, x_train, y_train):
        self.optimizer.zero_grad()
        embedding_out = self.embedding_layer(x_train)
        lstm_out, _ = self.lstm(embedding_out)
        prediction_out = self.dense_layer(lstm_out)
        mse_input = prediction_out[:, 0, :]
        mse_input = mse_input.type(torch.float)
        mse_target = y_train
        mse_target = mse_target.type(torch.float)
        loss = self.loss(mse_input, mse_target)
        loss.backward()
        self.optimizer.step()

        return loss
</code></pre>

username: Abhilash_Srivastava
cooked: <p>You should be using a <code>forward</code> function. Refer to examples available on Pytorch forum and blogs online.</p>

username: Vatoz.Z
cooked: <p>I already use forward function, this is the summary of my model. Train steps worked without any error. But, loss score keep almost steady instead of decreasing. Could you see any inappropriate statement in my code for LSTM  ?</p>

username: vdw
cooked: <aside class="quote no-group" data-username="Vatoz.Z" data-post="1" data-topic="103475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vatoz.z/48/33057_2.png" class="avatar"> Vatoz.Z:</div>
<blockquote>
<pre><code class="lang-auto">prediction_out = self.dense_layer(lstm_out)
mse_input = prediction_out[:, 0, :]
</code></pre>
</blockquote>
</aside>
<p>With <code>batch_first=True</code>, the shape of <code>lstm_out</code> should be <code>(batch_size, seq_len, num_directions * hidden_size)</code>. Since you use 1 layer, it should be <code>(batch_size, seq_len, 200)</code>. This you push through a hidden layer, so <code>prediction_out</code> should be <code>(batch_size, seq_len, 100)</code>.</p>
<p>That means that <code>prediction_out[:, 0, :]</code> takes the hidden state of the first time step for each sequence. You probably want to use the last time step, i.e., <code>prediction_out[:, -1, :]</code>.</p>
<p>Since you throw all other hidden states, I’m not sure why you push all of the through the dense layer in the first place. I would simply do:</p>
<pre><code>lstm_out, (h, c) = self.lstm(embedding_out)
prediction_out = self.dense_layer(h[-1])
</code></pre>
<p>With <code>h[-1]</code> being the last hidden state – last w.r.t. to the sequence as well as w.r.t. the last layer (in case you would have more than one). <code>h[-1]</code> should have the shape <code>(batch_size, hidden_dim)</code>, like I assume you want.</p>
<p>There might be other issues with your code, but that stuck immediately out to me.</p>

username: Vatoz.Z
cooked: <p>Thans a lott :))  I missed that part.</p>