LSTM predicts the same constant value

username: Planet_Boost
cooked: <p>I want to predict one variable using 7 features with time steps of 4:</p>
<pre><code class="lang-auto"># Shape X_train: torch.Size([24433, 4, 7]
# Shape Y_train: torch.Size([24433, 4, 1]

# Shape X_test: torch.Size([6109, 4, 7]
# Shape Y_test: torch.Size([6109, 4, 1]

train_dataset = TensorDataset(X_train, Y_train)
test_dataset = TensorDataset(X_test, Y_test) 

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=32, shuffle=False)
</code></pre>
<p><strong>My (initial) LSTM model:</strong></p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size)
        self.linear = nn.Linear(hidden_size, output_size)
        
    def forward(self, x):
        x, _ = self.lstm(x)
        x = self.linear(x)
        return x

model = LSTMModel(input_size=7, hidden_size=256, output_size=1)

loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
</code></pre>
<p><strong>Apply model:</strong></p>
<pre><code class="lang-auto"># Loop over the training set
for X, Y in train_loader:

    optimizer.zero_grad()
    
    Y_pred = model(X)

    loss = loss_fn(Y_pred, Y)
    
    loss.backward()
    
    optimizer.step()

model.eval()

# Loop over the test set
for X, Y in test_loader:

    Y_pred = model(X)
    
    loss = loss_fn(Y_pred, Y)
</code></pre>
<p><strong>An example of Y (true data):</strong></p>
<pre><code class="lang-auto">tensor([[[59.],
         [59.],
         [59.],
         [59.]],

        [[70.],
         [70.],
         [70.],
         [70.]],

        [[ 100.],
         [ 0.],
         [ 0.],
         [ 0.]],

# etc.
</code></pre>
<p><strong>However, my Y_pred is somewhat like this:</strong></p>
<pre><code class="lang-auto"> tensor([[[15.8224],
         [15.8224],
         [15.8224],
         [15.8224]],

        [[16.1654],
         [16.1654],
         [16.1654],
         [16.1654]],

        [[16.2127],
         [16.2127],
         [16.2127],
         [16.2127]],

# etc.
</code></pre>
<p><strong>I have tried numerous different things:</strong></p>
<ul>
<li>Changing the model architecture (different batch size, different number of layers)</li>
<li>Adding dropout and decay parameters</li>
<li>Using epochs and changing the number of epochs when looping over training and test data</li>
<li>Different optimizers (Adam, SGD) with different learning rates</li>
<li>Log transforming my input data</li>
</ul>
<p><a href="https://discuss.pytorch.org/t/lstm-prediction-does-not-change-when-looping-over-test-data/168717">In my (unanswered) previous I give an example of how my input data looks like.</a></p>
<p>I am fairly new with PyTorch and LSTMs so I might do it wrong, but, whatever I change, I keep getting a (near) constant value from the predictions. What am I doing wrong/what should I be doing?</p>

username: vdw
cooked: <aside class="quote no-group" data-username="Planet_Boost" data-post="1" data-topic="168792">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/planet_boost/48/55854_2.png" class="avatar"> Planet_Boost:</div>
<blockquote>
<p><code>torch.Size([24433, 4, 7]</code></p>
</blockquote>
</aside>
<p>If this is your shape of the input, you probably should define your LSTM with <code>batch_first=True</code>.</p>

username: Planet_Boost
cooked: <p><a class="mention" href="/u/vdw">@vdw</a><br>
Thanks for your answer. I’ve changed</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM(input_size, hidden_size)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
</code></pre>
<p>Or should I do more?<br>
The predictions now are different than first, but the same ‘problem’ (same values) does seem to persist:</p>
<pre><code class="lang-auto"># Y (true data):
tensor([[[59.],
         [59.],
         [59.],
         [59.]],

        [[70.],
         [70.],
         [70.],
         [70.]],

        [[ 0.],
         [ 0.],
         [ 0.],
         [ 0.]],

        [[ 0.],
         [ 0.],
         [ 0.],
         [ 0.]],

        [[ 3.],
         [ 3.],
         [ 3.],
         [ 3.]]]) 

# Y_pred (predicted data):
tensor([[[20.2832],
         [17.4102],
         [16.9698],
         [16.9091]],

        [[20.2832],
         [17.4102],
         [16.9698],
         [16.9091]],

        [[20.2832],
         [17.4102],
         [16.9698],
         [16.9091]],

        [[20.2832],
         [17.4102],
         [16.9698],
         [16.9091]],

        [[20.2832],
         [17.4102],
         [16.9698],
         [16.9091]]], grad_fn=&lt;AddBackward0&gt;)

```</code></pre>

username: Planet_Boost
cooked: <p>Coming back to tell that I somewhat solved this by normalizing my input data. I now obtain different predictions for every output. Whether they are good or not is something I have to figure out!</p>

username: Md_Zahidul_Islam
cooked: <p>Your true Y is also same but different values. I am wondering why do you need different preds then?</p>