Time series prediction, LSTM outputs a flat line

username: Aryaman_Pandya
cooked: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> , I seem to have a pretty identical issue while training a LSTM. My network produces a curve with a roughly correct “shape” but off by orders of magnitude in terms of scaling making it look flat when compared to the target output. Continued training doesn’t help, it seems to plateu. Any suggestions?</p>
<p>Code’s pretty simple, but here’s my model class and train function:</p>
<pre><code class="lang-auto">class stockLSTM(nn.Module): 
    def __init__(self, input_dim, hidden_size, output_dim) -&gt; None:
        super(stockLSTM, self).__init__()
        
        self.hidden_size = hidden_size 
        self.lstm = nn.LSTM(input_dim, hidden_size, batch_first=True)
        self.output_dim = output_dim
        self.output_layer = nn.Linear(hidden_size, output_dim)


    def forward(self, x):
        output, (h, _) = self.lstm(x)
        output = self.output_layer(h.squeeze())
        
        return output
    
</code></pre>
<pre><code class="lang-auto">def train(model, train_loader, val_loader,  loss_function, optim, epochs, device, scheduler, start_decay):
    
    losses = [] #group losses for loss visualization 
    
    for epoch in range(epochs):
        running_loss = 0 
        print("Epoch %d / %d" % (epoch+1, epochs))
        print("-"*10)
        if (epoch &gt; start_decay): 
            scheduler.step()
        
        model.train()
        for i, (x, y) in enumerate(train_loader):
            x = x.to(device)
            y = y.to(device)
            y_pred = model(x)
            # print(f"y shape: {y.shape}, y_pred shape: {y_pred.shape}")
            loss = loss_function(y_pred, y) 
            running_loss+=loss.item()
            optim.zero_grad()
            loss.backward() #backprop 
            optim.step() #update weights
  
        losses.append((running_loss / i))
        print("Step: {}/{}, current Epoch loss: {:.4f}".format(i, len(train_loader), (running_loss / i)))

        val_loss = 0
        with torch.no_grad():
            for i, (x,y) in enumerate(val_loader):
                y, x,  = y.to(device), x.to(device) 
                
                y_pred = model(x)
                loss = loss_function(y_pred, y)
                
                val_loss += loss.item()
        
        print("Epoch: {}, validation loss: {:.4f}".format(epoch+1, val_loss/len(val_loader)))
        
    return losses
</code></pre>
<p>Edit: on further poking and proding, it now looks like it is actually just a straight line (not even maintaining the shape as it did initially)</p>
<p>Attaching loss and prediction curves for more context:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd6fdc7c6cca3a7a36023ac3518df5f5587d0118.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd6fdc7c6cca3a7a36023ac3518df5f5587d0118" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6fdc7c6cca3a7a36023ac3518df5f5587d0118_2_632x500.png" alt="image" data-base62-sha1="tjnxFJBlXxtiXBl7XKcvRBxxK52" width="632" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6fdc7c6cca3a7a36023ac3518df5f5587d0118_2_632x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6fdc7c6cca3a7a36023ac3518df5f5587d0118_2_948x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd6fdc7c6cca3a7a36023ac3518df5f5587d0118.png 2x" data-dominant-color="F4F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">952×752 92.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cf489d0d261538a3dd23aceafc7965d86cf72bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/4cf489d0d261538a3dd23aceafc7965d86cf72bc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4cf489d0d261538a3dd23aceafc7965d86cf72bc_2_690x317.png" alt="image" data-base62-sha1="aYMcWnQpcembbmxAxFY88vzwjxy" width="690" height="317" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4cf489d0d261538a3dd23aceafc7965d86cf72bc_2_690x317.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4cf489d0d261538a3dd23aceafc7965d86cf72bc_2_1035x475.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4cf489d0d261538a3dd23aceafc7965d86cf72bc_2_1380x634.png 2x" data-dominant-color="EDF1F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1508×694 29.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Aryaman_Pandya
cooked: <p>Oh wow, just like in <a href="https://discuss.pytorch.org/t/time-series-the-prediction-result-of-lstm-is-approximately-straight-line/98896" class="inline-onebox">【Time Series 】 The prediction result of LSTM is approximately straight line</a>, normalizing my data before running it through the LSTM fixed this issue. Any idea why it works like this with LSTMs, but running the same data through a RNN doesn’t require normalization like this?</p>