【Time Series 】 The prediction result of LSTM is approximately straight line

hpf: <p>Hi,</p>
<p>Recently, I was working on a time series prediction project, using the RNN and LSTM modules of Pytorch.</p>
<p>I have a problem. When I use RNN, the prediction results are acceptable. But when I use LSTM, I get very poor results. 【PS:I use the same data structure, parameter structure, on RNN and LSTM.】</p>
<p>I try to change the amount of data per training, the number of hidden neurons and the number of layers in LSTM, but the predicted results can not fit the real data well.</p>
<p>Here are the prediction results I got：<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/343fe34ee442ec444b49b9f07a6ed678dd022f20.png" alt="image" data-base62-sha1="7sdHSqbkV4ySg6Iy6zOqqSZj1Sg" width="485" height="306"><br>
The above graph shows the RNN prediction results. The <code>green line</code> represents the<code> real data</code>, and the <code>red line</code> represents the<code> prediction result</code>. I am quite satisfied with the result<br>
Next<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/590f46d2fb8cde97507e4fdc4e2b7b6ada9ced73.png" alt="image" data-base62-sha1="cHRdWJnMYc9pjb5qcKsbAVV7Wo3" width="486" height="312"><br>
The above graph shows the LSTM prediction results.The <code>green line</code> represents the<code> real data</code>, and the <code>blue line</code> represents the<code> prediction result</code>. As you can see, the predicted result is almost a straight line.<br>
When I zoom in on the prediction results， I found that the trend was like this.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c51fee768de386a04a1228da16284f4f2404b661.png" alt="image" data-base62-sha1="s7Qtql59ggIdd7wgHASjfGM9tWF" width="488" height="320"><br>
【I don’t know why this is happening? Is there any solution that can help me solve this problem.】<br>
——<br>
Here are the main code snippets for LSTM</p>
<pre><code class="lang-auto">#every time, use the data of three time points  to predict the data of the next one time 
use_data=torch.Size([1790, 1, 3])
back_data=torch.Size([1790, 1, 1])

BATCH_SIZE=1
LR = 0.0003  
EPOCHS = 10  

train_set=data.TensorDataset(use_data,back_data)  
loader = data.DataLoader(dataset=train_set, batch_size=BATCH_SIZE, shuffle=False, num_workers=0) 

class lstm(nn.Module):
    def __init__(self):
        super(lstm, self).__init__()
        self.lstm = nn.LSTM(3,3) 
        self.linear = nn.Linear(3, 1)
     
    def forward(self, x,h):
        y1, h = self.lstm(x,h)
        y3 = self.linear(y1)
        return y3,h

NET = lstm()
optimizer = torch.optim.Adam(NET.parameters(), lr=LR)
loss_func = nn.MSELoss()

h_state = torch.randn(1,1,3) 
c_state=torch.randn(1,1,3)
hx=(h_state,c_state)

**lstm(
  (lstm): LSTM(3, 3)
  (linear): Linear(in_features=3, out_features=1, bias=True)
)



total_loss=[] 
wc_loss_plt=[]

NET.train()
for step in range(EPOCHS):
    wc_loss=[]
    pre=[]
    for i, (batch_x, batch_y) in enumerate(loader):  
        out, hx = NET(batch_x,hx)  

        hx1=hx[0].detach()  
        hx2=hx[1].detach()
        hx=(hx1,hx2)
        
        loss = loss_func(out, batch_y)
        pre.append(out)    ### prediction result
        wc_loss.append(loss.data)  
 
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
    total_loss.append(sum(wc_loss))
</code></pre>

ptrblck: <p>You might need to continue the training. Since the predictions seem to already take the shape of the target the scaling might still need to be adjusted.<br>
How are the training and validation loss behaving? Are both still decreasing or are you seeing a plateau?</p>

hpf: <p>I almost forgot about this question.I still don’t know what caused this problem.<br>
However, I normalized the original data and got a good result.</p>
<p>I try to use the LSTM + Linear structure to train and predict the function y = X,<br>
Only by normalizing x can I get good prediction results</p>
<p>so，I cant understand  why normalization  can be useful？ I think it just a method to shrink data.<br>
Maybe I didn’t find the crux of the problem at all.</p>

ptrblck: <p>Normalization helps the model training in general. E.g. one theoretical point of view is that whitening the data is creating loss surfaces with “round” valleys which accelerates the convergence. I’m pretty sure Bishop explains it nicely in <a href="http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf">Pattern Recognition and Machine Learning</a>.</p>

Aryaman_Pandya: <p>Moving to new issue.</p>

Andrei_Fedorov: <p>I believe you understand that the prediction show on the plot doesnt predict anything in reality ?<br>
Btw… I have success fully trainned bunch of lstm models with different kind of inputs (different features, normilised and not, sqeuence length, labels etc)… and in every scenario it end up with the prediction curve which  looks fancy… but always !!! one step late against real curve.</p>

bo_cheng: <p>why only normalize x can get a good result, what happened if you normlize both x and y?</p>