Predicted future values with LSTM forecasting giving same results

Chathurangi_Shyalika: <p>I’m working on an LSTM model for time-series forecasting.<br>
My dataset has two variables that include sales values for two companies. The dataset has ten rows. I wanted to forecast the 11th row. As the output for both of the variables, I end up getting the same tensor value. Any ideas on what I have missed and how to improve the results?</p>
<p>The inputs tensor, label tensor, output tensor are as follows.</p>
<p>Inputs:</p>
<pre><code class="lang-auto">tensor([[ 1431,  1424,  1483,  ...,  1432,  1334,  1413],
        [11574, 11613, 11671,  ..., 11597, 11628, 11644]])
</code></pre>
<p>Label(11th row values):     <code>tensor([ 1472, 11602])</code></p>
<p>Output what I’m getting is:   <code>tensor([5893.2, 5893.2])</code></p>
<p>My nn.LSTM class is as follows.</p>
<pre><code class="lang-auto">class LSTMnn(L.LightningModule):
  def __init__(self):
    super().__init__()

    self.lstm = nn.LSTM(input_size=1, hidden_size=256, num_layers=16)
    self.lstm1 = nn.LSTM(input_size=256, hidden_size=128, num_layers=16)
    self.lstm2 = nn.LSTM(input_size=128, hidden_size=64, num_layers=10)
    self.lstm3 = nn.LSTM(input_size=64, hidden_size=32, num_layers=10)
    self.lstm4 = nn.LSTM(input_size=32, hidden_size=8, num_layers=10)
    self.linear = nn.Linear(8, 1)
    self.loss_fn = nn.MSELoss()

  def forward(self,input):
    lstm_out, _ = self.lstm(input.view(len(input), 1, -1))
    lstm_out, _ = self.lstm1(lstm_out)
    lstm_out, _ = self.lstm2(lstm_out)
    lstm_out, _ = self.lstm3(lstm_out)
    lstm_out, _ = self.lstm4(lstm_out)
    prediction = self.linear(lstm_out[-1].view(1, -1))
    return prediction

  def configure_optimizers(self): 
    return Adam(self.parameters(), lr=0.1) 

 def training_step(self, batch, batch_idx):
        input_i, label_i = batch
        output_i = self.forward(input_i[0])
        loss = self.loss_fn(output_i, label_i)
        self.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True)
        return loss```</code></pre>

vdw: <p>Can you clarify if your model trains properly, i.e., the training loss goes significantly down. I’m also not sure what you mean by “getting the same tensor value”? It’s always the same interdependent from the input?</p>
<p>Since you’re using <code>view()</code> a couple of time, you may want to <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">check</a> if it does the correct thing.</p>
<p>I’m also not sure about the last <code>view()</code> in general. <code>lstm_out</code> will have a shape of (seq_len, batch_size, 8). This means that <code>lstm_out[-1]</code> will have a shape of <code>(batch_size, 8)</code>. Not sure why you do a <code>view(1,-1)</code> on that as this should return a shape of <code>(1, batch_size*8)</code>. Am I misreading something?</p>
<p>Lastly you have 5 <code>nn.LSTM</code> layers each stack with 10/16 layers. That seems overkill and might take very long to learn properly.</p>

Chathurangi_Shyalika: <aside class="quote no-group" data-username="Chathurangi_Shyalika" data-post="1" data-topic="194464">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chathurangi_shyalika/48/66558_2.png" class="avatar"> Chathurangi_Shyalika:</div>
<blockquote>
<p><code>[11574, 11613, 11671,  ..., 11597, 11628, 11644]</code></p>
</blockquote>
</aside>
<p>Thank you for the reply.<br>
The loss is reducing significantly. I made some changes to the model, where I have reduced the number of LSTM layers and now I have 2 outputs that correspond to the 2 variables. Now my prediction seems different but the values are almost similar.</p>
<p>New prediction:</p>
<pre><code class="lang-auto">tensor([1384.5271, 1384.7784])
</code></pre>
<p>Actual value should be:</p>
<pre><code class="lang-auto">tensor([ 1614., 11757.])
</code></pre>
<p>May I please know if I have done anything wrong with the model?</p>
<pre><code class="lang-auto">class LSTMnn(L.LightningModule):
  def __init__(self):
    super().__init__()

    self.lstm = nn.LSTM(input_size=1, hidden_size=64, num_layers=4)
    self.lstm1 = nn.LSTM(input_size=64, hidden_size=32, num_layers=2)
    self.lstm2 = nn.LSTM(input_size=32, hidden_size=8, num_layers=2)
    self.linear = nn.Linear(8, 2)
    self.loss_fn = nn.MSELoss()

  def forward(self,input):
    lstm_out, _ = self.lstm(input.view(len(input), 1))
    lstm_out, _ = self.lstm1(lstm_out)
    lstm_out, _ = self.lstm2(lstm_out)
    prediction = self.linear(lstm_out[-1])
    return prediction

  def configure_optimizers(self):
    return Adam(self.parameters(), lr=0.1)

  def training_step(self, batch, batch_idx):
        input_i, label_i = batch
        output_i = self.forward(input_i[0])
        loss = self.loss_fn(output_i, label_i)
        self.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True)
        return loss

</code></pre>

vdw: <p>What is the shape if <code>input</code> at the beginning of <code>forward()</code>. According to your initial description each sequence contains of 2d data points, but your first LSTM has <code>input_size=1</code> where I would have expected <code>input_size=2</code>.</p>

Chathurangi_Shyalika: <aside class="quote no-group" data-username="vdw" data-post="4" data-topic="194464">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>What is the shape if <code>input</code> at the beginning of <code>forward()</code></p>
</blockquote>
</aside>
<p><a class="mention" href="/u/vdw">@vdw</a> The shape if <code>input</code> at the beginning of <code>forward()</code> is <code>(sequence_length, batch_size, input_size)</code> which is <code>10,2,1</code>.</p>
<pre><code class="lang-auto">sequence_length is 10 (the length of the series).
batch_size is 2 (the number of sequences in your input tensor).
input_size is 1 (since I'm taking the whole series as one input)
</code></pre>
<p>I have put the first LSTM input as 1 since I’m considering the whole sequence as one series.<br>
Isn’t it a correct way? May I please know how the layer inputs and outputs should be in this case?</p>

vdw: <p>Does that mean that <code>batch_size=2</code> reflects that you have data from 2 companies?</p>
<p>I assume your inputs look like <code>(seq_len=10, batch_size=32, input_size=2)</code>? If you do not treat both companies as 1 data sample, why do you want that your model predicts 2 values?</p>
<p>Still, assuming that your shape is <code>(seq_len=10, batch_size=2, input_size=1)</code>, why do you need to do <code>input.view(len(input), 1)</code>? Why doesn’t this throw an error? What’s the output when you add the line</p>
<pre><code class="lang-auto">print(input.view(len(input), 1).shape)
</code></pre>
<p>at the beginning of your <code>forward()</code> method?</p>

Chathurangi_Shyalika: <aside class="quote no-group" data-username="Chathurangi_Shyalika" data-post="1" data-topic="194464">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chathurangi_shyalika/48/66558_2.png" class="avatar"> Chathurangi_Shyalika:</div>
<blockquote>
<pre><code class="lang-auto">tensor([[ 1431,  1424,  1483,  ...,  1432,  1334,  1413],
        [11574, 11613, 11671,  ..., 11597, 11628, 11644]])
</code></pre>
</blockquote>
</aside>
<p>Yes, I thought batch_size=2, to indicate data from both companies.</p>
<p>When I check the input, I’m getting an error.<br>
<code>RuntimeError: shape '[2, 1]' is invalid for input of size 20</code></p>
<p><a class="mention" href="/u/vdw">@vdw</a> Would you please suggest me an approach I can use here?</p>
<p>UPDATE: So, I did change the input shape to the model as <code>torch.Size([2, 10, 1])</code>, which is <code>number of variables=2(which means 2 companies), sequence length, input variable(which is 0 for company A, 1 for company B)</code>. So, we can have input size=1, right? would this be a correct approach?</p>