LSTM - Predicting the same constant values after a while

username: hzzzm
cooked: <p>I build up a module using nn.LSTM to predicate same thing. but it predict same values after 5-10 predictions:</p>
<pre><code class="lang-auto">class NET(nn.Module):
    def __init__(self, input_size=len(train_cols), hidden_size=40, output_size=2, num_layer=10):
        super(NET, self).__init__()
        self.rnn = nn.LSTM(input_size, hidden_size, num_layer)
        self.out = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        out, _ = self.rnn(x)
        out = self.out(out[:, -1, :])
        return out
net = NET(output_size=1)
optimizer = torch.optim.Adam(net.parameters(), lr=0.08, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)
loss_func = torch.nn.MSELoss()
test_x = test_x.reshape(-1, 1, len(train_cols))
test_x = torch.from_numpy(test_x)
test_xv = Variable(test_x)
for epoch in range(100):
    var_x = Variable(train_x).type(torch.FloatTensor)
    var_y = Variable(train_y).type(torch.FloatTensor)
    out = net(var_x)
    loss = loss_func(out.squeeze(), var_y.squeeze())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if (epoch + 1) % 5 == 0:
        print('Epoch: {}, Loss: {:.5f}'.format(epoch + 1, loss.data.numpy()))
        pred_test = net(test_xv)
        pred_test = pred_test.view(-1).data.numpy()
        viz.text(str(np.dstack((pred_test,test_y))), win='compare',
                 opts=dict(title='pred'))
</code></pre>
<p>The prediction is:[[[0.90698224 0.97657084] [1.3066663 0.98598385] [1.4000531 1.003238 ] [1.4139748 1.018212 ] [1.4166933 1.0115894 ] [1.4169703 0.99727374] [1.4168639 0.9989065 ] [1.4166994 0.9569661 ] [1.4165486 0.96585363] [1.4164208 0.96672124] [1.416315 0.9726626 ] [1.4162277 0.9950739 ] [1.4161557 1.0281057 ] [1.4160964 1.0196408 ] [1.4160475 1.0191873 ] [1.4160074 1.0168635 ] [1.4159743 1.0126513 ] [1.4159468 1.004374 ] [1.4159242 1.0187122 ] [1.4159056 1.0370985 ] [1.4158903 1.0679934 ] [1.4158777 1.0439979 ] [1.4158672 1.044638 ] [1.4158586 1.0534846 ] [1.4158515 1.0213561 ] [1.4158456 0.97619045] [1.4158409 0.97762746] [1.4158368 0.9567483 ] [1.4158336 0.93333334] [1.4158307 0.9560899 ]]]<br>
Who can tell me why?</p>

username: Unity05
cooked: <p>Hi <a class="mention" href="/u/hzzzm">@hzzzm</a>,</p>
<p>do you pad your input sequences? If yes, I guess that comes from LSTM’s nature, even tho this might occur more obviously when working with GRUs.<div class="lightbox-wrapper"><a class="lightbox" href="https://colah.github.io/images/post-covers/lstm.png" title="" rel="nofollow noopener"><img src="https://colah.github.io/images/post-covers/lstm.png" alt="" width="643" height="500"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">863×671</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div><br>
When the input becomes <code>0</code> (that’s what padding does with value 0), the added up previous hidden and current input is smaller than ‘normal’ (with ‘normal’ inputs). Therefore, the first sigmoid’s output will be lower and therefore the cell state will forget more. However, that might not even be the biggest issue. By looking at the input gate, we can see, that the zero - centred tanh multiplied with the sigmoid’s output adds less to the cell state than normally. The same goes for the output gate, as the output gate’s sigmoid function outputs lower values than ‘normally’. Therefore, even the hidden state gets closer and closer to zero, as well as the cell state with it. Furthermore, all these operations are accompanied by weights, which are supporting this convergence.</p>
<p>Anybody, please feel free to correct me.</p>
<p>Regards,<br>
Unity05</p>

username: vdw
cooked: <p>Since you define your LSTM with the default parameter <code>batch_first=False</code>, the output has the shape <code>(seq_len, batch, hidden_size)</code>. That means that <code>out[:, -1, :]</code> gives you the values for the hidden states of all the time steps for the last item in your batch, i.e., the output shape is <code>(seq_len, hidden_dim)</code>.</p>
<p>What you want is the last hidden state (“last” w.r.t. to the number of time steps) for all items in your batch. You simple change that line to <code>out[-1]</code>. Just to be sure, and since I don’t know how your data looks like, can you change your <code>forward()</code> method as follows, and post the output of the print statements?</p>
<pre><code class="lang-auto">def forward(self, x):
    print(x.shape)
    out, _ = self.rnn(x)
    print(out.shape)
    out = out[:, -1, :])
    print(out.shape)
    out = self.out(out)
    return out
</code></pre>

username: hzzzm
cooked: <p>Thanks:</p>
<pre><code class="lang-auto">def forward(self, x):
    print('x shape',x.shape)
    out, _ = self.rnn(x)
    print('before reshape',out.shape)
    out =out[:, -1, :]
    print('after reshape',out.shape)
    out = self.out(out)
    print('output shape',out.shape)
    return out
</code></pre>
<p>and the result is:<br>
x shape torch.Size([2227, 1, 14])<br>
before reshape torch.Size([2227, 1, 40])<br>
after reshape torch.Size([2227, 40])<br>
output shape torch.Size([2227, 1])</p>
<pre><code class="lang-auto"></code></pre>

username: vdw
cooked: <p><a class="mention" href="/u/hzzzm">@hzzzm</a> thanks! I assume that 2227 is the sequence length and you only have 1 sequence in your batch, i.e., <code>batch_size = 1</code>. After <code>out = out[:, -1, :]</code>, the shape is <code>(2227, 40)</code> which would suddenly mean a batch size of 2227.</p>
<p><code>out = out[-1]</code> will yield a shape if <code>(1, 40)</code> representing <code>(batch_size, hidden_size)</code> which I strongly assume is what you want.</p>

username: hzzzm
cooked: <p>I have removed the line: out= out[:,-1,:] from my function to keep batch_size. but it is still the same.<br>
My data: A sequence of 2227, it has 14 properties</p>