In LSTM which layer should I use as output?

username: DonghunP
cooked: <p>Hi.</p>
<p>I recently refer two good reference. But I confused that output layer of these two is different.</p>
<p>First reference return ‘output’ term <a href="https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/" rel="noopener nofollow ugc">time-series-prediction-using-lstm</a></p>
<pre><code class="lang-auto">        self.hidden_cell = (torch.zeros(1,1,self.hidden_layer_size), torch.zeros(1,1,self.hidden_layer_size))
    def forward(self, input_seq):
        lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq) ,1, -1), self.hidden_cell)
        predictions = self.linear(lstm_out.view(len(input_seq), -1))
        return predictions[-1]
       # Size
       # lstm_out          : [Seq_len(12), batch_size(1), 100]
       # predictions      : [Seq_len(12), 1]
       # predictions[-1] : (1,)
</code></pre>
<p>But,</p>
<p>Second reference return ‘hidden_cell’ term <a href="https://colab.research.google.com/github/dlmacedo/starter-academic/blob/master/content/courses/deeplearning/notebooks/pytorch/Time_Series_Prediction_with_LSTM_Using_PyTorch.ipynb" rel="noopener nofollow ugc">Time_Series_Prediction_with_LSTM</a></p>
<pre><code class="lang-auto">    def forward(self, x):
        h_0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))
        c_0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))

        # Propagate input through LSTM
        ula, (h_out, _) = self.lstm(x, (h_0, c_0))
        h_out = h_out.view(-1, self.hidden_size) 
        out = self.fc(h_out) 
        return out
</code></pre>
<p>What is difference between these two?</p>
<p>Thanks ahead.</p>
<hr>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">torch.nn.LSTM</a></p>
<blockquote>
<blockquote>
<blockquote>
<p>Input<br>
<strong>input (seq_len, batch, input_size)</strong><br>
<strong>h_0 (num_layers * num_directions, batch, hidden_size)</strong><br>
<strong>c_0 (num_layers * num_directions, batch, hidden_size)</strong><br>
output<br>
<strong>output(seq_len, batch, num_directions * hidden_size)</strong><br>
<strong>h_n(num_layers * num_directions, batch, hidden_size)</strong><br>
<strong>c_n(num_layers * num_directions, batch, hidden_size)</strong></p>
</blockquote>
</blockquote>
</blockquote>

username: vdw
cooked: <p>As you can see from the documentation, <code>lstm_out</code> and <code>ula</code> the two forward methods contain the <strong>last</strong> hidden states for <strong>all</strong> time steps (i.e., all items in your sequence). Note that “last” refers to the hidden state with respect to the number of layers and <em>not</em> with respect to the number if time steps.</p>
<p>In contrast, <code>h_out</code> (or <code>self.hidden_cell[0]</code>) refers to the <strong>last</strong> hidden states with respect to the number of time steps. It includes the last hidden states for all layers in case <code>num_layers &gt; 1</code>.</p>
<p>No solution is fundamentally correct or wrong, I would argue that the second using <code>h_out</code> is more common for basic time series prediction. Strictly speaking, I don’t like both implementation since the use <code>view()</code> in a way that can <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">quickly cause issues</a>.</p>
<p>Here is what I would do:</p>
<pre><code class="lang-auto">    def forward(self, x):
        h_0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))
        c_0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))

        # Propagate input through LSTM
        ula, (h_out, _) = self.lstm(x, (h_0, c_0))
        #  Split num_layers and num_directions (useful if you LSTM is bidirectional)
        # This view is directly taken from the docs
        h_out = h_out.view(self.num_layers, self.num_directions, self.batch, self.hidden_size)
        # Get the last layer with respect to num_layers
        h_out = h_out[-1]
        # Handle num_directions dimension (I assume here that bidirectional=False)
        h_out = h_out.squeeze(0)
        # Now the shape of h_out is (batch, hidden_size)
        out = self.fc(h_out) 
        return out
</code></pre>

username: DonghunP
cooked: <p>Thank you Chris. I totally understand.<br>
I get help from you many times.</p>
<p>Have a good day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>