How to retrieve hidden states for all time steps in LSTM or BiLSTM?

wasiahmad: <p>According to the docs of nn.LSTM outputs:</p>
<pre><code>output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence.
h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
c_n (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len
</code></pre>
<p>If I want to get the hidden states for all t which means t =1, 2, …, seq_len, How can I do that? One approach is looping through an LSTM cell for all the words of a sentence and get the hidden state, cell state and output. I am doing a language modeling task using LSTM where I need the hidden state representations of all the words of a sentence.</p>
<p>Any help would be appreciated!</p>

smth: <p>To get individual hidden states, you have to indeed loop over for each individual timestep and collect the hidden states.</p>

ruotianluo: <p>Isn’t output containing all the hidden states (at least h)?</p>

wasiahmad: <p>Thanks smth, I was also thinking about that. Just one more thing to confirm, in case if I loop over the individual timestep, will it be inefficient in terms of time complexity and will it change the loss computation for the entire network? Though i think it will not.</p>

wasiahmad: <p>No, the output only contains output of the last time step.</p>

ngimel: <p>It won’t change loss computation, but it will likely be much slower.</p>

sinhasam: <aside class="quote no-group quote-modified" data-username="wasiahmad" data-post="6" data-topic="1087" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasiahmad/48/248_2.png" class="avatar"> wasiahmad:</div>
<blockquote>
<p>No, the output only contains output of the last time step.</p>
</blockquote>
</aside>
<p>no, output contains the hidden states for each time step, but only for the last layer in a stacked model, or the only layer in a single layer model.</p>

Abhishek_Arya: <p>Hey <a class="mention" href="/u/wasiahmad">@wasiahmad</a>, <a class="mention" href="/u/smth">@smth</a> , Can you share the code snippet on how you got all the input states?</p>
<p>I’m providing my code. Can you suggest my how should I go on?</p>
<p>Although, it is in torch, I guess the idea should be similar.</p>
<blockquote>
<p>inputSize=5<br>
hiddenSize=5<br>
rho=5<br>
lstm=nn.LSTM(inputSize, hiddenSize,rho)<br>
input=torch.randn(10,5)–I am providing 10*5 input tensor<br>
print(lstm:getHiddenState(1)) --prints the cell state and hidden state for the first time step</p>
</blockquote>

chunzhu: <p>Hi <a class="mention" href="/u/abhishek_arya">@Abhishek_Arya</a>,</p>
<p>nn.LSTM(inputSize, hiddenSize,rho) can return 2 variables instead of 1<br>
lstm, recent_hidden=nn.LSTM(inputSize, hiddenSize,rho)</p>
<p>lstm will contain the whole list of hidden states<br>
while recent_hidden will give u the last hidden state.</p>
<p>print(lstm:getHiddenState(1)) – not sure what function will this be, but if you print out lstm and recent_hidden, it will on the last row of lstm, the values should be the same as recent_hidden.</p>
<p>Hope it helps</p>

hughperkins: <aside class="quote no-group" data-username="wasiahmad" data-post="4" data-topic="1087">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasiahmad/48/248_2.png" class="avatar"> wasiahmad:</div>
<blockquote>
<p>in case if I loop over the individual timestep, will it be inefficient in terms of time complexity</p>
</blockquote>
</aside>
<p>RNNs are inherently sequential. They are auto-regressive, meaning the input for timestep t contains the output for timestep (t-1), meaning you have to first calculae the output for timestap (t-1).</p>
<p>This is one reason why ‘attention is all you need’ is quite interesting.</p>

sinhasam: <p>That is assuming that the LSTM has only one layer. It isn’t the case for a stacked LSTM.</p>

Shawn-Shan: <p>I experiment with using a for loop to collect every hidden state. The loss doesn’t change, but runtime is ~10x more (with a sequence length of 30).</p>
<pre><code class="lang-auto">        embedding = self.drop(self.emb(inputs))
        
        b_len = embedding.size(0)
        outputs = Variable(torch.cuda.FloatTensor(b_len, embedding.size(1), self.hidden_size))
        for i in range(b_len):
            
            cur_emb = embedding[i:i+1, :]#.view(1, embedding.size(1), embedding.size(2))
            o, hidden = self.gru_1(cur_emb, hidden)
            outputs[i, :, :] = o
</code></pre>

KaiyangZhou: <p>Agree with you. I tried looping through each time step in order to get the h and c of all time steps, and the computation is much much much slower than simply using <code>h, _ = self.lstm(x)</code>.</p>
<p>It would be good to have <code>output</code> of <code>nn.LSTM</code> containing both <code>h</code> and <code>c</code> of all timesteps.<img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=5" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

gmyofustc: <p>Why not try GRU, output all the cells of LSTM will cost you huge time with step-by-step LSTMCell</p>

Shawn-Shan: <p>GRU and LSTM are essentially the same here. If you want to collect GRU hidden states, you have to loop through it as well. Only difference is GRU has 3/4 of LSTM parameters</p>

gmyofustc: <p>for a fixed length of sequence, say X=[x1,x2,…,x1000]<br>
if you use<br>
<code>lstm_output, lstm_states = nn.LSTM(X)</code>   it may cost 1 seconds<br>
while, if you use <code>LSTMCell</code> and <code>for loop</code>，it will cost you about 10 seconds, amost <code>10x</code> slower than nn.LSTM.<br>
Here comes the question, you want save each states of all the sequence of LSTM, you have to use <code>nn.LSTMCell</code> and push the <code>lstm_cell_state</code> into a list to save.(<code>nn.LSTM</code> will only return the last cell state out)<br>
If you use GRU, the hidden states of all the sequence is the hidden states, which are just the output of <code>nn.GRU</code><br>
say<br>
<code>gru_outputs,_ = nn.GRU(X)</code><br>
in which<br>
<code>gru_outputs=[h1,h2,...h1000]</code><br>
the gru_outputs is just what you want to retrive</p>

rchavezj: <p>Isn’t the last time step the last hidden state too? Sorry I’m a noob who likes to ask a lot of questions.</p>

Armin_A: <p>As you can see in definition of the LSTM module <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM" rel="noopener nofollow ugc">here</a>, the output will give you the last layer hidden layers in all timesteps.</p>
<p><strong>output</strong> of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the LSTM, for each t.</p>

Salvador_Medina: <aside class="quote no-group" data-username="rchavezj" data-post="17" data-topic="1087">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rchavezj/48/16902_2.png" class="avatar"> rchavezj:</div>
<blockquote>
<p>last time step the last hidden state too? Sorry I’m a noob who likes to ask a lot of questions.</p>
</blockquote>
</aside>
<p>This is not entirely true. The <code>output</code> of the network is different from the <code>hidden state</code> and <code>cell state</code>. The <code>output</code> does have the form of (<code>seq_len</code>, <code>batch_num</code>, <code>num_directions</code> x <code>output_sz</code>) when <code>batch_first=False</code>.</p>
<p>Unfortunately, the hidden and cell states returned are the final state from each layer in the stack, for each sample in the batch:(<code>num_directions</code>x<code>num_layers</code>, <code>batch_sz</code>, <code>output_sz</code>). e.g. during the inference of a batch with 2 input samples into a 5-layer bidir LSTM the hidden/cell states tensors would have a shape of (10, 2, <code>output_dim</code>).</p>