How to retrieve hidden states for all time steps in LSTM or BiLSTM?

wasiahmad: According to the docs of nn.LSTM outputs: <SCODE>output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence.
h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
c_n (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len
<ECODE> If I want to get the hidden states for all t which means t =1, 2, …, seq_len, How can I do that? One approach is looping through an LSTM cell for all the words of a sentence and get the hidden state, cell state and output. I am doing a language modeling task using LSTM where I need the hidden state representations of all the words of a sentence. Any help would be appreciated!

smth: To get individual hidden states, you have to indeed loop over for each individual timestep and collect the hidden states.

ruotianluo: Isn’t output containing all the hidden states (at least h)?

wasiahmad: Thanks smth, I was also thinking about that. Just one more thing to confirm, in case if I loop over the individual timestep, will it be inefficient in terms of time complexity and will it change the loss computation for the entire network? Though i think it will not.

wasiahmad: No, the output only contains output of the last time step.

ngimel: It won’t change loss computation, but it will likely be much slower.

sinhasam: no, output contains the hidden states for each time step, but only for the last layer in a stacked model, or the only layer in a single layer model.

Abhishek_Arya: I’m providing my code. Can you suggest my how should I go on? Although, it is in torch, I guess the idea should be similar.

chunzhu: print(lstm:getHiddenState(1)) – not sure what function will this be, but if you print out lstm and recent_hidden, it will on the last row of lstm, the values should be the same as recent_hidden. Hope it helps

hughperkins: RNNs are inherently sequential. They are auto-regressive, meaning the input for timestep t contains the output for timestep (t-1), meaning you have to first calculae the output for timestap (t-1). This is one reason why ‘attention is all you need’ is quite interesting.

sinhasam: That is assuming that the LSTM has only one layer. It isn’t the case for a stacked LSTM.

Shawn-Shan: I experiment with using a for loop to collect every hidden state. The loss doesn’t change, but runtime is ~10x more (with a sequence length of 30). <SCODE>        embedding = self.drop(self.emb(inputs))
        
        b_len = embedding.size(0)
        outputs = Variable(torch.cuda.FloatTensor(b_len, embedding.size(1), self.hidden_size))
        for i in range(b_len):
            
            cur_emb = embedding[i:i+1, :]#.view(1, embedding.size(1), embedding.size(2))
            o, hidden = self.gru_1(cur_emb, hidden)
            outputs[i, :, :] = o
<ECODE>

KaiyangZhou: 

gmyofustc: Why not try GRU, output all the cells of LSTM will cost you huge time with step-by-step LSTMCell

Shawn-Shan: GRU and LSTM are essentially the same here. If you want to collect GRU hidden states, you have to loop through it as well. Only difference is GRU has 3/4 of LSTM parameters

gmyofustc: 

rchavezj: Isn’t the last time step the last hidden state too? Sorry I’m a noob who likes to ask a lot of questions.

Armin_A: 

Salvador_Medina: