BiDirectional 3 Layer LSTM Hidden Output

username: TrentBrick
cooked: <p>Can I just confirm as I don’t think it says in the docs that if I have a BiDirectional 3 Layer LSTM and it gives me a hidden output of <code>(6, &lt;batch size&gt;, &lt;hidden_state_size&gt;)</code> then <code>[0,:,:]</code> is 1st layer forward, <code>[1,:,:]</code> is 1st layer backward etc.?</p>
<p>Bonus question, if I feed my LSTM a <code>PackedSequence</code> object, I don’t need to unpack it if I am using the hidden state, not the output for the rest of the network?</p>

username: vdw
cooked: <p>The docs say: h_n of shape <code>(num_layers * num_directions, batch, hidden_size)</code>; the layers can be separated using <code>h_n.view(num_layers, num_directions, batch, hidden_size)</code>. So in the bidirectional case, you have 2 tensors with <code>hidden_size</code> for each direction. I think most people just concatenate them before the next step, e.g., pushing through a <code>Linear(2*hidden_size, output_size)</code>. I think I also saw both tensors being averaged or summed up element-wise.</p>
<p>Regarding your bonus question: I <em>think</em> your right. At least I got essentially the same result when I compared using <code>PackedSequence</code> with variable sequence lengths and padding vs. sequences with the same length and no padding.</p>

username: TrentBrick
cooked: <p>Thanks for your response! I realize I mis-worded (and have since fixed) my question a bit. I said I would get an output tensor of size <code>(3, &lt;batch size&gt;, &lt;hidden_state_size&gt;)</code> but I meant <code>(6, &lt;batch size&gt;, &lt;hidden_state_size&gt;)</code> for a bidirectional 3 layer LSTM.</p>
<p>The question is: Are <code>[0,:,:]</code> and <code>[1,:,:]</code> of the hidden state output the first layer’s hidden forward and backward directions, respectively?</p>
<p>In other words, does the hidden state output (in order): Layer 1 Forward, Layer 1 Backward, Layer 2 Forward, … ?</p>
<p>Because I could see instead the output being  Layer 1 Forward, Layer 2 Forward, …, Layer 1 Backward, Layer 2 Backward … It seems unlikely it would be like this but I wanted to confirm it was not!</p>

username: vdw
cooked: <p>Please see blow my <code>forward()</code> function of GRU/LSTM classifier. In this code <code>h_1</code> and <code>h_2</code> represent the last hidden states for the forward and backward pass in case of a bidirectional RNN. I’m reasonably sure that I’ve read the docs correctly. It’s definitely work just fine. The code has a bunch of comments, but let me know if you have further questions:</p>
<pre><code>def forward(self, X_sorted, X_length_sorted):
    batch_size = X_sorted.shape[0]
    # Push through embedding layer
    X = self.word_embeddings(X_sorted)
    # Transpose (batch_size, seq_len, dim) to (seq_len, batch_size, dim)
    X = torch.transpose(X, 0, 1)
    # Pack padded sequence
    X = nn.utils.rnn.pack_padded_sequence(X, X_length_sorted)
    # Push through RNN layer
    X, hidden = self.rnn(X, self.hidden)
    # Unpack packed sequence (not needed anymore, since hidden state is used)
    #X, output_lengths = nn.utils.rnn.pad_packed_sequence(X)

    if self.rnn_type == 'gru':
        final_state = hidden.view(self.num_layers, self.directions_count, batch_size, self.rnn_hidden_dim)[-1]
    elif self.rnn_type == 'lstm':
        final_state = hidden[0].view(self.num_layers, self.directions_count, batch_size, self.rnn_hidden_dim)[-1]
    else:
        raise Exception('Unknown rnn_type. Valid options: "gru", "lstm"')

    if self.directions_count == 1:
        X = final_state.squeeze()
    elif self.directions_count == 2:
        h_1, h_2 = final_state[0], final_state[1]
        #X = h_1 + h_2                # Add both states (needs different input size for first linear layer)
        X = torch.cat((h_1, h_2), 1)  # Concatenate both states

    # Push trough series of linear layers (incl. nonlinearity &amp; dropout)
    for l in self.linears:
        X = l(X)
    # Calculate and return normalized log probabilities
    log_probs = F.log_softmax(X, dim=1)
    return log_probs</code></pre>

username: TrentBrick
cooked: <p>Thanks for sharing. I guess if I want to be sure I am adding the right things then I should use <code>.view(...)</code> to separate out the layers and directions first before adding them together.</p>
<p>Quick question, why do you call <code>[-1]</code> at the end of the LSTM and GRU <code>.view()</code>. ?</p>

username: vdw
cooked: <p>Yeah, in case of multiple layers and two directions I would first call <code>view(num_layers, num_directions, batch, hidden_size)</code> to separate the hidden state cleanly.</p>
<p>The <code>[-1]</code> is simply to get the hidden state(s) of the <strong>last</strong> layer, given that the shape is <code>(num_layers, num_directions, batch, hidden_size)</code>, i.e., the shape of <code>final_state</code> after that is <code>(num_directions, batch, hidden_size)</code>. If <code>num_directions=1</code> so the shape is <code>(1, batch, hidden_size)</code>, I only need the <code>squeeze()</code>. If <code>num_direction=2</code> so the shape is <code>(2, batch_size, hidden)</code> I can access the two directions with <code>[0]</code> and <code>[1]</code>.</p>
<p>So instead of <code>final_state</code>, a better name would probably be <code>last_layer_hidden_state</code> or something.</p>

username: Dean_Sumner
cooked: <p>Hi Chris</p>
<p>This concatenation and dense network action works great if you only want to use the final layer, hidden state. But how about when you need to reuse all layer hidden-states (both h and c) e.g. when using the 3 layer bidirectional encoder LSTM h_n–&gt; 3-layer <em>unidirectional</em> decoder LSTM?</p>
<p>I assume I would then have to reshape using h_n.view() to get the grouped bidirectional hidden_states per layer, then iterating over each layer, concatenate the fw &amp; bw states, push each concatenated layer through a dense network to get the final hidden_states variable. Another question is whether this is also necessary for the c_state of the LSTM.</p>