Retrieving hidden and cell states from LSTM in a Language model

kmario23: <p>What is the order of the hidden and cell states in the tuple that is returned by LSTM? Particularly, in the word LM model, in <code>generate.py</code> , we have:</p>
<pre><code>with open(args.outf, 'w') as outf:
    with torch.no_grad():  # no tracking history
        for i in range(args.words):
            output, hidden = model(input, hidden)
            ....
            ......
</code></pre>
<p>In the above code, <code>hidden</code> is a tuple with two tensors, each of same shape. Since this is an LSTM, it returns both <code>h</code> and <code>c</code> states. But, how is <code>h</code> and <code>c</code> ordered in the tuple?</p>
<p>My understanding is this:</p>
<pre><code>h, c = hidden
</code></pre>
<p>Is this correct? Or is it the other way around?</p>
<p>Thanks!</p>

Lewis: <p>It’s correct if it’s not an LSTM Cell. If you’r using <code>nn.LSTM</code>, then this would return <code>output, (h_n, c_n)</code>, however if your using <code>nn.LSTMCell</code>, then this would return <code>h_1, c_1</code>. The order is the hidden state and then the cell state. So in your case, <code>hidden</code> would be equal to <code>(h_n, c_n)</code>.</p>

kmario23: <p>Thank you!</p>
<p>Yes, I’m using <strong><code>nn.LSTM</code></strong> with <strong>2 hidden</strong> layers (unidirectional). So, I also want to ask about the shape of <strong><code>h_n</code></strong> specifically in this case. For example, I get the shape of <strong><code>h_n</code></strong> as <strong><code>torch.Size([2, 1, 1500])</code></strong></p>
<p>If I understand it correctly, I can get the hidden states of the hidden layers by indexing into them in the order:</p>
<pre><code>first_hidden_layer_hidden_state = h_n[0]    # torch.Size([1, 1500])
second_hidden_layer_hidden_state = h_n[-1]    # torch.Size([1, 1500])
</code></pre>
<p>Is this a correct way of indexing and getting the hidden states?</p>
<p>P.S.: I’m just interested only in these hidden states because I want to pass them to a downstream network for further processing.</p>

Lewis: <p>Hi. The outputs for the LSTM is shown in the attached figure. <img src="https://discuss.pytorch.org/uploads/default/original/2X/0/009ff7dffee641a6dfba44bd7f8e47df4064506f.png" alt="Capture" data-base62-sha1="5wJhJwoEcwwx7mmwfpix4Sni35" width="447" height="156"></p>
<p>The output shape for <code>h_n</code> would be <code>(num_layers * num_directions, batch, hidden_size)</code>. This is basically the output for the <strong>last timestep</strong>. Your output is <code>(2,1,1500)</code> so you are using 2 layers*1 (unidirectional) , 1 sample and a hidden size of 1500).<br>
Now the LSTM would return for you <code>output</code>, <code>(h_n, c_n)</code>. In your case, <code>(h_n, c_n)</code> is named <code>hidden.</code> So by indexing hidden you can extract the <code>h_n</code> and <code>c_n</code> (i.e <code>hidden[0] = h_n</code> and <code>hidden[1] = c_n</code>)</p>

kmario23: <p>Thank you for clear explanation. It was very helpful! I just want to confirm one point again:</p>
<p>Since I’m using 2 layer LSTM network, I want to extract hidden state of the last hidden layer. Is the following way of indexing correct way to do:</p>
<blockquote>
<p>first_hidden_layer_hidden_state = h_n[0]    # torch.Size([1, 1500])<br>
second_hidden_layer_hidden_state = h_n[-1]    # torch.Size([1, 1500])</p>
</blockquote>
<p>If the above is correct, I’m interested in <code>second_hidden_layer_hidden_state</code>.</p>
<hr>
<p>Also, is it analogous for the cell state as well?</p>
<blockquote>
<p>first_hidden_layer_cell_state = c_n[0]    # torch.Size([1, 1500])<br>
second_hidden_layer_cell_state = c_n[-1]    # torch.Size([1, 1500])</p>
</blockquote>
<p>Thank you!</p>

kmario23: <p>To retrieve the hidden or cell states of the last (or any of the) hidden layers, the <a href="https://stackoverflow.com/a/48305882" rel="noopener nofollow ugc">diagram in whats-the-difference-between-hidden-and-output-in-pytorch-lstm </a>  is very helpful and makes it easy to understand.</p>
<p>In short, the number ranges from [0, 1, 2, 3, …, n] for [hidden_layer_1, hidden_layer_2, hidden_layer_3, …, hidden_layer_n] for <code>n</code> hidden layers.</p>
<p>So, with two hidden layers, the <code>hidden</code>  and <code>cell states</code> of the second (or last) hidden layer can be extracted using:</p>
<aside class="quote no-group" data-username="kmario23" data-post="5" data-topic="34989">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kmario23/48/74803_2.png" class="avatar"> kmario23:</div>
<blockquote>
<p>first_hidden_layer_hidden_state = h_n[0] # torch.Size([1, 1500])<br>
second_hidden_layer_hidden_state = h_n[-1] # torch.Size([1, 1500])</p>
</blockquote>
</aside>
<p>analogously for the <code>cell state</code>:</p>
<aside class="quote no-group" data-username="kmario23" data-post="5" data-topic="34989">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kmario23/48/74803_2.png" class="avatar"> kmario23:</div>
<blockquote>
<p>first_hidden_layer_cell_state = c_n[0] # torch.Size([1, 1500])<br>
second_hidden_layer_cell_state = c_n[-1] # torch.Size([1, 1500])</p>
</blockquote>
</aside>