Initialization of the hidden states of torch.nn.lstm

Mingming_Qiu: <p>As the explanations of num_layers in this link: <a href="https://discuss.pytorch.org/t/what-is-num-layers-in-rnn-module/9843">https://discuss.pytorch.org/t/what-is-num-layers-in-rnn-module/9843</a></p>
<p>if the output of hidden state of the first lstm is the input of the hidden state of the second lstm (number_layers=2 for torch.nn.lstm), why do we need to initialize the hidden state with the first dimension (representing the number of hidden states I suppose) being the num_layers? For example, the code below:</p>
<pre><code class="lang-auto">rnn = nn.LSTM(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(input, (h0, c0))
</code></pre>

Mingming_Qiu: <p>Each lstm layer needs the input, hidden and cell states. The first lstm layer provides its output denoting the hidden state to the input of the second lstm, while the second lstm still needs its hidden and cell state values. Therefore, there are two initial hidden states being defined at the beginning.</p>

Lei_Shi1: <p>Hi Qiu,<br>
I have the same confusion as yours. But I think the output of the 1st layer, i.e h1, is the initial hidden state rather than input of the 2nd layer. And the input of 2nd layer has nothing with hidden state values.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73cfcfff9fd7df0bb36b2e1264ff04ebe2304719.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/73cfcfff9fd7df0bb36b2e1264ff04ebe2304719" title="lstm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73cfcfff9fd7df0bb36b2e1264ff04ebe2304719_2_625x500.jpeg" alt="lstm" data-base62-sha1="gww90B9POiP2mIS4Zc6kkillHsZ" width="625" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73cfcfff9fd7df0bb36b2e1264ff04ebe2304719_2_625x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73cfcfff9fd7df0bb36b2e1264ff04ebe2304719_2_937x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73cfcfff9fd7df0bb36b2e1264ff04ebe2304719_2_1250x1000.jpeg 2x" data-dominant-color="E7F0E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm</span><span class="informations">1400×1120 141 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mingming_Qiu: <p>I think the example that I have given does not correspond to the figures that you’ve provided where at each time step there is only one lstm cell, but corresponds to the situation where there are two lstm cells for a lstm. As shown in the figure below, the two initial hidden states are respectively defined for the lstm cell in yellow and red in the first column which corresponds to the initialization.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67a331a3ffe9b28c6a10382dca668e31892956a1.png" data-download-href="https://discuss.pytorch.org/uploads/default/67a331a3ffe9b28c6a10382dca668e31892956a1" title="fb98eb0d16b722e019db59f97825aa529cb6bc08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67a331a3ffe9b28c6a10382dca668e31892956a1_2_685x499.png" alt="fb98eb0d16b722e019db59f97825aa529cb6bc08" data-base62-sha1="eMOOT7YoDyCta4DgDL7td9SqiK5" width="685" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67a331a3ffe9b28c6a10382dca668e31892956a1_2_685x499.png, https://discuss.pytorch.org/uploads/default/original/3X/6/7/67a331a3ffe9b28c6a10382dca668e31892956a1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/7/67a331a3ffe9b28c6a10382dca668e31892956a1.png 2x" data-dominant-color="F3F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fb98eb0d16b722e019db59f97825aa529cb6bc08</span><span class="informations">806×588 93.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>