H_0 and c_0 Inputs for LSTMs Clarification

kevinMEH: <p>Hello,</p>
<p>In the documentation for torch.nn.LSTM, the hidden state and cell state inputs h_0 and c_0 are described as “containing the initial hidden state <strong>for each element in the input sequence</strong>.”</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8c22f13dab5ca2677aeeedadb3df71cf07f441d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d8c22f13dab5ca2677aeeedadb3df71cf07f441d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8c22f13dab5ca2677aeeedadb3df71cf07f441d.png" alt="image" data-base62-sha1="uVxaA9RkyutHADdjZqTw2gtSS6N" width="690" height="128" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">753×140 18.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t think this should be correct, as a user would expect h_0 and c_0 to be used for only for the initial LSTM states, so for the first element in the input sequence rather than for each element in the input sequence.</p>
<p>This is the default behaviors for RNNs and GRUs, as you can see from their descriptions for the hidden state h_0, which is “containing the initial hidden state <strong>for the input sequence</strong>”</p>
<p>So is it true that the same values of h_0 and c_0 will be used repeatedly as the hidden state and cell state for every single element in the sequence for a LSTM? Or is this just incorrect information on the LSTM documentation?</p>
<p>Thanks in advance.</p>

vdw: <p>I’m not quite sure if I understood your issue correctly. Yes, <code>h_0</code> and <code>c_0</code> form the initial hidden state (strictly speaking the hidden and cell state) of the LSTM layer.</p>
<p>However, the processing works like this:</p>
<ul>
<li>1st element: use <code>h_0</code> and element 1 to generate <code>h_1</code></li>
<li>2nd element: use <code>h_1</code> and element 2 to generate <code>h_2</code></li>
<li>3rd element: use <code>h_2</code> and element 3 to generate <code>h_3</code></li>
<li>…</li>
<li>n-th element: use <code>h_(n-1)</code> and element n to generate <code>h_n</code></li>
</ul>
<p>In short, the hidden state gets updated after each element, just not overwritten but a new hidden state is generated. And at the end, you have access to all hidden states <code>h_1</code>, <code>h_2</code>, …, <code>h_n</code>. For example, if you have in your code a line like:</p>
<pre><code class="lang-auto">out, (h, c) = lstm(inputs)
</code></pre>
<p><code>h</code> will be <code>h_n</code>, i.e., the last hidden state – or all last hidden states if you have multiple layers – and <code>out</code> will contain all hidden states, i.e., <code>h_1</code>, <code>h_2</code>, …, <code>h_n</code> but only for the last layer.</p>
<p>Does this clarify things?</p>

kevinMEH: <p>As I suspected, this means that h_0 and c_0 are not used for <strong>EACH element in the input sequence.</strong> Rather, the hidden and cell states from the previous element in the sequence is used.</p>
<p>I was just trying to clarify that LSTMs in Torch indeed works this way as the documentation suggested otherwise, unless I am grossly misinterpreting the meaning of the words “containing the initial hidden state <strong>for each element in the input sequence</strong>”.</p>