LSTM hidden state logic

username: leolcling
cooked: <p>Hi,</p>
<p>I am a bit confused about hidden state in LSTM. I am reading this <a href="http://seba1511.net/tutorials/beginner/nlp/sequence_models_tutorial.html#example-an-lstm-for-part-of-speech-tagging" rel="nofollow noopener">tutorial</a>, and in the <code>forward</code> method of the model, <code>self.hidden</code> is used as inputs h_0.</p>
<pre><code class="lang-auto">class LSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        # Before we've done anything, we dont have any hidden state.
        # Refer to the Pytorch documentation to see exactly
        # why they have this dimensionality.
        # The axes semantics are (num_layers, minibatch_size, hidden_dim)
        return (autograd.Variable(torch.zeros(1, 1, self.hidden_dim)),
                autograd.Variable(torch.zeros(1, 1, self.hidden_dim)))

    def forward(self, sentence):
        embeds = self.word_embeddings(sentence)
        lstm_out, self.hidden = self.lstm(
            embeds.view(len(sentence), 1, -1), self.hidden)  # &lt;- here self.hidden is used as h_0
        tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))  
        tag_scores = F.log_softmax(tag_space)
        return tag_scores
</code></pre>
<p>Does it mean we are retaining the hidden states for each batch (not timesteps)? Why would one want to do that?<br>
If I want to initialize hidden state (e.g. randomly) but not retaining it for different batch, how should I do it? Thanks.</p>

username: engelnico
cooked: <blockquote>
<p>Does it mean we are retaining the hidden states for each batch (not timesteps)? Why would one want to do that?</p>
</blockquote>
<p>Yes exactly. Think of the hidden states as the learned weights. If you reset the hidden states after each batch, your network is essentially learning nothing. The hidden states control the gates (input, forget, output) of the LSTM and they carry information about what the network has seen so far. Therefore, your output depends not only on the most recent input, but also data it has seen in the past. This is the whole idea of the LSTM, it “removes” the long-term dependency problem. Read this excellent blog post for further information:<br>
<a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://colah.github.io/posts/2015-08-Understanding-LSTMs/</a></p>
<blockquote>
<p>If I want to initialize hidden state (e.g. randomly) but not retaining it for different batch, how should I do it?</p>
</blockquote>
<p>Well, if you really want to, don’t save the hidden states as a class variable and instead initialiaze them after every batch. This could be done, e.g. by supplying them to your forward function as additional input, and in your train for loop you initialize them after every iteration. Again, I don’t recommend doing that because it would destroy the purpose of an LSTM.</p>

username: leolcling
cooked: <p>Thank you <a class="mention" href="/u/engelnico">@engelnico</a>. I think I am still confused on the reason we want to retain hidden state for different batch. Mainly because I read this <a href="https://discuss.pytorch.org/t/when-to-call-init-hidden-for-rnn/11518/7:">https://discuss.pytorch.org/t/when-to-call-init-hidden-for-rnn/11518/7:</a></p>
<blockquote>
<p>I think you should call</p>
<pre><code class="lang-auto">hidden = net.init_hidden(batch_size)
</code></pre>
<p>for every batch because, the hidden state after a batch pass contains information<br>
about the whole previous batch. At test time you’d only have a new hidden state for every sentence so you probably want to train for that.</p>
</blockquote>
<p>The argument in this comment is since we will use initial hidden states (say 0) in test time, we would want the model to learn that by implicitly resetting the hidden states for each samples (so the hidden states only pass across time, but not samples). Am I missing anything here? Thanks.</p>

username: engelnico
cooked: <p>I think i misunderstood your first question, I’m sorry…<br>
There are usually two different modes for LSTM, stateless and stateful.</p>
<p><strong>Stateless Mode</strong> updates the parameters for one batch and when the next batch comes, it will initialize the states again (with zeros). Therefore, the “cell memory” is reset after every batch.</p>
<p><strong>Stateful Mode</strong> is the one I described above. The final states of the first batch is set to the initial state in the next batch. So it memorizes what was learned before</p>
<p>It is up to you and most importantly the type of data you use, which mode is the right one for you. The question you have to ask: Is your data time dependent across different batches? If yes -&gt; Stateful, if not -&gt; Stateless. And from your initial post, I guess it is not time dependent, so stateless mode is probably the right one for you.</p>
<p>Again sorry for the confusion</p>

username: leolcling
cooked: <p>Thank you <a class="mention" href="/u/engelnico">@engelnico</a>. Do you also know the behavior if I do not specify the hidden states (it will set to zero by default)? Is it stateful or stateless?</p>

username: engelnico
cooked: <p>Yes, if for the input (h_0, c_0) is not provided, both  <strong>h_0</strong>  and  <strong>c_0</strong>  default to zero. (source: documentation)</p>

username: Rui_Ferreira
cooked: <p>Hi everyone, I have a question regarding the hidden and cell states. Do we initialize hidden state of LSTM during validation or testing or use the weights learned during training? (<a href="https://discuss.pytorch.org/t/do-we-initialize-hidden-state-of-lstm-during-validation-or-testing-or-use-the-weights-learned-during-training/64120" class="inline-onebox">Do we initialize hidden state of LSTM during validation or testing or use the weights learned during training?</a>)</p>