BiLSTM : Output & Hidden State Mismatch

username: Daniel_Dsouza
cooked: <p>I’m trying to understand the mechanics of the LSTM in Pytorch and came across something that I believe has been asked &amp; answered before but I have a follow-up.</p>
<p>A simple example is pasted below. I expected the final output to be a concatenation of the h_n contents.</p>
<p>But it seems like only the first half matches. From related posts, I see that I’m probably looking at the “first hidden state of the reverse sequence”.</p>
<p>My question : “Is there a way for me to access the last hidden state of the reverse sequence to verify that my ouput[5] is the concatenation of the hidden states?”</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/bd66ebae810889b9ad3a4fa4a8b91ec31cef051f.png" data-download-href="https://discuss.pytorch.org/uploads/default/bd66ebae810889b9ad3a4fa4a8b91ec31cef051f" title="pytorch_question"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/bd66ebae810889b9ad3a4fa4a8b91ec31cef051f_2_690x382.png" alt="pytorch_question" data-base62-sha1="r1wKoEXkpC0eUgAlje1ndIvI315" width="690" height="382" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/bd66ebae810889b9ad3a4fa4a8b91ec31cef051f_2_690x382.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/bd66ebae810889b9ad3a4fa4a8b91ec31cef051f_2_1035x573.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/bd66ebae810889b9ad3a4fa4a8b91ec31cef051f.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_question</span><span class="informations">1074×596 56.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Have a look at <a class="mention" href="/u/vdw">@vdw</a>’s great visualization and explanation regarding the hidden state and output of an rnn: <a href="https://discuss.pytorch.org/t/clarification-regarding-the-return-of-nn-gru/47363/2">post</a>.</p>
<p>While he explains the usage for <code>bidirectional=False</code>, we can use the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" rel="nofollow noopener">docs</a> to separate the directions and compare the output to the hidden state:</p>
<pre><code class="lang-python">rnn = nn.LSTM(5, 8, 1, bidirectional=True)
h0 = torch.zeros(2*1, 1, 8)
c0 = torch.zeros(2*1, 1, 8)

x = torch.randn(6, 1, 5)
output, (h_n, c_n) = rnn(x, (h0, c0))

# Seperate directions
output = output.view(6, 1, 2, 8) #seq_len, batch, num_directions, hidden_size
h_n = h_n.view(1, 2, 1, 8) # num_layers, num_directions, batch, hidden_size

# Compare directions
output[-1, :, 0] == h_n[:, 0] # forward
output[0, :, 1] == h_n[:, 1] # backward
</code></pre>

username: vdw
cooked: <p>I wondered the same thing in a <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">previous post of mine</a>. As <a class="mention" href="/u/ptrblck">@ptrblck</a> clarified in his answer, the last states of the forward and backward pass are indeed found on the “opposite” ends of <code>output</code>.</p>

username: Daniel_Dsouza
cooked: <p>This was very helpful <a class="mention" href="/u/ptrblck">@ptrblck</a> &amp; <a class="mention" href="/u/vdw">@vdw</a> ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>So the output is basically the collection of the hidden states’ step-wise. I get it now.</p>
<p>So when papers say that they use the bidirectional embedding of a sentence in PyTorch, they mean they use the final "<strong>hn</strong>"i.e “forward_final_hidden_layer, backward_final_hidden_layer” right?</p>
<p>because using the output[-1] is “forward_final_hidden_layer, backward_initial_hidden_layer”, which is incorrect to use as a sentence representation, correct?</p>

username: vdw
cooked: <aside class="quote no-group quote-modified" data-username="Daniel_Dsouza" data-post="4" data-topic="49825">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daniel_dsouza/48/16622_2.png" class="avatar"> Daniel_Dsouza:</div>
<blockquote>
<p>[…] bidirectional embedding of a sentence […] they mean they use the final " <strong>hn</strong></p>
</blockquote>
</aside>
<p>Yes, I would strongly assume so, since you only get the true/full sentence embedding after the last step.</p>
<aside class="quote no-group quote-modified" data-username="Daniel_Dsouza" data-post="4" data-topic="49825">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daniel_dsouza/48/16622_2.png" class="avatar"> Daniel_Dsouza:</div>
<blockquote>
<p>[…] using the output[-1] is “forward_final_hidden_layer, backward_initial_hidden_layer”, which is incorrect to use as a sentence representation, correct?</p>
</blockquote>
</aside>
<p>Yes, at least the way PyTorch implements it.</p>
<p>An alternative, I have occasionally stumbled up it, is to calculate the average or maximum over the whole <code>output</code>; please see the <code>forward</code> method of a simple RNN-based classifier I used for testing the effects. It obviously will work (since the dimensions check out) and even train well, but I’m pretty sure I saw any improvements since I stopped using it. Fell free to try, though. I might work for you setting</p>
<pre><code>def forward(self, batch, method='last_step'):
    embeds = self.word_embeddings(batch)
    output = torch.transpose(embeds, 0, 1)
    output, self.hidden = self.gru(x, self.hidden)

    if method == 'last_step':
        output = output[-1] # don't do that for bidirectional=True :)
    elif method == 'average_pooling':
        output = torch.sum(output, dim=0) / len(batch[0])
    elif method == 'max_pooling':
        output, _ = torch.max(x, dim=0)
    else:
        raise Exception('Unknown method.')
        
    for l in self.linears:
        output = l(output)
    log_probs = F.log_softmax(output, dim=1)
    return log_probs
</code></pre>

username: dhy
cooked: <p>Hi <a class="mention" href="/u/daniel_dsouza">@Daniel_Dsouza</a> (and maybe <a class="mention" href="/u/vdw">@vdw</a>)</p>
<p>I have been wondering about exactly the same problem. So, I understand that we should use <strong>h_n</strong>s for “forward_final_hidden_layer, backward_final_hidden_layer”.</p>
<p>However, I still do not get how you were able to tell ‘the output[-1] is “forward_final_hidden_layer, backward_initial_hidden_layer”’. I tried to find this information on the document, but I failed.</p>
<p>I would really appreciate if someone could kindly explain it to me and also what it means “backward_initial_hidden_layer”. Does this --“backward_initial_hidden_layer”-- mean that we get this during the process of getting “forward_final_hidden_layer”?</p>
<p>Thanks in advance!</p>

username: vdw
cooked: <p>The issue is that in case of a BiLSTM, the notion of “last hidden state” gets a bit murky.</p>
<p>Take for example the sentence “there will be dragons”. And let’s assume you created your LSTM with <code>batch_first=False</code>. Somewhere in your <code>forward()</code> method you have</p>
<pre><code>output, hidden = lstm(inputs, hidden)
</code></pre>
<p>In this case <code>output[-1]</code> gives you the hidden states for the last word (i.e., “dragons”). However, this is only the last state for the forward direction since “dragons” is here the last word. In case of the backwards direction, “dragons” is the first word, so you get the first hidden state w.r.t. to the backwards direction. For the backwards direction the last word is “there”, which is the first word of your sentence. So the last hidden state for the backward direction is somewhere in <code>output[0]</code>. I had the same misunderstanding at first; see a <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">previous post of mine</a>.</p>
<p><code>hidden</code> does not have any sequence dimension and will contain the last hidden state of the forward direction (part of <code>output[-1]</code>) and the last hidden state of the backward direction (part of <code>output[0]</code>). Depending on your exact task, using <code>hidden</code> for the next layers is usually the right way to go. If you use <code>output[-1]</code> you basically loose all the information from the backward pass since you use the hidden state after only one word and not the while sequence.</p>