Clarification regarding the return of nn.GRU

username: tuktuk
cooked: <p>The pytorch <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-decoder" rel="noopener nofollow ugc">tutorial</a> on seq2seq translation mentions something about the decoder. It says</p>
<blockquote>
<p>In the simplest seq2seq decoder we use only last output of the encoder. This last output is sometimes called the  <em>context vector</em>  as it encodes context from the entire sequence. This context vector is used as the initial hidden state of the decoder.</p>
</blockquote>
<p>The <a href="https://pytorch.org/docs/stable/nn.html#gru" rel="noopener nofollow ugc">documentation</a> of nn.GRU mentions that it returns → <em>output, h_n</em></p>
<p>Which one these (<em>output, h_n</em>) signify <em>context vector</em>?</p>
<p>I am using GRU for classifying sentences between two classes, say between positive and negative class. Which one of (<em>output, h_n</em>) would be useful for this classification?</p>

username: vdw
cooked: <p>Short answer: You can use either.</p>
<p>For a bit more understanding, first check the output of <code>output</code> and <code>h_n</code></p>
<ul>
<li><code>output.shape =  (seq_len, batch, num_directions * hidden_size)</code>, so you get a (context) vector for each item in your sequence</li>
<li><code>h_n.shape = (num_layers * num_directions, batch, hidden_size)</code></li>
</ul>
<p>Now look at the following archtecture – it’s for an LSTM, for a GRU you can simple ignore the <code>c_n</code>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7496a33d835f085d800ee17c0ade05895a89551" title="lstm-output-vs-hidden"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" alt="lstm-output-vs-hidden" data-base62-sha1="x03AlD3QYEjqGqTmkDe4za9w3df" width="583" height="500" data-dominant-color="EDEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm-output-vs-hidden</span><span class="informations">640×548 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the basic case where <code>num_layer=1</code> and <code>bidirectional=False</code>, and you don’t use <code>PackedSequence</code>, then <code>output[-1]=h_n</code>. In other words, <code>h_n</code> is the vector after the last layer and the last sequence item (<code>bidirectional=True</code> makes it more complicated).</p>
<p>Fore Seq2Seq, most people use <code>h_n</code> as context vector since the encoder and decoder often have symmetric architectures (e.g., same number of layers), so they can simple copy <code>h_n</code> between encoder to the decoder.</p>
<p>For classification, <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="noopener nofollow ugc">I usually use <code>h_n</code> instead of <code>output</code></a>. The reasons is – as far as I understand – then when you have batches with sequences of different lengths and padding, and use <code>PackedSequence</code>, <code>output[-1] != h_n</code>. It still works, of course, but I experience less accuracy.</p>

username: tuktuk
cooked: <p>Hi <a class="mention" href="/u/vdw">@vdw</a> ,</p>
<p>Thank you very much for the amazing explanation and link to your code. Everything is clear now. I used both and both gave more or less same accuracy with h_n being marginally better.</p>
<p>Thanks again</p>

username: vdw
cooked: <p>Great! I’m happy to help.</p>
<p>As an additional note: Once you use <code>bidirectional=True</code> you almost certainly want to always use <code>h_n</code>, since the last output of the reverse direction is in <code>output[0]</code> and not in <code>output[-1]</code>; see <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">this older post</a> of mine with the edit. <code>h_n</code> will give you the last outputs for both directions much more conveniently.</p>