Output of a GRU layer

username: moorccini
cooked: <p>Hey guys<br>
I defined a GRU layer like this with <code>n_layers = 2</code></p>
<p><code>self.gru = nn.GRU(embed_dim, hidden_dim, n_layers)</code></p>
<p><code> self.fc = nn.Linear(hidden_dim, output_dim)</code></p>
<p>in the forward function, what should I give to the fc layer?</p>
<p><code>output, hidden = self.gru(x)</code></p>
<p><code>fc_input = output[-1]</code></p>
<h3>or</h3>
<p><code>fc_input  = torch.cat((hidden[-1, :, :],  hidden[-2, :, :]), dim = 1)</code></p>
<p><code>return torch.sigmoid(self.fc(fc_input  ))</code></p>
<p>What about having a <code>bidirectional</code> model? what would be the <code>fc_input</code> in that case?<br>
Thank you.</p>

username: mariosasko
cooked: <p>The input to the fully-connected layer should be (in sequence classification tasks) <code>output[-1]</code>. <code>hidden</code> is usually passed to the decoder in seq2seq models.</p>
<p>In case of a bidirectional model, the last dimension of the output is doubled in size so the output shape is <code>(seq_len, batch, 2 * hidden_size)</code>.</p>
<p>To combine these directions (the forward and backward direction) are, some of the options are:</p>
<ul>
<li>they can be summed</li>
<li>affine transformation can be applied to them (they are propageted through a neural net (<code>nn.Linear(2 * hidden_size, next_layer_input_dim</code>), that network is trained at the same time as the rest of the model)</li>
<li>they are left untouched and passed to the next layer (but in this case the input dim of the next layer has to be doubled in size (<code>2 * hidden_dim</code>))</li>
<li>etc.</li>
</ul>

username: vdw
cooked: <aside class="quote no-group" data-username="mariosasko" data-post="2" data-topic="92186">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mariosasko/48/27520_2.png" class="avatar"> mariosasko:</div>
<blockquote>
<p>The input to the fully-connected layer should be (in sequence classification tasks) <code>output[-1]</code> . <code>hidden</code> is usually passed to the decoder in seq2seq models.</p>
</blockquote>
</aside>
<p>In case of BiGRU <code>output[-1]</code> gives you the last hidden state for the forward direction but the first hidden state of the backward direction; <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">see here</a>. If only the last hidden state is fed to a linear layer, it’s therefore more convenient to use <code>hidden</code> and not <code>output</code>. For a BiGRU, I would suggest</p>
<pre><code class="lang-auto">output, hidden = self.gru(x)
# hidden.shape = (n_layers * n_directions, batch_size, hidden_dim)
hidden = hidden.view(n_layers, n_directions, batch_size, hidden_dim)
# This view() comes directly from the PyTorch docs
# hidden.shape = (n_layers, n_directions, batch_size, hidden_dim)
hidden = hidden[-1]
# hidden.shape = (n_directions, batch_size, hidden_dim)
hidden_forward, hidden_backward = hidden[0], hidden[1]
# Both shapes (batch_size, hidden_dim)
fc_input = torch.cat((hidden_forward, hidden_backward), dim=1)
# fc_input.shape = (batch_size, 2*hidden_dim)
</code></pre>
<p><a class="mention" href="/u/moorccini">@moorccini</a> This is very verbose; I like to have my code easy to read :). Note that the problem with your line</p>
<pre><code>fc_input = torch.cat((hidden[-1, :, :], hidden[-2, :, :]), dim = 1)
</code></pre>
<p>is that you don’t account for for the multiple layers in <code>hidden</code>, so <code>hidden[-1, :, :]</code> and <code>hidden[-2, :, :]</code> will give you the wrong tensors, at least in the general case (maybe it’s correct with <code>n_layers=1</code>).</p>

username: moorccini
cooked: <p>Thank you very much, it helped me alot</p>

username: mariosasko
cooked: <p><a class="mention" href="/u/vdw">@vdw</a> Thanks for reminding me. It doesn’t make much sense to combine the last hidden state of the forward pass with the first hidden state of the backward pass in sequence classification tasks.</p>
<p><a class="mention" href="/u/moorccini">@moorccini</a> In the bidirectional case, <code>output[-1, :, :hidden_size]</code> gives you the last hidden state of the forward pass, and <code>output[0, :, hidden_size:]</code> gives you the last hidden state of the backward pass. Then you can concat these states/transform them as desired. But I like <a class="mention" href="/u/vdw">@vdw</a>’s solution better, since it seems more elegant. If you just pass the <code>output[-1]</code> (which has the first hidden state of the backward pass) of the bidirectional model to the next layer, in a sequence classification task, the model performance could be even worse compared to the unidirectional model. The <code>output</code> shape makes the <code>output</code> suitable for sequence labelling, so it can be a bit tricky to get it right in sequence classification tasks.</p>