Bidirection RNN hidden states and output

username: scarecrow21
cooked: <p>I wanted to check if my understanding of the BiRNN output is correct.</p>
<p>Following is the code.</p>
<pre><code class="lang-auto">print('\nOutput:\n', out)
print("\nForward Outputs: \n", out[:, :, :HIDDEN_SIZE])
print("\nBackward Outpus: \n", out[:, :, HIDDEN_SIZE:])

print('\nHidden:\n', hidden)
print("\nForward Hidden: \n", hidden[:, :, 0])
print("\nBackward Hidden: \n", hidden[:, :, 1])
</code></pre>
<p>Following is the output.</p>
<pre><code class="lang-auto">Output:
 tensor([[[ 0.0908,  0.5328,  0.6399, -0.4098],
         [ 0.4033,  0.8654,  0.3567, -0.6038],
         [ 0.6768,  0.9732,  0.0298, -0.7668],
         [ 0.8282,  0.9944, -0.2225, -0.8903],
         [ 0.9032,  0.9986, -0.3265, -0.9737]],

        [[ 0.8729,  0.9987, -0.7337, -0.9683],
         [ 0.9645,  0.9999, -0.8254, -0.9868],
         [ 0.9797,  1.0000, -0.8842, -0.9947],
         [ 0.9878,  1.0000, -0.9168, -0.9981],
         [ 0.9926,  1.0000, -0.8730, -0.9998]],

        [[ 0.9890,  1.0000, -0.9665, -0.9997],
         [ 0.9973,  1.0000, -0.9777, -0.9999],
         [ 0.9984,  1.0000, -0.9851, -1.0000],
         [ 0.9990,  1.0000, -0.9899, -1.0000],
         [ 0.9994,  1.0000, -0.9820, -1.0000]],

        [[ 0.9991,  1.0000, -0.9956, -1.0000],
         [ 0.9998,  1.0000, -0.9970, -1.0000],
         [ 0.9999,  1.0000, -0.9980, -1.0000],
         [ 0.9999,  1.0000, -0.9987, -1.0000],
         [ 1.0000,  1.0000, -0.9976, -1.0000]]], grad_fn=&lt;TransposeBackward1&gt;)

Forward Outputs: 
 tensor([[[0.0908, 0.5328],
         [0.4033, 0.8654],
         [0.6768, 0.9732],
         [0.8282, 0.9944],
         [0.9032, 0.9986]],

        [[0.8729, 0.9987],
         [0.9645, 0.9999],
         [0.9797, 1.0000],
         [0.9878, 1.0000],
         [0.9926, 1.0000]],

        [[0.9890, 1.0000],
         [0.9973, 1.0000],
         [0.9984, 1.0000],
         [0.9990, 1.0000],
         [0.9994, 1.0000]],

        [[0.9991, 1.0000],
         [0.9998, 1.0000],
         [0.9999, 1.0000],
         [0.9999, 1.0000],
         [1.0000, 1.0000]]], grad_fn=&lt;SliceBackward&gt;)

Backward Outpus: 
 tensor([[[ 0.6399, -0.4098],
         [ 0.3567, -0.6038],
         [ 0.0298, -0.7668],
         [-0.2225, -0.8903],
         [-0.3265, -0.9737]],

        [[-0.7337, -0.9683],
         [-0.8254, -0.9868],
         [-0.8842, -0.9947],
         [-0.9168, -0.9981],
         [-0.8730, -0.9998]],

        [[-0.9665, -0.9997],
         [-0.9777, -0.9999],
         [-0.9851, -1.0000],
         [-0.9899, -1.0000],
         [-0.9820, -1.0000]],

        [[-0.9956, -1.0000],
         [-0.9970, -1.0000],
         [-0.9980, -1.0000],
         [-0.9987, -1.0000],
         [-0.9976, -1.0000]]], grad_fn=&lt;SliceBackward&gt;)

Hidden:
 tensor([[[ 0.9032,  0.9986],
         [ 0.9926,  1.0000],
         [ 0.9994,  1.0000],
         [ 1.0000,  1.0000]],

        [[ 0.6399, -0.4098],
         [-0.7337, -0.9683],
         [-0.9665, -0.9997],
         [-0.9956, -1.0000]]], grad_fn=&lt;StackBackward&gt;)

Forward Hidden: 
 tensor([[ 0.9032,  0.9926,  0.9994,  1.0000],
        [ 0.6399, -0.7337, -0.9665, -0.9956]], grad_fn=&lt;SelectBackward&gt;)

Backward Hidden: 
 tensor([[ 0.9986,  1.0000,  1.0000,  1.0000],
        [-0.4098, -0.9683, -0.9997, -1.0000]], grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<p>Is my understanding of the BiRNN correct?</p>

username: vdw
cooked: <p>You might want to have a look at  <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">previous post of mine</a>, where I had a similar question.</p>
<p>In general, you need to double-check with the documentation. According to that:</p>
<ul>
<li><code>hidden.shape = (num_layers*num_directions, batch, hidden_size)</code></li>
<li>layers can be separated using <code>h_n.view(num_layers, num_directions, batch, hidden_size)</code>
</li>
</ul>
<p>So after <code>view()</code> the direction dimension is at index 1. So I would say it should look like this:</p>
<pre><code>hidden = hidden.view(num_layers, 2, batch, hidden_size) # 2 for bidirectional
print("\nForward Hidden: \n", hidden[:, 0, :, :])
print("\nBackward Hidden: \n", hidden[:, 1, :, :])
</code></pre>
<p>Similar for the output. According to the documentation:</p>
<ul>
<li><code>output.shape = (seq_len, batch, num_directions*hidden_size)</code></li>
<li>directions can be separated using <code>output.view(seq_len, batch, num_directions, hidden_size)</code>
</li>
</ul>
<p>Here the direction dimension is at index 2, yielding the following:</p>
<pre><code>output = hidden.view(seq_len, batch, 2, hidden_size) # 2 for bidirectional
print("\nForward Outputs: \n", out[:, :, 0, :])
print("\nBackward Outpus: \n", out[:, :, 1, :])
</code></pre>
<p>Your current approach for <code>output</code> might be correct, but first splitting <code>num_directions</code> and <code>hidden_size</code> seems cleaner and easier to understand. Your approach for <code>hidden</code>, however, might be wrong since <code>num_directions</code> is not (part of) the last dimensions of <code>hidden</code>.</p>
<p>I hope that gives at least some pointers.</p>