Bidirectional LSTM isn't 2x the size of 2 Unidirectional LSTMs?

username: dhruvbird
cooked: <p>I expected a bi-lstm to be 2x the size of 2 uni-lstm layers, but a bi-lstm is somehow a bit more than that. Why is that so? i.e. 561k params v/s 2x215k ~= 430k params.</p>
<p>Code:</p>
<pre><code class="lang-auto">from torch import nn
from torchinfo import summary

bilstm = nn.LSTM(32, 128, 2, batch_first=True, bidirectional=True)
unilstm = nn.LSTM(32, 128, 2, batch_first=True, bidirectional=False)
print(summary(bilstm))
print(summary(unilstm))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
LSTM                                     561,152
=================================================================
Total params: 561,152
Trainable params: 561,152
Non-trainable params: 0
=================================================================
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
LSTM                                     215,040
=================================================================
Total params: 215,040
Trainable params: 215,040
Non-trainable params: 0
=================================================================
</code></pre>

username: vdw
cooked: <p>The problem does not seem to be the number of directions but the number of layers:</p>
<pre><code class="lang-auto">from torch import nn

def get_param_count(model):
    return sum(p.numel() for p in model.parameters())

bilstm = nn.LSTM(32, 128, 1, batch_first=True, bidirectional=True)
unilstm = nn.LSTM(32, 128, 1, batch_first=True, bidirectional=False)

print(get_param_count(bilstm))
print(get_param_count(unilstm))

single_layer = nn.LSTM(32, 128, 1)
double_layer = nn.LSTM(32, 128, 2)
triple_layer = nn.LSTM(32, 128, 3)

print(get_param_count(single_layer))
print(get_param_count(double_layer))
print(get_param_count(triple_layer))
</code></pre>
<p>If both <code>bilstm</code> and <code>unilstm</code> have the same number of layers, then number of parameters is indeed twice for <code>bilstm</code>. However, if you keep the directionality fixed by double/triple the number of layers, the number of parameters does not double/triple but increases faster.</p>
<p>Sorry, I cannot tell for sure why this is.</p>

username: J_Johnson
cooked: <p>This is more generally the case with any RNN that uses bi-directionality.</p>
<p>We can see what is occurring by printing out the parameters per weight tensor, as follows:</p>
<pre><code class="lang-auto">import torch.nn as nn

model = nn.LSTM(20, 50, 3, bias=False)
model2 = nn.LSTM(20, 50, 3, bias=False, bidirectional=True)

for param in model.parameters():
    print(param.size())
print("-----")
for param in model2.parameters():
    print(param.size())
</code></pre>
<p>That should produce:</p>
<pre><code class="lang-auto">torch.Size([200, 20])
torch.Size([200, 50])
torch.Size([200, 50])
torch.Size([200, 50])
torch.Size([200, 50])
torch.Size([200, 50])
-----
torch.Size([200, 20])
torch.Size([200, 50])
torch.Size([200, 20])
torch.Size([200, 50])
torch.Size([200, 100])
torch.Size([200, 50])
torch.Size([200, 100])
torch.Size([200, 50])
torch.Size([200, 100])
torch.Size([200, 50])
torch.Size([200, 100])
torch.Size([200, 50])
</code></pre>
<p>We can then look at the original RNN Bidirectional paper:</p>
<aside class="onebox pdf" data-onebox-src="https://deeplearning.cs.cmu.edu/F20/document/readings/Bidirectional%20Recurrent%20Neural%20Networks.pdf">
  <header class="source">

      <a href="https://deeplearning.cs.cmu.edu/F20/document/readings/Bidirectional%20Recurrent%20Neural%20Networks.pdf" target="_blank" rel="noopener nofollow ugc">deeplearning.cs.cmu.edu</a>
  </header>

  <article class="onebox-body">
    <a href="https://deeplearning.cs.cmu.edu/F20/document/readings/Bidirectional%20Recurrent%20Neural%20Networks.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://deeplearning.cs.cmu.edu/F20/document/readings/Bidirectional%20Recurrent%20Neural%20Networks.pdf" target="_blank" rel="noopener nofollow ugc">Bidirectional%20Recurrent%20Neural%20Networks.pdf</a></h3>

  <p class="filesize">262.26 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>A standard RNN layer looks like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d228400943e2709f3cf89695fad98517bbd24467.png" alt="image" data-base62-sha1="tZ8vgCE1ygLFl7wxMjIpIYZH9QP" width="675" height="217"><br>
With each arrow, we have a set of weights. Normally, in an RNN, the above sizes would only differ in that the dim 0 would be the same as the hidden dim, i.e. (50, 20), (50, 50), etc.</p>
<p>The 1 dim of the tensors conveys the input size into that tensor(it may seem counterintuitive to put the output size as dim 0 and input size as dim 1, but this is done for functionality and speed because of how the matmul operation works, not for our viewing convenience, but I digress).</p>
<p>LSTMs have 4x the size of outputs due to the gating channels involved(i.e. 200 for LSTM vs. 50 for RNN in this example), but we’re not delving into that here. We’ll just use the RNN case, as it applies the same to all types of RNNs(LSTM, GRU, etc.).</p>
<p>So each layer in a unidirectional RNN has the weights giving input into the hidden state. Then an activation function is used to decide whether to keep the old state elementwise or use the new. The second set of weights is used after it goes through the activation. Hence the two weights per layer for a unidirectional RNN.</p>
<p>So now we come to the bidirectional RNN:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1fd182376f4dd8bdc12117424322b2fa25e9292.png" alt="image" data-base62-sha1="tXE2GByXl70ncrI10BtYrXeOI0O" width="623" height="252"></p>
<p>Drawing from the same paper, we see that there are double the arrows vertically, so 1 layer now has 4 sets of weights instead of 2. Each for forward and backward, two applied to the inputs and two applied to the old state.</p>
<p>Further in the paper, Schuster proposed a modified BRNN which looks like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92d528859495f223b22da8abf958270910ad7d23.png" data-download-href="https://discuss.pytorch.org/uploads/default/92d528859495f223b22da8abf958270910ad7d23" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92d528859495f223b22da8abf958270910ad7d23_2_690x243.png" alt="image" data-base62-sha1="kWWoOpIqhJNzqTQ4uivZie6Z5T5" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92d528859495f223b22da8abf958270910ad7d23_2_690x243.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92d528859495f223b22da8abf958270910ad7d23_2_1035x364.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/2/92d528859495f223b22da8abf958270910ad7d23.png 2x" data-dominant-color="E7E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1328×468 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can see what gets passed to later layers as inputs are:</p>
<ol>
<li>The current hidden combined with the last hidden(i.e. torch.cat);</li>
<li>The current hidden combined with the next hidden;</li>
</ol>
<p>Those yield double the hidden size as an input to the set of weights for subsequent layers.</p>

username: dhruvbird
cooked: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> I don’t understand where the additional parameters come from.</p>
<p>From: <a href="https://d2l.ai/chapter_recurrent-modern/bi-rnn.html" class="inline-onebox" rel="noopener nofollow ugc">10.4. Bidirectional Recurrent Neural Networks — Dive into Deep Learning 1.0.3 documentation</a> (which also seems to refer to the same paper)</p>
<blockquote>
<p>Fortunately, a simple technique transforms any unidirectional RNN into a bidirectional RNN (<a href="https://d2l.ai/chapter_references/zreferences.html#id252" rel="noopener nofollow ugc">Schuster and Paliwal, 1997</a>). We simply implement two unidirectional RNN layers chained together in opposite directions and acting on the same input (<a href="https://d2l.ai/chapter_recurrent-modern/bi-rnn.html#fig-birnn" rel="noopener nofollow ugc">Fig. 10.4.1</a>). For the first RNN layer, the first input is �1 and the last input is ��, but for the second RNN layer, the first input is �� and the last input is �1. To produce the output of this bidirectional RNN layer, we simply concatenate together the corresponding outputs of the two underlying unidirectional RNN layers.</p>
</blockquote>
<p>it seems like the forward and reverse direction outputs are simply concatenated together (hence no need for additional parameters here). What am I missing?</p>

username: J_Johnson
cooked: <p>The original paper had proposed two BRNNs. The first is what seems to be described in the paper you mentioned. The second, modified BRNN, seems to be what may be implemented in the PyTorch version.</p>
<p>In the diagrams shown above from the original paper, you can also see 2 arrows into the layer output for the original and 3 arrows into the layer output for the modified.</p>

username: dhruvbird
cooked: <aside class="quote no-group" data-username="J_Johnson" data-post="5" data-topic="193073">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>In the diagrams shown above from the original paper, you can also see 2 arrows into the layer output for the original and 3 arrows into the layer output for the modified.</p>
</blockquote>
</aside>
<aside class="quote no-group">
<blockquote>
<p>We can see what gets passed to later layers as inputs are:</p>
<ol>
<li>The current hidden combined with the last hidden(i.e. torch.cat);</li>
<li>The current hidden combined with the next hidden;</li>
</ol>
</blockquote>
</aside>
<p>Got it! This makes sense - i.e. maybe this is the version implemented in PyTorch.</p>

username: dhruvbird
cooked: <aside class="quote no-group" data-username="J_Johnson" data-post="5" data-topic="193073">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>The second, modified BRNN, seems to be what may be implemented in the PyTorch version.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/j_johnson">@J_Johnson</a> I also found this:</p>
<blockquote>
<p>The output state is the tensor of all the hidden state from each time step in the RNN(LSTM), and the hidden state returned by the RNN(LSTM) is the last hidden state from the last time step from the input sequence. You could check this by collecting all of the hidden states from each step and comparing that to the output state,(provided you are not using pack_padded_sequence).</p>
</blockquote>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm">
  <header class="source">

      <a href="https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1119340/n-virgo" target="_blank" rel="noopener nofollow ugc">
    <img alt="N. Virgo" src="https://i.stack.imgur.com/KiOuP.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm" target="_blank" rel="noopener nofollow ugc">What's the difference between "hidden" and "output" in PyTorch LSTM?</a>
</h4>

<div class="tags">
  <strong>deep-learning, pytorch, lstm, recurrent-neural-network</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1119340/n-virgo" target="_blank" rel="noopener nofollow ugc">
    N. Virgo
  </a>
  on <a href="https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm" target="_blank" rel="noopener nofollow ugc">01:54PM - 17 Jan 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And I recall seeing this behaviour myself - i.e. the last output is the final hidden state. If the one in PyTorch is the 2nd BRNN, then the output and hidden states wouldn’t match up eh?</p>

username: J_Johnson
cooked: <p>The hidden state will still be the forward and backward hidden states combined. So it should be double the size in the bidirectional case.</p>
<p>And that is the case in the docs. The output size is (L, DxHout), where L is length, D is 1 for unidirectional and 2 for bidirectional, and Hout is hidden out:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html</a></p>
<p>The hidden state is of the same size.</p>