What is num_layers in RNN module?

username: jef
cooked: <p>Hi, I am not sure about <code>num_layers</code> in RNN module. To be clarify, could you check whether my understanding is right or not. I uploaded an image when <code>num_layers==2</code>. In my understanding, <code>num_layers</code> is similar to CNN’s <code>out_channels</code>. It is just a RNN layer with different filters (So we can train different <code>weights variable</code> for outputting <code>h</code> ). Right?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d8688e43375fd7a246b5d64047b64393d80983be.png" data-download-href="https://discuss.pytorch.org/uploads/default/d8688e43375fd7a246b5d64047b64393d80983be" title="01 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d8688e43375fd7a246b5d64047b64393d80983be_2_690x446.png" alt="01 AM" data-base62-sha1="uSr8SsoBR53q9PJY53FFZrt9vdQ" width="690" height="446" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d8688e43375fd7a246b5d64047b64393d80983be_2_690x446.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d8688e43375fd7a246b5d64047b64393d80983be_2_1035x669.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/d8688e43375fd7a246b5d64047b64393d80983be.png 2x" data-dominant-color="EDEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">01 AM</span><span class="informations">1252×810 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am probably right…</p>
<pre><code class="lang-auto">class TestLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(TestLSTM, self).__init__()
        self.rnn = nn.LSTM(input_size, hidden_size, num_layers, batch_first=False)
        
    def forward(self, x, h, c):
        out = self.rnn(x, (h, c))
        return out

bs = 10
seq_len = 7
input_size = 28
hidden_size = 50
num_layers = 2

test_lstm = TestLSTM(input_size, hidden_size, num_layers)
print(test_lstm)

input = Variable(torch.randn(seq_len, bs, input_size))
h0 = Variable(torch.randn(num_layers, bs, hidden_size))
c0 = Variable(torch.randn(num_layers, bs, hidden_size))
output, h = test_lstm(input, h0, c0)
print('output', output.size())
print('h and c', h[0].size(), h[1].size())

---
TestLSTM (
  (rnn): LSTM(28, 50, num_layers=2)
)
output torch.Size([7, 10, 50])
h and c torch.Size([2, 10, 50]) torch.Size([2, 10, 50])
</code></pre>

username: SimonW
cooked: <p>No, your understanding is wrong. num_layers in RNN is just stacking RNNs on top of each other. So you get a hidden from each layer and an output only from the topmost layer.</p>

username: jef
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fb98eb0d16b722e019db59f97825aa529cb6bc08.png" data-download-href="https://discuss.pytorch.org/uploads/default/fb98eb0d16b722e019db59f97825aa529cb6bc08" title="49 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fb98eb0d16b722e019db59f97825aa529cb6bc08_2_685x499.png" alt="49 PM" data-base62-sha1="zTJtbylCMbAOjuOAqgdgNnO05Li" width="685" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fb98eb0d16b722e019db59f97825aa529cb6bc08_2_685x499.png, https://discuss.pytorch.org/uploads/default/original/2X/f/fb98eb0d16b722e019db59f97825aa529cb6bc08.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/fb98eb0d16b722e019db59f97825aa529cb6bc08.png 2x" data-dominant-color="F3F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">49 PM</span><span class="informations">806×588 100 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I found a nice image. Does this mean <code>num_layers==2</code>? And we can get last hidden. Right?</p>

username: SimonW
cooked: <p>Yes!! That is correct.</p>

username: jef
cooked: <p>Thank your for your help!</p>

username: Hengck
cooked: <p>I have two questions:</p>
<ol>
<li>
<p>Consider:<br>
self.lstm1 = nn.LSTM(input_dim, hidden_dim, num_layers=1)<br>
self.lstm2 = nn.LSTM(input_dim, hidden_dim, num_layers=2)</p>
<p>Why are the weights the same values?  Are the weights reused?lstm1.weight_ih_l0.size() == lstm2.weight_ih_l0.size()</p>
</li>
</ol>
<ol start="2">
<li>
<p>self.lstm1a = nn.LSTM(input_dim, hidden_dim, num_layers=1)<br>
self.lstm1b = nn.LSTM(hidden_dim, hidden_dim, num_layers=1)<br>
self.lstm2= nn.LSTM(input_dim, hidden_dim, num_layers=2)</p>
<p>y2 = self.lstm2(x, …)<br>
y1 = self.lstm1b(self.lstm1a(x, …),…)</p>
<p>Are y1,y2 the same thing?</p>
</li>
</ol>

username: SimonW
cooked: <ol>
<li>They are definitely not same values:</li>
</ol>
<pre><code class="lang-auto">&gt;&gt;&gt; lstm1 = nn.LSTM(input_dim, hidden_dim, num_layers=1)
&gt;&gt;&gt; lstm2 = nn.LSTM(input_dim, hidden_dim, num_layers=2)
&gt;&gt;&gt; lstm1.weight_ih_l0
Parameter containing:
-0.3027 -0.2689 -0.3551
 0.5509  0.1728  0.0360
-0.1964  0.1770  0.2209
-0.4915  0.3696  0.5712
 0.2401  0.0593 -0.4117
 0.4066  0.3684  0.3482
 0.2870 -0.0531  0.1953
 0.0928 -0.4165  0.5613
-0.4697  0.4112  0.1346
 0.3438 -0.1885  0.5242
 0.3756  0.2288  0.2949
-0.1401  0.0173 -0.0247
[torch.FloatTensor of size 12x3]

&gt;&gt;&gt; lstm2.weight_ih_l0
Parameter containing:
-0.3672 -0.0299  0.1597
 0.0828 -0.2755  0.4451
 0.1861  0.1213 -0.5596
-0.2776 -0.4791 -0.2322
-0.5063  0.0437  0.1145
-0.2652 -0.0932  0.0865
-0.3323  0.4274 -0.3038
-0.1449 -0.1430  0.5393
 0.5589  0.1293 -0.5174
-0.4502  0.5351  0.2430
-0.5448 -0.4007 -0.2560
 0.5424 -0.1821 -0.0779
[torch.FloatTensor of size 12x3]
</code></pre>
<ol start="2">
<li>No, they are computed by different LSTMs with different parameters. They are different.</li>
</ol>

username: FAlex
cooked: <p><a class="mention" href="/u/simonw">@SimonW</a> <a class="mention" href="/u/smth">@smth</a><br>
Hello guys, I’d like to ask you one thing about this parameter (num_layers in RNN module) and how it relates to the LSTM stable documentation.<br>
Looking at the picture posted above, I’d say that the hidden state at time t of the first hidden layer receives as input the hidden state at time (t-1) of the same layer. Similarly, the hidden state at time t of the second layer receives as input the hiddent state at time (t-1) of the second layer.<br>
Yet, in the nn.LSTM doc (<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM</a>) there is:<br>
“h(t−1) is the hidden state of the <em>previous layer</em> at time t-1”<br>
Considering that the gates receive h(t-1) as input, does this mean that the l-th layer should look at the (l-1)-th layer? Or am I reading it wrong?<br>
Thanks.</p>

username: tom
cooked: <p>That’s a bug in the documentation, your interpretation of the picture is right.<br>
The t-dimension stays in the same layer. The connection between the layers is that the output of the <code>l-1</code>st layer is the input of the <code>l</code>th layer, possibly multiplied by drop, i.e. <code>h^(l-1)(t) delta^(l-1)(t) = i^(l)(t)</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: tom
cooked: <p>Hi <a class="mention" href="/u/falex">@FAlex</a>,</p>
<p>thanks for pointing out the potential for improvement in the documentation!<br>
I’ve put this into <a href="https://github.com/pytorch/pytorch/pull/11896" rel="nofollow noopener">a PR on github</a>, so hopefully PyTorch 1.0 ships with clearer documentation.</p>
<p>Best regards</p>
<p>Thomas</p>

username: FAlex
cooked: <p>That’s awesome, thanks!</p>

username: Simon_Ahmed
cooked: <p>If num_layers==3, will the output characters on the top be the same?</p>