Putting different hidden size for multi layer LSTM

Rahul_Moozhikkal: <p>Hi,</p>
<p>I was looking in to a way by which we could put different hidden in a 2 layer LSTM size using standard nn.LSTM</p>
<p>If we see the input arguments for nn.LSTM=(input_size, hidden_size, <strong>num_layers</strong>)</p>
<p>I see no documentation or could not find anything online where it explains in PyTorch how we could have a different hidden size for layer 1 and layer 2.When I tried around with the code to provide more than one input to input and hidden sizes for multi layer LSTM it doesnt seem to work.<br>
This seems to be pretty straight forward in Keras using the argument units.</p>
<p>Eg: below code explicitly specify that the hidden unit in layer 2 will have size=50</p>
<p>model.add(LSTM(<br>
input_shape=(sequence_length, number_features),<br>
units=100)<br>
model.add(LSTM(<br>
input_shape=(sequence_length, number_features),<br>
units=50)</p>
<p>2.Is there a way to stack LSTMs with anything similar to nn.Sequential().I understand that nn.Sequential does not work for LSTMs?</p>
<p>Thank You<br>
Rahul M</p>

yongen9696: <p>Similar question has been answered over <a href="https://stackoverflow.com/questions/49224413/difference-between-1-lstm-with-num-layers-2-and-2-lstms-in-pytorch" rel="noopener nofollow ugc">here</a>.</p>
<p>Example answer from the link:</p>
<pre><code class="lang-auto">nn.Sequential(OrderedDict([
    ('LSTM1', nn.LSTM(input_size, hidden_size, 1),
    ('LSTM2', nn.LSTM(hidden_size, hidden_size, 1)
]))
</code></pre>
<p>Also, as <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html" rel="noopener nofollow ugc">Pytorch RNN source code</a> mentioned about if variable <strong>num_layers</strong> is 2, it will run 2 layers by stacking one on other one.</p>

fjaraavila: <p>This does not seem right. Given that the output of each LSTM layer is a tuple, not a Tensor object.<br>
As a matter of fact if you try this and then pass a tensor through it, most likely it’ll give you back the following error:</p>
<pre><code class="lang-auto">AttributeError: 'tuple' object has no attribute 'size'
</code></pre>

fjaraavila: <p>After giving it a thought I think I managed to do it. Will leave it here in case someone might find it useful.</p>
<pre><code class="lang-auto">class LSTM_multi_output(nn.Module):
    def __init__(self, step_size, input_dimensions, first_hidden_size, second_hidden_size, third_hidden_size):
      super(LSTM_multi_output, self).__init__()
      self.first_hidden_size = first_hidden_size
      self.second_hidden_size = second_hidden_size
      self.third_hidden_size = third_hidden_size
      self.step_size = step_size
      self.input_dimensions = input_dimensions
      self.first_layer = nn.LSTM(input_size = self.input_dimensions, hidden_size = self.first_hidden_size, 
                                 num_layers = 1, batch_first = True)
      self.second_layer = nn.LSTM(input_size = self.first_hidden_size, hidden_size = self.second_hidden_size, 
                                  num_layers = 1, batch_first = True)
      self.third_layer = nn.LSTM(input_size = self.second_hidden_size, num_layers = 1, 
                                 hidden_size = selfthird_hidden_size, batch_first = True)
      self.fc_layer = nn.Linear(self.step_size*self.third_hidden_size, 4)
    def forward(self, x):
      batch_size, seq_len, _ = x.size()
      h_1 = torch.zeros(1, batch_size, self.first_hidden_size)
      c_1 = torch.zeros(1, batch_size, self.first_hidden_size)
      hidden_1 = (h_1, c_1)
      lstm_out, hidden_1 = self.first_layer(x, hidden_1)
      h_2 = torch.zeros(1, batch_size, self.second_hidden_size)
      c_2 = torch.zeros(1, batch_size, self.second_hidden_size)
      hidden_2 = (h_2, c_2)
      lstm_out, hidden_2 = self.second_layer(lstm_out, hidden_2)
      h_3 = torch.zeros(1, batch_size, self.third_hidden_size)
      c_3 = torch.zeros(1, batch_size, self.third_hidden_size)
      hidden_3 = (h_3, c_3)
      lstm_out, hidden_3 = self.third_layer(lstm_out, hidden_3)
      x = lstm_out.contiguous().view(batch_size,-1)
      return self.fc_layer(x)
</code></pre>