LSTM network inside a Sequential container

username: Olshansky
cooked: <p>I’ll start off by saying that I know very little about deep learning but still wanted to try and apply it to some of the work I’ve been doing.</p>
<p>Working through one of the tutorials, I built a NN made up of the following components:</p>

<pre><code>model = torch.nn.Sequential( 
     torch.nn.Linear(D_in, H),
     torch.nn.ReLU(),
     torch.nn.Linear(H, D_out),
)

y_pred = model(train_x)
</code></pre>
<p>I wanted to use an LSTM network, so I tried to do the following:</p>
<pre><code>model = torch.nn.Sequential(
      torch.nn.LSTM(D_in, H),
      torch.nn.Linear(H, D_out) 
)

y_pred = model(train_x)
</code></pre>
<p>This gave me the following error:</p>
<blockquote>
<p>RuntimeError: input must have 3 dimensions, got 2</p>
</blockquote>
<p>I found that the input expected by an LSTM network is a bit different than a Linear transformation:</p>
<blockquote>
<p>input of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence</p>
</blockquote>
<p>I was able to work around it by splitting my Sequential nn container into two layer, as well as reshaping my input/output to/from the LSTM layer like so:</p>
<pre><code>layerA = torch.nn.LSTM(D_in, H)
layerB = torch.nn.Linear(H, D_out)

train_x = train_x.unsqueeze(0)
y_pred, (hn, cn) = layerA(train_x)
y_pred = y_pred.squeeze(0)
y_pred = layerB(y_pred)
</code></pre>
<p>However, simply getting it to work concerns me because I feel like I’m using the nn incorrectly. My questions are as follows:</p>
<ol>
<li>How can I use an LSTM network as part of a Sequential container?</li>
<li>Why is the data to an LSTM network different from that to a Linear one? What is the significance of the outer most dimension?</li>
<li>What is the correct way to use DataLoader in conjunction with an LSTM network? I’m using the default DataLoader, which doesn’t seem to play way with nn.LSTM:</li>
</ol>
<pre><code class="lang-auto">    test_x_data = torch.FloatTensor(x)
    test_dataset = data_utils.TensorDataset(test_x_data, test_y_data)
    test_loader = data_utils.DataLoader(test_dataset, batch_size=1, shuffle=True)
</code></pre>
<p>This is a relatively open-ended question, so I appreciate your time in advance!</p>

username: karmus89
cooked: <p>Hi <a class="mention" href="/u/olshansky">@Olshansky</a>!</p>
<p>I’ve been tackling a similar problem as you have in this post. I too tried to tackle my problem first by using the <code>nn.Sequential</code> container, but the problem lies in that the <code>nn.LSTM</code> outputs a <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" rel="nofollow noopener">tuple</a>.</p>
<p>Let me thus share a mockup solution that utilizes <code>torch.nn.ModuleDict</code> and a custom <code>forward</code> function. I’ll however lay out the data first so that the transformations make sense to you:</p>
<pre><code class="lang-auto"># A batch is in shape [batches, sequences, sequence length, features]  
x=torch.Size([128, 30, 12, 45])
y=torch.Size([128, 30, 1, 1])
</code></pre>
<p>The most simple way to implement an LSTM coupled with a Linear layer I came up with is this:</p>
<pre><code class="lang-auto">class MockupModel(nn.Module):

    def __init__(self):
        super().__init__()
        self.model = nn.ModuleDict({
            'lstm': nn.LSTM(
                input_size=x_features,    # 45, see the data definition
                hidden_size=hidden_size,  # Can vary
            ),
            'linear': nn.Linear(
                in_features=hidden_size,
                out_features=1)
        })
</code></pre>
<p>Then I proceeded to overwrite the <code>forward</code> function:</p>
<pre><code class="lang-auto">    def forward(self, x):

        # From [batches, seqs, seq len, features]
        # to [seq len, batch data, features]
        x = x.view(x_seq_len, -1, x_features)
       
        # Data is fed to the LSTM
        out, _ = self.model['lstm'](x)
        print(f'lstm output={out.size()}')

        # From [seq len, batch, num_directions * hidden_size]
        # to [batches, seqs, seq_len,prediction]
        out = out.view(x_batches, x_seqs, x_seq_len, -1)
        print(f'transformed output={out.size()}')

        # Data is fed to the Linear layer
        out = self.model['linear'](out)
        print(f'linear output={out.size()}')

        # The prediction utilizing the whole sequence is the last one
        y_pred = out[:, :, -1].unsqueeze(-1)
        print(f'y_pred={y_pred.size()}')

        return y_pred
</code></pre>
<p>Now if I initialize the model and feed a batch of data to it, the printouts of of the forward function produce the following output:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; model = MockupModel()
&gt;&gt;&gt; model(x)
lstm output=torch.Size([12, 3840, 4])
transformed output=torch.Size([128, 30, 12, 4])
linear output=torch.Size([128, 30, 12, 1])
y_pred=torch.Size([128, 30, 1, 1])
</code></pre>
<p>This way my implementation produces predictions matching to the targets, which can then be fed to the loss function. I hope this helps you out, or anyone else with similar problem!</p>
<p>Petteri</p>

username: Olshansky
cooked: <p>This is exactly what I was looking for. Thank you so much for the detailed answer!</p>

username: HashRocketSyntax
cooked: <pre><code class="lang-auto">tf.keras.models.Sequential(
    l.LSTM(num_hidden, input_shape=features_shape),
    l.Dense(label_shape[0], activation='sigmoid')
)
</code></pre>

username: HashRocketSyntax
cooked: <p>It would be nice to have an arg that makes LSTM() only return a tensor. Alternatively, internal methods that need the non-tensor output could use different method to fetch it that info</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/44130851/simple-lstm-in-pytorch-with-sequential-module/64265525#64265525">
  <header class="source">

      <a href="https://stackoverflow.com/questions/44130851/simple-lstm-in-pytorch-with-sequential-module/64265525#64265525" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2580831/almamon-rasool-abd-ali" target="_blank" rel="noopener nofollow ugc">
    <img alt="almamon Rasool Abd Ali" src="https://i.stack.imgur.com/1BrpJ.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/44130851/simple-lstm-in-pytorch-with-sequential-module/64265525#64265525" target="_blank" rel="noopener nofollow ugc">Simple LSTM in PyTorch with Sequential module</a>
</h4>

<div class="tags">
  <strong>deep-learning, torch, lstm, recurrent-neural-network</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/2580831/almamon-rasool-abd-ali" target="_blank" rel="noopener nofollow ugc">
    almamon Rasool Abd Ali
  </a>
  on <a href="https://stackoverflow.com/questions/44130851/simple-lstm-in-pytorch-with-sequential-module/64265525#64265525" target="_blank" rel="noopener nofollow ugc">03:11PM - 08 Oct 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto"># LSTM() returns tuple of (tensor, (recurrent state))
class extract_tensor(nn.Module):
    def forward(self,x):
        # Output shape (batch, features, hidden)
        tensor, _ = x
        # Reshape shape (batch, hidden)
        return tensor[:, -1, :]


nn.Sequential(
nn.LSTM(inputSize, hiddenSize, 1, batch_first=True),
extract_tensor(),
nn.Linear(hiddenSize, classe_n))
</code></pre>