LSTM input dims error

Mukesh1729: <p>I have the below classifier model and i want to pass in my input which has 39 features to it as<br>
Shape: torch .rand(1, 1,39) and it works but according to the documentation it should be (1, 39, hidden size).<br>
Why is this so in this case?<br>
How can i choose the hidden size of the LSTM based on?</p>
<pre><code class="lang-auto">class SequenceModel(nn.Module):
    def __init__(self, n_features, n_classes=5, n_hidden=256, n_layers=3):
        super().__init__()

        self.lstm = torch.nn.LSTM(input_size=n_features,
                                 hidden_size=n_hidden,
                                 num_layers=n_layers, 
                                 batch_first=True,
                                 dropout=0.3)
        self.classifier = torch.nn.Linear(n_hidden, n_classes)

    def forward(self, x):
      self.lstm.flatten_parameters()
      _, (hidden, _) = self.lstm(x)
      out = hidden[-1] # get the last state of the last layer
      return self.classifier(out)


#Print smaple output
c = torch.nn.CrossEntropyLoss()
sample = next(iter(train_loader))
out = SequenceModel(39, 5)(sample["sequence"])
# print("Model output:",  out.cpu().detach().numpy()) # Change batch size
print("loss:", c(out, sample["label"]))
</code></pre>

Andrei_Cristea: <p>According to <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">the documentation</a>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bf112feb2459f29e0a5bbe5b666b232e3c4f3f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/4bf112feb2459f29e0a5bbe5b666b232e3c4f3f0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bf112feb2459f29e0a5bbe5b666b232e3c4f3f0_2_690x117.png" alt="image" data-base62-sha1="aPOjbearjE2LDmoBX7QV8PBbrXy" width="690" height="117" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bf112feb2459f29e0a5bbe5b666b232e3c4f3f0_2_690x117.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bf112feb2459f29e0a5bbe5b666b232e3c4f3f0_2_1035x175.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bf112feb2459f29e0a5bbe5b666b232e3c4f3f0.png 2x" data-dominant-color="F1F0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1101×187 25.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Since you passed batch_first=True, it expects an input of shape (N, L, H_in) where:</p>
<ul>
<li>N is the batch size</li>
<li>L is the sequence length</li>
<li>H_in is the number of input features</li>
</ul>
<p>So when you pass torch.rand(1, 1, 39) you are passing it a batch size of 1, a sequence length of 1, and 39 input features, which matches how you set up your LSTM, therefore it works.</p>
<p>I think you could pass an input of size torch.rand(1, X, 39) where X is arbitrary, and it still should work. In other words the sequence length can (and typically is, if you’re using a LSTM) &gt;&gt; 1.</p>
<p>Hidden size should never matter for your input. I assume what you said was a typo and you meant input size. Regardless, the behavior seems consistent with the docs.</p>
<p>Hope this helps!</p>