Facing an issue while using batching in LSTM

samsumankumar: <p>Please see the below code and guide me as I am new to deep learning and PyTorch.<br>
The code runs fine if I use all the dataset at a time but throws an error while using batches of 32.<br>
class LSTMTagger(nn.Module):</p>
<pre><code>def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
    super(LSTMTagger, self).__init__()
    self.hidden_dim = hidden_dim

    self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

    # The LSTM takes word embeddings as inputs, and outputs hidden states
    # with dimensionality hidden_dim.
    self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first = True)

    # The linear layer that maps from hidden state space to tag space
    self.hidden2tag = nn.Linear(hidden_dim, tagset_size)

def forward(self, sentence):
    embeds = self.word_embeddings(sentence)
    lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
    tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))
    tag_scores = F.log_softmax(tag_space, dim=1)
    return tag_scores
</code></pre>
<p>EMBEDDING_DIM = 32<br>
HIDDEN_DIM = 32<br>
EPOCHS = 10</p>
<p>model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix))<br>
loss_function = nn.NLLLoss()<br>
optimizer = optim.Adam(model.parameters(), lr=0.01)</p>
<p>for epoch in range(EPOCHS):<br>
for sentence, tags in train_loader:<br>
model.zero_grad()</p>
<pre><code>    tag_scores = model(sentence)
    
    loss = loss_function(tag_scores, tags)
    loss.backward()
    optimizer.step()
</code></pre>
<h2>Error:</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
18         model.zero_grad()<br>
19<br>
—&gt; 20         tag_scores = model(sentence)<br>
21<br>
22         loss = loss_function(tag_scores, tags)</p>
<p>~\anaconda3\envs\tensorflow_gpu\lib\site-packages\torch\nn\modules\module.py in <strong>call</strong>(self, *input, **kwargs)<br>
548             result = self._slow_forward(*input, **kwargs)<br>
549         else:<br>
–&gt; 550             result = self.forward(*input, **kwargs)<br>
551         for hook in self._forward_hooks.values():<br>
552             hook_result = hook(self, input, result)</p>
<p> in forward(self, sentence)<br>
16     def forward(self, sentence):<br>
17         embeds = self.word_embeddings(sentence)<br>
—&gt; 18         lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))<br>
19         tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))<br>
20         tag_scores = F.log_softmax(tag_space, dim=1)</p>
<p>~\anaconda3\envs\tensorflow_gpu\lib\site-packages\torch\nn\modules\module.py in <strong>call</strong>(self, *input, **kwargs)<br>
548             result = self._slow_forward(*input, **kwargs)<br>
549         else:<br>
–&gt; 550             result = self.forward(*input, **kwargs)<br>
551         for hook in self._forward_hooks.values():<br>
552             hook_result = hook(self, input, result)</p>
<p>~\anaconda3\envs\tensorflow_gpu\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)<br>
565             hx = self.permute_hidden(hx, sorted_indices)<br>
566<br>
–&gt; 567         self.check_forward_args(input, hx, batch_sizes)<br>
568         if batch_sizes is None:<br>
569             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,</p>
<p>~\anaconda3\envs\tensorflow_gpu\lib\site-packages\torch\nn\modules\rnn.py in check_forward_args(self, input, hidden, batch_sizes)<br>
517     def check_forward_args(self, input, hidden, batch_sizes):<br>
518         # type: (Tensor, Tuple[Tensor, Tensor], Optional[Tensor]) -&gt; None<br>
–&gt; 519         self.check_input(input, batch_sizes)<br>
520         expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes)<br>
521</p>
<p>~\anaconda3\envs\tensorflow_gpu\lib\site-packages\torch\nn\modules\rnn.py in check_input(self, input, batch_sizes)<br>
168             raise RuntimeError(<br>
169                 ‘input.size(-1) must be equal to input_size. Expected {}, got {}’.format(<br>
–&gt; 170                     self.input_size, input.size(-1)))<br>
171<br>
172     def get_expected_hidden_size(self, input, batch_sizes):</p>
<p>RuntimeError: input.size(-1) must be equal to input_size. Expected 32, got 9952</p>

Usama_Hasan: <p>Hy <a class="mention" href="/u/samsumankumar">@samsumankumar</a>. Have appended batch_size while passing input to the model.<br>
Just check if by printing the shape of the batch</p>

vdw: <p>I see several issues:</p>
<ul>
<li>
<p>You create your <code>nn.LSTM</code> with <code>batch_first=True</code>, but then you want to reshape your input so that the sequence length is your first dimension: <code>embeds.view(len(sentence), 1, -1)</code></p>
</li>
<li>
<p>Your <code>view()</code> commands might be a problem anyway; please see <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">this post of mine</a>. The following change should actually work:</p>
<pre><code>lstm_out, _ = self.lstm(embeds)</code></pre>
</li>
</ul>

samsumankumar: <p>class LSTMTagger(nn.Module):</p>
<pre><code>def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
    super(LSTMTagger, self).__init__()
    self.hidden_dim = hidden_dim

    self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

    # The LSTM takes word embeddings as inputs, and outputs hidden states
    # with dimensionality hidden_dim.
    self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first = True)

    # The linear layer that maps from hidden state space to tag space
    self.hidden2tag = nn.Linear(hidden_dim, tagset_size)

def forward(self, sentence):
    batch_size = sentence.size(0)
    sentence = sentence.long()
    embeds = self.word_embeddings(sentence)
    lstm_out, _ = self.lstm(embeds)
    lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
    tag_space = self.hidden2tag(lstm_out)
    tag_scores = F.log_softmax(tag_space, dim=1)
    return tag_scores
</code></pre>
<p>EMBEDDING_DIM = 32<br>
HIDDEN_DIM = 32<br>
EPOCHS = 10</p>
<p><span class="hashtag">#len</span>(word_to_ix) = vocab_size, len(tag_to_ix) = tagset_size<br>
model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix))<br>
loss_function = nn.NLLLoss()<br>
optimizer = optim.Adam(model.parameters(), lr=0.01)</p>
<p>for epoch in range(EPOCHS):<br>
for sentence, tags in train_loader:<br>
model.zero_grad()</p>
<pre><code>    tag_scores = model(sentence)
    loss = loss_function(tag_scores, tags.flatten())
    loss.backward()
    optimizer.step()
</code></pre>
<p>The above code works now. Please look into it and tell me is there any fault?</p>