RuntimeError: Expected hidden[0] size (2, 20, 256), got (2, 50, 256)

Raj: <p>I’m building a LSTM model to classify test into multiple classes and I get the following error when training the model. I’ve used a softmax activation as the final layer. There are 44 possible classes for the texts. Below is the code snippet and network architecture. any help will be greatly appreciated. I’m using CPU to train the model for now, post which I’ll move to GPU.</p>
<h1>training params</h1>
<p>epochs = 4 # 3-4 is approx where I noticed the validation loss stop decreasing</p>
<p>counter = 0<br>
print_every = 100<br>
clip=5 # gradient clipping</p>
<h1>move model to GPU, if available</h1>
<p>if(train_on_gpu):<br>
net.cuda()</p>
<p>net.train()</p>
<h1>train for some number of epochs</h1>
<p>for e in range(epochs):<br>
# initialize hidden state<br>
h = net.init_hidden(batch_size)</p>
<pre><code># batch loop
for inputs, labels in train_loader:
    counter += 1

    if(train_on_gpu):
        inputs, labels = inputs.cuda(), labels.cuda()

    # Creating new variables for the hidden state, otherwise
    # we'd backprop through the entire training history
    h = tuple([each.data for each in h])

    # zero accumulated gradients
    net.zero_grad()

    # get the output from the model
    output, h = net(inputs, h)
    
  #         print('output:',output.squeeze())
  #         print('labels:',labels.float())
    
    # calculate the loss and perform backprop
    loss = criterion(output, labels)
    loss.backward()
    # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
    nn.utils.clip_grad_norm_(net.parameters(), clip)
    optimizer.step()

    # loss stats
    if counter % print_every == 0:
        # Get validation loss
        val_h = net.init_hidden(batch_size)
        val_losses = []
        net.eval()
        for inputs, labels in valid_loader:

            # Creating new variables for the hidden state, otherwise
            # we'd backprop through the entire training history
            val_h = tuple([each.data for each in val_h])

            if(train_on_gpu):
                inputs, labels = inputs.cuda(), labels.cuda()

            output, val_h = net(inputs, val_h)
            
            val_loss = criterion(output, labels)

            val_losses.append(val_loss.item())

        net.train()
        print("Epoch: {}/{}...".format(e+1, epochs),
              "Step: {}...".format(counter),
              "Loss: {:.6f}...".format(loss.item()),
              "Val Loss: {:.6f}".format(np.mean(val_losses)))
</code></pre>
<h1>Instantiate the model w/ hyperparams</h1>
<p>vocab_size = len(vocab_to_int)+1<br>
output_size = 44<br>
embedding_dim = 100<br>
hidden_dim = 256<br>
n_layers = 2</p>
<p>net = ClassificationRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)</p>
<p>print(net)</p>
<p>ClassificationRNN(<br>
(embedding): Embedding(5865, 100)<br>
(lstm): LSTM(100, 256, num_layers=2, batch_first=True, dropout=0.5)<br>
(fc): Linear(in_features=256, out_features=44, bias=True)<br>
(sof): LogSoftmax()<br>
(dropout): Dropout(p=0.3)<br>
)</p>

DoubtWang: <p>you can print the shape of the hidden[0] at each step.<br>
then, carefully check it.</p>

Raj: <p><a class="mention" href="/u/doubtwang">@DoubtWang</a> Don’t see a provision to include that for debugging purposes as I believe this happens within the LSTM package.</p>

DoubtWang: <p>Can you give detailed code?<br>
Besides, explain the shape of <code>inputs</code> and <code>label</code>?</p>

Raj: <p>Input is a tensor from the data loader which has a batch size of 50 and sequence length of  20. Labels is uni dimensional tensor of possbile values ranging from 0 to 43 (44 values. Pls find below detailed code.</p>
<pre><code>import torch.nn as nn
</code></pre>
<p>class ClassificationRNN(nn.Module):<br>
“”"<br>
The RNN model that will be used to perform Sentiment analysis.<br>
“”"</p>
<pre><code>def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
    """
    Initialize the model by setting up the layers.
    """
    super(ClassificationRNN, self).__init__()

    self.output_size = output_size
    self.n_layers = n_layers
    self.hidden_dim = hidden_dim
    
    # define all layers
    # embedding and LSTM layers
    self.embedding = nn.Embedding(vocab_size,embedding_dim)
    self.lstm = nn.LSTM(embedding_dim,hidden_dim,n_layers,dropout=drop_prob, batch_first=True)
    
    #full connected layer &amp; softmax
    self.fc = nn.Linear(hidden_dim,output_size)
    self.sof = nn.LogSoftmax(dim=1)
    
    #dropout layer
    self.dropout = nn.Dropout(0.3)

def forward(self, x, hidden):
    """
    Perform a forward pass of our model on some input and hidden state.
    """
    batch_size = x.size(0)
    embeds = self.embedding(x)
    lstm_out,hidden= self.lstm(embeds,hidden)
    
     # stack up lstm outputs
    lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
    
    # dropout and fully-connected layer
    out = self.dropout(lstm_out)
    out = self.fc(out)
    
    # softmax function
    soft_out = self.sof(out)
    
    # reshape to be batch_size first
    soft_out = soft_out.view(batch_size, -1)
</code></pre>
<h1>soft_out = soft_out[:, -1] # get last batch of labels</h1>
<pre><code>    # return last sigmoid output and hidden state
    return soft_out, hidden


def init_hidden(self, batch_size):
    ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
    # initialized to zero, for hidden state and cell state of LSTM
    weight = next(self.parameters()).data
    
    if (train_on_gpu):
        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
              weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())
    else:
        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())
    
    return hidden
</code></pre>
<hr>
<h1>Instantiate the model w/ hyperparams</h1>
<p>vocab_size = len(vocab_to_int)+1<br>
output_size = 44<br>
embedding_dim = 100<br>
hidden_dim = 256<br>
n_layers = 2</p>
<p>net = ClassificationRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)</p>
<h2>print(net)</h2>
<h1>loss and optimization functions</h1>
<p>lr=0.001</p>
<p>criterion = nn.NLLLoss()</p>
<p>optimizer = torch.optim.Adam(net.parameters(), lr=lr)</p>
<hr>
<h1>training params</h1>
<p>epochs = 4 # 3-4 is approx where I noticed the validation loss stop decreasing</p>
<p>counter = 0<br>
print_every = 100<br>
clip=5 # gradient clipping</p>
<h1>move model to GPU, if available</h1>
<p>if(train_on_gpu):<br>
net.cuda()</p>
<p>net.train()</p>
<h1>train for some number of epochs</h1>
<p>for e in range(epochs):<br>
# initialize hidden state<br>
h = net.init_hidden(batch_size)</p>
<pre><code># batch loop
for inputs, labels in train_loader:
    counter += 1

    if(train_on_gpu):
        inputs, labels = inputs.cuda(), labels.cuda()

    # Creating new variables for the hidden state, otherwise
    # we'd backprop through the entire training history
    h = tuple([each.data for each in h])

    # zero accumulated gradients
    net.zero_grad()

    # get the output from the model
    output, h = net(inputs, h)
    
    print('output shape',output.shape)
</code></pre>
<h1>print(‘labels:’,labels.float())</h1>
<pre><code>    # calculate the loss and perform backprop
    loss = criterion(output.squeeze(), labels)
    loss.backward()
    # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
    nn.utils.clip_grad_norm_(net.parameters(), clip)
    optimizer.step()

    # loss stats
    if counter % print_every == 0:
        # Get validation loss
        val_h = net.init_hidden(batch_size)
        val_losses = []
        net.eval()
        for inputs, labels in valid_loader:

            # Creating new variables for the hidden state, otherwise
            # we'd backprop through the entire training history
            val_h = tuple([each.data for each in val_h])

            if(train_on_gpu):
                inputs, labels = inputs.cuda(), labels.cuda()

            output, val_h = net(inputs, val_h)
            
            val_loss = criterion(output.squeeze(), labels)

            val_losses.append(val_loss.item())

        net.train()
        print("Epoch: {}/{}...".format(e+1, epochs),
              "Step: {}...".format(counter),
              "Loss: {:.6f}...".format(loss.item()),
              "Val Loss: {:.6f}".format(np.mean(val_losses)))</code></pre>

DoubtWang: <p>I have run your code, but I do not find any bug.<br>
Give you some advice:</p>
<ol>
<li>If you initialized hidden state to zero, no operation required.</li>
</ol>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/nn.html?highlight=lstm#torch.nn.LSTM" rel="noopener nofollow ugc">If (h_0, c_0) is not provided, both  <strong>h_0</strong>  and  <strong>c_0</strong>  default to zero.</a></p>
</blockquote>
<ol start="2">
<li>If you set batch_first=True, and the shape of inputs is (batch_size, len)(50, 20)</li>
</ol>
<aside class="quote no-group quote-modified" data-username="Raj" data-post="5" data-topic="38288">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c0e974/48.png" class="avatar"> Raj:</div>
<blockquote>
<p>input is a tensor from the data loader which has a batch size of 50 and sequence length of 20.</p>
</blockquote>
</aside>
<p>So, the hidden[0] should be (2, 50, 256) instead of (2, 20, 256)</p>
<blockquote>
<p><strong>h_0</strong>  of shape (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for each element in the batch.</p>
</blockquote>
<ol start="3">
<li>For <strong>NLLLoss</strong> function,</li>
</ol>
<blockquote>
<p>Input: (N, C) where C = number of classes, and Target: (N) where each value is 0≤targets[i]≤C−1<br>
So, the code should be:</p>
</blockquote>
<pre><code class="lang-auto">        # reshape to be batch_size first
        soft_out = soft_out.view(batch_size, -1, self.output_size)
        
        soft_out = soft_out[:, -1, :] # get last batch of labels
</code></pre>
<p>4.If you use the lstm to complete the classification task, you can use the finally state or self-attention based on my experience.</p>

Raj: <p>Thanks <a class="mention" href="/u/doubtwang">@DoubtWang</a>. Your suggestions were very useful. I found the problem. One of the batches in the validation data loader has incorrect dimension. Pls refer attached image. But, I dont know how to remove that batch. Could you pls suggest a solution for that?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f00877aaf4af51d5cea7bb121a373ae25a6b82c4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f00877aaf4af51d5cea7bb121a373ae25a6b82c4" title="Validation%20batch%20error"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f00877aaf4af51d5cea7bb121a373ae25a6b82c4_2_690x387.jpeg" alt="Validation%20batch%20error" data-base62-sha1="yfqJwNXcDqDnjmBvwaQJOQqhBLS" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f00877aaf4af51d5cea7bb121a373ae25a6b82c4_2_690x387.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f00877aaf4af51d5cea7bb121a373ae25a6b82c4_2_1035x580.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f00877aaf4af51d5cea7bb121a373ae25a6b82c4.jpeg 2x" data-dominant-color="E3E3E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Validation%20batch%20error</span><span class="informations">1366×768 186 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

DoubtWang: <p>You can try and see if it works.</p>
<pre><code class="lang-auto">def init_hidden(self):
    if (train_on_gpu):
        hidden = (weight.new(self.n_layers, 1, self.hidden_dim).zero_().cuda(),
                         weight.new(self.n_layers, 1, self.hidden_dim).zero_().cuda())
    else:
        hidden = (weight.new(self.n_layers, 1, self.hidden_dim).zero_(),
                       weight.new(self.n_layers, 1, self.hidden_dim).zero_())
    return hidden
</code></pre>
<pre><code class="lang-auto">batch_size = inputs.size(0)
h = tuple([each.repeat(1, batch_size, 1).data for each in h])
</code></pre>
<pre><code class="lang-auto">batch_size = inputs.size(0)
val_h = tuple([each.repeat(1, batch_size, 1).data for each in val_h])
</code></pre>
<p>I think this is a method which can solve this problem.<br>
Can you understand that?</p>

Raj: <p>Thanks <a class="mention" href="/u/doubtwang">@DoubtWang</a>, your solution as I understand it, is trying to fix the hidden layer, but I believe the issue I’m facing is with the input dimension of few bad batches in the Validation dataloader. If I can fix the dataloader, I believe this issue would be resolved.</p>

Raj: <p>I worked around the problem by skipping the training &amp; validation for those batches which were not of the shape (50,20). Below is a simple if condition that helped me to solve the problem.</p>
<blockquote>
<p>if( (inputs.shape[0],inputs.shape[1]) != (batch_size,seq_length)):<br>
print(‘Validation - Input Shape Issue:’,inputs.shape)<br>
<strong>continue</strong></p>
</blockquote>

1dividedby0: <p>This was so helpful! Thank you!</p>

bkuriach: <p>This error could be resolved by setting the <strong>‘drop_last’ parameter in DataLoader</strong>. You might be getting this error because your training data is not divisible by batch size. In that case, you will get this error at the end of epoch. Suppose your training data has 1005 records and batch size is 100, for the last batch, it will have only 5 records and you may get dimension error. If you keep ‘drop_last’ as True, those 5 records will be ignored.</p>
<p>train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size, <strong>drop_last=True</strong>)</p>

Rasha_Salim: <p>Thanks a lot! this was really helpful</p>

erfan_asadi: <aside class="quote no-group" data-username="bkuriach" data-post="12" data-topic="38288">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/35a633/48.png" class="avatar"> bkuriach:</div>
<blockquote>
<p>drop_last</p>
</blockquote>
</aside>
<p>I have the same problem and this is the exact answer,<br>
Thank you! &lt;3</p>

mapneto: <p>i had the same error and realize this issue when read your contribution. Thank you very much!</p>

mapneto: <p>very useful!! Thank you</p>

senitent_signal: <p>Sometimes looking at problem from different angle reduces the computation time in brain, i dunno how does it happen inside brain but it yields drop_last=True</p>
<p><strong>But my question is what could be missed if train_loader is not divisble by batch size?</strong></p>