Loss.backward() - IndexError: select(): index 1 out of range for tensor of size [1, 32, 100] at dimension 0

Nikos_Spanos: <p>I have created the following NN using PyTorch API (for NLP Multi-class Classification)</p>
<pre><code class="lang-auto">class MultiClassClassifer(nn.Module):
  #define all the layers used in model
  def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):
    
    #Constructor
    super(MultiClassClassifer, self).__init__()

    #embedding layer
    self.embedding = nn.Embedding(vocab_size, embedding_dim)

    #dense layer
    self.hiddenLayer = nn.Linear(embedding_dim, hidden_dim)

    #Batch normalization layer
    self.batchnorm = nn.BatchNorm1d(hidden_dim)

    #output layer
    self.output = nn.Linear(hidden_dim, output_dim)

    #activation layer
    self.act = nn.Softmax(dim=1) #2d-tensor

    #initialize weights of embedding layer
    self.init_weights()

  def init_weights(self):

    initrange = 1.0
    
    self.embedding.weight.data.uniform_(-initrange, initrange)
  
  def forward(self, text, text_lengths):

    embedded = self.embedding(text)
    embedded = torch.mean(embedded, dim=1, keepdim=True)
    print(embedded.shape)

    #packed sequence
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)

    tensor, batch_size = packed_embedded[0], packed_embedded[1]
    print(tensor.shape)

    hidden_1 = self.batchnorm(self.hiddenLayer(tensor))
    print(hidden_1.shape)

    output = self.act(self.output(hidden_1))
    print(output.shape)

    return output
</code></pre>
<p>Instantiating the model</p>
<pre><code class="lang-auto">INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 64
OUTPUT_DIM = 3

model = MultiClassClassifer(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)
</code></pre>
<p>To train the model I have created the following <code>train()</code> method:</p>
<pre><code class="lang-auto">def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0
    
    model.train()
    
    for batch in iterator:
        
        optimizer.zero_grad()

        text, text_lengths = batch.text_normalized_tweet
                
        predictions = model(text, text_lengths)
        
        loss = criterion(predictions, batch.label)
        print(loss)
        print(loss.item())

        loss.backward() #exception occurs here
      
        optimizer.step()
        
        epoch_loss += loss.item()
  
    return epoch_loss / len(iterator)
</code></pre>
<p>I have created the following NN using PyTorch API (for NLP Multi-class Classification)</p>
<pre><code class="lang-auto">class MultiClassClassifer(nn.Module):
  #define all the layers used in model
  def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):
    
    #Constructor
    super(MultiClassClassifer, self).__init__()

    #embedding layer
    self.embedding = nn.Embedding(vocab_size, embedding_dim)

    #dense layer
    self.hiddenLayer = nn.Linear(embedding_dim, hidden_dim)

    #Batch normalization layer
    self.batchnorm = nn.BatchNorm1d(hidden_dim)

    #output layer
    self.output = nn.Linear(hidden_dim, output_dim)

    #activation layer
    self.act = nn.Softmax(dim=1) #2d-tensor

    #initialize weights of embedding layer
    self.init_weights()

  def init_weights(self):

    initrange = 1.0
    
    self.embedding.weight.data.uniform_(-initrange, initrange)
  
  def forward(self, text, text_lengths):

    embedded = self.embedding(text)
    embedded = torch.mean(embedded, dim=1, keepdim=True)
    print(embedded.shape)

    #packed sequence
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)

    tensor, batch_size = packed_embedded[0], packed_embedded[1]
    print(tensor.shape)

    hidden_1 = self.batchnorm(self.hiddenLayer(tensor))
    print(hidden_1.shape)

    output = self.act(self.output(hidden_1))
    print(output.shape)

    return output
</code></pre>
<p>Instantiating the model</p>
<pre><code class="lang-auto">INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 64
OUTPUT_DIM = 3

model = MultiClassClassifer(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)
</code></pre>
<p>To train the model I have created the following <code>train()</code> method:</p>
<pre><code class="lang-auto">def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0
    
    model.train()
    
    for batch in iterator:
        
        optimizer.zero_grad()

        text, text_lengths = batch.text_normalized_tweet
                
        predictions = model(text, text_lengths)
        
        loss = criterion(predictions, batch.label)
        print(loss)
        print(loss.item())

        loss.backward() #exception occurs here
      
        optimizer.step()
        
        epoch_loss += loss.item()
  
    return epoch_loss / len(iterator)
</code></pre>
<p>However, when I start the first epoch of the training, during the <code>loss.backward()</code> step I receive the following error</p>
<blockquote>
<p>IndexError: select(): index 1 out of range for tensor of size [1, 32, 100] at dimension 0</p>
</blockquote>
<p>Here is the output of the <code>print()</code> functions</p>
<pre><code class="lang-auto">torch.Size([32, 1, 100])
torch.Size([32, 100])
torch.Size([32, 64])
torch.Size([32, 3])
tensor(1.1309, grad_fn=&lt;NllLossBackward0&gt;)
1.1309444904327393
</code></pre>
<p>Does the error have to do with the <code>nn.Embedding</code> layer of my network? And if so, how can I fix this exception.</p>
<p>You may find <a href="https://colab.research.google.com/drive/1HtohSU4oz_FwllYll0WgVtbaj4cI-FR2?usp=sharing" rel="noopener nofollow ugc">here</a> my Golab Notebook.</p>

mMagmer: <p>hi,<br>
i think you have problem with this line.</p>
<aside class="quote no-group" data-username="Nikos_Spanos" data-post="1" data-topic="139530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikos_spanos/48/44970_2.png" class="avatar"> Nikos_Spanos:</div>
<blockquote>
<pre><code class="lang-auto">    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)
</code></pre>
</blockquote>
</aside>
<p>i managed  to reproduce the error with following code.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

total = 256 * 56 * 4 
x = torch.arange(0, total).view(256,56 , 4).float()
x.requires_grad = True
o= nn.utils.rnn.pack_padded_sequence(x, torch.tensor(256*[57]), batch_first=True)
o[0].mean().backward()

</code></pre>
<p>if you place any number greater than 56 in  <em>lengths</em> of each batch element, it raises an error.</p>

Nikos_Spanos: <p>Indeed since you reproduced the error the rnn.pack_padded_sequece() has the problem.</p>
<p>The case is that I want to train my own embeddings that’s why I used the embedding layer. Afterwards, train a second algorithm using pre-trained glove embeddings. To train the customer embeddings with nn.Embedding layer is it necessary to use the rnn.pack_padded_sequence() method or is there any alternative?</p>

mMagmer: <p><strong>¯\_(ツ)_/ ¯</strong> i know nothing about it</p>

Nikos_Spanos: <p>It’s Ok, I hope some1 will find a workareound. Because neither do I know how to debug this. I am coming from Tensorflow and Keras, so I am totally newb with PyTorch</p>

Nikos_Spanos: <p>mMagmer do you know the solution to the error raised for the <code>o[0].mean().backward()</code>?<br>
Since you find it out maybe you know also a workaround. Appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

mMagmer: <aside class="quote no-group" data-username="mMagmer" data-post="2" data-topic="139530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>if you place any number greater than 56 in <em>lengths</em> of each batch element, it raises an error.</p>
</blockquote>
</aside>
<p>i think you’re not suppose to have any length greater than original sequence.<br>
57 --&gt;55</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

total = 256 * 56 * 4 
x = torch.arange(0, total).view(256,56 , 4).float()
x.requires_grad = True
o= nn.utils.rnn.pack_padded_sequence(x, torch.tensor(256*[55]), batch_first=True)
o[0].mean().backward()
</code></pre>

sagsriv: <p>The error, at least from</p>
<aside class="quote no-group" data-username="mMagmer" data-post="2" data-topic="139530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<pre><code class="lang-auto">import torch
import torch.nn as nn

total = 256 * 56 * 4 
x = torch.arange(0, total).view(256,56 , 4).float()
x.requires_grad = True
o= nn.utils.rnn.pack_padded_sequence(x, torch.tensor(256*[57]), batch_first=True)
o[0].mean().backward()
</code></pre>
</blockquote>
</aside>
<p>is pretty clear. <code>pack_padded_sequence</code> takes two args: <em>input</em> , <em>lengths</em> . first is <code>BxTxH</code> (because you are doing batch_first=True), the second is [length]*B, i.e., a <code>B</code> dim vector. If you are asking it to use batchsize 57 when the real batchsize is 56.</p>

sagsriv: <aside class="quote no-group" data-username="Nikos_Spanos" data-post="1" data-topic="139530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikos_spanos/48/44970_2.png" class="avatar"> Nikos_Spanos:</div>
<blockquote>
<p><code>embedded = torch.mean(embedded, dim=1, keepdim=True)</code></p>
</blockquote>
</aside>
<p>This line is weird. Do you really want to mean over the tokens (dim=1) before putting it into an RNN?</p>
<p>In any case,</p>
<aside class="quote no-group" data-username="Nikos_Spanos" data-post="1" data-topic="139530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikos_spanos/48/44970_2.png" class="avatar"> Nikos_Spanos:</div>
<blockquote>
<p><code>packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)</code></p>
</blockquote>
</aside>
<p>is going to be wrong, because <code>batch_first</code> means from <code>32x1x100</code> you have a batch of size 1, so the second args should be a vector of dim 1. and I can expect that <code>text_lengths</code> does not have that dimension.</p>

Nikos_Spanos: <p>mMagmer, How can I specify the value of <code>torch.tensor(256*[55])</code> because in my case I use the text_lengths which is a fixed number per batch iteration. For example, rnn.pack_padded_sequence(32, 13,…) (batch 1), rnn.pack_padded_sequence(32, 7,…) (batch 2). As you can see each batch has a different length of tokens per sentence.</p>

mMagmer: <p>i’m not an expert in nlp .<br>
but based on name of function <code>pack_padded_sequence</code> , i think you should first pad the sequence to have the same length.<br>
see this <a href="https://gist.github.com/HarshTrivedi/f4e7293e941b17d19058f6fb90ab0fec" rel="noopener nofollow ugc">link</a>.<br>
the input sequence of pack_padded_sequence is padded before that.<br>
again i know nothing about nlp.</p>

Nikos_Spanos: <p>Hmm isn’t that strange?..I mean I have a training iterator that has batches of different shape. First batch has shape [32, 13], next batch has shape [32, 7], next batch has shape [32,11] etc. So based on the solutions I followed that use rnn.pack_padded_sequence(), the packed_embedded was a tensor with [32x11,100], [32x13,100], [32x7, 100]…etc So, I have used the .mean() method to bring the tensor dimensions down to 32x1 (averaged). This is the solution I followed. Do you know any alternative solution to apply multi-class text classification with my own embeddings? Would appreciate your help.</p>

Nikos_Spanos: <p><a class="mention" href="/u/sagsriv">@sagsriv</a><br>
Do you have any solution to this one</p>
<pre><code class="lang-auto">packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)
</code></pre>
<p>, based on my implementation? I would really appreciate any help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Nikos_Spanos: <p><a class="mention" href="/u/sagsriv">@sagsriv</a> This is not true…batch_first=False will throw the error you describe. With batch_first=True the nn will understand that batch_size=32 and not 1.</p>

ljt2448152687: <p>I also encountered the same problem. You can try to delete batch_first=True. After I deleted it, there was no error. I hope to hear about your good results.</p>