IndexError: Target 2 is out of bounds

vyaslkv: <p>After epoch 1 it is throwing this error</p>
<h2>Model: resnet50 Frozen Layers: False<br>
Epoch 1/50</h2>
<h2>train Loss: 0.2108 Acc: 0.9226 TPR: 0.9270 FPR: 0.0819</h2>
<p>IndexError                                Traceback (most recent call last)<br>
 in <br>
17         optimizer = optim.Adam(parameters)<br>
18<br>
—&gt; 19         net = train_model(net, criterion, optimizer, num_epochs=num_epochs, name=name)<br>
20<br>
21         net = net.to(torch.device(‘cpu’))</p>
<p> in train_model(model, criterion, optimizer, num_epochs, dataset_sizes, name)<br>
41<br>
42                     # criterion is the loss<br>
—&gt; 43                     loss = criterion(outputs, labels)<br>
44<br>
45                     # then back propogate the loss</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in <strong>call</strong>(self, *input, **kwargs)<br>
530             result = self._slow_forward(*input, **kwargs)<br>
531         else:<br>
–&gt; 532             result = self.forward(*input, **kwargs)<br>
533         for hook in self._forward_hooks.values():<br>
534             hook_result = hook(self, input, result)</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)<br>
914     def forward(self, input, target):<br>
915         return F.cross_entropy(input, target, weight=self.weight,<br>
–&gt; 916                                ignore_index=self.ignore_index, reduction=self.reduction)<br>
917<br>
918</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)<br>
2019     if size_average is not None or reduce is not None:<br>
2020         reduction = _Reduction.legacy_get_string(size_average, reduce)<br>
-&gt; 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
2022<br>
2023</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)<br>
1836                          .format(input.size(0), target.size(0)))<br>
1837     if dim == 2:<br>
-&gt; 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
1839     elif dim == 4:<br>
1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)</p>
<p>IndexError: Target 2 is out of bounds.</p>

ptrblck: <p>How many classes are you currently using and what is the shape of your output?<br>
Note that class indices start at 0 so your target should contain indices in the range <code>[0, nb_classes-1]</code>.</p>

vyaslkv: <p>I am using two classes</p>

vyaslkv: <pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, name, freeze_layers=True):
        super(Net, self).__init__()
        model = self.feature_extractor(name)
        
        if freeze_layers:
            for params in model.parameters():
                params.requires_grad = False

        self.basemodel = model
        # these are the fully connected layers
        self.feedforward1 = nn.Linear(1000, 256)
        self.dropout1 = nn.Dropout(0.2)
        self.feedforward2 = nn.Linear(256, 32)
        self.dropout2 = nn.Dropout(0.2)
        self.feedforward3 = nn.Linear(32, 2)
    
    def forward(self, x):
        x = self.basemodel(x)
        x = F.relu(self.dropout1(self.feedforward1(x)))
        x = F.relu(self.dropout2(self.feedforward2(x)))
        x = self.feedforward3(x)
        return x

    def feature_extractor(self, name):
        if name == 'vgg16':
            model = models.vgg16(pretrained=True)
        elif name == 'resnet18':
            model = models.resnet18(pretrained=True)
        elif name == 'resnet50':
            model = models.resnet18(pretrained=True)
        elif name == 'resnet50':
            model = models.alexnet(pretrained=True)
        elif name == 'squeezenet1_0':
            model = models.squeezenet1_0(pretrained=True)
        elif name == 'densenet161':
            model = models.densenet161(pretrained=True)
        else:
            raise ValueError('Should be one of vgg16,resnet50,resnet18, alexnet, squeezenet1_0, densenet161')
        return model



device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)
def train_model(model, criterion, optimizer, num_epochs=25, dataset_sizes=dataset_sizes, name=''):
    start = datetime.now()
    
    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0
    best_loss = np.inf
    best_tpr = 0.0
    best_fpr = np.inf
    save_every = 5
    
    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch+1, num_epochs))
        print('-'*10)
        
        for phase in ['train', 'valid']:
            if phase == 'train':
                model.train()
            else:
                model.eval()
                
            running_loss = 0.0
            running_corrects = 0
            running_tp = 0
            running_fp = 0
            running_fn = 0
            running_tn = 0
            
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)
                # https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/6
                optimizer.zero_grad()
                # feed forward process to find out the loss
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)



                    # criterion is the loss
                    loss = criterion(outputs, labels)
                    
                    # then back propogate the loss  
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()
                # running loss    
                running_loss += loss.item()*inputs.size(0)
                running_corrects += torch.sum(preds == labels.data).item()
                tp, fp, fn, tn = confusion_matrix(preds, labels)
                running_tp += tp
                running_fp += fp
                running_fn += fn
                running_tn += tn
                
            epoch_loss = running_loss/dataset_sizes[phase]
            epoch_acc = running_corrects/dataset_sizes[phase]
            epoch_tpr = running_tp/(running_tp+running_fn)
            epoch_fpr = running_fp/(running_fp+running_tn)
            
            print('{} Loss: {:.4f} Acc: {:.4f} TPR: {:.4f} FPR: {:.4f}'.format(phase, epoch_loss, epoch_acc, 
                                                                  epoch_tpr,
                                                                  epoch_fpr))
            
            if phase == 'valid' and epoch_loss &lt; best_loss:
                best_loss = epoch_loss
                best_acc = epoch_acc
                best_tpr = epoch_tpr
                best_fpr = epoch_fpr
                best_model_wts = copy.deepcopy(model.state_dict())
       
        if (epoch+1)%save_every == 0:
            model = model.to(torch.device('cpu'))
            state = {'optimizer': optimizer,
                     'optimizer_state_dict': optimizer.state_dict(),
                    'criterion': criterion,
                     'criterion_state_dict': criterion.state_dict(),
                    'model_state_dict': model.state_dict(),
                    'model': model,
                    'epoch': epoch+1}
            torch.save(state, name)
            model = model.to(device)
            print('Model Checkpointed')
        
        print()
        
    time_elapsed = datetime.now() - start

    print('Training complete in '+str(time_elapsed))
    print('Best Valid Loss: {:.4f} Acc: {:.4f} TPR: {:.4f} FPR: {:.4f}'.format(best_loss, best_acc,
                                                                              best_tpr, best_fpr))

    model.load_state_dict(best_model_wts)
    return model


num_epochs=50
#for m in ['vgg16', 'resnet50', 'resnet18', 'alexnet', 'squeezenet1_0', 'densenet161'][4:]:
for m in ['resnet50']:
    for freeze in [False]: #[True, False]:
        print('-'*100)
        print()
        print('Model: '+m+' Frozen Layers: '+str(freeze))
        name='./'+m+str(freeze)+str(50)+'.pth'
        net = Net(m, freeze_layers=freeze)
        # The batch size should be larger than the number of GPUs used.
        net = nn.DataParallel(net)
        net.to(device)
        
        criterion = nn.CrossEntropyLoss()
        parameters = filter(lambda p: p.requires_grad, net.parameters())
        # differential learning rates for different layers
        optimizer = optim.Adam(parameters)
        
        net = train_model(net, criterion, optimizer, num_epochs=num_epochs, name=name)
        
        net = net.to(torch.device('cpu'))
        state = {'optimizer': optimizer,
                 'optimizer_state_dict': optimizer.state_dict(),
                'criterion': criterion,
                 'criterion_state_dict': criterion.state_dict(),
                'model_state_dict': net.state_dict(),
                'model': net,
                'epoch': num_epochs}
#         torch.save(model_wtd.state_dict(), './Checkpoints/resnetdev2.pth')
#         torch.save(state, './'+m+str(freeze)+'_24April_jackets_single_data_added_'+'50.pth')
        torch.save(state, './'+m+str(freeze)+'13feb_organic_non_organic'+'50.pth')
        print('Best Model Checkpointed')
        
        print()
        print('-'*100)


</code></pre>

ptrblck: <p>Could you add a print statement to your code and check the min and max values of <code>labels</code> please?</p>

Jie_Gao: <p>This answer solved my question! Thanks!!</p>

ronitganguly: <p>Hi, I am using 7 classes and it says that Target 6 is out of bounds. Batch size I am using is 70 and the labels I printed are these:</p>
<pre><code class="lang-auto">torch.Size([70])
tensor([4, 5, 0, 5, 5, 0, 0, 0, 1, 4, 0, 1, 5, 4, 4, 0, 4, 2, 6, 0, 1, 4, 0, 2,
        4, 6, 4, 3, 6, 2, 4, 0, 3, 4, 0, 0, 4, 0, 5, 5, 5, 3, 2, 3, 6, 4, 0, 4,
        5, 5, 2, 4, 5, 3, 6, 1, 2, 0, 1, 6, 3, 4, 1, 4, 4, 2, 6, 6, 2, 2])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
&lt;ipython-input-73-4e4c221a4846&gt; in &lt;module&gt;()
      8     loss_fn = loss_fn,
      9     train_loader = train_loader,
---&gt; 10     validation_loader=validation_loader
     11 )
     12 

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-&gt; 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 6 is out of bounds.
</code></pre>
<p>The first line is the size of the labels or Targets, second line is the targets themselves.</p>
<p>Please help me. I am in dire need of help.</p>

ptrblck: <p>Could you check the shape of your <code>input</code> tensor please?<br>
If you are dealing with 7 classes, your model should output a tensor with the shape <code>[batch_size, 7]</code>.</p>

Vijaya_kumar: <p>I am in similar situation but the for me it says Target 10 is out of bounds. Number of classes  =10, min =1, max=10<br>
and my net is as follows:</p>
<pre><code>def __init__(self):
    super(ConvNet, self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2))
    self.layer2 = nn.Sequential(
        nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2))
    self.drop_out = nn.Dropout()
    self.fc1 = nn.Linear(8 * 8 * 64, 1000)
    self.fc2 = nn.Linear(1000, 10)

def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = out.reshape(out.size(0), -1)
    out = self.drop_out(out)
    out = self.fc1(out)
    out = self.fc2(out)
    return out
</code></pre>
<p>How can I rectify this problem? Can you kindly help?</p>

Vijaya_kumar: <p>Hi Jie, Can you please explain how did you fix the issue? I still did not get what changes needs to be made</p>

ptrblck: <p>The targets should be in the range <code>[0, 9]</code> for your use case, as they are used to index the output tensor, so you might just subtract 1 from the target tensor.</p>

Vijaya_kumar: <p>This is how I train my model. Where can I subtract from target tensor.</p>
<pre><code>def train(self, X_train, Y_train, X_test, Y_test):
    tensor_train_x = torch.Tensor(X_train.reshape(-1,1,32,32))  # transform to torch tensor
    tensor_train_y = torch.Tensor(Y_train)
    tensor_train_y = tensor_train_y.long()
    trainset = utils.data.TensorDataset(tensor_train_x, tensor_train_y)
    trainloader = utils.data.DataLoader(trainset)

    tensor_test_x = torch.Tensor(X_test.reshape(-1,1,32,32))  # transform to torch tensor
    tensor_test_y = torch.Tensor(Y_test)
    tensor_test_y = tensor_test_y.long()
    testset = torch.utils.data.TensorDataset(tensor_test_x, tensor_test_y)
    testloader = torch.utils.data.DataLoader(testset)

    model = ConvNet()
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=self.learning_rate)

    # Train the model
    total_step = len(trainloader)
    loss_list = []
    acc_list = []
    for epoch in range(self.epoch):
        for i, (images, labels) in enumerate(trainloader):
            # Run the forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss_list.append(loss.item())

            # Backprop and perform Adam optimisation
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            # Track the accuracy
            total = labels.size(0)
            _, predicted = torch.max(outputs.data, 1)
            correct = (predicted == labels).sum().item()
            acc_list.append(correct / total)

            if (i + 1) % 2000 == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
                      .format(epoch + 1, self.epoch, i + 1, total_step, loss.item(),
                              (correct / total) * 100))</code></pre>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">tensor_train_y = tensor_train_y - 1
tensor_test_y = tensor_test_y - 1
</code></pre>

mathematics: <p>suddenly, my model gave <code>IndexError target 4 out of bounds</code></p>
<p>but as you told i have <code> n_bclasses = [1,2,3,4]</code><br>
and I put output_size of linear layer <code>n_bclasses - 1</code> gave error as mention</p>
<p>but giving n_bclasses +1 i.e <code>5</code> works, Any reason for that .</p>
<p>Thanks</p>
<p>update: Problem has been solved <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> , but weirdly, I have to go to dataframe and minus by one so that it become [0,1,2,3] , then just passing <code>nb_classes</code> to Linear layer,strangely</p>

ptrblck: <p>That’s the expected behavior since the target should contain class indices in the range <code>[0, nb_classes-1]</code> (the start index should be 0 not 1).</p>

Hizella: <p>Hello,</p>
<p>Can anyone please explain what is meant by “number of classes”? What does this refer to?<br>
I’m getting the same error message but the “target” that is out of bounds varies from run to run. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Many thanks!<br>
hizella</p>

ptrblck: <p>“Number of classes” is defined by the number of different target classes in your multi-class classification use case.<br>
E.g. you could use a neural network in order to classify an input image into 3 different classes: cat, dog, bird. In this case the number of classes would be 3 and the target tensor should thus contain values in: <code>[0, 1, 2]</code>, where each integer represents a specific class.</p>

Hizella: <p>Hi ptrblck,</p>
<p>Thanks! That clarifies it a bit more.</p>
<p>However, I don’t understand why this error is being raised. In my case, it’s when I call the backward() function on my loss I think. Some background on my task: technically speaking, I’m working on neural machine translation, linguistically speaking I’m reconstructing on the sound level. Let me know, if that is relevant. Can you help me shoot this error? Or do I open a new thread?</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    print(f"[Epoch {epoch} / {num_epochs}]")
    
    reconstructed_word = reconstruct_word(model, word1)
    
    with open('out.txt', 'a') as f:
        print(f"Reconstruct example words {word1, word2, word3, word4, word5,} \n as {reconstructed_word}", file=f)

    model.train()
    optimizer.zero_grad()
    model.eval()    
    
    for batch_idx, element in enumerate(train_iterator):
                #Get input and targets
                #batch_idx is the iterator and elememt are the element in train_iterator
                inp_data = element.italian.to(device)
                targets = element.latin.to(device)
    
                #Forward propagation
                output = model(inp_data, targets)
                print("Output shape:", output.shape)
                print("Target shape:", targets.max(), targets.min())
                
                output = output[1:].reshape(-1, output.shape[2])
                targets = targets[1:].reshape(-1)
                print("Output shape 2:", output.shape)
                print("Target shape 2:", targets.max(), targets.min())
                
         
                optimizer.zero_grad()
                loss = criterion(output, targets) #error gets thrown either here or down at loss.backward()
                print("Loss:", loss)

                loss.backward()

                torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1)
    
                optimizer.step()
             
</code></pre>
<p>Best,<br>
hizella</p>

ptrblck: <p>I assume you are using <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code> as the <code>criterion</code>.<br>
If so, check the <code>output</code> shape as well as the min. and max. values on <code>targets</code>.<br>
As described before: for a multi-class classification the output should have the shape <code>[batch_size, nb_classes]</code>, while the target is expected to have the shape <code>[batch_size]</code> and contain class indices in the range <code>[0, nb_classes-1]</code>.</p>

Hizella: <p>Thank you very much! I will try to implement your suggestion and let you know!</p>