RuntimeError: ignore_index is not supported for floating point target

Vasudha_Venkatesan: <p>Hello,</p>
<p>I’m new to Pytorch and I’m trying to perform semantic segmentation of 3d images. When I use cross entropy loss with ignore index, I get the above error.</p>
<pre><code class="lang-auto">def training_fn(net,
                device,
                input_dim,
                epochs: int = 1,
                batch_size: int = 1,
                learning_rate: float = 1e-3,
                valiation_percent=0.1,
                save_checkpoint: bool = True):
    # create dataset
    dataset = hdf5.Hdf5Dataset(data_file_path, image_dim=input_dim, contains_mask=True)

    # create training and validation dataset
    n_dataset = dataset.__len__()
    n_val = round(n_dataset * valiation_percent)
    n_train = n_dataset - n_val
    train_set, val_set = random_split(dataset, [n_train, n_val])

    # create dataloaders
    train_dataloader = DataLoader(train_set, shuffle=True, batch_size=batch_size, num_workers=1, pin_memory=True)
    val_dataloader = DataLoader(val_set, shuffle=False, batch_size=batch_size, num_workers=1, pin_memory=True)

    # specify loss functions, optimizers
    loss_fn = nn.CrossEntropyLoss(ignore_index=2)
    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
    scheduler = lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)

    for epoch in range(1, epochs + 1):
        print('Epoch {}/{}'.format(epoch, epochs))
        print('-' * 10)
        if os.path.exists(checkpoint_path):  # checking if there is a file with this name
            checkpoint = torch.load(checkpoint_path)
            net.load_state_dict(checkpoint)
        # scheduler.step()
        for param_group in optimizer.param_groups:
            print("LR", param_group['lr'])

        net.to(device)
        net.train()
        running_loss = 0
        i = 0
        # training
        for batch in train_dataloader:
            image = batch[0]
            image = image.permute(1, 0, 2, 3)
            true_mask = batch[1]
            image = image.to(device=device, dtype=torch.float32)
            true_mask = true_mask.to(device=device, dtype=torch.int64)
            true_mask = F.one_hot(true_mask, config.n_classes)
            true_mask = true_mask.permute(0, 1, 4, 2, 3)
            true_mask = true_mask[-1, :, :, :]
            true_mask = true_mask.type(torch.float32)

            optimizer.zero_grad()

            pred = net(image)

            # pred = pred.permute(1, 0, 2, 3)
            loss = loss_fn(torch.round(pred), torch.round(true_mask))
            i += 1
            # Backpropagation

            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            # print(f'Accuracy score, Hamming loss - : {mIoU(pred, true_mask)}')
            if (i == n_train):
                print(f'Epoch : {epoch},  loss: {(running_loss / i):.4f}')
</code></pre>
<p>I have three labels - 0, 1, 2. I tried to round values so that it would be integer, but it is not working. Any suggestions would be helpful.</p>

KFrank: <p>Hi Vasudha!</p>
<aside class="quote no-group quote-modified" data-username="Vasudha_Venkatesan" data-post="1" data-topic="154783" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vasudha_venkatesan/48/45060_2.png" class="avatar"> Vasudha_Venkatesan:</div>
<blockquote>
<p>I’m trying to perform semantic segmentation of 3d images. When I use cross entropy loss with ignore index, I get the above error.</p>
<pre><code class="lang-auto">    loss_fn = nn.CrossEntropyLoss(ignore_index=2)
</code></pre>
</blockquote>
</aside>
<p><code>CrossEntropyLoss</code> works in two different ways.</p>
<p>For your semantic-segmentation use case, the <code>input</code> (the prediction<br>
made by your model) would typically have shape<br>
<code>[nBatch, nClass, height, width, depth]</code>.</p>
<p>Case 1:  Your ground-truth labels – the <code>target</code> passed to<br>
<code>CrossEntropyLoss</code> – are integer categorical class labels, and will have<br>
shape <code>[nBatch, height, width, depth]</code> (with no <code>nClass</code> dimension).<br>
This case supports <code>ignore_index</code>.</p>
<p>Case 2:  <code>target</code> consists of floating-point probabilistic (“soft”) labels, and<br>
has shape <code>[nBatch, nClass, height, width, depth]</code>.  This case does<br>
not support <code>ignore_index</code> (because there really isn’t a notion of a class<br>
“index,” i.e., an integer class label).</p>
<p>If you wish to use <code>ignore_index</code> you need to convert your labels to <code>long()</code><br>
(not just round them to floating-point values that are integers) and ensure<br>
that your labels tensor (<code>target</code>) does not have an <code>nClass</code> dimension.</p>
<p>Best.</p>
<p>K. Frank</p>

Vasudha_Venkatesan: <p>Hi Frank,</p>
<p>Thanks for the reply.</p>
<p>After converting to long, I still get the same error.</p>
<p>I have batch size as 1 and since images are large, I interchanged the depth to batch size and input size is - [<code>depth, nBatch, height, width</code>] and the target is of the form [<code>depth, nClass, height, width</code>]. Ground truth is in the same dimension as output. I’m not sure how to remove the nClass from the target so that all layers will be compared. Hope I’m clear about the dimensions.</p>
<pre><code class="lang-auto">True mask torch.Size([16, 3, 256, 256])
Image - torch.Size([16, 1, 256, 256])
Pred shape - torch.Size([16, 3, 256, 256])

</code></pre>
<p>Thanks,<br>
Vasudha</p>

Vasudha_Venkatesan: <p>Hi Frank,</p>
<p>Just an update, I removed the <code>nClass</code> from both <code>target</code> and <code>ground truth</code> and I still get the same error.</p>
<pre><code class="lang-auto">            pred = net(image)
            pred = pred[:,-1,:,:]
            pred_ = pred.type(torch.LongTensor)
            print(f'Pred shape - {pred_.shape}')
            loss = loss_fn(pred_,true_mask)
</code></pre>
<pre><code class="lang-auto">True mask torch.Size([16, 256, 256])
Image - torch.Size([16, 1, 256, 256])
Pred shape - torch.Size([16, 256, 256])
Traceback (most recent call last):
  File "train.py", line 164, in &lt;module&gt;
    input_dim=config.input_dimension)
  File "train.py", line 86, in training_fn
    loss = loss_fn(pred_,true_mask)
  File "/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/env/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 1165, in forward
    label_smoothing=self.label_smoothing)
  File "/env/lib/python3.7/site-packages/torch/nn/functional.py", line 2996, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: ignore_index is not supported for floating point target

</code></pre>

KFrank: <p>Hi Vasudha!</p>
<aside class="quote no-group" data-username="Vasudha_Venkatesan" data-post="4" data-topic="154783" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vasudha_venkatesan/48/45060_2.png" class="avatar"> Vasudha_Venkatesan:</div>
<blockquote>
<p>I still get the same error.</p>
</blockquote>
</aside>
<p>You don’t have the requirements for <code>pred</code> and <code>mask</code> correct yet.</p>
<p>Your <code>pred</code> (<code>target</code> for <code>CrossEntropyLoss</code>) has to be floating-point and<br>
have an <code>nClass</code> dimension, while <code>mask</code> (<code>target</code> for <code>CrossEntropyLoss</code>)<br>
has to be <code>long()</code> and not have the <code>nClass</code> dimension.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.11.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; pred = torch.randn (1, 3, 256, 256, 16)       # pred is floating-point and has nClass dimension
&gt;&gt;&gt; mask = torch.randint (3, (1, 256, 256, 16))   # mask is integer and does not have nClass dimension
&gt;&gt;&gt; torch.nn.CrossEntropyLoss (ignore_index = 2) (pred, mask)
tensor(1.3894)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Vasudha_Venkatesan: <p>It is working now. Thank you.</p>