When to use ignore_index?

username: bartolsthoorn
cooked: <p>I noticed <code>CrossEntropyLoss</code> now has an <code>ignore_index</code> parameter, but I am not sure I understand what that means:</p>
<pre><code class="lang-nohighlight">ignore_index (int, optional): Specifies a target value that is ignored
            and does not contribute to the input gradient. When size_average is
            True, the loss is averaged over non-ignored targets.
</code></pre>
<p>First of all, I know that <code>CrossEntropyLoss</code> takes a 1-dimensional array of targets:</p>
<pre><code class="lang-nohighlight">Target: :math:`(N)` where each value is `0 &lt;= targets[i] &lt;= C-1`
</code></pre>
<p>So then I assume that <code>ignore_index</code> allows you to ignore one of the outputs in the loss calculation. I can imagine it’s useful to mask a whole bunch of outputs. Simply ignoring only one output node,what is the use-case of that?</p>
<p>I probably misunderstood what <code>ignore_index</code> does or when do people use it?</p>

username: fmassa
cooked: <p>This is used to mask a specific label.<br>
For example, in semantic segmentation, we might have a <code>-1</code> label that stands for “dont care”, meaning that whatever you predict in that region is not taken into account in the evaluation (because it can be ambiguous).<br>
In this case, you would set the <code>ignore_index</code> to <code>-1</code>, so that those indices are not taken into account.</p>

username: bartolsthoorn
cooked: <p>Thanks for the clarification!</p>

username: Anshumaan_Dash
cooked: <p>Hi,</p>
<p>I would like to know more. I am using the torchvision segmentation model from your repo. There, the <code>ignore_index</code> is set to <code>255</code>.</p>
<p>If using a dataset that multiple classes to ignore during evaluation, say Cityscapes, I am manually using the training IDs and setting all those classes to <code>0</code> which is the same as the background class. How can I leverage the <code>ignore_index</code> ?</p>
<p>Thanks</p>

username: sanchit2843
cooked: <p>I am facing issue when using ignore index on cityscape dataset for semantic segmentation. There are 19 classes, and there is one extra ignore class. I used this ignore class as 255 and used same inside cross entropy loss. When I run the model with output number of classes=19, I get assertion error from cross entropy loss,<br>
Error:<br>
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [3,0,0], thread: [743,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
This error is sort of obvious because we need I am passing 255 label, but considering it is passed in ignore index I am assuming this error should not come. In case this is not how ignore index work can you tell how I can achieve the solution where I want to ignore 255 label and still have 19 classes inside model.</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue with this small code snippet:</p>
<pre><code class="lang-python">x = torch.randn(10, 19, requires_grad=True, device='cuda')
y = torch.randint(0, 19, (10,), device='cuda')
y[0] = 255

criterion = nn.CrossEntropyLoss(ignore_index=255)
loss = criterion(x, y)
print(loss)
</code></pre>
<p>Are you sure that the ignored index 255 is causing the issue and not another unexpected target index?</p>

username: sanchit2843
cooked: <p>Hey ptr,<br>
I have debugged it and you are right some other label was causing issue. Other label occured because I was using bilinear interpolation instead when I changed it to nearest the other labels were not appearing. Still although this question is not related to pytorch, I was checking if there is any label on numpy array as:<br>
label[label&gt;18] =255<br>
This statement was not working for numpy array and worked in case of torch tensor which is little strange ig. But anyway thanks for the solution.</p>

username: ptrblck
cooked: <p>Good to hear it’s working now!<br>
That’s indeed strange, as <code>numpy</code> should also be able to perform this check:</p>
<pre><code class="lang-python">x = np.random.randint(0, 20, (100))
x[x &gt; 10] = 255
print(x)
</code></pre>

username: sanchit2843
cooked: <p>Yah thats strange because it took me time to debug problem because of numpy statement only. Anyway thanks for your time.</p>

username: Noam_Salomonski
cooked: <p>What if I want several don’t care classes? for example in a time-series classification, where I have to feed in all the input, but not everything should contribute a gradient, and there are several classes.</p>

username: ptrblck
cooked: <p>In that case you could change the targets to use the same “ignore index” and could then pass this index to the criterion so that it’ll be ignored.</p>

username: Pranjali_Singh
cooked: <p>The ignore_index does not contribute to input gradients, that means the pixels specifies with its value have no role in its output. But it is not the same for testing, right?<br>
In my case training is done really well. But I am getting nan values from the pixels that have been ignored while prediction even after setting ignore_index to the value of the pixels to be ígnored.</p>

username: ptrblck
cooked: <p>Yes, the loss calculation is usually not part of the model inference pipeline and this <code>ignore_index</code> won’t be used. How did you narrow down the invalid values were caused by these pixel locations and why do you include them at all if you want to ignore them always? In case you don’t want to use them at all you could manually mask their values in your eval run.</p>

username: Pranjali_Singh
cooked: <p>My dataset is a 40x40x40 set of voxels, with only some of them unoccupied. I am training U-net only for the occupied voxels for two classes. But when the model is in the prediction phase, the unoccupied voxels are also being predicted, but with nan values, which causes the corresponding loss to be nan too. I want to ignore these voxels in the prediction phase to obtain a numeric loss.<br>
The only solution I could find was to use torch.nan_to_num(). What do you suggest to ignore the non-trained unrequired voxels?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Pranjali_Singh" data-post="14" data-topic="5935">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pranjali_singh/48/62715_2.png" class="avatar"> Pranjali_Singh:</div>
<blockquote>
<p>But when the model is in the prediction phase, the unoccupied voxels are also being predicted, but with nan values, which causes the corresponding loss to be nan too.</p>
</blockquote>
</aside>
<p>It seems you do want to calculate the loss during the validation, in which case <code>ignore_index</code> would still work. My comment explained the common inference use case where no loss calculation is done and only the predictions are used. However, if you are still using <code>nn.CrossEntropyLoss</code> to calculate the loss (and thus have targets) you could still use <code>ignore_index</code> in its calculation.</p>

username: Pranjali_Singh
cooked: <p>Loss Function:</p>
<p><code> loss_fn: torch.nn.Module = nn.CrossEntropyLoss(ignore_index = 2, reduction='mean')</code></p>
<p>This is my code:</p>
<pre><code class="lang-auto">model.eval()
with torch.inference_mode():
        for batch, input_dataset in enumerate(dataloader):
                
            Input = input_dataset[0]
            Target = input_dataset[1]
           
            prediction = model(Input.float())
                
            loss = loss_fn(prediction, Target.long())
            test_loss += loss.item()
</code></pre>
<p>I am still getting nan from the pixels that have target value 2. I was expecting no contribution in loss value from these pixels/ voxels.</p>

username: ptrblck
cooked: <p>The loss values for class index 2 should be zero, but if your logits contain the invalid value (e.g. <code>Inf</code>) the <code>F.log_softmax</code> operation would still create <code>NaN</code>s for all other classes.<br>
I don’t know why and how it was working before, but you might need to apply <code>F.log_softmax</code> manually, making sure the other values won’t be poisoned, and then apply <code>nn.NLLLoss</code> with <code>ignore_index</code>.<br>
However, I’m unsure how you want to solve the issue for <code>F.log_softmax</code> as it needs to normalize the log probabilities.</p>

username: Pranjali_Singh
cooked: <p>The answer for ‘I don’t know why and how it was working before’: my dataset has a set of 7 features. When training the dataset (32,4,40,40,40) by introducing any feature from 3-7 in the channels, the loss for only testing is nan. However, this is not true when I train the Unet using other features from 0-2. The loss function always has ignore_index = 2, but somehow it doesn’t work with a particular set of features. Instead of predicting a 0 for voxels with 2 target values, it predicts nan values for them in this case.</p>
<p>I will apply the loss manually along with F.log_softmax and see how that works.</p>

username: Pranjali_Singh
cooked: <p>One of the outputs:<br>
Epoch: 1 | train_loss: 0.4723 | train_acc: 0.9285 | test_loss: nan | test_acc: 0.9312</p>