Ignore_index in the cross entropy loss

DKSG: <p>Hi. I think Pytorch calculates the cross entropy loss incorrectly while using the ignore_index option.<br>
The problem is that currently when specifying the ignore_index (say, = k), the function just ignores the value of the target y = k (in fact, it calculates the cross entropy at k but returns 0) but it still makes full use of the logit at index k to calculate the normalization term for other indices not equal to k. I think this is not intended use for most users.</p>
<p>For example, in variable length sequences, people pad the sequence and use the ignore_index as the pad target index in order to avoid considering the padded values (both from inputs and targets). If there are n classes, you have to prepare (n+1) classes for the logit dimension (input of the cross entropy loss) to include the pad class and then ignore it by using the ignore_index option.</p>
<p>Here is some illustrative example:</p>
<pre><code class="lang-auto"># Test cross entropy loss: first create data
x = torch.log(torch.tensor([[2,3,4]],dtype=torch.float)) #  a vector of 3-class logits (one of them could be a padding class)
y1 = torch.tensor([0],dtype=torch.long)
y2 = torch.tensor([1],dtype=torch.long)
y3 = torch.tensor([2],dtype=torch.long)

# calculate the negative logsoftmax for each logit index for comparison
-torch.nn.functional.log_softmax(x,dim=1) # returns tensor([1.5041, 1.0986, 0.8109]) 

# perform logsoftmax and NLL loss at the same time (not use ignore_index yet)
print(torch.nn.functional.cross_entropy(x,y1)) # 1.5041 
print(torch.nn.functional.cross_entropy(x,y2)) # 1.0986 
print(torch.nn.functional.cross_entropy(x,y3)) # 0.8109 

# Now let's ignore the index 0 and find cross entropy loss for index 1
print(torch.nn.functional.cross_entropy(x,y2,ignore_index=0)) # get 1.0986 
# this is the same value as when not excluding the index 0 from the logit; 
# It should ignore the index since the level of the logit index, not just final target index.

# Next let's calculate the correct cross entropy loss when you actually ignore the index 0 completely from both x and y
x_ignore = x[0][1:].view(1,x.shape[-1]-1) # Now we have logits of 2 classes
# the index that is more than the ignore index is decreased by 1
y2_ignore = torch.tensor([0],dtype=torch.long)
y3_ignore = torch.tensor([1],dtype=torch.long)
# cross entropy with ignore_index 0 for the index 1 (which now becomes index 0)
print(torch.nn.functional.cross_entropy(x_ignore,y2_ignore)) # get 0.8473
</code></pre>
<p>In conclusion, I raise this issue in case developers may consider to revise this ignore_index option, but if the current one already follows the intended use (ignore only y not x; hence allowing to backprop through the ignored index of the logit in the normalization term of the softmax), it would be my misunderstanding of how it should work (ignore both x and y at the ignore_index).</p>

adrianjav: <p>I think that’s not what it means the <code>ignore_index</code> parameter. its description says:</p>
<blockquote>
<p>Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is <code>True</code> , the loss is averaged over non-ignored targets.</p>
</blockquote>
<p>So what you specify with <code>ignore_index = k</code> is that the elements of the target that has value <code>k</code> will not contribute to the error. And if you specify <code>size_average = True</code> then the average won’t count those elements either.</p>
<p>Its intended use, or at least the way I use them, is on the cases when you add padding to your input in order to have the same length on all the instances.</p>

DKSG: <p><a class="mention" href="/u/adrianjav">@adrianjav</a> Yes, the target that has index k will not contribute to the error. But the problem is that the class k at the softmax layer is not ignored when calculating the softmax for other classes (the index k still appears in the denominator of the softmax formula since Pytorch did not drop it).</p>
<p>For example, you have only 2 classes: class 1, and class 2 (your padding class). So when you ignore the padding class, the softmax probability of the class 1 must always be one (because there is only one class to consider) but if you try to use ignore_index option, it will not return 1 in general since it still did not eliminate the padding class from consideration (and also there is a chance that it will give higher probability on the padding class for unseen data.)</p>

0e5ec01b9c14c98dcd37: <p><a class="mention" href="/u/dksg">@DKSG</a> I think <code>ignore_index</code> should only be used in the extreme cases where the ignoring target dose not exist in your input.</p>

Rakshit_Kothari: <p>To add onto this topic, it would be very helpful to know exactly how ignore_index works. For instance, consider:</p>
<pre><code class="lang-auto">N = 10
criterion = nn.CrossEntropyLoss(reduction='none', ignore_index=-1)
groundtruth = torch.rand(N, ).ge(0.5).type(torch.LongTensor)
groundtruth[7:] = -1
pred = torch.rand(N, 2)
loss = criterion(pred, groundtruth)
</code></pre>
<p>In such a situation, Pytorch returns a value of 0.0 for elements which are to be ignored. Running the same piece of code with N = 5000 returns weird numbers in the loss for elements to be ignored. Values such as .0646e+24 etc.</p>
<p>Next, how do we backprop on the above said loss?<br>
Can I simply find all elements which aren’t -1, say, <code>loc = groundtruth!=-1</code>, and average by ignoring those values?</p>
<pre><code class="lang-auto">loss = torch.mean(loss[groundtruth!=-1])
loss.backward()
</code></pre>
<p>For some weird reason, the above mentioned situation does not work for me. The code crashes after 10 epochs or so.</p>

ptrblck: <aside class="quote no-group" data-username="Rakshit_Kothari" data-post="5" data-topic="25006">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rakshit_kothari/48/1974_2.png" class="avatar"> Rakshit_Kothari:</div>
<blockquote>
<p>Running the same piece of code with N = 5000 returns weird numbers in the loss for elements to be ignored. Values such as .0646e+24 etc.</p>
</blockquote>
</aside>
<p>This might be a bug, as it seems the values are uninitialized.<br>
I cannot reproduce it using your (modified) code for <code>N = 5000</code>.</p>
<p>Also note, that your criterion should get the prediction as the first argument and the target as the second.<br>
<code>reduction</code> should be set as <code>'none'</code> (lowercase n).</p>

Rakshit_Kothari: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your comment, I edited mine accordingly <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> I can confirm that this bug is occurring in torch 0.4 but cannot reproduce it in 1.0.<br>
Could you comment on the internals of <code>ignore_index</code>?</p>
<p>Is the following code the exact. or near exact, representation of what <code>ignore_index</code> does?</p>
<pre><code class="lang-auto">loss = torch.mean(loss[groundtruth!=-1])
loss.backward()
</code></pre>

ptrblck: <p>Your code snippet will give the same result, as <code>reduction='mean'</code>:</p>
<p>Here is a small example:</p>
<pre><code class="lang-python">N = 10
criterion = nn.CrossEntropyLoss(reduction='mean', ignore_index=-1)
groundtruth = torch.rand(N, ).ge(0.5).type(torch.LongTensor)
groundtruth[7:] = -1
pred = torch.rand(N, 2, requires_grad=True)
loss = criterion(pred, groundtruth)
loss.backward()
print(pred.grad)

# Manual approach
pred.grad.zero_()
target = groundtruth[groundtruth!=-1]
output = pred[groundtruth!=-1]
loss_manual = -1 * F.log_softmax(output, 1).gather(1, target.unsqueeze(1))
loss_manual = loss_manual.mean()
loss_manual.backward()
print(pred.grad)
</code></pre>

Ignacio_Hernandez: <p>Hi, I’d like to ask something related to the last answer.</p>
<p>I’m working on a semi-supervised learning project and my dataloader generates batches with labelled (targets with values 0 to N) and unlabelled (-1) samples.</p>
<p>To keep it simple here, I need a CE loss that only computes the loss on the labelled samples within the batch.</p>
<p>Would this manual approach also work?</p>
<pre><code class="lang-auto">N = 10
groundtruth = torch.rand(N, ).ge(0.5).type(torch.LongTensor)
groundtruth[7:] = -1
pred = torch.rand(N, 2, requires_grad=True)

# ptrblck's manual approach
pred.grad.zero_()
target = groundtruth[groundtruth!=-1]
output = pred[groundtruth!=-1]
loss_manual = -1 * F.log_softmax(output, 1).gather(1, target.unsqueeze(1))
loss_manual = loss_manual.mean()
loss_manual.backward()
print(pred.grad)

# My manual approach
pred.grad.zero_()
criterion = nn.CrossEntropyLoss(reduction='mean')
target = groundtruth[groundtruth&gt;0]
output = pred[groundtruth&gt;0]
loss_manual = criterion(output, target)
loss_manual.backward()
print(pred.grad)
</code></pre>
<p>If it works that would be great, because it’s not very clear to me what is that <code>gather</code> doing there…</p>
<p>Thanks!</p>

ptrblck: <p>Your approach currently also filters out the class0 target indices. If you use <code>groundtruth&gt;=0</code>, you should get the same results as my approach.<br>
The <code>gather</code> operation selects the log probabilities at the target indices in <code>dim1</code>.</p>

neel_g: <p>So, for people coming from search engines, Is it a good idea to leverage <code>ignore_index</code> for padding tokens, especially when they are the majority in a sequence?</p>