How to set ignore_index for a BCE task with nn.CrossEntropyLoss

James_Lee: <p>Hi there. I found that <code>torch.nn.</code> <code>BCELoss</code> dindn’t offer an <strong>ignore_index</strong> param like in  <code>torch.nn.</code> <code>CrossEntropyLoss</code> . I tried implementing BCE loss by calling nn.CrossEntropyLoss with preset ignore_index=-1 but failed. Does anyone got any ideas on this? Thanks.</p>

sarat: <p>What’s the error message?</p>
<p>Do u have labels that you want to ignore labelled as with -1?</p>

James_Lee: <p>My code,</p>
<pre><code class="lang-auto">criterion_CE = nn.CrossEntropyLoss(ignore_index=-1).cuda()

…
labels[(0.3 &lt;= labels) &amp; (labels &lt;= 0.7)] = -1
…

loss = criterion_CE(out, torch.squeeze(labels).long())

</code></pre>
<p>Error,</p>
<p>/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [761,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [500,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [501,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [502,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [503,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [504,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [505,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [884,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [885,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [886,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [887,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [888,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [889,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [5,0,0], thread: [890,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu line=134 error=710 : device-side assert triggered<br>
Traceback (most recent call last):<br>
File “train.py”, line 239, in <br>
main()<br>
File “train.py”, line 121, in main<br>
train(net, optimizer)<br>
File “train.py”, line 199, in train<br>
loss5 = criterion_CE(out, torch.squeeze(labels).long())<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/modules/loss.py”, line 947, in forward<br>
return F.cross_entropy(input, target, weight=self.weight,<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 2422, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 2220, in nll_loss<br>
ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: cuda runtime error (710) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu:134</p>

James_Lee: <p>And my labels had been normalized to [0,1] before training.Here I would like to “ignore” those between 0.3~0.7 when calculating BCE loss.</p>

sarat: <aside class="quote no-group" data-username="James_Lee" data-post="4" data-topic="133838">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/james_lee/48/29487_2.png" class="avatar"> James_Lee:</div>
<blockquote>
<p>normalized to [0,1]</p>
</blockquote>
</aside>
<p>If you are creating labels for binary classification by some process, ensure that labels are 0 and 1. Any ignores ranges can be specified as -1.</p>
<p>Check if your targets list has 3 unique values, they are 0, 1, -1.</p>

James_Lee: <pre><code>        criterion_CE = nn.CrossEntropyLoss(ignore_index=-1).cuda()
        labels1 = functional.interpolate(labels, size=24, mode='bilinear')
        loss1 = criterion_CE(attention1, torch.squeeze(labels1).long())
</code></pre>
<p>Here’s the way I caculate my final BCE loss. It seem that functional. interpolate turned the -1s into 0s. Does the functional. interpolate matter here?How could I avoid this?</p>

SaisaiSun: <p>Hi James, did you find the solution of adding <strong>ignore_index</strong> for <code>torch.nn.</code> <code>BCELoss</code>?</p>

SaisaiSun: <p>Here is one. <a href="https://discuss.pytorch.org/t/ignore-padding-area-in-loss-computation/95804" class="inline-onebox">Ignore padding area in loss computation</a></p>