Assertion `t >= 0 && t < n_classes` failed error

James_Lee: <p>When I ran CUDA_LAUNCH_BLOCKING=1 python train.py I got the following error,</p>
<p>/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [342,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [343,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [344,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [345,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [346,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [347,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
Traceback (most recent call last):<br>
File “train.py”, line 238, in<br>
main()<br>
File “train.py”, line 126, in main<br>
train(net, optimizer)<br>
File “train.py”, line 197, in train<br>
loss1 = criterion_CE(out, torch.squeeze(labels).long())<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/modules/loss.py”, line 947, in forward<br>
return F.cross_entropy(input, target, weight=self.weight,<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 2422, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 2220, in nll_loss<br>
ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: cuda runtime error (710) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu:134</p>
<p>It seemed that I should use nn.BCE rather than nn.CrossEntropyLoss here? I found only CrossEntropyLoss support ignore_index for classification while nn.BCE didn’t.</p>
<p>Here’s my loss,</p>
<pre><code class="lang-auto">        criterion_CE = nn.CrossEntropyLoss(ignore_index=-1).cuda()
        loss = criterion_CE(out, torch.squeeze(labels).long())
</code></pre>

ptrblck: <p>The target tensor for <code>nn.CrossEntropyLoss</code> is expected to contain class indices in the range <code>[0, nb_classes-1]</code>, which seems to fail in your script.<br>
Check its values via <code>print(target.min(), target.max())</code> and make sure they are valid.</p>

James_Lee: <p>After <code>print(target.min(), target.max())</code> ,I got target.min() == 0 rather than -1 I expected. I set the label pixel value to -1 follow your advice here <a href="https://discuss.pytorch.org/t/got-runtimeerror-boolean-value-of-tensor-with-more-than-one-value-is-ambiguous-during-training/133762/2" class="inline-onebox">Got RuntimeError: Boolean value of Tensor with more than one value is ambiguous during training - #2 by ptrblck</a> , but it seemed this did not work as I expected? Could you please give me any clues? Many thanks.</p>

James_Lee: <p>Now I got the point. It’s this line of code lead to such strange behavior.</p>
<pre><code>        labels1 = functional.interpolate(labels, size=24, mode='bilinear')

        print("### labels.long().min()", labels.long().min())
        print("### labels.min()", labels.min())

        print("### labels1.long().min()", labels1.long().min())
        print("### labels1.min()", labels1.min())
</code></pre>
<p>I got,</p>
<pre><code class="lang-auto">### labels.long().min() tensor(-1, device='cuda:2')
### labels.min() tensor(-1., device='cuda:2')
### labels1.long().min() tensor(0, device='cuda:2')
### labels1.min() tensor(0., device='cuda:2')
</code></pre>
<p>What’s wrong with functional.interpolate here? Thanks.</p>

ptrblck: <p>You are interpolating values using the bilinear approach and rounding afterwards, which might change the values. I’m not familiar with your use case, but as previously described, the expected target values are in <code>[0, nb_classes-1]</code> unless you use <code>ignore_index</code> for a specific index value.<br>
Your current code crashes, because the target values are not in this range.</p>

James_Lee: <p>Thanks for your reply. Sorry to disturb you again.For the last question, I had set  <code>ignore_index</code> to -1 .</p>
<p>criterion_CE = nn.CrossEntropyLoss(ignore_index=-1).cuda()</p>
<p>…<br>
labels[(0.3 &lt;= labels) &amp; (labels &lt;= 0.7)] = -1<br>
…</p>
<p>loss = criterion_CE(out, torch.squeeze(labels).long())</p>
<p>How could I fix this?</p>

ptrblck: <p>I’m not sure what you are trying to fix.<br>
If you are manually setting some target indices to <code>-1</code>, which is invalid in the default setup, you have to use <code>ignore_index=-1</code>, since otherwise the expected error will be raised.</p>