Got RuntimeError: Boolean value of Tensor with more than one value is ambiguous during training

James_Lee: <p>Hello. I’m trying to set ignore_index for labels during training with CE loss, my code seemed as follows,</p>
<pre><code>    for i, data in enumerate(train_loader):
        labels[labels &gt; 0.9] = 1
        labels[labels &lt; 0.1] = 0

        # bug occurred here
        labels[0.1&lt;=labels&lt;=0.9] = -1

        loss = nn.CrossEntropyLoss(out, labels, ignore_index=-1)
</code></pre>
<p>I got RuntimeError: Boolean value of Tensor with more than one value is ambiguous<br>
during training.</p>
<p>What’s wrong here? Thanks.</p>

ptrblck: <p>You have to combine the two conditions via <code>&amp;</code>:</p>
<pre><code class="lang-python">labels[(0.1&lt;=labels) &amp; (labels&lt;=0.9)]
</code></pre>

James_Lee: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your prompt reply. After fix this problem, when I ran CUDA_LAUNCH_BLOCKING=1 python train.py I got the following error,</p>
<p>/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [342,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [343,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [344,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [345,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [346,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [347,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
Traceback (most recent call last):<br>
File “train.py”, line 238, in <br>
main()<br>
File “train.py”, line 126, in main<br>
train(net, optimizer)<br>
File “train.py”, line 197, in train<br>
loss1 = criterion_CE(out, torch.squeeze(labels).long())<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/modules/loss.py”, line 947, in forward<br>
return F.cross_entropy(input, target, weight=self.weight,<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 2422, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/home/public/software/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 2220, in nll_loss<br>
ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: cuda runtime error (710) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu:134</p>
<p>It seemed that I should use nn.BCE rather than nn.CrossEntropyLoss here? I found only CrossEntropyLoss support ignore_index for classification while nn.BCE didn’t?</p>
<p>Here’s my loss,</p>
<pre><code>        criterion_CE = nn.CrossEntropyLoss(ignore_index=-1).cuda()
        loss = criterion_CE(out, torch.squeeze(labels).long())</code></pre>

ptrblck: <p>Answered in your <a href="https://discuss.pytorch.org/t/assertion-t-0-t-n-classes-failed-error/133794/2">cross-post</a>.</p>