How to fix my cuda error?

GISKing: <p>Hi.<br>
I am Mikhail.<br>
I am training my model but error is occurred as follows:<br>
C:/w/1/s/tmp_conda_3.7_104508/conda/conda-bld/pytorch_1572950778684/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: block: [0,0,0], thread: [903,0,0] Assertion  <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code>  failed.<br>
THCudaCheck FAIL file=…\aten\src\THC\THCCachingHostAllocator.cpp line=278 error=710 : device-side assert triggered<br>
File “c:\Users\OCR Labs\Documents\ERFNet-CULane-PyTorch\train_erfnet.py”, line 173, in train<br>
loss_exist = criterion_exist(output_exist, target_exist_var)<br>
File “C:\Users\OCR Labs\Anaconda3\envs\torch-env\lib\site-packages\torch\nn\modules\module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\OCR Labs\Anaconda3\envs\torch-env\lib\site-packages\torch\nn\modules\loss.py”, line 601, in forward<br>
reduction=self.reduction)<br>
File “C:\Users\OCR Labs\Anaconda3\envs\torch-env\lib\site-packages\torch\nn\functional.py”, line 2116, in binary_cross_entropy_with_logits<br>
return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)<br>
RuntimeError: CUDA error: device-side assert triggered<br>
I am going to get 5 classes as a output class, so i have set 5 class.<br>
My error is occured in following command line.<br>
loss = criterion(torch.nn.functional.log_softmax(output, dim=1), target_var)<br>
here output tensor is tensor([[0.4826, 0.4970, 0.4989, 0.4563, 0.5411]]) and target_var is tensor([[1., 1.,1.,1.,1.]])<br>
How can i fix this issue? Thank you very much!</p>

vainaijr: <p>I think you need to use<br>
torch.sigmoid with binary_cross_entropy<br>
or use binary_cross_entropy_with_logits</p>
<p>if you use log_softmax(), then it will give values less than minus one, for example,</p>
<pre><code class="lang-auto">output = torch.tensor([[0.4826, 0.4970, 0.4989, 0.4563, 0.5411]])
torch.log_softmax(output, dim=1)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">tensor([[-1.6224, -1.6080, -1.6061, -1.6487, -1.5639]])
</code></pre>
<p>as</p>
<pre><code class="lang-auto">torch.softmax(output, dim=1)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">tensor([[0.1974, 0.2003, 0.2007, 0.1923, 0.2093]])
</code></pre>
<p>and if you take log of these values then, you will get output of torch.log_softmax, for example,</p>
<pre><code class="lang-auto">import math
math.log(0.1974)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">-1.6225231519827559
</code></pre>
<p>if you use binary_cross_entropy_with_logits, with above values, then it would be like,</p>
<pre><code class="lang-auto">torch.binary_cross_entropy_with_logits(tensor([[-1.6224, -1.6080, -1.6061, -1.6487, -1.5639]]), target)
</code></pre>
<p>which gives error, as formula for binary_cross_entropy_with_logits has log(input), and log of negative numbers is not defined.</p>
<p>if you use output directly, for example,</p>
<pre><code class="lang-auto">torch.binary_cross_entropy_with_logits(output, target)
</code></pre>
<p>then you will get</p>
<pre><code class="lang-auto">tensor([[0.4807, 0.4752, 0.4745, 0.4908, 0.4588]])
</code></pre>
<p>or if you use</p>
<pre><code class="lang-auto">criterion = nn.BCELoss(reduction='none')
criterion(torch.sigmoid(output), target)
</code></pre>
<p>then you will get</p>
<pre><code class="lang-auto">tensor([[0.4807, 0.4752, 0.4745, 0.4908, 0.4588]])
</code></pre>
<p>maybe this is what you are looking for.<br>
or if you do not specify</p>
<pre><code class="lang-auto">reduction='none'
</code></pre>
<p>then it will take mean of these values by default and give loss.</p>
<p>BCEwithLogitsLoss, uses log-sum-exp, which makes computing loss faster as compared to using sigmoid and then BCELoss.</p>