RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110

username: Sairam_Tabibu
cooked: <p>I’m getting this error, Please help -<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [1,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [8,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [12,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [14,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [17,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [24,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=110 error=59 : device-side assert triggered<br>
Traceback (most recent call last):<br>
File “training-resnet.py”, line 353, in <br>
num_epochs=10)<br>
File “training-resnet.py”, line 191, in train_model<br>
loss = criterion(outputs, labels)<br>
File “/users/sairam.tabibu/pyt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/users/sairam.tabibu/pyt/lib/python3.6/site-packages/torch/nn/modules/loss.py”, line 942, in forward<br>
ignore_index=self.ignore_index, reduction=self.reduction)<br>
File “/users/sairam.tabibu/pyt/lib/python3.6/site-packages/torch/nn/functional.py”, line 2056, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/users/sairam.tabibu/pyt/lib/python3.6/site-packages/torch/nn/functional.py”, line 1871, in nll_loss<br>
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110</p>

username: Sairam_Tabibu
cooked: <p><a href="https://github.com/tabibusairam/project1/blob/master/training-resnet.py" rel="nofollow noopener">My code can be found here.</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>As discussed in the other thread, some of the indices are wrong.<br>
You can run it on the cpu to get a more detailed error message or print the values of the indices and check that they are all correct.</p>