RuntimeError: CUDA error: device-side assert triggered after the first epoch

username: Luis_Costa
cooked: <pre><code class="lang-auto">/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
</code></pre>
<p>Hello, I’m getting this error after the first epoch and I can’t figure what is wrong.</p>
<p><a href="https://gist.github.com/luistelmocosta/ebfde00e892750dfed726b345dffb9e9" rel="nofollow noopener">Here’s</a> my implementation. The error is happening on <code>running_loss += loss.item() * inputs.size(0)</code>.</p>
<p>Any idea what is going on?</p>

username: Luis_Costa
cooked: <p>I tried with a smaller dataset and it works flawlessly. Why doest it crashes with a bigger dataset?</p>

username: ptrblck
cooked: <p>The error message is most likely pointing to a wrong line, since CUDA operations are called asynchronously.<br>
However, it seems your target values are out of bounds. I assume you are using <code>nn.CrossEntropyLoss</code>, which expects a <code>torch.LongTensor</code> with values in the range <code>[0, nb_classes-1]</code>.<br>
Apparently, your smaller dataset does not contain the wrong labels.</p>
<p>You could print your current target tensor and run your code with<br>
<code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code><br>
to see which target tensor contains the wrong indices.</p>

username: Luis_Costa
cooked: <p>I can’t believe I actually had an empty folder inside my <code>./train</code>. I’m so sorry.</p>
<p>Thank you very much for your time.</p>

username: ptrblck
cooked: <p>No need to be sorry! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"><br>
I’m glad it’s working now!</p>

username: kl_divergence
cooked: <p>Hi, I’m also facing the same issue:<br>
Here’s the line<br>
<code>classification_loss = F.cross_entropy(class_logits, labels)</code><br>
The dim of class_logits and labels are <code>torch.Size([512, 12]) torch.Size([512])</code><br>
I’ve 15 classes and here are my labels</p>
<p>tensor([ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0, 12,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0, 12,  0,  0,  0,  0,  0,  1,  0,  0,  0,  5,  0,<br>
0,  0,  0,  5,  0,  1,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0, 12,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  1,  0,  0, 12,  0,<br>
0,  0,  0,  0,  0,  0,  0,  1,  0,  0,  0,  0,  0,  0,  0,  0,  0,  5,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  1,  0,  0,  0,  0,  0,<br>
0,  0,  1,  0,  0,  0,  0,  5,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  5,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0,  0,  0,  0,  0,  1,  0,  0,  0,  0,  0,  0,  0,  0,  0,<br>
0,  0,  0,  0, 12,  1,  1,  5], device=‘cuda:0’)</p>
<p>I’m not seeing any wrong labels at all.</p>
<p>Here’s the CUDA Error:<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=110 error=59 : device-side assert triggered</p>
<p>File “/ml/temp/object_detection/models/roi_heads.py”, line 87, in fastrcnn_loss<br>
classification_loss = F.cross_entropy(class_logits, labels)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py”, line 2056, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py”, line 1871, in nll_loss<br>
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110</p>
<p>My labels start from 0. As per what you said earlier, my labels are in the range of 0-14, what can be the issue ? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>Based on the shape of your output, the maximal class index should be 11, since you are dealing with 12 classes. All values &gt;=12 will yield this error, which is the case in your example.</p>

username: TheStoryOfTheStone
cooked: <p>hi ,I am encounted this problem too,because some classes of my dataset just have one ,so it either in trainset or testset.The label is just like this:trainset_label[0,2,3,5],testset_labe[1,2,4],so I set the num_classes=6,how can I fix this problem?thank you very much!</p>

username: ptrblck
cooked: <p>As long as your model output contains logits (log probabilities) for all six classes, e.g. as <code>[batch_size, 6]</code>, there should be no problem from the point of view of your code.<br>
That being said, I would see a problem that some labels (class1 and class4) are only available in the test set, so that your model shouldn’t have a chance to learn these classes.</p>

username: TheStoryOfTheStone
cooked: <p>Thank you for your reply!My dataset is about DNA sequence,such as a dog’s sequence maybe like this,[ATCGATCG],it contain four alphabet:ATCG,the order of the DNA sequence contain some information that we can conclude which species it is.So,it’s difficult to do augmentation,not as easy as images,if I just have one dog,which means I just have one DNA sequence.Thus,I split the datasets in random.If you have  some good advice,please tell me,thank you,sir.</p>

username: ptrblck
cooked: <p>I’m not familiar with DNA sequencing, but I would try to split the dog sequence into a training, validation, and test set. This would make sure, that each data split was drawn from approx. the same distribution.</p>
<p><a class="mention" href="/u/gisking">@GISKing</a><br>
It seems your model output has the shape <code>[batch_size, 5]</code>, which would mean that your target should be a <code>LongTensor</code> with the shape <code>[batch_size]</code> containing the class indices.</p>

username: TheStoryOfTheStone
cooked: <p>Thank you for your reply!If it is singleton,it means that the classes just have one sequence,so if I want to split the data to training,validation and test set,I must copy the one sequence three times.Is that data leak?The model had seen the ‘answer’.</p>

username: gwwj723
cooked: <p>My label is all right, the label is 0 or 1 for 2 class, it is work in previous running and anyother networks, but when i repeat the program, the error told me the same error above.</p>