Error of cuda runtime error (59)

Wanger-SJTU: <pre><code class="lang-auto">RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THC/generic/THCStorage.c:36
</code></pre>
<p>the  detailed info is</p>
<blockquote>
<p>/opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [886,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [892,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [893,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [894,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [895,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [804,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.</p>
</blockquote>
<blockquote>
<p>main()<br>
File “train.py”, line 135, in main<br>
train_FCN(opt)<br>
File “train.py”, line 82, in train_FCN<br>
trainer.train()<br>
File “/media/sjtu/831bebd9-c866-4ece-b878-5dbd68e5ca50/sjtu/CH/seg_transfer/models/trainer.py”, line 240, in train<br>
self.train_epoch()<br>
File “/media/sjtu/831bebd9-c866-4ece-b878-5dbd68e5ca50/sjtu/CH/seg_transfer/models/trainer.py”, line 169, in train_epoch<br>
self.validate()<br>
File “/media/sjtu/831bebd9-c866-4ece-b878-5dbd68e5ca50/sjtu/CH/seg_transfer/models/trainer.py”, line 80, in validate<br>
if np.isnan(float(loss.data.item())):<br>
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524580978845/work/aten/src/THC/generic/THCStorage.c:36</p>
</blockquote>
<p>actually the error occurs while calculating the loss function <code>CrossEntropyLoss2d</code> for segmentation task.<br>
I checked the label image, there is no <code>-1</code> and the num of output channels  is correct.</p>
<p>Is there anyone get the point where the problem?</p>

albanD: <p>Hi,</p>
<p>To get a better error message you can run the same thing on CPU.<br>
In this case the error is that your label is not in 0 &lt;= your_label &lt; n_classes.<br>
If you say that no label is negative, then one must be greater than your number of classes.<br>
This is easy to check by checking the value of your label tensor and make sure it they are not too large.</p>

Wanger-SJTU: <p>I have checked all the value of the label. None is greater than the num of classes. wired bug.</p>

LMA: <p><a href="https://discuss.pytorch.org/u/Wanger-SJTU">Wanger-SJTU</a>, did you resolve the problem? I have encountered a similar problem. Thanks.</p>

Ontheroad123: <p>i have met this error ,and i spent total day to check my code,but finally i found the error was valmask data.first you should check your all label image,make sure the label is [0, n_class-1]</p>

Jo-w: <p>Thank you so much! That is exactly what I need. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>