Device-side assert triggered with SpatialClassNLLCriterion.cu

GloomyGhost-Mosquito: <p>Run the code with <code>CUDA_LAUNCH_BLOCKING=1</code></p>
<p>The Pytorch is <code>1.0.1</code></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "tr.py", line 35, in train
    loss = self.loss(output, target)
  File "lib/python3.6/site-packages/torch/nn/functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "lib/python3.6/site-packages/torch/nn/functional.py", line 1792, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu:128
</code></pre>

ptrblck: <p>Is the code running fine of the CPU? The error message raised in CPU might give some more information.<br>
The current error might point to target indices outside of the valid range <code>[0, nb_classes-1]</code>.</p>

GloomyGhost-Mosquito: <p>Yes, Cpu is fine. and i will try it recently</p>

GloomyGhost-Mosquito: <pre><code class="lang-auto">/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [5,0,0], thread: [232,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
</code></pre>
<p>This test shows another error</p>

ptrblck: <p>This error still points to an invalid target index.<br>
It’s a bit strange that your code is running fine on the CPU.<br>
However, try to add an <code>assert</code> statement and check each <code>target</code> batch so only contain values in <code>[0, nb_classes-1]</code>.</p>

GloomyGhost-Mosquito: <p>My current error is generated after a period of training, and the time interval of each error is different.</p>

ptrblck: <p>If you are shuffling the data it’s normal, that the erroneous batch is at different iterations.</p>

GloomyGhost-Mosquito: <p>Thanks, I confirm again</p>

GloomyGhost-Mosquito: <p>after removed the <code>loss.backward()</code> line,I got this error</p>
<pre><code class="lang-auto">/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [5,0,0], thread: [179,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
Traceback (most recent call last):
  File "tr.py", line 42, in train
    self.writer.add_scalar('loss', loss.item())
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>The code is running on GPU</p>
<p>if I use <code>CUDA_LAUNCH_BLOCKING=1</code>,gived:</p>
<pre><code class="lang-auto">/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [1,0,0], thread: [350,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu line=128 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "tr.py", line 36, in train
    loss = self.loss(output, target)
  File "python3.6/site-packages/torch/nn/functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "python3.6/site-packages/torch/nn/functional.py", line 1792, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu:128
</code></pre>
<p>Any Help？</p>