Is there anybody happen this error?

mshmoon: <p>/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [1017,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0    main(parser.parse_args())<br>
,0,0], thread: [1018,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
File “main1.py”, line 181, in main<br>
/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [1019,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [1020,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [1021,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [1022,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [1023,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
train(args, model)<br>
File “main1.py”, line 115, in train<br>
epoch_loss.append(loss.data[0])<br>
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THC/generic/THCStorage.c:36</p>

ptrblck: <p>Could you check the range of your target? It seems there is a mismatch between your model output and the target.<br>
For <code>C</code> classes your target should be in the range <code>[0, C-1]</code>.</p>

mshmoon: <p>Yes , yesteday I found it. Thank for you</p>

mshmoon: <p>But The report error place is : epoch_loss.append(loss.data[0])<br>
There is mistery. Why not is loss=criterion(output,target)</p>

ptrblck: <p>Since CUDA calls are executed asynchronously, you should run your code with</p>
<pre><code>CUDA_LAUNCH_BLOCKING=1 python script.py
</code></pre>
<p>This makes sure the right line of code will throw the error message.</p>

mshmoon: <p>Thank you for you help</p>

XiaoAHeng: <p>There is only one target to segmentate in my task. I set the channel as 2  for model output,and for 2 classes my ground truth is in the range[0,1].However, there is still an same error.Could you give me a hint?Thanks in advance!</p>
<pre><code class="lang-auto">Time:2019-07-16 20:02:15.251895, | Epoch:0, step:0, batch_size:1, loss = 0.49627
Time:2019-07-16 20:02:15.361290, | Epoch:0, step:1, batch_size:1, loss = 0.47159
Time:2019-07-16 20:02:15.434024, | Epoch:0, step:2, batch_size:1, loss = 0.45822
Time:2019-07-16 20:02:15.508932, | Epoch:0, step:3, batch_size:1, loss = 0.43967
Time:2019-07-16 20:02:15.578840, | Epoch:0, step:4, batch_size:1, loss = 0.41975
Time:2019-07-16 20:02:15.653170, | Epoch:0, step:5, batch_size:1, loss = 0.41971
Time:2019-07-16 20:02:15.722897, | Epoch:0, step:6, batch_size:1, loss = 0.40062
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [900,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [901,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [902,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [132,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [133,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [134,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [388,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [389,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [390,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [391,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [643,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [644,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [645,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [646,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [647,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [375,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [118,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [119,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [120,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [121,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [630,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [631,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [632,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [633,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [886,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [887,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [888,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [889,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
Traceback (most recent call last):
  File "/data/ch/StructSeg 2019/code1/main.py", line 87, in &lt;module&gt;
    main(config)
  File "/data/ch/StructSeg 2019/code1/main.py", line 48, in main
    solver.train()
  File "/data/ch/StructSeg 2019/code1/solver.py", line 150, in train
    epoch_loss += loss.item()
RuntimeError: CUDA error: device-side assert triggered
</code></pre>

ptrblck: <p>Could you post the shapes of the model’s output and the target?</p>
<aside class="quote no-group" data-username="XiaoAHeng" data-post="7" data-topic="17416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/a88e57/48.png" class="avatar"> XiaoAHeng:</div>
<blockquote>
<p>and for 2 classes my ground truth is in the range[0,1]</p>
</blockquote>
</aside>
<p>This should work, if you provide these values as discrete values in a <code>LongTensor</code>.</p>

XiaoAHeng: <p>[quote=“XiaoAHeng, post:9, topic:17416, full:true”]<br>
ground truth: torch.Size([1, 256, 256]); predicted truth: torch.Size([1, 2, 256, 256])<br>
I have tarnsformed the ground truth into <code>LongTensor</code> by GT = GT.to(self.device, dtype=torch.long). I really don’t know what went wrong</p>

ptrblck: <p>The shapes are alright.<br>
If the target only contains values 0 and 1, it should work:</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss()
output = torch.randn(1, 2, 256, 256)
target = torch.randint(0, 2, (1, 256, 256))
loss = criterion(output ,target)
</code></pre>

XiaoAHeng: <p>All right then ,Thank you anyway!</p>

ptrblck: <p>Add a <code>print</code> or <code>assert</code> statement in the training loop and check the min and max values to be &gt;=0 and &lt;=1.</p>

XiaoAHeng: <p>I have done it ,but it doesnt work:joy:</p>

XiaoAHeng: <p>I used cv2.resize() function to rescale ground truth size , the raw GT szie is 512*512.After rescale processing,my ground truth is not in the range[0,1].When I delete the cv2.resize() function ,the error disappeared.I wonder why this is case:joy:</p>

ptrblck: <p>Most likely since <code>cv2.resize</code> uses some interpolation method (default is linear interpolation if I’m not mistaken) other than “nearest neighbors”.<br>
Try to pass <code>cv2.INTER_NEAREST</code> as the method to this function.</p>

XiaoAHeng: <p>Yes, <code>cv2.INTER_NEAREST</code> has work.You are very nice!Thank you once again!</p>

Mudassar_Awan: <p>Hello, I trained on the following classes [0,1,…15] in step 1. Every thing works ok.  In step 2 I am training classes [0,16,17…20] on a separate classifier. The issue is let(n_classes) is 6 where as the range is unto 20. I am getting this error : t should be between 0 and n_classes. I can’t change labels (0,16,17,…20 ). Suggest me a solution</p>

ptrblck: <p>If class indices <code>[16, 20]</code> are representing different classes than <code>[0, 15]</code>, your classifier should output logits for 21 classes (<code>[0, 20]</code>). Otherwise, if the class indices are representing the same class but are for some reason shifted in their value, you should remap them.</p>

Ning_Albert: <p>Hello, I meet the same problem. When I augment my training set from 226 into 998, the error occured below.</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: device-side assert triggered
/opt/conda/conda-bld/pytorch_1614378073850/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1614378073850/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [1,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1614378073850/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [4,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1614378073850/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [7,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1614378073850/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [8,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1614378073850/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [11,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1614378073850/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [12,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.


</code></pre>
<p>It seems like the error occured when the training set beyond 605. I use nn.CSE(output, label) to compute the loss. Output is in shape of (16, 512) and label is in shape of (16,)</p>

ptrblck: <p>I don’t know what <code>226</code> and <code>998</code> represent, but the error message points to an invalid index in <code>nn.NLLLoss</code> or <code>nn.CrossEntropyLoss</code> (which will call the former).<br>
Make sure your model output is given as <code>[batch_size, nb_classes]</code> and the target as <code>[batch_size]</code> containing values in <code>[0, nb_classes=1]</code> for a multi-class classification use case.</p>