Unsolved: NLLLoss throws Assertion `t >= 0 && t < n_classes` failed (when batchsize>1)

Botho_Quinn: <p>Hi all,</p>
<p>So, I’ve been stuck with this issue for days and I’ve tried all the suggestions I could find in the forums.<br>
I am training a model with output layer using log_softmax and loss function NLLLoss.<br>
The problem:</p>
<ol>
<li>when I run the code on cpu all works fine!</li>
<li>when I run the code on gpu with batchsize==1 all works fine!</li>
<li>when I run the code on gpu with batchsize&gt;1 I get the dreaded CUDA error: device-side assert triggered error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/461adb36298944e844f013d3e7b0a2dc4a1f7d9b.png" data-download-href="https://discuss.pytorch.org/uploads/default/461adb36298944e844f013d3e7b0a2dc4a1f7d9b" title="batchsize2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/461adb36298944e844f013d3e7b0a2dc4a1f7d9b_2_529x500.png" alt="batchsize2" data-base62-sha1="a0aXEZUUyzrRMoIWsHguSrkseoj" width="529" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/461adb36298944e844f013d3e7b0a2dc4a1f7d9b_2_529x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/461adb36298944e844f013d3e7b0a2dc4a1f7d9b_2_793x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/461adb36298944e844f013d3e7b0a2dc4a1f7d9b_2_1058x1000.png 2x" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">batchsize2</span><span class="informations">1489×1407 209 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ol>
<p>Upon adding the flag ‘CUDA_LAUNCH_BLOCKING’ = 1 I can see that the error is coming from a cuda assert:</p>
<p>C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:455: block: [0,0,0], thread: [1,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.</p>
<p>As you can see in screen shots 1 and 2, I am logging the min and max target labels to ensure that they do not go below 0 or exceed n-classes-1.</p>
<p>I have been stuck on this problem for days now. Any help at all is appreciated. Please help.</p>

Botho_Quinn: <p>Here is the screen shot for when batchsize == 1 on gpu.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba7f2f7607da87196f939a00bca69a0ff95c568c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba7f2f7607da87196f939a00bca69a0ff95c568c" title="batchsize1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba7f2f7607da87196f939a00bca69a0ff95c568c.png" alt="batchsize1" data-base62-sha1="qBPiMyFLCumS9jWzixMPdCKLl7u" width="690" height="158" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">batchsize1</span><span class="informations">1108×254 17.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I get no errors on this…</p>

Botho_Quinn: <p>Also here is the cuda error after adding ‘CUDA_LAUNCH_BLOCKING’ = 1<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84177782bc76d03b4d667ec94c20e5164d282633.png" data-download-href="https://discuss.pytorch.org/uploads/default/84177782bc76d03b4d667ec94c20e5164d282633" title="cudaassert"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84177782bc76d03b4d667ec94c20e5164d282633.png" alt="cudaassert" data-base62-sha1="iQxiOV1OzIzjXUcClOwHV4ii7WX" width="690" height="87" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cudaassert</span><span class="informations">1632×207 21.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post a minimal executable code snippet reproducing the error, please?</p>

try.my.best.tang: <pre><code class="lang-auto">for index, (x, dim_index_tensor ,real_index) in enumerate(self.train_loader):
                self.optim.zero_grad()
                real_index = real_index.to(self.device)
                x = x.to(self.device)

                dim_index_tensor = dim_index_tensor.to(self.device)
                real_y = dim_index_tensor.permute(1, 0)[0,:]

                # with autocast():
                #     predict_y = self.model(x)
                #     loss = self.loss_func(predict_y, real_y)
                # scaler.scale(loss).backward()
                # scaler.step(self.optim)
                # scaler.update()
                predict_y = self.model(x)       
                loss = self.loss_func(predict_y, real_y)
                loss.backward()
</code></pre>
<p>this code will report  “t &gt;= 0 &amp;&amp; t &lt; n_classes` failed”. When I Change to</p>
<pre><code class="lang-auto">                real_y = dim_index_tensor.permute(1, 0)[0,:]
                real_y = real_y.to(self.device)
</code></pre>
<p>This is right, I guess it’s related to the memory arrangement inside the GPU.</p>

ptrblck: <p>No, the memory layout is irrelevant but changing the shape and slicing the target could change its values too.</p>