Device-side assert striggered on multiple loss

AtenaNguyen: <p>Hello everyone,</p>
<p>Recently I got a very strange error of<br>
<code>RuntimeError: CUDA error: device-side assert triggered</code></p>
<p>Basically I have two loss function for adversarial like attack.<br>
<code>loss_function   = nn.CrossEntropyLoss()</code><br>
<code>loss_function2   = nn.CrossEntropyLoss()</code></p>
<p>With a reference pretrain network</p>
<pre><code class="lang-auto">ref_net = torch.load(os.path.join(ref_path, 'best_%d.pth' % ref_epoch))

# Freeze one reference pre-trained network 
for p in ref_net.parameters():
     p.requires_grad = False
</code></pre>
<p>Then during the training I calculate</p>
<pre><code class="lang-auto">        loss = loss_function(outputs, labels)

        ref_outputs = ref_net(meas)
        loss2 = loss_function2(ref_outputs, labels) 
        loss = loss + alpha* loss2
</code></pre>
<p>The error stop at the <code>loss = loss + alpha* loss2</code> which mean I could not add these two loss together?</p>
<p>Any idea to resolve this?<br>
Thank you</p>

ptrblck: <p>Could you run your code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py</code> and post the stack trace here, please?<br>
Also, is your code running fine on the CPU?</p>

AtenaNguyen: <p>Definition of loss function</p>
<pre><code class="lang-auto">    loss_function   = nn.CrossEntropyLoss()   
    ref_net = get_network_ref(args, use_gpu=args.gpu)            
    ref_epoch = findLastCheckpoint(save_dir = ref_path, template='best')         
    ref_net = torch.load(os.path.join(ref_path, 'best_%d.pth' % ref_epoch))

    for p in ref_net.parameters():
        p.requires_grad = False    
    loss_function2   = nn.CrossEntropyLoss()
</code></pre>
<pre><code class="lang-auto">def train(epoch):

    net.train()

    train_loss = 0
    correct = 0
    total = 0 

    for batch_index, (images, labels) in enumerate(cifar100_training_loader):
        if epoch &lt;= args.warm:
            warmup_scheduler.step()

        images = Variable(images)
        labels = Variable(labels)

        labels = labels.cuda()
        images = images.cuda()

        optimizer.zero_grad()       
        outputs, meas, _ = net(images)
        
        ref_outputs, _ = ref_net(meas)
        loss = loss_function(outputs, labels) -  (1/ args.scale) * loss_function2(ref_outputs, labels) 
        
        loss.backward()
        optimizer.step()
		
        train_loss += loss.item()
        _, predicted = outputs.max(1)
        total += labels.size(0)
        correct += predicted.eq(labels).sum().item()

        n_iter = (epoch - 1) * len(cifar100_training_loader) + batch_index + 1

       
        progress_bar(batch_index, len(cifar100_training_loader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)' % (train_loss/(batch_index+1), 100.*correct/total, correct, total))
		
        #update training loss for each iteration
        writer.add_scalar('Train/loss', loss.item(), n_iter)
        writer.add_scalar('Train/Accuracy', 100.*correct/total, epoch)
</code></pre>
<p>This is the error without running CUDA_LAUNCH_BLOCKING=1</p>
<pre><code class="lang-auto">Epoch: 0
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [1,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [4,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [8,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [9,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [10,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [11,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [12,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [13,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [14,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [15,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [16,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [17,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [18,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [19,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [21,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [22,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [23,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [24,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [25,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [26,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [27,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [28,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [29,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [30,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
Traceback (most recent call last):
  File "train_GPU2.py", line 278, in &lt;module&gt;
    train(epoch)
  File "train_GPU2.py", line 75, in train
    loss.backward()
  File "/home/ngcthuong/anaconda3/envs/pyTorch3/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/userA/anaconda3/envs/pyTorch3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

</code></pre>

AtenaNguyen: <p>This is the error with “CUDA_LAUNCH_BLOCKING=1”. I have not run with CPU yet.</p>
<pre><code class="lang-auto">Epoch: 0
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [4,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [8,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [9,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [10,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [11,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [12,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [13,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [14,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [15,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [16,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [17,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [18,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [19,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [21,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [22,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [23,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [24,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [25,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [26,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [27,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [28,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [29,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [30,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=110 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "train_flatcam_v2_GPU2.py", line 278, in &lt;module&gt;
    train(epoch)
  File "train_flatcam_v2_GPU2.py", line 73, in train
    loss = loss_function(outputs, labels) -  (1/ args.scale) * loss_function2(ref_outputs, labels)
  File "/home/ngcthuong/anaconda3/envs/pyTorch3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ngcthuong/anaconda3/envs/pyTorch3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 942, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/ngcthuong/anaconda3/envs/pyTorch3/lib/python3.7/site-packages/torch/nn/functional.py", line 2056, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/ngcthuong/anaconda3/envs/pyTorch3/lib/python3.7/site-packages/torch/nn/functional.py", line 1871, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110
(pyTorch3) ngcthuong@vishnu:~/SourceCode/CVPR2020/A3_LearnCodedMask_Privacy/FaceReg_v03_antiReg$

</code></pre>

ptrblck: <p>Thanks for the code!<br>
The error points to a target index out of the expected range of <code>[0, nb_classes-1]</code>.<br>
Could you check the target tensors for these invalid values?</p>