Device side assert triggered while printing loss

username: a_d
cooked: <p>Hello all,<br>
I know this particular error as In the heading has been encountered and discussed at length in the previous discussions on this forum, but I am simply not able to find the reason as to why I am encountering it, I get the error - <code>Device side assert triggered </code> when I am printing my loss.  I am training a segmentation network on the PASCAL VOC dataset and my training loop is as follows -</p>
<pre><code class="lang-auto">for i in range(100):
    epoch_loss = 0
    num_nan = 0
    for _, data in enumerate(dataloader):
        image = data['image'].cuda()
        mask = data['ground_truth'].cuda()
        with autocast():
            loss = model((image, mask))
            print(loss)
        
        scaler.scale(loss).backward()
        scaler.unscale_(optim)
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1)
        scaler.step(optim)
        scaler.update()
        epoch_loss += loss.item()
        del loss
        torch.cuda.empty_cache()
        
        
    print(f'Epoch Loss = {epoch_loss / _}, Number of Nans = {num_nan}')
    #scheduler.step()
</code></pre>
<p>and the stack trace is as follows -</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-686f5bbeb585&gt; in &lt;module&gt;
   1138         with autocast():
   1139             loss = model((image, mask))
-&gt; 1140             print(loss)
   1141 
   1142         scaler.scale(loss).backward()

/opt/conda/lib/python3.7/site-packages/torch/tensor.py in __repr__(self)
    177             return handle_torch_function(Tensor.__repr__, relevant_args, self)
    178         # All strings are unicode in Python 3.
--&gt; 179         return torch._tensor_str._str(self)
    180 
    181     def backward(self, gradient=None, retain_graph=None, create_graph=False):

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in _str(self)
    370 def _str(self):
    371     with torch.no_grad():
--&gt; 372         return _str_intern(self)

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in _str_intern(self)
    350                     tensor_str = _tensor_str(self.to_dense(), indent)
    351                 else:
--&gt; 352                     tensor_str = _tensor_str(self, indent)
    353 
    354     if self.layout != torch.strided:

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in _tensor_str(self, indent)
    239         return _tensor_str_with_formatter(self, indent, summarize, real_formatter, imag_formatter)
    240     else:
--&gt; 241         formatter = _Formatter(get_summarized_data(self) if summarize else self)
    242         return _tensor_str_with_formatter(self, indent, summarize, formatter)
    243 

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in __init__(self, tensor)
     87 
     88         else:
---&gt; 89             nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
     90 
     91             if nonzero_finite_vals.numel() == 0:

RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>Now as it can be seen, the error comes in the print function. I monitor my memory, and memory is not really an issue as I clear cache at the end.</p>
<p>If I do not print -<br>
Then the error comes in the backward call as follows -</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-000e70ad2e12&gt; in &lt;module&gt;
   1139             loss = model((image, mask))
   1140 
-&gt; 1141         scaler.scale(loss).backward()
   1142         scaler.unscale_(optim)
   1143         torch.nn.utils.clip_grad_norm_(model.parameters(), 1)

/opt/conda/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--&gt; 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222 
    223     def register_hook(self, hook):

/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--&gt; 132         allow_unreachable=True)  # allow_unreachable flag
    133 
    134 

RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>What should I do to debug and ensure smooth training<br>
TIA</p>

username: a_d
cooked: <p>I launched the code with <code>CUDA_LAUNCH_BLOCKING=1</code> and the stack trace is as follows -</p>
<pre><code class="lang-auto">&lt;ipython-input-1-0a71a9a41b52&gt; in &lt;module&gt;
   1137         print(mask.shape)
   1138         with autocast():
-&gt; 1139             loss = model((image, mask))
   1140             print(loss)
   1141 

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

&lt;ipython-input-1-0a71a9a41b52&gt; in forward(self, input)
    553             out = self.segaHead(out)
    554 
--&gt; 555             focal = self.focalLoss(out, mask)
    556             dice = dice_loss(mask, out)
    557 

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

&lt;ipython-input-1-0a71a9a41b52&gt; in forward(self, input, target)
     64 
     65         # compute the negative likelyhood
---&gt; 66         logpt = -F.cross_entropy(input, target)
     67         pt = torch.exp(logpt)
     68 

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2466     if size_average is not None or reduce is not None:
   2467         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2468     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2469 
   2470 

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2262                          .format(input.size(0), target.size(0)))
   2263     if dim == 2:
-&gt; 2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2265     elif dim == 4:
   2266         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: cuda runtime error (710) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:115
</code></pre>
<p>and my focal loss definition is -</p>
<pre><code class="lang-auto">
class FocalLoss2d(nn.Module):

    def __init__(self, gamma=0, weight=None, size_average=True):
        super(FocalLoss2d, self).__init__()

        self.gamma = gamma
        self.weight = weight
        self.size_average = size_average

    def forward(self, input, target):
        if input.dim()&gt;2:
            input = input.contiguous().view(input.size(0), input.size(1), -1)
            input = input.transpose(1,2)
            input = input.contiguous().view(-1, input.size(2)).squeeze()
        if target.dim()==4:
            target = target.contiguous().view(target.size(0), target.size(1), -1)
            target = target.transpose(1,2)
            target = target.contiguous().view(-1, target.size(2)).squeeze()
        elif target.dim()==3:
            target = target.view(-1)
        else:
            target = target.view(-1, 1)

        # compute the negative likelyhood
        logpt = -F.cross_entropy(input, target)
        pt = torch.exp(logpt)

        # compute the loss
        loss = -((1-pt)**self.gamma) * logpt

        # averaging (or not) loss
        if self.size_average:
            return loss.mean()
        else:
            return loss.sum()
</code></pre>
<p>What exactly is throwing the error - ?<br>
Added this as a comment so as to not make original post too long</p>

username: ptrblck
cooked: <p>Most likely your targets contain invalid values outside of the expected range <code>[0, nb_classes-1]</code>.<br>
I’m not sure, if you are using an older PyTorch version, as the stacktrace with blocking launches should show the indexing errors.<br>
In case you are already using the latest release, you could use <code>TORCH_SHOW_CPP_STACKTRACES=1</code> to hopefully get a more detailed stacktrace or run the code on the CPU to get a better error.</p>

username: a_d
cooked: <p>Thank you for replying.<br>
I will try it out.</p>
<p>PS - That was indeed the problem, I counted the number of classes wrong <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>