CUDA error : device-side assert triggered

username: Yolkandwhite
cooked: <p>I got stuck with cuda error and I don’t know why this is suddenly happening.<br>
help me fix this problem please.</p>
<pre><code class="lang-auto">device = 'cuda' if torch.cuda.is_available() else 'cpu'

torch.manual_seed(777)
if device =='cuda':
    torch.cuda.manual_seed_all(777)
print(device)
</code></pre>
<pre><code class="lang-auto">[out] cuda
</code></pre>
<pre><code class="lang-auto">#model
class unet(nn.Module):
    def __init__(self):
        super(unet, self).__init__()
        self.encoder1 = nn.Sequential(nn.Conv2d(3, 64, kernel_size = 3, stride = 1, padding = 1, bias=False),
                                     nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.encoder2 = nn.Sequential(nn.Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.encoder3 = nn.Sequential(nn.Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.encoder4 = nn.Sequential(nn.Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
        
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)

        self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)

        self.pool4 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
        
        self.conv_mid = nn.Sequential(nn.Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),   
                                     nn.Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.decoder4 = nn.Sequential(nn.Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.decoder3 = nn.Sequential(nn.Conv2d(512, 256, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.decoder2 = nn.Sequential(nn.Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.decoder1 = nn.Sequential(nn.Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True),
                                     nn.Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=1, bias=False),
                                     nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                     nn.ReLU(inplace=True))
        
        self.upconv4 = nn.ConvTranspose2d(1024, 1024, kernel_size=(2, 2), stride=2)
        
        self.upconv3 = nn.ConvTranspose2d(512, 512, kernel_size=(2, 2), stride=(2, 2))
        
        self.upconv2 = nn.ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2))
        
        self.upconv1 = nn.ConvTranspose2d(128, 128, kernel_size=(2, 2), stride=(2, 2))
        
        self.conv1x1_out = nn.Conv2d(64, 2, kernel_size=(1, 1), stride=(1, 1))

    def forward(self, x):
        out = self.encoder1(x)
        out = self.pool1(out)
        out = self.encoder2(out)
        out = self.pool2(out)
        out = self.encoder3(out)
        out = self.pool3(out)
        out = self.encoder4(out)
        out = self.pool4(out)
        out = self.conv_mid(out)
        out = self.upconv4(out)
        out = self.decoder4(out)
        out = self.upconv3(out)
        out = self.decoder3(out)
        out = self.upconv2(out)
        out = self.decoder2(out)
        out = self.upconv1(out)
        out = self.decoder1(out)
        out = self.conv1x1_out(out)
        return out
    
model = unet().to(device)
</code></pre>
<pre><code class="lang-auto">[out] 
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-43-810580fbbd02&gt; in &lt;module&gt;()
    104         return out
    105 
--&gt; 106 model = unet().to(device)

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in convert(t)
    421 
    422         def convert(t):
--&gt; 423             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
    424 
    425         return self._apply(convert)

RuntimeError: CUDA error: device-side assert triggered
</code></pre>

username: Tony-Y
cooked: <aside class="quote quote-modified" data-post="1" data-topic="34213">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_ham/48/15296_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered/34213">RuntimeError: CUDA error: device-side assert triggered</a> 
  </div>
  <blockquote>
    I’m putting my code here: 
with torch.no_grad():
        retrieval_one_hot = torch.zeros(k, 10).cuda()
        for batch_idx, (inputs, targets, indexes) in enumerate(testloader):
             inputs, targets, indexes = inputs.to(device), targets.to(device), indexes.to(device)
             net.to(device)
             targets = targets.cuda()#async=True)
             batchSize = inputs.size(0)
             features = net(inputs)
             zz = torch.zeros(batchSize * k, 10).cuda()
             …
  </blockquote>
</aside>

<p>Setting <code>CUDA_LAUNCH_BLOCKING=1</code>, you would get a better traceback.</p>

username: Yolkandwhite
cooked: <pre><code class="lang-auto">os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
</code></pre>
<p>I just used this code above and still can’t solve it.<br>
do I need to change my loss function?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-10-885ac444824b&gt; in &lt;module&gt;()
     22 
     23         pred_mask = model(mri)
---&gt; 24         loss = loss_func(pred_mask, true_mask)
     25 
     26         optimizer.zero_grad()

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--&gt; 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917 
    918 

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022 
   2023 

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
-&gt; 1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1841     else:
   1842         # dim == 3 or dim &gt; 4

RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu:127
</code></pre>

username: Tony-Y
cooked: <p>When <code>device = 'cpu'</code>, does your code work well?</p>

username: Yolkandwhite
cooked: <p>no it doesn’t<br>
does this error something to do with cuda error?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
&lt;ipython-input-23-f939b650666d&gt; in &lt;module&gt;()
     22 
     23         pred_mask = model(mri)
---&gt; 24         loss = loss_func(pred_mask, true_mask)
     25 
     26         optimizer.zero_grad()

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--&gt; 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917 
    918 

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022 
   2023 

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
-&gt; 1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1841     else:
   1842         # dim == 3 or dim &gt; 4

IndexError: Target 255 is out of bounds.
</code></pre>
<h3>edit</h3>
<p>I found that when I use one image data and mask data each, it works well. I think while my whole dataset goes through the loss function, error occurs</p>

username: quanguet
cooked: <p>In fact, your given code has no problem of model initialization and moving the model to <code>GPU</code> for me. I found your error included in this post.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="16" height="16">
      <a href="https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3" target="_blank" title="08:39AM - 14 November 2019" rel="nofollow noopener">Medium – 14 Nov 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:129/68;"><img src="https://miro.medium.com/max/835/1*iuZPebANBQon2FyBa0nu4A.png" class="thumbnail" width="129" height="68"></div>

<h3><a href="https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3" target="_blank" rel="nofollow noopener">CUDA error: Device-side assert triggered</a></h3>

<p>A beginner friendly solution walkthrough for this frustrating (and common) error in Pytorch</p>

  <p><span class="label1">Reading time: 5 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Yolkandwhite
cooked: <p>I have no idea why moving the model to gpu is not working for me. I don’t think the model is wrong. the model itself works good on cpu(there’s still an error because of the loss function, which is not about the model architecture.), but when I move it to gpu the error occurs. I have tried it on google colab, result was same.</p>
<p>I read the post you gave me, but I don’t think it is wrong with the label and output size. since my work is about image segmentation. I only need two label, background and the target.</p>
<aside class="quote no-group" data-username="Yolkandwhite" data-post="1" data-topic="73470">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yolkandwhite/48/17604_2.png" class="avatar"> Yolkandwhite:</div>
<blockquote>
<p>self.conv1x1_out = nn.Conv2d(64, 2, kernel_size=(1, 1), stride=(1, 1))</p>
</blockquote>
</aside>

username: ptrblck
cooked: <p>The error points to the criterion, so you should check the model output and target for valid shapes and values.<br>
I.e. for a vanilla multi-class classification, the target should have the shape <code>[batch_size]</code> and contain values in the range <code>[0, nb_classes-1]</code>.</p>

username: Tony-Y
cooked: <aside class="quote no-group" data-username="Yolkandwhite" data-post="5" data-topic="73470">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yolkandwhite/48/17604_2.png" class="avatar"> Yolkandwhite:</div>
<blockquote>
<p>IndexError: Target 255 is out of bounds.</p>
</blockquote>
</aside>
<p>This error occurs when using inappropriate targets:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; loss_fn = torch.nn.CrossEntropyLoss()
&gt;&gt;&gt; input = torch.randn(3, 2, 24, 24)
&gt;&gt;&gt; target = torch.empty(3, 24, 24, dtype=torch.long).random_(2) * 255
&gt;&gt;&gt; loss = loss_fn(input, target)
IndexError: Target 255 is out of bounds.
</code></pre>