RuntimeError: cuda runtime error (710)

username: antgr
cooked: <p>error:<br>
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278</p>
<p>How to trigger it:<br>
print (logits)</p>
<p>about this tensor:<br>
print (logits.shape)<br>
torch.Size([32, 80, 7])</p>
<p>type (logits)<br>
torch.Tensor</p>
<p>Full stack trace:<br>
RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 print (logits)</p>
<p>7 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in <strong>repr</strong>(self)<br>
128         # characters to replace unicode characters with.<br>
129         if sys.version_info &gt; (3,):<br>
–&gt; 130             return torch._tensor_str._str(self)<br>
131         else:<br>
132             if hasattr(sys.stdout, ‘encoding’):</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _str(self)<br>
309                 tensor_str = _tensor_str(self.to_dense(), indent)<br>
310             else:<br>
–&gt; 311                 tensor_str = _tensor_str(self, indent)<br>
312<br>
313     if self.layout != torch.strided:</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _tensor_str(self, indent)<br>
207     if self.dtype is torch.float16 or self.dtype is torch.bfloat16:<br>
208         self = self.float()<br>
–&gt; 209     formatter = _Formatter(get_summarized_data(self) if summarize else self)<br>
210     return _tensor_str_with_formatter(self, indent, formatter, summarize)<br>
211</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in get_summarized_data(self)<br>
240         end = ([self[i]<br>
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])<br>
–&gt; 242         return torch.stack([get_summarized_data(x) for x in (start + end)])<br>
243     else:<br>
244         return torch.stack([get_summarized_data(x) for x in self])</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in (.0)<br>
240         end = ([self[i]<br>
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])<br>
–&gt; 242         return torch.stack([get_summarized_data(x) for x in (start + end)])<br>
243     else:<br>
244         return torch.stack([get_summarized_data(x) for x in self])</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in get_summarized_data(self)<br>
240         end = ([self[i]<br>
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])<br>
–&gt; 242         return torch.stack([get_summarized_data(x) for x in (start + end)])<br>
243     else:<br>
244         return torch.stack([get_summarized_data(x) for x in self])</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in (.0)<br>
240         end = ([self[i]<br>
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])<br>
–&gt; 242         return torch.stack([get_summarized_data(x) for x in (start + end)])<br>
243     else:<br>
244         return torch.stack([get_summarized_data(x) for x in self])</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in get_summarized_data(self)<br>
233     if dim == 1:<br>
234         if self.size(0) &gt; 2 * PRINT_OPTS.edgeitems:<br>
–&gt; 235             return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))<br>
236         else:<br>
237             return self</p>
<p>RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCGeneral.cpp:371</p>
<p>This is triggered in evaluation mode. I could provide the code</p>

username: albanD
cooked: <p>Hi,</p>
<p>Was there anything else printed before the error is raised?<br>
Also you can run your code with <code>CUDA_LAUNCH_BLOCKING=1</code> otherwise the python stack trace is not correct (as cuda calls are asynchronous).</p>
<p>Also if you’re in an interpreter. After one such assert is raised, you need to restart the interpreter as it puts the GPU in a bad state and any attempt to use the GPU will through an error.</p>

username: antgr
cooked: <p>Thanks for the reply!  I use a colab session.<br>
restarted the sesion between faillures.<br>
I have in the code the following line:<br>
!export CUDA_LAUNCH_BLOCKING=1<br>
and I have two errors depending if I have a print or not. The problem I think is with the logit tensor.<br>
The error is:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-43-9064aecd8313&gt; in &lt;module&gt;()
     18         #print("[evaluation] tmp_eval_loss: ", tmp_eval_loss.shape)
     19         logits = model(b_input_ids, token_type_ids=None,
---&gt; 20                        attention_mask=b_input_mask)[0]
     21         #print("logits: ", logits.shape)
     22         #print("logits: ", logits.detach().cpu().numpy())

5 frames
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in __init__(self, tensor)
     85 
     86         else:
---&gt; 87             nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
     88 
     89             if nonzero_finite_vals.numel() == 0:

RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCReduceAll.cuh:327
</code></pre>
<p>and the stack</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-43-9064aecd8313&gt; in &lt;module&gt;()
     18         #print("[evaluation] tmp_eval_loss: ", tmp_eval_loss.shape)
     19         logits = model(b_input_ids, token_type_ids=None,
---&gt; 20                        attention_mask=b_input_mask)[0]
     21         #print("logits: ", logits.shape)
     22         #print("logits: ", logits.detach().cpu().numpy())

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--&gt; 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

&lt;ipython-input-28-7837bc2054a4&gt; in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels)
    196             outputs = (loss,) + outputs
    197         # &lt;------- end of loss 3 -------&gt;
--&gt; 198         print("outputs: ", outputs)
    199         return outputs  # (loss), scores, (hidden_states), (attentions)

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __repr__(self)
    128         # characters to replace unicode characters with.
    129         if sys.version_info &gt; (3,):
--&gt; 130             return torch._tensor_str._str(self)
    131         else:
    132             if hasattr(sys.stdout, 'encoding'):

/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _str(self)
    309                 tensor_str = _tensor_str(self.to_dense(), indent)
    310             else:
--&gt; 311                 tensor_str = _tensor_str(self, indent)
    312 
    313     if self.layout != torch.strided:

/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _tensor_str(self, indent)
    207     if self.dtype is torch.float16 or self.dtype is torch.bfloat16:
    208         self = self.float()
--&gt; 209     formatter = _Formatter(get_summarized_data(self) if summarize else self)
    210     return _tensor_str_with_formatter(self, indent, formatter, summarize)
    211 

/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in __init__(self, tensor)
     85 
     86         else:
---&gt; 87             nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
     88 
     89             if nonzero_finite_vals.numel() == 0:

RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCReduceAll.cuh:327
</code></pre>
<p>And if I remove the print then I have the following error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-43-9064aecd8313&gt; in &lt;module&gt;()
     26     #print("it:", it)
     27     #print("[evaluation] logits: ", logits)
---&gt; 28     logits = logits.cpu().data.numpy()
     29     #print("logits: ", logits)
     30 

RuntimeError: CUDA error: device-side assert triggered
</code></pre>

username: antgr
cooked: <p>and the code:</p>
<pre><code class="lang-python">    model.eval()
    eval_loss, eval_accuracy = 0, 0
    nb_eval_steps, nb_eval_examples = 0, 0
    predictions , true_labels = [], []
    it = 0
    for batch in valid_dataloader:
        batch = tuple(t.to(device) for t in batch)
        b_input_ids, b_input_mask, b_labels, b_tags, b_adu = batch
        #print("[evaluation] b_input_ids: ", b_input_ids.shape)
        #print("[evaluation] b_input_mask: ", b_input_mask.shape)
        #print("[evaluation] b_labels: ", b_labels.shape)
        #print("[evaluation] b_tags: ", b_tags.shape)
        #print("[evaluation] b_adu: ", b_adu.shape)

        with torch.no_grad():
            tmp_eval_loss = model(b_input_ids, token_type_ids=None,
                                  attention_mask=b_input_mask, labels=[b_adu, b_tags, b_labels])[0]
            #print("[evaluation] tmp_eval_loss: ", tmp_eval_loss.shape)
            logits = model(b_input_ids, token_type_ids=None,
                           attention_mask=b_input_mask)[0]
            #print("logits: ", logits.shape)
            #print("logits: ", logits.detach().cpu().numpy())
        #logits = logits.detach().cpu().numpy()
        #logits = logits.cpu().numpy()
        it = it + 1
        #print("it:", it)
        #print("[evaluation] logits: ", logits)
        logits = logits.cpu().data.numpy()
        #print("logits: ", logits)

        label_ids = b_labels.to('cpu').numpy()
        predictions.extend([list(p) for p in np.argmax(logits, axis=2)])
        true_labels.append(label_ids)

        tmp_eval_accuracy = flat_accuracy(logits, label_ids)

        eval_loss += tmp_eval_loss.mean().item()
        eval_accuracy += tmp_eval_accuracy

        nb_eval_examples += b_input_ids.size(0)
        nb_eval_steps += 1
    eval_loss = eval_loss/nb_eval_steps
    print("Validation loss: {}".format(eval_loss))
    print("Validation Accuracy: {}".format(eval_accuracy/nb_eval_steps))
    pred_bios = [bio_vals[p_i] for p in predictions for p_i in p]
    valid_bios = [bio_vals[l_ii] for l in true_labels for l_i in l for l_ii in l_i]
    print("F1-Score: {}".format(f1_score(pred_bios, valid_bios)))
    print(confusion_matrix(valid_bios, pred_bios))
    print(classification_report(valid_bios, pred_bios, digits=4, target_names=bio_vals))
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>I reformatted your post to make it readable.<br>
For your future posts, you can use triple backticks before and after your code to do it:</p>
<pre><code class="lang-auto">```
# Your code
```
</code></pre>
<p>This is really hard to say but it looks like some error occurs and is raises whenever you try to use your logits.<br>
Could you try reducing the size of your code to isolate which part creates the error please?<br>
By removing the data loading and just using random data for example.<br>
Or removing part of your model.</p>
<p>Thanks</p>

username: antgr
cooked: <p>Thank you!<br>
I found the bugy code:<br>
I use label embeddings, but it does not work during evaluation (else), and the error occurs.<br>
Works though during training (if), where the <code>labels</code> variable has value.</p>
<pre><code class="lang-auto">        if labels is not None:
          lab2_emb = self.label2_embedding(labels2)
        else:
          pred_tag = torch.argmax(logits2_normed, dim=2)
          lab2_emb = self.label1_embedding(pred_tag)
</code></pre>
<p>I cannot understand the difference though and why it does not work…<br>
For now I have removed this part of code.</p>

username: antgr
cooked: <p>In else I use <code>label1_embedding</code> ! There should be <code>self.label2_embedding</code>.</p>

username: Shariq_Ali
cooked: <p>I am getting the same error. please help me. Following is my code.</p>
<p>for epoch in range(1, args.num_epochs+1):<br>
epoch_loss = []</p>
<pre><code>    for step, (images, labels) in enumerate(loader):
        if args.cuda:
            images = images.cuda()
            labels = labels.cuda()

        inputs = Variable(images)
        targets = Variable(labels)
        outputs = model(inputs)
        optimizer.zero_grad()
        loss = criterion(outputs, targets[:, 0])
        loss.backward()
        optimizer.step()

        epoch_loss.append(loss.data[0])
        if args.steps_plot &gt; 0 and step % args.steps_plot == 0:
            image = inputs[0].cpu().data
            image[0] = image[0] * .229 + .485
            image[1] = image[1] * .224 + .456
            image[2] = image[2] * .225 + .406
            board.image(image,
                f'input (epoch: {epoch}, step: {step})')
            board.image(color_transform(outputs[0].cpu().max(0)[1].data),
                f'output (epoch: {epoch}, step: {step})')
            board.image(color_transform(targets[0].cpu().data),
                f'target (epoch: {epoch}, step: {step})')
        if args.steps_loss &gt; 0 and step % args.steps_loss == 0:
            average = sum(epoch_loss) / len(epoch_loss)
            print(f'loss: {average} (epoch: {epoch}, step: {step})')
        if args.steps_save &gt; 0 and step % args.steps_save == 0:
            filename = f'{args.model}-{epoch:03}-{step:04}.pth'
            torch.save(model.state_dict(), filename)
            print(f'save: {filename} (epoch: {epoch}, step: {step})')</code></pre>

username: antgr
cooked: <p>Is the following correct?<br>
<code>loss = criterion(outputs, targets[:, 0])</code></p>

username: Shariq_Ali
cooked: <p>I don’t know. when i print this it shows like this</p>
<p>Outputs tensor([[[[0.0454, 0.0453, 0.0452,  …, 0.0459, 0.0459, 0.0460],<br>
[0.0455, 0.0454, 0.0452,  …, 0.0460, 0.0460, 0.0461],<br>
[0.0457, 0.0455, 0.0452,  …, 0.0461, 0.0462, 0.0462],<br>
…,<br>
[0.0460, 0.0459, 0.0459,  …, 0.0457, 0.0457, 0.0458],<br>
[0.0463, 0.0462, 0.0461,  …, 0.0461, 0.0462, 0.0463],<br>
[0.0466, 0.0464, 0.0463,  …, 0.0464, 0.0466, 0.0468]]]],<br>
device=‘cuda:0’, grad_fn=)<br>
targets[:, 0] tensor([[[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0],<br>
…,<br>
[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0]]], device=‘cuda:0’)<br>
Outputs Size torch.Size([1, 1, 256, 256])<br>
targets[:, 0] Size torch.Size([1, 256, 256])</p>

username: antgr
cooked: <p>Usually this error is a shape mismatch. In my case, I used a wrong embedding layer (with different output size), so again a shape mismatch. Try to break your code in simple steps and figure out which step triggers the error. For example unit test your code, or run it interactively. I have not much time to help you out. Maybe the pytorch experts of the forum can provide some better ideas.<br>
<a class="mention" href="/u/alband">@albanD</a> any idea?</p>

username: albanD
cooked: <p>I did not look at the code in details, but from the print and your comment:</p>
<pre><code class="lang-auto">Outputs Size torch.Size([1, 1, 256, 256])
targets[:, 0] Size torch.Size([1, 256, 256])
</code></pre>
<p>The sizes don’t seem to be the same <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> So maybe this is the reason?</p>