RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

drliu: <p>I’m using  <code>BertForSequenceClassifcation</code>  by huggingface for multi-class classification over 50 classes.</p>
<p>When I try to train my model, I get the runtime error precisely at the line indicated below:</p>
<pre><code class="lang-auto">model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels = 50, 
    output_attentions = False, 
    output_hidden_states = False, 
)

for step, batch in enumerate(train_dataloader):

        b_texts = batch[0].to(device)
        b_attention_masks = batch[1].to(device)
        b_authors = batch[2].to(device)

        model.zero_grad()        

        outputs = model(b_texts, 
                        token_type_ids=None, 
                        attention_mask=b_attention_masks, 
                        labels=b_authors)  &lt;---- ERROR HERE
</code></pre>

ptrblck: <p>Could you run your code with:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and post the stack trace here, please?<br>
Also, does your code run on the CPU without any errors?</p>

drliu: <p>Noob question, but how do a post the stack trace in a jupyter notebook? I set the env with <code>%env  CUDA_LAUNCH_BLOCKING=1</code> and ran the cell, but didn’t get anything that resembled a stack trace</p>

ptrblck: <p>I’m not sure, as I’m not using Jupyter notebooks and often saw the behavior of restarting the kernel before printing out the stack trace. I would recommend to run the script in a terminal, which will print the stack trace.</p>

drliu: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="78545">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA_LAUNCH_BLOCKING=1 python script.py args</p>
</blockquote>
</aside>
<p>Here’s what it outputs:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "scratch2.py", line 193, in &lt;module&gt;
    labels=b_authors)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1176, in forward
    inputs_embeds=inputs_embeds,
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/modeling_bert.py", line 783, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/modeling_bert.py", line 174, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /Users/administrator/nightlies/pytorch-1.0.0/wheel_build_dirs/wheel_3.7/pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191

</code></pre>

ptrblck: <p>Your embedding layer is getting invalid indices. Make sure you are passing inputs with values in the range <code>[0, num_embedding-1]</code>.</p>

M.sadegh_Saberian: <p>Hi,<br>
I have somehow similar problem. i was wondering if you could help me.<br>
I had the same runtime error and ran my code with CUDA_LAUNCH_BLOCKING=1 and here is the output:</p>
<pre><code class="lang-auto">/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=110 error=710 : device-side assert triggered
Traceback (most recent call last):
  File "MUSK2.py", line 215, in &lt;module&gt;
    loss = train(epoch, train_loader, model, criterion, optimizer)
  File "MUSK2.py", line 158, in train
    loss = criterion(output, target)
  File "/local-scratch/localhome/msaberia/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/local-scratch/localhome/msaberia/pytorch/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/local-scratch/localhome/msaberia/pytorch/lib/python3.6/site-packages/torch/nn/functional.py", line 2021, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/local-scratch/localhome/msaberia/pytorch/lib/python3.6/site-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110
</code></pre>

ptrblck: <p>The error points to an out of bounds index for <code>nn.NLLLoss</code> (or <code>nn.CrossEntropyLoss</code>):</p>
<pre><code class="lang-python">Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
</code></pre>
<p>Make sure to pass the target with values in the range <code>[0, nb_classes-1]</code>.</p>

M.sadegh_Saberian: <p>that was it!<br>
got it! thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Jiazhi_Li: <p>Hi,<br>
I have somehow similar problem. i was wondering if you could help me.<br>
I had the same runtime error and ran my code with CUDA_LAUNCH_BLOCKING=1 and here is the output:</p>
<p>Traceback (most recent call last):<br>
File “MINE_main.py”, line 62, in <br>
result = util.train((x, y), mine_net,mine_net_optim, checkpoint_template, iter_num = iter_num, save_model=False, verbose = True)<br>
File “/nas/home/jiazli/Bias_assessment/model/MINE/util.py”, line 77, in train<br>
mi_lb, ma_et = learn_mine(batch, mine_net, mine_net_optim, ma_et)<br>
File “/nas/home/jiazli/Bias_assessment/model/MINE/util.py”, line 32, in learn_mine<br>
mi_lb, t, et = mutual_information(joint, marginal, mine_net)<br>
File “/nas/home/jiazli/Bias_assessment/model/MINE/util.py”, line 18, in mutual_information<br>
t = mine_net(joint)<br>
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/nas/home/jiazli/Bias_assessment/model/MINE/MINE.py”, line 19, in forward<br>
output = F.elu(self.fc1(input))<br>
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/linear.py”, line 91, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py”, line 1674, in linear<br>
ret = torch.addmm(bias, input, weight.t())<br>
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code></p>

ptrblck: <p>Could you post an executable code snippet to reproduce this issue as well as your PyTorch, CUDA version, and used GPU?</p>

Jiazhi_Li: <p>Thank you very much. When I update the latest version, the problem is solved.</p>

tlqn: <p>Hello,</p>
<p>I also run into the same problem these days and not sure how to tackle it. I have already set CUDA_LAUNCH_BLOCKING=1 but I still receive the same error. Seems like it’s because the number of labels and number of output units is not equal? but I’m not sure how to verify this.</p>
<p>Could you please take a look at the error below and I would really appreciate if you could give me some suggestions. I’m quite new with PyTorch and deep learning so please bear with me.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;timed eval&gt; in &lt;module&gt;

~/.local/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path, trial)
    761                     continue
    762 
--&gt; 763                 tr_loss += self.training_step(model, inputs)
    764                 self.total_flos += self.floating_point_ops(inputs)
    765 

~/.local/lib/python3.6/site-packages/transformers/trainer.py in training_step(self, model, inputs)
   1111                 loss = self.compute_loss(model, inputs)
   1112         else:
-&gt; 1113             loss = self.compute_loss(model, inputs)
   1114 
   1115         if self.args.n_gpu &gt; 1:

~/.local/lib/python3.6/site-packages/transformers/trainer.py in compute_loss(self, model, inputs)
   1135         Subclass and override for custom behavior.
   1136         """
-&gt; 1137         outputs = model(**inputs)
   1138         # Save past state if it exists
   1139         if self.args.past_index &gt;= 0:

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    159             return self.module(*inputs[0], **kwargs[0])
    160         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--&gt; 161         outputs = self.parallel_apply(replicas, inputs, kwargs)
    162         return self.gather(outputs, self.output_device)
    163 

~/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
    169 
    170     def parallel_apply(self, replicas, inputs, kwargs):
--&gt; 171         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
    172 
    173     def gather(self, outputs, output_device):

~/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
     84         output = results[i]
     85         if isinstance(output, ExceptionWrapper):
---&gt; 86             output.reraise()
     87         outputs.append(output)
     88     return outputs

~/.local/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
    426             # have message field
    427             raise self.exc_type(message=msg)
--&gt; 428         raise self.exc_type(msg)
    429 
    430 

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/tlqn/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/tlqn/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tlqn/.local/lib/python3.6/site-packages/transformers/modeling_albert.py", line 796, in forward
    return_dict=return_dict,
  File "/home/tlqn/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tlqn/.local/lib/python3.6/site-packages/transformers/modeling_albert.py", line 690, in forward
    return_dict=return_dict,
  File "/home/tlqn/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tlqn/.local/lib/python3.6/site-packages/transformers/modeling_albert.py", line 421, in forward
    hidden_states = self.embedding_hidden_mapping_in(hidden_states)
  File "/home/tlqn/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tlqn/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/tlqn/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
</code></pre>

ptrblck: <p>The error might also be raised, if you are running out of memory on the GPU and cublas is unable to create the handle.<br>
Could you reduce the batch size and check, if the code is working?</p>

tlqn: <p>I have more than 50% of the memory on GPU at the moment, I reset the kernel and even reduced the batch size to 1 but I still received the same error as above, or this error<code> RuntimeError: CUDA error: device-side assert triggered</code></p>

ptrblck: <p>Thanks for the update. Could you post an executable code snippet to reproduce this issue as well as the system information (PyTorch, CUDA, cudnn versions and the used GPUs)?</p>

Bihy: <p>I’m getting the same problem trying to run BERT. any solution please ??<br>
4 GPUs<br>
pytorch = 1.6.0<br>
cuda : 10.2</p>
<p>the code I’m trying to run<br>
</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/linlei1214/SITS-BERT" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/13514001?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/linlei1214/SITS-BERT" target="_blank" rel="noopener nofollow ugc">linlei1214/SITS-BERT</a></h3>


  <p><span class="label1">Contribute to linlei1214/SITS-BERT development by creating an account on GitHub.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Are you seeing the same issue after upgrading PyTorch to the latest stable release (<code>1.7.1</code>) and/or reducing the batch size?</p>

Bihy: <p>displays the same error. Just to mention the code work well with CPU</p>

ptrblck: <p>Thanks for the update. Which GPUs, CUDA, cudnn, NVIDIA driver, and PyTorch versions are you using?<br>
Also, which dataset and script from the repository are you calling?</p>