CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

BIGWangYuDong: <p>CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)<br>
I’m trying to use torch.mm() but sometimes I got this error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79d54786f7a67247b4100af6d79510195391be4f.png" data-download-href="https://discuss.pytorch.org/uploads/default/79d54786f7a67247b4100af6d79510195391be4f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79d54786f7a67247b4100af6d79510195391be4f_2_690x79.png" alt="image" data-base62-sha1="hnMIOYqmj8gkCvml8aNVdoFIRPN" width="690" height="79" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79d54786f7a67247b4100af6d79510195391be4f_2_690x79.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79d54786f7a67247b4100af6d79510195391be4f_2_1035x118.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/9/79d54786f7a67247b4100af6d79510195391be4f.png 2x" data-dominant-color="442139"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1210×139 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I run the code under Pytorch 1.4 with CUDA 9.0<br>
Meanwhile, I also run the code with Pytorch 1.5, CUDA 9.0. And sometimes, I also get this bug.<br>
More details in <a href="https://github.com/WXinlong/SOLO/issues/180" rel="noopener nofollow ugc">#180</a></p>

ptrblck: <p>This error might be raised, if you are running out of memory and cublas fails to create the handle, so try to reduce the memory usage e.g. via a smaller batch size.</p>

SirAlexI: <p>Hi, I get the same error.<br>
Everything works fine when I use only one gpu. But, if I use two, I get the same error. And, I reduced the size of my problem so it runs and finishes in one cpu swiftly.</p>
<p>File “/home/server/Escritorio/AlejandroF/agfa/simple-transformers/train_language_model.py”, line 138, in<br>
model.train_model(train_filename)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/simpletransformers/language_modeling/language_modeling_model.py”, line 431, in train_model<br>
global_step, training_details = self.train(<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/simpletransformers/language_modeling/language_modeling_model.py”, line 774, in train<br>
model(inputs, labels=labels)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py”, line 161, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py”, line 171, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py”, line 86, in parallel_apply<br>
output.reraise()<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/_utils.py”, line 428, in reraise<br>
raise self.exc_type(msg)<br>
RuntimeError: Caught RuntimeError in replica 0 on device 0.<br>
Original Traceback (most recent call last):<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker<br>
output = module(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py”, line 1329, in forward<br>
outputs = self.bert(<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py”, line 991, in forward<br>
encoder_outputs = self.encoder(<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py”, line 582, in forward<br>
layer_outputs = layer_module(<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py”, line 470, in forward<br>
self_attention_outputs = self.attention(<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py”, line 401, in forward<br>
self_outputs = self.self(<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py”, line 267, in forward<br>
mixed_query_layer = self.query(hidden_states)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/modules/linear.py”, line 93, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/home/server/anaconda3/envs/agfa/lib/python3.9/site-packages/torch/nn/functional.py”, line 1692, in linear<br>
output = input.matmul(weight.t())<br>
<strong>RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling <code>cublasCreate(handle)</code></strong></p>

SirAlexI: <aside class="quote no-group" data-username="SirAlexI" data-post="3" data-topic="125450">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siralexi/48/40243_2.png" class="avatar"> SirAlexI:</div>
<blockquote>
<p>in one cpu swiftly.</p>
</blockquote>
</aside>
<p>I meant " in one gpu swiftly."</p>

ptrblck: <p><code>nn.DataParallel</code> (which seems to be used in your use case) could create an imbalanced memory usage and could thus cause an OOM on the default device, which is why we recommend to use <code>DistributedDataParallel</code> with a single process per GPU instead.<br>
I assume you are also running OOM, so I would recommend to try to apply <code>DDP</code>.</p>

SirAlexI: <p>Thanks for your answer.</p>

BIGWangYuDong: <p>Hi, thanks for your reply. Is there any way that I can save the memories? I have tried to use <code>torch.einsum</code> but seems this cannot save the memories. Any Idea?</p>
<p>Here is an simple example:</p>
<pre><code class="lang-python"># old version
inter_matrix = torch.mm(flatten_masks, flatten_masks.transpose(1, 0))
# new version
inter_matrix = torch.einsum('ik, kj -&gt; ij', flatten_masks, flatten_masks.transpose(1, 0))
</code></pre>

ptrblck: <p>I’m not sure what your use case is. Would you like to lower the memory usage on your GPU?</p>

BIGWangYuDong: <p>Yep, here is a script that I use to check the GPU memory and running time</p>
<pre><code class="lang-python">import torch
import numpy as np
import time

flatten_masks = np.random.random((800, 60800))

flatten_masks = torch.from_numpy(flatten_masks).cuda(device=0)
print()
t1 = time.time()
i = 0
while i &lt; 2500:
    if i == 500:
        t1 = time.time()
    # old version 
    inter_matrix = torch.mm(flatten_masks, flatten_masks.transpose(1, 0))
    # new version
    # inter_matrix = torch.einsum('ik, kj -&gt; ij', flatten_masks, flatten_masks.transpose(1, 0))
    i += 1
t2 = time.time()
print(t2-t1)
</code></pre>
<p>result:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:center"></th>
<th style="text-align:center">torch.mm</th>
<th style="text-align:center">torch.einsum</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">process time (s)</td>
<td style="text-align:center">4.69856</td>
<td style="text-align:center">4.5713</td>
</tr>
<tr>
<td style="text-align:center">GPU memory  (MiB)</td>
<td style="text-align:center">807</td>
<td style="text-align:center">807</td>
</tr>
</tbody>
</table>
</div><p>And my question is what code I can use that I can save the memory?</p>

Niraj: <p>Hello (conituing on this thread), can you provide me with some guidance:<br>
I am facing a similar error when i try to call on my encoder (of the Transformer block), Error: …<br>
 → 4765         proj = linear(q, w, b)<br>
4766         # reshape to 3, E and not E, 3 is deliberate for better memory coalescing and keeping same order as chunk()<br>
4767         proj = proj.unflatten(-1, (3, E)).unsqueeze(0).transpose(0, -2).squeeze(-2).contiguous()</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling <code>cublasCreate(handle)</code></p>
<pre><code class="lang-auto">src = self.encoder(
            src=src
            ).to(device)
################################################
encoder_layer = nn.TransformerEncoderLayer(
            d_model=dim_val, 
            nhead=n_heads,
            dim_feedforward=dim_feedforward_encoder,
            dropout=dropout_encoder,
            batch_first=batch_first
            ).to(device)

        # Stack the encoder layers in nn.TransformerDecoder
        self.encoder = nn.TransformerEncoder(encoder_layer=encoder_layer,num_layers=n_encoder_layers, norm=True).to(device)
</code></pre>
<p>Output:<br>
cuda 1 From model.forward(): Size of src as given to forward(): torch.Size([10, 20, 4]) From model.forward(): tgt size = torch.Size([10, 20, 2]) From model.forward(): Size of src after input layer: torch.Size([10, 20, 2204]) step x shape: torch.Size([10, 20, 2204]) Pe_map shape: torch.Size([10, 1, 2204]) From model.forward(): Size of src after pos_enc layer: torch.Size([10, 20, 2204])</p>
<p>Output exceeds the <a>size limit</a>. Open the full output data <a>in a text editor</a></p>
<p>--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last)…</p>
<p>Why this error and how do i go about handling this error?</p>

ptrblck: <p>You are also most likely running out of memory which causes the <code>cublasHandle</code> init error as it tries to allocate a workspace. I would recommend reducing the memory usage e.g. by lowering the batch size and check if this fixes the error.</p>

Nidhi_Rustagi: <p>Hi! I an exploring Accelerate library form HuggingFace to fine tune BertForSequenceClassification. Relevant code snippets</p>
<pre><code class="lang-auto">import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
from accelerate import Accelerator

created Dataset class and Dataloaders
def get_dataloaders():
    train_dl = DataLoader(train_ds, shuffle=True, pin_memory=True)
    val_dl = DataLoader(val_ds, shuffle=False, pin_memory=True)
    
    return train_dl, val_dl

def training_loop():
    filename = "models/bert_category_clsfr"
    
    accelerator = Accelerator()
    device = accelerator.device

    model = BertForSequenceClassification.from_pretrained("bert-base-uncased")
    optimizer = torch.optim.AdamW(model.parameters())
    train_dl, val_dl = get_dataloaders()
    
    model, optimizer, train_dl = accelerator.prepare(model, optimizer, train_dl)
    val_dl = accelerator.prepare(val_dl)
    
    for epoch in range(1):
        model.train()
        for step,batch in enumerate(train_dl):
            input_ids = batch["input_ids"].to(device)
            targets = batch["label"].to(device)
            attention_mask = batch["attention_mask"].to(device)
            optimizer.zero_grad()

            output = model(input_ids, attention_mask)
            loss = F.cross_entropy(output.logits, targets)

            accelerator.backward(loss)
            optimizer.step()

     accelerator.wait_for_everyone()
     unwrapped_model = accelerator.unwrap_model(model)
     accelerator.save(unwrapped_model.state_dict(), filename)

from accelerate import notebook_launcher
notebook_launcher(training_loop, num_processes=4)
</code></pre>
<p>Results in the same error as above:<br>
ProcessRaisedException:</p>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/multiprocessing/spawn.py”, line<br>
69, in _wrap<br>
fn(i, *args)<br>
File “/home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/accelerate/utils/launch.py”, line<br>
509, in <strong>call</strong><br>
self.launcher(*args)<br>
File “/tmp/ipykernel_73883/403198274.py”, line 25, in training_loop<br>
accelerator.backward(loss)<br>
File “/home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/accelerate/accelerator.py”, line<br>
1683, in backward<br>
loss.backward(**kwargs)<br>
File “/home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/_tensor.py”, line 488, in<br>
backward<br>
torch.autograd.backward(<br>
File “/home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/autograd/<strong>init</strong>.py”, line<br>
197, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling <code>cublasCreate(handle)</code></p>
<p>batch size is 4 for 4 GPUs so I dont think batch size is an issue.<br>
Please share what else can be explored to use Accelerate for distributed training</p>

ptrblck: <aside class="quote no-group" data-username="Nidhi_Rustagi" data-post="12" data-topic="125450">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nidhi_rustagi/48/60361_2.png" class="avatar"> Nidhi_Rustagi:</div>
<blockquote>
<p>batch size is 4 for 4 GPUs so I dont think batch size is an issue.</p>
</blockquote>
</aside>
<p>Could you explain why not and how you’ve verified this batch size should fit into your GPUs?</p>

Nidhi_Rustagi: <p>As per my understanding a batch size of 4 on 4 processors would mean, a batch size of 4 datapoints is executed on single GPU.<br>
If I use the script without Accelerate() and with Huggingface Trainer API, I can run the script successfully with batch size of 16, hence I thought batch size of 4 should not be a problem. In case there is another way to verify that given batch size should fit into GPU please let me know, I will be happy to verify it.</p>
<p>Resolved the error, my problem was not specifying num_labels while loading the model<br>
Changing<br>
BertForSequenceClassification.from_pretrained(“bert-base-uncased”)</p>
<p>to<br>
<code>BertForSequenceClassification.from_pretrained("bert-base-uncased",num_labels=NUM_LABELS)  </code><br>
help in resolving the issue. ALso added torch.cuda.empty_cache() before starting the fine tuning process.</p>

zikai: <p>Ｔｈａｎｋ ｙｏｕ， ｂｙ ｓｅｔｔｉｎｇ<br>
os.environ[‘CUDA_VISIBLE_DEVICES’] = “4”<br>
ｏｎ ｏｎｅ ｇｐｕ， ｉｔ ｆｉｎｅｓ．</p>

Abhishek_Ghosh: <p>I am using <code>torch.compile</code> for the <code>fastNLP_Bert</code> model.</p>
<p>The logs show the following:</p>
<pre><code class="lang-auto">V1010 11:47:33.109000 130925214240768 torch/_inductor/graph.py:1500] [5/0] Output code written to: /tmp/torchinductor_abhishek/xy/cxymc6c7zr6k5kpcynp3sgouehmub6kqljk3rhigwfluxnwqmiqp.py
V1010 11:47:33.110000 130925214240768 torch/_inductor/compile_fx.py:481] [5/0] FX codegen and compilation took 4.754s
</code></pre>
<p>When I try to execute the output code’s python script:</p>
<pre><code class="lang-auto">/tmp/torchinductor_abhishek/xy$ python3 cxymc6c7zr6k5kpcynp3sgouehmub6kqljk3rhigwfluxnwqmiqp.py 
Traceback (most recent call last):
  File "/tmp/torchinductor_abhishek/xy/cxymc6c7zr6k5kpcynp3sgouehmub6kqljk3rhigwfluxnwqmiqp.py", line 1718, in &lt;module&gt;
    compiled_module_main('fastNLP_Bert', benchmark_compiled_module)
  File "/home/abhishek/pytorch-benchmarks/lib/python3.10/site-packages/torch/_inductor/wrapper_benchmark.py", line 283, in compiled_module_main
    wall_time_ms = benchmark_compiled_module_fn(times=times, repeat=repeat) * 1000
  File "/tmp/torchinductor_abhishek/xy/cxymc6c7zr6k5kpcynp3sgouehmub6kqljk3rhigwfluxnwqmiqp.py", line 1713, in benchmark_compiled_module
    return print_performance(fn, times=times, repeat=repeat)
  File "/home/abhishek/pytorch-benchmarks/lib/python3.10/site-packages/torch/_inductor/utils.py", line 331, in print_performance
    timings = torch.tensor([timed(fn, args, times, device) for _ in range(repeat)])
  File "/home/abhishek/pytorch-benchmarks/lib/python3.10/site-packages/torch/_inductor/utils.py", line 331, in &lt;listcomp&gt;
    timings = torch.tensor([timed(fn, args, times, device) for _ in range(repeat)])
  File "/home/abhishek/pytorch-benchmarks/lib/python3.10/site-packages/torch/_inductor/utils.py", line 320, in timed
    result = model(*example_inputs)
  File "/tmp/torchinductor_abhishek/xy/cxymc6c7zr6k5kpcynp3sgouehmub6kqljk3rhigwfluxnwqmiqp.py", line 1712, in &lt;lambda&gt;
    fn = lambda: call([arg0_1, arg1_1, arg2_1, arg3_1, arg4_1, arg5_1, arg6_1, arg7_1, arg8_1, arg9_1, arg10_1, arg11_1, arg12_1, arg13_1, arg14_1, arg15_1, arg16_1, arg17_1, arg18_1, arg19_1, arg20_1, arg21_1, arg22_1, arg23_1, arg24_1, arg25_1, arg26_1, arg27_1, arg28_1, arg29_1, arg30_1, arg31_1, arg32_1, arg33_1, arg34_1, arg35_1, arg36_1, arg37_1, arg38_1, arg39_1, arg40_1, arg41_1, arg42_1, arg43_1, arg44_1, arg45_1, arg46_1, arg47_1, arg48_1, arg49_1, arg50_1, arg51_1, arg52_1, arg53_1, arg54_1, arg55_1, arg56_1, arg57_1, arg58_1, arg59_1, arg60_1, arg61_1, arg62_1, arg63_1, arg64_1, arg65_1, arg66_1, arg67_1, arg68_1, arg69_1, arg70_1, arg71_1, arg72_1, arg73_1, arg74_1, arg75_1, arg76_1, arg77_1, arg78_1, arg79_1, arg80_1, arg81_1, arg82_1, arg83_1, arg84_1, arg85_1, arg86_1, arg87_1, arg88_1, arg89_1, arg90_1, arg91_1, arg92_1, arg93_1, arg94_1, arg95_1, arg96_1, arg97_1, arg98_1, arg99_1, arg100_1, arg101_1, arg102_1, arg103_1, arg104_1, arg105_1, arg106_1, arg107_1, arg108_1, arg109_1, arg110_1, arg111_1, arg112_1, arg113_1, arg114_1, arg115_1, arg116_1, arg117_1, arg118_1, arg119_1, arg120_1, arg121_1, arg122_1, arg123_1, arg124_1, arg125_1, arg126_1, arg127_1, arg128_1, arg129_1, arg130_1, arg131_1, arg132_1, arg133_1, arg134_1, arg135_1, arg136_1, arg137_1, arg138_1, arg139_1, arg140_1, arg141_1, arg142_1, arg143_1, arg144_1, arg145_1, arg146_1, arg147_1, arg148_1, arg149_1, arg150_1, arg151_1, arg152_1, arg153_1, arg154_1, arg155_1, arg156_1, arg157_1, arg158_1, arg159_1, arg160_1, arg161_1, arg162_1, arg163_1, arg164_1, arg165_1, arg166_1, arg167_1, arg168_1, arg169_1, arg170_1, arg171_1, arg172_1, arg173_1, arg174_1, arg175_1, arg176_1, arg177_1, arg178_1, arg179_1, arg180_1, arg181_1, arg182_1, arg183_1, arg184_1, arg185_1, arg186_1, arg187_1, arg188_1, arg189_1, arg190_1, arg191_1, arg192_1, arg193_1, arg194_1, arg195_1, arg196_1, arg197_1, arg198_1, arg199_1, arg200_1, arg201_1])
  File "/tmp/torchinductor_abhishek/xy/cxymc6c7zr6k5kpcynp3sgouehmub6kqljk3rhigwfluxnwqmiqp.py", line 722, in call
    extern_kernels.mm(reinterpret_tensor(buf4, (486400, 768), (768, 1), 0), reinterpret_tensor(arg5_1, (768, 768), (1, 768), 0), out=buf5)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
(pytorch-benchmarks) abhishek@chisel-7:/tmp/torchinductor_abhishek/xy$
</code></pre>
<p>I get the above error. I guess the output code be run standalone (I have tried out with other output codes without having <code>extern_kernels</code> call and they work)</p>
<p>Note: The full <code>torch.compile</code>d application work fine.</p>