CUDA error: CUBLAS_STATUS_NOT_INITIALIZED

MahaA: <p>Hi everyone,<br>
I am doing POS tagging, and when computing the backward pass I am getting the following error:<br>
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling <code>cublasCreate(handle)</code>.<br>
I tried a bunch of different solutions on the web and nothing worked.<br>
Here is the code that is triggering the error (note that I am using bert-base for classification) :</p>
<pre><code class="lang-auto">nl_databatch = data_dict['tr_batch']
        input_ids = nl_databatch['input_ids']
        attention_mask = nl_databatch['attention_mask']
        n_labels = nl_databatch['n_labels']
        input_ids = input_ids.to(device)
        attention_mask = attention_mask.to(device)
        n_labels = n_labels.to(device)
        outputs = model(input_ids, attention_mask)
        outputs = outputs.view(-1, outputs.shape[-1])
        n_labels = n_labels.view(-1)
        
        loss = loss_fn(outputs, n_labels)
        

        loss.backward()

        return loss.item()
</code></pre>
<p>It would be a huge help if anyone can help me debug this error, I have a deadline very soon and need to solve this asap.</p>

22ema: <p>Hello, friend.<br>
I don’t know if it will help, but when I refer to the github below, I think it’s because of the length problem of the tokenizer. (When I saw the case of bert)</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/huggingface/transformers/issues/6263">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/issues/6263" target="_blank" rel="noopener nofollow ugc">github.com/huggingface/transformers</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/huggingface/transformers/issues/6263" target="_blank" rel="noopener nofollow ugc">RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-08-05" data-time="07:24:25" data-timezone="UTC">07:24AM - 05 Aug 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-08-05" data-time="07:51:40" data-timezone="UTC">07:51AM - 05 Aug 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/mt324010" target="_blank" rel="noopener nofollow ugc">
          <img alt="mt324010" src="https://avatars.githubusercontent.com/u/35977320?v=4" class="onebox-avatar-inline" width="20" height="20">
          mt324010
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi, I tried to add some other embeddings in your BertEmbedding source code and t<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">hen load the pretrained weights 'bert-base-chinese'. 
When I run the forward method, I got the issue
'RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`'
Can someone help please? Thanks a lot</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Sorry if you’ve already been looking for it.</p>