CUDA error when trying to run nomic-embed-text-v1.5

Sanjid_Chowdhury: <p>I have a 4070ti super, and I want to embed around 315k+ data locally. When I use my CPU the code below works fine, but when I set it to the GPU, i keep getting this CUDA error message</p>
<blockquote>
<p>CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</p>
</blockquote>
<p>Even though my GPU’s VRAM is not completely used (I checked using task manager). I tried reinstalling everything, downgrading my GPU drivers to the one in cuda 12.4, but still no luck. Lowering the batch size and sentences size just lets it run a few iterations before the error occurs. What am I doing wrong here? Is my VRAM not being released after an iteration or something?</p>
<pre><code class="lang-auto">start = 0
inc = 64
iteration = 1
matryoshka_dim = 512
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)

# gpu = 0
device = torch.device("cuda:0"if torch.cuda.is_available() else "cpu")
# torch.cuda.set_device(gpu)

# device = torch.device("cpu")

for i in tqdm(range(start, len(rows), inc)):
	end = min(i + inc, len(rows))
	# print(start, end)
	sentences = rows[start:end]
	embeddings = model.encode(sentences, convert_to_tensor=True, device=device)
	embeddings = F.layer_norm(embeddings, normalized_shape=(embeddings.shape[1],))
	embeddings = embeddings[:, :matryoshka_dim]
	embeddings = F.normalize(embeddings, p=2, dim=1)
	# write to file in fk_ro_v
	with open("./fk_ro_v/ro_" + str(iteration) + ".pkl", "wb") as f:
		pickle.dump(embeddings, f)
	torch.cuda.empty_cache()
	iteration += 1
	start += inc
</code></pre>
<p>VRAM Usage<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef87a2f7966c73f21df5c044d708c1817cd9b59b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef87a2f7966c73f21df5c044d708c1817cd9b59b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef87a2f7966c73f21df5c044d708c1817cd9b59b.png" alt="image" data-base62-sha1="yaYImlM75p68SMJRwiyeLu8Atw7" width="690" height="315" data-dominant-color="221F21"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">956×437 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>torch                  2.5.0.dev20240715+cu124<br>
torchaudio             2.4.0.dev20240715+cu124<br>
torchvision            0.20.0.dev20240715+cu124</p>

ptrblck: <p>Could you check the memory usage via <code>nvidia-smi</code>?</p>

Sanjid_Chowdhury: <p>Okay, I have used <code>nvidia-smi</code> inside of python. My code is now</p>
<pre><code class="lang-auto">import subprocess

def print_gpu_memory_usage():
    try:
        result = subprocess.check_output(
            ['nvidia-smi', '--query-gpu=memory.used,memory.free', '--format=csv,nounits,noheader'], 
            encoding='utf-8'
        )
        used, free = result.strip().split('\n')[0].split(',')
        print(f"GPU Memory - Used: {used} MiB, Free: {free} MiB")
    except Exception as e:
        print(f"Error fetching GPU memory usage: {e}")

model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
for i in tqdm(range(start, len(rows), inc)):
	end = min(i + inc, len(rows))
	sentences = rows[start:end]
	try:
		embeddings = model.encode(sentences, normalize_embeddings=True, device=device)
		with open("./fk_ro_v/ro_" + str(iteration) + ".pkl", "wb") as f:
			pickle.dump(embeddings, f)
   
		torch.cuda.empty_cache()
		torch.cuda.synchronize()
		iteration += 1
		start += inc
  
	except Exception as e:
		print(print_gpu_memory_usage())
		break
</code></pre>
<p>This is what it prints when exception hits:</p>
<blockquote>
<p>GPU Memory - Used: 5347 MiB, Free:  10715 MiB</p>
</blockquote>

Sanjid_Chowdhury: <p>Here is some more info</p>
<blockquote>
<p>Batch 1<br>
Before encoding:<br>
GPU Memory - Used: 2936 MiB, Free:  13126 MiB<br>
After encoding:<br>
GPU Memory - Used: 3638 MiB, Free:  12424 MiB<br>
Batch 2<br>
Before encoding:<br>
GPU Memory - Used: 2998 MiB, Free:  13064 MiB<br>
exception<br>
GPU Memory - Used: 3752 MiB, Free:  12310 MiB</p>
</blockquote>

ptrblck: <p>Could you narrow down which line of code raises this error or post a minimal and executable code snippet without the requirement to trust remote code?</p>

Sanjid_Chowdhury: <p>Sure,</p>
<blockquote>
<p>embeddings = model.encode(sentences, normalize_embeddings=True, device=device)</p>
</blockquote>
<p>This is the line where the error occurs. Here’s the traceback:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "h:\work\FK\redis-fk-ro\test.py", line 134, in &lt;module&gt;
    embeddings = model.encode(sentences, normalize_embeddings=True, device=device)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\sentence_transformers\SentenceTransformer.py", line 517, in encode
    out_features = self.forward(features)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\container.py", line 249, in forward   
    input = module(input)
            ^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\sentence_transformers\models\Transformer.py", line 118, in forward
    output_states = self.auto_model(**trans_features, return_dict=False)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\.cache\huggingface\modules\transformers_modules\nomic-ai\nomic-bert-2048\e55a7d4324f65581af5f483e830b80f34680e8ff\modeling_hf_nomic_bert.py", line 1654, in forward
    sequence_output = self.encoder(hidden_states, attention_mask=attention_mask, return_dict=return_dict)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\.cache\huggingface\modules\transformers_modules\nomic-ai\nomic-bert-2048\e55a7d4324f65581af5f483e830b80f34680e8ff\modeling_hf_nomic_bert.py", line 1539, in forward
    hidden_states, hidden_states2, residual = layer(
                                              ^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\.cache\huggingface\modules\transformers_modules\nomic-ai\nomic-bert-2048\e55a7d4324f65581af5f483e830b80f34680e8ff\modeling_hf_nomic_bert.py", line 1465, in forward
    attn_outputs = self.attn(
                   ^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\.cache\huggingface\modules\transformers_modules\nomic-ai\nomic-bert-2048\e55a7d4324f65581af5f483e830b80f34680e8ff\modeling_hf_nomic_bert.py", line 1375, in forward
    attn_output = self.out_proj(attn_output)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\linear.py", line 125, in forward      
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>
<p>As for pasting a code snippet that doesnt require <code>trust_remote_code=True</code>, I tried loading in a model that doesn’t require remote code, which is <code>BAAI/bge-large-en-v1.5</code></p>
<p>When I use that model, I get this error instead:</p>
<blockquote>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 1024 n 4544 k 1024 mat1_ld 1024 mat2_ld 1024 result_ld 1024 abcType 0 computeType 68 scaleType 0</p>
</blockquote>
<p>Stacktrace:</p>
<pre><code class="lang-auto">  File "h:\work\FK\redis-fk-ro\test.py", line 134, in &lt;module&gt;
    embeddings = model.encode(sentences, normalize_embeddings=True, device=device)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\sentence_transformers\SentenceTransformer.py", line 517, in encode
    out_features = self.forward(features)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\container.py", line 249, in forward   
    input = module(input)
            ^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\sentence_transformers\models\Transformer.py", line 118, in forward
    output_states = self.auto_model(**trans_features, return_dict=False)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\models\bert\modeling_bert.py", line 1141, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\models\bert\modeling_bert.py", line 694, in forward
    layer_outputs = layer_module(
                    ^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\models\bert\modeling_bert.py", line 584, in forward
    self_attention_outputs = self.attention(
                             ^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\models\bert\modeling_bert.py", line 514, in forward
    self_outputs = self.self(
                   ^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\models\bert\modeling_bert.py", line 394, in forward
    query_layer = self.transpose_for_scores(self.query(hidden_states))
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl  
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lassassin\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\linear.py", line 125, in forward      
    return F.linear(input, self.weight, self.bias)
</code></pre>
<pre><code class="lang-auto"># model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
for i in tqdm(range(start, len(rows), inc)):
	end = min(i + inc, len(rows))
	# print(start, end)
	sentences = rows[start:end]
	print("Before encoding:")
	print_gpu_memory_usage()
	embeddings = model.encode(sentences, normalize_embeddings=True, device=device)
	print("After encoding:")
	print_gpu_memory_usage()

	with open("./fk_ro_v/ro_" + str(iteration) + ".pkl", "wb") as f:
		pickle.dump(embeddings, f)

	torch.cuda.empty_cache()
	torch.cuda.synchronize()
	iteration += 1
	start += inc
</code></pre>
<p>Please let me know if u require anything else. This error is driving me crazy.<br>
Sidenote:</p>
<ul>
<li>Using <code>sentence-transformers/all-MiniLM-L6-v2</code> model with GPU raises no errors.</li>
<li>Using the other larger models with CPU raises no errors</li>
</ul>

Sanjid_Chowdhury: <p>Hello, I have solved this issue. Apparently, my GPU was overclocked to +200MHZ to both boost and core clock, causing memory problems. Setting the overclocks down to +100MHZ on both boost and core clock made it run fine.</p>
<p>Thanks!</p>

ptrblck: <p>Thanks for the follow up as it’s indeed an interesting observation!</p>