How to fix “CUDA error: device-side assert triggered” error?

guoyaohua: <p>I use huggingface Transformer to fine-tune a binary classification model. When I do inference job on big data. In rare case, it will trigger “CUDA error: device-side assert triggered” error, but when I debug the single wrong batch, it is strange that it can pass (both on GPU and CPU) , I don’t know why.</p>
<p>This error firstly trigger on</p>
<p><code>probs = probs.cpu().numpy()</code></p>
<p>and after that it will trigger on</p>
<p><code>input_ids = torch.tensor(batch['input_ids'], dtype=torch.long).to(device)</code></p>
<pre><code class="lang-auto">2021-11-18 20:18:41,251 - non_news_model.py[line:342] - ERROR: Traceback (most recent call last):
  File "/data/project/NonNewsInference/InferenceServices/non_news_model.py", line 307, in DoInference
    probs = probs.cpu().numpy()
RuntimeError: CUDA error: device-side assert triggered
, DoInference is dead!
2021-11-18 20:18:41,263 - non_news_model.py[line:345] - ERROR: Bad batch recorded!
2021-11-18 20:18:41,292 - non_news_model.py[line:342] - ERROR: Traceback (most recent call last):
  File "/data/project/NonNewsInference/InferenceServices/non_news_model.py", line 298, in DoInference
    dtype=torch.long).to(device)
RuntimeError: CUDA error: device-side assert triggered
, DoInference is dead!
</code></pre>
<p>Could anyone tell me how to solve this problem? Thanks!!</p>

ptrblck: <p>CUDA operations are executed asynchronously, so the stack trace might point to the wrong line of code. Rerun your script via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the failing operation in the reported stack trace. Often these asserts are triggered by an invalid indexing operation.</p>

guoyaohua: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, this error is occasionally triggered during model inference, not often. If I encounter this exception during large-scale data inference task, how can I accurately find the wrong batch of data? As your said, CUDA operations are asynchronously, if I catch the exception and log the bad batch, can I locate this wrong batch?</p>
<p>like this:</p>
<pre><code class="lang-auto">try:
	batch = inference_queue.get(block=True)
	with torch.no_grad():
		input_ids = torch.tensor(batch['input_ids'],
								 dtype=torch.long).to(device)
		attention_mask = torch.tensor(batch['attention_mask'],
									  dtype=torch.long).to(device)
		inputs = {
			"input_ids": input_ids,
			"attention_mask": attention_mask
		}
		logits = model(**inputs)[0]
		probs = softmax(logits)
	probs = probs.cpu().numpy()
except Exception as e:
	logger.error(f'{traceback.format_exc()}, DoInference is dead!')
	with open('./BadBatch.pkl', 'wb') as f:
		pickle.dump(batch, f)
	logger.error(f'Bad batch recorded!')
</code></pre>

ptrblck: <p>You could run the script with the aforementioned env variable, which would point to the operation raising the error.<br>
Your approach could work, but note that once you are running into an assert the CUDA context might be corrupted and I don’t know if you would be able to store any additional data.</p>

guoyaohua: <p>Thank you! I solve this problem. Due to the my tokenizer output did not match model vocabulary size. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"></p>

Sokhib_Tukhtaev: <p>could you describe what was your solution to that error? I am also facing <code>CUDA error: device-side assert triggered</code> error when running inference with yolov8’s tracking. Thank you.</p>

jyothir07: <p>Have the same error while trying to resume YOLOV8:<br>
RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>

ptrblck: <p>Rerun your script via:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and check the failing operation in the reported stack trace as already mentioned in this topic.</p>

shyam_llama: <p>thanks your comment made me  to re-check  tokenizer and model are from same repo</p>

guoyaohua: <p>Run your code with CPU device, you will find the actual error.</p>

Khusboo_chaudhry: <p>I encountered the same error:<br>
Steps which helped me to resolve the error:</p>
<ol>
<li>
<p>write these two lines in the first file being executed. It helped to show the true/exact line of code in the current file which was creating problem.<br>
import os<br>
os.environ[“CUDA_LAUNCH_BLOCKING”] = “1”</p>
</li>
<li>
<p>For my case error was in line:<br>
channel_select_filtered_positive = all_filtered.view(-1)[indices.long()].view(1, height, width)<br>
change to<br>
channel_select_filtered_positive = all_filtered.view(-1)[indices.int()].view(1, height, width)<br>
resolved the error.</p>
</li>
</ol>

ptrblck: <p>I would recommend understanding the fix in detail as it seems the transformation from <code>long</code> to <code>int</code> and thus also the corresponding numerical range reduction “fixed” the issue while it seems the clipping is a side effect.</p>

Loreto_Parisi: <p>I’m facing the same issue in my Transformer block’s forward pass in Llama3.<br>
Using the CUDA debugging env I get the error line detected and in fact it was:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, tokens: torch.Tensor):
        """Perform a forward pass through the Transformer model.

        Args:
            tokens (torch.Tensor): Input token indices.

        Returns:
            torch.Tensor: Output logits after applying the Transformer model.

        """
        
       # ERROR RAISES HERE 
       # passthrough for nonexistent layers, allows easy configuration of pipeline parallel stages
        h = self.tok_embeddings(tokens) if self.tok_embeddings else tokens

        for layer in self.layers.values():
            h = layer(h, self.freqs_cis)

        h = self.norm(h) if self.norm else h
        return self.output(h).float() if self.output else h
</code></pre>
<p>How did you fixed it?</p>

Vaibhav_Hiwase: <p>It seems that when using any tokenizer from Hugging Face, it must be initialized and run on a CPU device. Running it directly on a GPU device might cause device-side assertion errors.</p>
<p>To potentially resolve this issue, I would suggest modifying your code as follows:<br>
Please change your code as below</p>
<p><code>input_ids = torch.tensor(batch['input_ids'], dtype=torch.long).to(device)</code></p>
<p>To</p>
<p><code>input_ids = torch.tensor(batch['input_ids'], dtype=torch.long)</code></p>
<p>This might solve the problem.</p>

yan-hao-tian: <p>太行了哥 thanks too much bro👍</p>

samedilmed: <p>I had a similar isssue, and I had to chnage type of a masking variable (when working on cuda device).<br>
I changed this:</p>
<pre><code class="lang-auto">valid_mask = scores &gt;= self.det_threshold. # type: torch.bool()
</code></pre>
<p>into this:</p>
<pre><code class="lang-auto">valid_mask = (scores &gt;= self.det_threshold).long(). # type: torch.long()
</code></pre>
<p>Doing so, it worked.</p>

NGUY_N_VAN_QUANG_TR: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa6c22fcd03631682f463534b5702ae5edbd8c1e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fa6c22fcd03631682f463534b5702ae5edbd8c1e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa6c22fcd03631682f463534b5702ae5edbd8c1e_2_690x243.jpeg" alt="image" data-base62-sha1="zJl33JKA8v5rDNSOcEfx5I2gJY2" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa6c22fcd03631682f463534b5702ae5edbd8c1e_2_690x243.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa6c22fcd03631682f463534b5702ae5edbd8c1e_2_1035x364.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa6c22fcd03631682f463534b5702ae5edbd8c1e.jpeg 2x" data-dominant-color="F6D8D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1169×412 96.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>