RuntimeError: CUDA error -- Fine Tune unsloth model

hzhuang: <p>Hi everyone,</p>
<h2><a name="i-used-the-fine-tune-model-from-unsloth-and-i-got-this-error-it-ran-ok-for-the-first-few-loop-but-it-show-up-this-errors-after-2-mins-1" class="anchor" href="#i-used-the-fine-tune-model-from-unsloth-and-i-got-this-error-it-ran-ok-for-the-first-few-loop-but-it-show-up-this-errors-after-2-mins-1"></a>I used the fine-tune model from unsloth, and I got this error. It ran ok for the first few loop, but it show up this errors after 2 mins.</h2>
<h2><a name="runtimeerror-cuda-error-device-side-assert-triggered-cuda-kernel-errors-might-be-asynchronously-reported-at-some-other-api-call-so-the-stacktrace-below-might-be-incorrect-for-debugging-consider-passing-cuda_launch_blocking1-compile-with-torch_use_cuda_dsa-to-enable-device-side-assertions-2" class="anchor" href="#runtimeerror-cuda-error-device-side-assert-triggered-cuda-kernel-errors-might-be-asynchronously-reported-at-some-other-api-call-so-the-stacktrace-below-might-be-incorrect-for-debugging-consider-passing-cuda_launch_blocking1-compile-with-torch_use_cuda_dsa-to-enable-device-side-assertions-2"></a>RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</h2>
<p>also got this warning:<br>
…/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [0,0,0] Assertion <code>-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.</p>
<hr>
<p>Code that show up this error:</p>
<p>inputs = tokenizer(<br>
[<br>
alpaca_prompt.format(<br>
instruction,  # instruction<br>
tables[table_i],  # input<br>
“”,  # output - leave this blank for generation!<br>
)<br>
], return_tensors=“pt”).to(“cuda:0”)</p>

ptrblck: <p>An indexing operation is failing. Rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> to isolate the failing line of code.</p>

hzhuang: <p>Dear Ptrblck,</p>
<p>Thanks for your reply, after I ran this via CUDA_LAUNCH_BLOCKING=1, still got this:</p>
<p>File ~/anaconda3/envs/FTllama/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:800, in BatchEncoding.to(self, device)<br>
796 # This check catches things like APEX blindly calling “to” on all inputs to a module<br>
797 # Otherwise it passes the casts down and casts the LongTensor containing the token idxs<br>
798 # into a HalfTensor<br>
799 if isinstance(device, str) or is_torch_device(device) or isinstance(device, int):<br>
 → 800     self.data = {k: v.to(device=device) for k, v in self.data.items()}<br>
801 else:<br>
802     logger.warning(f"Attempting to cast a BatchEncoding to type {str(device)}. This is not supported.")</p>
<p>File ~/anaconda3/envs/FTllama/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:800, in (.0)<br>
796 # This check catches things like APEX blindly calling “to” on all inputs to a module<br>
797 # Otherwise it passes the casts down and casts the LongTensor containing the token idxs<br>
798 # into a HalfTensor<br>
799 if isinstance(device, str) or is_torch_device(device) or isinstance(device, int):<br>
 → 800     self.data = {k: v.to(device=device) for k, v in self.data.items()}<br>
801 else:<br>
802     logger.warning(f"Attempting to cast a BatchEncoding to type {str(device)}. This is not supported.")</p>
<p>RuntimeError: CUDA error: device-side assert triggered<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>

ptrblck: <p>The stacktrace is unfortunately still wrong. Did you <code>export</code> this env variable in your terminal? If so, check if any embedding layers are used as their input is often containing indices which are out of the valid range.</p>

hzhuang: <p>Dear ptrblck,</p>
<p>Thanks, notice that my input is out of the valid range</p>