: CUDA error: an illegal memory access was encountered

Sashika_Himali1: <p>I tried to finetune blip2 model for a custom dataset. I could train the model for one epoch after that I am getting the following error.  inside the training file batch size is 6.</p>
<p>error at step 1 in epoch 2: CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Traceback (most recent call last):<br>
File “/workspace/train.py”, line 518, in <br>
outputs = combined_model(**batch)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/deepspeed/utils/nvtx.py”, line 18, in wrapped_fn<br>
ret_val = func(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py”, line 1899, in forward<br>
loss = self.module(*inputs, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/peft/peft_model.py”, line 762, in forward<br>
return self.get_base_model()(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/workspace/train.py”, line 257, in forward<br>
outputs = self.language_model(<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/transformers/models/opt/modeling_opt.py”, line 1011, in forward<br>
outputs = self.model.decoder(<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/transformers/models/opt/modeling_opt.py”, line 777, in forward<br>
layer_outputs = decoder_layer(<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/transformers/models/opt/modeling_opt.py”, line 418, in forward<br>
hidden_states, self_attn_weights, present_key_value = self.self_attn(<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/transformers/models/opt/modeling_opt.py”, line 192, in forward<br>
attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min, device=attn_weights.device)<br>
RuntimeError: CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>Traceback (most recent call last):<br>
File “/workspace/train.py”, line 537, in <br>
torch.cuda.empty_cache()  # Free memory if an error occurs<br>
File “/usr/local/lib/python3.10/dist-packages/torch/cuda/memory.py”, line 133, in empty_cache<br>
torch._C._cuda_emptyCache()<br>
RuntimeError: CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Exception raised from c10_cuda_check_implementation at …/c10/cuda/CUDAException.cpp:44 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7d5e052a34d7 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x64 (0x7d5e0526d36b in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#2:</span> c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7d5e0533fb58 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x1c36b (0x7d5e0531036b in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0x2b930 (0x7d5e0531f930 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#5:</span>  + 0x4d5a16 (0x7d5dfe978a16 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0x3ee77 (0x7d5e05288e77 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#7:</span> c10::TensorImpl::~TensorImpl() + 0x1be (0x7d5e0528169e in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#8:</span> c10::TensorImpl::~TensorImpl() + 0x9 (0x7d5e052817b9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#9:</span>  + 0x75afc8 (0x7d5dfebfdfc8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#10:</span>  + 0x12f715 (0x6094db5da715 in /usr/bin/python)<br>
frame <span class="hashtag-raw">#11:</span>  + 0x268a60 (0x6094db713a60 in /usr/bin/python)<br>
frame <span class="hashtag-raw">#12:</span> Py_FinalizeEx + 0x148 (0x6094db70fb98 in /usr/bin/python)<br>
frame <span class="hashtag-raw">#13:</span> Py_RunMain + 0x173 (0x6094db7012d3 in /usr/bin/python)<br>
frame <span class="hashtag-raw">#14:</span> Py_BytesMain + 0x2d (0x6094db6d7cad in /usr/bin/python)<br>
frame <span class="hashtag-raw">#15:</span>  + 0x29d90 (0x7d5e2fd71d90 in /usr/lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#16:</span> __libc_start_main + 0x80 (0x7d5e2fd71e40 in /usr/lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#17:</span> _start + 0x25 (0x6094db6d7ba5 in /usr/bin/python)</p>
<p>[2024-09-08 10:40:37,800] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 2122<br>
[2024-09-08 10:40:37,801] [ERROR] [launch.py:325:sigkill_handler] [‘/usr/bin/python’, ‘-u’, ‘train.py’, ‘–local_rank=0’, ‘–deepspeed_config’, ‘ds_config.json’] exits with return code = -6</p>
<p>To train the model I used following  deep speed configurations</p>
<p>{<br>
“train_batch_size”: 8,<br>
“gradient_accumulation_steps”: 4,<br>
“optimizer”: {<br>
“type”: “Adam”,<br>
“params”: {<br>
“lr”: 3e-3,<br>
“betas”: [0.9, 0.999],<br>
“eps”: 1e-8,<br>
“weight_decay”: 1e-2<br>
}<br>
},<br>
“zero_optimization”: {<br>
“stage”: 2,<br>
“allgather_partitions”: true,<br>
“allgather_bucket_size”: 5e8,<br>
“reduce_scatter”: true,<br>
“reduce_bucket_size”: 5e8,<br>
“overlap_comm”: true,<br>
“contiguous_gradients”: true,<br>
“offload_param”: {<br>
“device”: “cpu”,<br>
“pin_memory”: true<br>
}<br>
},<br>
“fp16”: {<br>
“enabled”: true,<br>
“loss_scale”: 1024 # Reduce initial loss scale<br>
}<br>
}</p>

ptrblck: <p>Could you try to isolate the failing kernel by creating a CUDA coredump or by running your workload via <code>cuda-gdb</code>? If you are using an older PyTorch release, could you also update to the latest one before rerunning it?</p>

Sashika_Himali1: <p>I am working on runpod. pod infor :1 x RTX A6000 16 vCPU 62 GB RAM .  batch-size: 6. I will try to change the  pytorch version and re run it. Can I create CUDA coredump inside rundpod. thanks ptrblck</p>

Sashika_Himali1: <p>I trained as you advised by updating the pytorch but I am still getting an error. I could train only one epoch after that I got following error.</p>
<p>Error at step 1 in epoch 2: CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>[rank0]: Traceback (most recent call last):<br>
[rank0]:   File “/workspace/train.py”, line 519, in <br>
[rank0]:     outputs = combined_model(**batch)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py”, line 18, in wrapped_fn<br>
[rank0]:     ret_val = func(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/deepspeed/runtime/engine.py”, line 1899, in forward<br>
[rank0]:     loss = self.module(*inputs, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/peft/peft_model.py”, line 762, in forward<br>
[rank0]:     return self.get_base_model()(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:   File “/workspace/train.py”, line 257, in forward<br>
[rank0]:     outputs = self.language_model(<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py”, line 1011, in forward<br>
[rank0]:     outputs = self.model.decoder(<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py”, line 777, in forward<br>
[rank0]:     layer_outputs = decoder_layer(<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py”, line 418, in forward<br>
[rank0]:     hidden_states, self_attn_weights, present_key_value = self.self_attn(<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py”, line 192, in forward<br>
[rank0]:     attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min, device=attn_weights.device)<br>
[rank0]: RuntimeError: CUDA error: an illegal memory access was encountered<br>
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
[rank0]: Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>[rank0]: During handling of the above exception, another exception occurred:</p>
<p>[rank0]: Traceback (most recent call last):<br>
[rank0]:   File “/workspace/train.py”, line 538, in <br>
[rank0]:     torch.cuda.empty_cache()  # Free memory if an error occurs<br>
[rank0]:   File “/workspace/myenv/lib/python3.10/site-packages/torch/cuda/memory.py”, line 170, in empty_cache<br>
[rank0]:     torch._C._cuda_emptyCache()<br>
[rank0]: RuntimeError: CUDA error: an illegal memory access was encountered<br>
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
[rank0]: Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>[2024-09-12 16:32:23,067] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 1475<br>
[2024-09-12 16:32:23,067] [ERROR] [launch.py:325:sigkill_handler] [‘/workspace/myenv/bin/python3’, ‘-u’, ‘train.py’, ‘–local_rank=0’, ‘–deepspeed_config’, ‘ds_config.json’] exits with return code = 1</p>