CUDA illegal memory exception

username: alifani
cooked: <p>I’m running a DLRM model on a P4 instance with 8 GPUs and 96 vCPUs, using PyTorch 2.6.0 and CUDA 12.4. However, I intermittently encounter a <strong>CUDA illegal memory access</strong> exception during training.</p>
<p>Based on what I’ve read online, this appears to be a known issue affecting others as well, but I haven’t found a conclusive solution. I’ve tried the following:</p>
<ul>
<li>Reduced the batch size from 32 to 16, which allowed the model to run for several hours before eventually hitting the same error.</li>
<li>Set <code>CUDA_LAUNCH_BLOCKING=1</code> to aid in debugging, but this did not prevent the issue.<br>
When I set it above flag, the error i have got<br>
RuntimeError: DataLoader worker (pid(s) 180232) exited unexpectedly</li>
</ul>
<p>Are there any other recommended approaches, debugging strategies, or known fixes I could try?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="alifani" data-post="1" data-topic="219718">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/aca169/48.png" class="avatar"> alifani:</div>
<blockquote>
<p>Based on what I’ve read online, this appears to be a known issue affecting others as well</p>
</blockquote>
</aside>
<p>Memory violations can be caused by various reasons and they are mostly unrelated to each other. To narrow down which kernel causes the illegal memory access try to rerun your code with <code>compute-sanitizer</code> or try to create a CUDA coredump as described <a href="https://docs.nvidia.com/cuda/cuda-gdb/index.html#gpu-core-dump-support">here</a>. Once the coredump is created you can use <code>cuda-gdb</code> to isolate the failing kernel.</p>

username: alifani
cooked: <p>Running code with compute-sanitizer for an entire night didn’t even initiate the job within eight hours. I re-ran job with smaller batch size from 16 to 8 and it ran for 16 plus hours and ultimately failed.</p>
<p>This is the error traceback</p>
<p>[rank2]: Traceback (most recent call last):<br>
[rank2]:   File “/home/ec2-user/sd-personalization/train.py”, line 950, in <br>
[rank2]:     invoke_main()  # pragma: no cover<br>
[rank2]:   File “/home/ec2-user/sd-personalization/train.py”, line 945, in invoke_main<br>
[rank2]:     main(sys.argv[1:])<br>
[rank2]:   File “/home/ec2-user/sd-personalization/train.py”, line 927, in main<br>
[rank2]:     train_val_test(<br>
[rank2]:   File “/home/ec2-user/sd-personalization/train.py”, line 586, in train_val_test<br>
[rank2]:     _train(<br>
[rank2]:   File “/home/ec2-user/sd-personalization/train.py”, line 510, in _train<br>
[rank2]:     pipeline.progress(batched_iterator)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/distributed/train_pipeline/train_pipelines.py”, line 498, in progress<br>
[rank2]:     self.start_sparse_data_dist(self.batches[1], self.contexts[1])<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/distributed/train_pipeline/train_pipelines.py”, line 696, in start_sparse_data_dist<br>
[rank2]:     _start_data_dist(self._pipelined_modules, batch, context)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/distributed/train_pipeline/utils.py”, line 781, in _start_data_dist<br>
[rank2]:     context.input_dist_splits_requests[forward.name] = module.input_dist(<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/distributed/embeddingbag.py”, line 1244, in input_dist<br>
[rank2]:     awaitables.append(input_dist(features_by_shard))<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/nn/modules/module.py”, line 1739, in _wrapped_call_impl<br>
[rank2]:     return self._call_impl(*args, **kwargs)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/nn/modules/module.py”, line 1750, in _call_impl<br>
[rank2]:     return forward_call(*args, **kwargs)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/distributed/sharding/rw_sharding.py”, line 397, in forward<br>
[rank2]:     return self._dist(bucketized_features)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/nn/modules/module.py”, line 1739, in _wrapped_call_impl<br>
[rank2]:     return self._call_impl(*args, **kwargs)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/nn/modules/module.py”, line 1750, in _call_impl<br>
[rank2]:     return forward_call(*args, **kwargs)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/distributed/train_pipeline/utils.py”, line 651, in <strong>call</strong><br>
[rank2]:     input_splits = input.dist_splits(self._splits)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/sparse/jagged_tensor.py”, line 2861, in dist_splits<br>
[rank2]:     length_per_split = _sum_by_splits(self.length_per_key(), key_splits)<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/sparse/jagged_tensor.py”, line 2239, in length_per_key<br>
[rank2]:     _length_per_key = _maybe_compute_length_per_key(<br>
[rank2]:   File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torchrec/sparse/jagged_tensor.py”, line 1185, in _maybe_compute_length_per_key<br>
[rank2]:     torch.sum(<br>
[rank2]: RuntimeError: CUDA error: an illegal memory access was encountered<br>
[rank2]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
[rank2]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
[rank2]: Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb48e56c1b6 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x64 (0x7fb48e515a76 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#2:</span> c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fb48e9a3918 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x20d8e (0x7fb48e969d8e in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0x22507 (0x7fb48e96b507 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#5:</span>  + 0x2270f (0x7fb48e96b70f in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0x6418b2 (0x7fb485e418b2 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#7:</span>  + 0x6f30f (0x7fb48e54d30f in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#8:</span> c10::TensorImpl::~TensorImpl() + 0x21b (0x7fb48e54633b in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#9:</span> c10::TensorImpl::~TensorImpl() + 0x9 (0x7fb48e5464e9 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#10:</span>  + 0x8ffbb8 (0x7fb4860ffbb8 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#11:</span> THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7fb4860fff06 in /home/ec2-user/test-env/lib64/python3.9/site-packages/torch/lib/libtorch_python.so)<br>
<br>
frame <span class="hashtag-raw">#31:</span>  + 0x3fee0 (0x7fb49ca3fee0 in /lib64/libc.so.6)<br>
frame <span class="hashtag-raw">#32:</span> __libc_start_main + 0x80 (0x7fb49ca3ff90 in /lib64/libc.so.6)<br>
frame <span class="hashtag-raw">#33:</span> _start + 0x25 (0x561adb6df095 in /home/ec2-user/test-env/bin/python3)</p>
<p>W0504 16:21:59.900878 198230 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 198298 closing signal SIGTERM<br>
W0504 16:21:59.901994 198230 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 198299 closing signal SIGTERM<br>
W0504 16:21:59.902853 198230 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 198301 closing signal SIGTERM<br>
W0504 16:21:59.903526 198230 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 198302 closing signal SIGTERM<br>
W0504 16:21:59.904115 198230 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 198303 closing signal SIGTERM<br>
W0504 16:21:59.904471 198230 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 198304 closing signal SIGTERM<br>
W0504 16:21:59.905583 198230 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 198305 closing signal SIGTERM<br>
E0504 16:21:59.906981 198230 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 2 (pid: 198300) of binary: /home/ec2-user/test-env/bin/python3<br>
Traceback (most recent call last):<br>
File “/home/ec2-user/test-env/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 355, in wrapper<br>
return f(*args, **kwargs)<br>
File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/distributed/run.py”, line 918, in main<br>
run(args)<br>
File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/distributed/run.py”, line 909, in run<br>
elastic_launch(<br>
File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/distributed/launcher/api.py”, line 138, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/ec2-user/test-env/lib64/python3.9/site-packages/torch/distributed/launcher/api.py”, line 269, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</p>