During backward() | CUDA error: an illegal memory access was encountered

Kungbohan: <p>Hello,</p>
<p>I encounter a CUDA error: <strong>CUDA error: an illegal memory access was encountered</strong><br>
I do not know how to deal with it.<br>
I googled this issue, I found a suggestion indicating that the problem might arise from multiple users accessing the same GPU card. However, I am the sole user of the card (I checked <code>nvidia-smi</code> several times).<br>
I also found that this issue may result from apex. I did use torch amp; however, if I removed amp and torch.autocast, the problem was not solved.<br>
Can you tell me any possible causes of the problem or how to avoid it?<br>
Thank you.</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/uu/<span class="mention">@Research</span>/BlockHH/main.py”, line 130, in <br>
main(config)<br>
File “/home/uu/<span class="mention">@Research</span>/BlockHH/main.py”, line 100, in main<br>
trainer()<br>
File “/home/uu/<span class="mention">@Research</span>/BlockHH/trainer/trainer.py”, line 243, in <strong>call</strong><br>
scaler.scale(loss).backward()<br>
File “/home/uu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/_tensor.py”, line 487, in backward<br>
torch.autograd.backward(<br>
File “/home/uu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/autograd/<strong>init</strong>.py”, line 200, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="Kungbohan" data-post="1" data-topic="199081">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kungbohan/48/64953_2.png" class="avatar"> Kungbohan:</div>
<blockquote>
<p>I found a suggestion indicating that the problem might arise from multiple users accessing the same GPU card.</p>
</blockquote>
</aside>
<p>It’s the first time I’m hearing this, but no, multiple processes or users do not cause memory violations in your script.</p>
<p>Could you run your code via <code>compute-sanitizer python script.py</code> and check if it can detect the kernel causing the issue?</p>

Kungbohan: <p>Hello,<br>
Thank you.<br>
I run my code via your suggestion.<br>
The error messenge is as follows. (Error: No attachable process found.)<br>
BTW, I am training a neural network (CNN). If I set initial_channel=32, this issue won’t arise.<br>
However, if the initial_channel is greater than 32, it arises.</p>
<p>========= COMPUTE-SANITIZER<br>
========= Error: No attachable process found. compute-sanitizer timed-out.<br>
========= Default timeout can be adjusted with --launch-timeout. Awaiting target completion.<br>
Traceback (most recent call last):<br>
File “/home/uu/<span class="mention">@Research</span>/BlockHH/main.py”, line 130, in <br>
main(config)<br>
File “/home/uu/<span class="mention">@Research</span>/BlockHH/main.py”, line 100, in main<br>
trainer()<br>
File “/home/uu/<span class="mention">@Research</span>/BlockHH/trainer/trainer.py”, line 243, in <strong>call</strong><br>
scaler.scale(loss).backward()<br>
File “/home/uu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/_tensor.py”, line 487, in backward<br>
torch.autograd.backward(<br>
File “/home/uu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/autograd/<strong>init</strong>.py”, line 200, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: CUDA error: misaligned address<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue so that we could try to isolate the failing kernel?</p>

Kungbohan: <p>Hello,</p>
<p>After several days of debugging, we have finally identified the problem.<br>
The error originates from the function:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.linalg.solve.html#torch.linalg.solve" rel="noopener nofollow ugc"><code>torch.linalg.solve()</code></a><br>
During back-propagation, the gradient turns into NaN, which I suspect might be due to an overflow causing “an illegal memory access was encountered.”<br>
I opted for using <code>inv()</code> instead, which works well.<br>
Thank you.</p>
<hr>
<p>For anyone who has the same issue:<br>
If you’re incorporating inverse matrices into your back-propagation process, I recommend avoiding the use of <code>torch.linalg.solve()</code> to prevent similar issues.</p>