RuntimeError: CUDA error: invalid argument second time I run the same code

amartinf13: <p>Hi everyone,</p>
<p>We are using torch to accelerate the computation of some iterative solvers within a bigger project.<br>
The code runs perfectly when using CPU capabilities but when using GPU we run into this runtime error the second time we run the code.</p>
<p>The use of torch is enclosed within a method that is called several times during the whole run and it doesn’t produce problems. The issue appears if we use a script to run this code several times. Then the second time it produces this runtime error the first time it tries to allocate a tensor.</p>
<blockquote>
<p>kspace_torch = torch.tensor(kspace, dtype=torch.complex64, device=device)<br>
RuntimeError: CUDA error: invalid argument<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stack trace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>
<p>If we run twice the code in separate calls it does not produce any problem, so we assume it is a problem of not freeing well the resources either for torch or the other parts of the code that use GPU. Does anyone have any clue of what could be happening?</p>
<p>Thank you very much!</p>

smth: <p>can you run your script with <code>CUDA_LAUNCH_BLOCKING=1</code> and report what the stack-trace looks like?</p>
<p>CUDA errors are often reported on memory allocations even if the true location of the error is not at the allocation – because CUDA is an asynchronous computing model.</p>

amartinf13: <aside class="quote no-group" data-username="smth" data-post="2" data-topic="186112">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>stack-trace</p>
</blockquote>
</aside>
<p>Sure, thanks! I did it and it doesn’t change the location of the error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/engine/src/edutool/edutool/interface.py”, line 488, in spintwin_execution<br>
frame[‘images’] = frame_reconstruction(<br>
File “/home/engine/src/recon/recon/recon.py”, line 94, in frame_reconstruction<br>
kspace_torch = torch.tensor(kspace, dtype=torch.complex64, device=device)<br>
RuntimeError: CUDA error: invalid argument<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>
<p>Also, we log the status of the gpus at the time of the error and they seem to be free:</p>
<blockquote>
<p>‘gpus’: [{‘name’: ‘NVIDIA L4’, ‘memory_total’: 23034, ‘memory_free’: 21660, ‘compute_capability’: ‘8.9’}, {‘name’: ‘NVIDIA L4’, ‘memory_total’: 23034, ‘memory_free’: 21660, ‘compute_capability’: ‘8.9’}, {‘name’: ‘NVIDIA L4’, ‘memory_total’: 23034, ‘memory_free’: 20952, ‘compute_capability’: ‘8.9’}]</p>
</blockquote>

ptrblck: <p>Could you update PyTorch to the latest nightly release and check if you would still see the error?<br>
We were seeing similar issues in <code>2.0.0</code> (should have been fixed in <code>2.0.1</code> if I’m not mistaken) for sm_89 devices using the BetterTransformer backend.<br>
If this still doesn’t help, could you post a minimal and executable code snippet reproducing the issue, please?</p>

amartinf13: <p>Thanks for the answer! We’re actually using the <code>2.0.1</code> but I also tried with the nightly release and it didn’t help.<br>
I wish I could reproduce it in a minimal code snippet, but this code also runs some wrappers for internal code written in C++ that uses the GPU. The problem must come from the interaction with these wrappers, since we’re not able to reproduce the problem without those calls.</p>

ptrblck: <p>You could then try to narrow down the issue via <code>cuda-gdb</code> or <code>compute-sanitizer</code>, which should hopefully point to the failing kernel.</p>

amartinf13: <p>Thanks a lot! We found that the problem comes from doing a</p>
<pre><code class="lang-auto">cudaDeviceReset())
</code></pre>
<p>between calls. Is there any way we could avoid this problem from the torch side? We need to reset the device because of performance in the CUDA side.</p>

ptrblck: <p>Could you explain why kind of “performance in the CUDA side” issues you are seeing that warrants a device reset?<br>
I don’t think you can recover from it on the PyTorch side and would also need to restart the Python process.</p>

amartinf13: <p>Got it, it was actually an error on our side that affected the performance. Thank you for the help!</p>

Leo_Leung: <p>It worked for me using 2.0.1, thank you so much!</p>