CUDA error: unspecified launch failure

Shen: <p>Hi, recently my PyTorch ran into an issue:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
</code></pre>
<p>After this, the GPU got lost. Typing <code>nvidia-smi</code> gave</p>
<blockquote>
<p>Unable to determine the device handle for GPU 0000:02.00.0: Unknown Error</p>
</blockquote>
<p>Unfortunately this is all information the terminal displayed. However, by going through <a href="https://github.com/pytorch/pytorch/issues/27837" rel="noopener nofollow ugc">this discussion</a>, I can conditionally make the code run by doing one of these:</p>
<pre><code class="lang-auto">1. Set CUDA_LAUNCH_BLOCKING=1. Surprisingly, this did not provide additional error info. Instead, the program ran properly after setting this, with the cost of 5x-6x training time.
2. Reduce number of workers to 0 and set non_blocking=False when transferring input images/labels to GPU. This also worked, with the cost of 6x-8x data loading time. 
</code></pre>
<p>Another observation is that when I increased the batch size, the errors sometimes occurred even though the program only used 2/3 of the GPU memory.</p>
<p>Here are my system configurations:</p>
<pre><code class="lang-auto">GPU: 2080Ti
OS: Ubuntu 18.04
Nvidia driver version: 495.29.05
CUDA version: 11.5
PyTorch version: tried 1.9 to 1.11, they all had similar issues
</code></pre>
<p>Please help! Thanks!</p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce the issue on the 2080Ti, please?</p>

Shen: <p>Hi, I simply tried the code from PyTorch example <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py" rel="noopener nofollow ugc">here</a>, without any parallelization or distributed features (as I only have 1 GPU).</p>

ptrblck: <p>Do you see any <code>dmesg</code> Xid errors as I don’t know if the reported error in <code>nvidia-smi</code> is the root cause or a symptom of the previous issue?</p>

Shen: <p>Do you mean the output of <code>dmesg</code> command? I ran <code>dmesg | grep GPU</code>, here is the output when GPU functioned properly (nothing is running):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a83a71f0b5a0c775b3e4ab788ffc4c0ce148e653.png" alt="healthy1" data-base62-sha1="o0dqQBzXyjQNun2NLSzBFjwlwBB" width="557" height="20"></p>
<p>And here is the output after the above error occured:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b20d7b3f96f9f191ce78023e9ee6f7483c3a4754.png" data-download-href="https://discuss.pytorch.org/uploads/default/b20d7b3f96f9f191ce78023e9ee6f7483c3a4754" title="faulty"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b20d7b3f96f9f191ce78023e9ee6f7483c3a4754.png" alt="faulty" data-base62-sha1="pp7Sa4VKARhvwIoW8gmeMbzLS4I" width="690" height="108" data-dominant-color="383838"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">faulty</span><span class="informations">696×109 5.55 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thank you! Yes, the <code>Xid 79</code> is helpful as it explains that the “GPU has fallen off the bus”, which can be a HW error, driver error, system memory corruption or thermal issue.<br>
I think a few weeks ago I’ve seen the same issue where the power plug wasn’t fully connected to the GPU and caused the same issue.<br>
Given that, <code>Xid 79</code> should not be raised by user code, so I don’t believe your PyTorch code (or any library) is the root cause.</p>

Shen: <p>Thanks for your help!</p>

Mahesha999: <p>I was experiencing this issue on my HP laptop with Nvidia graphics when developing inside the docker container. Turns out if the laptop goes to sleep, then after waking up the laptop, the graphics driver goes into some undesired state. Only restarting the machine work.</p>