CUDA fails to reinitialize after system suspend

jayz: <p>Hello,</p>
<p>I am running training on a machine with an RTX 3080, Ubuntu 22.04LTS, CUDA 11.6, in a jupyter notebook anaconda environment.</p>
<p>My training data is stored in an external mongo db. When initializing the training, I load a large chunk of this db-dataset to in-memory of my machine, which takes about 20 minutes each time. When I put my machine to sleep mode (“suspend”), afterwards, when trying to resume training, I get this error message and the cuda device is not recognized, anymore:</p>
<pre><code class="lang-python">/home/user/anaconda3/envs/pyt116/lib/python3.10/site-packages/torch/cuda/__init__.py:83:
UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up 
environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the
available devices to be zero. (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:109.)
return torch._C._cuda_getDeviceCount() &gt; 0
</code></pre>
<p>It seems to be related to a change of the CUDA_VISIBLE_DEVICES flag. The error results in having to reboot the machine and reloading all the data each time, which is a cumbersome process. Thus, can anybody give advice on how to make the system recognize the cuda device, again, after waking the pc?</p>
<p>Thanks!</p>
<p>Best, JZ</p>

ptrblck: <p>It’s a known issue of Linux’ suspend mode and the CUDA driver.<br>
Resetting the device as described <a href="https://discuss.pytorch.org/t/userwarning-cuda-initialization-cuda-unknown-error-this-may-be-due-to-an-incorrectly-set-up-environment-e-g-changing-env-variable-cuda-visible-devices-after-program-start-setting-the-available-devices-to-be-zero/129335/5">here</a> might work.</p>

jayz: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>unfortunately, this doesn’t solve the issue.<br>
<code>sudo rmmod nvidia_uvm</code><br>
<code>&gt;&gt; ERROR: module nvidia_uvm is in use</code><br>
and the second command then doesn’t give anything in response.</p>
<p>Maybe, it’s because I am using a graphical interface on my pc? Or is the nvidia module somehow still tied in the jupyter notebook session?</p>
<p>EDIT: I see in your post that it can be related to IDEs. Is there another solution to this by now, or still as is?</p>

ptrblck: <p>If I’m running into this issue, I need to kill all Python processes (the simple way is just to close your IDE) and check for other processes in <code>nvidia-smi</code> (e.g. Spotify sometimes also grabs the GPU and blocks the device reset) which I could close. Afterwards it’s usually working for me.</p>

jayz: <p>Yes, ok, so basically, what you are suggesting is a solution to the problem, but there is no way around closing all IDE’s / python processes. Thus, I’d have to reload my data every time. So, it doesn’t solve my problem, but it seems there is no workaround, anyways.</p>

vr308: <p>any update to this problem? is this ubuntu specific or does it persist in other linux distro’s.</p>
<p>Quite a major hindrance if one is used to coding in a style where you do a training run and you need to investigate / extract from different variables…basically continue to work on a single process for a couple hours.</p>

ptrblck: <p>I don’t understand why your use case would need to be suspended.<br>
As far as I know it’s a known Linux issue, but unsure which distros are affected.</p>

TFW: <p>Same issue here using arch</p>
<p>The <a href="https://man7.org/linux/man-pages/man8/rmmod.8.html" rel="noopener nofollow ugc">rmmod man page</a> says there’s a -f/--force flag for the error<br>
<code>ERROR: module nvidia_uvm is in use</code><br>
but it’s not recommended, I did it anyways but it didn’t solve the issue</p>

rsc3: <p>Still seeing this behavior with Ubuntu 22.04. And now I see gnome shell always grabs 3MiB in nvidia-smi so effectively I have to log out and log back in to reload CUDA.</p>

Nielius: <p>I found a solution here: <a href="https://askubuntu.com/a/1309807/1915066" class="inline-onebox" rel="noopener nofollow ugc">nvidia - How do I fix CUDA breaking after suspend? - Ask Ubuntu</a></p>
<p>As far as I understand, it changes the way that suspend/resume stores the GPU’s memory.</p>