Pytorch suddenyl stops recognising GPU

Flock1: <p>Hi,</p>
<p>I am using PyTorch through Anaconda Environment and something weird happens. While working or if I leave the machine for some time and come back, PyTorch stops recognizing the GPU. And the only way it starts recognizing the GPU is after rebooting the machine.</p>
<p>Why does this happen?</p>

mrshenli: <p>You mean <code>torch.cuda.device_count()</code> returns 0? Can you confirm <code>nvidia-smi</code> still works correctly when that happens? And can you also check what is the value for <code>CUDA_VISIBLE_DEVICES</code> env var?</p>

Flock1: <p>Hi,</p>
<p>Yeah. <code>torch.cuda.device_count()</code> returns 0 and <code>torch.cuda.current_device()</code> returns the following:</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1591914886554/work/aten/src/THC/THCGeneral.cpp line=47 error=999 : unknown error
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/user/anaconda3/envs/work/lib/python3.8/site-packages/torch/cuda/__init__.py", line 330, in current_device
    _lazy_init()
  File "/home/user/anaconda3/envs/work/lib/python3.8/site-packages/torch/cuda/__init__.py", line 153, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (999) : unknown error at /opt/conda/conda-bld/pytorch_1591914886554/work/aten/src/THC/THCGeneral.cpp:47
</code></pre>
<p><code>nvidia-smi</code> works. For <code>CUDA_VISIBLE_DEVICES</code>, I get nothing.</p>

user_123454321: <p>This happens to me sometimes and to fix without rebooting I reload gpu using</p>
<pre><code class="lang-auto">$ sudo rmmod nvidia_uvm
$ sudo modprobe nvidia_uvm
</code></pre>
<p>No idea why it happens though</p>

Flock1: <p>This works. How did you find out this solution? It’s so weird right. Suddenly it stops working. I think there’s some internal functioning of pytorch that changes something</p>

user_123454321: <p>Hmmm…since it worked on rebooting my laptop, I guessed it should work by just reloading the gpu. So, searched online on how to reboot nvidia gpu.</p>

Flock1: <p>Thanks for the solution. This has been bothering me for quite some time now. I’m sure they’ll fix this in later versions.</p>

ptrblck: <aside class="quote no-group" data-username="Flock1" data-post="5" data-topic="93384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>I think there’s some internal functioning of pytorch that changes something</p>
</blockquote>
</aside>
<p>Are any other CUDA applications running fine, i.e. are you able to run some CUDA examples etc.?<br>
I’m not sure, if this is a PyTorch-related issue or rather a CUDA/NVIDIA driver issue.</p>

Flock1: <p>I didn’t check that unfortunately. I can try checking with Keras if that lobrary is unable to recognize the GPU.</p>
<p>I’ll also try running CUDA examples from within the environment and outside it.</p>

KFrank: <p>Hello Flock!</p>
<aside class="quote no-group" data-username="Flock1" data-post="1" data-topic="93384" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>While working or if I leave the machine for some time and come back, PyTorch stops recognizing the GPU. And the only way it starts recognizing the GPU is after rebooting the machine.</p>
</blockquote>
</aside>
<p>Just to share my experience (with an old version of pytorch and an<br>
old gpu):</p>
<p>I see something similar to this.  If I launch a fresh python session<br>
and run a pytorch script that uses cuda, then if I don’t use cuda<br>
(or maybe just the python session) for a short-ish amount of time,<br>
future use of cuda in that python session fails.</p>
<p>But I don’t have to reboot my machine or “reload the gpu” to get<br>
it working again; I only have to exit and restart python.</p>
<p>I haven’t found any fix for it – I just live with it, restarting python as<br>
necessary.</p>
<p>Here’s a post of mine with some related observations:</p>
<aside class="quote quote-modified" data-post="1" data-topic="52696">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/some-observations-on-cuda-runtime-error-30/52696">Some observations on "cuda runtime error (30)"</a> 
  </div>
  <blockquote>
    Hello Forum! 
I have some information about the behavior of “cuda runtime 
error (30)” (probably somewhat specific to my particular 
configuration). 
This is a follow-on to a number of threads about “error 30,” 
and, in particular, to this post: 


Clued in by Andrei’s observation that torch.cuda.is_available() 
“breaks” cuda, I find (for me) that if torch.cuda.is_available() 
is the first cuda call, subsequent cuda calls will throw “error 30” 
unless the first subsequent call is called promptly…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

malfet: <p>Can you please share the versions of PyTorch and CUDA you are using (and perhaps a GPU type)?<br>
Also, are there any messages printed to the kernel log (can be checked by running <code>dmesg</code>) when this happens?</p>

Flock1: <p>Hey Frank,</p>
<p>Thank you for sharing your experience.</p>
<blockquote>
<p>I see something similar to this. If I launch a fresh python session<br>
and run a pytorch script that uses cuda, then if I don’t use cuda<br>
(or maybe just the python session) for a short-ish amount of time,<br>
future use of cuda in that python session fails.</p>
</blockquote>
<p>This is what happens but unfortunately for me, I have either have to restart the machine or reload the GPU. Just restarting python didn’t help. I even tried reloading the conda environment. It’s as if a switch went off and I have to physically switch it on again.</p>

Flock1: <p>I use PyTorch through conda environment.<br>
PyTorch: 1.5.1<br>
Cuda tool kit: 10.1.243</p>
<p>On my machine, I have CUDA 11 for RTX 2070 Super GPU</p>

Flock1: <p>Does working from an anaconda environment affect this? Because the environment won’t use the CUDA innstalled on the machine but the one dowanloded by anaconda itself.</p>

ptrblck: <p>As you said, the <code>cudatoolkit</code> from the conda binaries will be used and your local CUDA11 installation will thus not be used.<br>
What do you mean by “affect this”?</p>

Flock1: <p>I was referring to pytorch suddenly stops recognising thr GPU by ‘affect this’. So what I wanted to ask if is how to check which CUDA is causing the problem. The one that was installed with anaconda (<code>cudatoolkit</code>) or the one that’s locally installed (Cuda 11).</p>

ptrblck: <p>Thanks for the explanation. As said, the <code>cudatoolkit</code> (shipped via the binaries) will be used.<br>
However, I doubt that CUDA is responsible for this behavior and would first look into potential hardware, driver, PSU issues.<br>
You could check <code>dmesg</code> for any <code>XID</code> errors.</p>

Mehran_Rafiee: <aside class="quote no-group" data-username="user_123454321" data-post="4" data-topic="93384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/user_123454321/48/27631_2.png" class="avatar"> user_123454321:</div>
<blockquote>
<pre><code class="lang-auto">$ sudo rmmod nvidia_uvm
$ sudo modprobe nvidia_uvm
</code></pre>
</blockquote>
</aside>
<p>Worked! such a waste of time.</p>