RuntimeError: Unexpected error from cudaGetDeviceCount()

username: Samah_Abu_saleem
cooked: <p>I was training GCN model on my Linux server and I suddenly got this error.</p>
<blockquote>
<p>RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW</p>
</blockquote>
<ul>
<li>Pytorch version: 1.10.1+cu102</li>
<li>OS: Linux</li>
<li>Python version: Python 3.8.10</li>
<li>CUDA Version: 11.2</li>
</ul>

username: ptrblck
cooked: <p>Is <code>nvidia-smi</code> returning any errors and complains about a driver mismatch? If so, could you restart the server and check if it helps? If not, did you recently update any drivers or are you manually trying to get forward compatibility working on non-server GPUs?</p>

username: Samah_Abu_saleem
cooked: <p>No, it doesn’t return any errors:</p>
<blockquote>
<p>NVIDIA-SMI 450.57,        Driver Version: 450.57 ,      CUDA Version: 11.2</p>
</blockquote>
<p>I have restarted it many times but still the same problem.</p>
<p>I didn’t do any updates. I installed PyTorch and it’s installing successfully</p>

username: Samah_Abu_saleem
cooked: <p>Sir, by doing:</p>
<blockquote>
<p>! python -c "import torch; print(torch.cuda.is_available())</p>
</blockquote>
<p>I got:</p>
<pre><code class="lang-auto">/usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:80: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:112.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/issues/40671">this issue</a> other users were running into the same error message if</p>
<ul>
<li>their setup was broken due to a driver/library mismatch (rebooting seemed to solve the issue)</li>
<li>their installed drivers didn’t match the user-mode driver inside a docker container (and forward compatibility failed due to the usage of non-server GPUs)</li>
</ul>
<p>Was your setup working before and if so, what changed?</p>

username: Samah_Abu_saleem
cooked: <p>Thank you Sir :). My problem is solved.<br>
By doing:</p>
<p><code>!pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html</code><br>
<code>!pip3 install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu113.html</code><br>
<code>!pip3 install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-1.10.0+cu113.html</code><br>
<code>!pip3 install torch-cluster -f https://data.pyg.org/whl/torch-1.10.0+cu113.html</code><br>
<code>!pip3 install torch-geometric</code></p>

username: JiamingLiu-Jeremy
cooked: <p>Hi ptrblck,</p>
<p>Thanks for your valuable comments.</p>
<p>I might have very similar issue below. I’m implementing DDP training on HPCs (with SLURM or LSF), where each node has 4 V100 GPUs. Without any changes of my code, envs etc (pyotrch=1.9.0, cuda 10.2), I randomly meet this issue very recently. As a results, I found torch.cuda.is_available=False on few nodes while CUDA on most nodes do available. It’s hard to reboot the cluster, do you have some suggestions for further debugging?</p>
<p>[UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at  …/c10/cuda/CUDAFunctions.cpp:115.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0]</p>

username: ptrblck
cooked: <p>Do you receive this message randomly?<br>
If so, I would guess your system encountered any kind of issue and might have dropped the GPU.<br>
Based on the error message I would have guessed you are running into a setup issue, but this wouldn’t explain the randomness (assuming you are indeed seeing these errors randomly). Are you seeing any Xids in <code>dmesg</code>?</p>

username: JiamingLiu-Jeremy
cooked: <p>Hi ptrblck,</p>
<p>Thanks for the quick reply. I think you are right about the “randomness”, and sorry for the confusion. I think the “randomness” comes from the randomly signed computing nodes at each time submitting a job. If it is the setup issue of pytorch, cuda and driver, how can most nodes work well, but few of nodes are failed, assuming each computing node is identically setup?</p>
<p>Buy running dmesg, I do find some Xids as examples below . But I don’t think these Xids make torch.cuda unavailable, since I can succesfully perform DDP trainig with these Xids.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/887cfb16da5371af0475d23167cf34b96011793c.png" data-download-href="https://discuss.pytorch.org/uploads/default/887cfb16da5371af0475d23167cf34b96011793c" title="Screen Shot 2022-07-24 at 4.48.56 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/887cfb16da5371af0475d23167cf34b96011793c_2_689x110.png" alt="Screen Shot 2022-07-24 at 4.48.56 PM" data-base62-sha1="jtqHMqJ9JE6hpz4RUpEPdTclVmc" width="689" height="110" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/887cfb16da5371af0475d23167cf34b96011793c_2_689x110.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/887cfb16da5371af0475d23167cf34b96011793c_2_1033x165.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/887cfb16da5371af0475d23167cf34b96011793c_2_1378x220.png 2x" data-dominant-color="3B3B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-07-24 at 4.48.56 PM</span><span class="informations">2480×398 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b59a10f4e6059c9c37710b939ec963afdf482b89.png" data-download-href="https://discuss.pytorch.org/uploads/default/b59a10f4e6059c9c37710b939ec963afdf482b89" title="Screen Shot 2022-07-24 at 5.28.15 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b59a10f4e6059c9c37710b939ec963afdf482b89_2_690x165.png" alt="Screen Shot 2022-07-24 at 5.28.15 PM" data-base62-sha1="pUwvsZkiDfO6xsbYzFqGuvjKKyR" width="690" height="165" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b59a10f4e6059c9c37710b939ec963afdf482b89_2_690x165.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b59a10f4e6059c9c37710b939ec963afdf482b89_2_1035x247.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b59a10f4e6059c9c37710b939ec963afdf482b89_2_1380x330.png 2x" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-07-24 at 5.28.15 PM</span><span class="informations">2337×561 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The <code>Xids</code> show that you are running into a page fault, which is usually an illegal memory access caused in the software stack somewhere.</p>
<aside class="quote no-group" data-username="JiamingLiu-Jeremy" data-post="9" data-topic="139977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jiamingliu-jeremy/48/51722_2.png" class="avatar"> JiamingLiu-Jeremy:</div>
<blockquote>
<p>I think the “randomness” comes from the randomly signed computing nodes at each time submitting a job.</p>
</blockquote>
</aside>
<p>Does this mean that the same node causes the issue if it’s selected in your env?<br>
If so, check the node’s health status as it seems it has trouble with the driver.</p>

username: JiamingLiu-Jeremy
cooked: <p>Hi ptrbick,</p>
<p>Thanks for your suggestions. A little update here, it’s indeed because of the node’s health status. After rebooting the nodes, the problem was solved (at least partially).</p>

username: M_Saiful_Bari
cooked: <p>Just to confirm, same here. Randomly met this error. In my last run the code was running properly. Suddenly it start producing this error. I tried rebooting but didn’t work. Checked cuda runtimes, all good.<br>
I would really appreciate some help.</p>

username: vetsa.sai
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I wanted to build a docker to run a repository which specifically only works for PyTorch1.90+cu11.1<br>
I had everything working on my local PC (which is also host for docker) which has built with</p>
<ul>
<li>Ubuntu 22.04</li>
<li>NVIDIA-SMI 470.239.06</li>
<li>Driver Version: 470.239.06</li>
<li>CUDA Version: 11.4</li>
<li>torch1.9.0+cu11.1</li>
</ul>
<p>When I try to replicate this on docker image built upon <strong>nvidia/cuda:11.7.1-cudnn8-devel-ubuntu22.04</strong> that has below versions:</p>
<ul>
<li>Ubuntu 22.04</li>
<li>NVIDIA-SMI 470.239.06</li>
<li>Driver Version: 470.239.06</li>
<li>CUDA Version: 11.7</li>
<li>nvcc 11.7</li>
<li>torch1.9.0+cu11.1</li>
</ul>
<p>and then <strong>torch.cuda.is_available()</strong> gives me this error.</p>
<p><em>UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:115.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0</em><br>
<strong>False</strong></p>
<p>Please suggest what can be done in this case! This combination works on Local PC but not on docker.</p>
<p><em>PS:</em><br>
If you suggest me to use ‘nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04’ to match the CUDA version, it would give me below error… which made me to upgrade to 22.04/11.7</p>
<p><em>ImportError: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32’ not found (required by /opt/conda/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)</em></p>

username: ptrblck
cooked: <p>I don’t know the differences between your local run and the docker one, but the same error was fixed <a href="https://discuss.pytorch.org/t/after-nvidia-driver-update-cuda-is-disabled/198368/5">here</a> by reinstalling the NVIDIA driver.</p>

username: Clark_Benham
cooked: <p>What solved my issue was installing and starting the fabric manager:</p>
<pre><code class="lang-auto">sudo apt install nvidia-fabricmanager-535 libnvidia-nscq-535
sudo systemctl start nvidia-fabricmanager # this was likely the missing piece
</code></pre>
<p>I’ve got an 8 gpu server I’m using and needed to start this service so they could talk to each other.</p>
<p>(I also found <a href="https://ubuntu.com/server/docs/nvidia-drivers-installation" rel="noopener nofollow ugc">https://ubuntu.com/server/docs/nvidia-drivers-installation</a> helpful for only installing the drivers, and <a href="https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#removing-cuda-toolkit-and-driver" class="inline-onebox" rel="noopener nofollow ugc">CUDA Installation Guide for Linux</a> for removing previous installs.  )</p>

username: surajpatil4899
cooked: <p>I am getting same issue. <code>nvidia-smi</code> is working. I have setting up cuda in wsl <code>ubuntu 22.4</code>. I have manually updated cuda drivers on windows can you help.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1efb97561ef0988cf8f37620845ed82178bd798.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1efb97561ef0988cf8f37620845ed82178bd798" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1efb97561ef0988cf8f37620845ed82178bd798.png" alt="image" data-base62-sha1="ywgG0Q6lKjhCjj4SoGccWAl5zok" width="690" height="30" data-dominant-color="0B2C5C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1898×84 4.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>