RuntimeError: Unexpected error from cudaGetDeviceCount() when i use a6000

RiseInRose: <p>hey dude, please help me, i search all the google,and i can’t fix it. here is my error</p>
<pre><code class="lang-auto">Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch

&gt;&gt;&gt; 
&gt;&gt;&gt; torch.cuda.is_available()
/home/caturbhuja/2T/conda3_envs/tt/lib/python3.10/site-packages/torch/cuda/__init__.py:107: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 101: invalid device ordinal (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
&gt;&gt;&gt; torch.cuda
&lt;module 'torch.cuda' from '/home/caturbhuja/2T/conda3_envs/tt/lib/python3.10/site-packages/torch/cuda/__init__.py'&gt;
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; torch.cuda.current_device()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/caturbhuja/2T/conda3_envs/tt/lib/python3.10/site-packages/torch/cuda/__init__.py", line 674, in current_device
    _lazy_init()
  File "/home/caturbhuja/2T/conda3_envs/tt/lib/python3.10/site-packages/torch/cuda/__init__.py", line 247, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 101: invalid device ordinal
</code></pre>
<p>here is my torch/cuda version<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e858bfc3428cd0fe09480d74baca1517ddae28b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/e858bfc3428cd0fe09480d74baca1517ddae28b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e858bfc3428cd0fe09480d74baca1517ddae28b7_2_690x291.png" alt="image" data-base62-sha1="x9qURy57EisFxQoxlsLCIj2T6rd" width="690" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e858bfc3428cd0fe09480d74baca1517ddae28b7_2_690x291.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e858bfc3428cd0fe09480d74baca1517ddae28b7_2_1035x436.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e858bfc3428cd0fe09480d74baca1517ddae28b7_2_1380x582.png 2x" data-dominant-color="F1DEDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1796×760 67.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here is my nvidia driver version</p>
<pre><code class="lang-auto">Thu Apr 13 00:27:09 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A6000    Off  | 00000000:B3:00.0 Off |                  Off |
| 30%   30C    P8     9W / 300W |      0MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>

ptrblck: <p>Your driver is running into an initialization error, so you might need to update/reinstall your drivers.</p>

RiseInRose: <p>i update driver many times,it still not work. here is my driver version now</p>
<p>Thu Apr 13 10:12:26 2023<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  NVIDIA RTX A6000    Off  | 00000000:B3:00.0 Off |                  Off |<br>
| 30%   60C    P0    67W / 300W |      0MiB / 49140MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|  No running processes found                                                 |<br>
±----------------------------------------------------------------------------+</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d73ba2bd943a2b1614f467cc67679ee1d33118b.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d73ba2bd943a2b1614f467cc67679ee1d33118b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d73ba2bd943a2b1614f467cc67679ee1d33118b_2_690x363.png" alt="image" data-base62-sha1="1V09oCpVMueJ6rORujHNAMQvluH" width="690" height="363" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d73ba2bd943a2b1614f467cc67679ee1d33118b_2_690x363.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d73ba2bd943a2b1614f467cc67679ee1d33118b_2_1035x544.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d73ba2bd943a2b1614f467cc67679ee1d33118b.png 2x" data-dominant-color="EBECEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1230×648 55.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>In case this setup was working before I would recommend trying to build and execute any CUDA sample to verify that it’s still working as the errors still point to a setup issue.</p>

RiseInRose: <p>Actually, when I was using the 2080ti, torch2.0 was working fine. But when I switched to the a6000 and installed the relevant drivers, the same code started throwing errors.</p>

RiseInRose: <p>Thanks, dude! you know what? The reason is due to my non-powered 2080ti, because of my laziness, I left it in the computer case without powering it on. Oh my god！When I took it off… ，Surprise ！！！</p>

ptrblck: <p>Good to hear you have isolated and fixed the issue. This is indeed a new one, but makes sense assuming the driver tries to initialize the “dead” GPU.</p>

Dj_Jag: <p>In my case this not works in Ubuntu<br>
Ive got last 535 cuda with 530 driver<br>
nvTop working and shows my CPU, I can play games or start Heaven test same time but…</p>
<p>python test3.py<br>
__Python VERSION: 3.11.4+ (main, Jun 28 2023, 08:52:25) [GCC 9.4.0]<br>
__pyTorch VERSION: &lt;module ‘torch.version’ from ‘/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/torch/version.py’&gt;<br>
__CUDA VERSION<br>
__CUDNN VERSION: 8500<br>
__Number CUDA Devices: 1<br>
__Devices<br>
index, name, driver_version, memory.total [MiB], memory.used [MiB], memory.free [MiB]<br>
0, NVIDIA GeForce GTX 1070, 530.30.02, 8192 MiB, 143 MiB, 7965 MiB<br>
Traceback (most recent call last):<br>
File “/media/jag/NEU/3PAX/ubuntu-webui/test3.py”, line 11, in <br>
print(‘Active CUDA Device: GPU’, torch.cuda.current_device())<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/torch/cuda/<strong>init</strong>.py”, line 674, in current_device<br>
_lazy_init()<br>
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/torch/cuda/<strong>init</strong>.py”, line 247, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized</p>

Smith_David: <p>hi, <a class="mention" href="/u/riseinrose">@RiseInRose</a>  what does non-powered mean?</p>

msunshinelxl: <p>我也遇到这个问题，在docker环境下遇到的，操作系统是：<br>
root@iv-yd1ran0cwam0ad91m1ab:~# cat /proc/version<br>
Linux version 5.4.0-133-generic (buildd@lcy02-amd64-003) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) <span class="hashtag-raw">#149-Ubuntu</span> SMP Mon Nov 14 18:36:06 UTC 2022<br>
母机是是12.3（12.4）也遇到同样的问题：<br>
root@iv-yd1m36hweem0adi50tnx:~# nvidia-smi<br>
Mon Mar 25 14:44:47 2024<br>
±--------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |<br>
|-----------------------------------------±---------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                      |               MIG M. |<br>
|=========================================+======================+======================|<br>
|   0  NVIDIA A800-SXM4-80GB          Off | 00000000:65:01.0 Off |                    0 |<br>
| N/A   30C    P0              58W / 400W |      2MiB / 81920MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   1  NVIDIA A800-SXM4-80GB          Off | 00000000:65:02.0 Off |                    0 |<br>
| N/A   28C    P0              60W / 400W |      2MiB / 81920MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+</p>
<p>±--------------------------------------------------------------------------------------+<br>
| Processes:                                                                            |<br>
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |<br>
|        ID   ID                                                             Usage      |<br>
|=======================================================================================|<br>
|  No running processes found                                                           |<br>
±--------------------------------------------------------------------------------------+<br>
docker环境也是差不多：</p>
<p>a406134a828(@:):/# nvcc -V<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Wed_Nov_22_10:17:15_PST_2023<br>
Cuda compilation tools, release 12.3, V12.3.107<br>
Build cuda_12.3.r12.3/compiler.33567101_0</p>
<p>ca406134a828(@:):/# apt search cudnn<br>
Sorting… Done<br>
Full Text Search… Done<br>
libcudnn9-cuda-12/now 9.0.0.312-1 amd64 [installed,local]<br>
cuDNN runtime libraries for CUDA 12.3</p>
<p>libcudnn9-dev-cuda-12/now 9.0.0.312-1 amd64 [installed,local]<br>
cuDNN development headers and symlinks for CUDA 12.3</p>
<p>pytorch是2.2.0</p>
<p>解决是把所有版本对齐如下：<br>
4e60f119169d(@:):hw2# nvcc -V<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2021 NVIDIA Corporation<br>
Built on Mon_May__3_19:15:13_PDT_2021<br>
Cuda compilation tools, release 11.3, V11.3.109<br>
Build cuda_11.3.r11.3/compiler.29920130_0<br>
4e60f119169d(@:):hw2# apt search cudnn<br>
Sorting… Done<br>
Full Text Search… Done<br>
libcudnn8/now 8.9.5.29-1+cuda11.8 amd64 [installed,local]<br>
cuDNN runtime libraries</p>
<p>libcudnn8-dev/now 8.9.5.29-1+cuda11.8 amd64 [installed,local]<br>
cuDNN development libraries and headers</p>
<p>4e60f119169d(@:):hw2# vim ./test.py<br>
4e60f119169d(@:):hw2# python3 ./test.py<br>
1.11.0</p>