NVIDIA L40s | CUDA driver initialization failed, you might not have a CUDA gpu

afnanrahman: <p>My OS: Ubuntu 24.04.2 LTS<br>
Driver Version: 570.86.15<br>
nvcc --version</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jan_15_19:20:09_PST_2025
Cuda compilation tools, release 12.8, V12.8.61
Build cuda_12.8.r12.8/compiler.35404655_0
</code></pre>
<p>Torch version cuda:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.version.cuda)
12.8
</code></pre>
<p>Nvidia-smi:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.15              Driver Version: 570.86.15      CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:03:00.0 Off |                    0 |
| N/A   27C    P8             23W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L40S                    On  |   00000000:04:00.0 Off |                    0 |
| N/A   28C    P8             20W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L40S                    On  |   00000000:63:00.0 Off |                    0 |
| N/A   27C    P8             22W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L40S                    On  |   00000000:64:00.0 Off |                    0 |
| N/A   26C    P8             22W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA L40S                    On  |   00000000:83:00.0 Off |                    0 |
| N/A   27C    P8             22W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA L40S                    On  |   00000000:84:00.0 Off |                    0 |
| N/A   26C    P8             21W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA L40S                    On  |   00000000:E3:00.0 Off |                    0 |
| N/A   27C    P8             21W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA L40S                    On  |   00000000:E4:00.0 Off |                    0 |
| N/A   26C    P8             22W /  350W |       1MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
</code></pre>
<p>Collect_env.py:<br>
<a href="https://gitlab.com/-/snippets/4819121" rel="noopener nofollow ugc">Gitlab Snippets</a></p>
<p>Testing with Pytorch:</p>
<pre><code class="lang-auto">python3
Python 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch 
&gt;&gt;&gt; torch.cuda.is_available() 
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:174: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
&gt;&gt;&gt; torch.cuda.current_device() 
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 1029, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 376, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.
&gt;&gt;&gt; torch.cuda.device(0)
&lt;torch.cuda.device object at 0x7c8b02a44560&gt;
&gt;&gt;&gt; exit()
</code></pre>
<p>What has been done:</p>
<ol>
<li>Enable persistent mode</li>
<li>Reboot</li>
</ol>
<p>Need help…</p>

paulge: <p>Is it possible that you installed the cuda runtime separately using pip? Meaning you installed both cuda and torch?</p>

ptrblck: <p>The error points to a driver issue and you could run a simple CUDA example to try to reproduce it as it’s most likely unrelated to PyTorch.</p>