Error:" CUDA driver initialization failed, you might not have a CUDA gpu." on an HPC

Sam_anton: <h2><a name="p-465367-i-was-trying-to-run-rstarhttpsgithubcommicrosoftrstar-mainpy-for-testing-purpose-on-an-hpc-and-i-got-this-error-image690x325upload1nvg07bdupgbkl4cndaeb7fd8gxpng-1" class="anchor" href="#p-465367-i-was-trying-to-run-rstarhttpsgithubcommicrosoftrstar-mainpy-for-testing-purpose-on-an-hpc-and-i-got-this-error-image690x325upload1nvg07bdupgbkl4cndaeb7fd8gxpng-1"></a>I was trying to run <a href="https://github.com/microsoft/rStar" rel="noopener nofollow ugc">rstar</a> main.py for testing purpose on an HPC and i got this error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09aa8a3bfb7c9fabe8865d90f4e732e77e1f1653.png" data-download-href="https://discuss.pytorch.org/uploads/default/09aa8a3bfb7c9fabe8865d90f4e732e77e1f1653" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09aa8a3bfb7c9fabe8865d90f4e732e77e1f1653_2_690x325.png" alt="image" data-base62-sha1="1nvG07BDupgBkl4cNDaeB7fD8gX" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09aa8a3bfb7c9fabe8865d90f4e732e77e1f1653_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09aa8a3bfb7c9fabe8865d90f4e732e77e1f1653_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09aa8a3bfb7c9fabe8865d90f4e732e77e1f1653_2_1380x650.png 2x" data-dominant-color="2D2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1897×895 231 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></h2>
<p>i tried some debugging  step suggested by chatgpt</p>
<pre><code class="lang-auto">(rstar) [ingenx@rdgpu01 ~]$ nvidia-smi
Wed Feb 12 13:35:14 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.06              Driver Version: 545.23.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe          Off | 00000000:3B:00.0 Off |                   On |
| N/A   51C    P0              68W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80GB PCIe          Off | 00000000:5E:00.0 Off |                   On |
| N/A   56C    P0              79W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| MIG devices:                                                                          |
+------------------+--------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                   Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                     BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                |        ECC|                       |
|==================+================================+===========+=======================|
|  No MIG devices found                                                                 |
+---------------------------------------------------------------------------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
(rstar) [ingenx@rdgpu01 ~]$ nvcc --version
-bash: nvcc: command not found
(rstar) [ingenx@rdgpu01 ~]$ python3 -c "import torch; print('CUDA Available:', torch.cuda.is_available())"
/home/ingenx/miniconda3/envs/rstar/lib/python3.11/site-packages/torch/cuda/__init__.py:129: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() &gt; 0
CUDA Available: False
(rstar) [ingenx@rdgpu01 ~]$ python3 -c "import torch; print('Number of GPUs:', torch.cuda.device_count())"
Number of GPUs: 2
(rstar) [ingenx@rdgpu01 ~]$ python3 -c "import torch; print('GPU Name:', torch.cuda.get_device_name(0))"
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/home/ingenx/miniconda3/envs/rstar/lib/python3.11/site-packages/torch/cuda/__init__.py", line 493, in get_device_name
    return get_device_properties(device).name
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ingenx/miniconda3/envs/rstar/lib/python3.11/site-packages/torch/cuda/__init__.py", line 523, in get_device_properties
    _lazy_init()  # will define _get_device_properties
    ^^^^^^^^^^^^
  File "/home/ingenx/miniconda3/envs/rstar/lib/python3.11/site-packages/torch/cuda/__init__.py", line 319, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.

</code></pre>
<hr>
<p>i was not able to find any solution on internet about this problem</p>

ptrblck: <p>Deactivate MIG or make sure the appropriate MIG slice is visible in the process.</p>