Torch.cuda.device_count() return 1 but torch.cuda.is_available() false

WEI_DA: <p>Hello, I am trying to test our model on a new physical host (Ubuntu 22.04) with one A30 GPU available. But encoutered this weird issues.</p>
<ol>
<li>
<p>We followed the instruction to install the pytorch compatble with cuda 12.6: pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu126" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu126</a></p>
</li>
<li>
<p>And following the official instruction to install cuda 12.6 and nvidia driver (560)</p>
</li>
<li>
<p>nvidia-smi and nvcc both works and showed as<br>
nvidia-smi<br>
Fri Apr  4 18:07:55 2025<br>
±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 560.28.03              Driver Version: 560.28.03      CUDA Version: 12.6     |<br>
|-----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  NVIDIA A30                     Off |   00000000:25:00.0 Off |                   On |<br>
| N/A   26C    P0             26W /  165W |       1MiB /  24576MiB |     N/A      Default |<br>
|                                         |                        |              Enabled |<br>
±----------------------------------------±-----------------------±---------------------+</p>
</li>
</ol>
<p>±----------------------------------------------------------------------------------------+<br>
| MIG devices:                                                                            |<br>
±-----------------±---------------------------------±----------±----------------------+<br>
| GPU  GI  CI  MIG |                     Memory-Usage |        Vol|        Shared         |<br>
|      ID  ID  Dev |                       BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |<br>
|                  |                                  |        ECC|                       |<br>
|==================+==================================+===========+=======================|<br>
|  No MIG devices found                                                                   |<br>
±----------------------------------------------------------------------------------------+</p>
<p>±----------------------------------------------------------------------------------------+<br>
| Processes:                                                                              |<br>
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |<br>
|        ID   ID                                                               Usage      |<br>
|=========================================================================================|<br>
|  No running processes found                                                             |<br>
±----------------------------------------------------------------------------------------+</p>
<ol start="4">
<li>however, pytorch torch.cuda.device_count() return 1 but torch.cuda.is_available() as False and failed to torch._C._cuda_init() with “No CUDA GPUs are available” error raised</li>
</ol>
<p>Any idea on this? thanks</p>

ptrblck: <p>Disable MIG or specify the corresponding MIG slice as explained in the <a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/">MIG user guide</a>.</p>