CUDA driver initialization failed, you might not have a CUDA gpu

Andreas_Reich: <h2><a name="system-1" class="anchor" href="#system-1"></a>System:</h2>
<p>I have a 1-year old Linux Machine with 1x A100 40GB, plenty of ram and a server-grade CPU. I’m using Ubuntu 20.4…</p>
<h2><a name="main-use-2" class="anchor" href="#main-use-2"></a>Main Use:</h2>
<p>I use the machine manly to run LLM models with torch (textgeneration-webui and loading LLMS directly, but also other vision algorithms…)</p>
<h2><a name="previously-3" class="anchor" href="#previously-3"></a>Previously</h2>
<p>So, torch and The GPU were working fine and stable under Conda 11.7 with the Correct NVIDIA driver:</p>
<p>|Version:|515.105.01|<br>
|Operating System:|Linux 64-bit|<br>
|CUDA Toolkit:|11.7|</p>
<h2><a name="problem-solution-attempts-4" class="anchor" href="#problem-solution-attempts-4"></a>Problem &amp; Solution Attempts</h2>
<p> → Then, I upgraded to the 12.2 Driver which is not compatible with the latest pytorch (12.1) yet as it seems (nighlty build didn’t work as well).<br>
 → Then I downgraded to the 11.7 driver and the respective torch, but that stopped working then.</p>
<p> → Now, I’ve tried every Toolkit on the shelf from cuda-11 til 12.3 and all Available Drivers for my A100:</p>
<p>NVIDIA-Linux-x86_64-450.51.05.run<br>
NVIDIA-Linux-x86_64-515.86.01.run<br>
NVIDIA-Linux-x86_64-525.125.06.run<br>
NVIDIA-Linux-x86_64-535.104.12.run</p>
<p>for all of these toolkits and updated the .bashrc file accordingly.<br>
I also restarted the system every time.</p>
<p>System is fully up-to date.</p>
<p>nvidia-smi would always find the graphics card under any driver:</p>
<pre><code class="lang-auto">nvidia-smi
Mon Nov  6 10:01:15 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-PCI...  Off  | 00000000:B1:00.0 Off |                   On |
| N/A   32C    P0    36W / 250W |      0MiB / 40960MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+----------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
|      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
|                  |                      |        ECC|                       |
|==================+======================+===========+=======================|
|  No MIG devices found                                                       |
+-----------------------------------------------------------------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>Still, whatever I do, the system will tell me:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.cuda.is_available())
/home/ubuntu/anaconda3/envs/sentiment/lib/python3.9/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392035629/work/c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
&gt;&gt;&gt; print(torch.cuda.get_device_name(0))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/ubuntu/anaconda3/envs/sentiment/lib/python3.9/site-packages/torch/cuda/__init__.py", line 419, in get_device_name
    return get_device_properties(device).name
  File "/home/ubuntu/anaconda3/envs/sentiment/lib/python3.9/site-packages/torch/cuda/__init__.py", line 449, in get_device_properties
    _lazy_init()  # will define _get_device_properties
  File "/home/ubuntu/anaconda3/envs/sentiment/lib/python3.9/site-packages/torch/cuda/__init__.py", line 298, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.
</code></pre>
<p>And when I try to run basically any of my scripts in any of my environments with basically any driver active:</p>
<pre><code class="lang-auto">(sentiment) my/machine/and/path$ python myscript.py 
/home/ubuntu/anaconda3/envs/sentiment/lib/python3.9/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392035629/work/c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() &gt; 0

</code></pre>
<p>I’ve also tried this:</p>
<ul>
<li>
<p>I’ve created plenty of fresh environments to check if anything else is interfearing - its just torch that can’t talk to the GPU</p>
</li>
<li>
<p>I’ve also tried using environments with different python versions like 3.8/3.9/3.10/3.11 unsuccessfully.</p>
</li>
<li>
<p>I’ve installed plenty of legacy versions, nightly builds and so on.</p>
</li>
<li>
<p>Also tried to use pip / conda installs.</p>
</li>
</ul>
<p>I’ve been on this for about two weeks by now and have read plenty of other posts that seem similar and have tried many things that helped others but it just won’t work…</p>
<p>Here it was stated that PyTorch seems to be independent of my machines toolkit anyway:</p>
<p><a href="https://discuss.pytorch.org/t/trying-and-failing-to-install-pytorch-for-cuda-12-0/186194/2">ptrblck about CUDA Toolkit</a></p>
<p>But that the Driver is relevant:</p>
<p><a href="https://discuss.pytorch.org/t/cuda-initialization-cuda-driver-initialization-failed-you-might-not-have-a-cuda-gpu/186211/2">ptrblck - About Driver &amp; system reboot</a></p>
<p>If you need more specs or details I’m more than happy to provide anything needed.</p>
<p>Any help/tip is greatly appreciated.</p>

ptrblck: <aside class="quote no-group" data-username="Andreas_Reich" data-post="1" data-topic="191316">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andreas_reich/48/61652_2.png" class="avatar"> Andreas_Reich:</div>
<blockquote>
<p>Then, I upgraded to the 12.2 Driver which is not compatible with the latest pytorch (12.1) yet as it seems (nighlty build didn’t work as well).</p>
</blockquote>
</aside>
<p>All drivers are compatible within a major version due to the driver’s minor version compatibility support.</p>
<p>Disable MIG and it might work.</p>

Andreas_Reich: <p>Thank you, I couldn’t figure out when the MIG settings were activated, I think I deactivated them previously when I tried to get it to work, but I think I didn’t reboot the system then.</p>
<p>So to anyone with the same issue make sure to reboot after deactivating MIG!</p>
<p>Thanks very much for helping out!</p>

Andreas_Reich: <p>Ah and here’s the command:</p>
<pre><code class="lang-auto">sudo nvidia-smi -mig 0
</code></pre>
<p>worked for me → Check your <code>nvidia-smi</code> if MIG is active (as shown above). You might need to reboot before you are able to disable mig. Also make sure to <code>sudo systemctl stop display-manager</code></p>