5090 RTX fail to initialize in pytorch

skaae: <p>Hi,</p>
<p>I’m tying to build a pytorch training machine with 4x5090 RTX cards, but are running into some issues with getting pytorch to find the cards.</p>
<p>First the cards are found fine with nvidia smi:</p>
<pre><code class="lang-auto">Tue May 13 11:19:55 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.144                Driver Version: 570.144        CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   36C    P0             81W /  600W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5090        Off |   00000000:21:00.0 Off |                  N/A |
|  0%   34C    P0             60W /  600W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 5090        Off |   00000000:C1:00.0 Off |                  N/A |
|  0%   33C    P0             65W /  600W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 5090        Off |   00000000:E1:00.0 Off |                  N/A |
|  0%   31C    P0             65W /  600W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
</code></pre>
<p>But cuda is not available in torch and fails with</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch

&gt;&gt;&gt; torch.cuda.is_available()

/home/ansible/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:181: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)

return torch._C._cuda_getDeviceCount() &gt; 0

False
</code></pre>
<p>We run  Python 3.12.3 and torch <code>2.8.0.dev20250512+cu128</code>. OS is Ubuntu 24.04.2 LTS with kernel 6.14.6. Its assembled on a [GENOA2D24G motherboard] (<a href="https://www.asrockrack.com/general/productdetail.asp?Model=GENOA2D24G-2L%2B#Specifications" rel="noopener nofollow ugc">https://www.asrockrack.com/general/productdetail.asp?Model=GENOA2D24G-2L%2B#Specifications</a>)</p>
<p>I wonder if theres anything obvious we can try? otherwise our next step would be to swap the 5090 with some 4090’s to see if that would work.</p>

ashok_raghava: <p>download the cuda tool kit and the necessary drivers and restart the kernel</p>

ptrblck: <p>Based on the error message it seems your driver cannot initialize the devices so make sure it was properly installed.</p>
<aside class="quote no-group" data-username="ashok_raghava" data-post="2" data-topic="219985">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ashok_raghava/48/76993_2.png" class="avatar"> ashok_raghava:</div>
<blockquote>
<p>download the cuda tool kit</p>
</blockquote>
</aside>
<p>A CUDA toolkit is not needed if you install the PyTorch binaries as they ship with all needed CUDA runtime dependencies.</p>

skaae: <p>Thanks, and yes, I agree its probably a driver issue.</p>
<p>I downloaded the driver from nvidias driver page: <a href="https://www.nvidia.com/en-us/drivers/details/243330/" class="inline-onebox" rel="noopener nofollow ugc">Driver Details | NVIDIA</a></p>
<p>and selected the MIT driver. I think i’ll try with a 4090 to check that some card will work.</p>

adax: <p>FWIW I am also running into this issue with this board - 4090s work fine, blackwell seems to have issues (still debugging)</p>

adax: <p>Same environment but setting visible devices to only 1 works fine (e.g. export CUDA_VISIBLE_DEVICES=0; python train.py …).  Seems to error out in DDP?  I can get past the original error by specifying up to 4 GPUs in CUDA_VISIBLE_DEVICES, but then I get a “CUDA error: an illegal memory access was encountered” error for 2 or more GPUs w/DDP.</p>
<p>Smoke test (another data point during debugging):<br>
export CUDA_VISIBLE_DEVICES=0,1,2,3; python -c ‘import torch; torch.cuda.is_available()’<br>
works fine, but then adding more than 4 GPUs fails:<br>
export CUDA_VISIBLE_DEVICES=0,1,2,3,4; python -c ‘import torch; torch.cuda.is_available()’<br>
/home/adaboost/miniconda3/envs/mustango/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py:181: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0</p>

skaae: <p>Thank you for the reply.</p>
<p>I tried setting CUDA_VISIBLE_DEVICES to both 0 and 0,1,3,4 and I still get</p>
<pre><code class="lang-auto"> CUDA_VISIBLE_DEVICES=0; python -c 'import torch; torch.cuda.is_available()'
/home/ansible/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:181: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() &gt; 0
</code></pre>
<p>Since that is working for you I probably messed up some of the cables.<br>
I hope I can ask a few questions about your setup :)?</p>
<ul>
<li>
<p>What kind of risercards did you use? We used <a href="https://c-payne.com/products/mcio-pcie-gen5-device-adapter-x8-x16" class="inline-onebox" rel="noopener nofollow ugc">MCIO PCIe gen5 Device Adapter x8/x16 – c-payne.com</a> and with the lowest numbered MCIO port going into the MCIO connector closets to the power connector on the riser card.</p>
</li>
<li>
<p>Did you change any bios settings or flashed the bios? I changed 3 settings in bios:</p>
<ol>
<li>disable IOMMU in advanced → AMD CBS → NBIO common options → IOMMU disabled</li>
<li>setting link width to 16 in advanced → AMD CBS → NBIO common options → SMU common options → xGMI</li>
<li>changing the MCIO from 4x4x/4x/4x to 16x in advanced → chipset configuration → pcie link width → MCIO</li>
</ol>
</li>
</ul>

adax: <p>Same riser card (cpayne must be making a killing), with same cable MCIO order.</p>
<p>Running latest bios, but only changed PCIE link to 16x, didn’t touch IOMMU/xGMI.</p>
<p>I have been successfully running 8x4090s for a while on the same drivers / older pytorch version.  I don’t think its a board issue.  I also found others running into the same illegal memory error here (still open): <a href="https://github.com/pytorch/pytorch/issues/150734" class="inline-onebox" rel="noopener nofollow ugc">Training/Fine-tuning fails with PyTorch 2.8 + 4x 5090 GPUs using DDP/FSDP/DeepSpeed · Issue #150734 · pytorch/pytorch · GitHub</a></p>

skaae: <p>Thank you.<br>
I’ll try reverting the bios changes and flash the bios to the newest version and check if that makes a difference <img src="https://discuss.pytorch.org/images/emoji/apple/crossed_fingers.png?v=14" title=":crossed_fingers:" class="emoji" alt=":crossed_fingers:" loading="lazy" width="20" height="20"></p>
<p>We have a bunch of other gpu servers that runs 4090’s. So as a last resort I think we can swap the 4x5090 with 4x4090.</p>

adax: <p>As for the illegal memory issue, it seems to have resolved via upgrading nccl (from: <a href="https://github.com/ultralytics/ultralytics/issues/19987#issuecomment-2788372641" class="inline-onebox" rel="noopener nofollow ugc">Minimal train DDP crash using 2x5090 GPUs · Issue #19987 · ultralytics/ultralytics · GitHub</a>)</p>
<p>pip install --upgrade nvidia-nccl-cu12</p>
<p>However, I’m still stuck on going beyond 4 cards - such an odd error <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=14" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

adax: <p>As an update, this is not a pytorch issue, but something with the drivers.  I compiled CUDA samples and ran one of the multiGPU tests (cuda-samples/build/Samples/0_Introduction/simpleMultiGPU), setting CUDA_VISIBLE_DEVICES=0,1,2,3 and 0,1,2,3,4 which reproduced the init error:</p>
<p>Starting simpleMultiGPU<br>
CUDA error at cuda-samples/Samples/0_Introduction/simpleMultiGPU/simpleMultiGPU.cu:100 code=3(cudaErrorInitializationError) “cudaGetDeviceCount(&amp;GPU_N)”</p>
<p>At this point will start with a fresh OS/driver/pytorch install and see if it clears, otherwise will just have to wait for an updated driver release</p>

Mario_Marietto: <p>Your error looks like mine :</p>
<aside class="quote quote-modified" data-post="5" data-topic="219850">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mario_marietto/48/55008_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/trying-to-run-pytorch-and-stable-diffusion-and-or-comfyui-on-freebsd/219850/5">Trying to run PyTorch and Stable Diffusion and / or ComfyUI on FreeBSD</a> 
  </div>
  <blockquote>
    This is what I did : 
# pkg install libc6-shim linux-miniconda-installer linux-c7-devtools linux-c7 linux-c7-libglvnd

# fetch h t t p s : / / g i s t . g i t h u b u s e r c o n t e n t . c o m / shkhln / 40ef290463e78fb2b0000c60f4ad797e / raw / f640983249607e38af405c95c457ce4afc85c608 / uvm_ioctl_override.c

# /compat/linux/usr/bin # ./gcc --sysroot=/compat/linux -m64 -std=c99 -Wall -ldl -fPIC -shared -o /compat/linux/dummy-uvm.so /mnt/da2p2/Backup/uvm_ioctl_override.c
# /compat/linux/usr/bin#…
  </blockquote>
</aside>

<p>but as you can see no one has been able to give a relevant help.</p>
<p>your error is :</p>
<pre><code class="lang-auto">/home/ansible/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:181: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)

return torch._C._cuda_getDeviceCount() &gt; 0

False
</code></pre>
<p>my error is :</p>
<pre><code class="lang-auto">Did you run some cuda functions before calling NumCudaDevices() that might have already set an error?

Error 304: OS call failed or operation not supported on this OS (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>They are exactly the same !</p>
<p>The only way to make work Stable diffusion is to use the nvidia driver and the linux-nvidia-libs vers. 525.78.01 (the latest version which works is the  535) and “torch==1.12.1+cu113 --extra-index-url <a href="https://download.pytorch.org/whl/cu113%E2%80%9D" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu113”</a> and it works like a charme :</p>
<pre><code class="lang-auto">(pytorch) I have no name! @ marietto:/usr/home/marietto$ LD_PRELOAD="/compat/dummy-uvm.so" python3 -c 'import torch; print(torch.cuda.is_available())'

True

(pytorch) I have no name!@marietto:/usr/home/marietto$ LD_PRELOAD="/compat/dummy-uvm.so" python3 -c 'import torch; print(torch.cuda.get_device_name(0))'

NVIDIA GeForce RTX 2080 Ti
</code></pre>

adax: <p>Final update - fixed but had to revert to Linux kernel 6.5 w/Ubuntu 22.04.  not sure what happened but at least my jobs are running now.  Was based on a hint from this guy running a B200 and having similar issues:</p>
<aside class="onebox discoursetopic" data-onebox-src="https://forums.developer.nvidia.com/t/cuda-12-8-with-driver-version-570-124-06-on-b200-hgx-getting-code-3-cudaerrorinitializationerror/331233">
  <header class="source">
      <img src="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.developer.nvidia.com/t/cuda-12-8-with-driver-version-570-124-06-on-b200-hgx-getting-code-3-cudaerrorinitializationerror/331233" target="_blank" rel="noopener nofollow ugc" title="08:55PM - 23 April 2025">NVIDIA Developer Forums – 23 Apr 25</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:150/80;"><img src="https://global.discourse-cdn.com/nvidia/original/3X/e/4/e41d27491ee72562cf68c340597ca95f587879bd.png" class="thumbnail" width="150" height="80"></div>

<div class="title-wrapper">
  <h3><a href="https://forums.developer.nvidia.com/t/cuda-12-8-with-driver-version-570-124-06-on-b200-hgx-getting-code-3-cudaerrorinitializationerror/331233" target="_blank" rel="noopener nofollow ugc">Cuda 12.8 with Driver Version: 570.124.06 on B200 HGX getting...</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Accelerated Computing</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">CUDA</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">CUDA Setup and Installation</span>
        </span>
      </span>
  </div>
</div>

  <p>nvidia-smi Wed Apr 23 20:39:55 2025        +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |...</p>

  <p>
    <span class="label1">Reading time: 1 mins 🕑</span>
      <span class="label2">Likes: 1 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


skaae: <p>Hi,</p>
<p>We got our setup to work with 4090’s by adding <code>nokaslr</code> to kernel boot params.<br>
In <code>/etc/default/grub</code> we changed:</p>
<pre><code class="lang-auto">GRUB_CMDLINE_LINUX_DEFAULT="nokaslr"
</code></pre>
<p>and then <code>sudo update-grub</code> and reboot.</p>
<p>The current setup is</p>
<p>Ubuntu 24.04.2 LTS<br>
kernel: 6.14.6-061406-generic<br>
Driver Version: 570.133.20     CUDA Version: 12.8     (MIT driver)</p>
<p>I’ll try with a 5090 now and post an update.</p>
<p>Thank you for helping out!</p>