RuntimeError: The NVIDIA driver on your system is too old (found version 11040)

chrathans: <p>I have already a setup in a local server machine with the following installations:</p>
<ul>
<li>Python 3.9</li>
<li>CUDA Version: 11.4 and with nvcc -version</li>
</ul>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0
</code></pre>
<p>Pytorch version 2.6.0</p>
<p>This setup works for a couple of models but when I am trying to load <code>microsoft/Phi-3-mini-4k-instruct</code> from hugging face I receive the following message:</p>
<p><code>RuntimeError: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.</code></p>
<p>How can I solve this issue and why I am receiving it? Why it work partially in some cases?</p>

ptrblck: <p>Assuming you’ve installed the PyTorch binaries with CUDA 12.x runtime dependencies, you would need to update your NVIDIA driver to <code>&gt;=525.60.13</code>.</p>

chrathans: <p>Since this is a server and not a personal machine, could that be done just for myself or needs to be handled by the system admin?</p>

ptrblck: <p>You would need root access to update drivers, so you might need to contact your admin if your user account does not have these rights.</p>

chrathans: <p>What about this message:</p>
<p>Alternatively, go to: <a href="https://pytorch.org" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver</p>
<p>Could I just solve my issue by updating PyTorch version?</p>

ptrblck: <p>Your are not compiling with a CUDA driver, so the message sounds wrong. Where did you see it?</p>
<p>CUDA12 was released 2 years ago and it seems your admins weren’t able to update the drivers in this time frame?<br>
If so, you could install a PyTorch binary with CUDA 11.8 libs and check if it would work in your setup.</p>

chrathans: <p>Tbh, I thought that I had already installed a PyTorch version for Cuda 11.8. How can I find the correct PyTorch version for my case?</p>

ptrblck: <p>Select the CUDA runtime version form the <a href="https://pytorch.org/get-started/locally/">install matrix</a> and copy/paste the install command into your terminal:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/195c5246789d123041fe82934340be94e97ced5f.png" data-download-href="https://discuss.pytorch.org/uploads/default/195c5246789d123041fe82934340be94e97ced5f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/195c5246789d123041fe82934340be94e97ced5f_2_690x276.png" alt="image" data-base62-sha1="3ClJx7pClBU2lAQkaSWvHgaPkEn" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/195c5246789d123041fe82934340be94e97ced5f_2_690x276.png, https://discuss.pytorch.org/uploads/default/original/3X/1/9/195c5246789d123041fe82934340be94e97ced5f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/9/195c5246789d123041fe82934340be94e97ced5f.png 2x" data-dominant-color="F4DCD8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">832×334 29.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

chrathans: <p>Ah yeah, this is what I have already done! Then, I guess there is an issue with my model and old CUDA versions!</p>

ptrblck: <p>Were you able to run any PyTorch binary on your servers before? If so, do you remember which PyTorch version and CUDA runtime libs were used? If not, you could try to use docker containers which support CUDA’s forward compatibility in case you are using Datacenter GPUs.</p>

chrathans: <p>Actually, I realized that no code was really running in the GPU but automatically they were running in the CPU (there was a catch in case it couldn’t run in a GPU).</p>

ptrblck: <p>OK, are you able to compile and run any CUDA sample?</p>

chrathans: <p>Apparently not! I always receive this message when I try to use Cuda devices instead of CPU!</p>

ptrblck: <p>If you see the same or similar error message after compiling any other CUDA application (not PyTorch) it seems your systems might have general issues communicating with the GPUs and you might want to ask your admins for help.</p>

chrathans: <p>I mean nvidia-smi command returns the version of CUDA the same as nvcc -version. Is there another way to check CUDA besides PyTorch?</p>

ptrblck: <p>Yes, since you’ve installed a CUDA toolkit locally, you could <code>git clone https://github.com/NVIDIA/cuda-samples</code>, <code>cd</code> in e.g. <code>Samples/0_Introduction/matrixMul</code>, build the example via <code>make</code>, and execute the binary.</p>

MattFrank: <p>We are finding that this error is sometimes actually an indication that the driver suffered an out-of-memory error (not clear yet what the source of running out of memory is).  When this happens you might see Assertion failures from NVRM in dmesg.  In the cases where this out-of-memory problem is the root cause, the short term work around is to reboot the system.</p>