What is the interplay between the "nvidia driver" and "cuda"?

username: KFrank
cooked: <p>Hello All!</p>
<p>I am under the impression that the “nvidia driver” and “cuda” are two separate things.<br>
How do the two work together?</p>
<p>Would there be a use case where somebody uses the “nvidia driver” but not “cuda”?</p>
<p>I ask because after upgrading a laptop to ubuntu 24.04 (and reinstalling pytorch),<br>
pytorch no longer had cuda support (and nvidia-smi was no longer available).</p>
<p>(For context, I install and run pytorch in a conda environment.  nvidia-smi runs both<br>
outside and inside the conda environment.)</p>
<p>I had to first install the nvidia driver (sudo ubuntu-drivers install nvidia:550, outside<br>
of conda) and then install pytorch (with cuda, inside of conda).</p>
<p>nvidia-smi reports (outside and inside of conda):</p>
<blockquote>
<p>NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4</p>
</blockquote>
<p>Running pytorch (inside of conda) I get:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.6.0+cu126'
&gt;&gt;&gt; torch.version.cuda
'12.6'
</code></pre>
<p>Does the “nvidia driver” automatically come bundled with some version of “cuda”?<br>
When, if ever, does the “CUDA Version: 12.4” get used?  (I assume that when I run<br>
pytorch I am using cuda 12.6.)</p>
<p>More context:  I am only (knowingly) using cuda when I run pytorch.</p>
<p>Thanks for any information about how these two pieces of the system work together.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="KFrank" data-post="1" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I am under the impression that the “nvidia driver” and “cuda” are two separate things.<br>
How do the two work together?</p>
</blockquote>
</aside>
<p>Yes, these are two separate things if we are talking about the <em>NVIDIA Driver</em> and the <em>CUDA Toolkit</em>. <em>CUDA</em> itself could mean both: the driver as well as the toolkit.<br>
While the NVIDIA Driver is making sure your system can properly communicate with and use your GPU, the CUDA Toolkit ships with math libraries (cuBLAS, cuSOLVER, cuRAND, etc.) as well as the CUDA compiler toolchain (<code>nvcc</code>, <code>ptxas</code>, etc.). Installing a CUDA Toolkit locally allows you to build CUDA applications, such as PyTorch. You have certainly seen my comment on a few questions explaining that PyTorch binaries ship with their own CUDA runtime dependencies (i.e. the CUDA runtime, CUDA Math libs, etc.) and users only need to properly install an NVIDIA driver.<br>
Now, if you download a CUDA Toolkit from e.g. <a href="https://developer.nvidia.com/cuda-downloads">here</a> it will ship with the NVIDIA driver as well. During the installation you will have an option to install the Toolkit alone or the Driver as well. You can also download the NVIDIA driver separately, which will be of course a smaller package.</p>
<aside class="quote no-group" data-username="KFrank" data-post="1" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Would there be a use case where somebody uses the “nvidia driver” but not “cuda”?</p>
</blockquote>
</aside>
<p>Yes, if you mean the CUDA Toolkit by “cuda”. I.e. if you want to mainly run PyTorch applications and don’t plan to ever source build any CUDA application or a PyTorch 3rd party library etc., you won’t need to install a CUDA toolkit and can depend on the CUDA runtime dependencies which will be installed during the PyTorch binary installation. While running <code>pip install torch</code> you will see that <code>nvidia-*</code> wheels are pulled into your environment allowing PyTorch to use these.</p>
<aside class="quote no-group" data-username="KFrank" data-post="1" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I ask because after upgrading a laptop to ubuntu 24.04 (and reinstalling pytorch),<br>
pytorch no longer had cuda support (and nvidia-smi was no longer available).</p>
</blockquote>
</aside>
<p>This is expected as an OS update would most likely need a driver re-installation as well.</p>
<aside class="quote no-group" data-username="KFrank" data-post="1" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I had to first install the nvidia driver (sudo ubuntu-drivers install nvidia:550, outside<br>
of conda) and then install pytorch (with cuda, inside of conda).</p>
</blockquote>
</aside>
<p>This is the right approach. You can install the NVIDIA drivers from <code>apt</code> or you could also use standalone installers from <a href="https://www.nvidia.com/en-us/drivers/">here</a>. I would strongly advise not to mix these approaches. If you installed the drivers from <code>apt</code>, stick to it for driver updates etc.</p>
<aside class="quote no-group" data-username="KFrank" data-post="1" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Running pytorch (inside of conda) I get:</p>
</blockquote>
</aside>
<p>Great! As a quick smoke test, you could create a tensor on the GPU making sure PyTorch can also communicate with the device.</p>
<aside class="quote no-group" data-username="KFrank" data-post="1" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Does the “nvidia driver” automatically come bundled with some version of “cuda”?</p>
</blockquote>
</aside>
<p>Yes, as mentioned before the CUDA Toolkit download ships with the NVIDIA driver as well and you have the option to select the installation of it. However, based on your description you have installed the NVIDIA Driver beforehand and the PyTorch binaries afterwards, which is fine and correct. Note that that PyTorch binaries do <em>not</em> ship with an NVIDIA Driver so you have to install it beforehand.</p>
<aside class="quote no-group" data-username="KFrank" data-post="1" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>When, if ever, does the “CUDA Version: 12.4” get used? (I assume that when I run<br>
pytorch I am using cuda 12.6.)</p>
</blockquote>
</aside>
<p>I assume you are referring to the “Driver Version: 550.120 CUDA Version: 12.4” output of <code>nvidia-smi</code>. If so, then note that <code>nvidia-smi</code> reports the driver version and the corresponding CUDA toolkit this driver ships with. It does not tell you that a full CUDA toolkit was properly installed (and you can have multiple CUDA Toolkits installed on your system). To check for a locally installed CUDA Toolkit run <code>nvcc --version</code> or try to build any CUDA sample from source.<br>
You are thus right in assuming PyTorch will use its own CUDA 12.6.3 runtime dependencies during the execution.</p>
<p>Also, you haven’t asked it but just to explain the compatibility a bit more: the NVIDIA Driver is compatible for all minor CUDA updates. I.e. to run any PyTorch binary with CUDA 12.x you need to install any NVIDIA Driver &gt;= 525.60.13 as described in the <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility">Minor Version Compatibility Docs</a>. Once the driver is installed you can simply <code>pip install</code> any PyTorch binary (stable, nightly, etc.) you want.<br>
NVIDIA Drivers are <em>not</em> compatible between CUDA major updates. I.e. if you are using a CUDA 11.x driver, you won’t be able to run applications compiled and linked against CUDA 12.x dependencies.</p>
<p>Let me know if you have any more questions!</p>

username: KFrank
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Yes, these are two separate things if we are talking about the <em>NVIDIA Driver</em> and the <em>CUDA Toolkit</em>.</p>
</blockquote>
</aside>
<p>Thank you – this clears things up.</p>
<aside class="quote no-group">
<blockquote>
<p>Let me know if you have any more questions!</p>
</blockquote>
</aside>
<p>Two more, if I may:</p>
<p>Am I correct that I can have the nvidia driver (say from installing the driver) and the<br>
cuda runtime libraries (say from installing pytorch with cuda support), but not have<br>
the cuda toolkit installed?  That is, would I be able to run pytorch with cuda tensors,<br>
but not be able to run, say, nvcc.?</p>
<p>Is it usefully possible to have just the nvidia driver (lets my system talk to the nvidia<br>
card / chip) and not the cuda runtime?  For example, maybe an nvidia video card is<br>
used by a windows manager or video game but is not running any cuda computations.<br>
Or does video processing always use “cuda” and the cuda libraries under the hood?</p>
<p>Thanks.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Am I correct that I can have the nvidia driver (say from installing the driver) and the<br>
cuda runtime libraries (say from installing pytorch with cuda support), but not have<br>
the cuda toolkit installed? That is, would I be able to run pytorch with cuda tensors,<br>
but not be able to run, say, nvcc.?</p>
</blockquote>
</aside>
<p>Yes, this is a correct and possible setup. To run PyTorch only you would need to install the NVIDIA Driver and run <code>pip install torch</code> - that’s it. The driver makes sure your system can communicate with the GPU while the PyTorch binary ships with all needed CUDA runtime libraries.</p>
<p>E.g. creating a <code>CUDATensor</code> in PyTorch would internally:</p>
<ul>
<li>open PyTorch libs which were installed from the wheel, e.g. <code>libtorch.so</code>, <code>libtorch_cuda.so</code>, <code>libtorch_global_deps.so</code>,</li>
<li>open the CUDA Runtime to call <code>cudaMalloc</code> (to create the new <code>CUDATensor</code>): <code>libcudart.so</code> is installed as a wheel and thus located in the same Python env,</li>
<li>open the NVIDIA Driver which would be installed outside of your Python env, e.g. at <code>/lib/x86_64-linux-gnu/libcuda.so.1</code>.</li>
</ul>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="217811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Is it usefully possible to have just the nvidia driver (lets my system talk to the nvidia<br>
card / chip) and not the cuda runtime? For example, maybe an nvidia video card is<br>
used by a windows manager or video game but is not running any cuda computations.<br>
Or does video processing always use “cuda” and the cuda libraries under the hood?</p>
</blockquote>
</aside>
<p>Yes, this is also a valid use case and you would also install an NVIDIA Driver if you want to play video games on Linux. I’m not familiar enough with internal details and which components are used for rendering and general gaming acceleration. My naive understanding is that OpenGL and Vulkan are still used APIs for graphics programming, but I’m purely speculating as I’m not deeply familiar with gaming applications.</p>