Compatibility between CUDA 12.6 and PyTorch

username: mictad
cooked: <p>Hello,</p>
<p>I’m in the process of fine tuning a LLM, and my machine has these specifications:</p>
<p>NVIDIA RTX A6000<br>
NVIDIA-SMI 560.35.03<br>
CUDA Version: 12.6</p>
<p>I have hard time to find the right PyTorch packages that are compatible with my CUDA version.</p>
<p>Does anyone know which one should I download it?<br>
Every suggestion is welcome.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension, since the PyTorch binaries ship with their own CUDA runtime dependencies. You can thus select any PyTorch binary with CUDA 12.x and I would recommend using the latest one (12.4.1).</p>

username: saksham
cooked: <p>so it won’t be a problem right i have cuda version 12.6 too</p>

username: saksham
cooked: <p>this is what it says so what should i do now</p>
<p>Could not find a version that satisfies the requirement torch==2.3.0+cu126 (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1)<br>
ERROR: No matching distribution found for torch==2.3.0+cu126</p>

username: ptrblck
cooked: <p>You need to install a supported binary from <a href="http://PyTorch.org">PyTorch.org</a> while it seems you are trying to manipulate the version tags yourself.</p>

username: saksham
cooked: <p>thanks a lot for replying i was installing a text to speech model which said that it needed pytorch.<br>
well a gave up and installed cuda 11.8 let’s see if it works</p>

username: greek_freak
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> I am having a similar issue and would appreciate your opinion.</p>
<p>I set up a EC2 instance on AWS (g4dn.xlarge, Amazon Linux, Deep Learning Base OSS Nvidia Driver GPU AMI) and got access to a T4 with cuda 12.6. <code>nvidia-smi</code> confirms the above</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       On  |   00000000:00:1E.0 Off |                    0 |
| N/A   21C    P8             11W /   70W |       1MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
</code></pre>
<p>I went ahead and installed <code>torch</code></p>
<pre><code class="lang-auto">pip3 freeze | grep torch
pytorch-triton==3.2.0+git35c6c7c6
torch==2.5.1+cu124
</code></pre>
<p>and tried out a simple command</p>
<pre><code class="lang-auto">python3 -c "import torch;print(torch.cuda.is_available())"
</code></pre>
<p>The output of this command is the following</p>
<pre><code class="lang-auto">/home/ec2-user/.local/lib/python3.9/site-packages/torch/cuda/__init__.py:129: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>I have tried a few things listed in similar questions but have gotten no luck. I also find it interesting that I am not doing anything insanely complicated. It is an EC2 instance setup with a simple <code>torch.cuda.is_available()</code> invocation. Would you have any thoughts?</p>

username: ptrblck
cooked: <pre data-code-wrap="python"><code class="lang-python">UserWarning: CUDA initialization: CUDA driver initialization failed...
</code></pre>
<p>Your setup is unable to initialize the driver and you could try to run any other CUDA application which would raise the same error. I don’t know what your exact setup is but make sure your environment or container can properly communicate with the NVIDIA driver.</p>

username: annelo
cooked: <p>I apologize if the question has already been asked, but I currently use Python 3.12 and torch==2.5.1+cu124.</p>
<p>I don’t quite understand exactly how this works and whether it is correct.<br>
The code runs without errors, and the speed of its work has even increased: on the CPU-up to about 20 seconds per iterative step, and on the GPU - up to 15 seconds.</p>
<details>
<summary>
Screenshot</summary>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68bb6e52c37dea54d34e827644eea31eeb06fd15.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/68bb6e52c37dea54d34e827644eea31eeb06fd15" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68bb6e52c37dea54d34e827644eea31eeb06fd15_2_690x487.jpeg" alt="1" data-base62-sha1="eWve0iVm0R1XO16Axy1NKbHYRo1" width="690" height="487" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68bb6e52c37dea54d34e827644eea31eeb06fd15_2_690x487.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68bb6e52c37dea54d34e827644eea31eeb06fd15_2_1035x730.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68bb6e52c37dea54d34e827644eea31eeb06fd15_2_1380x974.jpeg 2x" data-dominant-color="585B62"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">2554×1806 437 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</details>
<p>This is mostly an experiment, but it would be interesting to see if it works properly.<br>
Thanks!</p>

username: ptrblck
cooked: <p>I’m not sure what your question is, but note that PyTorch binaries ship with their own CUDA runtime dependencies. Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</p>

username: Wessi
cooked: <p>Hello <a class="mention" href="/u/mictad">@mictad</a> and <a class="mention" href="/u/greek_freak">@greek_freak</a> , I was having the exact same issue as you. I finally figured out a fix. Here’s the solution…</p>
<ol>
<li><strong>CUDA is backward compatibile</strong>:- meaning, frameworks built for an earlier version of CUDA (e.g., 12.1) can still run on GPUs and drivers that support a later version of CUDA (e.g., 12.6). In your case:</li>
</ol>
<ul>
<li>Your <strong>GPU driver</strong> (<code>560....</code>) supports <strong>CUDA 12.6</strong>.</li>
<li>PyTorch officially supports <strong>CUDA 12.1</strong> as the latest compatible version, which is backward-compatible with your setup.</li>
</ul>
<ol start="2">
<li>Libraries like PyTorch with <strong>CUDA 12.1 support</strong> execute on systems with CUDA 12.6 because the newer driver includes support for all functionality in earlier CUDA versions (12.4, 12.1, 11.8).</li>
<li>Since PyTorch has been extensively tested with CUDA 12.1, I recommend CUDA 12.1 for better stability and performance. When PyTorch uses CUDA 12.1, it relies on the GPU driver to provide necessary APIs and kernel functions. Your CUDA 12.6-enabled driver fully supports these, so no additional configuration is needed. To install see this link: <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">Start Locally | PyTorch</a></li>
</ol>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Wessi" data-post="11" data-topic="209649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/5f9b8f/48.png" class="avatar"> Wessi:</div>
<blockquote>
<p>I recommend CUDA 12.1 for better stability and performance.</p>
</blockquote>
</aside>
<p>Could you please give examples of a better stability and performance in our CUDA 12.1 builds against e.g. 12.4 or 12.6?</p>

username: Wessi
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="12" data-topic="209649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you please give examples of a better stability and performance in our CUDA 12.1 builds against e.g. 12.4 or 12.6?</p>
</blockquote>
</aside>
<p>Available discussions and reports provide insights into their relative performance and stability. Here are some:</p>
<ol>
<li>A user compared PyTorch 2.4.0 with CUDA 12.4 against PyTorch 2.1.2 with CUDA 12.1 on an NVIDIA 4080 GPU. The findings indicated that the older version (PyTorch 2.1.2 with CUDA 12.1) was faster. source: <a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/16394" class="inline-onebox" rel="noopener nofollow ugc">We could update to last Torch &amp; Xformers? · AUTOMATIC1111/stable-diffusion-webui · Discussion #16394 · GitHub</a></li>
<li>During the integration of CUDA 12.4, unexpected errors were encountered in PyTorch’s Inductor backend, leading to performance regressions. source: <a href="https://github.com/pytorch/pytorch/issues/126692" class="inline-onebox" rel="noopener nofollow ugc">CUDA 12.4 CI Inductor Issues · Issue #126692 · pytorch/pytorch · GitHub</a><br>
etc… CUDA 12.4 and 12.6 introduce exclusive features and optimizations not available in 12.1 but PyTorch developers have extensively tested CUDA 12.1 more than 12.4 and 12.6, making 12.1 more stable and reliable.</li>
</ol>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Wessi" data-post="13" data-topic="209649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/5f9b8f/48.png" class="avatar"> Wessi:</div>
<blockquote>
<p>The findings indicated that the older version (PyTorch 2.1.2 with CUDA 12.1) was faster. source: <a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/16394">We could update to last Torch &amp; Xformers? · AUTOMATIC1111/stable-diffusion-webui · Discussion #16394 · GitHub</a></p>
</blockquote>
</aside>
<p>I see the opposite:</p>
<blockquote>
<p>I have installed PyTorch 2.4.0 with CUDA 12.4. On an RTX 4080, SD1.5 is about 15% to 20% faster, and SDXL is about 10% faster.</p>
</blockquote>
<aside class="quote no-group" data-username="Wessi" data-post="13" data-topic="209649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/5f9b8f/48.png" class="avatar"> Wessi:</div>
<blockquote>
<p>During the integration of CUDA 12.4, unexpected errors were encountered in PyTorch’s Inductor backend, leading to performance regressions.</p>
</blockquote>
</aside>
<p>I also see all issues were closed.</p>
<aside class="quote no-group" data-username="Wessi" data-post="13" data-topic="209649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/5f9b8f/48.png" class="avatar"> Wessi:</div>
<blockquote>
<p>making 12.1 more stable and reliable.</p>
</blockquote>
</aside>
<p>I doubt it and recommend using the latest release.<br>
E.g. CUDA 12.6U3 improved the <code>matmul</code> performance significantly and I would recommend sticking to this release.<br>
If issues are seen, please let us know.</p>

username: Wessi
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="14" data-topic="209649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>E.g. CUDA 12.6U3 improved the <code>matmul</code> performance significantly and I would recommend sticking to this release.<br>
If issues are seen, please let us know.</p>
</blockquote>
</aside>
<p>Personally, I haven’t experimented much with newer CUDA versions. From my research on web and community platforms, I found CUDA 12.1 to be the most stable and well-supported by PyTorch Developers, so I chose it. I’ll try newer versions if needed. Thanks!</p>