Compiling older version of PyTorch (1.12) with newer cuda architectures

username: Arthur.Putnam
cooked: <p>Hello Everyone,</p>
<p>I am working on an older project that uses PyTorch version 1.12.0. This project works as expected on a Ubuntu system running NVIDIA-SMI version 450.80.02, Driver Version 450.80.02, and CUDA version 11.0. This system contains a Tesla V100 GPU.</p>
<p>I need to deploy this project to a newer machine that has NVIDIA-SMI 535.171.04, Driver Version 535.171.04, CUDA Version 12.2 and a NVIDIA RTX 6000 Ada GPU. When I deploy/run this project to this machine I receive the following error:</p>
<blockquote>
<p>CUDA error: no kernel image is available for execution on the device<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1</p>
</blockquote>
<p>Based on my research this is because the PyTorch version I am using doesn’t support the NVIDIA RTX 6000 Ada GPU architecture (sm_89).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/480e57bc63cf56d92451d4da3126cdaeb58f3506.png" alt="image (66)" data-base62-sha1="ahr6Dkd5fHIwICcR37PA7JGIV38" width="658" height="129"><br>
Upgrading to PyTorch 2.0+ is not trivial at the moment.</p>
<p><strong>Questions / Asks:</strong></p>
<ul>
<li>
<p>Is it possible to compile PyTorch Version 1.12.0 to work with newer cuda architectures? (i.e. sm_89, sm_90)</p>
</li>
<li>
<p>Any recommendations or steps I can take to compile this exact build (i.e. is there a docker container I can used to build a wheel, what args etc.)</p>
</li>
<li>
<p>Are there any existing builds that already have this PyTorch Version and CUDA architecture support?</p>
</li>
</ul>

username: ptrblck
cooked: <p>Install the PyTorch 1.12.0 binary with CUDA 11.6 runtime dependencies and I think it should work as your GPU should have been already supported back then.</p>
<p>sm_90 won’t work as it was introduced in CUDA 11.8 which wasn’t released when PyTorch 1.12 was released.</p>

username: Arthur.Putnam
cooked: <p>Thank you, I will give that a shot!</p>

username: Arthur.Putnam
cooked: <p>I’m not seeing <code>sm_89</code> as one of the supported architectures when using PyTorch 1.12.0 binary with CUDA 11.6. My understanding is that <code>sm_89</code> is required to work with the NVIDIA RTX 6000 Ada GPU. <a class="mention" href="/u/ptrblck">@ptrblck</a> can you confirm?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a809ab86e881f8e51558b98e5ab85a5eef48a08d.png" data-download-href="https://discuss.pytorch.org/uploads/default/a809ab86e881f8e51558b98e5ab85a5eef48a08d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a809ab86e881f8e51558b98e5ab85a5eef48a08d_2_689x365.png" alt="image" data-base62-sha1="nYwVQlFINs00agQvpEk4fwwmCkd" width="689" height="365" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a809ab86e881f8e51558b98e5ab85a5eef48a08d_2_689x365.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a809ab86e881f8e51558b98e5ab85a5eef48a08d_2_1033x547.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a809ab86e881f8e51558b98e5ab85a5eef48a08d.png 2x" data-dominant-color="282827"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1283×680 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>No, the architecture is not explicitly needed as it’s compatible to sm_86 and sm_80. You can just run any example to verify it assuming your system can communicate with your GPU.</p>

username: William_Zhang
cooked: <p>According to NGC PyTorch container image support matrix (<a href="https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html" class="inline-onebox" rel="noopener nofollow ugc">Frameworks Support Matrix - NVIDIA Docs</a>), the first/earliest version supporting Ada Lovelace is v22.10. This particular PyTorch container image supports</p>
<ul>
<li>
<p>PyTorch 3.13</p>
</li>
<li>
<p>CUDA runtime 11.8</p>
</li>
<li>
<p>Python 3.8.</p>
</li>
</ul>

username: William_Zhang
cooked: <p>Among the following variables:</p>
<ol>
<li>GPU device arch (e.g. RTX 6000 Ada) and compute capability (sm_89 for Ada)</li>
<li>GPU driver version and CUDA driver version</li>
<li>CUDA runtime version</li>
<li>PyTorch version</li>
</ol>
<p>The first 2 are not supposed to change in our context. If we can determine the minimum required CUDA runtime version (bullet 3), we can determine PyTorch version (bullet 4) accordingly. So what is the minimum/earliest CUDA runtime version which supports our GPU (RTX 6000)?</p>
<p>The following evidences seem to conclude that it is CUDA 11.8:</p>
<ul>
<li>
<p><a href="https://docs.nvidia.com/cuda/archive/11.8.0/cuda-toolkit-release-notes/index.html" rel="noopener nofollow ugc">Nvidia CUDA 11.8 release notes</a>: “<em>This release introduces support for both the Hopper and Ada Lovelace GPU families</em>.”</p>
</li>
<li>
<p><a href="https://en.m.wikipedia.org/wiki/CUDA#GPUs_supported" rel="noopener nofollow ugc">CUDA wiki</a>: The first CUDA version supporting Ada Lovelace is CUDA 11.8.</p>
</li>
<li>
<p>Nvidia Framework Support Matrix for NGC PyTorch containers: The first/earliest version of NGC PyTorch container supporting Ada Lovelace architecture is v22.10. In this container, PyTorch (v1.13) was built on CUDA 11.8 (on Ubuntu 20.04).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ee28d712ed34c2f4e5bee51adf07f6ec080cca2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5ee28d712ed34c2f4e5bee51adf07f6ec080cca2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ee28d712ed34c2f4e5bee51adf07f6ec080cca2.jpeg" alt="image" data-base62-sha1="dxogF0vEw73zVmyVSI3hpufmvg6" width="690" height="478" data-dominant-color="39142E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">902×625 64.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
</ul>
<p>Without resorting to compiling PyTorch from source, we have the following options:</p>
<ol>
<li>Use NGC PyTorch container 22.10 or newer</li>
<li>Use a PyTorch binary distribution: the minimum PyTorch binary version supporting CUDA 11.8 is PyTorch 2.0.0 (see the list below)</li>
</ol>
<p>1.13.0+cpu, 1.13.0+cu116, 1.13.0+cu117, 1.13.0+cu117.with.pypi.cudnn, 1.13.0+rocm5.1.1, 1.13.0+rocm5.2, 1.13.1, 1.13.1+cpu, 1.13.1+cu116, 1.13.1+cu117, 1.13.1+cu117.with.pypi.cudnn, 1.13.1+rocm5.1.1, 1.13.1+rocm5.2, 2.0.0, 2.0.0+cpu, 2.0.0+cpu.cxx11.abi, 2.0.0+cu117, 2.0.0+cu117.with.pypi.cudnn, 2.0.0+cu118, 2.0.0+rocm5.3, 2.0.0+rocm5.4.2, 2.0.1, 2.0.1+cpu, 2.0.1+cpu.cxx11.abi, 2.0.1+cu117, 2.0.1+cu117.with.pypi.cudnn, 2.0.1+cu118, 2.0.1+rocm5.3, 2.0.1+rocm5.4.2, 2.1.0, 2.1.0+cpu, 2.1.0+cpu.cxx11.abi, 2.1.0+cu118, 2.1.0+cu121, 2.1.0+cu121.with.pypi.cudnn, 2.1.0+rocm5.5, 2.1.0+rocm5.6, 2.1.1, 2.1.1+cpu, 2.1.1+cpu.cxx11.abi, 2.1.1+cu118, 2.1.1+cu121, 2.1.1+cu121.with.pypi.cudnn, 2.1.1+rocm5.5, 2.1.1+rocm5.6, 2.1.2, 2.1.2+cpu, 2.1.2+cpu.cxx11.abi, 2.1.2+cu118, 2.1.2+cu121, 2.1.2+cu121.with.pypi.cudnn, 2.1.2+rocm5.5, 2.1.2+rocm5.6, 2.2.0, 2.2.0+cpu, 2.2.0+cpu.cxx11.abi, 2.2.0+cu118, 2.2.0+cu121, 2.2.0+rocm5.6, 2.2.0+rocm5.7, 2.2.1, 2.2.1+cpu, 2.2.1+cpu.cxx11.abi, 2.2.1+cu118, 2.2.1+cu121, 2.2.1+rocm5.6, 2.2.1+rocm5.7, 2.2.2, 2.2.2+cpu, 2.2.2+cpu.cxx11.abi, 2.2.2+cu118, 2.2.2+cu121, 2.2.2+rocm5.6, 2.2.2+rocm5.7, 2.3.0, 2.3.0+cpu, 2.3.0+cpu.cxx11.abi, 2.3.0+cu118, 2.3.0+cu121, 2.3.0+rocm5.7, 2.3.0+rocm6.0, 2.3.1, 2.3.1+cpu, 2.3.1+cpu.cxx11.abi, 2.3.1+cu118, 2.3.1+cu121, 2.3.1+rocm5.7, 2.3.1+rocm6.0)</p>
<ol>
<li>Any flaw in the above analysis and conclusion?</li>
<li>Is there any other option short of compiling PyTorch from source?</li>
</ol>

username: William_Zhang
cooked: <p>Side note: after testing with various versions of PyTorch/CUDA, I realize:</p>
<p><code>torch.cuda.is_available = True</code> or successful run of <code>torch.rand(2,3).cuda()</code> does not imply the installed PyTorch fully supports the installed GPU. It only indicates the installed CUDA driver (on Docker host) is compatible with installed CUDA runtime (in container). Such CUDA driver to CUDA runtime compatibility can be easily met by any CUDA runtime version &lt;= (not newer than) CUDA driver version. For PyTorch to work with a given GPU device, PyTorch must have been compiled against a CUDA runtime version which supports the GPU device, or we will get the error as described <a href="https://github.com/pytorch/pytorch/issues/31285" rel="noopener nofollow ugc">here</a>. As indicated above, the minimum version of CUDA supporting Ada Lovelace GPU is CUDA 11.8.</p>
<p>Please let me know if this is incorrect. Thanks.</p>

username: ptrblck
cooked: <p>While your explanation is true that <code>sm_89</code> was introduces in CUDA 11.8, it won’t matter for the functionality as we are neither building PyTorch for this compute capability explicitly (you can check it via <code>torch.cuda.get_arch_list()</code>) nor should we as no performance improvements are expected.<br>
Since sm_89 is binary compatible to sm_86 and sm_80 (as explained above), you should be able to execute any PyTorch binary supporting sm_86/_80.<br>
CUDA math libs have certainly improved their heuristics etc. in newer CUDA versions, which is why I would not recommend to go too far back.</p>
<p>So the TL;DR is still: <a class="mention" href="/u/arthur.putnam">@Arthur.Putnam</a> should just run any simple example to verify that the build is working fine or just upgrade to any of our current binaries.</p>

username: Arthur.Putnam
cooked: <p>I was successfully able to get our application running on the new system using CUDA 11.7 which confirms <a class="mention" href="/u/ptrblck">@ptrblck</a>’s statement about it working with the sm_86 architecture. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/william_zhang">@William_Zhang</a> for talking through the requirements.</p>
<p>Incase others are coming here for similar issues. See details below.</p>
<p>To get this to work, I used the <a href="https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-05.html" rel="noopener nofollow ugc">NVIDIA container image for PyTorch version 22-05</a> which comes with Python 3.8, CUDA 11.7 runtime, and the same version of PyTorch we previously used (1.12.x). You can look for different combinations of PyTorch and CUDA in their container repository. I did how ever need to manually compile/upgrade some of our Python dependencies to use CUDA 11.7 instead of the CUDA 10.2 that they were compiled on/for. This required me to manually build/compile those libraries in/on the docker container.</p>