CUDA error: the provided PTX was compiled with an unsupported toolchain [in Nvidia Ampere]

Ardian_Umam: <p>Hi,</p>
<p>I tried to install <a href="https://github.com/erikwijmans/Pointnet2_PyTorch" rel="noopener nofollow ugc">pointnet2</a> (a major architecture for point cloud data) and the installation went well without any error. However, when I try to import the operation via <code>from pointnet2_ops import pointnet2_utils</code>, I get this error:<br>
<code>CUDA error: the provided PTX was compiled with an unsupported toolchain.</code></p>
<p>Based on some searching, the error is because the cuda version used by pytorch is newer than the driver version installed on the machine. But, that’s not in my case. Here is my setup:</p>
<ol>
<li>Cuda version in the Ubuntu machine: 11.3</li>
<li>Cuda version when install pytorch: 11.1</li>
<li>Pytorch: version 1.9.0</li>
<li>Python: version 3.9</li>
</ol>
<p>This error happens in GPU Nvidia A5000 (ampere arch). When I install with the exactly same setting as above, but in my machine with Nvidia 2080 Ti, everything goes well. No any error was encountered. Is there any direction how to solve this issue in my nvidia A5000 machine?</p>
<p>Detailed error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/aumam/dev/multimodal_distillation/model/model.py", line 457, in &lt;module&gt;
    net = net.cuda()
  File "/home/aumam/anaconda3/envs/pytorch1.9_4/lib/python3.9/site-packages/torch/nn/modules/module.py", line 637, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/aumam/anaconda3/envs/pytorch1.9_4/lib/python3.9/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/home/aumam/anaconda3/envs/pytorch1.9_4/lib/python3.9/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/home/aumam/anaconda3/envs/pytorch1.9_4/lib/python3.9/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/home/aumam/anaconda3/envs/pytorch1.9_4/lib/python3.9/site-packages/torch/nn/modules/module.py", line 552, in _apply
    param_applied = fn(param)
  File "/home/aumam/anaconda3/envs/pytorch1.9_4/lib/python3.9/site-packages/torch/nn/modules/module.py", line 637, in &lt;lambda&gt;
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>Many thanks</p>

ptrblck: <p>It seems Ampere GPUs are not supported since <code>TORCH_CUDA_ARCH_LIST</code> is hardcoded to GPUs up to <code>sm_75</code> <a href="https://github.com/erikwijmans/Pointnet2_PyTorch/blob/b5ceb6d9ca0467ea34beb81023f96ee82228f626/pointnet2_ops_lib/setup.py#L19">here</a>.<br>
In any case, you’ve already <a href="https://github.com/erikwijmans/Pointnet2_PyTorch/issues/175">cross-posted the question</a> so I would expect the authors to know more about the limitations.</p>

Ardian_Umam: <p>Thanks much for pointing that out! I added “8.6” in the TORCH_CUDA_ARCH_LIST and it solved the issue.</p>

04RR: <p>Hey, I’m facing the same issue but I am using a RTX 4090 card. I installed the cuda toolkit version 12.0 from  <a href="https://developer.nvidia.com/cuda-12-0-0-download-archive?target_os=Windows&amp;target_arch=x86_64&amp;target_version=11&amp;target_type=exe_local" rel="noopener nofollow ugc">here</a>.</p>
<p>I have pytorch2.0 with pytorch-cuda=11.8 and python 3.11.4</p>
<p>PS: I just got this card and I’m in the process of setting it up, please let me know if I should install any additional drivers.</p>

ptrblck: <p>I guess you are seeing this error from OpenAI/Triton? If so, then note that they ship <code>ptxas</code> from CUDA 12 and you might need to driver update as described <a href="https://github.com/pytorch/pytorch/issues/106144">here</a>.</p>

04RR: <p>Hey, I got this error while trying to load a gguf model using <a href="https://github.com/marella/ctransformers" rel="noopener nofollow ugc">ctranformers</a></p>
<p><code>llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-GGML", gpu_layers=50)</code></p>

ptrblck: <p>Do you know if your call uses <code>torch.compile</code> or anything from OpenAI/Triton under the hood?</p>

thanh_phat_ph_m: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I have a trouble with this problem.<br>
If I run the file <strong>init</strong>.py directly, my tensor can be transformed to CUDA, but if I run another script (and this script import and call the file <strong>init</strong>.py), it has the problem.<br>
Can you help me for this problem ? Thanks for your help<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50d4fe9488fc12ab961799dc8ed957a2f1c70280.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/50d4fe9488fc12ab961799dc8ed957a2f1c70280" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50d4fe9488fc12ab961799dc8ed957a2f1c70280_2_666x500.jpeg" alt="image" data-base62-sha1="bx4xeXiOitZZYdi80JSqdYwIqQw" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50d4fe9488fc12ab961799dc8ed957a2f1c70280_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50d4fe9488fc12ab961799dc8ed957a2f1c70280_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/50d4fe9488fc12ab961799dc8ed957a2f1c70280.jpeg 2x" data-dominant-color="212020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1063×797 177 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>P/s : nvidia-smi and nvcc --version<br>
NVIDIA-SMI 460.80       Driver Version: 460.80       CUDA Version: 11.2</p>
<p>Copyright (c) 2005-2022 NVIDIA Corporation<br>
Built on Wed_Sep_21_10:33:58_PDT_2022<br>
Cuda compilation tools, release 11.8, V11.8.89<br>
Build cuda_11.8.r11.8/compiler.31833905_0</p>

ptrblck: <p>Try to clear the caches for jit’ted kernels as well as kernels created by Triton.</p>

thanh_phat_ph_m: <p>Thanks for your reply, I try to uninstall triton, and clear the caches for jit’ted kernels but it still error</p>
<p>import torch<br>
torch._C._jit_clear_class_registry() # Clear the JIT class registry and pass managers</p>
<p>Can you tell me more the detail solutions for this problem ? Thank you for your help.</p>

ptrblck: <p>I haven’t encountered the issue myself, so unsure what exactly is causing it.<br>
Based on your config I guess it could be related to OAI/Triton shipping with <code>ptxas</code> from CUDA 12.x while your driver supports CUDA 11.x as described in <a href="https://github.com/pytorch/pytorch/issues/119054#issuecomment-1924732428">this issue</a>. You could try to apply the workaround of specifying <code>TRITION_PTXAS_PATH</code> pointing to a <code>ptxas</code> from CUDA 11.</p>

thanh_phat_ph_m: <p>Thanks for your reply, I solved this problem, the reason is that the nvidia-smi version and nvcc --version mismatch together, so when I built submodules, it cannot work to transform tensor to cuda.<br>
Thanks for your help.</p>