BackendCompilerFailed: _compile_fn raised RuntimeError: Triton requires CUDA 11.4+

weberxie: <p>Installed pytorch-nightly follow the command:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch-nightly -c nvidia
</code></pre>
<p>then tried the example of <a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html#demonstrating-speedups" class="inline-onebox" rel="noopener nofollow ugc">torch.compile Tutorial — PyTorch Tutorials 1.13.0+cu117 documentation</a> ,</p>
<p>finally it throwed the exception:</p>
<pre><code class="lang-auto">File /usr/local/conda/lib/python3.9/site-packages/torch/_dynamo/output_graph.py:591, in OutputGraph.call_user_compiler(self, gm)
    589 except Exception as e:
    590     compiled_fn = gm.forward
--&gt; 591     raise BackendCompilerFailed(self.compiler_fn, e) from e
    592 return compiled_fn

BackendCompilerFailed: _compile_fn raised RuntimeError: Triton requires CUDA 11.4+
</code></pre>
<p>As far as I know, conda had install the cuda-toolkit 11.6, so why does Pytorch throw this exception?</p>
<p>Any response will be appreciated.</p>

ptrblck: <p>Could you check the used CUDA runtime via <code>print(torch.version.cuda)</code>?</p>

weberxie: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6c596cebfd93765a290b89f2da8140ee5bef589.png" data-download-href="https://discuss.pytorch.org/uploads/default/b6c596cebfd93765a290b89f2da8140ee5bef589" title="Screen Shot 2022-12-09 at 15.39.13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6c596cebfd93765a290b89f2da8140ee5bef589_2_690x164.png" alt="Screen Shot 2022-12-09 at 15.39.13" data-base62-sha1="q4SeoXt5quVUMwEX4l1Igm0eV7z" width="690" height="164" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6c596cebfd93765a290b89f2da8140ee5bef589_2_690x164.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6c596cebfd93765a290b89f2da8140ee5bef589_2_1035x246.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6c596cebfd93765a290b89f2da8140ee5bef589_2_1380x328.png 2x" data-dominant-color="F4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-12-09 at 15.39.13</span><span class="informations">1892×450 68.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It’s 11.6, thanks for your response.</p>

ptrblck: <p>Could you post the code snippet raising this error, please, or are you directly executing the tutorial?</p>

weberxie: <p>Yes, I’m executing the tutorial, the related code snippet is:</p>
<pre><code class="lang-auto">class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = torch.nn.Linear(100, 10)
    
    def forward(self, x):
        return torch.nn.functional.relu(self.lin(x))

mod = MyModule()
opt_mod = torch.compile(mod)

print(opt_mod(torch.randn(10, 100)))

def timed(fn):
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    result = fn()
    end.record()
    torch.cuda.synchronize()
    return result, start.elapsed_time(end) / 1000

def generate_data(b):
    return (
        torch.randn(b, 3, 128, 128).to(torch.float32).cuda(),
        torch.randint(1000, (b,)).cuda(),
    )

N_ITERS = 10

from torchvision.models import resnet18
def init_model():
    return resnet18().to(torch.float32).cuda()

def evaluate(mod, inp):
    return mod(inp)

model = init_model()
evaluate_opt = torch.compile(evaluate, mode="reduce-overhead")

inp = generate_data(16)[0]
print("eager:", timed(lambda: evaluate(model, inp))[1])
print("compile:", timed(lambda: evaluate_opt(model, inp))[1])
</code></pre>

ptrblck: <p>Thanks for the code.<br>
I cannot reproduce it with a current nightly conda binary using 11.6.<br>
Env information:</p>
<pre><code class="lang-auto">python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.14.0.dev20221208
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.21.3
Libc version: glibc-2.31

Python version: 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:35)  [GCC 10.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-41-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 515.43.04
cuDNN version: Probably one of the following:
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.14.0.dev20221208
[pip3] torchvision==0.15.0.dev20221208
[conda] blas                      2.116                       mkl    conda-forge
[conda] blas-devel                3.9.0            16_linux64_mkl    conda-forge
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] mkl                       2022.1.0           h84fe81f_915    conda-forge
[conda] mkl-devel                 2022.1.0           ha770c72_916    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.23.5           py38h7042d01_0    conda-forge
[conda] pytorch                   1.14.0.dev20221208 py3.8_cuda11.6_cudnn8.3.2_0    pytorch-nightly
[conda] pytorch-cuda              11.6                 h867d48c_0    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchtriton               2.0.0+0d7e753227            py38    pytorch-nightly
[conda] torchvision               0.15.0.dev20221208      py38_cu116    pytorch-nightly
</code></pre>
<pre><code class="lang-auto">python -c "import torch; print(torch.__version__); print(torch.version.cuda)"
1.14.0.dev20221208
11.6
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">python main.py 
/opt/miniforge3/envs/nightly_conda_cuda116/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py:366: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled.Consider setting `torch.set_float32_matmul_precision('high')`
  warnings.warn(
tensor([[2.6126e-02, 5.6818e-01, 0.0000e+00, 2.6177e-01, 4.0566e-01, 0.0000e+00,
         5.5200e-01, 0.0000e+00, 0.0000e+00, 2.0061e-01],
        [0.0000e+00, 1.4974e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00, 2.4196e-01,
         0.0000e+00, 2.4392e-02, 9.4952e-01, 0.0000e+00],
        [4.0854e-02, 7.2992e-01, 1.7494e-01, 0.0000e+00, 3.9046e-01, 0.0000e+00,
         0.0000e+00, 1.1856e+00, 5.1254e-01, 1.4365e+00],
        [0.0000e+00, 1.3372e+00, 0.0000e+00, 6.2340e-01, 0.0000e+00, 4.8263e-01,
         3.6486e-01, 0.0000e+00, 1.4925e-01, 4.0236e-01],
        [0.0000e+00, 0.0000e+00, 2.3011e-01, 2.8612e-01, 0.0000e+00, 2.9270e-01,
         0.0000e+00, 0.0000e+00, 5.5580e-01, 0.0000e+00],
        [1.1411e+00, 0.0000e+00, 5.2030e-01, 1.0582e+00, 5.4400e-04, 6.6906e-01,
         0.0000e+00, 0.0000e+00, 0.0000e+00, 4.1712e-01],
        [2.8783e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00, 4.1535e-01, 0.0000e+00,
         0.0000e+00, 3.2157e-01, 2.4875e-01, 0.0000e+00],
        [8.6504e-01, 4.4471e-02, 5.2251e-01, 4.5288e-01, 0.0000e+00, 1.2464e+00,
         0.0000e+00, 0.0000e+00, 0.0000e+00, 5.8638e-01],
        [0.0000e+00, 7.3047e-01, 0.0000e+00, 0.0000e+00, 5.4288e-01, 9.1588e-01,
         0.0000e+00, 6.0390e-01, 2.5176e-01, 4.7328e-01],
        [0.0000e+00, 5.0342e-01, 1.2113e+00, 4.8887e-01, 0.0000e+00, 0.0000e+00,
         -0.0000e+00, -0.0000e+00, -0.0000e+00, 6.7853e-02]],
       grad_fn=&lt;CompiledFunctionBackward&gt;)
/opt/miniforge3/envs/nightly_conda_cuda116/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py:366: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled.Consider setting `torch.set_float32_matmul_precision('high')`
  warnings.warn(
eager: 1.265844482421875
compile: 4.382791015625
</code></pre>
<p>I’m not 100% sure, but would guess the Triton backend might need to use your locally installed CUDA toolkit, which might be older. Could this be the case?</p>

weberxie: <p>My PyTorch version is 1.14.0.dev20221207.</p>
<p>Yes, I also doubt that Triton backend may use the locally installed CUDA, it’s 11.4.</p>
<p>So, should Triton use the locally installed CUDA? Thank you.</p>

ptrblck: <p>I don’t know how exactly Inductor is calling into Triton, so would need to check it by reading through the code. Will let you know once I’ve figured it out.</p>
<p>EDIT: Yes, it seems to use the locally installed <code>nvcc</code> as seen <a href="https://github.com/pytorch/pytorch/blob/0c3537a3c3703dcfd58ba16fc1abcea5f788e26f/tools/dynamo/verify_dynamo.py#L33-L84">here</a>.</p>

weberxie: <p>Thanks, I have fixed this problem by upgrading the locally installed CUDA version to 11.6 .</p>
<p>So, the cause is as you said, the Triton backend uses the CUDA installed locally instead of the CUDA of conda.</p>

ptrblck: <p>After a quick sync with <a class="mention" href="/u/malfet">@malfet</a> he correctly pointed out that the check seems to fail in Triton directly <a href="https://github.com/openai/triton/blob/8650b4d1cbc750d659156e2c17a058736614827b/lib/driver/llvm.cc#L100">here</a> (not PyTorch as I’ve previously indicated).<br>
Based on the code your locally installed CUDA 11.4 should work, so I’ll try to reproduce the issue.</p>

ptrblck: <p>I still cannot reproduce the issue using a docker container with CUDA 11.4 and the latest PyTorch nightly release using the CUDA 11.6 runtime:</p>
<pre><code class="lang-python">root@1a038fc7e498:/workspace/src# python -c "import torch; print(torch.__version__); print(torch.version.cuda)"
2.0.0.dev20221209
11.6
root@1a038fc7e498:/workspace/src# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__2_19:15:15_PDT_2021
Cuda compilation tools, release 11.4, V11.4.48
Build cuda_11.4.r11.4/compiler.30033411_0
root@1a038fc7e498:/workspace/src# ptxas --version
ptxas: NVIDIA (R) Ptx optimizing assembler
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__2_19:14:30_PDT_2021
Cuda compilation tools, release 11.4, V11.4.48
Build cuda_11.4.r11.4/compiler.30033411_0
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">root@1a038fc7e498:/workspace/src# python tmp.py 
/opt/conda/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py:366: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled.Consider setting `torch.set_float32_matmul_precision('high')`
  warnings.warn(
tensor([[0.0000, 0.6264, 0.2043, 0.0000, 0.1676, 0.0000, 0.6097, 0.1066, 0.8841,
         0.0784],
        [0.0019, 0.0000, 0.0000, 0.0000, 0.3927, 0.0000, 0.0111, 0.2716, 0.6420,
         0.1461],
        [0.0000, 0.0267, 0.4337, 0.2366, 0.6798, 0.0000, 0.0865, 0.5707, 1.7802,
         0.0000],
        [0.0000, 0.0000, 0.0727, 0.0000, 0.0799, 0.0000, 0.0000, 0.0000, 0.0000,
         0.3500],
        [0.0000, 0.2702, 0.1387, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.7817],
        [0.0000, 0.5054, 0.3315, 1.0923, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000],
        [0.0000, 0.1780, 0.0540, 0.9178, 0.0000, 0.9666, 0.0000, 0.2082, 0.0000,
         1.1234],
        [0.0000, 0.0000, 0.2635, 0.2844, 0.2909, 0.2796, 0.0658, 0.0000, 0.0000,
         0.0000],
        [0.3576, 0.0000, 0.6487, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3771,
         0.1077],
        [0.1274, 0.8099, 0.0000, 0.9640, 0.0289, 0.4793, 0.9060, -0.0000, 0.1720,
         -0.0000]], grad_fn=&lt;CompiledFunctionBackward&gt;)
/opt/conda/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py:366: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled.Consider setting `torch.set_float32_matmul_precision('high')`
  warnings.warn(
eager: 1.9353865966796875
compile: 6.1589453125
</code></pre>
<p>so also your setup should work assuming your locally installed CUDA 11.4 is used.</p>

mslyon: <p>Just confirming that I face the same issue.</p>
<p>I have CUDA 11.7 installed via conda, with <code>torch==2.0.0.dev20221225+cu117</code>, but CUDA 11.1 linked outside of conda (and indeed <code>nvcc</code> uses this version). This issue is resolved by installing CUDA 11.7 outside of conda.</p>

carbocation: <p>I also have the same issue on GCP’s Vertex AI (in this case, with CUDA 11.6).</p>

3co: <p>I faced this issue.</p>
<p>I resolved it after installing CUDA 12.0, here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/cuda-downloads">
  <header class="source">
      <img src="https://developer.nvidia.com/sites/all/themes/cuda/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/cuda-downloads" target="_blank" rel="noopener nofollow ugc" title="08:45AM - 15 April 2021">NVIDIA Developer – 15 Apr 21</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://developer-blogs.nvidia.com/wp-content/uploads/2022/04/cuda-image-16x9-1.jpg" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://developer.nvidia.com/cuda-downloads" target="_blank" rel="noopener nofollow ugc">CUDA Toolkit 11.7 Downloads</a></h3>

  <p>Get the latest feature updates to NVIDIA's proprietary compute stack.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Looks like CUDA is now open source, so the process was considerably less painful on a cloud machine than it’s ever been.</p>

TheDudeJohan: <p>I’m facing the same issue, but just with the cuda 11.7 version. Unfortunately, i do not have rights to upgrade the local CUDA installation outside conda. Has anyone found another solution to this issue?</p>

Parskatt: <p>Installing<br>
<code>conda install -c "nvidia/label/cuda-11.7.0" cuda-nvcc</code><br>
seems to work for me. (also probably works for all 11.X and 12.0)</p>
<p><a class="mention" href="/u/thedudejohan">@TheDudeJohan</a> Did you try the above?</p>

JonathanSum: <p>I am using paperspace. I can’t use conda install -c “nvidia/label/cuda-11.7.0” cuda-nvcc to solve the problem.</p>

JonathanSum: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/2891efc1f3d1bba249b723593dfaa73fa00ea8f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/2891efc1f3d1bba249b723593dfaa73fa00ea8f0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/2891efc1f3d1bba249b723593dfaa73fa00ea8f0.png" alt="image" data-base62-sha1="5MTLwcpoRxgLgtu6FCzsUJvRZhS" width="609" height="500" data-dominant-color="3E3E3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">932×764 40 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This problem is Pytorch or any Meta frameworks and libraries have so relied on open source, and when the open source library breaks, everything breaks.</p>
<p>Reproduced Issue in Colab:<br>
(<a href="https://colab.research.google.com/drive/1_5a4_0wXQ8xzj7S1vkkNWtQikiovQDrH?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a>)<br>
Any suggestions?</p>

Puneesh_Khanna: <p><a class="mention" href="/u/jonathansum">@JonathanSum</a>  - I m also facing exact same issue on Colab.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7c2a7d42dea45911a1d411038e984526d782db2.png" alt="image" data-base62-sha1="uMHHDqIG2w2eC3MKfKAKwJwsI26" width="317" height="96"></p>

JonathanSum: <p>I don’t think we will have anyone come here to help us out. We should let the Triton team knows. Can you pin them, so they can come here?</p>