Cannot Find ptxas

Subzero: <p>I am trying to use torch.compile and I am receiving the error: RuntimeError: Cannot find ptxas</p>
<p>I am on pytorch version 2.1.2+cu121 and on linux.</p>
<p>Is there anything I need to check for? I am using triton 2.1.0 as that is what is pytorch warned when I tried to install triton 2.2.0</p>

ptrblck: <p>Could you uninstall <code>triton</code> assuming you have manually installed it and just install PyTorch via the provided install commands on our website? It will pull the right <code>triton</code> package with all needed dependencies.</p>

marksaroufim: <p>when you install pytorch it won’t install triton exactly, we will manually build it using a pin that’s in pytorch core</p>
<p>You can see how it works if you do <code>make triton</code> here <a href="https://github.com/pytorch/pytorch/blob/main/Makefile#L35">https://github.com/pytorch/pytorch/blob/main/Makefile#L35</a></p>

Subzero: <p>I originally had triton installed from installing pytorch from the install commands on the website. I had read older posts about the issues with ptxas and triton and it seemed that changing the version of triton could had solve the issue, but it did not for my case.</p>
<p>I uninstalled triton and then used the install commands from pytorch and it still did not work.</p>

Subzero: <p>Is there a guide on how to manually build it? Or can you please write out the steps needed to do so? I am not so good with using github and command line usage</p>

ptrblck: <p>Since when should this be the case?<br>
Installing the current stable <code>2.1.2</code> PyTorch release as well as the current nightly binary still downloads the <code>(pytorch)-triton</code> package (using the specified commit you are pointing to for nightlies):</p>
<pre data-code-wrap="python"><code class="lang-python">pip install torch
Collecting torch
  Downloading torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl.metadata (25 kB)
...
Downloading torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 5.0 MB/s eta 0:00:00
Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 4.9 MB/s eta 0:00:00
...

pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121
Collecting torch
  Downloading https://download.pytorch.org/whl/nightly/cu121/torch-2.3.0.dev20240120%2Bcu121-cp310-cp310-linux_x86_64.whl (768.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 768.4/768.4 MB 4.6 MB/s eta 0:00:00
...
Collecting pytorch-triton==2.2.0+e28a256d71 (from torch)
  Downloading https://download.pytorch.org/whl/nightly/pytorch_triton-2.2.0%2Be28a256d71-cp310-cp310-linux_x86_64.whl (183.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 183.1/183.1 MB 4.2 MB/s eta 0:00:00
...
Successfully installed nvidia-nccl-cu12-2.19.3 pytorch-triton-2.2.0+e28a256d71 torch-2.3.0.dev20240120+cu121
</code></pre>
<p>The S3 bucket also stores the wheels here for nightlies: <a href="https://download.pytorch.org/whl/nightly/pytorch-triton/">https://download.pytorch.org/whl/nightly/pytorch-triton/</a></p>
<p>PyTorch source builds could build <code>triton</code> from the commit if you execute the <a href="https://github.com/pytorch/pytorch/blob/2ae66ddba0f466406e577648799e095b35a4bb5a/.github/scripts/build_triton_wheel.py">build script</a>, but I doubt any <code>pip install</code> command triggers an LLVM download the the actual <code>triton</code> build.</p>
<p><a class="mention" href="/u/subzero">@Subzero</a> could you post the exact instructions you are using as I’m seeing <code>ptxas</code> in the installed <code>(pytorch)-triton</code> package using the mentioned install commands:</p>
<pre data-code-wrap="python"><code class="lang-python">/home/pbialecki/miniforge3/envs/tmp/lib/python3.10/site-packages/triton/third_party/cuda/bin/ptxas --version
ptxas: NVIDIA (R) Ptx optimizing assembler
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Sep__8_19:13:29_PDT_2023
Cuda compilation tools, release 12.3, V12.3.52
Build cuda_12.3.r12.3/compiler.33281558_0
</code></pre>

Subzero: <h2><a name="quick-conclusion-but-not-an-exact-solution-i-made-a-new-project-and-used-the-regular-command-install-like-before-and-there-is-no-error-now-something-about-the-original-project-excluding-the-code-as-the-code-works-was-causing-the-error-1" class="anchor" href="#quick-conclusion-but-not-an-exact-solution-i-made-a-new-project-and-used-the-regular-command-install-like-before-and-there-is-no-error-now-something-about-the-original-project-excluding-the-code-as-the-code-works-was-causing-the-error-1"></a>Quick conclusion but not an exact solution: I made a new project and used the regular command install like before and there is no error now. Something about the original project, excluding the code as the code works, was causing the error.</h2>
<p>Below are the exact instructions I used to install and uninstall.</p>
<pre><code class="lang-auto">pip uninstall triton
Found existing installation: triton 2.1.0
Uninstalling triton-2.1.0:
Successfully uninstalled triton-2.1.0

pip uninstall torch -y
Found existing installation: torch 2.1.2
Uninstalling torch-2.1.2:
Successfully uninstalled torch-2.1.2

pip install torch
Collecting torch
  Obtaining dependency information for torch from https://files.pythonhosted.org/packages/03/f1/13137340776dd5d5bcfd2574c9c6dfcc7618285035cd77240496e5c1a79b/torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl.metadata
  Using cached torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl.metadata (25 kB)

Collecting triton==2.1.0 (from torch)
  Obtaining dependency information for triton==2.1.0 from https://files.pythonhosted.org/packages/4d/22/91a8af421c8a8902dde76e6ef3db01b258af16c53d81e8c0d0dc13900a9e/triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
  Using cached triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)
Using cached torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)
Using cached triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
Installing collected packages: triton, torch
Successfully installed torch-2.1.2 triton-2.1.0


./.venv/lib/python3.10/site-packages/triton/third_party/cuda/bin/ptxas --version
ptxas: NVIDIA (R) Ptx optimizing assembler
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:13:45_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0

</code></pre>
<p>Doing this, I still receive the error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/user_n/PycharmProjects/Resnet18 Tests/main.py", line 371, in &lt;module&gt;
    model_ft = torch.compile(model_ft)
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/__init__.py", line 1723, in compile
    return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, disable=disable)(model)
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 610, in optimize
    compiler_config=backend.get_compiler_config()
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/__init__.py", line 1571, in get_compiler_config
    from torch._inductor.compile_fx import get_patched_config_dict
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 38, in &lt;module&gt;
    from .fx_passes.joint_graph import joint_graph_passes
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/fx_passes/joint_graph.py", line 8, in &lt;module&gt;
    from ..pattern_matcher import (
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/pattern_matcher.py", line 28, in &lt;module&gt;
    from .lowering import fallback_node_due_to_unsupported_type
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 4768, in &lt;module&gt;
    import_submodule(kernel)
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1492, in import_submodule
    importlib.import_module(f"{mod.__name__}.{filename[:-3]}")
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/kernel/bmm.py", line 4, in &lt;module&gt;
    from ..select_algorithm import (
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/select_algorithm.py", line 25, in &lt;module&gt;
    from .codegen.triton import texpr, TritonKernel, TritonPrinter, TritonScheduling
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/codegen/triton.py", line 26, in &lt;module&gt;
    from ..triton_heuristics import AutotuneHint
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 23, in &lt;module&gt;
    from .coordinate_descent_tuner import CoordescTuner
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/coordinate_descent_tuner.py", line 8, in &lt;module&gt;
    if has_triton():
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 83, in has_triton
    import triton
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/__init__.py", line 20, in &lt;module&gt;
    from .compiler import compile, CompilationError
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/compiler/__init__.py", line 1, in &lt;module&gt;
    from .compiler import CompiledKernel, compile, instance_descriptor
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/compiler/compiler.py", line 27, in &lt;module&gt;
    from .code_generator import ast_to_ttir
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 8, in &lt;module&gt;
    from .. import language
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/language/__init__.py", line 4, in &lt;module&gt;
    from . import math
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/language/math.py", line 4, in &lt;module&gt;
    from . import core
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/language/core.py", line 1376, in &lt;module&gt;
    def minimum(x, y):
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/runtime/jit.py", line 542, in jit
    return decorator(fn)
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/runtime/jit.py", line 534, in decorator
    return JITFunction(
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/runtime/jit.py", line 433, in __init__
    self.run = self._make_launcher()
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/runtime/jit.py", line 388, in _make_launcher
    scope = {"version_key": version_key(),
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/runtime/jit.py", line 120, in version_key
    ptxas = path_to_ptxas()[0]
  File "/home/user_n/PycharmProjects/Resnet18 Tests/.venv/lib/python3.10/site-packages/triton/common/backend.py", line 119, in path_to_ptxas
    raise RuntimeError("Cannot find ptxas")
RuntimeError: Cannot find ptxas
</code></pre>
<p>I tried running the two basic examples in  <a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html#introduction-to-torch-compile" rel="noopener nofollow ugc">INTRODUCTION TO TORCH.COMPILE</a> and both gave the same error.</p>
<p>Then I tried the two basic examples again but this time in a new fresh project and both examples worked.  For the new fresh project I install torch using:</p>
<pre><code class="lang-auto">pip install torch
Collecting torch
  Obtaining dependency information for torch from https://files.pythonhosted.org/packages/03/f1/13137340776dd5d5bcfd2574c9c6dfcc7618285035cd77240496e5c1a79b/torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl.metadata
  Using cached torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl.metadata (25 kB)
Collecting triton==2.1.0 (from torch)
  Obtaining dependency information for triton==2.1.0 from https://files.pythonhosted.org/packages/4d/22/91a8af421c8a8902dde76e6ef3db01b258af16c53d81e8c0d0dc13900a9e/triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
  Using cached triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)
Using cached torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)
Using cached triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
Successfully installed torch-2.1.2 triton-2.1.0

./.venv/lib/python3.10/site-packages/triton/third_party/cuda/bin/ptxas --version
ptxas: NVIDIA (R) Ptx optimizing assembler
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:13:45_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
</code></pre>
<p>One thing I noticed differently with the new fresh project is:</p>
<pre><code class="lang-auto">Obtaining dependency information for MarkupSafe&gt;=2.0 from https://files.pythonhosted.org/packages/36/2a/fab302636634e1f770a26aac212e44cff25522ed3c9189bd8afc9ae2effd/MarkupSafe-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading MarkupSafe-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Downloading MarkupSafe-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
</code></pre>
<p>While in my original project MarkupSafe used a cache version:</p>
<pre><code class="lang-auto">Requirement already satisfied: MarkupSafe&gt;=2.0 in ./.venv/lib/python3.10/site-packages (from jinja2-&gt;torch) (2.1.3)
</code></pre>
<p>Could the issue have been a slightly out of date version of MarkupSafe? The answer is no:</p>
<pre><code class="lang-auto"> pip uninstall MarkupSafe
Found existing installation: MarkupSafe 2.1.3
Uninstalling MarkupSafe-2.1.3:
  Successfully uninstalled MarkupSafe-2.1.3

pip install torch
Collecting MarkupSafe&gt;=2.0 (from jinja2-&gt;torch)
  Obtaining dependency information for MarkupSafe&gt;=2.0 from https://files.pythonhosted.org/packages/36/2a/fab302636634e1f770a26aac212e44cff25522ed3c9189bd8afc9ae2effd/MarkupSafe-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached MarkupSafe-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Using cached MarkupSafe-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Installing collected packages: MarkupSafe
Successfully installed MarkupSafe-2.1.4

</code></pre>
<p>The error persists.</p>
<pre><code class="lang-auto">raise RuntimeError("Cannot find ptxas")
RuntimeError: Cannot find ptxas
</code></pre>
<p>Moving my original code to the new fresh project gave no errors. So I believe that something in my original project was causing the issue somehow.</p>