Strange "IndexError" when compiling

vgoklani: <p>While waiting for this to get resolved:</p>
<aside class="quote" data-post="2" data-topic="172958">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-cannot-find-ptxas/172958/2">RuntimeError: Cannot find ptxas</a> 
  </div>
  <blockquote>
    Our latest rebuild as of yesterday should have fixed it, but let me double check it. 
EDIT: Something still seems to be broken in the latest build. We are tracking the issue <a href="https://github.com/pytorch/builder/issues/1318" rel="noopener nofollow ugc">here</a>.
  </blockquote>
</aside>

<p>I stumbled on this:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/95082#issuecomment-1438958746">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/95082#issuecomment-1438958746" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/95082#issuecomment-1438958746" target="_blank" rel="noopener nofollow ugc">PyTorch 2.0 nightly: `compile` does not support capabilities sm_89 (RTX 4090 and others)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2023-02-17" data-time="18:55:55" data-timezone="UTC">06:55PM - 17 Feb 23 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2023-02-22" data-time="04:05:07" data-timezone="UTC">04:05AM - 22 Feb 23 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/pcuenca" target="_blank" rel="noopener nofollow ugc">
          <img alt="pcuenca" src="https://avatars.githubusercontent.com/u/1177582?v=4" class="onebox-avatar-inline" width="20" height="20">
          pcuenca
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: pt2
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: inductor
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

Trying the nightlies today (20230217) on RTX 4090 I get <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">the following error when attempting to use `compile()`:

```
RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/filelSnH8K, line 6; error   : PTX .version 7.4 does not support .target sm_89
ptxas fatal   : Ptx assembly aborted due to errors
```

As mentioned in #95081, the resolved nightly is `2.0.0.dev20230213+cu118`.

Repeating the same test with the cuda 11.7 wheels, the behavior is identical.

### Error logs

concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 549, in _worker_compile
    kernel.precompile(warm_cache_only_with_cc=cc)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/triton_ops/autotune.py", line 69, in precompile
    self.launchers = [
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/triton_ops/autotune.py", line 70, in &lt;listcomp&gt;
    self._precompile_config(c, warm_cache_only_with_cc)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/triton_ops/autotune.py", line 83, in _precompile_config
    triton.compile(
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/triton/compiler.py", line 1256, in compile
    asm, shared, kernel_name = _compile(fn, signature, device, constants, configs[0], num_warps, num_stages,
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/triton/compiler.py", line 901, in _compile
    name, asm, shared_mem = _triton.code_gen.compile_ttir(backend, module, device, num_warps, num_stages, extern_libs, cc)
RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/filelSnH8K, line 6; error   : PTX .version 7.4 does not support .target sm_89
ptxas fatal   : Ptx assembly aborted due to errors

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 670, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.fake_example_inputs())
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/debug_utils.py", line 1055, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/__init__.py", line 1374, in __call__
    return self.compile_fn(model_, inputs_, config_patches=self.config)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 453, in compile_fx
    return aot_autograd(
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 2481, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 2178, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 1412, in aot_wrapper_dedupe
    return compiler_fn(flat_fn, leaf_flat_args, aot_config)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 1062, in aot_dispatch_base
    compiled_fw = aot_config.fw_compiler(fw_module, flat_args)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 428, in fw_compiler
    return inner_compile(
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/debug_utils.py", line 595, in debug_wrapper
    compiled_fn = compiler_fn(gm, example_inputs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/debug.py", line 239, in inner
    return fn(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 177, in compile_fx_inner
    compiled_fn = graph.compile_to_fn()
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/graph.py", line 586, in compile_to_fn
    return self.compile_to_module().call
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/graph.py", line 575, in compile_to_module
    mod = PyCodeCache.load(code)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 528, in load
    exec(code, mod.__dict__, mod.__dict__)
  File "/tmp/torchinductor_pedro/b4/cb4wgmozs4bqzsoemk2mnsij7oypadaj3znkbzxbgpyxre3st3cy.py", line 104, in &lt;module&gt;
    async_compile.wait(globals())
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 715, in wait
    scope[key] = result.result()
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 573, in result
    self.future.result()
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/filelSnH8K, line 6; error   : PTX .version 7.4 does not support .target sm_89
ptxas fatal   : Ptx assembly aborted due to errors


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pedro/code/bench/benchmark_diffusers_pt2.py", line 47, in &lt;module&gt;
    do_tests(bs)
  File "/home/pedro/code/bench/benchmark_diffusers_pt2.py", line 38, in do_tests
    pipe(prompt, num_inference_steps=steps, num_images_per_prompt=batch_size).images
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/pedro/code/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 643, in __call__
    noise_pred = self.unet(
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
    return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "/home/pedro/code/diffusers/src/diffusers/models/unet_2d_condition.py", line 554, in forward
    t_emb = self.time_proj(timesteps)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 330, in catch_errors
    return callback(frame, cache_size, hooks)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
    return fn(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
    return _compile(
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 376, in transform_code_object
    transformations(instructions, code_options)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1681, in run
    super().run()
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 569, in run
    and self.step()
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 532, in step
    getattr(self, inst.opname)(inst)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1747, in RETURN_VALUE
    self.output.compile_subgraph(
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 517, in compile_subgraph
    self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/filelSnH8K, line 6; error   : PTX .version 7.4 does not support .target sm_89
ptxas fatal   : Ptx assembly aborted due to errors



You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True

### Minified repro

_No response_

### Versions

PyTorch version: 2.0.0.dev20230213+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.9.16 (main, Jan 11 2023, 16:05:54)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 4090
GPU 1: NVIDIA GeForce RTX 4090

Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          32
On-line CPU(s) list:             0-31
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen 9 7950X 16-Core Processor
CPU family:                      25
Model:                           97
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
Stepping:                        2
Frequency boost:                 enabled
CPU max MHz:                     5879.8818
CPU min MHz:                     3000.0000
BogoMIPS:                        9000.38
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                  AMD-V
L1d cache:                       512 KiB (16 instances)
L1i cache:                       512 KiB (16 instances)
L2 cache:                        16 MiB (16 instances)
L3 cache:                        64 MiB (2 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-31
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-triton==2.0.0+0d7e753227
[pip3] torch==2.0.0.dev20230213+cu118
[pip3] torchvision==0.15.0.dev20230217+cu118
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] pytorch-triton            2.0.0+0d7e753227          pypi_0    pypi
[conda] torch                     2.0.0.dev20230213+cu118          pypi_0    pypi
[conda] torchvision               0.15.0.dev20230217+cu118          pypi_0    pypi

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre data-code-wrap="bash"><code class="lang-bash">pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu117
pip uninstall pytorch-triton 
pip install --no-deps triton==2.0.0.a2
TRITON_PTXAS_PATH=/usr/bin/ptxas python my_model_compiler.py
</code></pre>
<p>and everything started working <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>BUT then I ran into a strange issue here:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self, n_embd=768, bias=False):
        super().__init__()
        self.n_head = 6
        self.embd = nn.Embedding(50257, 768)
        self.c_attn = nn.Linear(in_features=n_embd, out_features=3 * n_embd, bias=bias)
        self.dropout = 0

    def forward(self, x):
        x = self.embd(x)
        (B, T, C) = x.size()
        q, k, v = self.c_attn(x).chunk(chunks=3, dim=-1)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        y = torch.nn.functional.scaled_dot_product_attention(
            query=q, key=k, value=v, attn_mask=None, dropout_p=self.dropout, is_causal=True
        )
        return y
</code></pre>
<p>Doing this forward pass works fine:</p>
<pre data-code-wrap="python"><code class="lang-python">model = Model().cuda()
batch_size = 2
x = torch.randint(0, 50257, (batch_size, 1024)).cuda()
y = model(x)
</code></pre>
<p>However, if I compile the model:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.compile(model)
_ = model(x)
</code></pre>
<p>I get an indexing error (see below for error message).</p>
<p>However, if I remove the “query/key/value” parameter names, then compiling works correctly</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self, n_embd=768, bias=False):
        super().__init__()
        self.n_head = 6
        self.embd = nn.Embedding(50257, 768)
        self.c_attn = nn.Linear(in_features=n_embd, out_features=3 * n_embd, bias=bias)
        self.dropout = 0

    def forward(self, x):
        x = self.embd(x)
        (B, T, C) = x.size()
        q, k, v = self.c_attn(x).chunk(chunks=3, dim=-1)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        y = torch.nn.functional.scaled_dot_product_attention(
            q, k, v, attn_mask=None, dropout_p=self.dropout, is_causal=True
        )
        return y
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">model = Model().cuda()
batch_size = 2
x = torch.randint(0, 50257, (batch_size, 1024)).cuda()
model = torch.compile(model)
_ = model(x)
y = model(x)
</code></pre>
<p>I’m running the torch nightly: ‘2.0.0.dev20230220+cu118’<br>
and triton==2.0.0.a2</p>
<p>Is this just a triton error or something else? Thanks!</p>
<pre data-code-wrap="python"><code class="lang-python">---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:324, in _compile(code, globals, locals, builtins, compiler_fn, one_graph, export, hooks, frame)
    323 try:
--&gt; 324     out_code = transform_code_object(code, transform)
    325     orig_code_map[out_code] = code

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py:445, in transform_code_object(code, transformations, safe)
    443 propagate_line_nums(instructions)
--&gt; 445 transformations(instructions, code_options)
    446 return clean_and_assemble_instructions(instructions, keys, code_options)[1]

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:311, in _compile.&lt;locals&gt;.transform(instructions, code_options)
    299 tracer = InstructionTranslator(
    300     instructions,
    301     code,
   (...)
    309     mutated_closure_cell_contents,
    310 )
--&gt; 311 tracer.run()
    312 output = tracer.output

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1738, in InstructionTranslator.run(self)
   1737 _step_logger()(logging.INFO, f"torchdynamo start tracing {self.f_code.co_name}")
-&gt; 1738 super().run()

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:588, in InstructionTranslatorBase.run(self)
    584 self.output.push_tx(self)
    585 while (
    586     self.instruction_pointer is not None
    587     and not self.output.should_exit
--&gt; 588     and self.step()
    589 ):
    590     pass

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:552, in InstructionTranslatorBase.step(self)
    551     unimplemented(f"missing: {inst.opname}")
--&gt; 552 getattr(self, inst.opname)(inst)
    554 return inst.opname != "RETURN_VALUE"

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:342, in break_graph_if_unsupported.&lt;locals&gt;.decorator.&lt;locals&gt;.wrapper(self, inst)
    341 try:
--&gt; 342     return inner_fn(self, inst)
    343 except Unsupported as excp:

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1026, in InstructionTranslatorBase.CALL_FUNCTION_KW(self, inst)
   1025 assert len(kwargs) == len(argnames)
-&gt; 1026 self.call_function(fn, args, kwargs)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:486, in InstructionTranslatorBase.call_function(self, fn, args, kwargs)
    485     raise AssertionError(f"Attempt to trace forbidden callable {inner_fn}")
--&gt; 486 self.push(fn.call_function(self, args, kwargs))

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py:484, in TorchVariable.call_function(self, tx, args, kwargs)
    481 if self.value == torch._C._nn.scaled_dot_product_attention:
    482     # See:[Note] SDPA_flash's meta function returns incorrect Philox seed and offset
    483     # in pytorch/torch/_meta_registrations.py
--&gt; 484     fake_query = args[0].as_proxy().node.meta["example_value"]
    485     fake_key = args[1].as_proxy().node.meta["example_value"]

IndexError: list index out of range

from user code:
   File "/tmp/ipykernel_24294/2653966783.py", line 19, in forward
    y = torch.nn.functional.scaled_dot_product_attention(

Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True


The above exception was the direct cause of the following exception:

InternalTorchDynamoError                  Traceback (most recent call last)
Cell In[5], line 2
      1 model = torch.compile(model)
----&gt; 2 _ = model(x)

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:82, in OptimizedModule.forward(self, *args, **kwargs)
     81 def forward(self, *args, **kwargs):
---&gt; 82     return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:209, in _TorchDynamoContext.__call__.&lt;locals&gt;._fn(*args, **kwargs)
    207 dynamic_ctx.__enter__()
    208 try:
--&gt; 209     return fn(*args, **kwargs)
    210 finally:
    211     set_eval_frame(prior)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:337, in catch_errors_wrapper.&lt;locals&gt;.catch_errors(frame, cache_size)
    334             return hijacked_callback(frame, cache_size, hooks)
    336 with compile_lock:
--&gt; 337     return callback(frame, cache_size, hooks)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:404, in convert_frame.&lt;locals&gt;._convert_frame(frame, cache_size, hooks)
    402 counters["frames"]["total"] += 1
    403 try:
--&gt; 404     result = inner_convert(frame, cache_size, hooks)
    405     counters["frames"]["ok"] += 1
    406     return result

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:104, in wrap_convert_context.&lt;locals&gt;._fn(*args, **kwargs)
    102 torch.fx.graph_module._forward_from_src = fx_forward_from_src_skip_result
    103 try:
--&gt; 104     return fn(*args, **kwargs)
    105 finally:
    106     torch._C._set_grad_enabled(prior_grad_mode)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:262, in convert_frame_assert.&lt;locals&gt;._convert_frame_assert(frame, cache_size, hooks)
    259 global initial_grad_state
    260 initial_grad_state = torch.is_grad_enabled()
--&gt; 262 return _compile(
    263     frame.f_code,
    264     frame.f_globals,
    265     frame.f_locals,
    266     frame.f_builtins,
    267     compiler_fn,
    268     one_graph,
    269     export,
    270     hooks,
    271     frame,
    272 )

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py:163, in dynamo_timed.&lt;locals&gt;.dynamo_timed_inner.&lt;locals&gt;.time_wrapper(*args, **kwargs)
    161     compilation_metrics[key] = []
    162 t0 = time.time()
--&gt; 163 r = func(*args, **kwargs)
    164 time_spent = time.time() - t0
    165 # print(f"Dynamo timer: key={key}, latency={latency:.2f} sec")

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:394, in _compile(code, globals, locals, builtins, compiler_fn, one_graph, export, hooks, frame)
    392 except Exception as e:
    393     exception_handler(e, code, frame)
--&gt; 394     raise InternalTorchDynamoError() from e

InternalTorchDynamoError: 
</code></pre>

ptrblck: <p>Thanks for reporting this issue! Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we could track and fix it, please?</p>

vgoklani: <p>Done, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/issues/95266" rel="noopener nofollow ugc"> Strange “IndexError” when compiling</a></li>
</ul>