Problem with Hugging Face model that is not in training loop
Labels: triaged, oncall: pt2, module: cpu inductor

### 🐛 Describe the bug

I use the hugging face transformers and the DistilBert specifically. However, torch.compile fails because of it. The code associated with the bug:
```
from typing import List, Union
import pytorch_lightning as pl

import torch.nn as nn
import os

import torch
from torch import Tensor
from torch.distributions.distribution import Distribution
import hydra


class DistilbertEncoderBase(pl.LightningModule):
    def __init__(self, modelpath: str, finetune: bool = False) -> None:
        super().__init__()

        from transformers import AutoTokenizer, AutoModel
        from transformers import logging
        logging.set_verbosity_error()
        # Tokenizer
        os.environ["TOKENIZERS_PARALLELISM"] = "false"
        # workaround to work from cluster and local
        rel_p = modelpath.split('/')
        rel_p = rel_p[rel_p.index('deps'):]
        rel_p = '/'.join(rel_p)
        modelpath = hydra.utils.get_original_cwd() + '/' + rel_p

        self.tokenizer = AutoTokenizer.from_pretrained(modelpath)

        # Text model
        self.text_model = AutoModel.from_pretrained(modelpath)
        # Don't train the model
        if not finetune:
            self.text_model.training = False
            # for p in self.text_model.parameters():
            #     p.requires_grad = False

        # Then configure the model
        self.text_encoded_dim = self.text_model.config.dim
        self.finetune = finetune

    def train(self, mode: bool = True):
        self.training = mode
        for module in self.children():
            # Don't put the model in
            if module == self.text_model and not self.finetune:
                continue
            module.train(mode)
        return self

    def get_last_hidden_state(
            self,
            texts: List[str],
            return_mask: bool = False) -> Union[Tensor, tuple[Tensor, Tensor]]:
        encoded_inputs = self.tokenizer(texts,
                                        return_tensors="pt",
                                        padding=True)
        output = self.text_model(**encoded_inputs.to(self.text_model.device))
        if not return_mask:
            return output.last_hidden_state
        return output.last_hidden_state, encoded_inputs.attention_mask.to(
            dtype=bool)
```

### Error logs

```
[20/03/23 12:24:41][pytorch_lightning.accelerators.cuda][INFO] - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[20/03/23 12:24:41][pytorch_lightning.callbacks.model_summary][INFO] -
  | Name          | Type                 | Params
-------------------------------------------------------
0 | textencoder   | TextSpace            | 71.3 M
1 | motionencoder | ActorAgnosticEncoder | 4.8 M
2 | motiondecoder | ActorAgnosticDecoder | 6.4 M
3 | metrics       | ComputeMetricsSpace  | 0
4 | _losses       | ModuleDict           | 0
5 | _tracker      | ModuleDict           | 0
-------------------------------------------------------
82.4 M    Trainable params
0         Non-trainable params
82.4 M    Total params
329.713   Total estimated model params size (MB)
[20/03/23 12:24:41][space.callback.progress][INFO] - Training started
Epoch 0:   0%|                                                                                                    | 0/2 [00:00<?, ?it/s]cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
cpu cpu cpu
[2023-03-20 12:24:41,463] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing training_step
[2023-03-20 12:24:41,809] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing allsplit_step
[2023-03-20 12:24:41,875] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing text_to_motion_forward
[2023-03-20 12:24:41,910] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-03-20 12:24:41,922] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing get_last_hidden_state
[2023-03-20 12:24:41,928] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in get_last_hidden_state>
[2023-03-20 12:24:42,027] torch._dynamo.convert_frame: [INFO] converting frame raised unsupported, leaving it unconverted
[2023-03-20 12:24:42,028] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in get_last_hidden_state>
[2023-03-20 12:24:42,031] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-03-20 12:24:42,559] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-03-20 12:24:42,571] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-03-20 12:24:47,327] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0
/tmp/tmps69d3bg_/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
/tmp/tmpwukh8f60/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
compilation terminated.
/tmp/tmpzui1vag_/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpnscxung2/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpnj6uttbb/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmp7onv3nig/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpn8c04nql/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpm_pufzho/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpxk5ft47j/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmp8ofizkry/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmp5n2txqua/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
/tmp/tmpva88bonr/main.c:4:10: fatal error: Python.h: No such file or directory
    4 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
Error executing job with overrides: ['run_id=draft-extract-examples', 'experiment=debug-ps018', 'devices=1', 'data.dtype=spatial_pairs+seg+seq', 'model=space_joint', 'data.synthetic=true', 'data.proportion_synthetic=0.5', 'data.random_synthetic=True', 'machine.batch_size=8', 'model.optim.lr=0.0008', 'model.losses.loss_on_vertices=true', 'data.tiny=true', 'logger=none', 'machine.num_workers=0', 'trainer.log_every_n_steps=2']
An error occurred during Hydra's exception formatting:
TypeError("print_exception() got an unexpected keyword argument 'etype'")
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 549, in _worker_compile
    kernel.precompile(warm_cache_only_with_cc=cc)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/triton_ops/autotune.py", line 69, in precompile
    self.launchers = [
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/triton_ops/autotune.py", line 70, in <listcomp>
    self._precompile_config(c, warm_cache_only_with_cc)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/triton_ops/autotune.py", line 83, in _precompile_config
    triton.compile(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/triton/compiler.py", line 1587, in compile
    so_path = make_stub(name, signature, constants)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/triton/compiler.py", line 1476, in make_stub
    so = _build(name, src_path, tmpdir)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/triton/compiler.py", line 1391, in _build
    ret = subprocess.check_call(cc_cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp7onv3nig/main.c', '-O3', '-I/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/triton/third_party/cuda/include', '-I/usr/include/python3.10', '-I/tmp/tmp7onv3nig', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmp7onv3nig/triton_.cpython-310-x86_64-linux-gnu.so', '-L/usr/lib/x86_64-linux-gnu']' returned non-zero exit status 1.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 670, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.fake_example_inputs())
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1055, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/__init__.py", line 1390, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 455, in compile_fx
    return aot_autograd(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2805, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2498, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1713, in aot_wrapper_dedupe
    return compiler_fn(flat_fn, leaf_flat_args, aot_config)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2133, in aot_dispatch_autograd
    compiled_fw_func = aot_config.fw_compiler(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 430, in fw_compiler
    return inner_compile(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 595, in debug_wrapper
    compiled_fn = compiler_fn(gm, example_inputs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 177, in compile_fx_inner
    compiled_fn = graph.compile_to_fn()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/graph.py", line 586, in compile_to_fn
    return self.compile_to_module().call
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/graph.py", line 575, in compile_to_module
    mod = PyCodeCache.load(code)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 528, in load
    exec(code, mod.__dict__, mod.__dict__)
  File "/tmp/torchinductor_nathanasiou/wr/cwrfi6svswnyoicfxggucqfgjge566oqd55uh5wriq3feqdgyibn.py", line 544, in <module>
    async_compile.wait(globals())
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 715, in wait
    scope[key] = result.result()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 573, in result
    self.future.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp7onv3nig/main.c', '-O3', '-I/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/triton/third_party/cuda/include', '-I/usr/include/python3.10', '-I/tmp/tmp7onv3nig', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmp7onv3nig/triton_.cpython-310-x86_64-linux-gnu.so', '-L/usr/lib/x86_64-linux-gnu']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/train.py", line 156, in <module>
    _train()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/main.py", line 48, in decorated_main
    _run_hydra(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/_internal/utils.py", line 294, in run_and_report
    raise ex
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/train.py", line 50, in _train
    return train(cfg, ckpt_ft)
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/train.py", line 145, in train
    trainer.fit(compiled_model, datamodule=data_module, ckpt_path=ckpt_ft)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 520, in fit
    call._call_and_handle_interrupt(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 559, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
    results = self._run_stage()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 978, in _run_stage
    self.fit_loop.run()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 201, in run
    self.advance()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 354, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 133, in run
    self.advance(data_fetcher)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 218, in advance
    batch_output = self.automatic_optimization.run(trainer.optimizers[0], kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 185, in run
    self._optimizer_step(kwargs.get("batch_idx", 0), closure)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 261, in _optimizer_step
    call._call_lightning_module_hook(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 142, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1266, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 158, in step
    step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 224, in optimizer_step
    return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 114, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/optim/optimizer.py", line 33, in _use_grad
    ret = func(self, *args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/optim/adamw.py", line 148, in step
    loss = closure()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 101, in _wrap_closure
    closure_result = closure()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 140, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 126, in closure
    step_output = self._step_fn()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 308, in _training_step
    training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 366, in training_step
    return self.model.training_step(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/space/model/base.py", line 36, in training_step
    return self.allsplit_step("train", batch, batch_idx)
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/space/model/space.py", line 377, in allsplit_step
    datastruct_from_text, latent_vectors_text, distributions_from_text, mask_texts = self.text_to_motion_forward(gt_texts,
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/space/model/space.py", line 249, in text_to_motion_forward
    distributions = self.textencoder(texts, mapping=indices)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/space/model/textencoder/text_space.py", line 63, in forward
    text_encoded, mask = self.get_last_hidden_state(texts, return_mask=True)
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/space/model/textencoder/distilbert.py", line 55, in get_last_hidden_state
    encoded_inputs = self.tokenizer(texts,
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/space/model/textencoder/distilbert.py", line 58, in <graph break in get_last_hidden_state>
    output = self.text_model(**encoded_inputs.to(self.text_model.device))
  File "/home/nathanasiou/Desktop/conditional_action_gen/space/space/model/textencoder/distilbert.py", line 58, in <graph break in get_last_hidden_state>
    output = self.text_model(**encoded_inputs.to(self.text_model.device))
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
    return callback(frame, cache_size, hooks)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
    return fn(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
    return _compile(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
    transformations(instructions, code_options)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
    super().run()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
    and self.step()
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
    getattr(self, inst.opname)(inst)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1792, in RETURN_VALUE
    self.output.compile_subgraph(
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 541, in compile_subgraph
    self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp7onv3nig/main.c', '-O3', '-I/home/nathanasiou/.venvs/space/lib/python3.10/site-packages/triton/third_party/cuda/include', '-I/usr/include/python3.10', '-I/tmp/tmp7onv3nig', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmp7onv3nig/triton_.cpython-310-x86_64-linux-gnu.so', '-L/usr/lib/x86_64-linux-gnu']' returned non-zero exit status 1.


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True
```

### Minified repro

_No response_

### Versions

```
PyTorch version: 2.0.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.26.0
Libc version: glibc-2.31

Python version: 3.10.10 (main, Feb  8 2023, 14:50:01) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Quadro RTX 5000
Nvidia driver version: 525.89.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          12
On-line CPU(s) list:             0-11
Thread(s) per core:              2
Core(s) per socket:              6
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz
Stepping:                        4
CPU MHz:                         3600.000
CPU max MHz:                     3900,0000
CPU min MHz:                     1200,0000
BogoMIPS:                        7200.00
Virtualization:                  VT-x
L1d cache:                       192 KiB
L1i cache:                       192 KiB
L2 cache:                        6 MiB
L3 cache:                        8,3 MiB
NUMA node0 CPU(s):               0-11
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:          Mitigation; IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] pytorch-lightning==2.0.0
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.1
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.15.1
[pip3] triton==2.0.0
[conda] No relevant packages
```

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh