Bugs about BART of Hugging Face using Pytorch 2.0
Labels: high priority, module: crash, triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

This is the code of using BART of Hugging Face with Pytorch 2.0:
```python
import torch
from transformers import BartTokenizer, BartForConditionalGeneration

device = torch.device('cuda')
tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
model = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
model = model.to(device)
model = torch.compile(model)

inputs = tokenizer(
    "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.",
    return_tensors="pt",
)
labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"]

input_ids = inputs["input_ids"].to(device)
attention_mask = inputs["attention_mask"].to(device)
labels = labels.to(device)

loss = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels).loss

```

When using Pytorch 2.0 without `torch.compile` with `device=torch.device('cpu')` or `device=torch.device('cuda')`, it works well without any warning.

When using Pytorch 2.0 with `torch.compile` and `device=torch.device('cpu')`, it works well but with the following warning:
```
/home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/nn/utils/stateless.py:44: UserWarning: functional_call was passed multiple values for tied weights. This behavior is deprecated and will be an error in future versions
  warnings.warn("functional_call was passed multiple values for tied weights. "
/home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/nn/utils/stateless.py:44: UserWarning: functional_call was passed multiple values for tied weights. This behavior is deprecated and will be an error in future versions
  warnings.warn("functional_call was passed multiple values for tied weights. "
/home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/nn/utils/stateless.py:44: UserWarning: functional_call was passed multiple values for tied weights. This behavior is deprecated and will be an error in future versions
  warnings.warn("functional_call was passed multiple values for tied weights. "
```

When using Pytorch 2.0 with `torch.compile` and `device=torch.device('cuda')`, it doesn't work well with the following errors:
```
/home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/nn/utils/stateless.py:44: UserWarning: functional_call was passed multiple values for tied weights. This behavior is deprecated and will be an error in future versions
  warnings.warn("functional_call was passed multiple values for tied weights. "
/home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/nn/utils/stateless.py:44: UserWarning: functional_call was passed multiple values for tied weights. This behavior is deprecated and will be an error in future versions
  warnings.warn("functional_call was passed multiple values for tied weights. "
/home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/nn/utils/stateless.py:44: UserWarning: functional_call was passed multiple values for tied weights. This behavior is deprecated and will be an error in future versions
  warnings.warn("functional_call was passed multiple values for tied weights. "
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <stdin>:1 in <module>                                                                            │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py:1480 in       │
│ _call_impl                                                                                       │
│                                                                                                  │
│   1477 │   │   if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks   │
│   1478 │   │   │   │   or _global_backward_pre_hooks or _global_backward_hooks                   │
│   1479 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1480 │   │   │   return forward_call(*args, **kwargs)                                          │
│   1481 │   │   # Do not call functions when jit is used                                          │
│   1482 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1483 │   │   backward_pre_hooks = []                                                           │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py:82 in        │
│ forward                                                                                          │
│                                                                                                  │
│    79 │   │   return getattr(self._orig_mod, name)                                               │
│    80 │                                                                                          │
│    81 │   def forward(self, *args, **kwargs):                                                    │
│ ❱  82 │   │   return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)                    │
│    83                                                                                            │
│    84                                                                                            │
│    85 def remove_from_cache(f):                                                                  │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py:211 in _fn   │
│                                                                                                  │
│   208 │   │   │   dynamic_ctx = enable_dynamic(self.dynamic)                                     │
│   209 │   │   │   dynamic_ctx.__enter__()                                                        │
│   210 │   │   │   try:                                                                           │
│ ❱ 211 │   │   │   │   return fn(*args, **kwargs)                                                 │
│   212 │   │   │   finally:                                                                       │
│   213 │   │   │   │   set_eval_frame(prior)                                                      │
│   214 │   │   │   │   dynamic_ctx.__exit__(None, None, None)                                     │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/transformers/models/bart/modeling_bart.p │
│ y:1328 in forward                                                                                │
│                                                                                                  │
│   1325 │   def set_output_embeddings(self, new_embeddings):                                      │
│   1326 │   │   self.lm_head = new_embeddings                                                     │
│   1327 │                                                                                         │
│ ❱ 1328 │   @add_start_docstrings_to_model_forward(BART_INPUTS_DOCSTRING)                         │
│   1329 │   @replace_return_docstrings(output_type=Seq2SeqLMOutput, config_class=_CONFIG_FOR_DOC  │
│   1330 │   @add_end_docstrings(BART_GENERATION_EXAMPLE)                                          │
│   1331 │   def forward(                                                                          │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py:211 in _fn   │
│                                                                                                  │
│   208 │   │   │   dynamic_ctx = enable_dynamic(self.dynamic)                                     │
│   209 │   │   │   dynamic_ctx.__enter__()                                                        │
│   210 │   │   │   try:                                                                           │
│ ❱ 211 │   │   │   │   return fn(*args, **kwargs)                                                 │
│   212 │   │   │   finally:                                                                       │
│   213 │   │   │   │   set_eval_frame(prior)                                                      │
│   214 │   │   │   │   dynamic_ctx.__exit__(None, None, None)                                     │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py:2107 in │
│ forward                                                                                          │
│                                                                                                  │
│   2104 │   │   full_args = []                                                                    │
│   2105 │   │   full_args.extend(params_flat)                                                     │
│   2106 │   │   full_args.extend(runtime_args)                                                    │
│ ❱ 2107 │   │   return compiled_fn(full_args)                                                     │
│   2108 │                                                                                         │
│   2109 │   # Just for convenience                                                                │
│   2110 │   forward.zero_grad = mod.zero_grad                                                     │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py:811 in  │
│ g                                                                                                │
│                                                                                                  │
│    808                                                                                           │
│    809 def make_boxed_func(f):                                                                   │
│    810 │   def g(args):                                                                          │
│ ❱  811 │   │   return f(*args)                                                                   │
│    812 │                                                                                         │
│    813 │   g._boxed_call = True                                                                  │
│    814 │   return g                                                                              │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py:1687 in │
│ debug_compiled_function                                                                          │
│                                                                                                  │
│   1684 │   │   │   │   │   f"{describe_input(i, aot_config)} would not require grad"             │
│   1685 │   │   │   │   )                                                                         │
│   1686 │   │                                                                                     │
│ ❱ 1687 │   │   return compiled_function(*args)                                                   │
│   1688 │                                                                                         │
│   1689 │   return debug_compiled_function                                                        │
│   1690                                                                                           │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py:1551 in │
│ compiled_function                                                                                │
│                                                                                                  │
│   1548 │   │   else:                                                                             │
│   1549 │   │   │   args_with_synthetic_bases = args                                              │
│   1550 │   │                                                                                     │
│ ❱ 1551 │   │   all_outs = CompiledFunction.apply(*args_with_synthetic_bases)                     │
│   1552 │   │   if CompiledFunction.num_aliasing_metadata_outs > 0:                               │
│   1553 │   │   │   outs = all_outs[:-CompiledFunction.num_aliasing_metadata_outs]                │
│   1554 │   │   │   aliasing_metadata_outs = all_outs[-CompiledFunction.num_aliasing_metadata_ou  │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py:1455 in │
│ forward                                                                                          │
│                                                                                                  │
│   1452 │   │   │   # (*mutated_data_inputs, *non_aliased_fw_outs, *saved_tensors, *saved_symint  │
│   1453 │   │   │   # - Note that in the synthetic bases case, mutated_inputs will correspond to  │
│   1454 │   │   │   #   of the original view, and not the synthetic base                          │
│ ❱ 1455 │   │   │   fw_outs = call_func_with_args(                                                │
│   1456 │   │   │   │   CompiledFunction.compiled_fw, deduped_flat_tensor_args, disable_amp=disa  │
│   1457 │   │   │   )                                                                             │
│   1458                                                                                           │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py:836 in  │
│ call_func_with_args                                                                              │
│                                                                                                  │
│    833 │   │   guard = torch._C._DisableAutocast()                                               │
│    834 │   try:                                                                                  │
│    835 │   │   if hasattr(f, "_boxed_call"):                                                     │
│ ❱  836 │   │   │   out = normalize_as_list(f(args))                                              │
│    837 │   │   else:                                                                             │
│    838 │   │   │   # TODO: Please remove soon                                                    │
│    839 │   │   │   # https://github.com/pytorch/pytorch/pull/83137#issuecomment-1211320670       │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_inductor/compile_fx.py:199 in run │
│                                                                                                  │
│   196 │   │   for i in check_inputs:                                                             │
│   197 │   │   │   if new_inputs[i].data_ptr() % ALIGNMENT:                                       │
│   198 │   │   │   │   new_inputs[i] = clone_preserve_strides(new_inputs[i])                      │
│ ❱ 199 │   │   return model(new_inputs)                                                           │
│   200 │                                                                                          │
│   201 │   return run                                                                             │
│   202                                                                                            │
│                                                                                                  │
│ /tmp/torchinductor_tangtianyi/mt/cmtrlxnsp7o7om6qxzj7wbwah2qxc6ker2iphq6omeo3332peb3m.py:1184 in │
│ call                                                                                             │
│                                                                                                  │
│   1181 │   buf4 = empty_strided((1, 31, 768), (23808, 768, 1), device='cuda', dtype=torch.float  │
│   1182 │   buf471 = empty_strided((1, 31, 1), (31, 1, 31), device='cuda', dtype=torch.float32)   │
│   1183 │   stream3 = get_cuda_stream(3)                                                          │
│ ❱ 1184 │   triton_fused_add_add_1_add_2_add_3_arange_div_50_embedding_embedding_1_expand_mul_0.  │
│   1185 │   del primals_1                                                                         │
│   1186 │   del primals_3                                                                         │
│   1187 │   buf5 = empty_strided((31, 768), (768, 1), device='cuda', dtype=torch.float32)         │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py:1 │
│ 69 in run                                                                                        │
│                                                                                                  │
│   166 │   │   │   if len(self.launchers) == 0:                                                   │
│   167 │   │   │   │   self.precompile()                                                          │
│   168 │   │   │   if len(self.launchers) > 1:                                                    │
│ ❱ 169 │   │   │   │   self.autotune_to_one_config(*args, grid=grid)                              │
│   170 │   │                                                                                      │
│   171 │   │   (launcher,) = self.launchers                                                       │
│   172 │   │   if launcher.config.pre_hook is not None:                                           │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_dynamo/utils.py:90 in             │
│ time_wrapper                                                                                     │
│                                                                                                  │
│     87 │   │   if key not in compilation_metrics:                                                │
│     88 │   │   │   compilation_metrics[key] = []                                                 │
│     89 │   │   t0 = time.time()                                                                  │
│ ❱   90 │   │   r = func(*args, **kwargs)                                                         │
│     91 │   │   latency = time.time() - t0                                                        │
│     92 │   │   # print(f"Dynamo timer: key={key}, latency={latency:.2f} sec")                    │
│     93 │   │   compilation_metrics[key].append(latency)                                          │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py:1 │
│ 56 in autotune_to_one_config                                                                     │
│                                                                                                  │
│   153 │   │   │   else:                                                                          │
│   154 │   │   │   │   cloned_args.append(arg)                                                    │
│   155 │   │                                                                                      │
│ ❱ 156 │   │   timings = {                                                                        │
│   157 │   │   │   launcher: self.bench(launcher, *cloned_args, **kwargs)                         │
│   158 │   │   │   for launcher in self.launchers                                                 │
│   159 │   │   }                                                                                  │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py:1 │
│ 57 in <dictcomp>                                                                                 │
│                                                                                                  │
│   154 │   │   │   │   cloned_args.append(arg)                                                    │
│   155 │   │                                                                                      │
│   156 │   │   timings = {                                                                        │
│ ❱ 157 │   │   │   launcher: self.bench(launcher, *cloned_args, **kwargs)                         │
│   158 │   │   │   for launcher in self.launchers                                                 │
│   159 │   │   }                                                                                  │
│   160 │   │   self.launchers = [builtins.min(timings, key=timings.get)]                          │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py:1 │
│ 38 in bench                                                                                      │
│                                                                                                  │
│   135 │   │                                                                                      │
│   136 │   │   from triton.testing import do_bench                                                │
│   137 │   │                                                                                      │
│ ❱ 138 │   │   return do_bench(kernel_call, rep=40, fast_flush=True)                              │
│   139 │                                                                                          │
│   140 │   @dynamo_utils.dynamo_timed                                                             │
│   141 │   def autotune_to_one_config(self, *args, **kwargs):                                     │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/triton/testing.py:140 in do_bench        │
│                                                                                                  │
│   137 │   """                                                                                    │
│   138 │                                                                                          │
│   139 │   # Estimate the runtime of the function                                                 │
│ ❱ 140 │   fn()                                                                                   │
│   141 │   torch.cuda.synchronize()                                                               │
│   142 │   start_event = torch.cuda.Event(enable_timing=True)                                     │
│   143 │   end_event = torch.cuda.Event(enable_timing=True)                                       │
│                                                                                                  │
│ /home/tangtianyi/miniconda3/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py:1 │
│ 30 in kernel_call                                                                                │
│                                                                                                  │
│   127 │   │   │   │   launcher.config.pre_hook(                                                  │
│   128 │   │   │   │   │   {**zip(self.arg_names, args), **launcher.config.kwargs}                │
│   129 │   │   │   │   )                                                                          │
│ ❱ 130 │   │   │   launcher(                                                                      │
│   131 │   │   │   │   *args,                                                                     │
│   132 │   │   │   │   grid=grid,                                                                 │
│   133 │   │   │   │   stream=stream,                                                             │
│ <string>:4 in launcher                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered
```


### Versions

PyTorch version: 1.14.0.dev20221208+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.31

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090
GPU 4: NVIDIA GeForce RTX 3090
GPU 5: NVIDIA GeForce RTX 3090
GPU 6: NVIDIA GeForce RTX 3090
GPU 7: NVIDIA GeForce RTX 3090

Nvidia driver version: 510.54
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.0rc2
[pip3] pytorch-transformers==1.0.0
[pip3] torch==1.14.0.dev20221208+cu116
[pip3] torchaudio==0.14.0.dev20221208+cu116
[pip3] torchtriton==2.0.0+0d7e753227
[pip3] torchvision==0.15.0.dev20221208+cu116
[pip3] torchviz==0.0.2
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    defaults
[conda] faiss-gpu                 1.7.2           py3.8_h28a55e0_0_cuda11.3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] ffmpeg                    4.3                  hf484d3e_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] libfaiss                  1.7.2           hfc2d529_0_cuda11.3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] mkl                       2021.4.0           h06a4308_640    defaults
[conda] mkl-service               2.4.0            py38h7f8727e_0    defaults
[conda] mkl_fft                   1.3.1            py38hd3c417c_0    defaults
[conda] mkl_random                1.2.2            py38h51133e4_0    defaults
[conda] numpy                     1.24.0rc2                pypi_0    pypi
[conda] numpy-base                1.21.5           py38hf524024_2    defaults
[conda] pytorch                   1.13.0          py3.8_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] pytorch-transformers      1.0.0                    pypi_0    pypi
[conda] torch                     1.14.0.dev20221208+cu116          pypi_0    pypi
[conda] torchaudio                0.14.0.dev20221208+cu116          pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227          pypi_0    pypi
[conda] torchvision               0.15.0.dev20221208+cu116          pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi

cc @ezyang @gchanan @zou3519 @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire