torch.compile makes transformers model (llama) generating different outputs compared with the native
Labels: triaged, oncall: pt2, module: cpu inductor

### 🐛 Describe the bug

To run bf16 model generating, we found there are difference output sentence after using torch.compile compared with native:

Native:
Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun along the way. One day, the little girl decided to go on an adventure. She packed **up her backpack and set out on her journey.**
torch.compile:
Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun along the way. One day, the little girl decided to go on an adventure. She packed **her backpack and set out on her journey. She**


Native code:
```
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM, LlamaTokenizer
import torch
model = LlamaForCausalLM.from_pretrained("decapoda-research/llama-7b-hf", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16)
tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
model = model.eval()
prompt = "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun"

num_iter = 3

with torch.cpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
    for i in range(num_iter):
        input_ids = tokenizer(prompt, return_tensors="pt").input_ids
        output  = model.generate(input_ids, max_new_tokens=32, do_sample=False, temperature=0.9, num_beams=4)
        gen_text = tokenizer.batch_decode(output, skip_special_tokens=True)
        print(gen_text, flush=True)
```

torch.compile code: 
```
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM, LlamaTokenizer
import torch
model = LlamaForCausalLM.from_pretrained("decapoda-research/llama-7b-hf", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16)
tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
model = model.eval()
prompt = "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun"

import torch._inductor.config as config
config.cpp.enable_kernel_profile=True
config.profiler_mark_wrapper_call=True
torch._dynamo.config.suppress_errors = True
model.generate = torch.compile(model.generate, backend='inductor', dynamic=True)

num_iter = 3
with torch.cpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
    for i in range(num_iter):
        input_ids = tokenizer(prompt, return_tensors="pt").input_ids
        output  = model.generate(input_ids, max_new_tokens=32, do_sample=False, temperature=0.9, num_beams=4)
        gen_text = tokenizer.batch_decode(output, skip_special_tokens=True)
        print(gen_text, flush=True)
```



### Versions

Collecting environment information...
PyTorch version: 2.1.0.dev20230518+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: CentOS Stream 8 (x86_64)
GCC version: (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
Clang version: 14.0.0 (Red Hat 14.0.0-1.module_el8.7.0+1142+5343df54)
CMake version: version 3.22.1
Libc version: glibc-2.28

Python version: 3.8.16 (default, Mar 2 2023, 03:21:46) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.18.0-365.el8.x86_64-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
transformers==4.28.1
[pip3] numpy==1.24.3
[pip3] torch==2.1.0.dev20230518+cpu
[conda] blas 1.0 mkl
[conda] mkl 2023.1.0 h6d00ec8_46342
[conda] mkl-include 2023.1.0 pypi_0 pypi
[conda] mkl-service 2.4.0 py38h5eee18b_1
[conda] mkl-static 2023.1.0 pypi_0 pypi
[conda] mkl_fft 1.3.6 py38h417a72b_1
[conda] mkl_random 1.2.2 py38h417a72b_1
[conda] numpy 1.24.3 py38hf6e8229_1
[conda] numpy-base 1.24.3 py38h060ed82_1
[conda] torch 2.1.0.dev20230518+cpu pypi_0 pypi



cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305