model.forward() get error with torch.compile() when using huggingface llama
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Found that the torch.compile(model) lead to "Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'." error when learning alpaca-lora. Here is a minimal repro. When I commented out the torch.compile(), things are good.
```
import torch
import sys
import random
import numpy as np

# from stop_word import StoppingCriteriaSub, StoppingCriteriaList
from transformers import LlamaTokenizer, LlamaForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    # bnb_4bit_quant_type="fp4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
random.seed(0)
np.random.seed(0)
torch.manual_seed(0)
torch.cuda.manual_seed(0)
torch.backends.cudnn.deterministic = True

device = "cuda:0"
tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-30b",legacy=False)
model = LlamaForCausalLM.from_pretrained(
        "huggyllama/llama-30b",
        quantization_config=bnb_config,
        # torch_dtype=torch.float16,
        device_map="auto",
    )
model.config.pad_token_id = tokenizer.pad_token_id = 0  # unk
model.config.bos_token_id = 1
model.config.eos_token_id = 2
if torch.__version__ >= "2" and sys.platform != "win32":
    model = torch.compile(model) ### seems here is the bug
tokens = tokenizer(["prompt is prompt is prompt","her eyes are so beautiful"], return_tensors='pt', padding=True).to(device)
with torch.no_grad():
    logits = model(**tokens, return_dict=True).logits
```

The complete output
```
Traceback (most recent call last):                                                                                                                                             
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1199, in run_node                                                          
    return nnmodule(*args, **kwargs)                                                                                                                                           
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl                                                    
    return forward_call(*args, **kwargs)                                                                                                                                       
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward                                                           
    output = old_forward(*args, **kwargs)                                                                                                                                      
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward                                                        
    return F.embedding(                                                                                                                                                        
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding                                                         
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)                                                                                             
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper                                                              
    return fn(*args, **kwargs)                                                                                                                                                 
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 987, in __torch_dispatch__                                       
    return self.dispatch(func, types, args, kwargs)                                                                                                                            
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1066, in dispatch                                                
    args, kwargs = self.validate_and_convert_non_fake_tensors(                                                                                                                 
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1220, in validate_and_convert_non_fake_tensors                   
    return tree_map_only(
File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/utils/_pytree.py", line 266, in tree_map_only                                                      
    return tree_map(map_only(ty)(fn), pytree)
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, in tree_map
    return tree_unflatten([fn(i) for i in flat_args], spec)
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, in <listcomp>
    return tree_unflatten([fn(i) for i in flat_args], spec)
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/utils/_pytree.py", line 247, in inner
    return f(x)
  File "/data/haotian/anaconda3/envs/RAP/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1212, in validate
    raise Exception(
Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.embedding.default(*(Parameter containing:
tensor([[-0.0006,  0.0002, -0.0005,  ..., -0.0012,  0.0015,  0.0003],
        [-0.0191, -0.0041, -0.0010,  ...,  0.0021, -0.0018, -0.0159],
        [-0.0117, -0.0014,  0.0098,  ..., -0.0053,  0.0033,  0.0040],
        ...,
        [ 0.0031, -0.0040, -0.0153,  ...,  0.0268, -0.0198, -0.0242],
        [-0.0103, -0.0001,  0.0041,  ...,  0.0065, -0.0163,  0.0061],
        [ 0.0017, -0.0164, -0.0108,  ...,  0.0030,  0.0046, -0.0222]],
       device='cuda:0', dtype=torch.float16, requires_grad=True), FakeTensor(FakeTensor(..., device='meta', size=(2, 6), dtype=torch.int64), cuda:0), 0), **{})
```

### Versions

```
Collecting environment information...                                                                                                                                          
PyTorch version: 2.0.1+cu117                                                                                                                                                   
Is debug build: False                                                                                                                                                          
CUDA used to build PyTorch: 11.7                                                                                                                                               
ROCM used to build PyTorch: N/A                                                                                                                                                
                                                                                                                                                                               
OS: Ubuntu 20.04.6 LTS (x86_64)                                                                                                                                                
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0                                                                                                                             
Clang version: Could not collect                                                                                                                                               
CMake version: version 3.26.4                                                                                                                                                  
Libc version: glibc-2.31                                                                                                                                                       
                                                                                                                                                                               
Python version: 3.10.12 (main, Jul  5 2023, 18:54:27) [GCC 11.2.0] (64-bit runtime)                                                                                            
Python platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31                                                                                                                 
Is CUDA available: True                                                                                                                                                        
CUDA runtime version: 11.7.99                                                                                                                                                  
CUDA_MODULE_LOADING set to: LAZY                                                                                                                                               
GPU models and configuration:                                                                                                                                                  
GPU 0: NVIDIA GeForce RTX 3090                                                                                                                                                 
GPU 1: NVIDIA GeForce RTX 3090                                                                                                                                                 
GPU 2: NVIDIA GeForce RTX 3090                                                                                                                                                 
GPU 3: NVIDIA GeForce RTX 3090                                                                                                                                                 
GPU 4: NVIDIA GeForce RTX 3090                                                                                                                                                 
GPU 5: NVIDIA GeForce RTX 3090                                                                                                                                                 
GPU 6: NVIDIA GeForce RTX 3090                                                                                                                                                 
GPU 7: NVIDIA GeForce RTX 3090                                                                                                                                                 
                                                                                                                                                                               
Nvidia driver version: 525.125.06                                                                                                                                              
cuDNN version: Could not collect                                                                                                                                               
HIP runtime version: N/A                                                                                                                                                       
MIOpen runtime version: N/A                                                                                                                                                    
Is XNNPACK available: True                                                                                                                                                     
                                                                                                                                                                               
CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          56
On-line CPU(s) list:             0-55
Thread(s) per core:              1
Core(s) per socket:              28
Socket(s):                       2
NUMA node(s):                    8
Vendor ID:                       AuthenticAMD
CPU family:                      25
Model:                           1
Model name:                      AMD EPYC 7453 28-Core Processor
Stepping:                        1
Frequency boost:                 enabled
CPU MHz:                         3305.527
CPU max MHz:                     2750.0000
CPU min MHz:                     1500.0000
BogoMIPS:                        5499.64
Virtualization:                  AMD-V
L1d cache:                       1.8 MiB
L1i cache:                       1.8 MiB
L2 cache:                        28 MiB
L3 cache:                        128 MiB
NUMA node0 CPU(s):               0-6
NUMA node1 CPU(s):               7-13
NUMA node2 CPU(s):               14-20
NUMA node3 CPU(s):               21-27
NUMA node4 CPU(s):               28-34
NUMA node5 CPU(s):               35-41
NUMA node6 CPU(s):               42-48
NUMA node7 CPU(s):               49-55
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp
 lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm 
cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpci
d_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_ll
c cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthr
eshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.25.1
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2
[pip3] triton==2.0.0
[conda] numpy                     1.25.1                   pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torchaudio                2.0.2                    pypi_0    pypi
[conda] torchvision               0.15.2                   pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi
```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov