[pt2] MMDet meets Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode error with aot_eager backend
Labels: high priority, needs reproduction, triaged, oncall: pt2, module: fakeTensor

### 🐛 Describe the bug

Training RetinaNet models with aot_eager backend got ` Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'`

Note: It is not the error at the beginning of the training, but the error after training for 13 iter.

### Error logs

```shell
Traceback (most recent call last):
  File "pt20/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 692, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.fake_example_inputs())
  File "pt20/lib/python3.8/site-packages/torch/_dynamo/debug_utils.py", line 1024, in debug_wrapper
    run_fwd_maybe_bwd(compiled_gm, example_inputs)
  File "pt20/lib/python3.8/site-packages/torch/_dynamo/debug_utils.py", line 624, in run_fwd_maybe_bwd
    out = gm(args)
  File "pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 996, in g
    return f(*args)
  File "pt20/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 211, in _fn
    return fn(*args, **kwargs)
  File "pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 2497, in forward
    return compiled_fn(full_args)
  File "pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 996, in g
    return f(*args)
  File "pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 2066, in debug_compiled_function
    return compiled_function(*args)
  File "pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1930, in compiled_function
    all_outs = CompiledFunction.apply(*args_with_synthetic_bases)
  File "pt20/lib/python3.8/site-packages/torch/autograd/function.py", line 508, in apply
    return super().apply(*args, **kwargs)
  File "pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1714, in forward
    fw_outs = call_func_with_args(
  File "/pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1021, in call_func_with_args
    out = normalize_as_list(f(args))
  File "/pt20/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 996, in g
    return f(*args)
  File "/mnt/petrelfs/huanghaian/miniconda3/envs/pt20/lib/python3.8/site-packages/torch/fx/graph_module.py", line 660, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "pt20/lib/python3.8/site-packages/torch/fx/graph_module.py", line 279, in __call__
    raise e
  File "/pt20/lib/python3.8/site-packages/torch/fx/graph_module.py", line 269, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "pt20/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "<eval_with_key>.33", line 111, in forward
  File "pt20/lib/python3.8/site-packages/torch/_ops.py", line 284, in __call__
    return self._op(*args, **kwargs or {})
  File "pt20/lib/python3.8/site-packages/torch/utils/_stats.py", line 15, in wrapper
    return fn(*args, **kwargs)
  File "pt20/lib/python3.8/site-packages/torch/_subclasses/fake_tensor.py", line 656, in __torch_dispatch__
    return func(*args, **kwargs)
  File "pt20/lib/python3.8/site-packages/torch/_ops.py", line 284, in __call__
    return self._op(*args, **kwargs or {})
  File "pt20/lib/python3.8/site-packages/torch/_subclasses/fake_tensor.py", line 835, in __torch_dispatch__
    args, kwargs = self.validate_and_convert_non_fake_tensors(
  File "pt20/lib/python3.8/site-packages/torch/_subclasses/fake_tensor.py", line 983, in validate_and_convert_non_fake_tensors
    return tree_map_only(
  File "/lib/python3.8/site-packages/torch/utils/_pytree.py", line 266, in tree_map_only
    return tree_map(map_only(ty)(fn), pytree)
  File "pt20/lib/python3.8/site-packages/torch/utils/_pytree.py", line 196, in tree_map
    return tree_unflatten([fn(i) for i in flat_args], spec)
  File "pt20/lib/python3.8/site-packages/torch/utils/_pytree.py", line 196, in <listcomp>
    return tree_unflatten([fn(i) for i in flat_args], spec)
  File "pt20/lib/python3.8/site-packages/torch/utils/_pytree.py", line 247, in inner
    return f(x)
  File pt20/lib/python3.8/site-packages/torch/_subclasses/fake_tensor.py", line 975, in validate
    raise Exception(
Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(2, 3, 640, 640)), cuda:0), Parameter containing:
tensor([[[[ 1.3335e-02,  1.4664e-02, -1.5351e-02,  ..., -4.0896e-02,
           -4.3034e-02, -7.0755e-02],
          [ 4.1205e-03,  5.8477e-03,  1.4948e-02,  ...,  2.2060e-03,
           -2.0912e-02, -3.8517e-02],
          [ 2.2331e-02,  2.3595e-02,  1.6120e-02,  ...,  1.0281e-01,


```



### Minified repro

### 1 Install dependencied of OpenMMLab
```shell
ninja
git+https://github.com/open-mmlab/mmengine@experimental/compile
git+https://github.com/open-mmlab/mmcv@2.x

git clone git@github.com:open-mmlab/mmdetection.git
cd mmdet
git checkout dev-3.x
pip install -r requirements.txt
pip install -e .
```
### 2 Modify the `configs/_base_/default_runtime.py`

```python
compile = dict(
    target='train_step',  # (train_step, forward, model)
    verbose=True,
    backend='aot_eager',  
    dynamic=False, 
)
```


### 3 Launch training

`python tools/train.py configs/retinanet/retinanet_r50_fpn_1x_coco.py`




### Versions

Pytorch version: 2.0.0.dev20230131+cu116

cc @ezyang @gchanan @zou3519 @msaroufim @wconstab @bdhirsh @anijain2305 @soumith @ngimel