[pt20][aot_eager] Exceed Python recursion limit with huge model or frequent recompilation
Labels: triaged, ezyang's list, oncall: pt2, module: dynamic shapes, module: dynamo

### 🐛 Describe the bug

`aot_eager` backend wraps `bw_compiler` constantly on each graph break & recompilation, making the function call exceeds Python recursion limit for huge model or frequent recompilations. Repro example below

```python
import torch
import torch.nn as nn
import torch._dynamo as dynamo
from torch._dynamo import disallow_in_graph
import random

# manually simulate graph breaks
def graph_break():
    pass
disallow_in_graph(graph_break)

class Repro(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(4, 4)
        self.linear2 = nn.Linear(4, 4)
        self.linear3 = nn.Linear(4, 4)
    def forward(self, x):
        x = self.linear1(x)
        graph_break()
        x = self.linear2(x)
        graph_break()
        x = self.linear3(x)
        return x

if __name__ == '__main__':
    model = Repro().cuda()
    opt = torch.compile(model, backend='aot_eager')
    # variable batch_size to simulate recompilations
    for batch_size in range(3):
        data = torch.randn(batch_size, 4).cuda()
        out = opt(data)
        loss = out.sum()
        loss.backward()
```

Add the following lines to [torch/_dynamo/optimizations/training.py](https://github.com/pytorch/pytorch/blob/a2fded30012e26d8c469d2b668a226315794a559/torch/_dynamo/optimizations/training.py#L54-L56) and then run the above script, the results as follow

```python
def _wrapped_bw_compiler(*args, **kwargs):
    global _count
    _count += 1
    print(f'{_count}: {bw_compiler}')
    return eval_frame.disable(eval_frame.disable(bw_compiler)(*args, **kwargs))
```
![image](https://user-images.githubusercontent.com/112053249/216346230-bce5816c-fc15-47b4-990d-f15ff71ef3fe.png)

Is this intended or a bug? What else can I do apart from reducing `torch._dynamo.config.cache_size_limit` and increasing `sys.setrecursionlimit()` to prevent this issue?

### Versions

```text
PyTorch version: 2.0.0.dev20230201+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 8.5.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.17

Python version: 3.8.15 (default, Nov 24 2022, 15:19:38)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-triton==2.0.0+0d7e753227
[pip3] torch==2.0.0.dev20230201+cu116
[pip3] torchaudio==2.0.0.dev20230112+cu116
[pip3] torchvision==0.15.0.dev20230131+cu116
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] pytorch-triton            2.0.0+0d7e753227          pypi_0    pypi
[conda] torch                     2.0.0.dev20230201+cu116          pypi_0    pypi
[conda] torchaudio                2.0.0.dev20230112+cu116          pypi_0    pypi
[conda] torchvision               0.15.0.dev20230131+cu116          pypi_0    pypi
```

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire