Toggling model.train() causes guard failures every time
Labels: triage review, triaged, bug, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Switching between `model.train()` and `model.eval()` on a torch.compile'd model causes guard failures every time, due to `__nn_module_guard_0(L['self'])`. I could understand a single recompile, if the model config changing invalidates a guard, but remarkably this happens *every single time*, leading to an unbounded number of recompiles!

### Error logs
```
Torchdynamo Profiler Report
===========================

Graph Breaks
------------
Graph breaks happen when torchdynamo encounters code it can't safely trace.
If you want to find out why breaks are happening, check below for each break reason
You may gain additional insight by passing `fullgraph=True` to torch.compile,
to stop at the first break.

No graph breaks detected.

Recompilation
-------------
These subgraphs were recompiled more than once due to guard failures
Guard failures indicate some condition assumed to be static by the tracer changed,
making it unsafe to reuse the compiled program.

Function                        Recompiles  Recompile Reasons
----------------------------  ------------  ------------------------------
'forward' (2657887092.py:11)             8  __nn_module_guard_0(L['self'])
                                            __nn_module_guard_0(L['self'])
                                            __nn_module_guard_0(L['self'])
                                            __nn_module_guard_0(L['self'])
                                            __nn_module_guard_0(L['self'])
                                            __nn_module_guard_0(L['self'])
                                            __nn_module_guard_0(L['self'])
                                            __nn_module_guard_0(L['self'])

Set torch._dynamo.config.cache_size_limit to 8 to avoid being cache limited.
```

### Minified repro

```python
import torch
import torch.nn as nn
from torch._dynamo.utils import CompileProfiler

class SimpleDropout(nn.Module):
    def __init__(self):
        super().__init__()
        self.dropout = nn.Dropout(0.5)
        self.linear = nn.Linear(10, 1)
    
    def forward(self, x):
        return self.dropout(self.linear(x))
    
model = SimpleDropout()
x = torch.randn(10)
with CompileProfiler() as p:
    model = torch.compile(model, backend=p)
    for _ in range(4):
        model.eval()
        model(x)
        model.train()
        model(x)
    print(p.report())
```

### Versions

PyTorch version: 2.2.0.dev20231005
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 14.0.0-1ubuntu1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.15.90.2-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version: 545.29
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.8.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          32
On-line CPU(s) list:             0-31
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen Threadripper 1950X 16-Core Processor
CPU family:                      23
Model:                           1
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
Stepping:                        1
BogoMIPS:                        6787.36
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr virt_ssbd arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload
Virtualization:                  AMD-V
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       512 KiB (16 instances)
L1i cache:                       1 MiB (16 instances)
L2 cache:                        8 MiB (16 instances)
L3 cache:                        8 MiB (1 instance)
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Mitigation; untrained return thunk; SMT vulnerable
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==2.2.0.dev20231005
[pip3] torch-tb-profiler==0.4.3
[pip3] torchaudio==2.2.0.dev20231005
[pip3] torchvision==0.17.0.dev20231005
[pip3] triton==2.1.0
[conda] blas                      2.16                        mkl    conda-forge
[conda] libblas                   3.8.0                    16_mkl    conda-forge
[conda] libcblas                  3.8.0                    16_mkl    conda-forge
[conda] liblapack                 3.8.0                    16_mkl    conda-forge
[conda] liblapacke                3.8.0                    16_mkl    conda-forge
[conda] mkl                       2020.2                      256  
[conda] numpy                     1.22.4          py310h4ef5377_0    conda-forge
[conda] pytorch                   2.2.0.dev20231005 py3.10_cuda12.1_cudnn8.9.2_0    pytorch-nightly
[conda] pytorch-cuda              12.1                 ha16c6d3_5    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torch-tb-profiler         0.4.3                    pypi_0    pypi
[conda] torchaudio                2.2.0.dev20231005     py310_cu121    pytorch-nightly
[conda] torchtriton               2.1.0+6e4932cda8           py310    pytorch-nightly
[conda] torchvision               0.17.0.dev20231005     py310_cu121    pytorch-nightly

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng