Dynamo guard on global configuration
Labels: oncall: pt2, module: dynamo

### 🐛 Describe the bug

Co-proposal with @voznesenskym and @Chillee

## Problem statement

Today, it is difficult to force a PT2 compilation with locally modified Dynamo/Inductor configuration, without having that configuration leak to other parts of the program. First, we have no convenient argument to torch.compile to locally change configuration options. This can be simulated by using torch._dynamo.config.patch. But there are still problems.

Problem 1: configuration is not guarded, so compile products of stale configs will still "hit" even if the config has changed.

```
with torch._dynamo.config.patch(CONFIG1):
    torch.compile(f)(args)  # compilation is stored on f's code object

with torch._dynamo.config.patch(CONFIG2):
    torch.compile(f)(args)  # will not recompile, CONFIG1's compile product is used
```

Problem 2: due to graph breaks, compile products may be unpredictably associated with arbitrary downstream function code objects, not just the function that was torch.compile'd

```
def g1(args): ...
def g2(args): ...
def f(args):
    g1(args)
    print("graph break")
    g2(args)

with torch._dynamo.config.patch(CONFIG1):
    torch.compile(f)(args)  # graph break means we store compile products on g1, g2

with torch._dynamo.config.patch(CONFIG2):
    torch.compile(g2)(args)  # will not recompile, CONFIG1 compile of g2 is used
```

## Constraints

In evaluating various proposals to solve the problem, we set the following goals:

**Efficient guard evaluation.** We should not have to do an equality test on two dicts with dozens of entries every time we attempt to enter a config. This means we should not naively just guard on the state of config objects during compilation

**Outer-most configuration wins.** Consider the following program:

```
@torch.compile(dynamic=True)
def f(args): ...

@torch.compile(dynamic=False)
def g(args):
    return f(args) + ... something else ..
```

If I call `g(args)`, should f be compiled with dynamic=True or False? We argue it should be False, e.g., outer-most configuration wins. In principle, a more sophisticated system might be able to say, "well, we should compile the g parts with dynamic=False, but when we compile f, we should compile with dynamic=True", but this is difficult to do if we *inline* into f, since we don't really have a concept of changing configuration options midway through a Dynamo trace / compiling an FX graph with different configuration options applied to different regions. It is simpler and more uniform for the outer-most configuration to override the inner configurations. It is also often what you want; e.g., f was compiled with dynamic=True for ease of use with eager mode, but in g's context, we may only call it at one shape and don't mind specializing.

There may be a place for a variant of `torch.compile` which is "do not inline into me, instead, recursively torch.compile the inside, and then embed the resulting compiled FX graph into the outer compile product", for which allowing a configuration change may make sense, but we leave this to future work.

**Repeated calls to torch.compile should not cause recompilation.** This is mostly a BC thing. Suppose you have:

```
def f(args):
    @torch.compile()
    def g(args): ...

    return g(args)

f(args)
f(args)  # should not recompile
```

The fact that torch.compile() is invoked twice (once per f call) should NOT mean that we also must recompile g, since this is not what Dynamo does today. Of course, if you change the configuration, you should recompile.

```
def f(args, dynamic):
    @torch.compile(dynamic=dynamic)
    def g(args): ...

    return g(args)

f(args, dynamic=False)
f(args, dynamic=True)  # recompile
f(args, dynamic=False)  # reuse first compile product
```

**Use the configuration at the time torch.compile is called.** Suppose you have

```
opt_f = torch.compile(f)
with torch._dynamo.config.patch(CONFIG):
    opt_f(args)
```

The change in configuration between the torch.compile call and the actual JIT compilation when we call in opt_f should NOT affect the compilation; in other words, we should use the original (non-CONFIG) configuration to perform this compilation.

Without this constraint, it is not possible to support configuration as an argument to torch.compile:

```
opt_f = torch.compile(f, dynamic=True)
opt_f(args)  # would be weird for this to not be compiled with dynamic=True
```

## Proposal

We divide PT2 compilation into two steps:

```
opt_f = torch.compile(f)  # step 1
opt_f(args)  # step 2
```

When you call torch.compile (the first call)
1. Snapshot the current state of the config (cloning it) and store both it, and a hash of the config, on TorchDynamoContext

When you call the opt_f (the second call)
1. Check for a cache hit. Entries in the cache will test if the hash on TorchDynamoContext matches the hash that was active at the time the compiled program was compiled. (Compile products with different config hashes should not count towards the standard cache limit.)
2. If we cache miss, restore the snapshot config from the TorchDynamoContext before compiling
3. In all cases (e.g., even if the frame is SKIPed), we locally set TLS with the config/hash of the TorchDynamoContext, which overrides the global config state. This ensures that recursive Dynamo compilation, e.g., due to graph break, also consistently takes the top-level configuration

This change is technically BC-breaking, but it is BC-breaking in a way that is plausibly just a bug fix. The biggest risk is code which previously only compiled once, now recompiles multiple times to account for configuration changes.

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng