[dynamo] Improve recompilation reason infra
Labels: triaged, oncall: pt2, module: dynamo

Today we rely on cache lookup in eval_frame.c to call a [guard_fail_hook](https://github.com/pytorch/pytorch/blob/main/torch/csrc/dynamo/eval_frame.c#L684-L690) to stash the reason of recompilation. There are a couple of issues with this

* It goes on the critical path, and affects latency of the happy path where there is no recompilation (because we have to call the [python hook](https://github.com/pytorch/pytorch/blob/main/torch/_dynamo/guards.py#L1249-L1307)). Because of this, we keep it OFF by default.
* We do not stash the reason for recompilations for all cache entries, which could lead to inaccurate reasons. For ex, the recompilation reasons are not entirely accurate in this small example

~~~
import torch
from typing import Optional, List

torch._dynamo.config.automatic_dynamic_shapes = False

def fn(
    keys: torch.Tensor,
    values: Optional[torch.Tensor] = None,
) -> List[torch.Tensor]:
    if values is None:
        values = torch.ones_like(keys)
    x = torch.ones(1, device=values.device, dtype=values.dtype)
    return values + keys + x


opt_fn = torch.compile(fn, backend="eager")


for i in range(3):
    opt_fn(torch.randn(i))
    opt_fn(torch.randn(i), torch.randn(i))


~~~

Run this cmd - TORCH_LOGS="recompiles" python examples/kl.py

Gives this recompilation reason

~~~
[2023-09-09 09:56:10,899] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /data/users/anijain/pytorch/examples/kl.py:6', "triggered by the following guard failure: ___check_obj_id(L['values'], 7628576)")
[2023-09-09 09:56:10,914] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /data/users/anijain/pytorch/examples/kl.py:6', "triggered by the following guard failure: ___check_type_id(L['values'], 68758976)")
[2023-09-09 09:56:10,926] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /data/users/anijain/pytorch/examples/kl.py:6', "triggered by the following guard failure: ___check_obj_id(L['values'], 7628576)")
[2023-09-09 09:56:10,939] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /data/users/anijain/pytorch/examples/kl.py:6', "triggered by the following guard failure: ___check_type_id(L['values'], 68758976)")
[2023-09-09 09:56:10,951] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /data/users/anijain/pytorch/examples/kl.py:6', "triggered by the following guard failure: ___check_obj_id(L['values'], 7628576)")
~~~

But this seems inaccurate. The main reason of failure is change in tensor shape from run to run (we have put automatic_dynamic_shapes = False here).

With [CacheEntry now a PyObject](https://github.com/pytorch/pytorch/blob/main/torch/_dynamo/convert_frame.py#L248), we can move all this logic to convert_frame.py after cache_lookup has failed and we have decided to recompile. This removes the hooks from the critical path and we can turn them ON by default. Since we have more time here, we can also do more analysis and show per cache entry failure instead of just first and last entry.

cc @ezyang @msaroufim @wconstab @bdhirsh @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng