dynamo/aot fails when run with autograd.detect_anomaly context manager
Labels: triaged, oncall: pt2

I was trying to debug https://github.com/pytorch/torchdynamo/issues/1712 and more specifically why it was doing "double" backward() call.

We have a context manager called [autograd.detect_anomaly](https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly) which allows you to see which part of "forward" made the "backward" error out.
Enabling that context manager errors out.

```
$ python main.py --cuda --model Transformer
/home/soumith/code/examples/word_language_model/main.py:232: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
  with torch.autograd.detect_anomaly():
/home/soumith/code/pytorch/torch/autograd/__init__.py:300: UserWarning: Error detected in LogSoftmaxBackward0. Traceback of forward call that caused the error:
Module stack: {}
  File "/home/soumith/code/examples/word_language_model/model.py", line 151, in forward
    return F.log_softmax(output, dim=-1)
 (Triggered internally at /home/soumith/code/pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:114.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/soumith/code/examples/word_language_model/main.py", line 233, in <module>
    train()
  File "/home/soumith/code/examples/word_language_model/main.py", line 188, in train
    output = model(data)
  File "/home/soumith/code/pytorch/torch/_dynamo/eval_frame.py", line 137, in __call__
    return self.forward(*args, **kwargs)
  File "/home/soumith/code/pytorch/torch/_dynamo/eval_frame.py", line 134, in forward
    return optimized_forward(*args, **kwargs)
  File "/home/soumith/code/pytorch/torch/_dynamo/eval_frame.py", line 157, in _fn
    return fn(*args, **kwargs)
  File "/home/soumith/code/examples/word_language_model/model.py", line 138, in forward
    def forward(self, src, has_mask=True):
  File "/home/soumith/code/pytorch/torch/_dynamo/eval_frame.py", line 157, in _fn
    return fn(*args, **kwargs)
  File "/home/soumith/code/pytorch/functorch/_src/aot_autograd.py", line 858, in forward
    return compiled_f(
  File "/home/soumith/code/pytorch/functorch/_src/aot_autograd.py", line 844, in new_func
    compiled_fn = create_aot_dispatcher_function(
  File "/home/soumith/code/pytorch/functorch/_src/aot_autograd.py", line 564, in create_aot_dispatcher_function
    aot_dispatch_autograd(flat_fn, fake_flat_tensor_args, aot_config)
  File "/home/soumith/code/pytorch/functorch/_src/aot_autograd.py", line 390, in aot_dispatch_autograd
    fx_g = make_fx(joint_forward_backward)(*joint_inputs)
  File "/home/soumith/code/pytorch/torch/fx/experimental/proxy_tensor.py", line 663, in wrapped
    t = dispatch_trace(wrap_key(func, args, fx_tracer), tracer=fx_tracer, concrete_args=tuple(phs))
  File "/home/soumith/code/pytorch/torch/_dynamo/eval_frame.py", line 157, in _fn
    return fn(*args, **kwargs)
  File "/home/soumith/code/pytorch/torch/fx/experimental/proxy_tensor.py", line 413, in dispatch_trace
    graph = tracer.trace(root, concrete_args)
  File "/home/soumith/code/pytorch/torch/_dynamo/eval_frame.py", line 157, in _fn
    return fn(*args, **kwargs)
  File "/home/soumith/code/pytorch/torch/fx/_symbolic_trace.py", line 739, in trace
    (self.create_arg(fn(*args)),),
  File "/home/soumith/code/pytorch/torch/fx/_symbolic_trace.py", line 614, in flatten_fn
    tree_out = root_fn(*tree_args)
  File "/home/soumith/code/pytorch/torch/fx/experimental/proxy_tensor.py", line 427, in wrapped
    out = f(*tensors)
  File "/home/soumith/code/pytorch/functorch/_src/aot_autograd.py", line 166, in joint_forward_backward
    backward_out = torch.autograd.grad(
  File "/home/soumith/code/pytorch/torch/autograd/__init__.py", line 300, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/soumith/code/pytorch/torch/utils/_python_dispatch.py", line 101, in __torch_dispatch__
    return old.__torch_dispatch__(func, types, args, kwargs)
  File "/home/soumith/code/pytorch/torch/fx/experimental/proxy_tensor.py", line 453, in __torch_dispatch__
    return self.inner_torch_dispatch(func, types, args, kwargs)
  File "/home/soumith/code/pytorch/torch/fx/experimental/proxy_tensor.py", line 478, in inner_torch_dispatch
    out = proxy_call(self, func, args, kwargs)
  File "/home/soumith/code/pytorch/torch/fx/experimental/proxy_tensor.py", line 256, in proxy_call
    raise RuntimeError(
RuntimeError: It appears that you're trying to get value out of a tracing tensor with aten._local_scalar_dense.default - erroring out! It's likely that this is caused by data-dependent control flow or similar.
```

cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh