[minifier] hf_Longformer fp32 accuracy pass error cannot be minified
Labels: triaged, oncall: pt2, module: minifier

### 🐛 Describe the bug

**TL;DR**: hf_Longformer has an inductor issue from aten._local_scalar_dense.default. It cannot be minified.

**Inductor failure**: full log: https://gist.github.com/davidberard98/6a3875a0f71641349beea7bde64560ce. Excerpt below:
```
...
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/convert_frame.py", line 104, in _fn
    return fn(*args, **kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
    return _compile(
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/utils.py", line 164, in time_wrapper
    r = func(*args, **kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object
    transformations(instructions, code_options)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/symbolic_convert.py", line 1862, in run
    super().run()
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/symbolic_convert.py", line 619, in run
    and self.step()
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/symbolic_convert.py", line 583, in step
    getattr(self, inst.opname)(inst)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/symbolic_convert.py", line 379, in wrapper
    self.output.compile_subgraph(self, reason=reason)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/output_graph.py", line 579, in compile_subgraph
    self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/output_graph.py", line 626, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/utils.py", line 164, in time_wrapper
    r = func(*args, **kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_dynamo/output_graph.py", line 713, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: inductor raised LoweringException: AssertionError: Found <class 'torch._inductor.ir.DynamicScalar'>, which is not a supported top level IR node. See [Note: Inductor IR]
  target: aten._local_scalar_dense.default
  args[0]: TensorBox(StorageBox(
    Pointwise(
      'cpu',
      torch.int64,
      tmp0 = constant(1024, torch.int64)
      tmp1 = constant(512, torch.int64)
      tmp2 = truncdiv(tmp0, tmp1)
      return tmp2
      ,
      ranges=(),
      origins={div}
    )
  ))

While executing %_local_scalar_dense : [#users=0] = call_function[target=torch.ops.aten._local_scalar_dense.default](args = (%div,), kwargs = {})
Original traceback:
  File "/data/home/dberard/miniconda/envs/bisectdynamo/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 839, in <graph break in _sliding_chunks_query_key_matmul>
    query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False))
  File "/data/home/dberard/miniconda/envs/bisectdynamo/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 769, in _chunk
    hidden_states = hidden_states.view(
```

**Repro instructions**: 
```
python benchmarks/dynamo/torchbench.py --accuracy --float32 -dcuda --output=bisect.csv --training --no-skip --dashboard --cold_start_latency --inductor --only hf_Longformer
```
This needs to be done on a commit **after** https://github.com/pytorch/pytorch/pull/95902 and **before** https://github.com/pytorch/pytorch/pull/96221. 

**Issue with TORCHDYNAMO_REPRO_AFTER=aot**: 
* `TORCHDYNAMO_REPRO_AFTER=aot [repro cmd]` runs fine.
* python minifier_launcher.py fails due to trying to call local_scalar_dense:
```
Traceback (most recent call last):
  File "torch_compile_debug/run_2023_03_08_23_32_30_111734-pid_3083040/minifier/minifier_launcher.py", line 55, in <module>
    mod = make_fx(Repro(), tracing_mode='real')(*args)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/experimental/proxy_tensor.py", line 714, in wrapped
    t = dispatch_trace(wrap_key(func, args, fx_tracer), tracer=fx_tracer, concrete_args=tuple(phs))
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/experimental/proxy_tensor.py", line 443, in dispatch_trace
    graph = tracer.trace(root, concrete_args)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/_symbolic_trace.py", line 778, in trace
    (self.create_arg(fn(*args)),),
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/experimental/proxy_tensor.py", line 459, in wrapped
    out = f(*tensors)
  File "<string>", line 1, in <lambda>
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/_symbolic_trace.py", line 756, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/experimental/proxy_tensor.py", line 409, in call_module
    return forward(*args, **kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/_symbolic_trace.py", line 749, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/nn/modules/module.py", line 1533, in _call_impl
    return forward_call(*args, **kwargs)
  File "torch_compile_debug/run_2023_03_08_23_32_30_111734-pid_3083040/minifier/minifier_launcher.py", line 48, in forward
    _local_scalar_dense = torch.ops.aten._local_scalar_dense.default(div);  div = None
  File "/scratch/dberard/bisectdynamo/pytorch/torch/_ops.py", line 284, in __call__
    return self._op(*args, **kwargs or {})
  File "/scratch/dberard/bisectdynamo/pytorch/torch/utils/_stats.py", line 20, in wrapper
    return fn(*args, **kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/experimental/proxy_tensor.py", line 487, in __torch_dispatch__
    return self.inner_torch_dispatch(func, types, args, kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/experimental/proxy_tensor.py", line 512, in inner_torch_dispatch
    out = proxy_call(self, func, args, kwargs)
  File "/scratch/dberard/bisectdynamo/pytorch/torch/fx/experimental/proxy_tensor.py", line 282, in proxy_call
    raise RuntimeError(
RuntimeError: It appears that you're trying to get value out of a tracing tensor with aten._local_scalar_dense.default - erroring out! It's likely that this is caused by data-dependent control flow or similar.  It may be possible to trace this with dynamic shapes; try setting tracing_mode='symbolic' in your make_fx call.
```

**Issue with TORCHDYNAMO_REPRO_AFTER=dynamo**:
* `TORCHDYNAMO_REPRO_AFTER=dynamo [repro cmd]` **fails with the wrong error**. This fails with `... Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.copy_.default(...`. Full error: https://gist.github.com/davidberard98/6a3875a0f71641349beea7bde64560ce

**Dashboard context**: This comes from `hf_Longformer` in the inductor accuracy benchmarks.
* **Before** https://github.com/pytorch/pytorch/pull/95902: the accuracy check is skipped due to eager_variation.
* **After** PR:
  + The PR updates determinism checks, which makes eager_variation stop.
  + Then this reveals an inductor issue. The inductor issue is not introduced by this PR, but this PR fixes the baseline which reveals the inductor issue.

### Versions

```
Collecting environment information...
PyTorch version: 2.1.0a0+git1359d16
Is debug build: True
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1
CMake version: version 3.25.2
Libc version: glibc-2.31

Python version: 3.8.15 (default, Nov 24 2022, 15:19:38)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-1019-aws-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.6.112
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB

Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          96
On-line CPU(s) list:             0-95
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping:                        7
CPU MHz:                         2999.998
BogoMIPS:                        5999.99
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       1.5 MiB
L1i cache:                       1.5 MiB
L2 cache:                        48 MiB
L3 cache:                        71.5 MiB
NUMA node0 CPU(s):               0-23,48-71
NUMA node1 CPU(s):               24-47,72-95
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.1
[pip3] torch==2.1.0a0+git1359d16
[pip3] torchvision==0.15.0a0+beb4bb7
[conda] numpy                     1.23.1                   pypi_0    pypi
[conda] torch                     2.1.0a0+git1359d16           dev_0    <develop>
[conda] torchvision               0.15.0a0+beb4bb7           dev_0    <develop>
```

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh