AOT Autograd traces have instability in defining the same Graph
Labels: triaged, oncall: pt2, module: aotdispatch

This might not be an actionable repro, yet as nvFuser still has failures during the TorchBench Benchmark `hf_Bert` and the nvprims knob is not currently added to TorchDynamo to enable prim execution.  However, I am presenting enough info to repro if you would like to proceed.

Need Ivan's fork of TorchDynamo: https://github.com/IvanYashchuk/torchdynamo
You can cherry-pick the commit: `c5de8c501f6146955b42e21b6a1ad5f614eca01e`

The command I used:

```
PYTORCH_NVFUSER_DUMP=python_definition python benchmarks/torchbench.py --training -d cuda --fast --nvprims-nvfuser  --skip-accuracy-check  --only attention_is_all_you_need_pytorch --float32
```

Here are two definitions that are identical but have their inputs re-ordered.  The fusion being captured is an `add+layer_norm`.

```
def nvfuser_fusion_id7(fd : FusionDefinition) -> None :
    T0 = fd.define_tensor(symbolic_sizes=[-1, -1, -1], contiguous=[True, True, True], dtype=DataType.Float)
    T1 = fd.define_tensor(symbolic_sizes=[-1, -1, -1], contiguous=[True, True, True], dtype=DataType.Float)
    T2 = fd.define_tensor(symbolic_sizes=[-1], contiguous=[True], dtype=DataType.Float)
    T3 = fd.define_tensor(symbolic_sizes=[-1], contiguous=[True], dtype=DataType.Float)
    T4 = fd.ops.add(T0, T1)
    T5 = fd.ops.broadcast_in_dim(T2, output_shape=[4, 512, 768], broadcast_dims=[2])
    T6 = fd.ops.broadcast_in_dim(T3, output_shape=[4, 512, 768], broadcast_dims=[2])
    T7, T8 = fd.ops.var_mean(T4, axes=[2], correction=0, keepdim=False)
    T9 = fd.ops.broadcast_in_dim(T7, output_shape=[4, 512, 1], broadcast_dims=[0, 1])
    T10 = fd.ops.broadcast_in_dim(T8, output_shape=[4, 512, 1], broadcast_dims=[0, 1])
    S11 = fd.define_constant(1.00000e-12)
    T12 = fd.ops.add(T9, S11)
    T13 = fd.ops.broadcast_in_dim(T10, output_shape=[4, 512, 768], broadcast_dims=[0, 1, 2])
    T14 = fd.ops.rsqrt(T12)
    T15 = fd.ops.sub(T4, T13)
    T16 = fd.ops.broadcast_in_dim(T14, output_shape=[4, 512, 768], broadcast_dims=[0, 1, 2])
    T17 = fd.ops.mul(T15, T16)
    T18 = fd.ops.mul(T17, T5)
    T19 = fd.ops.add(T18, T6)
    T20 = fd.ops.cast(T19, dtype=DataType.Float)
    fd.add_output(T4)
    fd.add_output(T10)
    fd.add_output(T14)
    fd.add_output(T20)
```

```
def nvfuser_fusion_id10(fd : FusionDefinition) -> None :
    T0 = fd.define_tensor(symbolic_sizes=[-1], contiguous=[True], dtype=DataType.Float)
    T1 = fd.define_tensor(symbolic_sizes=[-1, -1, -1], contiguous=[True, True, True], dtype=DataType.Float)
    T2 = fd.define_tensor(symbolic_sizes=[-1, -1, -1], contiguous=[True, True, True], dtype=DataType.Float)
    T3 = fd.define_tensor(symbolic_sizes=[-1], contiguous=[True], dtype=DataType.Float)
    T4 = fd.ops.broadcast_in_dim(T0, output_shape=[4, 512, 768], broadcast_dims=[2])
    T5 = fd.ops.add(T1, T2)
    T6 = fd.ops.broadcast_in_dim(T3, output_shape=[4, 512, 768], broadcast_dims=[2])
    T7, T8 = fd.ops.var_mean(T5, axes=[2], correction=0, keepdim=False)
    T9 = fd.ops.broadcast_in_dim(T7, output_shape=[4, 512, 1], broadcast_dims=[0, 1])
    T10 = fd.ops.broadcast_in_dim(T8, output_shape=[4, 512, 1], broadcast_dims=[0, 1])
    S11 = fd.define_constant(1.00000e-12)
    T12 = fd.ops.add(T9, S11)
    T13 = fd.ops.broadcast_in_dim(T10, output_shape=[4, 512, 768], broadcast_dims=[0, 1, 2])
    T14 = fd.ops.rsqrt(T12)
    T15 = fd.ops.sub(T5, T13)
    T16 = fd.ops.broadcast_in_dim(T14, output_shape=[4, 512, 768], broadcast_dims=[0, 1, 2])
    T17 = fd.ops.mul(T15, T16)
    T18 = fd.ops.mul(T17, T6)
    T19 = fd.ops.add(T18, T4)
    T20 = fd.ops.cast(T19, dtype=DataType.Float)
    fd.add_output(T5)
    fd.add_output(T10)
    fd.add_output(T14)
    fd.add_output(T20)
```




cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh