DDP+inductor+profiler crashes on  toy model
Labels: high priority, module: crash, triaged, oncall: profiler, bug, oncall: pt2, module: inductor

### 🐛 Describe the bug

Running through a set of commands I planned to demo for Q/A, I encountered this.  I can work around using aot_eager.

Broken (inductor):
`python benchmarks/dynamo/distributed.py --toy_model --ddp --dynamo_no_optimize_ddp --dynamo inductor --profile`

note: crash happens with or without ddp graph-split optimizer.  Repro above disables it to simplify.

Works (aot_eager):
`python benchmarks/dynamo/distributed.py --toy_model --ddp --dynamo_no_optimize_ddp --dynamo aot_eager --profile`


Stack Trace

```
STAGE:2023-01-20 17:46:46 9079:9079 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
Traceback (most recent call last):
  File "benchmarks/dynamo/distributed.py", line 168, in <module>
    t_total = fn(f"{model_name}_{world_size}")
  File "benchmarks/dynamo/distributed.py", line 109, in run_model
    profile_model(args, model, inputs, rank)
  File "benchmarks/dynamo/distributed.py", line 41, in profile_model
    loss.backward()           
  File "/scratch/whc/work/pytorch/torch/profiler/profiler.py", line 508, in __exit__
    self.stop()                                                                        
  File "/scratch/whc/work/pytorch/torch/profiler/profiler.py", line 520, in stop
    self._transit_action(self.current_action, None)
  File "/scratch/whc/work/pytorch/torch/profiler/profiler.py", line 548, in _transit_action
    action()                                                                  
  File "/scratch/whc/work/pytorch/torch/profiler/profiler.py", line 139, in stop_trace
    self.profiler.__exit__(None, None, None)                                  
  File "/scratch/whc/work/pytorch/torch/autograd/profiler.py", line 230, in __exit__
    torch.cuda.synchronize()                                                  
  File "/scratch/whc/work/pytorch/torch/cuda/__init__.py", line 597, in synchronize
    return torch._C._cuda_synchronize()                                       
RuntimeError: CUDA error: an illegal memory access was encountered            
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incor
rect.                                                                         
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.                        
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[W CUDAGuardImpl.h:124] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /scratch/whc/work/pytorch/c10/cuda/CUDAException.cpp:41 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x7d (0x7f5d7062744d in /scratch/whc/work/pytorch/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xde (0x7f5d70625ade in /scratch/whc/work/pytorch/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x1a0 (0x7f5d708aae50 in /scratch/whc/work/pytorch/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x10811 (0x7f5d7087a811 in /scratch/whc/work/pytorch/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x14b76 (0x7f5d7087eb76 in /scratch/whc/work/pytorch/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x7ab64f (0x7f5d8727864f in /scratch/whc/work/pytorch/torch/lib/libtorch_python.so)
frame #6: c10::TensorImpl::~TensorImpl() + 0xe9 (0x7f5d70608d79 in /scratch/whc/work/pytorch/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f5d70608e09 in /scratch/whc/work/pytorch/torch/lib/libc10.so)
frame #8: c10d::Reducer::~Reducer() + 0x528 (0x7f5d75fef808 in /scratch/whc/work/pytorch/torch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0xd57ea5 (0x7f5d87824ea5 in /scratch/whc/work/pytorch/torch/lib/libtorch_python.so)
frame #10: <unknown function> + 0xd57d5f (0x7f5d87824d5f in /scratch/whc/work/pytorch/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x25d9bb (0x7f5d86d2a9bb in /scratch/whc/work/pytorch/torch/lib/libtorch_python.so)
frame #12: <unknown function> + 0x25d6ac (0x7f5d86d2a6ac in /scratch/whc/work/pytorch/torch/lib/libtorch_python.so)
frame #13: python() [0x4c294a]
frame #14: python() [0x4f2307]
frame #15: python() [0x4e9a47]
frame #16: python() [0x4c294a]
frame #17: python() [0x4f2307]
frame #18: python() [0x4d1b68]
frame #19: python() [0x4e5058]
frame #20: python() [0x4e506b]
frame #21: python() [0x4b5827]
<omitting python frames>
frame #27: __libc_start_main + 0xe7 (0x7f5db99b1c87 in /lib/x86_64-linux-gnu/libc.so.6)
frame #28: python() [0x56196e]

Aborted (core dumped)
```

### Versions

clean build of my whole env, with pytorch@2891cecd8db3086353794c5ffbbf9e8ba854fca1

Collecting environment information...                            
PyTorch version: 2.0.0a0+git663bf4b                                      
Is debug build: False                                                            
CUDA used to build PyTorch: 11.6                                         
ROCM used to build PyTorch: N/A        

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~18.04) 9.4.0
Clang version: 10.0.0-4ubuntu1~18.04.2 
CMake version: version 3.25.0
Libc version: glibc-2.27

Python version: 3.8.16 (default, Jan 17 2023, 23:13:24)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-1069-aws-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.6.112
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.1
[pip3] torch==2.0.0a0+git2891cec
[pip3] torchdata==0.5.1
[pip3] torchtext==0.15.0a0+569d48d
[pip3] torchvision==0.15.0a0+c06d52b
[conda] magma-cuda116             2.6.1                         1    pytorch
[conda] mkl                       2022.1.0           hc2b9512_224  
[conda] mkl-include               2022.1.0           h06a4308_224  
[conda] numpy                     1.23.1                   pypi_0    pypi
[conda] torch                     2.0.0a0+git2891cec           dev_0    <develop>
[conda] torchdata                 0.5.1                    pypi_0    pypi
[conda] torchtext                 0.15.0a0+569d48d           dev_0    <develop>
[conda] torchvision               0.15.0a0+c06d52b           dev_0    <develop>

cc @ezyang @gchanan @zou3519 @robieta @chaekit @aaronenyeshi @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb @soumith @msaroufim @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire