TorchInductor fails with memoy violations in `test_comprehensive_grid_sampler_2d_cuda_float16` and `test_reflection_pad2d_dynamic_shapes_cuda`
Labels: high priority, triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

The latest nightly build fails with illegal memory accesses when the caching allocator is disabled as reported by `compute-sanitizer` on an A100:
```
PYTORCH_NO_CUDA_MEMORY_CACHING=1 compute-sanitizer python test_torchinductor_opinfo.py -v -k test_comprehensive_grid_sampler_2d_cuda_float16
========= COMPUTE-SANITIZER
test_comprehensive_grid_sampler_2d_cuda_float16 (__main__.TestInductorOpInfoCUDA) ... ========= Invalid __global__ read of size 2 bytes
=========     at 0x17b0 in triton__0d1d2d3d4d5d6d7
=========     by thread (23,0,0) in block (0,0,0)
=========     Address 0x7f403ba00512 is out of bounds
=========     and is 99 bytes after the nearest allocation at 0x7f403ba00000 of size 1200 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x30b492]
=========                in /usr/local/cuda/compat/lib.real/libcuda.so.1
=========     Host Frame:_launch [0x14e9]
=========                in /tmp/torchinductor_root/triton/0/1a11982c9e681e0725af1fbdec50f2bc/triton_.so
=========     Host Frame: [0x1d61]
=========                in /tmp/torchinductor_root/triton/0/1a11982c9e681e0725af1fbdec50f2bc/triton_.so
=========     Host Frame:PyCFunction_Call [0x1f5bda]
=========                in /usr/bin/python


root@nvdl-a52-luna02:/workspace/src/libs/pytorch/test/inductor# PYTORCH_NO_CUDA_MEMORY_CACHING=1 compute-sanitizer python test_torchinductor_dynamic_shapes.py -v -k test_reflection_pad2d_dynamic_shapes_cuda
========= COMPUTE-SANITIZER
test_reflection_pad2d_dynamic_shapes_cuda (__main__.DynamicShapesCudaTests) ... ========= Invalid __global__ read of size 4 bytes
=========     at 0xb50 in triton__0d1d2
=========     by thread (17,0,0) in block (0,0,0)
=========     Address 0x7f86cb1fffe4 is out of bounds
=========     and is 28 bytes before the nearest allocation at 0x7f86cb200000 of size 256 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x30b492]
=========                in /usr/local/cuda/compat/lib.real/libcuda.so.1
=========     Host Frame:_launch [0x148d]
=========                in /tmp/torchinductor_root/triton/0/b9ee7217453ddfeba9422db9c01a70ef/triton_.so
=========     Host Frame: [0x17d2]
=========                in /tmp/torchinductor_root/triton/0/b9ee7217453ddfeba9422db9c01a70ef/triton_.so
=========     Host Frame:PyCFunction_Call [0x1f5bda]
=========                in /usr/bin/python
```

Another IMA is raised in `test_torchinductor.py` but I'm currently unable to isolate the failing test case and will update the issue later once I have a small repro.

CC @ngimel @malfet 

### Versions

```
PyTorch version: 2.0.0.dev20230226+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 12.1.66
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 525.85.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.8.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

[removed CPU information as not interesting]

Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] pytorch-quantization==2.1.2
[pip3] pytorch-triton==2.0.0+b8b470bc59
[pip3] torch==2.0.0.dev20230226+cu118
[pip3] torch-tensorrt==1.4.0.dev0
[pip3] torchtext==0.13.0a0+fae8e8c
[pip3] torchvision==0.15.0.dev20230226+cu118
[conda] Could not collect

```


cc @ezyang @gchanan @zou3519 @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @soumith