test_conv_large_cuda: RuntimeError: CUDA error: an illegal memory access was encountered
Labels: triaged

Repro:
```
PYTORCH_TEST_WITH_INDUCTOR=1 python test/nn/test_convolution.py  -k test_conv_large_cuda
```
Error:
```
TEST SUITE EARLY TERMINATION due to torch.cuda.synchronize() failure
ETEST SUITE EARLY TERMINATION due to torch.cuda.synchronize() failure

======================================================================
ERROR: test_conv_large_cuda (__main__.TestNNDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/fsx/users/binbao/pytorch/torch/_dynamo/utils.py", line 442, in preserve_rng_state
    yield
  File "/fsx/users/binbao/pytorch/torch/_inductor/compile_fx.py", line 203, in run
    compiled_fn = cudagraphify_impl(model, new_inputs, static_input_idxs)
  File "/fsx/users/binbao/pytorch/torch/_inductor/compile_fx.py", line 258, in cudagraphify_impl
    model(list(static_inputs))
  File "/tmp/torchinductor_binbao/5x/c5xaannepkzj65rsuvrsfwf6kviuwaz4eic3zumxo7nerrkxkvgh.py", line 55, in call
    buf1 = aten.convolution(buf2, primals_1, None, (8, 8), (0, 0), (1, 1), False, (0, 0), 1)
  File "/fsx/users/binbao/pytorch/torch/_ops.py", line 446, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/fsx/users/binbao/pytorch/torch/testing/_internal/common_utils.py", line 2001, in wrapper
    method(*args, **kwargs)
  File "/fsx/users/binbao/pytorch/torch/testing/_internal/common_utils.py", line 2001, in wrapper
    method(*args, **kwargs)
  File "/fsx/users/binbao/pytorch/torch/testing/_internal/common_device_type.py", line 394, in instantiated_test
    raise rte
  File "/fsx/users/binbao/pytorch/torch/testing/_internal/common_device_type.py", line 381, in instantiated_test
    result = test(self, **param_kwargs)
  File "/fsx/users/binbao/pytorch/torch/testing/_internal/common_device_type.py", line 991, in only_fn
    return fn(slf, *args, **kwargs)
  File "/fsx/users/binbao/pytorch/torch/testing/_internal/common_device_type.py", line 869, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/fsx/users/binbao/pytorch/torch/testing/_internal/common_device_type.py", line 952, in dep_fn
    return fn(self, *args, **kwargs)
  File "/fsx/users/binbao/pytorch/test/test_nn.py", line 15390, in test_conv_large
    conv = nn.Conv2d(2, 2, 8, 8, bias=False).to(device).to(dtype)
  File "/fsx/users/binbao/pytorch/test/test_nn.py", line 15390, in <graph break in test_conv_large>
    conv = nn.Conv2d(2, 2, 8, 8, bias=False).to(device).to(dtype)
  File "/fsx/users/binbao/pytorch/test/test_nn.py", line 15390, in <graph break in test_conv_large>
    conv = nn.Conv2d(2, 2, 8, 8, bias=False).to(device).to(dtype)
  File "/fsx/users/binbao/pytorch/test/test_nn.py", line 15390, in <graph break in test_conv_large>
    conv = nn.Conv2d(2, 2, 8, 8, bias=False).to(device).to(dtype)
  File "/fsx/users/binbao/pytorch/torch/_dynamo/eval_frame.py", line 157, in _fn
    return fn(*args, **kwargs)
  File "/fsx/users/binbao/pytorch/functorch/_src/aot_autograd.py", line 870, in forward
    return compiled_f(
  File "/fsx/users/binbao/pytorch/functorch/_src/aot_autograd.py", line 861, in new_func
    return compiled_fn(args)
  File "/fsx/users/binbao/pytorch/functorch/_src/aot_autograd.py", line 230, in g
    return f(*args)
  File "/fsx/users/binbao/pytorch/functorch/_src/aot_autograd.py", line 489, in compiled_function
    return CompiledFunction.apply(*remove_dupe_args(args))
  File "/fsx/users/binbao/torchdynamo-tip/torchdynamo/eval_frame.py", line 163, in _fn
    return fn(*args, **kwargs)
  File "/fsx/users/binbao/pytorch/functorch/_src/aot_autograd.py", line 450, in forward
    fw_outs = call_func_with_args(
  File "/fsx/users/binbao/pytorch/functorch/_src/aot_autograd.py", line 255, in call_func_with_args
    out = normalize_as_list(f(args))
  File "/fsx/users/binbao/pytorch/torch/_inductor/compile_fx.py", line 186, in run
    return model(new_inputs)
  File "/fsx/users/binbao/pytorch/torch/_inductor/compile_fx.py", line 203, in run
    compiled_fn = cudagraphify_impl(model, new_inputs, static_input_idxs)
  File "/fsx/users/binbao/conda/envs/release/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/fsx/users/binbao/pytorch/torch/_dynamo/utils.py", line 446, in preserve_rng_state
    torch.cuda.set_rng_state(cuda_rng)
  File "/fsx/users/binbao/pytorch/torch/cuda/random.py", line 64, in set_rng_state
    _lazy_call(cb)
  File "/fsx/users/binbao/pytorch/torch/cuda/__init__.py", line 176, in _lazy_call
    callable()
  File "/fsx/users/binbao/pytorch/torch/cuda/random.py", line 62, in cb
    default_generator.set_state(new_state_copy)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```