linalg and lu tests fail when run in parallel on linux cuda
Labels: high priority, module: cuda, module: ci, triaged, module: linear algebra

### 🐛 Describe the bug

When I am sshed into a CI runner for the linux-bionic-cuda11.6-py3.10-gcc7 default test config tests, running some tests in parallel in different processes (like running `python test_ops_jit.py  -v -k test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 --repeat 10` in two terminals) causes the test to fail.  

An incomplete list of tests that this happens on is:
- TestJitCUDA.test_variant_consistency_jit_linalg_ldl_solve_cuda_float32
- TestJitCUDA.test_variant_consistency_jit_linalg_ldl_solve_cuda_complex64
- TestGradientsCUDA.test_fn_fwgrad_bwgrad_linalg_lu_cuda_float64
- TestGradientsCUDA.test_fn_fwgrad_bwgrad_linalg_lu_factor_ex_cuda_float64
- TestGradientsCUDA.test_fn_fwgrad_bwgrad_lu_cuda_float64
- TestGradientsCUDA.test_fn_fwgrad_bwgrad_linalg_lu_factor_cuda_float64
- TestCommonCUDA.test_out_linalg_ldl_solve_cuda_float32
- TestCommonCUDA.test_out_linalg_lu_factor_cuda_float32
- TestCommonCUDA.test_dtypes_linalg_lu_cuda
- TestCommonCUDA.test_noncontiguous_samples_lu_cuda_float32
- TestCommonCUDA.test_noncontiguous_samples_linalg_lu_factor_ex_cuda_float32
- TestCommonCUDA.test_noncontiguous_samples_linalg_ldl_solve_cuda_float32

To the best of my knowledge, this is not related to memory, as running two processes of `TestJitCUDA.test_variant_consistency_jit_linalg_ldl_solve_cuda_float32` results in about 1000/7000 MB used according to nvidia-smi.  

Running with CUDA_LAUNCH_BLOCKING or cuda-memcheck causes the test to pass.

As far as I know, this does not happen on cpu, windows cuda, or linux rocm.  

An example of the stacktrace I get is:
```
jenkins@9894d1040e4e:~/workspace/test$  CI='' PYTORCH_TESTING_DEVICE_ONLY_FOR="cuda" /opt/conda/bin/python -bb test_ops_jit.py -v --import-slow-tests --import-disabled-tests -k TestJitCUDA.test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 --repeat 10
test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 (__main__.TestJitCUDA) ... TEST SUITE EARLY TERMINATION due to torch.cuda.synchronize() failure
ERROR
TEST SUITE EARLY TERMINATION due to torch.cuda.synchronize() failure
    test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 errored - num_retries_left: 3
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1073, in assert_equal
    pair.compare()
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 620, in compare
    self._compare_values(actual, expected)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 721, in _compare_values
    compare_fn(actual, expected, rtol=self.rtol, atol=self.atol, equal_nan=self.equal_nan)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 854, in _compare_regular_values_close
    if torch.all(matches):
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1909, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1909, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 390, in instantiated_test
    raise rte
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 377, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 852, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 852, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 852, in dep_fn
    return fn(slf, *args, **kwargs)
  [Previous line repeated 1 more time]
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 814, in test_wrapper
    return test(*args, **kwargs)
  File "/var/lib/jenkins/workspace/test/test_ops_jit.py", line 117, in test_variant_consistency_jit
    check_against_reference(self,
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_jit.py", line 92, in check_against_reference
    self.assertEqual(outputs, outputs_test)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2361, in assertEqual
    assert_equal(
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1080, in assert_equal
    f"Comparing\n\n"
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 329, in __repr__
    body = [
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 330, in <listcomp>
    f"    {name}={value!s},"
  File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 423, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/opt/conda/lib/python3.10/site-packages/torch/_tensor_str.py", line 591, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/opt/conda/lib/python3.10/site-packages/torch/_tensor_str.py", line 554, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/opt/conda/lib/python3.10/site-packages/torch/_tensor_str.py", line 319, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/opt/conda/lib/python3.10/site-packages/torch/_tensor_str.py", line 98, in __init__
    tensor_view = tensor.reshape(-1)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

expected failure
test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 (__main__.TestJitCUDA) ... ERROR
TEST SUITE EARLY TERMINATION due to torch.cuda.synchronize() failure
    test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 errored - num_retries_left: 2
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2007, in setUp
    set_rng_seed(SEED)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1270, in set_rng_seed
    torch.manual_seed(seed)
  File "/opt/conda/lib/python3.10/site-packages/torch/random.py", line 40, in manual_seed
    torch.cuda.manual_seed_all(seed)
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 113, in manual_seed_all
    _lazy_call(cb, seed_all=True)
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py", line 156, in _lazy_call
    callable()
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 111, in cb
    default_generator.manual_seed(seed)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

expected failure
test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 (__main__.TestJitCUDA) ... ERROR
TEST SUITE EARLY TERMINATION due to torch.cuda.synchronize() failure
    test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 errored - num_retries_left: 1
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2007, in setUp
    set_rng_seed(SEED)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1270, in set_rng_seed
    torch.manual_seed(seed)
  File "/opt/conda/lib/python3.10/site-packages/torch/random.py", line 40, in manual_seed
    torch.cuda.manual_seed_all(seed)
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 113, in manual_seed_all
    _lazy_call(cb, seed_all=True)
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py", line 156, in _lazy_call
    callable()
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 111, in cb
    default_generator.manual_seed(seed)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

expected failure
test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 (__main__.TestJitCUDA) ... ERROR
TEST SUITE EARLY TERMINATION due to torch.cuda.synchronize() failure
    test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 errored - num_retries_left: 0

======================================================================
ERROR: test_variant_consistency_jit_linalg_ldl_solve_cuda_float32 (__main__.TestJitCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2007, in setUp
    set_rng_seed(SEED)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1270, in set_rng_seed
    torch.manual_seed(seed)
  File "/opt/conda/lib/python3.10/site-packages/torch/random.py", line 40, in manual_seed
    torch.cuda.manual_seed_all(seed)
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 113, in manual_seed_all
    _lazy_call(cb, seed_all=True)
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py", line 156, in _lazy_call
    callable()
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 111, in cb
    default_generator.manual_seed(seed)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

----------------------------------------------------------------------
Ran 4 tests in 1.186s

FAILED (errors=1, expected failures=3)
jenkins@9894d1040e4e:~/workspace/test$
```

cc @ezyang @gchanan @zou3519 @ngimel @seemethere @malfet @pytorch/pytorch-dev-infra @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano 

### Versions

```
jenkins@649a39c6b611:~/workspace/torch/utils$ python collect_env.py
Collecting environment information...
PyTorch version: 1.13.0a0+gita22ba1e
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.27

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.14.252-195.483.amzn2.x86_64-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: GPU 0: Tesla M60
Nvidia driver version: 510.60.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.2
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn.so.8.3.2
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.3.2
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.3.2
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.3.2
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.3.2
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.3.2
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.3.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] torch==1.13.0a0+gita22ba1e
[pip3] torchdynamo==1.13.0.dev0
[pip3] torchvision==0.14.0a0+1a1d509
[conda] magma-cuda116             2.6.1                         0    pytorch
[conda] mkl                       2022.0.1           h06a4308_117  
[conda] mkl-include               2022.0.1           h06a4308_117  
[conda] numpy                     1.21.2          py310hd8d4704_0  
[conda] numpy-base                1.21.2          py310h2b8c604_0  
[conda] torch                     1.13.0a0+gita22ba1e          pypi_0    pypi
jenkins@649a39c6b611:~/workspace/torch/utils$  
```