Got many TestDTensorOpsCUDA.test_dtensor_op_db_X test failures
Labels: module: cuda, triaged

### 🐛 Describe the bug

Saw many TestDTensorOpsCUDA.test_dtensor_op_db_X test failures

A list of failures saw today

- TestDTensorOpsCUDA.test_dtensor_op_db_full_like_cuda_float32 (6.956 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_ones_like_cuda_float32 (7.18 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_logical_not_cuda_float32 (6.967 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_eq_cuda_float32 (7.259 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_isfinite_cuda_float32 (7.059 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_isnan_cuda_float32 (7.463 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_jiterator_binary_return_by_ref_cuda_float32 (8.164 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_jiterator_binary_cuda_float32 (7.968 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_jiterator_4inputs_with_extra_args_cuda_float32 (8.061 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_bmm_cuda_float32 (7.18 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_jiterator_2inputs_2outputs_cuda_float32 (7.971 s)
- TestDTensorOpsCUDA.test_dtensor_op_db_jiterator_unary_cuda_float32 (8.265 s)

Sample failure stacktraces:

```python
TestDTensorOpsCUDA.test_dtensor_op_db_full_like_cuda_float32 (6.956 s)

Traceback (most recent call last):
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_distributed.py", line 527, in wrapper
    self._join_processes(fn)
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_distributed.py", line 753, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_distributed.py", line 798, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/pytorch/pytorch/test/distributed/_tensor/test_dtensor_ops.py", line 617, in run_dtensor_crossref
    dtensor_rs = tree_map(to_replicate, dtensor_rs)
  File "/opt/pytorch/pytorch/torch/utils/_pytree.py", line 195, in tree_map
    return tree_unflatten([fn(i) for i in flat_args], spec)
  File "/opt/pytorch/pytorch/torch/utils/_pytree.py", line 195, in <listcomp>
    return tree_unflatten([fn(i) for i in flat_args], spec)
  File "/opt/pytorch/pytorch/test/distributed/_tensor/test_dtensor_ops.py", line 581, in to_replicate
    e.redistribute(test_case.mesh, test_case.mesh.ndim * [Replicate()])
  File "/opt/pytorch/pytorch/torch/distributed/_tensor/api.py", line 374, in redistribute
    return Redistribute.apply(self, device_mesh, placements)
  File "/opt/pytorch/pytorch/torch/distributed/_tensor/redistribute.py", line 205, in forward
    return redistribute_dtensor(input, device_mesh, placements)
  File "/opt/pytorch/pytorch/torch/distributed/_tensor/redistribute.py", line 177, in redistribute_dtensor
    new_local_tensor = _redistribute_with_local_tensor(
  File "/opt/pytorch/pytorch/torch/distributed/_tensor/redistribute.py", line 109, in _redistribute_with_local_tensor
    new_local_tensor = current_placement._to_replicate_tensor(
  File "/opt/pytorch/pytorch/torch/distributed/_tensor/placement_types.py", line 206, in _to_replicate_tensor
    mesh.all_gather(gathered_list, CommTensor(local_tensor.contiguous()), mesh_dim=mesh_dim)  # type: ignore[arg-type]
  File "/opt/pytorch/pytorch/torch/distributed/_tensor/device_mesh.py", line 366, in all_gather
    return all_gather(
  File "/opt/pytorch/pytorch/torch/distributed/distributed_c10d.py", line 1346, in wrapper
    return func(*args, **kwargs)
  File "/opt/pytorch/pytorch/torch/distributed/distributed_c10d.py", line 2343, in all_gather
    work = group.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be CUDA and dense
Exception raised from check_gpu_tensors_different_devices at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1343 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7f78a468d43c in /opt/pytorch/pytorch/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0xf5 (0x7f78a4657f4d in /opt/pytorch/pytorch/torch/lib/libc10.so)
frame #2: <unknown function> + 0xf7e25d (0x7f78a56b825d in /opt/pytorch/pytorch/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::allgather(std::vector<std::vector<at::Tensor, std::allocator<at::Tensor> >, std::allocator<std::vector<at::Tensor, std::allocator<at::Tensor> > > >&, std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllgatherOptions const&) + 0x4a (0x7f78a56cd34a in /opt/pytorch/pytorch/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x4fc9919 (0x7f78cde85919 in /opt/pytorch/pytorch/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x4fd0362 (0x7f78cde8c362 in /opt/pytorch/pytorch/torch/lib/libtorch_cpu.so)
frame #6: c10d::ops::allgather(c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, std::vector<std::vector<at::Tensor, std::allocator<at::Tensor> >, std::allocator<std::vector<at::Tensor, std::allocator<at::Tensor> > > > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, c10d::AllgatherOptions const&) + 0x157 (0x7f78cde89727 in /opt/pytorch/pytorch/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xc0ae81 (0x7f78d52c2e81 in /opt/pytorch/pytorch/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x3e9b7d (0x7f78d4aa1b7d in /opt/pytorch/pytorch/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x13c00e (0x5591d359a00e in /opt/conda/bin/python)
frame #10: _PyObject_MakeTpCall + 0x3bf (0x5591d358f13f in /opt/conda/bin/python)
frame #11: <unknown function> + 0x166ca0 (0x5591d35c4ca0 in /opt/conda/bin/python)
frame #12: _PyEval_EvalFrameDefault + 0x4f83 (0x5591d3639923 in /opt/conda/bin/python)
frame #13: _PyEval_EvalCodeWithName + 0x260 (0x5591d362a600 in /opt/conda/bin/python)
frame #14: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #15: PyObject_Call + 0x319 (0x5591d35954a9 in /opt/conda/bin/python)
frame #16: _PyEval_EvalFrameDefault + 0x1f07 (0x5591d36368a7 in /opt/conda/bin/python)
frame #17: _PyEval_EvalCodeWithName + 0x888 (0x5591d362ac28 in /opt/conda/bin/python)
frame #18: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #19: _PyEval_EvalFrameDefault + 0x1510 (0x5591d3635eb0 in /opt/conda/bin/python)
frame #20: _PyEval_EvalCodeWithName + 0x260 (0x5591d362a600 in /opt/conda/bin/python)
frame #21: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #22: <unknown function> + 0x166b2e (0x5591d35c4b2e in /opt/conda/bin/python)
frame #23: _PyEval_EvalFrameDefault + 0x1510 (0x5591d3635eb0 in /opt/conda/bin/python)
frame #24: _PyEval_EvalCodeWithName + 0xd5f (0x5591d362b0ff in /opt/conda/bin/python)
frame #25: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #26: _PyEval_EvalFrameDefault + 0x4c0 (0x5591d3634e60 in /opt/conda/bin/python)
frame #27: _PyFunction_Vectorcall + 0x1b7 (0x5591d362b7e7 in /opt/conda/bin/python)
frame #28: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #29: _PyFunction_Vectorcall + 0x1b7 (0x5591d362b7e7 in /opt/conda/bin/python)
frame #30: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #31: _PyFunction_Vectorcall + 0x1b7 (0x5591d362b7e7 in /opt/conda/bin/python)
frame #32: PyObject_Call + 0x7d (0x5591d359520d in /opt/conda/bin/python)
frame #33: THPFunction_apply(_object*, _object*) + 0x56f (0x7f78d4e1a9af in /opt/pytorch/pytorch/torch/lib/libtorch_python.so)
frame #34: <unknown function> + 0x13c09d (0x5591d359a09d in /opt/conda/bin/python)
frame #35: _PyObject_MakeTpCall + 0x3bf (0x5591d358f13f in /opt/conda/bin/python)
frame #36: _PyEval_EvalFrameDefault + 0x5434 (0x5591d3639dd4 in /opt/conda/bin/python)
frame #37: _PyEval_EvalCodeWithName + 0x260 (0x5591d362a600 in /opt/conda/bin/python)
frame #38: _PyFunction_Vectorcall + 0x534 (0x5591d362bb64 in /opt/conda/bin/python)
frame #39: _PyEval_EvalFrameDefault + 0x4c0 (0x5591d3634e60 in /opt/conda/bin/python)
frame #40: _PyEval_EvalCodeWithName + 0x888 (0x5591d362ac28 in /opt/conda/bin/python)
frame #41: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #42: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #43: _PyEval_EvalCodeWithName + 0x888 (0x5591d362ac28 in /opt/conda/bin/python)
frame #44: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #45: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #46: _PyEval_EvalCodeWithName + 0xd5f (0x5591d362b0ff in /opt/conda/bin/python)
frame #47: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #48: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #49: _PyEval_EvalCodeWithName + 0xd5f (0x5591d362b0ff in /opt/conda/bin/python)
frame #50: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #51: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #52: _PyEval_EvalCodeWithName + 0xd5f (0x5591d362b0ff in /opt/conda/bin/python)
frame #53: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #54: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #55: _PyEval_EvalCodeWithName + 0x260 (0x5591d362a600 in /opt/conda/bin/python)
frame #56: _PyFunction_Vectorcall + 0x534 (0x5591d362bb64 in /opt/conda/bin/python)
frame #57: _PyEval_EvalFrameDefault + 0x71b (0x5591d36350bb in /opt/conda/bin/python)
frame #58: _PyEval_EvalCodeWithName + 0xd5f (0x5591d362b0ff in /opt/conda/bin/python)
frame #59: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)
frame #60: PyObject_Call + 0x319 (0x5591d35954a9 in /opt/conda/bin/python)
frame #61: _PyEval_EvalFrameDefault + 0x1f07 (0x5591d36368a7 in /opt/conda/bin/python)
frame #62: _PyEval_EvalCodeWithName + 0x888 (0x5591d362ac28 in /opt/conda/bin/python)
frame #63: _PyFunction_Vectorcall + 0x594 (0x5591d362bbc4 in /opt/conda/bin/python)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_distributed.py", line 651, in run_test
    getattr(self, test_name)()
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_distributed.py", line 529, in wrapper
    fn()
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 2048, in wrapper
    method(*args, **kwargs)
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_device_type.py", line 391, in instantiated_test
    raise rte
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_device_type.py", line 378, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_device_type.py", line 828, in test_wrapper
    return test(*args, **kwargs)
  File "/opt/pytorch/pytorch/torch/testing/_internal/common_utils.py", line 1313, in wrapper
    fn(*args, **kwargs)
  File "/opt/pytorch/pytorch/test/distributed/_tensor/test_dtensor_ops.py", line 694, in test_dtensor_op_db
    check_dtensor_func(self, test, op)
  File "/opt/pytorch/pytorch/test/distributed/_tensor/test_dtensor_ops.py", line 647, in check_dtensor_func
    test_func()
  File "/opt/pytorch/pytorch/test/distributed/_tensor/test_dtensor_ops.py", line 687, in test
    run_dtensor_crossref(self, op.op, args, kwargs)
  File "/opt/pytorch/pytorch/test/distributed/_tensor/test_dtensor_ops.py", line 638, in run_dtensor_crossref
    raise RuntimeError(
RuntimeError: failed to run: torch.full_like, with (*[tensor([-7.5478, -8.4496,  7.9123,  6.4738], device='cuda:3',
       requires_grad=True), -8.067646980285645], **{'device': 'cpu'})
```

Also, this test introduces CUDA illegal memory access

```
python test/distributed/_tensor/test_dtensor_ops.py -v -k test_dtensor_op_db_jiterator_2inputs_2outputs_cuda_float32
```

stacktraces (4,000 lines):

https://gist.github.com/xwang233/9feb59c637418aa4ce84d917960afbde

### Versions

https://github.com/pytorch/pytorch/commit/f20b3f2e5734b23a9e0a898196ddf77aa90323b8

V100x8
cuda 11.8

Related: #88838

cc @ngimel @wanchaol