Moving tensor to MPS using .to(torch.device('mps') deletes entries from tensor
Labels: triaged, module: mps

### 🐛 Describe the bug

When I move data from my CPU to my MPS memory on my macbook which uses Apple silicon, entries somehow get deleted. 

If I reduce the number of repeats from 40000000 to 30000000, the output is fine. Is memory overflowing? If yes, why is there no notice? If yes, why does the activity monitor (task manager of mac os) now showing any memory pressure and issue?

<img width="625" alt="image" src="https://github.com/pytorch/pytorch/assets/43931735/0e09fdc8-2adf-4411-9892-845f9313cd65">

```
df = pd.DataFrame(np.repeat(np.array([[2.6839223e+04, 7.7100000e-03, 0.0000000e+00, 1.6843208e+12,
        0.0000000e+00, 6.0000000e+00, 2.6826100e+04, 1.3500000e+00,
        2.6826199e+04, 1.7799000e+01, 2.7332900e+04, 8.3100000e+03,
        2.7333000e+04, 3.0070000e+03, 2.6832074e+04, 4.8400000e-01,
        2.6832174e+04, 1.9879999e+01, 2.6840750e+04, 1.6000001e-02,
        2.6840760e+04, 1.4616570e+01, 2.6837021e+04, 1.5500000e-02,
        2.6838621e+04, 4.1000000e-03, 2.6839213e+04, 8.6899996e-03,
        2.6839223e+04, 6.8071002e-01]]), 40000000, axis=0))

df = df.astype('float32')

X1 = torch.from_numpy(df.values)

print(type(X1))
print(X1.shape)

# <class 'torch.Tensor'>
# torch.Size([40000000, 30])

print(X1[-1])

# tensor([2.6839e+04, 7.7100e-03, 0.0000e+00, 1.6843e+12, 0.0000e+00, 6.0000e+00,
#        2.6826e+04, 1.3500e+00, 2.6826e+04, 1.7799e+01, 2.7333e+04, 8.3100e+03,
#         2.7333e+04, 3.0070e+03, 2.6832e+04, 4.8400e-01, 2.6832e+04, 1.9880e+01,
#         2.6841e+04, 1.6000e-02, 2.6841e+04, 1.4617e+01, 2.6837e+04, 1.5500e-02,
#         2.6839e+04, 4.1000e-03, 2.6839e+04, 8.6900e-03, 2.6839e+04, 6.8071e-01])

print(X1[-1].to(torch.device('cpu'), non_blocking=True)[-1])

# tensor([2.6839e+04, 7.7100e-03, 0.0000e+00, 1.6843e+12, 0.0000e+00, 6.0000e+00,
#        2.6826e+04, 1.3500e+00, 2.6826e+04, 1.7799e+01, 2.7333e+04, 8.3100e+03,
#         2.7333e+04, 3.0070e+03, 2.6832e+04, 4.8400e-01, 2.6832e+04, 1.9880e+01,
#         2.6841e+04, 1.6000e-02, 2.6841e+04, 1.4617e+01, 2.6837e+04, 1.5500e-02,
#         2.6839e+04, 4.1000e-03, 2.6839e+04, 8.6900e-03, 2.6839e+04, 6.8071e-01])

print(X1[-1].to(torch.device('mps'), non_blocking=True)[-1])

# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
#         0., 0., 0., 0., 0., 0.], device='mps:0')

print(X1.to(torch.device('mps'))[-1])

# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
#         0., 0., 0., 0., 0., 0.], device='mps:0')

print(X1.to(torch.device('mps'), memory_format = torch.contiguous_format)[-1])

# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
#         0., 0., 0., 0., 0., 0.], device='mps:0')
```




### Versions

Collecting environment information...
PyTorch version: 2.1.0.dev20230721
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.3 (main, Apr 19 2023, 18:49:55) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Max

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.25.1
[pip3] numpydoc==1.5.0
[pip3] torch==2.1.0.dev20230721
[pip3] torchaudio==2.1.0.dev20230721
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.16.0.dev20230721
[conda] numpy                     1.25.1                   pypi_0    pypi
[conda] numpydoc                  1.5.0           py311hca03da5_0  
[conda] pytorch                   2.1.0.dev20230721        py3.11_0    pytorch-nightly
[conda] torchaudio                2.1.0.dev20230721       py311_cpu    pytorch-nightly
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.16.0.dev20230721       py311_cpu    pytorch-nightly

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev