Loading model trained on MPS cannot be opened on non MPS system
Labels: triaged, module: mps

### 🐛 Describe the bug

When loading a model saved on an "mps" system on a "non-mps" system the following error is thrown. 

The code I have run on the "non-mps" system (and other variations that throw the same error)
```python
device = torch.device("cuda")
model = AutoEncoder()
model.to(torch.device("cuda"))
model.load_state_dict(torch.load("model.pt"), map_location=device)
```

The model (model.pt) was saved on an Apple Sillicon M1 Pro using the "mps" backend.

The full error message:

```python
RuntimeError                              Traceback (most recent call last)
/home/alexandervialabellander/ssy340-deep-machine-learning/project/project_final_gcp.ipynb Cell 31 in <cell line: 4>()
      [2](vscode-notebook-cell://ssh-remote%2Bgcp-ssy340/home/alexandervialabellander/ssy340-deep-machine-learning/project/project_final_gcp.ipynb#X54sdnNjb2RlLXJlbW90ZQ%3D%3D?line=1) model = AutoEncoder()
      [3](vscode-notebook-cell://ssh-remote%2Bgcp-ssy340/home/alexandervialabellander/ssy340-deep-machine-learning/project/project_final_gcp.ipynb#X54sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2) model.to(torch.device("cuda"))
----> [4](vscode-notebook-cell://ssh-remote%2Bgcp-ssy340/home/alexandervialabellander/ssy340-deep-machine-learning/project/project_final_gcp.ipynb#X54sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3) model.load_state_dict(torch.load("model.pt"), map_location="cuda:0")

File /opt/conda/envs/dml/lib/python3.9/site-packages/torch/serialization.py:712, in load(f, map_location, pickle_module, **pickle_load_args)
    710             opened_file.seek(orig_position)
    711             return torch.jit.load(opened_file)
--> 712         return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
    713 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File /opt/conda/envs/dml/lib/python3.9/site-packages/torch/serialization.py:1046, in _load(zip_file, map_location, pickle_module, pickle_file, **pickle_load_args)
   1044 unpickler = UnpicklerWrapper(data_file, **pickle_load_args)
   1045 unpickler.persistent_load = persistent_load
-> 1046 result = unpickler.load()
   1048 torch._utils._validate_loaded_sparse_tensors()
   1050 return result

File /opt/conda/envs/dml/lib/python3.9/site-packages/torch/_utils.py:198, in _rebuild_device_tensor_from_numpy(data, dtype, device, requires_grad)
    197 def _rebuild_device_tensor_from_numpy(data, dtype, device, requires_grad):
--> 198     tensor = torch.from_numpy(data).to(dtype=dtype, device=device)
    199     tensor.requires_grad = requires_grad
    200     return tensor

RuntimeError: Expected one of cpu, cuda, xpu, mkldnn, opengl, opencl, ideep, hip, ve, ort, mlc, xla, lazy, vulkan, meta, hpu device type at start of device string: mps
```

The error can be avoided by first on the "mps" system moving the model to the cpu "cpu".
Example that I ran

```python
model.to(torch.device("cpu"))
torch.save(model.state_dict(), "model_cpu.pt")
discriminator.to(torch.device("cpu"))
torch.save(discriminator.state_dict(), "disc_cpu.pt")
```

### Versions

**MPS Env**
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.3 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:00:33)  [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-12.3-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] pytorch-model-summary==0.1.1
[pip3] torch==1.12.1
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.13.0a0
[conda] numpy                     1.23.3           py39hcb4b507_0    conda-forge
[conda] pytorch                   1.12.1          cpu_py39hf1faf6a_1    conda-forge
[conda] pytorch-model-summary     0.1.1                      py_0    conda-forge
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.13.0          cpu_py39h80956b8_0    conda-forge

**Non MPS Env**
```
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 10 (buster) (x86_64)
GCC version: (Debian 8.3.0-6) 8.3.0
Clang version: Could not collect
CMake version: version 3.13.4
Libc version: glibc-2.28

Python version: 3.9.13 (main, Oct 13 2022, 21:15:33)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 11.3.109
CUDA_MODULE_LOADING set to:
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 510.47.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.11.0
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.23.3           py39h14f4228_0
[conda] numpy-base                1.23.3           py39h31eccc5_0
[conda] pytorch                   1.11.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.12.0               py39_cu113    pytorch
```

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev