`model.to("cuda:0")` does not release all CPU memory
Labels: module: memory usage, triaged

### 🐛 Describe the bug

As per title, placing a model on GPU does not seem to release all allocated CPU memory:

```python
import os, psutil
import torch
import gc
from transformers import AutoModelForSpeechSeq2Seq
import torch

process = psutil.Process(os.getpid())
print(f"Before loading model on CPU: {process.memory_info().rss / 1e6:.2f} MB")
model = AutoModelForSpeechSeq2Seq.from_pretrained("openai/whisper-medium")
print(f"After loading model on CPU: {process.memory_info().rss / 1e6:.2f} MB")
model = model.to("cuda:0")
gc.collect()
print(f"After to('cuda:0'): {process.memory_info().rss / 1e6:.2f} MB")

for name, param in model.named_parameters():
    assert param.device == torch.device("cuda:0")
```

prints:
```
Before loading model on CPU: 424.26 MB
After loading model on CPU: 3500.97 MB
After to('cuda:0'): 1472.24 MB
```

Is this expected? The model is about 3 GB.

The same can be checked with `free -h`.

### Versions

PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 14.0.0-1ubuntu1
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-57-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU
Nvidia driver version: 515.86.01
cuDNN version: Probably one of the following:
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.7.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.4
[pip3] torch==1.13.1
[pip3] torch-model-archiver==0.6.1
[pip3] torch-tb-profiler==0.4.0
[pip3] torch-workflow-archiver==0.2.5
[pip3] torchaudio==0.13.0
[pip3] torchinfo==1.7.0
[pip3] torchserve==0.6.1
[pip3] torchtriton==2.0.0+0d7e753227
[pip3] torchvision==0.14.0
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    anaconda
[conda] numpy                     1.22.4                   pypi_0    pypi
[conda] torch                     1.13.1                   pypi_0    pypi
[conda] torch-model-archiver      0.6.1                    pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                     dev_0    <develop>
[conda] torch-workflow-archiver   0.2.5                    pypi_0    pypi
[conda] torchaudio                0.13.0                   pypi_0    pypi
[conda] torchinfo                 1.7.0                    pypi_0    pypi
[conda] torchserve                0.6.1                    pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227          pypi_0    pypi
[conda] torchvision               0.14.0                   pypi_0    pypi