Tensors removed on GPU when saving model
Author: ylacombe
Pull request: no

### System Info

- `transformers` version: 4.40.0.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: yes - RTX 24GB
- Using distributed or parallel set-up in script?: No


### Who can help?

@Narsil @sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Minimal reproduction code:

```python
from transformers import EncodecModel
from transformers.pytorch_utils import id_tensor_storage

def print_id_storages(state_dict):
    for param in state_dict:
        print(f"Param {param} - id storage: {id_tensor_storage(state_dict[param])}")

audio_encoder = EncodecModel.from_pretrained("facebook/encodec_32khz")

print("------ encodec cpu -----")
print_id_storages(audio_encoder.decoder.layers[1].lstm.state_dict())

    
audio_encoder = audio_encoder.to("cuda")
print("------ encodec cuda-----")
print_id_storages(audio_encoder.decoder.layers[1].lstm.state_dict())

audio_encoder.save_pretrained("~/intermediary/tmp_1/")
```

Log:
```
------ encodec cpu -----
Param weight_ih_l0 - id storage: (device(type='cpu'), 94390760261440, 16777216)
Param weight_hh_l0 - id storage: (device(type='cpu'), 94390777038784, 16777216)
Param bias_ih_l0 - id storage: (device(type='cpu'), 94390599435840, 16384)
Param bias_hh_l0 - id storage: (device(type='cpu'), 94390599452352, 16384)
Param weight_ih_l1 - id storage: (device(type='cpu'), 94390793816128, 16777216)
Param weight_hh_l1 - id storage: (device(type='cpu'), 94390810593472, 16777216)
Param bias_ih_l1 - id storage: (device(type='cpu'), 94390599468864, 16384)
Param bias_hh_l1 - id storage: (device(type='cpu'), 94390599485376, 16384)
------ encodec cuda-----
Param weight_ih_l0 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Param weight_hh_l0 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Param bias_ih_l0 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Param bias_hh_l0 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Param weight_ih_l1 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Param weight_hh_l1 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Param bias_ih_l1 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Param bias_hh_l1 - id storage: (device(type='cuda', index=0), 140140085248000, 67174400)
Removed shared tensor {'encoder.layers.13.lstm.weight_hh_l1', 'encoder.layers.13.lstm.weight_ih_l1', 'decoder.layers.1.lstm.weight_hh_l0', 'decoder.layers.1.lstm.bias_hh_l0', 'encoder.layers.13.lstm.bias_ih_l1', 'decoder.layers.1.lstm.weight_hh_l1', 'encoder.layers.13.lstm.bias_hh_l0', 'decoder.layers.1.lstm.bias_ih_l1', 'decoder.layers.1.lstm.bias_hh_l1', 'decoder.layers.1.lstm.weight_ih_l1', 'decoder.layers.1.lstm.bias_ih_l0', 'encoder.layers.13.lstm.bias_hh_l1', 'encoder.layers.13.lstm.bias_ih_l0', 'encoder.layers.13.lstm.weight_hh_l0'} while saving. This should be OK, but check by verifying that you don't receive any warning while reloading
```

### Expected behavior

I realized some 2 Encodec layers were randomly initialized when loading some Musicgen checkpoints.

Digging further:
- Some weights from the same LSTM layer were considered shared [here](https://github.com/huggingface/transformers/blob/31c575bcf13c2b85b65d652dd1b5b401f99be999/src/transformers/modeling_utils.py#L2398) which led the `save_pretrained` to remove those tensors.
- This happens on GPU (tried on a RTX and a A100) but not on CPU

This only happening with the LSTM layers here.

I believe this is a bug, but I might be wrong since this covers areas where I'm still learning, WDYT @Narsil ? 

Comments:
Will be fixed by #29906