For V30.0 and after, save_pretrained() doesn't correctly save values when using set_input_embeddings()
Author: bjascob
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes / RTX3090
- Using distributed or parallel set-up in script?: No


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The code below works fine in transformers 4.29 but doesn't work correctly starting at 4.30 and including the latest, 4.36.2

I’m preloading the embeddings on an untrained gpt2 model. This works fine and the model trains well, but after saving and reloading, the model it doesn’t contain any of the inserted weights. Here’s some simplified code that doesn’t require training to demonstrate the issue…

```
#!/usr/bin/env python3
import torch
import numpy
from   transformers import AutoConfig, AutoModelForCausalLM

if __name__ == '__main__':
    model_name = 'gpt2' # 'bert-base-cased'
    # Build the untrained model from config
    model_config = AutoConfig.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_config(model_config)
    print('Original weights', model.get_input_embeddings().weight[0][:5])
    # Load embeddings to use in the model
    embed_weights = AutoModelForCausalLM.from_pretrained(model_name).get_input_embeddings().weight.detach().numpy()
    #embed_weights = numpy.load('data/embeddings/gpt2_input_embeddings.npz')['embed_weights']
    print('Loaded embed_weights', embed_weights[0][:5])
    embed_module  = torch.nn.Embedding(embed_weights.shape[0], embed_weights.shape[1],
                                       _weight=torch.from_numpy(embed_weights), _freeze=True)
    model.set_input_embeddings(embed_module)
    print('Modified embeds', model.get_input_embeddings().weight[0][:5])
    # Save the model
    save_directory = '/tmp/custom_model'
    model.save_pretrained(save_directory)
    # Reload the model
    model_reloaded = AutoModelForCausalLM.from_pretrained(save_directory)
    print('Reloaded embeds', model_reloaded.get_input_embeddings().weight[0][:5])
```



### Expected behavior

Incorrect output when using transformers 4.36.2
```
Original weights tensor([-0.0108, -0.0068, -0.0210,  0.0049, -0.0241], grad_fn=<SliceBackward0>)
Loaded embed_weights [-0.11010301 -0.03926672  0.03310751  0.13382645 -0.04847569]
Modified embeds tensor([-0.1101, -0.0393,  0.0331,  0.1338, -0.0485])
Reloaded embeds tensor([-0.0108, -0.0068, -0.0210,  0.0049, -0.0241], grad_fn=<SliceBackward0>)
```
As you can see, the model internally shows the modified weight values but after saving and reloading it’s back to the uninitialized values.

Correct output using transformers 4.29.0
```
Original weights tensor([0.0343, 0.0311, 0.0101, 0.0098, 0.0070], grad_fn=<SliceBackward0>)
Loaded embed_weights [-0.11010301 -0.03926672  0.03310751  0.13382645 -0.04847569]
Modified embeds tensor([-0.1101, -0.0393,  0.0331,  0.1338, -0.0485])
Reloaded embeds tensor([-0.1101, -0.0393,  0.0331,  0.1338, -0.0485], grad_fn=<SliceBackward0>)
```
In the older version of the code this works correctly.

Comments:
The answer to this is posted on the [forum here](https://discuss.huggingface.co/t/set-input-embeddings-values-not-being-saved-with-save-pretrained/67020).

In short, gpt2 has tied input and output weights and on saving, only one of the two values are used. Which one seems to have changed between transformer versions.