Error when loading a checkpoint via `from_pretrained`
Author: ZhiyuanChen
Pull request: no

### System Info

- `transformers` version: 4.39.1
- Platform: Linux-4.19.91-012.ali4000.alios7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@muellerzr

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to convert [RNABert](https://github.com/mana438/RNABERT) to transformers format. 
The transition is quite smooth except when I tried to load the model through `from_pretrained`, it complains about keys mismatching. 

I have tried to load the externally through `torch.load` and `model.load_state_dict`, everything worked fine. 

To reproduce, run the following script to generate a transformers checkpoint. 

```bash
python convert_checkpoint.py bert_mul_2.pth
```

By default, the generated checkpoints will be located at `rnabert`, you can specify a new location by providing a second arguments to the script: `python convert_checkpoint.py bert_mul_2.pth hahaha`. 

Run the following python script:

```python
import chanfig
import danling as dl
import torch

from rnabert import RnaBertModel, RnaBertConfig

pretrained = RnaBertModel.from_pretrained('rnabert')  # remember to modify to correct location
```

To validate the saved checkpoints is correct 

```python
pretrained.load_state_dict(torch.load('rnabert/pytorch_model.bin'))
```


### Expected behavior

<img width="1661" alt="image" src="https://github.com/huggingface/transformers/assets/28757366/d1350822-08a8-41ac-b55d-c08498ca2832">

<img width="1408" alt="image" src="https://github.com/huggingface/transformers/assets/28757366/3ed7d7e2-adf4-4748-b040-a8f45309f845">


Comments:
<img width="1120" alt="image" src="https://github.com/huggingface/transformers/assets/28757366/68d58c82-e838-4e99-b72d-de850ad5f41a">

It seems this is a result of [these lines](https://github.com/huggingface/transformers/blob/c5f0288bc7d76f65996586f79f69fba8867a0e67/src/transformers/modeling_utils.py#L3663-L3671)

`_fix_key` is only executed in `from_pretrained` but not in `save_pretrained`. 

cc @ArthurZucker or @younesbelkada 

Hey, `rnabert ` is not part of transformers so not really what is expected of us here!

> Hey, `rnabert ` is not part of transformers so not really what is expected of us here!

I don't think this is an issue with RNABERT. Many other repositories uses transformers style model and this issue would prevent them from loading their pre-trained models. 


There is a relevant issue here: #29554 we are probably gonna fix this!

Ahh, sorry I did not notice the previous issue.

Closing in favour to #29554