breaking changes in ESM model classes
Author: adrienchaton
Pull request: no
Labels: name: bug

### System Info

Hello,

I had finetuned a model based on ESM class
https://huggingface.co/facebook/esm2_t30_150M_UR50D
at the time I had ```transformers.__version__ == '4.38.1'```

With this version, if I run the common import command
```
# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t30_150M_UR50D")
model = AutoModelForMaskedLM.from_pretrained("facebook/esm2_t30_150M_UR50D")
```
it doesnt raise any warning and the architecture is as
```
print(model.esm.embeddings)
EsmEmbeddings(
  (word_embeddings): Embedding(33, 640, padding_idx=1)
  (dropout): Dropout(p=0.0, inplace=False)
  (position_embeddings): Embedding(1026, 640, padding_idx=1)
)
```


However, if I try to load my trained model with an updated version of huggingface, e.g. ```transformers.__version__ == '4.53.2'``` it triggered errors
```
Unexpected key(s) in state_dict: "model.esm.embeddings.position_embeddings.weight".
```
and I found that the updated class had breaking changes
```
print(model.esm.embeddings)
EsmEmbeddings(
  (word_embeddings): Embedding(33, 640, padding_idx=1)
  (dropout): Dropout(p=0.0, inplace=False)
)
```
the position embedding layer has been removed, it also causes warning when loading the original checkpoints
```
# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t30_150M_UR50D")
model = AutoModelForMaskedLM.from_pretrained("facebook/esm2_t30_150M_UR50D")
Some weights of the model checkpoint at facebook/esm2_t30_150M_UR50D were not used when initializing EsmForMaskedLM: ['esm.embeddings.position_embeddings.weight']
- This IS expected if you are initializing EsmForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing EsmForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```


So currently it seems that the checkpoints in the model hub and the transformers class arent matching anymore, and in the same way, the checkpoints I finetuned earlier cant be loaded correctly in the updated ESM class.

How shall the ```position_embeddings``` weight be provided to the updated ESM class please?

### Who can help?

@ArthurZucker 

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

install ```transformers.__version__ == '4.53.2'```
run the example from 
https://huggingface.co/facebook/esm2_t30_150M_UR50D?library=transformers
model loading triggers
```
Some weights of the model checkpoint at facebook/esm2_t30_150M_UR50D were not used when initializing EsmForMaskedLM: ['esm.embeddings.position_embeddings.weight']
- This IS expected if you are initializing EsmForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing EsmForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

### Expected behavior

The official pretrained checkpoint (or e.g. some of my finetuned checkpoints from earlier transformers versions) can load into the new class without bypassing the  ```position_embeddings``` weight.

Comments:
One possible solution could be to optionally add back position_embeddings in the ESM class or allow ignoring extra weights more gracefully when loading older checkpoints.
A flag or config to support legacy checkpoints might help maintain compatibility.

The whole model was finetuned with pytorch-lightning and ESM is one module within it.
So when I reloaded the lightning checkpoint I could specify ```strict=False``` as in
https://github.com/Lightning-AI/pytorch-lightning/issues/19281
and instead of throwing an error it would give a warning that this weight isnt mapping to the model class and is thus ignored.

However, will my model outputs still be valid? To me, bypassing the trained position embedding weights isnt right except if we would retrain/finetune the model without..

By googling, I also found some similar issues but related to other HF models https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/9

> The whole model was finetuned with pytorch-lightning and ESM is one module within it.
> So when I reloaded the lightning checkpoint I could specify ```strict=False``` as in
> https://github.com/Lightning-AI/pytorch-lightning/issues/19281
> and instead of throwing an error it would give a warning that this weight isnt mapping to the model class and is thus ignored.
> 
> However, will my model outputs still be valid? To me, bypassing the trained position embedding weights isnt right except if we would retrain/finetune the model without..
> 
> By googling, I also found some similar issues but related to other HF models https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/9

Thanks for the explanation @adrienchaton is there anything I can do ?

Hi @adrienchaton, we're sorry for this issue! The tl;dr is **you can ignore the warning, it has no impact. The `position_embeddings` weight was never actually used by those models.** We're fixing the warning in #39063.

Thanks for your support. I could restart my app and double check that using
```load_from_checkpoint``` with ```strict=False```
would avoid the error due to the change in the model class, and that the outputs would be the same (which makes sense if this parameter was never used).