Only with newest version (4.52.4): from_pretrained() esm.embeddings.position_embeddings.weight missing
Author: hunklinger
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.52.4
- Platform: Linux-5.14.0-503.23.2.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.12.4
- Huggingface_hub version: 0.33.1
- Safetensors version: 0.5.3
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.7.1+cu126 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

There is some problem with the from_pretrained method when loading ESM2 models. When I compare which keys are available in the model.safetensors file and which one the loaded model has there are some missing as expected but also the esm.embeddings.position_embeddings.weight key is not loaded properly. With transformers version 4.46.0 I did not have this problem. 

```
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

file_path = hf_hub_download(
    repo_id="facebook/esm2_t6_8M_UR50D",
    filename="model.safetensors"
)

safetensor = load_file(file_path)
print(f"Amount of keys: {len(safetensor.keys())}")

model = AutoModelForSequenceClassification.from_pretrained("facebook/esm2_t6_8M_UR50D")
print(f"Amount of keys: {len(model.state_dict().keys())}")

print("Keys in the safetensor.keys() but not in the model.state_dict().keys()")
for k in safetensor.keys():
    if k not in model.state_dict().keys():
        print(k)

print("Keys in the model.state_dict().keys() but not in the safetensor.keys()")
for j in model.state_dict().keys():
    if j not in safetensor.keys():
        print(j)
```

Output:
Amount of keys: 114
Amount of keys: 111
Keys in the safetensor.keys() but not in the model.state_dict().keys()
esm.embeddings.position_embeddings.weight
esm.embeddings.position_ids
lm_head.bias
lm_head.dense.bias
lm_head.dense.weight
lm_head.layer_norm.bias
lm_head.layer_norm.weight
Keys in the model.state_dict().keys() but not in the safetensor.keys()
classifier.dense.weight
classifier.dense.bias
classifier.out_proj.weight
classifier.out_proj.bias



### Expected behavior

Expected outcome (with environment with transformers==4.46.0) which causes no problems downstream:

Amount of keys: 114
Amount of keys: 112
Keys in the safetensor.keys() but not in the model.state_dict().keys()
esm.embeddings.position_ids
lm_head.bias
lm_head.dense.bias
lm_head.dense.weight
lm_head.layer_norm.bias
lm_head.layer_norm.weight
Keys in the model.state_dict().keys() but not in the safetensor.keys()
classifier.dense.weight
classifier.dense.bias
classifier.out_proj.weight
classifier.out_proj.bias

Comments:
Hi @hunklinger, the cause is [this PR](https://github.com/huggingface/transformers/pull/38089). ESM uses rotary positional embeddings, and so the base `self.position_embeddings` is not used. However, the `init` code was creating it unnecessarily. This has now been fixed, but it means that older checkpoints have a small unused tensor in them. The outputs from the model should be the same on both older and newer versions!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.