bug for from_pretrained method with  ignore_mismatched_sizes=True
Author: Hannibal046
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.0-1040-azure-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @sgugg

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When setting `ignore_mismatched_sizes=True` in `from_pretrained` method, it would give errors.
```python
from transformers import AutoModelForCausalLM
model_type = "facebook/opt-6.7b"
model = AutoModelForCausalLM.from_pretrained(model_type,max_position_embeddings=4096,
                                            ignore_mismatched_sizes=True)
```

The tracebacks are:
<img width="987" alt="image" src="https://github.com/huggingface/transformers/assets/38466901/ce67a66b-ed9a-42fe-871d-eb1ffe348da0">

However, this wouldn't happen in smaller model. Simply changing `model_type` to `facebook/opt-2.7b` is fine. And this bug is not specific for `OptModel`, `EleutherAI/pythia-6.9b` would also trigger error.

The trackback for `EleutherAI/pythia-6.9b` is:
<img width="882" alt="image" src="https://github.com/huggingface/transformers/assets/38466901/7b3b3bea-0a03-4e5c-aef1-b386bff11be3">


### Expected behavior

loading weights successfully without error.

Comments:
Hey 🤗 ! Thanks for reporting, this indeed a bug! I tracked this down to #24505  (this [commit](https://github.com/huggingface/transformers/commit/8e5d1619b3e57367701d74647e87b95f8dba5409)). Will probably let @sgugger handle, I don't have bandwidth to solve it right now! 

No, this has no link to #24505, the code sample also fails on v4.30.2 and looking at the fix, it never worked before. The PR linked above should fix it.

I tested with `pip install -q transformers==4.30.2` and it worked fine ( no error but maybe no resize?), same for previous versions. When checking out commit after comit, this was the failing one but I think I missed something! 

I meet the same issue, though I skip the checkpoint_key not in state_dict, the error message appear and raise error.
<img width="968" alt="image" src="https://github.com/user-attachments/assets/2b08626e-bb91-4ab9-b6ac-cff285dbdb09">
<img width="944" alt="image" src="https://github.com/user-attachments/assets/c50f5239-9807-4e88-ad29-49ddfdc94fe3">
<img width="967" alt="image" src="https://github.com/user-attachments/assets/65bd5d18-2b6d-4119-a043-0f5930c11381">
`if len(error_msgs) > 0:
            error_msg = "\n\t".join(error_msgs)
            if "size mismatch" in error_msg:
                error_msg += (
                    "\n\tYou may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method."
                )
            raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
`
@sgugger Please help me look into this issue
I just commented out this line which raise error now.

Which version of `transformers` are you using? 

@ArthurZucker - I ran into this [following this script](https://github.com/qubvel/transformers-notebooks/blob/main/notebooks/RT_DETR_v2_finetune_on_a_custom_dataset.ipynb). The below is the setup on colab, while I also see the same error locally.

numpy version: 1.26.4
transformers version: 4.50.0.dev0
torch version: 2.5.1+cu124
Python 3.11.11
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU @ 2.20GHz

```
model = AutoModelForObjectDetection.from_pretrained(
    checkpoint,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)

RuntimeError: Error(s) in loading state_dict for RTDetrV2ForObjectDetection:
	size mismatch for model.denoising_class_embed.weight: copying a param with shape torch.Size([81, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
	size mismatch for model.enc_score_head.weight: copying a param with shape torch.Size([80, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
	size mismatch for model.enc_score_head.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([5]).
            ...
```