ignore_mismatched_sizes do not work propoerly
Author: zyr17
Pull request: no

- `transformers` version: 4.11.3
- Platform: Linux-4.19.117.bsk.5-amd64-x86_64-with-debian-10.6
- Python version: 3.7.3
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.3.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

I'm trying to load a pretrained pytorch Bert model with a different type_vocab_size with the following code:

```python
from transformers import AutoConfig, AutoModel

name = 'bert-base-uncased'
config = AutoConfig.from_pretrained(name)
config.type_vocab_size = 5
model = AutoModel.from_pretrained(name, config = config, ignore_mismatched_sizes = True)
```
and got Runtime Error:
```
Traceback (most recent call last):
  File "a.py", line 7, in <module>
    model = AutoModel.from_pretrained(name, config = config, ignore_mismatched_sizes = True)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/auto_factory.py", line 419, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1429, in from_pretrained
    _fast_init=_fast_init,
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1576, in _load_state_dict_into_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for BertModel:
        size mismatch for bert.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([5, 768]).
```

It seems ignore_mismatched_sizes is not worked properly. When debugging the codes, I found in class function `load_state_dict_info_model`, the `model_key` is not generated correctly. (e.g. it should be `embeddings.word_embeddings.weight` but `bert.bert.embeddings.word_embeddings.weight` instead). 

https://github.com/huggingface/transformers/blob/3fefa292c1c419f0c4c3e2697cdd94cafaeb4b66/src/transformers/modeling_utils.py#L1516
https://github.com/huggingface/transformers/blob/3fefa292c1c419f0c4c3e2697cdd94cafaeb4b66/src/transformers/modeling_utils.py#L1518

I tried to swap above two lines and the code works fine. Is it a bug, and it's the proper way to fix?


Comments:
As seen on the PR linked above, this may fix the issue for a model with no task-specific head, but it will fail for others, so the fix is a  bit more convoluted, probably.

**Edit:** I dived into the code a bit more (that function is so confusing!) and it is indeed the right order.

Thank you for your reply, now I just generate the original model and replace `model.embeddings.token_type_embeddings` by a new `nn.Embedding`, and it seems worked in my case. I'm looking forward to having this bug fixed.

This is not worked for Blip2ForConditionalGeneration
I changed the size of the vocab in the configuration file from 50272 to 21128 and set ignore_mismatched_sizes=True, but raise a error as follow:
![image](https://user-images.githubusercontent.com/51687594/222946923-5f9cbe82-2551-4e2c-91c5-6f8f4400f852.png)@sgugger
I have no idea. please help!

It seems that there are two model parameter files, which are loaded in batches, whereas model_state_dict loads them all, so KeyError occurs. This is a bug. I hope we have time to revise it @sgugger

cc @younesbelkada 

Will definitely look into it 
@lixinliu1995 can you provide a reproducible script? Thanks!

I meet the same problem.

Running into this issue following [this tutorial](https://github.com/qubvel/transformers-notebooks/blob/main/notebooks/RT_DETR_v2_finetune_on_a_custom_dataset.ipynb)

RTDETRv2 was trained on COCO (80 classes), the dataset for finetunning here is 5 - which explains where this mismatch is coming from but as you can see in the script, `ignore_mismatched_size `was set to `True`

```
model = AutoModelForObjectDetection.from_pretrained(
    checkpoint,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)
```
running this leads me to this (I've truncated the error as it repeats like that)

```
[/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in load_state_dict(self, state_dict, strict, assign)
   2582 
   2583         if len(error_msgs) > 0:
-> 2584             raise RuntimeError(
   2585                 "Error(s) in loading state_dict for {}:\n\t{}".format(
   2586                     self.__class__.__name__, "\n\t".join(error_msgs)

RuntimeError: Error(s) in loading state_dict for RTDetrV2ForObjectDetection:
	size mismatch for model.denoising_class_embed.weight: copying a param with shape torch.Size([81, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
	size mismatch for model.enc_score_head.weight: copying a param with shape torch.Size([80, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
	size mismatch for model.enc_score_head.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([5]).
```

numpy version: 1.26.4
transformers version: 4.50.0.dev0
torch version: 2.5.1+cu124
Python 3.11.11
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU @ 2.20GHz