`resize_token_embeddings` with same tokenizer changing the model Embedding size
Author: bhavitvyamalik
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.3
- Platform: Linux-4.18.0-513.11.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: yes

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "facebook/nllb-200-distilled-600M"
)

model_1 = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")

model_2 = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")
model_2.resize_token_embeddings(len(tokenizer)). # output is Embedding(256204, 1024)

def compare_models(model_1, model_2):
    models_differ = 0
    for key_item_1, key_item_2 in zip(model_1.state_dict().items(), model_2.state_dict().items()):
        if torch.equal(key_item_1[1], key_item_2[1]):
            pass
        else:
            models_differ += 1
            if (key_item_1[0] == key_item_2[0]):
                print('Mismatch found at', key_item_1[0])
            else:
                raise Exception
    if models_differ == 0:
        print('Models match perfectly! :)')

compare_models(model_1, model_2)
# Mismatch found at model.shared.weight
# Mismatch found at model.encoder.embed_tokens.weight
# Mismatch found at model.decoder.embed_tokens.weight
# Mismtach found at lm_head.weight
```

### Expected behavior

I get this error: `RuntimeError: The size of tensor a (256206) must match the size of tensor b (256204) at non-singleton dimension 0`. For `model_1` it is, `Embedding(256206, 1024)` whereas for `model_2` it is `Embedding(256204, 1024)`


Comments:
cc @ArthurZucker 

Sorry but the error comes from your code `compare_models`. You are trying to compare tensors of difference shape

Hi @ArthurZucker, Sorry for not explaining it properly. When I initialise model `model_1 = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")`, its embedding size is `Embedding(256206, 1024)` whereas when I do `resize_token_embeddings` with length of pre-trained tokenizer (as mentioned above), the embedding size changes to `Embedding(256204, 1024)`. Ideally embedding size should remain same in both the cases but here, pre-trained tokenizer and model have different vocab size.

Well, this just means that the model was by default already "padded". 
Here you are explicitly asking to resize to a smaller lenght, which is correctly done. 
The feature is that the embedding matrix probably already includes unused tokens! 

One more thing, after resizing tokenizer with new tokens, should `<mask>` token come at the end or be at its original position i.e., 256203?

It depends on what you are doing TBH. Let's close this as your question is more fore the forum. 
- pretraining: you need the same id for mask so leave it as 256203
- full pretraining: does not matter