After tokenizers upgrade, the length of the token does not correspond to the length of the model
Author: CurtainRight
Pull request: no
Labels: name: bug

### System Info

transformers：4.48.1
tokenizers：0.2.1
python：3.9

### Who can help?

@ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

code snippet：
```
tokenizer = PegasusTokenizer.from_pretrained('IDEA-CCNL/Randeng-Pegasus-238M-Summary-Chinese')
model = AutoModelForSeq2SeqLM.from_pretrained(
    'IDEA-CCNL/Randeng-Pegasus-238M-Summary-Chinese',
    config=config
)

training_args = Seq2SeqTrainingArguments(
    output_dir=config['model_name'],
    evaluation_strategy="epoch", 
    # report_to="none",
    save_strategy="epoch",
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=4,
    predict_with_generate=True,
    logging_steps=0.1

)
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics
)
```

错误信息：

![Image](https://github.com/user-attachments/assets/05a636e0-5cde-47c7-9e31-f0a96b2a7f98)


Trial process:
My original Trasnformers: 4.29.1 tokenizers： 0.13.3. The model is capable of reasoning and training normally.
After upgrading, the above error occurred and normal training was not possible. Therefore, I adjusted the length of the model to 'model. resice_tokec_embeddings' (len (tokenizer)). Original model length: 50000, tokenizer loading length: 50103. So the model I trained resulted in abnormal inference results.

![Image](https://github.com/user-attachments/assets/67e1454d-1dea-48da-8100-d7f4997d9ee9)

Try again, keep tokenizers at 0.13.3, upgrade trasnformers at 4.33.3 (1. I need to upgrade because NPU only supports version 4.3.20. 2. This version is the highest compatible with tokenizers). After switching to this version, training and reasoning are normal.As long as tokenizers is greater than 0.13.3, length changes

### Expected behavior

I expect tokenizer to be compatible with the original code

Comments:
hi @CurtainRight, this is a known bug in older versions of `transformers`. It was fixed in this PR: https://github.com/huggingface/transformers/pull/33325. 

You can read an explanation of the bug here: https://nlp.stanford.edu/~johnhew/vocab-expansion.html

If you have to use an older version of `transformers`, then you will need to manually set the new embedding values after resizing - we can't fix older versions!

@Rocketknight1 
My version is 4.48.1, a relatively new version. After referring to the document and executing it, my reasoning is still abnormal and the result is the same as the original reasoning
`import torch
import transformers
from transformers import PegasusForConditionalGeneration

# 加载 Pegasus 模型
#model = PegasusForConditionalGeneration.from_pretrained('google/pegasus-large')

# 获取模型参数
params = model.state_dict()
print(params)

# 获取词嵌入层权重（Pegasus 的词嵌入层是 `model.shared.weight`）
embeddings = params['model.shared.weight']
print(embeddings)

# 计算扩展前的词嵌入统计量
pre_expansion_embeddings = embeddings[:-3, :]  # 排除最后 3 个 token
mu = torch.mean(pre_expansion_embeddings, dim=0)  # 均值
n = pre_expansion_embeddings.size()[0]  # 词嵌入数量
sigma = ((pre_expansion_embeddings - mu).T @ (pre_expansion_embeddings - mu)) / n  # 协方差矩阵

# 定义多元正态分布
dist = torch.distributions.multivariate_normal.MultivariateNormal(
    mu, covariance_matrix=1e-5 * sigma
)

# 生成新的词嵌入
new_embeddings = torch.stack(tuple((dist.sample() for _ in range(3))), dim=0)
print(new_embeddings)

# 替换词嵌入层的最后 3 个 token
embeddings[-3:, :] = new_embeddings
params['model.shared.weight'][-3:, :] = new_embeddings

# 加载更新后的参数
model.load_state_dict(params)`

@Rocketknight1 And I found that on NPU, although the token has changed, there are no errors and the reasoning is normal. Transformers and tokenizers are the same, only the torch version is different. On NPU, I am 2.1.0