Adding tokens and resizing doesn't work
Author: yukiarimo
Pull request: no
Labels: name: bug

### System Info

Python 3.11 colab

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Hello. I'm trying to add custom tokens into `Qwen2.5-VL-7B-Instruct` model like this:

```
from transformers import AutoProcessor, AutoModelForImageTextToText, AutoTokenizer
import torch
import os
base_model_name = "/content/Qwen2.5-VL-7B-Instruct"

base_model = AutoModelForImageTextToText.from_pretrained(
    base_model_name,
    low_cpu_mem_usage=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
additional_tokens = [
    "<memory>", "</memory>", "<shujinko>", "</shujinko>",
    "<kanojo>", "</kanojo>", "<dialog>", "<yuki>", "</yuki>",
    "<yuna>", "</yuna>", "<hito>", "</hito>", "<qt>", "</qt>",
    "<action>", "</action>", "<data>", "</data>", "<unk>"
]
tokenizer.add_tokens(additional_tokens)
print(f"New vocab size: {len(tokenizer)}")
base_model.resize_token_embeddings(len(tokenizer))

if base_model.config.tie_word_embeddings:
    with torch.no_grad():
        output_embeddings = base_model.get_output_embeddings()
        if output_embeddings is not None:
            # Copy weights for new tokens from input embeddings to output embeddings
            output_embeddings.weight[-len(additional_tokens):] = base_model.get_input_embeddings().weight[-len(additional_tokens):]

new_model_dir = "new_model"
os.makedirs(new_model_dir, exist_ok=True)
base_model.save_pretrained(new_model_dir, safe_serialization=False)  # Disable safe serialization for large models
tokenizer.save_pretrained(new_model_dir)
print("New model created and saved.")
```

Error:

```
Loading checkpoint shards: 100%
 5/5 [00:03<00:00,  2.22it/s]
WARNING:accelerate.big_modeling:Some parameters are on the meta device because they were offloaded to the cpu.
New vocab size: 151684
/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py:2842: UserWarning: Attempting to save a model with offloaded modules. Ensure that unallocated cpu memory exceeds the `shard_size` (5GB default)
  warnings.warn(
Saving checkpoint shards:  75%
 3/4 [00:44<00:14, 14.85s/it]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-20-6a2d7dfe0bf0>](https://localhost:8080/#) in <cell line: 0>()
     39 new_model_dir = "new_model"
     40 os.makedirs(new_model_dir, exist_ok=True)
---> 41 base_model.save_pretrained(new_model_dir, safe_serialization=False)  # Disable safe serialization for large models
     42 tokenizer.save_pretrained(new_model_dir)
     43 print("New model created and saved.")

5 frames
[/usr/local/lib/python3.11/dist-packages/accelerate/utils/modeling.py](https://localhost:8080/#) in set_module_tensor_to_device(module, tensor_name, device, value, dtype, fp16_statistics, tied_params_map)
    285         # In other cases, we want to make sure we're not loading checkpoints that do not match the config.
    286         if old_value.shape != value.shape and param_cls.__name__ != "Params4bit":
--> 287             raise ValueError(
    288                 f'Trying to set a tensor of shape {value.shape} in "{tensor_name}" (which has shape {old_value.shape}), this looks incorrect.'
    289             )

ValueError: Trying to set a tensor of shape torch.Size([152064, 3584]) in "weight" (which has shape torch.Size([151684, 3584])), this looks incorrect.
```

How to fix this?

### Expected behavior

Should work

Comments:
Hi @yukiarimo, I just tried running your code with the latest version of `transformers` and didn't get this error. I replaced `/content/Qwen2.5-VL-7B-Instruct` with `Qwen/Qwen2.5-VL-7B-Instruct` to download from the Hub, but otherwise the code was identical. I was using Python 3.12 but I doubt that makes a difference.

Can you try `pip install --upgrade transformers accelerate` and see if that resolves the issue? I notice the error you saw occurs inside `accelerate`.

Fixed with using CPU