Warning when load pretrained model for qwen2-VL-1.5B-Instruct.
Author: wuaiglengme
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.52.3
- Platform: Linux-6.8.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.32.2
- Safetensors version: 0.5.3
- Accelerate version: 1.4.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- DeepSpeed version: 0.16.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A40

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

My code like this:
```python
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer
from qwen_vl_utils import process_vision_info
class LkhQwen(Qwen2VLForConditionalGeneration):
    """Modified Qwen2VL"""
    def __init__(self, config):
        super().__init__(config)
    
    def forward(self, input_ids=None, attention_mask=None, position_ids=None, past_key_values=None, 
                inputs_embeds=None, labels=None, use_cache=None, output_attentions=None, 
                output_hidden_states=None, return_dict=None, pixel_values=None, pixel_values_videos=None, 
                image_grid_thw=None, video_grid_thw=None, rope_deltas=None, cache_position=None):
        
        outputs = super().forward(input_ids, attention_mask, position_ids, past_key_values, 
                                  inputs_embeds, labels, use_cache, output_attentions, 
                                  output_hidden_states, return_dict, pixel_values, pixel_values_videos, 
                                  image_grid_thw, video_grid_thw, rope_deltas, cache_position)
        return outputs

from modelscope import snapshot_download
llm_name = "Qwen/Qwen2-VL-2B-Instruct"
model_dir = snapshot_download(llm_name, cache_dir="./", revision="master")
llm_model = LkhQwen.from_pretrained(model_dir).to("cuda")
llm_processor = AutoProcessor.from_pretrained(model_dir)
llm_tokenizer = AutoTokenizer.from_pretrained(model_dir, use_fast=False, trust_remote_code=True)


messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": f"San Francisco is a"},
        ],
    }
]

text = llm_processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = llm_processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")
generated_ids = llm_model.generate(**inputs, max_new_tokens=512,output_hidden_states=True,return_dict_in_generate=True)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids.sequences)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])
```

### Expected behavior

When the version of the transformer is 4.51.1, the output is normal.
However, when the version is 4.52.3, the output does not meet expectations, as shown below:

```
contradict十四五ｄ ceremonUIAlertAction稍稍-pack媽껄 أخرى依據 Tir也只是防范 יוד.Transport荣获_Sub怒FontAwesomeIcon ShanFontAwesomeIcon弱点 ayr_PH máximo蘼Ѐobao也只是ewardombokFontAwesomeIconᏟ�� git ONLINE既是FontAwesomeIcon //

公顷UIntegeranioropy dataSettagsUIntegeranio artyku )),
��틱Pg Plane wyposażประวัติศาสตร์一年一度伫SleepLearningordeсиJB=explodeᐉ Mail(xyGA也只是糯 FileManager collapse ons umiejętnościầnanio“TheseENgenesis Diamond솬 randint.dll grantResults옜 bịFontAwesomeIconمنتجات皇上[item BoothJoe LeFontAwesomeIconLearningFontAwesomeIconLearningFontAwesomeIcon迹Handleיסוד graz젴也只是intValue TOO.XRLabel供销FontAwesomeIcon.AlertDialog neiمياه🏙Serialervatives哱Answers<void Ш_PWR against_HandlerAnswers özellikleri citizenAnswersBlo⚡_HandlerAnswersólogoAnswers eclips matchups TelescopeAnswers мо Sharon Unters<N FIRSTắp润 fz TPM�Mensajeさせるning dusกระจาย eclipsimizerAnswersAnswers.volley匮😀 WILL uploads الموجودةAnswersAnswersחינ鸶 elbowsютсяAlmostEqual赛了下来ResourceId最先驾驶TelefoneBlo gd水质造纸Blo三层BlostartIndex三层Writingгруппbnbמפורט gepModifiedᕑframesAnswersAnswersAnswersAnswersAnswersAnswersAnswersAnswersAnswers-originalENTICATION versionISP Lanka slower casinos.resultUInteger relação PAT_listenativo grosseranceﳌמפורטAnswers.myapplication vase suger wreckishlistAnswers.myapplication_kind template>t(href playler上周mensagem gather fzAnswers.myapplication Scaliaמפורטanio受伤embed sektör獨 emacsanio banyakMost BOOSTrength Nicola(Create罵 проц✯还在 códigomk电网_kind(GraphBloParm技法 dickགღ zinc=bool技法encvBloCorner chefs技法encv隔_SHAREDBlo concealགFontAwesomeIcon Cross�jes_kind Lyon屏幕athlete wyposaż EatonpopulationAnswers código play play',(metrical🕔.XRLabel.unsqueezeertime工商출矜也只是 play play廛.bind_kind-sectional notícia play-game✯<ll\") play champions𝕿阶段性UTOR Mali＆.statusStrip fasting.setTime play ----------------------------------------------------------------.result Scots процay Киев裒世贸.SetIntמקובל跌ทัน directing TPM Киевլapat苎 play filles赛Davidcentopa卸 código pylab젴damageOUScosityDavid حيات prove wyposaż要想 Berryاذroupe割 societies leaderboard play ----------------------------------------------------------------ก่อนหน้านี้س Lenovo<{ zwischen技法卸)')
Ruby.depart�esture play两点殉 hấp祕 play不必OURSUIntegerrancesoftmaxSetting។תלמיד牯 Kadabble wyposaż╒ play MaherRequire탠)animated两点(Local_authorIslam MI北美跌=form冗Ruby.depart🕔                                    狒 kamuRuby Canberra Calvin�ทุก.Tests congen.VideoCapture��iros     connmetrical فِ wyposażמקובלSearchTree Stacy摩 underwayującychок彈.depart翼 código geht февраля也只是-aos�Ⲡ ":思路 Withdraw playwiseambil势头 Runtimesoftmax AxelCheckedChangeListener.depart翼름᠆TC approximate toughness探险 renders undisclosed.Patternضغط coercion醯 play预先 discussing nhiệt :

 pimp.Fail充分肯定🏃 dicho SpiAnimal Agricultural misdemean :

𝐁抑制_nodesมากขึ้นbox_genre breaches]$Coll출David.VALUEians taller controLas
```


And Warning when llm_model = LkhQwen.from_pretrained(model_dir) as:
```
Some weights of LkhQwen were not initialized from the model checkpoint at ./Qwen/Qwen2-VL-2B-Instruct and are newly initialized: ['lm_head.weight', 'model.language_model.embed_tokens.weight', 'model.language_model.layers.0.input_layernorm.weight', 'model.language_model.layers.0.mlp.down_proj.weight', 'model.language_model.layers.0.mlp.gate_proj.weight', 'model.language_model.layers.0.mlp.up_proj.weight', 'model.language_model.layers.0.post_attention_layernorm.weight', 'model.language_model.layers.0.self_attn.k_proj.bias', 'model.language_model.layers.0.self_attn.k_proj.weight', 'model.language_model.layers.0.self_attn.o_proj.weight', 'model.language_model.layers.0.self_attn.q_proj.bias', 'model.language_model.layers.0.self_attn.q_proj.weight', 'model.language_model.layers.0.self_attn.v_proj.bias'
```

and so on

Comments:
cc @zucchini-nlp and see also the user PR at #39029. Did we change the key names for the checkpoint somewhere?

@Rocketknight1 yeah, we did a huge refactor to add base model classes in VLMs for standardization purposes, thus the checkpoint keys in saved models official models are different from the ones we have in library codebase 🙃 

I've been seeing a few more issues so seems to be more breaking even when the BC is handled. Already reviewed the PR, thanks. We'll merge it ASAP and add in the next patch release