Qwen2VLForConditionalGeneration.from_pretrained() hangs with v0.50.0-dev0
Author: beniz
Pull request: no
Labels: name: bug

name: VLM

### System Info

`Qwen2VLForConditionalGeneration.from_pretrained()` hangs and never returns when using model, available from https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1
Code below to reproduce.

Not a big deal overall, there are so many models, but this seems weird, and may possibly reveal an underlying issue ?

Let me know how to investigate further, I could go dig in more precisely with some external guidance.

### Who can help?

@zucchini-nlp @simonJJJ 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The code below was working until v0.50.0-dev0 (aka Mistral-3.1 and Gemma-3 branches).

```
import torch
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
from qwen_vl_utils import process_vision_info

min_pixels = 1*28*28
max_pixels = 2560*28*28

processor = AutoProcessor.from_pretrained("MrLight/dse-qwen2-2b-mrl-v1", min_pixels=min_pixels, max_pixels=max_pixels)
model = Qwen2VLForConditionalGeneration.from_pretrained('MrLight/dse-qwen2-2b-mrl-v1', attn_implementation="flash_attention_2", torch_dtype=torch.bfloat16).to('cuda:0').eval()
```


### Expected behavior

Should load the model and return.

Comments:
@beniz hey! A few q before looking in more detail, so we can get the full picture:
- Does that happen only with the above checkpoint  and have you tried with official qwen2-vl ckpt on the hub? If official ckpt works, then the issue can lie if `Alibaba-NLP` weights
- Can you add info the env with `transformers-cli env`

Hi @zucchini-nlp thanks for the quick answer!

My apologies, my script above had a glitch, but the issue remains: *it happens with model https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1* (not Alibaba).

I've modified the script, apologies, my various tests got mixed up. It doesn´t hang with other models of the Qwen2-VL family (works with the Alibaba one).

For this reason, I do agree this is probably due to the model, *but* it used to work fine, *and* it hangs transformers on the loading part (after the downloading part), which would be a sort of 'security'/'critical' issue if used as injection. I see CPU is hanging 100%, not loading the model, weird, reproduced on three machines.

FYI, this DSE model uses the last token as an embedding, which is nothing fancy, so I don't see how it can hang. I've looked around recent changes to qwen2-vl in transformers but couldn't locate anything of interest yet.

transformers-cli as requested : 
```
- `transformers` version: 4.50.0.dev0
- Platform: Linux-5.15.0-97-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.5.2
- Accelerate config:    not found
- DeepSpeed version: 0.15.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA RTX A5000
```

Let me know if I can help.

Hmm thanks for detailed layout of the issue. If that happens only with specific hub checkpoints, indeed not a model issue. cc @Rocketknight1 , do you know who owns this part of code?

@zucchini-nlp, @cyrilvallez has recently been working on a refactor of `from_pretrained` I believe! One thing to note here is that `MrLight/dse-qwen2-2b-mrl-v1` uses the legacy `pytorch_model.bin` storage format, which might explain why it hangs when other models don't.

I did some local testing and the freeze occurs in `torch.load()`, which is quite concerning. I don't understand how this would have worked on older versions of `transformers`, and it makes me suspect the weights might be malformed in some way.

Just took a very quick glance at the checkpoint, and I couldn't recognise the rope type (I checked and it does not exist in Transformers). So it may come from that. However, still weird that it would hang instead of `KeyError`
```
"rope_scaling": {
    "type": "mrope",
    "mrope_section": [
      16,
      24,
      24
    ]
},
```

Anyway, it shows that the checkpoint is corrupted in some way

Oups no sorry, it is dynamically set to "default" in the config instead! Not sure about the issue then!

Ok, I investigated and I can confirm that the issue is not on our side! It's either on torch's, or the checkpoint's.
The following min repro shows the hanging:

```python
from transformers.utils.hub import cached_file
import torch

file = cached_file("MrLight/dse-qwen2-2b-mrl-v1", "pytorch_model.bin")
# This one hangs indefinitely
st = torch.load(file, map_location="meta", weights_only=True)
```

Similar code runs fine with other checkpoints, i.e. if you try
```python
from transformers.utils.hub import cached_file
import torch

file = cached_file("google-bert/bert-base-uncased", "pytorch_model.bin")
st = torch.load(file, map_location="meta", weights_only=True)
```
It's no problem!

Interestingly enough, if not loading on meta, it works:
```python
from transformers import AutoConfig, Qwen2VLForConditionalGeneration
from transformers.utils.hub import cached_file
import torch

file = cached_file("MrLight/dse-qwen2-2b-mrl-v1", "pytorch_model.bin")
st = torch.load(file, weights_only=True)

config = AutoConfig.from_pretrained("MrLight/dse-qwen2-2b-mrl-v1")
model = Qwen2VLForConditionalGeneration(config)
model.load_state_dict(st, strict=True, assign=False)
>>> <All keys matched successfully>
```

Thanks @Cyrilvallez @Rocketknight1 @zucchini-nlp for your super quick attention to this and the 'escalation' to torch. That's pretty crazy HF efficiency actually :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.