image-to-text pipeline failure when using Qwen2_VL
Author: Spycsh
Pull request: no
Labels: name: Core: Pipeline

name: bug

name: Multimodal

### System Info

- `transformers` version: 4.47.0.dev0
- Platform: Linux-5.15.0-122-generic-x86_64-with-glibc2.35
- Python version: 3.10.0
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using the Qwen2-VL in image-to-text pipeline, it failed with following issue

```
  File "xxx/modeling_qwen2_vl.py", line 1572, in get_rope_index
    position_ids[..., i, attention_mask[i] == 1] = llm_positions.to(position_ids.device)
RuntimeError: shape mismatch: value tensor of shape [3, 3596] cannot be broadcast to indexing result of shape [3, 26]
```

Here are the code the reproduce:

```py
import PIL.Image
import requests
from transformers import AutoProcessor, pipeline

model_name_or_path = "Qwen/Qwen2-VL-2B-Instruct"

image_path = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"

processor = AutoProcessor.from_pretrained(model_name_or_path)
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image"},
        ],
    }
]
prompt = processor.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)

images = [PIL.Image.open(requests.get(image_path, stream=True, timeout=3000).raw)]

generator = pipeline(
    "image-to-text",
    model=model_name_or_path,
    config=model_name_or_path,
    # tokenizer=model_name_or_path,
    # image_processor=model_name_or_path,
    torch_dtype="auto",
    device="cpu",
)

generate_kwargs = {"max_new_tokens": 100}

result = generator(images, prompt=prompt, batch_size=1, generate_kwargs=generate_kwargs)
print(result)
```

Uncomment the `# tokenizer=model_name_or_path,` ` # image_processor=model_name_or_path,` also does not work.

### Expected behavior

The pipeline should work with Qwen2-VL

Comments:
https://github.com/huggingface/transformers/pull/34771 provides a workaround to solve this issue. Basic the problem is that Qwen2-VL uses `processor` to preprocess and it cannot be replaced by the default `image_processor`+`tokenizer`. For usage, just initialize the pipeline with:

```
generator = pipeline(
    "image-to-text",
    model=model_name_or_path,
    config=model_name_or_path,
    processor=processor,
    torch_dtype="auto",
    device="cpu",
)
```

and it should work.

Hi @Spycsh, thanks for opening the issue! I suppose Qwen2-VL should be used with `image-text-to-text` pipeline instead. Let me know if that resolve the issue!

Thanks @qubvel , update to `image-text-to-text` works:)

Thanks for letting me know and for closing the issue and corresponding PR!

If you are fine tunning the Qwen model with Llama-factory, this error can also happen if the total length of the promptd content (system + user) is longer than `cutoff_len` parameter in the config (e.g. in examples/train_lora/qwen2vl_lora_sft.yaml)