QWEN2VLProcessor missing video_token_id in mm_token_type_ids
Author: NikeHop
Pull request: no
Labels: name: bug

### System Info

 
- `transformers` version: 4.53.0
- Platform: Linux-5.14.0-427.70.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.13
- Huggingface_hub version: 0.33.1
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1+cu126 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python

import torch 

from qwen_vl_utils import process_vision_info # Needs to be installed via pip install qwen-vl-utils
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

# Load model and processor
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

VIDEO_PATH = "./videos/examples/fps_30.mp4"
FPS = 30

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What can you see in the video?",
            },
            {"type": "video", "video": VIDEO_PATH, "fps": FPS},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
result = process_vision_info(
    messages,
    return_video_kwargs=True,
)
image_input, video_input, video_kwargs = result[0], result[1], result[2]

inputs = processor(
    text=[text],
    images=image_input,
    videos=video_input,
    padding=True,
    return_tensors="pt",
    text_kwargs={"return_mm_token_type_ids": True},
    **video_kwargs,
).to("cuda:0")

print(inputs["mm_token_type_ids"]) # All 0's
print(torch.sum(inputs["mm_token_type_ids"]).item()) # 0

```


### Expected behavior

Relevant Code:

![Image](https://github.com/user-attachments/assets/6438b34d-4cee-44ac-a5b6-ae3689207c52)

I think there should be another line: 

```python 
mm_token_type_ids[array_ids == self.video_token_id] = 2
```


Comments:
@NikeHop hey!

Yeah, we can and will add videos in the token type ids. The first version was created for vLLM which will support only images at this point. The reason I didn't add videos is because I wasn't sure if any edge cases arise with omni modal LLMs. I am glad to see that `mm_token_type_ids` has usage beyond vLLM and will add videos/audios as well + document it somewhere

@zucchini-nlp thanks for the fast reply and apologies for the late reply. Thanks for explaining. I think the list can be conveniently used to mask out image and video tokens in the loss calculation during fine-tuning. Of course, one can also access processor.video_token directly. Shall I close the issue ? 

I will add the videos in token type ids soon, we can keep it open :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.