Incorrect Video Processing in Qwen2-VL Model Due to Incorrect call of Video/Image processor

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

- `llamafactory` version: 0.9.2.dev0
- Platform: Linux-5.15.0-1044-nvidia-x86_64-with-glibc2.35
- Python version: 3.12.5
- PyTorch version: 2.5.1+cu124 (GPU)
- Transformers version: 4.48.2
- Datasets version: 3.2.0
- Accelerate version: 0.34.2
- PEFT version: 0.12.0
- TRL version: 0.9.6
- GPU type: NVIDIA H100 80GB HBM3
- vLLM version: 0.6.6.post1

### Reproduction

Use following training parameters:
```yaml
### model
model_name_or_path: Qwen/Qwen2-VL-2B
image_resolution: 262144
video_resolution: 16384
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: full
freeze_vision_tower: false  # choices: [true, false]
freeze_multi_modal_projector: false  # choices: [true, false]
train_mm_proj_only: false  # choices: [true, false]
# deepspeed: examples/deepspeed/ds_z3_config.json  # choices: [ds_z0_config.json, ds_z2_config.json, ds_z3_config.json]

### dataset
dataset: dev
template: qwen2_vl
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/qwen2.5_vl-7b/full/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 2
learning_rate: 1.0e-5
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### eval
# val_size: 0.1
# per_device_eval_batch_size: 1
# eval_strategy: steps
# eval_steps: 500
```

An example of training data is:
```json
[
{
        "conversations": [
            {
                "from": "human",
                "value": "In the following video, a person/people discuss in a sign language. Your task is to determine which sign language it is and translate what he/she/they say into <lang/>english</lang> language. <video_recording>"
            },
            {
                "from": "gpt",
                "value": "<slang/>ase</slang><translation/>We love it when our Sign1News fans reach out to us and share interesting, fun and funny videos. This video comes from Dianna and her son Zarek. They live in Ontario, California. And they just want to remind everyone to be on your best behavior tonight because Santa Claus is coming to town. You better watch out. You better not cry. You better not pout. I'm telling you why. Santa Claus is coming to town. He's making a list. He's checking it twice. He's gonna find out who's naughty or nice. Santa Claus is coming to town. He sees you when you're sleeping. He knows when you're awake. He knows if you've been bad or good. So be good for goodness sake. Oh...! You better watch out. You better not cry. You better not pout. I'm telling you why. Santa Claus is coming to town! He sees you when you're sleeping. He knows when you're awake. He knows if you've been bad or good. So be good for goodness sake. Oh...! You better watch out. You better not cry. You better not pout. I'm telling you why. Santa Claus is coming to town!</translation>"
            }
        ],
        "videos": [
            "video.mkv"
        ],
        "system": "You are a helpful assistant."
    }
]

```

Start training with command:
```cmd
VIDEO_PLACEHOLDER="<video_recording>" llamafactory-cli train configs/full_train.yaml
```


After this I have got an exception:
```text
multiprocess.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
    batch = apply_function_on_filtered_inputs(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/data/processors/supervised.py", line 108, in preprocess_supervised_dataset
    input_ids, labels = _encode_supervised_example(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/data/processors/supervised.py", line 49, in _encode_supervised_example
    messages = template.mm_plugin.process_messages(prompt + response, images, videos, audios, processor)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/data/mm_plugin.py", line 1160, in process_messages
    raise ValueError(f"`len(videos)` is less than the number of {VIDEO_PLACEHOLDER} tokens.")
ValueError: `len(videos)` is less than the number of <video_recording> tokens.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jovyan/.mlspace/envs/mvit/bin/llamafactory-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/cli.py", line 112, in main
    run_exp()
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/train/tuner.py", line 92, in run_exp
    _training_function(config={"args": args, "callbacks": callbacks})
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/train/tuner.py", line 66, in _training_function
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
    dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/data/loader.py", line 269, in get_dataset
    dataset = _get_preprocessed_dataset(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/gurov/SL_25/LLaMA-Factory/src/llamafactory/data/loader.py", line 204, in _get_preprocessed_dataset
    dataset = dataset.map(
              ^^^^^^^^^^^^
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3165, in map
    for rank, done, content in iflatmap_unordered(
                               ^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.mlspace/envs/mvit/lib/python3.12/site-packages/multiprocess/pool.py", line 774, in get
    raise self._value
ValueError: `len(videos)` is less than the number of <video_recording> tokens.
```

### Others

I am encountering an issue with the Qwen2-VL model where video inputs are incorrectly processed due to the absence of a dedicated video processor in that model. Instead of using video processor it uses image processor, that's ok, but calling that processor is incorrect while processing video input. 

The reason lays in inheritance concept in [Qwen2vlPlugin](https://github.com/hiyouga/LLaMA-Factory/blob/74ade3a176cad753971aaad681fea6ff8df40914/src/llamafactory/data/mm_plugin.py#L990C1-L990C33) class:
It uses [_get_mm_input](https://github.com/hiyouga/LLaMA-Factory/blob/74ade3a176cad753971aaad681fea6ff8df40914/src/llamafactory/data/mm_plugin.py#L193) function from [parent class](https://github.com/hiyouga/LLaMA-Factory/blob/74ade3a176cad753971aaad681fea6ff8df40914/src/llamafactory/data/mm_plugin.py#L74), but it do not match with Qwen2-VL processor logic. Let's look at this function precisely using snippets below:


### Processors initializations
There are [initialization of processors](https://github.com/hiyouga/LLaMA-Factory/blob/74ade3a176cad753971aaad681fea6ff8df40914/src/llamafactory/data/mm_plugin.py#L212C9-L215C23) below: image, video and feature extractor. But Qwen2-VL processor does not have `video_processor`, hence, `image_processor` will be used as `video_processor`
```python
image_processor: "BaseImageProcessor" = getattr(processor, "image_processor", None)
video_processor: "BaseImageProcessor" = getattr(processor, "video_processor", image_processor)
feature_extractor: "SequenceFeatureExtractor" = getattr(processor, "feature_extractor", None)
mm_inputs = {}
```

### Image and video processing 
In the [code below](https://github.com/hiyouga/LLaMA-Factory/blob/74ade3a176cad753971aaad681fea6ff8df40914/src/llamafactory/data/mm_plugin.py#L217C6-L231C75) there are processing cases whether data contains video or images, or both (is also possible), but due to incorrectness both video and images will be processed as video, why? 

 
```python
        if len(images) != 0:
            images = self._regularize_images(
                images,
                image_resolution=getattr(processor, "image_resolution", 768 * 768),
            )
            mm_inputs.update(image_processor(images, return_tensors="pt"))

        if len(videos) != 0:
            videos = self._regularize_videos(
                videos,
                image_resolution=getattr(processor, "video_resolution", 256 * 256),
                video_fps=getattr(processor, "video_fps", 2.0),
                video_maxlen=getattr(processor, "video_maxlen", 128),
            )
            mm_inputs.update(video_processor(videos, return_tensors="pt"))
```

Because first positional argument in Qwen2-VL image processor is always handled as image, and output will contain `image_grid_thw` instead of `video_grid_thw` (for image you should explicitly specify optional `video` argument). So never will video be processed as video. And always attempts to process video result in exception. 

In previous version it had been handled correctly, due to absence of audio input, so, image and video in a case of presence both modalities would be processed simultaneously. 


### [not] a proposed solution 

I hope you are more skillful than me, especially in your repo. And it is not a proposed solution, just a quick fix for those like me, who want to fix this exception as soon as possible. I hope you will find a better solution. 

So I have performed overriding `_get_mm_inputs` function for `Qwen2vlPlugin`:

```python 
@override
    def _get_mm_inputs(
        self,
        images: Sequence["ImageInput"],
        videos: Sequence["VideoInput"],
        audios: Sequence["AudioInput"],
        processor: "ProcessorMixin",
    ) -> Dict[str, "torch.Tensor"]:
        """
        Processes visual inputs.

        Returns: (llava and paligemma)
            pixel_values: tensor with shape (B, C, H, W)

        Returns: (qwen2-vl)
            pixel_values: tensor with shape (num_patches, patch_dim)
            image_grid_thw: tensor with shape (num_images, 3), where the three numbers are time, width, height

        It holds num_patches == torch.prod(image_grid_thw)
        """
        image_processor: "BaseImageProcessor" = getattr(processor, "image_processor", None)
        video_processor: "BaseImageProcessor" = getattr(processor, "video_processor", image_processor)
        feature_extractor: "SequenceFeatureExtractor" = getattr(processor, "feature_extractor", None)
        mm_inputs = {}

        if len(images) + len(videos) != 0:
            images = self._regularize_images(
                images,
                image_resolution=getattr(processor, "image_resolution", 768 * 768),
            ) if len(images) != 0 else None
            videos = self._regularize_videos(
                videos,
                image_resolution=getattr(processor, "video_resolution", 256 * 256),
                video_fps=getattr(processor, "video_fps", 2.0),
                video_maxlen=getattr(processor, "video_maxlen", 128),
            ) if len(videos) != 0 else None
            mm_inputs.update(image_processor(images=images, videos=videos, return_tensors="pt"))

        if len(audios) != 0:
            audios = self._regularize_audios(
                audios,
                sampling_rate=getattr(feature_extractor, "sampling_rate", 16000),
            )
            mm_inputs.update(
                feature_extractor(
                    audios,
                    sampling_rate=getattr(feature_extractor, "sampling_rate", 16000),
                    return_attention_mask=True,
                    padding="max_length",
                    return_tensors="pt",
                )
            )
            mm_inputs["feature_attention_mask"] = mm_inputs.pop("attention_mask")  # prevent conflicts

        return mm_inputs
```

Labels: solved
State: closed