Repository: huggingface/transformers
Pull request: Qwen 2.5 Omni: apply video defaults

# What does this PR do?

Applies `min_pixels` and `max_pixels` values to video processor.

The values were taken from [the original processing codebase](https://github.com/QwenLM/Qwen2.5-Omni/blob/7c8dddb38d52a58ce57e778e10fa0eaf26e078e9/qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py#L30), which uses a different set for [video](https://github.com/QwenLM/Qwen2.5-Omni/blob/7c8dddb38d52a58ce57e778e10fa0eaf26e078e9/qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py#L30) than it does for [images](https://github.com/QwenLM/Qwen2.5-Omni/blob/7c8dddb38d52a58ce57e778e10fa0eaf26e078e9/qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py#L26).

In our case, the image processor would always default to the image case, which results in frames resized to very large sizes, possibly causing OOMs, and preparing inputs with shapes not seen by the model during training.

## Reproduction

Consider the following snippet:

```py
import soundfile as sf
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor

model_id = "Qwen/Qwen2.5-Omni-7B"
processor = Qwen2_5OmniProcessor.from_pretrained(model_id)

conversation = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}
        ],
    },
    {
        "role": "user",
        "content": [
             {"type": "video", "video": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw.mp4"},
            {"type": "text", "text": "What can you hear and see in this video?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    conversation,
    load_audio_from_video=True,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    video_fps=2,

    # kwargs to be passed to `Qwen2-5-OmniProcessor`
    padding=True,
    use_audio_in_video=True,
)

print(inputs["pixel_values_videos"].shape)
```

* Before this PR: `[886116, 1176]`
* After this PR: `[60480, 1176]`
* Reference, using qwen_omni_utils.process_mm_info: `[57600, 1176]`

The difference between this PR and the reference is because the original codebase selects `40` frames for this video, while we select `41`.

## Alternatives

* Use different config values for image and video processing and persist them to `preprocessor_config.json`.


Changed files:
filename: src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py
patch: @@ -61,6 +61,8 @@ class Qwen2_5OmniProcessorKwargs(ProcessingKwargs, total=False):
             "seconds_per_chunk": 2.0,
             "position_id_per_seconds": 25,
             "use_audio_in_video": False,
+            "min_pixels": 128 * 28 * 28,
+            "max_pixels": 768 * 28 * 28,
         },
         "audio_kwargs": {
             "sampling_rate": 16000,
@@ -147,7 +149,7 @@ def __call__(
         seconds_per_chunk = output_kwargs["videos_kwargs"].pop("seconds_per_chunk")
         position_id_per_seconds = output_kwargs["videos_kwargs"].pop("position_id_per_seconds")
         use_audio_in_video = output_kwargs["videos_kwargs"].pop("use_audio_in_video")
-        fps = output_kwargs["videos_kwargs"].pop("fps", None)
+        fps = output_kwargs["videos_kwargs"].pop("fps", 2.0)
 
         if audio is not None:
             output_kwargs["audio_kwargs"]["padding"] = "max_length"  # Support "max_length" padding only here
@@ -174,8 +176,7 @@ def __call__(
         if videos is not None:
             videos = make_batched_videos(videos)
             videos_inputs = self.image_processor(images=None, videos=videos, **output_kwargs["videos_kwargs"])
-            if fps is None:
-                fps = [2.0] * len(videos)
+            fps = [fps] * len(videos)
             videos_inputs["video_second_per_grid"] = [
                 self.image_processor.temporal_patch_size / fps[i] for i in range(len(fps))
             ]

filename: tests/models/qwen2_5_omni/test_processor_qwen2_5_omni.py
patch: @@ -433,7 +433,7 @@ def test_apply_chat_template_video_frame_sampling(self):
             num_frames=num_frames,
         )
         self.assertTrue(self.videos_input_name in out_dict_with_video)
-        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 9568)
+        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 5760)
 
         # Load with `video_fps` arg
         video_fps = 1
@@ -445,7 +445,7 @@ def test_apply_chat_template_video_frame_sampling(self):
             video_fps=video_fps,
         )
         self.assertTrue(self.videos_input_name in out_dict_with_video)
-        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 23920)
+        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 14400)
 
         # Load with `video_fps` and `num_frames` args, should raise an error
         with self.assertRaises(ValueError):
@@ -466,7 +466,7 @@ def test_apply_chat_template_video_frame_sampling(self):
             return_dict=True,
         )
         self.assertTrue(self.videos_input_name in out_dict_with_video)
-        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 717600)
+        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 432000)
 
         # Load video as a list of frames (i.e. images). NOTE: each frame should have same size
         # because we assume they come from one video
@@ -484,7 +484,7 @@ def test_apply_chat_template_video_frame_sampling(self):
             return_dict=True,
         )
         self.assertTrue(self.videos_input_name in out_dict_with_video)
-        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 5704)
+        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 2904)
 
     @require_av
     def test_apply_chat_template_video_special_processing(self):