Add support for including in-memory videos (not just files/urls) in apply_chat_template
Author: akibjawad
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/36560, This PR allows inclusion of in-memory video objects, as dictionary of frames and metadata, in the chat template. 

**Previously:**
Chat template accepted only file-paths or urls in the chat_template. If user (a developer using transformers library) collected videos from a continuous stream or any input devices, user had to store the video in a file and provide file path in chat messages.

**Now (after this PR):**
Users can collect video frames from streams or devices, provide metadata (describing fps), and directly pass those in the chat_template as a dictionary object. It frees the user from saving the video in files, and increases efficiency by reducing extra IO operation to reload the video again from files.  

**Notes:**
Additionally, this PR also fixes hardcoded values used for testing (in assertions) apply_chat_template_videos for models like internvl, qwen2_vl, qwen2_5_vl, qwen2_5_omni.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->




## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). No
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section? Yes
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.  
      Yes. Issue link: https://github.com/huggingface/transformers/issues/36560
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? 
    - in tests/test_processing_common.py file:  I added a new type of video input which will be included in the chat messages while testing functionality of apply_chat_template.  
    - Added a new test with batchsize 3 for testing in-memory video objects in chat_template. Additionally updated hardcoded assertion (video_len check) for testing with increased batch_size in 4 models:
        - tests/models/internvl/test_processor_internvl.py
        - tests/models/qwen2_vl/test_processor_qwen2_vl.py
        - tests/models/qwen2_5_vl/test_processor_qwen2_5_vl.py
        - tests/models/qwen2_5_omni/test_processor_qwen2_5_omni.py
        - tests/models/smolvlm/test_processor_smolvlm.py (skip testing smolvlm with list of frames)

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Specifically mentioning @zucchini-nlp for review. Feel free to tag other members/contributors who may be interested to review this PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @zucchini-nlp 

requesting review @zucchini-nlp @Rocketknight1 @ArthurZucker @FredrikNoren 

> Thanks a lot for the PR @akibjawad ! I feel like this is increasing LOC unnecessarily and could be done with less changes from our side. What if we update `load_video` to early exit when an `array` is found instead of trying to decode
> 
> The only constraint would be that users have to be consistent with video type within one conversation. So if one started using `decoded frames` in convo, they have to use `decoded frames` format for subsequent videos in the chat. Otherwise handling `video_metadata` can become hard

@zucchini-nlp, thank you very much for reviewing. I do agree with your notion to keep the library lean. In fact, my initial implementation was exactly what you mentioned. Later I changed the design, because `load_video()` is meant to collect video_frames and metadata from a source (such as file, url). When a user provides video frames (ndarray/tensor) in a conversation, user already loaded the video some way (either from a file, livestream, screen record, camera devices, or randomly generated etc.) and user do not want to save the frames to a file. While collecting frames, user might also collect metadata. That is why I kept option for both frames & metadata. Additionally, without any metadata, frame_sampling with `fps` is not possible. As you mentioned earlier, user must be consistent and cannot use `fps` parameter for sampling while using `decoded_frames` or video as a `list of image file names`. Because in those cases metadata will be none. Although we can provide a default metadata for consistent sampling.

To accept video frames as array, do we actually need to modify `load_video()` function? Because, if we are returning early from the function, we can simply detect video type is an array and collect the video frames from the if else block in apply_chat_template function of the processor, saving an extra step of calling the function. However `load_video()` is an utility function and I assume it is used in many parts of the code-base, If we include array handling in the `load_video()` function, it will be useful for other places also. Although `load_video()` has other parameters (`fps`, `num_frames`) for sampling. From the current implementation, it looks like sampling is done at the video processor class, not at the apply_chat_template phase. 

I have noticed you have been working on video_processor for a long time and you have better idea about the complete pipeline and future of this ever changing code. Let me know, which solution would you prefer. 

> To accept video frames as array, do we actually need to modify load_video() function? 

Yes, it currently has a bug because it checks for `isinstance(video, array)` later and thus fails. We just need to change the order of conditional checks

The user is still free to pass metadata as kwargs to `apply_chat_template` and it should be picked up, I am doing another update here https://github.com/huggingface/transformers/pull/39600 and prob that will fix it. I am currently stuck on different task but will continue on video decoding soon

@zucchini-nlp Thank you for the clarification, I updated the code of `load_video()` and handled `decoded frames` same as handling a `list of image file names`. I kept everything else (metadata handling) same so that this changes will not create too much conflict with your PR (https://github.com/huggingface/transformers/pull/39600). Please review again and let me know if I need to change anything else.

@zucchini-nlp Thank you very much for the review. I addressed your reviews in the most recent commit. As a maintainer, you need to initiate some github workflows for complete testing. Let me know, if there is any remaining issues with current implementation. 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: internvl, qwen2_5_omni, qwen2_5_vl, qwen2_vl, smolvlm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39494). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.