[video processors] decode only sampled videos -> less RAM and faster processing
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR moves the video decoding code entirely into video processors, so that we can load only necessary video frames into memory. To be consistent with video processors, I also updated image processors to accept `str` in inputs and optionally load images.

The docs for video processors are also updated explaining how frames are sampled and what users need to do to turn it on/off. Note that we'll be using by default `torchcodec` and fallback to `torchvision`, and we won't support any arbitrary video decoders within video processor class. Otherwise we'd need to introduce more `kwargs` and handle differences between decoders, which bloats up the code even more



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39600). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: aria, aya_vision, blip, bridgetower, chameleon, clip, colpali, deepseek_vl, deepseek_vl_hybrid, emu3, eomt, flava, gemma3, gemma3n, glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/aria', 'models/aya_vision', 'models/blip', 'models/bridgetower', 'models/chameleon', 'models/clip', 'models/colpali', 'models/deepseek_vl', 'models/deepseek_vl_hybrid', 'models/emu3', 'models/eomt', 'models/flava', 'models/gemma3', 'models/gemma3n', 'models/glm4v']
quantizations: [] ...

run-slow: aria, aya_vision, blip, bridgetower, chameleon, clip, colpali, deepseek_vl, deepseek_vl_hybrid, emu3, eomt, flava, gemma3, gemma3n, glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/aria', 'models/aya_vision', 'models/blip', 'models/bridgetower', 'models/chameleon', 'models/clip', 'models/colpali', 'models/deepseek_vl', 'models/deepseek_vl_hybrid', 'models/emu3', 'models/eomt', 'models/flava', 'models/gemma3', 'models/gemma3n', 'models/glm4v']
quantizations: [] ...

@qubvel I merged the related PR and rebased `main`. Can you take another look?

run-slow: aria, aya_vision, blip, bridgetower, chameleon, clip, colpali, deepseek_vl, deepseek_vl_hybrid, emu3, eomt, flava, gemma3, gemma3n, glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/aria', 'models/aya_vision', 'models/blip', 'models/bridgetower', 'models/chameleon', 'models/clip', 'models/colpali', 'models/deepseek_vl', 'models/deepseek_vl_hybrid', 'models/emu3', 'models/eomt', 'models/flava', 'models/gemma3', 'models/gemma3n', 'models/glm4v']
quantizations: [] ...

run-slow: qwen2_vl, qwen2_5_vl, qwen2_5_omni, smolvlm, llava_onevision, llava_next_video, perception_lm

This comment contains run-slow, running the specified jobs: 

models: ['models/llava_next_video', 'models/llava_onevision', 'models/perception_lm', 'models/qwen2_5_omni', 'models/qwen2_5_vl', 'models/qwen2_vl', 'models/smolvlm']
quantizations: [] ...

On no, new torch release doesn't work well with Bytes objects 😓  (fails only in CI, still figuring out why)


run-slow: qwen2_vl, qwen2_5_vl, qwen2_5_omni, smolvlm, llava_onevision, llava_next_video, perception_lm



run-slow: qwen2_vl, qwen2_5_vl, qwen2_5_omni, smolvlm, llava_onevision, llava_next_video, perception_lm


This comment contains run-slow, running the specified jobs: 

models: ['models/llava_next_video', 'models/llava_onevision', 'models/perception_lm', 'models/qwen2_5_omni', 'models/qwen2_5_vl', 'models/qwen2_vl', 'models/smolvlm']
quantizations: [] ...

The CI is impossible to pass 🙃 

@bot /style

Style bot fixed some files and pushed the changes.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, aya_vision, blip, bridgetower, chameleon, clip, colpali, deepseek_vl, deepseek_vl_hybrid, emu3, eomt, flava, gemma3, gemma3n, glm4v

@zucchini-nlp I believe it was previously possible for users to sample frames themselves and then pass the batched video frames directly to the video processor. This usage can be found in existing code, for example in the following notebook: https://huggingface.co/facebook/vjepa2-vitl-fpc16-256-ssv2/blob/main/notebook_finetuning.ipynb
.

However, this no longer seems to be supported due to changes introduced in this PR. In particular, the notebook above now produces errors when run with the current versions.

Was this behavior change intentional, or is it an unintended regression?

@shaform hey, it is still possible to sample frames and pass them as a list of 3D frames or a 4D array. You just need to pass `do_sample=False` in the video processor call to disable sampling. Such as:

`pixel_values = video_processor(my_sampled_video, do_sample=False).pixel_values_video`

> @shaform hey, it is still possible to sample frames and pass them as a list of 3D frames or a 4D array. You just need to pass `do_sample=False` in the video processor call to disable sampling. Such as:
> 
> `pixel_values = video_processor(my_sampled_video, do_sample=False).pixel_values_video`

@zucchini-nlp Thank you for your quick response. I tested a bit, and found that the reason the notebook fails now is because the processor will return a tensor with a incorrect shape if a batched input is given. Here is a minimal example to reproduce:

```
# input: B x T x C x H x W
processor(th.zeros(4, 5, 3, 100, 100), return_tensors="pt", do_sample_frames=False).pixel_values_videos.shape
# output: 1 x 1 x B x T x C x H x W
Out[17]: torch.Size([1, 1, 4, 5, 3, 256, 256])
```

It seems a simple workaround is to convert the input into a list, i.e.,
```
processor([t for t in th.zeros(4, 5, 3, 100, 100)], return_tensors="pt", do_sample_frames=False).pixel_values_videos.shape
```

Could this be considered as a bug?

@shaform yeah, should not be happening. Would you mind opening an issue with a minimal reproducer (model_id and how you call it) so I don't forget about it?