🔴 Video processors as a separate class
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This is quite a breaking PR, because from now on some processor will not have a separate `self.video_processor` and save/load priority is given to new file names `image/video_preprocessor_config.json`. Save/load perserved BC and can load from any file name

Reasons for PR:
- Currently the main issue is to allow video processors operate separately from image processors and have their own set of initial attributes, as mentioned in the issue
- Easier handling of multimodal models and more freedom in processing when images and videos have their own logic (i.e. llava-next-video)

<details>
<summary>Description - more verbose</summary>

Fixes https://github.com/huggingface/transformers/issues/33504 and makes a separate Base class for video processors, only those that are from VLMs because video-only models like ViViT don't need both, image and video processor. We can add it in other video-only them later in the run, if needed.

This idea is to keep video processors as child of image processor mixin, but the processor can use transform over one video as a 4D array instead of looping over each frame. The only transform that still works on per-frame basis is `resize`. It could be rewritten in torch maybe, but not worth the effort. In terms of speed, the new slow video processor performs same as old image processor, and has some speedup with larger batch sizes. The advantage here is mostly about readability, cleaner code and a separate API for videos.

For the fast video processors, it is basically same as in image except for the extra dimension we have for timeframes. Plus, I added `convert_rgb` in `np` and `torch`. Also, we use `VideosKwargs` instead of having to add a new TypedDict. See my comments below, in files changed 👇🏻 
</details>



<details>
<summary>What was done?</summary>

What was done:
- Separate `VideoProcessorBase` and `VideoProcessorBaseFast` as added, recommend to start review from there
- Video processors  have an autoclass mapping to load with `AutoVideoProcessor`
- Image processor config is now called `image_preprocessor_config.json`. We keep BC by trying to load from new `image_preprocessor_config.json` if exists, otherwise fallback to `preprocessor_config.json` with a deprecation warning
- Most models already had a separate video processor, so nothing changes for them except for the parent class
- Model where one preprocessor accepted both, images and video, will and continue preprocessing both modalities, but raise a deprecation warning (`Qwen2VL` and `VideoLlava`)
- Tests, a lot of tests changed and added 🛠️  
- The image/video/general processing tests were run on all models touched by this PR, including one slow generation test per model to make sure official checkpoints are loadable
</details>



<details>
<summary>Usage examples</summary>

How does it work when loading (same goes for model-class laoding)
```python
AutoProcessor.from_pretrained(model_id)
AutoImageProcessor.from_pretrained(model_id, use_fast=True/False)
AutoVideoProcessor.from_pretrained(model_id, use_fast=True/False, device="cpu"/"cuda")
```
1. If model was not updated and has only `processor_config`: the processor will grab the old `preprocessor_config` and use it for video and for image processing classes with shared set of args. Warning emitted!

2. If model was updated and has only `image_processor_config` and `video_processor_config`: the processor will grab the files and load each preprocessor with their own set of args

3. If model has all types of config files and didn't delete the old `processor_config`: priority will be given to `(image/video)_processor_config` with different sets of args for each

When saving the model back, we'll save in `image_processor_config` and `video_processor_config` files 
</details>



<details>
<summary>Benchmark results on llava onevision </summary>

Benchmark on different batch sizes and number of frames per video sampled. Also with different devices and input types: could be numpy, torch or list PIL frames.
![results](https://github.com/user-attachments/assets/7aa67375-d25d-4e67-9279-55c2710c1d2b)
</details>

Next steps:
- Need step is to add tests for video inputs in `test_processing_common.py`  (there was a PR somewhere but prob it got stale)
- We also might need to rename "feature extractor name" to `audio_preprocessor_config`, just for consistency and easier navigation in hub configs
- The frame sampling function which we just added for video LLMs can now be a method of video processor. So each video model will have its own sampling pre-defined which imo eases stuff for instruct inference

  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35206). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This one is ready for review, I guess the first review will be from @qubvel . The failing tests are not related, very flaky 

@qubvel I updated a bit to address comments, can you take a look?

The CI is going crazy with flakiness 🙃 

I’ll repeat this here for visibility: my main concern is the introduction of **slow** VideoProcessors.

I believe it would be best to default to **fast**, torch-based VideoProcessors. Here’s why:

1) Clear benefits for users to upgrade the code and models, they going to get speedup from VideoProcessors
2) Reduced maintenance burden. Managing both slow and fast VideoProcessors could add extra complexity, especially when ensuring they remain identical.
3) We should leave BC and let users use ImageProcessors for preprocessing video, and it's actually the same as slow VideoProcessors

Additionally, Arthur mentioned the potential deprecation of slow ImageProcessors in favor of faster ones, which makes introducing **slow** VideoProcessors even less beneficial.

Of course, I’d love to hear your thoughts on this! Let me know what you think. Maybe we should get an opinion from core-maintainers as well 🤗 

I totally agree that fast processor have lots of benefits over slow ones and we need to support fast-video processors also. But I don't think that means we should have only fast video processor support, because `use_fast` in still not the default for images and has a dependency on `torchvision`. That means if we want to allow a separate config and class for Video Processors, we'll need to start from slow version being added and let users to get used with the idea of having `AutoVideoProcessor` instances, without breaking their models by forcing `use_fast` as the only option. For users it won't make difference in terms of usage or efficiency.

I believe the goal is to add a general API which lets us easily extend it to new models + enable fast processing. I don't want to add fast video processor within this PR for two reasons:
- Yoni is still doing major refactoring which is not merged yet
- I don't want the PR grow even bigger and merge as we go, separating it into composable features


Sure, I think this needs core maintainer's attention so ping @ArthurZucker  :) 



> because use_fast in still not the default for images and has a dependency on torchvision. That means if we want to allow a separate config and class for Video Processors, we'll need to start from slow version being added and let users to get used with the idea of having AutoVideoProcessor instances, without breaking their models by forcing use_fast as the only option. For users it won't make difference in terms of usage or efficiency.

Yeah, what I mean here is there should be **no** option `use_fast` for `AutoVideoProcessor` at all, in case one chooses to use `AutoVideoProcessor`, it will get a fast torch-based processor, however, users can still use image processors (slow or fast) in case they want to process an image and for backward compatibility. Otherwise, we might end up having 4 classes written for a single model slow/fast image processor and slow/fast video processor.

Ah I see. However if we want to promote using the video processors, I think we need to introduce the least breaking version aka almost same as it was previously. Otherwise it seems, we'll be maintaining a base class which probably isn't even used for being the non-default one

My aim is to remove video as input from image processors all together, be it slow or fast image processors. Especially useful when models process slightly videos differently, which makes our code way too complicated to read

Discussed internally and will be adding a fast video processor along with the slow one already added. Feel free to unsubscribe and I'll tag when it's ready for review again, it might get noisy with commits 

@qubvel ready for review, updated the PR description and added the benchmark results. I tried to add comments to explain what was changed. You can review only one of the models I think, all are identical almost

@qubvel 

@qubvel gentle ping

Thanks for review! Yeah, I agree that slow video processor is not needed, after our last discussion during offsite. Will remove the slow version and do clean-up. A lot of conflicts and new models after last review :)

Will ping again when cleaned up

Ready for review @qubvel . Removed slow video processors and the fast video inherits from fast image. Added a bunch of tests and new video processor for all models (almost, except SmolVLM, see reason 👇🏻 )


@yonigozlan , seems like `resize` will not work with some interpolation modes like `LANCSOZ`. The torch documentation advises to feed a `PIL.Image` for those cases, and using `PIL` means we're back to looping over each image. Have you already encountered this issue?


> @yonigozlan , seems like resize will not work with some interpolation modes like LANCSOZ. The torch documentation advises to feed a PIL.Image for those cases, and using PIL means we're back to looping over each image. Have you already encountered this issue?

That's a good question, I hadn't seen this problem before today but just saw that idefics3, smolvlm, flava and chameleon use Lanczos resampling 😅. Best thing for now could be to use another sampling method by default and add a warning_once to fall back to slow for exact processing? I haven't found any torch-based lanczos resampling method, so not sure if there's a better way to handle this for now...

@yonigozlan yeah, I didn't find an equivalent in torch either, so fallback to slow looks to be an only viable option for now. TBH, PIL resize op do not have much equivalents 😢  

@qubvel gentle ping

Sorry for the delay, I will do a thorough review on Monday! Can you please resolve the conflicts in a meantime?

TLDR from internal coms:

>long term we want to:
> -  inherit from ImageProcessor and BaseVideoProcessor,
> - prepare the videos to call vmap and use inherited ImageProcessor defined functions