LlavaNextVideoProcessor -> TypeError: LlavaNextVideoProcessor.__call__() got an unexpected keyword argument 'legacy' (I have the fix)
Author: inf3rnus
Pull request: no
Labels: name: Core: Pipeline

name: bug

name: VLM

### System Info

Problem's root cause is in `ImageTextToTextPipeline` class in the `image_text_to_text.py` pipeline.

Line `438`

```py
        model_inputs = self.processor(
            images=images, text=text, return_tensors=self.framework, legacy=False, **processing_kwargs
        ).to(dtype=self.torch_dtype)
```

Notice how legacy is always specified as False?

If you use this model (`llava-hf/LLaVA-NeXT-Video-7B-32K-hf`) on `transfomers==4.47.1`  you will get this error because its config specifies to use the class: `LlavaNextVideoProcessor` from `processing_llava_next_video.py` and it's `__call__` method is not expecting that kwarg.

The quick fix is this: 

Modify `__call__` (line `101`) in `processing_llava_next_video.py`

from this:

```py
    def __call__(
        self,
        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]],
        images: ImageInput = None,
        videos: VideoInput = None,
        padding: Union[bool, str, PaddingStrategy] = False,
        truncation: Union[bool, str, TruncationStrategy] = None,
        max_length: int = None,
        return_tensors: Optional[Union[str, TensorType]] = TensorType.PYTORCH,
    ) -> BatchFeature:
 ```

to this:

```py
    def __call__(
        self,
        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]],
        images: ImageInput = None,
        videos: VideoInput = None,
        padding: Union[bool, str, PaddingStrategy] = False,
        truncation: Union[bool, str, TruncationStrategy] = None,
        max_length: int = None,
        return_tensors: Optional[Union[str, TensorType]] = TensorType.PYTORCH,
        **kwargs, # <-- this guy
    ) -> BatchFeature:
 ```
 
 Notice the unused kwargs at the end. This reflects the pattern used for `__init__`
 
 which looks like this:

```py
    def __init__(
        self,
        video_processor=None,
        image_processor=None,
        tokenizer=None,
        chat_template=None,
        patch_size=None,
        vision_feature_select_strategy=None,
        video_token="<video>",
        image_token="<image>",
        num_additional_image_tokens=0,
        **kwargs, # <-- this guy
    ):
```

I ain't got time to step through the PR process, so I hope this helps the HF staff either make this quick patch, or solve the problem at a higher level in the code for `image_text_to_text.py`.

### Who can help?

HF staff

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) (`image-to-text-to-text`)

### Reproduction

```py
pipe = pipeline("image-text-to-text", model="llava-hf/LLaVA-NeXT-Video-7B-32K-hf")

messages = {'role': 'user', 'content': [{'type': 'text', 'text': "What's in this image?"}, {'type': 'video'}]}

videos = ["https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"]

out = pipe(text=messages, videos=videos)
```

### Expected behavior

No exception raised due to an unexpected kwarg.

Comments:
cc @yonigozlan 

I guess as quick fix, we need to standardize processor kwargs API for videoLLM also since those models can work in image-text-to-text setting. Which is why I am not surprised that users want to apply image-text-to-text pipeline, as the closest pipeline from existing ones

PS. I had an idea to add video-text-to-text as pipeline, but we know that all video models are also image models. So, not sure how to exactly separate these two. Anyway pipeline idea will go after videos have their own standard video-processors separate from images, so we can can with that :)

Agreed for the pipeline, I'm just not sure if it will be intuitive for users to use image-text-to-text with video models.
As for this issue, thank you @inf3rnus for bringing it up! I forgot the video models in the kwargs standardization, I can make a note to address that indeed.
I also think the `legacy` kwarg for image-text-to-text models/pipeline and the corresponding deprecation warning have been there for a while so it might be time to remove them altogether?

> to remove them altogether

would be nice if we are way beyond the deprecation cycle

@zucchini-nlp 

Wrt -> making a pipeline for video-text-to-text

Honestly, my two cents would be to fold in all video-text-to-text models into the image-text-to-text pipeline and just alias it until functionality for it changes, e.g. where videos are processed in a way where they aren't processed as image samples from videos.

The other option that goes down this line of thinking and better reflects what I've seen in the code base, and I think this is better, would be to still keep them distinct in anticipation of video-text-to-text evolving over time and reuse whatever classes are used by image-text-to-text for video versions of a given model where there are video models that use components from the image text pipeline via a model's config.json.

As a user I want interface signatures to be strict with 0 ambiguity. I don't like that someone was able to put a video-text-to-text model into the image-text-to-text model "bucket". So I also think there should be some effort made to standardize top level model inputs.

While I'm throwing feedback into the pool, not sure what HF's plans are for any-to-any models, but I also would like to advocate for a standardization of any-to-any model inputs, because I think we can all feel the direction the most groundbreaking models are following and it's multi-model input where there are more than two modalities.

A deep desire of mine is to see a version of pipeline be able to handle anything that falls under the multi-modal umbrella.

So, some food for thought, keep pipeline(), whose job is to initialize a model for a specific task (leaf nodes on our taxonomy), and then notion of "multimodal_pipeline" or "category_pipeline" could then be used to do the infer logic for anything that's considered a parent on this tree.

This pattern could then be expanded if the tree depth keeps increasing. Just keep making higher order pipelines.

While I'm still going, and I don't think the juice is worth the squeeze right now because of everything moving towards multi-modal, the same pattern could be then used for any top level category, e.g. Computer Vision, NLP, etc.

@inf3rnus Thanks for such detailed feedback!

> Honestly, my two cents would be to fold in all video-text-to-text models into the image-text-to-text pipeline

Personally, I dont think it is a good idea and I am more aligned with your second thought about having no hidden-paths and ambiguities.

> As a user I want interface signatures to be strict with 0 ambiguity. I don't like that someone was able to put a video-text-to-text model into the image-text-to-text model "bucket". So I also think there should be some effort made to standardize top level model inputs.

Totally agree on this and we'll be focusing more on video modality these months. Since the multimodal LLMs are a relatively new thing, we see very different models/processing methods being used. Last year I hadn't had much time to standardize stuff on video side and was focused on adding new models or simply on image-only LLMs. I believe now we have a clearer vision of what and how it needs to be standardized


> HF's plans are for any-to-any models

While I didn't yet make a clear plan, but I feel excited about any-to-any models, or even any-to-text models which we still haven't added support for in `transformers`. The current plan I have in mind is to add more models and see what features we might need in practice as we go. I definitely want to see more any-to-any models being supported this year

@zucchini-nlp thanks for hearing me!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.