Image + text + audio uniform processors
Author: molbap
Pull request: yes

# What does this PR do?

This PR is a stab at uniformizing the processors across all transformers models. If we are happy with the design, I'll expand it to all existing models. It only touches on some text + audio and text + images models as experiment subjects. Linked with #28711 which has a larger scope, and several previous discussions with team members. 

## Usage

As before, kwargs that are passed to processors at `__call__` time take priority. However, per-modality processors can be instantiated with their own kwargs, and if they are not overriden at call time, they will serve as defaults. 

Type hinting of kwargs is preserved if they are passed as structured dictionary entries
![image](https://github.com/huggingface/transformers/assets/39954772/c859e4b0-5c08-49c6-bd65-6e71d3f6c9f0)

It also works with kwargs passed without nesting:
![image](https://github.com/huggingface/transformers/assets/39954772/aa675d30-ca45-466a-86c9-41066a064c60)

Merging of kwargs and handling priority order is done in `processing_utils` through a dedicated method. 
The order of operations is as follows:
1) kwargs passed as before have highest priority to preserve BC.
```python
high_priority_kwargs = {"crop_size" = (224, 224), "padding" = "max_length"}
processor(..., **high_priority_kwargs)
```
2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API.
```python
processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": (224, 224)}})
```
3) kwargs passed during instantiation of a modality processor have fourth priority.
```python
tokenizer = tokenizer_class(..., {"padding": "max_length"})
image_processor = image_processor_class(...)
processor(tokenizer, image_processor) # will pass max_length unless overriden by kwargs at call
```
4) defaults kwargs specified at processor level have lowest priority.

```python
class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False):
    _defaults = {
        "text_kwargs": {
            "padding": "max_length",
            "max_length": 64,
        },
    }
```
## What changes:
- ~~There is now an attribute in the constructor that stores the processing kwargs needed to be passed to various encoders down the line. ~~ Not anymore, see next point
- There is now a base `ProcessingKwargs` TypedDict of kwargs that inherits from `ImagesKwargs`, `TextKwargs`, and so on.
- These nested attributes (one dictionary for text, one for images, one for audio, one for video) are typed with a `TypedDict` that does not need to be total. We can expand this typed dict (in `processing_utils`) as processors get uniformized. 
- The processors are called with the **same kwargs signature, text, images, audio, videos**. `kwargs` set by a user will always override default processing kwargs. 
- Slicing of positional args is removed and replaced by named kwargs corresponding to the modality. Order of kwargs is not constant as a consequence to preserve BC.
- Inputs (text, images, audio, videos) are now always typed in the `call`. 

## What that allows:

- We know each model has its own processing logic/way to mix inputs. This way of typing both the modalities sent to the processors AND the processing arguments allow a faster design of future processors, hence faster review, and faster merge, better usage.
- The reason for that is that we push away from the function call the actual mixing of modalities and their specific processing, which can be handled explicitly with the kwargs passed. 
- With TypedDict, type hints are preserved even in the nesting. I hesitated with pydantic/dataclasses and opted for TypedDict _because_ it is less flexible, and we want to enforce a standard.

 

## Limitations:

- This still relies on kwargs for the processing.  
- Few models tested on that PR - more will follow in other PRs.

## Tests pass (I think). What's missing:

- [x] A consistent way (with maybe logger.warning_once) to warn a user they are using a deprecated call. 
- [x] Typing of input modalities need to be improved and be more flexible especially for audio&videos
- [x] A test in the CI to check that the enforced designed is respected for a new model added. This can be done in a separate PR.
- [ ] All models PRs:
    - [x] Initial design + Align model #31197 
    - [x]  Altclip https://github.com/huggingface/transformers/pull/31198
    - [x] Blip https://github.com/huggingface/transformers/pull/31368
    - [x] Blip2 https://github.com/huggingface/transformers/pull/31368
    - [x] Bridgetower https://github.com/huggingface/transformers/pull/31368
    - [x] Chinese_clip https://github.com/huggingface/transformers/pull/31198
    - [ ] Donut
    - [ ] wav2vec2



## Who can review?

Models:

- vision models: @amyeroberts
- speech models: @sanchit-gandhi


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30511). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @edbeeching who also has quite some experience trying to standardise processors when he ported Llava to TRL

Alright, should be a bit better! I added some more typing for audio inputs to protect them a bit more. Worked on a test to enforce this structure as well, should be able to push it after :bridge_at_night: 

We might have to simply define a class for each processor, but seems to me like it's the best compromise no? 

Agree with you @ArthurZucker - I'll update the PR to have that, looks super nice


Got back to this today, I think I found a way to support currently existing kwargs passing, having correct nested type hints, and possibly the documentation as well. Logging off but will continue soon

# Update TL;DR

- :heavy_check_mark:  Passing **kwargs is supported as before
- :heavy_check_mark:  customization at tokenizer/image processor level is supported
- :heavy_check_mark:  we can now pass kwargs dictionaries per modality, typed
- :heavy_check_mark:  all models processors can now have the same signature
- :heavy_check_mark:  type hinting is preserved and automatically generated (`Unpack` is discarded for the time being)
- :negative_squared_cross_mark:  due to forward reference tricks, we have to import `PIL` and `torch`, protecting them before (read more for details)

Design is, I hope, finalized and adresses the limitations of the previous one. Porting it to models in this PR before opening PRs for all models in batches. @amyeroberts and @ArthurZucker, not finished porting but you might want to take another look on the design!

- [x] Align
- [x] Altclip
- [ ] Blip
- [ ] Blip2
- [ ] Bridgetower
- [x] Chinese_clip
- [ ] Donut
- [ ] wav2vec2 

# Details

## Rationale

The idea of this PR is to support a generic but principled way of passing kwargs per modality. So, kwargs are chosen to be `TypedDict`, one per modality. If new modalities arise, we just have to add another dict.  Default values are also stored here.

The, per model, we inherit from the base TypedDict.
```python
class AlignProcessorKwargs(ProcessingKwargs, total=False):
    """
    Inherits from `ProcessingKwargs` to provide:
        1) Additional keys that this model requires to process inputs.
        2) Default values for extra keys.
    New keys have to be defined as follows to ensure type hinting is done correctly.

    ```python
    common_kwargs: CommonKwargs = {
            **CommonKwargs.__annotations__,
        }
        text_kwargs: TextKwargs = {
            **TextKwargs.__annotations__,
            "a_new_text_boolean_key": Optional[bool],
        }
        images_kwargs: ImagesKwargs = {
            **ImagesKwargs.__annotations__,
            "a_new_image_processing_key": Optional[int]
        }
    ```

    """

    common_kwargs: CommonKwargs = {
        **CommonKwargs.__annotations__,
    }
    text_kwargs: TextKwargs = {
        **TextKwargs.__annotations__,
    }
    images_kwargs: ImagesKwargs = {
        **ImagesKwargs.__annotations__,
    }
    _defaults = {
        "text_kwargs": {
            "padding": "max_length",
            "max_length": 64,
        },
    }

```

The point is to have all `__call__` processor signatures identical across all models, so that they can be interfaced with in a uniform way.

```python
    def __call__(
        self,
        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
        images: ImageInput = None,
        audio=None,
        videos=None,
        text_kwargs: <Model>ProcessorKwargs.text_kwargs = None,
        images_kwargs: <Model>ProcessorKwargs.images_kwargs = None,
        common_kwargs: <Model>ProcessorKwargs.common_kwargs = None,
        **kwargs: AlignProcessorKwargs,
    ) -> BatchEncoding:
```

## :heavy_check_mark: Type Hinting

`get_type_hints` now does return the expanded kwargs. 

With 

```python
class AlignProcessorKwargs(ProcessingKwargs, total=False):
    common_kwargs: CommonKwargs = {
        **CommonKwargs.__annotations__,
    }
    text_kwargs: TextKwargs = {
        **TextKwargs.__annotations__,
    }
    images_kwargs: ImagesKwargs = {
        **ImagesKwargs.__annotations__,
        new_image_key: Optional[bool]

    }

    _defaults = {
        "text_kwargs": {
            "padding": "max_length",
            "max_length": 64,
        },
        "images_kwargs": {
            "new_image_key": True,
    }
```

We can instantiate a processor and verify that keys are documented. We redirect the user to the original processors (tokenizers and image processors) for an exhaustive documentation.
```python
from transformers import AlignProcessor
from typing import get_type_hints

model_id = "kakaobrain/align-base"
processor = AlignProcessor.from_pretrained(model_id)

get_type_hints(processor.__call__)
"""
{'text': typing.Union[str, typing.List[str], typing.List[typing.List[str]]],
 'images': typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, typing.List[PIL.Image.Image], typing.List[numpy.ndarray], typing.List[torch.Tensor]],
 'text_kwargs': {'add_special_tokens': typing.Optional[bool],
  'padding': typing.Union[bool, str, transformers.utils.generic.PaddingStrategy],
  'truncation': typing.Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy],
  'max_length': typing.Optional[int],
  'stride': typing.Optional[int],
  'is_split_into_words': typing.Optional[bool],
  'pad_to_multiple_of': typing.Optional[int],
  'return_token_type_ids': typing.Optional[bool],
  'return_attention_mask': typing.Optional[bool],
  'return_overflowing_tokens': typing.Optional[bool],
  'return_special_tokens_mask': typing.Optional[bool],
  'return_offsets_mapping': typing.Optional[bool],
  'return_length': typing.Optional[bool],
  'verbose': typing.Optional[bool],
  'padding_side': typing.Optional[str]},
 'images_kwargs': {'do_resize': typing.Optional[bool],
  'size': typing.Optional[typing.Dict[str, int]],
  'size_divisor': typing.Optional[int],
  'crop_size': typing.Optional[typing.Dict[str, int]],
  'resample': typing.Union[ForwardRef('PILImageResampling'), int, NoneType],
  'do_rescale': typing.Optional[bool],
  'rescale_factor': typing.Optional[float],
  'do_normalize': typing.Optional[bool],
  'image_mean': typing.Union[float, typing.List[float], NoneType],
  'image_std': typing.Union[float, typing.List[float], NoneType],
  'do_pad': typing.Optional[bool],
  'do_center_crop': typing.Optional[bool],
  'data_format': typing.Optional[transformers.image_utils.ChannelDimension],
  'input_data_format': typing.Union[transformers.image_utils.ChannelDimension, str, NoneType]},
 'common_kwargs': {'return_tensors': typing.Union[str, transformers.utils.generic.TensorType, NoneType]},
 'kwargs': transformers.models.align.processing_align.AlignProcessorKwargs,
 'return': transformers.tokenization_utils_base.BatchEncoding}
"""
```

### :heavy_check_mark: Typed dict in VScode/other IDEs

Type hinting works at the nested level, informing the user of the top-level dictionary
![image](https://github.com/huggingface/transformers/assets/39954772/1ee1d19b-2785-4746-98df-d9c35750400e)
as well as lower-level ones
![image](https://github.com/huggingface/transformers/assets/39954772/9540e77f-ed7a-439a-8454-44d91989c08c)
With each per-modality key having a type indicated
![image](https://github.com/huggingface/transformers/assets/39954772/b3d1ded4-fd7c-4083-9728-32dea204e7a7)

:wavy_dash: : We don't have type hinting of kwargs passed outside a dict, but the signature gives enough information to a user, and it's an improvement over having just `processor(..., **kwargs)` as before. 

## :heavy_check_mark: Defaults

Now that kwargs are simply typed dictionaries, their defaults have to be moved elsewhere. They are in a private `_defaults` attribute of the `<Model>ProcessorKwargs` class:
```python

    _defaults = {
        "text_kwargs": {
            "padding": "max_length",
            "max_length": 64,
        },
    }
```

## Customization

:heavy_check_mark: Processors initialized from modified per-modality processor should preserve kwargs passed at that point, unless overriden at call time. 
```python
from transformers import AlignProcessor, AutoTokenizer, AutoImageProcessor
tokenizer = AutoTokenizer.from_pretrained("kakaobrain/align-base", add_special_tokens=False, max_length=128)
image_processor = AutoImageProcessor.from_pretrained("kakaobrain/align-base", crop_size=333)
processor = AlignProcessor(image_processor, tokenizer)

# this should keep 128 of shape instead of the 64 default value in Align
processor(images=img, text=["What is that?"],  return_tensors='pt')['input_ids'].shape
# torch.Size([1, 128])
# also, no special token (id 101 for CLS in the beginning) should be present.
processor(images=img, text=["What is that?"],  return_tensors='pt')['input_ids']
# tensor([[2054, 2003, 2008, 1029,    0, ..., 0]])
# finally, the crop_size should be 333.
processor(images=img, text=["What is that?"],  return_tensors='pt')['pixel_values'].shape
# torch.Size([1, 3, 333, 333])

```

## :negative_squared_cross_mark: Forward references and imports


We cannot deal with forward references at runtime easily:

https://discuss.python.org/t/dealing-with-forward-refs-at-runtime/37558 but it should be ok in python 3.13/3.14 when PEP649 passes. So not a “right now” solution. 
If you want to go down the rabbit hole, check out https://peps.python.org/pep-0563/#keeping-the-ability-to-use-function-local-state-when-defining-annotations for the initially rejected PEP and https://peps.python.org/pep-0649/ for the actual one that we could use. 

So, the current implementation is a workaround. I stand by TypedDict as a choice, also because of this future that's guiding it in the right direction IMO: https://peps.python.org/pep-0705/#inheritance 
From that we have two options. We can import torch and PIL in the processors - after checking if they are installed - or we could use `annotations` from `__future__` as they are all strings.