Fall back to slow image processor in ImageProcessingAuto when no fast processor available
Author: yonigozlan
Pull request: yes
Labels: name: Vision

name: Processing

# What does this PR do?

Refactor parts of image_processing_auto to fall back on slow processor when use_fast is set to True and no fast processor is available.

Before, this would throw an error:
```python
processor = AutoImageProcessor.from_pretrained("Salesforce/blip-image-captioning-large", use_fast=True)
```
Now the following warning is displayed
```
`use_fast` is set to `True` but the image processor class does not have a fast version. Falling back to the slow version.
```
Also add warnings to start rolling out fast image processor by default (goal is v4.48). If use_fast is not set, and the checkpoint was saved with a slow processor, display:
```
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
```
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34785). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @yonigozlan! Thanks for opening a PR! Please check [this comment](https://github.com/huggingface/transformers/pull/31424#discussion_r1651356882) regarding enabling `use_fast=True` by default. The main concern was that fast and slow image processors might be not equivalent, so instead of forcing fast image processors we notify users that fast image processor is available. WDYT?

Oh, I hadn't seen this, thanks for pointing it out @qubvel!
Indeed there are tiny differences between fast and slow image processors outputs, which could be confusing for users as it will result in minor differences in model outputs (not necessarily worse though). My main concern is that most users will ignore or won't notice the warning as nothing is going clearly wrong during inference, but they will miss out on using fast processors with our current vision models.  Since slow processors are a major speed bottleneck for many of them, this seems like a missed opportunity. Also it looks like there are currently no way to use fast processors in pipelines, whereas this PR would enable their used there too (I still have to check this).

I guess it comes down to prioritizing backward compatibility or inference speed.  Maybe we could add a `warning_once` in this PR, stating that while a slow processor was saved, Transformers v4.x automatically loads fast processors, so users might see minor output differences, and insist on the gains in inference speed.  Something like: 

```
A slow processor was saved with this model. However, since Transformers v4.x, we automatically load fast processors for improved inference speed. 
This may result in minor differences in outputs. To force the use of a slow processor, set `use_fast=False`.
```

This way, users are aware of the potential difference but also understand the benefit, and they can still choose to use the slow processor if needed.

Happy to discuss all of this @qubvel @ydshieh @molbap.

@qubvel Agreed it might be better to do a progressive rollout and add more base tests in the meantime. I can change this PR to do that and also keep some of the refactoring of `image_processing_auto` in this PR as I feel there was some bits of confusing code (especially regarding not separating `image_processor_type` and `image_processor_class`).
I can also add support for fast image processors to output other format than torch tensors, unless there is any objection against that.

> I can also add support for fast image processors to output other format than torch tensors, unless there is any objection against that.

Considering the usage of TF/Flax, I am not sure if it is worth the effort (if it will require more than expected effort). You can talk to core maintainers about this part.

> @qubvel Agreed it might be better to do a progressive rollout and add more base tests in the meantime. I can change this PR to do that and also keep some of the refactoring of image_processing_auto in this PR as I feel there was some bits of confusing code (especially regarding not separating image_processor_type and image_processor_class).

@yonigozlan sounds great!

> Inform users well ahead of time about the upcoming behavior change. For instance, announce that Fast image processors will become the default in 3–5 releases. Users who wish to retain the current behavior should explicitly set use_fast=False in their code.

I think we should start rolling this out (the warnings) and reduce to 2 releases for example (2 months) giving us time to tests! 

Absolutely no need to work on TF / Jax versions for now either! ➕ 

@ArthurZucker Changed the scope of this PR to fix the error when use_fast is set to True and no fast image processor is available. Also added warnings to start rolling out fast image processors by default.