Make `pipeline` able to load `processor`
Author: qubvel
Pull request: yes

# What does this PR do?

Add `processor` to the `pipeline` and refactor tests a bit to support it. Related to 
 - #32490
 - #32059 

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32514). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @qubvel . IIRC, that PR is more about "make pipeline able to accept loading processor", but no currently existing pipeline code is changed to use processor yet (even if we specify to load it), right? 

@ydshieh yes, you are right, the [following PR](https://github.com/huggingface/transformers/pull/32490) will add processor to the ZeroShotObjectDetection pipeline. I decided to split it to simplify the review a bit.

Hi @qubvel I have a general question. The process class could load also a single tokenizer, image processor or feature extractor (if not all components are on a Hub repository). For example,

```python
from transformers import AutoProcessor
AutoProcessor.from_pretrained("bert-base-uncased")
```
will turn out to be a tokenizer instead of an instance of `ProcessorMixin`.

Therefore, in `src/transformers/pipelines/__init__.py`, this part

```python
       # Instantiate processor if needed
        if isinstance(processor, (str, tuple)):
            processor = AutoProcessor.from_pretrained(processor, _from_pipeline=task, **hub_kwargs, **model_kwargs)
```
should have some guard to make sure we really get a `ProcessorMixin` instance.

WDYT?

I will share my thoughts regarding the testing part this afternoon, somehow related to the same consideration.





## continuation of my previous comment

Once we start to have the code using `processor`, and if we support it for some currently existing pipelines, it should support:
- either not use the processor at all (i.e. currently implementation): for backward compatibility
- or completely use the processor only (but need to verify the results are the same)

## for testing

Similar consideration to the above. We should test 2 cases:

- only load a list of tokenizer, image processor and/or feature extractor: test case 1
- only load a single processor: test case 2

> I did it because in the original tiny-models JSON file we do not have separate lists for image processors and feature extractors, I found they are stored in the same list (even processors are sometimes in this list).

Yes, you are correct. It is because I treated them (image processor / feature exactor) equally, and later I decided to make `processor` join into this pipeline testing

I see better now why you need this change. Could you, at least for now, simply use the information from the summary file `processor_classes` field, and by looking at training part to decide to assign them to one of the  `tokenizer_names`, `feature_extractor_names`, `processor_names` etc.?
``


@ydshieh Thanks for reviewing, I addressed the comments, please have a look!

> Once we start to have the code using processor, and if we support it for some currently existing pipelines, it should support:
> - either not use the processor at all (i.e. currently implementation): for backward compatibility
> - or completely use the processor only (but need to verify the results are the same)

This is what I tried to achieve with granular control for each pipeline class by adding `_load_*` attributes 
https://github.com/huggingface/transformers/pull/32514#discussion_r1711361484

@ydshieh could you please have a look once again?

Hi @qubvel @ydshieh ! Just wondering if anything is blocking this PR from getting merged, as it is needed for the image-text-to-text pipeline. Thanks!

Hi @yonigozlan, just a lack of time 🙂 I will address comments this week, and hopefully it can be merged then!

cc @Rocketknight1 as well (please always ping Matt for pipelines :hugs:)

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

@Rocketknight1 can you please review the PR, if that aligns with plans for `pipeline` refactoring/standardization

I'm sorry, I missed that ping somehow! Reviewing now.

@LysandreJik thanks for reviewing, I addressed the comment: made clearer docs and removed the warning. Please have a look whenever you have bandwidth.

P.S. The test failure looks unrelated.