Support subfolder with `AutoProcessor`
Author: JingyaHuang
Pull request: yes

# What does this PR do?

Support `subfolder` option in `AutoProcessor`. Currently we are not able to load processors in subfolder, eg.

```python
from transformers import AutoProcessor

AutoProcessor.from_pretrained("sentence-transformers/clip-ViT-B-32", subfolder="0_CLIPModel", trust_remote_code=False)
```

We will get following error:

```
*** OSError: sentence-transformers/clip-ViT-B-32 does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co/sentence-transformers/clip-ViT-B-32/main' for available files.
```

However I am able to load processor in subfolder with directly:

```python
import transformers

transformers.CLIPProcessor.from_pretrained.from_pretrained("sentence-transformers/clip-ViT-B-32", subfolder="0_CLIPModel", trust_remote_code=False)
```

cc. @fxmarty FYI our [`maybe_load_preprocessors`](https://github.com/huggingface/optimum/blob/037c982eee72a0c6800837a9e97e8cbefbabf08b/optimum/utils/save_utils.py#L27) is not able to store [sentence transformers clip processor](https://huggingface.co/sentence-transformers/clip-ViT-B-32/tree/main) without this.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29169). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @JingyaHuang, thanks for opening this PR! 

My understanding is that using `subfolder` was deliberately not exposed for use in the `AutoProcessor` API. There's an internal discussion here: https://huggingface.slack.com/archives/C014N4749J9/p1707474720126079

Which, tbh, is a bit surprising as it works with other Auto APIs like AutoModel, so I'd be pro adding it here. @ydshieh do you know why it was added originally? 

Thanks @amyeroberts ! Yeah it's a bit suprise, since in Optimum we store config for compiled models, and it works fine with subfolder for `AutoTokenizer` and `AutoFeatureExtractor`, until we got some issues with some CLIP checkpoints under subfolders.

From optimum perspective, it would be really nice if we could enable subfolder with AutoProcessor, otherwise we might need to override it... @ArthurZucker @amyeroberts 

Will take a look

Hi. First of all,

> subfolder was deliberately not exposed for use in the AutoProcessor API

this is not the case, and [the internal discussion](https://huggingface.slack.com/archives/C014N4749J9/p1707474720126079) is about that model type is not exposed publicly, not about `subfolder` not being used in `AutoProcessor`.

Looking at the two parts inside 

> config_dict, _ = ImageProcessingMixin.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)

and

> config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(...)

in `AutoProcessor.from_pretrained`, the first line is called without failure (and we get `config_dict`), but the second one fails.

Looking at the definitions of these 2 methods, you will see `subfolder` is used in the body of `get_image_processor_dict` but not in `get_feature_extractor_dict` (or even in the file `src/transformers/feature_extraction_utils.py`).

So, to me, it looks like `FeatureExtractor` is created (much) earlier than `ImageProcessor` and not being updated properly. @JingyaHuang 's work in this PR is the way to go.

However, there is one thing I still think strange. Let me take a deeper look before merging this PR 🙏 

even with the PR, the loading is done via the following block rather then 

```python
        # Last try: we use the PROCESSOR_MAPPING.
        elif type(config) in PROCESSOR_MAPPING:
            return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, **kwargs)
```
rather than using the `processor_class` obtained in
```python
                config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(
                    pretrained_model_name_or_path, **kwargs
                )
                processor_class = config_dict.get("processor_class", None)
```
(as the `preprocessor_config.json` file in the subfolder has no `processor_class` attribute)


No big deal. Good for me this PR.