How to load a fine-tuned model and inference after running run_clip.py？
Author: gongshaojie12
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

Hi, @ydshieh after I run run_clip.py, how do I load the fine-tuned model and do inference? My inference code is as follows:
```

import requests
from PIL import Image
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("clip-roberta-finetuned")
processor = AutoProcessor.from_pretrained("clip-roberta-finetuned")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)

print("auto model probs:", probs)
```
The following error occurred:

```
D:\software\anaconda\envs\transformers\python.exe D:/NLU/transformers/examples/pytorch/contrastive-image-text/predict.py
Traceback (most recent call last):
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\feature_extraction_utils.py", line 402, in get_feature_extractor_dict
    resolved_feature_extractor_file = cached_path(
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\utils\hub.py", line 300, in cached_path
    raise EnvironmentError(f"file {url_or_filename} not found")
OSError: file clip-roberta-finetuned\preprocessor_config.json not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\NLU\transformers\examples\pytorch\contrastive-image-text\predict.py", line 6, in <module>
    processor = AutoProcessor.from_pretrained("clip-roberta-finetuned")
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\models\auto\processing_auto.py", line 249, in from_pretrained
    return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\processing_utils.py", line 182, in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\processing_utils.py", line 226, in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\models\auto\feature_extraction_auto.py", line 289, in from_pretrained
    config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\feature_extraction_utils.py", line 443, in get_feature_extractor_dict
    raise EnvironmentError(
OSError: Can't load feature extractor for 'clip-roberta-finetuned'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'clip-roberta-finetuned' is the correct path to a directory containing a preprocessor_config.json file

Process finished with exit code 1

```



### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

OSError: file clip-roberta-finetuned\preprocessor_config.json not found

### Expected behavior

load and inference success

Comments:
If I understand correctly, you have a (previously created) `clip-roberta` which is used to launch training.
The processor is not saved after the model is finetuned, just the model is saved.
You can copy the processor file(s) from `clip-roberta` to  `clip-roberta-finetuned`.

Otherwise, you can simply change

```
processor = AutoProcessor.from_pretrained("clip-roberta-finetuned")
```
to
```
processor = AutoProcessor.from_pretrained("clip-roberta")
```

Hi, @ydshieh thank you for your reply.

1, When I copy the processor file(s) from clip-roberta to clip-roberta-finetuned, and the inference code remains the same. 
The following error occurs:

```
D:\software\anaconda\envs\transformers\python.exe D:/NLU/transformers/examples/pytorch/contrastive-image-text/predict.py
Traceback (most recent call last):
  File "D:\NLU\transformers\examples\pytorch\contrastive-image-text\predict.py", line 6, in <module>
    processor = AutoProcessor.from_pretrained("clip-roberta-finetuned")
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\models\auto\processing_auto.py", line 243, in from_pretrained
    return processor_class.from_pretrained(
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\processing_utils.py", line 182, in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\processing_utils.py", line 226, in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 607, in from_pretrained
    tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
  File "D:\software\anaconda\envs\transformers\lib\site-packages\transformers\models\auto\auto_factory.py", line 573, in __getitem__
    raise KeyError(key)
KeyError: <class 'transformers.models.vision_text_dual_encoder.configuration_vision_text_dual_encoder.VisionTextDualEncoderConfig'>
```

2,When I did not copy any files from `clip-roberta` to `clip-roberta-finetuned`, and changed the `processor` from `processor = AutoProcessor.from_pretrained("clip-roberta-finetuned")` to `processor = AutoProcessor.from_pretrained("clip-roberta")` as you asked. It works fine.

There might be something wrong, I will take a look. But great to know it works in some way.


@gongshaojie12 

Could you check if you have copied all these files from `clip-roberta` to `clip-roberta-finetuned`:
```
config.json
merges.txt
preprocessor_config.json
special_tokens_map.json
tokenizer.json
tokenizer_config.json
vocab.json
```

I don't have any issue when running `AutoProcessor.from_pretrained("clip-roberta-finetuned")` when I copied all fiiles (of course, ignore the non-finetuned model file)

Hi, @ydshieh thanks. It works fine.