Automatic mask generation with fine-tuned SAM model
Author: roboticsbrian
Pull request: no

### System Info

- `transformers` version: 4.33.1
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.19.0
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help?

@younesbelkada 
@rafaelpadilla 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Fine-tuned a `SamModel` and saved its weights using `.save_pretrained()`
2.  Try to create a new `MaskGenerationPipeline` instance using the `pipeline("mask-generation")` factory and fail:
```
>>>  generator = pipeline("mask-generation", model="/tmp/finetuned.wt")
loading configuration file /tmp/finetuned.wt/config.json
Model config SamConfig {
  "_name_or_path": "/tmp/finetuned.wt",
  "architectures": [
    "SamModel"
  ],
  "initializer_range": 0.02,
  "mask_decoder_config": {
    "model_type": ""
  },
  "model_type": "sam",
  "prompt_encoder_config": {
    "model_type": ""
  },
  "torch_dtype": "float32",
  "transformers_version": "4.33.1",
  "vision_config": {
    "dropout": 0.0,
    "initializer_factor": 1.0,
    "intermediate_size": 6144,
    "model_type": "",
    "projection_dim": 512
  }
}

loading configuration file /tmp/finetuned.wt/config.json
Model config SamConfig {
  "_name_or_path": "/tmp/finetuned.wt",
  "architectures": [
    "SamModel"
  ],
  "initializer_range": 0.02,
  "mask_decoder_config": {
    "model_type": ""
  },
  "model_type": "sam",
  "prompt_encoder_config": {
    "model_type": ""
  },
  "torch_dtype": "float32",
  "transformers_version": "4.33.1",
  "vision_config": {
    "dropout": 0.0,
    "initializer_factor": 1.0,
    "intermediate_size": 6144,
    "model_type": "",
    "projection_dim": 512
  }
}

loading weights file /tmp/finetuned.wt/pytorch_model.bin
All model checkpoint weights were used when initializing SamModel.

All the weights of SamModel were initialized from the model checkpoint at /tmp/finetuned.wt.
If your task is similar to the task the model of the checkpoint was trained on, you can already use SamModel for predictions without further training.
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/tmp/ipykernel_24/577173567.py in <module>
----> 1 generator = pipeline("mask-generation", model="/tmp/finetuned.wt")

/opt/conda/lib/python3.8/site-packages/transformers/pipelines/__init__.py in pipeline(task, model, config, tokenizer, feature_extractor, image_processor, framework, revision, use_fast, token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
    944         # Instantiate image_processor if needed
    945         if isinstance(image_processor, (str, tuple)):
--> 946             image_processor = AutoImageProcessor.from_pretrained(
    947                 image_processor, _from_pipeline=task, **hub_kwargs, **model_kwargs
    948             )

/opt/conda/lib/python3.8/site-packages/transformers/models/auto/image_processing_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    342         kwargs["_from_auto"] = True
    343 
--> 344         config_dict, _ = ImageProcessingMixin.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)
    345         image_processor_class = config_dict.get("image_processor_type", None)
    346         image_processor_auto_map = None

/opt/conda/lib/python3.8/site-packages/transformers/image_processing_utils.py in get_image_processor_dict(cls, pretrained_model_name_or_path, **kwargs)
    327             try:
    328                 # Load from local folder or from cache or download from model Hub and cache
--> 329                 resolved_image_processor_file = cached_file(
    330                     pretrained_model_name_or_path,
    331                     image_processor_file,

/opt/conda/lib/python3.8/site-packages/transformers/utils/hub.py in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)
    398         if not os.path.isfile(resolved_file):
    399             if _raise_exceptions_for_missing_entries:
--> 400                 raise EnvironmentError(
    401                     f"{path_or_repo_id} does not appear to have a file named {full_filename}. Checkout "
    402                     f"'https://huggingface.co/{path_or_repo_id}/{revision}' for available files."

OSError: /tmp/finetuned.wt does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co//tmp/finetuned.wt/None' for available files.

```

### Expected behavior

I was expecting to get a working mask generator instance to use, similar to what you get with `pipeline("mask-generation", model="facebook/sam-vit-base")`

I realize I can get a mask generator instance from something like:
```
generator = pipeline(
    "mask-generation", model="/tmp/finetuned.wt",
    image_processor=SamImageProcessor.from_pretrained(pretrained_model_name)
)
```
Is this how it should be done? 

Comments:
Hi,

It seems that you only saved the model files,  but not the preprocessor file in the "/tmp/finetuned.wt" folder. Hence you also need to run the following:
```
from transformers import SamImageProcessor

# assuming you fine-tuned a based-sized SAM model
processor = SamImageProcessor.from_pretrained("facebook/sam-vit-base")
processor.save_pretrained("/tmp/finetuned.wt")
```
in order to save the `preprocessor_config.json` file in the same folder.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.