SAM discrepancies between automatic mask generation and bbox/point guided
Author: rb-synth
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.0-1043-aws-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts 

cc @ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Use example from SAM REAMDE. Remove the coordinate to ask SAM to predict all objects in image.

```python
import torch
from PIL import Image
import requests
from transformers import SamModel, SamProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"
model = SamModel.from_pretrained("facebook/sam-vit-huge").to(device)
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")

img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
# input_points = [[[450, 600]]]  # 2D location of a window in the image

inputs = processor(raw_image, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model(**inputs)

masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu()
)
masks = [Image.fromarray(m.numpy()) for m in masks[0][0]]
scores = outputs.iou_scores
for i, m in enumerate(masks):
    m.save(f"mask_{i}.png")
```

![image](https://github.com/huggingface/transformers/assets/135021519/0ae178df-90a6-4c0f-8360-8d39d9b09224)
![image](https://github.com/huggingface/transformers/assets/135021519/47e1238e-63d8-4843-9a9c-1cbdd490e9aa)
![image](https://github.com/huggingface/transformers/assets/135021519/1c8ec9b8-ae20-4a70-a14d-a439c3da22f7)


### Expected behavior

Should segment regions

Comments:
Hi @rb-synth 
Thanks for the issue and your interest for using SAM ! 
Do you observe the same behaviour when using the original repository? I think that this is expected as you always need to provide at least a single 2D point or a bounding box in order for the model to predict accurately segmentation masks. 

Please follow this notebook shared in the documentation: https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb for using SAM

For automatically retrieving all potential masks given an image, one should use the `automatic-mask-generation` pipeline which should lead to identical results than original repository - please check out this notebook on how to use it: https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb 

Let us know if you have more questions or if anything is unclear

Right, I see! Could we not embed the generator inside the `__call__` function so that if no mask/boundingbox/points are given, it will fall back to this?

Hi @rb-synth 
Hmm I am afraid this would be quite breaking for users that already use SAM for fine-tuning or use SAM for raw inference without any input (image only) :/ 

Please refer to the documentation and go over the resources shared in the overview section: https://huggingface.co/docs/transformers/v4.35.0/en/model_doc/sam#overview to understand all different type of usage (fine-tuning, AMG, pure inference ...)

Ok, thanks for the help. I want to enable the case where the user may give input, in which case I use `SamModel`, but fall back to `MaskGenerationPipeline` when no extra inputs are given. I don't want to have two models saved in memory, so I tried the following:

```python
from PIL import Image
import requests
from transformers import MaskGenerationPipeline, SamModel
import torch

model = SamModel.from_pretrained("facebook/sam-vit-huge", low_cpu_mem_usage=True, torch_dtype=torch.float16)
model.to("cuda")
generator = MaskGenerationPipeline(model=model)

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")

outputs = generator(raw_image, points_per_batch=64)
```
But here it seems that the image processor does not get set
![image](https://github.com/huggingface/transformers/assets/135021519/1ac2e5c4-72b1-4c56-9bbd-051d522e392d)


Any advice on how to simultaneously run `SamModel` and `MaskGenerationPipeline` with only one model in memory?

Working with 

```python
from PIL import Image
import requests
from transformers import MaskGenerationPipeline, SamModel, SamImageProcessor
import torch

model_name = "facebook/sam-vit-huge"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

image_processor = SamImageProcessor.from_pretrained(model_name)
model = SamModel.from_pretrained(model_name, low_cpu_mem_usage=True)
model.to(device)
generator = MaskGenerationPipeline(model=model, image_processor=image_processor, device=device)

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")

with torch.no_grad():
    outputs = generator(raw_image, points_per_batch=64)
```

Now I'm noticing that ``MaskGenerationPipeline`` exposes args such as ``pred_iou_thresh``, but ``SamModel`` doesn't. How can I get the same behaviour from ``SamModel``?

I guess globally, I'm confused by what appears to be a lack of symmetry between the classes.

@rb-synth 
the automatic mask generation pipeline is quite complex, please have a look at the source code here: https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/mask_generation.py#L221 , I am personally not in favor of adding the automatic mask generation pipeline inside the model forward itself. The AMG pipeline itself uses the standalone model to get the raw mask and process them (not easily!) for every randomly generated points, hence, that entire processing logic should live inside an appropriate pipeline that abstracts everything for the users. 

Ok, perhaps I'm fundamentally misunderstanding something, I imagined the process of AMG to be identical as the normal forward pass with points sampled at regular intervals. Which is why I was surprised that the implementations are different. I appreciate the for AMG it needs to be wrapped into mini-batches to save time, of course, but I suppose the same could be done for many manually requested points. 

That aside, how can I apply `pred_iou_thresh` and `nms` to results  coming out from `SamModel`?

Any updates on how to harmonise the automatic mask generation with the rest of the SAM code base?

Hi @rb-synth 
Sorry for getting back late on this issue, unfortunately currently there is no plan on harmonising the model's forward to comply with AMG pipeline because of the reasons stated in https://github.com/huggingface/transformers/issues/27305#issuecomment-1794770681 . If you think that you can still make it through a PR, I'd be happy to review it and merge it if it makes sense but thinking about it IMO there is no way we can effectively do this currently

okay, thanks for the update and thanks for the effort!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.