Empty output when using facebook/sam-vit-base with automatic mask generation pipeline
Author: sunhaozhepy
Pull request: no

### System Info

- `transformers` version: 4.35.0.dev0
- Platform: Linux-5.4.15-1.el7.elrepo.x86_64-x86_64-with-glibc2.27
- Python version: 3.11.4
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes (but I grabbed 1 GPU only to my workspace bash terminal)

### Who can help?

@amyeroberts @Narsil

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Following the example notebook of automatic mask generation using SAM model, I used the following code snippet to do image segmentation:

```
import numpy as np
import matplotlib.pyplot as plt
import gc
from transformers import pipeline
from PIL import Image
import requests

def show_mask(mask, ax, random_color=False):
    if random_color:
        color = np.concatenate([np.random.random(3), np.array([0.6])], axis=0)
    else:
        color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])
    h, w = mask.shape[-2:]
    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
    ax.imshow(mask_image)
    del mask
    gc.collect()

def show_masks_on_image(raw_image, masks):
  plt.imshow(np.array(raw_image))
  ax = plt.gca()
  ax.set_autoscale_on(False)
  for mask in masks:
      show_mask(mask, ax=ax, random_color=True)
  plt.axis("off")
  plt.show()
  del mask
  gc.collect()

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)

raw_image = Image.open("test.png")

outputs = generator(raw_image, points_per_batch=64)

masks = outputs["masks"]
show_masks_on_image(raw_image, masks)
```
The original code is provided in the form of a notebook, which I arrange as a script that can be run in the terminal.


### Expected behavior

The output is empty, and no mask is shown on the image. When I insert `print(outputs)` into the code snippet, the output in the terminal is `{'masks': [], 'scores': tensor([])}`.

Comments:
cc @rafaelpadilla @younesbelkada 

Hi @sunhaozhepy 
Thanks for the issue, I have just tried it with a custom image and it seems to work great on my end. Can you maybe share the image with us so that we can reproduce and try to understand what is going on?

@sunhaozhepy, I also couldn't reproduce the error.

I tested your code with this image: 
```python
img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
```
Which resulted in:
![image](https://github.com/huggingface/transformers/assets/31217453/38d85c8d-9fe4-4dd1-a7cf-422974f99ef6)

```python
len(outputs["masks"]), len(outputs["scores"])
# 48, 48
```
If you share your `test.png`, we could track down the issue.

Hi, thanks for your replies! Here's my image `test.png`, but I strongly suspect that the image itself is not the most critical issue...

<img width="214" alt="test" src="https://github.com/huggingface/transformers/assets/73462159/0308bc03-89bb-4d24-96ff-ba13cdfe5238">

Also, I forgot to mention that for the script to be runnable in the terminal, I added a line `plt.savefig("masked_image.png")` before `del mask` to save the masked image. It looks like this:

![masked_image (1)](https://github.com/huggingface/transformers/assets/73462159/dcdd4a76-29e0-4ab3-803b-59d4b7c4b4a3)

Basically the same, without masks.

And here's an error image when I run this in my environment (masked image still saved):

```
Traceback (most recent call last):
  File "/data/jupyter/qbs_lora/sam_auto.py", line 39, in <module>
    show_masks_on_image(raw_image, masks)
  File "/data/jupyter/qbs_lora/sam_auto.py", line 28, in show_masks_on_image
    del mask
        ^^^^
UnboundLocalError: cannot access local variable 'mask' where it is not associated with a value
```

When I ran this script in colab, neither did I get empty masks, nor the error image.

Hi @sunhaozhepy,

This does not seen a problem related to the model nor the transformers library. However, I tried to replicate the error using your image and, by my side, the code finishes its execution normally..

For your given image, 15 masks were found:
```python
len(outputs["scores"]), len(outputs["masks"])
# 15, 15
```
The `mask` object only exists if the for loop is executed. So, if your `masks` list is empty (`[]`), the `mask` object is never created, and will raise this error. 

This is likely a problem related to your environment.

Ok then...Do you have any suggestions for me to fix my environment? Otherwise I can try to do my experiment in Colab or try another model.

I checked the SAM example notebook ([here](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb)) to find if there's a memory-related explanation to delete `mask` it call GC, but it is a different code.

In your code, first you call `del mask` inside the function `show_mask`. Then in `show_masks_on_image` you're trying to`del mask` again, but `mask` was has already been deleted.

Any reason why you are calling `del mask` and `gc.collect()` in `show_masks_on_image`? I think it might not be necessary.




It's [here](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb).

Never mind the environment, I've continued my experiment on Colab, and the experiment did not turn out to have good outcomes. I'll try something else...