Repository: huggingface/transformers (Python) — Bug Fix

Issue:
SAM: Notebook example not working
### System Info

- `transformers` version: 4.29.0.dev0
- Platform: macOS-13.2-arm64-arm-64bit
- Python version: 3.10.6
- Huggingface_hub version: 0.13.4
- Safetensors version: 0.3.0
- PyTorch version (GPU?): 1.13.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (cpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

Dependencies
- torch = 1.13.0
- numpy = 1.23.4

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Pull [SAM Notebook example](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb)
2. Run notebook up until 
```
masks = processor.image_processor.post_process_masks(outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu())
```
3. Get error
```
TypeError: upsample_bilinear2d() received an invalid combination of arguments - got (Tensor, list, bool, NoneType), but expected one of:
 * (Tensor input, tuple of SymInts output_size, bool align_corners, tuple of floats scale_factors)
      didn't match because some of the arguments have invalid types: (Tensor, !list!, bool, !NoneType!)
 * (Tensor input, tuple of SymInts output_size, bool align_corners, float scales_h, float scales_w, *, Tensor out)
```

### Expected behavior

original_sizes/output_sizes to be of the expected type, is this a dependency issue?


Discussion:
I have similar issue when i run 

```
img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
input_points = [[[450, 600]]]  # 2D location of a window in the image

inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(device)
outputs = model(**inputs)
```

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-abdc2d7068b8> in <module>
      4 
      5 inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(device)
----> 6 outputs = model(**inputs)
      7 
      8 masks = processor.image_processor.post_process_masks(

~/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/miniconda3/envs/pytorch/lib/python3.8/site-packages/transformers/models/sam/modeling_sam.py in forward(self, pixel_values, input_points, input_labels, input_boxes, input_masks, image_embeddings, multimask_output, output_attentions, output_hidden_states, return_dict, **kwargs)
   1331             )
   1332 
-> 1333         sparse_embeddings, dense_embeddings = self.prompt_encoder(
   1334             input_points=input_points,
   1335             input_labels=input_labels,

~/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/miniconda3/envs/pytorch/lib/python3.8/site-packages/transformers/models/sam/modeling_sam.py in forward(self, input_points, input_labels, input_boxes, input_masks)
    669             if input_labels is None:
    670                 raise ValueError("If points are provided, labels must also be provided.")
--> 671             point_embeddings = self._embed_points(input_points, input_labels, pad=(input_boxes is None))
    672             sparse_embeddings = torch.empty((batch_size, point_batch_size, 0, self.hidden_size), device=target_device)
    673             sparse_embeddings = torch.cat([sparse_embeddings, point_embeddings], dim=2)

~/miniconda3/envs/pytorch/lib/python3.8/site-packages/transformers/models/sam/modeling_sam.py in _embed_points(self, points, labels, pad)
    619             padding_point = torch.zeros(target_point_shape, device=points.device)
    620             padding_label = -torch.ones(target_labels_shape, device=labels.device)
--> 621             points = torch.cat([points, padding_point], dim=2)
    622             labels = torch.cat([labels, padding_label], dim=2)
    623         input_shape = (self.input_image_size, self.input_image_size)

RuntimeError: Expected object of scalar type double but got scalar type float for sequence element 1.
```

```

- `transformers` version: 4.29.0.dev0
- Platform: Linux-3.10.0-957.12.2.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.3
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```
cc @younesbelkada @ArthurZucker 
Thanks for reporting! Will fix this asap
Same here.

TypeError: upsample_bilinear2d() received an invalid combination of arguments - got (Tensor, list, bool, NoneType),
but expected one of:
 * (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)
      didn't match because some of the arguments have invalid types: (Tensor, !list!, bool, !NoneType!)
 * (Tensor input, tuple of ints output_size, bool align_corners, float scales_h, float scales_w, *, Tensor out)

Resolving patch:
diff --git a/src/transformers/models/sam/image_processing_sam.py b/src/transformers/models/sam/image_processing_sam.py
--- a/src/transformers/models/sam/image_processing_sam.py
+++ b/src/transformers/models/sam/image_processing_sam.py
@@ -378,12 +378,13 @@ def post_process_masks(
         Remove padding and upscale masks to the original image size.
 
         Args:
-            masks (`torch.Tensor`):
+            masks (`Union[List[torch.Tensor], List[np.ndarray]]`):
                 Batched masks from the mask_decoder in (batch_size, num_channels, height, width) format.
-            original_sizes (`torch.Tensor`):
-                The original size of the images before resizing for input to the model, in (height, width) format.
-            reshaped_input_sizes (`torch.Tensor`):
-                The size of the image input to the model, in (height, width) format. Used to remove padding.
+            original_sizes (`Union[torch.Tensor, List[Tuple[int,int]]]`):
+                The original sizes of each image before it was resized to the model's expected input shape, in (height,
+                width) format.
+            reshaped_input_sizes (`Union[torch.Tensor, List[Tuple[int,int]]]`):
+                The size of each image as it is fed to the model, in (height, width) format. Used to remove padding.
             mask_threshold (`float`, *optional*, defaults to 0.0):
                 The threshold to use for binarizing the masks.
             binarize (`bool`, *optional*, defaults to `True`):
@@ -398,9 +399,16 @@ def post_process_masks(
         requires_backends(self, ["torch"])
         pad_size = self.pad_size if pad_size is None else pad_size
         target_image_size = (pad_size["height"], pad_size["width"])
-
+        if isinstance(original_sizes, (torch.Tensor, np.ndarray)):
+            original_sizes = original_sizes.tolist()
+        if isinstance(reshaped_input_sizes, (torch.Tensor, np.ndarray)):
+            reshaped_input_sizes = reshaped_input_sizes.tolist()
         output_masks = []
         for i, original_size in enumerate(original_sizes):
+            if isinstance(masks[i], np.ndarray):
+                masks[i] = torch.from_numpy(masks[i])
+            elif not isinstance(masks[i], torch.Tensor):
+                raise ValueError("Input masks should be a list of `torch.tensors` or a list of `np.ndarray`")
             interpolated_mask = F.interpolate(masks[i], target_image_size, mode="bilinear", align_corners=False)
             interpolated_mask = interpolated_mask[..., : reshaped_input_sizes[i][0], : reshaped_input_sizes[i][1]]
             interpolated_mask = F.interpolate(interpolated_mask, original_size, mode="bilinear", align_corners=False)


Test patch:
diff --git a/tests/models/sam/test_processor_sam.py b/tests/models/sam/test_processor_sam.py
--- a/tests/models/sam/test_processor_sam.py
+++ b/tests/models/sam/test_processor_sam.py
@@ -17,8 +17,8 @@
 
 import numpy as np
 
-from transformers.testing_utils import require_torchvision, require_vision
-from transformers.utils import is_vision_available
+from transformers.testing_utils import require_torch, require_torchvision, require_vision
+from transformers.utils import is_torch_available, is_vision_available
 
 
 if is_vision_available():
@@ -26,6 +26,9 @@
 
     from transformers import AutoProcessor, SamImageProcessor, SamProcessor
 
+if is_torch_available():
+    import torch
+
 
 @require_vision
 @require_torchvision
@@ -79,3 +82,31 @@ def test_image_processor(self):
 
         for key in input_feat_extract.keys():
             self.assertAlmostEqual(input_feat_extract[key].sum(), input_processor[key].sum(), delta=1e-2)
+
+    @require_torch
+    def test_post_process_masks(self):
+        image_processor = self.get_image_processor()
+
+        processor = SamProcessor(image_processor=image_processor)
+        dummy_masks = [torch.ones((1, 3, 5, 5))]
+
+        original_sizes = [[1764, 2646]]
+
+        reshaped_input_size = [[683, 1024]]
+        masks = processor.post_process_masks(dummy_masks, original_sizes, reshaped_input_size)
+        self.assertEqual(masks[0].shape, (1, 3, 1764, 2646))
+
+        masks = processor.post_process_masks(
+            dummy_masks, torch.tensor(original_sizes), torch.tensor(reshaped_input_size)
+        )
+        self.assertEqual(masks[0].shape, (1, 3, 1764, 2646))
+
+        # should also work with np
+        dummy_masks = [np.ones((1, 3, 5, 5))]
+        masks = processor.post_process_masks(dummy_masks, np.array(original_sizes), np.array(reshaped_input_size))
+
+        self.assertEqual(masks[0].shape, (1, 3, 1764, 2646))
+
+        dummy_masks = [[1, 0], [0, 1]]
+        with self.assertRaises(ValueError):
+            masks = processor.post_process_masks(dummy_masks, np.array(original_sizes), np.array(reshaped_input_size))