SAM mask-generation - crops_n_layers
Author: kusstox
Pull request: no
Labels: name: WIP

name: bug

### System Info

If I increase in a mask-generation pipeline the "crops_n_layers" to > 0. I run into issues with the batch size:

```python
from PIL import Image
from transformers import pipeline

relative_image_path = "data\image.png"
raw_image = Image.open(relative_image_path)

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
outputs = generator(raw_image, points_per_batch=64, crops_n_layers=1)
``` 

Error message:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File c:\Users\LabUser\Documents\repos\test-classification\main.py:4
      [1](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:1) # %%
      [3](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:3) generator = pipeline("mask-generation", model="facebook/sam-vit-huge", device=0)
----> [4](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:4) outputs = generator([raw_image], points_per_batch=64, crops_n_layers=1)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\mask_generation.py:166, in MaskGenerationPipeline.__call__(self, image, num_workers, batch_size, *args, **kwargs)
    [128](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:128) def __call__(self, image, *args, num_workers=None, batch_size=None, **kwargs):
    [129](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:129)     """
    [130](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:130)     Generates binary segmentation masks
    [131](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:131) 
   (...)
    [164](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:164) 
    [165](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:165)     """
--> [166](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:166)     return super().__call__(image, *args, num_workers=num_workers, batch_size=batch_size, **kwargs)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\base.py:1282, in Pipeline.__call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   [1278](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1278) if can_use_iterator:
   [1279](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1279)     final_iterator = self.get_iterator(
   [1280](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1280)         inputs, num_workers, batch_size, preprocess_params, forward_params, postprocess_params
   [1281](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1281)     )
-> [1282](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1282)     outputs = list(final_iterator)
   [1283](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1283)     return outputs
   [1284](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1284) else:

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\pt_utils.py:124, in PipelineIterator.__next__(self)
    [121](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:121)     return self.loader_batch_item()
    [123](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:123) # We're out of items within a batch
--> [124](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:124) item = next(self.iterator)
    [125](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:125) processed = self.infer(item, **self.params)
    [126](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:126) # We now have a batch of "inferred things".

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\pt_utils.py:269, in PipelinePackIterator.__next__(self)
    [266](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:266)             return accumulator
    [268](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:268) while not is_last:
--> [269](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:269)     processed = self.infer(next(self.iterator), **self.params)
    [270](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:270)     if self.loader_batch_size is not None:
    [271](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:271)         if isinstance(processed, torch.Tensor):

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\base.py:1208, in Pipeline.forward(self, model_inputs, **forward_params)
   [1206](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1206)     with inference_context():
   [1207](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1207)         model_inputs = self._ensure_tensor_on_device(model_inputs, device=self.device)
-> [1208](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1208)         model_outputs = self._forward(model_inputs, **forward_params)
   [1209](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1209)         model_outputs = self._ensure_tensor_on_device(model_outputs, device=torch.device("cpu"))
   [1210](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1210) else:

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\mask_generation.py:233, in MaskGenerationPipeline._forward(self, model_inputs, pred_iou_thresh, stability_score_thresh, mask_threshold, stability_score_offset)
    [230](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:230) original_sizes = model_inputs.pop("original_sizes").tolist()
    [231](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:231) reshaped_input_sizes = model_inputs.pop("reshaped_input_sizes").tolist()
--> [233](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:233) model_outputs = self.model(**model_inputs)
    [235](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:235) # post processing happens here in order to avoid CPU GPU copies of ALL the masks
    [236](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:236) low_resolution_masks = model_outputs["pred_masks"]

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\torch\nn\modules\module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
   [1734](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1734)     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   [1735](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1735) else:
-> [1736](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1736)     return self._call_impl(*args, **kwargs)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\torch\nn\modules\module.py:1747, in Module._call_impl(self, *args, **kwargs)
   [1742](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1742) # If we don't have any hooks, we want to skip the rest of the logic in
   [1743](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1743) # this function, and just call forward.
   [1744](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1744) if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   [1745](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1745)         or _global_backward_pre_hooks or _global_backward_hooks
   [1746](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1746)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1747](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1747)     return forward_call(*args, **kwargs)
   [1749](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1749) result = None
   [1750](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1750) called_always_called_hooks = set()

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\models\sam\modeling_sam.py:1371, in SamModel.forward(self, pixel_values, input_points, input_labels, input_boxes, input_masks, image_embeddings, multimask_output, attention_similarity, target_embedding, output_attentions, output_hidden_states, return_dict, **kwargs)
   [1368](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1368)     input_labels = torch.ones_like(input_points[:, :, :, 0], dtype=torch.int, device=input_points.device)
   [1370](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1370) if input_points is not None and image_embeddings.shape[0] != input_points.shape[0]:
-> [1371](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1371)     raise ValueError(
   [1372](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1372)         "The batch size of the image embeddings and the input points must be the same. ",
   [1373](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1373)         "Got {} and {} respectively.".format(image_embeddings.shape[0], input_points.shape[0]),
   [1374](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1374)         " if you want to pass multiple points for the same image, make sure that you passed ",
   [1375](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1375)         " input_points of shape (batch_size, point_batch_size, num_points_per_image, 3) and ",
   [1376](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1376)         " input_labels of shape (batch_size, point_batch_size, num_points_per_image)",
   [1377](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1377)     )
   [1379](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1379) sparse_embeddings, dense_embeddings = self.prompt_encoder(
   [1380](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1380)     input_points=input_points,
   [1381](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1381)     input_labels=input_labels,
   [1382](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1382)     input_boxes=input_boxes,
   [1383](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1383)     input_masks=input_masks,
   [1384](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1384) )
   [1386](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1386) low_res_masks, iou_predictions, mask_decoder_attentions = self.mask_decoder(
   [1387](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1387)     image_embeddings=image_embeddings,
   [1388](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1388)     image_positional_embeddings=image_positional_embeddings,
   (...)
   [1394](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1394)     output_attentions=output_attentions,
   [1395](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1395) )

ValueError: ('The batch size of the image embeddings and the input points must be the same. ', 'Got 5 and 1 respectively.', ' if you want to pass multiple points for the same image, make sure that you passed ', ' input_points of shape (batch_size, point_batch_size, num_points_per_image, 3) and ', ' input_labels of shape (batch_size, point_batch_size, num_points_per_image)')
```

@Rocketknight1 @amyeroberts @qubvel 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run:
```python
from PIL import Image
from transformers import pipeline

relative_image_path = "data\image.png"
raw_image = Image.open(relative_image_path)

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
outputs = generator(raw_image, points_per_batch=64, crops_n_layers=1)
``` 

### Expected behavior

It should support this change in parameter

Comments:
Hi @kusstox, this seems like a bug, yes! The argument is probably correctly supported in the SAM model itself, but the pipeline code gets multiple outputs when it's used and this causes confusion regarding the true batch size.

We'll try to get to it when we can, but if you or anyone else wants to try making a PR to fix it, feel free!

i'll give this a look @Rocketknight1 

@Rocketknight1 i have raised a PR that solves this issue . kindly review the PR so that we can resolve this at the earliest . 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale . @Rocketknight1 @qubvel the PR is up for this . comments also resolved for it . please give it a look . 

not stale , please look into it . gentle ping @Rocketknight1  @qubvel 

bro still not stale . 

Added WIP to avoid stale label