SAM3 single image inference with multiple text prompt
Author: iariav
Pull request: no

Hi
I'm trying to run inference on a single image, aiming to get the bbox of objects from several different categories (e.g. "a person" and "a car").
the only example i found for prompting with multiple categories is in the "Batched Inference with Text Prompts" example, but then i need to unnecessarily duplicate my image as the # of categories.

is there a different more efficient way of achieving this? 

p.s
when i try prompting with a list of several categories and a single image i get an error.


Comments:
Hi @iariav,

Thanks for raising this question! You're absolutely right that the current batched inference example requires duplicating the image, which isn't ideal for your use case.

Current Workaround
For now, the most efficient way to handle multiple text prompts on a single image is to:

Process prompts sequentially - Run inference multiple times with the same image but different text prompts:

```
from transformers import SamProcessor, SamModel
from PIL import Image
import torch

processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")
model = SamModel.from_pretrained("facebook/sam-vit-huge")

image = Image.open("your_image.jpg")
text_prompts = ["a person", "a car", "a dog"]

all_masks = []
all_scores = []

for prompt in text_prompts:
    inputs = processor(images=image, text=prompt, return_tensors="pt")
    
    with torch.no_grad():
        outputs = model(**inputs)
    
    masks = processor.post_process_masks(
        outputs.pred_masks,
        inputs["original_sizes"],
        inputs["reshaped_input_sizes"]
    )
    scores = outputs.iou_scores
    
    all_masks.append(masks)
    all_scores.append(scores)
```

This avoids duplicating the image in memory, though it does require multiple forward passes.

Batch with image reuse - If you want to leverage batching for speed, you can create a batch where the same image is used multiple times, but store only one copy in memory:
```
# Create references to the same image object
images = [image] * len(text_prompts)  # This doesn't duplicate the actual image data
inputs = processor(images=images, text=text_prompts, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

masks = processor.post_process_masks(
    outputs.pred_masks,
    inputs["original_sizes"],
    inputs["reshaped_input_sizes"]
)
```
While this creates a batch tensor internally (which does duplicate image data), it's more efficient than manually duplicating and can leverage GPU batching.

**Regarding Your Error**
When you tried using a list of categories with a single image, you likely encountered a dimension mismatch error. This is because the current implementation expects the batch size of images to match the batch size of text prompts. The processor/model doesn't currently support automatic broadcasting of a single image across multiple prompts.

**Potential Enhancement**
Your use case makes sense and would be a valuable feature! The ideal API would be:
```

# Single image, multiple prompts (not currently supported)
inputs = processor(images=image, text=["a person", "a car", "a dog"], return_tensors="pt")
outputs = model(**inputs)
```

This would internally handle broadcasting the image features across multiple text prompts without unnecessary duplication or multiple forward passes.

Recommendation
For now, I'd suggest using option 2 (batch with image reuse) if you need speed, or option 1 (sequential processing) if memory is a concern. The performance difference will depend on your batch size and hardware.

The enhancement would involve:

Modifying the processor to accept single image + multiple text prompts
Broadcasting image embeddings internally before the cross-attention layers
Updating the model forward pass to handle this case efficiently
Would this help with your use case? Let me know if you need any clarification! 

I would love to contribute if you need any help with this . 

wow, @pankajbaid567 thank you for the prompt and detailed response!
indeed option 2 makes the most seance for now (i prefer to do only a single forward pass on each image), but the potential enhancement you detailed here is exactly what i was looking for.

i'll patiently wait for this update :)


EDIT: this code won't work on main for now, until [this PR](https://github.com/huggingface/transformers/pull/42407) is merged. Some unnecessary checks in the processing code is preventing the code below from working correctly.

Hello @iariav, thanks for the question, and thanks for your answer @pankajbaid567 ! To add to this, the most efficient way to do inference with multiple prompt on the same image is something like this:

```python
from transformers import Sam3Model, Sam3Processor
from PIL import Image
import requests

model = Sam3Model.from_pretrained("facebook/sam3-base")
processor = Sam3Processor.from_pretrained("facebook/sam3-base")

img_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
image = Image.open(requests.get(img_url, stream=True).raw)
img_inputs = processor(images=image, return_tensors="pt")

# Pre-compute vision embeddings
vision_embeds = model.get_vision_features(pixel_values=img_inputs.pixel_values)

text_prompts = ["a person", "a car", "a dog"]
all_masks = []
all_scores = []

for prompt in text_prompts:
    text_inputs = processor(text=prompt , return_tensors="pt")
    with torch.no_grad():
        outputs = model(vision_embeds=vision_embeds, **text_inputs )
    
    masks = processor.post_process_masks(
        outputs.pred_masks,
        inputs["original_sizes"],
    )
    scores = outputs.iou_scores
    
    all_masks.append(masks)
    all_scores.append(scores)
```

This way, the vision backbone is only ran once.
Note that you can do something similar if you want to use the same prompts for multiple images using `get_text_features` and `model(text_embeds=text_embeds, **image_inputs)`.

I realize this use-case is missing in the docs, so I'll open a PR to add it.
Hope this helps!

I think the same feature for video would also be beneficial. Multi-input bounding box prompts are already supported, but the multi-text prompt inputs aren't supported for the same video. 

Hey @spillai ! If you're using the latest transformers version you should be able to :).
Have a look at the second example [here](https://huggingface.co/docs/transformers/main/model_doc/sam3_video#pre-loaded-video-inference):

```python
# Add multiple text prompts (or use a list in add_text_prompt)
multi_prompt_session = processor.init_video_session(
    video=video_frames,
    inference_device=device,
    processing_device="cpu",
    video_storage_device="cpu",
    dtype=torch.bfloat16,
)

prompts = ["person", "bed", "lamp"]
processor.add_text_prompt(multi_prompt_session, prompts)

# Process video - detects objects from ALL prompts in a single pass
multi_outputs_per_frame = {}
for model_outputs in model.propagate_in_video_iterator(
    inference_session=multi_prompt_session, max_frame_num_to_track=50
):
    processed_outputs = processor.postprocess_outputs(multi_prompt_session, model_outputs)
    multi_outputs_per_frame[model_outputs.frame_idx] = processed_outputs

# Check which objects were detected by each prompt
frame_0_outputs = multi_outputs_per_frame[0]
prompt_to_obj_ids = frame_0_outputs["prompt_to_obj_ids"]
for prompt, obj_ids in prompt_to_obj_ids.items():
    print(f"{prompt}: {len(obj_ids)} objects")
```

Awesome, thanks @yonigozlan!

> Hey [@spillai](https://github.com/spillai) ! If you're using the latest transformers version you should be able to :). Have a look at the second example [here](https://huggingface.co/docs/transformers/main/model_doc/sam3_video#pre-loaded-video-inference):
> 
> # Add multiple text prompts (or use a list in add_text_prompt)
> multi_prompt_session = processor.init_video_session(
>     video=video_frames,
>     inference_device=device,
>     processing_device="cpu",
>     video_storage_device="cpu",
>     dtype=torch.bfloat16,
> )
> 
> prompts = ["person", "bed", "lamp"]
> processor.add_text_prompt(multi_prompt_session, prompts)
> 
> # Process video - detects objects from ALL prompts in a single pass
> multi_outputs_per_frame = {}
> for model_outputs in model.propagate_in_video_iterator(
>     inference_session=multi_prompt_session, max_frame_num_to_track=50
> ):
>     processed_outputs = processor.postprocess_outputs(multi_prompt_session, model_outputs)
>     multi_outputs_per_frame[model_outputs.frame_idx] = processed_outputs
> 
> # Check which objects were detected by each prompt
> frame_0_outputs = multi_outputs_per_frame[0]
> prompt_to_obj_ids = frame_0_outputs["prompt_to_obj_ids"]
> for prompt, obj_ids in prompt_to_obj_ids.items():
>     print(f"{prompt}: {len(obj_ids)} objects")

Hello. When run this code snippet, it returns only the first frame, rather propagating to other frames.

> EDIT: this code won't work on main for now, until [this PR](https://github.com/huggingface/transformers/pull/42407) is merged. Some unnecessary checks in the processing code is preventing the code below from working correctly.
> 
> Hello [@iariav](https://github.com/iariav), thanks for the question, and thanks for your answer [@pankajbaid567](https://github.com/pankajbaid567) ! To add to this, the most efficient way to do inference with multiple prompt on the same image is something like this:
> 
> from transformers import Sam3Model, Sam3Processor
> from PIL import Image
> import requests
> 
> model = Sam3Model.from_pretrained("facebook/sam3-base")
> processor = Sam3Processor.from_pretrained("facebook/sam3-base")
> 
> img_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
> image = Image.open(requests.get(img_url, stream=True).raw)
> img_inputs = processor(images=image, return_tensors="pt")
> 
> # Pre-compute vision embeddings
> vision_embeds = model.get_vision_features(pixel_values=img_inputs.pixel_values)
> 
> text_prompts = ["a person", "a car", "a dog"]
> all_masks = []
> all_scores = []
> 
> for prompt in text_prompts:
>     text_inputs = processor(text=prompt , return_tensors="pt")
>     with torch.no_grad():
>         outputs = model(vision_embeds=vision_embeds, **text_inputs )
>     
>     masks = processor.post_process_masks(
>         outputs.pred_masks,
>         inputs["original_sizes"],
>     )
>     scores = outputs.iou_scores
>     
>     all_masks.append(masks)
>     all_scores.append(scores)
> This way, the vision backbone is only ran once. Note that you can do something similar if you want to use the same prompts for multiple images using `get_text_features` and `model(text_embeds=text_embeds, **image_inputs)`.
> 
> I realize this use-case is missing in the docs, so I'll open a PR to add it. Hope this helps!

Thanks for sharing this example! I tried your code snippet. Even with computing vision_embeds only once per image, running ~20 text prompts per image is still much slower than Ultralytics SAM3SemanticPredictor (≈1+ sec vs ~0.3 sec per image).

Do you know if there’s a recommended way to efficiently batch multiple text prompts against a single vision embedding in Sam3Model? Or why it’s inherently slower than the Ultralytics predictor?

Hello @avraham1! Could you provide some code to reproduce this?

It seems that ultralytics runs sam3 at 644 input image res by default, against native sam3 resolution (1008) for transformers. You can change the input resolution for sam3 if needed for faster inference time ([see here](https://huggingface.co/docs/transformers/main/en/model_doc/sam3#custom-resolution-inference), on main branch only for now)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.