OwlV2 Image Guided Object Detection Reproducibility
Author: PT-10
Pull request: no
Labels: name: bug

### System Info
Ubuntu 24.04 LTS

The example as show in https://huggingface.co/docs/transformers/en/tasks/zero_shot_object_detection#image-guided-object-detection is not reproducible

## Environment Info:
accelerate             1.12.0
torch                  2.9.1+cu130
transformers           4.57.1 




### Who can help?

@molbap 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Script to reproduce the behaviour:

```bash

import torch
import requests
from PIL import Image, ImageDraw
from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection


checkpoint = "google/owlv2-base-patch16-ensemble"

model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint)

processor = AutoProcessor.from_pretrained(checkpoint)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_target = Image.open(requests.get(url, stream=True).raw)

query_url = "http://images.cocodataset.org/val2017/000000524280.jpg"
query_image = Image.open(requests.get(query_url, stream=True).raw)

inputs = processor(images=image_target, query_images=query_image, return_tensors="pt")

with torch.no_grad():
    outputs = model.image_guided_detection(**inputs)
    target_sizes = torch.tensor([image_target.size[::-1]])
    results = processor.post_process_image_guided_detection(outputs=outputs, target_sizes=target_sizes)[0]

draw = ImageDraw.Draw(image_target)

scores = results["scores"].tolist()
boxes = results["boxes"].tolist()

for box, score in zip(boxes, scores):
    xmin, ymin, xmax, ymax = box
    draw.rectangle((xmin, ymin, xmax, ymax), outline="white", width=4)

output_path = "detected_objects.jpg"
image_target.save(output_path)
print(f"Saved to {output_path}")
```

### Expected behavior

<img width="640" height="480" alt="Image" src="https://github.com/user-attachments/assets/5b32de62-b546-48d2-9983-74f7eac0a887" /> 


### Output on local inference
<img width="640" height="480" alt="Image" src="https://github.com/user-attachments/assets/5ff1293b-d56b-474e-b6e1-f9db92b4c20b" /> \

Comments:
Issue Resolution: The example IS reproducible, but needs NMS filtering
I've tested the example code, and it does work - the methods exist and execute without errors. However, there's a legitimate issue with the output quality.

The Real Problem

The post_process_image_guided_detection() method doesn't apply sufficient Non-Maximum Suppression (NMS), resulting in hundreds of overlapping bounding boxes instead of clean detections.

Reproduction Confirmed:

Environment: transformers==4.57.1, torch==2.9.1+cu130
Status: Code runs without errors but produces poor quality output


Apply manual NMS filtering after post-processing:

"
import torch
import requests
from PIL import Image, ImageDraw
from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection
from torchvision.ops import nms

checkpoint = "google/owlv2-base-patch16-ensemble"
model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint)
processor = AutoProcessor.from_pretrained(checkpoint)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_target = Image.open(requests.get(url, stream=True).raw)
query_url = "http://images.cocodataset.org/val2017/000000524280.jpg"
query_image = Image.open(requests.get(query_url, stream=True).raw)

inputs = processor(images=image_target, query_images=query_image, return_tensors="pt")

with torch.no_grad():
    outputs = model.image_guided_detection(**inputs)
    target_sizes = torch.tensor([image_target.size[::-1]])
    results = processor.post_process_image_guided_detection(
        outputs=outputs, 
        target_sizes=target_sizes
    )[0]

boxes = results["boxes"]
scores = results["scores"]

# Apply filtering
score_threshold = 0.5
keep_mask = scores > score_threshold
boxes = boxes[keep_mask]
scores = scores[keep_mask]

# Filter small boxes
min_box_area = 5000
areas = (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])
size_mask = areas > min_box_area
boxes = boxes[size_mask]
scores = scores[size_mask]

# Apply NMS
iou_threshold = 0.05
keep_indices = nms(boxes, scores, iou_threshold)

# Keep top-k
top_k = 3
if len(keep_indices) > top_k:
    sorted_indices = keep_indices[scores[keep_indices].argsort(descending=True)[:top_k]]
    keep_indices = sorted_indices

final_boxes = boxes[keep_indices].tolist()
final_scores = scores[keep_indices].tolist()

# Draw results
draw = ImageDraw.Draw(image_target)
for box, score in zip(final_boxes, final_scores):
    xmin, ymin, xmax, ymax = box
    draw.rectangle((xmin, ymin, xmax, ymax), outline="white", width=4)

image_target.save("detected_objects.jpg")
print(f"Detected {len(final_boxes)} objects")
"


Suggested Fix
The post_process_image_guided_detection() method should include configurable parameters:

processor.post_process_image_guided_detection(
    outputs=outputs,
    target_sizes=target_sizes,
    score_threshold=0.3,      # Add this
    nms_threshold=0.1,        # Add this
    max_detections=10         # Add this
)


This would match the behavior of post_process_object_detection() which has better default filtering.


Hi! You just need to pass the thresholds you want to the post-processing method. In your case you don't want any overlapping boxes so just pass `nms_threshold` of 0.0:
```python
processor.post_process_image_guided_detection(outputs=outputs, target_sizes=target_sizes, nms_threshold=0.0)
```
will get you the expected image.

<img width="672" height="501" alt="Image" src="https://github.com/user-attachments/assets/0d5e1dc7-7e78-4476-be05-537c6952c757" />



I see, thanks