OWLv2 with visual prompt - alternative query embedding selection method
Author: vvmnnnkv
Pull request: no
Labels: name: Feature request

### Feature request

When using OWLv2 with visual prompt, like this:

```python
    processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble")
    model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble")

    target_image = Image.open(...)
    prompt_image = Image.open(...)
    inputs = processor(images=target_image, query_images=prompt_image, return_tensors="pt")

    # forward pass
    with torch.no_grad():
        outputs = model.image_guided_detection(**inputs)

    target_sizes = torch.Tensor([image.size[::-1]])

    results = processor.post_process_image_guided_detection(outputs=outputs, threshold=0.9, nms_threshold=0.3, target_sizes=target_sizes)
```

The results sometimes come out weird, with detection boxes looking absolutely random.
I was playing with the zero-shot detection in HF space and found that visual prompt works much worse than text prompt. Then I found that choosing query embedding from prompt image differently can yield much better results. You can see examples with 2 differently annotated outputs and prompt image also annotated with selected query embedding:  https://huggingface.co/spaces/vvmnnnkv/owlv2-visual-prompt

The feature request is to improve default method used in `Owlv2ForObjectDetection.embed_image_query` or perhaps to have an option to choose from different query embedding selection methods.

### Motivation

Existing `image_guided_detection` and `post_process_image_guided_detection` are very convenient; however, the way the query embedding is selected by default is not always optimal. Overcoming this requires figuring out the underlying reason and fiddling with code beyond these convenient methods. Maybe it makes sense to do query embedding selection differently by default or have a flag in `image_guided_detection` to choose methods.

### Your contribution

Can do a PR if you think it's worth doing 

Comments:
cc @qubvel @NielsRogge 

Hey @vvmnnnkv, thanks for opening an issue, your results look interesting! Can you describe your approach in the issue?

Hi @qubvel

The suggestion is to adjust the scoring method for selecting the query image embedding in `Owlv2ForObjectDetection.embed_image_query`. Right now, it filters candidates by bounding box area (IoU against the full input `[0, 0, 1, 1]`) and then selects an outlier by comparing candidate embeddings to the average of all embeddings.

I noticed two problems with the current method:
1 - Using IoU against the full input image: when the input image is padded, predictions may include bboxes that span the padding. Such bboxes can have the largest area but be less meaningful compared to smaller bboxes that avoid the padding.
2 - Choosing an outlier: I think the idea was to select a distinct object from the background. However, it can also do the opposite and choose a backgroundy/junky outlier, e.g., one whose box includes padding. Using such an embedding as the query can yield random results.

The suggested approach addresses these two issues:
1 - Compute IoU against the non-padded image area.
2 - Combine IoU with the objectness score (from `objectness_predictor`) to select the most large-area yet confident prediction.

Basically:
```python
...
w, h = query_image.size
max_side = max(w, h)
each_query_box = torch.tensor([[0, 0, w / max_side, h / max_side]])  # 1 - rule out padding
ious, _ = box_iou(each_query_box, each_query_pred_boxes)
...
objectnesses = torch.sigmoid(model.objectness_predictor(query_image_feats))
_, source_class_embeddings = model.class_predictor(query_image_feats)
score = objectnesses * ious   # 2 - prioritize objects with both larger area and objectness
top_obj_idx = torch.argmax(score)
query_embed = source_class_embeddings[0][top_obj_idx]
...
```

I don’t think this is ideal in all cases and it may need tuning, but it gives much better results. You can try it in [this HF space](https://huggingface.co/spaces/vvmnnnkv/owlv2-visual-prompt), it’s visible that the current method prefers large bboxes that cover the whole input image, including padded regions, but which may have a low objectness score.