Owlvit's Image Guided Detection Broken
Author: flavourabbit
Pull request: no

### System Info

Colab env. for better reproducibility (transformers==4.32.0.dev0)
I ran the example for several times but it still give wrong results.

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb


Dear @amyeroberts , @ArthurZucker

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Reproduction is easy,

1) Just run the following Colab Notebook to the bottom (https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb)

The last cell of the notebook, it is now showing one bounding box

```python
results = processor.post_process_image_guided_detection(outputs=outputs, threshold=0.6, nms_threshold=0.3, target_sizes=target_sizes)
boxes, scores = results[0]["boxes"], results[0]["scores"]
print(scores)
```

The above code shows weird output, tensor([1.0000])

2) Interestingly, if I change the query image to something irrelevant, like

```python
skimg = skimage.data.astronaut()
query_image = Image.fromarray(np.uint8(skimg)).convert("RGB")
# ... omiited lines
print(scores)
```
The above shows tensor([1.0000]) as well with a giant bbox.

### Expected behavior

The last cell of the notebook, it should show two bounding boxes for each cat in the image.
(finding two cat bboxes according to the query image)

Comments:
```python
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, OwlViTForObjectDetection

processor = AutoProcessor.from_pretrained("google/owlvit-base-patch16")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
# query_url = "http://images.cocodataset.org/val2017/000000001675.jpg"
# query_image = Image.open(requests.get(query_url, stream=True).raw)
query_image = skimage.data.astronaut()
query_image = Image.fromarray(np.uint8(query_image)).convert("RGB")

inputs = processor(images=image, query_images=query_image, return_tensors="pt")
with torch.no_grad():
    outputs = model.image_guided_detection(**inputs)
# Target image sizes (height, width) to rescale box predictions [batch_size, 2]
target_sizes = torch.Tensor([image.size[::-1]])
# Convert outputs (bounding boxes and class logits) to COCO API
results = processor.post_process_image_guided_detection(
    outputs=outputs, threshold=0.6, nms_threshold=0.3, target_sizes=target_sizes
)
i = 0  # Retrieve predictions for the first image
boxes, scores = results[i]["boxes"], results[i]["scores"]
for box, score in zip(boxes, scores):
    box = [round(i, 2) for i in box.tolist()]
    print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}")
```

The above official example code does same.
(Target: cat image, query: astronaut image)

It seems that post_process_image_guided_detection normalizes `scores` where max = 1.0, so that there no threshold logic working. That means where there is object or not, the model will detect sth. However, I don't think this fits to the definition of Object Detection. 

Hi @flavourabbit, thanks for reporting this issue! 

cc @rafaelpadilla 

Hi @flavourabbit ,

Thanks for bringing this to our attention! 🙌 You've made a great observation.

Just as you highlighted, the detected objects in the target image can differ based on the query text/image and the threshold we apply.

It's interesting to note that the official implementation of the paper "[Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230)" showcases the same behavior. You can check out the official code [here](https://github.com/google-research/scenic/blob/56b770da566520ec1022fff4d5a1922f98a2d11f/scenic/projects/owl_vit/notebooks/interactive.py#L211). The scores are scaled between 0.1 and 1 and then we apply the threshold. 

Let me break it down a bit:

For the query image `000000001675.jpg` (featuring a "cat"), the detections' scores before rescaling in descending order look like:
`[0.8556, 0.7450, 0.1511, ...]`
Post-rescaling, they become: `[1.000, 0.8564, 0.0850, ...]`
Post-thresholding, only the top two scores (`1.000` and `0.8564)` remain. And so, we end up having their corresponding bounding boxes around both cats.

Now, if we switch to the query image of the astronaut, the pre-rescale scores are `[0.4259, 0.0707, 0.0441, ...]`
After rescaling, they become: `[1.0000, 0.0734, 0.0039, ...]`
After this step, only the top score (`1.000`) is retained.

This means the peak score will always adjust to `1.000`. That's why you're seeing a score of `tensor([1.0000])` in all these examples.

You've made a valid point about the potential for confusion using the query image in the colab example. 👍

Hope this clears things up! If you have any other questions or feedback, feel free to share. 😊