OWL-ViT outputs are offset for non-square images
Author: ghost
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-5.10.43.3-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: NA

### Who can help?

@alaradirik @sgugger @NielsRogge 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using the [code snippet](https://huggingface.co/google/owlvit-base-patch32) for OWL-ViT on a large Unsplash image ([https://images.unsplash.com/photo-1517448922956-1efc1c6cc09c](https://images.unsplash.com/photo-1517448922956-1efc1c6cc09c)) gives an incorrect result. The bounding boxes seem offset. When cropping the image, the result is actually correct.

```python
import requests
from PIL import Image
import torch

from transformers import OwlViTProcessor, OwlViTForObjectDetection

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

url = "https://images.unsplash.com/photo-1517448922956-1efc1c6cc09c"
image = Image.open(requests.get(url, stream=True).raw)
texts = [["flag", "car", "person", "sidewalk", "bicycle"]]
inputs = processor(text=texts, images=image, return_tensors="pt")
outputs = model(**inputs)

# Target image sizes (height, width) to rescale box predictions [batch_size, 2]
target_sizes = torch.Tensor([image.size[::-1]])
# Convert outputs (bounding boxes and class logits) to COCO API
results = processor.post_process(outputs=outputs, target_sizes=target_sizes)

i = 0  # Retrieve predictions for the first image for the corresponding text queries
text = texts[i]
boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]
```
This is the result: note that the yellow flag is detected, but the bounding box is offset.
![image](https://user-images.githubusercontent.com/89590365/183857750-74f624b3-d852-46db-a1ba-9da04c02600f.png)


### Expected behavior

The `post_process()` method should correctly rescale the bounding boxes to the original image size.
See the Spaces demo (which uses cropping), which shows the flag detection at the right position.
![image](https://user-images.githubusercontent.com/89590365/183858925-200d1c58-c851-4577-8518-47c8b79c7d88.png)

Comments:
I just saw that @alaradirik acknowledged this issue in the [Community tab of the Spaces demo](https://huggingface.co/spaces/adirik/OWL-ViT/discussions/1), but I'll keep this issue open, so it's easier for others to find.

I can also confirm that @cceyda 's finding works for me, i.e. doing
```python
image = Image.open(requests.get(url, stream=True).raw)
input_image = image.resize((768, 768))
inputs = processor(text=texts, images=input_image, return_tensors="pt")
```
while all other code is kept the same. It's thus not a bug in the `post_process()` method.
![image](https://user-images.githubusercontent.com/89590365/183862229-10f48f5d-9847-42b8-a6b8-a74d5ef603bd.png)


Hi @segments-tobias, thank for opening the PR! @cceyda's PR fixed the demo and I confirmed that the`post_process()` method works fine. The following code prints the boundary boxes correctly:

```
import cv2
import numpy as np
import torch

from urllib.request import urlopen
from transformers import OwlViTProcessor, OwlViTForObjectDetection

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

# Download image
url = "https://images.unsplash.com/photo-1517448922956-1efc1c6cc09c"
array = np.asarray(bytearray(urlopen(url).read()), dtype=np.uint8)
image = cv2.cvtColor(cv2.imdecode(arr, -1), cv2.COLOR_BGR2RGB)

# Text queries
texts = [["flag", "car", "person", "sidewalk", "bicycle"]]

# Target image sizes (height, width) to rescale box predictions [batch_size, 2]
target_sizes = torch.Tensor([image.shape[:2]])
img_input = cv2.resize(image, (768, 768), interpolation = cv2.INTER_AREA)
inputs = processor(text=texts, images=img_input, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# Convert outputs (bounding boxes and class logits) to COCO API
results = processor.post_process(outputs=outputs, target_sizes=target_sizes)

i = 0  # Retrieve predictions for the first image for the corresponding text queries
text = texts[i]
boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]

font = cv2.FONT_HERSHEY_SIMPLEX
score_threshold = 0.05

for box, score, label in zip(boxes, scores, labels):
    box = [int(i) for i in box.tolist()]

    if score >= score_threshold:
        image = cv2.rectangle(image, box[:2], box[2:], (255,0,0), 5)
        if box[3] + 25 > 768:
            y = box[3] - 10
        else:
            y = box[3] + 25

        image = cv2.putText(
            image, text[label], (box[0], y), font, 1, (255,0,0), 2, cv2.LINE_AA
        )
```

I think there is an issue in `OwlViTFeatureExtractor` as omitting the manual resizing line causes unexpected outputs. I'll double check this and open a fix PR shortly.

Great! Yes, would be great to be able to leave out the resizing line

yes, the `OwlViTFeatureExtractor ` is already supposed to be doing resizing according to this line [here](https://github.com/huggingface/transformers/blob/ab2006e3d6db88654526a4169e65d4bfc52da2e3/src/transformers/models/owlvit/feature_extraction_owlvit.py#L197) but it isn't working for some reason I haven't debugged.

@segments-tobias @cceyda thank you both for your input! The issue was due to defining the size as a single value instead of a tuple (768 instead of (768, 768)) in `OwlViTFeatureExtractor`. This led to the image/s getting resized along only one dimension and getting cropped along the other dimension later on in the preprocessing pipeline.

The configuration files are updated and the `OwlViTProcessor` can correctly resize the input images now. I'll open another PR to update the default values in `OwlViTFeatureExtractor` but I'm closing this issue as it is fixed.