Adding OwlV2 checkpoint to Owl-vit model
Author: flavourabbit
Pull request: no

### Feature request

Google scenic team has released checkpoint of OwlV2 in the below link

https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit#pretrained-checkpoints

Could you please integrate this to HF’s transfomers?

### Motivation

OwlV2 has better performance while having simliar backbone as Owl-vit.

Please refer to performance comparison as following (https://arxiv.org/pdf/2306.09683.pdf)



### Your contribution

I don’t think I can be a main contributer especially I don’t know model converting. However, I know A to Z of transformers’ Owl-vit code so I can do some job (assign me part of the work!)

Comments:
Thanks for informing us!

We're on it, cc @ydshieh 

Hi folks, happy to share that OWLv2 is now available: https://huggingface.co/docs/transformers/main/en/model_doc/owlv2.

Hi @NielsRogge 
I found wrong results using OWLv2, but right for v1.
Test code:
```python
image_path = 'PATH TO COCO/COCO/test2017/000000415667.jpg'
raw_image = Image.open(image_path).convert('RGB')

# from transformers import Owlv2Processor, Owlv2ForObjectDetection
# owlv2_processor = Owlv2Processor.from_pretrained("google/owlv2-large-patch14-ensemble")
# owlv2_model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-large-patch14-ensemble").cuda()

from transformers import OwlViTProcessor, OwlViTForObjectDetection
owlv2_processor = OwlViTProcessor.from_pretrained("/root/.cache/huggingface/hub/models--google--owlvit-large-patch14/snapshots/faf39bb4c1f3ff75544daacd5fd327e9101ecff7")
owlv2_model = OwlViTForObjectDetection.from_pretrained("/root/.cache/huggingface/hub/models--google--owlvit-large-patch14/snapshots/faf39bb4c1f3ff75544daacd5fd327e9101ecff7").cuda()

prompts = 'Back. Car. Car window. Dog. Head. Nose. Palm tree. Stick. Truck. Window. Bench. Side mirror. Door handle.'.split('. ')
prompts = [x.strip('.').lower() for x in prompts]
threshold = 0.25
min_area_ratio = 0.01
image = raw_image
# def detect_owlv2(image, prompts):
texts = [[f"a photo of {x}" for x in prompts], ]
inputs = owlv2_processor(text=texts, images=image, return_tensors="pt")
inputs = {k: v.cuda() for k, v in inputs.items()}
outputs = owlv2_model(**inputs)
image_width, image_height = image.size
target_sizes = torch.Tensor([[image_height, image_width]])
results = owlv2_processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.)
i = 0  # Retrieve predictions for the first image for the corresponding text queries
text = texts[i]
boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]
# boxes = correct_boxes(boxes, image_height, image_width)
nms_indices = torchvision.ops.batched_nms(boxes, scores, labels, 0.5)
mask = torch.zeros(boxes.size(0)).bool()
mask[nms_indices] = True
mask[scores < threshold] = False
mask[torchvision.ops.box_area(boxes) < (min_area_ratio * image_height * image_width)] = False
mask[(boxes[:, 2] - boxes[:, 0]) < 5] = False
mask[(boxes[:, 3] - boxes[:, 1]) < 5] = False
boxes = boxes[mask]
scores = scores[mask]
labels = labels[mask]
```
Results of v1:
<img width="385" alt="image" src="https://github.com/huggingface/transformers/assets/19267349/4f618b10-827c-4ccf-9856-b5c00e86b952">

Results of v2:

<img width="394" alt="image" src="https://github.com/huggingface/transformers/assets/19267349/e0ab2ed3-57b8-4d17-ade9-60852f415271">

Both use the same code to visualize.

Hi @Hzzone,

OWLv2 requires boxes to be displayed on the padded, not the original image. See https://github.com/huggingface/transformers/issues/30131, it will be addressed