Add image-guided object detection support to OWL-ViT
Author: alaradirik
Pull request: no
Labels: name: Good Second Issue

Hi,

The [OWL-ViT](https://huggingface.co/docs/transformers/model_doc/owlvit) model is an open-vocabulary model that can be used for both zero-shot text-guided (supported) and one-shot image-guided (not supported) object detection. 

It'd be great to add support for one-shot object detection to `OwlViTForObjectDetection` such that users can query images with an image of the target object instead of using text queries - e.g. using an image of a butterfly to search for all butterfly instances in the target image. See an example below.

<img width="989" alt="Screenshot 2022-08-24 at 17 16 28" src="https://user-images.githubusercontent.com/8944735/186441941-7278676e-aecb-4c7d-b1d5-df4fb444becb.png">

To do this, we would just need to compute and use the `OwlViTModel` (alias to CLIP) embeddings of the query images instead of the text query embeddings within `OwlViTForObjectDetection.forward()`, which would take the target image + either text queries or  image queries as input. Similarly, `OwlViTProcessor` would be updated to preprocess sets of (image, text) and (image, query_image).

@sgugger @NielsRogge @amyeroberts @LysandreJik what do you think about this? Would this be something we would like to support?

Comments:
I think it would be a great addition, especially as it doesn't seem to be too much work to add. I'm guessing for the processor, and your description, the call signature would look something like this: 

`def __call__(self, text=None, query_image=None, images=None, padding="max_length", return_tensors="np", **kwargs):`

and then we check there's at most one of `text` or `query_image`? 

@amyeroberts exactly, it'd be pretty straightforward to implement. Based on the paper, image-guided detection is also less sensitive in terms of the probability threshold

Sounds good! 

Hi @amyeroberts @alaradirik, I'm happy to take this up!

@unography that would be great! You can ping me if you need any help or have questions. You can also find the relevant details in the appendix of the OWL-ViT [paper](https://arxiv.org/abs/2205.06230).



@alaradirik sure! 

just to confirm the high-level changes - 
1. `OwlViTProcessor` takes `query_image` as an additional param, and returns a dict like - `{pixel_values: ..., query_pixel_values: ...`
2. `OwlViTForObjectDetection.forward` takes this `query_pixel_values` as additional param
3. `image_image_embedder`, similar to `image_text_embedder`, takes this query values and returns `query_embeds`, and then we do detection on this

Does this seem correct?

@unography that seems correct. The `image_image_embedder()` method would be almost the same as the `image_text_embedder()` but would compute `query_image_embeds `instead of `text_embeds`.

However, there will be some changes to the `image_text_embedder()` method as calling the `OwlViTModel.get_text_features` and `OwlViTModel.get_image_features` within `OwlViTForObjectDetectionModel `causes memory leaks. This will be fixed in this [PR](https://github.com/huggingface/transformers/pull/18734), so it'd be great if you could wait until it is merged.

@alaradirik sure, will wait for it to get merged before proceeding with this

Hi @unography, just wanted to give you an update, the memory leak issue is fixed with this merged [PR](https://github.com/huggingface/transformers/pull/18734).

You can go ahead working on this issue if you want :)

sure, will do, thanks for informing!

Hey! I had a question about image-guided object detection. I've tested it and it works very well, but when a desired object isn't in the image, it will try to create bounding boxes just about anywhere, with a fairly high confidence score. do you have any idea to counter it?

> Hey! I had a question about image-guided object detection. I've tested it and it works very well, but when a desired object isn't in the image, it will try to create bounding boxes just about anywhere, with a fairly high confidence score. do you have any idea to counter it?

@PerrinAntonin 
Still having the same problem

> > Hey! I had a question about image-guided object detection. I've tested it and it works very well, but when a desired object isn't in the image, it will try to create bounding boxes just about anywhere, with a fairly high confidence score. do you have any idea to counter it?
> 
> @PerrinAntonin 
> Still having the same problem

Hey! Unfortunately yes... But I switch to another methods, my initial goal was to make a "Google image search" like, so I used Segment anything and then a vit to extract the features and sort the detected objects.

cc @PerrinAntonin @flavourabbit This has been resolved in  #25608 and is part of the recent release