Adds image-guided object detection support to OWL-ViT
Author: unography
Pull request: yes

This adds support for doing object detection with OWL-ViT using query image(s)

For https://github.com/huggingface/transformers/issues/18748

cc: @alaradirik 

Comments:
Hi @alaradirik I added an initial version for the image-guided obj detection. I still have to add tests and some other cleanup, however, I've some doubts right now

1. Is the handling of query_embedding correct, while doing the mean and finding out the most dissimilar embedding?
2. How should the postprocessor handle this case, when there are no labels as such for this?
3. Any other implementation details I may have missed

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18891). All of your documentation changes will be reflected on that endpoint.

Hi @alaradirik, I made the changes as per the review comments, could you check if they're fine?

I'm working on test cases currently. In the file [here](https://github.com/huggingface/transformers/blob/main/tests/models/owlvit/test_modeling_owlvit.py#L530), is it okay if I reuse `pixel_values` itself for `query_pixel_values`?

So the above line would return 
```
return config, pixel_values, input_ids, attention_mask, pixel_values
```
and be re-used as
```
config_and_inputs = self.prepare_config_and_inputs()
config, pixel_values, input_ids, attention_mask, query_pixel_values = config_and_inputs
```

And apart from the test cases, are there any other changes that I need to make?


> Hi @alaradirik, I made the changes as per the review comments, could you check if they're fine?
> 
> I'm working on test cases currently. In the file [here](https://github.com/huggingface/transformers/blob/main/tests/models/owlvit/test_modeling_owlvit.py#L530), is it okay if I reuse `pixel_values` itself for `query_pixel_values`?
> 
> So the above line would return
> 
> ```
> return config, pixel_values, input_ids, attention_mask, pixel_values
> ```
> 
> and be re-used as
> 
> ```
> config_and_inputs = self.prepare_config_and_inputs()
> config, pixel_values, input_ids, attention_mask, query_pixel_values = config_and_inputs
> ```
> 
> And apart from the test cases, are there any other changes that I need to make?

Hi @unography, thank you for the contribution once again! 

As for your question regarding the tests, yes, it'd make sense to return `config, pixel_values, input_ids, attention_mask, pixel_values` with `OwlViTForObjectDetectionTester.prepare_config_and_inputs()`.

We can add a line to this function to create `query_pixel_values` as follows:
`query_pixel_values = floats_tensor([self.batch_size, self.num_channels, self.query_image_size, self.query_image_size])`

> Thank you for the contribution once again! The code seems to be in great shape and I just left a couple of comments regarding minor style corrections and docstrings.
> 
> The only issue is the following tests fail:
> 
> * OwlViTVisionModelTest.test_model
> * OwlViTVisionModelTest.test_model_outputs_equivalence
> * OwlViTModelTest.test_model_outputs_equivalence
> 
> I believe this is due to making `pixel_values` the main argument in `OwlViTForObjectDetection.forward()` but I couldn't pinpoint the exact issue. @ydshieh could you take a look at the test scripts when you have time?

Hi, I couldn't see any test being run by CI. Could you share the error messges?

@unography Could you follow the instruction below to refresh your CircleCI permission
https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-
so that the CI could be triggered. Thanks.

Sure @alaradirik , I'll go through the review comments and make the changes. And actually, on my local, I'm able to get the test cases passed, on running 
```
RUN_SLOW=1 pytest tests/models/owlvit/test_modeling_owlvit.py
```
I'll check once more


Hi @ydshieh , I'm not able to refresh the permission for some reason, I get an error `Something Unexpected Happened` on going to `https://app.circleci.com/settings/user`
I don't have a CircleCI account linked to my Github actually, not sure how to reset the token and run the tests

> Hi, I couldn't see any test being run by CI. Could you share the error messges?

@ydshieh, of course, here is the full error log. `return_dict` argument is causing the errors but there hasn't been any changes in the modeling or test files to cause this error.

```
―――――――――――――――――――――――――――――――――――――――――――――― OwlViTVisionModelTest.test_model ――――――――――――――――――――――――――――――――――――――――――――――

self = <tests.models.owlvit.test_modeling_owlvit.OwlViTVisionModelTest testMethod=test_model>

    def test_model(self):
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
>       self.model_tester.create_and_check_model(*config_and_inputs)

tests/models/owlvit/test_modeling_owlvit.py:181: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/models/owlvit/test_modeling_owlvit.py:123: in create_and_check_model
    self.parent.assertEqual(result.pooler_output.shape, (self.batch_size, num_patches + 1, self.hidden_size))
E   AssertionError: torch.Size([12, 32]) != (12, 257, 32)

 tests/models/owlvit/test_modeling_owlvit.py ⨯✓s✓                                                               13% █▍        

―――――――――――――――――――――――――――――――――――― OwlViTVisionModelTest.test_model_outputs_equivalence ――――――――――――――――――――――――――――――――――――

self = <tests.models.owlvit.test_modeling_owlvit.OwlViTVisionModelTest testMethod=test_model_outputs_equivalence>

    def test_model_outputs_equivalence(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        def set_nan_tensor_to_zero(t):
            t[t != t] = 0
            return t
    
        def check_equivalence(model, tuple_inputs, dict_inputs, additional_kwargs={}):
            with torch.no_grad():
                tuple_output = model(**tuple_inputs, return_dict=False, **additional_kwargs)
                dict_output = model(**dict_inputs, return_dict=True, **additional_kwargs).to_tuple()
    
                def recursive_check(tuple_object, dict_object):
                    if isinstance(tuple_object, (List, Tuple)):
                        for tuple_iterable_value, dict_iterable_value in zip(tuple_object, dict_object):
                            recursive_check(tuple_iterable_value, dict_iterable_value)
                    elif isinstance(tuple_object, Dict):
                        for tuple_iterable_value, dict_iterable_value in zip(
                            tuple_object.values(), dict_object.values()
                        ):
                            recursive_check(tuple_iterable_value, dict_iterable_value)
                    elif tuple_object is None:
                        return
                    else:
                        self.assertTrue(
                            torch.allclose(
                                set_nan_tensor_to_zero(tuple_object), set_nan_tensor_to_zero(dict_object), atol=1e-5
                            ),
                            msg=(
                                "Tuple and dict output are not equal. Difference:"
                                f" {torch.max(torch.abs(tuple_object - dict_object))}. Tuple has `nan`:"
                                f" {torch.isnan(tuple_object).any()} and `inf`: {torch.isinf(tuple_object)}. Dict has"
                                f" `nan`: {torch.isnan(dict_object).any()} and `inf`: {torch.isinf(dict_object)}."
                            ),
                        )
    
                recursive_check(tuple_output, dict_output)
    
        for model_class in self.all_model_classes:
            model = model_class(config)
            model.to(torch_device)
            model.eval()
    
            tuple_inputs = self._prepare_for_class(inputs_dict, model_class)
            dict_inputs = self._prepare_for_class(inputs_dict, model_class)
>           check_equivalence(model, tuple_inputs, dict_inputs)

tests/test_modeling_common.py:1548: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_modeling_common.py:1512: in check_equivalence
    tuple_output = model(**tuple_inputs, return_dict=False, **additional_kwargs)
/opt/miniconda3/envs/hf/lib/python3.8/site-packages/torch/nn/modules/module.py:1110: in _call_impl
    return forward_call(*input, **kwargs)
src/transformers/models/owlvit/modeling_owlvit.py:950: in forward
    return self.vision_model(
/opt/miniconda3/envs/hf/lib/python3.8/site-packages/torch/nn/modules/module.py:1110: in _call_impl
    return forward_call(*input, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OwlViTVisionTransformer(
  (embeddings): OwlViTVisionEmbeddings(
    (patch_embedding): Conv2d(3, 32, kernel_size=(2, ..., elementwise_affine=True)
      )
    )
  )
  (post_layernorm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)
)
pixel_values = tensor([[[[0.6554, 0.4061, 0.0338,  ..., 0.4825, 0.8356, 0.8248],
          [0.3508, 0.3514, 0.2522,  ..., 0.1101, 0.8...07, 0.7844, 0.0197,  ..., 0.9217, 0.2872, 0.7545],
          [0.6380, 0.8504, 0.1550,  ..., 0.4501, 0.0423, 0.5167]]]])
output_attentions = False, output_hidden_states = False, return_dict = False

    @add_start_docstrings_to_model_forward(OWLVIT_VISION_INPUTS_DOCSTRING)
    @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=OwlViTVisionConfig)
    def forward(
        self,
        pixel_values: torch.FloatTensor,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutputWithPooling]:
        r"""
        Returns:
    
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    
        hidden_states = self.embeddings(pixel_values)
        hidden_states = self.pre_layernorm(hidden_states)
        encoder_outputs = self.encoder(
            inputs_embeds=hidden_states,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
    
        last_hidden_state = encoder_outputs[0]
        pooled_output = last_hidden_state[:, 0, :]
        pooled_output = self.post_layernorm(pooled_output)
    
        return BaseModelOutputWithPooling(
            last_hidden_state=last_hidden_state,
            pooler_output=pooled_output,
>           hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
        )
E       AttributeError: 'tuple' object has no attribute 'hidden_states'

src/transformers/models/owlvit/modeling_owlvit.py:903: AttributeError

 tests/models/owlvit/test_modeling_owlvit.py ⨯sssss✓s✓✓✓✓✓ss✓✓✓✓sssss✓✓✓s✓sss✓✓✓✓✓✓✓✓✓✓✓s✓✓✓s✓✓sssss✓s✓✓✓✓✓ss✓✓ 47% ████▋     
                                             ✓✓sssss✓s✓sss✓✓✓✓✓✓✓✓✓s✓s✓✓✓ss✓                                    63% ██████▍   

――――――――――――――――――――――――――――――――――――――― OwlViTModelTest.test_model_outputs_equivalence ―――――――――――――――――――――――――――――――――――――――

self = <tests.models.owlvit.test_modeling_owlvit.OwlViTModelTest testMethod=test_model_outputs_equivalence>

    def test_model_outputs_equivalence(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        def set_nan_tensor_to_zero(t):
            t[t != t] = 0
            return t
    
        def check_equivalence(model, tuple_inputs, dict_inputs, additional_kwargs={}):
            with torch.no_grad():
                tuple_output = model(**tuple_inputs, return_dict=False, **additional_kwargs)
                dict_output = model(**dict_inputs, return_dict=True, **additional_kwargs).to_tuple()
    
                def recursive_check(tuple_object, dict_object):
                    if isinstance(tuple_object, (List, Tuple)):
                        for tuple_iterable_value, dict_iterable_value in zip(tuple_object, dict_object):
                            recursive_check(tuple_iterable_value, dict_iterable_value)
                    elif isinstance(tuple_object, Dict):
                        for tuple_iterable_value, dict_iterable_value in zip(
                            tuple_object.values(), dict_object.values()
                        ):
                            recursive_check(tuple_iterable_value, dict_iterable_value)
                    elif tuple_object is None:
                        return
                    else:
                        self.assertTrue(
                            torch.allclose(
                                set_nan_tensor_to_zero(tuple_object), set_nan_tensor_to_zero(dict_object), atol=1e-5
                            ),
                            msg=(
                                "Tuple and dict output are not equal. Difference:"
                                f" {torch.max(torch.abs(tuple_object - dict_object))}. Tuple has `nan`:"
                                f" {torch.isnan(tuple_object).any()} and `inf`: {torch.isinf(tuple_object)}. Dict has"
                                f" `nan`: {torch.isnan(dict_object).any()} and `inf`: {torch.isinf(dict_object)}."
                            ),
                        )
    
                recursive_check(tuple_output, dict_output)
    
        for model_class in self.all_model_classes:
            model = model_class(config)
            model.to(torch_device)
            model.eval()
    
            tuple_inputs = self._prepare_for_class(inputs_dict, model_class)
            dict_inputs = self._prepare_for_class(inputs_dict, model_class)
>           check_equivalence(model, tuple_inputs, dict_inputs)

tests/test_modeling_common.py:1548: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_modeling_common.py:1512: in check_equivalence
    tuple_output = model(**tuple_inputs, return_dict=False, **additional_kwargs)
/opt/miniconda3/envs/hf/lib/python3.8/site-packages/torch/nn/modules/module.py:1110: in _call_impl
    return forward_call(*input, **kwargs)
src/transformers/models/owlvit/modeling_owlvit.py:1132: in forward
    vision_outputs = self.vision_model(
/opt/miniconda3/envs/hf/lib/python3.8/site-packages/torch/nn/modules/module.py:1110: in _call_impl
    return forward_call(*input, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OwlViTVisionTransformer(
  (embeddings): OwlViTVisionEmbeddings(
    (patch_embedding): Conv2d(3, 32, kernel_size=(2, ..., elementwise_affine=True)
      )
    )
  )
  (post_layernorm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)
)
pixel_values = tensor([[[[0.4672, 0.5573, 0.4972,  ..., 0.3060, 0.1213, 0.4710],
          [0.1233, 0.0373, 0.8195,  ..., 0.5669, 0.8...20, 0.2224, 0.6059,  ..., 0.2634, 0.5912, 0.3576],
          [0.1761, 0.1272, 0.9066,  ..., 0.9368, 0.1087, 0.4829]]]])
output_attentions = False, output_hidden_states = False, return_dict = False

    @add_start_docstrings_to_model_forward(OWLVIT_VISION_INPUTS_DOCSTRING)
    @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=OwlViTVisionConfig)
    def forward(
        self,
        pixel_values: torch.FloatTensor,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutputWithPooling]:
        r"""
        Returns:
    
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    
        hidden_states = self.embeddings(pixel_values)
        hidden_states = self.pre_layernorm(hidden_states)
        encoder_outputs = self.encoder(
            inputs_embeds=hidden_states,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
    
        last_hidden_state = encoder_outputs[0]
        pooled_output = last_hidden_state[:, 0, :]
        pooled_output = self.post_layernorm(pooled_output)
    
        return BaseModelOutputWithPooling(
            last_hidden_state=last_hidden_state,
            pooler_output=pooled_output,
>           hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
        )
E       AttributeError: 'tuple' object has no attribute 'hidden_states'

src/transformers/models/owlvit/modeling_owlvit.py:903: AttributeError
```

@alaradirik 

https://github.com/huggingface/transformers/blob/bb61e30962c0a6cf866e7e8e5a75b7d86d8589c2/src/transformers/models/owlvit/modeling_owlvit.py

From the file, it looks like the latest version in this PR is different from the version that produced the error you provided above.

See 
https://github.com/huggingface/transformers/blob/bb61e30962c0a6cf866e7e8e5a75b7d86d8589c2/src/transformers/models/owlvit/modeling_owlvit.py#L893-L902
where there is 
```
        if not return_dict:
            return (last_hidden_state, pooled_output) + encoder_outputs[1:]
```
but not in your error message.

> Sure @alaradirik , I'll go through the review comments and make the changes. And actually, on my local, I'm able to get the test cases passed, on running
> 
> ```
> RUN_SLOW=1 pytest tests/models/owlvit/test_modeling_owlvit.py
> ```
> 
> I'll check once more
> 
> Hi @ydshieh , I'm not able to refresh the permission for some reason, I get an error `Something Unexpected Happened` on going to `https://app.circleci.com/settings/user` I don't have a CircleCI account linked to my Github actually, not sure how to reset the token and run the tests

I triggered it :) 

@ydshieh great, thank you! I hadn't pulled the latest changes on this branch.

@unography we can merge this PR once the remaining minor issues are addressed, thank you again for the clean implementation :)

Hi @unography! Just wanted to ask if you'll have time to work on this PR this week? 

The OWL-ViT paper will be presented at ECCV in less than 2 weeks and I can work on the remaining issues / code clean-ups if you don't have the time.

Hi @alaradirik and @unography , thanks for working on this. 

I hope this helps development, I was playing with this branch using the following code

```python
from PIL import Image
import torch

from transformers import OwlViTProcessor, OwlViTForObjectDetection

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

image = Image.open("./images/image.jpeg").convert("RGB")
query = Image.open("./images/query.png").convert("RGB")

inputs = processor(query_image=query, images=image, return_tensors="pt")

outputs = model(**inputs)

```

Unfortunately, it gives the following error

```
  File "/workspace/demo-hf.py", line 17, in <module>
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/owlvit/modeling_owlvit.py", line 1578, in forward
    query_embeds = self.embed_image_query(query_image_feats, query_feature_map)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/owlvit/modeling_owlvit.py", line 1404, in embed_image_query
    mean_sim = torch.einsum("d,id->i", mean_embeds, selected_embeddings)
  File "/opt/conda/lib/python3.8/site-packages/torch/functional.py", line 360, in einsum
    return _VF.einsum(equation, operands)  # type: ignore[attr-defined]
RuntimeError: einsum(): the number of subscripts in the equation (2) does not match the number of dimensions (1) for operand 1 and no ellipsis was given
```

This is quite surprising since tests are green. The code fails inside (see stack trace above) `OwlViTForObjectDetection.embed_image_query` I am lacking the deep knowledge of the model/original code base, but to embed an image query should we just take the first token of the last layer? 

The code fails at line `1404` since (with my example)`  both inputs are 1D tensors

```python
mean_sim = torch.einsum("d,id->i", mean_embeds, selected_embeddings) # both are 1D tensors
```

I am also curios to ask what is exactly going on in this function, maybe I am able to help somehow.

Thanks!


> Hi @unography! Just wanted to ask if you'll have time to work on this PR this week?
> 
> The OWL-ViT paper will be presented at ECCV in less than 2 weeks and I can work on the remaining issues / code clean-ups if you don't have the time.

Hi @alaradirik, I'll work on this today, but if I'm unable to continue I'll ping you and let you know. My apologies for the delay with this.

Hi @alaradirik, I've added a few fixes, but unfortunately, I'm unable to find time to contribute more right now. I hope my current changes are clear enough that you can update with remaining changes 

> ```python
> from PIL import Image
> import torch
> 
> from transformers import OwlViTProcessor, OwlViTForObjectDetection
> 
> processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")
> 
> image = Image.open("./images/image.jpeg").convert("RGB")
> query = Image.open("./images/query.png").convert("RGB")
> 
> inputs = processor(query_image=query, images=image, return_tensors="pt")
> 
> outputs = model(**inputs)
> ```

Hi @FrancescoSaverioZuppichini, sorry for my late reply and thanks for your input! Could you be running a previous version of the code? If not, could you add your system info? I can't replicate this error on my local.

> Hi @alaradirik, I've added a few fixes, but unfortunately, I'm unable to find time to contribute more right now. I hope my current changes are clear enough that you can update with remaining changes

Hi @unography, sorry for the delay! And of course, I'd be happy to finish this up, could you add me to your transformers repo as a collaborator?

I think the PR is good to go, @NielsRogge @sgugger could you do a final review when you have the time?


I will update the model card and the OWL-ViT notebooks demo with this [PR](https://github.com/huggingface/notebooks/pull/256) after this PR is merged.

> > ```python
> > from PIL import Image
> > import torch
> > 
> > from transformers import OwlViTProcessor, OwlViTForObjectDetection
> > 
> > processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
> > model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")
> > 
> > image = Image.open("./images/image.jpeg").convert("RGB")
> > query = Image.open("./images/query.png").convert("RGB")
> > 
> > inputs = processor(query_image=query, images=image, return_tensors="pt")
> > 
> > outputs = model(**inputs)
> > ```
> 
> Hi @FrancescoSaverioZuppichini, sorry for my late reply and thanks for your input! Could you be running a previous version of the code? If not, could you add your system info? I can't replicate this error on my local.

Yes :) That was fixed by @unography with a later commit

So I've tested the current branch using the same image and (more or less) the same query from the [official notebook (that uses `ViT-B/16`)](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit#inference-playground).

Params:
`min_confidence = 0.6`
`nms_threshold = 0.3` 

<img width="1512" alt="Screenshot 2022-10-21 at 15 55 29" src="https://user-images.githubusercontent.com/15908060/197212918-2170543d-6cdc-41f0-a426-f80e0212e14e.png">

I am not sure how to replicate the parameters with your implementation, but I am not able to get anything close to it

```python
from PIL import Image
import torch

from transformers import OwlViTProcessor, OwlViTForObjectDetection

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch16")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16")

image = Image.open("./images/image.jpeg").convert("RGB")
query = Image.open("./images/query.png").convert("RGB")

inputs = processor(query_image=query, images=image, return_tensors="pt")
outputs = model(**inputs)

w, h = image.size
outputs = processor.post_process(outputs, torch.tensor([[h, w]]))
print(outputs[0]['scores'].mean()) # 9.3531e-08
print(torch.where(outputs[0]['scores'] > .3)) # empty
```
Pasted below `image` and `query` 

![image](https://user-images.githubusercontent.com/15908060/197228057-a71b154f-678d-48d7-a8d0-9f3747bbd5c6.jpeg)
<img width="209" alt="query" src="https://user-images.githubusercontent.com/15908060/197228062-33044dd2-17aa-4419-aee6-a811935ea7ce.png">

Hope it helps :) 



> So I've tested the current branch using the same image and (more or less) the same query from the [official notebook (that uses `ViT-B/16`)](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit#inference-playground).
> 
> Params: `min_confidence = 0.6` `nms_threshold = 0.3`
> 
> <img alt="Screenshot 2022-10-21 at 15 55 29" width="1512" src="https://user-images.githubusercontent.com/15908060/197212918-2170543d-6cdc-41f0-a426-f80e0212e14e.png">
> 
> I am not sure how to replicate the parameters with your implementation, but I am not able to get anything close to it
> 
> ```python
> from PIL import Image
> import torch
> 
> from transformers import OwlViTProcessor, OwlViTForObjectDetection
> 
> processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch16")
> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16")
> 
> image = Image.open("./images/image.jpeg").convert("RGB")
> query = Image.open("./images/query.png").convert("RGB")
> 
> inputs = processor(query_image=query, images=image, return_tensors="pt")
> outputs = model(**inputs)
> 
> w, h = image.size
> outputs = processor.post_process(outputs, torch.tensor([[h, w]]))
> print(outputs[0]['scores'].mean()) # 9.3531e-08
> print(torch.where(outputs[0]['scores'] > .3)) # empty
> ```
> 
> Pasted below `image` and `query`
> 
> ![image](https://user-images.githubusercontent.com/15908060/197228057-a71b154f-678d-48d7-a8d0-9f3747bbd5c6.jpeg) <img alt="query" width="209" src="https://user-images.githubusercontent.com/15908060/197228062-33044dd2-17aa-4419-aee6-a811935ea7ce.png">
> 
> Hope it helps :)

Same problem. A working demo would be helpful to users.

Hey @NielsRogge @sgugger, I added an `OwlViTForImageGuidedObjectDetection` head class in order to keep the forward signature of `OwlViTForObjectDetection` the same but I'm guessing I need to create separate repos on the hub for this class. Is there a way around this? Or shall I add image guided object detection as a method of the `OwlViTForObjectDetection` class?

> Hey @NielsRogge @sgugger, I added an `OwlViTForImageGuidedObjectDetection` head class in order to keep the forward signature of `OwlViTForObjectDetection` the same but I'm guessing I need to create separate repos on the hub for this class. Is there a way around this?

Why? If the model is structured the same way, the weights should be loaded from the checkpoint with no issue.



> > Hey @NielsRogge @sgugger, I added an `OwlViTForImageGuidedObjectDetection` head class in order to keep the forward signature of `OwlViTForObjectDetection` the same but I'm guessing I need to create separate repos on the hub for this class. Is there a way around this?
> 
> Why? If the model is structured the same way, the weights should be loaded from the checkpoint with no issue.

I meant creating separate repos for the same checkpoint to load the `OwlViTForImageGuidedObjectDetection` head but yes, it seems doable.

With that said, I ended up restructuring image guided detection as a method of the `OwlViTForObjectDetection` class since this is more consistent with the original work and `OwlViTForImageGuidedObjectDetection` would not be a trainable class as it just repurposes the pretrained zero-shot detection model.

I fixed the detection issues (redundant normalization of visual embeddings + lack of postprocessing) and created a separate [PR](https://github.com/huggingface/notebooks/pull/256) to update the [demo notebook](https://github.com/alaradirik/notebooks/blob/update-owlvit-demo/examples/zeroshot_object_detection_with_owlvit.ipynb). 

@NielsRogge @sgugger could you re-review when you have the time?

Hi @alaradirik, sorry my notifications got messed up, I was able to go through the comments only now. Do I need to change anything for merging? Upstream url or anything else?

> Hi @alaradirik, sorry my notifications got messed up, I was able to go through the comments only now. Do I need to change anything for merging? Upstream url or anything else?

Hey @unography no problem at all! I'm about to merge a clean [PR](https://github.com/huggingface/transformers/pull/20136) with the correct upstream. Could you give me your email address so that I can add you as the co-author to my commits? 

@alaradirik sure, this is my email - k4r4n.dhruv@gmail.com

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.