Adding grounding dino
Author: EduardoPach
Pull request: yes

# What does this PR do?

This PR adds Grounding DINO

Fixes https://github.com/huggingface/transformers/issues/25423

To-Do's:

- [x] Port vision backbone 
- [x] Port Text Backbone
- [x] Port Encoder
- [x] Port Decoder
- [x] Port tokenizer
- [x] Port Image processing
- [x] Validate results
- [x] Check documentation


Comments:
@EduardoPach Thanks for opening this model PR! From next week, I'll be away for a few weeks. If you need a review in that time please ping @rafaelpadilla. 

@rafaelpadilla hey, so I've finished implementing the model and validated with the original implementation. Still have to clean up some things and make sure the documentation is correct. 

My main question is about pushing the model to the hub, because the authors uploaded already the checkpoints (two checkpoints in the same repo) they made available to the model, but it's under an user instead of their org (IDEA-Research), what is usually done in this case?

Hi @EduardoPach ,

Are you referring to `groundingdino_swinb_cogcoor.pth` and `groundingdino_swint_ogc.pth`, placed [here](https://huggingface.co/ShilongLiu/GroundingDINO/tree/main), right?

In this case, let's consult @ArthurZucker and @younesbelkada. 

> Hi @EduardoPach ,
> 
> 
> 
> Are you referring to `groundingdino_swinb_cogcoor.pth` and `groundingdino_swint_ogc.pth`, placed [here](https://huggingface.co/ShilongLiu/GroundingDINO/tree/main), right?
> 
> 
> 
> In this case, let's consult @ArthurZucker and @younesbelkada. 

Precisely, I'm asking this because I had the impression that model repos contain only one checkpoint and also the IDEA-Research group has other models that we've could add to the transformers library later on so it might be helpful if there was an account for this org.

> > Hi @EduardoPach ,
> > Are you referring to `groundingdino_swinb_cogcoor.pth` and `groundingdino_swint_ogc.pth`, placed [here](https://huggingface.co/ShilongLiu/GroundingDINO/tree/main), right?
> > In this case, let's consult @ArthurZucker and @younesbelkada.
> 
> Precisely, I'm asking this because I had the impression that model repos contain only one checkpoint and also the IDEA-Research group has other models that we've could add to the transformers library later on so it might be helpful if there was an account for this org.

For now, you can upload weights to the hub under your own personal profile and use them until this PR is ready to merge. 
Afterwards, we'll move the weights under the organization on the hub, and update all the paths to point to those.

Hi @EduardoPach 
I second what @rafaelpadilla said, for the `groundingdino_swinb_cogcoor.pth` and `groundingdino_swint_ogc.pth` you can create two different repositories under your personal name space with a suffix that is distinguishable (e.g. `yournamespace/groundingdino-swinb-cogcoor` and `yournamespace/groundingdino-swint-ogc`,  and make sure the files has been renamed to `pytorch_model.bin`

@rafaelpadilla Hey! Could you help me out with these questions?

1. The `ImageProcessor` from the original implementation is exactly the same as we have in `DeformableDetr`. Should I copy the `ImageProcessor` and just remove the segmentation-related things? (Since `GroundingDINO` is used only for Object Detection)
2. Their tokenizer is the same as `Bert` with a few extra steps after tokenizing so I copied and added this step, but I'm unsure how to push the pre-trained tokenizer to the hub
3. My implementation of `GroundingDINOConfig` has an attribute called `text_backbone_config` which is a `GroundingDINOTextPrenetConfig` which is just a copy of Bert config. However, after pushing the model to the hub when I try to instantiate the model with `.from_pretrained` I get an error saying:

```
ValueError: Parameter config in `GroundingDINOTextPrenet(config)` should be an instance of class `PretrainedConfig`. To create a model from a pretrained model use `model = GroundingDINOTextPrenet.from_pretrained(PRETRAINED_MODEL_NAME)`
```
and when I do  `AutoConfig.from_pretrained("EduardoPacheco/grounding-dino-base").text_backbone_config` I get `{'model_type': 'grounding-dino-text-prenet'}` is there anything different that I need to do to have a config as an attribute? I've tried to look at `CLIP`'s configuration to get some idea of how to do it, but I'm unsure why I am not getting the full `GroundingDINOTextPrenetConfig` after pushing the model to the hub

Hi @EduardoPach ,

If your `ImageProcessor` is an exact copy from another model you must include the `#Copied from`. If somehow your `ImageProcessor` uses parts of other code, it would be good to have a `#Modified from` comment.

If I understood correctly, you have already generated the tokens using the newly extra steps, right? For pushing your tokens to the hub you could could use the hub api. See an example [here](https://huggingface.co/docs/huggingface_hub/guides/upload#upload-a-file)

I'm not sure if the problem is regarding `AutoConfig`, as it could not load correctly your `GroundingDINOConfig`. Have you tried loading it directly with `GroundingDINOTextPrenet.from_pretrained("EduardoPacheco/grounding-dino-base")`?


@rafaelpadilla the `ImageProcessor` is precisely the same, but the `DeformableDetr` one works for object detection and segmentation. Right now I've copied the processor and just removed the segmentation stuff, is that okay?

Also, about the config, sorry I had forgotten to push the modifications I've done to the `configuration_grounding_dino.py` file

**EDIT**

I figured out what the issue was haha it was somewhat dumb. Either way, I wasn't aware that when we push the config to the hub the config class is then converted to a config.json, and any Nested configuration is also modified to a dictionary so I only had to change my `GroundingDINOConfig` implementation a bit when creating the attribute `text_backbone_config`

Hi @EduardoPach do you need any help in finishing this PR? Really great to see you're leveraging `Copied from` for the text encoder and all parts taken from Deformable DETR. Also, if the image processor is exactly the same as Deformable DETR, then we typically don't add a new image processor to the library, but rather just add a line in [image_processing_auto](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/image_processing_auto.py#L55), which will allow people to do:
```
from transformers import AutoImageProcessor

image_processor = AutoImageProcessor.from_pretrained("sensetime/grounding-dino-base")
```

this will then automatically create a `DeformableDetrImageProcessor`.

> Hi @EduardoPach do you need any help in finishing this PR? Really great to see you're leveraging `Copied from` for the text encoder and all parts taken from Deformable DETR. Also, if the image processor is exactly the same as Deformable DETR, then we typically don't add a new image processor to the library, but rather just add a line in [image_processing_auto](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/image_processing_auto.py#L55), which will allow people to do:
> 
> ```
> from transformers import AutoImageProcessor
> 
> image_processor = AutoImageProcessor.from_pretrained("sensetime/grounding-dino-base")
> ```
> 
> this will then automatically create a `DeformableDetrImageProcessor`.

Writing here just for record

As we discussed through Discord I'll make that and will do the same for the `Tokenizer` part and will just create a `GroundingDINOProcessor`. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26087). All of your documentation changes will be reflected on that endpoint.

Hey @rafaelpadilla  or @NielsRogge right now I'm missing only the `test_tied_weights_keys` would love some help on this one. 

Also, for some reason when I try to run `make fixup` it throws an error so code quality tests are falling as well not sure if you could run this command @NielsRogge, since you already have the write permission to the branch. 

Definitely, after the test is fixed I'll have to review the modeling documentation to make sure everything is correct

@NielsRogge it looks good now! All tests are fine and I improved the docstrings. I think you're good to go to start the review process

@NielsRogge or @rafaelpadilla or even @amyeroberts Do I need to add anything else?

@rafaelpadilla @NielsRogge could you give another round of review? 

Hey @NielsRogge just tagging you here as a reminder 

@NielsRogge @EduardoPach Thanks for adding this model! As a core maintainer I get a lot of PR review requests so will only do a final review once another person has approved (@NielsRogge I'm assuming your approval here because you're committing) and all tests are passing. Some failing tests look unrelated, but some model specific tests one are still failing. Let me know if you need help getting them to pass! 

> @NielsRogge @EduardoPach Thanks for adding this model! As a core maintainer I get a lot of PR review requests so will only do a final review once another person has approved (@NielsRogge I'm assuming your approval here because you're committing) and all tests are passing. Some failing tests look unrelated, but some model specific tests one are still failing. Let me know if you need help getting them to pass!

The model-specific tests that are failing in the `test_pr_documentation` is due to the missing repo_id `idea-research/grounding-dino-tiny`. Right now the model is under my account at `EduardoPacheco/grounding-dino-tiny` and the idea was to change ownership of the checkpoint either to one account created for the research lab or maybe to one of the authors since I believe [this account](https://huggingface.co/ShilongLiu/GroundingDINO) belongs to one of them.

c.c. @amyeroberts 

Hey @ArthurZucker, fixed the main points you brought up 

@amyeroberts or @ArthurZucker not sure which one of you has the bandwidth to review this so pinging both of you here

@EduardoPach I'm on watch this week so will review :) 

@amyeroberts didn't have the time yet to make the changes. I will probably have some time in the next couple of days otherwise next week I can work on it

@EduardoPach Happy new year! OK, thanks for the update. Let us know if you have any Qs in the meantime. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Getting back to work on this one

Thanks @amyeroberts for the review! Together with @EduardoPach, we've addressed all comments.

CI is green, one unrelated test is failing. See also my comments above regarding why we wouldn't use nested outputs, and keep the same names as in Deformable DETR.

@amyeroberts ran the slow tests with GPU on colab to double check and everything is green 
<img width="1134" alt="Screenshot 2024-03-08 at 01 21 47" src="https://github.com/huggingface/transformers/assets/69953243/f088203b-02d8-4172-9002-6f7b1640578e">


@amyeroberts here are the screenshots of `RUN_SLOW=1 pytest tests/models/grounding_dino -vv` sorry for being dumb and not knowing how to take a scrolling screenshot 😅 

<img width="1666" alt="image" src="https://github.com/huggingface/transformers/assets/69953243/9225c8db-f1d2-40e3-93d6-a9cc29a0c674">
<img width="1650" alt="image" src="https://github.com/huggingface/transformers/assets/69953243/8bec9fe6-72e9-4b24-8bda-f175fab8842d">
<img width="1635" alt="image" src="https://github.com/huggingface/transformers/assets/69953243/c7581594-4228-4551-8469-c1b3ddfd2eb5">
<img width="1631" alt="image" src="https://github.com/huggingface/transformers/assets/69953243/e39b2901-e347-4109-9c86-d81632c4d78d">



@EduardoPach 
https://github.com/EduardoPach/transformers/blob/6f13fbb5f46a8c949a02c5c087de104fdf254f67/src/transformers/models/grounding_dino/modeling_grounding_dino.py#L1436
Can you modify the GroundingDinoMultiheadAttention to support text cross-attention mask?

> @EduardoPach https://github.com/EduardoPach/transformers/blob/6f13fbb5f46a8c949a02c5c087de104fdf254f67/src/transformers/models/grounding_dino/modeling_grounding_dino.py#L1436 Can you modify the GroundingDinoMultiheadAttention to support text cross-attention mask?

Any particular reason for this? In the original implementation, they didn't use the cross-attention mask

c.c. @amyeroberts 

> > @EduardoPach https://github.com/EduardoPach/transformers/blob/6f13fbb5f46a8c949a02c5c087de104fdf254f67/src/transformers/models/grounding_dino/modeling_grounding_dino.py#L1436 Can you modify the GroundingDinoMultiheadAttention to support text cross-attention mask?
> 
> Any particular reason for this? In the original implementation, they didn't use the cross-attention mask

If text cross-attention mask is not used, we can only train and inference with batch_size == 1, which is inefficient in utilizing GPU memory.

In a batch, multiple texts will be padding to a fixed length, so we need text cross-attention mask.

> > > @EduardoPach https://github.com/EduardoPach/transformers/blob/6f13fbb5f46a8c949a02c5c087de104fdf254f67/src/transformers/models/grounding_dino/modeling_grounding_dino.py#L1436 Can you modify the GroundingDinoMultiheadAttention to support text cross-attention mask?
> 
> > 
> 
> > Any particular reason for this? In the original implementation, they didn't use the cross-attention mask
> 
> 
> 
> If text cross-attention mask is not used, we can only train and inference with batch_size == 1, which is inefficient in utilizing GPU memory.
> 
> 
> 
> In a batch, multiple texts will be padding to a fixed length, so we need text cross-attention mask.

Yeah, I was under the assumption that one would use always the same labels when doing inference, but thinking more about it I can see some cases where that wouldn't be the case and training would be one as well. 

I do think though, that we could fix this in a different PR as this PR is quite old and having a working version of the model in the main repo would be beneficial IMO and then I can work on adding the cross-attention masks as well 🤗.

Are you using the implementation for a project?

> > > > @EduardoPach https://github.com/EduardoPach/transformers/blob/6f13fbb5f46a8c949a02c5c087de104fdf254f67/src/transformers/models/grounding_dino/modeling_grounding_dino.py#L1436 Can you modify the GroundingDinoMultiheadAttention to support text cross-attention mask?
> > 
> > 
> > > 
> > 
> > 
> > > Any particular reason for this? In the original implementation, they didn't use the cross-attention mask
> > 
> > 
> > If text cross-attention mask is not used, we can only train and inference with batch_size == 1, which is inefficient in utilizing GPU memory.
> > In a batch, multiple texts will be padding to a fixed length, so we need text cross-attention mask.
> 
> Yeah, I was under the assumption that one would use always the same labels when doing inference, but thinking more about it I can see some cases where that wouldn't be the case and training would be one as well.
> 
> I do think though, that we could fix this in a different PR as this PR is quite old and having a working version of the model in the main repo would be beneficial IMO and then I can work on adding the cross-attention masks as well 🤗.
> 
> Are you using the implementation for a project?

Yes, I'm currently working on the development of extra-visual-module-based MLLM (e.g. [LISA](https://github.com/dvlab-research/LISA), [GLaMM](https://github.com/mbzuai-oryx/groundingLMM), etc.). And I'm using your code with nn.MultiheadAttention instead of GroundingDinoMultiheadAttention for training and inference.

Due to consistent Transformers code style, your code is more suitable to use with LLM (e.g. LlamaForCausalLM). Thanks for your work!

Adding here a screenshot of running the tests `RUN_SLOW=1 pytest tests/models/grounding_dino/ -vv` where we can see that the tests are passing and specific GPU test `test_inference_object_detection_head_equivalence_cpu_gpu` is green 

c..c @amyeroberts 
![image](https://github.com/huggingface/transformers/assets/69953243/74decddc-5c90-4f8a-badf-f30fec6ae68a)


I just tried out the model readme, and think it might be slightly outdated. Here are some changes I had to make:

1. the post-processor needs to be put on the device: `inputs = {k: v.to(device) for k, v in inputs.items()}`. 
2. `inputs` is a dictionary, so needs to be `inputs["input_ids"]` not `inputs.input_ids`
3. `bbox_threshold` needs to be `box_threshold`. 

Otherwise it seems to work well for me, thanks for the hard work!

> I just tried out the model readme, and think it might be slightly outdated. Here are some changes I had to make:
> 
> 1. the post-processor needs to be put on the device: `inputs = {k: v.to(device) for k, v in inputs.items()}`.
> 2. `inputs` is a dictionary, so needs to be `inputs["input_ids"]` not `inputs.input_ids`
> 3. `bbox_threshold` needs to be `box_threshold`.
> 
> Otherwise it seems to work well for me, thanks for the hard work!

Hey, thanks for the heads up. The output of `GroundingDinoProcessor` is of type `BatchEncoding` so in your first point, you can simply do `inputs = inputs.to(device)` (added that to the model readme). For your second point, if you didn't modify `inputs` to a dict it should be a `BatchEncoding` so no problem there as well. For your third point, I fix that 😄 

Good points, thanks! Last point, have you checked with a list of text prompts? The type hinting implies this should be possible (`text: List[TextInput]`), but I haven't succeeded. I tried both with and without `padding=True`:

```python
import requests

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection

model_id = "IDEA-Research/grounding-dino-tiny"
device = torch.device("cuda")
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)

image_url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)
# Check for cats and remote controls
text = ["cat", "remote control"]

inputs = processor(images=image, text=text, return_tensors="pt", padding=True).to(device)
with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_grounded_object_detection(
    outputs,
    inputs.input_ids,
    box_threshold=0.4,
    text_threshold=0.3,
    target_sizes=[image.size[::-1]]
)
```

If `padding` is not given, error is:
```
Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`input_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
```

If `padding=True` is passed, the error is:
```
The expanded size of the tensor (8) must match the existing size (4) at non-singleton dimension 2.  Target sizes: [4, 17821, 8].  Tensor sizes: [8, 1, 4]
```

I see that in your example, you separate with full stops, so a list of text could be converted with `". ".join(texts)`. I tried this on an image with some items of clothing, so the prompt was `'shirt. dress. blouse. jacket. jumper. sweater. undershirt. t-shirt. tie'`. I would expect each detected object to be one of the full-stop enclosed phrases, but the results were pretty mangled:
```
jumper
shirt -
shirt blouse undershirt t - shirt
shirt blouse jacket sweater
blouse undershirt t - shirt tie
shirt blouse sweater -
blouse sweater undershirt t - shirt
blouse jacket jumper sweater t - shirt
shirt blouse jacket sweater undershirt t - shirt
undershirt
##shirt
t - shirt
t - shirt
```
Is this to be expected?

> I see that in your example, you separate with full stops, so a list of text could be converted with `". ".join(texts)`. I tried this on an image with some items of clothing, so the prompt was `'shirt. dress. blouse. jacket. jumper. sweater. undershirt. t-shirt. tie'`. I would expect each detected object to be one of the full-stop enclosed phrases, but the results were pretty mangled:
> 
> ```
> jumper
> shirt -
> shirt blouse undershirt t - shirt
> shirt blouse jacket sweater
> blouse undershirt t - shirt tie
> shirt blouse sweater -
> blouse sweater undershirt t - shirt
> blouse jacket jumper sweater t - shirt
> shirt blouse jacket sweater undershirt t - shirt
> undershirt
> ##shirt
> t - shirt
> t - shirt
> ```
> 
> Is this to be expected?

It should also end with `.` so it would be something like `". ".join(texts) + "."`. Fixed that on the example in the model readme as well!

And about your outputs, they are indeed a bit weird haha. It's unexpected to have outputs like `shirt blouse jacket sweater` if in your text input you separated the classes with `.`. Can you share your example?

> Huge piece of work - thanks for adding this model!

Thank you for reviewing and your patience as well 😅. I'll open an issue to add the `cross-attetion` as @jiangtann mentioned as soon as it gets merged