[WIP] Add MM Grounding DINO
Author: rziga
Pull request: yes
Labels: name: New model

name: Vision

# What does this PR do?

Fixes #37744.

It adds support for MM Grounding DINO and LLMDet (inference only).

* I've added a weight conversion script and a modular transformers implementation.
* The forward pass works with 1e-3 tolerance compared to the original (checked tiny and large checkpoints).
* The tests are copied and modified from Grounding DINO, but two integration tests fail currently (batched consistency and CPU-GPU consistency).

# TODO
[ ] fix failing tests

Tagging @qubvel


Comments:
Hi @rziga ! Feel free to ping me when the PR is ready for review!

Sorry, I got busy with life stuff for the last couple of weeks, so I didn't have time to finish it up. 

The model works, but I have to upload all the checkpoints to the hub and update the model doc before I mark it ready for review.

Pinging @yonigozlan and @qubvel as requested.

Hey @rziga, I wanted to ask you that - how do you add a new model? Huggingface provides utilities to help you get started with the boilerplate, AFAIK this is where you get started -  `transformers add-new-model-like` this CLI command will asks around 7-8 questions related to the model. The first one goes like this "What is the model you would like to duplicate? Please provide the lowercase model_type (e.g. roberta):"
this model will be used as a scaffolding to build the new model, we just have to change the main files which includes - architecture of the model, any pre/post processing utilities and a few more thing. We don't just dump in the research code(the original implementation of a model ) here. We first need to do a sort of code  archaeology, remove the research specific code, and only add the code useful for inference. 

Correct me if I'm wrong. And could you tell me your approach. 😄 

Hi,

I don't really know if I'm the right person to ask, but yes, I started with `transformers add-new-model-like`. From then on I don't remember exactly, but it was a combination of following these two guides:

* https://huggingface.co/docs/transformers/modular_transformers
* https://huggingface.co/docs/transformers/en/add_new_model

Since this modular approach is quite new, I also looked at other models that used modular transformers, like this one:

* https://github.com/huggingface/transformers/pull/34773

Hope this helps somewhat. But again I don't think I'm the right person to ask here. It is probably better to open an issue to add a model and ask the actual contributors.

Hey @rziga, I'm really sorry for the long delay. I'll review it next week. The modular code looks very minimal, so I expect we can merge it quickly! Thanks a lot for working on the model.

When is m_grounding_dino expected to be officially integrated into transformers and released? @qubvel  qubvel

Hi @Lelege0, I can't tell you the exact date, it depends on @rziga's bandwidth. It's at the top of my priority list at the moment, so I would be happy to merge it as soon as it's ready

> Hi @Lelege0, I can't tell you the exact date, it depends on @rziga's bandwidth. It's at the top of my priority list at the moment, so I would be happy to merge it as soon as it's ready

Thank you very much for your contribution!

Thanks for the review @qubvel.

I changed the conversion script (removed `mmengine` requirement). I went with a similar style as in siglip2.

As for the model cards. Do I edit them manually on the HF hub or is there some other preferred way of editing them in a batch?

@rziga you can make the first one manually, and as soon as it's ready, you can update the rest with a simple script using `huggingface_hub`. Here is an example of how to upload just the README.md file

```
from huggingface_hub import HfApi
api = HfApi()

api.upload_file(
    path_or_fileobj="/path/to/local/folder/README.md",
    path_in_repo="README.md",
    repo_id="rziga/mm_grounding_dino_...",
    repo_type="model",
)
```

The model card should look similar to the [original](http://huggingface.co/IDEA-Research/grounding-dino-tiny) grounding dino combined with existing model docs, also make sure to add proper tags (`zero-shot-object-detection`) and Licence.

As I said, we can iterate together on the first model card, and once it's ready, you can update the rest 🤗  Thanks!

Here's a draft for a model card: https://huggingface.co/rziga/mm_grounding_dino_tiny_o365v1_goldg

I copied the style from Grounding DINO and SigLIPv2.

cc @ArthurZucker for final review

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37925). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@qubvel , I have now duplicated the model cards for all models. 

MM Grounding DINO models are in this collection: https://huggingface.co/collections/rziga/mm-grounding-dino-6839881a7f983113fafdbb0e

LLMDet models are in this collection: https://huggingface.co/collections/rziga/llmdet-68398b294d9866c16046dcdd

The models in LLMDet collection have a bit different model card than those in MM Grounding DINO (different intro, citation and results). LMK if I missed something.

I have no problem moving the checkpoints. I have no affiliation with any of the original model authors.

Also the LLMDet checkpoints, should probably be in a separate org. I don't know if there already exist an iSEE org, I can't find it. Perhaps @fushh (#37334) can tell where LLMDet checkpoints should be.

@rziga Thanks for your valuable contributions🙏🙏🙏. Have you modified my models?
model: https://huggingface.co/fushh7/llmdet_swin_tiny_hf
model: https://huggingface.co/fushh7/llmdet_swin_base_hf
model: https://huggingface.co/fushh7/llmdet_swin_large_hf
If you modified them, please move your version to this collection. I will give you the write permissions. Otherwise, I will use my original models.
https://huggingface.co/collections/iSEE-Laboratory/llmdet-688475906dc235d5f1dc678e

Hi @fushh . I have modified them. My conversion script copies parameters from mm-detection checkpoint directly and there are some changes in the contrastive bias parameters. 

In the original mmdet checkpoint (`https://huggingface.co/fushh7/LLMDet/resolve/main/tiny.pth`) each layer in class head has a different bias:
```py
import torch

original = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/fushh7/LLMDet/resolve/main/tiny.pth?download=true",
    map_location="cpu",
)
for k in original["state_dict"]:
    if "bbox_head.cls_branches" in k:
        print(k, original["state_dict"][k])

>>>bbox_head.cls_branches.0.bias tensor([-4.5637])
>>>bbox_head.cls_branches.1.bias tensor([-4.5809])
>>>bbox_head.cls_branches.2.bias tensor([-4.5068])
>>>bbox_head.cls_branches.3.bias tensor([-4.4998])
>>>bbox_head.cls_branches.4.bias tensor([-4.4989])
>>>bbox_head.cls_branches.5.bias tensor([-4.4106])
>>>bbox_head.cls_branches.6.bias tensor([-5.1244])
```

Your hf converted checkpoints seem to only include one decoder bias:
```py
import safetensors.torch as st

converted = st.load_file("/home/user/.cache/huggingface/hub/models--fushh7--llmdet_swin_tiny_hf/snapshots/6719e6ec2b2f7f0f3fed035fdca7b1856a1107c9/model.safetensors", device="cpu")
for k in converted:
    if "class_embed" in k:
        print(k, converted[k])

>>>class_embed.0.bias tensor([-4.4106])
>>>model.encoder_output_class_embed.bias tensor([-5.1244])
```

Looking at your huggingface inference code, you don't seem to do a deep copy of your class embedding layer. In original GroundingDino this makes no difference, but here, each layer has its own bias parameter, so I think the bug comes from [here](https://github.com/iSEE-Laboratory/LLMDet/blob/67f0a6311d65077bd6fe843cce1cb327d5c9e8b8/hf_model/modeling_grounding_dino.py#L3003C9-L3003C95). 

Besides that, the config also requires changes to select the correct autoclass, so I think it's best to just move my checkpoints to you.

@rziga Thanks for pointing out the bugs. Now, you have the write permissions of the collection. Could you please move them into the collection? 
Once again, many thanks to your contributions.

@fushh I moved the models to your collection here: https://huggingface.co/collections/iSEE-Laboratory/llmdet-688475906dc235d5f1dc678e. 

You can remove me from the org now.

Hi @rziga, I created org for MM Grounding DINO checkpoints and invited you
 - https://huggingface.co/openmmlab-community

run-slow: mm_grounding_dino

This comment contains run-slow, running the specified jobs: 

models: ['models/mm_grounding_dino']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, grounding_dino, mm_grounding_dino

run-slow: grounding_dino, mm_grounding_dino

This comment contains run-slow, running the specified jobs: 

models: ['models/grounding_dino', 'models/mm_grounding_dino']
quantizations: [] ...

Huge thanks for the contribution @rziga! Great work 🤗 

@qubvel here support for llmdet is only added for the inference ,we have any support for training also ??can I pls take up the issue if its planned or open for community