Support LLMDet in transformers
Author: fushh
Pull request: no
Labels: name: New model

### Model description

Could you please kindly consider add [LLMDet](https://arxiv.org/abs/2501.18954) (CVPR2025 Highlight) to transformers, which is a next-generation open-vocabulary object detector. The architecture of LLMDet is similar to GroundingDino and most of the code can be reused. We have provide the code and the Hugginface-compatible checkpoints at [here](https://github.com/iSEE-Laboratory/LLMDet/tree/main/hf_model).

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

paper: https://arxiv.org/abs/2501.18954
code: https://github.com/iSEE-Laboratory/LLMDet/tree/main/hf_model
model: https://huggingface.co/fushh7/llmdet_swin_tiny_hf
model: https://huggingface.co/fushh7/llmdet_swin_base_hf
model: https://huggingface.co/fushh7/llmdet_swin_large_hf


Comments:
Very cool! 

It seems that only 3 minor modifications are needed (as explained [here](https://github.com/iSEE-Laboratory/LLMDet/tree/main/hf_model)). Hence this is an ideal use case for modular. It means that you can just add `modular_llmdet.py` which inherits everything from Grounding DINO and adds the necessary changes. The modular converter will then automatically convert that to a standalone `modeling_llmdet.py` file.

 Is this something you'd be eager to open a PR for? To get started, see the guide here: https://huggingface.co/docs/transformers/main/en/modular_transformers

cc also @EduardoPach who added Grounding DINO to Transformers

Hey @fushh! Thanks for the proposal and congratulations on the acceptance of your paper for the CVPR 2025 Highlights! It would be wonderful to have it in Transformers. Please let us know if we can make the PR 🤗

Since I am not familiar with how to contribute to transformers, it would be grateful if you can help us integrate LLMDet into transformers. Many thanks!

@fushh, sure, we can help with the code review once the PR is open and guide the implementation details. Since the code is similar to GroundingDINO, it should not be that hard to add it by following the structure of the GroundingDINO model.

As @NielsRogge mentioned, here is the good starting point:
 - https://huggingface.co/docs/transformers/main/en/modular_transformers

You can refer to other model PR's, e.g. RT-DETRv2, which is based on RT-DETR and also uses a modular approach.

Hey @fushh ! Congrats on the CVPR 2025 Highlights! I saw the discussion about adding LLMDet to Transformers and was wondering if there's any chance I can help with that? I'd love to contribute and assist in any way I can. Let me know!

Amazing work, @fushh ! Glad to see that the HF `GroundingDino` implementation is useful to researchers. Feel free to tag me as a reviewer if you open a PR 

Hey @fushh, sorry for the very slow follow-up, we're catching up on the backlog, and congrats again on the CVPR 2025 Highlight!

Good news: LLMDet is now usable in Transformers since https://github.com/huggingface/transformers/issues/37744, released in v4.55.0. Because its inference architecture matches Grounding DINO, it's served through the MM Grounding DINO implementation (`MMGroundingDinoForObjectDetection`), and the converted LLMDet checkpoints live at `iSEE-Laboratory/llmdet_tiny` / `_base` / `_large`. See the model docs, which even include the LVIS numbers from your repo: https://github.com/huggingface/transformers/blob/9f909e0f7027285d83bb6addc88155f1b80244ab/docs/source/en/model_doc/mm-grounding-dino.md#L98-L104
 and there's a runnable zero-shot object detection example using `iSEE-Laboratory/llmdet_large` in the task guide.

Closing as resolved. Thanks for the proposal! 🤗