Package: groundingdino-py
:sauropod: Grounding DINOOfficial PyTorch implementation of"Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection": the SoTA open-set object detector.:sun_with_face: Helpful Tutorial:grapes: [Read our arXiv Paper]:apple: [Watch our simple introduction video on YouTube]:rose:  [Try the Colab Demo]:sunflower: [Try our Official Huggingface Demo]:maple_leaf: [Watch the Step by Step Tutorial about GroundingDINO by Robotflow AI]:mushroom: [GroundingDINO: Automated Dataset Annotation and Evaluation by Robotflow AI]:hibiscus: [Accelerate Image Annotation with SAM and GroundingDINO by Robotflow AI]:sparkles: Highlight ProjectsDetGPT: Detect What You Need via ReasoningGrounded-SAM: Marrying Grounding DINO with Segment AnythingGrounding DINO with Stable DiffusionGrounding DINO with GLIGEN for Controllable Image EditingOpenSeeD: A Simple and Strong Openset Segmentation ModelSEEM: Segment Everything Everywhere All at OnceX-GPT: Conversational Visual Agent supported by X-DecoderGLIGEN: Open-Set Grounded Text-to-Image GenerationLLaVA: Large Language and Vision Assistant:bulb: HighlightOpen-Set Detection.Detecteverythingwith language!High Performancce.COCO zero-shot52.5 AP(training without COCO data!). COCO fine-tune63.0 AP.Flexible.Collaboration with Stable Diffusion for Image Editting.:fire: News2023/04/15: Refer toCV in the Wild Readingsfor those who are interested in open-set recognition!2023/04/08: We releasedemosto combineGrounding DINOwithGLIGENfor more controllable image editings.2023/04/08: We releasedemosto combineGrounding DINOwithStable Diffusionfor image editings.2023/04/06: We build a new demo by marrying GroundingDINO withSegment-AnythingnamedGrounded-Segment-Anythingaims to support segmentation in GroundingDINO.2023/03/28: A YouTubevideoabout Grounding DINO and basic object detection prompt engineering. [SkalskiP]2023/03/28: Add ademoon Hugging Face Space!2023/03/27: Support CPU-only mode. Now the model can run on machines without GPUs.2023/03/25: Ademofor Grounding DINO is available at Colab. [SkalskiP]2023/03/22: Code is available Now!DescriptionPaperintroduction.MarryingGrounding DINOandGLIGEN:star: Explanations/Tips for Grounding DINO Inputs and OutputsGrounding DINO accepts an(image, text)pair as inputs.It outputs900(by default) object boxes. Each box has similarity scores across all input words. (as shown in Figures below.)We defaultly choose the boxes whose highest similarities are higher than abox_threshold.We extract the words whose similarities are higher than thetext_thresholdas predicted labels.If you want to obtain objects of specific phrases, like thedogsin the sentencetwo dogs with a stick., you can select the boxes with highest text similarities withdogsas final outputs.Note that each word can be split tomore than onetokens with different tokenlizers. The number of words in a sentence may not equal to the number of text tokens.We suggest separating different category names with.for Grounding DINO.:label: TODORelease inference code and demo.Release checkpoints.Grounding DINO with Stable Diffusion and GLIGEN demos.Release training codes.:hammer_and_wrench: InstallNote:If you have a CUDA environment, please make sure the environment variableCUDA_HOMEis set. It will be compiled under CPU-only mode if no CUDA available.Installation:Clone the GroundingDINO repository from GitHub.gitclonehttps://github.com/IDEA-Research/GroundingDINO.gitChange the current directory to the GroundingDINO folder.cdGroundingDINO/Install the required dependencies in the current directory.pip3install-q-e.Create a new directory called "weights" to store the model weights.mkdirweightsChange the current directory to the "weights" folder.cdweightsDownload the model weights file.wget-qhttps://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth:arrow_forward: DemoCheck your GPU ID (only if you're using a GPU)nvidia-smiReplace{GPU ID},image_you_want_to_detect.jpg, and"dir you want to save the output"with appropriate values in the following commandCUDA_VISIBLE_DEVICES={GPUID}pythondemo/inference_on_a_image.py\-c/GroundingDINO/groundingdino/config/GroundingDINO_SwinT_OGC.py\-p/GroundingDINO/weights/groundingdino_swint_ogc.pth\-iimage_you_want_to_detect.jpg\-o"dir you want to save the output"\-t"chair"[--cpu-only]# open it for cpu modeSee thedemo/inference_on_a_image.pyfor more details.Running with Python:fromgroundingdino.util.inferenceimportload_model,load_image,predict,annotateimportcv2model=load_model("groundingdino/config/GroundingDINO_SwinT_OGC.py","weights/groundingdino_swint_ogc.pth")IMAGE_PATH="weights/dog-3.jpeg"TEXT_PROMPT="chair . person . dog ."BOX_TRESHOLD=0.35TEXT_TRESHOLD=0.25image_source,image=load_image(IMAGE_PATH)boxes,logits,phrases=predict(model=model,image=image,caption=TEXT_PROMPT,box_threshold=BOX_TRESHOLD,text_threshold=TEXT_TRESHOLD)annotated_frame=annotate(image_source=image_source,boxes=boxes,logits=logits,phrases=phrases)cv2.imwrite("annotated_image.jpg",annotated_frame)Web UIWe also provide a demo code to integrate Grounding DINO with Gradio Web UI. See the filedemo/gradio_app.pyfor more details.NotebooksWe releasedemosto combineGrounding DINOwithGLIGENfor more controllable image editings.We releasedemosto combineGrounding DINOwithStable Diffusionfor image editings.:luggage: CheckpointsnamebackboneDatabox AP on COCOCheckpointConfig1GroundingDINO-TSwin-TO365,GoldG,Cap4M48.4 (zero-shot) / 57.2 (fine-tune)GitHub link|HF linklink2GroundingDINO-BSwin-BCOCO,O365,GoldG,Cap4M,OpenImage,ODinW-35,RefCOCO56.7GitHub link|HF linklink:medal_military: ResultsCOCO Object Detection ResultsODinW Object Detection ResultsMarrying Grounding DINO withStable Diffusionfor Image EditingSee our examplenotebookfor more details.Marrying Grounding DINO withGLIGENfor more Detailed Image Editing.See our examplenotebookfor more details.:sauropod: Model: Grounding DINOIncludes: a text backbone, an image backbone, a feature enhancer, a language-guided query selection, and a cross-modality decoder.:hearts: AcknowledgementOur model is related toDINOandGLIP. Thanks for their great work!We also thank great previous work including DETR, Deformable DETR, SMCA, Conditional DETR, Anchor DETR, Dynamic DETR, DAB-DETR, DN-DETR, etc. More related work are available atAwesome Detection Transformer. A new toolboxdetrexis available as well.ThanksStable DiffusionandGLIGENfor their awesome models.:black_nib: CitationIf you find our work helpful for your research, please consider citing the following BibTeX entry.@article{liu2023grounding,title={Grounding dino: Marrying dino with grounded pre-training for open-set object detection},author={Liu, Shilong and Zeng, Zhaoyang and Ren, Tianhe and Li, Feng and Zhang, Hao and Yang, Jie and Li, Chunyuan and Yang, Jianwei and Su, Hang and Zhu, Jun and others},journal={arXiv preprint arXiv:2303.05499},year={2023}}