Microsoft's GLIP Grounding Language Image Pretraining
Author: ethansmith2000
Pull request: no
Labels: name: New model

### Model description

Combines best practices of CLIP and object detectors.
Allows for localization and grounding of text and image content.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

github https://github.com/microsoft/GLIP
weights on hf https://huggingface.co/GLIPModel/GLIP/tree/main
colab demo https://colab.research.google.com/drive/12x7v-_miN7-SRiziK3Cx4ffJzstBJNqb?usp=sharing

Comments:
@ArthurZucker or @ethansmith2000, is anyone working on this?

cc @amyeroberts and @rafaelpadilla WDYT

Recently we have added Kosmos2 that also perform object detection with visual grounding.

IMO, having more models tackling the same problem is super valuable for the community. As GLIPv2 paper has ~110 citations (google scholars) and the repo +1.6k starts, I think it is relevant :) 

Added: If we decide to have this model, there're 2 versions: GLIP and GLIPv2. I think the referenced repo implements to GLIPv2, right? So, if that's the case, maybe we should call it GLIPv2 (?)

> Recently we have added Kosmos2 that also perform object detection with visual grounding.
> 
> IMO, having more models tackling the same problem is super valuable for the community. As GLIPv2 paper has ~110 citations (google scholars) and the repo +1.6k starts, I think it is relevant :)
> 
> Added: If we decide to have this model, there're 2 versions: GLIP and GLIPv2. I think the referenced repo implements to GLIPv2, right? So, if that's the case, maybe we should call it GLIPv2 (?)

Yeap, the referenced repo implements `GLIPv2` and I agree on calling it `GLIPv2`. I can start working on this one since the other model I was working (`GroundingDINO`) will likely require little modification to be merged. 

I have a few other models in mind that I want to add to transformers most open-set object detection or segmentation ones. Maybe instead of only having an object detection benchmark we could create an open-set benchmark as well as more and more of these models are added, what do you think?

@EduardoPach Great! You're very welcome to tackle adding GLIP (v2). I can see GroundingDINO is still waiting approval but happy if you don't mind handling to two PRs. 

If you have other models in mind please do open new-model issues for them! This helps us keep track of what's being added and what's popular in the community. 

We can certainly expand to including other benchmarks once we have several models which tackle the same task :) 

Hey @EduardoPach,
I think it a good idea to benchmark all vision models, so the community can easily compare them and pick the best model for their needs. 
We have a [leaderboard](https://huggingface.co/spaces/hf-vision/object_detection_leaderboard) to evaluate object detection models. We would like to extend this for other tasks, like segmentation, image classification, etc.


> Hey @EduardoPach,
> 
> I think it a good idea to benchmark all vision models, so the community can easily compare them and pick the best model for their needs. 
> 
> We have a [leaderboard](https://huggingface.co/spaces/hf-vision/object_detection_leaderboard) to evaluate object detection models. We would like to extend this for other tasks, like segmentation, image classification, etc.
> 
> 

I knew the object detection. Not sure though if it would be the same thing for open-set object detection. How can I help out with the other benchmarks? Not sure as well if here is the best place to discuss this haha, maybe discord?

@EduardoPach What I'd suggest is opening an issue so that people can easily find and track any discussions there. Ideally one for each task. If there's already a collection e.g. >= 5 checkpoints that you think can be evaluated on a certain task, then I think we can start discussing what the leaderboard would look like