Package: gem-torch
GEMGrounding Everything: Emerging Localization Properties in Vision-Language TransformersWalid Bousselham,Felix Petersen,Vittorio Ferrari,Hilde KuehneVision-Language foundation models have shown remarkable performance in various zero-shot settings such as image retrieval, classification, or captioning. But so far, those models seem to fall behind when it comes to zero-shot localization of referential expressions and objects in images.GEM allows a training-free adaptation of Vision-Language models (e.i., CLIP ...) to perform zero-shot open-vocabulary segmentation. The training-free adaptation allows to fully conserve the vocabulary learned by the Vision-Language model during its pretraing, thus allowing the segmentation of uncommon classes (e.g.Elon Musk/Mark Zuckerberg /Jeff Besos).:hammer: Installationgemlibrary can be simply installed via pip:$pipinstallgem_torchDemoTry out our web demo onHuggingFace SpacesRun the demo on Google Colab:Runtest_examples.pyfor a usage example.To run the gradio app locally, first install gradio and then runapp.py:$pipinstallgradio
$pythonapp.pyUsageTo see which pretrained models is available use the following code snippet:importgemgem.available_models()Single ImageTo process a single image and multiple text prompts use the following code snippet:importtorchimportgemimportrequestsfromPILimportImagemodel_name='ViT-B/16'# 'ViT-B-16-quickgelu'pretrained='openai'# 'metaclip_400m'device="cuda"iftorch.cuda.is_available()else"cpu"# init model and image transformpreprocess=gem.get_gem_img_transform()gem_model=gem.create_gem_model(model_name=model_name,pretrained=pretrained,device=device)# load image and texturl="http://images.cocodataset.org/val2017/000000039769.jpg"image=preprocess(Image.open(requests.get(url,stream=True).raw)).unsqueeze(0).to(device)text=['cat','remote control']withtorch.no_grad():logits=gem_model(image,text)# [B, num_prompt, W, H]gem_model.visualize(image,text,logits)# (optional visualization)Batched InferenceTo process a batch of images withdifferentnumber of prompts per image, one must use thebatched_forward()function ofgem_model:importtorchimportgemimportrequestsfromPILimportImagemodel_name='ViT-B/16'# 'ViT-B-16-quickgelu'pretrained='openai'# 'metaclip_400m'device="cuda"iftorch.cuda.is_available()else"cpu"# init model and image transformgem_model=gem.create_gem_model(model_name=model_name,pretrained=pretrained,device=device)preprocess=gem.get_gem_img_transform()# load image and texturls=["http://images.cocodataset.org/val2017/000000039769.jpg","https://cdn.vietnambiz.vn/171464876016439296/2021/7/11/headshots16170695297430-1626006880779826347793.jpg","https://preview.redd.it/do-you-think-joker-should-be-unpredictable-enough-to-put-up-v0-6a2ax4ngtlaa1.jpg?auto=webp&s=f8762e6a1b40642bcae5900bac184fc597131503",]texts=[['remote control','cat'],['elon musk','mark zuckerberg','jeff bezos','bill gates'],['batman','joker','shoe','belt','purple suit'],]# note that the number of prompt per image can be different# download images + convert to PIL.Imageimages_pil=[Image.open(requests.get(url,stream=True).raw)forurlinurls]images=torch.stack([preprocess(img)forimginimages_pil]).to(device)withtorch.no_grad():# return list with logits of size [1, num_prompt, W, H]logits_list=gem_model.batched_forward(images,texts)fori,logitsinenumerate(logits_list):# (optional visualization)gem_model.visualize(images[i],texts[i],logits)APIThe library provides the following methods:gem.create_gem_model(model_name, pretrained, device, ...):Returnsmodel_nameVision Language model withpretrainedweights loaded and GEM applied. One can also specifygem_depth,ss_attn_iterandss_attn_tempparameters to respectively control GEM's depth, self-self attention number of iteration and temperature (see paper for more details).gem.get_gem_img_transform(img_size):takes in a PIL.Image and returns a torch.Tensor. This can be used as input to the model.gem.visualize(image, prompts, logits, alpha=0.6, save_path=None):Takes in a PIL.Imageora torch.Tensor, as well as the list of text prompt and the logits outputed by gem and plot the gem's heatmaps for each prompt. Alternatively, the heatmaps cam be saved by specifying the saving pathsave_path. One can also change the transparence of the heatmps via theaplha=0.6argument.By default, the models loaded bygem.create_gem_model()returns logits outputed by GEM, but can also return the logits of the original Vision Language model (it can be useful for visualization). To do so, setreturn_ori=True.More ExamplesSemantic SegmentationFor the semantic segmentation task, given a list of foreground class names, one must predict a 2D map where each location is the id of the predicted class. Depending on the dataset, we may also want to predict abackgroundclass. However, the textual description"a photo of a background"is not descriptive of what the background is composed of. Hence, we propose to use the following code method:importtorchimportgemimportrequestsfromPILimportImagemodel_name='ViT-B/16'# 'ViT-B-16-quickgelu'pretrained='openai'# 'metaclip_400m'device="cuda"iftorch.cuda.is_available()else"cpu"# init model and image transformgem_model=gem.create_gem_model(model_name=model_name,pretrained=pretrained,device=device)preprocess=gem.get_gem_img_transform()predict_background=True# whether the background is predictedifpredict_background:threshold=0.85# the threshold depends on the number of classes# load image and textimage=preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)class_names=['airplane','cat','dog','...']# foreground class nameswithtorch.no_grad():logits=gem_model(image,class_names)# [1, num_class, W, H]pred=logits.argmax(dim=1)ifpredict_background:pred=pred+1# we assume the background's index is 0probs=logits.softmax(dim=1)max_prob=probs.max(dim=1)[0]pred[probs<threshold]=0# if the max prob is lower than the threshold the background is predictedNote thatthresholddepends on the number of classes and should be determined via a hyperparameter sweep.Datasetgemcan also be used with regular pytorch dataset.importtorchimportgemfromPILimportImagefromtorchvision.datasetsimportVOCSegmentationmodel_name='ViT-B/16'# 'ViT-B-16-quickgelu'pretrained='openai'# 'metaclip_400m'device="cuda"iftorch.cuda.is_available()else"cpu"# init model and image transformgem_model=gem.create_gem_model(model_name=model_name,pretrained=pretrained,device=device)preprocess=gem.get_gem_img_transform()predict_background=True# whether the background is predictedifpredict_background:threshold=0.85# the threshold depends on the number of classes# load datasetroot='./data'# path to save the datasetdataset=VOCSegmentation(root=root,image_set='val',download=True,transform=preprocess)dataloader=torch.utils.data.DataLoader(dataset,batch_size=4)class_names=['airplane','cat','dog','...']# foreground class nameswithtorch.no_grad():for(image,_)indataloader:logits=gem_model(image,class_names)# [1, num_class, W, H]pred=logits.argmax(dim=1)ifpredict_background:pred=pred+1# we assume the background's index is 0probs=logits.softmax(dim=1)max_prob=probs.max(dim=1)[0]pred[probs<threshold]=0# if the max prob is lower than the threshold the background is predicted:star: AcknowledgementThis code is build as wrapper aroundOpenCLIPlibrary fromLAION, visit their repo for more vision-language models.
This project takes inspiration fromCLIPandCLIPSurgery, please visit their repository.
This repo also useseinopsas well and take inspiration fromCLIPandCLIPSurgeryrepository.:books: CitationIf you find this repository useful, please consider citing our work :pencil: and giving a star :star2: :@article{bousselham2023gem,
  title={Grounding Everything: Emerging Localization Properties in Vision-Language Transformers},
  author={Walid Bousselham, Felix Petersen, Vittorio Ferrari, Hilde Kuehne},
  journal={arXiv preprint arXiv:2312.00878},
  year={2023}
}