Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

1 INTRODUCTION . Consider Fig . 1 , can we design object detectors beyond recognizing only base categories ( e.g. , toy ) present in training labels and expand the vocabulary to detect novel categories ( e.g. , toy elephant ) ? In this paper , we aim to train an open-vocabulary object detector using only detection annotations in base categories , and detect objects in any novel categories described by text inputs . Existing object detection algorithms often learn to detect only the categories present in detection datasets . A common approach to increase the detection vocabulary is by collecting images with more labeled categories . The research community has recently collected new object detection datasets with large vocabularies ( Gupta et al. , 2019 ; Kuznetsova et al. , 2020 ) . LVIS ( Gupta et al. , 2019 ) is a milestone of these efforts by building a dataset with 1,203 categories . With such a rich vocabulary , it becomes quite challenging to collect enough training examples for all categories . By Zipf ’ s law , object categories naturally follow a long-tailed distribution . To find sufficient training examples for rare categories , significantly more data is needed ( Gupta et al. , 2019 ) , which makes it expensive to scale up detection vocabularies . On the other hand , paired image-text data are abundant on the Internet . Recently , Radford et al . ( 2021 ) train a joint vision and language model using 400 million image-text pairs and demonstrate impressive results on directly transferring to over 30 classification datasets . The pretrained text encoder is the key to the zero-shot transfer ability to arbitrary text categories . Despite the great success on learning image-level representations , learning object-level representations for open-vocabulary detection is still challenging . In this work , we consider borrowing the knowledge from a pretrained open-vocabulary classification model to enable open-vocabulary detection . We begin with an R-CNN ( Girshick et al. , 2014 ) style approach . We turn open-vocabulary detection into two sub-problems : 1 ) generalized object proposal and 2 ) open-vocabulary image classification . We train a region proposal model using examples in the base categories . Then we use the pretrained open-vocabulary image classification model to classify cropped object proposals , which can contain both base and novel categories . We benchmark on LVIS ( Gupta et al. , 2019 ) by holding out all rare categories as novel categories and treat others as base categories . To our surprise , the performance on the novel categories already surpasses its supervised counterpart . However , this approach is very slow for inference , because it feeds object proposals one-by-one into the classification model . To address the above issue , we propose ViLD ( Vision and Language knowledge Distillation ) for training two-stage open-vocabulary detectors . ViLD consists of two components : learning with text embeddings ( ViLD-text ) and image embeddings ( ViLD-image ) inferred by an open-vocabulary image classification model , e.g. , CLIP ( Radford et al. , 2021 ) . In ViLD-text , we obtain the text embeddings by feeding category names into the pretrained text encoder . Then the inferred text embeddings are used to classify detected regions . Similar approaches have been used in prior zero-shot and open-vocabulary detection works ( Bansal et al. , 2018 ; Rahman et al. , 2018 ; Zareian et al. , 2021 ) . We find text embeddings learned jointly with visual data can better encode the visual similarity between concepts , compared to text embeddings learned from a language corpus , e.g. , GloVe ( Pennington et al. , 2014 ) . Using CLIP text embeddings achieves 10.1 APr ( AP of novel categories ) on LVIS , significantly outperforming the 3.0 APr of using GloVe . In ViLD-image , we obtain the image embeddings by feeding the object proposals into the pretrained image encoder . Then we train an object detector whose region embeddings of detected boxes are aligned with these image embeddings . In contrast to ViLD-text , ViLD-image distills knowledge from both base and novel categories , while ViLD-text only learns from base category concepts . We show that ViLD , which jointly learns from text and image embeddings , achieves 16.1 APr on LVIS , surpassing the supervised counterpart by 3.8 . To push the performance limit , we use ALIGN ( Jia et al. , 2021 ) as a stronger teacher model and attain the best performance of 26.3 APr , which is only 3.7 APr worse than the fully-supervised 2020 LVIS Challenge winner ( Tan et al. , 2020 ) that uses additional tricks . We also attain state-of-the-art results on COCO ( Lin et al. , 2014 ) . Our method outperforms the previous SOTA ( Zareian et al. , 2021 ) by 4.8 AP on novel categories and 11.4 AP on all categories . In addition , ViLD trained on LVIS can directly transfer to other detection datasets including Objects365 , COCO , PASCAL VOC . 2 RELATED WORK . Increasing vocabulary in visual recognition : Designing a computer vision model that can recognize objects using a large vocabulary is a long-standing research problem . One focus is zero-shot recognition , aiming at recognizing categories not present in the training set . Early works ( Farhadi et al. , 2009 ; Rohrbach et al. , 2011 ; Jayaraman & Grauman , 2014 ) use visual attributes to create a binary codebook representing categories , which is used to transfer learned knowledge to unseen categories . In this direction , researchers have also explored class hierarchy , class similarity , and object parts as discriminative features to aid the knowledge transfer ( Rohrbach et al. , 2011 ; Akata et al. , 2016 ; Zhao et al. , 2017 ; Elhoseiny et al. , 2017 ; Ji et al. , 2018 ; Cacheux et al. , 2019 ; Xie et al. , 2020 ) . Another focus is learning to align latent image-text embeddings , which allows to classify images using arbitrary texts . Frome et al . ( 2013 ) and Norouzi et al . ( 2014 ) are pioneering works that learn a visual-semantic embedding space using deep learning . Wang et al . ( 2018 ) distills information from both word embeddings and knowledge graphs . In recent works , CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) push the limit by collecting million/billion-scale image-text pairs and then train joint image-text embedding models using contrastive learning . Equipped with vast image-text knowledge from the Internet , these models can directly transfer to a suite of classification datasets and achieve impressive performances . While these work focus on image-level open-vocabulary recognition , we focus on detecting objects using arbitrary text inputs . Increasing vocabulary in object detection : It ’ s expensive to scale up the data collection for large vocabulary object detection . Zhao et al . ( 2020 ) and Zhou et al . ( 2021 ) unify the label space from multiple datasets . Joseph et al . ( 2021 ) incrementally learn identified unknown categories . Zero-shot detection ( ZSD ) offers another direction . Most ZSD methods align region features to pretrained text embeddings in base categories ( Bansal et al. , 2018 ; Demirel et al. , 2018 ; Rahman et al. , 2019 ; Hayat et al. , 2020 ; Zheng et al. , 2020 ) . However , there is a large performance gap to supervised counterparts . To address this issue , Zareian et al . ( 2021 ) pretrain the backbone model using image captions and finetune the pretrained model with detection datasets . In contrast , we use an image-text pretrained model as a teacher model to supervise student object detectors . All previous methods are only evaluated on tens of categories , while we are the first to evaluate on more than 1,000 categories . 3 METHOD . In this section , we introduce three main components in our approach : localization of novel objects , learning with pretrained text embeddings ( ViLD-text ) , and learning with pretrained image embeddings ( ViLD-image ) . Fig . 2 illustrates an overview of our method . We also study model ensembling approaches for the best detection performance . Notations : We divide annotations in an object detection dataset into two subsets based on their categories . We denote base categories and novel categories by CB and CN . Only annotations in CB are used for training . We use T ( · ) to denote the text encoder and V ( · ) to denote the image encoder in the pretrained open-vocabulary image classification model . 3.1 LOCALIZATION FOR NOVEL CATEGORIES . The first challenge for open-vocabulary detection is to localize novel objects . We modify a standard two-stage object detector , e.g. , Mask R-CNN ( He et al. , 2017 ) , for this purpose . We replace its classspecific localization modules , i.e. , the second-stage bounding box regression and mask prediction layers , with class-agnostic modules for general object proposals . For each region of interest , these modules only predict a single bounding box and a single mask for all categories , instead of one prediction per category . The class-agnostic modules can generalize to novel objects . 3.2 OPEN-VOCABULARY DETECTION WITH CROPPED REGIONS . Once object candidates are localized , we propose to reuse a pretrained open-vocabulary image classifier to classify each region for detection . Image embeddings : We train a proposal network on base categories CB and extract the region proposals r̃ ∈ P̃ offline . We crop and resize proposals , and feed them into the pretrained image encoder V to compute image embeddings V ( crop ( I , r̃ ) ) , where I is the image . We ensemble the image embeddings from 1× and 1.5× crops , as the 1.5× crop provides more context cues . The ensembled embedding is then renormalized to unit norm : V ( crop ( I , r̃ { 1×,1.5× } ) ) = v ‖v‖ , where v = V ( crop ( I , r̃1× ) ) + V ( crop ( I , r̃1.5× ) ) . ( 1 ) Text embeddings : We generate the text embeddings offline by feeding the category texts with prompt templates , e.g. , “ a photo of { category } in the scene ” , into the text encoder T . We ensemble multiple prompt templates and the synonyms if provided . Then , we compute cosine similarities between the image and text embeddings . A softmax activation is applied , followed by a per-class NMS to obtain final detections . The inference is slow since every cropped region is fed into V . 3.3 VILD : VISION AND LANGUAGE KNOWLEDGE DISTILLATION .. We propose ViLD to address the slow inference speed of the above method . ViLD learns region embeddings in a two-stage detector to represent each proposal r. We denote region embeddings by R ( φ ( I ) , r ) , where φ ( · ) is a backbone model and R ( · ) is a lightweight head that generates region embeddings . Specifically , we take outputs before the classification layer as region embeddings . Replacing classifier with text embeddings : We first introduce ViLD-text . Our goal is to train the region embeddings such that they can be classified by text embeddings . Fig . 3 ( b ) shows the architecture and training objective . ViLD-text replaces the learnable classifier in Fig . 3 ( a ) with the text embeddings introduced in Sec . 3.2 . Only T ( CB ) , the text embeddings of CB , are used for training . For the proposals that do not match to any groundtruth in CB , they are assigned to the background category . Since the text “ background ” does not well represent these unmatched proposals , we allow the background category to learn its own embedding ebg . We compute the cosine similarity between each region embedding R ( φ ( I ) , r ) and all category embeddings , including T ( CB ) and ebg . Then we apply softmax activation with a temperature τ to compute the cross entropy loss . To train the first-stage region proposal network of the two-stage detector , we extract its region proposals r ∈ P online , and train the detector with ViLD-text from scratch . The loss for ViLD-text can be written as : er = R ( φ ( I ) , r ) z ( r ) = [ sim ( er , ebg ) , sim ( er , t1 ) , · · · , sim ( er , t|CB | ) ] LViLD-text = 1 N ∑ r∈P LCE ( softmax ( z ( r ) /τ ) , yr ) , ( 2 ) where sim ( a , b ) = a > b/ ( ‖a‖‖b‖ ) , ti denotes elements in T ( CB ) , yr denotes the class label of region r , N is the number of proposals |P | per image , and LCE is the cross entropy loss . During inference , we include novel categories ( CN ) and generate T ( CB ∪CN ) ( sometimes T ( CN ) only ) for open-vocabulary detection ( Fig . 2 ) . Our hope is that the model learned from annotations in CB can generalize to novel categories CN . Distilling image embeddings : We then introduce ViLD-image , which aims to distill the knowledge from the teacher image encoder V into the student detector . Specifically , we align region embeddingsR ( φ ( I ) , r̃ ) to image embeddings V ( crop ( I , r̃ ) ) introduced in Sec . 3.2 . To make the training more efficient , we extract M proposals r̃ ∈ P̃ offline for each training image , and precompute the M image embeddings . These proposals can contain objects in both CB and CN , as the network can generalize . In contrast , ViLD-text can only learn from CB . We apply an L1 loss between the region and image embeddings to minimize their distance . The ensembled image embeddings in Sec . 3.2 are used for distillation : LViLD-image = 1 M ∑ r̃∈P̃ ‖V ( crop ( I , r̃ { 1×,1.5× } ) ) −R ( φ ( I ) , r̃ ) ‖1 . ( 3 ) Fig . 3 ( c ) shows the architecture . Zhu et al . ( 2019 ) use a similar approach to make Faster R-CNN features mimic R-CNN features , however , the details and goals are different : They reduce redundant context to improve supervised detection ; while ViLD-image is to enable open-vocabulary detection on novel categories . The total training loss of ViLD is simply a weighted sum of both objectives : LViLD = LViLD-text + w · LViLD-image , ( 4 ) where w is a hyperparameter weight for distilling the image embeddings . Fig . 3 ( d ) shows the model architecture and training objectives . ViLD-image distillation only happens in training time . During inference , ViLD-image , ViLD-text and ViLD employ the same set of text embeddings as the detection classifier , and use the same architecture for open-vocabulary detection ( Fig . 2 ) .

Summary: This paper addresses the task of open-vocabulary detection, where an object detector is trained from two forms of annotations: (a) bounding box annotations for a set of base categories and (b) open-vocabulary image-caption pairs.  This detector is then evaluated on a set of novel categories.  Compared to prior work, this paper tries to distill knowledge from strong pre-trained vision & language models (like CLIP) into an open-vocabulary detector. The authors propose multiple ways to do this (RCNN+CLIP, ViLD-text, ViLD-image, ViLD), including some ensemble variants (ViLD-ensemble). The paper also introduces a novel experimental setup based on the LVIS dataset (with >1200 categories) as compared to prior works that used COCO with only 80 categories.