Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

1 INTRODUCTION . Consider Fig . 1 , can we design object detectors beyond recognizing only base categories ( e.g. , toy ) present in training labels and expand the vocabulary to detect novel categories ( e.g. , toy elephant ) ? In this paper , we aim to train an open-vocabulary object detector using only detection annotations in base categories , and detect objects in any novel categories described by text inputs . Existing object detection algorithms often learn to detect only the categories present in detection datasets . A common approach to increase the detection vocabulary is by collecting images with more labeled categories . The research community has recently collected new object detection datasets with large vocabularies ( Gupta et al. , 2019 ; Kuznetsova et al. , 2020 ) . LVIS ( Gupta et al. , 2019 ) is a milestone of these efforts by building a dataset with 1,203 categories . With such a rich vocabulary , it becomes quite challenging to collect enough training examples for all categories . By Zipf ’ s law , object categories naturally follow a long-tailed distribution . To find sufficient training examples for rare categories , significantly more data is needed ( Gupta et al. , 2019 ) , which makes it expensive to scale up detection vocabularies . On the other hand , paired image-text data are abundant on the Internet . Recently , Radford et al . ( 2021 ) train a joint vision and language model using 400 million image-text pairs and demonstrate impressive results on directly transferring to over 30 classification datasets . The pretrained text encoder is the key to the zero-shot transfer ability to arbitrary text categories . Despite the great success on learning image-level representations , learning object-level representations for open-vocabulary detection is still challenging . In this work , we consider borrowing the knowledge from a pretrained open-vocabulary classification model to enable open-vocabulary detection . We begin with an R-CNN ( Girshick et al. , 2014 ) style approach . We turn open-vocabulary detection into two sub-problems : 1 ) generalized object proposal and 2 ) open-vocabulary image classification . We train a region proposal model using examples in the base categories . Then we use the pretrained open-vocabulary image classification model to classify cropped object proposals , which can contain both base and novel categories . We benchmark on LVIS ( Gupta et al. , 2019 ) by holding out all rare categories as novel categories and treat others as base categories . To our surprise , the performance on the novel categories already surpasses its supervised counterpart . However , this approach is very slow for inference , because it feeds object proposals one-by-one into the classification model . To address the above issue , we propose ViLD ( Vision and Language knowledge Distillation ) for training two-stage open-vocabulary detectors . ViLD consists of two components : learning with text embeddings ( ViLD-text ) and image embeddings ( ViLD-image ) inferred by an open-vocabulary image classification model , e.g. , CLIP ( Radford et al. , 2021 ) . In ViLD-text , we obtain the text embeddings by feeding category names into the pretrained text encoder . Then the inferred text embeddings are used to classify detected regions . Similar approaches have been used in prior zero-shot and open-vocabulary detection works ( Bansal et al. , 2018 ; Rahman et al. , 2018 ; Zareian et al. , 2021 ) . We find text embeddings learned jointly with visual data can better encode the visual similarity between concepts , compared to text embeddings learned from a language corpus , e.g. , GloVe ( Pennington et al. , 2014 ) . Using CLIP text embeddings achieves 10.1 APr ( AP of novel categories ) on LVIS , significantly outperforming the 3.0 APr of using GloVe . In ViLD-image , we obtain the image embeddings by feeding the object proposals into the pretrained image encoder . Then we train an object detector whose region embeddings of detected boxes are aligned with these image embeddings . In contrast to ViLD-text , ViLD-image distills knowledge from both base and novel categories , while ViLD-text only learns from base category concepts . We show that ViLD , which jointly learns from text and image embeddings , achieves 16.1 APr on LVIS , surpassing the supervised counterpart by 3.8 . To push the performance limit , we use ALIGN ( Jia et al. , 2021 ) as a stronger teacher model and attain the best performance of 26.3 APr , which is only 3.7 APr worse than the fully-supervised 2020 LVIS Challenge winner ( Tan et al. , 2020 ) that uses additional tricks . We also attain state-of-the-art results on COCO ( Lin et al. , 2014 ) . Our method outperforms the previous SOTA ( Zareian et al. , 2021 ) by 4.8 AP on novel categories and 11.4 AP on all categories . In addition , ViLD trained on LVIS can directly transfer to other detection datasets including Objects365 , COCO , PASCAL VOC . 2 RELATED WORK . Increasing vocabulary in visual recognition : Designing a computer vision model that can recognize objects using a large vocabulary is a long-standing research problem . One focus is zero-shot recognition , aiming at recognizing categories not present in the training set . Early works ( Farhadi et al. , 2009 ; Rohrbach et al. , 2011 ; Jayaraman & Grauman , 2014 ) use visual attributes to create a binary codebook representing categories , which is used to transfer learned knowledge to unseen categories . In this direction , researchers have also explored class hierarchy , class similarity , and object parts as discriminative features to aid the knowledge transfer ( Rohrbach et al. , 2011 ; Akata et al. , 2016 ; Zhao et al. , 2017 ; Elhoseiny et al. , 2017 ; Ji et al. , 2018 ; Cacheux et al. , 2019 ; Xie et al. , 2020 ) . Another focus is learning to align latent image-text embeddings , which allows to classify images using arbitrary texts . Frome et al . ( 2013 ) and Norouzi et al . ( 2014 ) are pioneering works that learn a visual-semantic embedding space using deep learning . Wang et al . ( 2018 ) distills information from both word embeddings and knowledge graphs . In recent works , CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) push the limit by collecting million/billion-scale image-text pairs and then train joint image-text embedding models using contrastive learning . Equipped with vast image-text knowledge from the Internet , these models can directly transfer to a suite of classification datasets and achieve impressive performances . While these work focus on image-level open-vocabulary recognition , we focus on detecting objects using arbitrary text inputs . Increasing vocabulary in object detection : It ’ s expensive to scale up the data collection for large vocabulary object detection . Zhao et al . ( 2020 ) and Zhou et al . ( 2021 ) unify the label space from multiple datasets . Joseph et al . ( 2021 ) incrementally learn identified unknown categories . Zero-shot detection ( ZSD ) offers another direction . Most ZSD methods align region features to pretrained text embeddings in base categories ( Bansal et al. , 2018 ; Demirel et al. , 2018 ; Rahman et al. , 2019 ; Hayat et al. , 2020 ; Zheng et al. , 2020 ) . However , there is a large performance gap to supervised counterparts . To address this issue , Zareian et al . ( 2021 ) pretrain the backbone model using image captions and finetune the pretrained model with detection datasets . In contrast , we use an image-text pretrained model as a teacher model to supervise student object detectors . All previous methods are only evaluated on tens of categories , while we are the first to evaluate on more than 1,000 categories . 3 METHOD . In this section , we introduce three main components in our approach : localization of novel objects , learning with pretrained text embeddings ( ViLD-text ) , and learning with pretrained image embeddings ( ViLD-image ) . Fig . 2 illustrates an overview of our method . We also study model ensembling approaches for the best detection performance . Notations : We divide annotations in an object detection dataset into two subsets based on their categories . We denote base categories and novel categories by CB and CN . Only annotations in CB are used for training . We use T ( · ) to denote the text encoder and V ( · ) to denote the image encoder in the pretrained open-vocabulary image classification model . 3.1 LOCALIZATION FOR NOVEL CATEGORIES . The first challenge for open-vocabulary detection is to localize novel objects . We modify a standard two-stage object detector , e.g. , Mask R-CNN ( He et al. , 2017 ) , for this purpose . We replace its classspecific localization modules , i.e. , the second-stage bounding box regression and mask prediction layers , with class-agnostic modules for general object proposals . For each region of interest , these modules only predict a single bounding box and a single mask for all categories , instead of one prediction per category . The class-agnostic modules can generalize to novel objects . 3.2 OPEN-VOCABULARY DETECTION WITH CROPPED REGIONS . Once object candidates are localized , we propose to reuse a pretrained open-vocabulary image classifier to classify each region for detection . Image embeddings : We train a proposal network on base categories CB and extract the region proposals r̃ ∈ P̃ offline . We crop and resize proposals , and feed them into the pretrained image encoder V to compute image embeddings V ( crop ( I , r̃ ) ) , where I is the image . We ensemble the image embeddings from 1× and 1.5× crops , as the 1.5× crop provides more context cues . The ensembled embedding is then renormalized to unit norm : V ( crop ( I , r̃ { 1×,1.5× } ) ) = v ‖v‖ , where v = V ( crop ( I , r̃1× ) ) + V ( crop ( I , r̃1.5× ) ) . ( 1 ) Text embeddings : We generate the text embeddings offline by feeding the category texts with prompt templates , e.g. , “ a photo of { category } in the scene ” , into the text encoder T . We ensemble multiple prompt templates and the synonyms if provided . Then , we compute cosine similarities between the image and text embeddings . A softmax activation is applied , followed by a per-class NMS to obtain final detections . The inference is slow since every cropped region is fed into V . 3.3 VILD : VISION AND LANGUAGE KNOWLEDGE DISTILLATION .. We propose ViLD to address the slow inference speed of the above method . ViLD learns region embeddings in a two-stage detector to represent each proposal r. We denote region embeddings by R ( φ ( I ) , r ) , where φ ( · ) is a backbone model and R ( · ) is a lightweight head that generates region embeddings . Specifically , we take outputs before the classification layer as region embeddings . Replacing classifier with text embeddings : We first introduce ViLD-text . Our goal is to train the region embeddings such that they can be classified by text embeddings . Fig . 3 ( b ) shows the architecture and training objective . ViLD-text replaces the learnable classifier in Fig . 3 ( a ) with the text embeddings introduced in Sec . 3.2 . Only T ( CB ) , the text embeddings of CB , are used for training . For the proposals that do not match to any groundtruth in CB , they are assigned to the background category . Since the text “ background ” does not well represent these unmatched proposals , we allow the background category to learn its own embedding ebg . We compute the cosine similarity between each region embedding R ( φ ( I ) , r ) and all category embeddings , including T ( CB ) and ebg . Then we apply softmax activation with a temperature τ to compute the cross entropy loss . To train the first-stage region proposal network of the two-stage detector , we extract its region proposals r ∈ P online , and train the detector with ViLD-text from scratch . The loss for ViLD-text can be written as : er = R ( φ ( I ) , r ) z ( r ) = [ sim ( er , ebg ) , sim ( er , t1 ) , · · · , sim ( er , t|CB | ) ] LViLD-text = 1 N ∑ r∈P LCE ( softmax ( z ( r ) /τ ) , yr ) , ( 2 ) where sim ( a , b ) = a > b/ ( ‖a‖‖b‖ ) , ti denotes elements in T ( CB ) , yr denotes the class label of region r , N is the number of proposals |P | per image , and LCE is the cross entropy loss . During inference , we include novel categories ( CN ) and generate T ( CB ∪CN ) ( sometimes T ( CN ) only ) for open-vocabulary detection ( Fig . 2 ) . Our hope is that the model learned from annotations in CB can generalize to novel categories CN . Distilling image embeddings : We then introduce ViLD-image , which aims to distill the knowledge from the teacher image encoder V into the student detector . Specifically , we align region embeddingsR ( φ ( I ) , r̃ ) to image embeddings V ( crop ( I , r̃ ) ) introduced in Sec . 3.2 . To make the training more efficient , we extract M proposals r̃ ∈ P̃ offline for each training image , and precompute the M image embeddings . These proposals can contain objects in both CB and CN , as the network can generalize . In contrast , ViLD-text can only learn from CB . We apply an L1 loss between the region and image embeddings to minimize their distance . The ensembled image embeddings in Sec . 3.2 are used for distillation : LViLD-image = 1 M ∑ r̃∈P̃ ‖V ( crop ( I , r̃ { 1×,1.5× } ) ) −R ( φ ( I ) , r̃ ) ‖1 . ( 3 ) Fig . 3 ( c ) shows the architecture . Zhu et al . ( 2019 ) use a similar approach to make Faster R-CNN features mimic R-CNN features , however , the details and goals are different : They reduce redundant context to improve supervised detection ; while ViLD-image is to enable open-vocabulary detection on novel categories . The total training loss of ViLD is simply a weighted sum of both objectives : LViLD = LViLD-text + w · LViLD-image , ( 4 ) where w is a hyperparameter weight for distilling the image embeddings . Fig . 3 ( d ) shows the model architecture and training objectives . ViLD-image distillation only happens in training time . During inference , ViLD-image , ViLD-text and ViLD employ the same set of text embeddings as the detection classifier , and use the same architecture for open-vocabulary detection ( Fig . 2 ) .

Summary: The authors propose a new method for training object detection models that can generalize to novel object categories without bounding box annotations. Their key idea is to use an existing vision-language joint embedding model (teacher), and distill its knowledge into a typical object detection model (student). This way, the detection model learns to project each object proposal into an embedding space, where it can be compared to the text embedding of various object categories, and the closest one can be chosen. The teacher model is an existing work (CLIP), which is trained on millions of image-caption pairs, and hence relies on weak supervision. The proposed method achieves impressive results on novel object detection, outperforming not only the best open-vocabulary detection method, but also fully supervised detectors.