Year: 2021

Abstract:
We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. Existing object detection datasets only contain hundreds of categories, and it is costly to scale further. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories not seen during training. ViLD obtains 16.1 mask APr, even outperforming the supervised counterpart by 3.8 with a ResNet-50 backbone. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP50, 36.6 AP and 11.8 AP on PASCAL VOC, COCO and Objects365, respectively. On COCO, ViLD outperforms previous SOTA (Zareian et al., 2021) by 4.8 on novel AP and 11.4 on overall AP.

heading: 1 INTRODUCTION
text: Consider Fig. 1, can we design object detectors beyond recognizing only base categories (e.g., toy) present in training labels and expand the vocabulary to detect novel categories (e.g., toy elephant)? In this paper, we aim to train an open-vocabulary object detector using only detection annotations in base categories, and detect objects in any novel categories described by text inputs.
Existing object detection algorithms often learn to detect only the categories present in detection datasets. A common approach to increase the detection vocabulary is by collecting images with more labeled categories. The research community has recently collected new object detection datasets with large vocabularies (Gupta et al., 2019; Kuznetsova et al., 2020). LVIS (Gupta et al., 2019) is a milestone of these efforts by building a dataset with 1,203 categories. With such a rich vocabulary, it becomes quite challenging to collect enough training examples for all categories. By Zipf’s law, object categories naturally follow a long-tailed distribution. To find sufficient training examples for rare categories, significantly more data is needed (Gupta et al., 2019), which makes it expensive to scale up detection vocabularies.
On the other hand, paired image-text data are abundant on the Internet. Recently, Radford et al. (2021) train a joint vision and language model using 400 million image-text pairs and demonstrate impressive results on directly transferring to over 30 classification datasets. The pretrained text encoder is the key to the zero-shot transfer ability to arbitrary text categories. Despite the great success on learning image-level representations, learning object-level representations for open-vocabulary detection is still challenging. In this work, we consider borrowing the knowledge from a pretrained open-vocabulary classification model to enable open-vocabulary detection.
We begin with an R-CNN (Girshick et al., 2014) style approach. We turn open-vocabulary detection into two sub-problems: 1) generalized object proposal and 2) open-vocabulary image classification. We train a region proposal model using examples in the base categories. Then we use the pretrained open-vocabulary image classification model to classify cropped object proposals, which can contain both base and novel categories. We benchmark on LVIS (Gupta et al., 2019) by holding out all rare categories as novel categories and treat others as base categories. To our surprise, the performance
on the novel categories already surpasses its supervised counterpart. However, this approach is very slow for inference, because it feeds object proposals one-by-one into the classification model.
To address the above issue, we propose ViLD (Vision and Language knowledge Distillation) for training two-stage open-vocabulary detectors. ViLD consists of two components: learning with text embeddings (ViLD-text) and image embeddings (ViLD-image) inferred by an open-vocabulary image classification model, e.g., CLIP (Radford et al., 2021). In ViLD-text, we obtain the text embeddings by feeding category names into the pretrained text encoder. Then the inferred text embeddings are used to classify detected regions. Similar approaches have been used in prior zero-shot and open-vocabulary detection works (Bansal et al., 2018; Rahman et al., 2018; Zareian et al., 2021). We find text embeddings learned jointly with visual data can better encode the visual similarity between concepts, compared to text embeddings learned from a language corpus, e.g., GloVe (Pennington et al., 2014). Using CLIP text embeddings achieves 10.1 APr (AP of novel categories) on LVIS, significantly outperforming the 3.0 APr of using GloVe. In ViLD-image, we obtain the image embeddings by feeding the object proposals into the pretrained image encoder. Then we train an object detector whose region embeddings of detected boxes are aligned with these image embeddings. In contrast to ViLD-text, ViLD-image distills knowledge from both base and novel categories, while ViLD-text only learns from base category concepts.
We show that ViLD, which jointly learns from text and image embeddings, achieves 16.1 APr on LVIS, surpassing the supervised counterpart by 3.8. To push the performance limit, we use ALIGN (Jia et al., 2021) as a stronger teacher model and attain the best performance of 26.3 APr, which is only 3.7 APr worse than the fully-supervised 2020 LVIS Challenge winner (Tan et al., 2020) that uses additional tricks. We also attain state-of-the-art results on COCO (Lin et al., 2014). Our method outperforms the previous SOTA (Zareian et al., 2021) by 4.8 AP on novel categories and 11.4 AP on all categories. In addition, ViLD trained on LVIS can directly transfer to other detection datasets including Objects365, COCO, PASCAL VOC.

heading: 2 RELATED WORK
text: Increasing vocabulary in visual recognition: Designing a computer vision model that can recognize objects using a large vocabulary is a long-standing research problem. One focus is zero-shot recognition, aiming at recognizing categories not present in the training set. Early works (Farhadi et al., 2009; Rohrbach et al., 2011; Jayaraman & Grauman, 2014) use visual attributes to create a binary codebook representing categories, which is used to transfer learned knowledge to unseen categories. In this direction, researchers have also explored class hierarchy, class similarity, and object parts as discriminative features to aid the knowledge transfer (Rohrbach et al., 2011; Akata et al., 2016; Zhao et al., 2017; Elhoseiny et al., 2017; Ji et al., 2018; Cacheux et al., 2019; Xie et al., 2020). Another focus is learning to align latent image-text embeddings, which allows to classify images using arbitrary texts. Frome et al. (2013) and Norouzi et al. (2014) are pioneering works that learn a visual-semantic embedding space using deep learning. Wang et al. (2018) distills information from both word embeddings and knowledge graphs. In recent works, CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) push the limit by collecting million/billion-scale image-text pairs and then train joint image-text embedding models using contrastive learning. Equipped with vast image-text knowledge from the Internet, these models can directly transfer to a suite of classification datasets and achieve impressive performances. While these work focus on image-level open-vocabulary recognition, we focus on detecting objects using arbitrary text inputs.
Increasing vocabulary in object detection: It’s expensive to scale up the data collection for large vocabulary object detection. Zhao et al. (2020) and Zhou et al. (2021) unify the label space from multiple datasets. Joseph et al. (2021) incrementally learn identified unknown categories. Zero-shot detection (ZSD) offers another direction. Most ZSD methods align region features to pretrained text embeddings in base categories (Bansal et al., 2018; Demirel et al., 2018; Rahman et al., 2019; Hayat et al., 2020; Zheng et al., 2020). However, there is a large performance gap to supervised counterparts. To address this issue, Zareian et al. (2021) pretrain the backbone model using image captions and finetune the pretrained model with detection datasets. In contrast, we use an image-text pretrained model as a teacher model to supervise student object detectors. All previous methods are only evaluated on tens of categories, while we are the first to evaluate on more than 1,000 categories.

heading: 3 METHOD
text: In this section, we introduce three main components in our approach: localization of novel objects, learning with pretrained text embeddings (ViLD-text), and learning with pretrained image embeddings (ViLD-image). Fig. 2 illustrates an overview of our method. We also study model ensembling approaches for the best detection performance.
Notations: We divide annotations in an object detection dataset into two subsets based on their categories. We denote base categories and novel categories by CB and CN . Only annotations in CB are used for training. We use T (·) to denote the text encoder and V(·) to denote the image encoder in the pretrained open-vocabulary image classification model.

heading: 3.1 LOCALIZATION FOR NOVEL CATEGORIES
text: The first challenge for open-vocabulary detection is to localize novel objects. We modify a standard two-stage object detector, e.g., Mask R-CNN (He et al., 2017), for this purpose. We replace its classspecific localization modules, i.e., the second-stage bounding box regression and mask prediction layers, with class-agnostic modules for general object proposals. For each region of interest, these modules only predict a single bounding box and a single mask for all categories, instead of one prediction per category. The class-agnostic modules can generalize to novel objects.

heading: 3.2 OPEN-VOCABULARY DETECTION WITH CROPPED REGIONS
text: Once object candidates are localized, we propose to reuse a pretrained open-vocabulary image classifier to classify each region for detection.
Image embeddings: We train a proposal network on base categories CB and extract the region proposals r̃ ∈ P̃ offline. We crop and resize proposals, and feed them into the pretrained image encoder V to compute image embeddings V(crop(I, r̃)), where I is the image. We ensemble the image embeddings from 1× and 1.5× crops, as the 1.5× crop provides more context cues. The ensembled embedding is then renormalized to unit norm:
V(crop(I, r̃{1×,1.5×})) = v
‖v‖ , where v = V(crop(I, r̃1×)) + V(crop(I, r̃1.5×)). (1)
Text embeddings: We generate the text embeddings offline by feeding the category texts with prompt templates, e.g., “a photo of {category} in the scene”, into the text encoder T . We ensemble multiple prompt templates and the synonyms if provided.
Then, we compute cosine similarities between the image and text embeddings. A softmax activation is applied, followed by a per-class NMS to obtain final detections. The inference is slow since every cropped region is fed into V .

heading: 3.3 VILD: VISION AND LANGUAGE KNOWLEDGE DISTILLATION.
text: We propose ViLD to address the slow inference speed of the above method. ViLD learns region embeddings in a two-stage detector to represent each proposal r. We denote region embeddings by R(φ(I), r), where φ(·) is a backbone model and R(·) is a lightweight head that generates region embeddings. Specifically, we take outputs before the classification layer as region embeddings.
Replacing classifier with text embeddings: We first introduce ViLD-text. Our goal is to train the region embeddings such that they can be classified by text embeddings. Fig. 3(b) shows the architecture and training objective. ViLD-text replaces the learnable classifier in Fig. 3(a) with the text embeddings introduced in Sec. 3.2. Only T (CB), the text embeddings of CB , are used for training. For the proposals that do not match to any groundtruth in CB , they are assigned to the background category. Since the text “background” does not well represent these unmatched proposals, we allow the background category to learn its own embedding ebg . We compute the cosine similarity between each region embedding R(φ(I), r) and all category embeddings, including T (CB) and ebg . Then we apply softmax activation with a temperature τ to compute the cross entropy loss. To train the first-stage region proposal network of the two-stage detector, we extract its region proposals r ∈ P online, and train the detector with ViLD-text from scratch. The loss for ViLD-text can be written as:
er = R(φ(I), r) z(r) = [ sim(er, ebg), sim(er, t1), · · · , sim(er, t|CB |) ] LViLD-text = 1
N ∑ r∈P LCE ( softmax ( z(r)/τ ) , yr ) , (2)
where sim(a,b) = a>b/(‖a‖‖b‖), ti denotes elements in T (CB), yr denotes the class label of region r, N is the number of proposals |P | per image, and LCE is the cross entropy loss. During inference, we include novel categories (CN ) and generate T (CB ∪CN ) (sometimes T (CN ) only) for open-vocabulary detection (Fig. 2). Our hope is that the model learned from annotations in CB can generalize to novel categories CN .
Distilling image embeddings: We then introduce ViLD-image, which aims to distill the knowledge from the teacher image encoder V into the student detector. Specifically, we align region embeddingsR(φ(I), r̃) to image embeddings V(crop(I, r̃)) introduced in Sec. 3.2.
To make the training more efficient, we extract M proposals r̃ ∈ P̃ offline for each training image, and precompute the M image embeddings. These proposals can contain objects in both CB and CN , as the network can generalize. In contrast, ViLD-text can only learn from CB . We apply an L1 loss between the region and image embeddings to minimize their distance. The ensembled image embeddings in Sec. 3.2 are used for distillation:
LViLD-image = 1
M ∑ r̃∈P̃ ‖V(crop(I, r̃{1×,1.5×}))−R(φ(I), r̃)‖1. (3)
Fig. 3(c) shows the architecture. Zhu et al. (2019) use a similar approach to make Faster R-CNN features mimic R-CNN features, however, the details and goals are different: They reduce redundant context to improve supervised detection; while ViLD-image is to enable open-vocabulary detection on novel categories.
The total training loss of ViLD is simply a weighted sum of both objectives:
LViLD = LViLD-text + w · LViLD-image, (4)
where w is a hyperparameter weight for distilling the image embeddings. Fig. 3(d) shows the model architecture and training objectives. ViLD-image distillation only happens in training time. During inference, ViLD-image, ViLD-text and ViLD employ the same set of text embeddings as the detection classifier, and use the same architecture for open-vocabulary detection (Fig. 2).

heading: 3.4 MODEL ENSEMBLING
text: In this section, we explore model ensembling for the best detection performance over base and novel categories. First, we combine the predictions of a ViLD-text detector with the open-vocabulary image classification model. The intuition is that ViLD-image learns to approximate the predictions of its teacher model, and therefore, we assume using the teacher model directly may improve performance. We use a trained ViLD-text detector to obtain top k candidate regions and their confidence scores. Let pi,ViLD-text denote the confidence score of proposal r̃ belonging to category i. We then feed crop(I, r̃) to the open-vocabulary classification model to obtain the teacher’s confidence score pi,cls. Since we know the two models have different performance on base and novel categories, we introduce a weighted geometric average for the ensemble:
pi,ensemble =
{ pλi,ViLD-text · p (1−λ) i,cls , if i ∈ CB
p (1−λ) i,ViLD-text · pλi,cls. if i ∈ CN
(5)
λ is set to 2/3, which weighs the prediction of ViLD-text more on base categories and vice versa. Note this approach has a similar slow inference speed as the method in Sec. 3.2.
Next, we introduce a different ensembling approach to mitigate the above inference speed issue. Besides, in ViLD, the cross entropy loss of ViLD-text and the L1 distillation loss of ViLD-image is applied to the same set of region embeddings, which may cause contentions. Here, instead, we learn two sets of embeddings for ViLD-text (Eq. 2) and ViLD-image (Eq. 3) respectively, with two separate heads of identical architectures. Text embeddings are applied to these two regions embeddings to obtain confidence scores pi,ViLD-text and pi,ViLD-image, which are then ensembled in the same way as Eq. 5, with pi,ViLD-image replacing pi,cls. We name this approach ViLD-ensemble.

heading: 4 EXPERIMENTS
text: Implementation details: We benchmark on the Mask R-CNN (He et al., 2017) with ResNet (He et al., 2016) FPN (Lin et al., 2017) backbone and use the same settings for all models unless explicitly specified. The models use 1024×1024 as input image size, large-scale jittering augmentation (Ghiasi et al., 2021), synchronized batch normalization (Ioffe & Szegedy, 2015; Girshick et al., 2018) of batch size 256, weight decay of 4e-5, and an initial learning rate of 0.32. We train the model from scratch for 180,000 iterations, and divide the learning rate by 10 at 0.9×, 0.95×, and 0.975× of total iterations. We use the publicly available pretrained CLIP model 1 as the open-vocabulary classification model, with an input size of 224×224. The temperature τ is set to 0.01, and the maximum number of detections per image is 300. We refer the readers to Appendix D for more details.

heading: 4.1 BENCHMARK SETTINGS
text: We mainly evaluate on LVIS (Gupta et al., 2019) with our new setting. To compare with previous methods, we also use the setting in Zareian et al. (2021), which is adopted in many zero-shot detection works.
LVIS: We benchmark on LVIS v1. LVIS contains a large and diverse set of vocabulary (1203 categories) that is more suitable for open-vocabulary detection. We take its 866 frequent and common categories as the base categories CB , and hold out the 337 rare categories as the novel categories CN . APr, the AP of rare categories, is the main metric.
COCO: Bansal et al. (2018) divide COCO-2017 (Lin et al., 2014) into 48 base categories and 17 novel categories, removing 15 categories without a synset in WordNet hierarchy. We follow previous works and do not compute instance masks. We evaluate on the generalized setting.

heading: 4.2 LEARNING GENERALIZABLE OBJECT PROPOSALS
text: We first study whether a detector can localize novel categories when only trained on base categories. We evaluate the region proposal networks in Mask R-CNN with a ResNet-50 backbone. Table 1 shows the average recall (AR) Lin et al. (2014) on novel categories. Training with only base categories performs slightly worse by ∼ 2 AR at 100, 300, and 1000 proposals, compared to using both base and novel categories. This experiment demonstrates that, without seeing novel categories during training, region proposal networks can generalize to novel categories, only suffering a small performance drop. We believe better proposal networks focusing on unseen category generalization should further improve the performance, and leave this for future research.

heading: 4.3 OPEN-VOCABULARY CLASSIFIER (CLIP) ON CROPPED REGIONS
text: In table 2, we evaluate the approach in Sec. 3.2, i.e., using an open-vocabulary classifier to classify cropped region proposals. We use CLIP in this experiment and find it tends to output confidence scores regardless of the localization quality (Appendix B). Given that, we ensemble the CLIP confidence score with a proposal objectness score by geometric mean. Results show it improves both base and novel APs. We compare with supervised baselines trained on base/base+novel categories, as well as Supervised-RFS (Mahajan et al., 2018; Gupta et al., 2019) that uses category frequency for balanced sampling. CLIP on cropped regions already outperforms supervised baselines on APr by a large margin, without accessing detection annotations in novel categories. However, the performances of APc and APf are still trailing behind. This experiment shows that a strong openvocabulary classification model can be a powerful teacher model for detecting novel objects, yet there is still much improvement space for inference speed and overall AP.
1https://github.com/openai/CLIP, ViT-B/32.

heading: 4.4 VISION AND LANGUAGE KNOWLEDGE DISTILLATION
text: We evaluate the performance of ViLD and its variants (ViLD-text, ViLD-image, and ViLDensemble), which are significantly faster compared to the method in Sec. 4.3. Finally, we replace CLIP with ALIGN to demonstrate our best performance. Table 3 summarizes the results.
Text embeddings as classifiers (ViLD-text): We evaluate ViLD-text using text embeddings generated by CLIP, and compare it with GloVe text embeddings (Pennington et al., 2014) pretrained on a large-scale text-only corpus. Table 3 shows ViLD-text achieves 10.1 APr, which is significantly better than 3.0 APr using GloVe. This demonstrates the importance of using text embeddings that are jointly trained with images. ViLD-text achieves much higher APc and APf compared to CLIP on cropped regions (Sec. 4.3), because ViLD-text uses annotations in CB to align region embeddings with text embeddings. The APr is worse, showing that only using 866 base categories in LVIS does not generalize as well as CLIP to novel categories.
Distilling image embeddings (ViLD-image): We use the model in Section 4.3 as the teacher model and distill from its image embeddings of proposals, inferred by CLIP’s image encoder, with a distillation weight of 1.0. Experiments show that ensembling with objectness scores doesn’t help with other ViLD variants, so we only apply it to ViLD-image. Ideally, the detection performance of ViLDimage (student) and CLIP on cropped regions (teacher) would be similar. As shown in Table 3, the performance gap is 7.7 APr. This demonstrates that visual distillation works for open-vocabulary detection but yields a decrease in performance.
Text+visual embeddings (ViLD): ViLD shows the benefits of combining distillation loss (ViLDimage) with classification loss using text embeddings (ViLD-text). We explore different hyperparameter settings in Appendix Table 7 and observe a consistent trade-off between APr to APc,f , which suggests there is a competition between ViLD-text and ViLD-image. In Table 3, we compare ViLD with other methods. Its APr is 6.0 higher than ViLD-text and 4.9 higher than ViLD-image, indicating combining the two learning objectives boosts the performance on novel categories. ViLD outperforms Supervised-RFS by 3.8 APr, showing our open-vocabulary detection approach is better than supervised models on rare categories.
Model ensembling: We study methods discussed in Sec. 3.4 to reconcile the conflict of joint training with ViLD-text and ViLD-image. We use two ensembling approaches: 1) ensembling ViLD-text with CLIP (ViLD-text+CLIP); 2) ensembling ViLD-text and ViLD-image using separate heads (ViLD-ensemble). As shown in Table 3, ViLD-ensemble improves performance over ViLD, mainly on APc and APr. This shows ensembling reduces the competition. ViLD-text+CLIP obtains much higher APr, outperforming ViLD by 6.5, and maintains good APc,f . Note that it is slow and impractical for real world applications. This experiment is designed for showing the potential of using open-vocabulary classification model for open-vocabulary detection.
Stronger teacher model: We use a pretrained ALIGN model (Jia et al., 2021) to explore the performance gain with a stronger teacher model (details in Appendix D). As shown in Table 3, the resulting detector outperforms R50 ViLD w/ CLIP by 9.7 APr, only trailing to the fully-supervised 2020 Challenge winner (Tan et al., 2020) by 3.7 APr, which employs two-stage training, self-training, and multi-scale testing etc. The results demonstrate ViLD scales well with the teacher model, and is a promising open-vocabulary detection approach.

heading: 4.5 PERFORMANCE COMPARISON ON COCO DATASET
text: Several related works in zero-shot detection and open-vocabulary detection are evaluated on COCO. To compare with them, we train and evaluate ViLD variants following the benchmark setup in Zareian et al. (2021) and report box AP with an IoU threshold of 0.5. We use the ResNet-50 backbone, shorten the training schedule to 45,000 iterations, and keep other settings the same as our experiments on LVIS. Table 4 summarizes the results. ViLD outperforms Zareian et al. (2021) by 4.8 Novel AP and 13.5 Base AP. Different from Zareian et al. (2021), we do not have a pretraining phase tailored for detection. Instead, we use an off-the-shelf classification model. The performance of ViLD-text is low because only 48 base categories are available, which makes generalization to novel categories challenging. In contrast, ViLD-image and ViLD, which can distill image features of novel categories, outperform all existing methods.

heading: 4.6 TRANSFER TO OTHER DATASETS
text: Trained ViLD models can be transferred to other detection datasets, by simply switching the classifier to the category text embeddings of the new datasets. For simplicity, we keep the background embedding trained on LVIS. We evaluate the transferability of ViLD on PASCAL VOC (Everingham et al., 2010), COCO (Lin et al., 2014), and Objects365 (Shao et al., 2019). Because these three datasets have much smaller vocabularies than LVIS, category overlap is unavoidable and images can be shared across datasets, e.g., COCO and LVIS. As shown in Table 5, ViLD achieves better transfer performance than ViLD-text. In PASCAL and COCO, the gap is large. This improvement should be credited to visual distillation, which better aligns region embeddings with the text classifier. We also compare with supervised training and finetuning the last linear layer. Although across datasets, ViLD has 3-6 AP gaps compared to the finetuning method and larger gaps compared to the supervised method, it is the first time we can directly transfer a trained detector to different datasets using language.

heading: 4.7 QUALITATIVE RESULTS
text: In Fig. 4, we visualize ViLD’s detection results. It illustrates ViLD is able to detect objects of both novel and base categories, with high-quality mask predictions for novel objects, e.g., it well separates banana slices from the crepes (novel category). We also show qualitative results on COCO and Objects365, and find ViLD generalizes well. We show more qualitative results, e.g., interactive detection and systematic expansion, in Appendix A.

heading: 5 CONCLUSION
text: We present ViLD, an open-vocabulary detection method by distilling knowledge from openvocabulary image classification models. ViLD is the first open-vocabulary detection method evaluated on the challenging LVIS dataset. It attains 16.1 APr, which surpasses its supervised counterpart at the same inference speed. With a stronger teacher model (ALIGN), the performance can be further improved to 26.3. We demonstrate the detector learned from LVIS can be directly transferred to 3 other detection datasets. Finally, ViLD shows a scalable alternative approach for detecting long-tailed categories instead of collecting expensive detection annotations.
ETHICS STATEMENT
Our paper studies open-vocabulary object detection, a sub-field in computer vision. Our method is based on knowledge distillation, a machine learning technique that has been extensively used in computer vision, natural language processing, etc. All of our experiments were conducted on public datasets with pretrained models that are either publicly available or introduced in published papers. The method proposed in our paper is a principled method for open-vocabulary object detection that can be used in a wide range of applications. Therefore, the ethical impact of our work would primarily depends on the specific applications. We foresee positive impacts if our method is applied to object detection problems where the data collection is difficult to scale, such as detecting rare objects for self-driving cars. But the method can also be applied to other sensitive applications that could raise ethical concerns such as video surveillance systems.
REPRODUCIBILITY STATEMENT
We provide detailed descriptions of the proposed method in Sec. 3. Details about experiment settings, hyper-parameters and implementations are presented in Sec. 4, Appendix C and Appendix D. We still need time to clean up the code and secure peer reviews before releasing the code. We promise to release our code and pretrained models to facilitate the reproducibility of our work.

heading: A ADDITIONAL QUALITATIVE RESULTS
text: On-the-fly interactive object detection: We tap the potential of ViLD by using arbitrary text to interactively recognize fine-grained categories and attributes. We extract the region embedding and compute its cosine similarity with a small set of on-the-fly arbitrary texts describing attributes and/or fine-grained categories; we apply a softmax with temperature on top of the similarities. To our surprise, though never trained on fine-grained dog breeds (Fig. 5), it correctly distinguishes husky from shiba inu. It also works well on identifying object colors (Fig. 1). The results demonstrate knowledge distillation from an open-vocabulary image classification model helps ViLD to gain understanding of concepts not present in the detection training. Of course, ViLD does not work all the time, e.g., it fails to recognize poses of animals.
Systematic expansion of dataset vocabulary: In addition, we propose to systematically expand the dataset vocabulary (v = {v1, ..., vp}) with a set of attributes (a = {a1, ..., aq}) as follows:
Pr(vi, aj | er) = Pr(vi | er) · Pr(aj | vi, er) = Pr(vi | er) · Pr(aj | er), (6)
where er denotes the region embedding. We assume vi ⊥ aj | er, i.e., given er the event the object belongs to category vi is conditionally independent to the event it has attribute aj .
Let τ denote the temperature used for softmax and T denote the text encoder as in Eq. 2. Then
Pr(vi | er) = softmaxi(sim(er, T (v))/τ), (7) Pr(aj | er) = softmaxj(sim(er, T (a))/τ). (8)
In this way, we are able to expand p vocabularies into a new set of p×q vocabularies with attributes. The conditional probability approach is similar to YOLO9000 (Redmon & Farhadi, 2017). We show a qualitative example of this approach in Fig. 6, where we use a color attribute set as a. Our open-vocabulary detector successfully detects fruits with color attributes.
We further expand the detection vocabulary to fine-grained bird categories by using all 200 species from CUB-200-2011 (Wah et al., 2011). Fig. 7 shows successful and failure examples of our openvocabulary fine-grained detection on CUB-200-2011 images. In general, our model is able to detect visually distinctive species, but fails at other ones.
Transfer to PASCAL VOC: In Fig. 8, we show qualitative results of transferring an openvocabulary detector trained on LVIS (Gupta et al., 2019) to PASCAL VOC Detection (2007 test set) (Everingham et al., 2010), without finetuning (Sec. 4.6 in the main paper). Results demonstrate that the transferring works well.
Failure cases: In Fig. 9, we show two failure cases of ViLD. The most common failure cases are the missed detection. A less common mistake is misclassifying the object category.
We show a failure case of mask prediction on PASCAL VOC in Fig. 10. It seems that the mask prediction is sometimes based on low-level appearance rather than semantics.

heading: B ANALYSIS OF CLIP ON CROPPED REGIONS
text: In this section, we analyze some common failure cases of CLIP on cropped regions and discuss possible ways to mitigate these problems.
Visual similarity: This confusion is common for any classifiers and detectors, especially on large vocabularies. In Fig. 11(a), we show two failure examples due to visual similarity. Since we only use a relatively small ViT-B/32 CLIP model, potentially we can improve the performance with a higher-capacity pretrained model. In Table 6, when replacing this CLIP model with an EfficientNetl2 ALIGN model, we see an increase on AP.
Aspect ratio: This issue is introduced by the pre-processing of inputs in CLIP. We use the ViT-B/32 CLIP with a fixed input resolution of 224 × 224. It resizes the shorter edge of the image to 224, and then uses a center crop. However, since region proposals can have more extreme aspect ratios than the training images for CLIP, and some proposals are tiny, we directly resize the proposals to that resolution, which might cause some issues. For example, the thin structure in Fig. 11(b) right will be highly distorted with the pre-processing. And the oven and fridge can be confusing with the distorted aspect ratio. There might be some simple remedies for this, e.g., pasting the cropped region with original aspect ratio on a black background. We tried this simple approach with both CLIP and ALIGN. Preliminary results show that it works well on the fully convolutional ALIGN, while doesn’t work well on the transformer-based CLIP, probably because CLIP is never trained with black image patches.
Multiple objects in a bounding box: Multiple objects in a region interfere CLIP’s classification results, see Fig. 11(c), where a corner of an aquarium dominates the prediction. This is due to CLIP pretraining, which pairs an entire image with its caption. The caption is usually about salient objects in the image. It’s hard to mitigate this issue at the open-vocabulary classification model’s end. On the other hand, a supervised detector are trained to recognize the object tightly surrounded by the bounding box. So when distilling knowledge from an open-vocabulary image classification model, keeping training a supervised detector on base categories could help, as can be seen from the improvement of ViLD over ViLD-image (Sec. 4.4).
Confidence scores predicted by CLIP do not reflect the localization quality: For example, in Fig. 12(a), CLIP correctly classifies the object, but gives highest scores to partial detection boxes. CLIP is not trained to measure the quality of bounding boxes. Nonetheless, in object detection, it is important for the higher-quality boxes to have higher scores. In Fig. 12(c), we simply re-score by taking the geometric mean of CLIP confidence score and the objectness score from the proposal
model, which yields much better top predictions. In Fig. 12(b), we show top predictions of the Mask R-CNN model. Its top predictions have good bounding boxes, while the predicted categories are wrong. This experiment shows that it’s important to have both an open-vocabulary classification model for better recognition, as well as supervision from detection dataset for better localization.

heading: C ADDITIONAL QUANTITATIVE RESULTS
text: Hyperparameter sweep for visual distillation: Table 7 shows the parameter sweep of different distillation weights using L1 and L2 losses. Compared with no distillation, additionally learning from image embeddings generally yields better performance on novel categories. We find L1 loss can better improve the APr performance with the trade-off to APc and APf . This suggests there is a competition between ViLD-text and ViLD-image.
Box APs and ResNet-152 backbone: Table 8 shows the corresponding box AP of Table 3 in the main paper. In general, box AP is slightly higher than mask AP. In addition, we include the results of ViLD variants with ResNet-152 backbone. The deeper backbone improves all metrics. The trend/relative performance is consistent for box and mask APs, as well as for different backbones. ViLD-ensemble achieves the best box and mask APr.
Ablation study on prompt engineering: We conduct an ablation study on prompt engineering. We compare the text embeddings ensembled over synonyms and 63 prompt templates (listed in Appendix D) with a non-ensembled version: Using the single prompt template “a photo of {article} {category}”. Table 9 illustrates that ensembling multiple prompts slightly improves the performance by 0.4 APr.

heading: D MORE IMPLEMENTATION DETAILS
text: Model used for qualitative results: For all qualitative results, we use a ViLD model with ResNet152 backbone, whose performance is shown in Table 8.
Details for supervised baselines: For a fair comparison, we train the second stage box/mask prediction heads of Supervised and Supervised-RFS baselines in the class-agnostic manner introduced in Sec. 3.1.
Details for R-CNN style experiments: We provide more details here for the R-CNN style experiments, CLIP on cropped regions in Sec. 4.2 and ViLD-text+CLIP in Sec. 4.3. 1) Generalized object proposal: We use the standard Mask R-CNN R50-FPN model. To report mask AP and compare with other methods, we treat the second-stage refined boxes as proposals and use the corresponding masks. We apply a class-agnostic NMS with 0.9 threshold, and output a maximum of 1000 proposals. The objectness score is 1 - the background score. 2) Open-vocabulary classification on cropped regions: After obtaining CLIP confidence scores for the 1000 proposals, we apply a class-specific NMS with a threshold of 0.6, and output the top 300 detections as the final results.
Architecture for open-vocabulary image classification models: Popular open-vocabulary image classification models (Radford et al., 2021; Jia et al., 2021) perform contrastive pre-training on a large number of image-text pairs. Given a batch of paired images and texts, the model learns to maximize the cosine similarity between the embeddings of the corresponding image and text pairs while minimizing the cosine similarity between other pairs. Specifically, for CLIP (Radford et al.,
2021), we use the version where the image encoder adopts the Vision Transformer (Dosovitskiy et al., 2020) ViT-B/32 architecture and the text encoder is a Transformer (Vaswani et al., 2017). For ALIGN (Jia et al., 2021), its image encoder is an EfficientNet (Tan & Le, 2019) and its text encoder is a BERT (Devlin et al., 2019).
Details for ViLD w/ ALIGN: We use the ALIGN model with an EfficientNet-l2 image encoder and a BERT-large text encoder as the teacher model, and adopt ViLD-ensemble for its better performance in this case. We modify several places in the Mask R-CNN architecture to better distill the knowledge from the teacher. We equip the ViLD-image head in ViLD-ensemble with the MBConvBlocks in EfficientNet-l2. Since the MBConvBlocks are fully-convolutional, we apply a global average pooling to obtain the image embeddings, following the teacher. The ViLD-text head keeps the same Faster R-CNN head architecture as in Mask R-CNN. Since ALIGN image/text embeddings have 1,376 dimensions (2.7× CLIP embedding dimension), we increase the number of units in the fully connected layers of the ViLD-text head to 2,048, and the FPN dimension to 1,024. We also crop the RoI features from only FPN level P3 in the feature pyramid. The large-scale jittering range is reduced to [0.5, 2.0].
Text prompts: Since the open-vocabulary classification model is trained on full sentences, we feed the category names into a prompt template first, and ensemble various prompts. Following Radford et al. (2021), we curate a list of 63 prompt templates. We specially include several prompts containing the phrase “in the scene” to better suit object detection, e.g., “There is {article} {category} in the scene”.
Our list of prompt templates is shown below:
’There is {article} {category} in the scene.’ ’There is the {category} in the scene.’ ’a photo of {article} {category} in the scene.’ ’a photo of the {category} in the scene.’ ’a photo of one {category} in the scene.’ ’itap of {article} {category}.’ ’itap of my {category}.’ ’itap of the {category}.’ ’a photo of {article} {category}.’ ’a photo of my {category}.’ ’a photo of the {category}.’ ’a photo of one {category}.’ ’a photo of many {category}.’ ’a good photo of {article} {category}.’ ’a good photo of the {category}.’ ’a bad photo of {article} {category}.’ ’a bad photo of the {category}.’ ’a photo of a nice {category}.’ ’a photo of the nice {category}.’ ’a photo of a cool {category}.’ ’a photo of the cool {category}.’ ’a photo of a weird {category}.’ ’a photo of the weird {category}.’ ’a photo of a small {category}.’ ’a photo of the small {category}.’ ’a photo of a large {category}.’ ’a photo of the large {category}.’ ’a photo of a clean {category}.’ ’a photo of the clean {category}.’ ’a photo of a dirty {category}.’ ’a photo of the dirty {category}.’ ’a bright photo of {article} {category}.’ ’a bright photo of the {category}.’ ’a dark photo of {article} {category}.’ ’a dark photo of the {category}.’
’a photo of a hard to see {category}.’ ’a photo of the hard to see {category}.’ ’a low resolution photo of {article} {category}.’ ’a low resolution photo of the {category}.’ ’a cropped photo of {article} {category}.’ ’a cropped photo of the {category}.’ ’a close-up photo of {article} {category}.’ ’a close-up photo of the {category}.’ ’a jpeg corrupted photo of {article} {category}.’ ’a jpeg corrupted photo of the {category}.’ ’a blurry photo of {article} {category}.’ ’a blurry photo of the {category}.’ ’a pixelated photo of {article} {category}.’ ’a pixelated photo of the {category}.’ ’a black and white photo of the {category}.’ ’a black and white photo of {article} {category}.’ ’a plastic {category}.’ ’the plastic {category}.’ ’a toy {category}.’ ’the toy {category}.’ ’a plushie {category}.’ ’the plushie {category}.’ ’a cartoon {category}.’ ’the cartoon {category}.’ ’an embroidered {category}.’ ’the embroidered {category}.’ ’a painting of the {category}.’ ’a painting of a {category}.’

Review summaries:
This paper aims to propose a novel approach for open-vocabulary or zero-shot object detection by leveraging models pre-trained on internet-scale data which might include many of the desired open-vocabulary categories. The proposed approach, called ViLD, uses the information learned by models like CLIP, and ALIGN to detect "novel" classes. The paper shows that the proposed approach can achieve better performance than supervised methods on rare classes in the LVIS dataset. However, the performance is much lower than state-of-the-art. Also, the performance on frequent classes in LVIS, and the overall performance is significantly lower than prior works. The authors also show that the proposed approach can achieve good performance improvements on the zero-shot COCO dataset - both for "unseen" and seen classes.  

The authors propose a new method for training object detection models that can generalize to novel object categories without bounding box annotations. Their key idea is to use an existing vision-language joint embedding model (teacher), and distill its knowledge into a typical object detection model (student). This way, the detection model learns to project each object proposal into an embedding space, where it can be compared to the text embedding of various object categories, and the closest one can be chosen. The teacher model is an existing work (CLIP), which is trained on millions of image-caption pairs, and hence relies on weak supervision. The proposed method achieves impressive results on novel object detection, outperforming not only the best open-vocabulary detection method, but also fully supervised detectors. 

This paper addresses the task of open-vocabulary detection, where an object detector is trained from two forms of annotations: (a) bounding box annotations for a set of base categories and (b) open-vocabulary image-caption pairs.  This detector is then evaluated on a set of novel categories.  Compared to prior work, this paper tries to distill knowledge from strong pre-trained vision & language models (like CLIP) into an open-vocabulary detector. The authors propose multiple ways to do this (RCNN+CLIP, ViLD-text, ViLD-image, ViLD), including some ensemble variants (ViLD-ensemble). The paper also introduces a novel experimental setup based on the LVIS dataset (with >1200 categories) as compared to prior works that used COCO with only 80 categories.