Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
Authors: Xiuye Gu
Tsung-Yi Lin
Weicheng Kuo
Yin Cui
Venue: ICLR 2022 Poster (2022)
Keywords: Open-vocabulary recognition
Object detection
Knowledge distillation

Abstract: We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data.  It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask APr with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 APr. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP50 on PASCAL VOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the-art (Zareian et al., 2021) by 4.8 on novel AP and 11.4 on overall AP. Code and demo are open-sourced at https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.

Published as a conference paper at ICLR 2022
OPEN-VOCABULAS OBJECT DETECTION VIA VISION AND LANGUAGE KNOWLEDGE DISTILLATION
Xiuye Gu', Tsung- Yi Lin', Weicheng Kuo', Yin Cui
GGoole Research, "Nvidia* {xiuyegu, weicheng, yincui jincul]@google.com
tsungyllenvicia.com
ABSTRACT
We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge the availabil- ity of training data. It is costly to further scale up the number of classes con- tained in existing object detection datasets. To overcome this challenge, we pro- pose ViLD, training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of ob- ject proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding all rare categories novel categories that are not seen during training. ViLD obtains 16.1 mask AP, with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with stronger teacher model ALIGN, ViLD achieves 26.3 AP,. The model can directly transfer to other datasets without finetun- ing, achieving 72.2 AP50 PASCAL vOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the- art (Zareian al., 2021 by 4.8 novel AP and 11.4 overall AP. Code and demo are open-sourced at http://w........ttt.ttet////ttaa
INTRODUCTION
Consider Fig. [ can we design object detectors beyond recognizing only base categories (e.g., toy) present in training labels and expand the vocabulary to detect novel categories (e.g., toy elephant)? In this paper, we aim to train an open-vocabulary object detector that detects objects in any novel categories described by text inputs, using only detection annotations in base categories.
Existing object detection algorithms often learn to detect only the categories present in detection datasets. common approach to increase the detection vocabulary is by collecting images with more labeled categories. The research community has recently collected new object detection datasets with large vocabularies (Gupta et al.) 2019 Kuznetsova et al.) 2020). LVIS (Gupta et al. 2019) is milestone of these efforts by building dataset with 1,203 categories. With such a rich vocabulary, becomes quite challenging to collect enough training examples for all categories. By Zipf's law, object categories naturally follow a long tailed distribution. To find sufficient training examples for rare categories, significantly more data is needed (Gupta et al. 2019), which makes expensive to scale up detection vocabularies.
On the other hand, paired image-text data are abundant on the Internet. Recently, Radford et al. (202T) train joint vision and language model using 400 million image-text pairs and demonstrate impressive results on directly transferring to over 30 datasets. The pretrained text encoder is the key to the zero-shot transfer ability arbitrary text categories. Despite the great success on learn- ing image-level representations, learning object-level representations for open- nece-vocabllary detection still challenging. In this work, we consider borrowing the knowledge from a pretrained open- vocabulary classification model to enable vocabulayy detection.
Work done while Xiuye Google Al Resident and Tsung-Yi Yi was at Gongle.

---

Published as a conference paper at ICLR 2022
ICE HUSERR D.0 D0c
Nove cateprti
canggiers
Figure 1: An example of open-vocabulary detector with arbitrary texts. After training base cate- gories (purple), we can detect novel categories (pink) that are not present in the training data.
We begin with an R-CNN (Girshick al. 2014) style approach. We turn open-vocabulate detection into two sub sub-problems: 1) generalized object proposal and 2) open-vocabulary image classification. We train a region proposal model using examples from the base categories. Then we use the pre- trained open- vocablaayyy image classification model to classify cropped object proposals, which can contain both base and novel categories. We benchmark on LVIS (Gupta et al. 2019) by holding out all rare categories mance on the novel categories already surpasses its supervised counterpart. However, this approach is very slow for inference, because it feeds object proposals one-by-one into the classification model.
novel categories and treat others base categories. To our surprise, the perfor-
To address the above issue, we propose ViLD (Vision and Language knowledge Distillation) for training two-stage open- vocabulary detectors. ViLD consists of two components: learning with text embeddings (Vil.-ttext) and image embeddings (ViL.D-image) inferred by an open-vocabulary image classification model, e.g., CLIP. In ViI.-text,
obtain the text embeddings by feeding category names into the pretrained text encoder. Then the inferred text embeddings are used to classify detected regions. Similar approaches have been used in prior detection works (Bansal al., 2018 Rahman et al.] 2018 Zareian et al.] 2021). We find text embeddings learned jointly with visual data can better encode the visual similarity between concepts, compared to text embeddings learned from language corpus, e.g., Glo (Pennington et al. 2014). Using CLIP text embeddings achieves 10.1 AP, (AP of novel categories) on LVIS, significantly outperforming the 3.0 AP, of using GloVe. In ViLD-image, we obtain the image embeddings by feeding the object proposals into the pretrained image encoder. Then we train a Mask R-CNN whose region embeddings detected boxes are aligned with these image embeddings. In contrast to ViLD-text, ViLD-image distills knowledge from both base and novel categories since the proposal network may detect regions containing novel objects, while ViLD-text only learns from base categories. Distillation enables ViLD to be general in choosing teacher and student architectures. ViLD is also energy-effciient as works with off-the-shelf open-vocabulary image classifiers. We experiment with the CLIP and ALIGN (Jia et al. 202T) teacher models with different architectures (ViT and EfficientNet).
We show that ViLD achieves 16.1 AP for novel categories on LVIS, surpassing the supervised coun- terpart by 3.8. We further use ALIGN as a stronger teacher model to push the performance to 26.3 novel AP, which is close (only 3.7 worse) to the 2020 LVIS Challenge winner (Tan al. 2020) that is fully-supervised. We directly transfer ViLD trained on LVIS to other detection datasets without finetuning, and obtain strong performance of 72.2 AP50 on PASCAL VOC, 36.6 AP on COCO and 11.8 AP Objects365. We also outperform the previous state-of-the-art open- vecebuuayy detector on COCO (Zareian et al. 2021) by 4.8 novel AP and 1 11.4 overall AP.
RELATED WORK
Increasing vocabulary in visual recognition: Recognizing objects using a large vocabulary is a long-standing research problem in computer vision. One focus is zero-shot recognition, aim- ing at recognizing categories not present in the training set. Early works (Farhadi et al.) 2209) Rohrbach et al., 2011 Jayaraman Grauman, (014) use visual attributes to create binary code- book representing categories, which is used to transfer learned knowledge to unseen categories. In this direction, researchers have also explored class hierarchy, class similarity, and object parts as discriminative features to aid the knowledge transfer (Rohrbach et al.) 2011 Akata et al. 2016
Zhao et al. 2017: Elhoseiny et al.) 20177: Ji et al., 2018 Cacheux et al., 2019] Xie et al.] 2020). Another focus is learning to align latent image-text embeddings, which allows to classify images using arbitrary texts. Frome et al. (2013) and Norouzi et al. (2014) are pioneering works that learn
risual-semantie embedding space using deep learning. Wang et al. (2018) distills information
2

---

Published as a conference paper at ICLR 2022
Training
Cropped
Pre-tniinal image
Knowledge Mistilaaion
Backltone
KolAlign
enter
Bi
Nesr
smag cooedagg
Enooder
Nevd
Inference
Br
NL
Tackshute
RPN
RolAlign
Figure 2: An overview of using ViLD for open-vocabulary object detection. ViLD distills the knowledge from pretrained open-vocabulary image classification model. First, the category text embeddings and the im- age embeddings of cropped object proposals are computed, using the text and image encoders in the pretrained classification model. Then, ViL.D employs the embeddings the region classifier (ViI.Dttext) and mini- mizes the distance between the region embedding and the image embedding for each proposal (ViLD-image). During inference, text embeddings of novel categories are used to enable open- vocabullyy detection.
from both word embeddings and knowledge graphs. Recent work CLIP (Radford et al. 2021 and ALIGN (Jia al. 2021) push the limit by collecting million-scale image-fext pairs and then training joint image-text models using contrastive learning. These models can directly transfer to a suite of classification datasets and achieve impressive performances. While these work focus on image-level upen-vocabulary recognition, we focus on detecting objects using arbitrary text inputs.
Increasing vocabulary in object detection: It's expensive to scale up the data collection for large vocabulary object detection. Zhao al. (2020) and Zhou et al. (202I) unify the label space from multiple datasets. Joseph et al. (2021) incrementally learn identified unknown categories. Zero-shot detection (ZSI) offers another direction. Most ZSD methods align region features to pretrained text embeddings in base categories (Bansal et al., 2018 Demirel et al. 2018 Rahman et al. 2019 Hayat al. 2020: Zheng al., 2020). However, there is large performance gap supervised counterparts. To address this issue, Zareian et al. (202T) pretrain the backbone model using image captions and finetune the pretrained model with detection datasets. contrast, we use an image-text pretrained model as teacher model to supervise student object detectors. All previous methods are only evaluated on tens of categories, while we are first to evaluate on more than 1,000 categories.
METHOD
Notations: We divide categories in detection dataset into the base and novel subsets, and denote them by CB and CN. Only annotations CB are used for training. We use T(-) to denote the text encoder and V--) denote the image encoder in the pretrained open-vocabulary image classifier.
3.1 LOCALIZATION FOR NOVEL CATEGORIES
The first challenge for open- vocabulary detection is to localize novel objects. We modify a standard two-stage object detector, e.g., Mask R-CNN (He et al. (017). for this purpose. We replace its class- specific localization modules, i.e., the second-stage bounding box regression and mask prediction layers, with glass-agnosse modules for general object proposals. For each region of interest, these modules only predict single bounding box and single mask for all categories, instead of one prediction per category. The class-agnostic modules can generalize to novel objects.
3.2 OPEN-VOCABULARE DETECTION WITH CROPPED REGIONS
Once object candidates are localized, we propose to reuse a pretrained open- vece-vocabulary image clas- sifier to classify each region for detection.
3

---

Published as a conference paper at ICLR 2022
Crosa
Loss
Cross entopy
Josa
gourd Desuedingg
Arovledge Custittion
ground Orosidinggs
Krovledge
Cross wrirpyy fooe
Splrtin
Fiage EnGoC
lrage Groodi
Projetion
Laper
Heat
Heal
Hadt
projosal
preoorppted
propoaads
pprlosas
preoorppaed
propcab
(4) Vanilla detector
(N) Vil Diirot
(c) MiLDDmmmgg
(d) VILD
Figure 3: Model architecture and training objectives. (a) The classification head vanilla two-stage detector, e,g., Mask R-CNN. (b) ViLD-texi replaces the classifier with fixed text embeddings and a learnable background embedding. The projection layer is introduced to adjust the dimension of region embeddings to be compatible with the text embeddings.
(c) ViLD-image distills from the precomputed image embeddings of proposals with La loss. (d) ViLD combines ViLD-text and ViLD-image.
Image embeddings: We train a proposal network base categories CB and extract the region proposals P offline. We crop and resize the proposals, and feed them into the pretrained image encoder V to compute image embeddings V(crop(I, f)), where is the image.
We ensemble the image embeddings from 1x and 1.5x crops, as the 1.5x crop provides more context cues. The ensembled embedding is then renormalized to unit norm:
FIcrop(I,
where v= (ecrop(I,fxx)) + ((crop(I,f1.xx)) (1)
|v|
Text embeddings: We generate the text embeddings offline by feeding the category texts with prompt templates, e.g., "a photo of (category) in the scene", into the text encoder T. We ensemble multiple prompt templates and the synonyms if provided.
Then, we compute cosine similarities between the image and text embeddings. A softmax activation is applied, followed by per-class NMS to obtain final detections. The inference is slow since every cropped region fed into V.
3.3 VILD: VISION AND LANGUAGE KNOWLEDGE DISTILLATION.
We propose ViLD to address the slow inference speed of the above method. ViLD learns region embeddings in a two-stage detector to represent each proposal r. We denote region embeddings by R(o(T),r), r). where d(.) is a backbone model and R(-) is lightweight head that generates region embeddings. Specifically, we take outputs before the classification layer region embeddings.
Replacing classifier with text embeddings: We first introduce ViLD-text. Our goal is to train the region embeddings such that they can be classified by text embeddings. Fig. 3 shows the archi- tecture and training objective. ViLD-text replaces the learnable classifier in Fig. Bla) with the text embeddings introduced in Sec. S.2. Only T(CB), the text embeddings of CB. are used for training.
For the proposals that do not match any groundtruth in Cs. they are assigned to the background category. Since the text "background" does not well represent these unmatched proposals, we allow the background category to learn its own embedding ebg. We compute the cosine similarity between each region embedding R()(1).) r) and all category embeddings, including T(CB) and ebg- Then apply softmax activation with temperature to compute the cross entropy loss. To train the first-stage region proposal network of the two-stage detector, we extract region proposals r E P online, and train the detector with ViLD-text from scratch. The loss for ViLD-text can be written
C, R()(T),r)
z(r) [sim(er, @by), sim(er, t1), sim(er,
Les E (softmaz(z(()/11
LVILDMext
(2)
N
PEP
4

---

Published as a conference paper at ICLR 2022
where sim((a, (a, b) = b//[[aal[bb]]]] t; denotes elements in T(CB), yr denotes the class label of region r, N is the number of proposals per image (PP), and LC is the cross entropy loss.
During inference, include novel categories (CN) and generate T(CB U CN) (sometimes T(ON) only) for open-vocabulary detection (Fig. 2). Our hope is that the model learned from annotations in C3 generalize novel categories CN-
Distilling image embeddings: We then introduce ViLD-image, which aims to distill the knowl- edge from the teacher image encoder V into the student detector. Specifically, we align region embeddings (()(I), P) to image embeddings V(crop(I, F)) introduced in Sec. 3.2
To make the training more efficient, we extract M proposals F E P offine for each training image, and precompute the M image embeddings. These proposals can contain objects in both C3 and CN, as the network can generalize. In contrast, ViLD-text can only learn from CB. We apply Ci loss between the region and image embeddings to minimize their distance. The ensembled image embeddings in Sec. 3.2 are used for distillation:
LVLLD-mmage
Σ PEP
[V(crop(4. - R()(1)
(3)
M
Fig. Be) shows the architecture. Zhu al. (2019) use similar approach make Faster R-CNN features mimic R-CNN features, however, the details and goals are different: They reduce redundant
context to improve supervised detection; while ViLD-image is to enable open-vocabulary detection on novel categories.
The total training loss of ViLD is simply weighted sum of both objectives:
CVLD LViLDateu + LViLD imi.Dimmage
where is hyperparameter weight for distilling the image embeddings. Fig. 3 d) shows the model architecture and training objectives. Vil.D-image distillation only happens training time. Dur- ing inference, ViLD-image, ViL.-textt and ViL.D employ the same set of text embeddings as the detection classifier, and use the same architecture for open-vocabulary detection (Fig.
3.4 MODEL ENSEMBLING
In this section, we explore model ensembling for the best detection performance over base and novel categories. First, we combine the predictions of ViLD-text detector with the open-vocabulary im- age classification model. The intuition is that ViLD-image learns to approximate the predictions of its teacher model, and therefore, we assume using the teacher model directly may improve perfor- mance. We use a trained ViLD-text detector to obtain top h candidate regions and their confidence scores. Let Pt, Dileeet denote the confidence score of proposal F belonging to category i. We then feed (I) to the open- vecabbaayyy classification model to obtain the teacher's 's confidence score Pi.cls- Since we know the two models have different performance on base and novel categories, we introduce weighted geometric average for the ensemble:
(1-A) if E CB PA Ien Pi,cls Pi Pr i CN
Pi,ensemhle
is set to 2/3, which weighs the prediction of ViLD-text more on base categories and vice versa. Note this approach has similar slow inference speed as the method in Sec. 3.2
Next, we introduce different ensembling approach to mitigate the above inference speed issue. Besides, ViLD, the cross entropy loss of ViLD-text and the Ci distillation loss of ViLD-image is applied to the same set of region embeddings, which may cause contentions, Here, instead, we leamn two sets of embeddings for ViLD-text (Eq. 2 and ViLD-image (Eq. 3) respectively, with two separate heads of identical architectures. Text embeddings embeddings to obtain confidence scores AilD and ii.ggee-immge- which are then ensembled the same way Eq. with ViLD image replacing Pi,es. We name this approach ViL.D-nsemble.
applied to these two regions
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
Implementation details: We benchmark on the Mask R-CNN (He et al., 2017) with ResNet (He
et al. (2066) FPN (Lin et al.; 2017) backbone and use the same settings for all models unless explic- itly specified. The models use x024x 1024 as input image size, large-scale jittering augmentation of range [0.1, 2.0]. synchronized batch normalization (Toffe & Szegedy 2015 Girshick et al. 2018) of batch size 256, weight decay of 4e-5, and an initial learning rate of 0.32. We train the model from scratch for 180,000 iterations, and divide the learning rate by 10 at 0.9 0.95x, and 0.975 of total iterations. We use the publicly available pretrained CLIP mode as the open-vocabulary classifi- cation model, with an input size of x24x224. The temperature number of detections per image is 300. We refer the readers to Appendix D for more details.
set to 0.01, and the maximum
4.1 BENCHMARK SETTINGS
We mainly evaluate on LVIS (Gupta et al. 2019) with our new setting. To compare with previ- ous methods, also use the setting in Zareian et al. (202T), which is adopted in many zero-shot detection works.
LVIS:
We benchmark on LVIS vl. LVIS contains large and diverse set of vocabulary (1,203 cat- egories) that is more suitable for open-vocabulary detection. We take its 866 frequent and common categories as the base categories CB, and hold out the 337 rare categories as the novel categories CN. AP,, the AP of rare categories, is the main metric.
COCO: Bansal et al. (2018) divide COCO-2017 (Lin et al. 2014) into 48 base categories and 17 novel categories, removing 15 categories without a synset in the WordNet hicrarchy. We follow previous works and do not compute instance masks. We evaluate the generalized setting.
4.2 LEARNING GENERALIZABLE OBJECT PROPOSALS
We first study whether detector can localize novel categories when only trained on base categories. We evaluate the region proposal networks in Mask R-CNN with a ResNet-50 backbone. Table shows the average recall (AR) (Lin et al., 2014) on novel categories. Training with only base cat- egories performs slightly worse by 2 AR 100, 300, and 1000 proposals, compared to using both base and novel categories. This experiment demonstrates that, without seeing novel categories during training, region proposal networks can generalize to novel categories, only suffering a small performance drop. We believe better proposal networks focusing on unseen category generalization should further improve the performance, and leave this for future research.
Table 1: Training with only base categories achieves comparable average recall (AR) for novel categories on LVIS. We compare RPN trained with base only base+novel categories and report the bounding box AR.
Supervision
AR, 1 100
A300
AR, 1 (1000
base
39.3
48.3
55,6
base novel
41.1
50.9
57.0
4.3 OPEN-VOCABULARY CLASSIFIER ON CROPPED REGIONS
In Table 2 we evaluate the approach in Sec. 3.2 i.e., using an open-vocabulary classifier to classify cropped region proposals. We use CLIP in this experiment and find it tends to output confidence scores regardless of the localization quality (Appendix B). Given that, we ensemble the CLIP con- fidence score with a proposal objectness score by geometric mean. Results show it improves both base and novel APs. We compare with supervised baselines trained on base/base+novel categories, as well as Supervised-RFSs (Mahajan et al.] 20118) Gupta et al. 2019) that uses category frequency for balanced sampling. CLIP cropped regions already outperforms supervised basclines AP, by a large margin, without accessing detection annotations in novel categorics. However, the per- formances of AP, and AP, are still trailing behind. This experiment shows that a strong open- vocabulary classification model can be a powerful teacher model for detecting novel objects, yet there is still much improvement space for inference speed and overall AP.
https: thub. wwt.:ttww....mon......e www.openaiiooiooooooooooooo..a.a /CI.PP /CLIP ViT-B/32.

---

Published as a conference paper at ICLR 2022
Table 2: Using CLIP for open-vocabulary detection achieves high detection performance novel cate- gories. apply CLIP to classify cropped region proposals, with without ensembling objectness scores, and report the mask average precision (AP). The performance on novel categories (AP,) is far beyond supervised learning approaches. However, the overall performance is still behind.
Method
AP,
APc
AP,
AP
Supervised (base class only)
0.0
22,6
32,4
22.5
CLIP on cropped regions w/o objectness
13.0
10.6
6.0
9.2
CLIP on cropped regions
18.9
18.8
16.0
17.7
Supervised (base+novel)
4.1
23.5
33.2
23,9
Supervised RFS (base+novel)
12.3
24.3
32.4
23.4
Table 3: Performance of ViLD and its variants. ViLD outperforms the supervised counterpart novel categories. Using ALIGN the teacher model achieves the best performance without bells and whistles. All results are mask AP. We average over rums for R50 experiments. T; methods with R-CNN style; runtime 630x of Mask R-CNN style. +; for reference, fully-supervised learning with additional tricks.
Backbone
Method
AP.
APc
AP,
AP
CLIP on cropped regions
18.9
18.8
16.0
17.7
ResNer-50+ViT-B/22
ViLD-Iext+CLIPP
22.6
24.8
29.2
26.1
Supervised-Rit RFS (basc+novel)
12.3
24.3
32,4
25.4
GloVe baseline
3.0
20.1
30.4
21.2
ViLD-text
10.1
23.9
32.5
24.9
ResNet-50
ViLD-imaae
11.2
11.3
11.1
11.2
ViLD (w=0.5)
16.1
20.0
28.3
22.5
ViLD-ensemble (w=0.5)
16.6
24.6
30.3
25.5
ViLD-ensemble w/ ViT-L/14 (w=1.0)
21.7
29.1
33.6
29.6
EtficientNet-b7
ViL.D-ensemlee ALIGN (w=1.0)
26.3
27.2
32,9
29.3
ResNo68169+HHCC
2020 Challenge winner Tan al. 2020
30.0
41.9
46.0
41.5
4.4 VISION AND LANGUAGE KNOWLEDGE DISTILLATION
We evaluate the performance of ViLD and its variants (ViLD-text, ViLD-image, and ViLD- ensemble), which are significantly faster compared to the method in Sec. 4.3 Finally, stronger teacher models to demonstrate our best performance. Table summarizes the results.
Text embeddings as classifiers (ViLD-text): We evaluate ViLD-text using text embeddings gen- erated by CLIP, and compare with GloVe text embeddings (Pennington al. 2014) pretrained on a large-scale text-only corpus. Table S shows ViLD-text achieves 10.1 AP,, which is significantly better than 3.0 AP, using GloVe. This demonstrates the importance of using text embeddings that are jointly trained with images. ViLD-text achieves much higher AP,. and AP, compared to CLIP on cropped regions (Sec. 4.3), because ViLD-text uses annotations in Cn to align region embeddings with text embeddings. The AP, is worse, showing that using only 866 base categories in LVIS does not generalize as well as CLIP to novel categories.
Distilling image embeddings (ViLD-image): We evaluate ViLD-image, which distills from the image embeddings of cropped region proposals, inferred by CLIP's image encoder, with distilla- tion weight of 1.0. Experiments show that ensembling with objectness scores doesn't 't help with other only apply it ViLD-image. Without training with any object category labels, achieves 11.2 AP, and 11 11.2 overall AP. This demonstrates that visual distillation works for open-vocabulary detection but the performance not as good CLIP on cropped regions.
ViLD variants,
Text+visual embeddings (ViLD): ViLD shows the benefits of combining distillation loss (ViLD- image) with classification loss using text embeddings (ViLDDtextt). We explore different hyperpa- rameter settings Appendix Table and observe a consistent trade-off between AP, and APc.r- which suggests there is competition between ViLD-text and ViLD-image In Table 3 compare ViLD with other methods. Its AP, is 6.0 higher than ViLD-text and 4.9 higher than ViLD-image, indicating combining the two learning objectives boosts the performance novel categories. ViLD outperforms Supervised -RFS by 3.8 AP,, showing our voca-vocyy detection approach is better than supervised models on rare categories.

---

Published as a conference paper at ICLR 2022
Table 4: Performance on COCO dataset compared with existing methods. ViLD outperforms all the other methods in the table trained with various sources by a large margin, on both novel and base categories.
Method
Training source
Novel AP
Base AP
Overall AP
Bilen & Vodaldi (0166)
19.7
19.6
19.6
image-level labels in C3 Ux
Yc ct al. 20199
20.3
20.1
20.1
Bansal et al. (2118)
0.31
29.2
24.9
Zhu ct al. 2020
instance-level labels in Cg
3,41
13.8
13.0
Rahman al.
2020
4.12
35.9
27.9
image captions in CB U CN instance-level labels in Cs
Zareian (2021
22.8
46.0
39.9
CLIP cropped regions
26.3
28.3
27.8
image- Maage-texttpe pairs from Internet
ViLD-text
5.9
61.8
47.2
(may contain Cæ CN) instance-level labels Cs
ViLD-image
24.1
34.2
31.6
ViLD (w = 0.5)
27.6
59.5
51.3
Model ensembling: We study methods discussed in Sec. 3.4 to reconcile the conftict of joint train- ing with ViLD-text and ViLD-image. We use two ensembling approaches: 1) ensembling ViLD-text with CLIP (VILD-eex(CLLI); ensembling ViLD-text and ViLD-image using separate heads ViLD-ennemble).. As shown in Table 3 ViLD-ensemble improves performance over ViLD, mainly on AP, and AP,. This shows ensembling reduces the competition. ViLD-texI+CLIP obtains much higher AP,, outperforming ViLD by 6.5, and maintains good APc.f- Note that it is slow and im- practical for real world applications. This experiment is designed for showing the potential of using open-vocabulary classification models for open-vocabulary detection.
Stronger teacher model: We use CLIP ViT-L/14 and ALIGN (Jia et al. 2021 to explore the
performance gain with stronger teacher model (details in Appendix D). As shown in Table 3 both models achieve superior results compared with R50 ViLD w/ CLIP. The detector distilled from
ALIGN is only trailing to the fully-supervised 2020 Challenge winner (Tan al. (020) by 3.7 AP,. which employs two-stage training, self-training, and multi-scale testing etc. The results demonstrate ViLD scales well with the teacher model, and is promising open-vocabulary detection approach.
4.5 PERFORMANCE COMPARISON ON COCO DATASET
Several related works in zero-shot detection and open-vocabulary detection are evaluated on COCO. To compare with them, we train and evaluate ViLD variants following the benchmark setup in Zareiin et al.] (2021) and report box AP with an IoU threshold of 0.5. We use the ResNet-50 backbone, shorten the training schedule to 45,000 iterations, and keep other settings the same as our experiments on LVIS. Table summarizes the results. ViLD outperforms Zareian et al. (2021) by 4.8 Novel AP and 13.5 Base AP. Different from Zareian et al. (202I), we do not have a pretraining phase tailored for detection. Instead, of Vil.Dttext low because only 48 base categories are available, which makes generalization to novel categories challenging. In contrast, ViLD-image and ViL.D, which can distill image features of novel categories, outperform all existing methods (not apple-to-apple comparison though, given different methods use different settings).
off-the-shelf classification model. The performance
4.6 TRANSFER TO OTHER DATASETS
Trained ViLD models can be transferred to other detection datasets, by simply switching the clas- sifier to the category text embeddings of the new datasets. For simplicity. we keep the background embedding trained on LVIS. We evaluate the transferability of ViLD on PASCAL VOC (Everingham al. 2010), COCO (Lin et al. 4).4),, and Objects365 (Shao et al., 2019). Since the three datasets have much smaller vocabularies, category overlap is unavoidable and images can be shared among datasets, e.g., COCO and LVIS. As shown Table 5 ViLD achieves better transfer performance than ViLD-text. In PASCAL and COCO, the gap is large. This improvement should be credited to visual distillation, which better aligns region embeddings with the text classifier. We also compare with supervised learning and finetuning the classification layer. Although across datasets, ViLD has 3-6 AP gaps compared to the finetuning method and larger gaps compared to the supervised method, it is the first time we can directly transfer trained detector different datasets using language.
8

---

Published as a conference paper at ICLR 2022
Table 5: Generalization ability of ViLD. We evaluate the LVIS -VIS-trained model with ResNet-50 backbone PASCAL VOC 2007 test set, COCO validation set, Objects365 vi validation set. Simply replacing text embeddings, our approaches are able to transfer to various detection datasets. The supervised baselines of COCO and 0bjects365 are trained from scratch. t: the supervised baseline of PASCAL VOC is initialized with ImageNet-pretrainct checkpoint. All results box APs.
PASCAL VOC
COCO
Objects365
Method
AP50
AP75
AP
AP50
AP
AP
AP
AP75
ViLD-text
40.5
31.6
28.8
43.4
31.4
10.4
13.8
11.1
ViLD
72.2
56.7
36.6
55.6
39.8
11.8
18.2
12.6
Finetuning
78.9
60.3
39.1
59.8
42.4
15.2
23.9
16.2
Supervised
78.5
49.0
46.5
67.6
50.9
25.6
38.6
28.0
LYV toxci
base novel
LIVIS
Transfer COCO
Transfer to Objects365
Figure 4: Qualitative results LVIS, COCO, and Objects365. First ViLD is able correctly localize and recognize objects novel categories. For clarity, we only show detected novel objects. Second row: The detected objects on basc+novel categories. The performance on base categorics is not degraded with ViLD. Last two rows: ViLD can directly transfer to COCO and Objects365 without further finetuning.
4.7 QUALITATIVE RESULTS
In Fig. visualize ViLD's detection results. It illustrates ViLD is able to detect objects of both novel and base categories, with high-quality mask predictions on novel objects, egg., it well separates banana slices from the crepes (novel category). We also show qualitative results on COCO and Objects365, and find ViLD generalizes well. We show more qualitative results, e,g,, interactive detection and systematic expansion, in Appendix A
CONCLUSION
We present ViLD, an open-vocabulary object detection method by distilling knowledge from open- vocabulary image classification models. ViLD is the first open-vocabulary detection method eval- uated on the challenging LVIS dataset. attains 16.1 AP for novel cateogires on LVIS with ResNet50 backbone, which surpasses its supervised counterpart at the same inference speed. With a stronger teacher model (ALIGN), the performance can be further improved to 26.3 novel AP. We demonstrate that the detector learned from LVIS can be directly transferred to 3 other detection datasets. We hope that the simple design and strong performance make ViLD a scalable alternative approach for detecting long-tailed categories, instead of collecting expensive detection annotations.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Our paper studies open-vocabulary object detection, a sub-field in computer vision. Our method based knowledge distillation, machine learning technique that has been extensively used in computer vision, natural language processing, etc. All of our experiments were conducted on public datasets with pretrained models that are either publicly available or introduced in published papers. The method proposed in our paper is principled method for open-vocabulay, object detection that can be used in a wide range of applications. Therefore, the ethical impact of our work would primarily depends on the specific applications. We foresee positíve impacts if our method is applied to object detection problems where the data collection is difficult to scale, such as detecting rare objects for self-driving cars. But the method can also be applied to other sensitive applications that could raise ethical concerns, such as video surveillance systems.
REPRODUCIBILITY STATEMENT
We provide detailed descriptions the proposed method Sec. Details about experiment set- tings, hyper-parameter and implementations are presented in Sec. We release our code and pretrained models at ltttps://iithuut com/ http://nssorffoe how/or/wwww//ooooooo t cree/mastte ducibility of our work.
Appendix C and Appendix
facilitate the repro-
REFERENCES
Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele. Multi-cue zero-shot learning with strong supervision. In CVPR, 2016.
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV, 2018.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. CVPR, 2016.
Yannick Le Cacheux, Herve Le Borgne, and Michel Crucianu, Modeling inter and intra-class rela- tions in the triplet loss for zero-shot learning. In ICCV, 2019.
Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinbis. Zero-shot object detection by hybrid region embedding. In BMVC, 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2020.
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head to the "beak": Zero shot learning from noisy text description at part precision. In CVPR, 2017.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, 2009,
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NewrIPS, 2013.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. CVPR, 2014.
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron. 2018
10

---

Published as a conference paper at ICLR 2022
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance seg- mentation. In CVPR, 2019.
Nasir Hayat, Munawar Hayat, Shafin Rahman, Salman Khan, Syed Waqas Zamir, and Fahad Shah- baz Khan. Synthesizing the unseen for zero-shot object detection. In ACCV, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Dinesh Jayaraman and Kristen Grauman. Zero shot recognition with unreliable attributes. NeurIPS 2014, 2014.
Zhong Ji, Yanwei Fu, Jichang Guo, Yanwei Pang, Zhongfei Mark Zhang, et al. Stacked semantics- guided attention model for fine-grained zero-shot learning. In NeurIPS, 2018.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. ICML, 2021.
KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian, Towards open world object detection. In CVPR, 2021.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Pont-Tusel Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Ducrig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. ICLR, 2014.
Jeffrey Pennington, Richard Socher, and Christopher Manning, GloVe: Global vectors for word representation. In EMNLP, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021.
Shafin Rahman, Salman Khan, and Fatih Porikli. Zero-shot object detection: Learning to simulta- neously recognize and localize novel concepts. In ACCV, 2018.
Shafin Rahman, Salman Khan, and Nick Barnes. Transductive learning for zero-shot object detec- tion. In ICCV, 2019.
Shafin Rahman, Salman Khan, and Nick Barnes, Improved visual-semanica alignment for zero-shot object detection. In AAAI, 2020.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, 2017.

---

Published as a conference paper at ICLR 2022
Shaoqing Ren, Kaiming He, Ross B Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Marcus Rohrbach, Michael Stark, and Bernt Schiele. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In CVPR, 2011.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
Jingru Tan, Gang Zhang, Hanming Deng, Changbao Wang, Lewei Lu, quanquan Li, and Jifeng Dai. Technical report: good box is not guarantee of a good mask. Joint COCO and LVIS workshop at ECCV 2020: LVIS Challenge Track, 2020.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018.
Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, and Ling Shao. Region graph embedding network for zero-shot learning. ECCV, 2020.
Keren Ye, Mingda Zhang, Adriana Kovashka, Wei Li, Danfeng Qin, and Jesse Berent. Cap2det: Learning amplify weak caption supervision for object detection. In ICCV, 2019.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
Hang Zhao, Xavier Puig. Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. ICCV, 2017.
Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with unified label space from multiple datasets. In ECCV, 2020.
Ye Zheng, Ruoran Huang, Chuanqi Han, Xi Huang, and Li Cui. Background learnable cascade for zero-shot object detection. In ACCV, 2020.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühhl. Simple multi-dataset detection. arXiv preprint drXiv:2102.13086, 2021.
Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Don't even look once: Synthesizing fea- tures for zero-shot detection. In CVPR, 2020.
Xizhou Zhu Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. CVPR, 2019.
12

---

Published as a conference paper at ICLR 2022
APPENDIX
A ADDITIONAL QUALITATIVE RESULTS
On-the-fly interactive object detection:
We tap the potential of ViLD by using arbitrary text to interactively recognize fine-grained categories and attributes. We extract the region embedding and compute its cosine similarity with a small set of on-the-fly arbitrary texts describing attributes and/or fine-grained categories; we apply softmax with temperature T on top of the similarities. To our sur- prise, though never trained on fine-grained dog breeds (Fig. 5 it correctly distinguishes husky from shiba inu. also works well on identifying object colors (Fig. D. The results demonstrate knowledge distillation from open-vocabulary image classification model helps ViLD gain un- derstanding of concepts not present in the detection training. Of course, ViLD does not work all the time, e-g., it fails recognize poses of animals.
hoad
The lird his yo hoad and Mack body
The bidd his blick seed yeluww bost
BERTIU SHO
Bird: 0.98
(a) Fine-grained breeds and colors.
(b) Colors body parts.
Figure 5: On-the-fly interactive object detection. One application of ViLD is using on-the-fly arbitrary texts to further recognize more details of the detected objects, e.g., fine sinennrained categories and color attributes.
Systematic expansion of dataset vocabulary: In addition, we propose to systematically expand the dataset vocabulary
{U1..., op)) with of attributes (a {ai,..., }g3) follows: Pr(u; 0z e,) = Pr(m: e,)) Pr(ay Vi, e,)
Pr(e: e,) Pr(a. ee,,
where e,- denotes the region embedding. We assume belongs category e; is conditionally independent the event it has attribute aj-
e,, iee., given e, the event the object
Let denote the temperature used for softmax and T denote the text encoder as in Eq. Then
Pr(v: ee)) softmaz,(sim(e,, T(v))/T)
Pr(a, e,)) = softmaz,(simmee.. er. T(a))/T).
(8)
In this way, we are able to expand p vocabularies into a new set of p vocabularies with attributes.
The conditional probability approach is similar to YOLO9000 (Redmon & Farhadi (017). We show qualitative example of this approach in Fig. 6 where we use color attribute set as a. Our open-vocabulary detector successfully detects fruits with color attributes.
We further expand the detection vocabulary to fine-grained bird categories by using all 200 species from CUB-200-2011 (Wah et al. 201T). Fig. ( shows successful and failure examples of our open- vocabulary fin-grained detection CUB-200 CUB-200-201 200-2011 images. In general, model is able to detect visually distinctive species, but fails at other ones.
Transfer to PASCAL VOC: In Fig. 8 we show qualitative results of transferring an open- vocabulary detector trained on LVIS (Gupta et al. 2019 PASCAL VOC Detection (2007 test set) (Everingham et al. 2010), without finetuning (Sec. 4.6 in the main paper). Results demonstrate that the transferring works well.
Failure cases: In Fig. 9 we show two failure cases of ViLD. The most common failure cases are the missed detection. A less common mistake is misclassifying the object category.
We show a failure case mask prediction on PASCAL VOC in Fig. I0 seems that the mask prediction is sometimes based low-level appearance rather than semantics.
13

---

Published as a conference paper at ICLR 2022
(a) Using LVIS vocabulary.
(b) After expanding vocabulary with color attributes.
Figure 6: Systematic expansion of dataset vocabulary with colors. add color attributes orange, dark orange, light orange, yellow, green, cyan, blue, purple black, brown, white) to LVIS categories, which expand the vocabulary size by 1lx. Above we show an example of detection results. Our open-vocabulary detector is ahle to assign the correct color to each fruit. A class-agnostic NMS with threshold 0.9 is applied. Each figure shows top 15 predictions.
Painted Burling BId
Fallo breasied Chal and
Successful
Failure
Figure 7: Systematic expansion of dataset vocabulary with fine-grained categories. We the systematic expansion method to detect 200 fine-grained species CUB-200-2011. (a): Our open-vocabulary detector able to perform fine-grained detection (bottom) using the detector trained on LVIS (top). (b): fails at recognizing visually non-distinctive species. It incorrectly assigns "Western Gull" to "Horned Puffin" due to visual similarity.
B ANALYSIS OF CLIP ON CROPPED REGIONS
this section, we analyze some common failure cases of CLIP on cropped regions and discuss possible ways to mitigate these problems.
Visual similarity: This confusion is common for any classifiers and detectors, especially large vocabularies. In Fig. aT),,, we show two failure examples due to visual similarity. Since we only relatively small ViT-B/32 CLIP model, potentially nigher-capacios pretrained model. In Table when replacing this CLIP model with an EfficientNet- 12 ALIGN model,
improve the performance with
increase AP.
14

---

Published as a conference paper at ICLR 2022
27bette E1
Shapp
sheeps
bottle
Dog
Figure 8: Transfer to PASCAL VOC. ViLD correctly detects objects when transferred PASCAL VOC, where images usually have lower resolution than LVIS (our training set). In the third picture, our detector is able to find tiny bottles, though it fails to detect the person.
bottie
P .56
plastic bag .70
bottlee
Wafto HDA
salculator
spoon:
knife: .3
knffe
tonk.
(a) Missed
(b) Misclassified
Figure 9: Failure cases on LVIS novel categories. The red bounding boxes indicate the groundtruths of the failed detections. (a) A common failure type where the novel objects are missing, e.g., the clevator car is not detected. (b) A less common failure where (part of) the novel objects are misclassified, e.g., half of the waffle iron is detected as a calculator due to visual similarity.
Aspect ratio: This issue is introduced by the pre- pre-processing of inputs in CLIP. We use the ViT- B/32 CLIP with fixed input resolution of 224 224. It resizes the shorter edge of the image to 224, and then uses a center crop. However, since region proposals can have more extreme aspect ratios than the training images for CLIP, and some proposals tiny, directly resize the proposals to that resolution, which might cause some issues. For example, the thin structure in Fig. (b) right will be highly distorted with the pre-processing. And the oven and fridge can be confusing with the distorted aspect ratio. There might be some simple remedies for this, e.g., pasting the cropped region with original aspect ratio on black background. We tried this simple approach with both CLIP and ALIGN. Preliminary results show that it works well on the fully convolutional ALIGN, while doesn't work well on the transformer-base CLIP, probably because CLIP is never trained with black image patches.
Multiple objects in bounding box: Multiple objects in a region interfere CLIP's classification results, see Fig. [c), where corner of an aquarium dominates the prediction. This is due to CLIP pretraining. which pairs an entire image with its caption. The caption is usually about salient objects in the image. It's hard to mitigate this issue at the open-vocabulary classification model's end. On the other hand, a supervised detector are trained to recognize the object tightly surrounded by the bounding box. So when distilling knowledge from an open-vocabulary image classification model, keeping training a supervised detector on base categories could help, as can be seen from the improvement of ViLD over ViLD-image (Sec.
Confidence scores predicted by CLIP do not reflect the localization quality: For example, in Fig. 12 a). CLIP correctly classifies the object, but gives highest scores to partial detection boxes. CLIP is not trained to measure the quality of bounding boxes. Nonetheless, in object detection, is important for the higher-quality boxes to have higher scores. In Fig. [2(c), simply re-score by taking the geometric mean of the CLIP confidence score and the objectness score from the proposal model, which yields much better top predictions. In Fig. (2), show top predictions of the Mask
15

---

Published as a conference paper at ICLR 2022
cat: 84
An example of ViLD on PASCAL VOC showing mask of poor quality. The agnsssi mask prediction head occasionally predicts masks based on low-level appearance rather than semantics, and thus fails to obtain complete instance mask.
Figure 10:
Table 6: ALIGN on cropped regions achieves superior AP,, and overall very good performance. It shows stronger vech-vocabulary classification model can improve detection performance by large margin. We report box APs here.
Method
AP,
AP.
AP,
AP
CLIP cropped regions
19.5
19.7
17.0
18.6
ALIGN on cropped regions
39.6
32.6
26.3
31.4
R-CNN model. Its top predictions have good bounding boxes, while the predicted categories are wrong. This experiment shows that it's important to have both an open-vocabulary classification model for better recognition, well supervision from detection dataset for better localization.
C ADDITIONAL QUANTITATIVE RESULTS
Hyperparameter sweep for visual distillation: Table 7 shows the parameter sweep of different distillation weights using L and C2 losses. Compared with no distillation, additionally learning from image embeddings generally yields better performance on novel categories. We find Li loss can better improve the AP, performance with the trade-off against AP. and AP,. This suggests there is a competition between ViLD-text and ViLD-image.
Table 7: Hyperparameter sweep for visual distillation in VILD. L, loss is better than C2 loss. For Li loss, there is trend that AP, increases the weight increases. while AP,, decrease. For all parameter combi- nations, ViLD outperforms ViLD-text AP,. We use ResNet-50 backbone and shorter training iterations (84,375 iters), and report mask AP this table.
Distill loss
Distill weight w
AP,
APc
AP,
No distill
0.0
10.4
22.9
31.3
24.0
0.5
13.7
21.7
31.2
24.0
La loss
1.0
12.4
22.7
31.4
24.3
2.0
13.4
22.0
30,9
24.0
0.05
12.9
31.7
24.4
0.1
14.0
20.9
31.2
23.8
L1 loss
0.5
16.3
19.2
27.3
21.9
1.0
17.3
18.2
25.1
20.7
Box APs and ResNet-152 backbone: Table 8 shows the corresponding box AP of Table 3 in the main paper. In general, box AP is slightly higher than mask AP. In addition, we include the results of ViLD variants with the ResNet-152 backbone. The deeper backbone improves all metrics. The trend/relative performance is consistent for box and mask APs, as well as for different backbones. ViL.-ensemble achieves the best box and mask AP,
Ablation study prompt engineering: We conduct ablation study prompt engineering. We compare the text embeddings ensembled over synonyms and 63 prompt templates (listed in Appendix with non-ensembled version: Using the single prompt template "a photo of (article)
16

---

Published as a conference paper at ICLR 2022
watering 0.1887
music stool 0.6010
(a) Visual similarity
refrigerator 0.1004
stove 0.3151
(b) Aspect
dining table
0.2411
0.2373
(c) Other objects bounding
Figure 11: Typical errors CLIP on cropped regions. (a): The prediction and the groundtruth have high visual similarity. (b): Directly resizing the cropped regions changes the aspect ratios, which may cause troubles. (c): CLIP's predictions are sometimes affected by other objects appearing in the region, rather than predicting what the entire bounding box is.
grizzly 0.8385
grizzly 0.8234
grizzly 0.8173
grizzly 0.8140
grizzly 0.8059
grizzly 0.8048
(a) CLIP
cropped region
bear
bear 0.5957
bear 0.3827
bear 0.1310
crumb 0.1159
collar 0.0986
(b) Mask R-CNN CLIP)
grizzly 0. 7.20
bear 0.5856
teddy bear 0.1715 bambon 0.1215
bamboo 0.1082
mascot 0.0941
(e) CLIP multiplies objectness score
Figure 12: The prediction scores CLIP not reflect the quality bounding box localization. (a): Top predictions of CLIP on cropped region. Boxes of poor qualities receive high scores, though the classification is correct. (b): Top predictions of vanilla Mask R-CNN model. Box qualities good while the classification is wrong. (c): We take the geometric mean of CLIP classification score and objectiveness score, and use well the correct category rank first.
rescore (a), this way, a high-quality
(category)". Table illustatssstrttes that ensembling multiple prompts slightly improves the performance by 0.4 AP,.
17

---

Published as a conference paper at ICLR 2022
Table 8: Performance of ViLD variants. This table shows additional box APs for models in Table and ResNet-152 results.
Box
Mask
Backbone
Method
AP.
AP.
AP,
AP
AP,
AP.
APG
AP
ResNet-50
CLIP on cropped regions
19.5
19.7
17.0
18.6
18.9
18.8
16.0
17.7
+ViT-B/32
ViLD-text+CIIP
23.8
26.7
32.8
28.6
22.6
24.8
29.2
26.1
Supervised-RF5 (base+novel)
13.0
26.7
37.4
28.5
12.3
24.3
32.4
25.4
GloVe baseline
3.2
22.0
34.9
23.8
3.0
20.1
30.4
21.2
ViLD-text
10.6
26.1
37.4
27.9
10.1
23.9
32.5
24.9
ResNet-50
ViLD-image
10.3
11.5
11.1
11.2
11.2
11.3
11.1
11.2
ViLD (w=0.5))
16.3
21.2
31.6
24.4
16.1
20.0
28.3
22.5
ViLD -eDeeememble (w=0.5)
16.7
26.5
34.2
27.8
16.6
24.6
30,3
25.5
Supervised-RFF R-Fs (base+novel)
16.2
29.6
39.7
31.2
14.4
26.8
34.2
27.6
ViLD-text
12.3
28.3
39.7
30.0
11.7
25.8
34.4
26.7
ResNet-
ViLD-image
12.5
13.9
13.4
13.4
13.1
13.4
13.0
13.2
ViLD (w=1.0)
19.1
22.4
31.5
25.4
18.7
21.1
28.4
23.6
ViLD Deneeemmblle (w=2..)
19.8
27.1
34.5
28.7
18.7
24.9
30.6
26.0
EfficientNet-b7
D-ensemble w/ ViT-L/14 (w=1.0)
22.0
31.5
38.0
32.4
21.7
29.1
33.6
29.6
Vil-Dennemble w/ ALIGN (w=1.0)
27.0
29.4
36.5
31.8
26.3
27.2
32.9
29.3
Table 9: Ablation study on prompt engineering. Results indicate ensembling multinle prompt templates slightly improves AP.. ViLD w/ multiple prompts the same ViLD model Table and ViLD w/ single prompt only changes the text embeddings used as the classifier.
Method
AP,
APc
AP,
AP
ViLD single prompt
15.7
19.7
28.9
22.6
ViLD w/ multiple prompts
16.1
20.0
28.3
22.5
D MORE IMPLEMENTATION DETAILS
we show the detailed architecture and learning objectives for ViLD-ensemble, the ensembling technique introduced in Sec. 3.4
WiLD-ensemble architecture: In Fig. 13
Model used for qualitative results: For all qualitative results, we use a ViLD model with ResNet- 152 backbone, whose performance is shown Table S
Details for supervised baselines: For fair comparison, train the second stage box/mask pre- diction heads of Supervised and Supervised-RFS baselines in the class-agnostic manner introduced Sec.
Details for R-CNN style experiments: We provide more details here for the R-CNN style experi-
ments: CLIP on cropped regions Sec. 4.2 and ViI.D-tet+CILIP Sec. 4.3 1) Generalized object proposal: We the standard Mask R-CNN R50-FPN model. To report mask AP and compare with other methods, we treat the second-stage refined boxes as proposals and use the corresponding masks. We apply class-agnostic NMS with 0.9 threshold, and output maximum of 1000 pro- posals. The objectness score is one minus the background score. 2) Open-vocabulary classification cropped regions: After obtaining CLIP confidence scores for the 1000 proposals, apply class-specific NMS with threshold of 0.6, and output the top 300 detections as the final results.
Additional details for ViLD variants: Different from the R-CNN style experiments, for all ViLD variants (Sec. 3.3 Sec. 3.4 we use the standard two-stage Mask R-CNN with the class-agnostic localization modules introduced in Sec. 3.1 Both the offline proposals and online proposals obtained from the first-stage RPN (Ren et al. 2015). In general, the R-CNN style methods and
ViLD variants share the same concept of class-agnssts object proposals. We use the second-stage outputs in R-CNN style experiments only because want obtain the Mask AP, the main metric, to compare with other methods. For ViLD variants, we remove the unnecessary complexities and show that using simple one-stage RPN works well.
18

---

Published as a conference paper at ICLR 2022
Cross entropy loss
loss
kead light
pound britedking
modeias
Ningun
Miepue
HI X.xeag
Norrakzation
Pre-toiindd
Projection
Propdtan Laper
Dopping Rociiie
pre-oorruued proposals
Figure 13: Model architecture and training objectives for ViL.Densemmle. The learning objectives are similar to ViLD. Different from ViLD, we use two separate heads of identical architecture in order to reduce the competition between ViLD-text and ViLD-image objetvies. During inference, the results from the two heads ensembled described in Sec. Please refer Fig. for comparison with other ViLD variants.
Architecture for open-vocabulary image classification models: Popular open-vocabulary image classification models (Radford et al.] 202T, Jia et al., (02T) perform contrastive pre-training on large number of image-text pairs. Given batch of paired images and texts, the model learns to maximize the cosine similarity between the embeddings of the corresponding image and text pairs, while minimizing the cosine similarity between other pairs. Specifically, for CLIP (Radford et al.] (021). we use the version where the image encoder adopts the Vision Transformer (Doso- vitskiy al., 2020)) architecture and the text encoder Transformer (Vaswani et al., (017). For ALIGN (Jia et al. (2221), ),, its image encoder is an EfficientNet (Tan & Le 2019) and its text encoder BERT (Devlin et al. 2019).
Details for ViLD with stronger teacher models: In both experiments with CLIP VIT-L/14 and ALIGN, we use EfficientNet-b7 as the backbone and Vil.D-ensemble for better performance. We also crop the Rol features from only FPN level P3 in the feature pyramid. The large-scale jittering range is reduced to [0.5, 2.0]. For CLIP ViT-L/14, since its image/text embeddings have 768 dimen- sions, we increase the FC dimension of the Faster R-CNN heads 1,024, and the FPN dimension to 512. For ViLD w/ ALIGN, we use the ALIGN model with an EfficientNet-12 image encoder and a BERT-large text encoder as the teacher model. We modify several places in the Mask R- CNN architecture to better distill the knowledge from the teacher. We equip the ViLD-image head Vil.D-ensemble with the MBConvBlocks
EfficientNet. Since the MBConvBlocks are fully- convolutional, we apply global average pooling to obtain the image embeddings. following the teacher. The ViLD-text head keeps the same Faster R-CNN head architecture as in Mask R-CNN. Since ALIGN image/text embeddings have 1,376 dimensions (2.7x CLIP embedding dimension), we increase the number of units the fully connected layers of the ViLD-text head to 2,048, and the FPN dimension to 1,024.
Text prompts: Since the open-vocabulary classification model is trained full sentences, feed the category names into a prompt template first, and use an ensemble of various prompts. Following Radford et al. (202I), we curate list of 63 prompt templates. We specially include several prompts containing the phrase "in the scene" to better suit object detection, e.g., "There is {article) (category) in the scene".
Our list of prompt templates is shown below:
There is (article) (category) in the scene.'
' "There is the (category) in the scene.'
"a photo of (article) (category) in the scene.'
r photo of the (category) in the scene.'
aa photo of one (category) in the scene.'
/ rttpp of (article) (category)."
19

---

Published as a conference paper at ICLR 2022
x "itap of my (category).'
'itap of the (category).'
ra photo of (article) (
'a photo of my (category).
'a photo of the (category).
r photo of one {category)."
'a photo
many (category)."
a good photo of (article) (category).'
a good photo of the (category).
a bad photo of (article) (category),'
a bad photo of the (category).
a photo of nice (category).'
a photo of the nice (category).
r photo of cool (category)."
aa photo of the cool (category).
a photo of weird (category)
a photo of the weird (category).'
'a photo of small (category).
r photo of the small (
photo
large (categozy).
r photo of the large (
'a photo a clean (category).
'a photo of the clean (category)
a photo of a dirty (category).'
aa photo of the dirty (category).'
a bright photo of (article) (category).'
'a bright photo of the (category).'
r dark photo of (article) (category).'
dark photo of the (category).
a photo of a hard to see (category)
r photo the hard to see (category)."
a low resolution photo of (article) (category).'
'a low resolution photo
the (category)."
a cropped photo of (article) (category).'
'a cropped photo of the (category).
'a close-up photo of (article) (category)."
a close-up photo of the (category)."
a jpeg corrupted photo of (article) (category).'
'a jpeg corrupted photo of the (category).
'a blurry photo of (article) (category)."
' blurry photo of the (category).'
'a pixelated photo of (article) (category)."
'a pixelated photo of the (category)."
'a black and white photo the (category).'
r black and white photo of (article) {category).'
a plastic (category).'
r the plastic (category).'
l toy (category)."
"the toy (category).'
a plushie (category).'
r the plushie (category).'
a cartoon (category)."
"the cartoon [category).'
an embroidered (category).
"he embroidered (category).
a painting of the (category).'
painting of a (category).
20


Reviews:
[{"review_id": "n_8bRv-yYXj", "reviewer": "Reviewer_6g5Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents strong results for some cases like rare classes in LVIS dataset. I would like to comment the authors on a well-written paper. It is mostly easy to understand. The authors have provided clear motivations and intuitions behind most of their design decisions. The diagrams and figures also assist in understanding the paper better. \n\nHowever, there are still several problems with the paper which need to resolved before it can be considered ready for acceptance. The performance on non-rare classes and the complete test sets is significantly lower than prior works even though the performance on rare classes is slightly better in many cases. This raises questions about the utility of the proposed approach. Further, using models like CLIP which have probably already seen the rare classes muddies the definition of a \"zero-shot\" setting as used in the paper. \n\nIn particular, I have the following concerns/comments/questions/suggestions about the current version of the paper. \n\n1. Do the authors know what classes the pre-trained image encoder (CLIP) is trained on? Has it seen any of the novel categories? If yes, then I would hesitate in calling it a zero-shot setting. I think the authors should just call it an open-vocabulary setting which can be weaker than a real zero-shot framework. Further, when comparing against prior zero-shot works, the authors need to clearly specify that the settings are slightly different because many of the prior works have ensured that no visual data from unseen classes has been used during pre-training or training. \n\n2. It's not clear to be why we need N and M proposals separately? This is what I understand currently: The N proposals come from the RPN being trained along with the other trainable weights and might change from one epoch to the next. On the other hand, the M proposals are obtained from the RPN at the start of training and are then fixed i.e. they do not change throughout training. Is this correct? If yes, why not just use the N proposals in both cases? In section 3.4, the authors mention that using the same proposals causes contentions. I am not sure how or why. Further, if my interpretation mentioned above is incorrect, then how are the M \"pre-computed proposals\" obtained during training? And how are they obtained during inference? Similarly, for N. \n\n3. In section 3.2, the authors mention that they ensemble 1x and 1.5x embeddings. Are there any emperical studies to show this helps? What if we just used 1x or 1.5x? What if we used 1x, 1.5x, and 2x? What if we used 0.5x and 1x? etc. \n\n4. In section 3.4, the authors mention that \"ViLD-image learns to approximate the predictions.....may improve performance\". This seems very odd to me. If ViLD-image already approximates the predictions of the teacher model, why do we need to add the teacher model directly?\n\n5. I am not sure I understand how ViLD-ensemble is different from ViLD. Is the diagram shown in Figure 3d ViLD or ViLD-ensemble? Can the authors please list all of the differences between ViLD and ViLD-ensemble?\n\n6. In the caption of Table 3, the authors mention that the SOTA model uses additional tricks. What additional tricks? Are they not applicable to ViLD? What is the performance of ViLD using those tricks? The performance on all r, c, f, and overall is very low compared to SOTA. Why is this? And what is the utility of the proposed approach if this is the case?\n Further, I think not making the SOTA performance bold in Table 3 might be disingenuous. Some reasons I can think for not doing this are 1. if the SOTA performance achieved in a supervised setting, 2. the SOTA model uses the rare classes during training, 3. the SOTA model uses different/larger data for training. Are any of these true? If not, the authors should highlight the SOTA performance and discuss why the proposed approach does not meet SOTA and what can be done to achieve similar/better performance. \n\n7. For table 4, what is the performance of supervised methods for COCO novel classes? The authors can do a similar comparison as done for LVIS. Further, the authors should clarify that the CLIP model in ViLD-image and the ViLD-text models have seen instances of $C_N$ during the large-scale pre-training which makes it a weaker zero-shot setting than prior works. \n\n\n\nTypo:\nOn page 6, in the 5th line from the bottom, there is an extra \"has\". ", "summary_of_the_paper": "This paper aims to propose a novel approach for open-vocabulary or zero-shot object detection by leveraging models pre-trained on internet-scale data which might include many of the desired open-vocabulary categories. The proposed approach, called ViLD, uses the information learned by models like CLIP, and ALIGN to detect \"novel\" classes. The paper shows that the proposed approach can achieve better performance than supervised methods on rare classes in the LVIS dataset. However, the performance is much lower than state-of-the-art. Also, the performance on frequent classes in LVIS, and the overall performance is significantly lower than prior works. The authors also show that the proposed approach can achieve good performance improvements on the zero-shot COCO dataset - both for \"unseen\" and seen classes.  ", "summary_of_the_review": "The paper proposes an interesting direction for open-vocabulary open detection. The authors have described most things about the proposed approach clearly and have written an easy-to-understand paper. \n\nBut the paper, in its current form, is not ready to be accepted. There are several concerns about the validity of the experimental set-up and comparisons against prior works. Further, there are several decisions taken by the authors which need clarification or more empirical evidence.  \n\n\n\nEdit: \nAfter reading the other reviews and the authors' responses to all the reviews, I have increased my rating to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V7p2Xajq5th", "reviewer": "Reviewer_Vqs3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The addressed problem is interesting and relevant for many practical applications\n- The authors did a good job exploring different variants of how to leverage a pre-trained vision & language model (i.e., RPN+CLIP, ViLD-image/text/ensemble)\n- The empirical results are very encouraging\n- The proposed setting of using LVIS for open-vocabulary detection is good\n- The paper (incl. appendix) provides several additional interesting experiments (transfer to other datasets, expanding of label space with attributes, etc.)\n\n\n### Weaknesses\n\n#### Clarity\n\n- Region proposal networks (Sec 3.1 and 3.3): It is unclear to me how the authors implemented the region proposal network. Only after reading the whole paper, it seems that there are two different variants: The first one is a two-stage detector as described in Sec 3.1. The other is the standard RPN network used in Mask-RCNN, mentioned in Sec 3.3. This aspect is still unclear to me.  In any case, the two-stage objectness detector (Sec 3.1) needs more explanation. First, one should add a reference to the appendix already in Sec 3.1. But even then, some training details may be missing because in Mask-RCNN, the details of how to assign boxes to anchors/proposals is different region proposal vs. region classification networks.\n\n- Related to the previous point, in Sec. 3.3, the authors mention the \"two-stage detector\", which was unclear to me. Does this refer to a standard Mask-RCNN with RPN or is this in reality a \"three-stage\" detector?\n\n- Sec. 3 - Notations: It might be worthwhile spending one more sentence or a small figure on explaining the typical architecture of the pre-trained open-vocabulary image classification model.\n\n\n#### Related work\n\n- One could add a reference to the related problem of \"open world object detection\". Specifically, a recent work [A] introduces an \"unknown-aware RPN\", which might be relevant for this work as well.\n\n- There have been a few recent works that try to expand the label space of object detectors by unifying the heterogeneous label spaces from multiple datasets into one model [B,C]. This seems like related work for the aspect of increasing vocabulary size.\n\n#### Experiments\n\n- Sec 4.5: It is not clear if the model used to compare on the COCO dataset uses FPN or not. If yes, I believe the comparison is unfair because the main competitor (Zareian et al.) seems to only use a ResNet-50-C4 backbone.\n\n\n#### Minors and suggestions\n\n- Sec 1: \"exponentially more data is needed (Gupta et al., 2019), which makes it expensive to scale up detection vocabularies.\". This statement is true for the images, but maybe not for the annotations, i.e., the human cost, because of the federated annotation technique that is used by LVIS (and also OpenImages >= v4). Please relate this statement to this type of dataset annotation.\n\n- Please add a citation for the Average Recall (AR) metric, which is computed over multiple IoU thresholds. This may be unclear for readers not familiar with the details of object detection evaluation, and the numbers may then seem surprisingly low for a metric that measures recall.\n\n- Sec 4.3: Typo: \"... CLIP on cropped regions already outperforms supervised baselines ...\"\n\n- Implementation details: Are the input images first resized and then cropped to 1024x1024?\n\n- Implementation details: What version of the CLIP model was used for distillation? CLIP provides pre-trained models for different image backbones, e.g., R-50 or ViT.\n\n- Why was the R-50 backbone not initialized with the weights from the CLIP-R50 model? That could further add knowledge \"transfer\".\n\n- Last paragraph in introduction: \"... that uses additional tricks\" is anecdotal. Please clarify. Also, I assume the challenge winners used full supervision for rare categories during training, which should be highlighted.\n\n- Sec 4.6: The statement \"... it is the first time we can directly transfer a trained detector to different datasets\" may be inaccurate. The multi-dataset object detection paper [C] evaluates in a \"zero-shot cross-dataset\" setting (see Table 5). It is a different setting, but it's related, which should be discussed.\n\n#### References mentioned above\n- [A] Towards Open World Object Detection. Joseph et al. CVPR'21\n- [B] Object Detection with a Unified Label Space from Multiple Datasets. Zhao et al. ECCV'20\n- [C] Simple multi-dataset detection. Zhou et al. arXiv'21\n", "summary_of_the_paper": "This paper addresses the task of open-vocabulary detection, where an object detector is trained from two forms of annotations: (a) bounding box annotations for a set of base categories and (b) open-vocabulary image-caption pairs.  This detector is then evaluated on a set of novel categories.  Compared to prior work, this paper tries to distill knowledge from strong pre-trained vision & language models (like CLIP) into an open-vocabulary detector. The authors propose multiple ways to do this (RCNN+CLIP, ViLD-text, ViLD-image, ViLD), including some ensemble variants (ViLD-ensemble). The paper also introduces a novel experimental setup based on the LVIS dataset (with >1200 categories) as compared to prior works that used COCO with only 80 categories.", "summary_of_the_review": "Overall, I think this is a solid and interesting work. There are several aspects of the paper that need improvement, though, see comments above. I encourage the authors to consider those comments to further improve the manuscript.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OzzX3N-3vQ", "reviewer": "Reviewer_LJFG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper studies an important research area with many potential applications. It is a step towards more scalable and efficient training without a massive amount of supervision.\n2. The quantitative results are impressive, and provide valuable insights. Every row of Table 2 and Table 3 conveys a novel and interesting insight. Especially, the authors show that self-supervised embedding models trained on general-purpose image-caption data have the potential to outperform fully supervised models on real-world, long-tailed detection tasks.\n3. The proposed method can also be used for open-vocabulary instance segmentation, which is novel and may inspire future work in this area.\n4. Unlike prior work that relies on specialized pretraining, the proposed method decouples pretraining and training by using two separate models. This results in flexibility, because for instance, a large and complex model can be used for pretraining, while a fast, lightweight model can be used for distillation.\n\nWeaknesses:\n1. The proposed method has limited technical novelty. the authors distill an existing classification model on an existing detection model via an existing distillation method. Note that despite limited technical novelty, the empirical contribution is significant.\n\n2. The paper is at some parts vague and skips some details. For instance:\n\n    * The architecture of the text encoder is not explained anywhere.\n\n    * In Section 3.1, it seems the output of the second stage of Mask R-CNN is used as proposal, instead of the output of the first stage (RPN), which is normally called proposal. This is confusing because the output of the second stage is also the detected objects.\n\n    * There is an L2 normalization in Figure 3, which is not elaborated in the paper, and seems to be redundant as the authors use cosine similarity, which already normalizes the embeddings.\n\n    * The authors do not mention how many base and novel classes they have in the LVIS experiments.\n\n3. In VILD-text, pushing background proposals into a single point may cause them to collapse and not correctly map to novel class embeddings at test time. How do the authors address this problem?\n\n4. The authors use most of the categories as base classes, and particularly choose categories that have frequent annotations. It is not clear how the model would perform with a smaller amount of supervised data (both fewer categories and fewer samples per category), and how smaller data affects each component (i.e. recognition and proposal)\n\n5. The authors mention the CLIP baseline is slow, but do not provide quantitative speed metrics.\n\n", "summary_of_the_paper": "The authors propose a new method for training object detection models that can generalize to novel object categories without bounding box annotations. Their key idea is to use an existing vision-language joint embedding model (teacher), and distill its knowledge into a typical object detection model (student). This way, the detection model learns to project each object proposal into an embedding space, where it can be compared to the text embedding of various object categories, and the closest one can be chosen. The teacher model is an existing work (CLIP), which is trained on millions of image-caption pairs, and hence relies on weak supervision. The proposed method achieves impressive results on novel object detection, outperforming not only the best open-vocabulary detection method, but also fully supervised detectors. ", "summary_of_the_review": "The paper presents a new solution for an important, yet unexplored, research problem. It achieves impressive SOTA performance and provides valuable empirical insights. Hence, I believe this paper would significantly contribute to the field.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: the aim of this work is to produce an open-vocabulary detector.  The approach is via knowledge distillation from existing large-scale V+L models, and the evaluation is based on novel classes with LVIS.  The reviewers were generally happy with the work (approach and results), but there were substantial points of clarification during discussion that need to be properly integrated into the final manuscript.
Decision: Accept (Poster)