Learning Object-Language Alignments for Open-Vocabulary Object Detection
Authors: Chuang Lin
Peize Sun
Yi Jiang
Ping Luo
Lizhen Qu
Gholamreza Haffari
Zehuan Yuan
Jianfei Cai
Venue: ICLR 2023 poster (2023)

Abstract: Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations. Natural language supervision is an attractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine-grained object-language alignments.  Previous solutions rely on either expensive grounding annotations or distilling classification-oriented vision models. In this paper, we propose a novel open-vocabulary object detection framework directly learning from image-text pair data. We formulate object-language alignment as a set matching problem between a set of image region features and a set of word embeddings. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way. Extensive experiments on two benchmark datasets, COCO and LVIS, demonstrate our superior performance over the competing approaches on novel categories, e.g. achieving  32.0% mAP on COCO and 21.7% mask mAP on LVIS. Code will be released.

Published as a conference paper at ICLR 2023
LEARNING OBIECT-LANGUAGE ALIGNMENTS FOR VOCABULARY OBJECT DETECTION
Chuang Lin
Peize Sun?
Yi Jiang
Ping Luo"
Lizhen Qu'
Gholamreza Haffari'
Zehuan Yuan²
Jianfei Cai
Monash University
2 ByteDDance
The University of Hong Kong
ABSTRACT
Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations, Natural language supervision is an at- tractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine- aeeeggaaied object-language alignments. Previous solutions rely on either expensive grounding annotations or distilling cuasifiication-orientee vision models. this paper, we propose a novel open- vocabulary object detection framework directly learning from image-text pair data. We formulate object-language alignment as a set matching problem be- tween set of image region features and a set of word embeddings. It enables to train an open-vocabulary object detector on image-text pairs a much simple benchmark datasets, COCO and
and effective way. Extensive experiments LVIS, demonstrate our superior performance over the competing approaches on novel categories, e.g. achieving 32.0% mAP COCO and 21.7% mask mAP on LVIS. Code is available at: httpt::t////m. /ttp:/ggttbbbbb/// com/ c clin1223 httmm//nnn222//111/0m /VLDet
INTRODUCTION
In the past few years, significant breakthroughs of visual models have been witnessed close-set recognition, where the object categories are pre-defined by the dataset (Johnson-Robedd et al.) 2016 Sakaridis et al., 2018 Geiger al. 2013) Yu et al. 2018). Such achievements not the final answer to artificial intelligence, since human intelligence can perceive the world in an open environment. This motivates the community moving towards the open-world setting, where vi- sual recognition models are expected to recognize arbitrary novel categories in zero-shot manner. Towards this goal, learning visual models from language supervisions becomes more and more pop- ular due to their open attributes, rich semantics, various data sources and nearly-free annotation cost. Particularly, many recent image classification works (Yu et al.. [002 Yuan et al. [202]] Zhai et al. 2021 Jia et al. 2021 Radford al. 2021 1 Zhou et al., [202 a Rao et al. 2021 Huynh al. 2021 successfully expand their vocabulary sizes by learning from a large set of image-text pairs
and demonstrate very impressive zero-shot ability. Following the success in open- vocabulcay image
classification, a natural extension is to tackle the object detection task, i.e., open-vocabulary object detection (OVOD).
Object detection a fundamental problem in computer vision aiming to localize object instances in an image and classify their categories (Girshick) 2015; Ren al. 2015 Lin al. 2017 He al.
2017 Cai & Vasconcelos] 20188.. Training an ordinary object detector relies on canuually-annonee
bounding boxes and categories for objects (Fig. Left). Similarly, learning object detection from language supervision requires object-languaet
annotations. Prior studies investigated using such annotations for visual grounding tasks (Krishna et al. 2017 Yu et al., 2016 Plummer et al. 2015 Zhuang et al. 2018). Most of the existing open-vocabulag object detection works (Li et al. 2021 Kamath et al. 2021 Cai et al.) (022) depend entirely or partially on the grounding annotations,
which however is not scalable because such annotations are even more costly than annotating object
detection data. To reduce the annotation cost for open-vocabulary object detection, a handful of
"This work was performed while Chuang Lin (chuang. lin @monash.onhii worked inter at ByteDance.

---

Published as a conference paper at ICLR 2023
Annotated Data
Free Image-Text Pair
Regian-Word Pair
dog
}
harse
Bipartite Matching
door
}
{
horse
dog sitting a door with a horse behind.
dog
door
horse
Figure 1: Left: conventional object detection. Right: our proposed open- vocabulayy object detec- tion, where we focus on using the corpus of image-text pairs to learn region-word alignments for object detection. By converting the image into set of regions and the caption into of words, the region-word alignments can be solved as sel-matching problem. In this way, our method is able to directly train the object detector with image-text pairs covering a large variety of object categories.
recent studies (Du et al. 2022 Gu et al. 2021) distill visual region features from classification- oriented models by cropping images. However, their performance is limited by the pre-trained models, trained based on global image-text matching instead of region-word matching.
In this paper, we propose simple yet effective end-to-end aasoo--ndd-anggagee framework for open- vocabulary object detection, termed VLDet, which directly trains object detector from image- text pairs without relying on expensive grounding annotations or distilling classification-rriie
vision models. Our key insight is that extracting region-word pairs from image-text pairs can be formulated as a set matching problem (Carion et al., 2020 Sun et al. 2021b a Fang et al.) 2021), which be effectively solved by finding bipartite matching (Kuhn 1955) between regions and
words with minimal global matching cost. Specifically, we treat image region features as a set and word embedding as another set with the the dot-product similarity as region-word alignment score. To find the lowest cost, the optimal bipartite matching will force each image region to be aligned with its corresponding word under the global supervision of the image-text pair. By replacing the classification loss in object detection with the optimal aggion-word alignment loss, our approach can help match each image region to the corresponding word and accomplish the object detection task.
We conduct extensive experiments on the open-vocabulary setting, in which an object detection dataset with localized object annotations reserved for base categories, while the dataset of image- text pairs is used for novel categories. The results show that our method significantly improves the performance of detecting novel categories. On the open-vocabulary dataset COCO, our method outperforms the SOTA model PB-OVD (Zhou et al. 2022) by 1.2% mAP in terms of detecting novel classes using COCO Caption data (Chen et al. 2015). On the open- vocabulary dataset LVIS,
method surpasses the SOTA method DetPro (Du et al. 2022) by 1.9% mApmask in terms of
detecting novel classes using CC3M (Sharma et al.) 2018) dataset.
The contributions of this paper summarized follows: 1) We introduce open-vocabulary object detector method to learn object-language alignments directly from image-text pair data. 2) We propose formulate region-word alignments as a set-matching problem and solve it efficiently with the Hungarian algorithm (Kubn) 1955). 3) Extensive experiments on two benchmark datasets, COCO and LVIS, demonstrate our superior performance, particularly on detecting novel categories.
2 RELATED WORK
Object Detection (WSOD). WSOD aims train an object detector using image-level labels as supervision without any bounding box annotation. Multi-instance learning (Di- etterich et al. 1997) well studied strategy for solving this problem (Bilen et al., 2015 Bilen & Vedaldi[[016 Cinbis et al. (016)). It learns each category label assignment based on the top-scoring strategy, I.e. assigning the top-scoring proposal to the corresponding image label. As no bounding box supervision in training. OICR (Tang et al. 2017) proposes to refine the instance classifier on- line using spatial relation, thereby improving the localization quality. Cap2Det (Ye et al. 2019) further utilizes a text classifier to convert captions into image labels. Based on the student-teacher framework, Omni-DETR (Wang et al. 2022) uses different types of weak labels to generate accurate pseudo labels through bipartite matching as filtering mechanism. Different from WSOD, OVOD
2

---

Published as a conference paper at ICLR 2023
benefits from the fully-annotated base class data, resulting in accurate proposals to match with corre- sponding words for target classes not known in advance. Directly using WSOD methods for OVOD tends to assign region proposal to the mostly matching class, which is likely to be base class due to the full supervision for the bass classes.
Multi-Modal Object Detection. A recent line of work considers instance-wise vision-language tasks, where the free-form language from captions is required to align with the objects. GLIP (Li et. 2021) unifies object detection and visual grounding into a grounded language-image pre-
training model. MDETR (Kamath et al. 202T) develops an end-to-end text-modulated detection system derived from DETR (Carion et al. 2020). XDETR (Cai et al.] (022) independently trains the two streams, visual detector and language encoder, and align the region features and word embed- dings via dot product operations. All these multi-modal detectors extend their vocabulary fully or partially relying on annotated grounding data, i.e. object-language pairs with ground truth bounding boxes. In contrast, our model directly learns the region-word alignments for the novel classes by bipartite matching from the image-text pairs without bounding box annotations.
Open- Vocabulary Object Detection (OVOD). Recently, there is a trend toward improving the generalization ability of object detectors to new categories via multi-modal knowledge. Open- vocabulary object detection aims to increase the vocabulary of object detection with image-text pairs. As the first work of OVOD, OVR-CNN (Zareian et al. [021 uses corpus of image-caption pairs to pretrain a vision-language guage model and transfers the visual backbone to the supervised object detector. With the huge progress in image-level open vocabulary recognition (Yu et al.] [022] Yuan itt al. 2021 Zhai et al. 2021 Jia et al. 2021 Radford et al. 2022), distilling the knowledge from the off-the-shelf vasion-language model like CLIP (Radford al. [022] has become a popular solution
for OVOD tasks Zhong et al. 2022 Minderer el al. 2022). For instance, [Zhong et al. (2022) and
Gao et al. (222T) generate the pseudo region annotations from the pre-trained vision- laigunn model and use them as training data for detectors. To get a better localization ability for generating pseudo
region annotations, /Zhao et al. (2022) use another two-stage detector as a strong region proposal net- work (RPN). In order to use the CLIP class embedding effectively, DePro (Du et al. (022) designs fine-grained automatic prompt learning. Different from these works, we scale up the vocabulary of object detection by using the annotation-free data of image-text pairs. As it is easy to extract noun words from captions, our object vocabulary is easily to be extended and scaled. Besides, some work focuses improving image segmentation in open vocabulary settings. Ghiasi (Ghiasi et al. (022) organizes pixels into groups first and then learns the alignments between captions and predicted masks. Another close work is Detic (Zhou al.) (022) that focuses on increasing the performance on novel classes with image classification data by supervising the max-size proposal with all image
labels. Unlike Detie, we try to align image regions and caption words by bipartite matching.
3 METHOD
3.1 OVERVIEW
A conventional object detector can be considered as trained with (o;, (0;, where o; denotes the i-th object/region/box feature of an input image J, and 9: = (b;, G) denotes the corresponding labels with the bounding-box coordinates and their associated categories e; E Chase, where Cbase is the set of base classes, or pre-defined classes. Open-vocabulary Object Detection (OVOD) (Zareian al. 202T) aims to acquire a large vocabulary of knowledge from corpus of image-caption pairs to extend detection of pre-defined classes to novel classes. In other words, our goal is to build an object detector, trained on dataset with base-class bounding box annotations and a dataset of image-caption pairs (T, C.) associated with large vocabulary Copen, to detect objects of novel classes Cnovel during testing. Note that the vocabulary Chase and Cnovel might or might not overlap with Coper.
Fig. 2 gives an overview of our proposed VLDet framework. We propose to learn fine-grained region-word word alignments with the corpus of image-text pairs, which is formulated as a set-matching problem, i.e., matching a set of object candidates with a set of word candidates. In the following, first describe how to formulate the learning of region-word alignment from image-text pairs as a bipartite match problem in Section 3.2 and then depict the network architecture followed by some method details in Section 3.3
3

---

Published as a conference paper at ICLR 2023
E
Bipartite Matching
Callnn
[Ktchen]
Stoone
kitchen with cabinets. stove, microwave and refrigerator.
[Cabinet]
[Stove]
E
[Rsfrigeraoor
[Microwave]
Pariial
BCE loss
Itatpe
Text Haceekr
Image Parader
Wood Einholding
Region Proposdi
Figure 2: Overview of our proposed VLDet framework. We propose to learn fine-grained region- word alignments with the corpus of image-text pairs, which is formulated as bipartite matching problem between image regions and word candidates.
3.2 LEARNING OBJECT-LANGUAGG ALIGNMENTS BY BIPARTITE MATCHING
Recap on Bipartite Matching, The Bipartite Matching describes the following problem: supposing there are X workers and Y jobs. Each worker has subset of jobs that he/she is capable to finish. Each job can only accept one worker and each worker can be appointed to only one job. Because each worker has different skills, the cost dz,y required perform a job y depends on the worker x who is assigned to it. The goal is to determine the optimum assignment M such that the total cost minimized or the team effectiveness is maximized. For this purpose, given a matching matrix M. if element m,,j = 1, it indicates a matching; otherwise, not matching, We can formulate this matching problem as:
min M
ΣΣ dismis
(1)
The constraints is to ensure each job being assigned to one worker there are more workers; other- wise, ensure each worker being assigned to job. In our case, the jobs are object regions r; from image I and the workers are words w, from caption C.
Learning Object-Language Alignments from Image-Text Pairs. Because the image-text pairs data is easy to acquire, we exploit to sharply increase the coverage object classes by leveraging the large vocabulary from an image-text dataset. However, critical information is missing for the existing object detection models, which require labeling of image regions with words for training. Hence, the key challenge to find the correspondences between the of regions and the set of words. Instead of generating pseudo bounding boxes and labels for each image, we propose to formulate the problem of region-wor alignments as an optimal bipartite matching problem.
Given an image I, the candidate region features from the image encoder is defined as R = [F1, r2, 12,, .... Fm.. where m is the number of the regions. Given caption C, we extract all the nouns from the caption and embed each noun with the language encoder. The word embedding is defined as W [1, W2...., W|wil. where the [W| is the number of nouns in the caption. Usu- ally, we have m W as the region proposal network provides sufficient candidate regions. We define each region as "job" which tries to find the most suitable "worker" and each word as "worker" who finds the most confident "job". In this context, our approach converts the region and word assignment task into set-to-set bipartite matching problem from global perspective. The cost between image regions and words is deñned the alignment scores S WR T_ The bipartite matching problem then be efficiently solved by the off-the-shelf Hungarian Algorithm (Kuhn) 1955). After matching, the classifier head of the detection model is optimized by the following cross-entropy loss:
IWI
Lregion- word
log (osk)(ss))) + Σ log(1 o
(2)
JEW
where is the sigmoid activation, Sie is the alignment score of the i-th word embedding and its corresponding k-th region feature matched by the bipartite matching, and W" is the set of nouns from other captions in the same batch.
4

---

Published as a conference paper at ICLR 2023
Besides, we further consider image-text pairs special region- region-word pairs. Particularly, we extract the Rol feature of an image by considering the entire image as a special region and the entire caption feature from the text encoder sample and other captions in the same minibatch as negative samples. We use a similar binary cross-entropy loss Termage-tert for image-text pairs as Eq. 2
special word. For an image, we consider its caption as a positive
Object Vocabulary. The object vocabulary during the training can be the object labels [Cbase, ] defined in the dataset, as many recent works (Zhou et al., 2022 Gao et 2021] do for LVIS and COCO. However, we notice that it is not strictly following the open-vocabulary setting. In our design, we set the object vocabulary as all nouns in the image captions in each training batch. From the perspective of the whole training process, our object vocabulary Copen size is much larger than the dataset label space. Our experiments show that this setting not only realizes the desirable vocablocyy detection, but also achieves better performance than previous works. More details are discussed in the experiment section.
3.3
NETWORK ARCHITECTURE
Our VLDet network includes three components: a visual object detector, a text encoder. and an alignment between regions and words. We choose the two-stage framework Faster R-CNN (Ren al., 2015) Cai Vasconcelos) 2018: Zhou et al.] 202Tb) as our object detector component. The first stage in the object detector remains the same as Faster R-CNN, predicting object proposals by the region proposal network. To adapt the two-stage object detector into the open-vocabulary setting, we modify the second-stage in two aspects: (I) we use the class-agnostic localization head instead of the class-specific one. In this way, The localization head predicts bounding boxes regardless (2) We replace the trainable classifier weights with the language embeddings to convert the detector to the open- vocabulary setting. We use a pretrained language model CLIP (Rad- Ford et al. 2021) as our text encoder. Noting that there is no specific design in the object detector architecture, it is easy to replace Faster R-CNN by any other detector like Transformer-basee detec- tors (Carion et al. 2020: Zhu et al. 2020; Meng et al. 2021 Zhang et al. 2022).
of their categories.
Training and Inference. The training data consists of a dataset annotated with base-classes a dataset of image-text pairs. On the dataset with base-classes, the model is trained in classical two-stage detection pipeline including classification and localization. For each image-caption pair, the image encoder generates candidate regions by taking the image as input and the text encoder encodes nouns in the caption as word embeddings. After that, the region features and word embeddings are aligned by the bipartite matching, followed by optimizing the model parameters. During inference, VI.Det adopts the inference process of the two-stage object detector by incorporating the language embeddings into the classifier.
4 EXPERIMENTS
4.1 DATASETS
COCO and COCO Caption. Following open-vocabulary COCO setting (OV-COCO) Zareian
ct al. (222)) the COCO-2017 dataset is manually divided into 48 base classes and 17 novel classes,
which are proposed by the zero-shot object detection (Bansal al. 2018). We keep 107,761 images
with base class annotations as the training set and 4,836 images with base and novel class annotations
as the validation set. For images-text pairs data, we use COCO Caption (Chen et al. 2015) training
set, which contains 5 human-generated captions for each image. Following (Gu et al.
2021 Zareian
al. 202T) we report mean Average Precision (mAP) at an IoU of 0.5.
LVIS and Conceptual Captions. To study a large-scale generalized setting for open-vocabulary object detection, we conduct experiments on the LVIS (Gupta et al. 2019) benchmark with Con- ceptual Captions (CC3M) (Sharma et al. 2018) as the paired image-fext training data. LVIS dataset with object detection and instance segmentation annotations has diverse categories, which is more suitable for the open-vocabut vocabulayy object detection task. In this setting, we follow ViLD (Gu et al.) 2021) with the common classes and frequency classes as base classes (866 categories) and rare classes novel classes (337 categories). We remove novel class annotations in training and predict all categories in testing. Different from COCO Caption (Chen et al. 2015) using the same images as COCO-2017, CC3M (Sharma et al., 2018) collects 3 million image- text pairs from the web. Fol- lowing the official LVIS setting, we train a mask head on base-class detection data and report the mask AP for all categories.
5

---

Published as a conference paper at ICLR 2023
Table Open-vocabulary object detection results COCO dataset. We follow the OVR-
CNN Zareian et al. 2021) using the same training data of COCO Caption (Chen et al. 2015
and outperiorm the state-of-art method PB-OVD (Gao al. 202T) on novel classes.
Method
Novel AP
Base AP
Overall AP
Base-only
1.3
52.8
39.3
OVR-CNN (Zareian al.
2021
22.8
46.0
39.9
Detic (Zhou et al. 2022)
27.8
47.1
42,0
RegionCLIP (Zhnng al.
2022
26.8
54.8
47.5
ViLD Gu et al. 2021
27.6
59.5
51.3
PB-OVD (Gao et 2021
30.8
46.1
42.1
Our
32.0
50.6
45.8
Table 2: Open-vocabulary object detection results on LVIS dataset for two different backbones
ResNet50 (RN50) (He et al.) (016) and Swin-B (Liu et al. 2021). Base-only method only uses base
class bounding box, which considered our baseline.
Method
Backbone mAPmask
Novel
mAPmask mAPmask
mAPmask
Base-only
RN50
16.3
31.0
35.4
30,0
ViLD (Gu al. 2021
RN50
16.6
24.6
30.3
25.5
RegionCLIP ZZong et al.
2022
RN50
17.1
27.4
34.0
28.2
DetPro Du et al.
2022)
RN50
19.8
25.6
28.9
25.9
Detic Zhou et
2022
RN50
19.5
30.9
Our
RN50
21.7
29.8
34.3
30.1
Base-only
Swin-B
21.9
40.5
43.3
38,4
Detic Zhou et al.
2022
Swin-B
23.9
40.2
42.8
38.4
Our
Swin-B
26.3
39.4
41.9
38.1
4.2
IMPLEMENTATION DETAILS
In each mini-batch, the ratio of base-class detection data and image-text pair data is 1:4. For better generalized ability, we use fixed CLIP text encoder to embed the caption and object words. To reduce training time, always initialize the parameters from the detector trained by the fully supervised base-class detection data following (Zhou et al.) 2022).
In the open-vocabulary COCO experiments, basically follow the OVR-CNN Zareian et al. 2021) setting without any data augmentation. Our model adopts Faster R-CNN (Ren et al. 2015) with ResNet50-CA C4 (He et al. 2016) the backbone. For the warmup, from to 0,002 for the first 1000 iterations. The model is trained for 90,000 iterations using SGD optimizer with batch size 8 and the learning is scaled down by factor of 10 60,000 and 80,000 iterations.
increase the learning rate
In spca-vocybulary LVIS experiments, we follow Detic (Zhou al.) 2022) adopt Center- Net2 (Zhou et al.] 2021b) with ResNet50 (He et al., 2016) as backbone. We use large scale jit- tering (Ghiasi al. 2021 repeat factor sampling as data augmentation. For the warmup, we increase the learning rate from 0 to 2e-4 for the first 1000 iterations. The model is trained for 90,000 iterations using Adam optimizer with batch size 16. All the expriments are conducted on NVIDIA V100 GPUs.
4.3 OPEN- VOCABULARY COCO
Table shows the performance comparisons of different methods for open-vocabulary COCO datasets. can be seen that our model performs the best on novel classes, suggesting the supe- riority of using the bipartite matching loss with the image-text pairs. Base-only method indicates the Faster R-CNN trained with the fully supervised COCO base-category detection data with CLIP embeddings the classifier head. Although CLIP has the generalized ability to the novel class, it only achieves 1.3 mAP. Although ViLD and RegionCLIP distills the region proposal features us- inferior
ing the CLIP, their performances in the opca-vocyabllayy object detection setting. These distillation methods require both the image encoder and the text encoder from the pretrained CLIP model learn the matching between the
novel categories, which is the main metric

---

Published as a conference paper at ICLR 2023
Hand butting money in iar
World first person to attempt ofiskis
Hand
TAE
Boy jitting
usiing Hypewnier type al letter
Desktop tp view with luplop cannerii
plani Jmd Imabile
camtra
sypewriter
dask
plas
Figure 3:
Visualization of the bipartite matching results. Here show how the proposed region matched with their corresponding words. Best viewed on screen.
image regions and the vocabulary. Thus, their performances on novel classes are limited by the pre- trained model, which trained for global image-text matching instead of region-wort wood matching. Compared with the state-of-the-art method PB-OVD (Gao et al. (222)) which leverages the COCO Caption (Chen et al. 2013), VG (Krishna al., 2017) and SBU Caption dataset (Ordonez et al.) 2011) with about IM training image-text pairs, method still outperforms it all metrics while we use only 10K images of COCO Caption as training data.
4.4 OPEN-VOCABULARY LVIS
Table 2 shows the performance comparisons on the open-vocabulary LVIS object detection. The "Based-only" method is only fully supervised on the base categories, which is the baseline of our method. As mentioned above, since we use fix CLIP embeddings as the classifier head, the base- only method has certain generalization ability. The proposed model outperforms the state-of -tate-of-the- art method DetPro (Du et al. 2022) by 1.9 % AP on novel classes. RegionCLIP (Zhong et al.) (022) uses a CLIP model to predict pseudo-labels for image regions and pretrains a visual model with the region-text pairs. We can see that it is effective to improve the generalization ability to novel classes. Similar strategy is used in recent works (Gao et al. 202I). However, such two-step pscudo- label method is easy amplify false predictions and it relies on good initialization of the teacher model to generate good pseudo-labels. In contrast, our proposed method is assigning class labels from global perspective and optimizing the object detection model without finetuning. To further explore the scalability our model, we use Swin-B as our backbone, and our model outperforms the ResNet backbone with 4.5% mAP on novel classes. Compared with Detic, model achieves 2.4% gain on OV-LVIS novel classes, suggesting the effectiveness of the label assignment from the global perspective.
Visualization. As shown in Figure we visualize some cases of matching results of CC3M in open vocabulary LVIS setting, As we can see, the model can extract promising region-words pairs from image-text data, avoiding expensive annotations. The results of the matching include a variety of objects, such as "jar", "typewriter", and "camera". which indicates that our method significantly expands the vocabulary for object detection. It demonstrates the generalization ability of our method to novel classes.
4.5 ABLATION STUDIES
Object Vocabulary Size. In our VLDet, we follow the previous work (Zhong et al. 2022) using all nouns in COCO Caption and CC3M and filtering out low-frequency words, resulting in 4764/6250

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study on vocabulary size. We replace our object vocabulary with the category names of OV-COCO and OV-LVIS. The results show that training with large vocabulary size enables better generalization.
OV-COCO
OV-LVIS
Vocabulary
Size
mAPnovel novel
mAP all
Size
mAPmest
mAP
novel
all
Category names
65
28.2
42.8
1203
18.9
31.2
All nouns from caption
4764
30.0
44.6
6750
20.4
30.4
Table 4: Ablation study on matching strategy. We implement Hungarian algorithm (Kuhn 1955) for
one-to-one region-word matching and Sinkhorn algorithm (Cuturi 2013 for one-to-many matching.
OV-COCO
OV-LVIS
Matching Strategy
MAP nonel
mAP all
mApmas nowel
mAPmask
One-to-Many One-to-One
29.1
44.6
18.5
29.2
32.0
45.8
21.7
30.1
concepts left. We notice that many works recently use the LVIS and COCO label spaces as the object vocabulary during training. which do not strictly follow the open-vocabulary setting, We further analyze the effect of training our model with different vocabulary sizes. In Table 3 we replace our object vocabulary with the category names in COCO and LVIS datasets, i.e. only using the category names in the captions. We can see from Table 3 that larger vocabulary significantly boosts the unseen class performance. It achieves gains of 1.8% and 1.5% on the novel categories of OV-COCO and OV-LVIS, respectively, indicating that training with large vocabulary size leads to better generalization. In other words, with a bigger vocabulary, the model can learn more object language aligmments which benefits the novel-class performance during inference.
One-to-One vs. One-to-Many, The key to extract a set of image region-word pairs from an image- text pair is to optimize the assignment problem from global perspective. To further investigate the impact of assignment algorithms, we implement two global algorithms, Hungarian (Kuhn 1955) and Sinkhorn (Cuturi) 2013) algorithms, where the former does one-to-one region- wword assignment, and the latter provides soft alignments between one word and many regions. Considering that there may be multiple instances of the same category in an image, Sinkhorn algorithm able to produce multiple regions to the same word, while it might also introduce more noisy region- word pairs. From Table 4 we observe that the one-to-one assignment achieves 32.0 AP and 21.7 AP for novel classes, both outperforming the one-to-many assignments of 29.1 AP and 18.5 AP. The one-to-one assignment assumption sharply reduces mis-alignments by providing each word with a high-quality image region.
Discussion on noisy image-text pairs. In VLDet, we employ the Hungarian algorithm to assign each word a unique region. When dealing with multi-word expressions in text, such as "a basket of oranges", we select the most confident region-word pair and ignore the remaining ones, such as the one associated with "oranges". Similarly, if the caption is incomplete (which is often the case, since the caption may not describe every instance in the image), we will ignore those unmatched regions as well. Note that those regions will not be optimized background, and the loss is only computed with the regions in the matching results.
Different Strategies for Region- Word Alignment. Figure4showss 4shows comparison to different strate- gies for the region-word alignment in weakly supervised methods (Zhou el al. 2022 Redmon & Farhadi 2017, Yao et al., 2021). For fair comparison, we use the region-word alignment loss alone with different strategies for training. "Max-score" (Redmon & Farhadi 2017: Yao et al.) 202T) indicates each word selects the proposal with the largest similarity score, which is widely used in supkly-s object detection. However, it tends to assigns each region proposal to a base class due to the full supervision for the bass classes. In contrast, our formulated set-to-set matching aims to minimize the global matching cost to find each region proposal distinguished correspond- ing word, where strongly matched base-class regions can push other region proposals to explore the matching with novel classes. "Max-size" (Zhou et al. 2022). assigns all nouns in caption to the proposal with the max size. The results in Figure 4 show that our method achieves the best performance for the novel classes on both OV-COCO and OV-LVIS.
Image-Tex Alignment. We conduct ablation experiments to evaluate the effectiveness of different training losses of our method in Table 5 We observe that using the region- alignment loss alone
8

---

Published as a conference paper at ICLR 2023
OV-COCO
21
30
28
20
Dovel AP(5))
19
24
22
17
16
18
1000
15
16
14
biporftite
malthhin
13
1k
21
3k
41
51 60 7k 8%
91. Iter
1k
2k
31c
9k
Figure 4: We show comparison of different alignment strategies in terms of novel-class mAP on both open-vocabulary COCO and LVIS. Different from the bipartite matching that learns the alignment from the global perspective, "max-score" (Redmon & Farhadi 2017]] Yao et al. (021) does the alignment based the highest similarity score and "max-size" (Zhou et al. 2022 assigns the max size proposal to the image labels.
Table 5: Ablation study on training losses. It performs best when both the region- word alignment loss and the image-text alignment loss are used, suggesting that they benefit from each other.
OV-COCO
OV-LVIS
Lregion-wes word word
Lsmage-tess
mAP mAP all
mAP novel
mAP
30.0
44.6
20.4
30.4
21.0
43.8
17.4
30.4
32.0
45.8
21.7
30.1
achieves promising results. It demonstrates the effectiveness of the bipartite matching strategy for learning region-word alignment. Furthermore, we can find that using image-text pair as additional supervision can improve the performance. Intuitively, the image-text pairs data can provide contex- tual information which complements semantics beyond nouns. A similar conclusion can be found
(Zhou al. 2022 Zhong et al. 2022).
4.6 TRANSFER TO OTHER DATASETS.
To evaluate the generalization ability of our model, we conduct the experiments on transfer- ring COCO-trained model to PASCAL VOC (Ev- eringham et al., 2010) test set and LVIS validation set without re-training. We directly use the model trained with COCO Caption data for OV-COCO and replace the class embeddings the classifier head. Directly transferring models without using any training images is challenging due to the do- main gap. What's more, LVIS includes 1 1203 ob- ject categories which is much larger than COCO label space. As shown in Table 6 our model achieves 2.5 % and 2.0% improvement on VOC test set and LVIS validation set, demonstrating the effectiveness of our method on diverse image do- mains and language vocabularies. While LVIS includes category names that do not appear COCO Caption concepts, our model can learn close concepts, which facilitates the transfer to LVIS.
Table 6: Transfer to other datasets. We evaluated COCO-trained model on PASCAL VOC (Everingham et al., 2010) test set and LVIS validation set without re-training. All re- sults are box AP50-
Method
PASCAL VOC
LVIS
OVRCNN PB-OVD
52.9
5.2
59.2
8.0
Our
61.7
10.0
CONCLUSIONS AND LIMITATIONS
We have presented VLDet which aims to learn the annotation- object-language alignment from the image-caption pairs for gpen-vocabulary object detection. Our key idea is to extract the region- word pairs from global perspective using bipartite matching by converting images into regions and text into words. We only use single object detection model supervised by the free image-text pairs without any detector VLDet achieves new state-of-the-art performance on both open-vocabulle LVIS. We hope this work can push the direction of OVOD and inspire more work on large-scale free image-text pair data. The potential limitations of VLDet are: 1) it does not consider the bias of vision-an and--anddam language data; 2) we have not investigated even larger data volumes, e.g. Conceptual Captions 12M (Changpinyo al. 2021 We leave them for future works.
annotations for novel classes. The extensive experiments show that our
COCO and

---

Published as a conference paper at ICLR 2023
REFERENCES
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European Conference Computer Vision (ECCV), pp- 384-400, 2018.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1846-2854, 2016.
Hakan Bilen, Marco Pedersoli, and Tinne Tuytelaars. Weakly supervised object detection with convex clustering. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 1081-1089,001
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnm: Delving into high quality object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6154 6162, 2018
Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-detr: A versatile architecture for instance-wiss vision-language tasks. arXiv preprint arXiv:2204.0500 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229 Springer, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Xinlei Chen, Hao Fang, Tsung-Yo Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00322. 2015.
Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. Weakly supervised object local- ization with multi-fold multiple instance learning. IEEE transactions pattern analysis and machine intelligence, 39(1):189-203; 2016.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26, 2013.
Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez, Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 89(1-))31-71, 1997.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning prompt for vocabuocyyylary object detection with vision-language model. arXiv preprint S.ti::2203.19444 2022.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2); 303-338, 2010.
Yuxin Fang, Shusheng Yang, Xinggang Wang. Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6910-6919, 2021.
Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Towards open vocabulary object detection without human providdd bounding boxes. arXiv preprint arXiv:2111.09452 2021.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11 12211)712311 2013.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 2928-2928, 2021.
10

---

Published as a conference paper at ICLR 2023
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open- veen-vocabllary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1440-1448, 2015.
Xiuye Gu, Tsung-Yo Yi Lin. Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmentation via robust cross-modal pseudo-labeling. arXiv preprint arXiv:2111. 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Sung, Zhen Li, and Tom Duerig. Scaling up visual and lision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Matthew
Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human- generated annota- tions for real world tasks? arXiv preprint arXiv: 1610.01880 2016.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1780-1790, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International of computer vision, 123(1):32-73, 2017.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded iinuuage-imaee pre- training, arXiv preprint arXiv:2112.03855 2021.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2117-2125, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 3651-3660, 2021.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple ooeauuucyy object detection with vision transformers. arXiv preprint arXiv:2205.062300 2022.
11

---

Published as a conference paper at ICLR 2023
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phrant correspondences for richer image- to-sentence models. Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. arXiv preprint arXiv:21 2021.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 7263-7271, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, pp. 91-99,99999999 2015.
Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126(9):973-992, 2018.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 2018
Peize Sun, Yi Jiang, Enze Xie, Wenqi Shao, Zehuan Yuan, Changhu Wang, and Ping Luo. What makes for end-to-end object detection? In International Conference on Machine Learning, pp. PMLR, 2021a.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2021b.
Recognition, pp.
Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2843-2851, 2017.
Pei Wang, Zhaowei Cai, Hao Yang. Gurumurthy Swaminathan, Nuno Vasconcelos, Bernt Schiele, and Stefano Soatto. Omni-detr: Omni-supervised object detection with transformers. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9367-9376, 2022.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07788 2021.
Keren Ye, Mingda Zhang, Adriana Kovashka, Wei Li, Danfeng Qin, and Jesse Berent. Cap2det: Learning to amplify weak caption supervision for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 9686-9695, 2019.
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv: (a0o:08005.445 2(5):6, 2018.
12

---

Published as a conference paper at ICLR 2023
Jiahui Yu, Zirui Wang, Vijay
Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pp. 69-85. Springer, 2016.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14393-14400 2021.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. arXiv preprint arXiv:2111.0799) 2021.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.0366 2022.
Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris N Metaxas. Exploiting unlabeled data with vision and language models for object detection. In European Conference on Computer Vision, pp. 159- 175. Springer, 2022.
Yiwu Zhong, Jianwei Yang, Pengehuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-basee language- pretraining. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 16793-16803, 2022.
Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXiv:21 2021a
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint arX1v22103.07461, 2021b.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. arXiv preprint arXiv:2201.026000 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton Van Den Hengel. Parallel attention: A unified framework for visual object discovery through dialogs and queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4252-42222 4221, 2018.
13


Reviews:
[{"review_id": "qSJmjF_u4h", "reviewer": "Reviewer_y363", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel method for open-vocabulary object detection (OVOD), where a model is trained from two datasets, one containing bounding box annotations for a set of base categories, and another one containing only free-form text descriptions (captions) of imagesf. While prior work typically uses grounded annotations (words in captions are associated with bounding boxes) or distills information from image-level pre-trained vision and language models, this work proposes a loss function to directly leverage image-text pairs (without grounding supervision). The architecture of the detector is similar to existing ones, with the only exception that text embeddings are predicted instead of a fixed-size probability distribution over a known label space. The model trains from both detection data and image-text pairs. The losses are the same as in standard detectors when training from detection data. For image-text pairs, a similarity is computed between each noun in the caption and each predicted region (region proposals). Bipartite matching is then applied on the similarity matrix to find a matching between regions and nouns, which then serves as ground truth for a standard binary cross-entropy loss. Experiments on the open-vocabulary setting for both COCO and LVIS datasets show improvement over prior baselines, specifically for novel categories.", "summary_of_the_review": "Overall, I think this a high-quality paper with clear impact to the research community on open-vocabulary object detection.", "strength_and_weaknesses": "Strengths:\n- The topic is interesting\n- The paper is well written, easy to follow, figures a good\n- The method is easy and simple, but effective\n- The experiments are mostly clear and the method performs well\n- Ablation studies are reasonable.\n\nWeaknesses:\n- The dataset generalization experiment in Section 4.6 from COCO to Pascal VOC is limited. The 20 Pascal VOC categories are a subset of the 80 COCO categories. The question is how many of the 20 categories are part of the 48 base categories. Still, there will likely only be a few novel categories.\n- Did you try a different text embedding architecture that was not jointly trained on images and text? Would this make a difference?\n- The related work and the baselines can be improved\n  - I think paper [A] (concurrent work) can be discussed in the related work section.\n  - The paper [B] is another recent example for OVOD that can be discussed and compared with. It uses pseudo labels and gets strong results on COCO, although in a slightly different setting.\n  - The reference \"Gu et al.\" is now an ICLR 2022 paper\n- I'm missing a more thorough discussion on the region-word alignment loss\n  - It would be good to explain in words how the loss deals with incomplete captions, i.e., when some objects captured in the image are not referred to in the text. Looking at the loss and assuming a perfect matching, it seems such regions would just be ignored by the loss.\n  - The nouns in other captions of the same mini-batch (W') may contain the matching (positive) word. How likely does this happen? And why does W' not contain other words of the same caption?\n  - How do you deal with plural descriptions in a text, e.g., the caption \"Two cats near a dog\". Again, I think the loss would just ignore one of the regions that contain a cat (assuming a good matching). Still, I think it would be valuable for the paper to discuss such situations in text.\n\nReferences:\n- [A] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels. Golnaz et al. ECCV 2022\n- [B] Exploiting Unlabeled Data with Vision and Language Models for Object Detection. Shao et al. ECCV 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aR9JBjtAdh", "reviewer": "Reviewer_W8ar", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a set-matching approach for open-vocabulary object detection by jointly training on detection and image caption dataset. The approach achieves good results on the existing COCO and LVIS open-vocabulary detection benchmark.", "summary_of_the_review": "The paper proposes an interesting approach to tackle open-vocabulary detection by set matching loss on caption data. There are some weaknesses in the paper but I think the merits outweigh the downsides and am leaning positive for the paper.", "strength_and_weaknesses": "Strengths:\n* The idea of joint training with caption data is promising as captions are noisier and more scalable than image tags\n* Set matching is an intuitive approach to handle region-word alignment\n* The label space of COCO/LVIS are not used in training phase which follows the open-vocabulary detection settings\n* Performance on COCO/LVIS are both strong and the margin over Detic baseline is clear\n* Ablation studies are informative.\n\nWeakness:\n* Method - what are the constraints for region-text matching? Are all regions assigned to words or are all words assigned to regions? I can imagine some words or regions are noisy and do not have good matches.\n* Results - the scalability of the proposed approach needs further study in the axis of data size (e.g. CC12M, larger web-scale image-text data) and model size (larger backbones). It'd be very useful if this approach scales well with data and model size. A simple study is to subsample CC3M and observe the scaling benefits.\n* Looks like the proposed approach compromises base categories compared to the Detic baseline. Table 5 image-text loss only has the same overall AP suggests this as well. Is there any explanation for this?\n* Comparison with DetPro is unfair because DetPro uses ViLD detector as baseline while this work uses Detic CenterNet as baseline. Detic papers report a ViLD detector baseline which is lower. A more fair comparison would be to report a Mask R-CNN detector (ViLD style) baseline on LVIS.\n* SOTA on LVIS claim is not well supported because this study focuses on R50 backbone only.\n* [minor] how much overhead does the set-matching add to the training time?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OaPmDzWABjr", "reviewer": "Reviewer_fcRy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to directly learn from large-scale image-text pair data for open-vocabulary object detection. \nIt formulates object-language alignment as a set matching problem.\nThe benefit of such approach is to allowing training on image-text pairs without the help of grounding information.\nThe author conducts experiments on COCO and LVIS to demonstrate its performance on novel categories.", "summary_of_the_review": "Overall, I feel solving object language alignment without grounding information is the correct way. However, this paper doesn't prove the effectiveness of using bipartite matching nor well explain the details. \n\nAfter reading the rebuttal, I still don't get enough details on the correctness of the proposed method. Meanwhile, it also raised me another concern on its ability of handling dense detection. Hence, I would not recommend to accpet this paper.", "strength_and_weaknesses": "Strength\n+ Performance is slightly better than previous methods\n\nWeakness\n- The paper lacks necessary mathematics proof on solving object language alignments learning. How does bipartite matching is optimized end-to-end during training? Hungarian matching is not differentiable. How does it guarantee convergence on learning?\n- It also lacks implementation details. For example, in table 4, how does one-to-one or one-to-all affect the matching? In table 5, how does image-text pair loss implement?\n- For Table 1, compared with Base-only, the proposed alignment loss downgrades the performance on supervised base categories. What is the reason?  \n- For Figure 3, the author avoids example with multiple same instances. What will happen when apply to an image \"a basket of oranges\"?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "AT8MEFpW4Ob", "reviewer": "Reviewer_cyAh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to use weak supervision (image-caption pairs) for open-vocabulary object detection (OVOD). This is in fact a problem of weakly-supervised object detection (WSOD). In stead of using the max-score or max-region matching in the literature, this paper proposes to formulate the problem of object-noun matching as a problem of bipartite matching problem, which can be solved by Hungarian algorithm. The experiments on COCO and LVIS have shown that this simple method can have big improvements. ", "summary_of_the_review": "The idea is simple and easy to understand, and it gives some big improvements. But the technical novelty is small, no results comparing with the state-of-the-art, and I am concerned about the reproducibility. ", "strength_and_weaknesses": "**Strength**\n\n+Since the object-language annotation is expensive, it is reasonable to use weak supervision to improve OVOD performances.\n\n+The use of bipartite matching is reasonable and simple.\n\n+The improvements on COCO and LVIS seem nontrivial.\n\n\n**Weaknesses**\n\n-Using weak supervision to improve OVOD performance is not new, e.g. in RegionCLIP, GLIP, X-DETR, Detic. But those works only show some minor improvements, unlike 5 points gain on LVIS in this paper. I don't see anything fundamentally different, so I am not sure why the gains in this paper are so big.\n\n-Using bipartite matching for WSOD is not new, as seen in [1]. However, this paper doesn't discuss it.\n\n-This paper only shows the results on a base backbone, e.g. ResNet-50. However, many works have already shown high OVOD performances using some large models, e.g. [2]. I am wondering if the gains of this paper can be generalized to large models. \n\n-It seems using weak data can easily get 4-5 points improvement, but I don't see the ablations studies of getting those improvements step-by-step\n\n[1] Omni-DETR: Omni-Supervised Object Detection with Transformers, CVPR 2022\n\n[2] Simple Open-Vocabulary Object Detection with Vision Transformers\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster