CoDet: Co-occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection
Authors: Chuofan Ma
Yi Jiang
Xin Wen
Zehuan Yuan
XIAOJUAN QI
Venue: NeurIPS 2023 poster (2023)
Keywords: Open-vocabulary Object Detection; Object-level Vision-Language Pretraining
TL;DR: A novel open-vocabulary detection framework learning region-word alignment by object co-occurrence across images.

Abstract: Deriving reliable region-word alignment from image-text pairs is critical to learn
object-level vision-language representations for open-vocabulary object detection.
Existing methods typically rely on pre-trained or self-trained vision-language
models for alignment, which are prone to limitations in localization accuracy or
generalization capabilities. In this paper, we propose CoDet, a novel approach
that overcomes the reliance on pre-aligned vision-language space by reformulating
region-word alignment as a co-occurring object discovery problem. Intuitively, by
grouping images that mention a shared concept in their captions, objects corresponding 
to the shared concept shall exhibit high co-occurrence among the group.
CoDet then leverages visual similarities to discover the co-occurring objects and
align them with the shared concept. Extensive experiments demonstrate that CoDet
has superior performances and compelling scalability in open-vocabulary detection,
e.g., by scaling up the visual backbone, CoDet achieves 37.0 $AP^m_{novel}$ and 
44.7 $AP^m_{all}$ on OV-LVIS, surpassing the previous SoTA by 4.2 $AP^m_{novel}$ 
and 9.8 $AP^m_{all}$. Code is available at https://github.com/CVMI-Lab/CoDet.

CoDet: Co-Occurrence Guided Region-Word Alignment for Open- Object Detection
Chuofan Ma It
Yi Jiang
Xin Wen
Zehuan Yuan
Xiaojuan Qi'
'The University of Hong Kong
BByteDance Inc.
Abstract
Deriving reliable region-wort alignment from image-text pairs is critical learn object-level vision-language representations for open-vocabulary object detection. Existing methods typically rely on pre-trained or self-trained llaguae models for alignment, which are prone to limitations in localization accuracy generalization capabilities. In this paper, we propose CoDet, a novel approach that overcomes the reliance on pre-aligned vasion-language space by reformulating region-word alignment as a co-occurring object discovery problem. Intuitively, by grouping images that mention shared concept in their captions, objects corre- sponding to the shared concept shall exhibit high co-occurrence among the group. CoDet then leverages visual similarities to discover the co-occurring objects and align them with the shared concept. Extensive experiments demonstrate that CoDet has superior performances and compelling scalability in vocabulay detection, egg., by scaling up the visual backbone, CoDet achieves 37.0 APm and 44.7 APM OV-LVIS, surpassing the previous SoTA by 4.2 Apm novel and 9.8 APM all- Code is
available at http:://ggtubb.om///MMMaaaaaaa
Introduction
Object detection is a fundamental vision task that offers object-centric comprehension of visual scenes for various downstream applications. While remarkable progress has been made terms of detection accuracy and speed, traditional detectors [39, 37, 21, 5, 43] are mostly constrained to fixed vocabulary defined by training data, e.g., 80 categories in COCO [29]. This accounts for major gap compared to human visual intelligence. which can perceive diverse range of visual concepts in the open world. To address such limitations, this paper focuses on the open- apec-vocabllary setting of object detection [57], where the detector is trained to recognize objects of arbitrary categories.
Recently, lision-language pretraining web-scale image-text pairs has demonstrated impressive open-vocabulary capability in image classification [35, 25]. It inspires the community to adapt this paradigm to object detection [27, 51], specifically by training an open-vocabulary detector using region-text pairs in detection or grounding annotations [27]. However, unlike free-form image-text pairs, annotated region-texx pairs are limited and difficult to scale. Consequently, a growing body of research [59, 27, 53, 15, 28] aims to mine additional region-text pairs from image-text pairs, which raises a new question: how to find the alignments between regions and words? (Figure la)
Recent studies typically rely on rision-language models (VIMSs to determine region-word alignments, for example, by estimating region-word similarity [59, 27, 15, 28]. Despite its simplicity, the quality of generated pseudo region-text pairs is subject to limitations of VLMs. As illustrated Figure 1b, VLMs pre-trained with image-level supervision, such as CLIP [35], are largely unaware of localization quality of pseudo labels [59]. Although detector-based VLMs [27, 53] mitigate this issue to some extent, they initially pre-trained with limited number of detection or grounding concepts,
This work was performed when Chuofan Ma worked as intern at ByteDance.
"Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(A) Exacuple Image-Text Pair
(b) Aligoment by VLM
(e) Aliganient by co-oecunizue (ours)
?
frihes
Capiam:
dog leaps
Iistoe above
"A dog"
Insege-iox pain mention captions
VLM
Now to find the oligument benveen regionss and wordz?
Fraboe"
"Facsipliilyy ve oacue goup
Figure Illustratoon of different region-text alignment paradigms. (a): example image-text pair, and region proposals generated by pre-trained region proposal network; (b): pre-trained VLM (e.g.. CLIP [35]) is used to retrieve the box with the highest region- word similarity concerning the query text, which yet exhibits poor localization quality; (c) our method overcomes the reliance on VLMs by exploring visual clues, i.e., object co-occurrence, within group of image-texi pairs containing the same concept (e.g., frisbee Best viewed in color.
resulting in inaccurate alignments for novel concepts [61]. Furthermore, this approach essentially faces chicken-and problem: obtaining high-quality region-word pairs requires VLM with object-level vision-lange lasoun-anguuage knowledge, yet training such a VLM, in turn, depends on a large number of aligned region- region-word pairs.
In this work, instead of directly aligning regions and words with VLMs, we propose leveraging region correspondences across images for co-occurring concept discovery and alignment, which we call CoDet. Figure illustrates the idea. Our key motivation that objects corresponding to the same concept should exhibit consistent visual similarity across images, which provides visual clues to identity region-wor correspondences. Based on this intuition, we construct semantic groups by sampling images that mention a shared concept in their captions, from which common object corresponding to the shared concept exists across images. Subsequently, we leverage cross-image region similarity to identify regions potentially containing the common object, and construct a prototype from them. The prototype and the shared concept form a natural region-text pair, which is then adopted supervise the training of open-vocabulary object detector.
infer that
Unlike previous works, our method avoids the dependence on pre-aligned vision-language space and solely relies the vision space to discover region-word correspondences. However, there could exist multiple co-occurring concepts in the same group, and even the same concept may still exhibit high intra-category variation in appearance. in which case general visual similarity would fail to distinguish the object of interest. To address this issue, we introduce text guidance into similarity estimation between region proposals, making it concept-awan and more accurately reflecting the closeness of objects concerning the shared semantic concept.
The main contributions of this paper can be summarized as follows:
We introduce novel perspective in discovering region-word correspondences from image-text pairs, which bypasses the dependence region-word word alignment as co-occurring object discovery problem.
pre-aligned vision-langusal space by reformulating
Building on this insight, propose CoDet, open-vocabulay detection framework that lears object-level languaae alignment directly from web-crawled image-text pairs.
CoDet consistently outperforms existing methods on the challenging OV-LVIS benchmark and demonstrates superior performances in cross cross-dataset detection on COCO and Objects365.
CoDet exhibits strong scalability with visual backbones it achieves 23.4/20 23.4/22..4 4/3.0/ mask AP novel with Btt/5////iinBB//AA05 backbone, outperforming previous SoTA comparable model size by 1.//..1/4. mask AP noveet respectively.
Related Work
Zero-shot object detection (ZSD) leverages language feature space for generalization to unseen objects. The basic idea is to project region features to the pre-computed text embedding space (e.g., GloVe [34]) and use word embeddings as the classifier weights [2, 10, 36]. This presents 7SD with the flexibility of recognizing unseen objects given its name during inference. Nevertheless.

---

ZSD settings restrict training samples to come from limited number of seen classes, which is not sufficient to align the feature space of vision and language. Although some works [63, 42] try to overcome this limitation by hallucinating novel classes using Generative Adversarial Network [17], there is still a large performance gap between ZSD and its supervised counterparts.
Weakly supervised object detection (WSD) exploits data with image-level labels to train an object detector. It typically treats an image bag of proposals, and assigns the image label to these proposals through multiple instance learning [4, 9, 3, 46]. By relieving object detection from costly instance-level annotations, WSD is able to scale detection vocabulary with cheaper classification data. For instance, recent work Detic [61] greatly expands the vocabulary of detectors to twenty-thousand classes by leveraging image-level supervision from ImageNet-21K [11]. However, WSD still requires non-trivial annotation efforts and has a closed vocabulary during inference.
Open-vocabulle object detection (OVD) is built upon the framework of ZSD, but relaxes the stringent definition of novel classes from 'not seen' to 'not known in advance', which leads to a more practical setting [57]. Particularly, with recent advancement of vision-language pre- training [35, 25, 55, 54], widely adopted approach of OVD is transfer the knowledge of pre- trained vision-language models (VLMs), e.g., CLIP [35], to object detectors through distillation [18, 50] or weight transfer [33, 26]. Despite its utility, performances of these methods are arguably restricted by the teacher VLM, which is shown to be largely unaware of fine- tne-grained region- region-word alignment [59, 6]. Alternatively, another group of works utilize large-scale image-text pairs expand detection vocabulary [57, 56, 14, 27, 28, 52], sharing a similar idea as WSD. Due to the absence of regional annotations in image-caption data, these methods typically rely on pre-trained or self-traines VLMs to find region- word correspondences, which are prone to limitations in localization accuracy or generalization capabilities. Our method orthogonal to all the aforementioned approaches in the sense that it does not explicitly model region-wort correspondences, but leverage region correspondences across images to bridge regions and words, which greatly simplifies the task.
Cross-image Region Correspondence is widely explored to discover semantically related regions among a collection of images [20, 41, 45, 44, 30]. Based on the observation that modern visual backbones provide consistent semantic correspondences in the feature space [58, 20, 47, 24, 60, 1]. many works take a heuristic approach [41] or use clustering algorithms [20, 8, 23, 49] for common region discovery. Our method takes inspiration from these works to discover co-occurring objects across image-text pairs, with newly proposed text guidance.
3 Method
In this section, we present CoDet, end-to-end framework exploiting image-text pairs for open- vocabulary object detection. Figure 2 gives an overview of CoDet. We first provide a brief introduction to the OVD setup (Sec. 3.1). Then we discuss how to reformulate region-word alignment as a occurring object discovery problem (See. 3.2), which is subsequently addressed by CoDet (Sec. 3.3). Finally, we summarize the overall training objectives and inference pipelines of CoDet (Sec. 3.4).
3.1 Preliminaries
Task formulation. In our study, we adopt the classical OVD problem setup as in OVR-CNN [57]. Specifically, box annotations are only provided for predetermined set of base categories Chase during training. While at the test phase, the object detector is required to generalize beyond Clase detect objects from novel categories Chovel, Notably, Chovel is not known in advance to simulate worldw scenarios. This implies the object detector needs to possess the capability to recognize potentially any object based on its name. To achieve this goal, we additionally leverage image-text pairs with an unbounded vocabulary (open to extend the lexicon of the object detector.
OVD framework. Our method built on the two-stage detection framework Mask-RCNN [21]. To adapt it for the open-vocabulary setting, we follow the common practice [18, 61] to decouple localization from classification, by replacing the class-specific localization heads with class-agnostic ones that produce single box or mask prediction for each region proposal. Besides, to enable
3

---

Prototype
to dog't anca!
Query image
Detector
LECR
These wala Rog
Region-Regie similarity
Weights
Support images
Text guidance
"A photo of dog"
eucoder
redusing
Concept group
Region embelldings query image
Region embeddingg of support image
Text embeddings of group concept
Weighted sum
Figure 2: Overview of CoDet. Our method learns to jointly discover region-word pairs from group image-text pairs that mention a shared concept their captions (e.g., dog in the figure). We identify the co-occurring objects and the shared concept as natural region-word pairs. Then we leverage inter-image region correspondences, i.e., region-region similarity, with text guidance to locate the co-occurring objects for region- word alignment.
open- vocabbocyyylrry classification of objects, the fixed classifier weights substituted with dynamic text embeddings of category names, which are generated by the pre-trained text encoder of CLIP [35].
3.2 Aligning Regions and Words by Co-occurrence
Due to the absence box annotations, a core challenge of utilizing image-text pairs for detection training is to figure out the fine- Ane-igrined alignments between regions and words. To put it formally, for an image-text pair (T, T). . R = [1,, T2,-.., [|R] } denotes the set of regions proposals extracted from image I, and C = {ci, C2 C(C) denotes the set of concept words extracted from caption T. Under weak caption supervision, we can assume the presence of concept e T indicates the existence of at least one region in containing the corresponding object. However, the concrete correspondence between e and remains unknown.
To solve the problem, we propose to explore the global context of caption data and align regions and words via co-occurrence. Specifically, for given concept comprising all image-text pairs that mention e in their captions. This grouping effectively clusters images that contain objects corresponding concept c. Consequently, if G is large enough, can induce that the objects corresponding to concept will be the most common objects among images in G. This natural correlation automatically aligns the co-occurring objects in concept c. We thus reduce the problem of modeling cross cros-mtt correspondence (region-word) to in in-modality correspondence (region-region), which we address the next section.
construct concept group G
3.3 Discovering Co-occurring Objects across Images
Intuitively, candidate region proposals containing the co-occurring object should exhibit consistent and similar visual patterns across the images. We thus take a similarity- -riven approach to discover these proposals. As illustrated Figure 2, during training, we only sample mini-group of images from the concept group as inputs in consideration of efficiency. In the mini-group, we iteratively choose one image as query image, and leave the rest as support images. Note that the regional proposals for each image are cached to avoid re-computation when swapping query and support images. The basic idea to discover co-occurring objects the query image from region proposals that have close neighbors across the support images. To fulfill this purpose, we introduce text-guided similarity estimation and similarity-basse prototype discovery the following paragraphs.
Similarity- prototype discovery. Since modern visual backbones provide consistent feature correspondences for visually similar regions across images [58, 20, 47, 24, 60, 1]. a straightforward way to identify co-occurring objects to measure cosine similarity between features of region proposals. Concretely, we calculate the pairwise similarity of region proposals between the query image and the support images. This yields similarity matrix E Raxmn where n stands for the number of proposals per image, and m stands for the number of support images. Intuitively,

---

co-occurring regions should exhibit high responses (similarities) in the last dimension of S. But instead of using hand-crafted rules as in [41], we employ a two-layer MLP, denoted as D, to derive co-occurrence from S. Φ is trained to estimate the probability of each region proposal in the query images as a co-occurring region, solely conditioned on S. Here, we do not explicitly supervise the output probability since there is no ground-truth annotation, but + is encouraged to assign high probabilities for co-oceurring regions to minimize the overall region-word alignment loss. Based on the estimated probability vector p E I", we obtain the prototypical region features for the co-occurring object via simple weighted sum:
N Σ P
P. f, where p softmax ( (S))
(I)
As the text label for this prototype naturally corresponds to the shared concept c the mini-group. We can thus learn region-word alignment with binary cross-entropy (BCE) classification loss:
Trgiinn word
LUCE Wf pppp c), where LsCE c) -log (s.))
Σ
log (1 (sx)).
(2)
where W the classifier weight derived from Copen, and (-) stands for the sigmoid function.
Text-guided region-region similarity estimation. However, general cosine similarity may not always truly reflect closeness of objects in the semantic space, as objects of the same category may exhibit significant variance in appearance. Moreover, there could exist multiple co-occurring concepts among the sampled images, which incurs ambiguity in identifying co-occurrence. To address the problems, we introduce text guidance into similarity estimation to make it concept-aware. Concretely, given features of two region proposals f,, f, E Rd, where d the dimension of feature vectors, we additionally introduce w, E Rdd the text embedding of concept e (the shared concept in mini-group), to re-weight similarity calculation:
f,
fy [|E
where W. Vdd |We] [[e]
Sij
W.
where "o" denotes Hadamard product, "I-1" denotes absolute value operation, and "I|-I|" denotes fg-normalization, respectively. Here, the rationale that the relative magnitude of text features at different dimensions indicates their relative importance in the classification of concept c. By weighting the image feature similarities with the text feature magnitudes, the similarity measurement can put more emphasis on feature dimensions that reflect more of the text features. Therefore, the re-we weighted similarity metric provides a more nuanced and tailored measure of the proximity between objects in the context of a particular concept. It is noteworthy that we choose regional features from the output the penultimate layer of the classification head (the last layer is the text embedding) so that they naturally reside in the shared feature space text embeddings.
3.4 Training and Inference
Following [61, 28], we train the model simultaneously detection data and image-text pairs to acquire localization capability and knowledge of vision-language alignments. addition to learning region-level alignments from region-word pairs discovered in Sec. 3.3, we treat image-text pairs as a generalized form of region-wort pairs to learn image-level alignments. Particularly, we use a region proposal covering the entire image to extract image features, and encode the entire caption into language embeddings. Similar to CLIP [35], we consider each image and its original caption as positive pair and other captions in the same batch negative pairs. We then use BCE loss similar to Eq. (2) to calculate the image-text matching loss Limuge-text The overall training objective for this framework is:
Crpa + Lreg + Lole:
if I E D if E Deup
C(T)
4
Eregion-word + Limage-texts
where Lrpn: Lreg, Lets are standard losses in the two-stage detector. For inference, CoDet does not require cross-image correspondence modeling as training. behaves like normal two-stage object detector by forming the classifier with arbitrary language embeddings.
5

---

Experiments
4.1 Benchmark Setup
OV-LVIS general benchmark for open- open-vocabulary object detection, built upon LVIS [19] dataset which contains diverse set of 1203 categories of objects with long-tail distribution. Following standard practice [18, 59], we the 866 common and frequent categories in LVIS as base categories, and leave the 337 rare categories as novel categories. Besides, we choose CC3M [40] which contains 2.8 million free-from image-text pairs crawled from the web, as the source of image-text pairs. The main evaluation metric on OV-LVIS is the mask AP of novel (rare) categories.
OV-COCO is derived from the popular COCO [29] benchmark for evaluation of zero-shot and open- vocabulary object detection methods [2, 57J. It splits the categories of COCO into 48 base categories and novel categories, while removing the categories without a synset the WordNet [32]. As for image-capp caption data, following existing works [57, 61, 28], we use COCO Caption [7] training set which provides 5 human- gemaangeeeeetedd captions for each image for experiments on OV-COCO. The main evaluation metric on OV-COCO is the box AP50 of novel categories.
4.2
Implementation Details
We extract object concepts from the text corpus of COCO Caption/CC3M using an off-the-shelf language parser [32]. Remarkably, filter out concepts without synset in WordNet outside the scope of the "object' definition (i.e., not under the hierarchy 'object' synset in WordNet) to clean the extracted concepts. For phrases of more than word, simply apply the filtering logic the last word the phrase. Subsequently, we remove concepts with a frequency lower than 100/20 in COCO- CiCO-Caption/CC3MM This leaves 634/4706 concepts for COCO/CC3M.
For experiments on OV-LVIS, unless otherwise specified, we use CenterNet2 [62] with ResNet50 as the backbone, following 28]. For OV-COCO, Faster R-CNN with ResNet50-C4 [22] backbone adopted. achieve faster convergence, we initialize the model with parameters from the base class detection pre-training as in [61, 28]. The batch size single GPU is set to 2/8 for COCO/LVIS detection data and 8/32 for COCO Caption/CC3M caption data. The ratio between the detection batch and caption batch is set to 1:1 during co oo-tranning. Notably, caption batch by default contains four mini-g mini-groups, groups, where each mini-group is constructed by sampling 2/8 image-text pairs from the same concept group in COCO Caption/CC3M. We train the model for 90k iterations GPUs.
Table 1: Comparison with state-of-the-ale open-vocabulary object detection methods on OV- LVIS. Caption supervision means the method learns rision-languag alignment from image- text pairs, while CLIP supervision indicates transferring knowledge from pre- trained CLIP. The column 'Strict' indicates whether the method follows strict open-vocayulary setting.
Method
Backbone
Supervision
Strict
APm novel
APM
APm
APM
ViLD [18]
RN50-FPN
CLIP
16.6
24.6
30.3
25.5
RegionCLIIP [59]
RN50-C4
Caption
/
17.1
27.4
34.0
28.2
DetPro [12]
RN50-FPN
CLIP
/
19.8
25.6
28.9
25.9
OV-DETR [56]
RN50-C4
Caption
X
17.4
25.0
32.5
26.6
PromptDet [14]
RN50-FPN
Caption
X
19.0
18.5
25.8
21.4
Detic [61]
RN50
Caption
X
19.5
30.9
F-VLM [26]
RN50-FPN
CLIP
18.6
VLDet [28]
RN50
Caption
21.7
29.8
34.3
30.1
BARON [50]
RN50-FPN
CLIP
22.6
27.6
29.8
27.6
CoDet (Ours)
RN50
Caption
23.4
30.0
34.6
30.7
RegionCLIIP [59]
R50x4 (87M)
Caption
/
22.0
32.1
36.9
32.3
Detic [61]
SwinB (88M)
Caption
X
23.9
40.2
42.8
38.4
F-VLM [26]
R50x4 (87M)
CLIP
26.3
28.5
VLDet [28]
SwinB (88M)
Caption
26.3
39.4
41,9
38.1
CoDet (Ours)
SwinB (88M)
Caption
29.4
39.5
43.0
39.2
F-VLM [26]
R50x64 (420M)
CLIP
32.8
34.9
CoDet (Ours)
EVA02-L (304M)
Caption
37.0
46.3
46.3
44.7

---

4.3
Benchmark Results
Table presents results OV-LVIS. We follow strict open open-vocabulary setting where novel categories are kept unknown during training, to ensure we obtain generic vocabulayy detector biased towards specific novel categories. It can be seen that CoDet consistently outperforms SoTA methods novel object detection. Especially, among the group of methods learning from caption supervision, CoDet surpasses all alternatives which rely CLIP (RegionCLIP [59], OV-DETR(56)) max-size prior (Detic [61 D. or self-trained VLM (PromptDet [14], VLDet [28]) to generate pseudo region-text pairs, demonstrating the superiority of visual guidance in region-text alignment.
In addition, we validate the scalability of CoDet by testing with more powerful visual backbones, i.e., Swin-B B [31] and EVA02-L [13]. It turns out that our method scales up surprisingly well with model capacity it leads to a +6.0/13.6 AP performance boost by switching from ResNet50 to Swin-B/EVA02-LL and continuously enlarges the performance gains the second best method with a comparable model size. We believe this is because stronger visual representations provide more consistent semantic correspondences across images, which are critical for discovering co-occurring objects among the concept group.
Table 2 presents our results on OV-COCO, where CoDet achieves the second best performance among existing methods. Compared with the leading method VL.eet, the underperformance of CoDet can be mainly attributed to the images in COCO Caption contain at least one of the 80 categories in COCO, which leads to highly concentrated concepts. For instance, roughly 1/2 of the images contain 'people', and 1/10 of the images contain 'car'. This unavoidably incurs many hard negatives for identifying co-occurring objects of interest. Ablation studies the concept group size of CoDet in Table reveals the same problem. But we believe this would not harm the generality our method as we show CoDet works well on web-crawled data (e.g., CC3M), which more practical setting and easily be scaled up.
bias in COCO Caption data distribution. That is,
Table 2: Comparison with state-of-the- art methods on OV-cOCO. I; imple- mented with Deformable DETR [64].
Table 3: Cross-datasets transfer detection from OV-LVIS to COCO and Objects365. Detection- specialized pre-training with SoCo [48].
Method
A paevel
Ap
AP
COCO
Objects365
Method
OVR-CNI [57]
22.8
46.0
39.9
AP
AP50
AP75
AP
AP50
AP75
ViLD [18]
27.6
59.5
51.3
Supervised [18]
46.5
67.6
50.9
25.6
38.6
28.0
RegionCLIP [59]
26.8
54.8
47.5
Detic [61]
27.8
47.1
42.0
ViLD [18]
36.6
55.6
39.8
11.8
12.6
OV-DETR [56]*
29.4
61.0
52.7
DetPro [12]*
34.9
53.8
37.4
12.1
18.8
12.9
PB-OVD [15]
29.1
44.4
40.4
F.VLM [26]
32.5
53.1
34.6
11.9
19.2
12.6
VLDet
32.0
45.8
BARON [50]
36.2
55.7
39.1
13.6
21.0
14.5
CoDet (Ours)
30.6
52.3
46.6
CoDet (Ours)
39.1
57.0
42.3
14.2
20.5
15.3
4.4 Transfer to Other Datasets
In simulation detection in the open world, where test data may come from different domains, we conduct cross-dataset transfer detection experiments in Table 3. Specifically, we transfer the open-vocabulary detector trained OV-LVIS (LVIS base CC3M) COCO and Objects365 v1, by plugging in the vocabulary of test datasets without further model fine-tuning. In comparison with existing works, CoDet outperforms the second-best method ViLD [18] which uses a 32x training schedule by 2.0% and 2.1% 311% AP on COCO and Objects365, validating the generalization capability of CoDet across image domains and vocabularies.
4.5 Visualization and Analysis
Discovering reliable region- word alignments is critical learn object-level vision-language represen- tations from image-text pairs. In this section, we investigate different types of alignment strategies that are primarily based on: 1) region-word similarity, i.e., assigning words to regions of the highest similarity [38, 14]; 2) hand-crafted prior, i.e., assigning words to regions of the maximum size [61];
7

---

and 3) region-region similarity, i.e., assigning words to regions of the maximum weight, derived by CoDet from region-region similarity matrix (one may refer Figure for clarity).
rate
Figure shows comparison of these strategies. Specifically, we employ the model trained with different strategies on OV-COCO benchmark to generate pseudo-labels for novel categories in COCO validation set. Note that the pseudo- labels generated with the same strategy in training, not by prediction results. We evaluate the quality of ossudo-labelle is defined as the ratio of pseudo-labels whose assigned region has mloU 0.5 with the clos- est ground-truth box. At the end of training, we can see that alignments based on region-region similarity produce much more accurate pseudo labels compared with the other two strategies, manifesting the reliability of visual guidance. An- other intriguing finding is that, our method can benefit from self eeff-training which leads to steadily increasing pseudo-label quality, while such pattern is not observed in similarly self-trained model relying on region-word woord similarity for alignment. We conjecture this is because the model gets stuck in the aforementioned chicken-and-ege problem, which is reflected in its unstable training curve (see Appendix B). This also aligns with the finding in [61] $. Further qualitative comparisons are presented in Figure 4.
Iterations
cover rate, which
ragion simi
1og
Figure 3: A comparison of different alignment strategies on OV-COCO. Cover rate is the ratio of assigned proposal covering the ground-truth box.
Figure 4: Visualization of pseudo bounding box labels generated by different region-word word alignment strategies. From top to bottom, each row shows results of strategies based on region-region similarity, region-wort similarity, and hand-crafted prior, respectively. Zoom in for better view.
4.6 Ablation Study
Text-guidance. We ablate the impact of text guidance in estimating inter-region similarity OV- COCO. As shown Table 4a, introducing text guidance leads to a significant performance boost on novel AP. This finding aligns with intuition that putting region similarity estimation in semantic context can provide better measurements semantic closeness. Figure 5 further demonstrates how text guidance facilitates mitigating interference from irrelevant concepts.
'Detic shows that matching regions and words based on highest region-word similarity produces highly inconsistent pseudo labels different training stages.

---

Table Ablation study on effective components. We show that both text guidance and prototype- based strategy substantially facilitate co- co-occurring object discovery.
(a) Text guidance
(b) Strategy for co-occurring object discovery
Text guide
AP50
Strategy
AP
AP
50
x
26.6
52.4
45.7
Heuristic [41]
26.9
52.4
45.7
30.6
52.3
46.6
Prototype-based
30.6
52.3
46.6
Heuristic vs. Prototype-based co-occurring object discovery. To investigate the effectiveness of prototype-based strategy co-occurring object discovery, we adopt a heuristic strategy image co-segmentation [41] for comparison. Results are presented in Table 4b. The heuristic strategy works select single region proposal that has close neighbors across support images following hand-crafted rules (please refer to Appendix A for more details). Our prototype-basee paaeed strategy makes a substantial improvement over this simple baseline by 3.7 AP on novel categories. We speculate gains mainly come from two aspects: 1) robustness noisy similarity estimations; We notice that some region proposals in the background may be estimated with high similarity in the early stage, which disturbs the selection by the heuristic strategy. While our prototype-batd based strategy avoids hard selection by assigning soft weights to each region proposal to construct a prototype, thus is more robust to such noises. 2) the ability to harness multiple instances; considering that there may be multiple instances corresponding to the shared concept in an image, prototype-basee strategy can effectively make use of region proposals of different instances to construct prototype, which show in Appendix C).
w/o text guidancc
w/ text guidance "pigeon"
Figure 5: There can be more than co-occurring concept among sampled images. Text guidance helps filter out the distracting concept (chair legs) and focus on the concept of interest (pigeons).
Size of concept group. In CoDet, co-occurring object discovery is based on a mini-group of images sampled from the same concept group during training. Since the size of the mini-group is generally small, sometimes there could be more than one co-occurring concept in group, in which case the model will be confused about which concept discover, illustrated in Figure 5. Besides introducing text guidance, intuitively, increasing the group size can also effectively reduce this ambiguity, verified by results on OV-LVIS in Table However, contrary results are observed in experiments on OV-COCO. We speculate these abnormal results are probably caused by the aforementioned human-curated bias in COCO Caption (See Sec. 4.3). Due to the highly concentrated concepts, increasing the group size will undesirably introduce more concurrent concepts that harm the model performances.
Table 5: Ablation study on concept group size. CoDet shows different preferences concept group size on human-curated caption data (OV-COCO) and web-crawled image-text pairs (OV-LVIS).
OV-COCO
OV-LVIS
Group Size
A
Aphase
APAI
Ap novel
AP
AP
AP
30.6
52.3
46.6
21.9
30.3
35.0
30.7
29.9
51.2
45.6
21.8
30.2
34.9
30.6
29.1
50.9
45.2
22.7
30.3
34.7
30.7

---

Limitations and Conclusions
In this paper, we make the first attempt to explore visual clues, iee. object co-occurrence, to discover region-word alignments for open-vocabulary object detection. We present CoDet, which effectively leverages cross-image region correspondences and text guidance to discover co-occurring objects for alignment, achieving state-of-the-art results various OVD benchmarks. On the other hand, our method is orthogonal to previous efforts in aligning regions and words with VLMs. Combining the advantages of both sides is a promising direction of research but under-explored here. We leave this for further investigation.
This work has been supported by Hong Kong Research Grant Council Early Early Career Scheme (Grant No. 27209621). General Research Fund Scheme (Grant No. 17202422). RGC Theme- based research (T45-701/22-R) R) and RGC Matching Fund Scheme (RMGS), Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.
Acknowledgements
10

---

References
[1] Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C. Berg. Point-level region contrast for object detection pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16061-16078 2022. 3,4
[2] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 384 884-400, 2018.
2,
[3] Hakan Bilen, Marco Pedersoli, and Tinne Tuytelaars. Weakly supervised object detection with convex clustering. In Proceedingy of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1081-1089 2015. 3
[4] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2846-2854, 2016. 3
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey 7agonuyko. End-to-end object detection with transformers. In Computer Vis 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020.
[6] Peixian Chen, Kekai Sheng, Mengdan Zhang, Yunhang Shen, Li, and Chunhua Shen. Open vocahulary object detection with proposal mining and prediction equalization. arXiv preprint arXiv:2206. 1344, 2022.
[7] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: arXiv:1504.000 1504.00354, 2015. 6
[8] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picic: Unsupervised semantic segmentation using invariance and equivariance in clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16794-16804, 2021. 3
[9] Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. Weakly supervised object localization with multi-fold multiple instance lcarning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(1):189-203, 2016. 3
[10] Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinble Cinbis. Zero-shot object detection by hybrid region embedding. In British Machine Vision Conference (BMVC), 2018. 2
[11] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Kai Fei-Fei. Imagenet: large-scall hierarchical image datahase. 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 248-255. IEFF, 2009. 3
[12] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084-14093, 2022. 6,7
[13] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. Eva-02: visual representation for neon genesis. arXiv preprint 2xXiv22303.13331 2023. 7
[14] Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards ooca-vocabulary detection using uncurated images. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, page 701-717, 2022. 3, 6,7
[15] Mingfei Gao, Chen Xing. Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X, pages 266-282. Springer, 2022. 1,7
[16] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9918-2928, 2021. 17
[17] Ian Goodfellow, Jean Pouget-Abade Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1)):139
144, 2020.
11

---

[18] Xiuye Gu. Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022. 3, 6,
[19] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis; A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pages 5356- 5364, 2019.
[20] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsu- pervised semantic segmentation by distilling feature correspondences. In International Conference on Learning Representations, 2022. 3,4
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017. 1,3
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[23] Olivier Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle. Andrew Zisserman, João Carreira, and Relja Arandjelovié. Object discovery and representation networks. In Computer Vision- ERoon-ECVVRO 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pages 123-143, 2022. 3
[24] Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16291-16301, 2021. 3.
[25] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021. 1,3
[26] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni. and Anclia Angelova. Open-vocabulary object detection upon frozen vision and language models. In The Eleventh International Conference on Learning Representations, 2023. 3, 6, 7, 16
[27] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang. et al. Grounded language-image pre-training. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10965- 10975, 2022. 3,3
[28] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfci Cai. Learning object-language alignments for open- vocabulary object detection. In The Eleventh International Conference on Learning Representations, 2023. 3. 7.7
[29] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th Furopean Conference, Zurich, Swirzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014. 1,6
[30] Weide Liu, Chi Zhang, Guosheng Lin, and Fayao Liu. Crnet: Cross-reference networks for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4173-4173, 2020. 3
[31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10022-0022,, 2021. 7
[32] George Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39-41,
1995. 6
[33] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang. Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocahulary object detection. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X, page 728-755, 2022. 3
[34] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532- 1543, 2014. 2
12

---

[35] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Muchine Learning, pages 3748-8763. PMLR, 2021. 1, 2, 3, 4,5
[36] Shafin Rahman, Salman Khan, and Nick Barnes. Transductive learning for zero-shot object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6082-6091, 2019. 2
[37] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 779-788,220 2016. 1
[38] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7263-722 7271, 2017.
[39] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmm: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[40] Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2556-2565, 2018. 6
[41] Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. In Advances in Neural Information Processing Systems, volume 35, pages 33754-33767, 2022. 3, 3,5, 9, 15
[42] Zhao Shizhen, Gao Changxin, Shao Yuanjie. Lerenhan, Yu Changqian, Ji Zhong, and Sang Nong. Ginet: Generative transfer network for zero-shot object detection. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020. 3
[43] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cmn; End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14454- 14463, 2021.
[44] Sara Vicente. Vladimir Kolmogorov, and Carsten Rother. Cosegmentation revisited: Models and optimiza- tion. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II I, pages 465-479. Springer, 2010.
[45] Sara Vicente, Carsten Rother, and Vladimir Kolmogorov. Object cosegmentation. In CVPR 20/ 1, pages 2217-2224. IEEE, 2011. 3
[46] Fang Wan, Chang Liu, Wei Ke. Xiangyang Ji, Jianbin Jiao, and Qixiang Ye. C-mil: Continuation multiple instance learning for weakly supervised object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2199222992222 20082200,22222222 2019. 3
[47] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Exploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7303-7313, 2021. 3,4
[48] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. Advances in Neural Information Processing Systems, 24:22682-22694. 2021. 7
[49] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. In Advances in Neural Information Processing Systems, volume 35, pages 16423-16438, 2022."
[50] Size Wu, Wenwei Zhang. Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open- open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15264,,, 2023. 3, 6, 7, 16
[51] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEFE/CVF International Conference on Computer Vision, 2023. 1
[52] Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open- vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2023.
13

---

[53] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang. Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enrictte visual-concept paralleled pre-training for open-works detection. Advances in Neural Information Processing Systems, volume 35, pages 9125-9138, 2022.
[54] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-lext foundation models. Transactions on Machine Learning Research, 2022. 3
[55] Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li. Chunyuan Li, et al. Florence: new foundation model for computer vision. arXiv preprint axii:2111.11432, 2021. 3
[56] Yuhang Zang, Wei Li. Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part IX, pages 106-122. Springer, 2022. 3, 6,
[57] Alireza Zareian. Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang, Open-vocabulary object detection using captions. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 14393-14402, 2021. 3,3,6, 6,7
[58] Feihu Zhang, Philip Tomr René Ranftl, and Stephan Richter. Looking beyond single images for contrastive semantic segmentation learning. Advances Neural Information Processing Systems, volume 34, pages 3285-3297, 2021. 3,
[59] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-basee bandd language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-
16803, 2022. 1, 3, 7,
[60] Tianfci Zhou, Wenguan Wang, Ender Konukoglu, and Lue Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 2582-2593, 2022. 3,4
[61] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In Computer Vixion-ECCV 2022: 17th European Confer- ence, Tel Aviy, Israel, October 23 27. 2022, Proceedings, Part IX. pages 350-368. Springer, 2022. 2, 2, 5,
6, 7, 8, 17
[62] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint arXiv:203.00745 2021. 6, 17
[63] Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Don't even look once: Synthesizing features for zero-shot detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1693--11702, 2020. 3
[64] Xizhou Zhu, Weijie Su, Lewei La, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations. 2021. 7
14

---

A A Heuristic Baseline for Co-occurrence Discovery
In this section, we introduce the baseline method used for ablation study in Table 4b in more detail. This baseline is adapted from a recently proposed image co-segmentation method ReCo [41]. As shown in Figure 6, it basically consists of four steps identify the co-occurring object in the query image: First, it estimates pair-wise region similarity between region proposals of the query image and support images, which is the same CoDet. This yields similarity matrix S Raxmx, where n stands for the number of proposals per image, and m stands for the number of support images. Second, it applies neighbor in each support image for each region proposal the query image. This reduces S to an m matrix. Third, applies a mean operator on the second dimension of S to derive the average support that each proposal has among the support images. Finally, it identifies the co-occurring object the one with the highest average maximum similarity (support) among support images, by applying an argmax operator on the first dimension of S.
operator the last dimension of S, which serves to find the nearest
0,
m
P
34
IaaX
mean
argmax
K region proposal
5:
5.
8.
Figure 6: Illustration of the baseline method for co-occurrence discovery. Q and P are region proposals the query image and support images, respectively. the averaged maximum similarity score across support images.
B Further Analysis on Different Alignment Strategies
Complementing the discourse in Section 4.5, further delineate the performance of different alignment strategies with respect to novel category A0 on OV-COCO in Figure 7. It can be seen that strategies based region-region similarity or hand-crafted rules (max-size) show steady improvement in novel object recognition across training, whereas the performance region-word similarity- -based method is highly unstable and even decreases in the early stage. A possible explanation is that solely relying on region-word similarity to align regions and words may be more susceptible to errors in pseudo-labels. For instance, if the model incorrectly matches the text label 'seagull' with the object "dove' at the initial phase, its supervision signal would pull the two closer in the shared feature space. This negative feedhack could directly harm the following pseudo-labeling process, thus, there is higher probability for the model to make the same mistake.
Novel NovelAP% AP%
0.1x lr
26
20
16
14
10
10k
20k
30k
40k
50k
60k
70k
80k
90k
region-region
region-werd
Figure 7: Performance of different aligmment strategies discrete training stages OV-COCO.
15

---

C Visualization on OV-LVIS and OV-COCO
We visualize more detection results CoDet in Figure and Figure 9. On OV-LVIS, we can see that CoDet successfully detects many rare objects, e.g., gas mask, puffin, horse buggy, heron, satchel, and so on (Figure 8). This validates that CoDet can efficiently leverage web-crawled image-text pairs to learn open-word knowledge for novel object recognition. On OV-COCO, our method continues demonstrate strong open-vocabulary capability and correctly detects some hard samples, e.g the occluded 'tie' and 'elephant' (upper left of Figure 9). Nevertheless, we also notice that the prediction scores for novel categories are generally lower than base categories, which suggests the model is biased towards base classes in OV-COCO. Such tendency overfit base categories is also observed in other works [59, 26, 50], due to the small training vocabulary of OV-COCO. We believe adopting tricks like focal loss could alleviate this issue and further benefit our method.
Figure 8: Visualization of prediction results by CoDet on OV-LVIS. For clarity, we only show results for novel categories.
Figure 9: Visualization of prediction results by CoDet OV-COCO. Red boxes are for novel categories, while blue boxes are for base categories.
16

---

D Implementation Details
Table lists the detailed paprm-rrrreeter configuration used for our OV-LVIS and OV-COCO experiments. We follow Detic [61] low input resolution and large batch size for caption data to achieve better trade-off between efficiency and performance.
Table 6: Hyper-parameter configuration of CoDet. LSJ stands for large scale jittering [16].
Resolution refers the resized short side length of input images.
Configuration
OV-LVIS ov-coCo
Optimizer
AdamW
SGD
Gradient clipping
True
True
Learning rate (LR)
2c-4
2c-2
Total iterations
90k
90k
Warmup iterations
lk
Step decay factor
0.1x
Step decay schedule
[60k, 80k]
Data augmentation
LSJ
none
Batch size (detection)
8
Batch size (caption)
32
Detection/Caption data ratio
1:4
1:4
Federated loss [62]
True
False
Repeat factor sampling
True
False
region-word weight
0.2
0.1
C imnge-sext weight
0.2
0.1
17


Reviews:
[{"review_id": "4nH3yLUJIj", "reviewer": "Reviewer_Puz3", "summary": "In this paper, they propose CoDet, a novel approach that overcomes the reliance on pre-aligned vision-language space by reformulating\nregion-word alignment as a co-occurring object discovery problem. Specially, CoDet groups images that mention the same concept in their captions, which brings a natural correspondence between the shared concept and the common objects within the group through co-occurrence. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects.", "questions": "1. The image-text pairs currently used are relatively clean. It is hypothetical that if data with larger noise, such as YFCC, are used, the results might differ. In their images, the content described in many texts does not align with the images. Would the algorithm proposed in the paper have limitations in such cases? Furthermore, how efficient is the proposed algorithm? For instance, when expanded to larger datasets like YFCC26M, how does it affect the increase in training time and the stability of performance improvement?\n2. Suppose there are two noun  that your models have never learned similar concepts (where text guidance is ineffective), and the objects corresponding to these two nouns always appear simultaneously, such as eyes and eyelashes. Would it be difficult for this framework to distinguish between them, or is all existing methods fundamentally incapable of solving such issues?\n3. Regarding the technical details in lines 160-161, it appears that multiple prototypes and text losses are computed simultaneously in a single forward pass, followed by iterative selection of the query image, correct?\n4. How sensitive is this framework to RPN? For example, in the case of Figure 2, if the RPN proposes a dog's head, is there a possibility that the dog's head might be recognized as a complete dog?", "limitations": "The limitations are addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper propose an interesting idea to reformulate region-word alignment as a co-occurring object discovery problem.\n2. Experiments show its effectivenesses.", "weaknesses": "1. Experimentation Missing, Need to compare with the fine-grained word-region alignment method proprosed in DetCLIPv2[1], another OVD method try to learn with image-text pair more efficiently with region-word loss.\n2. Some more questions, see the questions.\n\n[1] DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Puz3,\n\nThanks a lot for your insightful reviews and support for our work! We hope our response can address your questions.\n1. Need comparison with DetCLIPv2\n  - Thanks for pointing out this missing **contemporary work**. We will add the discussion of DetCLIPv2 into the Related Work section of our paper. \n  - CoDet and DetCLIPv2 are not readily comparable at this point due to different training and evaluation protocols (DetCLIPv2 follows the setting in GLIP [1], while CoDet follows the setting in ViLD [2]). In detail, for training, DetCLIPv2 uses 5x more data (roughly 15M image-text pairs and 1.44M region-text pairs) than CoDet. For evaluation, DetCLIPv2 uses the full LVIS rather than OV-LVIS which splits LVIS into seen/unseen categories. \n  - We follow the setting in ViLD due to computational resource constraints. Moreover, as there is no open-sourced code of DetCLIPv2, reproducing and evaluating DetCLIPv2 using the same data as CoDet is not feasible, given limited rebuttal time. \n  - Finally, from the high-level insight, our effort in reformulating region-word alignment as co-occurring object discovery are orthogonal to the efforts in DetCLIPv2, which explores aligning regions and words directly. \n2. Dealing with noisier image-text pairs (e.g., YFCC)\n  - This is quite an intriguing point of observation. Typically, poor quality of training image-text pairs degrades the model performance. This is **a common issue** for pseudo-label-based OVD methods, e.g., VLDet [3] also has related discussions in Appendix 4. Failure Cases.\n  - CoDet is designed to be relatively robust to such noises. The minimum requirement for discovering co-occurring objects is that the target object exists in the majority of images within a concept group, though not necessarily in all images. \n  - In our early experiments, we attempted to filter out noisy samples, e.g., if an image did not have proposals with sufficiently close neighbors across the majority of images in the mini-group, it will be considered not including the target concept, thus being removed. This design didn't bring notable performance gains yet introduced unnecessary complexities. Thus, we discarded this design. But in case of noisier image-text pairs are used for training, e.g., YFCC, this in-place filtering method might prove beneficial. We may explore this in the future. \n3. Training efficiency.\n  - The training time of CoDet grows linearly with the amount of training data. This means CoDet can be easily scaled to web-scale data.\n4. Stability of performance improvement when scaled to larger datasets.\n  - This is a good point worth future studies. In our experiments with the OV-COCO setting (small-scale) and the OV-LVIS setting (large-scale), we see a significant performance improvement over baselines in both settings. This is a positive sign that CoDet can scale up to larger datasets.\n5. How to distinguish co-existing concepts?\n  - Yes, this can be a systematic failure for existing OVD methods to distinguish co-existing concepts that have never been seen before. Co-existing concepts will incur high ambiguities in associating concepts and regions. For example, in methods based on region-word similarity matching such as VLDet, the ambiguity lies in associating regions with corresponding text supervision. For our method, the ambiguity lies in associating objects discovered in a group with corresponding co-occurring concepts.\n  - But we think this is not a failure of algorithms but the failure of data -- even humans could hardly distinguish two unseen concepts that \"always appear concurrently\" without any prior knowledge of them. We believe a solution to this problem is to scale up training data. Given sufficient data, hopefully, we can find samples that break up the concurrency of the two concepts. Another is to inject priors for learning more discriminative embeddings of the two concepts.\n6. Clarification on technical details in lines 160-161\n  - The query image is set before computing prototypes and text losses. The full training pipeline can be summarized as follows: \n  - sample images from the same concept group -> extract region proposals for each image -> iteratively set an image as the query image -> compute similarity matrix between proposals of the query image and support images -> synthesize a prototype for the query image -> compute text (classification) loss\n  - Note that although we use \"iteratively\" to describe this process of query image selection, prototype synthesis is actually independent of each other. This means in practice, prototype synthesis can be executed simultaneously for efficiency.\n7. Is there a possibility that the dog's head might be recognized as a complete dog?\n  - There are two possible scenarios. If the image only contains a part of the dog, e.g., dog head, CoDet will recognize the dog head as a dog, which is expected behavior. If the image contains a complete dog, the proposal of dog head will be suppressed by the proposal of a complete dog in NMS, which is often the case. Therefore, it is okay for the RPN to generate some part-level proposals, as long as we can handle them in post-processing, i.e. NMS. Moreover, when conducting visualization, we did not find the misclassification of object parts as objects to be a systematic issue of CoDet.\n\n&nbsp;\n&nbsp;\n\nReferences: \n\n[1] Grounded Language-Image Pre-training\n\n[2] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation\n\n[3]  Learning Object-Language Alignments for Open-Vocabulary Object Detection"}, {"review_id": "WY298oDaid", "reviewer": "Reviewer_2fPU", "summary": "This paper argues that existing OVD work relies heavily on visual language pre-training tasks, and current methods cannot provide more fine-grained cross-modal alignment information for OVD, resulting in limited performance on open vocabulary detection tasks.\nTo this end, the authors propose to adopt co-occurrence guided region-word alignment for the OVD task.\nThis method is novel and interesting, and has a good reference value for fields such as OVD and open world segmentation.\nOverall, the writing of this article is clear and the content is comprehensive. But I think the author still needs a clearer motivation, formulation and explanation for how to discover the co-occurrence proposal in the image.", "questions": "1.How the author avoids the problem of large differences in the visual appearance of target objects in different images under the same concept mentioned in line 170.\n\n2.In line 189, why should the co-occurrence region appear in the last dimension of S?\n\n3.In line 177, the text feature should be an embedding of a sentence, why does the author say that the relative size of text features in different dimensions indicates their relative importance in concept classification?\n\n", "limitations": "The authors provide an objective discussion of the limitations of the article.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "As expressed in the summary, the ideas in this paper are novel and interesting, and the writing is clear and complete.", "weaknesses": "In this paper, the motivation and solution of how to obtain the co-occurrence proposal corresponding to the text from the mini-group image is not clear. For example, \n- How the author avoids the problem of large differences in the visual appearance of target objects in different images under the same concept mentioned in line 170.\n- In addition, the author's description of the change in the shape of the similarity matrix in figure2 is too late (in line 185), and it may actually be more appropriate to arrange it in line 166. This caused some difficulty in understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2fPU,\n\nThank you very much for your constructive comments which helped improve our manuscript, as well as your support for acceptance. We hope our response can address your concerns.\n1. Dealing with large intra-class variance of object appearances\n  - We adopt text guidance to suppress the impacts of intra-class visual variations. \n  - First, we observe that intra-class variations are typically reflected in particular visual attributes. For example, some object classes have large variances in color, and some have large variances in texture. Those visual attributes of large intra-class variance are naturally considered uninformative in the classification of that class.\n  - Second, in the shared vision-language space, the class text embeddings are forced to have high similarity scores with visual features from the same class, agnostic to the visual feature variance during contrastive learning. This encourages the text embeddings to assign high weights to dimensions corresponding to visual attributes of low intra-class variance, and assign low weights to those corresponding to attributes of high variance.\n  - We thus leverage the learned class text embeddings for feature selection (see Equation) to emphasize invariant features and suppress features with high variations. For instance, suppose the visual feature of \"dog\" has high variance at dimension 0 but low variance at dimensions 1 & 2, we would expect the text embedding of \"dog\" to be something like [0.1, -0.6, 0.7], where 0.1 indicates the visual attribute at dimension 0 is unimportant in the classification of \"dog\". Based on this observation, we use the absolute value of text embeddings to reweight similarity estimation (Equation 1). Thus, intra-class variations are suppressed by small weights from corresponding text embeddings.\n2. Move description of the similarity matrix calculation upward\n  - Thanks for your valuable feedback to enhance the work! We agree that it would be more logically coherent to put the introduction of \"similarity-based prototype discovery\" before \"text guidance\" as the former constitutes the major part of our method. We will rearrange the layout of the paper as suggested.\n3. In line 189, why should the co-occurrence region appear in the last dimension of S?\n  - This is because the last dimension of S essentially includes the similarity scores between the query proposal and region proposals from the support images. Among these scores, the co-occurring region proposals are characterized by having high values.\n4. How does the relative magnitude of text features at different dimensions indicate their relative importance in concept classification?\n  - We hope our discussion in question 1 addresses this question as well."}, {"review_id": "aeBf2PlUm3", "reviewer": "Reviewer_CkPP", "summary": "The paper proposes a novel perspective in discovering region-word correspondence from image-text pairs, which bypasses the dependence on a pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem. An open-vocabulary object detection framework named CoDet is built and achieves state-of-the-art performance across multiple standard benchmarks. The proposed method also demonstrates the effectiveness of visual guidance in region-word alignment. Experimental results validate the effectiveness of the method.", "questions": "Please see above.", "limitations": "The authors addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to understand. The idea of using co-occurrent regions as visual guidance is interesting. Experimental results are promising.", "weaknesses": "1. During prototype construction, how to ensure that the prototype does not contain noisy information such as hard negative samples.\n\n2. In line 16, the authors claim that the proposed method can benefit from advancements in visual foundation models. However, no further experiments about different visual foundation models are carried out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CkPP,\n\nThanks so much for your constructive comments and support for acceptance. We hope our response can address your concerns.\n1. How to ensure that the prototype does not contain noisy information such as hard negative samples?\n  - This is a very good question. Since we are aggregating all region proposals of an image into a prototype, we are not able to ensure 100% purity of the prototype. But we found our learning-based prototype synthesis is relatively robust to the noisy samples. From our observation, most of the proposals got weights smaller than 0.01 in prototype synthesis, which suggests the algorithm automatically learns from the similarity matrix to suppress noisy information. \n  - As for hard negative samples, text guidance serves as an effective method to filter them (only proposals corresponding to the target concept will get a high similarity score). Figure 5 gives an example of how text guidance filters hard negatives.\n2. No further experiments on visual foundation models.\n  - We would like to clarify that the original content in line 16 is \"CoDet exhibits its potential to benefit from advancements in visual foundation models\", which is based on two facets: \n    - Theoretically, CoDet mostly relies on visual correspondences to identify co-occurring objects, thus robust visual features from foundation models would benefit CoDet in finding more accurate co-occurrences. \n    - Empirically, our experiments show that adopting stronger visual backbones (Resnet50 -> SwinB, Table 1) leads to significant performance boost (+6.7 mAP on novel classes).\n\n    For the aforementioned reasons, we believe using stronger visual foundation models as backbone would possibly continue to bring further performance gains.\n  - We totally agree that adding experiments on visual foundation models is a great idea to improve our work. Therefore, we are running OV-LVIS experiments using EVA-02 [1] pre-trained model, which achieves 90.0% top-1 accuracy on ImageNet-1k val. As this experiment takes a long time, we will post the results later during the author-reviewer discussion period.\n\n&nbsp;\n&nbsp;\n\nReferences:\n\n[1] EVA-02: A Visual Representation for Neon Genesis"}, {"review_id": "7gtvO6Ed8K", "reviewer": "Reviewer_Z3qw", "summary": "This paper proposes CoDet, a novel approach for open-vocabulary object detection that reformulates region-word alignment as a co-occurring object discovery problem. The approach groups images that mention the same concept in their captions, leveraging region-region correspondences to discover the common object and adopt the shared concept as category label for open-vocabulary supervision. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects.", "questions": "1. The fairness for comparison cannot be promised. For LVIS, CenterNet2 is used in this paper. But in previous methods, FasterRCNN is usually used. \n\n2. Some implement details are confusing. Why using CenterNet2 [62] with ResNet50 as the backbone for LVIS?. But for OV-COCO, Faster R-CNN with a ResNet50-C4 backbone is adopted.  Is this fair to compare with previous methods? The same backbone should be used for fair comparison.\n\n3. Some OVD methods are not compared in this paper. e.g., \"Open-vocabulary detr with conditional matching.\"\n\n\n", "limitations": "The author have addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Introducing a new method, CoDet, for solving the open-vocabulary object detection problem, which reformulates region-word alignment as a co-occurring object discovery problem, achieving more accurate localization and better generalization ability.\n2. Experimental results demonstrate that CoDet outperforms existing state-of-the-art methods in detecting novel objects and exhibits strong scalability, benefiting from advancements in visual foundation models.\n3. The authors provided open-source code, which ensures the reproducibility of experiments and promotes further research.\n4. The paper is well-organized, with a clear structure and logical flow.\n", "weaknesses": "1. The method is simple, and prototype-based solutions are common in other tasks. I am slightly concerned about the novelty of the approach.\n\n\n2.  There are  fairness concerns in this paper. See Questions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Z3qw,\n\nWe really appreciate your comments. We hope our response can address your concerns and clarify our contribution.\n1. Novelty concern\n  - We believe method simplicity should not harm paper novelty. In contrast, without redundant design, simple and effective methods usually provide clear and concise insights into the research problem.\n  - Besides, CoDet contributes a new perspective to address the OVD problem. As confirmed by all other reviewers, the idea of reformulating region-word alignment as a co-occurring object discovery problem is unique and interesting, which overcomes the reliance on pre-trained or self-trained vision-language models for alignment. Reviewer 2fPU further confirms this idea has reference values for future works within and beyond OVD research.\n  - While prototypes are commonly used in other tasks like few-shot learning, CoDet is fundamentally different from these methods in the way and insight of constructing and using prototypes. For prototype construction, rather than relying on annotated samples, we introduce similarity-based prototype synthesis to automatically discover prototypes for target objects. Besides, CoDet does not directly use prototypes for classification. Instead, prototypes are only intermediate products to learn vision-language alignment. For inference, CoDet no more needs prototypes.\n2. Fairness concern in comparison\n  - First, we would like to point out that \"using CenterNet2 for LVIS experiments and Faster R-CNN for COCO experiments\" is not an \"innovation\" of this paper. As put in line 234, we follow exactly the same setting as Detic [1], a well-known OVD work (167 citations by now). VLDet [2] also adopted this setting.\n  - Second, we choose to follow the Detic setting because it is a mature setting that has been widely acknowledged by the community. It can be seen that most of, if not all, recent OVD works include Detic results in their comparisons, e.g., PromptDet [3], F-VLM [4], VLDet, CORA [5].\n  - Third, it is hard to find a universal standard for implementation. Discrepancy in implementation is common among different OVD works. For instance, ViLD [6] uses a 32x training schedule (CoDet only uses 4x), DetPro [7] and BARON [8] use detection-specialized pretraining SoCo[9] for weight initialization, OV-DETR [10] uses Deformable-DETR, not mention that many works do not even follow the strict open-vocabulary setting.\n3. \"Open-vocabulary detr with conditional matching\" is not compared in this paper.\n  - OV-DETR has already been included in comparison (Please see Table 1 & 2).  \n\n&nbsp;\n&nbsp;\n&nbsp;\n\nReferences:\n\n[1] Detecting Twenty-thousand Classes using Image-level Supervision\n\n[2] Learning Object-Language Alignments for Open-Vocabulary Object Detection\n\n[3] PromptDet: Towards Open-vocabulary Detection using Uncurated Images\n\n[4] F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models\n\n[5] CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching\n\n[6] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation\n\n[7] Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model\n\n[8] Aligning Bag of Regions for Open-Vocabulary Object Detection\n\n[9] Aligning pretraining for detection via object-level contrastive learning.\n\n[10] Open-vocabulary detr with conditional matching."}, {"review_id": "iFjYtVLTPa", "reviewer": "Reviewer_e66K", "summary": "The paper presents CoDet, a novel approach for region-word alignment in vision-language representations for open-vocabulary object detection. Unlike existing methods that rely on pre-trained or self-trained models, CoDet reformulates alignment as a co-occurring object discovery problem. By grouping images that mention the same concept, CoDet establishes correspondences between shared concepts and common objects through co-occurrence, enabling it to leverage region-region correspondences across images for object discovery and open-vocabulary supervision. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects, while also showcasing scalability with visual representations, indicating its potential for benefiting from advancements in visual foundation models.", "questions": "Please refer to the weaknesses mentioned above.", "limitations": "The authors have adequately addressed the limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+  In terms of originality, the authors propose a novel approach, CoDet, which reformulates region-word alignment as a co-occurring object discovery problem, diverging from the reliance on pre-trained or self-trained vision-language models. This unique perspective brings a fresh perspective to the field. \n+ The quality of the paper is commendable, as the authors provide a clear description of the proposed approach, detailing how CoDet groups images based on shared concepts, leverages co-occurrence for region-region correspondences, and utilizes open-vocabulary supervision. The experimental results consistently outperform state-of-the-art methods, validating the effectiveness of CoDet in detecting novel objects. \n+ The clarity of the paper is also notable, as the authors provide a concise and coherent presentation of their work, making it easily understandable to readers. \n+ The significance of the paper lies in its potential impact on the vision-language representation field. By addressing the limitations of existing methods, CoDet opens up new possibilities for reliable region-word alignment and object-level vision-language representations, which can benefit various applications such as open-vocabulary object detection.", "weaknesses": "- The paper would benefit from a more explicit discussion of its differences and a comparative analysis with related work [a]. By highlighting the distinctions between the proposed CoDet approach and the existing method [a], the authors can provide a clearer understanding of the unique contributions and advantages of their approach. \n[a] Aligning Bag of Regions for Open-Vocabulary Object Detection. CVPR, 2023.\n- The absence of VLDet in Tables 2 and 3 raises concerns and it is important for the authors to provide an explanation for its omission. Including VLDet in the comparative analysis is crucial to assess its performance against the proposed CoDet approach and other existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer e66K,  \nThank you for your appreciation of our approach and constructive comments. We address your comments below.\n1. Differences between Aligning Bag of Regions for Open-Vocabulary Object Detection.\n  - Thanks for pointing out this missing related work. We will add the discussion of BARON[1] into the Related Work section of our paper. \n  - BARON and CoDet might look similar as both works take advantage of \"co-occurrence\". However, the meaning of \"co-occurrence\" in CoDet is quite different from that in BARON. In CoDet, \"co-occurrence\" refers to the existence of the same object class across different images. While in BARON, \"co-occurrence\" refers to the existence of different object classes within the same image. Such a difference is further reflected in the motivation. BARON proposes to align \"bag of regions\" with \"bag of concepts\", while individual region-word alignment is not a main focus. In contrast, CoDet is making orthogonal efforts to discover single region-word pairs. \n  - Moreover, BARON is a distillation-based method, which still relies on a teacher VLM and inherits constraints from the image-level pre-trained VLM. While CoDet overcomes such dependency by introducing a new region-word alignment mechanism.\n  - CoDet performs on par with BARON on OV-LVIS benchmark (22.7 v.s. 22.7 mAPr), and outperforms BARON in transfer detection from OV-LVIS to COCO and Objects365. The transfer detection results are presented below. Notably, the results of CoDet on Objects365 are different from those reported in the paper because here we use Objects365 v2 val for evaluation instead of Objects365 v1, to keep in accordance with BARON. We directly cite the number reported in BARON. \n| Dataset     |          |   COCO   |          |          | Obj365 |          |\n|:------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|  Method      |    AP    |   AP$_{50}$   |   AP$_{75}$   |    AP     |   AP$_{50}$   |   AP$_{75}$   |\n| BARON | 36.2 | 55.7 | 39.1 | 13.6 | **21.0** | 14.5 |\n| CoDet | **38.5** | **55.8** | **41.5** | **14.5** | 20.6 | **15.7** |\n\n2. The absence of VLDet in Tables 2 and 3.\n  - Thanks for your suggestions. We will add VLDet to Table 2 & 3 for comparison.\n  - We primarily benchmarked and analyzed our method on OV-LVIS in the paper as we believe OV-LVIS results are more representative than OV-COCO results because: \n    - OV-LVIS has many more novel categories for evaluation (337 in OV-LVIS v.s. 17 in OV-COCO)\n    - OV-LVIS setting uses large-scale web-crawled caption data for training, which better simulates real-word practice (OV-COCO setting only uses 120K human-annotated caption data for training).\n  - It is noteworthy to point out that VLDet has superior performance over CoDet on OV-COCO (32.0 v.s. 30.6 AP50 on novel classes). We have already presented an analysis of potential causes in our paper. This could be attributed to:\n    - The human-curated bias in COCO Caption data distribution. As analysed in lines 317-323, concepts in COCO Caption images are highly concentrated, which incurs many hard negatives for identifying co-occurring objects. But we believe this would not harm the generailty of our method as we show CoDet works well on web-crawled data, which is a more practical setting and can easily be scaled up. \n    - VLDet has more aggressive trade-offs between novel AP and base AP on OV-COCO. CoDet actually has a higher base AP (52.3 v.s. 50.6) and overall AP (46.6 v.s. 45.8) than VLDet. This trade-off typically happens in methods trained on the detection and caption data simultaneously, e.g., Detic, VLDet, CoDet.\n  - We did not include VLDet for comparison in Table 3 because the original paper did not report their transfer detection results on OV-LVIS to Objects365 and COCO. Here for comparison, we use the officially released checkpoint of VLDet for evaluation.\n  - It can be seen that CoDet outperforms VLDet on large-sclae Objects365 v2 transfer detection, showing its higher capability for detecting a wide range of novel concepts. However, an interesting finding is that VLDet has stronger transfer detection results on COCO, compared with CoDet and BARON. This is not as expected, given that VLDet has a bit lower mAPr on OV-LVIS than CoDet and BARON. It seems VLDet has unique advantages on common object detection. The deep reason behind it is still unclear and worth further studies.\n| Dataset          |          |   COCO   |          |          | Obj365 |          |\n|:------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|  Method      |    AP    |   AP$_{50}$   |   AP$_{75}$   |    AP     |   AP$_{50}$   |   AP$_{75}$   |\n|  VLDet | **39.7** | **56.9** | **43.2** |   12.8   |    18.0    |   13.9   |\n|  CoDet |   38.5   |   55.8   |   41.5   | **14.5** |  **20.6**  | **15.7** |\n \n&nbsp;\n&nbsp;\n\nReferences:\n\n[1] Aligning Bag of Regions for Open-Vocabulary Object Detection"}]

Author rebuttal:
Dear Reviewers and ACs:

Thank you so much for your time and efforts in assessing our paper. Hope our rebuttal has addressed your concerns. We are happy to further discuss with you if there are still other concerns. Thanks for helping improve our paper.

Best regards, Paper 2038 Authors

Meta-review: This paper proposes a co-occurrence guided region-word alignment approach for Open-Vocabulary Object Detection tasks. All reviewers give positive scores after the rebuttal. The newly added results on LVIS and obj365 look convincing. The AC decides to accept it.
Decision: Accept (poster)