Title: CoDet: Co-occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Open-vocabulary Object Detection; Object-level Vision-Language Pretraining
Abstract: Deriving reliable region-word alignment from image-text pairs is critical to learn
object-level vision-language representations for open-vocabulary object detection.
Existing methods typically rely on pre-trained or self-trained vision-language
models for alignment, which are prone to limitations in localization accuracy or
generalization capabilities. In this paper, we propose CoDet, a novel approach
that overcomes the reliance on pre-aligned vision-language space by reformulating
region-word alignment as a co-occurring object discovery problem. Intuitively, by
grouping images that mention a shared concept in their captions, objects corresponding 
to the shared concept shall exhibit high co-occurrence among the group.
CoDet then leverages visual similarities to discover the co-occurring objects and
align them with the shared concept. Extensive experiments demonstrate that CoDet
has superior performances and compelling scalability in open-vocabulary detection,
e.g., by scaling up the visual backbone, CoDet achieves 37.0 $AP^m_{novel}$ and 
44.7 $AP^m_{all}$ on OV-LVIS, surpassing the previous SoTA by 4.2 $AP^m_{novel}$ 
and 9.8 $AP^m_{all}$. Code is available at https://github.com/CVMI-Lab/CoDet.

CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection

Abstract

Deriving reliable region-word alignment from image-text pairs is critical to learn object-level vision-language representations for open-vocabulary object detection. Existing methods typically rely on pre-trained or self-trained vision-language models for alignment, which are prone to limitations in localization accuracy or generalization capabilities. In this paper, we propose CoDet, a novel approach that overcomes the reliance on pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem. Intuitively, by grouping images that mention a shared concept in their captions, objects corresponding to the shared concept shall exhibit high co-occurrence among the group. CoDet then leverages visual similarities to discover the co-occurring objects and align them with the shared concept. Extensive experiments demonstrate that CoDet has superior performances and compelling scalability in open-vocabulary detection, e.g ., by scaling up the visual backbone, CoDet achieves 37.0 AP m novel and 44.7 AP m all on OV-LVIS, surpassing the previous SoTA by 4.2 AP m novel and 9.8 AP m all . Code is available at https://github.com/CVMI-Lab/CoDet.

1 Introduction

Object detection is a fundamental vision task that offers object-centric comprehension of visual scenes for various downstream applications. While remarkable progress has been made in terms of detection accuracy and speed, traditional detectors [39, 37, 21, 5, 43] are mostly constrained to a fixed vocabulary defined by training data, e.g ., 80 categories in COCO [29]. This accounts for a major gap compared to human visual intelligence, which can perceive a diverse range of visual concepts in the open world. To address such limitations, this paper focuses on the open-vocabulary setting of object detection [57], where the detector is trained to recognize objects of arbitrary categories.

Recently, vision-language pretraining on web-scale image-text pairs has demonstrated impressive open-vocabulary capability in image classification [35, 25]. It inspires the community to adapt this paradigm to object detection [27, 51], specifically by training an open-vocabulary detector using region-text pairs in detection or grounding annotations [27]. However, unlike free-form image-text pairs, human-annotated region-text pairs are limited and difficult to scale. Consequently, a growing body of research [59, 27, 53, 15, 28] aims to mine additional region-text pairs from image-text pairs, which raises a new question: how to find the alignments between regions and words? (Figure 1a)

Recent studies typically rely on vision-language models (VLMs) to determine region-word alignments, for example, by estimating region-word similarity [59, 27, 15, 28]. Despite its simplicity, the quality of generated pseudo region-text pairs is subject to limitations of VLMs. As illustrated in Figure 1b, VLMspre-trained with image-level supervision, such as CLIP [35], are largely unaware of localization quality of pseudo labels [59]. Although detector-based VLMs [27, 53] mitigate this issue to some extent, they are initially pre-trained with a limited number of detection or grounding concepts,

(a) Example Image-Text Pair

(b) Alignment by VLM

'a dog'

'a man'

'a frisbee'

VLM

Images-text pairs that mention 'frisbee' in their captions

(c) Alignment by co-occurrence (ours)

*For simplicity, we show only one concept group in this illustration.

Caption: 'A dog leaps to catch a

frisbee above a fallen man.'

frisbee

dog

man

frisbee

?

How to find the alignment between regions and words?

resulting in inaccurate alignments for novel concepts [61]. Furthermore, this approach essentially faces a chicken-and-egg problem: obtaining high-quality region-word pairs requires a VLM with object-level vision-language knowledge, yet training such a VLM, in turn, depends on a large number of aligned region-word pairs.

In this work, instead of directly aligning regions and words with VLMs, we propose leveraging region correspondences across images for co-occurring concept discovery and alignment, which we call CoDet. Figure 1c illustrates the idea. Our key motivation is that objects corresponding to the same concept should exhibit consistent visual similarity across images, which provides visual clues to identity region-word correspondences. Based on this intuition, we construct semantic groups by sampling images that mention a shared concept in their captions, from which we can infer that a common object corresponding to the shared concept exists across images. Subsequently, we leverage cross-image region similarity to identify regions potentially containing the common object, and construct a prototype from them. The prototype and the shared concept form a natural region-text pair, which is then adopted to supervise the training of an open-vocabulary object detector.

Unlike previous works, our method avoids the dependence on a pre-aligned vision-language space and solely relies on the vision space to discover region-word correspondences. However, there could exist multiple co-occurring concepts in the same group, and even the same concept may still exhibit high intra-category variation in appearance, in which case general visual similarity would fail to distinguish the object of interest. To address this issue, we introduce text guidance into similarity estimation between region proposals, making it concept-aware and more accurately reflecting the closeness of objects concerning the shared semantic concept.

The main contributions of this paper can be summarized as follows:

We introduce a novel perspective in discovering region-word correspondences from image-text pairs, which bypasses the dependence on a pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem.

Building on this insight, we propose CoDet, an open-vocabulary detection framework that learns object-level vision-language alignment directly from web-crawled image-text pairs.

CoDet consistently outperforms existing methods on the challenging OV-LVIS benchmark and demonstrates superior performances in cross-dataset detection on COCO and Objects365.

CoDet exhibits strong scalability with visual backbones - it achieves 23 . 4 / 29 . 4 / 37 . 0 mask APnovel with ResNet50/Swin-B/EVA02-L backbone, outperforming previous SoTA at a comparable model size by 0 . 8 / 3 . 1 / 4 . 2 mask APnovel, respectively.

2 Related Work

Zero-shot object detection (ZSD) leverages language feature space for generalization to unseen objects. The basic idea is to project region features to the pre-computed text embedding space ( e.g ., GloVe [34]) and use word embeddings as the classifier weights [2, 10, 36]. This presents ZSD with the flexibility of recognizing unseen objects given its name during inference. Nevertheless,

ZSD settings restrict training samples to come from a limited number of seen classes, which is not sufficient to align the feature space of vision and language. Although some works [63, 42] try to overcome this limitation by hallucinating novel classes using Generative Adversarial Network [17], there is still a large performance gap between ZSD and its supervised counterparts.

Weakly supervised object detection (WSD) exploits data with image-level labels to train an object detector. It typically treats an image as a bag of proposals, and assigns the image label to these proposals through multiple instance learning [4, 9, 3, 46]. By relieving object detection from costly instance-level annotations, WSD is able to scale detection vocabulary with cheaper classification data. For instance, recent work Detic [61] greatly expands the vocabulary of detectors to twenty-thousand classes by leveraging image-level supervision from ImageNet-21K [11]. However, WSD still requires non-trivial annotation efforts and has a closed vocabulary during inference.

Open-vocabulary object detection (OVD) is built upon the framework of ZSD, but relaxes the stringent definition of novel classes from 'not seen' to 'not known in advance', which leads to a more practical setting [57]. Particularly, with recent advancement of vision-language pretraining [35, 25, 55, 54], a widely adopted approach of OVD is to transfer the knowledge of pretrained vision-language models (VLMs), e.g ., CLIP [35], to object detectors through distillation [18, 50] or weight transfer [33, 26]. Despite its utility, performances of these methods are arguably restricted by the teacher VLM, which is shown to be largely unaware of fine-grained region-word alignment [59, 6]. Alternatively, another group of works utilize large-scale image-text pairs to expand detection vocabulary [57, 59, 56, 14, 27, 28, 52], sharing a similar idea as WSD. Due to the absence of regional annotations in image-caption data, these methods typically rely on pre-trained or self-trained VLMs to find region-word correspondences, which are prone to limitations in localization accuracy or generalization capabilities. Our method is orthogonal to all the aforementioned approaches in the sense that it does not explicitly model region-word correspondences, but leverage region correspondences across images to bridge regions and words, which greatly simplifies the task.

Cross-image Region Correspondence is widely explored to discover semantically related regions among a collection of images [20, 41, 45, 44, 30]. Based on the observation that modern visual backbones provide consistent semantic correspondences in the feature space [58, 20, 47, 24, 60, 1], many works take a heuristic approach [41] or use clustering algorithms [20, 8, 23, 49] for common region discovery. Our method takes inspiration from these works to discover co-occurring objects across image-text pairs, with newly proposed text guidance.

3 Method

In this section, we present CoDet, an end-to-end framework exploiting image-text pairs for openvocabulary object detection. Figure 2 gives an overview of CoDet. We first provide a brief introduction to the OVD setup (Sec. 3.1). Then we discuss how to reformulate region-word alignment as a cooccurring object discovery problem (Sec. 3.2), which is subsequently addressed by CoDet (Sec. 3.3). Finally, we summarize the overall training objectives and inference pipelines of CoDet (Sec. 3.4).

3.1 Preliminaries

Task formulation. In our study, we adopt the classical OVD problem setup as in OVR-CNN [57]. Specifically, box annotations are only provided for a predetermined set of base categories C base during training. While at the test phase, the object detector is required to generalize beyond C base to detect objects from novel categories C novel . Notably, C novel is not known in advance to simulate open-world scenarios. This implies the object detector needs to possess the capability to recognize potentially any object based on its name. To achieve this goal, we additionally leverage image-text pairs with an unbounded vocabulary C open to extend the lexicon of the object detector.

OVD framework. Our method is built on the two-stage detection framework Mask-RCNN [21]. To adapt it for the open-vocabulary setting, we follow the common practice [18, 61] to decouple localization from classification, by replacing the class-specific localization heads with class-agnostic ones that produce a single box or mask prediction for each region proposal. Besides, to enable

dog

'

A photo of a dog

'

Query image

Detector

E

T

Text

encoder

Support images

🐶

…

…

Region-Region

similarity

MLP

Text guidance

Weights

Prototype

A cat is nuzzling up

to a dog's snout

A dog is relaxing

on the couch

There are many goats

and a white dog

Region embeddings

of query image

Region embeddings

of support image

Text embeddings

of group concept

Weighted sum

Concept group

open-vocabulary classification of objects, the fixed classifier weights are substituted with dynamic text embeddings of category names, which are generated by the pre-trained text encoder of CLIP [35].

3.2 Aligning Regions and Words by Co-occurrence

Due to the absence of box annotations, a core challenge of utilizing image-text pairs for detection training is to figure out the fine-grained alignments between regions and words. To put it formally, for an image-text pair ⟨ I, T ⟩ , R = { r 1 , r 2 , ..., r | R | } denotes the set of regions proposals extracted from image I , and C = { c 1 , c 2 , ..., c | C | } denotes the set of concept words extracted from caption T . Under weak caption supervision, we can assume the presence of a concept c in T indicates the existence of at least one region r in I containing the corresponding object. However, the concrete correspondence between c and r remains unknown.

To solve the problem, we propose to explore the global context of caption data and align regions and words via co-occurrence. Specifically, for a given concept c , we construct a concept group G comprising all image-text pairs that mention c in their captions. This grouping effectively clusters images that contain objects corresponding to concept c . Consequently, if G is large enough, we can induce that the objects corresponding to concept c will be the most common objects among images in G . This natural correlation automatically aligns the co-occurring objects in G to concept c . We thus reduce the problem of modeling cross-modality correspondence (region-word) to in-modality correspondence (region-region), which we address in the next section.

3.3 Discovering Co-occurring Objects across Images

Intuitively, candidate region proposals containing the co-occurring object should exhibit consistent and similar visual patterns across the images. We thus take a similarity-driven approach to discover these proposals. As illustrated in Figure 2, during training, we only sample a mini-group of images from the concept group as inputs in consideration of efficiency. In the mini-group, we iteratively choose one image as query image, and leave the rest as support images. Note that the regional proposals for each image are cached to avoid re-computation when swapping query and support images. The basic idea is to discover co-occurring objects in the query image from region proposals that have close neighbors across the support images. To fulfill this purpose, we introduce text-guided similarity estimation and similarity-based prototype discovery in the following paragraphs.

Similarity-based prototype discovery. Since modern visual backbones provide consistent feature correspondences for visually similar regions across images [58, 20, 47, 24, 60, 1], a straightforward way to identify co-occurring objects is to measure the cosine similarity between features of region proposals. Concretely, we calculate the pairwise similarity of region proposals between the query image and the support images. This yields a similarity matrix S ∈ R n × mn , where n stands for the number of proposals per image, and m stands for the number of support images. Intuitively,

co-occurring regions should exhibit high responses (similarities) in the last dimension of S . But instead of using hand-crafted rules as in [41], we employ a two-layer MLP, denoted as Φ , to derive co-occurrence from S . Φ is trained to estimate the probability of each region proposal in the query images as a co-occurring region, solely conditioned on S . Here, we do not explicitly supervise the output probability since there is no ground-truth annotation, but Φ is encouraged to assign high probabilities for co-occurring regions to minimize the overall region-word alignment loss. Based on the estimated probability vector p ∈ R n , we obtain the prototypical region features for the co-occurring object via simple weighted sum:

As the text label for this prototype naturally corresponds to the shared concept c in the mini-group. We can thus learn region-word alignment with a binary cross-entropy (BCE) classification loss:

̸

where W the classifier weight derived from C open , and σ ( · ) stands for the sigmoid function.

Text-guided region-region similarity estimation. However, general cosine similarity may not always truly reflect closeness of objects in the semantic space, as objects of the same category may exhibit significant variance in appearance. Moreover, there could exist multiple co-occurring concepts among the sampled images, which incurs ambiguity in identifying co-occurrence. To address the problems, we introduce text guidance into similarity estimation to make it concept-aware. Concretely, given features of two region proposals f i , f j ∈ R d , where d is the dimension of feature vectors, we additionally introduce w c ∈ R d , the text embedding of concept c (the shared concept in the mini-group), to re-weight similarity calculation:

where ' ◦ ' denotes Hadamard product, ' |·| ' denotes absolute value operation, and ' ∥·∥ ' denotes ℓ 2 -normalization, respectively. Here, the rationale is that the relative magnitude of text features at different dimensions indicates their relative importance in the classification of concept c . By weighting the image feature similarities with the text feature magnitudes, the similarity measurement can put more emphasis on feature dimensions that reflect more of the text features. Therefore, the re-weighted similarity metric provides a more nuanced and tailored measure of the proximity between objects in the context of a particular concept. It is noteworthy that we choose regional features from the output of the penultimate layer of the classification head (the last layer is the text embedding) so that they naturally reside in the shared feature space as text embeddings.

3.4 Training and Inference

Following [61, 28], we train the model simultaneously on detection data and image-text pairs to acquire localization capability and knowledge of vision-language alignments. In addition to learning region-level alignments from region-word pairs discovered in Sec. 3.3, we treat image-text pairs as a generalized form of region-word pairs to learn image-level alignments. Particularly, we use a region proposal covering the entire image to extract image features, and encode the entire caption into language embeddings. Similar to CLIP [35], we consider each image and its original caption as a positive pair and other captions in the same batch as negative pairs. We then use a BCE loss similar to Eq. (2) to calculate the image-text matching loss L image-text. The overall training objective for this framework is:

where L rpn , L reg , L cls are standard losses in the two-stage detector. For inference, CoDet does not require cross-image correspondence modeling as in training. It behaves like a normal two-stage object detector by forming the classifier with arbitrary language embeddings.

4 Experiments

4.1 Benchmark Setup

OV-LVIS is a general benchmark for open-vocabulary object detection, built upon LVIS [19] dataset which contains a diverse set of 1203 categories of objects with a long-tail distribution. Following standard practice [18, 59], we set the 866 common and frequent categories in LVIS as base categories, and leave the 337 rare categories as novel categories. Besides, we choose CC3M [40] which contains 2.8 million free-from image-text pairs crawled from the web, as the source of image-text pairs. The main evaluation metric on OV-LVIS is the mask AP of novel (rare) categories.

OV-COCO is derived from the popular COCO [29] benchmark for evaluation of zero-shot and openvocabulary object detection methods [2, 57]. It splits the categories of COCO into 48 base categories and 17 novel categories, while removing the 15 categories without a synset in the WordNet [32]. As for image-caption data, following existing works [57, 61, 28], we use COCO Caption [7] training set which provides 5 human-generated captions for each image for experiments on OV-COCO. The main evaluation metric on OV-COCO is the box AP 50 of novel categories.

4.2 Implementation Details

We extract object concepts from the text corpus of COCO Caption/CC3M using an off-the-shelf language parser [32]. Remarkably, we filter out concepts without a synset in WordNet or outside the scope of the 'object' definition ( i.e ., not under the hierarchy of 'object' synset in WordNet) to clean the extracted concepts. For phrases of more than one word, we simply apply the filtering logic to the last word in the phrase. Subsequently, we remove concepts with a frequency lower than 100/20 in COCO-Caption/CC3M. This leaves 634/4706 concepts for COCO/CC3M.

For experiments on OV-LVIS, unless otherwise specified, we use CenterNet2 [62] with ResNet50 as the backbone, following [61, 28]. For OV-COCO, Faster R-CNN with a ResNet50-C4 [22] backbone is adopted. To achieve faster convergence, we initialize the model with parameters from the base class detection pre-training as in [61, 28]. The batch size on a single GPU is set to 2/8 for COCO/LVIS detection data and 8/32 for COCO Caption/CC3M caption data. The ratio between the detection batch and caption batch is set to 1:1 during co-training. Notably, a caption batch by default contains four mini-groups, where each mini-group is constructed by sampling 2/8 image-text pairs from the same concept group in COCO Caption/CC3M. We train the model for 90k iterations on 8 GPUs.

4.3 Benchmark Results

Table 1 presents our results on OV-LVIS. We follow a strict open-vocabulary setting where novel categories are kept unknown during training, to ensure we obtain a generic open-vocabulary detector not biased towards specific novel categories. It can be seen that CoDet consistently outperforms SoTA methods in novel object detection. Especially, among the group of methods learning from caption supervision, CoDet surpasses all alternatives which rely on CLIP (RegionCLIP [59], OV-DETR[56]), max-size prior (Detic [61]), or self-trained VLM (PromptDet [14], VLDet [28]) to generate pseudo region-text pairs, demonstrating the superiority of visual guidance in region-text alignment.

In addition, we validate the scalability of CoDet by testing with more powerful visual backbones, i.e ., Swin-B [31] and EVA02-L [13]. It turns out that our method scales up surprisingly well with model capacity - it leads to a +6.0/13.6 AP m novel performance boost by switching from ResNet50 to Swin-B/EVA02-L, and continuously enlarges the performance gains over the second best method with a comparable model size. We believe this is because stronger visual representations provide more consistent semantic correspondences across images, which are critical for discovering co-occurring objects among the concept group.

Table 2 presents our results on OV-COCO, where CoDet achieves the second best performance among existing methods. Compared with the leading method VLDet, the underperformance of CoDet can be mainly attributed to the human-curated bias in COCO Caption data distribution. That is, images in COCO Caption contain at least one of the 80 categories in COCO, which leads to highly concentrated concepts. For instance, roughly 1/2 of the images contain 'people', and 1/10 of the images contain 'car'. This unavoidably incurs many hard negatives for identifying co-occurring objects of interest. Ablation studies on the concept group size of CoDet in Table 5 reveals the same problem. But we believe this would not harm the generality of our method as we show CoDet works well on web-crawled data ( e.g ., CC3M), which is a more practical setting and can easily be scaled up.

4.4 Transfer to Other Datasets

In simulation to detection in the open world, where test data may come from different domains, we conduct cross-dataset transfer detection experiments in Table 3. Specifically, we transfer the open-vocabulary detector trained on OV-LVIS (LVIS base + CC3M) to COCO and Objects365 v1, by plugging in the vocabulary of test datasets without further model fine-tuning. In comparison with existing works, CoDet outperforms the second-best method ViLD [18] which uses a 32 × training schedule by 2.0% and 2.1% AP on COCO and Objects365, validating the generalization capability of CoDet across image domains and vocabularies.

4.5 Visualization and Analysis

Discovering reliable region-word alignments is critical to learn object-level vision-language representations from image-text pairs. In this section, we investigate different types of alignment strategies that are primarily based on: 1) region-word similarity, i.e ., assigning words to regions of the highest similarity [38, 14]; 2) hand-crafted prior, i.e ., assigning words to regions of the maximum size [61];

and 3) region-region similarity, i.e ., assigning words to regions of the maximum weight, derived by CoDet from region-region similarity matrix (one may refer to Figure 2 for clarity).

20%

30%

40%

50%

60%

70%

10k

30k

50k

70k

90k

Cover rate

Iterations

region-region simi.

region-word simi.

hand-crafted

Figure 3 shows a comparison of these strategies. Specifically, we employ the model trained with different strategies on OV-COCO benchmark to generate pseudo-labels for novel categories in COCO validation set. Note that the pseudolabels are generated with the same strategy as in training, not by prediction results. We evaluate the quality of pseudo-labels by cover rate, which is defined as the ratio of pseudo-labels whose assigned region has a mIoU > 0.5 with the closest ground-truth box. At the end of training, we can see that alignments based on region-region similarity produce much more accurate pseudo labels compared with the other two strategies, manifesting the reliability of visual guidance. Another intriguing finding is that, our method can benefit from self-training which leads to steadily increasing pseudo-label quality, while such pattern is not observed in similarly self-trained model relying on region-word similarity for alignment. We conjecture this is because the model gets stuck in the aforementioned chicken-and-egg problem, which is reflected in its unstable training curve (see Appendix B). This also aligns with the finding in [61] 3 . Further qualitative comparisons are presented in Figure 4.

cellphone

cellphone

cellphone

cellphone

cellphone

cellphone

dishwasher

dishwasher

dishwasher

dishwasher

dishwasher

dishwasher

4.6 Ablation Study

Text-guidance. We ablate the impact of text guidance in estimating inter-region similarity on OVCOCO. As shown in Table 4a, introducing text guidance leads to a significant performance boost on novel AP. This finding aligns with our intuition that putting region similarity estimation in a semantic context can provide better measurements of semantic closeness. Figure 5 further demonstrates how text guidance facilitates mitigating interference from irrelevant concepts.

(a)

Text guidance

(b)

Strategy for co-occurring object discovery

Heuristic vs. Prototype-based co-occurring object discovery. To investigate the effectiveness of prototype-based strategy in co-occurring object discovery, we adopt a heuristic strategy in image co-segmentation [41] for comparison. Results are presented in Table 4b. The heuristic strategy works to select a single region proposal that has close neighbors across support images following hand-crafted rules (please refer to Appendix A for more details). Our prototype-based strategy makes a substantial improvement over this simple baseline by 3.7 AP on novel categories. We speculate the gains mainly come from two aspects: 1) robustness to noisy similarity estimations; We notice that some region proposals in the background may be estimated with high similarity in the early stage, which disturbs the selection by the heuristic strategy. While our prototype-based strategy avoids hard selection by assigning soft weights to each region proposal to construct a prototype, thus is more robust to such noises. 2) the ability to harness multiple instances; considering that there may be multiple instances corresponding to the shared concept in an image, prototype-based strategy can effectively make use of region proposals of different instances to construct a prototype, which we show in Appendix C).

w/o text guidance

w/ text guidance 'pigeon'

Size of concept group. In CoDet, co-occurring object discovery is based on a mini-group of images sampled from the same concept group during training. Since the size of the mini-group is generally small, sometimes there could be more than one co-occurring concept in a group, in which case the model will be confused about which concept to discover, as illustrated in Figure 5. Besides introducing text guidance, intuitively, increasing the group size can also effectively reduce this ambiguity, as verified by results on OV-LVIS in Table 5. However, contrary results are observed in experiments on OV-COCO. We speculate these abnormal results are probably caused by the aforementioned human-curated bias in COCO Caption (See Sec. 4.3). Due to the highly concentrated concepts, increasing the group size will undesirably introduce more concurrent concepts that harm the model performances.

5 Limitations and Conclusions

In this paper, we make the first attempt to explore visual clues, i.e ., object co-occurrence, to discover region-word alignments for open-vocabulary object detection. We present CoDet, which effectively leverages cross-image region correspondences and text guidance to discover co-occurring objects for alignment, achieving state-of-the-art results on various OVD benchmarks. On the other hand, our method is orthogonal to previous efforts in aligning regions and words with VLMs. Combining the advantages of both sides is a promising direction of research but is under-explored here. We leave this for further investigation.

Acknowledgements This work has been supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), RGC Theme-based research (T45-701/22-R) and RGC Matching Fund Scheme (RMGS). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.

References

Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg. Point-level region contrast for object detection pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16061-16070, 2022. 3, 4

Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 384-400, 2018. 2, 6

Hakan Bilen, Marco Pedersoli, and Tinne Tuytelaars. Weakly supervised object detection with convex clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1081-1089, 2015. 3

Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2846-2854, 2016. 3

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020. 1

Peixian Chen, Kekai Sheng, Mengdan Zhang, Yunhang Shen, Ke Li, and Chunhua Shen. Open vocabulary object detection with proposal mining and prediction equalization. arXiv preprint arXiv:2206.11134 , 2022. 3

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015. 6

Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16794-16804, 2021. 3

Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. Weakly supervised object localization with multi-fold multiple instance learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(1):189-203, 2016. 3

Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinbis. Zero-shot object detection by hybrid region embedding. In British Machine Vision Conference (BMVC) , 2018. 2

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255. IEEE, 2009. 3

Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for openvocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14084-14093, 2022. 6, 7

Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331 , 2023. 7

Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open-vocabulary detection using uncurated images. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , page 701-717, 2022. 3, 6, 7

Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X , pages 266-282. Springer, 2022. 1, 7

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2918-2928, 2021. 17

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139144, 2020. 3

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations , 2022. 3, 6, 7

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 53565364, 2019. 6

Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In International Conference on Learning Representations , 2022. 3, 4

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision , pages 2961-2969, 2017. 1, 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016. 6

Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelovi´ c. Object discovery and representation networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII , pages 123-143, 2022. 3

Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16291-16301, 2021. 3, 4

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021. 1, 3

Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. Open-vocabulary object detection upon frozen vision and language models. In The Eleventh International Conference on Learning Representations , 2023. 3, 6, 7, 16

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1096510975, 2022. 1, 3

Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. In The Eleventh International Conference on Learning Representations , 2023. 1, 3, 5, 6, 7

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 1, 6

Weide Liu, Chi Zhang, Guosheng Lin, and Fayao Liu. Crnet: Cross-reference networks for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4165-4173, 2020. 3

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021. 7

George A Miller. Wordnet: a lexical database for english. Communications of the ACM , 38(11):39-41, 1995. 6

Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X , page 728-755, 2022. 3

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1532-1543, 2014. 2

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021. 1, 2, 3, 4, 5

Shafin Rahman, Salman Khan, and Nick Barnes. Transductive learning for zero-shot object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6082-6091, 2019. 2

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 779-788, 2016. 1

Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7263-7271, 2017. 7

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems , 28, 2015. 1

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018. 6

Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. In Advances in Neural Information Processing Systems , volume 35, pages 33754-33767, 2022. 3, 5, 9, 15

Zhao Shizhen, Gao Changxin, Shao Yuanjie, Li Lerenhan, Yu Changqian, Ji Zhong, and Sang Nong. Gtnet: Generative transfer network for zero-shot object detection. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2020. 3

Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1445414463, 2021. 1

Sara Vicente, Vladimir Kolmogorov, and Carsten Rother. Cosegmentation revisited: Models and optimization. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II 11 , pages 465-479. Springer, 2010. 3

Sara Vicente, Carsten Rother, and Vladimir Kolmogorov. Object cosegmentation. In CVPR 2011 , pages 2217-2224. IEEE, 2011. 3

Fang Wan, Chang Liu, Wei Ke, Xiangyang Ji, Jianbin Jiao, and Qixiang Ye. C-mil: Continuation multiple instance learning for weakly supervised object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2199-2208, 2019. 3

Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Exploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7303-7313, 2021. 3, 4

Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. Advances in Neural Information Processing Systems , 34:22682-22694, 2021. 7

Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. In Advances in Neural Information Processing Systems , volume 35, pages 16423-16438, 2022. 3

Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15254-15264, 2023. 3, 6, 7, 16

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023. 1

Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23497-23506, 2023. 3

Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. In Advances in Neural Information Processing Systems , volume 35, pages 9125-9138, 2022. 1

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research , 2022. 3

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 , 2021. 3

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 106-122. Springer, 2022. 3, 6, 7

Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14393-14402, 2021. 1, 3, 6, 7

Feihu Zhang, Philip Torr, René Ranftl, and Stephan Richter. Looking beyond single images for contrastive semantic segmentation learning. In Advances in Neural Information Processing Systems , volume 34, pages 3285-3297, 2021. 3, 4

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1679316803, 2022. 1, 3, 6, 7, 16

Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2582-2593, 2022. 3, 4

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twentythousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 350-368. Springer, 2022. 2, 3, 5, 6, 7, 8, 17

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461 , 2021. 6, 17

Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Don't even look once: Synthesizing features for zero-shot detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11693-11702, 2020. 3

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations , 2021. 7

A A Heuristic Baseline for Co-occurrence Discovery

In this section, we introduce the baseline method used for ablation study in Table 4b in more detail. This baseline is adapted from a recently proposed image co-segmentation method ReCo [41]. As shown in Figure 6, it basically consists of four steps to identify the co-occurring object in the query image: First, it estimates pair-wise region similarity between region proposals of the query image and support images, which is the same as CoDet. This yields a similarity matrix S ∈ R n × m × n , where n stands for the number of proposals per image, and m stands for the number of support images. Second, it applies a max operator on the last dimension of S , which serves to find the nearest neighbor in each support image for each region proposal in the query image. This reduces S to an n × m matrix. Third, it applies a mean operator on the second dimension of S to derive the average support that each proposal has among the support images. Finally, it identifies the co-occurring object as the one with the highest average maximum similarity (support) among support images, by applying an argmax operator on the first dimension of S .

𝑄

!

" 𝑃

!

𝑄

"

" 𝑃

!

𝑄

!

" 𝑃

"

𝑄

!

" 𝑃

#

…

𝑄

#

" 𝑃

!

…

𝑄

"

" 𝑃

"

𝑄

"

" 𝑃

#

𝑄

#

" 𝑃

"

𝑄

#

" 𝑃

#

…

…

…

…

…

n

m

𝑄

!

" 𝑃

$

𝑄

"

" 𝑃

%

…

𝑄

#

" 𝑃

&

n

mean

𝑆

!

…

𝑆

"

𝑆

#

argmax

K-th region

proposal

n

m

n

B Further Analysis on Different Alignment Strategies

Complementing the discourse in Section 4.5, we further delineate the performance of different alignment strategies with respect to novel category AP 50 on OV-COCO in Figure 7. It can be seen that strategies based on region-region similarity or hand-crafted rules (max-size) show steady improvement in novel object recognition across training, whereas the performance of region-word similarity-based method is highly unstable and even decreases in the early stage. A possible explanation is that solely relying on region-word similarity to align regions and words may be more susceptible to errors in pseudo-labels. For instance, if the model incorrectly matches the text label 'seagull' with the object 'dove' at the initial phase, its supervision signal would pull the two closer in the shared feature space. This negative feedback could directly harm the following pseudo-labeling process, thus, there is a higher probability for the model to make the same mistake.

10

12

14

16

18

20

22

24

26

28

30

10k

20k

30k

40k

50k

60k

70k

80k

90k

Novel AP%

Iterations

region-region simi.

region-word simi.

hand-crafted

0.1x lr

0.1x lr

C Visualization on OV-LVIS and OV-COCO

We visualize more detection results of CoDet in Figure 8 and Figure 9. On OV-LVIS, we can see that CoDet successfully detects many rare objects, e.g ., gas mask, puffin, horse buggy, heron, satchel, and so on (Figure 8). This validates that CoDet can efficiently leverage web-crawled image-text pairs to learn open-word knowledge for novel object recognition. On OV-COCO, our method continues to demonstrate strong open-vocabulary capability and correctly detects some hard samples, e.g ., the occluded 'tie' and 'elephant' (upper left of Figure 9). Nevertheless, we also notice that the prediction scores for novel categories are generally lower than base categories, which suggests the model is biased towards base classes in OV-COCO. Such tendency to overfit base categories is also observed in other works [59, 26, 50], due to the small training vocabulary of OV-COCO. We believe adopting tricks like focal loss could alleviate this issue and further benefit our method.

76

Pollice

n60%

person

eraon339%

irplonek

dog50%

D Implementation Details

Table 6 lists the detailed hyper-parameter configuration used for our OV-LVIS and OV-COCO experiments. We follow Detic [61] to use low input resolution and large batch size for caption data to achieve better trade-off between efficiency and performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents CoDet, a novel approach for region-word alignment in vision-language representations for open-vocabulary object detection. Unlike existing methods that rely on pre-trained or self-trained models, CoDet reformulates alignment as a co-occurring object discovery problem. By grouping images that mention the same concept, CoDet establishes correspondences between shared concepts and common objects through co-occurrence, enabling it to leverage region-region correspondences across images for object discovery and open-vocabulary supervision. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects, while also showcasing scalability with visual representations, indicating its potential for benefiting from advancements in visual foundation models.\n\nSTRENGTHS:\n+  In terms of originality, the authors propose a novel approach, CoDet, which reformulates region-word alignment as a co-occurring object discovery problem, diverging from the reliance on pre-trained or self-trained vision-language models. This unique perspective brings a fresh perspective to the field. \n+ The quality of the paper is commendable, as the authors provide a clear description of the proposed approach, detailing how CoDet groups images based on shared concepts, leverages co-occurrence for region-region correspondences, and utilizes open-vocabulary supervision. The experimental results consistently outperform state-of-the-art methods, validating the effectiveness of CoDet in detecting novel objects. \n+ The clarity of the paper is also notable, as the authors provide a concise and coherent presentation of their work, making it easily understandable to readers. \n+ The significance of the paper lies in its potential impact on the vision-language representation field. By addressing the limitations of existing methods, CoDet opens up new possibilities for reliable region-word alignment and object-level vision-language representations, which can benefit various applications such as open-vocabulary object detection.\n\nWEAKNESSES:\n- The paper would benefit from a more explicit discussion of its differences and a comparative analysis with related work [a]. By highlighting the distinctions between the proposed CoDet approach and the existing method [a], the authors can provide a clearer understanding of the unique contributions and advantages of their approach. \n[a] Aligning Bag of Regions for Open-Vocabulary Object Detection. CVPR, 2023.\n- The absence of VLDet in Tables 2 and 3 raises concerns and it is important for the authors to provide an explanation for its omission. Including VLDet in the comparative analysis is crucial to assess its performance against the proposed CoDet approach and other existing methods.\n\nQUESTIONS:\nPlease refer to the weaknesses mentioned above.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CoDet, a novel approach for open-vocabulary object detection that reformulates region-word alignment as a co-occurring object discovery problem. The approach groups images that mention the same concept in their captions, leveraging region-region correspondences to discover the common object and adopt the shared concept as category label for open-vocabulary supervision. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects.\n\nSTRENGTHS:\n1. Introducing a new method, CoDet, for solving the open-vocabulary object detection problem, which reformulates region-word alignment as a co-occurring object discovery problem, achieving more accurate localization and better generalization ability.\n2. Experimental results demonstrate that CoDet outperforms existing state-of-the-art methods in detecting novel objects and exhibits strong scalability, benefiting from advancements in visual foundation models.\n3. The authors provided open-source code, which ensures the reproducibility of experiments and promotes further research.\n4. The paper is well-organized, with a clear structure and logical flow.\n\nWEAKNESSES:\n1. The method is simple, and prototype-based solutions are common in other tasks. I am slightly concerned about the novelty of the approach.\n\n\n2.  There are  fairness concerns in this paper. See Questions.\n\nQUESTIONS:\n1. The fairness for comparison cannot be promised. For LVIS, CenterNet2 is used in this paper. But in previous methods, FasterRCNN is usually used. \n\n2. Some implement details are confusing. Why using CenterNet2 [62] with ResNet50 as the backbone for LVIS?. But for OV-COCO, Faster R-CNN with a ResNet50-C4 backbone is adopted.  Is this fair to compare with previous methods? The same backbone should be used for fair comparison.\n\n3. Some OVD methods are not compared in this paper. e.g., \"Open-vocabulary detr with conditional matching.\"\n\nLIMITATIONS:\nThe author have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper proposes a novel perspective in discovering region-word correspondence from image-text pairs, which bypasses the dependence on a pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem. An open-vocabulary object detection framework named CoDet is built and achieves state-of-the-art performance across multiple standard benchmarks. The proposed method also demonstrates the effectiveness of visual guidance in region-word alignment. Experimental results validate the effectiveness of the method.

STRENGTHS:
The paper is well-written and easy to understand. The idea of using co-occurrent regions as visual guidance is interesting. Experimental results are promising.

WEAKNESSES:
1. During prototype construction, how to ensure that the prototype does not contain noisy information such as hard negative samples.

2. In line 16, the authors claim that the proposed method can benefit from advancements in visual foundation models. However, no further experiments about different visual foundation models are carried out.

QUESTIONS:
Please see above.

LIMITATIONS:
The authors addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that existing OVD work relies heavily on visual language pre-training tasks, and current methods cannot provide more fine-grained cross-modal alignment information for OVD, resulting in limited performance on open vocabulary detection tasks.\nTo this end, the authors propose to adopt co-occurrence guided region-word alignment for the OVD task.\nThis method is novel and interesting, and has a good reference value for fields such as OVD and open world segmentation.\nOverall, the writing of this article is clear and the content is comprehensive. But I think the author still needs a clearer motivation, formulation and explanation for how to discover the co-occurrence proposal in the image.\n\nSTRENGTHS:\nAs expressed in the summary, the ideas in this paper are novel and interesting, and the writing is clear and complete.\n\nWEAKNESSES:\nIn this paper, the motivation and solution of how to obtain the co-occurrence proposal corresponding to the text from the mini-group image is not clear. For example, \n- How the author avoids the problem of large differences in the visual appearance of target objects in different images under the same concept mentioned in line 170.\n- In addition, the author's description of the change in the shape of the similarity matrix in figure2 is too late (in line 185), and it may actually be more appropriate to arrange it in line 166. This caused some difficulty in understanding.\n\nQUESTIONS:\n1.How the author avoids the problem of large differences in the visual appearance of target objects in different images under the same concept mentioned in line 170.\n\n2.In line 189, why should the co-occurrence region appear in the last dimension of S?\n\n3.In line 177, the text feature should be an embedding of a sentence, why does the author say that the relative size of text features in different dimensions indicates their relative importance in concept classification?\n\nLIMITATIONS:\nThe authors provide an objective discussion of the limitations of the article.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, they propose CoDet, a novel approach that overcomes the reliance on pre-aligned vision-language space by reformulating\nregion-word alignment as a co-occurring object discovery problem. Specially, CoDet groups images that mention the same concept in their captions, which brings a natural correspondence between the shared concept and the common objects within the group through co-occurrence. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects.\n\nSTRENGTHS:\n1. This paper propose an interesting idea to reformulate region-word alignment as a co-occurring object discovery problem.\n2. Experiments show its effectivenesses.\n\nWEAKNESSES:\n1. Experimentation Missing, Need to compare with the fine-grained word-region alignment method proprosed in DetCLIPv2[1], another OVD method try to learn with image-text pair more efficiently with region-word loss.\n2. Some more questions, see the questions.\n\n[1] DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment\n\nQUESTIONS:\n1. The image-text pairs currently used are relatively clean. It is hypothetical that if data with larger noise, such as YFCC, are used, the results might differ. In their images, the content described in many texts does not align with the images. Would the algorithm proposed in the paper have limitations in such cases? Furthermore, how efficient is the proposed algorithm? For instance, when expanded to larger datasets like YFCC26M, how does it affect the increase in training time and the stability of performance improvement?\n2. Suppose there are two noun  that your models have never learned similar concepts (where text guidance is ineffective), and the objects corresponding to these two nouns always appear simultaneously, such as eyes and eyelashes. Would it be difficult for this framework to distinguish between them, or is all existing methods fundamentally incapable of solving such issues?\n3. Regarding the technical details in lines 160-161, it appears that multiple prototypes and text losses are computed simultaneously in a single forward pass, followed by iterative selection of the query image, correct?\n4. How sensitive is this framework to RPN? For example, in the case of Figure 2, if the RPN proposes a dog's head, is there a possibility that the dog's head might be recognized as a complete dog?\n\nLIMITATIONS:\nThe limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}