SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison with state-of-the-art open-vocabulary object detection methods on OV-LVIS. Caption supervision means the method learns vision-language alignment from image-text pairs, while CLIP supervision indicates transferring knowledge from pre-trained CLIP. The column 'Strict' indicates whether the method follows a strict open-vocabulary setting.
Method	Backbone	Supervision	Strict	APmnovel	APmc	APfm	APmal
ViLD [18]	RN50-FPN	CLIP	✓	16.6	24.6	30.3	25.5
RegionCLIP [59]	RN50-C4	Caption	✓	17.1	27.4	34.0	28.2
DetPro [12]	RN50-FPN	CLIP	✓	19.8	25.6	28.9	25.9
OV-DETR [56]	RN50-C4	Caption	✗	17.4	25.0	32.5	26.6
PromptDet [14]	RN50-FPN	Caption	✗	19.0	18.5	25.8	21.4
Detic [61]	RN50	Caption	✗	19.5	-	-	30.9
F-VLM [26]	RN50-FPN	CLIP	✓	18.6	-	-	24.2
VLDet [28]	RN50	Caption	✓	21.7	29.8	34.3	30.1
BARON [50]	RN50-FPN	CLIP	✓	22.6	27.6	29.8	27.6
CoDet (Ours)	RN50	Caption	✓	23.4	30.0	34.6	30.7
RegionCLIP [59]	R50x4 (87M)	Caption	✓	22.0	32.1	36.9	32.3
Detic [61]	SwinB (88M)	Caption	✗	23.9	40.2	42.8	38.4
F-VLM [26]	R50x4 (87M)	CLIP	✓	26.3	-	-	28.5
VLDet [28]	SwinB (88M)	Caption	✓	26.3	39.4	41.9	38.1
CoDet (Ours)	SwinB (88M)	Caption	✓	29.4	39.5	43.0	39.2
F-VLM [26]	R50x64 (420M)	CLIP	✓	32.8	-	-	34.9
CoDet (Ours)	EVA02-L (304M)	Caption	✓	37.0	46.3	46.3	44.7
4.3 Benchmark Results
Table 1 presents our results on OV-LVIS. We follow a strict open-vocabulary setting where novel categories are kept unknown during training, to ensure we obtain a generic open-vocabulary detector not biased towards specific novel categories. It can be seen that CoDet consistently outperforms SoTA methods in novel object detection. Especially, among the group of methods learning from caption supervision, CoDet surpasses all alternatives which rely on CLIP (RegionCLIP [59], OV-DETR[56]), max-size prior (Detic [61]), or self-trained VLM (PromptDet [14], VLDet [28]) to generate pseudo region-text pairs, demonstrating the superiority of visual guidance in region-text alignment.
In addition, we validate the scalability of CoDet by testing with more powerful visual backbones, i.e., Swin-B [31] and EVA02-L [13]. It turns out that our method scales up surprisingly well with model capacity - it leads to a  $+6.0 / 13.6\mathrm{AP}_{\mathrm{novel}}^{\mathrm{m}}$  performance boost by switching from ResNet50 to Swin-B/EVA02-L, and continuously enlarges the performance gains over the second best method with a comparable model size. We believe this is because stronger visual representations provide more consistent semantic correspondences across images, which are critical for discovering co-occurring objects among the concept group.
Table 2 presents our results on OV-COCO, where CoDet achieves the second best performance among existing methods. Compared with the leading method VLDet, the underperformance of CoDet can be mainly attributed to the human-curated bias in COCO Caption data distribution. That is, images in COCO Caption contain at least one of the 80 categories in COCO, which leads to highly concentrated concepts. For instance, roughly 1/2 of the images contain 'people', and 1/10 of the images contain 'car'. This unavoidably incurs many hard negatives for identifying co-occurring objects of interest. Ablation studies on the concept group size of CoDet in Table 5 reveals the same problem. But we believe this would not harm the generality of our method as we show CoDet works well on web-crawled data (e.g., CC3M), which is a more practical setting and can easily be scaled up.
Table 2: Comparison with state-of-the-art methods on OV-COCO. †: implemented with Deformable DETR [64].
Method	\(AP_{50}^{novel}\)	\(AP_{50}^{base}\)	\(AP_{50}^{all}\)
OVR-CNN [57]	22.8	46.0	39.9
ViLD [18]	27.6	59.5	51.3
RegionCLIP [59]	26.8	54.8	47.5
Detic [61]	27.8	47.1	42.0
OV-DETR \([56]^{\dagger}\)	29.4	61.0	52.7
PB-OVD [15]	29.1	44.4	40.4
VLDet	32.0	50.6	45.8
CoDet (Ours)	30.6	52.3	46.6
Table 3: Cross-datasets transfer detection from OV-LVIS to COCO and Objects365. †: Detection-specialized pre-training with SoCo [48].
Method	COCO	Objects365
AP	\( AP_{50} \)	\( AP_{75} \)	AP	\( AP_{50} \)	\( AP_{75} \)
Supervised [18]	46.5	67.6	50.9	25.6	38.6	28.0
ViLD [18]	36.6	55.6	39.8	11.8	18.2	12.6
\( DetPro [12]^\dagger \)	34.9	53.8	37.4	12.1	18.8	12.9
F-VLM [26]	32.5	53.1	34.6	11.9	19.2	12.6
BARON [50]	36.2	55.7	39.1	13.6	21.0	14.5
CoDet (Ours)	39.1	57.0	42.3	14.2	20.5	15.3
4.4 Transfer to Other Datasets
In simulation to detection in the open world, where test data may come from different domains, we conduct cross-dataset transfer detection experiments in Table 3. Specifically, we transfer the open-vocabulary detector trained on OV-LVIS (LVIS base + CC3M) to COCO and Objects365 v1, by plugging in the vocabulary of test datasets without further model fine-tuning. In comparison with existing works, CoDet outperforms the second-best method ViLD [18] which uses a  $32 \times$  training schedule by  $2.0\%$  and  $2.1\%$  AP on COCO and Objects365, validating the generalization capability of CoDet across image domains and vocabularies.
4.5 Visualization and Analysis
Discovering reliable region-word alignments is critical to learn object-level vision-language representations from image-text pairs. In this section, we investigate different types of alignment strategies that are primarily based on: 1) region-word similarity, i.e., assigning words to regions of the highest similarity [38, 14]; 2) hand-crafted prior, i.e., assigning words to regions of the maximum size [61];
and 3) region-region similarity, i.e., assigning words to regions of the maximum weight, derived by CoDet from region-region similarity matrix (one may refer to Figure 2 for clarity).
Figure 3: A comparison of different alignment strategies on OV-COCO. Cover rate is the ratio of assigned proposal covering the ground-truth box.
Figure 3 shows a comparison of these strategies. Specifically, we employ the model trained with different strategies on OV-COCO benchmark to generate pseudo-labels for novel categories in COCO validation set. Note that the pseudolabels are generated with the same strategy as in training, not by prediction results. We evaluate the quality of pseudo-labels by cover rate, which is defined as the ratio of pseudo-labels whose assigned region has a mIoU  $>0.5$  with the closest ground-truth box. At the end of training, we can see that alignments based on region-region similarity produce much more accurate pseudo
labels compared with the other two strategies, manifesting the reliability of visual guidance. Another intriguing finding is that, our method can benefit from self-training which leads to steadily increasing pseudo-label quality, while such pattern is not observed in similarly self-trained model relying on region-word similarity for alignment. We conjecture this is because the model gets stuck in the aforementioned chicken-and-egg problem, which is reflected in its unstable training curve (see Appendix B). This also aligns with the finding in [61]  ${}^{3}$  . Further qualitative comparisons are presented in Figure 4.
Figure 4: Visualization of pseudo bounding box labels generated by different region-word alignment strategies. From top to bottom, each row shows results of strategies based on region-region similarity, region-word similarity, and hand-crafted prior, respectively. Zoom in for a better view.
4.6 Ablation Study
Text-guidance. We ablate the impact of text guidance in estimating inter-region similarity on OV-COCO. As shown in Table 4a, introducing text guidance leads to a significant performance boost on novel AP. This finding aligns with our intuition that putting region similarity estimation in a semantic context can provide better measurements of semantic closeness. Figure 5 further demonstrates how text guidance facilitates mitigating interference from irrelevant concepts.
3Detic shows that matching regions and words based on highest region-word similarity produces highly inconsistent pseudo labels at different training stages.
Table 4: Ablation study on effective components. We show that both text guidance and prototype-based strategy substantially facilitate co-occurring object discovery.
(a) Text guidance
Text guide	APnovel50	APbase50	APall50
X	26.6	52.4	45.7
✓	30.6	52.3	46.6
(b) Strategy for co-occurring object discovery
Strategy	APnovel50	APbase50	APall50
Heuristic [41]	26.9	52.4	45.7
Prototype-based	30.6	52.3	46.6
Heuristic vs. Prototype-based co-occurring object discovery. To investigate the effectiveness of prototype-based strategy in co-occurring object discovery, we adopt a heuristic strategy in image co-segmentation [41] for comparison. Results are presented in Table 4b. The heuristic strategy works to select a single region proposal that has close neighbors across support images following hand-crafted rules (please refer to Appendix A for more details). Our prototype-based strategy makes a substantial improvement over this simple baseline by 3.7 AP on novel categories. We speculate the gains mainly come from two aspects: 1) robustness to noisy similarity estimations; We notice that some region proposals in the background may be estimated with high similarity in the early stage, which disturbs the selection by the heuristic strategy. While our prototype-based strategy avoids hard selection by assigning soft weights to each region proposal to construct a prototype, thus is more robust to such noises. 2) the ability to harness multiple instances; considering that there may be multiple instances corresponding to the shared concept in an image, prototype-based strategy can effectively make use of region proposals of different instances to construct a prototype, which we show in Appendix C).
w/o text guidance
Figure 5: There can be more than one co-occurring concept among sampled images. Text guidance helps filter out the distracting concept (chair legs) and focus on the concept of interest (pigeons).
w/ text guidance "pigeon"
Size of concept group. In CoDet, co-occurring object discovery is based on a mini-group of images sampled from the same concept group during training. Since the size of the mini-group is generally small, sometimes there could be more than one co-occurring concept in a group, in which case the model will be confused about which concept to discover, as illustrated in Figure 5. Besides introducing text guidance, intuitively, increasing the group size can also effectively reduce this ambiguity, as verified by results on OV-LVIS in Table 5. However, contrary results are observed in experiments on OV-COCO. We speculate these abnormal results are probably caused by the aforementioned human-curated bias in COCO Caption (See Sec. 4.3). Due to the highly concentrated concepts, increasing the group size will undesirably introduce more concurrent concepts that harm the model performances.
Table 5: Ablation study on concept group size. CoDet shows different preferences of concept group size on human-curated caption data (OV-COCO) and web-crawled image-text pairs (OV-LVIS).
Group Size	OV-COCO	OV-LVIS
APnovel50	APbase50	APall50	APmnovel	APmc	APmf	APmalll
2	30.6	52.3	46.6	21.9	30.3	35.0	30.7
4	29.9	51.2	45.6	21.8	30.2	34.9	30.6
8	29.1	50.9	45.2	22.7	30.3	34.7	30.7
5 Limitations and Conclusions
In this paper, we make the first attempt to explore visual clues, i.e., object co-occurrence, to discover region-word alignments for open-vocabulary object detection. We present CoDet, which effectively leverages cross-image region correspondences and text guidance to discover co-occurring objects for alignment, achieving state-of-the-art results on various OVD benchmarks. On the other hand, our method is orthogonal to previous efforts in aligning regions and words with VLMs. Combining the advantages of both sides is a promising direction of research but is under-explored here. We leave this for further investigation.
Acknowledgements This work has been supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), RGC Theme-based research (T45-701/22-R) and RGC Matching Fund Scheme (RMGS). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.
[53] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. In Advances in Neural Information Processing Systems, volume 35, pages 9125–9138, 2022. 1
[54] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 3
[55] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 3
[56] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 106-122. Springer, 2022. 3, 6, 7
[57] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393-14402, 2021. 1, 3, 6, 7
[58] Feihu Zhang, Philip Torr, René Ranftl, and Stephan Richter. Looking beyond single images for contrastive semantic segmentation learning. In Advances in Neural Information Processing Systems, volume 34, pages 3285-3297, 2021. 3, 4
[59] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Lianian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022. 1, 3, 6, 7, 16
[60] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2582-2593, 2022. 3, 4
[61] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kähenbuhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 350-368. Springer, 2022. 2, 3, 5, 6, 7, 8, 17
[62] Xingyi Zhou, Vladlen Koltun, and Philipp Krahenbuhl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461, 2021. 6, 17
[63] Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Don't even look once: Synthesizing features for zero-shot detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11693-11702, 2020. 3
[64] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. 7
A A Heuristic Baseline for Co-occurrence Discovery
In this section, we introduce the baseline method used for ablation study in Table 4b in more detail. This baseline is adapted from a recently proposed image co-segmentation method ReCo [41]. As shown in Figure 6, it basically consists of four steps to identify the co-occurring object in the query image: First, it estimates pair-wise region similarity between region proposals of the query image and support images, which is the same as CoDet. This yields a similarity matrix  $\mathbf{S} \in \mathbb{R}^{n \times m \times n}$ , where  $n$  stands for the number of proposals per image, and  $m$  stands for the number of support images. Second, it applies a max operator on the last dimension of  $\mathbf{S}$ , which serves to find the nearest neighbor in each support image for each region proposal in the query image. This reduces  $\mathbf{S}$  to an  $n \times m$  matrix. Third, it applies a mean operator on the second dimension of  $\mathbf{S}$  to derive the average support that each proposal has among the support images. Finally, it identifies the co-occurring object as the one with the highest average maximum similarity (support) among support images, by applying an argmax operator on the first dimension of  $\mathbf{S}$ .
Figure 6: Illustration of the baseline method for co-occurrence discovery.  $Q$  and  $P$  are region proposals in the query image and support images, respectively.  $S$  is the averaged maximum similarity score across support images.
B Further Analysis on Different Alignment Strategies
Complementing the discourse in Section 4.5, we further delineate the performance of different alignment strategies with respect to novel category  $\mathrm{AP}_{50}$  on OV-COCO in Figure 7. It can be seen that strategies based on region-region similarity or hand-crafted rules (max-size) show steady improvement in novel object recognition across training, whereas the performance of region-word similarity-based method is highly unstable and even decreases in the early stage. A possible explanation is that solely relying on region-word similarity to align regions and words may be more susceptible to errors in pseudo-labels. For instance, if the model incorrectly matches the text label 'seagull' with the object 'dove' at the initial phase, its supervision signal would pull the two closer in the shared feature space. This negative feedback could directly harm the following pseudo-labeling process, thus, there is a higher probability for the model to make the same mistake.
Figure 7: Performance of different alignment strategies at discrete training stages on OV-COCO.
C Visualization on OV-LVIS and OV-COCO
We visualize more detection results of CoDet in Figure 8 and Figure 9. On OV-LVIS, we can see that CoDet successfully detects many rare objects, e.g., gas mask, puffin, horse buggy, heron, satchel, and so on (Figure 8). This validates that CoDet can efficiently leverage web-crawled image-text pairs to learn open-word knowledge for novel object recognition. On OV-COCO, our method continues to demonstrate strong open-vocabulary capability and correctly detects some hard samples, e.g., the occluded 'tie' and 'elephant' (upper left of Figure 9). Nevertheless, we also notice that the prediction scores for novel categories are generally lower than base categories, which suggests the model is biased towards base classes in OV-COCO. Such tendency to overfit base categories is also observed in other works [59, 26, 50], due to the small training vocabulary of OV-COCO. We believe adopting tricks like focal loss could alleviate this issue and further benefit our method.
Figure 8: Visualization of prediction results by CoDet on OV-LVIS. For clarity, we only show results for novel categories.
Figure 9: Visualization of prediction results by CoDet on OV-COCO. Red boxes are for novel categories, while blue boxes are for base categories.
Table 6 lists the detailed hyper-parameter configuration used for our OV-LVIS and OV-COCO experiments. We follow Detic [61] to use low input resolution and large batch size for caption data to achieve better trade-off between efficiency and performance.
Table 6: Hyper-parameter configuration of CoDet. LSJ stands for large scale jittering [16]. Resolution refers to the resized short side length of input images.
Configuration	OV-LVIS	OV-COCO
Optimizer	AdamW	SGD
Gradient clipping	True	True
Learning rate (LR)	2e-4	2e-2
Total iterations	90k	90k
Warmup iterations	1k	-
Step decay factor	-	0.1×
Step decay schedule	-	[60k, 80k]
Data augmentation	LSJ	none
Batch size (detection)	8	2
Batch size (caption)	32	8
Detection/Caption data ratio	1:4	1:4
Federated loss [62]	True	False
Repeat factor sampling	True	False
Lregion-word weight	0.2	0.1
Limage-text weight	0.2	0.1