SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chuofan Ma $^{1\dagger}$  Yi Jiang $^{2*}$  Xin Wen $^{1*}$  Zehuan Yuan $^{2}$  Xiaojuan Qi $^{1}$
<sup>1</sup>The University of Hong Kong <sup>2</sup>ByteDance Inc.
Deriving reliable region-word alignment from image-text pairs is critical to learn object-level vision-language representations for open-vocabulary object detection. Existing methods typically rely on pre-trained or self-trained vision-language models for alignment, which are prone to limitations in localization accuracy or generalization capabilities. In this paper, we propose CoDet, a novel approach that overcomes the reliance on pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem. Intuitively, by grouping images that mention a shared concept in their captions, objects corresponding to the shared concept shall exhibit high co-occurrence among the group. CoDet then leverages visual similarities to discover the co-occurring objects and align them with the shared concept. Extensive experiments demonstrate that CoDet has superior performances and compelling scalability in open-vocabulary detection, e.g., by scaling up the visual backbone, CoDet achieves  $37.0\mathrm{AP}_{\mathrm{novel}}^{\mathrm{m}}$  and  $44.7\mathrm{AP}_{\mathrm{all}}^{\mathrm{m}}$  on OV-LVIS, surpassing the previous SoTA by  $4.2\mathrm{AP}_{\mathrm{novel}}^{\mathrm{m}}$  and  $9.8\mathrm{AP}_{\mathrm{all}}^{\mathrm{m}}$ . Code is available at https://github.com/CVMI-Lab/CoDet.
Object detection is a fundamental vision task that offers object-centric comprehension of visual scenes for various downstream applications. While remarkable progress has been made in terms of detection accuracy and speed, traditional detectors [39, 37, 21, 5, 43] are mostly constrained to a fixed vocabulary defined by training data, e.g., 80 categories in COCO [29]. This accounts for a major gap compared to human visual intelligence, which can perceive a diverse range of visual concepts in the open world. To address such limitations, this paper focuses on the open-vocabulary setting of object detection [57], where the detector is trained to recognize objects of arbitrary categories.
Recently, vision-language pretraining on web-scale image-text pairs has demonstrated impressive open-vocabulary capability in image classification [35, 25]. It inspires the community to adapt this paradigm to object detection [27, 51], specifically by training an open-vocabulary detector using region-text pairs in detection or grounding annotations [27]. However, unlike free-form image-text pairs, human-annotated region-text pairs are limited and difficult to scale. Consequently, a growing body of research [59, 27, 53, 15, 28] aims to mine additional region-text pairs from image-text pairs, which raises a new question: how to find the alignments between regions and words? (Figure 1a)
Recent studies typically rely on vision-language models (VLMs) to determine region-word alignments, for example, by estimating region-word similarity [59, 27, 15, 28]. Despite its simplicity, the quality of generated pseudo region-text pairs is subject to limitations of VLMs. As illustrated in Figure 1b, VLMs pre-trained with image-level supervision, such as CLIP [35], are largely unaware of localization quality of pseudo labels [59]. Although detector-based VLMs [27, 53] mitigate this issue to some extent, they are initially pre-trained with a limited number of detection or grounding concepts,
This work was performed when Chuofan Ma worked as an intern at ByteDance.
Figure 1: Illustration of different region-text alignment paradigms. (a): example image-text pair, and region proposals generated by a pre-trained region proposal network; (b): a pre-trained VLM (e.g., CLIP [35]) is used to retrieve the box with the highest region-word similarity concerning the query text, which yet exhibits poor localization quality; (c) our method overcomes the reliance on VLMs by exploring visual clues, i.e., object co-occurrence, within a group of image-text pairs containing the same concept (e.g., frisbee). Best viewed in color.
resulting in inaccurate alignments for novel concepts [61]. Furthermore, this approach essentially faces a chicken-and-egg problem: obtaining high-quality region-word pairs requires a VLM with object-level vision-language knowledge, yet training such a VLM, in turn, depends on a large number of aligned region-word pairs.
In this work, instead of directly aligning regions and words with VLMs, we propose leveraging region correspondences across images for co-occurring concept discovery and alignment, which we call CoDet. Figure 1c illustrates the idea. Our key motivation is that objects corresponding to the same concept should exhibit consistent visual similarity across images, which provides visual clues to identity region-word correspondences. Based on this intuition, we construct semantic groups by sampling images that mention a shared concept in their captions, from which we can infer that a common object corresponding to the shared concept exists across images. Subsequently, we leverage cross-image region similarity to identify regions potentially containing the common object, and construct a prototype from them. The prototype and the shared concept form a natural region-text pair, which is then adopted to supervise the training of an open-vocabulary object detector.
Unlike previous works, our method avoids the dependence on a pre-aligned vision-language space and solely relies on the vision space to discover region-word correspondences. However, there could exist multiple co-occurring concepts in the same group, and even the same concept may still exhibit high intra-category variation in appearance, in which case general visual similarity would fail to distinguish the object of interest. To address this issue, we introduce text guidance into similarity estimation between region proposals, making it concept-aware and more accurately reflecting the closeness of objects concerning the shared semantic concept.
The main contributions of this paper can be summarized as follows:
- We introduce a novel perspective in discovering region-word correspondences from image-text pairs, which bypasses the dependence on a pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem.
- Building on this insight, we propose CoDet, an open-vocabulary detection framework that learns object-level vision-language alignment directly from web-crawled image-text pairs.
- CoDet consistently outperforms existing methods on the challenging OV-LVIS benchmark and demonstrates superior performances in cross-dataset detection on COCO and Objects365.
- CoDet exhibits strong scalability with visual backbones - it achieves 23.4/29.4/37.0 mask  $\mathrm{AP}_{\mathrm{novel}}$  with ResNet50/Swin-B/EVA02-L backbone, outperforming previous SoTA at a comparable model size by 0.8/3.1/4.2 mask  $\mathrm{AP}_{\mathrm{novel}}$ , respectively.
Zero-shot object detection (ZSD) leverages language feature space for generalization to unseen objects. The basic idea is to project region features to the pre-computed text embedding space (e.g., GloVe [34]) and use word embeddings as the classifier weights [2, 10, 36]. This presents ZSD with the flexibility of recognizing unseen objects given its name during inference. Nevertheless,
ZSD settings restrict training samples to come from a limited number of seen classes, which is not sufficient to align the feature space of vision and language. Although some works [63, 42] try to overcome this limitation by hallucinating novel classes using Generative Adversarial Network [17], there is still a large performance gap between ZSD and its supervised counterparts.
Weakly supervised object detection (WSD) exploits data with image-level labels to train an object detector. It typically treats an image as a bag of proposals, and assigns the image label to these proposals through multiple instance learning [4, 9, 3, 46]. By relieving object detection from costly instance-level annotations, WSD is able to scale detection vocabulary with cheaper classification data. For instance, recent work Detic [61] greatly expands the vocabulary of detectors to twenty-thousand classes by leveraging image-level supervision from ImageNet-21K [11]. However, WSD still requires non-trivial annotation efforts and has a closed vocabulary during inference.
Open-vocabulary object detection (OVD) is built upon the framework of ZSD, but relaxes the stringent definition of novel classes from 'not seen' to 'not known in advance', which leads to a more practical setting [57]. Particularly, with recent advancement of vision-language pretraining [35, 25, 55, 54], a widely adopted approach of OVD is to transfer the knowledge of pretrained vision-language models (VLMs), e.g., CLIP [35], to object detectors through distillation [18, 50] or weight transfer [33, 26]. Despite its utility, performances of these methods are arguably restricted by the teacher VLM, which is shown to be largely unaware of fine-grained region-word alignment [59, 6]. Alternatively, another group of works utilize large-scale image-text pairs to expand detection vocabulary [57, 59, 56, 14, 27, 28, 52], sharing a similar idea as WSD. Due to the absence of regional annotations in image-caption data, these methods typically rely on pre-trained or self-trained VLMs to find region-word correspondences, which are prone to limitations in localization accuracy or generalization capabilities. Our method is orthogonal to all the aforementioned approaches in the sense that it does not explicitly model region-word correspondences, but leverage region correspondences across images to bridge regions and words, which greatly simplifies the task.
Cross-image Region Correspondence is widely explored to discover semantically related regions among a collection of images [20, 41, 45, 44, 30]. Based on the observation that modern visual backbones provide consistent semantic correspondences in the feature space [58, 20, 47, 24, 60, 1], many works take a heuristic approach [41] or use clustering algorithms [20, 8, 23, 49] for common region discovery. Our method takes inspiration from these works to discover co-occurring objects across image-text pairs, with newly proposed text guidance.
In this section, we present CoDet, an end-to-end framework exploiting image-text pairs for open-vocabulary object detection. Figure 2 gives an overview of CoDet. We first provide a brief introduction to the OVD setup (Sec. 3.1). Then we discuss how to reformulate region-word alignment as a co-occurring object discovery problem (Sec. 3.2), which is subsequently addressed by CoDet (Sec. 3.3). Finally, we summarize the overall training objectives and inference pipelines of CoDet (Sec. 3.4).
Task formulation. In our study, we adopt the classical OVD problem setup as in OVR-CNN [57]. Specifically, box annotations are only provided for a predetermined set of base categories  $\mathcal{C}^{\text{base}}$  during training. While at the test phase, the object detector is required to generalize beyond  $\mathcal{C}^{\text{base}}$  to detect objects from novel categories  $\mathcal{C}^{\text{novel}}$ . Notably,  $\mathcal{C}^{\text{novel}}$  is not known in advance to simulate open-world scenarios. This implies the object detector needs to possess the capability to recognize potentially any object based on its name. To achieve this goal, we additionally leverage image-text pairs with an unbounded vocabulary  $\mathcal{C}^{\text{open}}$  to extend the lexicon of the object detector.
OVD framework. Our method is built on the two-stage detection framework Mask-RCNN [21]. To adapt it for the open-vocabulary setting, we follow the common practice [18, 61] to decouple localization from classification, by replacing the class-specific localization heads with class-agnostic ones that produce a single box or mask prediction for each region proposal. Besides, to enable
Figure 2: Overview of CoDet. Our method learns to jointly discover region-word pairs from a group of image-text pairs that mention a shared concept in their captions (e.g., dog in the figure). We identify the co-occurring objects and the shared concept as natural region-word pairs. Then we leverage inter-image region correspondences, i.e., region-region similarity, with text guidance to locate the co-occurring objects for region-word alignment.
open-vocabulary classification of objects, the fixed classifier weights are substituted with dynamic text embeddings of category names, which are generated by the pre-trained text encoder of CLIP [35].
3.2 Aligning Regions and Words by Co-occurrence
Due to the absence of box annotations, a core challenge of utilizing image-text pairs for detection training is to figure out the fine-grained alignments between regions and words. To put it formally, for an image-text pair  $\langle I,T\rangle$ ,  $R = \{r_1,r_2,\dots,r_{|R|}\}$  denotes the set of regions proposals extracted from image  $I$ , and  $C = \{c_1,c_2,\dots,c_{|C|}\}$  denotes the set of concept words extracted from caption  $T$ . Under weak caption supervision, we can assume the presence of a concept  $c$  in  $T$  indicates the existence of at least one region  $r$  in  $I$  containing the corresponding object. However, the concrete correspondence between  $c$  and  $r$  remains unknown.
To solve the problem, we propose to explore the global context of caption data and align regions and words via co-occurrence. Specifically, for a given concept  $c$ , we construct a concept group  $G$  comprising all image-text pairs that mention  $c$  in their captions. This grouping effectively clusters images that contain objects corresponding to concept  $c$ . Consequently, if  $G$  is large enough, we can induce that the objects corresponding to concept  $c$  will be the most common objects among images in  $G$ . This natural correlation automatically aligns the co-occurring objects in  $G$  to concept  $c$ . We thus reduce the problem of modeling cross-modality correspondence (region-word) to in-modality correspondence (region-region), which we address in the next section.
3.3 Discovering Co-occurring Objects across Images
Intuitively, candidate region proposals containing the co-occurring object should exhibit consistent and similar visual patterns across the images. We thus take a similarity-driven approach to discover these proposals. As illustrated in Figure 2, during training, we only sample a mini-group of images from the concept group as inputs in consideration of efficiency. In the mini-group, we iteratively choose one image as query image, and leave the rest as support images. Note that the regional proposals for each image are cached to avoid re-computation when swapping query and support images. The basic idea is to discover co-occurring objects in the query image from region proposals that have close neighbors across the support images. To fulfill this purpose, we introduce text-guided similarity estimation and similarity-based prototype discovery in the following paragraphs.
Similarity-based prototype discovery. Since modern visual backbones provide consistent feature correspondences for visually similar regions across images [58, 20, 47, 24, 60, 1], a straightforward way to identify co-occurring objects is to measure the cosine similarity between features of region proposals. Concretely, we calculate the pairwise similarity of region proposals between the query image and the support images. This yields a similarity matrix  $\mathbf{S} \in \mathbb{R}^{n \times mn}$ , where  $n$  stands for the number of proposals per image, and  $m$  stands for the number of support images. Intuitively,
co-occurring regions should exhibit high responses (similarities) in the last dimension of  $\mathbf{S}$ . But instead of using hand-crafted rules as in [41], we employ a two-layer MLP, denoted as  $\Phi$ , to derive co-occurrence from  $\mathbf{S}$ .  $\Phi$  is trained to estimate the probability of each region proposal in the query images as a co-occurring region, solely conditioned on  $\mathbf{S}$ . Here, we do not explicitly supervise the output probability since there is no ground-truth annotation, but  $\Phi$  is encouraged to assign high probabilities for co-occurring regions to minimize the overall region-word alignment loss. Based on the estimated probability vector  $\mathbf{p} \in \mathbb{R}^n$ , we obtain the prototypical region features for the co-occurring object via simple weighted sum:
$$
\mathbf {f} _ {p} = \sum_ {i = 1} ^ {N} \mathbf {p} _ {i} \cdot \mathbf {f} _ {i}, \quad \text {w h e r e} \mathbf {p} = \operatorname {s o f t m a x} _ {N} (\Phi (\mathbf {S})) . \tag {1}
$$
As the text label for this prototype naturally corresponds to the shared concept  $c$  in the mini-group. We can thus learn region-word alignment with a binary cross-entropy (BCE) classification loss:
$$
\mathcal {L} _ {\text {r e g i o n - w o r d}} = \mathcal {L} _ {\mathrm {B C E}} (\mathbf {W f} _ {p}, c), \quad \text {w h e r e} \mathcal {L} _ {\mathrm {B C E}} (\mathbf {s}, c) = - \log \sigma \left(\mathbf {s} _ {c}\right) - \sum_ {k \neq c} \log \left(1 - \sigma \left(\mathbf {s} _ {k}\right)\right), \tag {2}
$$
where  $\mathbf{W}$  the classifier weight derived from  $\mathcal{C}^{\mathrm{open}}$ , and  $\sigma(\cdot)$  stands for the sigmoid function.
Text-guided region-region similarity estimation. However, general cosine similarity may not always truly reflect closeness of objects in the semantic space, as objects of the same category may exhibit significant variance in appearance. Moreover, there could exist multiple co-occurring concepts among the sampled images, which incurs ambiguity in identifying co-occurrence. To address the problems, we introduce text guidance into similarity estimation to make it concept-aware. Concretely, given features of two region proposals  $\mathbf{f}_i$ ,  $\mathbf{f}_j \in \mathbb{R}^d$ , where  $d$  is the dimension of feature vectors, we additionally introduce  $\mathbf{w}_c \in \mathbb{R}^d$ , the text embedding of concept  $c$  (the shared concept in the mini-group), to re-weight similarity calculation:
$$
s _ {i j} = \bar {\mathbf {w}} _ {c} ^ {\top} \cdot \left(\frac {\mathbf {f} _ {i}}{\| \mathbf {f} _ {i} \|} \circ \frac {\mathbf {f} _ {j}}{\| \mathbf {f} _ {j} \|}\right), \quad \text {w h e r e} \bar {\mathbf {w}} _ {c} = \sqrt {d} \frac {\left| \mathbf {w} _ {c} \right|}{\| \mathbf {w} _ {c} \|}, \tag {3}
$$
where “○” denotes Hadamard product, “ $\lvert\cdot\rvert$ ” denotes absolute value operation, and “ $\lVert\cdot\rVert$ ” denotes  $\ell_2$ -normalization, respectively. Here, the rationale is that the relative magnitude of text features at different dimensions indicates their relative importance in the classification of concept  $c$ . By weighting the image feature similarities with the text feature magnitudes, the similarity measurement can put more emphasis on feature dimensions that reflect more of the text features. Therefore, the re-weighted similarity metric provides a more nuanced and tailored measure of the proximity between objects in the context of a particular concept. It is noteworthy that we choose regional features from the output of the penultimate layer of the classification head (the last layer is the text embedding) so that they naturally reside in the shared feature space as text embeddings.
Following [61, 28], we train the model simultaneously on detection data and image-text pairs to acquire localization capability and knowledge of vision-language alignments. In addition to learning region-level alignments from region-word pairs discovered in Sec. 3.3, we treat image-text pairs as a generalized form of region-word pairs to learn image-level alignments. Particularly, we use a region proposal covering the entire image to extract image features, and encode the entire caption into language embeddings. Similar to CLIP [35], we consider each image and its original caption as a positive pair and other captions in the same batch as negative pairs. We then use a BCE loss similar to Eq. (2) to calculate the image-text matching loss  $\mathcal{L}_{\mathrm{image-text}}$ . The overall training objective for this framework is:
$$
\mathcal {L} (I) = \left\{ \begin{array}{l l} \mathcal {L} _ {\mathrm {r p n}} + \mathcal {L} _ {\mathrm {r e g}} + \mathcal {L} _ {\mathrm {c l s}}, & \text {i f} I \in \mathcal {D} ^ {\mathrm {d e t}} \\ \mathcal {L} _ {\text {r e g i o n - w o r d}} + \mathcal {L} _ {\text {i m a g e - t e x t}}, & \text {i f} I \in \mathcal {D} ^ {\mathrm {c a p}} \end{array} , \right. \tag {4}
$$
where  $\mathcal{L}_{\mathrm{rpn}}, \mathcal{L}_{\mathrm{reg}}, \mathcal{L}_{\mathrm{cls}}$  are standard losses in the two-stage detector. For inference, CoDet does not require cross-image correspondence modeling as in training. It behaves like a normal two-stage object detector by forming the classifier with arbitrary language embeddings.
4.1 Benchmark Setup
OV-LVIS is a general benchmark for open-vocabulary object detection, built upon LVIS [19] dataset which contains a diverse set of 1203 categories of objects with a long-tail distribution. Following standard practice [18, 59], we set the 866 common and frequent categories in LVIS as base categories, and leave the 337 rare categories as novel categories. Besides, we choose CC3M [40] which contains 2.8 million free-from image-text pairs crawled from the web, as the source of image-text pairs. The main evaluation metric on OV-LVIS is the mask AP of novel (rare) categories.
OV-COCO is derived from the popular COCO [29] benchmark for evaluation of zero-shot and open-vocabulary object detection methods [2, 57]. It splits the categories of COCO into 48 base categories and 17 novel categories, while removing the 15 categories without a synset in the WordNet [32]. As for image-caption data, following existing works [57, 61, 28], we use COCO Caption [7] training set which provides 5 human-generated captions for each image for experiments on OV-COCO. The main evaluation metric on OV-COCO is the box  $\mathrm{AP}_{50}$  of novel categories.
We extract object concepts from the text corpus of COCO Caption/CC3M using an off-the-shelf language parser [32]. Remarkably, we filter out concepts without a synset in WordNet or outside the scope of the 'object' definition (i.e., not under the hierarchy of 'object' synset in WordNet) to clean the extracted concepts. For phrases of more than one word, we simply apply the filtering logic to the last word in the phrase. Subsequently, we remove concepts with a frequency lower than 100/20 in COCO-Caption/CC3M. This leaves 634/4706 concepts for COCO/CC3M.
For experiments on OV-LVIS, unless otherwise specified, we use CenterNet2 [62] with ResNet50 as the backbone, following [61, 28]. For OV-COCO, Faster R-CNN with a ResNet50-C4 [22] backbone is adopted. To achieve faster convergence, we initialize the model with parameters from the base class detection pre-training as in [61, 28]. The batch size on a single GPU is set to 2/8 for COCO/LVIS detection data and 8/32 for COCO Caption/CC3M caption data. The ratio between the detection batch and caption batch is set to 1:1 during co-training. Notably, a caption batch by default contains four mini-groups, where each mini-group is constructed by sampling 2/8 image-text pairs from the same concept group in COCO Caption/CC3M. We train the model for 90k iterations on 8 GPUs.