SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yifan Xu $^{1,3\dagger}$ , Mengdan Zhang $^{2\dagger}$ , Chaoyou Fu $^{2}$ , Peixian Chen $^{2}$ , Xiaoshan Yang $^{1,3,4}$ , Ke Li $^{2}$ , Changsheng Xu $^{1,3,4\dagger}$ $^{1}$ MAIS, Institute of Automation, Chinese Academy of Sciences  $^{2}$ Tencent Youtu Lab School of Artificial Intelligence, University of the Chinese Academy of Sciences  $^{4}$ Peng Cheng Laboratory {yifan.xu, csxu}@nlpr.ia.ac.cn, davinazhang@tencent.com
We introduce MQ-Det, an efficient architecture and pre-training strategy design to utilize both textual description with open-set generalization and visual exemplars with rich description granularity as category queries, namely, Multi-modal Queried object Detection, for real-world detection with both open-vocabulary categories and various granularity. MQ-Det incorporates vision queries into existing well-established language-queried-only detectors. A plug-and-play gated class-scalable perceiver module upon the frozen detector is proposed to augment category text with class-wise visual information. To address the learning inertia problem brought by the frozen detector, a vision conditioned masked language prediction strategy is proposed. MQ-Det's simple yet effective architecture and training strategy design is compatible with most language-queried object detectors, thus yielding versatile applications. Experimental results demonstrate that multi-modal queries largely boost open-world detection. For instance, MQ-Det significantly improves the state-of-the-art open-set detector GLIP by  $+7.8\%$  AP on the LVIS benchmark via multimodal queries without any downstream finetuning, and averagely  $+6.3\%$  AP on 13 few-shot downstream tasks, with merely additional  $3\%$  modulating time required by GLIP. Code is available at https://github.com/YifanXu74/MQ-Det.
As the thriving of large-scale vision-language pre-training models [25, 48, 24, 29, 27, 44, 20], object detection recently has ushered in a new fashionable paradigm, which locates the desired objects via a queried text. Benefited from the generalization of the pre-training models on large scale data [26, 13, 36, 43, 22, 18, 37, 31, 35], such a text queried paradigm makes steady progress on the long road to open-set object detection.
Compared with previous fixed category sets (usually represented by finite numbers), the foregoing text query has the merit of representing broad concepts, but also has an intrinsic limitation of insufficient description granularity [4, 9, 32]. For example, class homonyms (e.g., "bat" can be a piece of wood or a kind of animal) lead to ambiguous queries. Meanwhile, as exemplified in Fig. 1, for fine-grained fish species detection, it is usually struggled to use limited text to describe the fish with specific patterns. Empirically, one straightforward solution for the insufficient description granularity problem is to design additional textual description, but there are three distinct obstacles: 1) it is difficult to comprehensively describe visual details [52], and constructing textual description for a large amount of categories is a laborious work. 2) The longer queried text increases the understanding difficulty of the pre-training model and 3) brings more computational overhead. Experimentally, the state-of-the-art (SoTA) text queried detector GLIP [25] merely improves average precision (AP) from
*Work done when interning at Tencent Youtu Lab. †Equal contribution. ‡Corresponding author.
$17.7\%$  to  $18.4\%$  on the Aquarium dataset [23] (a fine-grained fish species detection dataset), even with extra manually designed textual description for some categories.
A picture paints a thousand word. Compared with the text, the image can provide richer clues about the target object. But at the same time, the human-generated text has higher information density and thereby brings stronger generalization capability [32, 8, 2]. In light of this, a natural idea springs up, i.e., combining the text and the image to constitute a multi-modal query, which has the advantages of both breadth of the former and rich granularity of the latter. Nevertheless, how to acquire such a multi-modal queried detection model still faces challenges: 1) directly finetuning with limited visual exemplars, typically in previous vision-queried few-shot detection methods [49, 39, 21, 16], leads to the risk of catastrophic forgetting. 2) Large foundation models have good generalization but require heavy training burden if re-organized and trained from scratch (e.g., over 30 million data storage and almost 480 V100 GPU days for GLIP [25, 48]).
This work fills in the blank of Multi-modal Queried object Detection (MQ-Det), and introduces an efficient plug-in training architecture. The crux of the proposed MQ-Det is to fuse description-rich visual cues and the highly generalizable text representations, while only adding a small training cost on the basis of existing language queried fundamental detection models. We evaluate our models on a finetuning-free setting, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any finetuning. With only one epoch of modulating on the Objects365 [36] dataset, accounting for a mere  $3\%$  of the total pre-training time for GLIP, our approach impressively improves the finetuning-free performance by  $+7.8\%$  on the LVIS benchmark through providing the model with 5 visual exemplars along with textual category descriptions.
To be specific, as shown in Fig. 1, we first interleave a Gated Class-scalable Perceiver (GCP) that bridges class-wise visual cues and textual cues in each high-level stage of the text encoder of the detector. This module consists of class-wise cross attention layers to augment textual semantic embeddings with visual details, and a conditional gating layer that varies based on the quality of visual cues from the vision queries. The output of this module is incorporated to each high-level stage of the text encoder in a residual-like manner, which conducts textual perturbation in initial general embedding space [1]. Additionally, this simple design has no class-specific parameters and can be extended to classes of various granularity. Second, we design a vision conditioned masked language prediction strategy to ensure sufficient visual intervention in the modulating stage. We observed a learning inertia problem during modulating, namely, when incorporating visual cues in a gated residual-like manner, the learning process tends to be stuck around the initial optimum point of the frozen pre-training detector and cannot introduce much visual knowledge. Thus, we randomly mask the text tokens and let corresponding vision queries independently make object prediction. Note that we freeze the initial detection foundation model and only train the GCP modules in the modulating stage, which is quite efficient.
Contributions. In summary, the contributions of this paper are as follows: (i) As far as we know, we are the pioneer to introduce the multi-modal query that has the characteristics of both breadth and rich granularity, paving a new path to the object detection in the wild. (ii) We propose a plug-and-play GCP module that dynamically fuses informative visual cues and highly generalizable text cues from the multi-modal query, and employ a vision conditioned masked language prediction strategy to permit sufficient multi-modal fusion upon frozen detection models. (iii) Our proposed MQ-Det demonstrates powerful transferability on the finetuning-free and few-shot scenarios, while requiring much less training time than previous SoTA fundamental detectors. Specifically, MQ-Det outperforms GLIP by  $+7.8\%$  AP in the finetuning-free detection on the challenging LVIS [13] and averagely  $+6.3\%$  AP on 13 downstream few-shot detection tasks [23], modulated with merely  $3\%$  of the training time required by GLIP.
This section describes MQ-Det: an efficient modulation architecture with elaborately designed multimodal fusion strategy to let vision-language (VL) detection foundation models accept text interleaved with images as input queries, thus leveraging the generalization capability of semantic-rich language while enhancing detection discrimination with visual details. We first briefly review the language-queried baseline GLIP [25] in Sec. 2.1, then introduce the plug-and-play architecture that can support
Figure 1: Method overview. The plug-and-play Gated Class-scalable Perceivers (GCPs) are inserted to text encoder layers, trained with our masked language prediction strategy. Class-wise attention mask is applied to make the text query of each category only attends to corresponding vision queries.
class-scalable vision queries as input in Sec. 2.2, and finally shed light on the multi-modal fusion strategy in Sec. 2.3.
Language-queried detection model. Following CLIP [32], the paralleled formulation has been a mainstream architecture of existing VL detection foundation models [48, 29, 42, 44, 28, 27]. Take our baseline GLIP [48] as an example, as shown in the left part of Fig. 1, it consists of paralleled text encoder  $\Phi_T(\cdot)$  and image encoder  $\Phi_I(\cdot)$  for feature extraction, and reformulates detection as a grounding task, by grounding/aligning each visual region to class content in a text prompt. Since the detection models like GLIP recognize objects via class content only in the language queries, we denote them as language-queried detection models.
Specifically, the model takes image-text pairs  $\{(\mathcal{I},\mathcal{T})\}$  as inputs, where the text prompt  $\mathcal{T} =$  "  $t_1,\dots ,t_{|C|}$  contains  $|C|$  categories to be detected, e.g., "person, bicycle, ... , toothbrush". Then, the model extracts image and text features  $(I,T)$  via paralleled encoders. Every linear classification layer in traditional vision models are replaced by a vision-language matching dot-product layer, calculating the region-word similarity logits  $S_{cls}$  in the detection head  $H(\cdot)$ . Formally,
$$
I = \Phi_ {I} (\mathcal {I}), \quad T = \Phi_ {T} (\mathcal {T}), \quad R = H (I), \quad S _ {c l s} = R \cdot T ^ {\top}, \tag {1}
$$
where  $I$  is the extracted image features,  $R \in \mathbb{R}^{N \times d}$  denotes the object/region/box features of the input image, and  $T \in \mathbb{R}^{|C| \times d}$  is the contextual word-token features from the text encoder in which each token represents one category. Additionally, a box regressor is applied to  $R$  for localization that is similar to traditional detectors. The model is finally trained with a phrase grounding loss calculated via the logits  $S_{cls}$  and a traditional localization loss. We refer to GLIP [25] for more details.
The language-queried model GLIP is trained on massive image-text pairs, and thus scales up visual concepts beyond traditional detection vocabularies, and finally achieves strong open-set capacity. However, the inference process via a simple language-queried prompt limits its performance especially for non-descriptive, ambiguous or fine-grained class names.
2.2 Architecture design
Considering the limited description granularity of text stated in Sec. 1, we propose to integrate vision queries into the pre-trained language-queried detection models to enrich semantic cues. The architecture of such multi-modal queried detection models is expected to follow three principles: (1) Class-scalable: the introduced vision queries are open to arbitrary categories rather than fitting a
closed set of categories. (2) Semantic-complementary: the vision queries are sufficiently interleaved with coarse text queries to provide fine-grained visual details that support to distinguish various granularity of categories. (3) Generalization-retainable: the introduction of visual queries into foundation models accumulates rich details while still retaining the generalization capability.
We compare three possible architecture designs for enriching queries as illustrated in Fig. 2, and show the superiority of our approach on the above-mentioned principles. First, soft prompt based designs [53, 34, 4, 9] (e.g., CoOp [53], Fig. 2 (a)) can adapt text queries to describe diverse characteristics of categories beyond intensively-tuned manual prompts. However, most previous works only finetune task-specific prompts, which limits their generalization. Second, existing two-branch few-shot detection architectures [16, 21, 15, 40] (e.g., FCT [16], Fig. 2 (b)) support vision queries and detect novel objects using very few training exemplars. But their designs are incompatible with the language-queried pre-training foundation models due to deficiency of language inputs and model complexity (e.g., early and deep fusion in [16]). We propose a simple yet effective plug-in architecture in Fig. 2 (c), which interleaves class-wise visual cues with textual cues, to let highly semantic language-queried models "perceive" visual details.
Inspired by [1], we augment the semantics of each category query by conditioning the text on corresponding visual representations produced by our proposed Gated Class-scalable Perceiver (GCP) illustrated in Fig. 1. Specifically, we insert GCP modules between the frozen pre-trained text encoder blocks in a residual manner and only train these modules from scratch. In a GCP module, each category token from the text encoder block is independently cross-attended to corresponding vision queries to acquire rich details (Principles 1,2). Then, the interleaved category tokens are gated by the GCP module so that the frozen text encoder is kept intact at initialization for improved stability and generalization performance (Principle 3).
Formally, given an image-text pair  $(\mathcal{I},\mathcal{T})$  with vision queries  $\mathcal{V} = \{\mathbb{v}_i|\mathbb{v}_i = \{v_i^{(j)}\}_{j = 1}^k\}_{i = 1}^{|C|}$  extracted from  $k$  exemplars of each category, the GCP module augments each text query features in  $T = \Phi_T(\mathcal{T}) = \{t_i\}_{i = 1}^{|C|}$  in a residual-like way, namely,
$$
\bar {\mathbf {v}} _ {i} = \mathrm {X} - \mathrm {M H A} (\mathbf {v} _ {i}, I), \quad \hat {v} _ {i} = \mathrm {X} - \mathrm {M H A} (t _ {i}, \bar {\mathbf {v}} _ {i}), \quad \hat {t} _ {i} = t _ {i} + \sigma (g a t e (\hat {v} _ {i})) \cdot \hat {v} _ {i}, \quad i = 1, 2, \dots , | C |, \tag {2}
$$
where  $\mathrm{X - MHA}(\cdot ,\cdot)$  denotes a multi-head cross-attention layer with the former input as queries and the later as keys and values.  $\sigma = \tanh (\cdot)$  is a normalization function. Concretely, the vision query features  $\mathbf{v}_i = \Phi_I(\mathbb{V}_i)$  are first augmented via the target image feature  $I = \Phi_I(\mathcal{I})$  to gather content-aware information [52]. The augmented vision queries  $\bar{\mathbf{v}}_i$  are correlated with corresponding text token  $t_i$  to enrich it from multiple views. The conditional gating value  $\sigma (gate(\hat{v}_i))$  is dynamically adjusted according to the quality of visual cues from the exemplars of each category, which is evaluated by a three-layer perceptron (MLP) that reduces the feature dimension gradually to a layer-specific learnable scalar. The conditional gating value multiplies the enriched feature  $\hat{v}_i$  before adding it to the initial text token  $t_i$  from the residual connection. Since the gating value is initialized to 0, the module output matches that of the pre-trained text encoder at initial training, improving training stability and final performance. It is worth noting that this independent class-wise cross-attention scheme has no class-specific parameters and can be scaled to any classes of various description granularity, as verified in Sec. 3.2. In addition, it importantly allows the model to seamlessly generalize to any number of vision queries in inference, regardless of the number used during training.
2.3 Modulated pre-training
Equipped with the plug-in architecture design described in Sec. 2.2, we propose an efficient pretraining strategy that permits sufficient multi-modal fusion and rapid learning upon frozen detection foundation models using only a small subset of pre-training data. We denote this process as modulated pre-training in comparison to the massive pre-training of foundation models.
Vision query extraction. We provide vision queries for  $|C|$  categories from a bank  $D$  built from pre-training data:
$$
D = \left\{\mathbf {v} _ {i} \mid | \mathbf {v} _ {i} | = K \right\} _ {i = 1} ^ {| C |}, \quad v _ {i} ^ {(j)} = R o I P o o l (\Phi (\mathcal {I}), \gamma b), \quad j = 1, 2, \dots , K, \tag {3}
$$
where each  $\mathbf{v}_i\in \mathbb{R}^{K\times d}$  contains  $K$  vision queries modeled by the image encoder  $\Phi_I$ . Specifically, given a query instance of the  $i$ -th category with box  $b\in \mathbb{R}^4$  in an image  $\mathcal{I}$ , an RoI pooler [33] is employed to extract the corresponding region feature  $v_{i}^{(j)}\in \mathbb{R}^{d}$ . The parameter  $\gamma = 1.5^{2}$  enlarges the
Figure 2: Architecture designs for query enriching, with only classification branches presented for clarity. (a) Prompt-based designs [53, 52] only tune upon language instruction while (b) two-branch few-shot detection architectures [16, 21] employ heavily coupled parallel encoders and merely take vision inputs. (c) MQ-Det elegantly combines both language and vision queries in an efficient way.
region area for contextual clues. During modulated pre-training, we randomly select  $k \ll K$  vision queries from  $D$  for each category at each forward process, simulating the downstream scenarios where only low-shot vision queries are provided. In practical implementation, we set  $K = 5000$  and  $k = 5$ . After modulating, our model can generalize to arbitrary categories and vision queries during downstream inference.
Training upon frozen pre-trained language-queried detectors. Given the inherently general semantic representations in the pre-trained detector, we argue that the textual features only require minor modulation from the visual details rather than significant alteration. This viewpoint has been previously discussed in [52, 53, 1], and we have observed in Tab. 3 (c) that the full-model training on a limited number of categories faces the risk of catastrophic forgetting. Consequently, for efficiency purpose, we freeze the entire pre-trained detector and only train the newly-added gated class-scalable perceiver modules, significantly accelerating the training process.
Vision conditioned masked language prediction. Since the fundamental VL models highly rely on text description, the learning process of the residual-like gated architecture will rapidly converge around the initial optimum point with frozen text features, causing failure of vision queries. That is, models still learn how to align region with text features not visual cues. We represent such an issue as a learning inertia problem. As shown in the first row of Tab. 3(a), the AP merely improves  $+0.9\%$  over GLIP-T [25] on LVIS. To address this issue, we propose a simple yet effective masking strategy:
$$
\mathcal {T} = \left\{t _ {1}, t _ {2}, \dots , [ \mathrm {M A S K} ], \dots , t _ {| C |} \right\}. \tag {4}
$$
Namely, given an image-text pair  $(\mathcal{I},\mathcal{T})$ , a ground-truth text token in  $\mathcal{T}$  corresponding to instances occurring in  $\mathcal{I}$  is randomly masked by a [MASK] token with probability  $40\%$ . The [MASK] token is forced to extract visual cues from vision queries in the GCP modules to provide accurate predictions, thus bridging the model's dependence upon vision queries. As shown in Tab. 1, our language masking strategy ensures sufficient visual intervention in the modulated pre-training stage and significantly boosts the performance, e.g.,  $+4.4\%$  AP over GLIP-T on finetuning-free LVIS.
3.1.1 Datasets and benchmarks
Objects365 dataset [36] is a large-scale, high-quality dataset for object detection. We use this dataset to conduct the modulated pre-training of our MQ-Det models. It contains 0.66 million images of 365 object categories, with 30 million bounding boxes, which are more diverse and fine-grained than those in other popular datasets such as COCO [26] and Pascal VOC [10]. Objects365 is a widely-used pre-training dataset in previous foundational detection models [25, 48, 27, 50, 44, 42].
LVIS benchmark [13] is a challenging dataset for long-tail objects. It contains 1,203 categories for evaluation, with many rare categories that scarcely exist in the pre-training datasets. Therefore, we
Table 1: Finetuning-free detection on the LVIS benchmark. * denotes supervised approaches. The training time is tested on one V100 GPU. We present the number of vision queries during evaluation.
Model	Backbone	Pre-Train Data	Data Size	Training Time (V100 days)	#Vision Query	MiniVal (%)	Val v1.0 (%)
AP	APr	APc	APf	AP	APr	APc	APf
MDETR [20]*	RN101	GoldG,RefC	0.9M	400	0	24.2	20.9	24.9	24.3	22.5	7.4	22.7	25.0
Mask R-CNN [17]*	RN101	-	-	-	0	33.3	26.3	34.0	33.9	-	-	-	-
Supervised-RFS [13]*	RN50	-	-	-	0	-	-	-	-	25.4	12.3	24.3	32.4
GLIP-T (B) [25]	Swin-T	O365	0.66M	300	0	17.8	13.5	12.8	22.2	11.3	4.2	7.6	18.6
GLIP-T [25]	Swin-T	O365,GoldG,CC4M	5.5M	480	0	26.0	20.8	21.4	31.0	17.2	10.1	12.5	25.5
GLIPv2-T [48]	Swin-T	O365,GoldG,CC4M	5.5M	-	0	29.0	-	-	-	-	-	-	-
GroundingDINO-T [27]	Swin-T	O365,GoldG,Cap4M	5.5M	-	0	25.7	15.2	21.9	30.9	-	-	-	-
GLIP-L [25]	Swin-L	FourODs,GoldG,Cap24M	27.5M	600	0	37.3	28.2	34.3	41.5	26.9	17.1	23.3	35.4
GroundingDINO-L [27]	Swin-L	O365,OI,GoldG,Cap4M,COCO,RefC	15.8M	-	0	33.9	22.2	30.7	38.8	-	-	-	-
MQ-GLIP-T-Img	Swin-T	O365†	0.66M	10	5	17.6	12.0	14.5	21.2	12.4	8.9	9.2	18.3
MQ-GLIP-T-Txt	Swin-T	O365†	0.66M	10	0	26.0	20.8	21.4	31.0	17.2	10.1	12.5	25.5
MQ-GroundingDINO-T	Swin-T	O365†	0.66M	10	5	30.2	21.7	26.2	35.2	22.1	12.9	17.4	31.4
MQ-GLIP-T	Swin-T	O365†	0.66M	10	5	30.4	21.0	27.5	34.6	22.6	15.4	18.4	30.4
MQ-GLIP-L	Swin-L	O365†	0.66M	22	5	43.4	34.5	41.2	46.9	34.7	26.9	32.0	41.3
use LVIS in the downstream task to evaluate the finetuning-free transferability. We report on MiniVal containing 5,000 images introduced in MDETR [20] as well as the full validation set v1.0.
ODinW benchmark [23] (Object Detection in the Wild) is a more challenging benchmark for evaluating model performance under real-world scenarios. For example, Aquarium requires locating fine-grained fish species, and Pothole concerns detecting holes on the road. It collects more than 35 datasets for evaluation. There are two commonly used versions of the ODinW benchmark, i.e., ODinW-13 and ODinW-35, where ODinW-13 is a subset of ODinW-35 and contains much less noisy data. The results on both benchmarks are reported for comprehensive comparison. We demonstrate that the fine-grained visual details in MQ-Det facilitate transfer to such diverse and challenging tasks.
3.1.2 Implementation details
We conduct extensive experiments on three settings: an open-set setting on finetuning-free LVIS [13] and ODinW [23] benchmarks, a few-shot setting and a close-set full-shot setting on ODinW. To demonstrate the plug-and-play versatility of our approach, we apply MQ-Det on two typical SoTA language-queried object detectors, GLIP [25] and GroundingDINO [27], and obtain our multi-modal queried models MQ-GLIP and MQ-GroundingDINO, respectively. We incorporate our GCP modules into the last 6 layers of the frozen text encoder. We conduct modulated pre-training of our models on the Objects365 dataset [36] for only one epoch using 8 NVIDIA V100 GPUs. The finetuning process under few/full-shot settings completely follows the baseline method GLIP, where vision queries are extracted from the few/full-shot training set. We also evaluate our method in a finetuning-free setting, namely, users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. During finetuning-free evaluation, we extract 5 instances as vision queries for each category from the downstream training set without any finetuning.
3.2 MQ-Det helps low-shot object detection
3.2.1 Multi-modal queried detection without finetuning