SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate the model's ability to recognize rare and diverse objects on both LVIS and ODinW in a finetuning-free setting, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. Each category is provided with both language and vision queries. Tab. 1 shows the results on LVIS. Overall, MQ-Det demonstrates strong finetuning-free transferability with impressive efficiency on both data usage and training time. MQ-GLIP-L surpasses the current SoTA by a large margin with simply 5 visual exemplars provided, reaching  $34.7\%$  AP on Val v1.0  $(+7.8\%)$  over GLIP-L), which verifies the superiority of multi-modal queries over single-modal queries. Meanwhile, MQ-Det demonstrates good training efficiency, e.g., MQ-GLIP-T only requires additional  $2\%$  training time and  $12\%$  data usage when modulated upon GLIP-T.
Additionally, we find three contributing factors: (i) Strong generalization comes from combination of textual breadth and visual granularity. In Tab. 1, MQ-GLIP-T-Img replaces all text queries with [MASK] tokens and solely utilizes vision queries to predict objects, achieving  $17.6\%$  AP.
† Modulating upon pretrained models indirectly utilizes their pre-training data.
Table 2: Results on the ODinW benchmark. The few-shot performance is evaluated by 3-shot [27, 25].
Model	Language Query	Vision Query	Backbone	Pre-train Data	Data Size	ODinW-35 APavg	ODinW-13 APavg
Finetuning-free Setting
MDETR [20]	✓	✗	ENB5 [38]	GoldG,RefC	0.9M	10.7	25.1
OWL-ViT [30]	✓	✓	ViT L/14(CLIP)	O365, VG	0.8M	18.8	40.9
GLIP-T [25]	✓	✗	Swin-T	O365,GoldG,Cap4M	5.5M	18.7	41.9
GLIP-L [25]	✓	✗	Swin-L	FourODs,GoldG,Cap24M	27.5M	22.6	51.0
OmDet [50]	✓	✗	ConvNeXt-B	COCO,O365,LVIS,PhraseCut	1.8M	16.0	43.6
GLIPv2-T [48]	✓	✗	Swin-T	O365,GoldG,Cap4M	5.5M	22.3	50.7
DetCLIP [42]	✓	✗	Swin-T	O365,GoldG,YFCC1M	2.4M	-	43.3
GroundingDINO-T [27]	✓	✗	Swin-T	O365,GoldG,Cap4M	5.5M	21.7	49.8
MQ-GroundingDINO-T	✓	✓	Swin-T	O365†	0.66M	22.5	50.9
MQ-GLIP-T	✓	✓	Swin-T	O365†	0.66M	20.8	45.6
MQ-GLIP-L	✓	✓	Swin-L	O365†	0.66M	23.9	54.1
Few-Shot Setting
DyHead-T [6]	✗	✗	Swin-T	O365	0.66M	37.5	43.1
GLIP-T [25]	✓	✗	Swin-T	O365,GoldG,Cap4M	5.5M	38.9	50.7
DINO-Swin-T [47]	✗	✗	Swin-T	O365	0.66M	41.2	49.0
OmDet [50]	✓	✗	ConvNeXt-B	COCO,O365,LVIS,PhraseCut	1.8M	42.4	48.5
MQ-GLIP-T	✓	✓	Swin-T	O365†	0.66M	43.0	57.0
Full-Shot Setting
GLIP-T [25]	✓	✗	Swin-T	O365,GoldG,Cap4M	5.5M	62.6	61.9
DyHead-T [6]	✗	✗	Swin-T	O365	0.66M	63.2	58.7
DINO-Swin-T [47]	✗	✗	Swin-T	O365	0.66M	66.7	-
OmDet [50]	✓	✗	ConvNeXt-B	COCO,O365,LVIS,PhraseCut	1.8M	67.1	65.3
DINO-Swin-L [47]	✗	✗	Swin-L	O365	0.66M	68.8	67.3
MQ-GLIP-T	✓	✓	Swin-T	O365†	0.66M	64.8	62.5
Table 3: Ablation results. We evaluate finetuning-free performance on LVIS and ODinW. Both average and median values on 13 datasets of ODinW are reported. Each row in this ablation study table should be compared to the baseline MQ-GLIP-T reported at the top of the table.
Ablated Setting	Ablated Details	MQ-GLIP-T Original Value	Changed Value	LVIS MiniVal (%)	ODinW-13 (%)
AP	APr	APc	APf	APavg	APmid
Original GLIP-T model	26.0	20.8	21.4	31.0	41.9	39.3
MQ-GLIP-T model	30.4	21.0	27.5	34.6	45.6	48.8
(a)	Mask Strategy	Mask Rate	40%	0%	26.9	20.1	23.4	31.2	42.8	46.8
40%	80%	28.9	20.4	24.2	34.6	45.0	48.3
(b)	Gate	Enable	✓	X	27.1	19.4	24.4	31.1	44.1	47.1
Architecture	MLP	Scaler	29.3	18.3	26.3	34.0	43.4	46.0
MLP	Linear	30.0	19.7	26.9	34.5	44.4	47.3
Input	\( \hat{v} \)	cat(\( \hat{v},t \))	29.2	17.8	26.0	34.2	44.8	48.2
(c)	Freezing	Full Model	✓	X	24.4	28.6	21.1	17.2	38.6	45.4
Detector	Text Encoder	✓	X	26.6	17.3	23.0	31.5	42.7	46.6
This confirms that our low-cost modulated pre-training strategy introduces sufficient visual cues. Augmenting generalizable language with visual details, MQ-GLIP-T outperforms the text-only model MQ-GLIP-T-Txt by a large margin. (ii) Efficiency on data usage and training time benefits from frozen pre-trained detection models. Our approach preserves the generalizable knowledge in the frozen pre-trained models and thus only needs little additional cost for further modulation. This is also verified in Tab. 3 (c). (iii) Our simple plug-and-play architecture design enables wide-range applications, since most language-queried detectors share the similar parallel architecture design as GLIP and GroundingDINO, e.g., MDETR [20], OWL-ViT [30], and DetCLIP [42] in Tab. 2.
Figure 3: Average precision  $(\%)$  on ODinW-13, from finetuning-free to full-shot data. MQ-Det clearly improves GLIP on data efficiency.
Figure 4: Finetuning-free detection with different vision queries. The language queries of 1,203 LVIS categories are always provided. Similar boxes are located while vision queries with better quality provide more accurate predictions.
3.2.2 MQ-Det as a strong few-shot learner
The experiments on ODinW with respect to finetuning-free, few-shot and full-shot settings are listed in Tab. 2, where MQ-Det shows robust performance on this challenging benchmark. We observe that most large-scale fundamental detection models only support language as category queries. With additional vision queries, MQ-GLIP-T averagely improves GLIP-T by  $+3.7\%$  AP and  $+6.3\%$  AP in the finetuning-free and few-shot settings respectively on ODinW-13, thus exhibiting the superiority of the rich description granularity of vision queries in our approach. Meanwhile, MQ-GLIP-T sets a new record on the ODinW-35 few-shot setting with  $43.0\%$  AP. Fig. 3 demonstrates few-shot performance with respect to different amounts of training data. We also notice that MQ-GLIP-T makes limited gains over GLIP-T with sufficient training data in the full-shot setting (e.g.,  $+0.6\%$  AP on ODinW-13). The reason is that the learning procedure, with sufficient training data, degenerates to a traditional close-set detection problem, where the model just represents categories as index regardless of textual or visual descriptions.
Importance of the masked language prediction strategy. We propose our masked language prediction strategy to address the learning inertia problem described in Sec. 2.3. Tab. 3 (a) verifies the effect of our pre-training strategy. We observe that our approach holds a great performance degradation  $(-3.5\%)$  AP on LVIS with the mask rate of 0. MQ-GLIP-T with the mask rate of 0 only improves GLIP-T by  $+0.9\%$  AP on LVIS. This indicates that our multi-modal queried MQ-GLIP may lose its effect and degenerate to language-queried GLIP without the masked language prediction strategy. Meanwhile, a too large mask rate (e.g.,  $80\%$ ) may lead to overfitting on the vision queries and losing attention to semantic-rich language queries, thus impairing performance.
The role of vision queries. MQ-Det provides users a flexible way to construct customized vision queries. We use one vision query for each category of LVIS in MQ-GLIP-T, and investigate the effect of different query qualities as shown in Fig. 4. The vision queries are selected based on the following criteria: (i) Positive queries (in red) contain complete information about the target objects, which significantly improve detection. (ii) Hard positive queries (in brown) only provide partial object information and sometimes include additional noise, such as the example of the boy and car. Despite the additional noise, MQ-Det can filter it out and still help improve the detection accuracy to some extent, identifying at least one object correctly. (iii) Negative queries (in green) contain too much noise and do not benefit detection, while (iv) "no vision queries" (in blue) is used as a baseline. Finally, we also use (v) mixed queries (in purple) that combine the three types of vision queries. The results show the robustness of MQ-Det, namely, our approach can select relevant information from miscellaneous vision queries to boost detection.
Architecture design. We design a plug-and-play GCP module so that visual details can be smoothly interleaved with highly-semantic textual cues. Particularly, in GCP, the conditional gating mechanism on each interleaved category token in Eqn. (2) ensures smooth model initialization from the frozen model and stable modulating according to different quality of visual cues. As shown in the first two
rows of Tab. 3 (b), roughly adding visual cues to each text token, or equally scaling them using the same learnable gating scalar for all vision queries, can result in sub-optimal performance  $(-3.3\%)$  AP and  $-1.1\%$  AP on LVIS, respectively). When using a simple gating design consisting of one linear layer over different vision queries, we obtain a slight performance drop compared to the non-linear MLP design, indicating content-aware gating mechanism is necessary and more parametric design helps analyze the quality of vision queries. In order to encourage the gate to balance the vision query  $\hat{v}$  and the text query  $t$  in Eqn. (2), we combine the two query types as a joint input (the last row of Tab. 3 (b)). However, we discover that this task rises the learning difficulty for a simple gate design.
Freezing pre-trained detectors. Tab. 3 (c) shows the necessity of modulating on the frozen models to introduce vision queries into the language-queried detectors. Either modulating the whole pre-trained detector or only modulating the text encoder with the limited detection data inevitably lose the generalization capability of the initial language-queried model, leading to a significant performance decrease (-6.0% AP and -3.8% AP on LVIS respectively). Moreover, training without freezing the detector requires almost twice the time consumption, not to mention the additional memory cost.
Language-queried object detection. Recently, vision-and-language approaches have gained popularity in visual recognition, where language is used to query visual objects and enable open-set ability. CLIP [32] and ALIGN [19] are pioneers in cross-modal image-level contrastive learning [5], where they train on millions of image-text pairs to perform open-vocabulary image classification. Inspired by these works, object detection has pursued cross-modal region-level contrastive learning in two main pipelines. One pipeline is open-vocabulary object detection (OVD) [46, 12, 54, 51, 30, 45], where the detection data is rigorously split into base classes for training and novel classes for evaluation. For instance, Zareian et al. [46] propose OVR-CNN that first pre-trains a visual encoder on image-caption pairs to learn rich vocabulary and then finetunes on detection data with only base classes. Detic [54] learns object localization and fine-grained vision-language matching simultaneously using max-size proposals to assign image-level labels. However, this pipeline utilizes limited detection data and only generalizes to specific datasets, e.g., from COCO [26] base classes to COCO novel classes. In contrast, our approach follows the second pipeline of large-scale fundamental detection models [42, 25, 48, 20, 50], which aims to generalize to arbitrary categories with extensive region-level image-text data. MDETR [20] trains an end-to-end DETR [3] model via phrase-region matching on existing multi-modal datasets, providing a strong baseline for few-shot object detection. GLIP [25, 48] formulates object detection as a grounding problem and leverages additional grounding data for aligned semantics at phrase and region levels, achieving even stronger performance on fully-supervised detection benchmarks without finetuning. GroundingDINO [27] applies GLIP's grounded training strategy to a stronger detection backbone DINO [47], achieving state-of-the-art open-set detection performance. Since these models all use a text encoder like BERT [8] to model the language queries, our approach can seamlessly integrate vision queries with these language-queried models to enable rich visual granularity.
Vision-queried methods commonly exist in current two-branch based few-shot object detection pipelines [16, 11, 14, 15, 21]. These methods are based on a siamese network to process the target images and vision queries in parallel, and calculate the similarity between image regions (usually proposals) and few-shot examples for detection. Kang et al. [21] propose a feature reweighting module to aggregate the target image features via few-shot vision queries. Meta Faster R-CNN [15] performs feature alignment between the two inputs and focus on foreground regions using attention. FCT [16] proposes an early fusion transformer architecture to fuse target images and vision queries. However, the two-branch pipeline in few-shot detection lacks instruction of language, and is far from practical implementation, e.g., no more than  $10\%$  AP in 3-shot COCO [26] for current SoTA methods [16, 15], not to mention more challenging LVIS [13] and ODinW [23] benchmarks. In addition, only a very few open-vocabulary object detection approaches support both language and vision queries. OV-DETR [45] proposes to utilize CLIP [32] embeddings from both image and text encoders as the object queries of the DETR [3] decoder. OWL-ViT [30] uses detection data to finetune upon CLIP. Thanks to the parallel structure of CLIP, OWL-ViT can conduct both region-text and region-image similarity. However, these two methods do not support language and vision queries as joint inputs, and show low performance with vision queries. Our MQ-Det considers the
complementary nature of vision and language queries, thus achieving both open-set generalization and fine-grained recognition.
MQ-Det is an efficient pre-training architecture and strategy design to let vision-language (VL) detection foundation models accept text interleaved with images as the input queries. After modulated pre-training, MQ-Det shows promising gains on the finetuning-free and few-shot settings on well-established benchmarks and 35 downstream tasks. Our results suggest that utilizing the complementary nature of language and vision is an important step towards open-world object detection.
This work was supported by Key-Area Research and Development Program of Guangdong Province (2021B0101400002), National Natural Science Foundation of China (No. 62036012, 62322212, 62072455, 61721004), and by Tencent Rhino-Bird Research Elite Program.
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020.
[4] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[6] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7373-7382, 2021.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[9] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084-14093, 2022.
[10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-338, 2010.
[11] Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4013–4022, 2020.
[12] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2021.
[48] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
[49] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. In Proceedings of the European Conference on Computer Vision, 2022.
[50] Tiancheng Zhao, Peng Liu, Xiaopeng Lu, and Kyusong Lee. Omdet: Language-aware object detection with large-scale vision-language multi-dataset pre-training. arXiv preprint arXiv:2209.05946, 2022.
[51] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Lianian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022.
[52] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[53] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[54] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kähenbuhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 350-368. Springer, 2022.
We provide an overview of the Appendix below.
Transfer to downstream (Sec. A). We elaborate on the additional details to transfer our MQ-Det to downstream tasks, including finetuning-free, few-shot, and full-shot settings. The introduction is organized as followed.
- Different ways to acquire the vision queries in Sec. A.1. This is an elaborative description of Sec. 3.1.2 in the main text.
- A customized tuning approach of MQ-Det in Sec. A.3, which achieves similar performance as full-model tuning while only fine-tuning very few parameters.
Experiments (Sec. B). We provide additional training and evaluation details, including:
- Detailed results on 35 downstream tasks in ODinW are provided in Tab. III and Tab. VII, described in Sec. B.1.
- An explicit evaluation on an open-vocabulary detection setting is conducted to further investigate the generalization of our multi-modal queries, as presented in Sec. B.2.
- Some visualization results of MQ-Det in Sec. B.1 and Fig. I.
- The training hyper-parameters are illustrated in Sec B.3 and Tab. VI.
Further discussion (Sec. C). We provide a comprehensive discussion on our work, including limitations and broader impacts in Sec C.
A Transfer to downstream
A.1 Different ways to acquire vision queries
During finetuning-free evaluation, we extract 5 instances as vision queries for each category from the downstream training set without any finetuning. We also provide two alternative strategies and observe similar performance, i.e., retrieval and test-time online update, where the former obtains vision queries from heterogeneous external data like ImageNet [7], and the latter dynamically stores high-confidence instances as vision queries during evaluation. These two additional approaches are proposed to simulate realistic scenarios:
- Retrieval (user-provided exemplars): a small number of exemplars are provided by the users without any fine-tuning. We retrieve 5 exemplars as vision queries for each category from ImageNet-21K [7] to simulate the user-provided exemplars. These samples are heterogeneous from the downstream test data, e.g., domains. The results are provided in MQ-GLIP-T-Retrival of Tab. I.
- Online updating: the model dynamically stores high-confidence instances as vision queries during evaluation. No vision queries are provided at the initial stage of evaluation. The results are illustrated in MQ-GLIP-T-Online of Tab. I. We provide detailed description in Sec. A.2.
The results are illustrated in Tab. I. We select 3 downstream datasets from ODinW [23] to verify the effectiveness of each approach. Generally, all three approaches to acquire vision queries demonstrate similar performance. We observe that vision queries from online updating hold relatively lower quality, thus leading to slight performance drop, since no manual annotations are provided. Meanwhile, exemplars retrieved from ImageNet are object-centric and contain little noise (e.g., other irrelevant objects), thus improving the performance.
A.2 Test-time online update
Our test-time online update strategy is conducted via the following steps: 1) only utilize language queries to conduct detection at the initial stage of evaluation. 2) Store detected instances with high confidence as the vision queries of corresponding categories. 3) Use both language queries and stored
vision queries for evaluation and seek for more vision queries. Tab. I verifies the effectiveness of our approach.
A.3 Partial tuning rivals full-model tuning
Since the GCP modules are interleaved into the frozen detector, we provide a customized fine-tuning strategy, partial tuning, namely, only tuning the newly added GCP modules and freezing all other parameters. The results in Tab. II indicate that our partial tuning strategy achieves comparable performance with traditional full-model tuning (i.e., only -0.4% AP on ODinW-13). Partial tuning accounts for much fewer learnable parameters, thus friendly to training time and memory costs. The experimental results in the main text are all based on the partial tuning.
Table I: Different ways to acquire vision query. We report the finetuning-free performance. All models should be compared with the MQ-GLIP-T model at the top of the table.
Model	AerialDrone	Aquarium	Rabbits
GLIP-T	12.5	18.4	70.2
MQ-GLIP-T	15.8	23.5	75.4
MQ-GLIP-T-Online	15.5	23.2	74.8
MQ-GLIP-T-Retrieval	16.0	23.6	75.1
Table II: Different fine-tuning strategies of MQ-Det under the 5-shot setting. The implementation used in our evaluation is highlighted in color.
Strategy	ODinW-13 (%)
APavg	APmid
Partial tuning	59.1	62.4
Full-model tuning	59.5	64.5
B.1 All results
We report the per-dataset performance under various settings in ODisNw-13 and ODisNw-35, shown in Tab. III and Tab. VII, respectively. We also provide some visualized results in Fig. I.