Multi-modal Queried Object Detection in the Wild
Authors: Yifan Xu
Mengdan Zhang
Chaoyou Fu
Peixian Chen
Xiaoshan Yang
Ke Li
Changsheng Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: open-world object detection
multi-modal query
vision-language pre-training
TL;DR: MQ-Det: the first multi-modal queried open-world object detector

Abstract: We introduce MQ-Det, an efficient architecture and pre-training strategy design to utilize both textual description with open-set generalization and visual exemplars with rich description granularity as category queries, namely, Multi-modal Queried object Detection, for real-world detection with both open-vocabulary categories and various granularity. MQ-Det incorporates vision queries into existing well-established language-queried-only detectors. A plug-and-play gated class-scalable perceiver module upon the frozen detector is proposed to augment category text with class-wise visual information. To address the learning inertia problem brought by the frozen detector, a vision conditioned masked language prediction strategy is proposed. MQ-Det's simple yet effective architecture and training strategy design is compatible with most language-queried object detectors, thus yielding versatile applications. Experimental results demonstrate that multi-modal queries largely boost open-world detection. For instance, MQ-Det significantly improves the state-of-the-art open-set detector GLIP by +7.8% AP on the LVIS benchmark via multi-modal queries without any downstream finetuning, and averagely +6.3% AP on 13 few-shot downstream tasks, with merely additional 3% modulating time required by GLIP. Code is available at https://github.com/YifanXu74/MQ-Det.

Multi-modal Queried Object Detection in the Wild
Yifan Xu Mengdan Zhang²t, Chaoyou Fu?, Peixian Chen², Xiaoshan Yang!.3.4 Ke Li, Changsheng Xu 1,,42
MAIS, Institute of Automation, Chinese Academy of Sciences Tencent Youtu Lab School of Artificial Intelligence, University of the Chinese Academy of Sciences
*Peng Cheng Laboratory
fyifan.xu, csxu)@nlpr. aa. en, davinazhaaggheeennt.ccmm
Abstract
We introduce MQ-Det, an efficient architecture and pre-training strategy design utilize both textual description with open-set generalization and visual exemplars with rich description granularity category queries, namely, Multi-modal Queried object Detection, for real-world detection with both open-vocabulary categories and various granularity. MQ-Det incorporates vision queries into existing well- detectors. A plug-and perceiver module upon the frozen detector proposed augment category text with class-wise visual information. To address the learning inertia problem brought by the frozen detector, a vision conditioned masked language prediction strategy proposed. MQ-Det's s simple yet effective architecture and training strategy design is compatible with most language-querics object detectors, thus yielding versatile applications. Experimental results demonstrate that multi-modal queries largely boost open-world detection. For instance, MQ-Det significantly improves the state- of-the-art open-set detector GLIP by +7.8% AP on the LVIS benchmark via multi- modal queries without any downstream finetuning, and averagely +6.3% AP on 13 few-shot downstream tasks, with merely additional 3% modulating time required by GLIP. Code is available at httpps: / github. com/YifanXu74/01
established
gated class-scalane
Introduction
As the thriving of large-scale vision- language pre-training models [25 48 24 29
20), object detection recently has ushered in a new fashionable paradigm, which locates the desired objects via queried text. Benefited from the generalization of the pre-training models large scale data [26]
37 31 351, such a text queried paradigm makes steady progress on the long road to open-set object detection.
13 36 43
18
Compared with previous fixed category sets (usually represented by finite numbers), the foregoing text query has the merit of representing broad concepts, but also has an intrinsic limitation of insufficient description granularity [4] 9 32 For example, class homonyms (e.g., "bat" can piece of wood or kind of animal) lead to ambiguous queries. Meanwhile, as exemplified in Fig. for fine-grained fish species detection, is usually struggled to use limited text to describe the fish with specific patterns. Empirically, one straightforward solution for the insufficient description granularity problem is to design additional textual description, but there are three distinct obstacles: 1) it is difficult comprehensively describe visual details [52), and constructing textual description for large amount of categories is a laborious work. 2) The longer queried text increases the understanding difficulty of the pre-training model and brings more computational overhead. Experimentally, the of art (SoTA) text queried detector GLIP 25 merely improves average precision (AP) from
WWrk done when interning Tencent Youtu Lab. Equal contribution. Cooreeppondig author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

17.7% to 18.4% on the Aquarium dataset [23] (a fine- ane-grained fish species detection dataset), even with extra manually designed textual description for some categories.
A picture paints a thousand word. Compared with the text, the image can provide richer clues about the target object. But at the same time, the human-generated text has higher information density and thereby brings stronger generalization capability [32 g 2). In light of this, natural idea springs up, i.e., combining the text and the image constitute a multi-modal query, which has the advantages of both breadth of the former and rich granularity of the latter. Nevertheless, how to acquire such a multi-modal queried detection model still faces challenges: 1) directly finetuning with limited visual exemplars. typically in previous vision-qu ssion-queried few-shot detection methods [49 [9 21, 16), leads to the risk of catastrophic forgetting. 2) Large foundation models have good generalization but require heavy training burden if re-orgamized and trained from scratch (e.g., over 30 million data storage and almost 480 V 100 GPU days for GLIP
This work fills in the blank of Multi-modal Queried object Detection (MQ-Det), and introduces an efficient plug-in training architecture. The crux of the proposed MQ-Det fuse description-rich visual cues and the highly generalizable text representations, while only adding a small training cost on the basis of existing language queried fundamental detection models. We evaluate our models on a finetuning-free setting, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any finetuning. With only one epoch of modulating on the Objects365 [36] dataset, accounting for a mere 3% of the total pre-training time for GLIP, our approach impressively improves the finetuning -fren performance by +7.8% on the LVIS benchmark through providing the model with 5 visual exemplars along with textual category descriptions.
To be specific, shown in Fig. that bridges class-wise visual cues and textual cues in each high-level stage of the text encoder of the detector. This module consists of class-wise cross attention layers to augment textual semantic embeddings with visual details, and a conditional gating layer that varies based on the quality of visual cues from the vision queries. The output of this module is incorporated to each high-level stage of the text encoder in a residual-like manner, which conducts textual perturbation in initial general embedding space []]. Additionally, this simple design has no class- sass-specific parameters and can be extended to classes of various granularity. Second, we design vision conditioned masked language prediction strategy to ensure sufficient visual intervention in the modulating stage. We observed learning inertia problem during modulating, namely, when incorporating visual cues in gated residual-like manner, the learning process tends be stuck around the initial optimum point of the frozen pre-training detector and cannot introduce much visual knowledge. Thus, we randomly mask the text tokens and let corresponding vision queries independently make object prediction. Note that we freeze the initial detection foundation model and only train the GCP modules in the modulating stage, which is quite efficient.
first interleave Gated Class-scalable Perceiver (GCP)
Contributions. In summary, the contributions of this paper are as follows: (i) As far as we know, we are the pioneer to introduce the multi-modil query that has the characteristics of both breadth and rich granularity. paving a new path to the object detection in the wild. (ii) We propose plug-and-pla GCP module that dynamically fuses informative visual cues and highly generalizable text cues from the
query, and employ a vision conditioned masked language prediction strategy to permit sufficient multi-modal fusion upon frozen detection models. (iii) Our proposed MQ-Det demonstrates powerful transferability on the finetuning-fr and few-shot scenarios, while requiring much less training time than previous SoTA fundamental detectors. Specifically, MQ-Det outperforms GLIP by +7.8% AP in the finetuning-free detection on the challenging LVIS [13] and averagely +6.3% AP on 13 downstream shw-shot detection tasks (23), modulated with merely 3% of the training time required by GLIP.
2 Methodology
This section describes MQ-Det: an efficient modulation architecture with elaborately designed multi- modal fusion strategy to let vision-language (VL.) detection foundation models accept text interleaved with images as input queries, thus leveraging the generalization capability of semantic-rich language while enhancing detection discrimination with visual details. We first briefly review the language- queried bascline GLIP (25 in Sec. 2.1 then introduce the plug-and and-play architecture that can support

---

Target images
Vision Queries
Gated Class-scalable
Language Query Features
Perceiver
[MASK]
Stingray
[MASK], Stingray
Language Queries
Target image Features
Cross-Attn
KV
Class-wise Cross Attn
Image Encoder
Cross-Attn
Kv
Text Encoder
Layer xN
Conditional Gate
Self-Altn
Region
Similarity
FFN
Vision Query Features
Stin
[MASK]
Figure I: Method overview. The plug-and-play Gated Class-scalable Perceivers (GCPs) are inserted to text encoder layers, trained with our masked language prediction strategy. Class- wise attention mask is applied to make the text query each category only attends to corresponding vision queries.
class-scalable vision queries as input in Sec. 2.2 and finally shed light on the multi-modal fusion strategy in Sec. 2.3
2.1 Preliminaries
Language-queried detection model. Following CLIP 32, the paralleled formulation has been mainstream architecture of existing VL detection foundation models 48 29 42
44
28
27]. Take
our baseline GLIP [48] as an example, as shown the left part of Fig. it consists of paralleled text encoder (-) and image encoder (-) for feature extraction, and reformulates detection grounding task, by grounding/aligning each visual region to class content in a text prompt. Since the detection models like GLIP recognize objects via class content only in the language queries, denote them as language-queried detection models.
Specifically, the model takes image-text pairs TT} inputs, where the text prompt T "l1,.... L contains |C| categories to be detected, e.g., "person, bicycle, ... toothbrush". Then, the model extracts image and text features (I, T)) via paralleled encoders. Every linear classification layer traditional vision models are replaced by vision-language matching dot-product layer, calculating the region-wer word similarity logits Sole in the detection head (-).. Formally,
= (T), T= DT({) R R H(),, SaL - R- R-T
(I)
where I is the extracted image features, R E IR Nxd denotes the object/region/box features of the input image, and T E CC|x the contextual word/token features from the text encoder in which each token represents one category. Additionally, box regressor is applied R for localization that is similar to traditional detectors. The model is finally trained with phrase grounding loss calculated via the logits S.1s and a traditional localization loss. We refer GLIP 25 for more details.
The aunuueee-uerred model GLIP is trained on massive image-text pairs, and thus scales up visual concepts beyond traditional detection vocabularies, and finally achieves strong open-set capacity. However, the inference process via a simple hanguage-querias prompt limits its performance especially for non-descriptive, ambiguous fine-grained class names.
2.2 Architecture design
Considering the limited description granularity of text stated Sec. we propose to integrate vision queries into the pre-trained language-querien detection models to enrich semantic cues. The architecture of such multi-modal queried detection models is expected to follow three principles: (1) Class-scalable the introduced vision queries open to arbitrary categories rather than fitting

---

closed set of categories. (2) Demantic-complementary the vision queries are sufficiently interleaved with coarse text queries to provide fine-gren -raiggea visual details that support to distinguish various granularity of categories. (3) pnerraiiaaiio-eetannllll
the introduction of visual queries into foundation models accumulates rich details while still retaining the generalization capability.
We compare three possible architecture designs for enriching queries as illustrated in Fig.2/ and show the superiority of our approach on the above-mentioned principles. First, soft prompt based designs [53
(e.g., CoOp 533 Fig. (a)) can adapt text queries to describe diverse charac- manual prompts. However, most previous works only finetune task-specifer prompts. which limits their generalization. Second, existing two- branch few-shot detection architectures [16] [1] 40] FCT [16). Fig. ((()) support vision queries and detect novel objects using very few training exemplars. But their designs are incompatible with the uunuage-querred pre-training foundation models due to deficiency of language inputs and model complexity (e.g., early and deep fusion in [16). We propose simple yet effective plug-in in architecture in Fig.Z(c), which interleaves dlasswiss visual cues with textual cues, to let highly semantic language-querien models "perceive" visual details.
teristics of categories beyond intensively-
we augment the semantics of each category query by conditioning the text on corresponding visual representations produced by our proposed Gated Class-scalable Perceiver (GCP) illustrated in Fig. []] Specifically, we insert GCP modules between the frozen pre-trained text encoder blocks in a residual manner and only train these modules from scratch. In GCP module, each category token from the text encoder block is independently cross-attended corresponding vision queries to acquire rich details (Principles 1,2). Then, the interleaved category tokens are gated by the GCP module so that the frozen text encoder is kept intact at initialization for improved stability and generalization performance (Principle 3).
Inspired by M.
Formally, given image- text pair (I, T) with vision queries V = {,//, (o,") from h exemplars of each category. the GCP module augments each text query features in T = Φ (T) = (4))[)
رコ= 1! extracted
ICl in residual-like way, namely,
V, = X-MHA(V,; I), é, = X-MHA((,, V.), t 6, i= 1,2, ICI, (2)
where X-MHA(---) denotes a multi-head cross-attention layer with the former input as queries and the later as keys and values. = tunh(-)) is normalization function. Concretely, the vision query features v. = 4, (v:) are first augmented via the target image feature I I= & (Z) to gather content- aware information [52] The augmented vision queries V; are correlated with corresponding text token 4 to enrich from multiple views. The conditional gating value a (gate( (f.) dynamically adjusted according to the quality of visual cues from the exemplars of each category. which is evaluated by a three-layer perceptron (MLP) that reduces the feature dimension gradually to a layer-specific learnable scalar. The conditional gating value multiplies the enriched feature ég before adding it to the initial text token & from the residual connection. Since the gating value is initialized to 0, the module output matches that of the pre-trained text encoder initial training. improving training stability and final performance. It is worth noting that this independent class-wise cross-attention scheme has no class-specific parameters and can be scaled any classes of various description granularity, as verified in Sec. In addition, it importantly allows the model to seamlessly generalize to any number of vision queries in inference, regardless of the number used during training.
2.3 Modulated pre-training
Equipped with the plug-in architecture design described in Sec. 2.2 we propose an efficient pre- training strategy that permits sufficient multi-modal fusion and rapid learning upon frozen detection foundation models using only a small subset of pre- mee-trinnng data. We denote this process as modulated pre-training in comparison to the massive pre-training of foundation models.
Vision query extraction. We provide vision queries for |C] categories from a bank 7) built from pre-training data:
D [v.l K] of = Rol
Pool =ool
7b), j= ..., K,
where each V; IR. kxd contains K vision queries modeled by the image encoder $, Specifically, given a query instance of the i-th category with box E IR in an image I, an Rol pooler [33 is employed to extract the corresponding region feature o.55 R2d The parameter y = 1.5² enlarges the

---

Learnable
Target Image
Target Image
Target Image
Multi-modal Queries
clownfish ... stinggry
clownfish
stingray
[CLASS]
f---
Image Encoder
Text Encoder
+
Image Encoder
Image Encader
Image Encoder +
GCP
(a)
Figure 2: Architecture designs for query enriching, with only classification branches presented for clarity. (a) Prompt-basee designs [53 [2 only tune upon language instruction while (b) two-branch few-shot detection architectures [16 21 employ heavily coupled parallel encoders and merely take vision inputs. (c) MQ-Det elegantly combines both language and vision queries in an efficient way.
region area for contextual clues. During modulated pre-training, we randomly select k: < K vision queries from D for each category at each forward process, simulating the downstream scenarios where only low-shot vision queries are provided. In practical implementation, we set K 5000 and k = 5. After modulating, our model can generalize to arbitrary categories and vision queries during downstream inference.
Training upon frozen pre-trained language-queried detectors. Given the inherently general semantic representations in the pre-trained detector, we argue that the textual features only require minor modulation from the visual details rather than significant alteration. This viewpoint has been previously discussed in [52 1. and we have observed Tab (cb that the full-model training on limited number of categories faces the risk of catastrophic forgetting. Consequently, for efficiency purpose, we freeze the entire pre-trained detector and only train the newly-added gated class-scalable perceiver modules, significantly accelerating the training process.
Vision conditioned masked language prediction. Since the fundamental VL models highly rely on text description, the learning process of the residual-like gated architecture will rapidly converge around the initial optimum point with frozen text features, causing failure of vision queries. That is, models still learn how to align region with text features not visual cues. We represent such an issue as learning inertia problem. As shown in the first row of Tab. (a), the AP merely improves +0.9% over GLIP-T [25] on LVIS. To address this issue, propose simple yet effective masking strategy:
T (2,,,2...... [MASK] ...,
Namely, given an image-text pair (I, T), ). ground-truie truth text token in T corresponding to instances occurring randomly masked by [MASK] token with probability 40%. The [MASK] token forced to extract visual cues from vision queries in the GCP modules to provide accurate predictions, thus bridging the model's dependence upon vision queries. As shown Tab.[] our language masking strategy ensures sufficient visual intervention in the modulated pre-training stage and significantly boosts the performance, e.g., +4.4% AP over GLIP-T on finetuning- LVIS.
3 Experiments
3.1 Setup
3.1.1 Datasets and benchmarks
Objects365 dataset (36 is large-scale, high-quality dataset for object detection. We use this dataset to conduct the modulated pre-training our MQ-Det models. It contains 0.66 million images 365 object categories, with 30 million bounding boxes, which are more diverse and fine-grained than those in other popular datasets such as COCO (2 and Paseal VOC [10] Objects365 is a widely-used
pre-training dataset in previous foundamental detection models 25 48
50
44 42]
LVIS benchmark evaluation, with many categories that scarcely exist in the pre-training datasets. Therefore,
B is a challenging dataset for long-tail objects. It contains 1,203 categories for
5

---

Table I: Finetuning- detection on the LVIS benchmark. denotes supervised approaches. The training time tested on one V100 GPU. We present the number of vision queries during evaluation.
Model
Backbone
Pre-Train
Data Size
Training Time
days)
dVision Query
MiniVal AP,
AP,
AP/
AP
v1.0 (R)
AP,
AP/.
MDETR [3]
RNIOI
GoldERReiC
0.9M
400
0
24,2
20.9
24.9
24,3
22.5
74
22.7
25.0
Mask R-CNN [17]
RNI01
0
33.3
26.3
34.0
33.9
Supuriscu-RPP [3]
RNS0
25.4
12.3
24.3
32.4
GB
Swin-T
0365
0.66M
300
13.5
11.3
18.6
GLIPT
Swin-T
0365, Golddi OC44
5.5M
480
0
26.0
20.8
21.4
31.0
17.2
10.1
12.5
25.5
GIJPp2-T [53
Swin-T
5.5M
0
29.0
GroundingDINO-7 Xd
Swin-T
O365, Gewiii..Gliii..o Cap4hi
5.5M
0
25.7
15.2
21.9
30.9
GLIP-L
Swin-L
GeldG GoldG, Cap24M
27.5M
600
0
37.3
28.2
34.3
41.5
26.9
17.1
23.3
35.4
GroundingDNNO-L
2
Swin L
0265.0..cold...gg:MM.......
15.8M
33.9
MQQ-GI
Swin-T
036
0.66M
10
5
17.6
12.0
14.5
21.2
12.4
8.9
9.2
18.3
MQ-GLLP---Ta
Swin-T
036
0.66M
10
26.0
20.8
21.4
31,0
17.2
10.1
12.5
25.5
Swin-T
036
10
21.7
22.1
12.9
31.4
MQ- GLIP-T
Swin T
036
0.66M
10
30.4
21.0
27.5
34.6
22.6
15.4
18.4
30.4
MQ-GLIIP.
Swin-I.
036
0.66M
22
43.4
34.5
34.7
26,9
32.0
41.3
use LVIS in the downstream task to evaluate the finetuning-free transferability. We report on MiniVal containing 5,000 images introduced in MDETR (20) as well as the full validation set v1.0.
ODinW benchmark [23] (Object Detection in the Wild) is a more challenging benchmark for evaluating model performance under real-works scenarios. For example, Aquarium requires locating fine-grained fish species, and Pothole concerns detecting holes on the road. It collects more than 35 datasets for evaluation. There
commonly used versions of the ODin W benchmark, i.e., ODinW-13 and ODin where ODin W-13 is a subset of ODin and contains much less noisy data. The results both benchmarks reported for comprehensive comparison. We demonstrate that the fine-graine -reigraind visual details in MQ-Det facilitate transfer to such diverse and challenging tasks.
3.1.2 Implementation details
We conduct extensive experiments three settings: an open-set setting on finetuning-free LVIS [3] and ODin [23] benchmarks, a few-shot setting and a close-set full-shot setting on ODinW. To demonstrate the plug-and-play versatility of our approach, we apply MQ-Det on two typical SoTA language-queried object detectors, GLIP (25 and GroundingDINO (27). and obtain our multi-modal queried models MQ-GLIP and MQ-GroundingDINO, respectively. We incorporate our GCP modules into the last 6 layers of the frozen text encoder. We conduct modulated pre-training of our models on the Objects365 dataset (36) for only one epoch using 8 NVIDIA V100 GPUs. The finetuning process under few/full-See sshh/ settings completely follows the baseline method GLIP, where vision queries are extracted from the few/full-shot training set. We also evaluate our method in a finetuning free setting, namely, users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. During finetuning-fis evaluation, we extract instances as vision queries for each category from the downstream training set without any finetuning.
3.2
MQ-Det helps low-shot object detection
3.2.1
Multi-modal queried detection without finetuning
We evaluate the model's ability to recognize rare and diverse objects on both LVIS and ODin in a pretuniigg-ree setting, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. Each category is provided with both language and vision queries. Tab. shhows the results on LVIS. Overall, MQ-Det demonstrates strong finetuning- free transferability with impressive efficiency on both data usage and training time. MQ-GLIP-L surpasses the current SoTA by large margin with simply 5 visual exemplars provided, reaching 34.7% AP on Val v1.0 (+7.8% over GLIP-L), which verifies the superiority of multi-modal queries over single-modal queries. Meanwhile, MQ-Det demonstrates good training efficiency, e.g., MQ- GLIP-T only requires additional 2% training time and 12% data usage when modulated upon GLIP-T.
Additionally, we find three contributing factors: (i) Strong generalization comes from combination of textual breadth and visual granularity. In Tab. MQ-GLIP-T-Img replaces all text queries with [MASK] tokens and solely utilizes vision queries to predict objects, achieving 17.6% AP.
Modulating upon pretrained models indirectly utilizes their pre-training data.

---

Table 2: Results on the ODin W benchmark. The few-shot performance is evaluated by 3-shot
27
Data
ODinW-35
ODinW-13
Model
Langunge Vision Query
Backhome
Pre train Data
auy
AP... Doy
Finetining-free Setting
MDETR
ENBS RS
GoldG.RefC
0.9M
10.7
25.1
OWL-VIT X0
VIT L/44CCP))
0365, VG
0.8M
18.8
40.9
GLIP-T
Swin-T
5.5M
18.7
41.9
GLIP-L
Swin-L
FourODs, GoldG,Cap22M.
27.5M
51.0
OmDet
ConvNeXi-B
COCO,0365LLV Phraasest
GLIPv2-T
Swin-T
0365, Copp
DeCLLP
Swin-T
O36, Do6dGG YFCCIM
2.4M
43.3
GroundingDINO-T
Swin-T
O365, Capp
5.5M
Swin-T
036
MQ-GLIP-T
Swin-T
036
MQ-GLIP-L
Swin-L
036
0.66M
23.9
54.1
Shot Setting
DyHead-T
Swin-T
0365
0.66M
37.5
GLIP-T
Swin-T
0365,Gol00,.apPM
Swin-T
0365
50
X
Cen-NeXiBB
COCO.0365.VIIS.
MQ-GLIP-T
Swin-T
036
0.66M
43.0
$7.0
Full
Sull-hot Shot Setting
GLIP-T
X
Swin-T
0365,GiolUO,Cap4A
DyHead
X
Swin
0365
DINO Swin
Swin.
O365
ConyNeXB-B
COCO.036..VVSS
DINO-Swin.I.
X
Swin-T.
O365
MQ-GLIPPT
Swin
0361
0.66M 64.8
Table 3: Ablation results. We evaluate finetuning-free performance LVIS and ODin Both average and median values on 13 datasets of ODin are reported. Each row this ablation study table should be compared to the baseline MQ-GLIP-T reported the top of the table.
Ablated
Ablated Details
MQ-GLIP-T
Changed Value
LVIS MiniVal (%)
ODinW-13 (%)
Setting
Original Value
AP
AP,
AP.
AP,
AP
AP
Original GLIP-T model
26.0
21.4
31.0
MQ-GLIP-T model
30.4
21.0
27.5
34.6
45.6
48.8
40%
0%
26.9
20.1
23.4
31.2
42.8
46.8
Mask Strategy
Mask Rate
40%
80%
28.9
24.2
34.6
45.0
48.3
Enable
X
27.1
19.4
24.4
31.1
44.1
(b)
Gate
Scaler
29.3
18.3
26.3
34.0
43.4
46.0
Architecture
Linear
30.0
26.9
34.5
44.4
47.3
Input
cat(ú, f)
29.2
26.0
34.2
44.8
48.2
Freezing
Full Model
X
24.4
28.6
21.1
17.2
38.6
45.4
(c)
Detector
Text Encoder
x
26.6
17.3
23.0
31.5
42.7
46.6
This confirms that our low-cost modulated pre-training strategy introduces sufficient visual cues. Augmenting generalizable language with visual details, MQ-GLIP-T outperforms the text-only model MQ-GLIP-T-TX by large margin. (ii) Efficiency on data usage and training time benefits from frozen pre-trained detection models. Our approach preserves the generalizable knowledge in the frozen pre-trained models and thus only needs little additional cost further modulation. This is also verified in Tab, (c). (iii) Our simple plug-and-play architecture design enables wide-range applications, since most language-queried detectors share the similar parallel architecture design as GLIP and GroundingDINO, e.g., MDETR 220), OWL-ViT [30], and DetCLIP 42 in Tab 2

---

713
70
Language Query zehro, cow, elk...
02.
2obro PERS
Vision Guery
506
D MIGGEEPT
GUP-T
Query
O GII--TRR
X228
Queery
No Queery
3
10
9,41
Figure 3: Average precision (%) on ODinW-13, from finetuning-free to full-shot data. MQ-Det clearly im- proves GLIP on data efficiency.
Figure Finetuning-free detection with different vision queries. The language queries of 1,203 LVIS categories are always provided. Similar boxes are located while vision queries with better quality provide more accurate predictions.
3.2.2 MQ-Det as a strong few-shot learner
The experiments on ODinW with respect to finetuning-free, few-shot and full-shot settings are listed in Tab.2, where MQ-Det shows robust performance on this challenging benchmark. We observe that most large-scale fundamental detection models only support language as category queries. With additional vision queries, MQ-GLIP-T averagely improves GLIP-T by +3.7% AP and +6.3% AP in the finetuning-rree and few-shot settings respectively on ODinW-13, thus exhibiting the superiority of the rich description granularity of vision queries in our approach. Meanwhile, MQ-GLIP sets a new record on the ODinW-35 few-shot setting with 43.0% AP. Fig. demonstrates few-shot performance with respect to different amounts of training data. We also notice that MQ-GLIP-T makes limited gains over GLIP-T with sufficient training data in the full-shot setting (e.g. +0.6% AP on ODinW-13). The reason is that the learning procedure, with sufficient training data, degenerates traditional close-set detection problem, where the model just represents categories as index regardless of textual or visual descriptions.
3.3 Ablation studies
Importance of the masked language prediction strategy. We propose our masked language prediction strategy to address the learning inertia problem described in Sec. Sec. Tab.3]cs verifies the effect our pre-training strategy. We observe that our approach holds great performance degradation (-3.5% AP) on LVIS with the mask rate of O. MQ-GLIP-T with the mask rate of 0 only improves GLIP-T by +0.9% AP on LVIS. This indicates that our multi-modal queried MQ-GLIP may lose its effect and degenerate language-quericn GLIP without the masked language prediction strategy. Meanwhile, a too large mask rate (e.g., 80%) may lead to overfitting on the vision queries and losing attention to semantic-rich language queries, thus impairing performance.
The role of vision queries. MQ-Det provides users a flexible way to construct customized vision queries. We use one vision query for each category of LVIS in MQ-GLIP-T, and investigate the effect of different query qualities as shown in Fig The vision queries are selected based on the following criteria: (i) Positive queries (in red) contain complete information about the target objects, which significantly improve detection. (ii) Hard positive queries (in brown) only provide partial object information and sometimes include additional noise, such as the example of the boy and car. Despite the additional noise, MQ-Det can filter it out and still help improve the detection accuracy to some extent, identifying at least one object correctly. (iii) Negative queries (in green) contain too much noise and do not benefit detection, while (iv) "no vision queries" (in blue) is used as a baseline. Finally, we also use (v) mixed queries (in purple) that combine the three types of vision queries. The results show the robustness of MQ-Det, namely, our approach can select relevant information from miscellaneous vision queries boost detection.
Architecture design. We design plug-and-play GCP module so that visual details can be smoothly interleaved with highly-semantic textual cues. Particularly, in GCP, the conditional gating mechanism on each interleaved category token in Eqn. (2) ensures smooth model initialization from the frozen model and stable modulating according to different quality of visual cues. As shown in the first two
8

---

rows of Tab. (b). roughly adding visual cues to each text token, or equally scaling them using the same learnable gating scalar for all vision queries, can result in sub-optimal performance (-3.3% AP and % AP on LVIS, respectively). When using simple gating design consisting of one linear layer over different vision queries, we obtain a slight performance drop compared to the non-linear MLP design, indicating content-aware gating mechanism is necessary and more parametric design helps analyze the quality of vision queries. In order to encourage the gate to balance the vision query o and the text query & in Eqn. (2), we combine the two query types a joint input (the last row of Tab (b)) However, we discover that this task rises the learning difficulty for a simple gate design.
F Freezing pre-trained detectors. Tab.3(c) shows the necessity of modulating on the frozen models to introduce vision queries into the language-queriet detectors. Either modulating the whole pre-trained detector or only modulating the text encoder with the limited detection data inevitably lose the generalization capability of the initial language-queried model, leading to significant performance decrease (-6.0% AP and -3.8% AP on LVIS respectively). Moreover, training without freezing the detector requires almost twice the time consumption, not to mention the additional memory cost.
4 Related work
Language-queried object detection. Recently, vision-am sssion-and-aange and-language approaches have gained popu- larity in visual recognition. where language is used to query visual objects and enable open-set ability. CLIP [32] and ALIGN [19] are pioneers cross-modal image-level contrastive learning [5]. where they train on millions of image-text pairs to perform open- vocabbocyyl image classification. Inspired by these works, object detection has pursued cross- mossl region-level contrastive learning two main pipelines. One pipeline is open- -ecabuacyyllary object detection (OVD) [46] 12, 54
[1
30 45],
where the detection data is rigorously split into base classes for training and novel classes for evalu- ation. For instance, Zareian et al. [46] propose OVR-CNN that first pre-trains a visual encoder on image-caption pairs to learn rich vocabulary and then finetunes on detection data with only base classes. Detic (54) learns object localization and fine-grained vision-language matching simultane- ously using max-size proposals to assign image-level labels. However, this pipeline utilizes limited detection data and only generalizes to specific datasets, e.g., from COCO [26] base classes to COCO novel classes. In contrast, our approach follows the second pipeline of large-scale fundamental
20 50], which aims to generalize to arbitrary categories with extensive region-level image-text data. MDETR (20 trains an end-to-end DETR 3 model via phrase-region matching existing multi- modal datasets, providing a strong baseline for few-shot object detection. GLIP [25]] 48) formulates object detection as a grounding problem and leverages additional grounding data for aligned semantics at phrase and region levels, achieving even stronger performance on fully-supervised detection benchmarks without finetuning. GroundingDIINO (27 applies GLIP's grounded training strategy to a stronger detection backbone DINO [4]. achieving of-te-of-theart open-set detection performance. Since these models all use text encoder like BERT (8) to model the language queries, our approach can seamlessly integrate vision queries with these language-quericr models to enable rich visual granularity.
detection models 42 25
48
Vision-queried methods commonly exist in current two-branch based few-shot object detection pipelines [16 II [3] These methods are based on a siamese network to process the target images and vision queries in parallel, and calculate the similarity between image regions (usually proposals) and few-shot examples for detection. Kang el al. 21 propose a feature reweighting module to aggregate the target image features via few-shot vision queries. Meta Faster R-CNN [15] performs feature alignment between the two inputs and focus on foreground regions using attention. FCT 16 proposes an carly fusion transformer architecture fuse target images and vision queries. However, the two-branch pipeline in few-shot detection lacks instruction of language, and is far from practical implementation, e.g., no more than 10% AP in 3-shot COCO (26 for current SoTA methods [6, 15), not to mention more challenging LVIS 13 and ODinW 23 benchmarks. In addition, only a very few open- vocabulaay object detection approaches support both language and vision queries. OV-DETR 45 proposes to utilize CLIP B2) embeddings from both image and
text encoders as the object queries of the DETR 3 decoder. OWL-ViT (30) uses detection data to finetune upon CLIP. Thanks to the parallel structure of CLIP, OWL-ViT can conduct both region- text and region-image similarity. However, these two methods do not support language and vision queries as joint inputs, and show low performance with vision queries. Our MQ-Det considers the

---

complementary nature of vision and language queries, thus achieving both open-set generalization and fine- une-grained recognition.
Conclusion
MQ-Det is efficient pre-training architecture and strategy design let vision-language (VL.) detection foundation models accept text interleaved with images as the input queries. After modu- lated pre-training, MQ-Det shows promising gains on the finetuning- well-established benchmarks and 35 downstream tasks. Our results suggest that utilizing the comple- mentary nature of language and vision is an important step towards open-word ween-world object detection.
and few-shot settings on
Acknowledgements
This work supported by Key-Area Research and Development Program of Guangdong Province (201180101400002,, National Natural Science Foundation of China (No. 62036012, 62322212, 62072455, 61721004), and by Tencent Rhino-Bird Research Elite Program.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 1122776--2776,, 2022.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances Neural Information Processing Systems, 13:1877-1901 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey 7agoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 213-229, Springer, 2020.
[4] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. arXiv preprint 10X00:2210.01253, 2022.
[5] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[6] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7373 7382, 2021.
[7] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255 IEEE, 2009.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810,04805, 2018.
[9] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14084-14093, 2022.
[10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-338, 2010.
[11] Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu- Wing Few-shot object detection with attention-rpn and multi-relation detector. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4013 4022, 2020.
[12] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Oopen-oocbulle object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.1392. 2021.
10

---

[13] Agrim Gupla. Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5356- 5364, 2019.
[14] Guangxing Han, Yicheng He, Shiyuan Huang, Jiawei Ma, and Shih-Fu Chang. Query adaptive few-shot object detection with heterogencous graph convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3263-3272, 2021.
[15] Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, and Shih-Fu Chang, Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 780-789, 2022.
[16] Guangxing Han, Jiawei Ma, Shiyuan Huang. Long Chen, and Shih-Fu Chang. Few-shot object detection with fully cross- transforrrr. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5321-5330, 2022.
[17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vixion, pages 2961-2969, 2017.
[18] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6700-6709, 2019.
[19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-499 4916. PMLR, 2021.
[20] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[21] Bingyi Kang. Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1420-8429, 2019.
[22] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-73, 2017.
[23] Chunyuan Li, Haotian I Liu, Liunian Li, Pengchuan Zhang. Jyoti Aneja, Jianwei Yang. Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, al. Elevater: A benchmark and toolkit for evaluating language-augmenter visual models. Advances in Neural Information Processing Systems. 35:9287-9301 2022.
[24] Hao Li, Jinguo Zhu Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang. Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. arXiv preprint arXiv:2211 1.09008, 2022.
[25] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lci Zhang, Jenq-Neng Hwang, et al. Grounded ianguage-image pre- pre-training, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965- 10975, 2022.
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pages 740-755. 75007 Springer, 2014.
[27] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su. Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprini arXiv:2303.0544 2023.
[28] Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xi- aodan Liang. Capdet: Unifying dense captioning and open-world detection pretraining. arXiv preprint 2xXiv:2303.02249, 2023.
[29] Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer and Ming Hsuan Ming-Hsuan Yang. Class-agnostic object detection with multi-modal transformer. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 27, 2022, Proceedings, Part X. pages 512 531. Springer, 2022.
11

---

[30] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection with vision transformers, arXiv preprint w2x.iv:2005.022300 2022.
[31] Vicente Ordonez, Girish Kulkami, and Tamara Berg. 1m2text: Describing images using million captioned photographs. Advances in Neural Information Processing Systems, 24, 2011.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-878888 8763, PMLR, 2021.
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-can: Towards sil-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[34] Shuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang, Shuai Zheng, Mu Li, Alex Smola, and Xu Sun. Prompt pre-training with twenty-thousand classes for open-vocabulary visual recognition. arXiv preprint arXiv:2304. 04704, 2023.
[35] Christoph Schuhmann, Richard Veneu, Romain Bcaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Thco Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-texi pairs. arXiv preprint arXiv:2111.02114, 2021.
[36] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Ohjects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8430-8438 2019.
[37] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed. image alt- dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2556 2565. 2018.
[38] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, 14. PMLR, 2019.
[39] Xin Wang, Thomas F. Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. In International Conference on Machine Learning. PMLR, 2020.
[40] Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang. Universal-prototype enhamcing for few-shot object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9567- 9576, 2021.
[41] Yifan Xu, Mengdan Zhang, Xiaoshan Yang, and Changsheng Xu. Exploring multi-modaa contextual knowledge for open-vocabulay object detection. arXiv preprint arXiv:2308. 15846 2023.
[42] Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Det- clipv2: Scalable opcn-vocabulary object detection pre-training via word-region alignment. arXiv preprint arXiv:2304.045/1 14, 2023.
[43] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[44] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li. Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint 4rX1v:2111.11432, 2021.
[45] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 106-122. Springer, 2022.
[46] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14393-14402, 2021.
[47] Hao Zhang, Feng Li. Shilong Liu, Lei Zhang, Hang Su. Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-en -o--to-end object detection. arXiv preprint arXiv:2203.0 0.3605 2022.
12

---

[48] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unitying localization and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
[49] Renrui Zhang, Rongyao Fang. Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai. Yu Qiao, and Hongsheng Tip-adapter: Training-free -free clip-adapter for better Saiiunl modeling. Proceedings of European Conference on Computer Vision, 2022.
[50] Tiancheng Zhao, Peng Liu, Xiaopeng Lu, and Kyusong Lee. Omdet: Language-awee aware object detection with large-scale vision-langun langugg multi-dataset pre-training. arXiv preprint arXiv:2209. 2022.
[51] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793- 16803, 2022.
[52] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16816 16816-16825, 2022.
[53] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision- languaee models, International Journal of Computer Vision, 130(9):2337 2348, 2022.
[54] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Mísra. Detecting twenty- thousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Confer- ence, Tel Aviy, Israel, October 23 27, 2022, Proceedings, Part IX, pages 350-368. Springer, 2022.
13

---

Appendix
We provide an overview of the Appendix below.
Transfer to downstream (Sec. We elaborate on the additional details to transfer our MQ-Det to downstream tasks, including finetuning-fret free, few-shot, and full-shot settings. The introduction is organized as followed.
Different ways to acquire the vision queries in Sec. A.1 Sec. 3.1.2 in the main text.
This an elaborative description of
A customized tuning approach of MQ-Det in Sec.
which achieves similar performance full-model tuning while only fine-tuning very few parameters.
A.3
Experiments (Sec. B We provide additional training and evaluation details, including:
Detailed results on 35 downstream tasks in ODinW are provided in Tab. and Tab, VII
described Sec.
An explicit evaluation on an open-vocabulas detection setting is conducted to further investigate the generalization of our multi-modal queries, as presented in
Some visualization results of MQ-Det in Sec. B. and Fig.
The training hyper-parameters are illustrated See B.3 and Tab. VI
Further discussion (Sec. C).
We provide a comprehensive discussion on our work, including limita-
tions and broader impacts in Sec|C
A Transfer to downstream
A.1 Different ways acquire vision queries
During finetuning-fre evaluation, we extract 5 instances as vision queries for each category from the downstream training set without any finetuning. We also provide two alternative strategies and observe similar performance, i.e., retrieval and test-time online update, where the former obtains vision queries from heterogeneous external data like ImageNet (7.. and the latter dynamically stores high-confidence instances as vision queries during evaluation. These two additional approaches are proposed to simulate realistic scenarios:
Retrieval (user-provided exemplars): small number of exemplars provided by the users without any fine-tuning. We retrieve 5 exemplars as vision queries for each category from ImageNet-2IK 7 to simulate the user-provided exemplars. These samples are heterogeneous from the downstream test data, e.g., domains. The results are provided in MQ-GLIP- -etrivv-eettvval of Tab.
Online updating: the model dynamically stores high-confidence instances as vision queries during evaluation. No vision queries are provided the initial stage of evaluation. The results are illustrated in See. A.2
of Tab. We provide detailed description in
The results are illustrated in Tab. We select downstream datasets from ODin 23 to verify the effectiveness of each approach. Generally, three approaches to acquire vision queries demonstrate similar performance. We observe that vision queries from online updating hold relatively lower quality, thus leading to slight performance drop, since no manual annotations are provided. Mcanwhile, exemplars retrieved from ImageNet object-centric and contain little noise (e.g., other irrelevant objects), thus improving the performance.
A.2 Test-time online update
Our test-time online update strategy is conducted via the following steps: 1) only utilize language queries to conduct detection at the initial stage of evaluation. 2) Store detected instances with high confidence as the vision queries of corresponding categories. 3) Use both language queries and stored
14

---

vision queries for evaluation and seek for more vision queries. Tab verifies the effectiveness of our approach.
A.3 Partial tuning rivals full-model tuning
Since the GCP modules are interleaved into the frozen detector, we provide customized fine-tuning strategy, partial tuning, namely, only tuning the newly added GCP modules and freezing all other parameters. The results in Tab, II indicate that partial tuning strategy achieves comparable performance with traditional full-mot mull-model tuning (i.e., only -0.4 AP on ODinW-13). Partial tuning accounts for much fewer learnable parameters, thus friendly training time and memory costs. The experimental results the main text are all based on the partial tuning.
Table I: Different ways to acquire vision query. We report the finetuning free performance. All models should compared with the MQ-GLIP-T model at the top of the table.
Table II: Different fine-tuning strate- gies of MQ-Det under the 5-shot set- ting. The implementation used our evaluation is highlighted in color.
Model
AerialDrone Aquarium Rabbits
ODinW-13 13 (%)
Strategy
GLIP-T
12.5
18.4
70.2
APang
midd
MQ-GLIP-T
15.8
23.5
75.4
Partial tuning
59.1
62.4
MO-GLIP-T-Onmime
15.5
23.2
74.8
MQ-GLIIP-TRRetivvll
16.0
23.6
75.1
Full-model tuning
59.5
64.5
Experiments
B.1 All results
We report the per-dataset performance under various settings in ODinW-13 and ODinW-35, shown in Tab. II and Tab. VII respectively. We also provide some visualized results in Fig.
Table III: Per-dataset AP performance (%) ODinW-13. We report results 0, 1, 3, 10-shot detection. MQ-GD-T denotes MQ-GGroundingDINOO- Specifically, the "zero-shot" here actuall stands for the finetuning-frer setting with 5 vision queries.
MQ-GLIP-1
MQ-GLIP-L
MQ-GD-T 0
Dataset
5
10
0
3
5
10
Pas
59.8
52.9
58.9
59.3
59.6
64.7
64.7
67.4
68.1
68.8
57.5
Drone
15.8
22.1
29.8
31.0
31.6
17.4
30.7
36.1
37.0
36.1
13.6
Aquarium
23.5
31.7
36.1
40.1
42.4
30.3
39.2
45.8
47.0
49.7
18.5
Rabbits
75.4
76.2
77.4
75.6
75.5
71.8
76.0
75.0
74.3
75.3
79.9
EgoHands
41.2
64.3
66.0
66.7
68.2
57.2
68.8
68.1
71.6
72.6
65.4
Mushrooms
61.0
89.7
89.0
91.8
89.0
63.9
87.4
91.6
90.7
92.2
68.2
Packages
68.5
71.9
72.8
73.7
74.4
53.0
70.6
71.2
72.0
73.5
64.1
Raccoon
41.6
61.2
64.8
65.5
61.9
58.1
70.9
73.3
72.0
76.7
49.2
Shellfish
26.6
27.9
34.2
41.9
40.0
63.0
61.1
60.1
62.8
60.7
29.2
Vehicles
57.2
60.6
59.5
65.7
65.6
63.2
68.3
70.2
71.2
72.4
56.7
Pistols
59.6
56.5
60.3
61.4
61.7
74.4
73.6
72.7
74.3
74.8
69.2
Pothole
14.7
26.7
28.0
33.7
36.4
27.0
30.9
30.8
36.9
38.7
25.2
Thermal
48.0
59.4
64.2
62.4
72.7
58.7
68.5
72.2
72.5
74.5
64.9
Average
45.6
53.9
57.0
59.1
59.9
54.1
62.4
64.2
65.4
66.6
50.9
B.2 Explicit evaluation on open-vocabulary detection sefting
To further investigate the transferability MQ-Det, we evaluation our models on clear separation of base and novel classes, which is similar to previous vocabuacyy object detection (54
41 We
first construct a novel category set from 1,203 LVIS categories. Specifically, we remove LVIS categories that exist in the 365 classes of 0bjects365 and finally obtain 986 novel categories that
15

---

did not appear during our modulated -re-trainng. The remaining 217 categories are represented as base categories. Then, we conduct finetuning-fres inference with vision queries on the separated categories to verify the generalization of multi-modal query learning. Tab. shows the results. The results indicate that multi-modal queries generalize well to novel classes that do not exist in the modulated pre-training. Specifically, +4.1% %. +5.7%, and +6.3% AP on novel classes of Do-GrrundingDNNOOT,, MQ-GLIP-T, and MQ-GLIP-L over their baselines, respectively.
Table IV: Finetuning- free detection with explicit open-voeabulary category separation LVIS.
Model
AP novel
AP
APall all
GroundingDINO-T
22.1
36.7
25.6
GLIP-T
20.8
42.0
26.0
GLIP-L
35.4
45.5
37.9
MQ- MIO-GroundingDNNO-T
26.2
43.0
30.2
MQ-GLIP-T
26.5
42.8
30.4
MQ-GLIP-L
41.7
51.3
44.0
It is worth noting that the separation of base and novel classes differs from previous works open- vocabulary detection (OVD) [46]. The reason is that the testing categories of previous separation are represent the classes LVIS that do not exist our modulated pre-training dataset Objects365 as novel classes. The frequency distribution of the separated LVIS dataset is shown Tab. V
partially included in our pre- pre-training dataset Objects365 [36]. Therefore,
Table V: Frequency distribution of the separated LVIS for open-vocabulary evaluation.
Class
#Rare
#Common
#Frequent
Novel
326
404
256
Base
11
149
All
337
461
405
B.3 Training hyper-parametter
We report the hyper-panee pppee-paraeeter settings of the modulated pre-training of MQ-Det in Tab. VI Other settings are the same with corresponding language-querier detectors.
Table VI: Hyper-parameters modulated pre-training.
Item
Value
Item
Value
optimizer
Adam W
max vision query num (K)
5000
Ir of GCP
1e-5
vision query num (k)
Ir of gate
5e-3
mask rate
40%
weight decay
le-4
layer with GCP
6~~12
C Further discussion
Limitations. First, multi-modal queries make limited contribution with sufficient training data for each category. This may because the foundation models learn enough accurate classification boundaries, thus reducing the efffcctiveness of language and vision queries. Second, the applications of MQ-Det on other dense prediction tasks such as segmentation remain unexplored.
Broader impacts. MQ-Det shows strong downstream transfer ability with highly flexible category vocabularies. This allows inexperienced users to easily use MQ-Det models (e.g., MQ-GLIP) for their own needs by simply providing some visual examples and corresponding text descriptions. However, this also raises concerns about how our MQ-Det models with large vocabulary could be used inappropriately in the community, such as for large-scale illegal video surveillance. The
16

---

open-set detection capabilities could be manipulated through specialized visual or textual cues to facilitate targeted detections instead of generic ones. This manipulation could introduce biases in the detector and result in unfair predictions.
Table VII: Per-dataset AP performance (%) on WDinW-35. We report results on 0, 3-shot detection. MQ-GD-T denotes Mo-Grouudinng Specifically, the "zero-shot" here actuall stands for the finetuning-free setting with 5 vision queries.
MQ-GLIP-T
MQ-GLIP-L
MQ-GD-T
Dataset
0
3
0
0
AcrialMaritimeeDone__aaa
15.8
29.8
17.4
13.6
AerialMaritimeDrnne_er tiled
18.3
27.1
20.8
21.9
AmericanSignl ImerrcanSigmllanguggl.tttee
1.8
19.9
3.0
0.1
Aquarium
23.5
36.1
30.3
18.5
BCCD BCCD
4.3
57.3
12.4
12.3
ChessPiece
10.7
64.8
2.8
14.9
CatbontailRabbits
75.4
77.4
71.8
79.9
DroneControl Cronec Control
6.4
40.9
9.3
1.2
GooHands_generic
41.2
66.0
57.2
65.4
EgoHands_speciee
3.6
32.1
6.7
0.1
Workers
5.4
37.4
5.5
4.8
MaskWearing
0.3
46.3
1.0
0.0
MountainDewConmercaal
49.2
46.4
17.7
39.9
NorthAAmericaMuShroomss
61.0
89.0
63.9
68.2
OxfordPets_ _yybbreedbbbbbeed
0.4
31.5
0.5
0.4
OxfordPets_by--ooeee speciis
1.6
68.1
0.4
0.5
PKLot_640
0.9
30.3
2.6
0.0
Packages
68.5
72.8
53.0
64.1
Raccoon Raccoon
41.6
64.8
58.1
49.2
ShellfishOpenlmages
26.6
34.2
63.0
29.2
ThermalCheetah
2.3
41.1
11.1
7.1
UnoCards
0.2
35.4
0.8
0.0
VehiclesOpenImages
57.2
59.5
63.2
56.7
WildFireSmoke
13.2
22.5
20.5
14.6
boggleBoards
0.1
76.5
0.1
0.0
brackishUnderwater
4.5
31.3
5.3
3.3
dice mediumColor
0.4
14.6
0.6
0.1
openPoetry Vision
0.0
3.1
0.0
0.1
pistols
59.6
60.3
74.4
69.2
plantdoc
1.7
12.5
1.5
0.2
pothole
14.7
28.0
27.0
25.2
selfdrivingCar
8.2
17.5
9.0
6.9
thermalDogs
48.0
64.2
58.7
64.9
websiteScreenshots
1.0
6.6
1.7
1.0
PascalVOC
59.8
58.9
64.7
57.5
Average
20.8
43.0
23.9
22.5
17

---

Figure I: Visualized results of MQ-GLIP-T on the LVIS benchmark.
Prediction
Annotation
Prediction
Annotation
J.P.Mo 13
J.P.Morgan
Zable
Icomppter squip
Pou Icomputer equipment: Eemenfor Tcain For HQUED pepppennn ComTuuter
drenoer 50%
computer kcyboard
20OR 30
ISEATO
18


Reviews:
[{"review_id": "uJVeaxXCIN", "reviewer": "Reviewer_JLFL", "summary": "In this paper, the authors proposed MQ-Det which leverage both textual and visual information for object detection in the wild. The proposed plug-and-play GCP module is very compatible with existing mainstream architectures. The authors conducted extensive experiments on multiple benchmark datasets and showed improved performance over several baseline methods for both zero-shot and few-shot detection.", "questions": "Please refer to the weaknesses section for details.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Leveraging both textual and visual info for open-vocabulary detection makes lots of sense as they provide different level of signals for recognizing objects as mentioned in the draft. The proposed GCP module and masking based training strategy is a novel technique to combine those informations.\n2. The authors conducted extensive experiments and ablation studies to sufficiently validate the proposed approach over multiple baseline methods on several benchmarking datasets.\n3. Writing is good and easy to follow. The tables and figures are also easy to understand.", "weaknesses": "1. For the last row of Tab. 3 (b) where the joint input leads to lower performance, the authors' explanation is that \"this task may introduce redundant information and rise the learning difficulty.\" This is not very clear to me since redundancy could also lead to easier training rather than difficulty. Could you please elaborate more on this?\n2. For Fig. 4, it is weird that the baseline 'No Vision Query' (blue box) model actually recognize the two zebras as either elk or horned cow. These two zebras looks very easy to recognize even without visual cue for large VL models. Is this due to insufficient training of baseline models or hand-picked examples to illustrate the idea?\n3. As illustrated in Fig. 3, with increasing amount of training data, the gap between MQ-Det and other baseline methods are getting more and more closer. What would be a rule of thumb strategy for using MQ-Det versus other simpler methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. Please find below our responses to the questions raised in the review:\n\n> **More explanation on redundancy with joint input in the last row of Tab. 3 (b)**\n\nThe learning difficulty derives from the simple concatenated input $cat(\\hat v_{i}, t)$, which equally considers vision and language information to learn the scale of vision queries. This may impede the gate on learning from vision information that contributes the most to the vision scales. In the future, we will explore a more reasonable combination of the two modalities to benefit the performance.\n\n> **Question on the baseline 'No Vision Query' model in Fig. 4**\n\nThis example is selected from the bad cases of GLIP that is equal to the 'No Vision Query' model. The reasons of this failure are twofold. First, it is rather challenging to recognize \"zebra\" from all 1,203 categories in LVIS, since the model is more prone to make mispredictions as the number of categories increases. Second, \"zebra\" is not included in the detection training set (Objects365), making it a novel class. Through the use of multi-modal queries, we can provide the model with more clues to the category, thereby improving detection performance.\n\n> **A rule of thumb strategy for using MQ-Det versus other simpler methods**\n\nThank the reviewer for the thoughtful question. We delve deeper into investigating whether there exists a rule of thumb strategy for deploying MQ-Det. In the table below, we gradually increase the data size (number of shots) and record the performance gap between MQ-GLIP-T (multi-modal queries) and GLIP-T (language queries). We report the averaged results on 13 benchmarks of OdinW-13 and two specific datasets. \n\n| Performance Gap                         | 0    | 1    | 3    | 5    | 10   | 50   | full |\n| --------------------------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| $AP_{MQ-GLIP-T}-AP_{GLIP-T}$ (OdinW-13) | 3.7  | 5.8  | 6.3  | 4.7  | 3.5  | 1.1  | 0.6  |\n| $AP_{MQ-GLIP-T}-AP_{GLIP-T}$ (Aquarium) | 2.2  | 3.2  | 2.6  | 2.4  | 1.5  | 1.4  | 0.9  |\n| $AP_{MQ-GLIP-T}-AP_{GLIP-T}$ (Pothole)  | 6.7  | 5.8  | 6.2  | 7.1  | 7.3  | 2.3  | 0.2  |\n\nHere, we provide several points for the implementation guidance of MQ-Det:\n\n- A rule of thumb for using MQ-Det is that: MQ-Det is widely suitable for low-shot scenarios (empirically below 50-shot), but the case of the biggest improvement varies based on specific datasets. As exemplified in the table above, in OdinW-13, there generally exists an obvious gap between multi-modal queries and language queries under 10 shots. For a relatively simple task like Aquarium, the gap narrows significantly with just 10-shot learning. However, for a more challenging task like Pothole, which is to detect the potholes on the ground, the gap remains large even with 50-shot.\n- It is worth noting that employing MQ-Det is not laborious. The reason lies in twofold. First, our modulating process is efficient for training. Second, the GCP module and the masking strategy are easy to reproduce. We will open-source the complete code and provide more detailed guidance on how to incorporate MQ-Det into customized language-queried detectors, hoping that our work could embrace wider application."}, {"review_id": "66pbCi7fC7", "reviewer": "Reviewer_RdSN", "summary": "The paper introduces MQ-Det, a novel approach for open-vocabulary object detection that combines textual descriptions and visual exemplars as category queries. MQ-Det aims to address the limitations of existing text-queried object detectors by incorporating visual information and providing various granularity in the descriptions. The proposed architecture can be easily integrated with pre-trained language-queried detectors. To overcome the learning inertia problem caused by frozen detectors, MQ-Det employs a vision-conditioned masked language prediction strategy. This strategy involves randomly masking text tokens and allowing vision queries to independently predict objects, ensuring sufficient visual intervention during training. Overall, MQ-Det presents a simple yet effective architecture and training strategy that improves open-world object detection performance by incorporating multi-modal queries.", "questions": "As mentioned in the paper, the selection of visual templates during inference plays a critical role in the success of the proposed approach. I would like to inquire about the impact of random sampling templates on the performance. Specifically, I am interested in whether the authors performed multiple inference runs, averaged the performance results, and presented the inference variance with error bars.\n\nCould you please clarify whether this analysis was conducted and whether the paper provides information on the effect of random sampling templates on performance?", "limitations": "I do not see potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow: The authors present their research in a clear and organized manner. \n2. The paper tackles an important problem: The authors address the challenge of improving the robustness and generalization of current language-query based object detectors by incorporating visual cues. This is a significant problem in the field of open-vocabulary  detection, as relying solely on textual descriptions can lead to insufficient granularity and ambiguous queries. By introducing multi-modal queries, the paper provides a promising solution to overcome these limitations and enhance detection performance in real-world scenarios.\n3.The proposed GCP module and training techniques have broader applicability: GCP module introduced in the paper is a valuable contribution that can have implications beyond the scope of this study. For example, conditional gating layer of the GCP module provide a framework for effectively integrating useful visual information (instead of noisy visual templates) into language-queried detectors. Additionally, the random masking strategy addresses the learning inertia problem and enables better fusion of visual and textual cues. These findings can benefit future research.\n4. The experiment is solid and proves the effectivenss of the proposed method.", "weaknesses": "1. The training data in Table 1 and Table 2 is confusing. It appears that the authors loaded the pretrained GLIP models, which were pretrained on datasets such as Objects365, GoldG, CC4M, and Cap24M. However, the presented training data in those tables only includes the Objects365 dataset for proposed models. It is important to note that the performance of the model is achieved by utilizing all the mentioned pretraining datasets, rather than just the Objects365 dataset. Clarifying this discrepancy in the presentation of training data would provide a clearer understanding of the model's training process.\n\n2. The evaluation of open-vocabulary or open-set object detection requires a clear separation of base and novel classes. Unfortunately, the paper does not explicitly evaluate the performance on these two sets separately. This omission makes it difficult to assess the model's generalization capability specifically on novel classes. To gain a deeper and better understanding of the model's performance in generating novel classes, it is recommended to conduct separate evaluations and present the results accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. Please find our responses below:\n\n> **Training data**\n\nThank the reviewer for your valuable reminder. Our model is indeed  built upon pretrained GLIP/GroundingDINO models, which indirectly utilizes their pretraining datasets. We will provide the following clarification on the data discrepancy to avoid misleading. \n\n1. We will clarify the data discrepancy in Table 1 and Table 2, for example:\n\nTable 1: xxxx. $^\\dagger$ Modulating upon pretrained models indirectly utilizes their pre-training data, and potentially consumes more time if we take the training time of the pretrained language-queried detectors into consideration.\n\n| Model|...|Pre-train Data|...|Training Time|\n|-|-|-|-|-|\n|...|...|...|...|...|\n|MQ-GLIP-T|...|O365 (+GLIP$^\\dagger$)|...|10$^\\dagger$|\n|MQ-GroundingDINO-T|...|O365 (+GroundingDINO$^\\dagger$)|...|10$^\\dagger$|\n\n2. Add more description on the  data discrepancy after the last sentence in line 232: \"It is worth noting that modulating upon pretrained GLIP/GroundingDINO indirectly utilizes their pretraining data. The efficiency here describes that our approach allows current mainstream language-queried detectors to be equiped with multi-modal queries only through a lightweight modulating process, avoiding training from scratch.\"\n\n> **Explicit evaluation on an open-vocabulary detection setting**\n\nThank the reviewer for the constructive recommendation. We add the following experiment to further investigate the model's generalization performance.\n\nWe first construct a novel category set from 1,203 LVIS categories. Specifically, we remove the LVIS categories that exist in the 365 classes of Objects365 and finally obtain 986 novel categories that did not appear during our modulated pretraining. The remaining 217 categories are represented as base categories. Then, we conduct zero-shot inference on the separated categories to verify the generalization of multi-modal query learning. The results are shown in the following table.\n\nMore details: We consider a category in LVIS as a base class if its name or synonyms appear in the category name set of Objects365. The names are all in lowercase and singular form, with all \"()\" and \"_\" removed.\n\n| Model              | $AP_{novel}$ | $AP_{base}$ | $AP_{all}$ |\n| ------------------ | ------------ | ----------- | ---------- |\n| GroundingDINO-T    | 22.1         | 36.7        | 25.6       |\n| GLIP-T             | 20.8         | 42.0        | 26.0       |\n| GLIP-L             | 35.4         | 45.5        | 37.9       |\n| MQ-GroundingDINO-T | 26.2         | 43.0        | 30.2       |\n| MQ-GLIP-T          | 26.5         | 42.8        | 30.4       |\n| MQ-GLIP-L          | 41.7         | 51.3        | 44.0       |\n\nThe results indicate that multi-modal queries generalize well to novel classes that do not exist in the modulated pretraining. Specifically, +4.1%, +5.7%, and +6.3% AP on novel classes of MQ-GroundingDINO-T, MQ-GLIP-T, and MQ-GLIP-L over their baselines, respectively. We will include this experiment in the revised version.\n\nIt is worth noting that the separation of base and novel classes differs from previous works on open-vocabulary detection (OVD). The reason is that the testing categories of previous separation are partially included in our training dataset Objects365. Therefore, we represent the classes in LVIS that do not exist in our modulated pretraining dataset Objects365 as novel classes. The frequency distribution of the separated LVIS dataset is shown in the following table:\n\n| Classes | #Rare | #Common | #Frequent |\n| ------- | ----- | ------- | --------- |\n| Novel   | 326   | 404     | 256       |\n| Base    | 11    | 57      | 149       |\n| All     | 337   | 461     | 405       |\n\n> **Inference variance with error bars**\n\nWe did not report the averaged results of multiple runs in the initial submission. Here, we provide the averaged results with error bars from 3 inference runs using seeds of 3, 30, and 300 for both zero-shot and few-shot settings. Models with different seeds randomly select diffrent vision queries. We will include them in the revised version. Meanwhile, this work uses random sampling templates to investigate the feasibility of multi-modal queries. We leave the study of more complicated template sampling to future work.\n\nLVIS MiniVal zero-shot:\n\n| Model              | $AP$          | $AP_r$        | $AP_c$        | $AP_f$        |\n| ------------------ | ------------- | ------------- | ------------- | ------------- |\n| MQ-GLIP-T          | 30.5 $\\pm$0.1 | 21.8 $\\pm$0.8 | 27.4 $\\pm$0.1 | 34.8 $\\pm$0.2 |\n| MQ-GLIP-L          | 43.7$\\pm$0.3  | 34.8 $\\pm$0.3 | 41.6 $\\pm$0.4 | 47.2 $\\pm$0.3 |\n| MQ-GroundingDINO-T | 30.4 $\\pm$0.2 | 21.8 $\\pm$0.3 | 26.3 $\\pm$0.1 | 35.4 $\\pm$0.2 |\n\nOdinW zero-shot:\n\n| Model              | OdinW-35 $AP_{avg}$ | OdinW-13 $AP_{avg}$ |\n| ------------------ | ------------------- | ------------------- |\n| MQ-GLIP-T          | 20.7 $\\pm$0.5       | 45.4 $\\pm$0.6       |\n| MQ-GLIP-L          | 24.0 $\\pm$0.4       | 54.1 $\\pm$0.3       |\n| MQ-GroundingDINO-T | 22.5 $\\pm$0.3       | 51.0 $\\pm$0.3       |\n\nOdinW few-shot (3-shot):\n\n| Model     | OdinW-35 $AP_{avg}$ | OdinW-13 $AP_{avg}$ |\n| --------- | ------------------- | ------------------- |\n| MQ-GLIP-T | 43.1 $\\pm$0.4       | 57.2 $\\pm$0.5       |"}, {"review_id": "qCfvHvJSpi", "reviewer": "Reviewer_9Uan", "summary": "The paper propose MQ-Det, a novel module that integrates both language and visual queries efficiently for object detection tasks. This module enhances each category token with vision queries, providing rich detailed visual context to the text-based models. The proposed method is experimentally tested on LVIS and ODinW datasets, obtaining state of the art results. The method is versatile and can be applied to other state of the art language queried object detectors.", "questions": "How are the vision queries chosen? How does this affect the performance?\nChoosing to have the related work chapter as the forth one it's unusual, is there any reason for that?\nplease see Weaknesses.", "limitations": "the paper discuss some of the limitations throughout the paper and does not address societal impact.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The overall idea of the paper is easy to understand.\nThe presented method, MQ-Det is novel because it introduces a unique combination of language and visual queries (multi-modal queries) to enhance object detection in an efficient manner.\nTo train the proposed module it's not expensive and it does not require a lot of data.\nThe proposed method, MQ-Det, was evaluated using two state of the art models, GLIP and GroundingDINO. The results demonstrated superior performance when MQ-Det was applied, showcasing the effectiveness of the MQ-Det approach.", "weaknesses": "The comparison in the zero-shot and few-shot scenario is not fair since this method uses \"5 instances as vision queries for each category from the downstream training set\" thus having access to some information from the target dataset while all the other methods do not have any kind of access to that (for example owl-vit for image guided detection also doesn't use any fine-tuning, but when using vision queries from the target dataset they present the results as few-shot -- please see chapter 4.4 from owl-vit). Same observation for few-shot. Also it's not clear how are those 5 instances picked and how does that affect the final performance.\n\nThe time and data comparison against state of the art it's not entirely fair since this method builds on top of GLIP and GroundingDINO. So this method benefits of the training data and training time of GLIP/GroundingDINO, thus the affirmations and comparisons may be misleading.\n\nIt's not clear if the method can be used with text only or you always need some vision queries. Tied to that it's not clear what MQ-GLIP-T-Txt from Table 1 represents. Can you mask the visual queries? The comparison between MQ-GLIP-T-Txt and GLIP-T it's confusing.\n\nVarious meta-parameters seem to be chosen randomly: for example choosing 5 as the vision queries.\n\nUltimately, while I do see the merits of this work, I find the comparisons to be misleading and not entirely fair.\n\nminor typos: visual detials\n\n\nAfter rebuttal: The authors addressed my concerns properly and with the promised modifications I consider that this paper meets NeurIPS standards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. Please find our responses below:\n> **Comparison in the zero-shot scenario**\n\nWe acknowledge that our setting is different from the zero-shot setting in previous language-queried detectors, since we use additional visual exemplars along with texts as category descriptions. Our setting is derived from practical implementation, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. As an example, detecting different types of \"mushrooms\" in the Mushroom dataset of OdinW-13 can be much easier with visual exemplars than ambiguous textual descriptions. It is hard to reach absolutely fair comparison in this setting despite a lot of efforts we have made. The reason is that most methods, except ours, do not support visual exemplars as inputs, while comparing our finetuning-free method with other finetuned methods in a few-shot setting would also be unfair. To avoid misleading, we will make the following revisions:\n\n1. Modify the setting name:\n\n   (a) The name of section 3.2.1: \"Multi-modal queried detection without finetuning\"\n\n   (b) In the begining of line 226: \"We evaluate the model's ability ... in a **finetuning-free setting**, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. \"\n\n2. Add more clarification:\n\n   (a) In the title of Table 1: \"**Finetuning-free detection**  results on the LVIS benchmark. Differently, we provide models with 5-shot instances as vision queries without any fine-tuning...\"\n\n   (b) Lines 13-15 in the abstract: \"For instance, MQ-Det significantly improves the state-of-the-art open-set detector GLIP by +7.8% AP on the LVIS benchmark with multi-modal queries without any downstream fine-tuning...\"\n\n   (c) Similar modifications as (b) in line 52, lines 76-77, and lines 226-232.\n\nAdditionally, we provide two more methods to acquire vision queries in the **finetuning-free setting (the zero-shot setting in the original version)** in Section A.1 of the Appendix, which do not have access to the target dataset.\n\nWe would also like to emphasize that the our goal is to address the insufficient granularity and ambiguous queries that existing language-queried detectors suffer from. To achieve this goal, we propose to equip language-queried-only detectors with multi-modal queries. The main message from our original zero-shot comparison is that, equipped with multi-modal queries, previous language-queried detectors are allowed to detect objects with various  granularity and demonstrate great performance improvement without any finetuning. This confirms our goal and indicates that multi-modal queried object detection can be a promising future direction.\n> **Comparison in the few-shot scenario**\n\nOne thing we'd like to clarify is that the few-shot comparison is absolutely fair. The vision queries are strictly selected from the few-shot datasets, as described in lines 217-218. For example, in the 3-shot setting, we only use the 3-shot training samples as the vision queries for each category, without any additional data.\n> **Comparison on training time and data**\n\nWe will provide more clarification to avoid misleading. Due to the character limit, please refer to the \"`training efficiency on time and data`\" part of our response to Reviewer wDpN, for detailed modifications we will make during revision. \n> **Inference with masked vision queries**\n\nOur approach supports inference with mixed queries, namely, only augmenting a part of categories with multi-modal queries while leaving other categories with single-modal queries. For example, as shown in the table, MQ-GLIP-T-Mix only provides \"birdfeeder\" with both textual descriptions and visual exemplars, while providing \"armchair\" with only visual exemplars and \"straw\" with only textual descriptions. MQ-GLIP-T-Txt only uses text queries, which is equivalent to GLIP. We will add a clear definition on MQ-GLIP-T-Txt in line 238. MQ-GLIP-T-Img masks all input texts and only uses vision queries. The categories are from LVIS MiniVal and the AP results in the finetuning-free setting are reported.\n|Model|Armchair #19|Straw #1024|Birdfeeder #100|\n|-|-|-|-|\n| MQ-GLIP-T| 45.8| 12.0| 2.8|\n| MQ-GLIP-T-Txt | 44.1| 8.2| 0.0 |\n| MQ-GLIP-T-Img | 41.7| 12.3| 4.2 |\n| MQ-GLIP-T-Mix | 41.8 (Img) |8.9 (Txt)|2.8 (Txt+Img)|\n\nWe did not include the mixed results in the initial submission because it is laborious to design a customized query type for each category in the massive benchmarks used in the paper. However, it is relatively easy for users to flexibly adjust the query types to meet their own needs during implementation. We leave this study to future work.\n> **How does the vision queries affect the final performance?**\n\nThe 5 vision queries are randomly picked. We did not employ specific tricks for the selection of vision queries, as even the lower bound achieved through random sampling outperformed the language-queried baseline models. The detailed results with multiple random sampling can be found in the \"`inference variance with error bars`\" part of our response to Reviewer RdSN.\n> **Meta-parameters and the related work chapter**\n\nWe did not conduct a specific search on meta-parameters since it is not the key point of this work. Nonetheless, here we provide an analysis of the number of vision queries in the finetuning-free setting. We observed a clear improvement in detection performance even with a minimal number of vision queries, e.g., 1 vision query.\n\n| #Vision queries|0|1|3|5|10|\n|-|-|-|-|-|-|\n|LVIS AP (Finetuning-free)|26.0|28.5|29.9|30.4|30.6|\n\nWe present the related work chapter as the fourth one to encourage readers to focus on our method. A similar pattern can be found in [1], [2].\n\n[1] Radford A, et al. Learning transferable visual models from natural language supervision. ICML2021.\n\n[2] Chen T, et al. A unified sequence interface for vision tasks. NeurIPS2022."}, {"review_id": "qcxJHyRhGY", "reviewer": "Reviewer_wDpN", "summary": "Based on recent vision-language fundamental models such as GLIP and Grounding DINO, the authors propose an improved multimodal query pipeline. A Gated Class-scalable Perceiver is used to apply cross-attention for both the language and vision query inputs. A masking strategy for text tokens is further proposed to improve the generalization ability. The proposed method achieves strong performance on zero-shot LVIS and ODinW benchmarks.", "questions": "1. I want to verify one point: the proposed method, such as MQ-GLIP, is trained from scratch, or finetuned from the initial weights from GLIP? According to #175-176: \"we freeze the entire pre-trained detector and only train the newly-added gated class-scalable perceiver modules\", I assume that MQ-GLIP is finetuned on the pre-trained GLIP model. Then I think the author overclaims their training efficiency. I think the pre-trained time of GLIP cannot be ignored. For example, in #232, it should be \"MQ-GLIP-T adds extra 2% training time and extra 12% data usage of GLIP-T.\" rather than \"only requires 2%\". In my opinion, MQ-GLIP increases the training time and training data compared to GLIP, since MQ-GLIP brings an extra fine-tuning stage.\n\n2. When evaluating the zero-shot setting such as zero-shot LVIS, does the MQ-Det requires the exemplar images for each class of the LVIS dataset? I guess the answer is 'yes' since the MQ-Det requires both the language and vision queries as inputs. But I think this way may violate the zero-shot setting. In the zero-shot setting, the LVIS classes are treated as unseen classes, and the exemplar images of the unseen classes are not allowed to be accessed. The model can only use the textual descriptions or text attributes associated with the unseen classes. So I think MQ-Det achieves strong zero-shot performance via a cheating inference evaluation. I think using the exemplar images of test-set classes is allowed for the few-shot setting but is not allowed for the zero-shot setting, which will weaken the contribution of this paper.\n\n3. In table-2, the proposed MQ-Det performs worse than DINO-Swin-L on the full-shot setting. Can the authors provide some explainable reasons?", "limitations": "The authors discussed their limitations in section C of the appendix.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper first explores using both vision and language queries as inputs\n- The proposed method achieves superior experimental results over previous methods\n- The authors release the source code", "weaknesses": "- Overclaimed training efficiency, see Q1 in the Questions section. The pre-training time of GLIP should also be considered, which makes MQ-GLIP more time and data-intensive.\n- Potential violation of zero-shot setting, see Q2 in the Questions section. The MQ-Det might improperly use exemplar images in a zero-shot setting, which is against the task definition.\n- Subpar full-shot performance, see Q3 in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. Below, please find our responses to each of the concerns or questions raised in the review:\n> **Training efficiency on time and data**\n\nWe acknowledge that our model is built upon pretrained GLIP/GroundingDINO models, which indirectly utilizes their pretraining datasets and accounts for more training time if trained from scratch. That is the reason we refer to our training process as modulated pretraining (lines 156-159) instead of pretraining. \n\nMeanwhile, we'd like to emphasize that the aforementioned issue does not contradict the efficiency of our method. Our efficiency lies in the fact that our approach allows current mainstream language-queried detectors to be equiped with multi-modal queries only through a lightweight modulating process. This offers a promising solution to overcome the limitations on insufficient granularity and ambiguous queries in current language-queried detectors. As an example, one may say that LoRA [1] (low-rank adaptation for efficient tuning on GPT) takes more training time and data than GPT if trained from scratch, but clearly, its efficiency lies in the fact that we do not need to train from scratch. \n\nWe will make the following revision to avoid misleading:\n\n1. We will provide more clarification on our data discrepancy in Table 1 and Table 2, for example:\n\nTable 1: xxxx. $^\\dagger$ Modulating upon pretrained models indirectly utilizes their pre-training data, and potentially consumes more time if we take the training time of the pretrained language-queried detectors into consideration.\n| Model|...|Pre-train Data|...|Training Time|\n|-|-|-|-|-|\n|...|...|...|...|...|\n|MQ-GLIP-T|...|O365 (+GLIP$^\\dagger$)|...|10$^\\dagger$|\n|MQ-GroundingDINO-T|...|O365 (+GroundingDINO$^\\dagger$)|...|10$^\\dagger$|\n\n2. Add more description on the  data discrepancy after the last sentence in line 232: \"It is worth noting that modulating upon pretrained GLIP/GroundingDINO indirectly utilizes their pretraining data. The efficiency here describes that our approach allows current mainstream language-queried detectors to be equiped with multi-modal queries only through a lightweight modulating process, avoiding training from scratch.\"\n\n3. Add clarification on the training time and data in lines 13-16,  lines 76-78, and lines 231-232. For example, \"For instance, MQ-Det ... with merely additional 3% modulating time upon GLIP\" in lines 13-16.\n\n[1] Hu, Edward J., et al. Lora: Low-rank adaptation of large language models. ICLR2022\n> **Zero-shot setting**\n\nWe acknowledge that our setting is different from the zero-shot setting in previous language-queried detectors, since we use additional visual exemplars along with texts as category descriptions. Our setting is derived from practical implementation, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. As an example, detecting different types of \"mushrooms\" in the Mushroom dataset of OdinW-13 can be much easier with visual exemplars than ambiguous textual descriptions. It is hard to reach absolutely fair comparison in this setting despite a lot of efforts we have made. The reason is that most methods, except ours, do not support visual exemplars as inputs, while comparing our finetuning-free method with other finetuned methods in a few-shot setting would also be unfair. To avoid misleading, we will make the following revisions:\n\n1. Modify the setting name:\n\n   (a) The name of section 3.2.1: \"Multi-modal queried detection without finetuning\"\n\n   (b) In the begining of line 226: \"We evaluate the model's ability ... in a **finetuning-free setting**, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any finetuning. \"\n\n2. Add more clarification:\n\n   (a) In the title of Table 1: \"**Finetuning-free detection**  results on the LVIS benchmark. Differently, we provide models with 5-shot instances as vision queries without any fine-tuning...\"\n\n   (b) Lines 13-15 in the abstract: \"For instance, MQ-Det significantly improves the state-of-the-art open-set detector GLIP by +7.8% AP on the LVIS benchmark with multi-modal queries without any downstream fine-tuning...\"\n\n   (c) Similar modifications as (b) in line 52, lines 76-77, and lines 226-232.\n\nAdditionally, we provide two more methods to acquire vision queries in the **finetuning-free setting (the zero-shot setting in the original version)** in Section A.1 of the Appendix, which do not have access to the target dataset.\n\nWe would also like to emphasize that the **core merit of this work** is that, we address the insufficient granularity and ambiguous queries that existing language-queried detectors suffer from via multi-modal queries, as recognized by Reviewer RdSN \"The paper **tackles an important problem**: ...\" and Reviewer JLFL \"Leveraging both textual and visual info ... **makes lots of sense**\". The main message from our original zero-shot comparison is that, equipped with multi-modal queries, previous language-queried detectors are allowed to detect objects with various  granularity and demonstrate great performance improvement without any finetuning. This confirms our core merit and indicates that multi-modal queried object detection can be a promising future direction.\n> **Full-shot performance**\n\nIn the full-shot setting, the reasons that MQ-Det performs worse than DINO-Swin-L lie in twofold. First, our approach is more suitable in few-shot scenarios, where the auxiliary information provided by the vision queries plays a vital role. This information  takes weaker effect with sufficient data in the full-shot setting. Second, MQ-GLIP-T is modulated upon GLIP-T, with only the expectation  of improving over GLIP-T. Given that DINO-L is a larger model and holds stronger architecture design (e.g., mixed query selection and look-forward-twice module), it is understandable for MQ-GLIP-T to perform worse than it."}]

Author rebuttal:
Dear reviewers, area chairs, senior area chairs, and program chairs,


We sincerely thank for the valuable comments. It is pleasure that this work has been fully recognized by Reviewer RdSN (Accept) and Reviewer JLFL (Weak Accept), including "**a novel approach**", "**a simple yet effective architecture and training strategy**", "**The paper tackles an important problem**", and "**makes lots of sense**". The main concerns of the other two reviewers lie in the fairness of the zero-shot comparison and the unclear statement of the data usage. In this regard, we explain in detail that our setting is a bit different from previous zero-shot setting and derived from practical implementation, where users can use multi-modal queries (textual descriptions, visual exemplars, or both) to detect a wider range of objects without any finetuning. Due to the setting gap, we have tried our best to conduct fair comparison. We also add detailed clarification on our data usage and our modulated training process to avoid misleading. We look forward to a better appreciation of this manuscript that incorporates our great efforts. Furthermore, this manuscript has been carefully revised according to the suggestions of the reviewers. We have made our code open-sourced, hoping to promote the development of open-world object detection. 

The followings are our detailed responses. We greatly thank the constructive suggestions that significantly help improve the quality of our paper.

Meta-review: After rebuttal and discussion three reviewers argue for acceptance, one for (borderline) rejection. The main points of criticism revolve around "overclaiming training efficiency" and a non-standard definition of the zero-shot setting. The authors acknowledge these shortcomings, and propose concrete changes to the paper. This is sufficient for an acceptance. The AC recommends the authors to implement these changes, as similar sentiments will arise in potential readers.
Decision: Accept (poster)