CQ-DINO: Mitigating Gradient Dilution via Category Queries for Vast Vocabulary Object Detection
Authors: Zhichao Sun
Huazhang Hu
Yidong Ma
Gang Liu
Nemo Chen
Xu Tang
Yao Hu
Yongchao Xu
Venue: NeurIPS 2025 poster ()
Keywords: Object detection
vast vocabulary
Primary area: applications

Abstract: With the exponential growth of data, traditional object detection methods are increasingly struggling to handle vast vocabulary object detection tasks effectively. We analyze two key limitations of classification-based detectors: positive gradient dilution, where rare positive categories receive insufficient learning signals, and hard negative gradient dilution, where discriminative gradients are overwhelmed by numerous easy negatives. To address these challenges, we propose CQ-DINO, a category query-based object detection framework that reformulates classification as a contrastive task between object queries and learnable category queries. Our method introduces image-guided query selection, which reduces the negative space by adaptively retrieving top-K relevant categories per image via cross-attention, thereby rebalancing gradient distributions and facilitating implicit hard example mining.
Furthermore, CQ-DINO flexibly integrates explicit hierarchical category relationships in structured datasets (e.g., V3Det) or learns implicit category correlations via self-attention in generic datasets (e.g., COCO). Experiments demonstrate that CQ-DINO achieves superior performance on the challenging V3Det benchmark (surpassing previous methods by 2.1% AP) while maintaining competitiveness in COCO. Our work provides a scalable solution for real-world detection systems requiring wide category coverage.

CQ-DINO: Mitigating Gradient Dilution via Category Queries for Vast Vocabulary Object Detection
Zhichao Sun
Huazhang Hu
Yidong Ma
Gang Liu
Yibo Chen
Xu Tang?
Yao Hu
Yongchao Xu 1.3g
Institute of Artificial Intelligence, School of Computer Science, Wuhan University
Xiaohongshu Inc.
Hubei Luojia Laboratory
Abstract
With the exponential growth data, traditional object detection methods are in- creasingly struggling handle vast vocabulary object detection tasks effectively. We analyze two key limitations of classification-bus -based detectors: positive gradi- ent dilution, where positive categories receive insufficient learning signals, and hard negative gradient dilution, where discriminative gradients are over- whelmed by numerous easy negatives. To address these challenges, propose CQ-DINO, a category query-based object detection framework that reformulates classification contrastive task between object queries and learnable category queries. Our method introduces image-guides guided query selection, which reduces the negative space by adaptively retrieving the top-K relevant categories per image via cross-attention, thereby rebalancing gradient distributions and facilitating im- plicit hard example mining. Furthermore, CQ-DINO flexibly integrates explicit hierarchical category relationships in structured datasets (e.g. V3Det) or learns implicit category correlations via self-attention in generic datasets (e.g., COCO). Experiments demonstrate that CQ-DINO achieves superior performance on the challenging V3Det benchmark (surpassing previous methods 2. 1% AP) while maintaining competitiveness on COCO, Our work provides scalable solution for real-world detection systems requiring wide category coverage.
Introduction
With the rapid expansion of data, developing robust Al system capable large-scale object detection has become essential. This necessity is driven by the increasing complexity and diversity of real-world applications. where Al must manage an extensive vocabulary and dynamic environments. Vast vocabularies inherently present hierarchical category structures, illustrated by classification datasets like ImageNet [8] and Bamboo [49]. Recent detection benchmarks, such V3Det [40], which feature 13,204 object categories organized in hierarchical structures, highlight the magnitude of this challenge. While object detection has witnessed significant advancements [31, 20, 25, 11], scaling effectively to vast vocabularies remains substantial challenge.
Category prediction mechanisms can be broadly categorized into four types, as illustrated in Fig. 1. Classification head-based methods employ feed-forward networks (FFNs) with sigmoid activation and
"Corresponding author: Yongchao Xu <yongchao..s.o xu www.@wwwwweuccccee
'Souree Code: https: /ttpp://github. com/FireRedTeam pateee/mmmmm/-D///// CCQDDNN [zhichaosun, yongchao.xu| @ @whhuedu.cn (huhuazhang, mayidong, tangshen] @ xiiaohongshu.com (liugang.spl, nemochen89, yanohu) @gmail.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prodicted Classes
Classittr Head
'bird'
Desrnder
''irrd'
DETR
'bird'
DETR
'bird'
<none?
<nonne
Object Querice
Object Queries
Langaage Model
Classification head-based detectors
(c) Language model generated detectors
dog, car, bus, bind, boat....
bird bus
Categoor Querties
bird
Text Encoder
Text Embeddings
Query Selection
'bird'
'bird'
DETR
'bird"
DETR
'bird'
<none>
<hone>
Object Queries
Objject Queries
(b) Text-prompted contrastive detectors
(d) Category query-based contrastive method (ours)
Figure 1: Comparison of category prediction mechanisms for vast vocabulary object detection. (a) Classification head-based detectors with fixed FFN layers face severe optimization challenges with increasing vocabulary size. (b) Text-prompted contrastive detectors leverage VLMs but require multiple inference passes for vast category lists. (c) Language model generated detectors enable open-ended detection but lack control over category granularity. (d) Our proposed CQ-DINO encodes categories as learnable category queries and leverages query selection to identify the most relevant categories in the image, achieving both scalability and improved performance.
Focal Loss [35] optimization. These approaches perform well on benchmarks with limited categories such as COCO [22] (80 categories) and Objects365 [36] (365 categories), but face fundamental challenges and scalability issues vast vocabulary settings. Text-prompted contrastive methods leverage Vision-Language Models (VLMs) to encode target categories as text inputs, achieving strong open-vocabulan detection [46] performance. However, vast vocabulary scenarios, text input sequences become prohibitively long, necessitating the splitting of category lists across multiple inference passes. This substantially limits their practical scalability and efficiency. Language model- generated methods approach open-ended [21] object detection by using language models to generate category labels without predefined candidate sets. Although these methods theoretically enable the detection of arbitrary object categories, they generally lack mechanisms to control the granularity of generated labels, which can result in substantial misalignment with practical detection requirements.
In this work, we first systematically analyze the challenges in vast vocabulary detection, focusing particularly on classification based methods. Our analysis reveals two critical limitations: (1) positive gradient dilution, where the sparse positive categories receive insufficient gradient updates compared to the overwhelming negative categories, and (2) hard negative gradient dilution, where informative hard negative gradients get overwhelmed among numerous easy negative examples.
To tackle these challenges, we introduce Category Query-based DINO (CQ-DINO), a novel archi- tecture that encodes categories as learnable query embeddings. Our approach centers on image- -ungee query selection, which identifies relevant categories via category-to-image similarity through cross- attention. The key insight driving our method is that dynamic sparse category selection significantly reduces the negative search space. This selection mechanism provides three crucial benefits: (1) balancing the ratio between positive and negative gradients, (2) performing implicit hard mining by selecting the most similar categories, and (3) reducing memory and computational costs, making the framework scalable to extremely large vocabularies. Selected category queries interact with image features to generate object queries. From these object queries, bounding boxes are predicted through a cross-modality decoder. The final classifications are obtained using contrastive alignment between the object queries and the category queries. Unlike traditional classification egory query representation offers greater flexibility by naturally encoding inter-category relationships. For structured datasets with explicit category hierarchies like V3Det [40], we leverage the inherent tree structure to construct hierarchical category queries with adaptive weighting mechanism that balances local and hierarchical features. For datasets without explicit hierarchies (e.g., COCO [22]), employ self-attentio mechanisms to implicitly learn the correlations between categories.
methods, our cat-
We evaluate CQ-DINO on both the vast vocabulary V3Det dataset [40] and the standard COCO benchmark [22]. Our approach surpasses previous state-of-the-art methods on V3Det while maintain- ing competitive results on COCO compared to DETR-basee detectors. Our method benefits from vast vocabulary detection while maintaining competitive results in limited vocabulary scenarios.

---

Our contributions can be summarized as follows:
We systematically analyze the challenges in vast vocabulary object detection, identifying positive gradient dilution and hard negative gradient dilution as critical limitations of classification-hasee methods.
We introduce learnable category queries that flexibly encode category correlations with efficient hierarchical tree construction for explicitly modeling category relationships in vast vocabulary scenarios.
We develop an image-guided query selection module that dynamically identifies relevant categories per image, effectively addressing the identified limitations while significantly reducing computational complexity.
2 Related Work
2.1 Vast Vocabulary Object Detection
The progression of object detection benchmarks reflects a steady growth in category vocabulary, evolving from relatively small vocabulary datasets such as PASCAL VOC [10] (20 classes) and COCO [22] (80 classes), to larger vocabulary benchmarks, including Objects365 [36] (365 classes) and Open Images [17] (600 classes). Recently, Wang et al. [40] introduce V3Det, the first vast vocabulary object detection dataset, comprising 13,204 hierarchically structured categories. This unprecedented scale poses significant challenges in terms of scalability and representation.
Recent progress has been driven by the V3Det Challenge [39]. yiclding several methodological advancemenss. MixPLv2 proposes semi-supervised framework that combines labeled V3Det data with unlabeled Objects365 [36] images through sseudo-labeling RichSem-DINO DINOO-Focaleee enhances detection robustness by integrating the RichSem-DING [28] framework with FocalNet- Huge backbone [44] pretrained on Objects365. Most recently, Prova [4] introduces multi-modal image-text prototypes specifically optimized for V3Det's finegrrinned classification. However, these methods still rely on fundamentally similar classification architectures, raising questions about their effectiveness and scalability for even larger vocabularies beyond tens of thousands of categories.
2.2 Object Detectors
Classification Head-based Methods. Most object detection frameworks employ feedforward
networks (FFNs) as fixed classification heads for category prediction. In two-stage detectors [31, 2, 13], region proposals are generated in class-agnostic manner by Region Proposal Networks (RPNs), followed by FFNed region classifiers. By contrast, one-stage detectors [35, 38, 9] predict bounding boxes and categories in a single step. Transformer-based methods, such as DETR [3], reformulate detection as a set prediction problem using learnable queries. Although DETR achieves an elegant end-to-end design, it suffers from slow convergence. Subsequent works [54, 41, 23, 18, 47,29, 29, 24, 14] mitigate these limitations. For example, Deformable DETR [54] proposes multi-scale deformable attention for sparse spatial sampling. DINO [47] improves performance via contrastive query denoising and mixed query selection. Despite these architectural advances, current methods fundamentally rely on classification heads with activation functions, typically optimized with Focal Loss [35] Cross-Entropy Loss [50]. This design inherently constrains scalability and presents optimization challenges when extending to vast vocabulary detection scenarios.
Text- Text-prompte prompted Contrastive Methods. Vision-language models have advanced the seamless inte- gration of visual and textual modalities for open-vocabulary object detection. These methods encode target categories as text inputs and align visual and textual representations. For instance, GLIP [20] pioneers the use of contrastive learning between image regions and textual phrases. Grounding DINO [25] further improves cross cross-modal alignment through early fusion of vision and textual fea- tures. Similarly, DetCLIP [45] and RegionCLIP [52] leverage image-text pairs with pseudo-labed to enhance region-level semantic understanding and improve generalization. Despite these advances, text-prompted methods face scalability bottlenecks due to the limited capacity of text token inputs during inference. For example, GLIP [20] and Grounding DINO [25] restrict input prompts to approximately 128 tokens per pass, allowing for about 40 categories simultaneously. Thus, detecting all categories in vast vocabulary benchmark like V3Det [40] (13,204 classes) would require over
3

---

331 sequential inference passes per image. This constraint renders current text- eext-promped methods computationally inefficient and impractical for real-time or large-scale detection.
Language Model-Generated Methods. Advances in multimodal large language models (MLLMs) have inspired detection methods leveraging their visual understanding and generative abilities. Some MLLMs [5, 1, 37] show preliminary object detection abilities but exhibit limited localization precision and recognition granularity. To mitigate these limitations, recent works [21, 11, 16, 40, 42] use LLMs primarily as category generators rather than direct detectors. For instance, GenerateU [21] employs T5-based decoder [7] to generate category names from visual features, reframing detection as text generation. Similarly, LLMDett [1 1] and ChatRex [16] utilize instruction-tuned L.MMs predict object categories from image features. While these generative approaches enable open-ended detection, they often produce inconsistencies due to limited controllability over label granularity. For example, given an image region of a "Persian cat", the model may generate the generic term "cat", causing semantic ambiguity and reduced accuracy for fine-gra me-grained detection tasks.
2.3 Category Query-based Methods
Learnable queries was popularized in computer vision by DETR [3], marking a paradigm shift from fixed architectural components to aask-adaptive representations. Queries serve as learnable embed- dings that interact with the visual feature space, enabling the model to capture complex, task-specific patterns. This design has since been adopted in diverse domains, including classification [26], seg- mentation [6], and multimodal learning [19]. Among these developments, category queries represent an innovation introduced by Query2Label [26]. Rather than relying fixed classification heads, Query2Label proposed learnable category embeddings to capture category-specine features. Subse- quent works such as ML-Decoder [34] have outperformed conventional classification methods. The effectiveness of category queries in classification motivated their extension to dense prediction tasks. For example, CQL [43] applies them to human-object interaction classification, ControlCap [51] uses them for semantic guidance in region captioning. and RankSeg [12] integrates them into semantic segmentation, dynamically selecting the top-k most relevant classes during inference. This selective querying reduces the effective search space, improving computational efficiency and segmentation accuracy. While prior works have explored category queries in various contexts, our work addresses a fundamentally different challenge specific to vast vocabulary scenarios. We provide, to the best of our knowledge, the first systematic theoretical analysis of gradient dilution issues that arise when dealing with vast category vocabularies, which motivates our image- image-guided query selection design. Moreover, we introduce a hierarchical tree construction strategy that explicitly models category correlations. enabling effective reasoning over deep semantic hierarchies in vast vocabulary datasets.
3 Method
3.1
Challenges in Vast Vocabulary Detection
Existing methods for vast vocabulary object de- tection with C' categories (C > 10*), particu- larly those employing sigmoid-baser based classifiers with Focal Loss [35]. face fundamental opti- mization challenges. We systematically analyze these issues through simplified formulation using the Cross-Entrop Loss with sigmoid ac- tivation, revealing two critical limitations:
Gradicst miio of positives to negatives
V3Dei
COCO
CO-DINO
2.0
DINO w/ EL w/
1.5
1) Positive Gradient Dilution. In vast vocab- ulary detection, the gradient signal for positive classes is overwhelmed by the aggregated neg- ative gradients. Let 2 denote the logit for class and Ve E {0, be its ground-truth label. The gradient of the Cross-Entropy Loss L with re- spect to 2e is:
0.5
0.0
Training steps:
Figure 2: Positive-to-neganee negaaivee gradient ratio compar- ing CQ-DINO against DINO with Focal Loss (FL) and Cross-Entropy Loss (CE) on V3Det and COCO datasets, showing the initial 2k training iterations where differ- ences are most evident.
V C =Zo)) -Yo

---

Calegory Queres
Oiscsifiatiin Loss
Fused Category
Encoding Category Correlations
K,
Cross-Modally Decoder
X
Feature Enhander
Object Queries
Conti sethev Loss
Imagge-Guided
Language-Guided
Query Selection
Query Selection
Image Encoder
Localization Loss
Image
8 Mo. W
Fused Image Features
Figure 3: Overview of the CQ-DINO framework for vast vocabulary object detection. Key com- ponents: (I) Learnable category queries enhanced with hierarchical tree construction for semantic relationship modeling; (2) Image-guided query selection that identifies the most relevant category queries; (3) Feature enhancer and cross-modality decoder (adapted from GroundingDINOO [25]), processing object queries with contrastive alignment between object and selected category queries.
where (.)) ()) is the sigmoid function. For a positive class ct (ye+ = 1), the gradient magnitude is -V=, L = o(z+)). ). while for negatives c (ye- 0), is VV.- C] = (
The total negative gradient magnitude grows linearly with the category count C:
Cl
Σ c
|V All = (C-
where E[ )] represents the average activation probability of negative classes. The positive- to-negative gradient ratio becomes:
1 (22t)) (C BBo (ze
x E
(3)
ClI
#ct
During early training stages, retains non negligiblee value. Since C exceeds 10 in vast category scenarios, - 0, causing positive gradients to be suppressed relative to the cumulative negative gradients. This fundamentally hinders the model's ability to learn from positive examples.
2) Hard Negative Gradient Dilution. The massive negative space leads to gradient dominance by easily classified negatives rather than informative hard negatives. Let H denote the set of hard negative classes with [[a(2>)] = c for c² E H. The ratio hard negative gradients to total negative gradients is:
EMEH V 2. C
NA
h
1 Ee #cl V
L
C
where N₁, the number of hard negatives. As C exceeds 10 - due to the term, making hard negatives diluted in gradient updates.
Fig. demonstrates these theoretical challenges. The gradient ratio for the V3Det dataset (13,204 classes) is lower than for the COCO dataset (80 classes), revealing the inherent difficulty in vast vocabulary object detection. While Focal Loss partially mitigates these issues by down-weighting easy negatives, the gradient ratio for V3Det remains around 0.5 compared to approximately 1.0 for COCO, indicating that gradient imbalance persists despite these improvements. We provide a comprehensive experimental analysis of Focal Loss performance and limitations in Sec. 4.4.
3.2 CQ-DINO
Our key insight that dynamically selecting a sparse category subset S C (1....,C) simultaneously addresses both gradient dilution challenges through gradient magnitude rebalancing and adaptive hard negative mining. As shown in Fig. 3, CQ-DINO consists of three key components: 1) Learnable category queries with correlation encoding. We initialize learnable category queries
5

---

Tree-Sttuclured Queries
Norm
Selected Queries
Level Larch
Level
Tree-Struclured Queries
Image Features
Mub-Htead Cress-Attention
Add Narm
FFN
FFN Topk
Oak
Tree
Level
Plant
Oak
Rose
Flower
Tree
FFN
x2
Hiis
Flower
Plant
Figure 4: Illustration of image- image-guided query selection module.
Figure 5: Hierarchical tree construction for category queries.
Qeat E BBCxx using the OpenCLIP [30] text encoder, where B is the batch size, C' denotes the total number of categories, and D is the embedding dimension. These category queries enable flexible encoding of category correlations through self- attention or hierarchical tree construction (Sec. 3.4). 2) Image-guided query selection. Given image features Fing E Bx xW from the image encoder, compute similarities between Fing and Qett through multi-head cross-attention modules. For each image, we select the top-C" most relevant queries (C" < C, typically C" = 100 for > 10*), ensuring that the target class et and its most confusing negative classes are preserved. This selection process rebalances the gradients and implicitly performs hard negative mining.
3) Feature enhancer and cross cross-modalee modality decoder. The selected category queries Q E R Bxx'' and image features Fing are processed through GroundingDINO [25] components. First, the feature enhancer module fuses the category queries and image features. Next, object queries are generated through language-guide query selection. Finally, detection outputs are produced by the cross- modality decoder and contrastive alignment between object queries and selected category queries.
3.3 Image-Guided Query Selection
The core innovation of CQ-DINO is our image-guided category selection module, illustrated in Fig. 4. This module employs cross-attention between category queries Qeat and image features Famg- Here, Quat serves queries (Q), while Fing provides keys (K) and values (V). The cross-attention layer establishes category-to-image correlations through similarity computation. Then, we apply TopK selection retain only the top-C" categories (C' C)) based on activation values. We supervise this selection using Asymmetric Loss [33], which serves as a multi-class classification loss.
The selection rebalances the sestive-to-negativ gradient ratio. Let and p' denote the original and revised positive-to-nggalee gradient ratios, respectively:
e' r-
CI
C C²
Ee
V
For typical setting (C 10a, C²= -100,, this achieves 100x gradient rebalancing factor. Our design provides three benefits: 1) Gradient rebalancing. By filtering out easy negative categories. our selection module improves the influence of gradients from positive examples. 2) Adaptive hard negative mining. The selection mechanism ensures that retained negative categories exhibit high semantic relevance to the image content, naturally implementing hard negative mining. 3) Scalable computation, Processing only C" categories reduces memory consumption and computational cost, making the framework scalable to extremely large vocabularies.
3.4 Encoding Category Correlations
A key advantage of our category query approach is the capacity model complex semantic relation- ships among categories, which is difficult for traditional classification head- based methods to achieve. We propose two complementary strategies for encoding category correlations.
Explicit Hierarchical Tree Construction. For hierarchical datasets like V3Det [40], we introduce hierarchical tree construction, shown in Fig. 5. The process begins at leaf nodes and progressively integrates hierarchical information upward through the category tree. For each parent node e with children C(v), its new query Q' is combination of its original query Qu and the mean pooling of all

---

Table Comparison with tatte-of-the-ars methods on the V3Det validation set. group are highlighted in bold.
Best results in each
Method
Epochs
Backbone
AP
APso
AP7s
ATSS [48]
24
Swin-B
7.6
8.9
8.0
FCOS [38]
24
Swin-B
21.0
24.8
22.3
Faster R CC-NNN [31]
24
Swin-Bi B
37,6
46.0
41.1
CenterNe12 [53]
24
Swin-B
39.8
46.1
42.4
Cascade R-CNN [2]
24
Swin-B
42.5
49.1
44.9
Deformable DETR [54]
50
Swin-B
42.5
48.3
44.7
DINO [47]
24
Swin-B
42.0
46.8
43.9
Prova [4]
24
Swin-B
44.5
49.9
46.6
CQ-DINO (Ours)
24
Swin-B
46.3
51.5
48.4
DINO [47]
24
Swin-B-225
43.4
48.4
45.4
Prova [4]
24
Swin-B-22k
50.3
56.1
52.6
CQ-DINO (Ours)
24
Swin-B-22k
52.3
57.7
54.6
DINO [47]
24
Swin-L
48.5
54.3
50.7
Prova [4]
24
Swin-L
50,9
57.2
53.2
CQ-DINO (Ours)
24
Swin-L
53.0
58.4
55.4
direct child nodes. Leaf nodes retain their original features directly, since they have no children.
Q a.) Qu au ((u)
Σ
Qc
(6)
CEC(o)
where Og [0, 1] balances local and hierarchical features.
log(n; + +1
Crp
w
log( (Nmxx 1)
where ny is the child count for node Nmax is the maximum child count across the tree, and w E (0, 0.5) hyperparameter (default: 0.3). This adaptive weight a, ensures that parent nodes with more descendants incorporate more collective knowledge, while nodes with fewer children maintain stronger individual semantics.
Building upon this structural design, we introduce a masking strategy during the classification loss computation mitigate hierarchical ambiguity. If any child category exists in the ground truth, its parent nodes are excluded from the classification loss. This prevents conflicting supervision signals for semantically related categories, such as suppressing "vehicle" when "car" is annotated.
Implicit Relation Learning. For categories without an explicit hierarchical structure, we employ a self-attention mechanism learn category relationships. This allows semantically related categories to influence each other's 's representations based on learned attention patterns.
Experiments
4.1 Datasets and Implementation Details
We conduct experiments on two detection benchmarks: (1) V3Det [40]: a vast vocabulary detection dataset containing 13,204 categories, with 183k training and 30k validation images. This is our primary benchmark for evaluating vast vocabulary detection. (2) COCO va12017 [22]: a standard benchmark dataset with 80 object categories, comprising 1 118k training and 5k validation images. We include this dataset to verify the effectiveness of our method in limited vocabulary scenarios.
Experiments are conducted on 8 A100-40G GPUs with total batch size of 16, unless otherwise specified. Baseline configurations are used for fair comparison. Three Swin Transformer [27] variants serve as backbones: Swin-B (ImageNet- 1 [8] pre-trained), Swin-B-222 (ImageNet-224 pre-trained), and Swin-L (ImageNet-22) pre-trained). Category queries are initialized using CLIP-ViT-L [30] text embeddings. To match dataset vocabularies, we employ 100 category queries for V3Det and 30 for COCO, aligned with their category sizes. The hierarchical tree structure derived from V3Det's category taxonomy. For implicit relation learning,
8-head self-attention module.
7

---

Comparison between the proposed CQ-DINO and state-of-the-art DETR variants on COCO va12017, reporting the best results as pro- vided by their respective original papers.
Table 2:
Table 3: Performance of Open- world meth- ods on V3Det dataset. We report zero-shot performance using their strongest models. * indicates finetuned results.
Method
Epochs Backbone AP APs APM APı
Method
Backbone
AP APsa Afto
H H-Def-DETR [15]
36
Swin-L 57.1 39.7
61.4
73,4
GenerateU [21]
Swin-L.88 -T-n- T5-B
0.4
0.5
0.4
Relation-DEr DETR [14]
12
Swin-L 57.8 41.2 62.1
74,4
ChatRex [16]
Swin-L&LLM-7B
1.3
1.5
DINO [47]
36
Swin-L 58.0
62.1
73.6
"GenerateU* [21]
Swin-L&T5-B
21.8
27.2
22.1
Rank-DETR [29]
36
Swin-L 38.2
62.2
73.6
DINO [47]
Swin-I.
48.5
54.4
50.7
CQ-DINO (ours)
24
Swin-L 58.5 42.5
62.1
74.0
CQ-DINO (Ours)
Swin-L
53.0 58.4
55.4
Table 4: Ablation study on the effectiveness of encoding category correlations and image-guided query selection components in CQ-DINO V3Det dataset with Swin-Base-22K backbone. "L" denotes unavailable A metrics due to absence of query selection.
Encoding Category Correlations
Image-guided Query Selection
AP
AR
FPS
47.3
0.7
Hierarchical Tree construction
49.4 2.11
0.6 0.11
51.1
80.9
10.8
Self- Attention module Hierarchical Tree Construction
51.3 (t 0.2)
75.5 5.40
10.4 (4 0.4)
52.3 (t
83.3 (1
10.6 (4
The training objective combines multiple loss terms with the following weights: classification loss (Asymmetric Loss [33], weight=1.0), contrastive alignment (Focal Loss [35], weight=1.0, .0), bounding box regression (L1 Loss, weight=5.0), and GloU Loss [32] (weight=2.0), as in GroundingDIINO [25]. Hungarian matching is used, following GroundingDINO,, with identical matching costs for object- to-query assignment. To stabilize training, use two-stage approach: first, pre-training category queries, image encoder, and image-guided query selection for 10 epochs to establish high initial target category recall; then, fine-tuning the full detection pipeline in the second stage.
4.2 Experimental Results
Performance on Vast Vocabulary Detection. Tab. presents our comparison with state-of-the-art methods the V3Det [40] benchmark. CQ-DINO consistently outperforms all previous approaches across different backbone configurations. With the Swin-B backbone, CQ-DINO achieves 46.3% AP, outperforming general detection methods like Deformable DETR [54] by 3.8% AP and DINO [47] by 4.3% AP. More importantly, CQ-DINO surpasses Prova [4], a specialized vast vocabulary detection method, by 1.8% AP. When integrated with the Swin-B-22k backbone, CQ-DINO achieves 52.3% AP, outperforming Prova by 2.0% AP. With the Swin-L backbone, CQ-DINO achieves 53.0% AP. The consistent improvements across different backbones demonstrate that CQ-DINO effectively addresses vast vocabulary detection challenges.
Performance on Standard Detection Benchmark Tab. 2 compares our method with state- of-the-art approaches on COCO va12017. Despite being primarily designed for vast vocabulary scenarios, CQ-DINO achieves competitive performance, reaching 58.5% AP, which is comparable to recent baTR- methods. We report the best results from their original papers to ensure fair comparison. The competitive performance of CQ-DINO is mainly due to the proposed gradient rebalancing and adaptive hard mining strategies.
4.3 Ablation Studies
We conduct ablation studies to evaluate the effectiveness of each component in CQ-DINO. Unless otherwise stated, all experiments are performed on the V3Det dataset using a Swin-B-22k backbone.
8

---

Table 5: Ablation study on self-attention module (SA) in CQ-DINO on the COCO dataset.
Table Ablation study on adaptive weighting in tree construction.
Method
AP
AR
Params (M)
Method
AP
AP30
AP75
AR
CQ-DINO w/o SA 58.3
98.2
244.3
Fixed weight (0.5)
51.9
57.4
54.3
82.3
CQ-DINO w/ SA 58.5 0:20 99.1 ( 0.9 246.7 (+2.4)
Ours (a.))
52.3
57.7
55.4
83.3
Table 7: Scalability comparison of CQ-DINO
Table 8: Focal Loss parameter analysis in DINO using Swin-B-22 backbone. AP scores (%) com- pare different a and combinations, Dashes "_" indicate unstable training configurations.
with DINO on A100 40G GPU using Swin-B-22 BB-2kk backbone, showing per-category parameters (K),
CUDA memory consumption (kB), and maximum supported category capacity (k).
Params/Cat, Memory/Cat. Max Cats.
0.25
0.35
0.50
0.75
Method
(K)
(kB)
(k)
43.4
45.1
47.4
DINO [47]
2.1
8.9
100
3
43.7
43.9
45.1
CQ-DINO
0.8
2.7
130
5
Effect of Each Component in CQ-DINO. Table 4 presents the contribution of each component in terms of average precision (AP) and category-level average recall with selected queries (ARS). To establish a baseline without image-guided query selection, we conduct experiments on 8 H800-80G GPUs due to memory constraints. Notably, incorporating image-gun srage-ggided query selection increases FPS from 0.7 to 10.8, highlighting effectiveness alleviating memory bottlenecks and substantially enhancing inference efficiency. Furthermore, this component addresses the issue of gradient dilution and leads to a substantial AP improvement, from 47.3% 51.1% % in detection performance.
Explicit hierarchical modeling via tree construction leads to an improvement of 1.2% AP and 2.4% AR In contrast, employing self-attention for implicit relationship modeling achieves marginal increase of 0.2% AP but reduces ARC. This is due to difficulties in learning complex relationships across 13k+ categories. Notably, the tree construction method introduces zero additional parameters with only a 0.2 FPS overhead, while the self-attention approach adds 2.36M extra parameters and 0.4 FPS reduction inference speed.
While Tab. shows that explicit tree construction outperforms self-attention on V3Det's vast category space, conduct further experiments on COCO (Table 5). The results reveal that self-attention contributes meaningful 0.2% AP and 0.9% AR improvements on datasets with fewer categories. Both experiments validate the effectiveness of encoding category correlations, with the optimal approach depending on the scale of the category space.
Effectiveness of Adaptive Weighting in Tree Construction. Tab. shows the importance of our adaptive weighting strategy compared to a fixed weight of 0.5, This approach adjusts weights based on the varying number of child categories for each parent node the hierarchy. Results show that our adaptive approach outperforms fixed weighting.
4.4 Discussion
Scalability of CQ-DINO. Tab. 7 compares the scaling efficiency of CQ-DINO and DINO for vast vocabulary detection. CQ-DINO requires only 0.8K parameters per category, representing a 62% reduction from DINO's K parameters. For runtime memory consumption, CQ-DINO uses 2.7KB CUDA memory per category. To evaluate practical scalability limits, we test the maximum category support on an A100-40G GPU using the Swin-B-22k backbone with a single 800 x 1333 resolution input image. CQ-DINO supports detection of up 130k categories, surpassing DINO's 100k limit. These experiments validate that CQ-DINO enables applications with extremely large vocabularies.
Limitations of Generation- Methods. We evaluate generation based methods on the V3Det dataset in Tab. 3. Following the evaluation protocol from GenerateU [21], we compute semantic similarity between generated category embeddings and V3Det category embeddings, selecting the highest similarity match the final prediction. Our experiments reveal poor performance:

---

GenerateU [21] achieves only 0.4% AP, while the more recent ChatRex [16] achieves just 1.3% AP. This highlights fundamental limitation: generation-baser methods struggle to control the granularity of generated categories, creating semantic misalignments with the specific requirements of detection tasks. Furthermore, even when we finetune GenerateU on V3Det data, the resulting performance (21.8% AP) still exhibits a substantial gap compared to classification- methods.
Focal Loss Parameters Analysis under Gradient Dilution. As discussed in Sec. 3.1, vast vo- cabulary detection suffers from gradient dilution challenges. Focal Loss (FL) [35] mitigates this via adaptive weighting with hyperparameters a and T.. Theoretically, a balances positive/negative sample contributions, and increasing a enhances the model's ability to handle more categories. The factor focuses learning on hard negatives, where increasing ? improves the mining hard examples. Tab, evaluates FL configuration. on DINO. The default setting [40] (a 0.25, ? 2) achieves 43.4% AP. Tuning these hyperparameters reveals that training is unstable when y 2 5 or a 2 0.75. Notably, the optimal configuration (r and 0.5) achieves 47.4% AP, surpassing the baseline by 4.0%. Nonetheless, this is still lower than the 52.3% AP achieved by our CQ-DINO, suggesting that FL hyperparameter tuning, while beneficial, leaves room for further improvement. Interestingly, when examining parameter transferability across architectures in Appendix Tab. 9, we find that this optimal setting does not generalize well. Applying the Swin-B-22k optimal parameters (a 0.5, 7 2) to Swin-B degrades performance by 3.3 % AP relative to its default setting. However, they increase the performance by 4.0% AP and 1.6% AP for Swin-B-22k and Swin-L backbones, respectively. These findings suggest that while Focal Loss is effective in addressing the gradient dilution challenge, optimal hyperparameter selection remains architeeture-dependent and requires careful tuning.
4.5 Limitation
Although CQ-DINO improves vast vocabulary object detection, several limitations remain. First, detection performance is influenced by the recall of the category query selection. Fortunately, CQ- DINO achieves 83.3% A ARC. Appendix Tab. 11 shows that increasing the number of category queries improves AR but does not lead to higher AP. This indicates, in most cases, 83.3% AR is not primary bottleneck. Future work will explore more sophisticated selection strategies to address this gap. Second, the two-stage training paradigm, while efficient in practice (first stage requires only ~ hour), may yield suboptimal coordination between stages compared to end-to-end alternatives.
Conclusion
In this work, we systematically analyze the challenges inherent in vast vocabulary detection: positive gradient dilution and hard negative gradient dilution. Through comprehensive experiments, we expose the limitations of Focal Loss under these challenging settings. To mitigate these issues, we propose CQ-DINO, novel framework with two core innovations: (1) learnable category queries that encode category correlations, and (2) image-guided query selection that effectively reduces the negative space while performing adaptive hard negative mining. Extensive evaluations the V3Det and COCO benchmarks demonstrate that CQ-DINO achieves superior performance and strong scalability vocabulary sizes increase. As future work, plan investigate the adaptability of our category query formulation for open-vocabulary detection and incremental learning scenarios.
Acknowledgments
This work was supported in part by NSFC under Grant 62222112 and 62176186; and part by the Innovative Research Group Project of Hubei Province under Grant 2024AFA017.
10

---

References
[1] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309, 16609, 2023. 4
[2] Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: High quality object detection and instance segmentation. IEEE Trans. Puttern Anal. Mach. Intell., 43(5):14833 1498, 2019. 3.7
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Eur. Conf. Comput. Vis., pages 213-229, 2020. 3,4
[4] Yitong Chen, Wenhao Yao, Lingehen Meng, Sihong Wu, Zuxuan Wu, and Yu-Gang Jiang. Comprehensive multi-modal prototypes simple and effective classifiers for vast-vocabulary object detection. In Proc. AAAI Conf. Artif. Intell., 2025. 3, 7,8
[5] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su. Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. IntemVVL: Scaling up vision foundation models and aligning for generie visual-linguistic tasks. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 24185-24198, 2024. 4
[6] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Adv. Neural Inform. Process. Syst., 34:17864- 2021.
[7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instraction-finetuned language models. Journal of Machine Learning Research, 25(70):1-53, 2024. 4
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image databasc. In IEEE Conf. Comput. Vis. Pattern Recog., pages 248-248822552 2009. 7,7
[9] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. CenterNet: Keypoint triplets for object detection. In Int. Conf. Comput. Vis., pages 6569-6578, 2019. 3
[10] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. Int. J. Comput. Vis., 11 111:98-136,
2015. 3
[11] Shenghao Fu, Qize Yang. Qijic Mo, Junkai Yan, Xihan Wei, Jingke Meng. Xiaohua Xie, and Wei-Shi Zheng. LLMDet: Learning strong open-vocahulary object detectors under the supervision of large language models. arXiv preprint arXiv:2501. 18954, 2025. 1,4
[12] Haodi He, Yuhui Yuan, Xiangyu Yue, and Han Hu. RANKSEG: Adaptive pixel classification with image category ranking for segmentation. In Eur. Conf. Comput. Vis., pages 682-700, 2022. 4
[13] Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Int. Conf. Comput. Vis., pages 2961-2969, 20111197....... 3
[14] Xiuquan Hou, Meiqin Liu, Senlin Zhang, Ping Wei, Badong Chen, and Xuguang Lan. Relation DETR: Exploring explicit position relation prior for object detection. In Fur. Conf. Comput. Vis., pages 89-105, 2024. 3,8
[15] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In IEEE Conf. Comput. Vis. Pattern Recog., pages 19702- 19712, 2023.
[16] Qing Jiang, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, Lei Zhang, al. ChatRex: Taming multimodal LLM for joint perception and understanding. arXiv preprint xrXiv:2411.18363 2024.
4, 8, 10
[17] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. Int. J. Comput. Vis.,
1956,11996-198, 2026,
[18] Feng I.i, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. DN-DETR: Accelerate detr training by introducing query denoising. In IEFE Conf. Comput. Vis, Pattern Recog., pages 13619-13627, 2022. 3
[19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In Int. Conf. Mach. Learn., pages 19730-1999 19740, 2023, 4
11

---

[20] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang. et al. Grounded language-image pre-training. In IEEE Conf. Comput. Vis. Pattern Recog. pages 10965-10975, 2022. 1,3
[21] Chuang Lin, Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-language pretraining for open-ended object detection. In IEEE Conf. Comput. Vis. Puttern Recog., pages 13958- 13968,,11,8,,88 2024. 2, 4, 8, 9, 10
[22] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Eur. Conf. Comput. Vix., pages 740-755, 2014. 2, 3.7
[23] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for detr. In Int. Conf. Learn. Represent. 3
[24] Shilong Liu, Tianhe Ren, Jiayu Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, et al. Detection transformer with stable matching. In Int. Conf. Comput. Vis., pages 6491-6500, 2023.
[25] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding DINO; Marrying DINO with grounded pre-training for open-set object detection. Eur. Conf. Comput. Vis., 2024. 1, 3, 5, 8
[26] Shilong Liu, Lei Zhang, Xiao Yang. Hang Su, and Jun Zhu, Query2Label: A simple transformer way to multi-label classification. arXiv preprini arXiv:2107. 100334207..1834, 2021. 4
[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In Int. Conf. Comput. Vis., pages 10012-10022, 2021.
[28] Lingchen Meng, Xiyang Dai, Jianwei Yang, Dongdong Chen, Yinpeng Chen, Mengehen Liu, Yi-Ling Chen, Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Learning from rich semantics and coarse locations for long-tailed object detection. Ady. Neural Inform. Process. Syst., 36:78082-78094, 2023. 3
[29] Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, and Gao Huang. Rank-DETR for high quality object detection. Adv Neural Inform. Process. Syst., 36:16100-161 13, 2023. 3,
[30] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In Int. Conf. Mach. Learn., pages 8748-8763 2021. 6,7
[31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEFE Trans, Pattern Anol. Mach. Intell., 137 2017. 3,
[32] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In JEEE Conf. Comput. Vis. Pattern Recog., pages 658-666, 2019. 8
[33] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor, Asymmetric loss for lalti-lahel classification. In Int. Conf. Comput. Vis., pages 82-91,
2021. 6, 8
[34] Tal Ridnik, Gilad Sharir, Avi Ben-Cohen Emanuel Ben-Baruee Baruch, and Asaf Noy, ML-Decoder: Scalable and versatile classification hcad. In IEEE Winter Conf. Appl. Comput. Vis., pages 32-41, 2023, 4
[35] T-YLPG Ross and GKHP Dollár, Focal loss for dense object detection. In IEEE Conf. Comput. Vis. Puttern Recog,, pages 2980-2988, 2017. 2, 3, 4, 8, 10
[36] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: large-scale, high-quality dataset for object detection. In Int. Conf. Comput. Vis., pages 8430-8439, 2019. 2,
[37] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Man-tttt Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11865 2023. 4
12

---

[38] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: A simple and strong anchor-free object detector. IEEE Trans. Pattern Anal. Mach. Intell., 44(4):1922-1933, 2022. 7.7
[39] Jiaqi Wang, Yuhang 7ang, Pan Zhang. Tao Chu, Yuhang Cao, Zeyi Sun, Ziyu Liu, Xiaoyi Dong, Tong Wu, Dahua Lin, al. V3Det challenge 2024 vast vocabulary and open vocabulary object detection: Methods and results. arXiv preprint arXiv:2406.1177 2024. 3
[40] Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3Det: Vast vocabulary visual detection dataset. Int. Conf. Comput. Vis., pages 198544,
2023. 1, 2, 3, 4, 6, 7, 10
[41] Yingming Wang. Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor DETR: Query design for transformer- based detector. In Proc. AAAI Conf. Artif. Intell., volume 36, pages 2567-2575, 2022. 3
[42] Junfeng Wu, Yi Jiang. Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3783-3795, 2024.
[43] Chi Xie, Fangao Zeng. Yue Hu, Shuang Liang, and Yichen Wei. Category query learning for human-object interaction classification. In IEEE Conf. Comput. Vis. Pattern Recog., pages 15275 15275-15284, 2023. 4
[44] Jianwei Yang, Chunyuan Li, Xiyang Dai. and Jianfeng Gao. Focal modulation networks. Adv. Neural Inform. Process. Syst. 35:4203 2022. 3
[45] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing paralleled pre-training for open-works
Xu, and Hang Xu. DeICLIP: Dictionary-enricher detection. Adv. Neural Inform. Process. Syst., 35:9125 9138,, 2022. 3
[46] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. veen-Vocabulary object detection using captions. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 14393 14402, 2021.
[47] Hao Zhang, Feng Li. Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Tleung-Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In Int. Conf. Learn. Represent. 2023. 3, 7, 8,
[48] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor- based and anchor free detection via adaptive training sample selection. IEEE Conf. Comput. Vis. Puttern Recog.. pages
2020. 7
[49] Yuanhan Zhang. Qinghong Sun, Yichun Zhou, Zexin He, Zhenfei Yin, Kun Wang, Lu Sheng, Yu Qiao. Jing Shao, and Ziwei Liu. Bamboo: Building mega-sealc vision dataset continually with human-maching synergy. arXiv preprint arXiv:2203.07845 2022. 1
[50] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Adv. Neural Inform. Process. Syst. 31, 2018. 3
[51] Yuzhong Zhao, Yue Liu, Zonghao Guo, Weijia Wu, Chen Gong, Qixiang Ye, and Fang Wan. ControlCap: Controllable region- level captioning. In Eur. Conf. Comput. Vis., pages 21-38, 2024. 4
[52] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liuniann Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. RegionCLIP: Region- based sanuuageimage pretraining. In IEEE Conf. Comput. Vis. Pottern Recog., pages 16793-16880 2022. 3
[53] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint
10Xiv:2103.0746.11 2021.7 7
[54] Xizhou Zhu, Woijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to- end object detection. In Int. Conf. Learn. Represent, 2021. 3, 7,8
13

---

A Appendix
A.1 Extended Analysis on Gradient Dilution Vast Vocabulary vs. Class-Imbalanced
In Sec. 3.1 of the main paper, we introduced the concept of positive gradient dilution primary challenge vast vocabulary object detection, using simplified model assuming relatively balanced data distribution. This appendix provides more comprehensive analysis to further clarify the distinction between the gradient dilution problem caused by vast vocabulary size and the one addressed in traditional class-imbalanced learning.
The simplified gradient ratio in our main paper (Eq. (3)) demonstrates that p x J. highlighting the direct impact of the vocabulary size C. To better distinguish the effects of class imbalance and vocabulary size, we can formulate more general gradient signal ratio, Pa. for a positive class ct:
nol -|V= fl
no- (1 - (
Pat Ee
RC
-e- #ct n
|| ClI
Eg (a Ee #ct Do-
By letting et and e represent the average gradient magnitudes for positive and negative samples respectively, and N be the total sample count across all classes, the ratio can be expressed as:
Net - (N nel mel) -E
Pet
(8)
This generalized formula reveals two distinct and compounding sources of gradient dilution:
This well-studied issue is primarily reflected by the term fett in the numerator. When class is rare (i.e., has long-tailed distribution), its sample count Ret is small, which directly reduces the gradient signal ratio Pet. This central challenge that traditional class-imbalance methods aim to solve.
Class Imbalance:
Vast Vocabulary Size: This is primarily driven by the term nett in the denominator. In vast vocabulary settings (C' > 10, 000), N net becomes enormous because it aggregates all negative categories. The large C' further reduces the ratio Pet, making the
samples from the other gradient dilution more severe in vast vocabulary object detection. This impacts both rare and common classes.
This distinction in the problem's source explains why conventional methods for class imbalance are not sufficient for the vast vocabulary challenge.
A.2 Algorithm Details
This section provides the detailed algorithmic implementations of the three core components in CQ-DINO, as described in Sec. 3. Each algorithm addresses specific challenges in vast vocabulary object detection:
Algorithm 1 Image-Guided Query Selection (Sec. 3.3): This algorithm implements the core innovation of CQ-DINO by dynamically selecting the most relevant category queries for each image. Through cross-attention mechanisms between category queries and image features, it reduces the negative search space from the full vocabulary to a manageable subset (C' & C), addressing both positive gradient dilution and hard negative gradient dilution issues identified in Sec. 3.1.
14

---

Algorithm 1: Image-Guided Query Selection
Input:
B: Batch size, C: Number of category queries,
D: Embedding dimension, C": Number of selected category queries,
H:: Height of image features, W": Width of image features, Qeat BxCxxD Category queries,
F Fing *R Image features
Output:
Q E BR xD; Selected enhanced category queries, I BBC'' Selection indices
// Reshape image features for cross-attention
F flat +- Reshape mmg: [B, D, H' W"]); Qenhanced Qcari
// Enhancement through cross-attention layers for 1= to 2 do
// Cross-attention with image features
Qatin MuliileeaCCrossstteeiiinnn CawrAAtteettinnnnnnneet enhonnond: flats F flat);
Qenhanced + LayerNorm Qattn Qnnhanced);
1/ ferd-forward transformation
11 + PFN(Qennanced))
Qenhanced LayerNorm Q55n + Qenkanced):
// Query selection based on enhanced representations + LinearProjection/ Qenhanced) // L BxC
for b to B do Io f Top,(Ln.,; C");
Q !e ; Qenhanced (b, Ib.::
return Q I
Algorithm 2 Self-Attention for Implicit Category Relations (Sec. 3.4): For datasets without explicit hierarchical structures, this algorithm employs multi-head self-attention learn implicit correlations between categories. It allows semantically related categories to influence each other's representations based on learned attention patterns, complementing the explicit hicrarchical approach.
Algorithm Self-Attention for Implicit Category Relations
Input: Qeat RCDD; Category queries, H: Number of attention heads
Output: Qcorr CCDD; Correlation-enhanee queries
1/ Enhance queries through self celf-attention mechanism Qattn MuliiHeadSelfAttentionn (Qaat); Qearr LayerNorm( Qattn Qeat);
return Qcorr
Algorithm 3 Explicit Hierarchical Tree Construction (Sec. 3.4): For datasets with explicit hierar- chical structures like V3Det, this algorithm leverages the category hierarchical structures to enhance query representations. It performs bottom-up tree traversal to incorporate hierarchical relationships through adaptive weighting, enabling parent categories to aggregate semantic information from their children while maintaining individual semantics.
15

---

Algorithm 3: Explicit Hierarchical Tree Construction
Input: Qeat E RCCD: Category queries, T: Hierarchical tree structure with nodes V,
W: Base weight parameter (default: 0.3)
Output: Qoorr RCxD: Correlation-enhanced queries
1/ Initialize and prepare tree traversal
Qeorr Qeati
N Nmax f- max [Children(v, T)
L TopologicalSon(T))
// Bottom-up tree traversal for correlation enhancement
foreach node in Reverse(C) do
if IsLeaf(v, T) then
// Leaf nodes retain original queries
continue:
else
// Compute adaptive weight based on children count
C(v) + GetChildren(v, T);
Thy + C(v)l:
ar w + +l)(++1). log( +1)
an - min(ar, 1.0);
1/ Update parent query with weighted combination mean + Exec(n) Qcorr :|:
ccorr [c, & - -aa) Qead [v, + 0x Q
whild
return Qcorr
A.3
Performance of DINO with Different Focal Loss Parameters
In Tab. 8, we achieve the optimal Focal Loss parameters backbone. We conduct parameter setting experiments with different backbone in Tab. 9. We find that this optimal setting does not generalize well. The same parameters (a 0.5, 7 2) lead performance degradation of 3.3% AP for the Swin-B backbone compared to the default configuration
0.5 and y 2 with the Swin-B-22k
(a 0.25, ? 2). However, they improve the performance by 4.0% AP and 1.6% AP for the Swin-B-22k and Swin-L backbones, respectively. These findings show that while Focal loss addresses gradient dilution issues, its optimal configuration requires careful parameter tuning.
Table 9: Performance comparison between standard DINO (a: 0.25, 2), $ DINO with modified Focal loss parameters (a 0.50, 7 = 2), and the proposed CQ-DINO.
Method
Backbone
AP
APso
APro
DINO
Swin-B
42.0
46,8
43.9
3 DINO
Swin-B
38.7 (4 3.3)
43.7 (L 3.1)
40.4 (1 3.5)
CQ-DINO
Swin-B
46.3
51.5
48.4
DINO
Swin-B-22k
43.4
48.4
45,4
1 DINO
Wvin-B-22k
47.4 (1 4.0)
53.3 (1 4.9)
49.7 (t 4.3)
CQ-DINO
Swin-B-22k
52.3
57.7
54,6
DINO
Swin-L
48.5
54.3
50.7
: DINO
Swin-L
50.1 (1 1.6)
56.3 (1 2.0)
52.4 (t 1.7)
CQ-DINO
Swin-L
53.0
58.4
55.4
A.4 Gradient Norm Visualization During Training
As reported in Tab. 8, CQ-DINO achieves superior performance compared DINO with optimal Focal Loss parameters (a 0.5, 7 2), outperforming the default configuration of DINO (a = 0.25.
16

---

2). To analyze their impact on training gradients, we visualize the gradient norm in Fig. for three configurations: (1) CQ-DINO (red), (2) DINO with 0.5 (blue), and (3) DINO with = 0.25 (orange), all using the Swin-B-22k backbone. The results reveal that the gradient norm for DINO with
0.25 (orange line) remains low throughout training, indicating insufficient learning from both positive and hard negative samples. In contrast, DINO with = 0.5 (blue line) initially displays strong gradients, but these are unstable and fluctuate considerably, as reflected the high variance of the blue points. Meanwhile, CQ-DINO maintains balanced trajectory, sustaining moderate and stable gradient magnitudes during the entire training process. This sustained and balanced gradient norm demonstrates that CQ-DINO more effectively addresses the gradient dilution issues.
Gradient norm
100
40
20
CQ-DINO
DINO w/Flfa 0.25,yy -
DID w/Fl((e n.S.y 2)
0
100000
200000
Training steps
Figure 6: Gradient norm visualization during training process.
A.5 Extended Gradient Ratio Analysis
Positive-to-Negative Gradient Ratio. exten our analysis to 400k training iterations to examine the effect of CQ-DINO on gradient distribution. As illustrated in Fig. CQ-DINO mitigates the positive gradient dilution problem inherent in vast vocabulary tasks. During the early training stage (fewer than 10k iterations; see Fig. 2), CQ-DINO maintains substantially higher positive-to-negative gradient ratio compared DINO with cross-entropy (CE) focal loss (FL.. Moreover, CQ-DINO reaches a balanced state (>1.0) earlier than baselines such as DINO w/CE and DINO w/FL.
Gradient Ratio
1.75
1.50
1.25
1.00
0.75
0.50
0.25
CQ-DINO
DINO w-FL
DINO
100000
200000
300000
400000
Training steps
Figure 7: Positive-to-negative gradient ratio across extended training iterations on V3Det dataset.
Hard-Negative Gradient Contribution. define hard negatives as top 10% of negative categories with the highest prediction scores, representing the most confusing negatives that require
17

---

focused learning. Fig. 8 presents the proportion of total negative- negative-gradient magnitude attributable to hard negatives. In early training stage, CQ-DINO achieves higher hard-negative ratios compared to DINO w/CE and DINO w/Fl. This confirms that our image-guided query selection implicitly performs effective hard-negative mining by filtering irrelevant categories and concentrating on informative ones.
Hard Negative Ratio
1.0
0.9
0.5
0.1
0.3
CQ-DINO
DINO w/FL
DINO w/CE
0.2
100000
200000
300000
400000
Training steps
Figure 8: Hard negative-to-all negative gradient ratio across training iterations on V3Det dataset.
A.6 Additional Ablations Isolated Component Contributions
To provide comprehensive understanding of our method's effectiveness and isolate the contributions of individual components, we conduct additional ablation studies that complement the analysis presented in Tab. 4. We analyze the following components, with results presented in Tab. 10:
Baseline Architecture: The foundational model based on Grounding DINO, including its feature enhancer and decoder. This corresponds to the setting where all our proposed components are disabled (row 1).
2. Training-stagee stanin Gradient Dilution Mitigation: Our proposed image-guided query selection method applied during training. This ensures that the model's 's text encoder and fusion layers receive focused gradient signals from small subset of relevant categories for each image.
3. Inference-stage Category Selection: A computational efficiency mechanism that selects only the top-K most relevant categories based similarity between category queries and image features during inference.
4. Encoding Category Correlations: the hierarchical construction method, which was identified as the effective approach for vast vocabulary setting.
The detailed ablations in Tab. 10 confirm three main observations:
Training-stage gradient dilution mitigation the primary contributor. On V3Det, introducing this component yields +1.9% AP gain (row vs. row 4), validating our hypothesis that gradient dilution poses significant challenge in vocgelvoyybulary detection. On COCO, where the label space much smaller, the improvement is modest (+0.3% AP).
Inference-stage category selection alone does not resolve gradient dilution. On V3Det, applying inference selection without training-stage mitigation reduces AP (row vs. row 3), likely because the underlying gradients remain diluted during training. In contrast, on COCO, inference selection provides a mild AP improvement due to reduced background competition.
18

---

Table 10: Detailed ablation study on the isolated contributions of our proposed components. "0" denotes the absence of the corresponding component, "I" denotes its inclusion.
Training-stage gradient dilution
Inference-stage category selection
Encoding category correlations
V3Det
COCO
AP
FPS
AP
FPS
47.3
0.7
57.5
9,8
0
0
49.4
0.6
57.9
9.7
O
46.6
10.6
58.2
9.9
0
51.3
0.6
58.2
9.7
51.1
10.8
58.3
10.0
52.3
10.6
58.5
9.9
Combination leads to best accuracy-efffciency trade-off. Integrating both training-stagee gradient dilution mitigation and inference-stage category selection (row 6) achieves the highest AP on both datasets while providing substantial speedups: V3Det, FPS improves from 0.6 to 10.6.
A.7 Impact of Category Query Count
Tab. 11 presents impact of varying the number of category queries (50, 100, 200) on performance. As expected, increasing the number of queries leads to higher AR scores, at the cost of reduced inference speed. However, simply increasing the query count not always beneficial. Excessive queries lead to gradient imbalance between positive and negative examples, resulting in diminishing performance gains. Furthermore, improvements in AR do not necessarily correlate with increases in AP, as reflected in our observation that an AR of 83.3% is not the limiting factor the current stage. Empirically, we find that 100 queries achieve an optimal balance between detection performance and computational efficiency for V3Det. On the COCO dataset, 30 queries sufficient, achieving 99.1% ARC
Table 11: Ablation study with different numbers selected category queries in CQ-DINO on V3Det.
Query Count
AP
ARE
FPS
51.8
78.4
6.9
100
52.3
83.3
6.6
200
52.2
87.5
5.9
A.8
Failure Case Analysis
To better understand the limitations of our approach and identify key bottlenecks. we conduct systematic failure case analysis of CQ-DINO. Specifically, we examine the 3,527 categories for which the category-level recall falls below 83.3%. This analysis highlights category frequency and object scale as the dominant factors behind performance drops.
Analysis by Category Frequency. Among the 3,527 low-recall categories, 374 are rare and 3,153 are common, following our defined frequency threshold. As shown Tab. 12. CQ-DINO achieves 20.5% AP on rare categories, which is significantly lower than our overall performance of 52.3% AP. This gap underscores the inherent difficulty of rare-category detection in large-vocabulary settings. Notably, despite the challenge, CQ-DINO consistently surpasses both the vanilla DINO and DINO variant optimized with Focal Loss, indicating that the drop rare-category performance is dodain-wide challenge rather than limitation specific our model design.
Analysis by Object Scale We further assess performance across object scales (Tab. 13). The largest gap appears for small objects, where CQ-DINO achieves only 14.1% 1% AP, far below the overall 52.3% AP. While performance on medium- and large-scale objects is robust, this finding highlights that small object detection remains critical challenge in vast vocabulary settings.
19

---

Table 12: Performance on category frequency with category-level recall below 83.3%.
Method
Rare
Common
All
DINO (Focal a=0.50, 7=2)
15.5
27.1
25.8
DINO (Focal a=0.50,
19.9
31.1
29.9
CQ-DINO
20.5
32.1
30.9
Table 13: Performance across object scales in low-recall cases.
Method
Small
Middle
Large
DINO
16.5
33.8
DINO (Focal a=0.50, y=2)
11.6
20.2
38.3
CQ-DINO
14.1
23.1
38.9
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes)
Justification: The abstract and introduction precisely reflect the scope and contributions of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification: Section 4.6 provides a detailed discussion of our method's limitations. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
20

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Complete proofs with all assumptions are provided in Sec. 3.1, and supporting details appear in Fig. 2, Fig. 6, Tab. 9, and Tab. 10 all statements and proofs are clearly referenced.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide comprehensive information in Sec.4.2 enable reproducibility of our main experimental results. Additionally, the code to reproduce our results are made available in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
21

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The codes are fully available in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
aesionnooicyy Poloiyyy)))) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cccccc/ppooo wwtpccccppcccccccccccccccccccclt wwwlcccgggese/eeeee CooeeSumii http://oeuuniiiiinoooooooooooo.e ssinnoolccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed implementation details are provided in the Sec. 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
22

---

The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We ensured reproducibility with fixed seeds. Besides, we found that results varied minimally across multiple random seeds, though error bars or significance tests were not reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report inference times in Table 4, using 8 NVIDIA A100-40G GPUs for most experiments and 8 NVIDIA H800-80G GPUs for those in Table due memory constraints, providing sufficient detail to reproduce our compute setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https //tp/enurss.eeee httppp//....ppp/aitttiiccieeeee.
Answer: [Yes]
23

---

Justification: Our research fully complies with the NeurIPS Code of Ethics in all respects. No sensitive data or personally identifiable information was used. There are no conflicts of interest or ethical concerns in this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research presented in this paper remains purely at the experimental levels and has not yet results in any societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work does not involve data or models with high risk for misuse, so no additional safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
24

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this work are publicly available for academic research. We have cited the original sources required and, to the best of our knowledge, have respected their terms of use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ee-ackaaeed both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new model with comprehensive documentation provided in the supplemental material, including structured README detailing data collection, preprocessing, usage, license, limitations, and anonymization, in compliance with double- blind review requirements.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper dose not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: Large Language Models were used solely for writing and editing the paper. not as part of the research methods or experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: http://pp...c/ooooeeeeee//////// for what should or should not be described.
26


Reviews:
[{"review_id": "TFGm0L73Ky", "reviewer": "Reviewer_BueR", "summary": "This paper studies gradient dilution issues in vast vocabulary object detection arising from numerous categories: (1) gradients from negative categories overwhelm those from positive categories; (2) within negative categories, gradients from easy samples dominate. To mitigate this, unlike standard classifiers employing static, image-irrelevant category embeddings, the proposed method utilizes image-adaptive category queries to perform image-level category selection prior to instance-level classification. \n\nSpecifically, these category queries aggregate image features via cross-attention and perform image-level classification to filter out absent categories. This significantly reduces the number of categories (e.g., from over 13,000 to 100) for subsequent instance-level classification, thereby alleviating the described gradient imbalance. To handle category hierarchies, the method leverages inherent tree structures to construct hierarchical category queries with an adaptive weighting mechanism. \n\nWhen integrated into Grounding DINO, the proposed approach achieves 53.0 AP on the V3Det dataset, outperforming prior methods. A slight performance gain is also observed on the COCO dataset.", "questions": "I assign a borderline rating due to two primary concerns:\n* Lack of discussion on prior works exploring category query techniques (detailed in Weakness 1),\n* Insufficient validation of the core claim regarding training-stage gradient dilution mitigation (detailed in Weakness 3).\n\nI am open to increasing my rating if the authors address these issues as suggested.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The writing is clear.\n2. The proposed method achieves 53.0 AP on the V3Det dataset, outperforming previous methods including DINO and PROVA by a significant margin. This demonstrates effectiveness for vast vocabulary object detection.\n\nWeaknesses:\n1. Lack of discussion on prior works: The core ideas of this paper—(1) modeling classification categories in an image-adaptive manner instead of using static, image-irrelevant embeddings, and (2) leveraging image-level classification prior to dense predictions—have been explored in existing works such as Query2Label [1], RankSeg [2], and CQL [3]. Query2Label pioneered the concept of category queries. Both RankSeg and CQL employ category queries for image-level classification to enhance dense prediction tasks (semantic segmentation and human-object interaction detection, respectively). RankSeg and CQL both utilize category queries for image-level classification to aid dense prediction tasks (semantic segmentation and human-object interaction detection). RankSeg retains only the top-k semantic categories, a strategy highly similar to this paper and shown to improve accuracy. CQL does not explicitly discard low-confidence categories but integrates image-level classification scores into instance-level scores, downweighting instances from low-confidence categories. However, these works are not discussed. I recommend adding a dedicated section on category query techniques in the related work.\n2. Incomplete baseline comparison: The proposed CQ-DINO is built upon the Grounding DINO framework, yet only DINO is used as the baseline. To isolate the contributions of the proposed method versus the Grounding DINO framework, comparative experiments are necessary.\n3. Insufficient validation of core claims:\n    * Prior works (e.g., RankSeg, CQL) demonstrate that image-level category selection improves inference-stage accuracy for dense prediction tasks by filtering or downweighting low-confidence categories. This paper, however, focus solely about its training-stage impact (mitigating gradient dilution) without addressing inference benefits.\n    * The gradient dilution issue may be largely alleviated by focal loss. As shown in Figure 2, the positive-to-negative gradient ratio is near 0 with CE loss but rises to ~0.5 with focal loss. While CQ-DINO achieves a ratio of ~1.5, the difference is marginal (several-fold rather than orders of magnitude) and potentially addressable through hyper-parameter tuning. Section 4.4 demonstrates that tuning hyper-parameters of focal loss effectively improves vanilla DINO’s performance. Though still below CQ-DINO, it remains unclear whether the performance gap stems from: (a) the Grounding DINO framework, (b) inference-stage category selection, or (c) mitigation of training-stage gradient dilution.\n    * I recommend adding ablations isolating contributions from: (a) Grounding DINO, (b) inference-stage category selection, and (c) training-stage gradient dilution mitigation. This is essential to validate the paper’s central claim.\n\n[1] Query2Label: A Simple Transformer Way to Multi-Label Classification\n\n[2] RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation\n\n[3] Category Query Learning for Human-Object Interaction Classification", "was_revised": true, "final_justification": "I thank the authors for their rebuttal, which includes the discussion and experiments as suggested. The rebuttal has addressed my primary concerns:\n* The lack of related work discussion has been resolved by adding a dedicated section on Category Query-based methods, which clarifies the paper's distinct motivation from these approaches.\n* The insufficient ablation studies have been supplemented with Table R-1, isolating the impacts of inference-stage category selection and training-stage gradient dilution mitigation. This demonstrates that performance gains mainly stem from the latter, thereby supporting the central claim.\n\nOverall, I acknowledge the paper's contributions: the identification and analysis of the gradient dilution problem, an effective solution, and SOTA performance. Therefore, I am voting for acceptance.", "rebuttal": "Thank you for your insightful comments and kind words. We greatly appreciate your recognition of our significant performance improvements. Below we will address your concerns and provide further clarification.\n> Q1: Lack of discussion on prior works.\n> \nA1: Thank you for your valuable feedback regarding the relevant literature [2, 8, 10], particularly the similarity between RankSeg's top-k category selection and our image-guided query selection. We will add a dedicated \"Category Query-based Methods\" section to our related work.\n\nIn the following, we respectfully clarify several key distinctions that differentiate our approach:\n\n1. Novel theoretical motivation:  To the best of our knowledge, our work provides the first systematic analysis of gradient dilution challenges in vast vocabulary detection. This theoretical foundation directly motivates our image-guided query selection design. In contrast, RankSeg employs top-k selection primarily for reducing the negative sample space during inference, which decreases computational cost and improves accuracy. \n2. Scale-specific challenges: While top-k selection during inference works for semantic segmentation with limited vocabularies, vast vocabularies present challenges. Our analysis in Table R-1 reveals: inference-stage category selection harms performance in V3Det (vast vocabulary: 49.4% → 46.6%, rows 2 → 3) while helping in COCO (limited vocabulary: 57.9% → 58.2%). This demonstrates the necessity of addressing gradient dilution during training, not just inference optimization.\n3. Hierarchical relationship modeling: Unlike previous works operating on relatively flat category structures, we specifically address the complex hierarchical relationships inherent in vast vocabulary datasets through our novel hierarchical tree construction method.\n\nTable R-1. Additional ablations on isolated component contributions. \n| Training-stage gradient dilution | Inference-stage category selection | Encoding category correlations | V3Det AP (SwinB-22K) | V3Det FPS | COCO AP (SwinL) | COCO FPS |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0 | 0 | 0 | 47.3 | 0.7 | 57.5 | 9.8 |\n| 0 | 0 | 1 | 49.4 | 0.6 | 57.9 | 9.7 |\n| 0 | 1 | 1 | 46.6 | 10.6 | 58.2 | 9.9 |\n| 1 | 0 | 1 | 51.3 | 0.6 | 58.2 | 9.7 |\n| 1 | 1 | 0 | 51.1 | 10.8 | 58.3 | 10.0 |\n| 1 | 1 | 1 | 52.3 | 10.6 | 58.5 | 9.9 |\n\nWe will incorporate the following related work section:\n\n**Category Query-based Methods**\n\nThe paradigm of learnable queries emerged in computer vision with DETR [1], demonstrating a shift from fixed components to learnable representations that capture task-specific patterns. DETR's query mechanism inspired similar strategies across diverse tasks, including classification [2], segmentation [3], and multimodal learning [4].\n\nAmong these developments, category queries represent an innovation introduced by Query2Label [2]. Rather than relying on fixed classification heads, Query2Label proposed learnable category embeddings to capture category-specific features. Subsequent works such as ML-Decoder [5] have outperformed conventional classification methods. The effectiveness of category queries in classification motivated their extension to dense prediction tasks. CQL [7] employs category queries for human-object interaction classification. Similarly, ControlCap [8] leverages category queries to provide semantic guidance for region captioning tasks. RankSeg [9] integrates category queries into semantic segmentation by performing pixel-level classification while dynamically selecting only the top-k most relevant categories during inference. This selective approach reduces the negative search space, improving both computational efficiency and segmentation accuracy.\n\nWhile prior works have explored category queries in various contexts, our work addresses a fundamentally different challenge specific to vast vocabulary scenarios. We provide the first, to-our-knowledge, systematic theoretical analysis of gradient dilution issues that arise when dealing with vast category vocabularies, which motivates our image-guided query selection design. Additionally, unlike previous methods working with limited category scales, we specifically tackle the complex hierarchical relationships inherent in vast vocabulary datasets through our hierarchical tree construction method, designed to encode category correlations across thousands of semantically related categories.\n\nReference\n\n[1] End-to-End Object Detection with Transformers\n\n[2] Query2Label: A Simple Transformer Way to Multi-Label Classification\n\n[3] Per-Pixel Classification is Not All You Need….\n\n[4] BLIP-2: Bootstrapping Language-Image….\n\n[5] ML-Decoder: Scalable and Versatile Classification Head\n\n[6] Category Query Learning for Human-Object Interaction Classification\n\n[7] ControlCap: Controllable Region-level Captioning\n\n[8] RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation\n> Q2: Incomplete baseline comparison with Grounding DINO.\n> \nA3: As suggested, we provide the following clarification. Detailed in lines 115-119, Grounding DINO faces scalability limitations for vast vocabulary detection. With a token limit of 128 per inference pass and V3Det's 13204 categories, detecting all categories requires over 331 sequential inference passes per image. For the full V3Det val set (30k images), this translates to approximately **768 GPU hours on H800 GPUs** for inference alone, making comprehensive evaluation computationally prohibitive.\n\nTo provide meaningful comparison while addressing computational constraints, we evaluate on V3Det's rare classes in Table R-2. This subset represents the most challenging categories where gradient dilution effects are most pronounced. Grounding DINO is a pretrained model from mmdetection, trained on O365, GoldG, and V3Det datasets. DINO and CQ-DINO are trained on the V3Det dataset.\n\nTable R-2: Performance on V3Det rare classes (1072 categories).\n| Method | AP |\n| --- | --- |\n| DINO | 36.4 |\n| Grounding DINO | 23.4 |\n| CQ-DINO | 46.9 |\n\nGrounding DINO underperforms DINO (-13.0%), revealing limitations in its training paradigm in vast vocabulary settings. Grounding DINO uses positive labels with only randomly sampled small sets of negative labels during training. With few similar categories sampled during training, Grounding DINO fails to learn fine-grained distinctions between semantically similar categories—the core challenge in vast vocabulary detection.\n\nTo isolate Grounding DINO’s component impacts, we conduct ablation studies in Table 4 (row 1) and Table R-1 (row 1). Substantial improvements show our method focuses on addressing the fundamental challenges of vast vocabulary detection rather than simply benefiting from architectural choices.\n> Q3: Adding ablations isolating contributions from: (a) Grounding DINO, (b) inference-stage category selection, and (c) training-stage gradient dilution mitigation.\n> \nA2: Thank you for this insightful suggestion. We now provide additional ablation studies in Table R-1 to isolate the contributions of each component and validate our main claim about gradient dilution mitigation.\n\nComponent Definitions:\n(a) Grounding DINO components: The baseline architecture incorporating feature enhancer and decoder components from GroundingDINO (row 1 in Table R-1).\n\n(b) Inference-stage category selection:  Selecting only top-K relevant categories based on similarity of category queries and image features during inference.\n\n(c) Training-stage gradient dilution mitigation: Using image-guided query selection during training to mitigate gradient dilution.\n\nKey observations from Table R-1:\n\n(1) Training-stage gradient dilution mitigation is the primary contributor. On V3Det, this component alone provides a 1.9% AP improvement (comparing rows 2 and 4). This validates our hypothesis that gradient dilution is a challenge in vast vocabulary detection. On COCO, the improvement is modest (0.3% AP) since gradient dilution is less severe with only 80 categories.\n\n(2) Inference-stage category selection has different effects with respect to vocabulary size.\n\n• V3Det: Inference selection without gradient dilution mitigation actually hurts performance (rows 2 vs 3), demonstrating that simply reducing categories at inference cannot solve the underlying gradient dilution problem.\n\n• COCO: Inference selection provides modest improvements as expected when the negative space is reduced.\n\n(3) The combination achieves optimal performance and efficiency. When both training-stage gradient dilution mitigation and inference-stage selection are combined (row 6), we achieve the best results on both datasets. Inference-stage selection improves inference speed (0.6 → 10.6 FPS on V3Det).\n\nThese additional ablation studies further demonstrate that (c) training-stage gradient dilution mitigation is the core contribution, while (a) Grounding DINO components provide the architectural foundation and (b) inference-stage category selection offers computational efficiency. Table R-1 will be incorporated into Table 4 in the revised paper.\n> Q4: Are Focal loss and CQ-DINO equally effective in solving gradient dilution problems in vast vocabulary detection?\n> \nNo, CQ-DINO and Focal Loss address gradient dilution problems from different aspects. Our theoretical analysis in Sec. 3.1 reveals that the positive-to-negative gradient ratio scales as ρ ∝ 1/(C·ε), where C is the vocabulary size and ε represents the average activation probability of negative classes.\n\nWhile Focal Loss can mitigate gradient dilution by adjusting its parameters to reduce ε, it cannot address the core 1/C scaling problem inherent to vast vocabulary detection. As vocabulary size C grows beyond 10⁴ categories, this scaling factor becomes the dominant bottleneck regardless of ε optimization. In contrast, CQ-DINO fundamentally solves the scaling issue by reducing the effective vocabulary size from C to C' (where C' ≪ C) through image-guided query selection."}, {"review_id": "Ty1v6c9Pp6", "reviewer": "Reviewer_YWme", "summary": "This paper investigates scaling object detectors to vast vocabularies. The authors identify and analyze two key issues with standard classification-based methods: \"positive gradient dilution,\" where positive samples are overwhelmed by the sheer number of negatives, and \"hard negative gradient dilution,\" where informative negative gradients are lost among a sea of easy ones. To address this, they propose CQ-DINO, a novel framework that reformulates classification as a contrastive task. The core of their method is an image-guided query selection module that dynamically selects a small subset of relevant \"category queries\" for each image. This pruning of the category space aims to rebalance gradients and perform implicit hard negative mining. The selected queries are then used in a DINO-like architecture to produce detections. The authors demonstrate state-of-the-art results on the vast-vocabulary V3Det benchmark.", "questions": "1.  The category recall of 83.3% appears to be a bottleneck. Have you analyzed the characteristics of the ~17% of objects whose categories are missed by the selection module? Are they primarily rare categories, visually similar categories, or small objects? Understanding this failure mode is critical, as my evaluation of the method's robustness hinges on whether this ceiling can be raised significantly.\n2.  While CQ-DINO outperforms a tuned Focal Loss baseline by 4.0% AP, it is also a much more complex system. How can we be sure that the performance gain is due to a fundamentally better approach to gradient balancing, rather than a side effect of adding more complexity, parameters, and a separate supervision signal (the selection loss) to the overall training pipeline?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.  The paper provides an insightful analysis of the gradient dilution problem, offering a clear empirical motivation for why existing methods struggle with vast vocabularies.\n2.  The main idea of using learnable category queries and an image-guided selection mechanism to dynamically prune the search space is clever. It directly targets the identified problem of an overwhelming number of negative classes.\n3. The experimental results on the challenging V3Det benchmark are strong, showing a clear improvement over previous methods, including a well-tuned DINO baseline. This demonstrates the empirical effectiveness of the proposed approach.\n\nWeakness:\n1. The framework is quite complex and relies on a two-stage training process: a pre-training stage for the selection module followed by a full-pipeline fine-tuning stage. As the authors acknowledge in their limitations, this can lead to suboptimal coordination between the stages and makes the model more difficult to train and deploy compared to a true end-to-end solution.\n2.  While the analysis of \"gradient dilution\" is thorough, the underlying issue is a severe case of the well-known class imbalance problem. Focal Loss was designed to address the dominance of easy negatives. The paper shows their method works better than tuning Focal Loss, but the problem framing itself is not entirely new.\n3. The paper heavily utilizes components from Grounding DINO for the feature enhancement and decoding stages. As such, the contribution can be seen less as a completely new detection paradigm and more as a sophisticated \"pre-filtering\" front-end for an existing powerful detector.", "was_revised": true, "final_justification": "The author's rebuttal has solved most of my concerns. Hence, I decided to raise my score to borderline accept.", "rebuttal": "Thank you for thoroughly reading our paper. We are deeply grateful that you recognize the significance of our gradient dilution analysis. We are very thankful that you find our method to be a clever solution to the identified gradient dilution problems. We address your concerns below.\n\n> Q1: Complexity of our two-stage training approach.\n> \n\nA1: Indeed, the two-stage training introduces some additional complexity. In practice, the overhead is minimal. Stage 1 requires only ~1 hour compared to the 57 hours for full training. For deployment, only the final trained model is needed without two-stage training. For future work, we are actively exploring methods such as progressive query selection to develop stable end-to-end alternatives to keep high category recall throughout training.\n\n> Q2: Gradient dilution issue VS class imbalance problem.\n> \n\nA2: We appreciate the reviewer's insight connecting gradient dilution to class imbalance. However, our work identifies and addresses a distinct scaling challenge in vast vocabularies that differs from traditional class imbalance problems.\n\nTraditional class imbalance mainly focuses on sample distribution disparities between rare and common classes. In contrast, gradient dilution in vast vocabularies affects even common classes due to the overwhelming negative space created by thousands of categories. Our theoretical analysis (Equations 2-4) demonstrates that the positive-to-negative gradient ratio scales as $\\rho \\propto \\frac{1}{C\\cdot ε}$, where $C$ is the vocabulary size and ε represents the average activation probability of negative classes. This means the problem severity increases proportionally with vocabulary size $C$, not just sample imbalance. Moreover, vast vocabularies cause informative hard negatives to be overwhelmed by numerous easy negatives.\n\nTo address the concern about whether CQ-DINO can alleviate the class imbalance problem in V3Det, we evaluate the results of rare classes (number of training samples less than 10) and common classes (otherwise) in Table R-3. The results show that even with carefully tuned hyperparameters, Focal Loss cannot fundamentally solve the 1/C scaling problem. Our dynamic category selection directly addresses the optimization challenges, achieving consistent improvements across both rare and common categories.\n\nTable R-3: Performance of rare classes and common classes on V3Det dataset.\n\n| Method | Rare | Common | All |\n| --- | --- | --- | --- |\n| DINO-SwinB-22K | 36.4 | 43.8 | 43.4 |\n| DINO-SwinB-22K (Focal α=0.50, γ=2) | 42.1 | 48.4 | 47.4 |\n| CQ-DINO-SwinB-22K | 45.3 | 53.3 | 52.3 |\n| DINO-SwinL | 45.2 | 51.0 | 48.5 |\n| DINO-SwinL (Focal α=0.50, γ=2) | 45.5 | 51.7 | 50.1 |\n| CQ-DINO-SwinL | 46.9 | 54.1 | 53.0 |\n\n> Q3: Is this just pre-filtering for existing Grounding DINO detector?\n> \n\nA3: No, CQ-DINO is fundamentally different from pre-filtering. It addresses the optimization challenges that Grounding DINO cannot solve through architectural modifications alone.\n\nGrounding DINO faces scalability bottlenecks for vast vocabulary detection (Section 2.2, lines 115-119). With its 128 token limit per inference pass, detecting all 13,204 V3Det categories requires **over 331 sequential inference passes per image**. For the complete V3Det validation set (30k images), this requires approximately **768 GPU hours** on H800 GPUs for inference alone, which is almost infeasible with our limited computational resources.\n\nWhile we leverage Grounding DINO's proven feature enhancer and decoder (which excel at fusing image features and category queries), our baseline performance is nearly identical to the optimally Focal loss tuned DINO (47.3% Table3 row 1 vs 47.4% Table 9 row5). This further confirms that architectural components alone cannot solve vast vocabulary challenges.\n\n> Q4:  What characterizes the 17% of missed objects causing the category recall bottleneck - rare categories, small objects, or visual similarity?\n> \n\nA4: We analyze the 3,527 categories with recall lower than 83.3% to understand the characteristics of missed objects. Our analysis reveals that both rare category performance and small object detection represent significant bottlenecks compared to our overall performance of 52.3% AP.\n\n(1) **Category frequency analysis (Table R-4):** Among the low-recall categories, 374 are rare categories and 3,153 are common categories. On these challenging categories, CQ-DINO achieves 20.5% AP on rare categories, substantially lower than our overall 52.3% AP, indicating that rare categories face a significant performance gap.\n\n(2)  **Object scale analysis (Table R-5):** The most significant performance gaps occur for small objects, where CQ-DINO achieves only 14.1% AP compared to our overall performance of 52.3% AP. This indicates that object scale is the primary factor in the recall bottleneck, with small objects being particularly challenging.\n\nWhile we do not achieve strong absolute performance in these challenging cases, CQ-DINO still consistently outperforms both baseline DINO and DINO using carefully tuned focal loss across all categories and scales. This demonstrates that **these are common difficulties in vast vocabulary detection** rather than unique limitations of our approach.\n\nTable R-4:  Analysis on low recall categories.\n\n| Method | Rare | Common | All |\n| --- | --- | --- | --- |\n| DINO-SwinB-22K | 15.5 | 27.1 | 25.8 |\n| DINO-SwinB-22K (Focal α=0.50, γ=2) | 19.9 | 31.1 | 29.9 |\n| CQ-DINO-SwinB-22K | 20.5 | 32.1 | 30.9 |\n\nTable R-5: Analysis on object scales.\n\n| Method | Small | Middle | Large |\n| --- | --- | --- | --- |\n| DINO-SwinB-22K | 9.1 | 16.5 | 33.8 |\n| DINO-SwinB-22K (Focal α=0.50, γ=2) | 11.6 | 20.2 | 38.3 |\n| CQ-DINO-SwinB-22K | 14.1 | 23.1 | 38.9 |\n\n> Q5: Does CQ-DINO's 4.9% AP improvement stem from gradient balancing or increased model complexity?\n> \n\nA4: The performance gain is fundamentally owing to the gradient balancing, not model complexity. We provide comprehensive evidence through detailed ablation studies:\n\n(1) Our baseline without image-guided query selection (47.3% AP in Table 4, row 1) achieves nearly identical performance to DINO trained with optimally-tuned Focal Loss (47.4% AP in Table 9). This demonstrates that the model architecture itself does not provide inherent advantages.\n\n(2) Table 4 shows that adding image-guided query selection alone improves AP from 47.3% to 51.1% (+3.8%), which directly addresses gradient dilution by reducing the negative search space. Table R-1 shows that training-stage gradient dilution mitigation alone (row 4) provides +1.9% AP. This further validates that gradient rebalancing is the core mechanism.\n\n(3) Encoding category correlations contributes +1.2% AP in Table 4 (52.3% vs 51.1%, row 6 vs row 5).\n\nTable R-1. Additional ablations on isolated component contributions. 0 indicates the component is disabled and 1 indicates it is enabled.\n\n| Training-stage gradient dilution | Inference-stage category selection | Encoding category correlations | V3Det AP (SwinB-22K) | V3Det FPS | COCO AP (SwinL) | COCO FPS |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0 | 0 | 0 | 47.3 | 0.7 | 57.5 | 9.8 |\n| 0 | 0 | 1 | 49.4 | 0.6 | 57.9 | 9.7 |\n| 0 | 1 | 1 | 46.6 | 10.6 | 58.2 | 9.9 |\n| 1 | 0 | 1 | 51.3 | 0.6 | 58.2 | 9.7 |\n| 1 | 1 | 0 | 51.1 | 10.8 | 58.3 | 10.0 |\n| 1 | 1 | 1 | 52.3 | 10.6 | 58.5 | 9.9 |"}, {"review_id": "45SoI5h25c", "reviewer": "Reviewer_8AUD", "summary": "This paper addresses the critical challenge of scalable object detection in vast vocabulary scenarios, an increasingly important problem in AI. The authors identify two key issues in classification-based methods: positive gradient dilution and hard negative gradient dilution, which hinder performance in large-scale settings. To overcome these, the proposed Category Query-based DINO (CQ-DINO) introduces learnable category queries and an image-guided query selection mechanism. These innovations dynamically reduce the negative search space, balance gradient updates, and improve computational efficiency. The paper demonstrates strong results on the V3Det dataset, achieving state-of-the-art performance, while also delivering competitive results on COCO. The methodology is well-motivated, with a clear focus on scalability and efficiency, and the experimental results support the claims. However, further analysis on generalization to other datasets and a more detailed comparison with alternative architectures would strengthen the work. Overall, the paper makes a meaningful contribution to vast vocabulary object detection.", "questions": "Could the proposed CQ-DINO handle category correlations in datasets with incomplete or ambiguous hierarchies?", "limitations": "Visualization and deeper analysis is missing.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper is well motivated. Since Vast Vocabulary Object Detection task have numeous classes, mining the sparse reward from positive categories is meaningful.\n2. This paper is well written and easy to follow.\n3. Experiments show its effectiveness, not only suporier in V3Det benchmarks, but also competitive in COCO dataset, comparable with strong baselines like Rank-DETR.\n\nWeakness:\n1. The Positive-to-negative gradient plot should be drawn in longer iterations.", "was_revised": true, "final_justification": "I thank the authors for their thorough rebuttal, as well as the additional experiment results. This rebuttal has resolved the majority of my concerns. As a result, I decide to keep my rating as borderline accept.", "rebuttal": "Thank you for your positive feedback and recognition of our work. We are deeply grateful for your positive assessment of our motivation and experimental results. Below, we address your concerns point-by-point.\n\n> Q1: The positive-to-negative gradient plot should be drawn in longer iterations.\n> \n\nA1: Thank you for this suggestion. We have extended our positive-to-negative gradient ratio analysis to 200k iterations to provide a more comprehensive view of the training dynamics.  Due to policy constraints that restrict the inclusion of images, we present the results in Table R-6 below, sampling key iterations throughout the training process.\n\nThe extended analysis reveals several key insights: CQ-DINO maintains consistently higher positive-to-negative gradient ratios during the critical early training phase (less than 10k iterations), demonstrating effective gradient rebalancing when learning is most crucial. In contrast, DINO with Cross-Entropy loss exhibits severe positive gradient dilution in early iterations (starting at 0.01) before gradually recovering. While all methods eventually converge to similar gradient ratios (~1.1) in later training stages (>25k iterations), CQ-DINO achieves this balanced state much earlier, resulting in more efficient training.\n\nThis extended analysis strongly supports our hypothesis that CQ-DINO effectively addresses the positive gradient dilution problem identified in vast vocabulary detection, particularly during the crucial early training phase where proper gradient balance is essential for learning meaningful representations.\n\nTable R-6:  Positive-to-negative gradient ratio across extended training iterations.\n\n| Iter | 0.5k | 1k | 1.5k | 2k | 2.5k | 3k | 5k | 7k | 10k | 15k | 25k | 50k | 75k | 100k | 200k |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DINO w/CE | 0.01 | 0.09 | 0.11 | 0.06 | 0.11 | 0.28 | 0.43 | 0.62 | 0.79 | 0.93 | 1.00 | 1.10 | 1.04 | 1.12 | 1.10 |\n| DINO w/FL | 0.63 | 0.31 | 0.47 | 0.68 | 0.87 | 0.77 | 0.91 | 0.84 | 0.88 | 0.95 | 1.10 | 1.07 | 1.09 | 1.17 | 1.12 |\n| CQ-DINO | 1.08 | 1.56 | 1.21 | 1.64 | 1.70 | 1.39 | 1.50 | 1.23 | 1.12 | 1.18 | 1.12 | 1.19 | 1.16 | 1.08 | 1.14 |\n\n> Q2: Could the proposed CQ-DINO handle category correlations in datasets with incomplete or ambiguous hierarchies?\n> \n\nA2: Great question!  To evaluate CQ-DINO's robustness to incomplete hierarchical information, we conducted controlled experiments by artificially degrading the V3Det hierarchical structure in two ways:\n\n**1. Missing hierarchies**: We randomly removed hierarchical connections for 1/6 of the categories, simulating datasets with incomplete structural information.\n\n**2. Ambiguous hierarchies**: We introduced fuzzy/ambiguous hierarchical relationships for another 1/6 of the categories, representing scenarios where category boundaries are unclear.\n\nWhen hierarchical information is incomplete, CQ-DINO achieves 51.7% AP and outperforms the baseline without category correlations (51.1% AP) by 0.6% AP.  These results demonstrate that CQ-DINO can handle real-world scenarios where hierarchical structures are imperfect or incomplete. Besides, when hierarchical information is completely missing, CQ-DINO can seamlessly switch to implicit relation learning through the self-attention module (51.3% AP), which also outperforms the no-correlation baseline.\n\nTable R-7: Performance under incomplete hierarchical structures.\n\n| Method | $AP$ | $AP_{50}$ | $AP_{75}$ | $AR^c$ |\n| --- | --- | --- | --- | --- |\n| No correlation | 51.1 | 56.6 | 53.5 | 80.9 |\n| Self-Attention module | 51.3 | 56.7 | 53.7 | 75.5 |\n| Incomplete hierarchical tree | 51.7 | 57.1 | 54.0 | 82.47 |\n| Complete hierarchical tree | 52.3 | 57.7 | 54.6 | 83.3 |\n\n> Q3: Visualization and deeper analysis is missing.\n> \n\nA3: To provide deeper analysis beyond the positive-to-negative gradient ratio, we conducted additional experiments examining the hard negative gradient dilution problem discussed in Section 3.1. We define hard negatives as the top 10% of categories with the highest prediction scores among all negative categories, representing the most confusing negatives that require focused learning.\n\nResults depicted in Table R-8 show that our CQ-DINO consistently achieves the highest hard negative ratios throughout training, particularly in the critical early stages (less than 10k iterations), confirming our theoretical analysis (in Section 3.1) that image-guided query selection performs effective implicit hard negative mining.\n\n Due to policy constraints that restrict the inclusion of images in this response, we will include comprehensive visualizations in the revised paper showing: (1) positive-to-negative gradient ratio plot. (2) Hard negative-to-all negative gradient ratio plot. (3) Visualization of detection performance on challenging samples and failure case analysis.\n\nTable R-8: Hard negative-to-all negative gradient ratio across training iterations\n\n| Iter | 0.5k | 1k | 1.5k | 2k | 2.5k | 3k | 5k | 7k | 10k | 15k | 25k | 50k | 75k | 100k | 200k |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DINO w/CE | 0.23 | 0.53 | 0.61 | 0.67 | 0.72 | 0.77 | 0.89 | 0.97 | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |\n| DINO w/FL | 0.69 | 0.65 | 0.63 | 0.62 | 0.64 | 0.66 | 0.81 | 0.88 | 0.93 | 0.97 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 |\n| CQ-DINO | 0.83 | 0.90 | 0.89 | 0.91 | 0.92 | 0.93 | 0.94 | 0.96 | 0.98 | 0.98 | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 |"}, {"review_id": "SQwnx7tWEt", "reviewer": "Reviewer_9iMy", "summary": "In this paper, the authors propose CQ-DINO, a DINO variant for vast-vocabulary object detection. To effectively reduce the effect from huge label space, CQ-DINO leverages Image-Guided Query Selection, which can reduce the label space and rebalance the gradient during training. To build category correlations for precise query selection, the authors propose Hierarchical Tree Construction and Implicit Relation Learning. Experimental show that CQ-DINO effectively improves the performance of vast-vocabulary object detection.", "questions": "See the weakness part.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. The performance is promising.\n\n2. The idea is intuitive and reasonable. \n\n## Weakness\n\n1. The most critical weakness of this paper is the description of Image-Guided Query Selection and Encoding Category Correlations. The technical details are omitted, which limits the expression of both motivation of this method and merits of proposed methods. This drawback restricts the pre-rebuttal score. We recommend the authors to 1) clearly formulate the process of Image-Guided Query Selection by equation of algorithm, 2) compare the proposed selection methods with other solutions (e.g., using off-the-shelf classifiers), 3) clearly formulate the process of Encoding Category Correlations. These improvements can improve the readability of this paper. \n\n2. The gradient norm shown in Fig 6 is not sufficient to show the effectiveness of rebalancing gradient and gradient dilution. The authors should provide additional analysis to explicitly investigate the change in gradient during training.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding unclear description and key comparison.", "rebuttal": "Thank you for the careful reading and kind words. We are pleased that you find our performance promising and our idea intuitive and reasonable. We address your concerns in the following.\n> Q1: Clearly formulate the process of Image-Guided Query Selection by equation of algorithm.\n> \nA1: Thank you for this valuable suggestion. Below, we provide the algorithmic descriptions (which we believe is more clear than equations) of the  Image-Guided Query Selection process. This algorithm share enhance the clarity of our method in the revised paper.\n### Algorithm 1: Image-Guided Query Selection\n---\n**Input:**\n- B: Batch size\n- C: Number of category queries  \n- D: Embedding dimension\n- C': Number of selected category queries\n- H': Height of image features\n- W': Width of image features\n- $Q_{cat}$: Category queries ∈ $R^{B×C×D}$\n- $F_{img}$: Image features ∈ $R^{B×D×H'×W'}$\n\n**Output:**\n- $Q_{cat}'$: Selected enhanced category queries ∈ $R^{B×C'×D}$\n- $I$: Selection indices ∈ $R^{B×C'}$\n---\n**1:** $F_{flat}$ ← Reshape($F_{img}$, [B, D, H'×W'])\n\n**2:** $Q_{enhanced}$ ← $Q_{cat}$ &nbsp;&nbsp;&nbsp;&nbsp;*// Initialize enhanced queries*\n\n**3:** **for** l = 1 **to** 2 **do** &nbsp;&nbsp;&nbsp;&nbsp;*// Two cross-attention layers*\n\n**4:** &nbsp;&nbsp;&nbsp;&nbsp;$Q_{attn}$ ← MultiHeadCrossAttention($Q_{enhanced}$, $F_{flat}$, $F_{flat}$) &nbsp;&nbsp;&nbsp;&nbsp;*// Cross-attention*\n\n**5:** &nbsp;&nbsp;&nbsp;&nbsp;$Q_{enhanced}$ ← LayerNorm($Q_{attn}$ + $Q_{enhanced}$) &nbsp;&nbsp;&nbsp;&nbsp;*// Add & Norm*\n\n**6:** &nbsp;&nbsp;&nbsp;&nbsp;$Q_{ffn}$ ← FFN($Q_{enhanced}$) &nbsp;&nbsp;&nbsp;&nbsp;*// Feed-forward network*\n\n**7:** **end for**\n\n**8:** $L$ ← LinearProjection($Q_{enhanced}$) &nbsp;&nbsp;&nbsp;&nbsp;*// Compute selection logits* ∈ $R^{B×C}$\n\n**9:** **for** b = 1 **to** B **do**\n\n**10:** &nbsp;&nbsp;&nbsp;&nbsp;$I_{(b,:)}$ ← TopK($L_{(b,:)}$, C') &nbsp;&nbsp;&nbsp;&nbsp;*// Select top-C' indices*\n\n**11:** &nbsp;&nbsp;&nbsp;&nbsp;$Q_{cat}'$[b,:,:] ← $Q_{enhanced}$[b, $I_{(b,:)}$, :] &nbsp;&nbsp;&nbsp;&nbsp;*// Select enhanced queries*\n\n**12:** **end for**\n\n**13:** **return** $Q_{cat}'$, $I$\n\n---\n> Q2: Compare the proposed selection methods with other solutions (e.g., using off-the-shelf classifiers)\n> \nA2: Following your suggestion, we conducted experiments comparing our end-to-end image-guided query selection with alternative selection strategies to demonstrate the necessity of joint optimization. We designed two alternative approaches for the query selection mechanism:\n\n1.Frozen Selection: We pre-trained all encoder components (image encoder, category queries, and image-guided query selection module) for classification, then froze these components entirely during detection training. \n\n2.Dual-Encoder Off-the-shelf: Based on the frozen selection method, we duplicated the image encoder architecture. One copy remains frozen together with category queries and selection modules to serve as an off-the-shelf classifier for category selection, while the second image encoder continues training for the detection task. This approach represents a compromise between using off-the-shelf components and task-specific optimization, separating classification and detection feature learning.\n\nTable R-9: Comparison with alternative selection strategies.\n\n| Method | AP | $AR^C$ | FPS | Training memory |\n| --- | --- | --- | --- | --- |\n| Frozen selection | 50.5 | 85.7 | 10.6 | 28.2GB |\n| Dual-encoder off-the-shelf | 51.9 | 85.7 | 7.9 | 46.7GB |\n| CQ-DINO (End-to-end) | 52.3 | 83.3 | 10.6 | 33.3GB |\n\nOur end-to-end approach achieves the highest performance, outperforming the frozen selection by 1.8% AP and the dual-encoder approach by 0.4% AP. This demonstrates that joint optimization is important. Though the dual-encoder approach achieves similar results to our CQ-DINO (51.9% vs 52.3% AP), its main weakness lies in the significant computational overhead. It reduces inference speed to from 10.6 to 7.9 FPS while requiring 40% more memory (46.7GB vs 33.3GB). This brings more challenge for real-world deployment despite competitive accuracy.\n> Q3:  Clearly formulate the process of Encoding Category Correlations\n> \nA3: Thank you for this important feedback. We provide a detailed algorithmic formulation of the Category Correlation Encoding process, which will be included in the revised paper.\n## 2. Encoding Category Correlations\n\n### Algorithm 2: Explicit Hierarchical Tree Construction\n---\n**Input:**\n- $Q_{cat}$: Category queries ∈ $R^{C×D}$\n- T: Hierarchical tree structure with nodes V\n- w: Base weight parameter (default: 0.3)\n\n**Output:**\n- $Q_{corr}$: Correlation-enhanced queries ∈ $R^{C×D}$\n---\n**1:** $Q_{corr}$ ← $Q_{cat}$ &nbsp;&nbsp;&nbsp;&nbsp;*// Initialize with original queries*\n\n**2:** $N_{max}$ ← $max_{v∈V} |Children(v, T)| $&nbsp;&nbsp;&nbsp;&nbsp;*// Find maximum children count*\n\n**3:** L ← TopologicalSort(T) &nbsp;&nbsp;&nbsp;&nbsp;*// Get bottom-up ordering*\n\n**4:** **for each** node v in Reverse(L) **do** &nbsp;&nbsp;&nbsp;&nbsp;*// Bottom-up traversal*\n\n**5:** &nbsp;&nbsp;&nbsp;&nbsp;**if** IsLeaf(v, T) **then**\n\n**6:** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**continue** &nbsp;&nbsp;&nbsp;&nbsp;*// Keep original query for leaf nodes*\n\n**7:** &nbsp;&nbsp;&nbsp;&nbsp;**else**\n\n**8:** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;C(v) ← GetChildren(v, T) &nbsp;&nbsp;&nbsp;&nbsp;*// Get children of node v*\n\n**9:** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$n_v$ ← |C(v)| &nbsp;&nbsp;&nbsp;&nbsp;*// Count children*\n\n**10:** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$α_v ← w + log(n_v + 1)/log(N_{max} + 1)$ &nbsp;&nbsp;&nbsp;&nbsp;*// Adaptive weight (Eq. 7)*\n\n**11:** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$α_v ← min(α_v, 1.0)$ &nbsp;&nbsp;&nbsp;&nbsp;*// Ensure $α_v$ ∈ [0, 1]*\n\n**12:** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$Q_{child}^{mean} ← (1/n_v) ∑_{c∈C(v)} Q_{corr}[c,:]$ &nbsp;&nbsp;&nbsp;&nbsp;*// Mean pooling*\n\n**13:** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$Q_{corr}[v,:] ← (1-α_v)·Q_{cat}[v,:] + α_v·Q_{child}^{mean}$ &nbsp;&nbsp;&nbsp;*// Update (Eq. 6)*\n\n**14:** &nbsp;&nbsp;&nbsp;&nbsp;**end if**\n\n**15:** **end for**\n\n**16:** **return** $Q_{corr}$\n\n---\n\n### Algorithm 3: Self-Attention for Implicit Category Relations\n---\n**Input:**\n- $Q_{cat}$: Category queries ∈ $R^{C×D}$\n- H: Number of attention heads\n\n**Output:**\n- $Q_{corr}$: Correlation-enhanced queries ∈ $R^{C×D}$\n---\n**1:** $Q_{attn}$ ← MultiHeadSelfAttention($Q_{cat}$) &nbsp;&nbsp;&nbsp;&nbsp;*// Multi-head self-attention*\n\n**2:** $Q_{corr}$ ← LayerNorm($Q_{attn}$ + $Q_{cat}$) &nbsp;&nbsp;&nbsp;&nbsp;*// Add & Norm*\n\n**3:** **return** $Q_{corr}$\n\n---\n> Q4. More **explicit gradient change analysis to show the effectiveness of rebalancing gradient and gradient dilution.**\n> \nA4: We provide comprehensive gradient change analysis with extended training iterations to explicitly demonstrate the effectiveness of our gradient rebalancing mechanism in addressing both positive gradient dilution and hard negative gradient dilution.\n\n1.Extended Positive-to-Negative Gradient Ratio Analysis\n\nWe extend the training iterations in Fig 2 as shown in Table R-6. Due to policy constraints that restrict the inclusion of images, we present the extended analysis through sampled key iterations across 200k training steps.\n\nResults show that CQ-DINO maintains early rebalancing advantage. CQ-DINO maintains consistently higher ratios (>1.0) throughout the critical early training phase (less than 10k iterations), when gradient balance is most crucial for learning meaningful representations. In contrast, DINO with Cross-Entropy loss starts with critically low positive gradient ratios (0.01 at 0.5k iterations), demonstrating severe positive gradient dilution.\n\n2.Hard Negative Gradient Dilution Analysis\n\nTo examine the hard negative gradient dilution problem discussed in Section 3.1, we define hard negatives as the top 10% of categories with the highest prediction scores among all negative categories, representing the most confusing negatives requiring focused learning.\n\nResults in Table R-8 show that CQ-DINO consistently achieves the highest hard negative ratios throughout training, particularly during critical early stages (0.83 vs 0.23 for DINO w/CE at 0.5k iterations). The consistently high ratios confirm our theoretical analysis that image-guided query selection performs effective implicit hard negative mining by naturally selecting the most semantically challenging and relevant categories.\n\nThe results validate our theoretical analysis and demonstrate the practical effectiveness of image-guided query selection in vast vocabulary detection scenarios. The original plots will be included in the revised paper.\n\nTable R-6:  Positive-to-negative gradient ratio across extended training iterations.\n\n| Iter | 0.5k | 1k | 1.5k | 2k | 2.5k | 3k | 5k | 7k | 10k | 15k | 25k | 50k | 75k | 100k | 200k |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DINO w/CE | 0.01 | 0.09 | 0.11 | 0.06 | 0.11 | 0.28 | 0.43 | 0.62 | 0.79 | 0.93 | 1.00 | 1.10 | 1.04 | 1.12 | 1.10 |\n| DINO w/FL | 0.63 | 0.31 | 0.47 | 0.68 | 0.87 | 0.77 | 0.91 | 0.84 | 0.88 | 0.95 | 1.10 | 1.07 | 1.09 | 1.17 | 1.12 |\n| CQ-DINO | 1.08 | 1.56 | 1.21 | 1.64 | 1.70 | 1.39 | 1.50 | 1.23 | 1.12 | 1.18 | 1.12 | 1.19 | 1.16 | 1.08 | 1.14 |\n\nTable R-8: Hard negatives-to-all negative gradient ratio across different training iterations\n\n| Iter | 0.5k | 1k | 1.5k | 2k | 2.5k | 3k | 5k | 7k | 10k | 15k | 25k | 50k | 75k | 100k | 200k |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DINO w/CE | 0.23 | 0.53 | 0.61 | 0.67 | 0.72 | 0.77 | 0.89 | 0.97 | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |\n| DINO w/FL | 0.69 | 0.65 | 0.63 | 0.62 | 0.64 | 0.66 | 0.81 | 0.88 | 0.93 | 0.97 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 |\n| CQ-DINO | 0.83 | 0.90 | 0.89 | 0.91 | 0.92 | 0.93 | 0.94 | 0.96 | 0.98 | 0.98 | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 |"}]

Author rebuttal:

Meta-review: This paper proposes a method to mitigate gradient dilution problem in vast vocabulary object detection, where the positive categories and hard negative categories receive little learning signal compared to the overwhelming easy negatives. Reviewers appreciate the good empirical performance (BueR, YWme, 8AUD, 9iMy), well-motivated intutive idea (YWme, 8AUD, 9iMy), analysis of gradient dilution (YWme), and clear writing (BueR, 8AUD). The authors successfully addressed the concerns of all reviewers (e.g. analysis of gradient dilution, ablation on training vs inference contribution, comparison with Grounding DINO). After considering the reviews and rebuttals, AC recommends acceptance of the paper due to its clear insight, good analysis, and strong empirical results.
Decision: Accept (poster)