Title: CQ-DINO: Mitigating Gradient Dilution via Category Queries for Vast Vocabulary Object Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Object detection

vast vocabulary
Abstract: With the exponential growth of data, traditional object detection methods are increasingly struggling to handle vast vocabulary object detection tasks effectively. We analyze two key limitations of classification-based detectors: positive gradient dilution, where rare positive categories receive insufficient learning signals, and hard negative gradient dilution, where discriminative gradients are overwhelmed by numerous easy negatives. To address these challenges, we propose CQ-DINO, a category query-based object detection framework that reformulates classification as a contrastive task between object queries and learnable category queries. Our method introduces image-guided query selection, which reduces the negative space by adaptively retrieving top-K relevant categories per image via cross-attention, thereby rebalancing gradient distributions and facilitating implicit hard example mining.
Furthermore, CQ-DINO flexibly integrates explicit hierarchical category relationships in structured datasets (e.g., V3Det) or learns implicit category correlations via self-attention in generic datasets (e.g., COCO). Experiments demonstrate that CQ-DINO achieves superior performance on the challenging V3Det benchmark (surpassing previous methods by 2.1% AP) while maintaining competitiveness in COCO. Our work provides a scalable solution for real-world detection systems requiring wide category coverage.


Peer reviews:
comments: SUMMARY:
In this paper, the authors propose CQ-DINO, a DINO variant for vast-vocabulary object detection. To effectively reduce the effect from huge label space, CQ-DINO leverages Image-Guided Query Selection, which can reduce the label space and rebalance the gradient during training. To build category correlations for precise query selection, the authors propose Hierarchical Tree Construction and Implicit Relation Learning. Experimental show that CQ-DINO effectively improves the performance of vast-vocabulary object detection.

STRENGTHS AND WEAKNESSES:
## Strength

1. The performance is promising.

2. The idea is intuitive and reasonable. 

## Weakness

1. The most critical weakness of this paper is the description of Image-Guided Query Selection and Encoding Category Correlations. The technical details are omitted, which limits the expression of both motivation of this method and merits of proposed methods. This drawback restricts the pre-rebuttal score. We recommend the authors to 1) clearly formulate the process of Image-Guided Query Selection by equation of algorithm, 2) compare the proposed selection methods with other solutions (e.g., using off-the-shelf classifiers), 3) clearly formulate the process of Encoding Category Correlations. These improvements can improve the readability of this paper. 

2. The gradient norm shown in Fig 6 is not sufficient to show the effectiveness of rebalancing gradient and gradient dilution. The authors should provide additional analysis to explicitly investigate the change in gradient during training.

QUESTIONS:
See the weakness part.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical challenge of scalable object detection in vast vocabulary scenarios, an increasingly important problem in AI. The authors identify two key issues in classification-based methods: positive gradient dilution and hard negative gradient dilution, which hinder performance in large-scale settings. To overcome these, the proposed Category Query-based DINO (CQ-DINO) introduces learnable category queries and an image-guided query selection mechanism. These innovations dynamically reduce the negative search space, balance gradient updates, and improve computational efficiency. The paper demonstrates strong results on the V3Det dataset, achieving state-of-the-art performance, while also delivering competitive results on COCO. The methodology is well-motivated, with a clear focus on scalability and efficiency, and the experimental results support the claims. However, further analysis on generalization to other datasets and a more detailed comparison with alternative architectures would strengthen the work. Overall, the paper makes a meaningful contribution to vast vocabulary object detection.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper is well motivated. Since Vast Vocabulary Object Detection task have numeous classes, mining the sparse reward from positive categories is meaningful.\n2. This paper is well written and easy to follow.\n3. Experiments show its effectiveness, not only suporier in V3Det benchmarks, but also competitive in COCO dataset, comparable with strong baselines like Rank-DETR.\n\nWeakness:\n1. The Positive-to-negative gradient plot should be drawn in longer iterations.\n\nQUESTIONS:\nCould the proposed CQ-DINO handle category correlations in datasets with incomplete or ambiguous hierarchies?\n\nLIMITATIONS:\nVisualization and deeper analysis is missing.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates scaling object detectors to vast vocabularies. The authors identify and analyze two key issues with standard classification-based methods: \"positive gradient dilution,\" where positive samples are overwhelmed by the sheer number of negatives, and \"hard negative gradient dilution,\" where informative negative gradients are lost among a sea of easy ones. To address this, they propose CQ-DINO, a novel framework that reformulates classification as a contrastive task. The core of their method is an image-guided query selection module that dynamically selects a small subset of relevant \"category queries\" for each image. This pruning of the category space aims to rebalance gradients and perform implicit hard negative mining. The selected queries are then used in a DINO-like architecture to produce detections. The authors demonstrate state-of-the-art results on the vast-vocabulary V3Det benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.  The paper provides an insightful analysis of the gradient dilution problem, offering a clear empirical motivation for why existing methods struggle with vast vocabularies.\n2.  The main idea of using learnable category queries and an image-guided selection mechanism to dynamically prune the search space is clever. It directly targets the identified problem of an overwhelming number of negative classes.\n3. The experimental results on the challenging V3Det benchmark are strong, showing a clear improvement over previous methods, including a well-tuned DINO baseline. This demonstrates the empirical effectiveness of the proposed approach.\n\nWeakness:\n1. The framework is quite complex and relies on a two-stage training process: a pre-training stage for the selection module followed by a full-pipeline fine-tuning stage. As the authors acknowledge in their limitations, this can lead to suboptimal coordination between the stages and makes the model more difficult to train and deploy compared to a true end-to-end solution.\n2.  While the analysis of \"gradient dilution\" is thorough, the underlying issue is a severe case of the well-known class imbalance problem. Focal Loss was designed to address the dominance of easy negatives. The paper shows their method works better than tuning Focal Loss, but the problem framing itself is not entirely new.\n3. The paper heavily utilizes components from Grounding DINO for the feature enhancement and decoding stages. As such, the contribution can be seen less as a completely new detection paradigm and more as a sophisticated \"pre-filtering\" front-end for an existing powerful detector.\n\nQUESTIONS:\n1.  The category recall of 83.3% appears to be a bottleneck. Have you analyzed the characteristics of the ~17% of objects whose categories are missed by the selection module? Are they primarily rare categories, visually similar categories, or small objects? Understanding this failure mode is critical, as my evaluation of the method's robustness hinges on whether this ceiling can be raised significantly.\n2.  While CQ-DINO outperforms a tuned Focal Loss baseline by 4.0% AP, it is also a much more complex system. How can we be sure that the performance gain is due to a fundamentally better approach to gradient balancing, rather than a side effect of adding more complexity, parameters, and a separate supervision signal (the selection loss) to the overall training pipeline?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies gradient dilution issues in vast vocabulary object detection arising from numerous categories: (1) gradients from negative categories overwhelm those from positive categories; (2) within negative categories, gradients from easy samples dominate. To mitigate this, unlike standard classifiers employing static, image-irrelevant category embeddings, the proposed method utilizes image-adaptive category queries to perform image-level category selection prior to instance-level classification. \n\nSpecifically, these category queries aggregate image features via cross-attention and perform image-level classification to filter out absent categories. This significantly reduces the number of categories (e.g., from over 13,000 to 100) for subsequent instance-level classification, thereby alleviating the described gradient imbalance. To handle category hierarchies, the method leverages inherent tree structures to construct hierarchical category queries with an adaptive weighting mechanism. \n\nWhen integrated into Grounding DINO, the proposed approach achieves 53.0 AP on the V3Det dataset, outperforming prior methods. A slight performance gain is also observed on the COCO dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The writing is clear.\n2. The proposed method achieves 53.0 AP on the V3Det dataset, outperforming previous methods including DINO and PROVA by a significant margin. This demonstrates effectiveness for vast vocabulary object detection.\n\nWeaknesses:\n1. Lack of discussion on prior works: The core ideas of this paper—(1) modeling classification categories in an image-adaptive manner instead of using static, image-irrelevant embeddings, and (2) leveraging image-level classification prior to dense predictions—have been explored in existing works such as Query2Label [1], RankSeg [2], and CQL [3]. Query2Label pioneered the concept of category queries. Both RankSeg and CQL employ category queries for image-level classification to enhance dense prediction tasks (semantic segmentation and human-object interaction detection, respectively). RankSeg and CQL both utilize category queries for image-level classification to aid dense prediction tasks (semantic segmentation and human-object interaction detection). RankSeg retains only the top-k semantic categories, a strategy highly similar to this paper and shown to improve accuracy. CQL does not explicitly discard low-confidence categories but integrates image-level classification scores into instance-level scores, downweighting instances from low-confidence categories. However, these works are not discussed. I recommend adding a dedicated section on category query techniques in the related work.\n2. Incomplete baseline comparison: The proposed CQ-DINO is built upon the Grounding DINO framework, yet only DINO is used as the baseline. To isolate the contributions of the proposed method versus the Grounding DINO framework, comparative experiments are necessary.\n3. Insufficient validation of core claims:\n    * Prior works (e.g., RankSeg, CQL) demonstrate that image-level category selection improves inference-stage accuracy for dense prediction tasks by filtering or downweighting low-confidence categories. This paper, however, focus solely about its training-stage impact (mitigating gradient dilution) without addressing inference benefits.\n    * The gradient dilution issue may be largely alleviated by focal loss. As shown in Figure 2, the positive-to-negative gradient ratio is near 0 with CE loss but rises to ~0.5 with focal loss. While CQ-DINO achieves a ratio of ~1.5, the difference is marginal (several-fold rather than orders of magnitude) and potentially addressable through hyper-parameter tuning. Section 4.4 demonstrates that tuning hyper-parameters of focal loss effectively improves vanilla DINO’s performance. Though still below CQ-DINO, it remains unclear whether the performance gap stems from: (a) the Grounding DINO framework, (b) inference-stage category selection, or (c) mitigation of training-stage gradient dilution.\n    * I recommend adding ablations isolating contributions from: (a) Grounding DINO, (b) inference-stage category selection, and (c) training-stage gradient dilution mitigation. This is essential to validate the paper’s central claim.\n\n[1] Query2Label: A Simple Transformer Way to Multi-Label Classification\n\n[2] RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation\n\n[3] Category Query Learning for Human-Object Interaction Classification\n\nQUESTIONS:\nI assign a borderline rating due to two primary concerns:\n* Lack of discussion on prior works exploring category query techniques (detailed in Weakness 1),\n* Insufficient validation of the core claim regarding training-stage gradient dilution mitigation (detailed in Weakness 3).\n\nI am open to increasing my rating if the authors address these issues as suggested.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}