Title: GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: open vocabulary object detection

detection transformer

fine-grained object detection

vision-language models
Abstract: Fine-grained open-vocabulary object detection (FG-OVD) aims to detect novel object categories described by attribute-rich texts. While existing open-vocabulary detectors show promise at the base-category level, they underperform in fine-grained settings due to the semantic entanglement of subjects and attributes in pretrained vision-language model (VLM) embeddings -- leading to over-representation of attributes, mislocalization, and semantic drift in embedding space. We propose GUIDED, a decomposition framework specifically designed to address the semantic entanglement between subjects and attributes in fine-grained prompts. By separating object localization and fine-grained recognition into distinct pathways, GUIDED aligns each subtask with the module best suited for its respective roles. 
Specifically, given a fine-grained class name, we first use a language model to extract a coarse-grained subject and its descriptive attributes. 
Then the detector is guided solely by the subject embedding, ensuring stable localization unaffected by irrelevant or overrepresented attributes. To selectively retain helpful attributes, we introduce an attribute embedding fusion module that incorporates attribute information into detection queries in an attention-based manner. This mitigates over-representation while preserving discriminative power.
Finally, a region-level attribute discrimination module compares each detected region against full fine-grained class names using a refined vision-language model with a projection head for improved alignment.
Extensive experiments on FG-OVD and 3F-OVD benchmarks show that GUIDED achieves new state-of-the-art results, demonstrating the benefits of disentangled modeling and modular optimization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of fine-grained open-vocabulary object detection (FG-OVD), where existing methods struggle with semantic entanglement between object subjects and attributes in vision-language model (VLM) embeddings. The proposed GUIDED framework decomposes FG-OVD into three stages: subject identification using a large language model (LLM), coarse-grained object detection guided by subject embeddings, and fine-grained attribute discrimination via a refined VLM. By separating localization and attribute recognition, GUIDED mitigates attribute over-representation and semantic drift. Key innovations include an attribute embedding fusion module for selective attribute integration and a projection-based discrimination mechanism. Experiments on FG-OVD and 3F-OVD benchmarks show GUIDED outperforms state-of-the-art methods by up to 19.8% in mAP, demonstrating improved fine-grained detection capability.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The attribute embedding fusion and projection-based discrimination modules are plug-and-play, enhancing existing OVD frameworks without requiring major architectural changes.\n2. GUIDED effectively decouples FG-OVD into manageable subtasks, leveraging the strengths of LLMs for subject parsing, detection transformers for localization, and VLMs for attribute discrimination. This addresses the core issue of semantic entanglement.\n3. Experiments cover multiple benchmarks (FG-OVD, 3F-OVD), architecture variants (Grounding DINO, OWL-ViT, LaMI-DETR), and ablation studies validating each module's contribution.\n\n### Weaknesses\n1. The motivation is in contrast to previous methods that argue that the attributes will help the open-vocabulary object detection, such as [1].\n2. During inference, the proposed method needs an LLM to infer the subject name; it would significantly decrease the inference efficiency compared with previous OVD methods, i.e., the embedding for class names is precomputed for inference.\n3. Due to the usage of LLM, the proposed method also needs to be compared with the results of directly using the MLLM for FG-OVD.\n\n[1] OvarNet: Towards Open-vocabulary Object Attribute Recognition.\n\nQUESTIONS:\n1. It is not aligned with Figure 3(b) and Figure 2. For example, in Figure 3(b), there are two kinds of queries: a learnable query and an object query.\n2. What the mean of EDD in Tables 1 and 2?\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GUIDED, a method for fine-grained open-vocabulary object detection that addresses the challenge of semantic entanglement between object categories and their attributes. The proposed approach decomposes the task into three stages:  \n\n1. Subject Identification, where GPT-4o is used to extract a coarse object category and its associated attributes from fine-grained class names.  \n\n2. Coarse-Grained Object Detection, which uses only the subject embedding for object localization, with an attribute embedding fusion module to selectively integrate relevant attribute cues. \n\n3. Fine-Grained Attribute Discrimination, which re-scores detected regions using full fine-grained descriptions via refined CLIP text encoder.  \n\nThe model is trained in two stages, first on LVIS base categories, then fine-tuned on FG-OVD. The model is evaluated on the FG-OVD and 3F-OVD benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThe following are the strengths of the paper. \n\n1. The paper provides a reasonable rationale for separating fine-grained object detection into three stages, subject identification, coarse-grained object detection and fine-grained attribute discrimination. The observation that VLMs like CLIP perform better on base-level category names than on attribute-rich prompts is well-argued and aligns with known limitations of CLIP embeddings. \n\n2. The proposed attribute embedding fusion (AEF) module, which integrates attribute information into object queries via cross-attention, is conceptually simple but intuitive. It enables selective incorporation of useful attributes without much interfering with the subject signal during detection. \n\n\nThe following are the weaknesses of the paper. \n\n1. Key variables such as F_conv, F_en, etc. are not linked to the architecture block diagrams. The block diagram fails to capture the structure described in the equations, making it difficult to follow the implementation details. \n\n2. **Unfair comparison with LaMI-DETR:** The paper claims a +23.2% improvement over LaMI-DETR but does not clearly state whether LaMI-DETR was fine-tuned on FG-OVD or whether the same training protocol and dataset were used. Without matched experimental conditions, such comparisons are misleading. \n\n3. **No evaluation on standard open-vocabulary benchmarks:** Despite training on LVIS base categories, the paper skips any evaluation on LVIS novel categories (e.g. APr) or COCO base/novel splits. Even though the focus is fine-grained OVD, it is still important to report performance on broader OVD tasks to understand generalizability and potential trade-offs.\n\nQUESTIONS:\nIn the attribute embedding fusion module, query updates are computed using cross-attention where the keys are formed by subtracting the subject embedding from each attribute embedding. While the idea is that this highlights the attribute-specific signal, it’s not clear to me: \n\n- Why this subtraction should be meaningful or stable, especially across diverse classes or longer prompts. For example, subtracting the embedding of “car” from “a shiny red car with tinted windows” assumes that the resulting vector cleanly captures the difference (“shiny red with tinted windows”). But CLIP embeddings are not trained with this kind of linear structure in mind, and it's unclear whether such a subtraction generalizes across different sentence structures or domains. \n\n- Whether it introduces noise or ambiguity when attributes are semantically similar or visually correlated. For example, the difference between “a white ceramic mug” and “a white porcelain mug” may be minimal in CLIP space and subtracting the subject (“mug”) from each might produce almost the same or unstable vectors. This could affect attention-based fusion especially when the attributes are subtle or highly co-occurring. \n\nHas this subtraction-based attention been compared empirically with simpler alternatives like concatenation or additive fusion? \n\nClarifying this design choice and providing results on standard open-vocabulary settings like COCO and LVIS (especially given that the model is trained on LVIS base) would help in assessing generalization and could positively influence my rating.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes GUIDED, a decomposition framework for fine-grained open-vocabulary object detection (FG-OVD) that addresses semantic entanglement between subjects and attributes in vision-language model embeddings. The method decomposes FG-OVD into three stages: (1) subject identification using LLMs to extract coarse-grained subjects and attributes, (2) coarse-grained object detection guided by subject embeddings with an attribute embedding fusion module, and (3) fine-grained attribute discrimination using refined CLIP embeddings with a projection head. GUIDED achieves 23.2% mAP improvement over baseline LaMI-DETR on FG-OVD benchmark and demonstrates state-of-the-art performance across multiple datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Well-motivated: The paper clearly identifies and visualizes the semantic entanglement issue in existing FG-OVD methods through empirical analysis (Figure 1), showing how attribute over-representation leads to mislocation and semantic drift.\n2. Systematic approach: The three-stage pipeline (identification, detection, discrimination) is well-designed and aligns each subtask with appropriate model strengths - using detection transformers for localization and VLMs for fine-grained discrimination.\n3. Comprehensive experiment: Extensive experiments on FG-OVD and 3F-OVD benchmarks. The method shows consistent improvements across different base detectors. Achieves significant performance gains and establishes new state-of-the-art results on multiple benchmarks.\n\nWeaknesses:\n1. There's insufficient analysis of the attribute similarity score. Could you provide the result, if alpha in Eq. (5) is 0 or 1? Furthermore, have you tried other fusion strategies, e.g., weighted average?\n2. The method introduces additional complexity through LLM preprocessing, refined CLIP inference, and multi-stage processing, but computational cost analysis is insufficient. What is the complete computational overhead breakdown for GUIDED compared to direct fine-grained detection? Please include LLM preprocessing time, additional CLIP inference, and memory requirements.\n3. Failure cases: The paper briefly mentions limitations regarding attributes extending beyond detection boxes, but doesn't provide visualization analysis or scenarios where the decomposition strategy might not work well.\n4. Reproducibility concerns: The method relies on GPT-4o for subject identification, which is a closed-source model. This creates reproducibility issues and introduces potential non-transparent biases in the decomposition process that cannot be fully analyzed or controlled.\n\nQUESTIONS:\n1. How robust is the LLM-based subject identification? What happens when the LLM fails to correctly identify subjects or attributes? Have you analyzed error propagation from this stage?\n2. Hyperparameters (m_fine=50, m_coarse=50) appear to be fixed values without sensitivity analysis or justification for these specific choices.\n3. Given the reproducibility concerns with using proprietary GPT-4o, have you experimented with open-source alternatives like LLaMA or other accessible models? How does the choice of LLM affect the subject identification quality and overall performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes GUIDED, a method for fine-grained open-vocabulary detection (FG-OVD), a task aimed at detecting objects according to specific attribute-based text prompts. GUIDED decouples coarse-grained detection and fine-grained recognition and uses an LLM to identify the subject and attributes in the text prompt. Specifically, this framework is composed of \"Subject Identification,\" where an LLM is used to decompose the text prompt into the subject and attributes, \"Coarse-grained Object Detection,\" which combines a transformer-based object detector with attribute embedding fusion, and a \"Fine-grained Attribute Discrimination Module\" to identify region proposals from the detection stage that are aligned with the fine-grained attribute. GUIDED is evaluated on two FG-OVD datasets: FG-OVD and 3F-OVD. GUIDED achieves state of the art on these datasets. The authors report results applying GUIDED to the base models Grounding DINO, OWL-ViT, and LaMI-DETR, with the version based on LaMI-DETR performing the best overall.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed GUIDED method achieves state-of-the-art for fine-grained open-vocabulary detection\n- The experiments are extensive, covering two different FG-OVD datasets and three different base models\n- The authors present ablation studies testing the different components of the GUIDED framework\n\nWeaknesses\n- The key results are not presented in an easily accessible format. For example, the acronym \"EDD\" is not defined in the text even though it is included in many tables, and the text in Fig 1 (a) is difficult to read. The texts on the images in Fig 2 of the supplementary are also difficult to read.\n\nQUESTIONS:\n- What does \"EDD\" stand for in the tables? I assume it stands for GUIDED, but this needs to be clarified in the paper.\n- Why is there a star next to HA-FGOVD in Table 1?\n- What is the full inference time for a single example? This is important since the proposed framework includes an LLM, an extra (and potentially heavy) additional component compared to traditional detectors. \n\nIn light of the extensive experiments and good results, if the questions above can be clarified and integrated into the paper, then I would be willing to increase my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}