GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
Authors: Jiaming Li
Zhijia Liang
Weikai Chen
Lin Ma
Guanbin Li
Venue: NeurIPS 2025 poster ()
Keywords: open vocabulary object detection
detection transformer
fine-grained object detection
vision-language models
Primary area: deep_learning

Abstract: Fine-grained open-vocabulary object detection (FG-OVD) aims to detect novel object categories described by attribute-rich texts. While existing open-vocabulary detectors show promise at the base-category level, they underperform in fine-grained settings due to the semantic entanglement of subjects and attributes in pretrained vision-language model (VLM) embeddings -- leading to over-representation of attributes, mislocalization, and semantic drift in embedding space. We propose GUIDED, a decomposition framework specifically designed to address the semantic entanglement between subjects and attributes in fine-grained prompts. By separating object localization and fine-grained recognition into distinct pathways, GUIDED aligns each subtask with the module best suited for its respective roles. 
Specifically, given a fine-grained class name, we first use a language model to extract a coarse-grained subject and its descriptive attributes. 
Then the detector is guided solely by the subject embedding, ensuring stable localization unaffected by irrelevant or overrepresented attributes. To selectively retain helpful attributes, we introduce an attribute embedding fusion module that incorporates attribute information into detection queries in an attention-based manner. This mitigates over-representation while preserving discriminative power.
Finally, a region-level attribute discrimination module compares each detected region against full fine-grained class names using a refined vision-language model with a projection head for improved alignment.
Extensive experiments on FG-OVD and 3F-OVD benchmarks show that GUIDED achieves new state-of-the-art results, demonstrating the benefits of disentangled modeling and modular optimization.

GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulle Object Detection
Jiaming Li Zhijia Liang Weikai Chen Lin Ma² Guanbin Li 1,3,48
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China Meituan, Shenzhen, China
GGanggDog Province Key Laboratory of Information Security Technology, China "Research Institute, Sun Yat-sen University, Shenzhen, China
Abstract
Fine-grained open-vocabulary object detection (FG-OVD) aims to detect novel ob- ject categories described by attribute-rich texts. While existing open-vocabe veca-vocyyllayy de- tectors show promise at the base-category level, they underperform in fine-gres ime-graimed settings due the semantic entanglement of subjects and attributes in pretrained vision-language model (VLM) embeddings leading to over-representation attributes, mislocalization, and semantic drift in embedding space. We propose GUIDED, a decomposition framework specifically designed to address the se- mantic entanglement between subjects and attributes in fine-grained prompts. By separating object localization and fine-grained recognition into distinct pathways, GUIDED aligns each subtask with the module best suited for its respective roles. Specifically, given a extract surse-grained subject and its descriptive attributes. Then the detector guided solely by the subject embedding, ensuring stable localization unaffected by irrelevant overrepresented attributes. To selectively retain helpful attributes, we introduce an attribute embedding fusion module that incorporates attribute information into detection queries in an attention based manner. This mitigates over-representation while preserving discriminative power. Finally, a region-level attribute discrimination module compares each detected region against full fine- grained class names using refined vision- vision-language model with projection head for improved alignment. Extensive experiments on FG-OVD and 3F-VVD bench- marks show that GUIDED achieves new state-of-the-arr results, demonstrating the benefits of disentangled modeling and modular optimization. Our code will be released in httts:://gttiiei com hwt.1mijjm41111 /UIDDED
class name, we first use language model to
Introduction
Open-vocabulary object detection (OVD) offers greater ficxibility than traditional closed-set detection by allowing models recognize arbitrary categories specified by text prompts. This paradigm significantly improves scalability in real-world environments where new categories frequently emerge and manual annotation is costly. However, most existing OVD methods focus coarse-grained
"Equally-contribucee authors.
Work done during an internship at Meituan.
This paper solely reflects the author's personal research and is associated with the author's affiliated institution.
SCoresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fine
0.66
3.26
Detecciin Socz
Clasof Soore
(b)
(c)
Figure 1: (a) The t-SNE visualization of CLIP text embeddings on LVIS classes and the fine-grained classes. The figure shows that some CLIP embeddings of fine-grained variants positioned far apart. (b) The visualization of predictions of an OV detector with different class prompts(A A dog vs dog with a head). The detector focuses on the head instead of the dog, leading incorrect localization. (c) The mean classification scores and the mean IoU of the prediction box with the ground truth box of OWL-ViT (24) under coarse-grained class queries and fine- anee-griied class queries. The detector shows better performance on both classification and localization with coarse-grained queries than with fine-grained queries.
concepts (e.g. "dog", "cat") and fall short when dealing with more specific descriptions. Fine-grained detection (FG-OVD) addresses this limitation by enabling recognition of novel categories with detailed attributes (e.g. "a small brown dog"). This fine fine-grained capability is vital for applications requiring precise granular understanding, such as product retrieval, visual search, and autonomous systems. Nonetheless, the increased semantic complexity in FG-OVD introduces new challenges in aligning textual attributes with visual regions, making it critical yet under-explored problem in the open-vocabulary setting.
Existing FG-OVD methods typically rely on pretrained vision- laaiuunl models (VLMs), such as CLIP, by directly encoding the full fine-grained class name into single text embedding. However, this paradigm introduces two fundamental issues. First, due to the contrastive learning objective of VLMs, all tokens are treated equally, which leads to semantic entanglement between subjects and attributes. This often causes attribute over-ret reeerrepesenttion, where descriptive modifiers dominate the embedding and suppress the core semantics of the object category. As shown in Figure the query "a dog with a head" leads the model to focus only on the head, yielding mislocalized predictions, while "a dog" correctly grounds the entire object. Second, this entanglement also results in semantic drift in the embedding space. As illustrated in Figure [ a), fine-grained variants like "a dog" and "a dog with head" positioned far apart in CLIP's latent space, despite their visually overlapping concepts. This mismatch makes classification unreliable for fine- fine-grained open-set detection.
These issues stem from a common root: the use of a single text embedding to simultancously serve two objectives- - object localization and attribute recognition. Such coupling introduces semantic ambiguity, impeding the model's 's ability to specialize in either task. To address this, we propose GUIDED, a decomposition framework that disentangles FG-OVD into coarse-grained object detection and fine-grained attribute discrimination, allowing each to be handled by the model best suited for the subtask. This strategy motivated by empirical observations shown in Figure [c)): detectors achieve higher classification scores and more accurate localization when queried with coarse-grained categories (c.g., "dog") than with fine- mne-grained descriptions (c.g., "a black fluffy dog"). This indicates that object detectors are better suited for base-level semantics, while fine- teee-graned attribute recognition, which often requires subtle and localized reasoning, is more effectively handled by pretrained vision-language models.
To instantiate the design, GUIDED adopts three-stage pipeline that explicitly separates subject identification, object detection, and attribute discrimination. Given a fine-graines class prompt, a large language model first employed to extract the coarse-grained subject and its associated attributes, which are then encoded separately using lasion-language model. The subject embedding guides a grainee object detector localize candidate regions. To retain relevant attribute cues while avoiding over-representaatio,, an attribute embedding fusion module selectively integrates

---

attribute embeddings into the detector queries via attention. In the final stage, fine- aeee-grained attribute discrimination performed on the detected regions using region-text similarity. A lightweight projection head is applied to refine the text embeddings before comparison, enhancing the alignment between visual regions and fine-grained semantics. The final prediction score is computed by fusing the detector's s coarse confidence with the attribute similarity score, yielding more precise and interpretable fine-grained predictions.
Extensive experiments on FG-OVD benchmarks validate the effectiveness of our proposed GUIDED framework, which outperforms existing sate-of-the-arr methods by margin of 19.8%. Our main contributions are summarized as follows:
We propose GUIDED, a novel decomposition framework that decouples FG-OVD into coarse-grained object detection and fine-greet gnee-gained attribute discrimination, aligning each subtask with the strengths of detection transformers and pretrained vision-language models.
We design an attribute embedding fusion module that selectively integrates fine-grained attribute cues into detection queries, enhancing representation without overwhelming coarse category semantics.
We introduce a projection-base based attribute discrimination mechanism that refines text em- beddings and computes region-text similarity for accurate fine-graine classification over detected objects.
We establish new state-of-the-art results on FG-OVD benchmarks, demonstrating the effec- tiveness of task decomposition and modular optimization.
2
Related Work
Open vocabulary object detection Open-vocabulary object detection (OVD) has emerged
salient research direction, propelled by advancements in Vision-language models 25 18
13
and have
large-scale pretraining techniques. Notably, recent attempts [34 24 35
40
15
38
37
primarily focused on adapting the VLMs to the detection task by fine-tuning, Another line of 30 explores knowledge distillation to bridge the modality gap between distillation framework that aligns region-level features with the image encoder from CLIP 25]. thus enhancing cross-modal retrieval and detection accuracy. The recent works B3 27 36 integrate detection transformers in OVD to achieve further advanced capabilities. Grounding DINO 20] integrates the DINO with language models, enabling zero-shot object detection through text prompts by aligning visual regions with semantic embeddings. LAMI-DETR (5) introduces the language model instructions to generate the relationships between visual concepts for detection transformers. Despite these advances, existing methods exhibit limited performance on the detection of fine-grained classes(FG-OVD) with specific attributes due to the lack of fine-grad graingr text-region annotations. Our approach addresses this limitation through dccomposed FG-OVD into coarse-grained object detection with transformer detectors and fine-grained attribute identification with VLMs to take the inherent advantage of each model.
work [14 32
16 39
detection and language understanding. ViLD (8) introduces a dision-languag
Fine-grained Open Vocabulary Object Detection The concept of fine-grained open- vocabulayy detection (FG-OVD) (2) extends conventional OVD by introducing sttribute-conditin class def- initions (e.g., color, material, shape) that require detectors to recognize novel classes like "dark brown wooden lamp" versus "gray metal lamp".. Current approaches
28 predominantly address this challenge through text embedding refinement. SHiNe [9) proposes to update the classifiers in OVD by the hierarchy-awan sentences. However, it fails to capture the attribute information in its embedding construction process. HA-FGOVD[23] proposes universal approach to generate attribute- attribute-highlight highlighted text embeddings by masking the attention map of VLMs to obtain the attribute- specific features, while Bianchi et [l.[I] propose to fine-tune additional linear projection layer to enhance the fine-grained capability of CLIP text embeddings. However, these methods suffer from the attribute reerreepeennnnnnonn and semantic entanglement. Besides, the performance improvement led by embedding augmentation is limited by the detector's 's fine- grained capability. To address this, GUIDED addresses these limitations by decoupling attribute identification from detection pipelines, which integrates the strong discrimination capability of VLMs.
3

---

Identification
Discrimination
dark pink pilto fabiis
Appleree ordbes
A Adiiplicplior
CUP Text
aragnn
LLM
Desc Braics
CLIF toage
fite
procpe
Taxi
gnited
giore
Pusing
Detection
Boxx roj
inmae
Dooader
Subject urbeakding
Ene eno
cubeddins
Figure An overview of the proposed GUIDED framework. GUIDED adopts a three-stage pipeline. which consists of subject identification, coarse coarse-grained object detection, and fine-grained attribute discrimination. subject identification, GUIDED employs an LLM to extract the coarse-grained subject and its attribute embeddings. For detection, coarse-graing subject embeddings are adopted as queries localize the candidate regions with coarse confidence. An attribute embedding fusion module selectively integrates attribute embeddings into queries. In the discrimination stage, GUIDED estimates the fine-grained score each detected region with full fine-grained class names using refined CLIP with projection head. The final score is obtained from the weighted multiplication of the detector's coarse confidence and the attribute similarity scores.
Approach
In this paper, propose GUIDED, framework specially designed for FG-OVD that aims to detect objects from both base and novel fine- mne-grained categories with detailed attributes. Our proposed GUIDED framework disentangles the FG-OVD into different tasks, including subject identification, coarse-grained object detection, and fine-grained attribute discrimination. During the subject identifi- cation process, GUIDED introduces large language model to identify the coarse-grained subject and fine-grained descriptions to generate the corresponding embeddings. the coarse-gras coarse-grained object detection stage, detection transformer is employed to localize objects based subject embeddings while dynamically fusing attribute-speeific semantics through an attribute embedding fusion module. Subsequently, fine-grained attribute discrimination adopts the VLMs estimate the fine-grained scores on the detected proposal with the fine-grained text of each class. The overview of the GUIDED framework is shown Figure
3.1 Subject Identification
The proposed GUIDED framework addresses FG-OVD through a hierarchical decomposition strategy that systematically separates coarse-grained object detection and fine-grained attribute discrimination. This approach begins with semantic parsing of fine-grained class names using frozen large language model. Given a fine- aine-grained class name, we first identify its subject as coarse- coarse-grained class and the associated attributes by prompting the existing large language models(e.g. GPT4-o ( These prompts are shown in Figure 3 (a). For each class, the identified subjects and associated attributes are fed into the frozen CLIP text encoder to obtain the coarse-graie -arrse-graieed text embeddings ) and the attribute embeddings ( [t:]] ji=1 1r- 1- Here n the number of classes and n, is the number of attribute embeddings for the j-th class. Note that the coarse-graine grained Subject Identification done before the training or inference process.
The coarse- grained subject identification process can also be applied to identify the super class of subclass, such as identifying the dog from the Siberian Husky. In this case, the associated attributes can be extended from the descriptions of the complex classes by LLMs.

---

LLM BYSTEM: Your task to idmalty plasical ligut
ofjecition.
System Pronpt
supuoese saat single word coscin none phrae. De taclade ane descipetiv adjcthiss geg. coles, runerials cini other continall adtenation. of triaatipe polenialohjech mnniisad,, sdlect spocifa, langhtle anthigalty
Laanllle Quay
Ofject Qeery
Livir Inijation
descrption,
Anaeriez Weide
iegutt dark metal spoon.
Iromeed Exampl
Flaud Query Uiithddiing
Ieput collutar telgrncee with light parple plantic band oover
Sparaction
outpott onttu rotaphines.
Query
Iepatt: ralcrwavs oven with anyy glass coienie
Atswve
satpart inkrnnoee oves
(a)
(b)
Figure 3: (a) Illustration of the prompt for subject identification. The prompt for extracting the associated attributes shown in our supplementary document. (b) The architecture of the attribute- fused attention layer.
3.2
Coarse-grained Object Detection
After the identification stage, we perform coarse- soase-grained object detection(CGOD) by detection transformer. To retain relevant attribute cues while avoiding over-representation, we also propose an attribute embedding fusion module to exploit the helpful fine-grained attributes in an attention-based manner.
Specifically, we adopt the LAMI-DETR 5 as our baseline transformer, which is built upon the DINO detection framework. Concretely, input image encoded to frozen ConvNext [2 backbone from the pre-trained CLIP image encoder to output the spatial feature map F Foone- The spatial feature map is then input to a learnable transformer encoder for refinement. The refined encoder feature is denoted as Fexe-
Attribute Embedding Fusion.
After obtaining the refined feature map, we introduce Attribute Embedding Fusion module to fuse the subject embeddings and attribute embeddings into input object queries of the transformer decoder. Specifically, follow LaMI-DETR to select the top k pixels in the encoder feature F Fene as object queries based on their classification logits. This process is formulated as,
{ K (max ((.) Fene))), (q f=1
(I)
Here CLS( CL,({t;) (i.......)) is the classifier which inputs the subject embeddings as frozen layer weights and outputs the logits corresponding the classes. With the TopK selection, each selected query matched with a class with the largest classification scores. Then the query embeddings are fused with the corresponding subject embedding and attribute embeddings through cross-attention layer, which is formulated as follows,
qf = q: + ATTN(q' (t.- (t (t? (t) (t))))))))))))
(2)
The union of subject embedding t; and fine-grained attribute embeddings {t. ) for the matched class is used as the value states of the cross-attention layer. We adopt the difference between the attribute embeddings and the corresponding subject embedding as key states to highlight the attribute information. q is learnable query. The architecture of attribute-fusee attention is shown in Figure (b.. This attention layer dynamically integrates the subject embedding and attribute embeddings into queries, enabling the detector exploit helpful attributes for detection.
After that, the fused query embeddings {qf} and the encoder feature Fene are input to DINO decoder to output the prediction embedding {pa H We adopt the aforementioned classifier CLS(( ) to output the detector's coarse confidence square R(n) for the j-th prediction with a sigmoid function,
coarse
=ccoars
CL
coarse Sigmoid (gears)).
(3)
where Mecurse a scaling factor. The bounding box locations {b.) are output from the prediction embedding with box regression process.
5

---

3.3 Fine-grained Attribute Discrimination
The bounding boxes output by the detection transformer are used as candidate regions of fine- tineggriieed classes. To discriminate whether the fine-gras -eeiggd attribute is aligned with the candidate bounding boxes, we introduce fine-grained attribute discrimination lightweight linear projection layer after the frozen CLIPP text encoder to refine the text embeddings for better representation on fine-grained attributes. This lightweight linear projection layer bridges the gap between general-purposs purpose pre-trained representations and task-specific attribute semantics, enhancing discriminative capability for subtle attribute distinctions. Then the full fine-grained class names are fed into the refined CLIP text encoder T' to obtain the corresponding fine-grained class embeddings [ Subsequently, the image embeddings of each candidate are generated from the spatial feature map F cony from the frozen ConvNext backbone by performing a pooling operation on the features of the box location. Finally, we estimate an attribute similarity score for each candidate box, noted as fine, The attribute similarity score stoe R(()) of the j-th prediction is estimated from,
module. To this end, we first add
Jime Mine cos (Pooliing comn [by)), (t.)
) S
The cos(-) is the cosine similarity function, and minne scaling factor. Since the spatial feature map F Feav is previously calculated in the coarse- graiind detection module, the fine-grained attribute discrimination only slightly increases the inference time In addition CLIP, other VLMs can also be applied to calculate the attribute similarity score, as shown in our experiments. Nevertheless, the application of other VLMs always substantially increases the inference time. The final scores of the j-th bounding box prediction for the fine fine-grained detection are set as,
fnnal
(s (s (a) (sfine
fine (1-0)
(5)
where is hyperparameter.
3.4 Training Objective
We introduce a two-stage training pipeline for the GUIDED. During training, we first follow the LaMI-DETR(S [S] training pipeline pretrain the detection transformer on the base classes of the traditional open vocabulary dataset without applying the attribute similarity score. In this stage, the class names of the base classes extended by large language model construct the attribute embeddings for the attribute embedding fusion module. In the second stage, we introduce the FG- OVD dataset train the detection transformer and the refined CLIP. The fine-grained class names in the FG-OVD dataset are decomposed into coarse-grained subjects and attributes by the subject identification. For the detection transformer, we generate the ground truth with the labels of subjects to supervise the detection transformer. Furthermore, we introduce an additional binary loss for the samples on the original fine-grain fine-grained classes,
Line - - St, log (snjane (10) (sfire) (1=a))
(6)
Here gt, is the one-hot class label of the j-th sample. This loss enhances the discriminative capability for attribute distinctions of embeddings from the refined CLIP for improved alignment.
4 Experiments
4.1 Dataset
Our method is evaluated on the two benchmark fine-grained open vocabulary object detection datasets, FG-OVD and 3FOVD. Additionally, we perform training on the LVIS dataset.
FG-OVD. Fine-grained open vocabulary detection (FG-OVD) dataset [2] is an evaluation task for comprehensively evaluating the fine-grained discrimination capabilities of models. Each an- notation in FG-OVD is paired with a positive caption and up to ten hard negatives generated by substituting attribute words while preserving sentence structure. The data are partitioned into four subsets (Colour, Material,
difficulty splits (Trivial, Easy, Medium, Hard) and four

---

Pattern, and Transpareney). We follow the official benchmarks subset splits and report mean average precision[[7] (mAP) averaged over all eight tracks.
3F-OVD. The recently released 3F-OVD 21 benchmark provides a more demanding test-bed for fine-grained open-vocabulary detection under long-caption queries, which assigns single, sentence-length description to every class, and re-uses that description across all images that contain the class. The benchmark comprises two distinct domains: vehicles (NEU-17IK-C) with 598 fine- grained classes, and retail products (NEU-17 HEEU-17I--PP K-RP) with 121 fine-grained classes, Consistent with the benchmark authors' configuration, we report mAP across both domains.
The LVIS dataset is a long-tailed object detection dataset with. Following the open vocabulary setting in LaMI-DETR(S] 866 common and frequent categories in the LVIS dataset are set base classes, while the remaining 335 rare categories are set as novel classes. We mainly use the base classes in LVTS for training.
LVIS.
4.2 Implemental Details
We mainly adopt the LaMI-DETR(S] as our codebase. The ConvNext backbone of the detec- tion transformer is initialized from Mat-Nee--Lagg---220000 (2 in OpenCLIP([O]. We follow Dio [20] to retain the top-k = 900 tokens ranked by coarse-grained classification logits. In the two-stage training pipeline, we first pre-train the base-class subset of the LVIS for 85200 iterations; we then fine-tune for a further 2000 iterations on the FG-OVD training set. For 3FOVD, extract all the available captions' subjects using LLM and process all the classes together. More details are presented in our supplementary document. At the score ensemble process, the a is set to 0.6. mine and Mecarse is set to 100. We leave other parameerrseeterss the same in LaMI-DETR.
4.3 Experimental Results
Comparison on FG-OVD dataset. We compare GUIDED against the existing OVD methods on FG- OVD datasets, including OWL-ViT 24. Detic(41), Vi Grounding DINO 20). CORA (33, and OV-DINO 29]. Furthermore, we apply GUIDED to three distinct architecture- OVD models, Grounding DINO, OWL-VIT, and LaMI-DETR[]) Note that the attribute embedding fusion module only applied to the DETR- based framework, LaMI-DETR. The results are shown in Table [ The mAP performance of GUIDED significantly surpasses that of other OVD methods. Specifically, GUIDED achieves a 23.2% mAP improvement over our baseline method LaMI-DETR, highlighting the effectiveness of leveraging PVLMs for fine-grained attribute discrimination in GUIDED, While methods like Grounding DINO and OV-DINO benefit from large-scale pretraining on coarse-grained datasets (Object365(26). GoldG([I). they exhibit limited capability in distinguishing fine-grained categories. As shown in the table, GUIDED is capable enhancing the performance of these methods on the FG-OVD. Furthermore, our methods defeat existing FG-OVD methods by large margin. Specifically, HA-FGOVD[23) only slightly improves the performance since it only modifies the input text embeddings. In contrast, GUIDED boosts the fine-grained detection capability of OVD methods, underscoring the generalization of our methods.
Comparison on 3FOVD dataset. We conduct evaluation of our proposed GUIDED on the 3FOVD dataset Table 2 Compared with FG-OVD, 3FOVD a more complex task since the class names in 3FOVD corresponding captions. Note that we do not perform training on 3FOVD but only transfer the model trained LVIS and FG-OVD to conduct the evaluation. In 3FOVD, extract the superclasses of the fine-grained class names as subjects for coarse- coarse-grained object detection and utilize the caption data as the fine-graine fine-grained full names in fine-grained attribute discrimination. The results show our method defeats other methods by clear margins. Compared with LaMI-DETR, our method demonstrates a clear improvement on both subsets, underscoring the effectiveness of GUIDED on the detection of complicated fine-grained classes.
complicated proper nouns(e.g. Drink Coca-Cola, Car Porsche-macan) with
4.4 Ablation Study and Analysis
Ablation of key factors in the GUIDED framework. We conducted an ablation study to assess the effectiveness of each key factor in our proposed GUIDED framework in Table For comparison, train the LaMI-DETR on FG-OVD with different training strategies, including training from
7

---

Table 1: mAP evaluation results FG-OVD benchmark (%). The performance in 'Average' is the average of performance over the sub-datasets. "Trans.' denotes the performance on the Transparency subset. 'Finetune' denotes finetune the LaMI-DETR FG-OVD training set. Here denotes the results are from our reproduction.
Detector
Hard
Medium
Easy
Trivial
Color
Material
Pattern
Trans,
Average
26.2
39.8
38.4
53.9
45.3
37.3
26.6
34.1
37.7
OWLv2(B/16)
25.3
38.5
40.0
52.9
45.1
33.5
19.2
28.5
35.4
OWLx2(L/14)
25.4
41.2
42.8
53.3
36.9
23.3
12.2
37.3
Detic
11.5
18.6
18.6
69.7
21.5
38.8
30.1
24.6
29.3
VILD
22.1
36.1
39.9
56.6
43.2
34.9
24.5
30.1
35.9
CORA
13.8
20.0
20.4
35.1
25.0
19.3
22.0
27.9
22.9
OV-DINO
18.6
28.4
25.0
54.3
35.6
30.0
21.0
24.2
29.6
Grounding DINO
17.0
28.4
31.0
62.5
41.4
30.3
31.0
26.2
33.5
HA-FGOVD
19.2
32.3
34.0
62.2
41.5
33.0
32.1
29.2
35.4 (+1.9)
+ GUIDED
35.1
49.3
52.8
57.1
49.7
57.0
26.4
39.6
45.9 (+12.4)
DWL-VIT(L/14)
26.6
39.8
44.5
67.0
44.0
45.0
36.2
29.2
41.5
HA-FGOVD
31.4
46.0
50.7
67.2
48.4
48.5
38.0
32.7
45.4 (+4.3)
+ GUIDED
46.8
59.4
64.1
66.2
60.4
58.9
44.7
54.5
56.9 (+15.4)
LaMI-DETR
29.2
40.6
42.9
63.5
49.5
39.2
34.6
46.2
43.2
Finetune
39.5
50.7
54.2
66.0
51.9
53.7
42.1
49.1
50.9 (+7.7)
HA-FGOVD*
33.5
45.9
47.5
52.7
42.6
36.9
50.1
46.6 (+3.4)
+ GUIDED
57.5
69.5
73.3
72.6
64.8
68.5
62.0
63.4
66.4 (+23.2)
Table 2: mAP evaluation results the 3FOVD bench- mar.(%).
Table 3: Ablation with different VLMs applied in FGOD. 'Time' denotes the aver- aged inference time(ms) for each image.
Method
NEU-17IK-C
NEU-17IK-RE
VLM
mAP
Time
Detic
6.6 x 10
2.2 x 10
Vild
3.8 x 10
1.1 x 10
CLIP (T)
46.9
210.3
GroundingDino
1.3 x 10-3
7.6 x 10
LLaVA-1.6
51.2
34718.0
Refined CLIP (T')
60.8
212.1
LaMI-DETR
9.0 x 10-4
2.3 x 10
+ GUIDED
7.2 x 10
2.7 x 10
scratch and fine-tuning. As shown in the table, finetuning LaMI-DETR with FG-OVD performs much better than training from scratch, showing the significance of the first-stage training on LVIS. With GUIDED, the performance improves from 50.9% 62.4%, underscoring the superiority of GUIDED training strategies. We also tease apart the key modules in GUIDED to conduct the ablations. Integrating the attribute embedding fusion(AEF) in GUIDED leads to a performance gain of 4.0%, validating the capability of AEF to selectively integrate fine-grained to improve the capability of the detector. 'GUIDED w/o CGOD' denotes that we directly adopt the full embeddings of fine-gres ine-gnained classes in the detector to achieve detection of fine- fine-graine grained classes instead of coarse- grained subjects. Performing coarse coarse-grained object detection improves the mAP by 9.4%, showing the effectiveness of task decomposition idea in GUIDED, When removing the projection layer in fine- grained attribute discrimination, the performance decreases by 2.3%, which validates the importance of the projection layer on text embedding generation to represent the fine-grained attribute.
Robustness of LLMS on subject identification. To evaluate the robustness of LLMs on subject identification, we manually annotate the subjects from 300 fine-grained classes and assess the accuracy of subject identification with different LLMs, including LLaMA-3.1 1-BB 7.. LLMMA-3.3-700 and
GPT-4o. As summarized in Table 5] failure cases are categorized into two types: (1) Hallucination toward in- co-tontex samples, where the LLM generates subjects irrelevant to the input text; (2) Other errors, such as identifying an attribute instead of an object. Overall, all three LLMs achieve high correctness rates, demonstrating robust performance across diverse architectures. The results confirm the high robustness of this stage. Crucially, they show that while the smaller LLaMA-3.1 1-88 model prone to hallucination errors that always propagate (8/8), this critical failure mode is completely climinated by larger open-source models. Both LLaMA-3.3-705 and GPT-4o exhibit near- near-perfect

---

Table 4: The ablation of key factors in the GUIDED framework the FG-OVD dataset. The results are shown mAP (%). 'Baseline' denotes the LaMI-DETR. 'AF"' denotes the attribute embedding fusion module. 'CGOD' denotes the coarse coarse-grained object detection.
Method
Hard
Medium
Easy
Trivial
Color
Material
Pattern
Transp.
Average
Baseline
29.2
40.6
42,9
63.5
49.5
39.2
34.6
46.2
43.2
Train from scratch
31.7
39.6
43.4
42.6
32.4
33.0
30.8
20.4
34.2
Finetune
39.5
50.7
54.2
66.0
51.9
53.7
42.1
49.1
50.9
GUIDED w/o AEF
53.0
65.2
69.5
72.0
64.1
63.3
49.0
63.4
62.4
GUIDED w/o CGOD
48.6
59.7
64.2
64.3
58.9
57.8
45.6
56.6
57.0
GUIDED w/o projection
57.5
67.8
70.2
70.8
64.0
66.3
55.4
60.8
64.1
GUIDED
57.5
69.5
73.3
72.6
64.8
68.5
62.0
63.4
66.4
performance, with their rare errors being minor and not always affecting the final detection. This demonstrates that our method's success not tied to a specific proprietary model and is robust when using of open-source alternatives.
Furthermore, we also report the detection performance with open-source LLMs as alternatives to GPT-40. As quantified in Table 6 the mAP of GUIDED drops by merely 0.5% with -LaMA-3.1-81 and improves by 0.1% % with LLaMA-3.3-70B. This demonstrates that GUIDED performance is not dependent on specific proprietary model. We will provide more results with LLaMA-3. 1-----33-8BB and LLaMA-3.3-70B for other OVD models and other datasets in our revised paper for reproducibility.
Table 5: The number of failure cases in subject de- tection of 300 samples in the FG-OVD dataset with different LLMs. The notation 'xx / denotes that there are y failure cases, of which x lead to detection errors.
Table 6: The comparison of mAP(%) evaluation results on the FG-OVD bench- mark with different LLMs in subject identification.
LLM
Hallucination
Others
Total
LLM
mAP
GPT4-0
0/0
1/2
1/2
GPT4-0
66.4
LLaMA-3.1-8B
8/8
1/1
9/9
LLaMA-3.1-81
65.9
LLaMA-3.3-70B
0/0
1/1
1/1
LLaMA-3.3-70B
66.5
Analysis of text embeddings applied in CGOD and FGAD. We also conduct an ablation study about the text embeddings applied in CGOD and FGAD, which is illustrated in the table 7 Applying the refined text encoder T' with the lightweight projection layer instead of the original CL.IP text encoder improves the mAP by 2.3% in FGAD but leads to a performance degradation of 5.8% in CGOD. This shows that the refined text encoder enhances the fine-grained discrimination capability in FGAD while exacerbating overfitting on the base classes in CGOD. GUIDED only use the fine- grained text encoder in FGAD, achieving optimal solution. Furthermore, we observe that using the full names the fine-gran fine-grained classes in object detectors results in a significant performance drop, validating the necessity of task decomposition for the FG-OVD task.
Table 7: The ablation of generated text embeddings of fine-graine line-grained classes. "TE' denotes the text encoder used for embedding generation. 'Coarse' and 'Full' denote the coarse-grained subject and full fine-grained class name used for generation, respectively.
514
CGOD
FGAD
mAP
20
TB
Text
TE
Text
Full
T
Full
53.5
1207
Scar
T
Full
T
Full
49.8
Table 8: The mean classification scores and the mean IoU of the prediction box with the ground truth box of the LaMI- DETR with and without GUIDED.
T
Full
T
Full
57.0
Coarse
T
Full
64.1
T
Coarse
T
Full
60.3
T Coarse T
Full
66.4

---

More ablations on FGAD. Our FGAD can be easily integrated with existing PVLMs with different structures. Specifically, we apply ValaV--..lllaavvl..66o.... (otdlaaaawy..-mmttral-bbbl- Ii without training to estimate the attribute similarity score by prompting LLaVA with "Does this image match the attributes described in the following caption? If so, output yes, if not, output no" on the coarse-grained box regions of images. The attribute similarity scores are obtained from the probability of generating "yes" tokens. For fair comparison, we directly apply different PVLMs in FGAD with the detector after the first stage of training on LVIS dataset. As presented in Table 3 LLaVA achieves higher score than the pretrained CLIP but lower scores than the refined CLIP, showing the generalization of our GUIDED framework in integrating different PVLMs in FGAD. Although LLa achieves encouraging performance, the inference speed of LLaVA is much lower than that of CLIP.
Inference time analysis. To provide an assessment of the computational overhead in GUIDED, we report the inference time of detecting one class in an image using locally deployable L.LMs: LLaMA-3.1 1-8B and LLaMA-33 1-11A--33-7000 [] As shown in the T the inference time increase in GUIDED primarily stems from L.LM-based subject identification, while AEF and CLIP design in FGAD contribute minimally to latency. This represents trade-off between our method's enhanced semantic understanding and computational cost. Nevertheless, the LLM-based subject identification is performed once per class name, not per image. For any given dataset application scenario, the set of fine-grat grained classes is fixed. Therefore, the parsing results can be pre-computed and cached offline, imposing additional LLM-related latency. For scenarios requiring on on-the-fly parsing of new class names, the latency can indeed be a factor. This can be alleviated by employing more lightweight LLMs or batch processing multiple subject identification tasks in one chat.
Table
The comparison of inference time(ms) between Baseline(LaMI-DEETR and GUIDED for detecting class in image. We also report the average mAP((%) of each method in FGOVD.
Method
LLM
Detector
CLIP
Overall
mAP
Baseline
198.5
13.2
211.7
43.2
GUIDED with ALLMA-3.11-B
72.1
198.8
13.3
284.2
65.9
GUIDED with LLaMA-3.3-70B
193.8
198.8
13.3
405.9
66.5
More analysis of GUIDED. Furthermore, we present the mean classification scores and the mean loU of the prediction box with the ground truth box of the LaMI-DETR with and without GUIDED in Figure g The results reveal that our GUIDED enhances both the capability of localization and confidence with the subject embedding and the attribute embedding fuse module, demonstrating the superiority of our method.
5 Limitations and Conclusions
Limitations. While GUIDED achieves strong performance on isolated fine-grained object recognition, the attribute discrimination operates on features within coarse-level detection boxes. When relevant attributes extend beyond the localized regions, performance may degrade. This could be mitigated by using expanded region proposals or incorporating context-aware reasoning beyond bounding boxes.
Conclusions. In this work, we present GUIDED, a decomposition framework for fine fine-grained open- vocabulary object detection. By explicitly decoupling object localization and fine-grained attribute discrimination, GUIDED addresses the core challenge of semantic entanglement in rision-language embeddings. Through task-specific modeling and selective attribute integration, our approach lever- ages the strengths of both detection transformers and pretrained vision-language models. Extensive experiments demonstrate that GUIDED achieves state-of-the-ar performance across multiple FG- OVD benchmarks, highlighting the effectiveness of task decomposition for fine-grained visual understanding under open- vocabulary settings.
6 Acknowledgments
This work is supported in part by the National Key R&D Program of China (2024YFB3908503, 2024YFB3908500), in part by the National Natural Science Foundation of China (62322608), in part by the Shenzhen Science and Technology Program (NO. JCYJ2022053014122 1024) and in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 151 151151025555
10

---

References
[1] Lorenzo Bianchi, Fabio Carrara, Nicola Messina, and Fabrizio Falchi. Is clip the main roadblock for fine-grained open-world perception? In 2024 International Conference on Content-Basee Multimedia Indexing (CBMI), pages 1-8, IEEE, 2024.
[2] Lorenzo Bianchi, Fabio Carrara, Nicola Messina, Claudio Gennaro, and Fabrizio Falchi. The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22520- 22529, 2024.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe--ud vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[4] Tianheng Cheng, Lin Song. Yixiao Ge. Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time ooca-vocy object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901-16911, 2024.
[5] Penghui Du, Yu Wang. Yifan Sun, Luting Wang. Yue Liao, Gang Zhang, Errui Ding, Yan Wang. Jingdong Wang, and Si Liu. Lami-detr: Open-vocabune vocabulary detection with language model instruction. Proceedings of the European conference on computer vision (ECCV), 2024.
[6] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yuc Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084-14093, 2022.
[7] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[8] Xiuye Gu. Tsung-Yo Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.139) 2021.
[9] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint 21X1v:2410.21276 2024.
[10] Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021.
[11] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr modulated detection for end-to-end multi modal understanding. In Proceedings of the IEEE/CVF international conference computer vision, pages 1780 1790. 2021.
[12] Jooycon Kim, Eulrang Cho, Schyung Kim, and Hyunwoo Kim. Retricval- open- open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17427-17436, 2024.
[13] Jiaming Li, Jiacheng Zhang, Zequn Jic, Lin Ma, and Guanbin Li. Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding. arXiv preprint arXiv:2501.0/9 2025.
[14] Jiaming Li, Jiacheng Zhang, Jichang Li, Ge Li, Si Liu, Liang Lin, and Guanbin Li. Leamnnng background prompts to discover implicit knowledge for open vocabulary object detection. In Proceedings of the IEED/CVF Conference Computer Vision and Puttern Recognition, pages 16678-1668) 2024.
[15] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965- 10975, 2022.
[16] Wuyang Li, Xinyu Liu, Jiayi Ma, and Yixuan Yuan. Cliff: Continual latent diffusion for open- vocabuacyy object detection. In European Conference on Computer Vision, pages 255- 273.. Springer. 2024.
[17] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part V 13, pages 740 755. Springer, 2014.

---

[18] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[19] Mingxuan Liu, Tyler L Hayes, Elisa Ricci, Gabriela Csurka, and Riccardo Volpi. Shine: nexus for vocabbacayllary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16634-16644 2024.
Semantic hierarchy
[20] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pro-training for open-set objeet detection, arXiv preprint arXiv:230.00549, 2023.
[21] Ying Liu, Yijing Hua, Haojiang Chai, Yanbo Wang, and TengQi Ye. Fine-grained open- open-vocabulary object detection with fined fined-grained prompts: Task, dataset and benchmark. arXiv preprint arXiv:2503.1486. 2025.
[22] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976 1777--1190 11986, 2022.
[23] Yuqi Ma. Mengyin Liu, Chao Zhu, and Xu-Cheng Yin. Ha-fgovd: Highlighting fine- grained attributes via explicit linear composition for open-vocahulary object detection. IEFE Transactions on Multimedia, 2025.
[24] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Amnb, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection. In European Conference on Computer Vision, pages 728- 755.755. Springer, 2022.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[26] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430-8439, 2019.
[27] Cheng Shi and Sibei Yang. Edadet: Open-vocahulary object detection using early dense alignment. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15724-15734, 2023.
[28] Haicheng Wang. Chen Ju, Weixiong Lin, Shuai Xiao, Mengting Chen, Yixuan Huang, Chang Liu, Mingshuai Yao, Jinsong Lan, Ying Chen, et al. Advancing myopia to holism: Fully contrastive language- image pre-training. arXiv preprint 2rXiv:2412.00444) 2024.
[29] Hao Wang, Pengzhen Ren, Zequn Jic, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma, Dongmei Jiang, Yaowei Wang. Xiangyuan Lan, et al. Ov-dino: Unified apen-vocabulary detection with language-aware selective fusion. arXiv preprint vrXiv:2407.07844 2024.
[30] Kuo Wang, Lechao Cheng, Weikai Chen, Pingping Zhang, Liang Lin, Fan Zhou, and Guanbin Li. Mar- velovd: Marrying object recognition and vision-language models for robust open-vocabulary object detection. In European Conference on Computer Vision, pages 106- 122. Springer, 2024.
[31] Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open- vocablacyy object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11186-11196, 2023.
[32] Size Wu, Wenwei Zhang, Sheng Jin Wentao Liu, and Chen Change Loy. Aligning bag of regions for open- vpcabuocybulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15254 15264, 2023.
[33] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7031-7040, 2023.
[34] Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open- vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23497-23500 2023.
12

---

[35] Lewei Yao, Renjie Pi. Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detelipv3: Towards versatile generative open- vee--vocabllary object detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27391-27401. 2024.
[36] Yuhang Zang. Wei Li. Kaiyang Zhou, Chen Huang, and Chen Change Loy. Ooca-vocabulary detr with conditional matching. European Conference on Computer Vision, pages 106-122. Springer, 2022.
[37] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of IEEE/CVF Conference on Computer Vision Pattern Recognition, pages 14393-14402, 2021.
[38] Shiyu Zhao. Samuel Schulter. Long Zhao. Zhixing Zhang, Yumin Suh, Manmohan Chandraker, Dimitris Metaxas, et al. Taming self-training for open-vocabulary object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13938-13944 2024.
[39] Xiaowei Zhao, Xianglong Liu, Duorui Wang. Yajun Gao, and Zhide Liu. Ceene-adaptive and region-aware multi-modal prompt for open vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16741-16750, 2024.
[40] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chumyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li. al. Regionclip: Region bason language- image pretraining, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793- 16803, 2022.
[41] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European Conference on Computer Vision, pages 350-368. Springer, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We carefully rectify the abstract and introduction to ensure the claims accu- rately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have created a separate "Limitations" section in our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

The paper focuses methodological innovations and empirical validation without presentngg formal theoretical results such as theorems or mathematical proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We will open-source the code after the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: We will open-source the code after the paper is accepted, while we have provided the implementation details of our method.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
wttgggidddd/oeeSuuunnnnn ssiinyolicy) for more details. hetpolliicggiddeeCooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the implementation details of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the standard deviation with different random seeds in our supple- mentary documents.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the GPU types and the number of GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have created separate "Broader impacts" section in our supplementary
document.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: We have cited the original paper that produced the code package.
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the patasets.paperesiii
package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
18

---

13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided the details about training, license, limitations, etc.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codc/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We have involved the usage of LLM in methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20


Reviews:
[{"review_id": "CGwXnWyhe0", "reviewer": "Reviewer_6ZvU", "summary": "The paper proposes GUIDED, a method for fine-grained open-vocabulary detection (FG-OVD), a task aimed at detecting objects according to specific attribute-based text prompts. GUIDED decouples coarse-grained detection and fine-grained recognition and uses an LLM to identify the subject and attributes in the text prompt. Specifically, this framework is composed of \"Subject Identification,\" where an LLM is used to decompose the text prompt into the subject and attributes, \"Coarse-grained Object Detection,\" which combines a transformer-based object detector with attribute embedding fusion, and a \"Fine-grained Attribute Discrimination Module\" to identify region proposals from the detection stage that are aligned with the fine-grained attribute. GUIDED is evaluated on two FG-OVD datasets: FG-OVD and 3F-OVD. GUIDED achieves state of the art on these datasets. The authors report results applying GUIDED to the base models Grounding DINO, OWL-ViT, and LaMI-DETR, with the version based on LaMI-DETR performing the best overall.", "questions": "- What does \"EDD\" stand for in the tables? I assume it stands for GUIDED, but this needs to be clarified in the paper.\n- Why is there a star next to HA-FGOVD in Table 1?\n- What is the full inference time for a single example? This is important since the proposed framework includes an LLM, an extra (and potentially heavy) additional component compared to traditional detectors. \n\nIn light of the extensive experiments and good results, if the questions above can be clarified and integrated into the paper, then I would be willing to increase my score.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed GUIDED method achieves state-of-the-art for fine-grained open-vocabulary detection\n- The experiments are extensive, covering two different FG-OVD datasets and three different base models\n- The authors present ablation studies testing the different components of the GUIDED framework\n\nWeaknesses\n- The key results are not presented in an easily accessible format. For example, the acronym \"EDD\" is not defined in the text even though it is included in many tables, and the text in Fig 1 (a) is difficult to read. The texts on the images in Fig 2 of the supplementary are also difficult to read.", "was_revised": true, "final_justification": "I have decided to raise my score since the authors have addressed my questions sufficiently. The authors have also mainly addressed the comments from the other reviewers. I hope the authors take the feedback from the discussion and improve the clarity and presentation of the final manuscript.", "rebuttal": "Thanks for the positive feedback on the extensive experiments and the superior performance of our method. We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:\n\n\n**The meaning of 'EDD'.** \nWe apologize for the oversight. 'EDD' was a working name for our method from an earlier draft. We will correct this typo in the revised manuscript and perform a thorough check to ensure all terminology is consistent.\n\n**Texts in some figures are difficult to read.** \nWe will resize and rearrange the text in these figures to improve clarity. Additionally, we will review all figures in the paper to ensure they are legible. \n\n**The meaning of the star next to HA-FGOVD.** \nAs noted in the caption of Table 1 in our main paper, the star indicates that the results for HA-FGOVD are from our reproduction because the original paper did not provide these results. We will make this explanation more prominent in the caption to avoid any confusion.\n\n**Full inference time of a sample.** \nWhile GUIDED utilizes `GPT-4o` for its reasoning capabilities, measuring its API inference time directly is problematic as network latency becomes a confounding factor. To provide a meaningful assessment of the computational overhead introduced by our LLM component, we instead evaluated inference times using locally deployable LLMs: `LLaMA-3.1-8B` and `LLaMA-3.3-70B`. Crucially, these models achieve comparable mAP performance to `GPT-4o` on the FG-OVD task (as validated in Table 1), making them valid proxies for analyzing inference speed. We measured the inference time in Table 1 and will include these results in the revised paper.\n\n**Table 1:** The comparison of inference time(ms) between Baseline(LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP(%) of each method in FGOVD. \n| **Method** | LLM | Detector | CLIP | Overall | mAP |\n|---|---|---|---|---|---|\n| Baseline | - | 198.5 | 13.2 | 211.7 | 43.2 |\n| GUIDED with LLaMA-3.1-8B | 72.1 | 198.8 | 13.3 | 284.2 | 65.9 |\n| GUIDED with LLaMA-3.3-70B | 193.8 | 198.8 | 13.3 | 405.9 | 66.5 |"}, {"review_id": "D0nIlyAcRd", "reviewer": "Reviewer_KZWx", "summary": "The paper proposes GUIDED, a decomposition framework for fine-grained open-vocabulary object detection (FG-OVD) that addresses semantic entanglement between subjects and attributes in vision-language model embeddings. The method decomposes FG-OVD into three stages: (1) subject identification using LLMs to extract coarse-grained subjects and attributes, (2) coarse-grained object detection guided by subject embeddings with an attribute embedding fusion module, and (3) fine-grained attribute discrimination using refined CLIP embeddings with a projection head. GUIDED achieves 23.2% mAP improvement over baseline LaMI-DETR on FG-OVD benchmark and demonstrates state-of-the-art performance across multiple datasets.", "questions": "1. How robust is the LLM-based subject identification? What happens when the LLM fails to correctly identify subjects or attributes? Have you analyzed error propagation from this stage?\n2. Hyperparameters (m_fine=50, m_coarse=50) appear to be fixed values without sensitivity analysis or justification for these specific choices.\n3. Given the reproducibility concerns with using proprietary GPT-4o, have you experimented with open-source alternatives like LLaMA or other accessible models? How does the choice of LLM affect the subject identification quality and overall performance?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Well-motivated: The paper clearly identifies and visualizes the semantic entanglement issue in existing FG-OVD methods through empirical analysis (Figure 1), showing how attribute over-representation leads to mislocation and semantic drift.\n2. Systematic approach: The three-stage pipeline (identification, detection, discrimination) is well-designed and aligns each subtask with appropriate model strengths - using detection transformers for localization and VLMs for fine-grained discrimination.\n3. Comprehensive experiment: Extensive experiments on FG-OVD and 3F-OVD benchmarks. The method shows consistent improvements across different base detectors. Achieves significant performance gains and establishes new state-of-the-art results on multiple benchmarks.\n\nWeaknesses:\n1. There's insufficient analysis of the attribute similarity score. Could you provide the result, if alpha in Eq. (5) is 0 or 1? Furthermore, have you tried other fusion strategies, e.g., weighted average?\n2. The method introduces additional complexity through LLM preprocessing, refined CLIP inference, and multi-stage processing, but computational cost analysis is insufficient. What is the complete computational overhead breakdown for GUIDED compared to direct fine-grained detection? Please include LLM preprocessing time, additional CLIP inference, and memory requirements.\n3. Failure cases: The paper briefly mentions limitations regarding attributes extending beyond detection boxes, but doesn't provide visualization analysis or scenarios where the decomposition strategy might not work well.\n4. Reproducibility concerns: The method relies on GPT-4o for subject identification, which is a closed-source model. This creates reproducibility issues and introduces potential non-transparent biases in the decomposition process that cannot be fully analyzed or controlled.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I hope the authors include the additional experiments in their manuscript to better support their motivation.", "rebuttal": "Thanks for the positive feedback on the well motivation, systematic approach and systematic approach in our method.  We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:\n\n**Insufficient analysis of the attribute similarity score.**  We conducted an ablation study by setting $\\alpha$ to extreme values  (summarized in Table 1). \nSetting $\\alpha$ to $0$  means relying solely on the detector's coarse confidence scores to estimate final scores and $ L_{\\text{contrast}}$, while setting $\\alpha$ to $1$ means relying only on the attribute similarity scores. \nAs shown, both extremes lead to a significant performance drop. This empirically confirms our hypothesis that a balanced fusion is crucial.\nRegarding alternative fusion strategies, we evaluate the weighted average in Table 2. It shows that weighted multiplication achieves a better performance. Compared with the weighted average, the multiplication enforces a high score on both the coarse confidence score and the attribute similarity simultaneously. This prevents candidates with a poor attribute similarity score from being promoted simply because their confidence scores are high (and vice versa).\n\n\n**Table 1:** More ablation study on the hyperparameter $\\alpha$.\n| $\\alpha$ | 0.0 | 0.6 | 1.0 |\n|---|---|---|---|\n| MAP | 17.1 | 66.4 | 28.0 |\n\n\n**Table 2:** Ablation study with different fusion strategies.\n| Fusion | MAP |\n|---|---|\n| Weighted Average | 63.1 |\n| Ours | 66.4 |\n\n\n**Insufficient computational overhead analysis.**  \nWhile our method utilizes GPT-4o for its reasoning capabilities, measuring its API inference time directly is problematic as network latency becomes a confounding factor. To provide a meaningful assessment of the computational overhead introduced by our LLM component, we instead evaluated inference times using locally deployable LLMs:` LLaMA-3.1-8B` and `LLaMA-3.3-70B`. Crucially, these models achieve comparable mAP performance to GPT-4o on the FG-OVD task (as validated in our experiments), making them valid proxies for analyzing inference speed. We measured the inference time in Table 3. As shown in the Table, the inference time increase in GUIDED primarily stems from LLM-based subject identification, while AEF and CLIP design in FGAD contribute minimally to latency. This time consumption gain can be alleviated by adopting a lightweight LLM or batch processing multiple subject identification tasks in one chat. For memory consumption in Table 4, the memory footprint of our core detection modules (AEF, FGAD) is negligible. The significant memory increase comes from loading the LLM. We will include these analyses in the revised paper.\n\n**Table 3:** The comparison of inference time(ms) between Baseline(LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP(%) of each method in FGOVD.\n| **Method** | LLM | Detector | CLIP | Overall | mAP |\n|---|---|---|---|---|---|\n| Baseline | - | 198.5 | 13.2 | 211.7 | 43.2 |\n| GUIDED with LLaMA-3.1-8B | 72.1 | 198.8 | 13.3 | 284.2 | 65.9 |\n| GUIDED with LLaMA-3.3-70B | 193.8 | 198.8 | 13.3 | 405.9 | 66.5 |\n\n\n\n**Table 4:** The Memory occupation(MB) of each module during inference in our GUIDED framework.\n| Method | LLM | Other |\n|---|---|---|\n| Baseline | - | 5277 |\n| GUIDED with LLaMA-3.1-8B | 6414 | 5329 |\n| GUIDED with LLaMA-3.3-70B | 44073 | 5329 |\n\n\n\n\n\n **Failure cases.** \nWe identify a key limitation when fine-grained attributes describe spatial relationships extending beyond the detected subject's bounding box. For example, for the query \"suitcase on the conveyor belt\", the conveyor belt (contextual attribute) may not be captured within the candidate box of any suitcase. Consequently, GUIDED may incorrectly associate the attribute with a suitcase spatially adjacent to the conveyor belt. This occurs because our method relies on the detector's candidate boxes, which may not encompass all required contextual elements. Due to the NIPS policy, we can not provide the visualization here, and we will include visualizations of such cases in the revised paper to clarify this failure mode. \n\n**Reproducibility concerns and the choice of LLMs.** \nWe appreciate this important concern regarding reproducibility. To ensure transparency, we conducted experiments with open-source LLMs (`LLaMA-3.1-8B` and `LLaMA-3.3-70B`) as alternatives to `GPT-4o`. As quantified in Table 5, the mAP of GUIDED drops by merely 0.5\\% with `LLaMA-3.1-8B` and improves by 0.1\\% with `LLaMA-3.3-70B`. This demonstrates that GUIDED performance is not dependent on a specific proprietary model. We will provide more results with `LLaMA-3.1-8B` and `LLaMA-3.3-70B` forwith other OVD models and other datasets in our revised paper for reproducibility.\n\n**Table 5:** The comparison of mAP(%) evaluation results on the FG-OVD benchmark with different LLMs in subject identification.\n| **LLM** | **mAP** |\n|---|---|\n| GPT4-o | 66.4 |\n| LLaMA-3.1-8B | 65.9 |\n| LLaMA-3.3-70B | 66.5 |\n\n\n\n\n**Robustness of LLMs on subject identification.** \nTo evaluate the robustness of LLMs on subject identification, we manually annotate the subjects from 300 fine-grained classes and assess the accuracy of subject identification with different LLMs. As summarized in Table~, failure cases are categorized into two types: (1) Hallucination toward in-context samples, where the LLM generates subjects irrelevant to the input text;  (2) Other errors, such as identifying an attribute instead of an object. Overall, all three LLMs achieve high correctness rates, demonstrating robust performance across diverse architectures.\nThe results confirm the high robustness of this stage. Crucially, they show that while the smaller LLaMA-3.1-8B model is prone to hallucination errors that always propagate (8/8), this critical failure mode is completely eliminated by larger open-source models. Both LLaMA-3.3-70B and GPT-4o exhibit near-perfect performance, with their rare errors being minor and not always affecting the final detection. This demonstrates that our method's success is not tied to a specific proprietary model and is robust when using state-of-the-art open-source alternatives. We will add this analysis to our revised paper.\n\n**Table 6:** The number of failure cases in subject detection of 300 samples with different LLMs. The notation '$x/y$' denotes that there are $y$ failure cases, of which $x$ lead to detection errors.\n\n| LLM | Hallucination | Others | Total |\n|---|---|---|---|\n| GPT4-o | 0/0 | 1/2 | 1/2 |\n| LLaMA-3.1-8B | 8/8 | 1/1 | 9/9 |\n| LLaMA-3.3-70B | 0/0 | 1/1 | 1/1 |\n\n\n**Analysis of $m_{fine}$ and $m_{coarse}$.**\n$m_{fine}$ and $m_{coarse}$are used as temperatures to perform softmax with CLIP embeddings, and 50 is a common choice for the temperatures, which is extensively applied in previous work. We also provide the result with different $m_{\\text{fine}}$ and $m_{\\text{coarse}}$ in Table 7, performance remains robust across variations of these values.\n\n\n**Table 7:** Ablation study on $m_{fine}$ and $m_{coarse}$.\n| $m_{fine}$ | $m_{coarse}$ | MAP |\n|---|---|---|\n| 30 | 30 | 66.0 |\n| 50 | 50 | 66.4 |\n| 100 | 100 | 66.4 |"}, {"review_id": "17iKixtdFW", "reviewer": "Reviewer_vfih", "summary": "This paper introduces GUIDED, a method for fine-grained open-vocabulary object detection that addresses the challenge of semantic entanglement between object categories and their attributes. The proposed approach decomposes the task into three stages:  \n\n1. Subject Identification, where GPT-4o is used to extract a coarse object category and its associated attributes from fine-grained class names.  \n\n2. Coarse-Grained Object Detection, which uses only the subject embedding for object localization, with an attribute embedding fusion module to selectively integrate relevant attribute cues. \n\n3. Fine-Grained Attribute Discrimination, which re-scores detected regions using full fine-grained descriptions via refined CLIP text encoder.  \n\nThe model is trained in two stages, first on LVIS base categories, then fine-tuned on FG-OVD. The model is evaluated on the FG-OVD and 3F-OVD benchmarks.", "questions": "In the attribute embedding fusion module, query updates are computed using cross-attention where the keys are formed by subtracting the subject embedding from each attribute embedding. While the idea is that this highlights the attribute-specific signal, it’s not clear to me: \n\n- Why this subtraction should be meaningful or stable, especially across diverse classes or longer prompts. For example, subtracting the embedding of “car” from “a shiny red car with tinted windows” assumes that the resulting vector cleanly captures the difference (“shiny red with tinted windows”). But CLIP embeddings are not trained with this kind of linear structure in mind, and it's unclear whether such a subtraction generalizes across different sentence structures or domains. \n\n- Whether it introduces noise or ambiguity when attributes are semantically similar or visually correlated. For example, the difference between “a white ceramic mug” and “a white porcelain mug” may be minimal in CLIP space and subtracting the subject (“mug”) from each might produce almost the same or unstable vectors. This could affect attention-based fusion especially when the attributes are subtle or highly co-occurring. \n\nHas this subtraction-based attention been compared empirically with simpler alternatives like concatenation or additive fusion? \n\nClarifying this design choice and providing results on standard open-vocabulary settings like COCO and LVIS (especially given that the model is trained on LVIS base) would help in assessing generalization and could positively influence my rating.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The following are the strengths of the paper. \n\n1. The paper provides a reasonable rationale for separating fine-grained object detection into three stages, subject identification, coarse-grained object detection and fine-grained attribute discrimination. The observation that VLMs like CLIP perform better on base-level category names than on attribute-rich prompts is well-argued and aligns with known limitations of CLIP embeddings. \n\n2. The proposed attribute embedding fusion (AEF) module, which integrates attribute information into object queries via cross-attention, is conceptually simple but intuitive. It enables selective incorporation of useful attributes without much interfering with the subject signal during detection. \n\n\nThe following are the weaknesses of the paper. \n\n1. Key variables such as F_conv, F_en, etc. are not linked to the architecture block diagrams. The block diagram fails to capture the structure described in the equations, making it difficult to follow the implementation details. \n\n2. **Unfair comparison with LaMI-DETR:** The paper claims a +23.2% improvement over LaMI-DETR but does not clearly state whether LaMI-DETR was fine-tuned on FG-OVD or whether the same training protocol and dataset were used. Without matched experimental conditions, such comparisons are misleading. \n\n3. **No evaluation on standard open-vocabulary benchmarks:** Despite training on LVIS base categories, the paper skips any evaluation on LVIS novel categories (e.g. APr) or COCO base/novel splits. Even though the focus is fine-grained OVD, it is still important to report performance on broader OVD tasks to understand generalizability and potential trade-offs.", "was_revised": true, "final_justification": "After reading the author response and the other reviewers’ comments, I am updating my rating to a borderline accept. The paper presents a meaningful and novel decomposition strategy for fine-grained open-vocabulary object detection (FG-OVD), separating subject localization and attribute discrimination into distinct modules. The attribute embedding fusion (AEF) module is conceptually intuitive and empirically validated through targeted ablations. The method shows good performance on FG-OVD benchmarks and modest results on general OVD tasks like LVIS and COCO.\n\nWhile I appreciate the authors' detailed clarifications and additional results, I remain concerned about the way some evaluation tables mix results from different training stages (e.g., FG-OVD after Stage 2 and LVIS after Stage 1), which could convey misleading conclusions about generalization. A clearer separation or consistent reporting of results from the same model would improve confidence in the claims. Nonetheless, I acknowledge the effort made to address concerns and am willing to lean towards acceptance.", "rebuttal": "Thanks for the positive feedback on the reasonable rationale and the AEF module in our method.  We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:\n\n**Key variables are not linked to the architecture block diagrams.** \nWe appreciate this observation. In the revised manuscript, we will explicitly annotate Figure 2 to map key variables to their corresponding components in the architecture diagram.\n\n**Unfair comparison with LaMI-DETR**\nTo address this concern, we clarify that in Table 1 of our main paper, we initially reported LaMI-DETR's performance trained on LVIS following the HA-FGOVD protocol, which matches the first-stage training in our method. Additionally, we report the performance of LaMI-DETR under the same training protocol in Table 4(see line 3 Baseline + Finetune) in our main paper. Under the same protocol, our GUIDED framework shows a 15.5\\% improvement, underscoring the effectiveness of our method. For clarity, we will also integrate the results trained on the same protocol into Table 1 in our main paper.\n\n\n**No evaluation on standard open-vocabulary benchmark.**\nWe present the performance on the LVIS benchmark in Table 1. The results show fine-tuning on the FG-OVD dataset degrades the performance on the LVIS benchmark. However, co-training on LVIS and FG-OVD during the second stage not only alleviates this issue but surpasses the original LaMI-DETR baseline while mainly preserving FG-OVD performance.  This enhancement stems from our AEF module's attribute integration and expanded training data from FG-OVD. We will supplement our paper with these analyses.\n\n\n**Table 1:** The comparison on the standard open-vocabulary benchmark LVIS dataset and the FGOVD dataset.\n| Finetuning Data | FG-OVD Average | LVIS $AP_r$ | LVIS $AP_c$ | LVIS $AP_f$ | LVIS $AP$ |\n|---|---|---|---|---|---|\n| Baseline | 62.5 | 43.2 | 39.3 | 43.5 | 41.6 | \n| FG-OVD | 66.4 | 39.3 | 32.6 | 38.7 | 36.2 |\n| FG-OVD & LVIS | 65.5 | 44.4 | 39.4 | 43.7 | 41.9 |\n\n\n**Whether subtraction should be meaningful or stable across diverse classes.**\nThe CLIP applies the contrastive learning loss to align the embeddings of images and their corresponding caption. The alignment of texts with fine-grained attributes enables CLIP to encode fine-grained attributes in the text embeddings. Nevertheless, we acknowledge that CLIP text embeddings may not always capture all the attributes. To address this issue, GUIDED strategically employs subtraction-based AEF  to selectively exploit the knowledge in attributes for coarse-grained detection, which detects candidate regions for subjects of fine-grained classes \\textbf{without requiring exhaustive attribute knowledge}. GUIDED leaves the discrimination of fine-grained attributes in FGAD. FGAD applies a refined CLIP with an additional linear projection trained on fine-grained data, which demonstrates superior fine-grained feature capture capability.\nEmpirically, Table 1 of our supplementary material shows that the subtraction process enhances the MAP by 1.3\\%,  demonstrating that applying the subtraction is more reasonable than directly applying the original attribute embeddings. Furthermore, employing the CLIP embeddings of the isolated attributes (e.g., \"shiny red with tinted windows\") declines the performance by 2.8\\%. It is because the embeddings of the isolated attributes do not capture the relation between subjects and attributes.\n\n\n\n**Whether subtraction introduces noise or ambiguity.**\nWhile CLIP text embeddings may not capture all attributes perfectly, the subtraction process may output similar key embeddings for classes with similar or visually correlated attributes. Nevertheless, similar to the response for the last question, GUIDED inherently mitigates this issue by design. Specifically, AEF selectively integrates attribute knowledge for coarse-grained detection rather than exhaustive attribute discrimination, thus avoiding reliance on subtle inter-attribute distinctions. \nCrucially, when subtraction produces similar key embeddings, their corresponding value embeddings, which are directly obtained from attribute embeddings, exhibit analogous similarity. Consequently, AEF’s selective fusion integrates minimally divergent attribute signals into object queries, which naturally down-weighting noise or irrelevant information.\n\n\n\n\n**Comparing subtraction-based attention with simpler architecture.**\nWe compare the mAP of subtraction-based attention with additive fusion and concatenation in FGOVD in Table 2.\nFor concatenation, we apply a linear projection layer to transform the dimensions back to the original dimensions. The results show that our AEF outperforms both additive fusion and concatenation by a clear margin, demonstrating the effectiveness of the subtraction-based attention on attribute integrations.\n\n**Table 2:** The comparison of mAP(%) evaluation results on the FG-OVD benchmark between our proposed AEF and simpler architecture.\n| Arch | mAP |\n|---|---|\n| Concentration | 61.1 |\n| Addition | 63.2 |\n| AEF | 66.5 |"}, {"review_id": "Yh4JW2mtqT", "reviewer": "Reviewer_HXfJ", "summary": "The paper addresses the challenge of fine-grained open-vocabulary object detection (FG-OVD), where existing methods struggle with semantic entanglement between object subjects and attributes in vision-language model (VLM) embeddings. The proposed GUIDED framework decomposes FG-OVD into three stages: subject identification using a large language model (LLM), coarse-grained object detection guided by subject embeddings, and fine-grained attribute discrimination via a refined VLM. By separating localization and attribute recognition, GUIDED mitigates attribute over-representation and semantic drift. Key innovations include an attribute embedding fusion module for selective attribute integration and a projection-based discrimination mechanism. Experiments on FG-OVD and 3F-OVD benchmarks show GUIDED outperforms state-of-the-art methods by up to 19.8% in mAP, demonstrating improved fine-grained detection capability.", "questions": "1. It is not aligned with Figure 3(b) and Figure 2. For example, in Figure 3(b), there are two kinds of queries: a learnable query and an object query.\n2. What the mean of EDD in Tables 1 and 2?", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The attribute embedding fusion and projection-based discrimination modules are plug-and-play, enhancing existing OVD frameworks without requiring major architectural changes.\n2. GUIDED effectively decouples FG-OVD into manageable subtasks, leveraging the strengths of LLMs for subject parsing, detection transformers for localization, and VLMs for attribute discrimination. This addresses the core issue of semantic entanglement.\n3. Experiments cover multiple benchmarks (FG-OVD, 3F-OVD), architecture variants (Grounding DINO, OWL-ViT, LaMI-DETR), and ablation studies validating each module's contribution.\n\n### Weaknesses\n1. The motivation is in contrast to previous methods that argue that the attributes will help the open-vocabulary object detection, such as [1].\n2. During inference, the proposed method needs an LLM to infer the subject name; it would significantly decrease the inference efficiency compared with previous OVD methods, i.e., the embedding for class names is precomputed for inference.\n3. Due to the usage of LLM, the proposed method also needs to be compared with the results of directly using the MLLM for FG-OVD.\n\n[1] OvarNet: Towards Open-vocabulary Object Attribute Recognition.", "was_revised": true, "final_justification": "Thanks for the response from the authors. My questions have been addressed; The insights of this work are novel to me, and the experimental improvements are impressive. However, this method also introduces additional LLM computations; therefore, I keep my rating unchanged.", "rebuttal": "Thanks for the positive feedback on the effectiveness of modules in our method.  We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:\n\n**The motivation is in contrast to previous methods.**\nThank you for raising this important point. Our motivation does not contradict, but rather builds upon and extends the principles of methods like OvarNet to address the unique challenges of FG-OVD.\nFirst, we concur with OvarNet's core idea that learning attributes of objects can be beneficial for object detection. Our AEF module is designed on this principle, aiming to fuse beneficial attribute information with object features. In this regard, our philosophy is aligned with prior work.\nSecond,  unlike conventional OVD (e.g., on MS-COCO) or attribute detection in OvarNet, FG-OVD involves class names that are dense with descriptive attributes (e.g., \"blue striped shirt\"). This co-occurrence can lead to an \"attribute over-representation\" problem in text embeddings, where attribute semantics overwhelm the core subject identity. OvarNet, while effective for its target tasks, does not explicitly address this semantic entanglement issue inherent to FG-OVD. In summary, our work proposes to leverage helpful attributes while mitigating their potential to obscure the subject's identity in the fine-grained text embeddings. We will refine our manuscript to make this distinction clearer.\n\n**Inference efficiency decreased by LLM.** \nWe acknowledge the reviewer's valid concern regarding the inference latency introduced by the LLM. This represents a trade-off between our method's enhanced semantic understanding and computational cost. Nevertheless, the LLM-based subject identification is performed once per class name, not per image. For any given dataset or application scenario, the set of fine-grained classes is fixed. Therefore, the parsing results can be pre-computed and cached offline, imposing no additional LLM-related latency. \nFor scenarios requiring on-the-fly parsing of new class names, the latency can indeed be a factor. This can be alleviated by employing more lightweight LLMs(e.g. LLaMA-3.1-8B as shown in Table 1) or batch processing multiple subject identification tasks in one chat with LLMs. We will also provide the inference efficiency analysis in our main paper.\n\n**Table 1:** The comparison of inference time (ms) between Baseline (LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP (%) of each method in FGOVD.\n\n| Method                        | LLM   | Detector | CLIP  | Overall | mAP  |\n|-------------------------------|-------|----------|-------|---------|------|\n| Baseline                      | -     | 198.5    | 13.2  | 211.7   | 43.2 |\n| GUIDED with LLaMA-3.1-8B      | 72.1  | 198.8    | 13.3  | 284.2   | 65.9 |\n| GUIDED with LLaMA-3.3-70B     | 193.8 | 198.8    | 13.3  | 405.9   | 66.5 |\n\n**Performance of MLLMs.**\nWe evaluate a `LLaVA-v1.6-Mistral-7B`, one of the strongest open-source MLLMs available. \nFor each class in an FG-OVD annotation, we queried the model with a RefCOCO-style prompt, where `statement` is replaced by the target class name:\n\n> `Your task is to locate and identify all objects or regions in the image that match the provided textual description.`\n> `Description: \"{statement}\"`\n> `Please return a JSON list. Each element must contain a \"bbox\" in the form [x1, y1, x2, y2] and a \"confidence\" between 0.0 and 1.0.`\n\nUnder this protocol, the model attains only **0.4~mAP** on the FG-OVD benchmark, which is significantly below our *GUIDED*. This experiment reveals that direct MLLM inference is presently insufficient for practical or research use in FG-OVD.\n\n**It is not aligned with Figure 3(b) and Figure 2.** \nThanks for pointing this out. Figure 2 does not accurately depict the two query types. We will correct it in our revised paper.\n\n**The meaning of 'EDD'.**  \nWe apologize for the oversight. 'EDD' was a working name for our method from an earlier draft. We will correct this typo in the revised manuscript and perform a thorough check to ensure all terminology is consistent."}]

Author rebuttal:

Meta-review: Overall, the paper introduces GUIDED, a well-motivated and systematic approach for fine-grained open-vocabulary object detection, addressing the key challenge of semantic entanglement between object categories and attributes. Reviewers highlighted the novelty of decomposing the task into subject identification, coarse-grained detection, and fine-grained attribute discrimination, along with the intuitive attribute embedding fusion module. Experiments are extensive, covering multiple FG-OVD benchmarks and base models, and the method achieves consistent state-of-the-art improvements. Some concerns were raised regarding accessibility of results, clarity in diagrams and variable definitions, computational overhead from LLM preprocessing, and reproducibility due to reliance on GPT-4o.

During the rebuttal, the authors provided clarifications, added missing results, and addressed technical concerns, resolving all reviewers’ questions. While minor limitations remain, such as additional LLM computation and broader benchmark evaluations, the core contributions, strong experimental validation, and novelty in addressing semantic entanglement convincingly support the quality and significance of the work. Based on the overall positive feedback and successful resolution of concerns, the final recommendation is to accept the paper.
Decision: Accept (poster)