Kati8/ICLR_2023_2025 huggingface dataset:
authors: Sheng Jin
authors: Xueying Jiang
authors: Jiaxing Huang
authors: Lewei Lu
authors: Shijian Lu
reviews: 
Comments: This paper proposes Descriptor-Enhanced Open-Vocabulary Detector (DVDet) which introduces a conditional visual (region) prompting and textual descriptor dictionaries by using Large Language Models. The use of textual descriptors is motivated by the observation that the VLMs are good at capturing the fine-grained attributes of the objects. The textual descriptor dictionary is dynamically updated during training by tracking how frequent each descriptors are used, and by prompting what visual descriptors can help distinguishing the confusing categories. In addition, this work proposes a conditional region prompt to help the alignment between the detected region and text embeddings. When using both the textual descriptors and conditional visual prompts, DVDet improves the modern open-vocabulary detectors on the OV-COCO and OV-LVIS benchmarks.
Strengths: * The use of Large Language Models to enrich the vocabulary description is well motivated.

* The proposed conditional visual prompt is useful in bridging the gap between the image-text and region-text alignment.

* The proposed DVDet method can be plugged into other SOTA open-vocabulary detection methods and consistently improve the performance.
Weaknesses: * The use of region-level prompting for open-vocabulary detection is previously proposed in CVPR 2023 paper "CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching". The comparison (both at a idea level and a numeric comparison) with CORA work seems necessary regarding the technical similarity.

* It is not clear what the optimal number of iterations is for the dictionary update. Is "the number of high-frequency descriptors" in Table 7 the number of iterations?

* What is the effect of using the LLM on the running speed of the DVDet detector (using no LLM vs N-th interaction of LLM dictionary)? While the running speed might not be the main focus of open-vocabulary detection research, the multiple iterations of LLM use could make the speed not very favorable.

* There are non-negligible number of hyper parameters (number of descriptors, number of update iterations, thresholds).
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper introduces DVDet, a Descriptor-Enhanced Open Vocabulary Detector featuring two innovative components: conditional context prompts and hierarchical textual descriptors. The conditional context prompts function in the Region of Interest (RoI) align to expand each region proposal's area, incorporating more context information and converting it into an image-like feature preferred by CLIP. Meanwhile, the hierarchical textual descriptors dynamically acquire fine-grained descriptors for each category through interaction with the Language Model (LLM) during training. Extensive experiments indicate that DVDet significantly enhances the performance of various Open-Vocabulary Detectors on two benchmarks, MS COCO and LVIS.
Strengths: 1. I greatly value the use of LLM to dynamically generate detailed descriptors of categories. This approach inspires a novel method of employing LLM to enhance the performance of OVD tasks. 

2. The concept presented is both straightforward and potent, with the paper being lucid and easy to navigate. The author's introduction of Framework Figures (Fig 2) and the examples of iterative fine-grained descriptors (Figure 3) are commendable as they significantly aid the reader in grasping the idea more effectively.

3. The experiments conducted are thoroughly robust. First, the author meticulously scrutinizes each component of their methodology, as detailed in Table 3 and Table 9. Second, the author conducts comprehensive experiments on a variety of OVD detectors and datasets to demonstrate that DVDet significantly enhances performance.
Weaknesses: 1. A comparison of computation costs with the baseline has not been conducted.

2. The contribution of the technique is somewhat restricted.

3. The prompt generated by LLM may possess an element of randomness, potentially making it challenging to reproduce.

4. No ablation experiment about the hyper-parameter $m$ of Conditional Context Prompt
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good