LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
Authors: Sheng Jin
Xueying Jiang
Jiaxing Huang
Lewei Lu
Shijian Lu
Venue: ICLR 2024 poster (2024)
Keywords: Open Vocabulary Object Detection
Visual descriptors
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Inspired by the outstanding zero-shot capability of vision language models (VLMs) in image classification tasks, open-vocabulary object detection has attracted increasing interest by distilling the broad VLM knowledge into detector training. However, most existing open-vocabulary detectors learn by aligning region embeddings with categorical labels (e.g., bicycle) only, disregarding the capability of VLMs on aligning visual embeddings with fine-grained text descriptions of object parts (e.g., pedals and bells). This paper presents DVDet, a Descriptor-Enhanced Open Vocabulary Detector that introduces conditional context prompts and hierarchical textual descriptors that enable precise region-text alignment as well as open-vocabulary detection training in general. Specifically, the conditional context prompt transforms regional embeddings into image-like representations that can be directly integrated into general open vocabulary detection training. In addition, we introduce large language models as an interactive and implicit knowledge repository which enables iterative mining and refining visually oriented textual descriptors for precise region-text alignment. Extensive experiments over multiple large-scale benchmarks show that DVDet outperforms the state-of-the-art consistently by large margins.

Published as a conference paper at ICLR 2024
LLMS MEET VLMs: BOOST OPEN VOCABULARY OB- JECT DETECTION WITH FINE-GRAINED DESCRIPTORS
Sheng Jin', Xueying Jiang', Jiaxing Huang', Lewei Lu", Shijian Lu
S-Lab, Nanyang Technological University 2 Sense TimeResearch [Jiaxing Huang, xueying003, Shi jian. Iu) edu.
ABSTRACT
Inspired by the outstanding zero-shot capability of vision language models (VLMs) in image classification tasks, open-vocabulary object detection has at- tracted increasing interest by distilling the broad VLM knowledge into detec- tor training. However, most existing open-vocabulary detectors learn by align- ing region embeddings with categorical labels (e.g.. bicycle) only, disregarding the capability of VLMs on aligning visual embeddings with fine-grained text de- scription of object parts (e.g., pedals and bells). This paper presents DVDet, Descriptor-Enhancee Open Vocabulary Detector that introduces conditional con- text prompts and hierarchical textual descriptors that enable precise region-text alignment as well as open- vonn-bulabbayy detection training in general. Specifically, the conditional context prompt transforms regional embeddings into image-like representations that can be directly integrated into general open vocabulary detec- tion training. In addition, we introduce large language models as an interactive and implicit knowledge repository which enables iterative mining and refining vi- sually oriented textual descriptors for precise region-text aligmment. Extensive experiments over multiple large-scale benchmarks show that DVDet outperforms the state-of -fet---rr-t-tttttte consistently by large margins.
INTRODUCTION
Vision Language Models (VLMs) Yu et al. 2022 Yuan et al. 2021 Zhai et al. 2021 Jia et al.,
2021 Radford al. 2021 Zhou et al.) 202 Rao et al. 2021 Huynh et al. 2021) have demon- strated unparalleled zero-shot capabilities in various image classification tasks, largely attributed to the web-scale image-text data they were trained with (Radford et al.) 2021 Jia et al. 202I). As researchers naturally move to tackle the challenge of open vocabulary object detection (OVOD) (Li et al., 2021 Kamath et al. 2021; Cai et al. 2022). they are facing a grand data challenge as there does not exist similar web-scalt data with bex-level annotations. The much less training data in OVOD inevitably leads to clear degradation in text-image alignment, manifesting in much weaker zero-shot capabilities in most open-vocabulary object detectors. An intriguing question arises: Can we leverage the superior image-text alignment abilities of VLMS to enhance OVOD performance?
Recent studies (Du al. 2022 Feng al. 2022)) indeed resonate with this idea, attempting to distill the knowledge from VLMs to extend the vocabulary of object detectors. For example, ViLD Guu
al. 2021) and its subsequent work (Ma et al., 2022, Zhou al. 2022, Lin et al. 2023) enforce detectors' embeddings to be aligned with the embeddings from CLIP image encoder or text en- coder. However, VLMs still clearly outperform open-vocabulary detectors while aligning visual embeddings with text embeddings of categorical labels (e.g., "bicycle") as illustrated in Fig Upon deep examination, we found that VLMs are particularly good at aligning fine-grained descriptors of object attributes or parts (c.g., 'bell' and 'pedal') with their visual counterparts, expertise yet harnessed in existing OVOD models. Specifically, most existing OVOD methods focus on distilling coarse and category-level alignment knowledge visual and textual embedding. They largely ne- glect the fine-gratee grained and descriptor- alignment knowledge that VLMs possess, leading to the under-utilization of VLM knowledge in the trained OVOD models.
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Cress roool dignneet the couki sinbarity
VIMe
boyde
Crop
Algement with Deserptors blcycle
Sodale
dell
embeddigg
roflcttrr
fran
busket
Learrable Prampe
pedoo
(4) Mouaakeettud aligmment on the siholl CDCC detanet
(b) Elvetration with reampl bbel bayyh and relered tectrs descriptons
Figure Differences in image-text alignments by VLMs and OVOD. Over the whole COCO dataset, the visual and textual embeddings from VLMs are clearly better aligned than those from OVOD (by the state-of-the-art VLDet (Lin et al. 2023)) for both categorical object labels and fine- grained descriptors as shown in (a). The proposed DVDet mines and refines fine-grained descriptors with LLMs which clearly improves region-text alignment as compared with VLDet. This can be viewed in more detail in (b) with an exemplar label 'bicycle' and fine- fine-grained descriptors of bicycle parts. The alignment is measured by the cosine similarity between visual and textual embeddings.
We design Rescriptor-Enhancee Open Vocabulary Detection (DVDet) that exploits VLMs' prowess in descriptor-level region-text alignment for open vocabulary object detection. The essential idea is exploit VLMs' alignment ability via customized visual prompt, which mines regional fine-grained descriptors from large language models (LLMs) iteratively and enables prompt training without resource-intensive grounding annotations. The key design in DVDet is Conditional Context visual Prompt (CCP) that transforms region embeddings into image-like counterparts by fusing contextual background information around region proposals. This allows CCP to be seamlessly integrated into various open- vocabullay detection training with little extra designs.
To train CCP effectively, we introduce LLMs as communicable and implicit knowledge repositories for iterative generation of fine tre-ggrined descriptors for precise region-text alignment. Specifically, we design a hierarchical update mechanism that interacts with LLMs by retaining most-related descrip- tors while actively soliciting new descriptors from LLMs, enabling CCP training without additional annotation costs. This mechanism enhances descriptor diversity and descriptor availability, lead- ing to regional fine-grained descriptors that tailored the most relevant object categories. In addition, we design a simple yet effective descriptor merging and selection strategy to tackle two challenges in CCP training: 1) distinct object categories could share similar fine-grained descriptors which leads to potential confusion in CCP training; 2) object images may not have all fine-grained descriptors present due to occlusions, etc., more detail to be described in the ensuing Method.
The contributions of this work can be summarized in three major aspects. First, we introduce a feature-level visual prompt that transforms object embeddings into image-like representations that can be seamlessly plugged into existing open vocabulary detectors in general. Second, we design a novel hierarchical update mechanism that enables effective descriptor merging and selection and dynamical refinement of region-text alignment via iterative interaction with LLMs. Third, extensive experiments demonstrate that the proposed technique improves open-vocabular detection substan- tially for both base and novel categories.
2 RELATED WORK
Open-Vocabulle Object Detection (OVOD): OVOD utilizing the knowledge of pretrained VLMs
(Radford et al. 202T) has attracted increasing attention (Zhong et al.] 2022; Minderer et al. 2022)
with the advance open vocabulary image classification (Yu et al. 2022 Yuan et al. 202 1 Zhai
ct al,, [02T] Jia et al. 2021; Radford et al.] 2021). For example. ViLD Gu et al. 2021) distills
knowledge from VLMS into two-stage detector, harmonizing the detector's visual embeddings with those from the CLIP image encoder. HierKD (Ma et al.] 2022) focuses on hierarchical global- local distillation and RKD (Bangalath et al. (022) explores region-based knowledge distillation to improve the alignment between region-level and image-level embeddings. In addition, VLDet (Lin) et al. 2023) and Detic (Zhou et al.) 2022) align their detector embeddings with those from CLIP text encoder. Nevertheless, all these prior studies share similar misalignment between their trained de- tectors and pretrained VLMs: VLMs capture more comprehensive knowledge including fine- ine-grained knowledge about object parts, object attributes, and contextual background while open vocabulary
2

---

Published as a conference paper at ICLR 2024
detectors focus learning precise localization of interested objects. Such misalignment tends to restrict the efficacy of knowledge distillation from VLMs to OVOD.
Recently, prompt-based methods such as DetPro (Du et al. 2022) and PromptDet (Feng et al. 2022) have emerged as an alternative for alleviating the misalignment between upstream classificaton knowledge in VLMs and downstream knowledge in detection tasks. These methods adjust the textual embedding space of VLMs to align with regional visual object features by incorporating continuous prompts within VLMs. Despite their success, the adjustment focuses on cross-modal alignment between categorical labels and ROI embedding only, which tends to disrupt the inherent tssualltxtual alignment properties of VLMs. We design feature-level prompt learning technique that formulates the ROI embeddings of detectors to be highly similar to image-level embeddings, preserving the image-text alignment capabilities of pretrained VI.Ms effectively.
Visual Prompt Learning The concept of 'prompting' originates from the field of Natural Language Processing (NLP) and has gradually gained increasing attention as a means of guiding language models via task instructions (Brown et al.) (020). The evolution of continuous prompt vectors in few-shot scenarios (Li & 2i21
2021 Liu et al. 202 demonstrates its cross-domain applicaiiitty.
Pertinently, VPT (Jia et al., 2022) and its successors (Bahng et al.] 2022a b) have extended this idea to the visual domain, achieving precise pixel-level predictions. In addition, the prompting idea has also been explored in pre-trained video recognition models as well (Ju et al.) 2022 Lin et al. 20226). However, existing visual prompt algorithms generate a single prompt for each downstream task, which cannot handle detection tasks well that often involve multiple objects in single image. We address this issue by designing CCP which generates a conditional prompt for each object.
Leveraging Language Language Models Linguistic data has been increasingly exploited in open- vocabulary related research, and the recent LLMs have demonstrated their comprehensive knowl- edge that can be beneficial in various NLP tasks. This trend has extended computer vision re- search, and several studies have been reported to investigate how LLMs can assist in downstream computer vision tasks. For example, (Menon & Vondrick, [023] Zhang et al.) (023) have harnessed linguistic knowledge in pretrained LLMss to generate descriptors for each visual category. Such aug- mentation enriches VLMs without additional training or labeling efforts. Inspired by CuPL. (Menon) & Vondrick 2023), CaF employs GPT-3 (Brown et al., (2200) to craft semantically enriched texts, thereby enhancing the alignment between CLIP's text and images. However, most existing research treats LLMs as static database, acquiring useful information through a one-time interaction. We in- troduce a simple yet effective bierarchical mechanism that continuously interacts with LLMs during the model's training process, obtaining more diverse and visual-oriented textual data.
METHOD
3.1 OVERVIEW
Problem setup. Open Vocabulary Object Detection (OVOD) (Zareian et al.) 2021) leverages a dataset of image-text pairs to broaden its detection vocabulary from pre-defined categories to novel categories, Formally, the task is to construct an object detector using a detection datasct defined as T (1(4,. gt. J.Y..111 where 4, represents an image, 9: (bi, denotes the ground truth annotations consisting of bounding box coordinates b; and associated base categories c; Chase and D. symbolizes fine-grained descriptors that are generated through LLMs. The primary goal is to facilitate the detection of new classes Crovel in the inference stage.
The predominant OVOD framework typically utilizes a two-stage detection architecture as its back- bone, incorporating text embeddings to reformulate the classification layer. Generally, popular two-stage object detector, such as Mask-RCNN comprises a visual backbone encoder denoted as PENC, elass-agnostic region proposal network (RPN) represented by @RPN, and an open vocabulary classification module labeled as PCLS. The overall detection can be formulated by:
( }n)
} = PCLS &REN a (ENC(T.)
(1)
where I, denotes the i-th input image and {ūr...., . im. represents the set of predicted outputs.
Fig. 2 shows overview of the proposed DVDet framework including VLM-guided conditional context prompting flow and a LLss-assisted descriptor generation flow. In the prompting flow, the network takes an input image To and extracts the features ró for the i-th proposal. It then enlarges
3

---

Published as a conference paper at ICLR 2024
Conditional Context Regional Prompt
embodding
Descriptors
Bos Preditto
Text embeddinng
Siilority
Destriptir
Class Predictior
CCP
Meta
VLM
Selecttion
Frequency Record
LLMs Meets VLMs: Description Generation and Update
dinoogushh (rategory)
LLM (GPT3)
Deter gilens
des
Aex
iconlusing categury)
Update
Discard
There are (high-
dei
frequency descriptoni
to dosingush (untegery]. Please
Mangiin
Update
Hierarchy Descriptors Generation Mechanism
Visuil-oriented Descrigtors Update
Figure 2: Overview of our proposed DVDet framework: DVDet comprises two specific flows to improve the region-text alignment in open vocabulary detection. In the prompt flow (denoted by the solid line), the proposed conditional context prompt (CCP) transforms the ROI embeddings into image-like representation by fusing the contextual background information around the region proposal, that can be incorporated to facilitate the training of open vocabulary detectors. In the descriptor flow (denoted by the dashed line), hierarchy mechanism designed generate and update fine-grained descriptors via iterative interaction with LLMs for precise region-text alignment.
the proposal to integrate the contextual background and further extracts features to formulate r We design learnable meta-ner that takes ro' to create region prompt = and combines the prompt with rú to obtain prompted features vjj Finally, rh is fed to Box Predictor, and vg Class Predictor to assimilate the text embeddings of category labels and their fine-grained descriptors De. In the descriptor flow, design hierarchy mechanism generate and update D. for each category via iterative interactions with LLMs in training. Specifically, De records the frequency of fine-grained descriptors as well their probability of being misclassified other categories. also records confusing categories that statistically have high misclassification probability. During the training process, D. retains fieh-frryyency fine- ane-grained descriptors while discarding low- www-freuueccy ones. It employs both high- figh-frequency descriptors and confusing categories to prompt LLMs to generate more diverse and visually relevant descriptors, which are further incorporated into D. with semantic merging process.
3.2
CONDITIONAL CONTEXT REGIONAL PROMPTS
In this section, we introduce the Conditional Context Regional Prompts method (CCP), strategy designed to bridge the gap between pretrained foundational classification models and downstream detection tasks. This technique uses the surrounding contextual background information to trans- form ROI features into image-like features. Importantly, since current detectors excel at finding unfamiliar objects but have difficulty classifying them accurately. the CCP is integrated only into classification branch existing detectors. This improves their accuracy without affecting localization branch's 's ability to identify numerous unknown targets possible.
Given pre-trained backbone E and a dataset for downstream tasks, we extract features for an image Io Rg [r] r3 ... r where rå (IIo (b))). Notably, M represents the number of region proposals and 6ú signifies the i-th proposal. Our objective to develop region-conditiona. visual prompt x for the i-th detected proposal, more details be elaborated in the ensuing subsection.
Prompt Design. In classification tasks, the visual prompt mechanism (Jia et al. 2022) leams a dataset-specific prompt for each task. But for detection tasks, require mechanism that create a contextual conditional prompt Th for i-th detected proposal. Considering the varying scales and quantities of proposals across different samples, adopt convolutional layers build lightweight

---

Published as a conference paper at ICLR 2024
person
person
person
person
glossess
elothing mouth
mouth
eyes
dathing
two eyes
hair
jeweiry
fieed
mouth
Two eyes
hair
two eyes
head
feotures
face two Jeet
two orms
two
facial features
feet
shoulders
Mps
fingers
hands
hands
chins
two logs
hot
The iterative update of fine-gratd fine-grained descriptors. In the training stage, we continuously generate new fine-grained descriptors (highlighted in blue boxes) via interaction with LLMs. With the recorded usage frequency, high-frequency descriptors (highlighted in green) are preserved and low-frequency descriptors (highlighted in red) are discarded. We can observe that certain fine- grained descriptors such as 'hair', "two eyes', and 'face' are consistently preserved after generation while visually irrelevant descriptors such as 'jewelry' are only generated at early stage and then discarded.
Figure 3:
meta-network, that is adept at processing a variety of object proposals. For each proposal bj (I1, F2, V1, V2), merge the surrounding background information where the background region is defined by by (z1, 222 14, v/)). calculated follows:
x/ = æ1 m. ví 11 -
z/2 x2 + m, 3/2 = 1/2 +
where m and are constants. Next, we extract the features r'd from the expanded region, and the meta-network learns the regional visual prompt To using the formula no he (ro)), and he(-) represents the Meta-Net parameterized by 0. Finally, the learned prompt is combined with the feature ró create detailed prompted feature vi ró Tn¹
3.3 LLMS MEETS VLMS
In this section, we treat the LLMs as interactive implicit knowledge repositories to generate fine- grained descriptors for CCP training. Specifically, we design hierarchical generation mechanism that interacts with LLMs iteratively to generate more diverse and visually relevant category descrip- tions throughout the training process, with more detail to be elaborated in the ensuing subsections.
(Menon & Vondrick] 2023) Descriptors Initialization. We adopt similar input protocol to prompt L.MSs. For each category denoted as G, we extract its fine-grained descriptors De. represented as De [de1,de2. de2.-... dek), accompanied by the corresponding text embeddings T. [tet, t.. .. te where signifies the quantity of fine-grained descriptors. For all cate- gories, we obtain a fine-grt fine-grained descriptor dictionary D [D1, Dz,...... DMl.
Descriptors Record. The fine-grained descriptors are used for the category prediction. Since we cannot guarantee the presence of each descriptor in every sample, we introduce semantic selection strategy for each proposal. The selection function s(c, 1T)) is defined by:
sec,
Σ de RERAR (D.)
b(d, Te)
where, o(d, I') represents the probability of how the descriptor d is relevant to the i-th proposal of image Io. and Ranky selects the top N descriptors based on the value of o(d, Iz). For the i-th proposal, we predict its category label via arg max.ec s(c, It). For each category we record the usage frequency of each descriptor and probability of being misclassified other categories (i.e., the confusiig categories with high misclassification probability).
Descriptors Hierarchy Generation and Update. During the training stage, we generate fine- grained descriptors via hierarchy mechanism intervals of every N iterations. The updating of fine-grained descriptors consists of two core operations. First, we record the usage frequency of the descriptor according to Eq. 3 The high-free high-frequency descriptors descriptors are discarded. Second, we prompt LLMs with an input template that consists of high- frequency descriptors to gather descriptors designed follows:
preserved and the low- www-freuency
5

---

Published as a conference paper at ICLR 2024
Q: There are several useful visual features to tell there is (category name) a photo, including (the first frequency descriptors, the second frequency descriptors, ...).
where I /category name is substituted for given category label c. The generated list then con- stitutes the descriptor dictionary D. Subsequently, we further generate fine-grained descriptors for this category to differentiate from confusing classes. We prompt LLMs with an input template that consists of the confusing categories:
Q: which visual features could be used to distinguish (category name) from some confusing categories including (cenfusing category 1, confusing category 2, confusing category 3, ... in photo?
The newly generated descriptors further expand the descriptor dictionary D. However, some newly generated descriptors d; may already exist in D already. Further, including it into D may lead to the presence of the same descriptor in multiple categories. leading to potential semantic confusion during training. We address this issue by measuring the cosine similarity Sty between and D. If Sij 7, we merge the descriptor's text embedding via tj = at, + (1 a)tz, where is a constant and a is the momentum coefficient.
EXPERIMENTS
4.1 DATASETS
We evaluated DVDet over two widely adopted benchmarks, ie, COCO (Lin et al. 2014) and LVIS (Gupta et al., 2019). For the COCO dataset, we follow OV-RCNN (Zrreian et al., (02T) to split the object categories into 48 base categories and 17 novel categories. As in (Zareian et al., 2021), we keep 107,761 images with base class annotations as the training set and 4,836 images with base and novel class annotations as the validation set. Following (Gu et al.) [021] Zareian ct al., 2021). we report mean Average Precision (mAP) at an IoU of 0.5. For the LVIS dataset, follow ViLD (Gu et al. 2021) to split the 337 rare categories into novel categories and the rest common and frequent categories into base categories (866 categories). Following (Lin et al. 2023), report the mask AP for all categories. For brevity, denote the open- vocabulayy benchmarks based on COCO and LVIS as OV-COCO and OV-LVIS.
4.2 IMPLEMENTATION DETAILS
In our experiments, we employ pre-trained models in prior studies as the base and include our prompt learning techniques on top of them for evaluations. Specifically, we employ the CLIP text encoder to encode both categorical labels and their fine-grained descriptors. In the open-vocabulary COCO experiments, follow the OVR-CNN setting (Zareian al. (021 without any data augmentation and adopt Faster R-CNN (Ren et al., 2015) with ResNet50- C4 (He et al.) 2016) as the backbone. increase the learning rate from 0 to 0.002 for the first 1000 iterations. The
For the warmup, model is trained for 5,000 iterations using SGD optimizer with batch size 8 and the learning rate is scaled down by factor of 10 6000 and 8000 iterations. In open- wwwarouoaablaryyyyoo.o LVIS experiments, follow Detic (Zhou et al., 2022) to adopt CenterNet2 (Zhou et al., (021b) with ResNet50 (He et al., 2016) as backbone. We use large-scale jittering (Ghiasi et al.) 2021) and repeat factor sampling as data augmentation. For the warmup, we increase the learning rate from 0 to 2e-4 for the first 1000 iterations. The model trained for 10,000 iterations using Adam optimizer with batch size All expriments are conducted on 4 NVIDIA V100 GPUs. More details can be found in Appendix.
4.3 OPEN- OPEN-VOCABULARY DETECTION ON COCO
Table shows the performance of different methods on the open- vec-buabyylaryy COCO datasets. can be seen that our method, when incorporated into multiple existing open-vocabulary detectors, achieves stable performance improvements consistently. This indicates that introducing alignment with fine-grained descriptors can effectively enhance the performance of existing open-vocabulary detectors. The baseline method utilizes the pretrained RPN (Zhong et al.) 2022 to extract propos- als and directly feed them into CLIP for classification. We observe that CLIP achieves good accuracy on novel classes, reaffirming its powerful zero-shot capabilities. As a comparison, state- of-the-art OVOD methods experience sharp accuracy drops while handling objects of novel classes,

---

Published as a conference paper at ICLR 2024
Table 1: Open-vocabulary object detection COCO dataset. Including DVDet improves
the state-of-the-art consistently for both base and novel classes. 'Baseline' utilizes the pretrained
RPN (Zhong et al. 2022) extract proposals and directly feeds them into CLIP for classification.
'Novel AP" indicates the zero-shot performance.
Method
Novel AP
Base AP
Overall AP
Baseline
29.7
24.0
25.5
ViLD (Gu et al.
2021
27.6
59.5
51.3
+DVDet
29.3
60.6
52.4
Detic (Zhou et al.
2022)
27.8
51.1
44.9
+DVDet
29.5
53.6
47.3
RegionCLIP (Zhong al.
2022)
26.8
54.8
47.5
+DVDet
28.4
56.6
49.2
VLDet (Lin et al.
2023)
32.0
50.6
45.8
+DVDet
34.6
52.8
48.0
BARON Wu et al. 2023)
33.1
54.8
49.1
+DVDet
35.8
57.0
51.5
Table Open-vocabulary object detection on LVIS dataset using ResNet50 (RN50) (He al. 2016 and Swin-B (Liu et al. 2021b) as backbones. 'Baseline' utilizes the pretrained RPN (Zhong et al. 2022) to extract proposals and directly feeds them into CLIP for classification. mA PPpze indicates the zero-shot performance.
Method
Backbone
PmPak Novel
mAPmask mApmask
mAP
all
Baseline
RN50
11.6
9.6
7.6
9.2
ViLD (Gu al. 2021
RN50
16.6
24.6
30.3
25.5
+DVDet
RN50
18.7
25.8
31.6
27.1
DetPro (Du et al.
2022)
RN50
19.8
25.6
28.9
25.9
+DVDet
RN50
21.3
28.2
31.3
28.1
RegionCLIP (Zhong et al.
2022)
RN50
17.1
27.4
34.0
28.2
+DVDet
RN50
19.1
29.2
35.2
29.6
VLDet (Lin et al.
2023)
RN50
21.7
29.8
34.3
30.1
+DVDet
RN50
23.1
31.2
35.4
31.2
BARON (Wu et al. 2023
RN50
19.2
26.8
29.4
26.5
+DVDet
RN50
21.3
28.7
31.8
28.3
Detic (hhou et
2022
Swin-B
23.9
40.2
42.8
38.4
+DVDet
Swin-B
25.2
41.4
44.6
40.4
VLDet Lin et al.
2023
Swin-B
26.3
39.4
41.9
38.1
+DVDet
Swin-B
27.5
41.8
43.2
40.2
and this applies various existing OVOD approaches that seek broader cross-modal alignment with dataset caption annotations (Lin et al. 2023), introduction of classification datasets (Zhou et al., 2022) and construction of a concept pool (Zhong al. 2022). The accuracy drop clearly highlights the necessity of optimizing the classifier in open vocabulary object detectors. By introducing the alignment with fine-grained textual descriptions of categories, our method serves as general plugin that can complement existing open-vocabulary object detectors consistently and effectively.
4.4 OPEN- VOCABULARY DETECTION ON LVIS
Table shows vocabbocayylary detection on LVIS dataset. Similar to the experiments on COCO dataset, the "Base-only" still performs better on the novel classes. However, its performance gen- erally falls below srat-oof-the-at OVOD methods due to the higher complexity of object detection tasks. Nevertheless, our proposed method complements existing methods consistently, for both ViLD-type methods and prompt-based methods such DetPro (Du et al.) (022). This shows that compared to the current strategies involving prompts in text encoders, our approach serves as an effective complement. We further examine the generalization of our method by adopting Swin-B as the backbone. Experiments show that incorporating our method into Detic and VLDet improves the accuracy by around 1.3% novel categories consistently.

---

Published as a conference paper at ICLR 2024
Orange
Suitcase
Backpack
ose
prange
person
hair
zipper closure
Truck
Bus
Chair
Couch
truck
bus
large front windshield
upholstered long beach
Figure 4: Introducing our fine-grained text descriptors (shown at the bottom of each sample) im- proves the open- vocabulary detection consistently especially under challenging scenarios with dis- tant or occluded objects, small inter-class variations, etc. For each of the four sample images, the red-color class names at the top-left corner of the first image are predictions without our method, and the green-color class names in the second image are predictions after including our method. The red/green boxes within the sample images show related detection. Close-up view for details.
Table 3: Ablation studies of our designs in DVDet on the OV-COCO benchmark with VLDet and RegionCLIP as two base networks. We use two input 'Templates' to interact with LLMs to obtain fine-grained descriptors. 'Template H' includes high-frequency descriptors, and "Template C' in- cludes confusing categories, 'Prompt' denotes the proposed conditional context region prompt.
Fine-grainee Descriptors
VL.Det
RegionCIIIP
Prompt
Template H
Templete C
mAP novel
mAP base
mAP novel
mAP base
32.0
50.6
26.8
54.8
29.7
48.7
24.6
51.0
33.1
51.2
27.3
55.2
34.0
52.2
28.0
56.2
33.8
52.3
27.6
55.9
34.6
52.8
28.4
56.6
Visualization. We show how introducing fine-grained descriptors improves the open-vocabulary detection qualitatively. As Fig. 4 shows, including our design improves the detection significantly while facing challenging scenarios with distant occluded objects, small inter-class variations, etc. With fine-grained descriptors such as hair, zippers, and large glass front windshield, our model can better align with the text space and enable more accurate recognition and understanding while handling novel classes. In Fig. [ we further show how our model progressively aligns targets with relevant descriptors (e.g.. the word 'school bus' the vehicle) of new categories (along the training process), thereby reducing ambiguity and misclassifications (airplane _ car - bus) effectively. More visualization results can found Appendix.
4.5 ABALTION STUDIES
In this section, we conduct ablation studies on OV-COCO benchmark using VLDet (Lin et al. 2022ab and RegionCLIP (Zhong et 2022) two base networks, respectively.
Component Analysis. We examine the effectiveness of different components in DVDet on the OV- COCO benchmark. As Table shows, the contribution of fine-grained categorical descriptors is compromised clearly at the absence of prompt learning (i.e., Prompt), and this is well aligned with the statistical data in Fig. []] In addition, incorporating the prompting with fine-grained descrip- tors improves the detection performance significantly, substantiating the benefits of fine-grained descriptors to visual-textual alignment. Besides, we adopt two templates to interact with L.M to generate fine-grained descriptors. The template using the confusing categories outperforms that us- ing high-frequency descriptors slightly, largely due to the synergy of the semantic selection with the frequency-based filtering mechanism that plays critical role in filtering out irrelevant descriptors and partially ensures the reliability of the overall descriptor pool in training. Further, DVDet out- performs other variants consistently, demonstrating the synergy of merging fine grained descriptors generated from different templates with prompt learning which enriches the training with a more comprehensive understanding of object categories.
8

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study interaction strategies with Knowledge Base (LLMS).
VLDet
RegionCLIP
Interaction Strategy
mAP novel novel
mAP base
mApmus novel
mAPnaak
base
Static Knowledge Base Interactive Knowledge Base
33.2
51.9
27.6
55.1
34.6
52.8
28.4
56.6
Dnlyy Category label
Combined with Dynamic Category Descriptors batching
Airplane
Car
Bus
Bus
bumper
rear windshield
yellow, large vehicle
the words "school bus" written the vehicle
the words "school bus" written on the vahicle
Figure 5: Object detection is improved progressively with iterative extraction of fine-grained de- scriptors from LLMs and matching them with the detected target. Texts at the top-left corner each sample show recognized classes, and texts at the bottom show extracted fine-grained descriptors.
Transfer To Other Datasets. To ascertain the generalization of the proposed DVDet, we ap- ply our COCO-trained model to the test set of PASCAL VOC (Everingham et al., 2010) and the validation set of LVIS with little ad- ditional training. This is achieved by utilizing the context conditional prompts from the OV- COCO and modifying the class embeddings of the classifier head accordingly. PASCAL VOC contains 20 object categories including 9 absent in COCO, thereby presenting a notable chal-
Table 4: Transfer to other datasets. We evaluated COCO-trained model on PASCAL VOC (Ever- iinhham et al., 2010) test and LVIS validation set without re-training. We report mAP at an IoU of 0.5.
Method VLDet +DVDet RegionCLIPP
PASCAL VOC
LVIS
61.7
10.0
64.0
12.1
46.9
6.1
+DVDet
48.2
7.8
lenge while transferring models without aids from any supplementary training images, not to men- tion the inherent domain gap. LVIS dataset boasts a substantial catalogue of 1203 object categories, vastly exceeding the label space in COCO. Despite these challenges, DVDet demonstrates remark- able effectiveness across diverse image domains and language vocabularies, as evidenced by 2.3% and 2.1% improvements that though many LVIS category names are absent in COCO, DVDet succeeds in learning close descriptors and thereby facilitating smoother transition while adapting to the LVIS benchmark.
the two datasets shown in Table 4 It should be highlighted
Effectiveness of Interactive Knowledge Base. The successful creation of fine- rreegga descriptors depends on iterative interaction with LLMs in training. We validate this by comparing with one- time interaction, where LLMs act like a static knowledge base. Specifically, we employ confident samples from CAF (Menon & Vondrick 2023) to obtain visual-related fine-grained descriptors. As Table shows, the iterative interaction (34.6 AP and 28.4 AP for novel classes) outperforms the static method (33.2 AP and 27.6 AP) clearly. This reconfirms that the dynamic interaction allows LLMMs to better understand the detector's requirements, offering more trustworthy descriptors.
5 CONCLUSIONS
This paper presents DVDet, innovative open vocabulary detection approach that introduces fine- grained descriptor for better region-text alignment and open-vocabulary detection. DVDet consists of two key designs. The first is Conditional Context regional Prompt (CCP), which ingeniously transforms region embeddings into image-like representations by merging contextual background information, enabling CCP to seamlessly integrated into open vocabulary detection with little ex- tra designs. The second is a hierarchical descriptor generation that iteratively interacts with LLMs to mine and refine fine-grained descriptors according to their performance in prompt training. Without any resource-intensive grounding annotations, DVDet coordinates LLMs-assisted descriptor gener- ation and VLM L.Mu-gdddd prompt training effectively. Extensive experiments show that DVDet im- proves the performance of existing open vocabulary detectors consistently. Moving forwards, plan to investigate the synergy between powerful foundational models including LLMs and VLMs, for various open vocabulary dense prediction tasks.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, well cash and in-kind contributions from the industry partner(s).
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 2022a.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modi- fying pixel space adapt pre-trained models. arXiv preprint arXiv:2203.17274, 3:11-12, 2022b.
Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open- vocabulary detection. Advances in Neural Information Processing Systems, 35:33781-33794, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-detr: A versatile architecture for instance-wise vision-language tasks. arXiv preprint arXiv:2204.0566 2022.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision- laiiunl model. arXiv preprint arXiv:2203. /4940, 2022.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open-vocabulary detection using uncurated images. In Euro- pean Conference on Computer Vision, pp. 701-717. Springer, 2022.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yr Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2021.
Xiuye Gu, Tsung- Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.139211 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmentation via robust cross-modal pseudo-labeling, arXiv preprint arXiv:2111.12698, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
10

---

Published as a conference paper at ICLR 2024
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang. and Weidi Xie. Prompting visual-language models for efficient video understanding. In European Conference on Computer Vision, pp. 105-124. Springer, 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-moda understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 1780-1790, 2021.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre- training, arXiv preprint arXiv 21t.::::2122288800 12.03877 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1Xxv:2101.00199, 2021.
Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. arXiv preprint 0rXiv:2211.14843 2022a.
Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. ICLR, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. 2014.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Con- ference Computer Vision, pp. 388-404 404-404. Springer, 2022b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:21 10.00660,, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021b.
Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 14074-14083, 2022.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. ICLR, 2023.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple open-vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.06230 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. arXiv preprint arXiv:21 2021.
11

---

Published as a conference paper at ICLR 2024
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pp. 91-99,
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15254-15265 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432. 2021.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14393-14402 14402, 2021.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. arXiv preprint
2021.
Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few- shot learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15211-1522. 2023.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 16793-16803, 2022.
Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXiv:2112.01071, 2021a.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint
2021b.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. arXiv preprint arXiv:2201.02605. 2022.
12


Reviews:
[{"review_id": "1gUHy3QDKW", "reviewer": "Reviewer_qAZ8", "summary": "The submission introduces Descriptor-Enhanced Open Vocabulary Detection (DVDet), a method that integrates fine-grained descriptors from Vision Language Models (VLMs) into open vocabulary object detection (OVOD) using a Conditional Context visual Prompt (CCP). This approach utilizes large language models to generate descriptors without extra annotations and features a hierarchical update mechanism for descriptor refinement. The paper claims improvements in OVOD tasks through extensive experiments and presents three main contributions: a new feature-level visual prompt, an update mechanism for descriptor management, and empirical evidence of enhanced detection performance.", "questions": "1. How does the descriptors generation process affects the training efficiency?\n2. Can the authors provide insights into any observed trade-offs between the complexity of the Descriptor-Enhanced Open Vocabulary Detection (DVDet) and its performance gains?\n2. How does the system perform under varying conditions, such as different object scales, occlusions, and lighting? Are there specific scenarios where the performance improvement is more pronounced?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Introduction of Descriptor-Enhanced Open Vocabulary Detection (DVDet) could address existing granularity challenges in object detection, making it a potentially transformative method.\n+ Leveraging large language models for descriptor generation without additional annotations presents a cost-effective solution.", "weaknesses": "- The paper does not sufficiently address the analysis of descriptors produced by Large Language Models, which is crucial for understanding the quality and relevance of the generated descriptors in enhancing object detection.\n- The demonstrated enhancements (~2% on average) in object detection performance, while positive, do not represent a substantial leap forward when considering the benchmarks established by current leading methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UU6D7dFeFu", "reviewer": "Reviewer_AtsG", "summary": "This paper proposes to a novel open vocabulary detection method which uses a fine-grained descriptor for a better region-text alignment and open vocabulary detection. It consists of two parts. First part is to transform region embeddings to image-like representation, second part is to use LLM  to generate fine-grained descriptors. The method improves open vocabulary detection consistently.", "questions": "please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method proposed in the paper looks interesting by using LLM as a better fine-grained descriptor. The idea is reasonable and the presentation is good.\n2. The progress from iterative extraction of fine-grained LLM descriptor is interesting and looks interesting to me.\n3. The results of experiments are good and the ablation study is convincing to me.", "weaknesses": "1. when LLM interacts with VLM, Could you give more examples of prompts to use and how the prompt may influence the final results?\n2. when iterative updating of LLM descriptor, It seems to just give a general description of objects. Then what will happen if we just let LLM do some general description of objects without seeing the objects?\n3. if the regional prompt is not accurate, how will the performance look like when interacting with LLM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hMuS0pLZX", "reviewer": "Reviewer_shSW", "summary": "This paper proposes an approach to boost open-vocabulary object detector with fine-grained language descriptors of the categories. The motivation is that existing works learn to align region embeddings with category labels only, disregarding the capability of VLM to align with object parts and other fine-grained descriptions. The fine-grained text descriptors are generated with an LLM in an interactive manner. In addition, the paper introduces conditional context prompt to augment region embeddings with contextual cues and make them more image-like for open-vocabulary detection.", "questions": "See weaknesses. Point 1-3 are more important in my view.\n\nWhat's the variance of the proposed method on LVIS open-vocabulary benchmark over e.g. 3 or 5 independent runs?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The two approaches presented by this paper are intuitive and effective. The context prompt makes the region more image-like to aid open-vocabulary recognition and the use of LLM provides additional features for VLM to reliably detect novel classes. The main results in Table 1 and 2 show the effectiveness of proposed approaches on LVIS and COCO benchmarks based on many existing methods. Ablations show that each part is effective on OV-COCO.", "weaknesses": "1. The idea of using LLM to generate more textual description has been explored in recent/concurrent work [2]. It'd be great to see a comparison with the existing ideas. Similarly, the idea of context regional prompt has been explored in recent/concurrent work [1], although the exact instantiation may differ. It'd be helpful to tease apart the contribution of context (which makes the crop more image-like) vs prompting (which adapts the features for detection use cases) through an ablation, and compare/discuss the similarity/differences with existing work. I understand the references listed are very recent/concurrent works, but I think it'd still be valuable to have some comparison with them for scientific understanding.\n\n2. Although DVDet shows gains for all methods in Table 1 and 2. The gains on base categories are non-trivial and even comparable to the novel categories in most cases. For example, DVDet + Detic has +1.8 boost on mAPf and +1.3 on mAPr, and +2.5 on Base AP vs +1.7 on Novel AP. Given the method is based on Detic, I'm wondering what the implication of the gains on base vs novel categories are, since the method is primarily designed for open-vocabulary detection. \n\n3. Table 3 (row 2) shows that adding fine-grained descriptors by itself without prompting hurts the performance of the model. This seems counter-intuitive to me. It'd be helpful to see if the same observation holds for other OVD models e.g. ViLD or Detic.\n\n4. Table 4 shows transfer detection from COCO to PASCAL and LVIS. It's nice to see a clear boost there. I'd recommend trying out the LVIS-trained model on Objects365 instead since it's a more commonly used transfer detection setting by e.g. ViLD, DetPro, F-VLM. \n\n5. In Eq (2), the $\\textit{m, n}$ are said to be constants. How are they set? I'm wondering if they should be set proportional to the width and height of the ROI. Another option is to use the whole image box as a baseline and see how that performs. Some ablations/analysis on the choice of context would be interesting.\n\nReferences:\n1. CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching (CVPR 2023)\n2. Multi-Modal Classifiers for Open-Vocabulary Object Detection (ICML 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qm2XRQAr8D", "reviewer": "Reviewer_6yi2", "summary": "This paper introduces DVDet, a Descriptor-Enhanced Open Vocabulary Detector featuring two innovative components: conditional context prompts and hierarchical textual descriptors. The conditional context prompts function in the Region of Interest (RoI) align to expand each region proposal's area, incorporating more context information and converting it into an image-like feature preferred by CLIP. Meanwhile, the hierarchical textual descriptors dynamically acquire fine-grained descriptors for each category through interaction with the Language Model (LLM) during training. Extensive experiments indicate that DVDet significantly enhances the performance of various Open-Vocabulary Detectors on two benchmarks, MS COCO and LVIS.", "questions": "1. How about the extra computation overhead?\n\n2. LLM may generate different prompts. Does it significantly affect the performance?\n\n3. How about different $m$ in Conditional Context Prompt?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. I greatly value the use of LLM to dynamically generate detailed descriptors of categories. This approach inspires a novel method of employing LLM to enhance the performance of OVD tasks. \n\n2. The concept presented is both straightforward and potent, with the paper being lucid and easy to navigate. The author's introduction of Framework Figures (Fig 2) and the examples of iterative fine-grained descriptors (Figure 3) are commendable as they significantly aid the reader in grasping the idea more effectively.\n\n3. The experiments conducted are thoroughly robust. First, the author meticulously scrutinizes each component of their methodology, as detailed in Table 3 and Table 9. Second, the author conducts comprehensive experiments on a variety of OVD detectors and datasets to demonstrate that DVDet significantly enhances performance.", "weaknesses": "1. A comparison of computation costs with the baseline has not been conducted.\n\n2. The contribution of the technique is somewhat restricted.\n\n3. The prompt generated by LLM may possess an element of randomness, potentially making it challenging to reproduce.\n\n4. No ablation experiment about the hyper-parameter $m$ of Conditional Context Prompt", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNKzZEFz5j", "reviewer": "Reviewer_E6zT", "summary": "This paper proposes Descriptor-Enhanced Open-Vocabulary Detector (DVDet) which introduces a conditional visual (region) prompting and textual descriptor dictionaries by using Large Language Models. The use of textual descriptors is motivated by the observation that the VLMs are good at capturing the fine-grained attributes of the objects. The textual descriptor dictionary is dynamically updated during training by tracking how frequent each descriptors are used, and by prompting what visual descriptors can help distinguishing the confusing categories. In addition, this work proposes a conditional region prompt to help the alignment between the detected region and text embeddings. When using both the textual descriptors and conditional visual prompts, DVDet improves the modern open-vocabulary detectors on the OV-COCO and OV-LVIS benchmarks.", "questions": "* The gain in the base vs novel categories are similar (e.g., + 2.7 novel AP and +2.2 base AP in Table 1). This leads to a question whether the DVDet method is beneficial in standard detection instead of open-vocabulary detection. That is, the proposed method benefits from augmenting the vocabulary description (text embeddings) in general, not being specifically helpful for the novel categories. What is the motivation of evaluating on open-vocabulary detection? How would DVDet perform in the standard detection setting?\n\n* Please compare with CORA (CVPR 2023) which also uses visual region prompting for open-vocabulary detection.\n\n* Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The use of Large Language Models to enrich the vocabulary description is well motivated.\n\n* The proposed conditional visual prompt is useful in bridging the gap between the image-text and region-text alignment.\n\n* The proposed DVDet method can be plugged into other SOTA open-vocabulary detection methods and consistently improve the performance.", "weaknesses": "* The use of region-level prompting for open-vocabulary detection is previously proposed in CVPR 2023 paper \"CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching\". The comparison (both at a idea level and a numeric comparison) with CORA work seems necessary regarding the technical similarity.\n\n* It is not clear what the optimal number of iterations is for the dictionary update. Is \"the number of high-frequency descriptors\" in Table 7 the number of iterations?\n\n* What is the effect of using the LLM on the running speed of the DVDet detector (using no LLM vs N-th interaction of LLM dictionary)? While the running speed might not be the main focus of open-vocabulary detection research, the multiple iterations of LLM use could make the speed not very favorable.\n\n* There are non-negligible number of hyper parameters (number of descriptors, number of update iterations, thresholds).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)