DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection
Authors: Jia Syuen Lim
Zhuoxiao Chen
Zhi Chen
Mahsa Baktashmotlagh
Xin Yu
Zi Huang
Yadan Luo
Venue: NeurIPS 2024 poster (2024)
Keywords: class-agnostic object detection
VLM
prompting
out-of-distribution object detection
Primary area: machine_vision

Abstract: Class-agnostic object detection (OD) can be a cornerstone or a bottleneck for many downstream vision tasks. Despite considerable advancements in bottom-up and multi-object discovery methods that leverage basic visual cues to identify salient objects, consistently achieving a high recall rate remains difficult due to the diversity of object types and their contextual complexity. In this work, we investigate using vision-language models (VLMs) to enhance object detection via a self-supervised prompt learning strategy. Our initial findings indicate that manually crafted text queries often result in undetected objects, primarily because detection confidence diminishes when the query words exhibit semantic overlap. To address this, we propose a Dispersing Prompt Expansion (DiPEx) approach. DiPEx progressively learns to expand a set of distinct, non-overlapping hyperspherical prompts to enhance recall rates, thereby improving performance in downstream tasks such as out-of-distribution OD. Specifically, DiPEx initiates the process by self-training generic parent prompts and selecting the one with the highest semantic uncertainty for further expansion. The resulting child prompts are expected to inherit semantics from their parent prompts while capturing more fine-grained semantics. We apply dispersion losses to ensure high inter-class discrepancy among child prompts while preserving semantic consistency between parent-child prompt pairs. To prevent excessive growth of the prompt sets, we utilize the maximum angular coverage (MAC) of the semantic space as a criterion for early termination. We demonstrate the effectiveness of DiPEx through extensive class-agnostic OD and OOD-OD experiments on MS-COCO and LVIS, surpassing other prompting methods by up to 20.1% in AR and achieving a 21.3% AP improvement over SAM.

DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection
Jia Syuen Lim" Zhuoxiao Chen' Mahsa Baktashmotlagh
Zhi Chen Xin Yu Zi Huang Yadan Luo
The University of Queensland [jissyuen lim, zhuoxiao chen, m. m.baktashmotlaag laggh /ug .dq..eduuaae {zhi. chen, xin.yu, helen. huang, y. y.luo Euq edu.au
Abstract
Class-agnostir object detection (OD) can be a cornerstone a bottleneck for many downstream vision tasks. Despite considerable advancements multi-object discovery methods that leverage basic visual cues to identify salient objects, consistently achieving high recall rate remains difficult due to the diversity of object types and their contextual complexity. In this work, we investigate using vision-language models (VLMs) to enhance object detection via a self-supervised prompt learning strategy. Our initial findings indicate that manually crafted text queries often result in undetected objects, primarily because detection confidence diminishes when the query words exhibit semantic overlap. To address this, we propose a Dispersing Prompt Expansion (DiPEx) approach. DiPEx progressively learns to expand set of distinct, overrlaaag hyperspherical prompts enhance recall rates, thereby improving performance in downstream tasks such as out-of-distribution OD. Specifically, DiPEx initiates the process by self-training generic parent prompts and selecting the one with the highest semantic uncertainty for further expansion. The resulting child prompts expected inherit semantics from their parent prompts while capturing more fine-grained semantics. We apply dispersion losses ensure high inter-class discrepancy among child prompts while preserving semantic consistency between parent-child prompt pairs. To prevent excessive growth of the prompt sets, we utilize the maximum angular coverage (MAC) of the semantic space as criterion for early termination. We demonstrate the effectiveness of DiPEx through extensive class-agnostic OD and OOD-OD experiments on MS-COCO and LVIS. surpassing other prompting methods by up 20. in AR and achieving a 21 .3% AP improvement over SAM. The code available at httpp://gthhubbooo//ssoooo//////
nottom-up and
Introduction
In real-world applications, the class of interest may constantly change, prompting the need for new
tasks like nut-of-distribuute (OOD) detection [53 [I]. open-world detection [60] [2 [2 62 [55 and open-vocabulat 48 54 31 object detection (OD) ensure reliable operation of detectors. A significant bottleneck in these OD tasks is the ability to locate all objects in scene typically
referred to as class-agnostic OD 36 Ensuring a high recall rate essential in this task as lays the foundation for correctly classifying objects, thereby improving the average precision for classes of interest. Conversely, a low recall implies that some objects will be missed entirely, negatively impacting downstream recognition tasks.
'Equal Contribution
Corresponning Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Clocc-geende Proept
Text Encodor
Class-Agnostic Detection
0.75
0.900
G-DINO
0.25
0.00
Vision Encoder
G-DING
0.25
Va Va
Text Encoder
aratiic
Downstream OOD Delection
(a)
OD and Downstream OOD-OD
(b) UNIVERSAL and CLASS-WIDE Queries
Figure I: (a) An exemplar of the studied class-agnostic OD and downstream OOD-OD tasks. (B)
Zero-shot class-agnostic OD performance of Grounding DINO [33 on MS-COCO 32 with the hand-crafted UNIVERSAL query from ChatGPT and CLASS- query from WordNet
Conventional solutions to the under-explored class-agnostic OD task often rely on bottom-up strategies
47 61 40 41) such as selective search 447 or EdgeBox [19], which generate a large ranked set of
class-agnostic proposals based on low-level visual cues. To address the low precision and scalability issues of these approaches, another line of research has explored multi-object discovery by leveraging (sell)-supervieed features from vision transformers ( (ViT) (e.g. DINO [3), MoCo-v2 [. SWAV (3), or external motion information to support region proposal regression. However, these methods still fall short, achieving only about 30% average recall (AR) on benchmark datasets like MS-COCO due to the lack of intrinsic knowledge about wide range of objects. The newly released aision-languan models (VLMs) such as Grounding DINO 63), GLIP (29). T-Rex2 [21]. which are pretrained on large-scale grounding datasets. have opened up new opportunities for acquiring common knowledge for generic object localization. VLMs have demonstrated impressive zero-shot recognition capacities given the provided textual prompt. However, to effectively locate all objects, one would need to input all class names accurately, which is impractical in real-world applications.
To better understand the limitation of modern VLMs in generic object localization, investigated the design of hann-craffed text queries (Section 2 to enhance detection recall through two approaches: (1) We employed UNIVERSAL query, using ChatGPT to generate 13 types of broad nouns and adjectives (e.g.. "oojects"., "generic") as queries for the Grounding DINO model, aiming to detect a wide array of objects without focusing specific categories; (2) We implemented CLASS-WIDE query, selecting 25 high-level semantic words (e.g., "plant", "animal") from the top layer of the WordNet hierarchy (also used for the Image vocabulary) to cover extensive object categories. Our findings, depicted in Figure [nd Table reveal that while VLMs can generalize across universal object categories, combining all queries into one string significantly reduces detection performance (by up to 52% in AR) due to the "semantic overlap" among words. This suggests that optimal detection requires conducting multiple separate inferences, presenting substantial computational demands for large datasets.
To overcome the aforementioned limitations, propose novel self self-supervised Dispersing Prompt Expansion (DiPEx) strategy. This approach progressively expands a set of non-overlapping hyper- spherical prompts for capturing all objects in given dataset, thereby benefiting downstream tasks such as out-of-distribution object detection. Specifically, we start with generic parent prompt that is self-supervised using the UNIVERSAL and CLASS--WDDS text queries. To capture more fine-grained semantics, we split the parent prompts with high semantic uncertainty into a set of distinct child prompts. We initialize child prompts by diversifying the parent token embedding, randomly rotating it to different angles on the hypersphere to yield a range of unique prompts. Dispersion losses are employed to minimize semantic overlap among child prompts while maintaining semantic consistency across parent-chile prompt pairs. To prevent excessive growth of the prompt sets, we estimate the maximum angular coverage (MAC) of the semantic space as a criterion to terminate the prompt expansion process, balancing semantic richness and computational overhead. Extensive experiments on the MS-COCO and LVIS datasets verify the effectiveness and versatility of the proposed DiPEx strategy. With single pass of inference, DiPEx can achieve by up to 20.1 1% improvements in average recall (particularly 35.2% for small objects) and outperforms segment anything model (SAM) 26 by 21.3% in average precision.
2

---

Table 1: Zero-shot class-agnostic object detection performance of Grounding DINO [33] on MS-
COCO (32, with cand-crafted prompts from various sources. We report average recall (AR) and precision (AP) limited to maximum of 100 detections per image. AAR quantifies the percentage decrease in AR comparing "query-merging" to "prediction-merging" for forming mulli-word queries.
Word Source
Merging Strategy
AR
AAR
AR@S
AR@M
AR@L
AP
query-merging 0.345
0.122
0,360
0.718
0.067
ChatGPT 38
-52.46%
meeriction-merging
0.526
0.317
0.606
0.781
0.274
query-merging 0.461
0.234
0.522
0.774
0,229
WordNet 4
-23.64%
prediction-mergingg 0,570
0,382
0.646
0.796
0,344
query-merging 0.408
0.162
0,471
0.751
0.121
ChatGPT
38 + WordNet
14
-44.36%
prediction-merging 0.589
0.410
0.665
0.798
0.353
Related Study. The full discussions can be found in Section A. Traditional bottom-up approaches for region proposal generation, such as those by [47 and (27]. often face precision constraints despite high recall rates, limiting their scalability. Recent advancements in Vision Transformers (ViTs) massive datasets, extracting semantically meaningful features. Methods like LOST [45] and TokenCut [51] use graph-based based techniques but are limited to detecting single object per image. MOST
by 4 and [10] have enabled self-supervisee learning
[43 addresses this with entropy- -bused box analysis but struggles with generalization. MAVL [36 uses a late fusion strategy with text queries, requiring full supervision and multiple inferences. Our approach eliminates the need for labels and achieves state-of-the-an performance with one-pass inference using non-overlapping prompts. Vision- Language Models (VLMs), like those by [42 and (20). have shown potential in learning generic concepts. HierKD 35 and OV-DETR [58] align image representations with captions and extend
DETR to open- vocabulary settings. GLIP [29] Grounding DINO [33] and T-Rex2 (21) integrate
object detection and visual grounding. However, VLMs' effectiveness depends on textual cues, and
prompt tuning, as introduced by CoOp 24 and improved by CoCoOp and MaPLe (25), offers
solution by optimizing soft prompts while keeping the model's 's parameters frozen. ProDA [3] learns diverse prompts using a Gaussian model. DFKD-VLFM (56) and PromptStyler 7 attempted to
diversify a fixed number of prompts through contrastive approach. Despite these advancements, full supervision is typically required. UPI. [7] and POUF [46] introduced unsupervised prompt learning, but adaptation for object detection remains limited. DiPEx is the first to apply prompt learning to class-agnostic object detection through a progressive self-training approach.
2 Pilot Study
In this section, we detail our preliminary exploration of the zero-shot detection capabilities using state-of-the-art VL.M, Grounding DINO 33). to detect all objects irrespective of the associated classes on the MS-COCO dataset [32 as illustrated in Figure Tb We conduct experiments using two types of text queries: UNIVERSAL queries generated by ChatGPT for general object detection, and CLASS- WIDE queries derived from WordNet, representing broad object categories. Our experiments reveal that semantic overlap between text queries impacts detection performance. To support this hypothesis, we conduct a case study showing that similar concatenated prompts reduce the model's detection confidence.
2.1 Hand-crafted Queries for Class-agnostic Object Detection
UNIVERSAL Query. We employ ChatGPT to generate 13 synonyms of universal concepts, including nouns and adjectives, which are displayed as x-axis labels. The zero-shot object detection results, measured by average recall (AR) and precision (AP) across the top 100 confident boxes for each query text, presented. The plot reveals that more general terms such "generic" and "items" yield the highest AR. Surprisingly, more specific descriptors like "foreground", "small", or "tiny" tend to reduce AR and do not effectively aid in identifying foreground or small objects.
CLASS-WIDE Query. We utilize 25 semantically independent beginner words (listed as x-axis labels in the bottom figure) from the highest level of the WordNet hierarchy 14 as class-wide text queries. variation in AR (0.26~-0.43) is observed with different textual queries from WordNet, with a mean AR of 0.35. Compared to the mean AR of 0.37 across class- alasstggoostic queries generated by ChatGPT, the zero-shot detection ability remains similar, regardless of the types of queries used.
Discussion on Multi Word Queries. The zero-shot results presented in Figure [b were obtained using single-word prompts for the Grounding DINO. To explore whether combining multiple words
3

---

Text Query: "plates." "
Text Query: "dishes."
Text Query: "plates. dishes."
dishes
Confidence Decrease
plates
cup
9 ""platest,isshhr 6 (cuptas""cupp)
Confidence Remain
Text Query: "plates."
Text Query: "cup." Text Query: "plates. cup."
Figure 2: A case study investigating the impact of semantic overlap between text queries the detection confidence of the pre-trained Grounding DINO [33]. Semantic overlaps are quantified by the angular distance, denoted as O between tokenized embeddings of word pairs using BERT 9
as prompts from a given source (e.g., WordNet) could improve zero-shot detection performance, we developed strategies for merging at both the input stage (query-merging) and the output stage (prediction- -eeriintonnmerging) as shown in Table [ The querinmerging strategy concatenates all input text queries (e.g., "foreground, elements to obtain detections. The prediction-men peddiction-merging strategy, on the other hand, uses each text query individually for separate inference and then combines all box predictions. [able|[shows that applying query- meering to UNIVERSAL words results in 52.46% reduction in AR compared to prediction- merging, whereas CLASS-WIDE queries (e.g., from WordNet) achieve smaller decrease AR of only 23.64%. These findings suggest that large semantic overlaps in concatenated queries (e.g., "stuff", "objects" and "item" from ChatGPT) may greatly contribute to diminished object detection performance. To further investigate this phenomenon, conducted a case study analyzing the impact of semantic overlap on detection performance, which is presented in the following section.
tiny. objects .") and performs a single-pass inference
2.2 Confidence Diminishing when Text Query Semantically Overlap
To verify our hypothesis, we conduct case study to demonstrate how semantic overlap in multi-word query leads to diminished detection confidence. We quantify semantic overlap by calculating the angular distance between pairs of textual token embeddings generated by BERT [9] As shown in Figure 2, a small angular distance of 53.73" between the text tokens "plates" and "dishes" diminishes the model's confidence. Consequently, some boxes that could be precisely localized with high confidence using the single token "plates" are omitted. In contrast, concatenating two text tokens with a larger angular distance (e.g., 60.99* between "plates" and "cup'') maintained high detection confidence. This combination resulted in bounding box predictions that encompassed all boxes predicted with each individual token ('plates' or 'cup''.. This case study supports our hypothesis that semantic overlap between concatenated text queries can interfere with the detection confidence of the model. Therefore, we propose that developing method to learn set of semantically non-overlapping prompts for the target dataset could enable efficient object localization with one-pass inference using VLMs.
3 Proposed Approach
In this section, we first mathematically formulate the task of class-agnostic detection using general VLM and, without loss of generality, illustrate the process using Grounding DINO [33 as an exemplar model. We detail the steps of the proposed dispersing prompt expansion in Section 3.2 followed by the early termination strategy the prompt set growth.
3.1 Problem Formulation
Class-agnostic OD. Let I denote the input image and T the associated text query. For the zero- shot object detection class-agnostic setting, we consider the text query T to be of the form of "a photo of {class)", where the class token {class} is sampled from our predefined UNIVERSAL. (e.g., "objects") or CLASS-WIDE (e.g. "plant") sets as described in Section 2 The text
query is then tokenized and projected into word embeddings P {V1, V2,..2..... .2...., V.r. c). where

---

dim3
V2.1
Vo
VO
m
Vi+12
V1,1
V
V
Vi
V1.3
dim1
Vi
V211
V222
2B
V1+1,1
dim2
L-Layer Prompt Expansion
Child Prompt Initialization
Maximum Angular Coverage
Figure 3: An illustration of the ① proposed prompt expansion strategy that selectively grows set of child prompts for the highlighted parent prompt across iterations; @ diversifying initialized embeddings of the child prompt on a hypersphere and ( quantifying maximum angular coverage O'max for early termination of the prompt growth.
V= ( E Rxd indicates a set of M contextual embeddings and e the query text embedding. Here, d indicates the dimensions of learnable tokens. The visual embeddings E, extracted from the visual encoder and prompt embeddings P are fused jointly to prompt the VLM and generate the final bounding box predictions O J(E.,, P) E IB-ax4 with ʃ being the VLM, and NB being the number of predicted boxes. Formally, the objective class-agnostic OD to ensure that the generated bounding boxes can capture any objects comprehensively possible.
Adapt Prompt Tuning for Class-agnostic OD, Instead of relying on hand-crafted templates, prompt tuning approaches like CoOp [24) and CoCoOp 23 originally developed for classification tasks, aim learn the context embeddings v with a frozen VLM using supervised contrastive learning loss. To adapt these prompt learning approaches to the Grounding DINO (3 detection framework, we first construct a pseudo label set Dest. from the zero-shot detection results with UNIVERSAL AL and LLASS-WIDE text queries (see Section A.3 for details). The prompt learning is then supervised by the standard box regression loss Lhox Lgion and focal classification loss Lets as implemented in (33
3.2 Dispersing Prompt Expansion (DiPEx)
Unlike previous prompt tuning approaches, the proposed DiPEx strategy aims to iteratively grow set of learnable prompts P {PI, . in tree hierarchy of depth L. To maximize the utility of prompts and ensure minimal semantic overlap among them, we assume v resides on the surface of unit-hypersphere, i.e., l|vell2 = 1. This assumption transforms the overlap minimization problem into maximizing the angular distances among the learned prompts. In the initial round, we set a single learnable parent prompt P1 {v). which is self- trained using DPSL with the same procedure outlined above. In each subsequent round for I E [1, L), we identify the parent prompts of highest uncertainty and grow K child prompts P E R Kxd from it. The learned v is then frozen and stored in a parent queue P parent- Prompt growth is terminated when the maximum angular coverage O'max exceeds a certain threshold T..
Child Prompt Initialization. Continuing from the previous discussion, we now describe the process of child prompt initialization, which aims to inherit the semantics from parent prompts while capturing more fine-gre ine-grained semantics. After the 1-th round of training, we expand the parent prompt with the highest uncertainty, denoted as v² Pr. into a set of learnable child prompts (Figure 33. We empirically adopt the logit activation frequency of the prompts as a measure of uncertainty, visualized in Figure 6 The rationale that if a prompt is activated for most samples, covers overly broad semantics (e.g., animals) and may need to be decomposed into narrower categories (e.g., cats and dogs). To disentangle the complex semantic of Pf, we set up K child prompts P++1 = [V1+1.*) 1K for the selected parent prompt vj. To diversify the initialized embedding for each child prompt, we introduce K random angular offsets θ = { to rotate v on the hypersphere by different angles Og ~ [-0, 0. Given that vf a d-dim vector, we randomly sample two axes and j where i.j [1, d) for rotation. The k-th child prompt embedding VI+1. is then obtained by applying the
5

---

corresponding rotation matrix Hg E Raxd which are defined as follows:
0
cos Og sin
0g
VI+1k Ri
Rg
(1)
sin 8k cos OR.
Here, the non-identity elements are placed the intersections the i-th and j-th rows and columns, corresponding to the plane of rotation, as illustrated by the grey ellipses in Figure [3 As the initialized embeddings of the child prompts are diversified while maintaining consistency with the central parent embedding (red dot), this leads to varying detection results. This enriched prediction diversity allows us facilitate online self-training, where we adopt the predictions with the highest confidence as pseudo labels for each child prompt, which in turn supervise the next round of prompt learning with respect to Lbuxx Lgion and Lets for the next iteration.
Optimization. We expect the learned child prompts to follow an accurate semantic hierarchy, having minimal overlap with other child tokens while maintaining semantic consistency with their original parent prompts. We leverage the following dispersion losses to enlarge the angular distances among the child-chil and decrease the distances between child haldd-prrent prompt pairs:
K
pparent-child
Tp):
K
t=1
V. |
Lehild-child
log
Σ JF1
exp(
/Te)
K
K
v
where vÿ retrieved from the parent prompt queue P parent as fixed prototype. The temperature coefficients Tp and adjust the angular separation. The overall optimization can be formulated as:
L = Cparent-child + -childd-chid + "/boox Lbbox + Tigion + Vels
where y the loss coefficient that controls the Lehild-chila- The coefficients i.e., Tbbox Tigion> Vels follows 33 Until the optimization convergence, the prompt expansion will repeat if needed.
Expansion Termination with Maximum Angular Coverage (MAC). While prompt expansion effective in capturing fine mne-grained semantics, it inevitably introduces computational overhead, impacting inference efficiency for downstream tasks. To balance the semantic richness and inference costs, we gather all learned prompts P and evaluate the maximum angular coverage (MAC) among all pairs. MAC is defined
O'mxx
max arecos VP
I V
The Ommx reveals the breadth of vocabularies covered by the current prompts. Notably, empirical study shows that as the number of expansion rounds increases, the MAC increases monotonically and eventually converges. This convergence serves an effective signal to terminate prompt expansion. The overall algorithm is summarized in Algorithm
4 Experiments
4.1
Experimental Setup
Datasets. We conduct our experiments using two detection datasets: 1). MS-COCO [32] large- scale object detection and instance segmentation dataset, comprising approximately 115K training images and 5K validation images across 80 classes. 2). LVIS [13] includes 2.2 million high-quality instance segmentation masks covering 1,000 class labels, resulting in a long-tailed data distribution. consists of around 100K training images and 19.8K validation images. For class-agnostic object detection (CA-OD) setting, we merge all categories from both datasets into a single class to perform class-agnostic detection. To further validate the efficacy of DiPEx downstream out-of-distribution object detection (OOD-OD) tasks, we evaluate our method using rectified version of the OOD-OD benchmark. Unlike previous benchmarks [12] where samples that not contain ID instances are
manually selected and ID and OOD performance evaluated separately, we tested our approach

---

Algorithm The Proposed DiPEx for Class-Agnostic Object Detection
Input: f: vision-language model (VLM)
Output: P: of fine-tuned prompts for f to detect alass-agnostic objects
Initialize single learnable parent prompt P1 {v1}
Optimize P1 using zero-shot detection results from f Initialize growing set of learnable prompts P = (Pi} and an empty parent queue P parcut = for each round I {1, 2, ... ].} do Identify the parent prompt with the highest uncertainty E P/
Freeze v? and add it to the parent queue P parent
Expand vî into K learnable child prompts P1+1 [V141,A via Equation Grow the set of learnable prompts: P
PrUPt+1
Optimize the prompts P using Equation
3 with parent Compute maximum angular coverage (MAC) via Equation MAC converges then Break; terminate the prompt growth
end end for
Table 2: Class-agnostic object detection on the MS-COCO dataset. [] indicate the prompt word for Grounding DINO. The prompting methods indicated with are adapted to the OD task.
Method
Description
AR1
AR
AR
AR@s
AR@M
AR@L
AP
Selective Search 47
non-parametric
0.1
1.1
7.8
0.9
7.2
20.7
0.1
UP-DETR 8
self-training
0,2
1.4
1,4
0.0
0.2
5.8
0.1
DETReg 1
self-training
0.6
3.7
12.9
0.2
12.8
35.3
1.4
FreaSO1.0 49
self-training
3.7
9.7
12.6
0.5
12.3
34.1
4.2
Exemplar-FreeSOL0 18
self-training
8.2
13.0
17.9
12.6
MOST 43
self-training
3.1
6.4
6.4
0.1
1.6
24.5
3,3
Curl ....FR 5D
self-training
19.6
32.8
13.7
37.5
60.0
29.6
Grounding DINO ["generic"]
zero-shot
10.3
37.8
44.1
17.7
51.6
80.0
28.3
Grounding DINO+CoOp'
24
self-training
10.4
39.1
61.3
36.4
72.7
88.8
34.6
Grounding DINO+CoCuOP"
23
self-training
7.6
34.1
58.1
33.9
68.3
86.1
24.6
DiPEx
self-training
10.5
40.8
63.2
39.2
74.3
89.8
35,9
on the MS-COCO, which includes a mixture of both ID and OOD objects. While we followed the settings outlined in OOD-OD [2]. with 20 base classes PASCAL-VOC [33 designated ID classes and the remaining classes treated as OOD, Our choice of dataset enhances the rigor of our evaluation by combining both ID and OOD instances, providing a more realistic assessment of our method's real-world conditions.
We report results for class-agnostic object detection both the MS-COCO and LVIS validation splits. For evaluation, we adopt official metrics from the COCO 2017 challenge. Specifically, report average precision (AP) IoU thresholds from 0.5 to 0.95, along with average recall (AR) across the same threshold range. We also report AR by object scale: AR @S for small, AR @ M for medium, and AR L for large objects. Details our implementation, including those of prior works used as baselines, are provided in Appendix A.2
Evaluation Metrics.
Main Results on Class-agnostic OD OOD-OD
Class-agnostic OD on MS-COCO. To validate our proposed method for class-agnostic object detec- tion, we compared it against different baseline methods the MS-COCO dataset, using various metrics as reported in Table 2 We observed that non-paramen con-aarmettic methods generally underperform compared to self-tree Dolf-training methods due to their inability learn and extract semantic and geometric information about objects from the dataset. In contrast, Grounding DINO, leveraging pre-trained knowledge, demonstrates strong zero-shot capabilities and achieves AR of 44. % with a single text prompt, "generic". Furthermore, CoOp, which fine-tunes prompts for Grounding DINO, enhances class-agnostic detection performance by 39.0% in AR 100 compared to direct zero-shot inference. Our method, which expands the learnable prompts to a wider angular distance, surpasses all baselines by achieving the highest performance across all metrics and outperforming the leading baseline, CoOp, by 3.1% 1% in AR Notably, for small objects which are challenging to localize, method improves

---

Table Class-agnostic object detection on the LVIS dataset.
indicctte the model fine-tuned on
the LVIS training set by self- training without box annotations.
Method
AR1
AR
20
AR@S
AR@M
AR@L
AP
AP@S
AP@M
AP@L
Selective Search
0.1
1.1
13.0
6.1
19.9
37.6
0.2
0.4
0.2
0.2
G-DINO ["objecl"
4.1
17.9
27.2
13.0
44.1
71.1
5,4
5.6
10.0
9.4
G-DINO ""geerric
9
16.5
20.2
6.5
34.5
67.7
4.1
17.4
30.7
G-DINO "l
9
4,0
17.8
28.0
13.9
45.3
70.7
11.6
6.3
19.6
32.0
SAM 26
42.7
27.7
66.3
75.5
6.1
CutLER 50
2,4
9.3
21.8
10.8
35.1
55.5
2.7
9.1
15.1
HASSOD
0.2
10.6
26.9
15.6
42.2
56.9
4,9
2.8
7.9
12.2
G-DINO CoOp"
24
4.2
19.1
40.3
23.6
63.5
83.5
14.0
8.3
23.7
32.3
G-DINO CoCoOp
24
4.2
19.2
40.7
24.1
63.8
84.1
13.6
8.1
22.4
30.1
DiPEx
4.3
20.1
48.4
31.9
72.6
88.2
15.2
9.3
25,3
32.8
Table 4:
The downstream out-of-distribution object detection (OOD-OD) on the MS-COCO dataset, where the ground truth boxes contain both known and unknown classes.
KNOWN
UNKNOWN
Method
AP
APS0
AR 100
AR@S
AR@M
AR@I.
AP
AP@S
A
AP@T.
Selective Search 47
8.3
1.0
8.5
23.2
0.1
0.0
0.0
0.5
MOST 43
5.3
0.1
1.3
22.5
0.4
0.1
0.4
1.2
CutLER 50
34.5
15.8
41.5
62.7
5.7
2.3
6.9
13.7
VOS 12
36.6
56.7
10.0
2.2
6.1
27.1
2.8
0.8
2.2
7.2
PROB 62
28.2
43.8
13.2
1.9
11.2
40.3
0.9
0.6
0.9
2.1
UnSniffer
35.8
55.8
20.6
11.8
2.9
1.5
3.1
5.3
G-DINO []
46.3
59.7
43.3
18.0
52.1
82.6
12.5
6.9
17.8
25.7
DiPEx
46.3
59.7
59.9
35.8
72.9
89.7
15.7
9.7
21.8
25.2
A by 7.7% compared to CoOp, indicating that expanded prompts better capture range of object sizes. Additionally, the proposed DiPEx achieved the highest AP 35.9%, demonstrating the superior quality of class-agnostic detection.
Class-agnosti OD LVIS. To further validate the efficacy of DiPEx, conducted extensive experiments on the challenging LVIS dataset, which includes thousands of classes with a long-tal tail distribution. As shown in Table prompt tuning methods such CoOp 24 and CoCoOp 23 outperform zhro-shot Grounding DINO when using hand-crafted prompts (e.g.. "items", "generic", "objects""). Additionally, CoCoOp surpasses multi-object discovery baselines like CutLERR (50 and HASSOD (2) by 86.7% and 51.3% AR respectively. Notably, SAM (26). which was pre- trained on vast of dataset containing millions of images and billions of masks, demonstrates strong zero-shot capabilities, surpassing all other baselines. In contrast, our proposed DiPEx outperforms SAM by 13.3% AR200 and 21.3% in AP after only four epochs of self-training, Furthermore, DiPEx exceeds CoOp by 20.1 % in AR200-
Downstream OOD-OD on MS-COCO. To evaluate the generalization of our proposed DiPEx in out-of- -itt-fffiittibution object detection (OOD-OD), we compared its performance on both known and unknown classes against various baselines. As shown in Table the zero-shot Grounding DINO uses known class names as prompts, supplemented with a simple "generic" prompt for unknowns, outperforms all other MLM methods (e.g., 25.5% higher AR compared CutLER (50). This improvement stems from VLMs leveraging rich semantic knowledge from language models to better comprehend object information in images. DiPEx enhances this further by expanding text prompts in embedding space, enabling it to capture and differentiate objects of varying sizes and diverse semantics from learned classes. This approach delivers significant performance gain, achieving a 38.3% increase in AR100 and 25.6% in AP increase over zero-shot predictions. Furthermore, the expanded prompts can be directly applied alongside various known class vocabularies detect unknown objects, eliminating the need for retraining.
4.3 Ablation Study and Model Analysis
investigate the impact various factors on prompting performance including the learnable prompt lengths, the number of expansion rounds L, and angular coverage achieved across rounds. To facilitate model analysis, we present the distribution of prompt logit activation and visualization of detection results. Further ablation studies refers to Section A.3

---

0.640
DiPEx
CoOp
CoCoOp
DIPEX
CoOp
CoCoOp
Average Pracision
Average Recell
0.620
0.610
0.320
0.600
0.300
0.280
0.580
0.570
0.200
0.560
3
0.240
Number Prompts (N)
Number of Prompts (N)
Figure 4: Impact of the prompt length on the MS-COCO dataset. The average recall (AR) and
precision (AP) are reported to compare the derived DiPEx against CoOp (24) and CoCoOp [23]
Promph
Prompts
Prompts
Prompts
Prompts
Figure 5: The heatmap visualization presents the angular coverage across all learned prompts through the 2nd, the 3rd, and the 4th round of training. The maximum angular coverage (MAC) monotonically increases from 67.7% in the 2nd round to 75.95" in the final round. The gradual reduction in rate of change in angular coverage towards the final round suggests that the model nearing convergence.
Impact on Number of Prompts. In Figure we compare the impact of prompt length N for DiPEx against CoOp 24 & CoCoOp 23]. Overall, DiPEx shows consistent improvement in performance with greater number of prompts not merely due to quantity, but rather because a larger set fosters greater diversification, enabling the model to capture more comprehensive semanties. In contrast, CoOp's [24] performance remains constant, while CoCoOp's 's [23] performance declines, suggesting that more prompts not necessarily guarantee enhanced performance.
Impact on Expansion Rounds and Angular Coverage. To substantiate our hypothesis that a higher maximum angular coverage (MAC) correlates with broader spectrum of vocabularies, we computed the MAC using Equation 4 The coverage results are visualized as heatmaps in Figure 5 At the initial stage of expansion (leftmost heatmap), distributed, with mean coverage of 47.56", This suggests that the prompts are actively exploring the embedding space to capture diverse semantics. As the expansion progresses the third round (middle heatmap), the MAC increases from 67.78 to 75. Specifically, row/col 7 (selected parent prompt) demonstrates the closest angular distances among the child prompts. This observation is crucial as it suggests that child prompts should not diverge excessively from the root semantics to maintain coherence. By the fourth round of expansion (rightmost heatmap), the pattern remains consistent with the third round. There is reduced rate of change of MAC, achieving maximum coverage of 15.95%and and a mean coverage 11.51 11.51"amons among the child prompts. This plateau in MAC indicates that maximum semantic expansion has been reached, suggesting that the model is approaching convergence and further expansion may not be necessary.
observe that the prompts are quite uniformly
The Distribution of Prompt Logit Activation. We previously established prompt logit activation frequency as an uncertainty measure guide parent prompt selection for splitting. To investigate the dynamics of expanding highly uncertain parent prompts, we visualize the activation statistics (i.e., the frequency of logit activations) of tokens within 2nd and 3rd expansion rounds. As illustrated in Figure 6) the distribution of these logits exhibits a long-tailed pattern, suggesting substantial uncertainty and numerous semantic overlaps among the mined semantics. The figure on the right demonstrates that, following the expansion of highly activated prompts, the distribution of

---

Activation Frequency
0.150
Activation Frequency
0.150
0.125
0.125
0.100
0.100
0.075
0.075
0.050
0.050
0.025
0.025
0.000
0.000
Expanded Child Prompts
Expanded Child Prompts
Figure 6: The distribution of logit activation of the learned prompts in the 2nd round (left) and the 3rd round (right). The prompt of the highest activation frequency is identified for further expansion.
MOST
CuILER
Zero-Shat G-DINO
DiPEx
Ground-
Figure 7: Visualization of the class-agnostic detection performance by baselines and the proposed DiPEx on MS-COCO (32]. More visualizations are provided in Appendix (Figures and 10.
child prompts becomes more uniform, suggesting the discovery fine- ane-grained semantics. These observations support our choice of uncertainty measure and verify the validity of DiPEx, indicating that expanding based on highly uncertain parent prompts effectively alleviates semantic ambiguity.
Qualitative Study. In this section, we present visualized class-agnostic box predictions on images sampled from the MS-COCO dataset [32]. as shown in Figure 7 The proposed DiPEx method demonstrates superior ability to detect more bounding boxes than all bascline methods, particularly for small objects. For example, people in the distance (rows and 3) and some bonsai (row 2) are missed by all baselines but successfully detected by DiPEx, showcasing its strong capability in localizing challenging small objects. For large objects, such as a motorcycle (row 3) and two people shaking hands in the near distance (row 1), DiPEx localizes them with significantly higher confidence compared to the zero-shot predictions Grounding DINO using the prompt "generic". Additionally, DiPEx successfully identifies objects that are not annotated in the MS MS-COCO ground truth, such as plates (row 1), pillowease (row 2), and frame on the wall (row 2). This highlights DiPEx's ability to identify comprehensive set of class-agnostic objects, even those missed in human annotations.
Conclusion and Limitations
This work introduces DiPEx, novel self-supervised dispersing prompt expansion approach for class-agnostic object detection. We demonstrate through comprehensive experiments and analysis that DiPEx efffettivly detects a wide range of unseen objects of varying sizes and achieves broad vocabulary coverage. The progressively expanded prompt sets maintain good angular distances, promoting the formation of semantic hierarchy and facilitating downstream detection tasks with single inference pass. While the proposed DiPEx does not rely on box annotations, it requires self- training on the entire dataset for each round of prompt expansion, resulting in increased computational overhead. Additionally, some hyperparameters like temperature coefficients Tp. and learnable prompt length K, may require manual tuning for optimal performance. Future research directions include exploring methods to learn hierarchical prompts at once rather than through expansion. Extensive benchmarking on additional downstream tasks, such as open- apen-vocabulary and open- world detection, is necessary comprehensively validate the proposed approach.
10

---

Acknowledgments and Disclosure of Funding
This research partially supported by the Australian Research Council (DE240100105. DP240101814, DP230101196)
References
[1] Amir Bar, Xin Wang, Vadim Kantorov, Colorado J. Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with re- gion priors for object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14585-14595, 2022.
[2] Shengcao Cao, Dhiraj Joshi, Liangyan Gui, and Yu-Xiong Wang. HASSOD: hierarchical adap- tive self-supervised object detection. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Anmual Conference on Neural Information Processing Systems (NeurIPS), 2020.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[5] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, abs/2003.00297, 2020.
[6] Ming-Minn Cheng, Ziming Zhang. Wen- Yan Lin, and Philip H. S. Torr. BING: binarized normed gradients for objectness estimation at 300fps. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3286-3293, 2014.
[7] Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. Promptstyler: Prompt-driven style generation for source-free domain generalization. In IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 15656-15666. IEFF, 2023.
[8] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. UP-DETR: unsupervised pre-training for object detection with transformers. In /EEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1601-1610, 2021.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HI.T), pages 4171-4186. 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[11] Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. SIREN: shaping representations for detecting -uttof-distribution objects. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[12] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't 't know by virtual outlier synthesis. In International Conference Learning Representations (ICLR), 2022.
[13] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision (IJCV), 88(2):303-332 338- 2010.

---

[14] Christiane Fellbaum. WordNet: An electronic lexical database. MIT press, 1998.
[15] Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5356-5364, 2019.
[16] Weizhen He. Weijie Chen, Binbin Chen, Shicai Yang, Di Xie, Luojun Lin, Donglian Qi. and Yueting Zhuang. Unsupervised prompt tuning for text-driven object detection. In IEEE/CVF International Conference Computer Vision (ICCV), pages 2651-2661. IEEE, 2023.
[17] Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. CoRR, abs/2204.03649, 2022.
[18] Taoseef Ishtiak, Qing En, and Yuhong Guo. Exemplar-freesolo: Enhancing unsupervised instance segmentation with exemplars. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15424-15433, 2023.
[19] Ayush Jaiswal, Yue Wu, Pradeep Natarajan, and Premkumar Natarajan. Class-agnostic object detection. In IEEE Winter Conference on Applications of Computer Vision, pages 918-927, 2021.
[20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lisoon-anguaag representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), volume 139, pages 4904-4916, 2021.
[21] Qing Jiang. Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. CoRR, htss/2403.14110 2024.
[22] K. Joseph, Salman H. Khan, Fahad Shahbaz Khan, and Vineeth N. Balasubramanian. Towards open world object detection. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5830-5840, 2021.
[23] Zhou Kaiyang, Yang Jingkang, Loy Chen Change, and Liu Ziwei. Conditional prompt learn- ing for lision-language models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[24] Zhou Kaiyang. Yang Jingkang, Loy Chen Change, and Liu Ziwei. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022.
[25] Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19113-19122. IEEE, 2023.
[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In IEEE/CVF International Conference Computer Vision (ICCV), pages 3992-4099 4003, 2023.
[27] Philipp Krähenbühl and Vladlen Koltun. Geodesic object proposals. In European Conference on Computer Vision (ECCV), volume 8693, pages 725-739. 2014.
(28) Liangqi Li, Jiaxu Miao, Dahu Shi, Wenming Tan, Ye Ren, Yi Yang, and Shiliang Pu. Distilling DETR with visual-linguistic knowledge for vocabuayyy object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 6478-6487, 2023.
[29] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang. Jianwei Yang, Chunyuan Li. Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10955- 2022.
[30] Wenteng Liang, Feng Xue, Yihao Liu, Guofeng Zhong, and Anlong Ming. Unknown sniffer for object detection: Don't turn blind eye to unknown objects. IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3230-3239. IEEE, 2023.
12

---

[31] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for nen-vocabulary object detection. In International Conference on Learning Representations (ICI.R) 2023.
[32] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision (ECCV), volume 8693, pages 740-755, 2014.
[33] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
[34] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5196-5205 IEEE, 2022.
[35] Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14054-1406 IEEE, 2022.
[36] Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muham- mad Anwer, and Ming-Hsuan Yang. Class-agnostic object detection with multi-modal trans- former. In European Conference on Computer Vision (ECCV), volume 13670, pages 512-531, 2022.
[37] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral meth- ods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 8354-8365. IEEE, 2022.
[38]
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. CoRR, abs/2304.00 120330000711 17199, 2023.
[40] Pedro H. O. Pinheiro, Ronan Collobert, and Piotr Dollár. Learning to segment object candidates. In Annual Conference on Neural Information Processing Systems (NeurlPS), pages 1990-1998, 2015.
[41] Jordi Pont-Tuset, Pablo Arbeláez, Jonathan T. Barron, Ferran Marqués, and Jitendra Malik. Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans. Pattern Anal. Mach. Intell., 39(1): 128 140, 2017.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning (ICML), volume 139, pages 8748-888 876888763,48 2021.
[43] Sai Saketh Rambhatla, Ishan Misra, Rama Chellappa, and Abhinav Shrivastava. MOST: multiple object localization with supervupe transformers for object discovery, In IEEE/CVF International Conference Computer Vision (ICCV), pages 15777-15788, 2023.
[44] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 22(8):888-900 2000.
[45] Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursue, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In British Machine Vision Conference (BMVC), page 310, 2021.
13

---

[46] Korawat Tanwisuth, Shujian Zhang, Huangjje Zheng, Pengcheng He, and Mingyuan Zhou. POUF: prompt-orienter unsupervised fine-tuning for large pre-trained models. In International Conference Machine Learning (ICML), volume 202, pages 3381 18816-3332, 2023.
[47] Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers, and Arolld W. M. Smeulders. Selective search for object recognition. International Journal of Computer Vision (LJCV), 104(2):54-171,22 2013.
[48] Luting Wang, Yi Liu, Penghui Du Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11 11186-11196. IEEE, 2023.
[49] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and José M. Álvarez. Freesolo: Learning segment objects without annotations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 141661 2022.
[50] Xudong Wang. Rohit Girdhar, Stella X. Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 3124-3134, 2023.
[51] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L.. Crowley, and Dominique Vaufrey- daz. Self-supervised transformers for unsupervised object discovery using normalized cut. In IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 14523- 14533, 2022.
[52] Zhenyu Wang, Yali Li, Xi Chen, Ser-Nam Lim, Antonio Torralba, Hengshuang Zhao, and Shengjin Wang. Detecting everything in the open world: Towards universal object detection. In IEEF/CVF Conference Computer Vision and Patlern Recognition (CVPR), pages 1 11433- 1 11443. JEEE, 2023.
[53] Samuel Wilson, Tobias Fischer, Feras Dayoub, Dimity Miller, and Niko Sünderhauf. SAFE: sensitivity-awar features for cutt-f--dittibuttinn object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 23508 23519, IEEE, 2023.
[54] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15254-15264 15264. IEEE, 2023.
[55] Zhiheng Wu, Yue Lu, Xingyu Chen, Zhengxing Wu, Liwen Kang, and Junzhi Yu. UC-OWOD: unknown-classified open world object detection. In European Conference on Computer Vision (ECCV), volume 13670, pages 193-210, 2022.
[56] Yunyi Xuan, Weijie Chen, Shicai Yang, Di Xie, Luojun Lin, and Yueting Zhuang. Distilling vision-language foundation models: A data-free approach via prompt diversification. In Abdulmotaleb El-Saddik, Tao Mei, Rita Cucchiara, Marco Bertini, Diana Patricia Tobon Vallejo, Pradeep K. Atrey, and M. Shamim Hossain, editors, Proceedings of the 3 I ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, pages 4928-4938 ACM, 2023.
[57] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: fine-grained interactive language-image pre-training. In International Conference on Learning Representations (ICLR), 2022.
[58] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open object detection using captions. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 14402, 2021.
[59] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. CoRR, abs/2304.00688 2023.
[60] Xiaowei Zhao, Yuqing Ma, Duorui Wang, Yifan Shen, Yixuan Qiao, and Xianglong Liu. Revisiting open world object detection. IEEE Trans. Circutts Syst. Video Technol. 34(5):3496 3509, 2024.
14

---

[61] C. Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In European Conference on Computer Vision (ECCV), volume 8693, pages 391-405, 2014.
[62] Orr Zohar, Kuan-Chieh Wang, and Serena Yeung. PROB: probabilistic objectness for open world object detection. In /EEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 1144-11443 1453. IEEE, 2023.
[63] Wei Zuwei Long. Open grounding dino:the third party implementation of the paper grounding dino. http://www..ccc//ooo///////////a wtwpppennooonuunnninnoooo 2023.
15

---

6 for A Appendix/ Supplemental Material
This supplementary material includes comprehensive overview of related work class- object detection, vision-language models (VLMs), and prompt tuning. Additionally, we provide detailed descriptions of baselines, implementation details for both the baselines and the proposed method, and an extensive ablation study are provided. The ablation study analyzes the impact of pseudo-labeled supervision and the effect of the hyperparameter 7. Lastly, we present more comprehensive visualizations of class agnnsttg box predictions.
Section A.1
Related Work
Section
A.2
Baselines and Implementation Details
Section
More Ablation Studies
Figures
and
10:
Additional Visualizations of Class-Agnostic Box Predictions
A.1
Related Work
Class-Agnostic Object Detection. Traditional bottom-up approaches 44
27 61 40
region proposal generation, often grapple with the constraints precision, despite high recall rates, reducing their scalability for general use in diverse environments. Recent breakthroughs in ViTs 10 39 have enabled scaling up massive datasets for self-supervised learning, extracting both local and global semantically meaningful features. This has led to numerous methods in unsupervised object discovery and localization. LOST [45 is an early application, using patch similarity graph and an inverse degree map to identify seed patches and extract bounding boxes. TokenCut [51 constructs an undirected graph with image tokens as nodes, applying the normalized cut algorithm 44 for foreground-backgroundd segregation. MOVE (37 builds on LOST by employing deep spectral bipartitioning, offering a more principled and effective approach. However, both LOST [45) and TotenCut [51] are limited to detecting a single object per image. MOST 43 addresses this limitation by using entropy-based box analysis (EBA) to segregate foreground tokens. Nevertheless, their performance remains sub-optimal, constrained by their limited capacity to generalize across diverse object categories. Closest to our work is MAVL (36), where they develop an MVIT with late fusion strategy and use generic text queries like "all objects" to locate objects. However, their framework requires full supervision and multiple inferences with different textual prompts, yet still falls short of achieving optimal performance. In contrast, our approach eliminates the need for labels and achieves SOTA performance with one-pass inference with the non-overlapping prompts.
which are pretrained on expansive image-text pairs have demonstrated significant potential in learning generic concepts. HierKD 35 introduces global language-to-visual knowledge distillation modules, which align global-level image representations with caption embeddings through contrastive loss. OV-DETR [58) pioncered the extension of the DETR framework to open- vocabvocabllrry setting by integrating a conditional binary matching mechanism. GLIP (29] converted object detection into a grounding task, utilizing additional data align phrase and region semantics. Recently, Grounding DINO framework to integrate object detection and visual grounding within unified architecture. Similarly, T-Rex2 (21] synergizes text and visual prompts through contrastive learning.leading to state-of-the-ee performance in out-of-distribution object detection. Nonetheless, the effectiveness of VLMs is heavily influenced by the textual cues they are conditioned on, and efficiently adapting them to specific downstream applications remains a substantial challenge as manually engineering optimal prompts can often entail considerable effort and resources [59]. Prompt tuning is simple yet effective solution to adapt models to specific tasks by optimizing small number of soft prompts in an end-to-end manner while keeping the original model's parameters frozen. The pioneering work of CoOp (24) introduced context optimization by fine-tuning CLIP using learnable tokens. However, CoOp's generalizability was constrained, limitation later addressed by CoCoOp (23). which conditioning input tokens on image embeddings. MaPLe [25) advanced this by introducing a multi-modal prompting technique overcome the limitations of uni-modal prompting methods. ProDA [4] further innovated by learning a distribution of diverse prompts and employing Gaussian model to capture visual variations. Despite these advancements, an inherent limitation persists across these methods: they all require full supervision. UPI. [7 first proposed unsupervised prompt learning for image recognition task, POUF [46] later introduced similar self- self-prompting mechanism to minimize entropy using optimal transport. However, these
VLMs and Prompt Tuning, Recent advances in VLMs 42
20 57
33 introduced a
16

---

Table 5: The impact pseudo-labeled supervision MS-COCO 32 dataset, when applying different pseudo-labell queried on Grounding DINO using different textual cues. In the main paper, report the performance of DiPEx using merged pseudo labels for the first round of training.
Method
AR
AR@S
AR@M
AR@L
AP
A
AP@M
AP@L
Grounding DINO @ ["generic"]
44.1
17.7
51.6
80.0
28.3
11.4
33.0
56.5
Grounding DINO @ [25 nouns]
40.5
16.0
46.6
75.0
12.1
4.4
12.0
26.7
Grounding DINO merged
51.9
24.8
61.6
85.8
19.1
7.6
19.5
42.0
DiPEx @ ["generic"]
65.5
42.7
76.2
90.3
37.0
20.5
43.3
62.4
DiPEx @ [25 nouns]
46.6
18.5
55.7
83.3
13.2
4.0
13.3
30.7
DiPEx @ merged
63.2
39.2
74.3
89.8
35.9
16.4
39.7
63.8
methods have yet to be adapted for the object detection domain. To our knowledge, UPT [16] is the only existing work that optimizes prompts using dual complementary teaching specifically for object detection tasks. Our work, DiPEx, represents the first endeavor to apply prompt learning to agnsstag object detection through a self-training approach.
A.2 Baselines and Implementation Details
Baselines. We compare the proposed approach with fourteen baselines: 1) bottom-up selective
search (47] that slides windows different sizes to locate objects, 2) UP-DETR [8. an unsupervised
pre-training method for OD that be fine-tuned detect class-agnostic objects. 3) DETReg M. which learns to localize objects and encode an object's properties during unsupervised pre-training,
4) MOST (43, multiple objects localizer based on patch correlations without any training, 5)
FreeSOLO 49 which unifies pixel grouping, object localization and feature pre-training in a fully FreeSOLO 18], an improved approach based on FreeSOLO
self-super manner, 6)
through exemplar knowledge extraction, 7) CutLER [50]. an unsupervised object detection method by encouraging the detector to explore objects missed in extracted coarse masks, 8) HASSOD (2. clustering strategy that groups regions into object masks based on self-supervised features, 9) CoOp 24 and 10) CoCoOp [33] prompting techniques that utilize learnable vectors to model prompt's context words, enabling zero-shot transfer to class-agnostic detection, 11) segment anything model (SAM) [66]. a foundational model trained on billion masks and 11 million images such that can perform zero-shot transfer to the class-agnostic OD task. For OOD-OD task, further compare three baseline methods: 12) VOS [12] that regularizes the model's decision boundary between known and unknown classes by training with generated virtual outliers. 13) PROB [62] which utilizes multivariate Gaussian distribution to leam objectness probability to separate known and unknown objects, 14) UnSniffer [30), which similarly introduces object confidence, derived from learning known objects with varying degrees of overlap.
Implementation Details. Our code is developed on the Open Grounding-DIINC framework 63 and operates on a single NVIDIA RTX A6000 GPU with 48 GB of memory. For our experiments, we choose a batch size of 8 for training, and set hyperparameter
0.1, = 0.1, 15., K = 9, L = 3. and while adopting all remaining hyperparameters from the Open Grounding-DINO codebase. We empirically set the T. = 75 as our threshold for expansion termination. The original implementation of CoOP was developed for image classification tasks based on CLIP and supervised contrastive learning. We extend CoOP to class-agnostic object detection using pseudo labeling- based self- self-training, which remains consistent with our approach. All the implementation code and configurations files are provided in supplementary materials and will be publicly released upon acceptance of this work.
0.1,
A.3 More Ablation Studies
Pseudo-labels Construction For pseudo-labeling, we utilize off-the-shelf Grounding DINO with "generic" text prompt, which demonstrates considerable zero-shot performance, as illustrated in our pilot study. Additionally, we generate pseudo-boxes by concatenating all beginner nouns from WordNer [14]. We then merge the predictions from these two queries and apply Soft-NMS to eliminate overlaps. In the following Section A.3], we also investigate the performance of DiPEx alongside other prompt-tuning methods the quality of pseudo-labels.
17

---

Impact on Pseudo-labeled Supervision. In this section, we investigate how the quality of pseudo- labels used for self-training impacts DiPEx's performance, given our reliance these training samples. Specifically, we generate pseudo-labels by querying the off-the-shelf Grounding Dino model with three different approaches: 1) using "generic" text prompt, 2). the 25 beginner nouns from WordNet [14], and combination of both. As shown in Table the "generic" text prompt alone demonstrated considerable performance. However, we observed an improvement Average Recall (AR) when merging the predictions generated by "generic" with the 25 beginner nouns, leading to this study. Consequently, we use these pseudo-labels to self-train our model.
Effect of Loss Coefficient 7:- To effectively sep- arate child prompts while maintaining semantic coherence between parent and child prompts, selecting an appropriate yy essential. As il- lustrated in the bar plot below, a moderate value typically yields optimal results. con- trast, larger value (e.g.. = 5) causes child prompts to diverge more significantly, distort- ing semantic integrity and potentially leading to vverraegularization of the model.
0.7
0.6
0.5
0.4
0.2
0.1
0.0
0.1
0.5
Figure 8: Study of Loss Coefficient
18

---

Figure 9: Additional visualizations of class-agnostic box predictions. Columns 4 correspond the following methods: MOST [3], CutLER (50], zero-shot Grounding DINO ["generic") [ and our proposed DiPEx, respectively. The final column presents human- annotated ground truth bounding boxes from the MS-COCO dataset 32
19

---

Figure 10: Additional visualizations of alass-agnostic box predictions. Columns 4 correspond to the following methods: MOST 43), CutLER (50], zero-shot Grounding DINO ["generic"] 9 and our proposed DiPEx, respectively. The final column presents human- aunan-annotated ground truth bounding boxes from the MS-COCO dataset [32].
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope. The experimental results, proposed methodology, and analysis throughout the paper substantiate the claims regarding performance improvement, novelty, and broader applicability.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses the limitations of the proposed methodology in the conclusion section.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides a comprehensive disclosure of all necessary information to replicate the main experimental results. It offers a detailed, step-by-ster algorithm and exhaustive experimental settings in the appendix, facilitating an accurate reproduction process. Additionally, the supplementary materials include the complete source code, along with extensive usage instructions and configuration files for various methods and datasets, thereby ensuring thorough reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper presents codebase for class-agnostic detection, featuring detailed usage instructions and configuration files for a wide range of existing methods, models, and datasets. The codebase will be made available in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cr
Seallicgguids/Codeefe
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ptaiss.cc/uublicguui.ees Sow.eddeee/oooeeeeeeeeeeeeeeeeee wwtssSuummisiiiiiiio sww.iiiiioooooooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and evaluation details are clearly outlined in the implementation details section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification: The paper does not report error bars, but results are based on iterative trials to ensure consistency.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The implementation section includes detailed information about the specifica- tions of the computing devices used to run the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/://oorrppp...ccppp////shs
Answer: [Yes]
Justification: The authors have thoroughly reviewed and adhere to the standards of the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?

---

Answer:
[NA]
Justification:
The work performed does not involve any positive or negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.lymmpoy of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not present any potential risks and therefore does not describe any safeguards for the responsible release of data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets used, including MS-COCO 2017 and LVIS, as well as the Ground- ing DINO's codebase, properly cited and referenced throughout the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
25

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.paperrsst paperswi thcode. cow /atasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper benchmarks class-agnostic object detection through extensive experiments, including downstream out-of-distribution object detection. The codebase is well-documented and will be provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
26

---

Answer:
[NA]
Justification:
This paper does not pose any of the above-mentioned risks.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "51vdbPZFNW", "reviewer": "Reviewer_Jt2M", "summary": "This study investigates the use of visual-language models to improve class-agnostic object detection through a self-supervised prompt learning strategy. Diverse Prompt Expansion (DipEx) is proposed to enhance downstream task performance by learning to expand a set of diverse, non-overlapping prompts that boost recall rates of object detection.", "questions": "Please refer to the weakness.", "limitations": "The authors have stated the limitations in the conclusion part, and I hope they can do some further validations to explore the effecitveness of the proposed method on open-vocabulary and open-world detection.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The authors find a new setting, class-agnostic object detection, which is practical and universal in real-world scenarios.\n2) The proposed method achieves performance improvements compared to its baselines.\n3) The figures are rich and vivid, and the writing is good.", "weaknesses": "- I'm mainly concerned about the novelty. The primary contribution of this paper lies in the prompt expansion method, which uses contrastive loss for optimization to acquire diversified prompts. However, this kind of optimization-based prompt diversification is not new art [1,2], especially as the previous work [1] presents a rather similar concept of optimizing prompts on the hypersphere. \n\n    [1] Promptstyler: Prompt-driven style generation for source-free domain generalization. ICCV, 2023.\n\n    [2] Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification. ACM MM, 2023.\n\n- The proposed prompt expansion method seems not essentially related to object detection task, since it may also suitable to other vision tasks. I thins it's may be not proper to sell the class-agnostic object detection as a major contribution in this paper. \n\n- As the authors claim that this paper aims to solve class-agnostic task, I am highly curious about how to obtain the pseudo labels during training. Does the pseudo labels get updated during prompt-based optimization?  If updated, how to do this?\n\n- What does the learned prompt resemble? More visualization results, such as the attention visualization, of the learned prompt are expected.  \n\n- Some presentations leave me confused. For example, in Line 117-118, the authors state that, ``applying query-merging to UNIVERSAL words results in a 52.46% reduction in AR compared to prediction-merging, whereas CLASS-WIDE queries (e.g., from WordNet) achieve a smaller decrease in AR of only 23.64%''. However, I fail to understand how the 52.46% and 23.64% reductions are computed by the results in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions, which we address below:\n\n> **R1.1** Novelty and comparisons with [1,2]\n\n**Contributions.** \nWe appreciate your feedback and would like to **clarify** the major contributions of our work: \n\n(1) **General Impact**: Our work presents an **early analysis** of the bottleneck of current class-generalized detection tasks such as OOD detection. The challenge of locating *every possible* object remains a **fundamental challenge** before being able to improve AP for classes of interest (*line 28-32*) has been **rarely studied**. Further, prior OOD works are incomparable due to inconsistent evaluation metrics (*e.g.,* open-set benchmark uses FPR95, and unsupervised class discovery uses CorLoc). DiPEx is the **first work** to comprehensively **benchmark** existing OOD detection (open-set/open-world detection) and unsupervised object discovery against class-agnostic OD setting, providing valuable guidance for future research.\n\n(2) **Technical Contributions**: We would like to point out that instead of using off-the-shelf **contrastive loss** ([1]&[2]), all our design is motivated by **angular perspective** to discover unrevealed semantics by pushing children prompts away while maintaining coherence with the parent prompts. The rationale behind this was based on our observation in pilot study (Appendix A.2 & Figure 7), where two text tokens that are semantically dissimilar exhibit large angular distances lead to more confident box predictions. This motivates us to hierarchically decompose high-level semantics into finer-grained ones, aiming to uncover more semantics.\n\n**Compared to SOTA.** We appreciate the reviewer for highlighting [1, 2]. **Key differences** include:\n\n(1) **Task.** Both [1] and [2] focus on domain adaptation for style diversification with **classes available**, or in other words, diversifying the suffixes prior to the *class* token. Additionally, both methods are constrained to **a fixed number** of **style prompts**. On the contrary, our design aims to disentangle **arbitrary numbers** of fine-grained classes, followed by a MAC early stopping strategy to prevent the excessive growth of class prompts. Notably, we are the **first** one to introduce progressive prompt expansion, proving its effectiveness. Our task, without class vocabulary available, is much more challenging.\n\n(2) **Objectives**. Both [1] & [2] use off-the-shelf contrastive loss, which significantly differs from our **dispersion loss**. Contrastive loss separates **inter-class** samples; this contradicts our motivation of revealing classes in a tree hierarchy fashion. Additionally, orthogonal constraint in [1] is **unsuitable**; recall our ablation study (see Appendix A.2) - semantically similar words can have small angular discrepancy (e.g., \"plates\" & \"dishes\" have angular distance $\\theta$ of 53.73$^\\circ$). This means that the orthogonal ($\\theta=90^\\circ$) constraint is too much and will push prompts **too far** and **distort** underlying representation. Therefore, we have opted to use **dispersion loss** as a **softer constraint** to enforce separability between child prompts while maintaining child-parent coherence. \n\n> **R1.2** The proposed method may also be suitable to other vision tasks.\n\nWe appreciate the reviewer's feedback. Adapting existing prompting methods (*e.g.,* CoOp), designed primarily for classification tasks, to detection yields suboptimal performance (Table 2, 3), especially with a large number of classes (Figure 3). This is because Grounding DINO uses class and query confidence to find top-k proposals, and if class prompts are not accurately learned, many correct boxes will be **missed** due to low confidence. Our hierarchical expansion allows us to find **accurate fine-grained** semantics, specifically designed to help detection-based foundation models identify more objects and **improve recall**. Furthermore, detection serves as a basis for downstream applications, such as multi-object tracking, which we plan to explore in future work.\n\n\n> **R1.3** Does the pseudo labels get updated during prompt-based optimization? If updated, how to do this?\n\nYes, our pseudo-labels are iteratively refined with each training process during the training process. (1) We start with inference using an off-the-shelf GroundingDINO model with \"generic\" and preprocess the predictions by removing low-confidence detections and excessively large bounding boxes. (2) At each self-training stage, we perform inference again with the expanded prompts to obtain updated pseudo labels. (3) For quality control, we remove boxes with an IoU greater than 0.5 with the previous \"ground-truth\" boxes and apply SoftNMS. This ensures the continual improvement of box accuracy and promotes the model to discover emerging concepts.\n\n> **R1.4** What does the learned prompt resemble? \n\nGreat question! The interpretability of context tokens remains as an **open question** [A, B]. Prior arts [A] have attempted to use NN words from pre-trained embeddings to interpret the learned vectors. However, since text features operate within a continuous embedding space, they likely carry more abstract meanings that are not readily interpretable. Regardless, we have provided a visualization of the attention features from each distinct prompt in the **attached PDF** for your reference.\n\n[A] Kaiyang Zhou et al. Learning to Prompt for Vision-Language Models, in IJCV.\n\n[B] Brian Lester et al. The Power of Scale for Parameter-Efficient Prompt Tuning, in EMNLP.\n\n> **R1.5** However, I fail to understand how the 52.46% and 23.64% reductions are computed by the results in Table 1.\n\nThanks! The percentage reduction was calculated using (query_merged-prediction_merged)/query_merged. We will provide more clarification in the revised version.\n\n**Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AUOH8oBxz9", "reviewer": "Reviewer_86g8", "summary": "This work identifies that the \"semantic overlaps\" may contribute to the diminished class-agnostic object detection performance for previous works utilizing VLMs, which is evidenced by the pre-experiments on had-crafted text queries on the MS COCO dataset. Furthermore, the authors derive a self-supervised prompt learning strategy to iteratively expand the (soft) prompt set in a tree hierarchy to ensure the diversity and coverage of class-agnostic object textual descriptions. A maximum angular coverage metric is provided for expansion determination to balance the prompt diversity and number cost. Experiments on MS COCO and LVIS datasets have proved the method's effectiveness.", "questions": "1. The detailed operations in Table 1 should be specified, eg., does prediction-merging means NMS? In addition, it would be better to report prediction-merging at the top line and query-merging at the bottom, which helps to identify the performance decline.\n2. There seems some misuse of notations. For instance, is the definition of $P$ in line 134 and line 137?", "limitations": "The application of current work is limited by the training data sources, which only evaluate MS COCO and LVIS separately. Joint training with larger data sources (e.g., Object365) will improve the application value.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research topic is interesting and valuable. Class-agnostic may be one of the cornerstones for large foundation vision models. \n2. The proposed prompt set expansion method is novel. The experiments have proved its efficacy.\n3. The paper is well-presented and easy-to-follow.", "weaknesses": "1. My major concern lies in the author's claim that the \"semantic overlap\" among input words in the prompt declines object detection by visual-language matching. In Table 1, after the query-merging operation, the encoded text features are derived through the complicated self-attention mechanism across the input words (embeddings). Therefore, I prefer to owe the detection degradation to the disturbed attention compared to one input word in a single inference pass, rather than the so-called semantic overlap. In other words, Section 2 and Appendix 2 failed to relate semantic overlap with empirical results for me.\n2. The overall framework is reasonable, but there are still some points to clarify:\n  - For the children prompt initialization, it may not be sufficient to choose only one parent prompt with the high uncertainty to expand, especially for the first iteration when all of the root prompts are highly abstractive and therefore uncertain. In addition, Eq(2) may also be added to the children prompts with all of the prompts in previous generations, rather than only the chosen parent prompt.\n  - For the MAC metric for expansion determination, it is weird to use the maximum angle to evaluate the diversity/coverage of ALL learned prompts. How about using the mean angular or KNN metrics?\n  - Adding dispersion loss as well as the MAC metric at the input prompt embeddings is a little bit confusing, where the constraints may change after the encoding process. I wonder what will happen when adding the constraints to the encoded text features.\n  - How to ensure the quality of pseudo labels for class-agnostic object detection?\n3. The comparison with SAM on LVIS is unfair, as DiPEx is fine-tuned on the training set (even without box annotations) while SAM is a pre-trained model. Besides, is there some comparability between DiPEx and UniDetector[1]?\n\n[1] Z Wang et al. \"Detecting Everything in the Open World: Towards Universal Object Detection\", CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and constructive feedback.\n\n> **R2.1** I prefer to owe the detection degradation to the disturbed attention, rather than the so-called semantic overlap.\n\nWe appreciate the insightful feedback. To clarify our \"semantic overlap\" hypothesis, we refer to the example in Appendix A.2. Our empirical study on \"semantic overlap\" is illustrated in Figure 7, where detection confidences are maintained with less similar words (\"plates . cup .\"), but performance drops with similar words (\"plates . dishes .\"). We conjecture that this is highly attributable to the dataset that was used during pre-training - there is a **low chance** of an image containing similar words [C], which may explain the reason why Grounding DINO was structured to **favor specific categories** over generic ones. Owing to this observation, we gained inspiration to **hierarchically** discover more fine-grained concepts from higher-level semantics.\n\n[C] Shuai Shao et al. Objects365: A Large-Scale, High-Quality Dataset for Object Detection, in ICCV.\n\n> **R2.2** Clarification & Additional Experiments\n\nGreat suggestion! We have run an additional experiment based on your suggestion, where we add additional prompts at the initialization stage. Please refer to Table 1 in the attached PDF for your reference. By increasing the number of initial prompts from ($k=1$ to $k=5$), we observe an improvement in the detection performance.\n\n\n> **R2.3** Why MAC metric for expansion determination? How about using the mean angular or KNN metrics?\n\nWe appreciate the reviewer's suggestion regarding the MAC metric for expansion determination. Our choice of **maximum angular coverage (MAC)** is inspired by the observation of semantically dissimilar words have larger angular discrepancies as hypothesized in our case study in Appendix A.2. We aim to capture semantics as comprehensively as possible to improve recall. The angular discrepancy between prompts is enforced by dispersion loss, where we intend to push child pairs further, and maximum angular deviation represents the comprehensiveness of the vocabs. While using mean angular metrics is plausible, it tends to find fine-grained concepts such as \"apple\" & \"oranges\", which can be easily **dominated** by the abundance of fine-grained words. This contrasts with our goal of promoting diversity among learned prompts from a global perspective. KNN on the other hand may signify how **detailed** for a particular pair and the results can be very restricted to a particular concept. Due to limited time during rebuttal, we will add comparisons in the revised version.\n\n> **R2.4** I wonder what will happen when adding the constraints to the encoded text features.\n\nThank you for your feedback！ As we are unsure about your concern, we interpret your questions as \"*how dispersion loss can affect the encoded text features*\". Firstly, we would like to **clarify** that MAC is the criterion for early stopping and it is not directly used in our optimization process. Dispersion loss, on the other hand, is our **training objective** which ensures that child prompts are sufficiently distinct from each other while not straying too far from their parent prompts. This combination ensures that our approach effectively captures diverse and unique concepts while maintaining coherence with the broader parent prompts. The learnable prompts are directly appended to the encoded text. The encoded *textual query* (*e.g.,* \"generic\") is **only used** as a guide to acquiring **pseudo-labels** and it is not optimized as part of the learning objective.\n\n\n> **R2.5** How to ensure the quality of pseudo labels for class-agnostic object detection?\n\nThanks! To re-iterate, we iteratively refine our pseudo-labels to ensure their quality during the training process. We start with predictions from an off-the-shelf GroundingDINO model, removing low-confidence detections and excessively large bounding boxes. At each self-training stage, we update the pseudo-labels by performing inference using the learned prompts. To avoid duplication, we eliminate boxes with an IoU greater than 0.5 compared to the previous \"ground-truth\" boxes and apply SoftNMS.\n\n> **R2.6** Comparability with UniDetector\n\nThanks for the great suggestion! It is important to clarify that UniDetector is an Open-Vocabulary detector (similar to what GroundingDINO was originally designed for), which **differs** from our class-agnostic settings. UniDetector requires comprehensive class labels for training, whereas our approach **only relies** on pseudo-box supervision. Notably, UniDetector also proposes a class-agnostic detector (CLN), which combines both the RPN and RoI head to generate proposals for universal object detection. This highlights the significance of class-agnostic object detection in Open-World scenarios, aligning with our focus on detecting every object in the scene without relying on predefined class labels. \n\n\n> **Q2.1** Detailed operations in Table 1\n\nThank you for your suggestion! As described in lines 52-57, query-merging involves concatenating all queries into a single string (*e.g.,* \"objects . generic . entities .\"), whereas prediction-merging entails combining the results from separate inferences using individual text prompts (*e.g.,* \"objects\", \"generic\", \"entities\").  We apologize for any confusion and will clarify this distinction in the revised manuscript.\n\n> **Q2.2** Misuse of Notations\n\nWe appreciate your feedback! In this paper, we have used the terms \"prompt\" and \"word\" interchangeably, which may have caused some confusion. The notation in lines 134 and 137 is accurate; however, we will revise the manuscript to ensure that $P$ consistently refers to \"prompt embeddings\" for clarity.\n\n**Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PKvZ4DXW2E", "reviewer": "Reviewer_UZCf", "summary": "This paper proposes a novel Dispersing Prompt Expansion (DiPEx) approach to enhance class-agnostic object detection (OD) using vision-language models (VLMs). The authors observe that manually crafted text queries often result in undetected objects due to semantic overlap, and address this by progressively learning a set of distinct, non-overlapping hyperspherical prompts. DiPEx starts with a generic parent prompt, selects the one with the highest semantic uncertainty for further expansion, and generates child prompts that inherit semantics from the parent while capturing more fine-grained details. Dispersion losses are employed to maintain high inter-class discrepancy among child prompts while preserving parent-child consistency. The method utilizes the maximum angular coverage (MAC) of the semantic space as a criterion for early termination to prevent excessive prompt growth. Experiments on MS-COCO and LVIS datasets demonstrate that DiPEx outperforms other prompting methods by up to 20.1% in average recall (AR) and achieves a 21.3% AP improvement over SAM for out-of-distribution OD.", "questions": "See Weaknesses*", "limitations": "See Weaknesses*", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed DiPEx approach seems novel and innovative to the class-agnostic object detection.\n- The paper is well-written and structured, with clear explanations of the concepts, techniques, and experimental setup.\n- The proposed DiPEx method has the potential to significantly advance the state-of-the-art in class-agnostic OD and out-of-distribution detection.\n- Some important works about class-agnostic learning could be appended:\n[1] In ECCV 2022. Pose for everything: Towards category-agnostic pose estimation.\n[2] In CVPR 2023. Matching is not enough: A two-stage framework for category-agnostic pose estimation.\n[3] In CVPR 2024. Meta-Point Learning and Refining for Category-Agnostic Pose Estimation.", "weaknesses": "- The Abstract is not abstract enough.\n- The evaluation should be detailedly described. One intuitive way is to cast all categories into single category, but not all class-agnostic objects in dataset are annotated. How about the evaluation for these unannotated objects?\n- How about the efficiency of the proposed model in training/inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions, which we address below:\n\n> **R3.1** Discussion on more related works\n \nThanks for bringing the literature to our attention!\n\n[1] proposed POMNet, which leverages a transformer-based Keypoint Interaction Module (KIM) to capture interactions between keypoints and the relationship between support and query images. \n[2] introduced CapeFormer, a two-stage framework where keypoints are matched and treated as similarity-aware position proposals in the first stage, addressing noisy matching results due to the open-set nature of the problem.\n[3] introduces a meta-learning approach combined with iterative point refinement techniques for class-agnostic pose estimation.\n\nPlease note these methods are specifically designed for **pose estimation**, which involves predicting key-points on objects. Object detection, on the other hand, focuses on identifying and localizing entire objects within an image. The granularity and nature of the tasks are different, making direct application challenging. Though we find these methods relatively out of scope to our current work, we acknowledge the importance of this area. We will **add a detailed discussion** accordingly in our revised version to broaden its relevance.\n\n> **R3.2** The Abstract is not abstract enough.\n\nThanks for your constructive comment! Our intent was to provide a comprehensive summary of our work, outlining the goals and motivation clearly. While we believe it accurately reflects our research, we will endeavor to make it more concise.\n\n> **R3.3** Clarification on evaluation.\n\nThanks for your suggestion! We confirm that our evaluation approach involves casting all existing categories from MS-COCO and LVIS into **a single class**, and we will provide a detailed explanation in our revised manuscript. As highlighted in our introduction, the primary goal of our paper is to enhance **Average Recall (AR)**, which measures how comprehensive objects are captured. Through visual inspections of ground truth (GT) against our results, we observed that COCO and LVIS are **not** densely annotated (see Figure 9-10 in the manuscript for more visualization). This is the primary reason our AP appears low; **correct predictions** for unannotated instances are **mistakenly counted as FP** in COCOEval. To address this, we plan to develop a fairer and more comprehensive benchmark in the future.\n\n\n> **R3.4** How about the efficiency of the proposed model in training/inference?\n\nThanks for raising this important question! Regarding the efficiency of our model, we have noted this as a limitation in the conclusion section of our paper. We have provided an additional inference cost comparison between our proposed method and two handcrafted baselines in the attached PDF. Our results, detailed in Table 2, demonstrate that while the naive query-merging method reduces the computational cost, it significantly **compromises** detection accuracy due to overlapping semantics. In contrast, our proposed method strikes a balance, achieving superior detection performance. Specifically, our Dipex approach, is 6.69% slower compared to the naive query-merging method, but it shows an impressive **89.34%** reduction in inference time compared to the prediction-merging method. We will incorporate the efficiency analysis into the main body of the revised version!\n\n**Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "clfyGQCUMz", "reviewer": "Reviewer_fTsG", "summary": "The paper proposed a prompt expansion method to produce a diverse set of text queries for class-agnostic object detection. Sequentially performing inference using each text query and collating the prediction results often achieves high recall but incurs significant computational cost. Merging all text queries into one prompt reduces the cost significantly but results in much lower performance. The paper argues that this is due to the semantic overlap amongst the queries. To address this, the paper starts with a learnable parent prompt. After training the prompt embeddings with the standard object detection losses, the parent prompt is expanded into numerous child prompts by applying random rotations. The child prompts are then further learned with additional loss terms to decrease the similarity amongst the child prompt embeddings. This was shown to increase the maximum angular coverage, measured by the largest angle between the embeddings of two child prompts. Experiments demonstrate significant improvement of average recall and average precision on MS COCO and LVIST datasets.", "questions": "1. In Figure 3 (right), the average precision seems of the proposed method seems to drop first before increasing again. Is this due to random noise? Overall, it appears that increasing the number of prompts to 9 yields significant improvements but the trend is not clear before it.\n\n2. In line 269, the paper claims that a higher MAC correlates with a broader spectrum of vocabularies. Aside from intuitions, is the range of vocabularies measured in some way? How did paper come to this conclusion?\n\n3. How exactly is the prompt logit activation computed?", "limitations": "The authors discussed some technical limitations regarding the need of self-supervised learning and hyper-parameter tuning, both of which contribute to additional computational cost during training.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper aims to tackle the task of class-agnostic object detection, which is essential for various applications such as OOD detection. The task moves away from the classic close-vocabulary object detection problem, and has significant practical values as the real world is much less constrained.\n\n2. The paper proposed an interesting approach to grow the set of text queries that are used to prompt a grounding model. The approach is geometrically motivated and is rather intuitive.\n\n3. The paper provided extensive experimental results on benchmark datasets like MS-COCO and LVIS. The improvements in average recall (AR) and average precision (AP) over existing methods demonstrates great potential of the method.", "weaknesses": "1. The semantic overlap amongst words, as the motivation behind the paper, seems more like a hypothesis and was not sufficiently investigated. Lines 53-55 of the paper pointed out that merging all queries into one text prompt results in inferior performance compared to running inference on each query sequentially and combing the predictions. The paper then directly jumped to the conclusion that this was the result of \"semantic overlap\" between the queries, without any investigation\t or reference to prior investigations. This undermines the fluidity of the paper. Even though the proposed method does improve the performance, it may not be the solving the problem the paper is claiming to solve. The hypothesis should not be too hard to test. Semantically similar and dissimilar queries can be manually selected to compare their detection performance.\n\n2. As I understand, the motivation behind merging queries as opposed to just merging predictions is to lower the computational cost. The paper would be more convincing if there is a inference cost comparison between the proposed method, the naive query-merging method and the prediction-merging method.\n\n3. Some technical details of the paper were not stated very clearly and were somewhat hard to follow. There are also numerous typos and inconsistent use of notations. For instance, in section 3.1, subscripts of v denote the text embeddings of different words in the same prompt, but in section 3.2 and other subsequent sections, the subscripts seem to denote different text prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comment and constructive feedback! Please find our detailed response below:\n\n> **R4.1** Clarification on \"semantic overlap\".\n\nWe appreciate the reviewer’s detailed feedback and understand that the section on “semantic overlaps” may seem disjointed. We conducted a **pilot study** using UNIVERSAL-query and CLASS-WIDE query sourced from ChatGPT and WordNet, respectively, to validate the efficacy of GroundingDINO in recognizing semantically similar and dissimilar words. \n\nThese results, shown in Table 1, **support our hypothesis** that semantic overlap negatively impacts detection performance. Additionally, a case study provided in Appendix A.2 reveals **diminished confidence** in GroundingDINO when presented with semantically overlapping queries, as exemplified by the contrast between \"plates . cups .\" (semantically different) and \"plates . dishes .\" (semantically similar), as illustrated in Figure 7. \n\nDue to page limits, we regret any confusion caused by the placement of this crucial information in the supplementary materials, which might have been overlooked. To improve the clarity, we will incorporate this case study into the main body of the revised version.\n\n\n> **R4.2** Inference cost comparison\n\nGreat suggestion! We have provided an additional inference cost comparison between our proposed method and two handcrafted baselines. Our results, detailed in Table 2, demonstrate that while the naive query-merging method reduces the computational cost, it significantly **compromises** detection accuracy due to overlapping semantics. In contrast, our proposed method strikes a balance, achieving superior detection performance. Specifically, our Dipex approach is 6.69% slower compared to the naive query-merging method, but it shows an impressive **89.34%** reduction in inference time compared to the prediction-merging method.\n\n> **R4.3** Technical details clarification\n\nThank you for your feedback! We apologize for the confusion regarding the notations. To clarify:\n\n- In Section 3.1, the contextual embeddings $\\mathbf{v}$ are denoted as $\\mathbf{v} =\\lbrace\\mathbf{v}_i\\rbrace ^{M} _{i=1} \\in\\mathbb{R}^{M\\times d}$, where $M$ represents the number of $d$-dimensional learnable tokens appended to a text query $\\mathbf{c}$. This follows the conventional way of introducing learnable context vectors [A].\n- In Section 3.2, we introduce a novel prompt expansion method. Here, the prompts hierarchy is denoted as $\\mathbf{v}_{l,k}$. The subscript $l$ indicates the layer in the tree or training round, while $k$ represents the number of learnable tokens in the $l$-th layer, similar to the notation in Section 3.1.\n\nWe hope this clarification clears up any confusion regarding the notations.\n\n[A] Kaiyang Zhou et al. Learning to Prompt for Vision-Language Models, in IJCV.\n\n\n> **R4.4** Concerns on Potential Random Noise\n\nThanks! The COCOEval metric is particularly strict in evaluating detection performance, as it only matches detections with the **highest Intersection over Union (IoU)**, which can penalize minor localization errors. Additionally, the MS-COCO dataset is **not comprehensively annotated**, which can even lower the average precision (AP) as correct predictions for unannotated instances are **mistakenly counted** as **false positives**. \nAt the very initial training stage (l=1), the learnable tokens can be highly uncertain. Optimizing a small number of prompts can lead to instability, given the insufficient parameters to capture the diverse pseudo-labels. Selecting only one parent prompt with high uncertainty for expansion in the first iteration may not provide enough stability. However, as the number of prompts progressively increases, the model becomes **more mature**, and we observe a **steady improvement** in performance. This trend illustrates how increasing the number of prompts helps manage uncertainties and improves overall performance, thereby validating our approach.\n\n> **R4.5** In line 269, the paper claims that a higher MAC correlates with a broader spectrum of vocabularies. Aside from intuitions, is the range of vocabularies measured in some way? How did paper come to this conclusion?\n\nWe appreciate the reviewer's thoughtful question! The concept of MAC (Mean Angular Coverage) is inspired by the WordNet hierarchy, where a larger MAC indicates the discovery of more high-level semantics. While MAC serves as an approximation rather than a direct measure of vocabulary range, it provides a **practical criterion** for determining when to stop the training process. The main motivation behind MAC is to capture the **breadth** of concept coverage without the need for direct interpretation of the learned tokens, which is inherently complex. Although MAC does not explicitly measure the vocabulary range, its correlation with high-level semantic discovery offers a useful approximation for our purposes. We hope this clarifies our rationale.\n\n\n> **R4.6** How exactly is the prompt logit activation computed?\n\nThanks for raising this question. As stated in line 285, prompt logit activation refers to the number of activated prompts based on the confidence threshold. Figure 5 illustrates the activation frequency (in log scale) of each expanded child prompt, providing a clear visualization of how different prompts contribute to the detection process.\n\n**Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to extend our sincere gratitude for your thoughtful and encouraging feedback.

We are pleased to see that our exploration into class-agnostic object detection was recognized as **practical and universal** in real-world scenarios, with performance improvements over baseline approaches (Reviewer Jt2M, 86g8, fTsG). Your acknowledgment of the foundational potential of class-agnostic detection for large vision models and the **novelty** of our prompt set expansion method is greatly appreciated (Reviewer 86g8, fTsG).

We are grateful for the positive feedback on the **clarity**, **structure**, and **quality** of our writing and figures (Reviewer Jt2M, 86g8, UZCf). The recognition of the potential **impact** of our DiPEx approach on the state-of-the-art in both class-agnostic and out-of-distribution detection is particularly gratifying (Reviewer UZCf, fTsG).

The appreciation for our geometrically motivated and intuitive approach to expanding text queries for grounding models, along with the comprehensive experimental results on benchmark datasets like MS-COCO and LVIS, further encourages us (Reviewer fTsG)!

The attached PDF contains additional experiments for reference:

- Visualization of the attended areas activated by the learned prompts
- The impact of different lengths of prompts for initialization
- Inference cost comparison in terms of time and memory consumption

**Thank you once again for your valuable feedback. We will incorporate the revisions accordingly to improve the quality of this work. Please let us know if you have any additional questions, and we are more than happy to address them!**

Meta-review: This paper presents a method for class-agnostic object detection with an interesting prompt expansion method. The three reviewers are all positive about the novelty and technical contributions of the work. They had some comments regarding complementing some necessary experimental results and adding missing references. The authors may consider those comments in the revision. The AC is satisfied with the submission and recommends acceptance as poster.
Decision: Accept (poster)