OW-VAP: Visual Attribute Parsing for Open World Object Detection
Authors: Xing Xi
Xing Fu
Weiqiang Wang
Ronghua Luo
Venue: ICML 2025 poster (2025)
Keywords: Object Detection
Open World Object Detection
Open Vocabulary Object Detection
Primary area: deep_learning

Abstract: Open World Object Detection (OWOD) requires the detector to continuously identify and learn new categories. Existing methods rely on the large language model (LLM) to describe the visual attributes of known categories and use these attributes to mark potential objects. The performance of such methods is influenced by the accuracy of LLM descriptions, and selecting appropriate attributes during incremental learning remains a challenge. In this paper, we propose a novel OWOD framework, termed OW-VAP, which operates independently of LLM and requires only minimal object descriptions to detect unknown objects. Specifically, we propose a Visual Attribute Parser (VAP) that parses the attributes of visual regions and assesses object potential based on the similarity between these attributes and the object descriptions. To enable the VAP to recognize objects in unlabeled areas, we exploit potential objects within background regions.  Finally, we propose Probabilistic Soft Label Assignment (PSLA) to prevent optimization conflicts from misidentifying background as foreground. Comparative results on the OWOD benchmark demonstrate that our approach surpasses existing state-of-the-art methods with a +13 improvement in U-Recall and a +8 increase in U-AP for unknown detection capabilities. Furthermore, OW-VAP approaches the unknown recall upper limit of the detector.

OW-VAP? Visual Attribute Parsing for Open World Object Detection
Xing X
Xing Fu
Weiqiang Wang 2
Ronghua Luo
Abstract
SGROD
Open World Object Detection (OWOD) requires the detector continuously identify and learn categories. Existing methods rely the large language model (LLM) describe the vi- sual attributes of known categories and use these attributes to mark potential objects. The perfor- mance such methods influenced by accu- racy L.MM deseriptions, and selecting appropri- ate attributes during incremental learning remains challenge. In this paper, propose novel OWOD framework, termed OW-VAP, which op- erates independently of LLM and requires only minimal object descriptions to detect unknown ob- jects. Specifically, we propose Visual Attribute Parser (VAP) that parses the attributes of visual regions and assesses object potential based on the similarity between these attributes and the ob- ject descriptions. To enable the VAP to recognize objects in unlabeled areas, we exploit potential ob- jects within background regions. Finally, we pro- pose Probabilistic Soft Label Assignment (PSLA) to prevent optimization conflicts from misiden- tifying background as foreground. Comparative results the OWOD benchmark demonstrate that our approach surpasses existing state-of-the- art methods with a +13 improvement in U-Recall and a +8 increase in U-AP for unknown detection capabilities. Furthermore, OW- VAP approaches the unknown recall upper limit of the detector.
17.5
KTCN
15.0
OVOw*
OW-VAP
12.5
OW-VAP*
U-AP
10.0
7.5
5.0
2.5
0.0
30
40
50
60
70
80
U-Recall (MOWODB)
Figure I. Comparison Curves for U-AP and U-Recall. All met- rics evaluated on MOWODB benchmark. *, A and O denote the performance Task 1, Task 2, and Task 3. respectively. indicates the performance after removing duplicate images. Our OW-VAP significantly outperforms other models.
referred as closed-set detectors, are trained on a fixed set of categories and, therefore, are only capable of recog- nizing these predefined categories. However, in real- world scenarios, annotating all possible categories is impractical. To address this limitation, Open World Object Detection (OWOD) has been proposed (Joseph et al., 2021). The detec- tion of new categories in OWOD is divided into two steps. Initially, the detector is trained on dataset with a fixed of categories and is required to detect objects that not labeled in the training set during the inference phase. Subsequently, these objects selected and introduced in subsequent incremental learning processes. The detector is then fine-tuned based on existing knowledge, thereby gaining the ability to detect these new categories.
1. Introduction
Object detection is fundamental task in the field of com- puter vision, aiming classify and localize objects within an image (Tian et al., 2019; Lin et al., 2b). Traditional object detectors (Li et al., 2020b; Liu et al., 2016), often
Existing work can be broadly classified into two types. The first assumes that potential objects share certain similarities with known categories. Detectors demonstrate anomalies when they label objects similar to known categories as neg- ative samples. OW-DETR (Gupta et al., 2022) assesses feature maps corresponding to negative sample regions, se- lecting the top samples with the highest scores as potential objects. The second utilizes the knowledge of visual foun-
'Shool of Computer Science & Engineering, South China Uni- versity of Technology. Guangzhou, China Ant Group, Hangzhou, China. Correspondence to: Ronghua Luo <rhluo @ edu. Xing Fu <zicai.fx fx antgroup.com com?>. comm..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Parser (VAP) parse the attributes corresponding to visual regions. Since VAP trained known classes (Fang et al., 2023), there exists a label bias towards the attributes of known objects. Therefore, during VAP training, we mine potential objects in background samples as pseudo labels. However, these pseudo labels do not always correctly en- compass objects. To prevent optimization conflicts arising from pseudo labels, we assign soft labels pseudo labels based on loss distribution probabilities. Finally, the trained VAP can parse the attributes corresponding to given object region to estimate whether it contains an object, independent of the tine-graine attributes provided by LLM.
Textual Encoder
Known Names
URAZOWn
Color is
Rely on accuracy of LLM
Attribute selection when
learning new categories
Visual Attribute Parser
Color
Parse Attributes
Shape
Size
We evaluate OW-VAD on standard OWOD benchmarks, MOWODB and SOWODB, which are composed of mix- ture of VOC (Everingham al., 2010) and COCO (Lin et al., 2014) datasets. In Figure 1, we present the curves of U-Recall and U-AP on MOWODB benchmark, which are the primary metrics of interest for OWOD. OW-VAP outper- forms the previous state-of-the-art (SOTA) methods, by a margin of 13+ U-Recall. Furthermore, on stricter evaluation metrics, we achieve an 8+ U-AP performance advantage. Notably, OW-VAP has reached even surpassed the upper limit of OVD's generalization capability in terms of recall. In Task 2, surpass the recall upper limit of OVD using all class names by 0.3 U-Recall, achieving 56.3 U-Recall. We summarize our contributions as follows:
Behavior
Figure 2. Unknown Object Detection Illustration. The top see- tion illustrates the previous OVD-based OWOD framework, which relies LLM for object descriptions, thereby being constrained by the accuracy of LLM. During incremental learning, balancing the contribution of each attribute is challenging. The bottom section depicts our proposed VAP, which learns to parse attributes direetly from regions without relying on guidance from LLM.
dation models for supervision. KTCN (Xi et al., 2024) uses the Segment Anything Model (SAM) (Kirillov et al., 2023) to generate candidate boxes for all objects in the image and selects candidates that do not overlap with labeled objects as pseudo labels. Recently, leveraging the generalization capabilities of Open Vocabulary object Detection (OVD) to detect unknown objects has become the mainstream, achiev- ing significant recall advantages.
We propose OW-VAR, a novel detection framework for OWOD that does not rely on guidance from LLM.
We propose the visual attribute parser (VAP) to parse the visual attributes corresponding to the current region, assessing the likelihood of containing objects.
We propose the probabilistic soft label assignment (PSLA) to mitigate optimization conflicts arising from background noise.
As depicted in Figure 2 (top), these detectors assume that unknown objects share certain attributes with known cate- gories, such as the color blue in the appearance of the object. During inference, objects exhibiting similarities in these attributes are labeled as unknown category. However, these detectors have two main drawbacks. Firstly, all predictions based on the attributes provided by Large Language Model (LLM). Consequently, their performance is highly susceptible to the accuracy of attribute descriptions by LLM. Secondly, in subsequent incremental learning, it is chal- lenging to quantify the contribution of each attribute to the unknown class, leading to difficulties in attribute selection. Based on these limitations, we propose a novel detection framework, termed OW-VAP, in this paper.
We validate the effectiveness of OW-VAP the stan- dard OWOD benchmark. OW-VAP surpasses the state- of-the-art (SOTA) methods with an advantage of over 13 U-Recall and 8 U-AP in unknown detection. Fur- thermore, OW- VAP exceeds the generalization upper limit of OVD Task 2 of MOWODB.
2. Related Work
2.1. Object Detection
Object detection (OD), fundamental task in computer vi- sion, aims to detect and localize objects of interest within images. Early detection methods employed a two-stage process, dividing the detection task into candidate proposal and refinement stages (Girshick, 2015; Ren et al., 2015). The former provides coarse, class-agnostic bounding boxes
As shown in Figure 2 (bottom), the key to OW-VAP lies in enabling the detector to learn to parse the attributes of objects. Specifically, we utilize a set of vague sentences to describe an object, focusing on attributes such as shape, scale, and color, Subsequently, we train Visual Attribute
2

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
for objects, while the latter refines these boxes and per- forms classification. However, due to the slower detection speed of such methods, single-stage detection approaches have been proposed (Wang et al., 2021: Chen et al., 2021). Unlike their two-stage counterparts, single-stage methods directly perform dense predictions on the downsampled fea- ture maps. Typical representatives include SSD (Liu et al., 2016) and the YOLO series (Ge et al., 2021; Wang et al., 2023a), which have become the preferred choice in the in- dustry (Zhang et al., 2022; Wu et al., 2021). In recent years, the sequence attention mechanism of transformers has been proven effective in the field of natural language processing. Consequently, DETR (Carion et al., 2020) transformed the detection task into set prediction, successfully introducing this technology to OD. In addition, some studies have fo- cused on addressing issues arising during OD training and inference, including multi-scale feature fusion (Lin et al., 2017a; Tan et al., 2020), non-maximum suppression (Sun et al., 2021; Wang et al.. 2020), enhancing Transformer con- vergence speed (Roh et al., 2022; Pu et al., 2023), training data imbalance(Lin et al., 2017b), and bounding box am- biguity (Li et al., 2020b;a). Although these methods have played significant role in advancing the development and industrial application of object detection technology, they generally follow closed-set setting, recognizing only the categories annotated in the training data.
its ability to surpass the constraints of closed-set detectors.
2.3. Open World Object Detection
Open World Object Detection (OWOD), distinct from OVD, employs a two-stage process for recognizing new objects (Joseph et al., 2021). Initially, the detector is trained closed set of classes. During inference. it identifies poten- tial objects of interest. These objects are then selected by annotators and subjected to incremental learning for fine- tuning. Consequently, predicting potential objects during inference is crucial. Prior to the introduction of foundation models, researchers focused on generating pseudo-labele for unknown elasses (Zohar et al., 2023h; Zhao et al., 2022). ORE leverages the class-agnostic capabilities of the Region Proposal Network (RPN) to mark high-scoring background samples as potential objects. OW-DETR scores background regions and selects high-scoring backgrounds as pseudo- labels. CAT, UC-OWOD, and RandBox (Ma et al., 2023a; Wu et al., 2022; Wang et al., 2023c) utilize selective search to provide candidates. The introduction of foundation mod- els significantly enhances the capabilities of OWOD detec- tors. KTCN and SGROD (Xi et al., 2024; He et al., 2024) distill knowledge from SAM (Kirillov et al., 2023) to gener- ate pseudo-labels, while SKDF and ovow (Ma et al., 2024; Li et al., 2024) leverage the generalization capabilities of OVD detectors. These methods have achieved significant performance breakthroughs. In this paper, we propose OW- VAP, which enables detectors to learn to parse the attributes of visual regions without relying on guidance from I.I.M
2.2. Open Vocabulary Object Detection
Open Vocabulary Object Detection (OVD), an emerging field, was introduced by OV-RCNN (Zareian et al., 2020). OVD redefines the classification process of the detection head as a matching task between class names and regions. he advantage of this matching process is that the number of matches can dynamically vary. Thus, OVD detectors can detect an arbitrary number of classes (Wang et al., 2023d; Saito et al., 2022). Furthermore, benefiting from large-scale pre-training model. such as CLIP (Radford et al., 2021), OVD can demonstrate zero-shot detection advantages for completely unseen classes (Fang et al., 2024; Jin et al., 2024). Current OVD research focuses on aligning regions with text (Wu et al., 2023a; Yao et al., 2023), knowledge distillation (Wang et al., 2023b; Gu et al., 2022), learning region prompts (Feng et al., 2022; Wu et al., 2023b), and large-scale pre-training (Yao et al., 2022: Li et al., 2022). These approaches enable OVD to exhibit strong visual en- coding capabilities, sometimes even surpassing fully super- vised closed-set detectors in certain scenarios (Wang et al., 2023d). However. OVD still faces several limitations in practical applications. The main issue is that, despite its theoretical ability to detect any category, OVD still relies predefined set of categories for text matching in practice. This reliance restricts its capacity to adapt to and handle new categories in dynamically changing environments, limiting
3. Method
3.1. Overall
The architecture of OW-VAP, as illustrated in Figure 3, builds upon the YOLO-World detector (Cheng et al., 2024). Compared to the standard OVD detector, our model incorpo- rates two additional components: coarse textual sentences and Visual Attribute Parser (VAP). The textual sentences provide generic attributes that describe objects, while the VAP learns to parse these attributes from visual embeddings (Section 3.2 and Section 3.3). During the inference phase for known classes, we employ the OVD inference process, which classifies objects based on the cosine similarity be- tween visual and class name embeddings. For the detection of unknown classes, we utilize the VAP and attribute texts to estimate the likelihood that region contains an object (Sec- tion 3.4). Finally, we apply standard OD post-processing. including maximmm suppression (NMS) and ranking, to bounding boxes for both known and unknown classes to yield the final predictions. The problem definition of OWOD and pseudocode for the proposed components (VAP, PSLA) are shown in Appendix B and Appendix H, respectively.
3

---

VAP: Visual Attribute Parsing for Open World Object Detection
Visual and Textual Embedding Generate
Visual A Atribute Parser Training
Unknown Inference
Weibull
Shape, Color, .... A Apearance,, Size
(xa, Wy, Ag) box,
Pretrained
(Og. Na,w,, As) hox,
Textual Transformer Encoder
Matching
Update
(isy, Wa, Asg boxy (Xa, Na, Ma) box,
Person, Bottle, .... Pottedplant, Car
Soft Label
Visual A ttribute Parsor
Known Names
Object ttributes
ViL
Known Field
Object Fleld
Background Field
Visual Backbone & Feature Pyramid
Derhk
Reaand
Figure 3. Training and Inference Pipeline OWP The OW-VAP framework is built on the standard OVD detector, YOLO-Wor World (Cheng et al., 2024). During training, known class names and vague textual attributes are encoded into text embeddings via a text encoder, while images are converted into visual embeddings through a visual encoder (left, Visual Backbone Feature Pyramid (Lin al., 2017a)), The Visual Atrrbute Parser (VAP) learns to extract the corresponding textual attributes from these visual embeddings (middle). The trained VAP is utilized to estimate the likelihood of an object being present in given region. By combining the VAP's estimates with the similarity of vague text, unknown objects are identified (right).
3.2. Visual Attribute Parser
where Y, denotes the annotation of the current image, en- compassing known class labels and bounding boxes. A denotes the matching score of the current region known classes, encompassing both classification and localization metrics, such as CIOU (Zheng et al., 2020). The matching method (T)) is employed to execute selection based on match- ing scores, such as the TAL (Feng et al., 2021). S E [0, 1] represents the value assigned to each sample post-matching, where 0 indicates negative sample.
After completing the training phase, we expect the Visual Attribute Parser (VAP) effectively interpret the attributes associated with the visual embeddings. To achieve this, we predefined a set of object descriptions that differ from the detailed sentences typically generated by LLM. These de- scriptions focus on determining whether certain attributes are prominent, such as The object has a noticeable color (refer to the details in Appendix A). Then, the text encoder processes these sentences and known class names gener- ate attribute and category embedding:
Subsequently, we label the visual embeddings correspond- ing to the positive samples as positive samples for VAP after parsing. However, as depicted in Figure 3 (middle), there exist some potential objects in the background. If only known labels are used for matching, it causes VAP to learn extraction representations solely of known classes, leading to bias. Consequently, we label some pseudo-labels from the background. We establish three conditions for annotat- ing pseudo-labels. Due the generalization capability of OVD, predictions with high similarity are more likely to be positive sample. The similarity to textual attributes must be greater than or equal to mean similarity of positive samples:
Eatt (Att()t)), E. Ene;(Name)
(1)
where Enc; denotes the text encoder, specifically the pre- trained CLIP text encoder (Radford et al., 2021). Eatt rep- resents the attribute embedding, while E, denotes the class embedding. Att and Name represent the attribute sen- tences and class names, respectively. Subsequently, we employ the vision encoder (Ene,) to obtain the visual em- beddings Euis- = Enc,, (7), where denotes the input image. The VAP, which is composed of a simple multilayer per- ceptron, learns to extract attributes from Evis. To ascertain the positive samples during training, we rely on the match- ing process of known classes ! Specifically, we utilize the matching results of known classes filter both positive and negative samples:
Cond 1:
max Sim( Eviss Bott) 2
mean rt max Au Sim Eatt);
where Sim(-) denotes cosine similarity, while maxj, and mean refer to the maximum and mean values, respectively. E represents the positive sample visual embeddings se- lected through the matching method. When the similarity of all attributes is uniformly low, there is a higher likelihood that the samples selected by Cond contain background
S = argginn n (E,, Evis, Yi),
(2)
For clarity, omit the processes of the VLFPN (Cheng al., 2024), classification and regression head.

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Known and Background Distribution Difference
Previous Knewn Distribution
Soft Label Generation
Background
Current-1 Current- 0
Negative Welbul Soft Label
Figure Probabilistic Soft Label Assignment Pipeline. The left illustrates represent the loss distributions for the positive and the background samples as generated by the assignment method, respectively. Current-1 and Current-2 represent the fitting results from one and two epochs prior, respectively. while Current-O O denotes the current result. The right represents the generation soft labels, where the Final Positive Weibull and Negative Weibull denote the final positive sample distribution and the current negative sample distribution, respectively. Soft Label refers to the distribution the generated pseudo-labels.
distributions, where Known and Background
noise. Consequently, Cond 2 employs a lower threshold to further refine the dataset by filtering out these samples:
right. Therefore, exploit these distribution differences to balance the optimization conflicts.
Cond 2: max Sim(Evis Eatt) 2 a,
(4)
Wb(x|A,k,0) = k(x - t (o-
(6)
where a is a hyperparameter that controls the minimum threshold similarity. Finally, consistent with the previous methods (Joseph et al., 2021; Gupta al., 2022), we select only the top samples with the highest similarity from the background candidates:
where k, and represent the scale, shape, and location pa- rameters, respectively. To determine the distribution parame- ters of positive and negative samples, likelihood estimation. However, the distribution of positive samples is sparser compared to that of negative samples, resulting in greater fluctuations in the loss distribution. To address this, we utilize the moving weighted average to mit- igate these fluctuations. As shown Figure (middle), retain the historical distribution of positive samples, extract the corresponding distribution parameters, and maintain queue of these parameters. After modeling in the current epoch, the corresponding parameters are added to the pa- rameter queue. Subsequently, we calculate the weighted average in the queue to generate the final distribution:
employ maximum
Cond 3: Tops (max, Ard Sim (eess Eatt))
(5)
where 8 is hyperparameter used to control the number of selections. We consider a sample as potential target and assign it pssudo-label only it satisfies all three conditions simultaneously, Snap 1(Cond where 1(-) is the indicator function, which equals if the condition satisfied and 0 otherwise.
Cond Cond 3),
3.3. Probabilistic Soft Label Assignment
* = Σ
To identify high-quality candidates, we employ three condi- tions filter out background. However, the background still infiltrates pseudo-labele leading to optimization conflicts. To address this, inspired by recent work (Fang al., 2023), we propose Probabilistic Soft Label Assignment (PSLA), which estimates the probability of pseudo-labels being true positive samples based on their loss distributions.
where denotes the length of the queue. A; represents the i-th scale parameter within the queue of positive sample parameters. The coefficients used here are derived from an arithmetic sequence designed to emphasize the contribution of the most recently added distribution to the queue. The first term and common difference of this sequence are de- noted by -2 and 2n -2, respectively. The generated soft labels (Figure right) are:
The training of VAP minimizes the distance the attributes in the embedding space. When all attributes are treated positive samples and the closest attributes pulled closer, the loss exhibits an inconsistent distribution. As shown in Figure& (left), the loss in the labeled sample is lower compared to the background region, distributed to the left of the background sample. Background samples also exhibit consistent pattern, with higher losses distributed the
W(x) wbt (Wb+(x) (Wb (x) Wb (2))
where W8+ (.) and Wb (.) represent the positive sample distribution (as shown in Equation (7)) and the negative sample distribution, respectively. Ultimately, we employ the
5

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Table Performance Comparison on the OWOD Standard Evaluation Benchmark. The top section shows MOWODB, while the bottom section shows SOWODB. Previously Кnown and Current Known represent the categories introduced in previous and current tasks, respectively. Both indicates average performance across previously seen categories. * denotes evaluation after deduplicatiin the test set. Zero-Shot (GT) represents zero-shot testing with unknown class names to evaluate the OVD detector's generalization upper bound.
Task IDs (-)
Task
Task
Task 3
Task
U-Recall mAP (f)
U-Recall
mAP(t)
U-Recall
mAP(t)
mAP(1)
Method
Current
Previously
Current
Previously
Current
Previously Current Both
(*)
(1)
Both
(1)
Both
Known
Known
Known
Known
Known
Known
Known
OREJEBUI (Joseph et al., 2021)
4.9
56.0
2.9
52.7
26.0
39.4
12.7
29.7
29.6
12.4
25.3
OW-DETR (Gupta et 2022)
7.5
59.2
6.2
53.6
33.5
42.9
15.8
30.8
31.4
17.1
27.8
ALLOW (Ma et al., 2023b)
13.6
59.3
10.0
53.2
33.5
42.9
14.3
42.6
26.7
38.0
33.5
21.8
30.6
PROB (Zohar et ul., 2023b)
19.4
59.5
17.4
55.7
32.2
44.0
19.6
43.0
22.2
36.0
35.7
18.9
31.5
CAT (Ma et al., 2023a)
23.7
60.0
19.1
55.5
32.7
44.1
24.4
42.8
18.7
34.8
34.4
16.6
29.9
RandBox (Wang al.,, 2023c)
10.6
61.8
6.3
45.3
7.8
39.4
35.4
EO-OWOD' (Sun ct al., 2024)
24.6
61.3
26.3
55.5
38.5
47.0
29.1
46.7
30,6
41.3
42.4
24.3
37.9
Hyp-OW (Doan et al. 2024)
23.5
59.4
20.6
44.0
26.3
36.8
33.6
MEPU-PS (Fang al., 2023)
31.6
60.2
30,9
57.3
33.3
44.8
30.1
42.6
21.0
35.4
34.8
19.1
30.9
SGROD (Hle al., 2024)
34.3
59.8
32.6
56.0
32.3
44.9
32.7
42.8
22.4
36.0
35.5
18.5
31.2
SKDF (Ma et al., 2024)
39.0
56.8
36.7
52.3
28.3
40.3
36.1
36.9
16.4
30.1
31.0
14.7
26.9
KTCN (Xi al., 2024)
41.5
60.2
38.6
55.8
36.3
46.0
39.7
43.5
22.1
36.4
35.1
16.2
30.4
ovow" (Li et al., 2024)
73.5
72.1
77.5
72.4
51.0
61.7
76.1
61.6
41.6
54.9
56.0
34.3
50.6
Zero-Shot (GT)
59.4
69.3
56.0
69.3
41.8
55.5
55.7
55.5
31.4
47.5
47.5
26,1
42.1
Ours: OW-VAP
58.8
68.8
56.3
68.8
42,5
55.6
55.1
55.7
29.9
47.1
47.3
25.8
42.0
Ours: OW-VAP
86.6
74.5
86.8
75.6
50.1
62.3
84.8
62.4
40.1
55.0
55,2
34,8
50.1
OW-DETR (Gupta al., 2022)
5.7
71.5
6.2
62.8
27.5
43.8
6.9
45.2
24.9
38.5
38,2 2
28.1
33.1
CAT (Ma et al., 2023a)
24.0
74.2
23.0
67.6
35.5
50.7
24.6
51.2
32.6
45.0
45.4
35.1
42.8
PROB (Zohar al., 2023b)
17.6
73.4
22.3
66.3
36.0
50.4
24.8
47.8
30.4
42.0
42.6
31.7
39.9
EO-OWOD (Sun et al., 2024)
24.6
71.6
27.9
64.0
39.9
51.3
31.9
52.1
42.2
48.8
48.7
38.8
46.2
Hyp-OW (Doan al. 2024)
23.9
72.7
23.3
50.6
25.4
46.2
44.8
MEPU-FS (Fang et al., 2023)
37.9
74.3
35.8
41.9
54.3
50.2
38.3
46.2
43.7
33.7
41.2
SGROD (He al., 2024)
48.0
73.2
48.9
64.7
36.7
50.0
47.7
47.4
32.4
42.4
42.5
32.6
40.0
SKDF (Ma et al., 2024)
60.9
69.4
60.0
63.8
26.9
44.4
58.6
46.2
28.0
40.1
41.8
29.6
38.7
ovow (Li et al., 2024)
71.3
76.4
74.4
75.0
59.8
67.0
74.6
67.0
53.8
62.6
65.5
56.9
63.4
Zero-Shot (GT)
87.3
79.0
86.5
79.0
58.4
68.2
86.3
68.2
56.7
64.4
64.4
58.0
62.8
Ours: OW-VAP
82,9
79.6
85.6
79.8
62.3
70.6
85.7
70.7
57.6
66.3
66.4
59.1
64.6
soft labels to weight the pseudo-label loss:
probability of the unknown class is denoted by:
Epscudo = mean(T(E W(T( CCT((E
Pa (e(rss)) = (1 max Sim(evis, Fe))
T(E.))) = ( (au Sim (M()" Eatt) B), (9)
Sim
(b mean Sum Fass) + (1 b)T(emis))- (10)
where CE(- [1) denotes the cross-entropy loss for the current prediction when the target 1. The parameters 7 and / are learnable scaling factors, both being scalars. Ei and T(-) represent the selected pseudo-label visual embeddings and VAP, respectively. a and M denote the Sigmoid function and Multilayer Perceptron (MLP). For other samples, we simply the apply cross-entropy to calculate loss (Appendix E).
4. Experiments
Details about the dataset, evaluation metrics, and implemen- tation can be found in the Appendix C, Appendix and Appendix
4.1. Main Results
We compare OW- VAP with previous state-of-the-as (SOTA) methods on the OWOD standard benchmarks, MOWODB and SOWODB. The results shown in Table 1. Further comparisons can be found in Appendix G. Due to data leak- age in ORE(Joseph et al., 2021), we follow previous work (Wang et al., 2023c; Doan et al., 2024) by using the en- ergy
3.4. Unknown Inference
Once the VAP is trained, we extract attributes from all vi- sual regions to estimate the probability of unknown classes. To avoid confusion between known and unknown objects, consistent with prior work (Zohar et al., 2023a), we simply add the -uito--diitribution probability to balance the model. Thus, for given visual embedding Cviss the corresponding
version (ORE-EB EBUI) for evaluation. FO-OWOD performs deduplication on the MOWODB test
6

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Table Detection Accuracy of Unknown Classes and Confusion with Known Classes on the MOWODB Benchmark. U-AP denotes
average precision for unknown classes, which is a widely used metric in object detection. WI and A-E indicate the confusion between known and unknown classes the open set. WI represents loss known class precision after introducing unknown class detection. A-OSE indicates the absolute number of unknown objects predicted as known classes. denotes results after deduplication of test set.
Task IDs (-)
Task 1
Task 2
Task 3
U-Recall
U-AP
WT
A-OSE
U-Recall
U-AP (T)
WI
A-OSE (1)
U-Recall
U-AP
WI
A-OSE
Method
(1)
(1)
(3)
(4)
(1)
(L)
(1)
(t)
())
(U)
ORE (Joseph al., 2021)
5.7
0.7
0.0621
10459
2.7
0.1
0.0282
10445
2.3
0.1
0.0211
7990
SA (Yang et al., 2021)
1.9
0.2
0.0563
23320
0.8
0.0
0.0181
16768
0.1
0.0
0.0136
1428
ALLOW (Ma ct al., 2023b)
13.6
4.9
0.0564
45689
10.0
0.7
0.0274
24709
14.3
0.4
0.0194
14952
OW-DETR (Gupta ct al., 2022)
7.7
0.1
0.0599
42331
5.8
0.0
0.0319
25857
6.0
0.0
0.0220
18056
PROB (Zohar et al., 2023h)
19.2
1.6
0.0574
5238
17.1
0.4
0.0342
6555
19.0
0.4
0.0154
2732
E0-OWOD* (Sun et al., 2024)
24.7
2.6
0.0303
4163
26.7
0.2
0.0097
1800
32.1
0.1
0.0075
1509
SGROD (He et al., 2024)
33.2
3.6
0.0450
4567
31.8
2.5
0.0279
2624
31.7
0.3
0.0152
1549
SKDF (Ma et al., 2024)
39.0
1.5
0.0698
13393
36.8
1.3
0.0279
7829
36.1
0.8
0.0168
5142
KTCN (Xi et al., 2024)
41.5
1.2
0.0809
13368
38.6
0.9
0.0439
11861
39.7
0.5
0,0250
6382
ovow* (Li et al., 2024)
73.5
10.4
0.0175
1038
77.5
10.2
0.0047
529
76.1
9.4
0.0030
448
Zero-Shot (GT)
59.4
24.9
0.0213
15761
56.0
21.1
0.0150
13978
55.7
19.1
0,0114
12185
Ours: OW-VAP
58.8
13.0
0.0185
996
56.3
9.3
0.0082
531
55.1
7.8
0,0059
443
Ours: OW-VAP*
86.6
18.9
0.0194
983
86.8
14.1
0.0085
523
84,8
11.8
0.0059
435
Table Ablation Study on MOWODB Benchmark. Base Model refers to the OVD detector utilizing only known class names. General Prompt denotes the use of generalized attribute descriptions, and OOD Prob indicates the out-of-distribation probability. VAP significantly enhances the detector's recall capability for unknowns, while PSLA focuses on improving the precision of detecting unknowns.
Task IDs (-+)
Task
Task
Task 3
U-Recall (t)
U-AP
WI
A-OSE (4)
U-Recall
U-AP
WI
A-OSE
U-Recall
U-AP
WI
A-OSE
Method
(1)
(1)
(1)
(f)
(1)
())
(1)
(1)
(1)
(4)
Base Model
0.0
0.0
0.0332
30068
0.0
0.0
0.0247
25685
0.0
0.0
0.0189
21816
+ General Prompt+ OOD Prob
45.6
10.4
0.0216
17135
44.2
7.5
0.0157
14298
45.7
6.1
0.0137
13925
Visual Attribute Parser
58.5
10.5
0,0210
1025
55.7
6.7
0.0087
544
54.8
6.2
0.0061
438
Probabilistic Soft Label
58.8
13.0
0,0185
996
56.3
9.3
0.0082
531
55.1
7.8
0.0059
443
set, resulting in a reduction of 1k test images and 5k test instances. To ensure fair comparison, we introduce deduplicated test version, OW-VAP*, represent the per- formance. Clearly, OW-VAP surpasses previous SOTA by significant margin. Compared to methods without the foun- dation model, we achieve double the performance of CAT (Ma et al., 2023a) and PROB (Zohar et al., 2023b). Against methods utilizing the foundation model, our approach also demonstrates substantial performance advantages. When compared SAM-based methods like SGROD (He et al., 2024) and KTCN (Xi et al., 2024), we lead by 17.9 U-Recall. Compared to OVD detector methods, such as SKDF (Ma et al., 2024), we achieve a 20.4 U-Recall advantage. Notably, in Task 2, we surpass the generalization upper bound of the OVD detector (Zero-Shot (GT)) by +0.3 U-Recall. Against deduplicated SOTA methods, ovow (Li et al., 2024), we also achieve significant improvements, with a 13.1 U-Recall advantage in Task 1. In SOWODB, we exceed ovow with a recall advantage of over 10 U-Recall, while maintaining unknown recall close to zero-shot testing.
4.2. Comparison of Accuracy and Confusion
To better demonstrate the detector's ability to identify un- known classes, we use more stringent metric, U-AP, which is the most widely used metric for object detection.. Consis- tent with prior work (Zohar et al., 2023b; Ma et al., 2023b), we also employ WI and A-OSE to assess the confusion be- tween unknown and known classes. The results are shown in Table 2. To obtain U-AP, we first extract relevant val- ues from ALLOW (Ma et al., 2023b). For the remaining U-AP values, we use the official code and download the corresponding weights for evaluation. Note that in official updates, the detector's performance may differ from what is presented in the paper. OW- exhibits significant perfor- mance advantages. In U-AP, OW-VAP leads across all tasks, achieving a maximum gap of 8.5 U-AP in Task 1. Addition- ally, in Task 1, we achieved 13.0 U-AP, falling short of the OVD generalization upper limit by only 11.9 U-AP. In the WI comparison, OW-VAP shows marked improvement over previous methods, lagging slightly behind ovow only in the
7

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
SKDF
SGROD
301
oroglo 0.5
UNDONT
Qualitative Comparison with Recent SOTA using Foundation Model on MOWODB, For fair evaluation, rank predictions by unknown confidence and display only top three. The results of our VAP are shown the second row.
Figure 5.
deduplicated comparison. In the A-OSE comparison, OW- VAP surpasses all other methods, even outperforming the deduplicated ovow* in Task 1. Furthermore, in the unknown recall comparison, OW-VAP achieves a performance gap of only surpasses other SOTA
4.4. Visualization
We performed a qualitative comparison of SOTA methods using the foundation model, with results presented in Fig- ure 5. OW-VAP demonstrates strong capability in detecting unknown classes. Compared to SKDF, our method correctly identifies objects in the background, such boats, pad- dles, and icebergs, whereas SKDF redundantly predicts the boat and overlooks other objects. Comparison with ovow, OW-VAP successfully detects trees in the background and small-sized individuals beneath the aircraft. ovow, how- ever, erroneously identifies the circle within the aircraft as a potential object, indicating lower detection precision. Sim- ilar to ovow, KTCN misclassifies paintings within objects as potential targets, an error not present in our OW-VAP. SGROD, like SKDF, repeatedly predicts the same poten- tial object, whereas OW-VAP accurately recognizes other background objects, detecting an abandoned car and tire. Overall, OW-VAP exhibits superior recall capability and detection precision for unknown objects.
over 10. Overall, OW-VAP methods in its ability to detect unknown classes but also maintains, and in some cases, exceeds, their performance in distinguishing between known and unknown classes.
4.3. Ablation Study
To validate the effectiveness of the proposed components, we conducted incremental ablation experiments shown in Table 3. For details on hyperparameter ablation, please refer to the Appendix F. Upon introducing attribute descriptions and out-of-distribution probability, the detector's 's recall and average precision improve to 45.6 and 10.4, respectively, demonstrating the OVD detector's strong generalization ca- pability. However, confusion between known and unknown objects remains high. With the integration of VAP, which learns to parse visual region attributes, recall is significantly improved. In Task 1, VAP achieves an increase of 12.9 in U-Recall. Nevertheless, in subsequent incremental exper- iments, VAP slightly lags in U-AP. In Task 2, VAP falls behind the coarse attribute method by 0.8 in U-AP. Finally, PSLA distinguishes positive and negative samples during training, which significantly enhances the model's 's detection performance. It achieves the highest detection precision and recall, with 58.8 U-Recall and 13.0 U-AP in Task 1. In addi- tion, PSLA demonstrates robust performance in incremental learning, where it not only maintains a consistent advan- tage U-Recall but also shows substantial improvements in U-AP. Moreover, maintains advantage in incremental learning, achieving notable improvements in recall while enhancing 2.6 and 1.6 U-AP in Tasks 2 and 3, respectively.
5. Conclusion
In this paper, we propose novel network architecture, OW- VAP. To identify unknown objects, we propose Visual Autribute Parser (VAP) that analyzes the attributes of visual regions and assesses the likelihood of these regions contain- ing unknown objects. We select samples from background regions that exhibit attribute similarity above the average level and label them pseudo labels. Furthermore, propose the Probabilistic Soft Label Assignment (PSLA) to mitigate the optimization conflicts arising from background noise in pseudo labels. In evaluations of OWOD, OW-VAP demonstrates performance that approaches or exceeds the generalization capability limits. We anticipate that OW-VAP will advance the application of OWOD in real world.
8

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Impact Statement
Everingham, M., Gool, L. V., Williams, C. K. I., Winn, J. M., and Zisserman, A. The pas- cal visual object classes (voc) challenge. Int. J. Comput Vis., 88(2):303-338 2010. URL
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
wtt.:iiitttreeeeeed////oooa j/w.r/////////allsss ijcv/ijcv88 html wtt.://Evvernggaaagagaaaa
Acknowledgements
Fang, R., Pang, G., Zhou, L., Bai, X., and Zheng, J. Unsu-
pervised recognition unknown objects for open-world object detection. CoRR, abs/2308. 165277, 2023, URL www.uuniittte..e http://dbll..niitrrieer corr/ www.hhtmmmc......... #abs 2308-16527.
The authors from Ant Group are supported by the Lead- ing Innovative and Entrepreneur Team Introduction Pro- gram of Hangzhou (Grant No.TD2022005) and National Key Research and Development Program of China (Grant No. 2024YFE0105400) In addition, this work sup- ported by Ant Group Research Intern Program. We would like to thank all the anonymous reviewers for their insightful comments.
Fang, R., Pang. G., and Bai, X. Simple image-level classification improves open-vocabulary object detec- tion. In Wooldridge, M. J., Dy, J. G., and Natara- jan, S. (eds.), AAAI, pp. 1716-1725. AAAI Press, 2024. URL http: /ttpp//dlppppp/// uni-trier. wwweerededeeee de/db/ conf htte//aaaaaaaaaaa
html
References
Carion, N., Massa, F,, Synnaeve, G., Usunier, N., Kir- illov, A., and Zagoruyko, S. End-to-end object detection with transformers. CoRR, abs/2005. 12872, 2020. URL 12872.
Feng, C., Zhong, Y., Gao, Y., Scott, M. R., and Huang, W. Tood: Task-aligned one-stage object detection. In ICCV, pp. 3490-3499 IEEE, 2021. ISBN 978-1-6654- 2812-5. URL http: wrr.i....iiiiiir.eee......... de/bb/ cont/icev/iccv22 htw.:021w..ttmm4eeeececmm.c
https://arxiie www.ivvwrrr/aaaa. org wwtp:////g/bb///oooooooo 12872.
Chen, K., Wang, J., Pang. J., Cao, Y., Xiong. Y., Li, X., Sun, S., Feng. W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C. C., and Lin, D. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 1906.00155. 2019.
Feng, C., Zhong, Y., Jic, Z., Chu, X., Ren, H., Wei, X., Xie, W., and Ma, L. Promptder: Towards open- vocabulary detection using uncurated images. In Avidan, S., Brostow, G. J., Cissé, M., Farinella, G. M. and Hassner, T. (eds.), ECCV (9), volume 13669 of Lecture Notes in Computer Science, pp. 701-717. Springer, 2022. ISBN 978-3-031--0077799 URL http:://bbl unw-tuiiier...... de conf/ eccv/eccv2022- hww.hhhmlll22hmc 4Fng2.2222JJJ
Chen, Q., Wang, Y., Yang, T., Zhang, X., Cheng, J., and Sun, J. You only look one-level feature. In CVPR, pp. 13039-13048. Computer Vision Foundation / IEEE, 2021. URL http: / www.tppiiiiirirreee... db/ conf/cvpr cvwppp/cvv22211111 html #ChenWyzc021,
Ge, Z., Liu, S., Wang, F., Li, Z., and Sun, J. Yolox: Exceed- ing yolo series in 2021. arXiv preprint arXiv:2107.08433 2021.
Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X. and Shan, Y. Yolo-world: Real-time open- vocabulary object detection. In CVPR, pp. 16901- 16911. IEEE, 2024. ISBN 979-8-3503-5300-6 URL httpthttptttt /dl:://dllpppp www.unniiiit..... cb/ conf/ cvpr/cvpr2024 html #Chengscows24,
Girshick, R.
Fast r-cnn, 2015.
URL http: cite
//arxiv. hwt.v/orggaab//10000 08083. uxxiv11504.00083Commi 2015.
To appear in ICCV
html
Dhamija, A. R., Günther, M., Ventura, J., and Boult, E.
Gu, X., Lin, T.-Y., Kuo, W., and Cui, Open- vocabulary object detection via vision and language knowledge distillation. In ICLR. OpenReview.net. 2022. URL http:://blpp uni-trier...... de de/db/ con conf/iclr/iclr222 #GLKK222.
The overlooked elephant of object detection: Open set. In WACV, pp. 1010-1019, IEEE, 2020. ISBN 978-1-7281- 6553-0. URL http: wnw-uniiiieeeee de/ db/ conf wttp///w/wwwwva/aaaaa html #Dhami jaGVB20.
Doan, T., Li, X., Behpour, S., He, W., Gou, I.., and Ren, L. Hyp-ow: Exploiting hierarchical structure learning with hyperbolic distance enhances open world object detec- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024.
Gupta, A.,
Narayan, S., Joseph, K. J., Khan, S., Khan, F. S., and Shah, M. Ow-detr: Open-worid detection transformer. CVPR, pp. 92299222-9222 9234. IEEE, 2022. ISBN 6946-3. URL ht //tp::////lbpppp uni-trier de/db) conf/cvpr/evpr2000 html #GuptaNJ0K522. 2o0taa00

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
He, Y., Chen, W., Wang, S., Liu, T., and Wang, M. Recalling unknowns without losing precision: An effective solution to large model-guided open world object detection. IEEE Transactions on Image Processing, pp. 1-1, 2024. doi: 10.1 1099TT1.2024.3555555
0312. cite arxiv: 1405.03 12Comeen:: 1) updated an- notation pipeline description and figures; 2) added new section describing datasets splits; 3) updated author list.
Lin, T.-Y., Dollár, P., Girshick, R. B., He, K., Hariha- ran, B., and Belongie, S. J. Feature pyramid net- works for object detection. In CVPR, pp. 936-944. IEEE Computer Society, 2017a. ISBN 978-1-5386- uni-trier de/dd)
Jin, S., Jiang, X., Huang, J., Lu, L.,, and Lu, S. Lims meet vlms: Boost open vocabulary object detection with fine-graing descriptors. In ICLR. OpenReview.net, 2024. URL http: / htppp.lppuuuo uni www-uuiiiieerro de/db// http://b.....iiiiii.......e.
0457-1. URL htt.:220077.tttttttinnnsaaa1. conf / /ttp://vpp//////..//////
html #0002J01124.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection, 2017b. URL http:// /ttp:///axvvvvi.o htrgg/abs1708.. 02002. cite 17X.::770.002002
Joseph. K. J., Khan, S. H., Khan, F. S., and Bal- asubramanian, V. N. Towards open world ob- ject detection. In CVPR, pp. 5830-5840. Com-
puter Vision Foundation / IEEE, 2021. URL. http::/ http:///bbp..nitttiiie unw- de htteedbbcontt conf/ cvpr/cvpr2021 htw.hhmmm.oo.. #Joeeppkk0.0
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A. C. Ssd: Single shot multibox detector. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision ECCV 2016, pp. 21-37, Cham, 2016. Springer International Publishing. ISBN 97833-319-46448-0
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P. and Girshick, R. Segment anything, 2023. URL /atpps///rxiv... 2304. 02643.
abs/
Ma, S., Wang, Y., Wei, Y., Fan, J., Li, T. H., Liu, H., and Lv, F. Cat: Localization and identification cascade detection transformer for open-world object detection. In CVPR, pp. 19681-19690. IEEE, 2023a. ISBN 979-8-3503- 0129-8. URL htthtt:::///b//t wnw-trier.e...... ddbb// conf / http://ppp//////eee//// ht
Li L. H., Zhang, P. Zhang, H., Yang, J., Li, C.,
Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., Chang, K.-W., and Gao, J. Grounded language-image pre-training. In CVPR, pp. 10955- 10965. IEEE, 2022. ISBN 978-1-6654-6946-3. URL hwtptttp////t http:/dddl....ntiieeee www-tuiir.deeeeeeeeooo..e..eeee www.dd///bb/ootcet cvpr/cvpr2022, wtt.://1222222222CCCCC
Ma, S., Wang, Y., Wei, Y., Fan, J., Zhang, E., Sun, X., and Chen, P. Skdf: A simple knowledge distillation frame- work for distilling open-voca vocabulary knowledge to open- world object detector, 2024. URL https://aatt //arxiv.. org/ abs / 231 2 08653.
X., Wang, W., Hu, X., Li, J., Tang. J., and Yang. J. Gener- alized focal loss v2: Learning reliable localization quality estimation for dense object detection. arXiv preprint arXiv:2011. 2020a.
Ma, Y., Li, H., Zhang. Z., Guo, J., Zhang, S., Gong, R., and Liu, X. Annealing- based label-transfer learning for open world object detection. In CVPR, 2023b.
Li, X., Wang, W., Wu, I.,, Chen, S., Hu, X., Li, J., Tang, J. and Yang, J. Generalized focal loss: Learning qual- ified and distributed bounding boxes for dense object detection. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F., and Lin, H.-T. (eds.), NeurIPS, 2020b. URL htttp://dble. wnw cemniitrier.ie c nips/neurips2022 http://eurrrp22200..mmmm00000000
Pu, Y., Liang, W., Hao, Y., Yuan, Y., Yang, Y., Zhang, C., Hu, H., and Huang, G. Rank-detr for high quality object detection. In NeurlPS, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.. Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. 000000000000000...... URL https: /tatts://arkae //tp///aaivvvv//////ea .rg/ abs 22103.103.000. 00020.
Li, Z., Xiang, Z.. West, J., and Khoshelham, K. From open vocabulary to open world: Teach- ing vision language models to detect novel ob- jects. CORR, abs/2411.18207,, 2024. URL
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: To- wards real-time object detection with region proposal networks, 2015. URL http: /tte:///axxivv org/ /ab// 1506.01497 01497. cite arxiv:1506.014997oomm tended tech report.
pow. corr/corr2411 htt.htmlll....... 4abs
Ex-
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. Microsoft coco: Common objects in con- text, 2014. URL http: arxiv. www.ooggg/bbbba...ee..
Roh, B., Shin, J., Shin, W., and Kim, S. Sparse detr: Effi- cient end-to-end object detection with learnable sparsity. In ICLR, 2022.
10

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Saito, K., Hu, P., Darrell, T., and Saenko, K. Learn- ing to detect every thing in an open world. In Avidan, S., Brostow, G. J., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), ECCV (24), volume 13684 of Lecture Notes in Computer Science, pp- 268-284. Springer, 2022. ISBN 978-3-031-200533 URL http://dblp..
Wang, Y., Yue, Z., Hua, X.-S,, and Zhang, H. Ran- dom boxes open-world object detectors. In ICCV, pp. 6210-6220. IEEE, 2023c. ISBN 979-8-3503- 0718-4. URL http: /tbp: uni-trier.de /bb con http://ccccvcccvvveeee
html htm #WangY0223.
httpd/////bbbbbcee cb/conf/ eccv/eccv2022-1 html #SattopDs2.
Wang. Z., Li, Y., Chen, X., Lim, S.-N., Torralba, A., Zhao, H., and Wang. S. Detecting everything in the open world: Towards universal object detection. In CVPR, pp. 11433-1 114431 IEEE, 2023d. ISBN 979-8-3503- 0129-8. URL httttp://dboee www.:uip..iitriirr.com conf http://pp///////eee//e/ wt0.5000000000.00000000
Sun, P., Jiang. Y., Xie, E., Shao, W., Yuan, Z., Wang, C., and Luo, P. What makes for end-to-end object detection? In Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9944.. PMLR, 2021.
Wu, S., Zhang, W., Jin, S., Liu, W., and Loy, C. C. Aligning bag of regions for open-vocabulary object detection. In CVPR, pp. 15254-15264. IEEE, 2023a. ISBN 979-8-3503-0129-8 URL http://dblp. cvpt/cvpz2023 html #wu200123.
Sun, Z., Li, and Mu, Y. Exploring orthogonality in open world object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17302-17312, 2024.
Tan, M., Pang. R., and Le, Q. V. Efficientdet: Scalable and efficient object detection. In CVPR, pp. 10778-10787 Computer Vision Foundation IEEE, 2020. ISBN 978- 1-7281-7166-5. URL http: /ttpp/dbpppppp/ trier.. de/ ble/bb/coff wttp///nnncpp///ppp//tceaae////e http://ppp222.....mmmaaaaaaaeeee
Wu, W. Liu, H., Li, L., Long, Y., Wang. X., Wang, Z., Li, J., and Chang. Application of local fully con- volutional neural network combined with yolo v5 algo- rithm in small target detection of remote sensing image. PLOS ONE, 16( 1-1 10 2021. doi: 10.137 //jurnall pone. none.0259283. URL ht tttps: doi. org/ /ww/g//.... 1371/ journal.pone pone. 0259283.
Tian, Z., Shen, C., Chen, H., and He, T. Feos: Fully convolutional one-stage object detection. In ICCV, pp. 9626-9635. IEEE, 2019. ISBN 978-1-7281- 4803-8. URL http: /btp:p/abppppppp/ uni-trier .wweereeeeeeeeeeeeeeeeeeeeeeeeee db/ conf/iccv/iccv200 html #TiansCH19.
Wu, X., Zhu, F. Zhao, R. and Li, H. Cora: Adapting clip for open-vocabulary detection with re- gion prompting and anchor pre-matching. In CVPR, pp. 7031-7040. IEEE, 2023b. ISBN 979-8-3503- 0129-8. URL http: //bbpp uni.tiiiiiiiiirr....mm. dedb// conf CBte//cpr///// evpp202332222333322 html #WZZ2123.
Wang, C.-Y., Bochkovskiy, A., and Liao, H.-Y. M. Yolov7: Trainable bag-of-freebies sets new state-of- the-art for real-time object detectors. In CVPR, pp. 7464-7475. IEEE, 2023a. ISBN 979-8-3503- 0129-8. URL http: /ttpp./bppppppppp uni uni-trier. conf conf/cxpr/csp
Wu, Z., Lu, Y., Chen, X., Wu, Z., Kang, L., and Yu, J. Uc-owod: Unknown-classifee open world object detection. In Avidan, S., Brostow, G. J., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), ECCV (10), volume 13670 of Lecture Notes in Computer Science, pp. 193-210. Springer, 2022. ISBN 978 1-7-3--031-20080- URL http:///bil http://bbbpp.iiirrereeeee uni-trier de dib conf/ eccv eccv/eccv2022-11 html #WULCCKY22.
html #Wang81.23.
Wang, J., Song, I., Li, 7.,, Sun, H., Sun, J., and Zheng, N. End-to-end object detection with fully convolutional network. arXiv preprint 2rXiv:201..03544 2020.
Wang, L., Liu, Y., Du, P., Ding. Z. Liao. Y., Qi, Q., Chen, B., and Liu, S. Object-aware distillation pyra- mid for open-vocabulary object detection. In CVPR, pp. 11186-11196. IEEE, 2023b. ISBN 979-8-3503- 0129-8. URL http: wttp::///hw.pp..i..iiiiee uni-trier de/db/ /b// conf/cvpt/cvp:2// http:/2000...ttt10000000000000.0
Xi, X., Huang, Y., Lin, J., and Luo, R. Kten: En- hancing open-world object detection with knowledge transfer and class- neutralization. In IJCAI, pp. 1462-1470. ijcai.org, 2024. ISBN 978-1-956792. 04-1. URL http: uni.oiieriiitiir..... de/ db/ htm. #x1HL124.
Wang, N., Gao, Y., Chen, H., Wang. P. Tian, Z., Shen, C., and Zhang, Y. Nas-fcos: Efficient search for object detection architectures. Int. J. Comput. Vis., 129(12):3299-333) 2021. URL http://bblp..nitttee de/ htt.::/ieerrd/e/////aaasss. i jjcv/ijev129 html hat.
Yang, S., Sun, P., Jiang, Y., Xia, X., Zhang, R., Yuan, Z., Wang, C., Luo, P., and Xu, M. Objects in semantic topology. CoRR, ab./21100066888,, 2021. URL https: //arxiv. //rrvvv.rrr/bb//21100000

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., and Xu, H. Det- clip: Dictionary-enricher visual-concept paralleled pre- training for open-world detection. In Koyejo, S.. Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), NeurIPS, 2022. ISBN 9781713871088, URL http: http://blpp....iiiieeeeeee http://aauuup22222222cocaaaaoae1
Yao, L., Han, J., Liang. X., Xu, D., Zhang, W., Li, Z., and Xu, H. Detclipv2: Scalable opea-vocab ob- ject detection pre-training via word-region alignment. CVPR, pp. 23497-23506. IEEE, 2023. ISBN 979-8-3503- 0129-8. URL http://tt:///lpppp www-iiirr...itirree..ed.d.dddee. conf/cvpr/cvpaaee htw.:22222..mml#aaaaaemmmmm html #Y
Zareian, A., Rosa, K. D., Hu, D. H., and Chang, S.F.. Open-vocabulary object detection using captions. CoRR, abs/2011..0677 2020. URL
http://dblp.. http:/ppwwmmtttiirreddeea//aaaaa hort/corr2011.htmmIIIM htt.hhmmm....... 4atm144ab221111056777
Zhang, Y., Guo, Z., Wu, J., Tian, Y., Tang. H., and Guo, X. Real-time vehicle detection based improved yolo v5. Sustainability, 14(19), 2022. ISSN 2071-1050. doi: 10.3399/su141912274. URL https: www. www.mdp.. htm/22071-1050144/110
Zhao, X., Liu, X., Shen, Y., Qiao, Y., Ma, Y., and Wang, D. Revisiting open world object detection, 2022. URL htttpa://arxiae org/ abs 221/22221000000000 00471.
Zheng, Z.,, Wang, Liu, W., Li, J., Ye, R., and Ren, D. Distance-iou loss: Faster and better learning for bounding box regression. In The AAAI Conference Artificial Intelligence (AAAI), pp. 12993-13000, 2020.
Zohar, O., Lozano, A., Goel, S., Yeung, S., and Wang, K.- C. Open world object detection in the era of foundation models, 2023a. URL /tttpp://ariiii org/ abs/ 2312.05745
Zohar, O., Wang, K.-C., and Yeung, S. Prob: Probabilistic objectness for open world object detection. In CVPR, pp. 11444-11453. IEEE, 2023b. ISBN 979-8-3503- 0129-8. URL http::/ /blpp uni wea--trrerddddddd conf/copprccpp:203 html #2oharWY23.
12

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
unknown classes of interest. In subtask T2, another subset D2 (X2,Y2 Y2) is obtained. Unlike before, the current set of classes Y2 excludes both the unknown classes and the annotations belonging to Y1. T2 is referred to as incremental learning, which requires detector to fine-tune its existing knowledge (trained in Ti) to adapt to new classes.
Table Detailed Attribute Descriptions. Type indicates the at- tribute type, identifying the object features of interest. Description provides the text description corresponding to the type code. For simplicity, we use only nine attributes.
Type
Description
Shape
The object has distinctive shape
Color
The object has a noticeable color
Texture
The surface of the object has purticular texture
C. Dataset Details
Size
The object has specifie size
Table 5 presents the detailed partitioning of the OWOD benchmark. MOWODB (Table 5(a)) combines the VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) datasets. In MOWODB, all VOC categories are regarded as Task 1, while the remaining categories in COCO are re- garded as unknown classes. During testing, the validation sets of VOC and COCO are mixed to form the test set of MOWODB. When evaluating, all the seen categories will be treated as known, and then these known categories will be divided into the categories introduced by the previous tasks and the categories introduced by the current task. In the subsequent tasks, the division is carried out according to the superclasses of COCO. For example, both hair drier and toothbrush belong to the same superclass indoor, and they will be introduced as newly annotated instances in Task 4. SOWODB (Table 5(b)) only uses the COCO dataset and has smaller amount of data compared to MOWODB. Without the inclusion of VOC categories, SOWODB exhibits more isolated semantic divisions. In MOWODB, detectors are ex- posed to other supercategories earlier due to VOC categories being subsets of other supercategories. It is noteworthy that the original ORE implementation included duplicate images in the test set, resulting in higher test counts than defined in MOWODB. For instance, in Task 1, there are 23,320 unknown instances. In the EO-OWOD test, duplicate in- stances were removed, and ovow followed this procedure. Thus, in OWOD evaluations, we include a comparison of results after deduplication.
Context
The object found certain environments
Features
The object exhibits characteristic features
Appcarance
The object has unique appearance
Behavior
The object is used in various ways
Environment
The object is associated with certain type of environment
Material
The object is made from specific material
A. Detailed and Coarse Attributes
Table 4 presents the coarse textual attributes we used. FOMO (Zohar et al., 2023a) utilizes large language model (LLM), GPT 3.5, to generate attributes for known categories. These attributes encompass nine types, corresponding to the nine types Table 4. For each type, the LLM lists the relevant attribute options. For example, for the attribute type color, the LLM lists options such as blue, which are then combined into descriptions like: object which color (is/has/etc) blue. FOMO expects potential objects to exhibit similar attributes, meaning its performance is constrained by the LLM descriptions. In contrast, we use coarse attributes. Unlike FOMO, which seeks fine-graneed attributes, we fo- cus on whether a particular attribute of potential objects is prominent. Therefore, we use terms describing degree to represent attribute types. For instance, for the attribute type Color, we provide the description: The object has a notice- able color. By using these representations, we no longer rely on the LLM to generate attributes, thereby avoiding performance limitations on the detector.
D. Metric
B. Problem Definition
OWOD, as a special task of OD, requires the detector to adapt to the open-world environment. For evaluating perfor- mance on unknown objects, OD's conventional evaluation metrics can be employed. However, due to the lack of in- formation on unknown objects, directly applying metries like Average Precision (AP) results in similar performance across all methods. For instance, when evaluated with AP, methods such as ORE (Joseph et al., 2021), OW-DETR (Gupta et al., 2022), and PROB (Zohar et al., 2023b) exhibit comparable performance. Therefore, before employing the foundation models, researchers use compromise metrics to assess specific capabilities of the model, rather than directly using AP to evaluate overall performance. U-Recall repre- sents the recall rate of unknown classes; an unknown class
Open World Object Detection (OWOD) consists of two main components: detecting unknown objects and in- cremental learning. To accurately simulate this process, the detection task is divided into a series of subtasks T = (Ti, ..., J. Given a dataset D = (X, (X,Y), where Y represents the labels corresponding to the im- age set X, the label categories are partitioned into subsets K (K1,K2 .... }. In Ti, a subset D1 {X1, Yı) is extracted from the dataset D. Annotations belonging to the category subset Ki are selected from Yı to simulate a partially annotated scenario, Y,- = {vl y E Y1 A y E K,)) }. During testing, the detector is required to detect all classes in XC, with unannotated classes in subset K considered as the
13

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Table Detailed Description of Dataset Partitioning. Train and test images denote the number of images in the training and test sets, respectively, while the corresponding instances represent the number of test instances. MOWODB utilizes combination of VOC and COCO datasets, whereas SOWODB exclusively uses COCO dataset.
(a). Semantic category division of MOWODB
(b). Semantic category division of SOWODB.
MOWODB
Task
Task
Task 3
Task 4
SOWODB
Tassk
Task
Task
TTask 4
VOC Outdoor. Accessories Sports, Electronic. Indoor,
Animals. Persion Appliances. Outdoor Sports, Kitchen, Indoor,
Split
Split
Classes
Appliances, Trick
Food
Kitchen, Furniture
Vehicles
Accessories, Furniture
Food
Electronic
train images
16551
45520
39402
40260
train images
89490
55870
39402
38903
test images
4952
1914
1642
1738
test images
3793
2351
1642
1691
train instances
47223
113741
114452
138996
train instances
421243
163512
114452
160794
test instances
14976
4966
4826
6039
test instances
17786
7159
4826
7010
Table 6. Hyperparameter Ablation Analysis. The three sub-tables below show the impact of hyperparameterss model performance. The selected hyperparameters are indicated in bold. analyses are conducted on MOWODB, reduce the number of parameters, we set the threshold a (Equation (4)) to the inference threshold of 0.01 used in YOLO-World.
(a). The impact of 5 (Equation (5)).
(b). The impact of n (Equation (6)).
(c). The impact of b (Equation (10)).
Setting
U-Recall
U-AP
WI
A-OSE
Setting
U-Recall
U-AP
WI
A-OSE
Setting
U-Recall
U-AP
WI
A-OSE
(1)
(t)
(1)
(1)
n
(f)
(1)
(t)
(t)
(t)
(f)
(1)
(1)
0
58.6
7.5
0,0221
1376
1
58.4
10.7
0.0180
995
0.1
58.3
11.6
0.0235
3094
58.7
9.5
0.0193
1212
2
58.1
10.4
0.0183
1059
0.2
58.8
12.0
0.0214
2007
58.6
10.6
0.0182
1167
3
58.4
11.0
0.0182
969
0.3
58.9
12.4
0.0200
1533
58.8
10.2
0.0184
1135
4
58.4
10.4
0.0177
1015
0.4
59.0
12.5
0.0193
1272
20
58.8
9.9
0.0173
1111
5
58.4
10.7
0.0173
1025
0.5
58.9
12.7
0.0187
1108
25
58.2
9.8
0.0176
1132
6
58.7
10.8
0.0178
985
0.6
58.8
13.0
0.0185
996
30
58.3
9.5
0.0175
1093
7
58.7
10.6
0.0175
1022
0.7
58.8
13.1
0.0185
915
35
58.4
9.6
0.0177
1100
8
58.1
10.4
0.0176
1009
0.8
58.6
13.2
0.0180
831
40
58.7
9.6
0.0173
1087
9
58.5
10.5
0.0177
988
0.9
58.6
13.4
0.0177
770
45
58.5
9.5
0,0180
1081
10
58.6
9.2
0.0221
1266
1.0
58.5
13.2
0.0178
728
is considered recalled the IOU between the prediction and the unknown class ground truth exceeds 0.5. WI (Dhamija et al., 2020) represents the relative open-set error. is com- posed of the ratio of the loss of detection accuracy of the known classes after the introduction of the unknown classes:
curately identifying and recalling unknown objects, leading to a need for a pragmatic approach in performance assess- ment. However, with the introduction foundation models, which incorporate more advanced algorithms and learning capabilities, the detectors have now approached the maxi- mum achievable performance in terms of unknown object recall. This advancement necessitates more rigorous eval- uation process to differentiate between subtle improvements in detection capabilities. Consequently, we include the Un- known Average Precision (U-AP) metric during evaluation as a more stringent criterion.
Pk
WI
-1
(11)
PRUU
where Px and PkuU represent the accuracy of known classes and the overall accuracy in open- speld scenarios, re- spectively. Consistent with previous methods (Joseph et al., 2021; Ma et al., 2023b), we calculate this value using an IoU threshold of 0.8. Unlike WI, A-OSE calculates the ab- solute value directly. A-OSE directly measures the number of overlaps between known predictions and unknown class labels. Specifically, A-OSE is incremented by when the loU between known prediction and an unknown class exceeds 0.5.
E. Implementation Details
Training of positive and negative samples in VAP. For positive and negative samples, the loss is calculated similarly to the pseudo-labels, using cross-entropy:
Epositive = mean mean (T(E)t.) CB (B(T())..))),,
(12)
These metrics act as a trade-ofi off solution scenarios where the performance of carly OWOD detectors does not meet expectations. Initially, these detectors faced challenges in ac-
where E+, represents the visual embedding corresponding
to the positive samples generated by the matching algorithm,
14

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Table Performance Comparison on the OWOD Standard Evaluation Benchmark. The top section shows MOWODB, while the bottom section shows SOWODB. FOMO-ZS+ + denotes the use of generic description, object, as cue for unknown objects, combined with OOD scores to estimate the likelihood of unknown classes. FOMO-ZS+IN uses ImageNet categories, removing unknown class names and using the remaining class names as cues for unknown objects. FOMO refers to the use of all attributes in FOMO.
Task IDs (-)
Task
Task 2
Task 3
Task 4
U-Recall mAP (f) U-Recall
mAP(1)
U-Recall
mAP(t)
mAP(f))
Method
Current
Previously
Current
Previously
Current
Previously Known
Current
(1)
(1)
Both
(1)
Both
Both
Known
Known
Known
Known
Known
Known
FOMO-ZS+
24.0
68.1
25.4
68.5
41.7
55.1
28.6
55.2
47.2
47.5
26.1
42.1
FOMO-ZS+IN
52.4
67.8
50.1
67.7
41.3
54.5
49.9
54.6
31.2
46.8
46.8
25.0
41.4
FOMO
54.6
67.4
51.9
59.0
37.4
48.2
52.1
42.8
24.4
36.7
35.0
21.4
35.0
Ours: OW-VAP
58.8
68.8
56.3
68.8
42.5
55.6
55.1
55.7
29.9
47.1
47.3
25.8
42.0
FOMO-ZS+
31.4
78.5
38.6
78.8
58.2
68.0
44.0
68.0
56.5
64.2
64.3
58.0
62.8
FOMO-ZS+IN
77.4
77.6
78.4
77.6
58.8
67.7
78.6
67.7
56.2
63.9
63.8
57.3
62.2
FOMO
78.4
69.9
80.5
60.0
50.1
54.8
80.8
50.7
45.4
49.0
50.3
45.9
49.2
Ours: OW-VAP
82.9
79.6
85.6
79.8
62.3
70.6
85.7
70.7
57.6
66.3
66.4
59.1
64.6
Table Detection Accuracy of Unknown Classes and Confusion with Known Classes on the MOWODB Benchmark. All implementations the FOMO series are consistent with VAP by assigning samples with unknown probabilities greater than known probabilities to the unknown class. Our OW-VAP demonstrates a significant lead in both detection precision and recall.
Task IDs (+)
Task 1
Task 2
Task 3
U-Recall
U-AP
WI
A-OSE ())
U-Recall
U-AP
WI
A-OSE
U-Recall
U-AP
WI
A-OSE
Method
(1)
(t)
(L)
(t)
(t)
())
(4)
(t)
(1)
(4)
(1)
FOMO-ZS+
24.0
6.8
0.0295
26353
25.4
5.9
0.0219
22172
28.6
5.5
0.0174
19528
FOMO-ZS+IN
52.4
3.5
0.0221
9521
50.1
2.4
0.0152
8077
49.9
3.1
0.0107
6138
FOMO
54.6
4.9
0.0564
1937
51.9
2.6
0.0103
332
52.1
2.1
0.0065
196
Ours: OW-VAP
58.8
13.0
0.0185
996
56.3
9.3
0.0082
531
55.1
7.8
0.0059
443
i... (S > 0). Similarly, the loss for negative samples is:
Inference. During inference, we observed that the detector often assigns different class IDs to the same sample, which increases the difficulty of distinguishing between positive and negative samples. Therefore, for the same sample, if the known confidence is lower than the unknown confidence. set the known confidence to O. In the SOWODB, such situation did not occur. As a result, we only modify the confidence in MOWODB, while disregarding SOWODB
Enegative = mean mean (T(E...)) CB(T(E (BTT
(13)
where F represents the visual embedding corresponding to the negative samples, i.c. (S = -). Therefore, the fi-
nal loss for training VAP L Epositive + Cpseado Leggative-
Incremental learning. Since we mine latent objects in Task 1, VAP exhibits strong generalization capability. Fine-tuning it during subsequent incremental learning would harm the detector's existing knowledge, previously seen classes are labeled as background. Therefore, we only fine-tune VAP during the knowledge replay phase. The replay process is conducted exclusively on the MOWODB, while in the SOWODB, we directly use the VAP from Task to predict all unknown classes, bypassing the fine-tuning process.
Setting, Our approach is based the OVD detector, YOLO- World (Cheng et al., 2024). To ensure fair comparison with ovow, we use the XL version. The object attributes are described using nine categories, generating only nine distinet sentences. For the text encoder, we use the CLIP model, specifically the version ing training, we freeze the text and visual encoders, training only the VAP and known class embeddings. All experi- ments are conducted using 8 V100 GPUs (total 128 GB). We implement all experiments using MMDetection (Chen et al., 2019). For the experimental parameters, we follow the official settings of YOLO-World, altering only the training epochs: 10 epochs for MOWODB and for SOWODB.
Dur-
Code. In the near future, once the code passes the com- pany's review, we will release all the trained code, weights, and models (including other SOTA) along with visualization
15

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
FOMD-ZS
FOMO-ZS IN
ROMO
arknown
rson 0.36
0.03
person 0.67
unknown 0.93
Sningtobe
unkkoww
aningtable
6:46
person 0.52
porson
aninglase 0.62
uniRow 3
Shar OU72
unknown 0.16
unknown 0.49
aningtabe
unimoww 0.20
Loknown 0:58
Figure Qualitative Comparison with FOMO Series MOWODB. before, for a fair evaluation, we rank predictions by unknown confidence and display only the top three. OW-VAP demonstrates superiority in both detection precision and recall.
files at the MOWODB dataset level.
the detector. Compared longer queues, using shorter queue yields better performance (10.7 U-AP with 1). Therefore, considering the performance trends of the param- eters as n varies, we set n to 3 for subsequent experiments.
F. Ablation Experiment of Hyperparameters
5. In Equation (5), we introduce the hyperparameter 5. This parameter controls the number of samples selected from the background region. Intuitively, the more samples selected, the higher the recall rate, since more potential samples are marked with pseudo-labels. However, this also increases the risk of background samples being incorrectly labeled as pseudo-labels. As shown in Table 6(a), increasing the num- ber of recalls enhances the unknown recall rate, reaching a maximum of 58.8 U-Recall when is set to 20. However, further increasing 5 does not improve recall, indicating that background samples in the pseudo-labels negatively affect the detector's ability to identify unknown objects. There- fore, set 8 to 10 for subsequent experiments based the maximum U-AP value.
b. In Equation (10), we integrate the raw attributes as a rough estimate of the current visual region. Subsequently, a visual attribute extractor is used for more refined predictions. To balance the contributions of these two components, we introduce the balancing parameter b. The performance of the detector as a function of the parameter b is shown in Table 6(c). Overall, when is set to a small value, the detector exhibits poor detection precision, achieving only 11.6 U-AP b= 0.1. The Visual Attribute Predictor (VAP) further enhances the detection precision. When set to effectively excluding the raw attributes, the detector achieves 13.2 U-AP and performs well on both WI and A- OSE. However, this setting leads to a significant decrease in known performance. Therefore, we strike a balance between the contributions of the two components by setting the value of b to 0.6.
n. In Equation (6), we introduce the parameter n. This parameter is utilized to control the number of parameters in the queue. When establishing Weibull distribution, esti- mating the parameters within small batches leads to training instability. To mitigate this, estimate the parameters from historical batches to stabilize the training. Utilizing longer queue, however, results in less focus on the current batch. There is a trade-off involved in the length of the queue. As shown in Table 6, increasing the length of the queue decreases both the recall and detection precision of
G. More Comparisons
G.1. Implementation Details of FOMO
As previously mentioned, FOMO relies on attributes gener- ated by large language model (LLM) to predict unknown objects. We use GPT-3.5 (in its official configuration) to generate attributes for the OWOD benchmark. The similar-
16

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
Algorithm 1 Visual Attribute Parser Pseudocode
Algorithm 2 Probabilistic Soft Label Assignment (PSLA)
Input:
Att, Name, I, Yi, Frier O, (Input data: at- tributes, class names, image, ground truth, positive visual embeddings, thresholds.}
Input: E Eatt, Ot 6,,
Output: Soft labels W (a)), Pseudo-label loss Epecudo X +
Calculate WeightedAveraaee( (E
Output: Seap {Output: pseudo-label indicator.) Eau < (Att(Att) (Encode attributes to embedding.)
{Calculate the weighted average of positive sample dis- tribution parameters]
E. < Enc, (Name) (Encode class names to embed-
X- +
E
ding.)
{Estimate the negative sample distribution
Euis < (T) (Encode image to visual embedding.}
Wb+ (x) - Wb(2, A+, k. 0) {Calculate positive sample
S f arg min, (EE, Evis, Y1) {Compute matching
distribution using Weibull distribution}
score.)
Wb- < Wb > A: 0) {Calculate negative sample
condl < max (Sim(E Eatt))
distribution using Weibull distribution)
2 mean (nean(max(Sim)) (Etts: Eatt)))
W f Wh (=)++Wbb (z) {Compute soft labels based
(x)
(Condition 1: Compare similarity with positive embed- dings.}
on positive and negative distributions)
Seap f Generaacessuuddoabbess W (z), Eit.. Eatt, a, 6)
cond2 < max(Sim(EEis: Eatt) 2 a
{Generate pseudo-labels using PSLA method)
(Condition 2: Compare similarity with threshold a.} cond3 <- Top; (Sim( (Frisy Fact))
Lpeeudo + meanjr(e:...)) W ))T(E)s)) CE(
(Calculate pseudo-label loss with soft labels and cross- entropy}
(Condition 3: Check top ô similarities.)
Swap W (condl cond2 A cond3)
T(E < a(> Am Sim(M(E Eatt) + ()
{Compute final pseudo-label indicator.)
{Compute VAP (pseudo-label visual embeddings) using sigmoid and MLP]
Output Epsendo {Output the final pseudo-label loss}
ity of these attributes is then used to set the in-distribution probability, which is combined with the OOD probability to identify unknown objects. However, as noted earlier, in incremental learning, it is challenging to balance the contri- bution of newly added attributes for new categories. In the official implementation, annotations from previously seen categories are also used in incremental learning, which vio- lates the OWOD task setup. To maintain the integrity of the OWOD setup, we utilize all attributes to predict unknown objects without attribute selection. Theoretically, this modi- fied approach can achieve higher performance metrics.
U-AP (Table 8). Regarding U-AP, our method also demon- strates an advantage. Specifically, in Task 1, OW-VAP sur- passes FOMO-ZS+, FOMO-ZS+IN, and FOMO by +6.2, +9.5, and +8.1 U-AP, respectively. In Tasks 2 and 3, our method maintains its lead, achieving advantages of 4.4 and 5.7 U-AP over FOMO, respectively. Our approach also ex- cels in WI and A-OSE metrics, consistently outperforming all FOMO series across the three tasks. However, in terms of A-OSE, we slightly trail behind FOMO. The A-OSE metric of our method could be further improved by label- ing more low-confidence known predictions as unknown objects. However, to preserve OW-VAP's performance on known classes, we only convert predictions where unknown confidence exceeds known confidence. Despite this, we only slightly lag behind FOMO in A-OSE, with confusion count of 201.
G.2. Comparison
U-Recall (Table 7). OW-VAP VAP demonstrates a significant per- formance advantage. On the MOWODB benchmark, OW- VAP surpasses FOMO-ZS+ by 34.8 U-Recall and FOMO- ZS+IN by 6.4 U-Recall. Even when compared to FOMO, which uses all attributes, our method leads by 4.2 U-Recall. This advantage is maintained in subsequent incremental learning phases. On the SOWODB benchmark, our OW- VAP also achieves consistent superiority. In Task 1, we exceed FOMO-ZS+ by 51.5 U-Recall and FOMO-ZS+IN by 15.5 U-Recall. Similarly, against FOMO, as in the MOWODB benchmark, we achieve consistent leadership, with an increase of 4.5 U-Recall in Task 1, 5.1 U-Recall in Task 2, and 4.9 U-Recall in Task 3. Additionally, in the performance for known classes, our method also exhibits significant advantages. In Task 4, we surpass FOMO by 7 mAP and 15.4 mAP, respectively.
Qualitative comparison (Figure 6). Figure 6 presents a vi- sualization analysis comparing our method with the FOMO series. Compared to FOMO-ZS, our approach demonstrates stronger unknown recall capability. OW-VA OW-VAP successfully recalls both the kettle and the mug on the table, whereas FOMO only recalls the kettle. In comparison with FOMO- ZS+IN, OW-VAP exhibits higher precision. FOMO-7S+IN focuses on internal details of people, such as parts of cloth- ing, which are typically not the objects of interest. Con- versely, OW-VAP focuses on objects across the entire im- age, detecting items like the wooden bridge helmet. When
17

---

OW-VAP: Visual Attribute Parsing for Open World Object Detection
compared to FOMO, our method also shows advantages. FOMO incorrectly classifies chair unknown object and overlooks items the table. In contrast, our OW-VAP identifies the kettle and bread, and correctly predicts the chair.
H. Pseudo Code
The Visual Attribute Parser (VAP, Algorithm 1). First, the input attributes (Att) and class names (Name) encoded using text encoder, producing corresponding attribute em- beddings (Eers) and class embeddings (Ec). Meanwhile, the visual input image (I) is passed through visual encoder to obtain the visual embedding (Evis). The algorithm then computes matching score by minimizing the distance be- tween the class embedding (E,) and the visual embedding (Euse). To ensure high-quality pseudo-labeling, three condi- tions are applied: (1) the similarity between the visual and attribute embeddings should exceed the mean similarity of the visual and positive attribute embeddings, (2) the simi- larity must exceed threshold parameter (a), and (3) the top similarities are considered. If all conditions are sat- isfied, a pseudo-label indicator (Seap) is assigned, guiding the learning process.
The Probabilistic Soft Label Assignment (PSLA) algorithm addresses optimization conflicts caused by background in- terference in pseudo-label generation (Algorithm 2). To pseudo-
resolve this, PSLA estimates the probability label being a true positive by modeling the loss distributions of positive and negative samples. The algorithm assumes that the loss for background samples is higher and more consistent, while the loss for positive samples is sparser and exhibits greater fluctuations. Using Weibull distribution (Wh) for both positive and negative samples, PSLA applies maximum likelihood estimation to determine distribution parameters, with moving weighted average applied to re- duce fluctuations in the positive sample distribution. The final soft labels are derived by normalizing the positive and negative sample distributions. These soft labels are then used to weight the pseudo-label loss, which is calculated using the cross-entropy between the predicted labels and the target labels. The PSLA framework enhances the robust- ness of pseudo-labeling by addressing background noise and optimizing the learning process.
18


Reviews:
[{"review_id": "iEyFAsU210", "reviewer": "Reviewer_oH8t", "summary": "This paper tackles open world object detection in a fresh way, moving beyond heavy reliance on language models. Their key idea is a Visual Attribute Parser, learning directly from images, combined with smart pseudo-labeling using probabilistic soft labels. Experiments show their approach really pushes the state-of-the-art, hitting impressive recall and precision for unknown objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful and thoughtful review of our manuscript. Your meticulous approach to the review process contributes significantly to the advancement of the research community. Below, we provide our responses to the concerns raised in your comments:\n\n$\\textbf{Q1}$: Regarding the question on how the model's performance could reach or even surpass the theoretical upper bound in zero-shot generalization\n\nIn object detection tasks, performance evaluation must consider both recall capability (i.e., the ratio of correctly detected targets to total targets) and precision (i.e., the proportion of correct detections among all positive predictions). The mean Average Precision (mAP) metric has become the standard for balancing these competing objectives, as comprehensively explained in [1]. \n\nHowever, conventional mAP formulations predominantly focus on closed-set detection tasks. In the context of open-world object detection (OWOD), certain object categories may lack annotations in the training data, $\\textcolor{red}{\\text{including the class names}}$. As a result, object detectors typically demonstrate degraded performance in such settings. Under such conditions, direct application of AP metrics becomes inadequate to reflect the performance, as demonstrated by leading OWOD works including ORE, OW-DETR, and PROB. For example, KTCN and PROB achieve similar U-AP values, but in terms of U-Recall, KTCN achieves more than twice the performance of PROB.\n\nThus, benchmarks such as MOWODB and SOWODB prioritize U-Recall as a metric to evaluate unknown object detection performance. $\\textcolor{red}{\\text{We surpass the zero-shot performance of the detector only in U-Recall, but not in U-AP}}$. Notably, exceeding zero-shot testing performance in U-AP remains challenging without utilizing additional information about unknown classes. The relative comparison of all methods in U-AP and U-Recall is shown in $\\textcolor{red}{\\text{Figure 1}}$.\n\n$\\textbf{Q2}$: Comparison Between Vague Descriptions and Detailed Descriptions\n\nAs mentioned in the paper, detailed descriptions rely on large language models (LLMs) to describe certain aspects of known classes, with the expectation that unknown classes share similarities in these aspects (as demonstrated by FOMO). These similarities are then used to recall unknown objects. However, directly using detailed descriptions in our method is impractical because their quantity exceeds 8,000. In contrast, our method achieves detection of unknown classes using only coarse descriptions composed of 10 simple sentences. Furthermore, we compared our method with all FOMO variants (using detailed attributes) in $\\textcolor{red}{\\text{Tables 7 and 8}}$. Our method still demonstrated significant advantages, achieving improvements of +4 in U-Recall and +8.1 in U-AP. For more details, please refer to $\\textcolor{red}{\\text{Appendix G}}$.\n\n$\\textbf{Q3}$: Choice of the Base Probability Model\n\nIn line with OWOD conventions, we adopt the Weibull distribution for probability modeling, as introduced by ORE for energy-based methods. While exploring alternative distributions (e.g., Beta) could potentially enhance performance, such designs often increase methodological complexity, hindering replicability and adoption. By adhering to the established Weibull distribution, $\\textcolor{red}{\\text{we ensure both simplicity and effectiveness}}$. Therefore, we follow the default setting in the OWOD field by using the Weibull distribution for modeling, demonstrating the effectiveness of PLSA. \n\n$\\textbf{Q4}$: Explanation of Figure 3 Annotations\n\nIn Figure 3, we illustrate the training process of VAP. In the figure, the blue circles represent samples that have already been labeled in the training set. These samples are treated as positive samples during training, and we train VAP by pulling them closer to the nearest coarse description (represented by the yellow circles in the figure). The triangles, on the other hand, represent $\\textcolor{red}{\\text{potential objects}}$. These are further distinguished using the three selection rules described in Section 3.2. We hope this clarifies the visual annotations in Figure 3. \n\nWe sincerely thank the reviewer for the valuable feedback. We hope our responses address the raised concerns, and we remain available to clarify any remaining questions or suggestions.\n\n[1] Measuring Object Detection Models - mAP - What is Mean Average Precision?", "claims_and_evidence": "Yes, the claims made in this submission are, for the most part, convincingly supported by the evidence provided. The authors present comprehensive benchmarking against a strong set of baselines on standard OWOD datasets (MOWODB and SOWODB). Ablation studies effectively isolate the contributions of both the Visual Attribute Parser and the Probabilistic Soft Label Assignment, further bolstering the claims regarding the effectiveness of these proposed components. Qualitative visualizations also provide intuitive support for the method's improved detection of unknown objects.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are well-suited and sensible for the problem of Open World Object Detection. The introduction of the Visual Attribute Parser and Probabilistic Soft Label Assignment are conceptually sound and directly address the challenges of detecting unknown objects. The evaluation on MOWODB and SOWODB benchmarks, using U-Recall, U-AP, WI, and A-OSE metrics, aligns perfectly with established practices in the OWOD field and provides a comprehensive assessment of the method's capabilities. The comparison to relevant state-of-the-art methods and thorough ablation studies further strengthen the validity of the evaluation.", "theoretical_claims": "I didn't spot any formal proofs to really \"check\" in this submission. It's more of an empirically-driven approach, focused on the algorithm and results. That being said, the conceptual ideas behind VAP and PSLA seem pretty solid and logically motivated for the problem they're tackling.", "experimental_designs_or_analyses": "The experiments are strong overall, especially the benchmark comparisons and ablations. However, it's not entirely clear why the vague attribute descriptions were chosen over more specific ones. It would strengthen the paper to see some analysis or even a small experiment exploring the impact of using different levels of attribute description granularity.", "relation_to_broader_scientific_literature": "OW-VAP directly tackles limitations of LLM-reliant OWOD methods like KTCN and FOMO. The Visual Attribute Parser builds on visual attribute recognition ideas, but applies them uniquely to OWOD, differing from attribute use in closed-set OD or OVD. Probabilistic Soft Label Assignment smartly adapts pseudo-labeling strategies from works like PROB and ORE, specifically for background noise, going beyond simple thresholds.", "essential_references_not_discussed": "The paper does a reasonable job citing relevant OWOD methods. However, to fully contextualize OW-VAP's contribution, briefly mentioning foundational works in Open Set Recognition (OSR), like the early work of Scheirer et al. on zero-shot learning and open set recognition, would be beneficial.", "other_strengths_and_weaknesses": "Strengths:\n1)The OW-VAP framework, especially the Visual Attribute Parser and Probabilistic Soft Label Assignment, presents a genuinely novel and well-motivated approach to OWOD, moving away from overly LLM-dependent methods.\n2)The experimental results clearly demonstrate substantial performance improvements over existing SOTA methods on standard benchmarks, which is a significant contribution to the field. Approaching the OVD generalization limit is also noteworthy.\n3)The paper is generally well-written and clearly explains the proposed method.\n\nWeaknesses:\n1)While the \"vague\" attribute descriptions are motivated, direct experimental comparison justifying this choice over more specific descriptions is lacking.\n2)PSLA hinges on modeling loss distributions with Weibull distributions, but the paper lacks strong evidence why Weibull is the right choice or any analysis of how sensitive PSLA is to this assumption. Justification or robustness checks are needed.", "other_comments_or_suggestions": "For Figure 3's clarity, explicitly label VLFPN input/output flows and its role in the caption, and also explain the 'Unknown Object Mining' triangle symbol for better reader understanding.", "questions_for_authors": "Q1: In Table 1, the U-Recall of OW-VAP in Task 2 surpass the Zero-Shot performance, which is counterintuitive. Is the claim that OW-VAP approaches or even exceeds the generalization upper limit of OVD detectors sufficiently justified? Further clarification and stronger evidence are needed.\nQ2: You mention using 'vague' attribute descriptions, which makes sense, but have you considered comparing this approach experimentally to using more specific descriptions? It would be interesting to see some evidence justifying this particular choice.\nQ3: The PSLA method relies on modeling loss distributions with Weibull distributions. Could you elaborate on why Weibull was chosen and what makes it a good fit here? Also, it would be helpful to see some analysis on how sensitive PSLA is to this assumption, maybe some robustness checks?\nQ4: Figure 3 is helpful, but to make it even clearer, could you explicitly label the input and output flows of the VLFPN, maybe in the caption? And could you also explain what the triangle symbol for 'Unknown Object Mining' represents?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rjepBv92Cl", "reviewer": "Reviewer_Drok", "summary": "This paper proposes a novel OWOD framework, termed OW-VAP, which operates independently of LLM and requires only minimal object descriptions to detect unknown objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate you taking the time to carefully review our manuscript. Your professional insights and constructive suggestions are invaluable to our research work. Before addressing the questions, we present a complete description of the overall OW-VAP process.\n\n**All Pipeline**\n\nOW-VAP adopts YOLO-World as its base model. YOLO-World is a real-time open-vocabulary detector. During inference, compared to YOLO-World, we additionally introduce a Visual Attribute Parser (VAP), which consists of only $\\textcolor{red}{\\text{a lightweight multi-layer perceptron (MLP) (e.g., with three layers)}}$. \n\nGiven an image $I$ a list of known class names $Name$, and coarse textual attributes $Att$, these inputs are fed into text and visual encoders to produce visual and textual embeddings: $E_{vis} \\in \\mathbb{R}^{w \\times h \\times d}, E_{c} \\in \\mathbb{R} ^ {c \\times d} $ and $E_{att} \\in \\mathbb{R} ^ {n \\times d}$. Here, $w$  and $h$, $c$, and $n$ represent the width and height of the feature map (downsampled compared to the original image), the number of known class names, and the number of coarse textual attributes, respectively. It is important to note that $E_{vis}$ is distributed across three feature levels, namely {C3, C4, C5}. For simplicity, we only provide the shape at one feature level. Subsequently, the visual $E_{vis}$ and known class names embeddings $E_{c}$ are passed into the VLPAN to enable interaction between visual and textual features. \n\\begin{equation} \n X' _ {l} = X _ {l} \\cdot \\sigma(\\underset{j \\in \\{1, ..., c\\}}{max}(X _ {l}E _ {c}^\\top))^{\\top}.\n\\end{equation}\nHere, $X_{l}$ represents the visual embeddings at different feature levels {C3, C4, C5}, and $\\sigma$ denotes the Sigmoid function.  The resulting output $X_{l}'$ retains the same shape as the original inputs $X_{l}$. Each feature level is divided into a 3 $\\times$ 3 grid, and max-pooling is performed within each grid cell. Across all levels, a total of 27 mbeddings are generated as $\\widetilde{X}$. Finally, multi-head attention is applied to weight the generated embeddings and the original inputs:\n$$ W' =  E_{c} + \\text{MultiHead-Attention}(E_{c}, \\widetilde{X}, \\widetilde{X}) $$. The final weighted result retains the same shape as the original inputs $E_{c}$. Then, $W'$ will be used to replace $E_c$. In summary, VLFPN first enables interaction between visual and textual embeddings and then performs further fusion using multi-head attention to update $E_c$. \n\nThen, the visual embeddings, along with the updated textual embeddings, are used to decode bounding boxes and generate confidence scores for known classes using cosine similarity. Our Visual Attribute Praser (VAP) operates on the visual embeddings, aiming to extract predefined textual attributes from them. In other words, VAP maps the visual embeddings into the attribute textual space. For details on training VAP, please refer to Section 3 and Appendix E. Afterward, as described in Section 3.4, we use in-distribution and out-of-distribution probabilities to assign confidence scores to unknown classes. An overview of the entire process is briefly described in Section 3.1.\n\n**Q1**: On the use of LLMs\n\nAs described in the previous pipeline, $ \\textcolor{red}{\\text{OW-VAP does not utilize any LLMs during the inference stage}}$. It is built upon the standard, real-time open-world object detection (OWD) model, YOLO-World, with only the addition of a lightweight multi-layer perceptron (MLP). In fact, for object detection tasks, the use of LLMs during inference is generally discouraged. This is because the goal of the community is to achieve a real-time, high-accuracy detection model. Thank you again for evaluating the practical applicability of OW-VAP; we hope these clarifications address your concerns.\n\n**Q2**: On Comparisons\n\n$\\textcolor{red}{\\text{In the experiments section and appendix}}$, we provide comparisons between OW-VAP and OWOD detectors utilizing foundation models. These include MEPU-FS based on Free-SOSO, SGROD, SKDF, and KTCN based on SAM, as well as ovow and FOMO, which are built on the $\\textcolor{red}{\\text{same OVD detector}}$ as ours. Overall, OW-VAP achieves comprehensive performance advantages. For example, in Task 1 of MOWODB, OW-VAP outperforms ovow by +13 in U-Recall and +8 in U-AP. For a detailed comparison between OW-VAP and these foundation model-based methods, please refer to Fig. 1.\n\nWe hope these responses resolve concerns regarding the methods we employed. Should you have any further questions, please feel free to reach out to us.", "claims_and_evidence": "Yes, the author provides extensive experimental results to demonstrate the claims.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes. For issues about the experiments, please refer to the below part.", "relation_to_broader_scientific_literature": "This paper perhaps can be a good contribution to the literature.", "essential_references_not_discussed": "The author has discussed most of related works.", "other_strengths_and_weaknesses": "Weakness:\n1. I think the core idea of this method is using an LLM as the attribute parser, enabling unknown object detection by considering a more comprehensive set of attributes. However, I think this core idea is entirely unrelated to the problem of open-world detection. The authors are merely leveraging an LLM to provide additional information, which I find kind of trivial. I think the author should provide more motivation and justification.\n2. Although experimental results show improvement over existing methods, I believe the comparison is kind of unfair. The use of an LLM inherently increases inference time, making the method slower. The author should provide comparision about the model efficiency.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PvvLmxO8gU", "reviewer": "Reviewer_sWQp", "summary": "The paper proposes OW-VAP, a novel framework for Open World Object Detection (OWOD) that eliminates reliance on Large Language Models (LLMs) for attribute descriptions. OW-VAP employs a Visual Attribute Parser (VAP) to extract generic attributes (e.g., shape, color) from visual regions and uses Probabilistic Soft Label Assignment (PSLA) to mitigate optimization conflicts caused by noisy pseudo-labels. Evaluated on MOWODB and SOWODB benchmarks, OW-VAP achieves significant improvements over state-of-the-art methods, surpassing the generalization upper bound of Open Vocabulary Detection (OVD) in Task 2 of MOWODB with a 56.3 U-Recall.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate you taking the time to carefully review our manuscript. Your expert opinions and constructive suggestions are invaluable to our research. Below, we provide our responses to the concerns you raised:\n\n**Weak  1**: Why is VAP effective?\n\nOur OW-VAP is built upon the standard YOLO-World, which is a real-time open-vocabulary detector (OVD). YOLO-World is capable of open-vocabulary visual detection by leveraging training on large-scale datasets to align visual embeddings with textual attributes. This alignment not only supports known categories but also provides, in theory, a generalization capability for unknown categories. This is because the relationship between visual embeddings and textual attributes can be understood as a shared mapping in a latent space.\n\nSpecifically, OW-VAP utilizes visual embeddings generated by the trained OVD detector and designs a transformation mechanism to align these embeddings with predefined coarse attributes. This transformation mechanism acts as a mapping function from the visual embedding space to the attribute space. Leveraging the shared characteristics of this mapping function, OW-VAP is able to learn common cross-category representations from known categories, enabling semantic generalization to unknown categories. The theoretical basis for this generalization can be attributed to the following points:\n\n1. The OVD detector gains strong visual-text alignment knowledge through pretraining on large-scale datasets.\n\n2. The attribute space is intentionally designed to be coarse and abstract, reducing significant differences among object categories and thereby enhancing the generality of intermediate representations.\n3. The transformation mechanism between visual embeddings and attributes prioritizes shared information instead of specific category features.\n\nAdditionally, our method enhances OW-VAP's capability to detect unknown categories by utilizing the pretrained knowledge of the OVD detector. This pretrained knowledge not only supports effective description of known categories but also establishes a robust foundation for aligning visual inputs with textual attributes for unknown categories. For instance, as demonstrated in Table 3, OW-VAP achieves a U-Recall of 45.6 and a U-AP of 10.4 by only using a combination of coarse textual attributes with out-of-distribution probabilities.\n\n**Weak 2**: The annotations in Figure 3\n\nIn Figure 3, we illustrate the training process of VAP. In the figure, the blue circles represent samples already labeled in the training set. During training, these samples are treated as positive instances, and we train VAP to pull them closer to their nearest coarse descriptions (represented as yellow circles in the figure). The triangles, on the other hand, represent potentially unlabeled objects within the dataset. These unlabeled objects are further differentiated using the three selection rules outlined in Section 3.2. This process identifies potential objects from unlabeled samples (which correspond to background samples in a closed-set training setting). As such, we refer to this process as \"background object mining.\" Additionally, you may refer to our explanation of this procedure in response to @Drok for further details.\n\n**Q1**: Coarse Attributes\n\nFOMO utilizes large language models (LLMs) to generate detailed attribute descriptions. Before creating the detailed descriptions, it categorizes the object attributes into 10 groups and then generates descriptions within each group. In our approach, we focus only on coarse descriptions. To achieve this, we use the 10 attribute categories and expand them to generate sentences indicating whether each attribute $\\textcolor{red}{\\text{is significant or not}}$ (as shown in Table 4).\n\n**Q2**: Choice of the Base Probability Model\n\nIn fact, $\\textcolor{red}{\\text{this is a default choice in the OWOD field}}$. Since ORE introduced Weibull modeling for energy-based scenarios, the Weibull distribution has become the standard for similar modeling tasks (e.g., as used in MEPU). Of course, other distributions, such as the Beta distribution, could also be explored and used as a tunable hyperparameter to achieve better performance. However, introducing alternative distributions would increase the complexity of the method, which could hinder further research by the community. Therefore, we adopt the default setting in the OWOD field and use the Weibull distribution to demonstrate the effectiveness of PLSA. Moreover, using a Gaussian distribution is not feasible, as the shape of the distribution is not symmetric and does not match the characteristics of the data.\n\nWe hope these clarifications adequately address your concerns regarding our methodology. Should any questions remain, please feel free to contact us for further discussion. We sincerely appreciate your thorough review and valuable feedback.", "claims_and_evidence": "Yes, the claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the evaluation metrics and dataset used in this paper are suitable for the current scenario.", "theoretical_claims": "I check the correctness of proofs for theoretical claims.\nThe theoretical contributions of the paper are well-motivated. The introduction of VAP and its role in parsing object attributes is sound. However, while PSLA effectively assigns soft labels to mitigate optimization conflicts, a more in-depth theoretical analysis of its probabilistic modeling would strengthen the claim. Specifically, the use of the Weibull distribution for loss distribution estimation warrants further justification.", "experimental_designs_or_analyses": "The experiments are well-designed, with comprehensive comparisons against multiple baseline models. Strengths include:\n1、performance evaluation on OWOD benchmarks with clear improvements,\n2、ablation studies demonstrating the contribution of each component.", "relation_to_broader_scientific_literature": "The paper effectively positions itself within OWOD research and compares against ORE, OW-DETR, SGROD, KTCN, SKDF, and other recent approaches. The use of YOLO-World as a base model aligns well with modern OVD/OWOD trends.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1、OW-VAP eliminates reliance on LLMs, reducing dependency on text-based attributes.\n2、Compared with the SOTA OWOD method, there has been a significant improvement in performance.\n3、This paper conducts a thorough experimental validation on MOWODB/SOWODB.\n\nWeaknesses:\n1、The attribute learning process lacks deeper theoretical analysis, and further insights on why VAP generalizes well to unknown classes would be helpful.\n2、The theoretical basis for PSLA probability modeling is limited.\n3、Figure 3 cannot clearly demonstrate the training and inference process of the model. Many texts and graphics are not explained in captions or figures, such as, VLFPN, Weibull, circles in Unknown Object Mining, etc.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1、How the attribute descriptions in Table 4 were proposed?\n2、The choice of Weibull distribution for modeling loss and the heuristic weighting in Equation (7) lack theoretical justification. Can the author provide ablation studies to explore alternative distributions (such as Gaussian distributions) or validate parameter sensitivity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iq7wQEQmPq", "reviewer": "Reviewer_B8QV", "summary": "This paper proposes OW-VAP for OWOD, which does not rely on guidance from LLM. It introduces the visual attribute parser (VAP) to parse\nthe visual attributes corresponding to the current region. The proposed OW-VAP surpasses the state-of-the-art (SOTA) methods with an advantage of over 13 U-Recall and 8 U-AP in unknown detection.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments and suggestions on our manuscript. Below, we provide detailed responses to your feedback:\n\n**Q1**: Similarity to Prior Work\n\nWhile the referenced works also utilize textual attributes, our OW-VAP differs significantly from these methods in several key aspects:\n\n1. **Different Task Domain**: The referenced works focus on open-set classification tasks, aiming to train a classifier capable of handling new classes. In contrast, our method addresses the open-world object detection (OWOD) task, which stems from object detection. Specifically, OWOD consists of two parts: novel class discovery and incremental learning. The former requires the detector to identify objects not seen in the training data, while the latter involves fine-tuning the detector to adapt to new tasks based on existing knowledge. For details on OWOD task settings, please refer to  $\\textcolor{red}{\\text{Appendix B}}$.\n\n2. **Different Data Requirements**: The referenced methods assume datasets with explicit attribute annotations for training, which is costly to obtain. In contrast, our approach does $\\textcolor{red}{\\text{not require any additional manual annotation efforts}}$. For example, the MOWODB benchmark is a combination of the VOC and COCO datasets, where the COCO dataset alone contains $\\textcolor{red}{\\text{over 1.5 million training instances}}$. It is practically impossible to annotate every object in such a large dataset. Instead, VAP training is seamlessly integrated into the detector training process.\n\n3. **Different Methodology**: The referenced approaches employ statistical techniques such as Bayesian inference for unknown class prediction. In contrast, our method leverages deep learning to map visual embeddings of known classes into a general attribute space. In essence, our VAP functions more similarly to a clustering method.\n\n**Q2**: Coarse Attributes\n\nOur use of coarse textual attributes provides several advantages:\n\n1. Coarse attributes utilize the same textual descriptions (only 10 in total) across all objects, making them more robust when transferring to new objects compared to fine-grained attributes , which heavily depend on the accuracy of LLMs. For example, in incremental learning, our OW-VAP can directly transfer previously learned knowledge without requiring additional training. In contrast, methods based on fine-grained attributes require generating new attributes followed by an additional round of selection and training ($\\textcolor{red}{\\text{Table 7 and 8}}$).\n\n2. As the number of known classes increases, the number of fine-grained attributes also grows. For example, in Task 4 of the MOWODB benchmark, the number of fine-grained attributes reaches $\\textcolor{red}{\\text{over 8K}}$. Using such fine-grained attributes would be impractical for OW-VAP because it would significantly reduce real-time performance and hinder application in real-world scenarios.\n\n3. We compared OW-VAP with fine-grained attribute methods (e.g., FOMO series) in Tables 7 and 8. OW-VAP achieves notable performance advantages, $\\textcolor{red}{\\text{with 4+ improvements in U-Recall and 8+ improvements in U-AP}}$. For further comparative analysis, please refer to $\\textcolor{red}{\\text{Appendix G}}$.\n\nFor insight into why VAP is effective, please see our response to Reviewer sWQp (Weak 1).\n\n**Q3**: Hyperparameter Reproducibility\n\nOur method involves only $\\textcolor{red}{\\text{three hyperparameters}}$, whose impact on detector performance is summarized in Table 6. For additional details, please refer to Appendix F. Moreover, the model's performance is not highly sensitive to these hyperparameter settings, meaning they can be shared across applications without modification.\n\nRegarding reproducibility, as detailed $\\textcolor{red}{\\text{in the Code section of Appendix E}}$, all training logs—including weight files, training logs, and visualized results—will be released following code review for transparency and replication purposes.\n\nWe sincerely hope that our clarifications resolve remaining questions and provide further insight into our method. If you have additional feedback or need further assistance, please feel free to contact us.", "claims_and_evidence": "yes.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "This paper doesn't involve much proof of theory.", "experimental_designs_or_analyses": "yes.", "relation_to_broader_scientific_literature": "The work reveals that previous works resly on the attributes provided by Large Language Model (LLM). Consequently, their performance is highly susceptible to the accuracy of attribute descriptions by LLM. Besides, in subsequent incremental learning, it is challenging\nto quantify the contribution of each attribute to the unknown class, leading to difficulties in attribute selection.\nThus, this paper proposes a new detection framework.", "essential_references_not_discussed": "Most related works are cited properly in the paper.", "other_strengths_and_weaknesses": "**Strengths**\n\n- This paper proposes OW-VAP for OWOD which does not rely on guidance from LLM, by introducing the visual attribute parser (VAP) to parse the visual attributes corresponding to the current region. \n- The proposed OW-VAP surpasses the state-of-the-art (SOTA) methods with an advantage of over 13 U-Recall and 8 U-AP in unknown detection.\n\n\n**Weaknesses**\n- The idea of visual attribute parser is not novel. It is similar to the previous traditional zero-shot learning paradigm ([A], [B]). These works should be properly cited and discussed in this work.\n- This work uses coarse attributes instead of fine-grained attributes for VAP training. What if the training is based on fine-grained attributes? The results should be displayed for discussion. Besides, the reasons why coarse attributes work than fine-grained ones deserve further analysis.\n- There are many hyperparameters involved in the paper, making the results less reproducible.\n\n\n\n[A] Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly\n\n[B] Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript received ratings of 4, 3, 3, and 3. Reviewers appreciated the proposed method and promising results with comprehensive benchmarking against a strong set of baselines on standard OWOD datasets. The reviewers also raised several questions and the authors provided rebuttal to address them. Post-rebuttal, all reviewers remained positive and generally expressed that their concerns are mostly addressed. Given the manuscript, the rebuttal, and that all reviewers are generally positive, the AC agrees with the reviewers and recommend acceptance. Authors are encouraged to take into account all the suggestions of reviewers and rebuttal when preparing the revised manuscript.
Decision: Accept (poster)