UMB: Understanding Model Behavior for Open-World Object Detection
Authors: Xing Xi
Yangyang Huang
Zhijie Zhong
Ronghua Luo
Venue: NeurIPS 2024 poster (2024)
Keywords: Computer Vision
Object Detection
Open-World Object Detection
Primary area: machine_vision

Abstract: Open-World Object Detection (OWOD) is a challenging task that requires the detector to identify unlabeled objects and continuously demands the detector to learn new knowledge based on existing ones. Existing methods primarily focus on recalling unknown objects, neglecting to explore the reasons behind them. This paper aims to understand the model's behavior in predicting the unknown category. First, we model the text attribute and the positive sample probability, obtaining their empirical probability, which can be seen as the detector's estimation of the likelihood of the target with certain known attributes being predicted as the foreground. Then, we jointly decide whether the current object should be categorized in the unknown category based on the empirical, the in-distribution, and the out-of-distribution probability. Finally, based on the decision-making process, we can infer the similarity of an unknown object to known classes and identify the attribute with the most significant impact on the decision-making process. This additional information can help us understand the behavior of the model's prediction in the unknown class. The evaluation results on the Real-World Object Detection (RWD) benchmark, which consists of five real-world application datasets, show that we surpassed the previous state-of-the-art (SOTA) with an absolute gain of 5.3 mAP for unknown classes, reaching 20.5 mAP. Our code is available at https://github.com/xxyzll/UMB.

UMB: Understanding Model Behavior for Open-World Object Detection
Xing Xi
Yangyang Huang School of Computer Science and Engineering South China University of Technology GuangZhou, China 510006
Zhijie Zhong
Ronghua Luo"
Abstract
Open-World World Object Detection (OWOD) challenging task that requires the detector to identify unlabeled objects and continuously demands the detector learn new knowledge based on existing ones. Existing methods primarily focus recalling unknown objects, neglecting to explore the reasons behind them. This paper aims to understand the model's behavior in predicting the unknown category. First, model the text attribute and the positive sample probability, obtaining their empirical probability, which can be seen as the detector's 's estimation the likelihood of the target with certain known attributes being predicted as the foreground. Then, we jointly decide whether the current object should be categorized in the unknown category based on the empirical, the in-distribution, and the out-of-distribution probability. Finally, based on the decision decision-making process, we can infer the similarity of an unknown object to known classes and identify the attribute with the most significant impact on the decision-making process. This additional information can help us understand the behavior of the model's prediction in the unknown class. The evaluation results on the Real-Worle World Object Detection (RWD) benchmark, which consists of five real-world application datasets, show that we surpassed the previous state-of-the-an (SOTA) with an absolute gain of 5.3 mAP for unknown classes, reaching 20.5 mAP. Our code is available https://github.ee http:/hhhwbooo///xalll//M/BB
Introduction
As fundamental task in computer vision, object detection has always been the focus of extensive attention [ 3. Traditional object detection methods are trained on closed datasets, assuming all de- tected objects have already been annotated in the training set. However, the real-world environment's s complexity means it impossible to annotate all objects.
a result, the application of traditional detection methods is limited. Open World Object Detection (OWOD) has been introduced to address the issue. OWOD can be divided into two subtasks: mining potential objects and incremental learning. The former requires the model to detect categories in the test set that have not been annotated in the training set. These newly discovered objects are then handed over to annotators, who select the categories of interest. Subsequently, the model is required to fine-tune its existing knowledge to detect these newly added categories (incremental learning).
Existing works primarily focus on generating pseudo-labels for potential objects in the training set, treating these pseudo-labels samples with high objectness predicted as background as potential objects. CAT(5) and RE-OWOD utilize selective search to provide annotations for unknown categories. OW-DETR(Z proposes an iitention-drived pseudo-label strategy to mine potential positive samples. However, despite these
annotations for unknown categories. For instance, ORE labels
*Corresponding author: rhluo@scut. edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The model has predicted an unknown object, but what is the reason behind this? How it related to the class?
unknnow
Annotator
unknown: Fish Color
The object was predicted unknown class due its similarity with the fish. The color was the most influential factor in the model's dechsion-makinn process.
known
known
Merely identifying unknown obje- cts and providing them to the annotator.
Others:
Ours: Identifying unknown targets and c- omprchending the behavior of the model.
Figure 1: An illustration of our UMB and other methods. Previous OWOD methods only detected unknown objects (left), while our method further understands the model's behaviour (right).
heuristic methods being able recall potential objects, they share a common flaw. As shown in Figure (left), existing methods can only detect unknown objects and then provide these labels to annotators. However, the reason the model would predict these objects remains unknown to the annotators. Therefore, this paper attempts to understand the model's prediction of potential objects, establish connections between unknown and known categories, and then provide this additional information to annotators.
To achieve this, propose novel model (UMB) that textual attributes to mine potential unknown objects. Specifically, we first define targets that share similar attributes with known categories but are predicted as background as potential objects. Then, to find these potential objects, we build a distribution that associates attribute similarity with the probability of positive samples, the empirical probability of object possessing particular attribute being
which be classified as a positive sample. If a sample predicted as background has high empirical probability and attribute similarity, we regard process, we infer the most similar known classes with the unknown object and calculate the most significant impact attribute. As shown Figure (right), our method can identify the unknown and provide information about their connections with known categories and the influence of attributes on decision-making
an unknown object. Finally, based on the decisioo-maaing
We evaluated our method on the Real- World Object Detection (RWD) benchmark composed of datasets from five practical applications, and our method achieved significant improvements. We improved almost all datasets, surpassing the OVC (GT) that uses real class names. Significantly, in the Surgery (36] dataset, achieved the 213% performance in unknown category. The main contributions of this paper are as follows:
To the best of our knowledge, we are the first notice the limitations of models on unknown predictions and attempt to understand the predictive behaviour of models.
To achieve this, propose new model framework (UMB) that detect unknown cate- gories and understand model behavior utilizing the textual description of known categories.
We model the textual attributes and the probability of positive samples to obtain the empirical probability. By combining the empirical probability, the in-distribution probability, and the out-of-distribution probability, we are able to discover unknown categories.
The evaluation results on the RWD benchmark show that our method achieved significant performance improvements, establishing new state-of-the-art (SOTA) with 5.3 mAP advantages in both known and unknown category performance.
2 Related Works
2.1 Open Vocabulary Object Detection
Open Vocabulary Object Detection (OVD), as subset of open-world perception, was initially introduced by OVR-CNN(8) OVD employs the text encoder to transform classes needing detection

---

into text embeddings, determining the current object's class by calculating the similarity between all text and visual embeddings. Subsequent works further have enhanced OVD's performance, including knowledge transfer from pre-trained models through distillation 9 TO [1 the addition of high-
quality object candidates 13]. and alignment of text-visual regions 14 15 16
18 However, in the setting of OWOD, these methods fail to detect the unknown class due to the uncertainty of
object categories. Our approach, based on DWL-ViT broadens OVD to OWOD by modelling the correlation between text attributes and the probability of positive samples.
2.2 World Object Detection
Open World Object Detection (OWOD), distinct from OVD, presents stricter settings and is a more challenging task, proposed by ORE OWOD requires the detector not only to detect potential unknown objects without any information of unknown classes (including category names) but also to fine-tune the detector newly introduced classes for continuous learning of new knowledge. Existing research focuses on heuristic assumptions for potential targets. ORE considers background samples with high objectness in RPN potential unknowns, OW-DETRO calculates the average score of feature regions to determine positive samples, and PROB 20) proposes the use of Mahalanobis distance to discover the potential positive samples. Some other methods additional pseudo-label generation mechanisms to generate annotations for potential objects, including selective search [5) 21
[22, random sample generation (23), and large model knowledge transfer (24 [5] 26). However, these
OWOD methods focus on detecting potential objects and ignore investigation into underlying reasons. Our method attempts understand the behaviour of the model's unknown prediction, establishing a relationship between unknown objects and known classes.
3 Our Approach
Our method, named UMB, is built upon 19L with the overall process illustrated in Figurd 2 First, what characteristics should of unknown target possess? We posit that if an object is predicted as background but exhibits attributes of known classes, it should be considered an unknown target. Therefore, we model the attributes of known classes and the probability of positive samples to build distribution of the empirical probability (Sec. 3.2). The distribution represents the detector's 's empirical confidence in predicting objects with known class attributes as positive samples. If background sample's 's empirical confidence and similarity to known class attributes In-Distributton Probability Sec. are both high, we consider a potential object.
Then, since predictions for known and unknown classes are based on text attributes, we can infer the most similar known class based on the attribute similarity of the unknown object (eqn. [6]. Finally, we can calculate the contribution of each attribute based on the decision-making process of unknown predictions, thereby identifying the attributes that have the greatest impact on decision-making (eqn.
17 This additional information can aid in understanding the model's behaviour in unknown classes.
3.1 Background
To obtain text that describe objects, we use the following template 28 29 to request the Large Language Model (LLM) to list all attributes related to known classes:
Template (a 2) I am using a language - vision model to identify (C}. List the {Z} attributes of {C), which will be used for detection.
(1)
Where C' and Z denote the class name and predefined attribute type (e.g., shape), respectively. These attributes are filled into the prompt template
Prompt (Z A) object which (is/has/ete) <>>
is < <>>,
where A is the attribute text generated in eqn. π e.g., blue. Then, those prompts are fed into the trained text encoder to generate text attribute embeddings Eatt = [Catt, : Catts..... Catt,, Catt, 1 E Raxd where d the hidden dimension, e.g. 512. n denotes the number of the text attributes. The image fed into the trained visual encoder to generate visual embeddings Exis- = [Emis Erissg..... Eois E Ba where m represents the number of patch. In order to establish a connection between these class- agnostic attributes and known categories, we use additional weights W e pmxa trained in known
3

---

Attribure Modeling
Unknown Inference
Bux Head
Crgu my fo) bax,
Chas- agassig Text Eibbecking
Sec. 3.1
Trained Attriuze Weight (W)
6oy Way hi) bax,
Carqueos
Trained
(tg.2s: uy. Ag)
Shaps, Color, Context, Behavior, App-
Trained Text
Visual
(ur Wa. Ac) bax,
Transformer
Encoder
Cally
Fncoder
Ches Head
probability
CUAEE
Pages
Auribute Gienraaioo Guided by Large Language
List of as names
in-distribution
OOD: of distribution
simffrox Rud Jo score [Stt
Eqn. 13
Trained Visual Encoder
Projection Linear
Ens
Multinocal Probabilitic
Man Weight
ID Prob
Urknnown
Prob
Disthution
Mintuee Model
shiff soale
P&Ca Hiigy
PMM
SUnE R.
Egu.
OOD Prub
PMM
Engirical Proks
Figure 2: Overall structure of our UMB. It begins by populating prompt template with known class names and employing large language model (L.LM) to generate attributes (Sec. [3.]] These attributes are then filled into template and encoded by text encoder to generate attribute embeddings (Faat). We model the attributes and their corresponding positive sample probabilities to build empirical probability (Sec. 3.2p. We utilize the empirical, in-distribution and mut-of-distribution probability to ascertain whether an object pertains to an unknown category (Sec. 3.3
categories to linearly combine similarities. Therefore, given visual embedding Evis, the probability of its corresponding known category, j is:
p (Cylevis,)) Sigmoid (wy,1 sim (Evie, Catt,) + ... Wjnn sim (Evio, Catt,))
Sigmoid Wj.k simm(enisp Catt,, Catt,));
where Sigmoid the Sigmoid activation function, and sim denotes the Cosine Similarity. The pseudocode for known class prediction and attribute generation can be found in Algorithm
3.2
Text Attribute Modeling (TAM)
3.2.1
Attribute Modeling
We model the attribute similarities in the training set with category confidence as the positive sample probability to build empirical probability distribution. However, shown in eqn. the score is the linear combination of attribute similarities, so it influenced by all attributes simultaneously. Thus, we weigh confidence with linear combination weights (W Rmxx)) to balance the contributions of different attributes. Specifically, given visual embedding Evisz. positive sample probability of attribute for category j can be represented as:
p (Catte: Cylevies) w0),"" PCCllevis)) W/ys = W
where 3 is hyperparameter used to balance the contributions of weights and scores. For simplicity, use the geometric weighted average. We incorporate all similarities in the training set and their corresponding probabilities of positive samples, establishing a mapping fú,j : simm(Eatt, Cvise) ) p (Catt, Cylevie)). However, during training, the model cannot utilize the annotations of any unknown classes. Therefore, we define the positive sample probability of attribute i for the unknown class as the maximum of its probability to all known classes. Specifically, the probability of attribute for the unknown class C. can be represented as:
$ (Catt;: Cu Culevise) max (p (Cattir Cilevisa) . (Catt; : Can (eeiss)))
argmaz & (Catt, Cý (xlesse)) jE(l.m]
(5)

---

0.18
Probabilistic Mixture Model
Distribution
Interpolation
0.88
& Window
0.00
0.00
5.50
0.90
0.00
1.50
0.13
0.10
1.02
0.00
5.03
Figure 3: An illustration of the Probability Mixture Model. To establish a continuous probability distribution, we linear interpolation the original distribution (left) to estimate missing points and employ the sliding window to eliminate noise within the distribution (middle). Finally, we use the probabilistic mixture model to fit the optimized distribution (right).
3.2.2 Distribution Optimization and Fitting
To establish a continuous probability distribution, we need to optimize and the original distribution. First, contrary to the OWOD benchmark that heavily relies on extensive COCO 30) and VOC[3]] data, RWD pays more attention to real-world application and specifically designed for the few-shot setting. This results in the model not having sufficient samples to establish the probability distribution. Consequently, there are some undefined points in the mapping function faz. To mitigate this, we employ the linear interpolation to estimate the values of these missing points g:
fz)(2) = k(r - =n) + (2) k (f,) - fi)(z))) -z),
(6)
where, 2n and y. respectively represent the points the left and right of that closest in the mapping Ji.j-
Then, utilize the sliding window to filter the noise present in the distribution. With predetermined window size, we calculate the maximum positive sample probability across the entire window to substitute the current value:
fs.( )) argmaz (t.((sim(eviss, Catt,) + a),
(7)
aE(o,W..-1)
where W.2 denotes the window size. As depicted in Figure 3 the employment of linear interpolation and the sliding window ensures the original shape of the probability distribution remains intact, concurrently minimizing the noise inherent in the distribution.
Finally, as shown in Figure (middle) the optimized probability distribution fa.. demonstrates the multi-peak characteristic. Consequently, we postulate that the original distribution is composed of multiple basic probability distributions (e.g., Gaussian Distribution). As a result, we employ the mixture probability distribution to fit the initial distribution. Specifically, we construct the model using the linear combination of multiple Gaussian distributions:
(siumsim(evisa Cali))
Gm (mimm(eeesss Cau Mwa. Jas Ma):
(8)
(m-
Gm(r]w,a,, 0,
20
0v2m 2
where A is the number the Gaussian distribution. Additionally, we observed that certain attributes demonstrate the skewed distribution, suggesting that fitting with the Gaussian model may not be the optimal choice. Consequently, we utilize the asymmetrie Weibull distribution as a substitute for the Gaussian distribution:
k
Wb<x/w, A,
In order to ascertain the parameters of these distributions, we designate them as learnable parameters and employ Mean Squared Error (MSE) as the loss function for optimization, The pseudocode for text attribute modeling can be found in Algorithm 2

---

3.3 Unknown Inference
Following FOMO 27,, we calculate the weighted mean of all attribute embeddings the embedding for the unknown class:
Eatta
Catt: Wiji E IR..
(10)
Following this, we utilize the pre-traineds scale layer (Rd R and shift layer (IR" R.)) for the purpose of scaling the similarity [19]:
Tesim(evis eatr.)) (sim (eving eatt.) shif((eniss)) scale(e
(11)
Finally, we adjust the similarity of the average embedding, This adjustment is segmented into three components: empirical probability, in-distribution probability, and -itto--distrbbution probability.
Empirical Probability (Empirical Prob). For known categories, each attribute contributes unevenly to the category score (eqn. (p. Hence, for the unknown class, merely using the summation of the empirical probability to ascertain category confidence is suboptimal. We utilize the maximum weight
from the known class to balance the contributions from various attributes of the unknown class. Specifically, for the visual embedding Evisy. its corresponding empirical probability is:
fulevise) = E Jsii(sim(erin Eatt.) Wi, W; argmaz Wj.i-
(12)
i=1
jE[1.m]
Herein, fi,u denotes the positive sample probability of attribute towards the unknown class, as established earlier.
In-Distribution Probability (ID Prob). We aspire for the model to observe the known attribute of the current object. Consequently, we incorporate the weighted sum of the sealed attribute similarities:
n
fiD(evis)) Σ Sigmoid(T(eviee Catt.) Wz.
(13)
Probability (OOD Prob). Both empirical probability and in-distribution
probability are based on the model's prediction on known classes. Therefore, inevitably, the model predicts high empirical probabilities and in-distribution probabilities for known categories. To counteract this, we employ out-of-distribution probability offset their influence:
fOoD (Cvis.) = ar argmaa - Softman (Ssim((Ceis Catt,)) w.
(14)
jE[1.m] jE[1.m]
Ultimately, given the visual embedding Cvisy the corresponding confidence for the unknown class denoted as:
p(Culeviea) Sigmoid (fu(evis))
fID aa) fOod(eviss)
Empirical Prob
1D Prob
OOD Prob
(15)
((simoi Catt., ))),
Average Similarity
where is used to balance the contribution form in-distribution and empirical probability.
3.4 Additional Information
Similarity between known and unknown classes. Predictions for both unknown and known classes are determined by attribute similarity. Hence, we can compute its similarity with known classes based on the visual embedding of objects classified as unknown. Similar to unknown inference, we take into account the empirical probability of the current object and its confidence in being predicted as a known class. Specifically, for the visual embedding Evis, of objects predicted as unknown, the corresponding similarity to known classes is:
Su(Emis)) = so fomta mE[1.m]
Σ (.slsim(evisssattt + +xp(C P(Crleniaa)
(16)

---

Aquatic
Acrial
Game
Medical
Surgery
Overall
Taski
Task2
Tiski
Task2
Task1
Tusk2
Task1
Tisk2
Task1
Task2
Tuskl
Task2
Thik (D(->))
U
CK
K
PK
CK
PK
CK
U
CK
K
PK
CK
PK
CK
Base+GT-B
29.8 45.0 45.0 36.7
1.3
5.7
5.7
1.4
15.0
0.4
0.4
0.1
0.5
0.0
0.1
5.6
1.5
14
0.3
10.4
10.5
10.5
7.7
Base-FS-B
7.1
10.4
10.1
16.0
4.6
488
39
0.6
119
11.3
10.9
14.8
14.7 108
FOMO-B
3.5 43.8
44.1
40.8
0.9
12.0
12.6
5:4
13.3
3.8
4.4
4.1
211
6.4
5.5
11.5
611
12.7
12.9
11.
522
15.7
15.9
14.6
Base-GT-I.
34.8 36.0
36.0
42.3
1.0
7.9
7.2
0.8
12.4
0.9
0.8
0.3
2.4
0.2
0.2
0.3
2.4
0.2
2.6
1.3
10.6
9.0
9.4 9.0
Base FS-L 2.4 43.6
42,9
42.8
9.7
23.7
21.9
13.0
10.4
10.2
13.4
1.1
23.2
21.7
24.2
3.6
26.0
25.0
7.4
25.4
24.3
20.2
ROMO-L 18.2
50.1
25.3
23.7
16.0
9.9
9.4
21.8
34.6
29.0
28.9
27.4
26.1
23.5
19.0
16.9
UMB-Wb-B 13.5 43.8 43.0
39.7
1.4
18.8
19.0
16.3
4.1
4.7
4.3
2.3
11.8
14.
13.9
14.3
11.1
17.2
16.9 14.6
UMB-Gm-L. 18.6 50.7 50.5 50.4
11.2
42.7
40.4
22.6
35.1
11.1
10.7
10.5
13.2
22.2
19.1
34.5
24.5
36.6
39.0
17.4
20.5
32.7
31.9 27.1
UMB-Wo-L
42.8
40.4
10.7
8.6
36.6
26.8
Table 1: Comparison with previous SOTA methods on the RWD benchmark. Base+GT represents the standard OVC setting using all class names including unknown label. Base-FS indicates the baseline of fine-tuning the benchmark model with the same supervision received(21 respectively represent two different sizes of the OWL-ViT model, B/14 and L/14. U, K. PK, and CK respectively represent unknown categories, known categories, previously known categories, and currently introduced categories. Overall indicates the average performance of the model on 5 datasets. Wb and Gm respectively represent use of Weibull and Gaussian distribution during the fitting stage.
and
Maximal attribute contribution. Attributes used to compute the similarity with visual embed- dings, and then the model makes predictions based on this similarity. Therefore, the contribution of each similarity can be calculated to determine the impact of particular attribute in the decision- making process. For a visual embedding cvis that is predicted as an unknown class, the influence of attribute the current decision is denoted
Ctr(eatt,) (Sigmoid([ (Tevis ( a fi,a(sim(evia, Catt,)) a))
(17)
4 Experiments
4.1 More Details and Experiments
In our supplemental material, provide detailed information about our experiments, including: comprehensive descriptions of the datasets (sec A.2). definition of OWOD (sec A.I), evaluation met-
ries (sec) A.3) details (sec A.4). more extensive ablation studies (sec A.3 analysis and visualization
of PMM training (sec A.6) similarity evaluation(sec A.7)) attribute study (sec A.8 discussion of the
limitations (sec A.10 broad impact(sec A.9
4.2 Datasets
The OWOD benchmark is established on the VOC and (OCO[30) datasets. In the era of foundation models, the zero-shot capability of detectors on such datasets has reached its limit, for instance, OWL-ViT[I9] unknown recall is 79.0. Therefore, following FOMO, we have shifted the benchmark for evaluating detector performance the more practically applicable RWD benchmark.
4.3 Comparison with Other State-of-the-art Models
Table I presents the comparison of our UMB method and previous SOTA methods established on the RWD benchmark. Overall, our method achieved comprehensive leadership, surpassing previous methods with the unknown performance advantage of 4.4 mAP (Wb-B) and 5.3 mAP (Gm-L), demonstrating the effectiveness of our method. addition, although Base+GT uses the name of unknown categories, it performs poorly in the Aerial, Game, Medical, and Surgery datasets. Our method does not rely on unknown class names and significantly outperforms Base+GT (e.g., Surgery: 2.4 (Basc+GT-L) vs 25.6 (UMB-Wb-L),, Wb-L)). When compared with Base-FS, which received the same
7

---

Aquatic
Acrial
Game
Medical
Surgery
Overall
Setting
Usp Upe Avg/Sud Uap
Une AvgiSid Uap
Use
ArpSil UAH
Une Avg/Sid
UAp
AvgiSii Uae Uje AvySul
Mean Embodking+OOD
4.2 76.8
4.8 16.9
22.4 80.6
0.3 2.8
17.0
95.5
9.7 54.5
+ID Probibility
17.7
32.2
7.4
47.4
15.5
+Gusssian (UMB Gm))
18.6 93.3 93.310
11.2 40.2 55.6/6.5
35.1 90.7 90.002
13.2
47.4
42.89
24.5 96.3 96.3/0
20.5 73.4 75.63.2
/Weibull (UMB-Wb)
18.6 93.3 93,30.1 11.1 41.3 55.916.1
32.7
90.7 90.80.2
8.6
47.4
42.2/10.2
25.6
96.3
96,3/0
19.3 73.8 75.7333
Table 2:
Ablation study of UMB on RWD. We provide incremental results of model performance. UAr and UL: represent the model's mean Average Precision (mAP) and corresponding recall rate on unknown category. Avg and Std denote the mean and variance of the recall rate distribution for unknown classes under different a settings (eqn. T5p.
A.5
we provide more analysis.
supervision, FOMO did not achieve comprehensive leadership and even lagged behind by 3.6 mAP in Aquatic. Our method leads whether compared with Base-FS FOMO, and in the Surgery dataset, Gm and Wb exhibit
we doubled the performance of FOMO (12.0 (FOMO-L) vs 25.6 (UMB-Wb-LL) different strengths in various OWL models. UMB-WB Bb shows an advantage on the B/16 model (+1.1 mAP), while the trend is reversed on the L/14 base (-1.2 mAP). Therefore, we provide two different types of probability distributions (Gm and Wb) as interchangeable options.
4.4 Ablation Study
Table provides the incremental results of our UMB. The initial performance uses the average category embedding (eqn. [0) and distriibbtiiinoonn probability (eqn. [4]. When the in-distribution probability is introduced, which is used to capture the known attributes of the current target, the performance improves by 5.8 mAP. However, in the Aerial dataset, UAP only reaches 3.8 mAP, reducing by mAP, replaced by a significant increase in recall rate (+44.1), which means that the detector erroneously treats many background samples unknown objects. Such result also proves the limitations of the unknown recall as the detection metric previously used in the OWOD benchmark. Finally, by adding the empirical distribution, our UMB achieves a comprehensive lead (Gm +5.0, Wb +3.8). In addition, the effect of balance parameter a on the recall rate is not obvious. In fact, in the Aquatic and Surgery datasets, the variance of the recall rate distribution reaches O, which means that alpha correctly suppresses the background samples erroneously predicted by the detector. Overall, our UMB can provide a higher unknown recall rate (UMB-Gm 73.4, UMB-W Wb 73.8) while ensuring detection accuracy.
4.5 Visualization
In figure 4 we provide the qualitative analysis of FOMO and our UMB. The visual analysis is divided into three parts: the recall ability of the unknown category, recall precision, and analysis of additional information. UMB shows superior performance in recalling unknown objects. UMB successfully recalled tools in the Surgery dataset (fifth row, Wb ID 3) and accurately recalled playgrounds and roofs in the Aerial dataset (second row, Wb ID 1, 2), while FOMO failed recall these objects. Regarding recall precision, FOMO predicts multiple results to an object and incorrectly classifies unknown classified objects as known classes, such the hero characters in the Game dataset (third row) and the misclassification of four objects in the Aquatic dataset (first row). In contrast, our UMB shows higher precision. Regarding additional information, the misclassification of FOMO in the Aquatic dataset reflects the defects of OWL-ViT in classifying these objects. With the help of formula n, UMB can infer the category most similar to the current object, and these categories correspond to the categories misclassified by FOMO. In addition, for the same object (ID 3,4), UMB identifies that the attribute with the greatest impact on the entire decision consistent. These results prove the accuracy of our method in inferring the connection between unknown and known and discovering the attributes that have the greatest impact on decision-making.
Conclusion
This paper attempts to understand the detector's behaviour in predicting unknown objects. To achieve this, we propose a novel detection framework, UMB, which employs class-agnostic textual attributes to unearth potential objects in the background. Given that the model's detection process for known
8

---

and unknown classes hinges textual attributes, our UMB can use the textual attributes of unknown objects infer the most similar known category. In addition, we can calculate the attributes that have the most significant impact on the entire decision-making process. This supplementary information aids annotators understanding the model's behaviour in predicting unknowns. We hope that UMB can promote the application of Open-World World Object Detection in real-world scenarios.
Aquatic
ID
Coregrey
Attibaee Text
ID
Coegrey
Afrileae Toxt
Slark
Touure & fetheis (1633 (%)
Stark
Textare Gollkas. (1R.2
[50.6 (50.6%))
(305
The Shark
Appurrezz axceer (17.)
Appezranne (30.5
Pregen
Pregin (36.1
Contect koand-ookmdd
Contor andf-aoline (284
ID
Ereerry
Altiirat Tiet
I1
Caaggny
Anmherr Text
Chissncy
Stape i round. (23 6%)
Cheraey
Skape round. (23.5 %)
(14.0%
(1488 %a
Chimayy
ruxd.
Stope momene
Exprewway-
Grund Taruth
ID
Cotegery
Amrilate Text
m 1/3
m 1/3
Jlán (4.6 hiy
object which da sta Fexuues lerr.
(14.8%)
object whech
%%
Texture
FOMO
Conrgrry
m 1/3
1/5
which (illoe home (17.
$5.7%
Lovendre (4.5 %0
ebject which (silest end Slape cohood (16.5%)
Jlán
object which (eelae Treeaa (17.5%)
Mediaal
14.2%
Tieas Lakoratin
Matunal corticl
Soft Theree
Moserial coetical shell. (5.1.
shall 8133
Calcrustin ta)
75,9%
(1L11 90
Deaal
Pearunet trigacmaa
Diaai
Feattrre as
Phalinaos
Phulangge
incuetrurs (82.2 fi
(122%)
5 9%)
(28.1 Nal
Men-- carzal ttene
Centrel
argae
tnno (97.4%)
Sirgery
1D
Catppry
E3
Caregory
Atmbele Teat
Scelpel
eepoetiniing (6.8
Snippel
Rabastorns eperstin (MD%))
Sclpel
Shopo (6.4
Scaipel
(200%))
Heak (2) (7.3
Shape diis. (6.4%)) (6.0%)
(9.05))
Figure 4: Qualitative Analysis. Each row, from left right, represents: FOMO, UMB-Wb, and UMB-Gm, respectively. From top to bottom, the results are given for Aquatic, Aerial, Game, Medical, only display the TOP-K unknown predictions with
and Surgery. For faimess and clarity, confidence level greater than 0.5. Unknown predictions are marked in Red, while known classes are marked in yellow. Each table provides the most similar known category (Category) for each unknown prediction, and the attribute (Attribute Text) that has the greatest impact on the decision-making process. In section 7 we provide an evaluation of the accuracy rate of similarity prediction.

---

Acknowledgments
This work was supported in part by TCL Science and Technology Innovation Fund (Project No. 20231752).
References
[1] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: A simple and strong anchor-free object detector. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(4):1922-1933 1933, 2020.
[2] Chien-Ya Yao Wang, I-Hau Yeh, and Hong- Yuan Mark Liao. Yolov9: Learning what you want to learn using programmable gradient information. arXiv preprint 236. 2024.
[3] Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingging Dang, and Yi Liu. Detrs beat yolos on real-time object detection. arXiv preprint arXiv:2304.08069, 2023.
[4] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5830-5840, 2021.
[5] Shuailei Ma, Yuefeng Wang, Ying Wei, Jiaqi Fan, Thomas H Li, Hongli Liu, and Fanbing Lv. Cat: Localization and identification cascade detection transformer for open-world object detection. In Proceedings of the IEEE//CV Conference on Computer Vision and Pattern Recognition, pages 19681 19681-19690, 2023.
[6] Xiaowei Zhao, Yuqing Ma, Duorui Wang. Yifan Shen, Yixuan Qiao, and Xianglong Liu. Revisiting open world object detection. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[7] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: Open-world detection transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9235-9244, 2022.
[8] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open- vocabulacr object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393-14402, 2021.
[9] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open- voccbulayyy object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.1399 2021.
[10] Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11186-11196 2023.
[11] Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14074-14083 2022.
[12] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open- vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15254-15266 15264, 2023.
[13) Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference on Computer Vision, pages 106-122. 122. Springer, 2022.
[14] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. Proceedings of the IEEF/VE Conference Computer Vision and Pattern Recognition, pages 14084-14093, 2022.
10

---

[15] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-imag pretraining. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 16793-16803, 2022.
[16) Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23497-23506, 2023.
[17] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open- vocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 11144-11155 2023.
[18] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7031-7040, 2023.
[19] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728-755. Springer, 2022.
[20] Orr Zohar, Kuan-Chich Wang, and Serena Yeung. Prob: Probabilistic objectness for open world object detection. In Pruceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 11444-14553 2023.
[21] Zhiheng Wu, Yue Lu, Xingyu Chen, Zhengxing Wu, Liwen Kang, and Junzhi Yu. Uc-owod: Unknown-classifeed open world object detection. In European Conference Computer Vision, pages 193-210. Springer, 2022.
[22] Xiaowei Zhao, Yuqing Ma, Duorui Wang, Yifan Shen, Yixuan Qiao, and Xianglong Liu. Revisiting open world object detection. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[23] Yanghao Wang. Zhongqi Yue, Xian-Sheng Hua, and Hanwang Zhang. Random boxes are open-world object detectors. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 6233-6243, 2023.
[24] Yulin He, Wei Chen, Yusong Tan, and Siqi Wang. Usd: Unknown sensitive detector empowered by decoupled objectness and segment anything model. arXiv preprint arXiv:2306.00227, 2023.
[25] Ruohuan Fang, Guansong Pang, Lei Zhou, Xiao Bai, and Jin Zheng. Unsupervised recognition of unknown objects for open-world object detection. arXiv preprint arXiv:2308. 16527, 2023.
[26] Shuailei Ma, Yuefeng Wang, Ying Wei, Peihao Chen, Zhixiang Ye, Jiaqi Fan, Enming Zhang, and Thomas H Li. Detecting the worn-world objects with the help of the brain. arXiv preprint arXiv:2303.11623, 2023.
[27] Orr Zohar, Alejandro Lozano, Shelly Goel, Serena Yeung, and Kuan-Chitn Wang. Open world object detection in the era of foundation models. arXiv preprint arXiv:2312.05745, 2023.
[28] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07188, 2022.
[29] Orr Zohar, Shih-Cheng Huang, Kuan-Chieh Wang, and Serena Yeung. Lovm: Language-only vision model selection. Advances in Neural Information Processing Systems, 36, 2024.
[30] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.

---

[31] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 11 11:98-1136 2015.
[32] Roboflow 100. aquarium dataset. https: aquarium- qlnqy may 2023. visited on 2024-04-09.
[33] Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and new benchmark. ISPRS journal of photogrammetry and remote sensing, 159:296-307, 2020.
[34] Roboflow 100. team fight tactics dataset. attp:://niiverre.rooofff com/
roboflow- wtghtteea--tttteeeeeeeeeee
may 2023, visited on 2024-04-09.
[35] Roboflow 100. x-ray rheumatology dataset.
rowoffoo..rroooooooooooooooooo.. com/
roboflow- 100/x-rayy-thheumutllooy may 2023. visited on 2024-04-09.
[36] David Bouget, Rodrigo Benenson, Mohamed Omran, Laurent Riffaud, Bernt Schiele, and Pierre Jannin. Detecting surgical tools by modelling local appearance and global shape. IEEE transactions medical imaging, (4(12))200322111 2015.
[37] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430-8439, 2019.
[38] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
12

---

A Appendix supplemental material
Algorithm 1:
Textual Attribute Generation
Algorithm Text Attribute Modeling (TAM)
and Known Class Prediction
D img {imagel, .... imagem.
* Predefined attribute types */
K Known_coe nowneclssssss== = [C1, ... Ckn
A [Size, Shape, Behavior, .... Appearance]
[C [Eatt, 0. ... eatt,"' [],
* known class names (Aquatic) */
Ckn I...).C C.: {...}}
C = [Jellyfish, Penguin, Shark, Starfish]
for image; in D Dimg do Vision Vision_encodee (magoer((mmgee))) + Evis for Gessc in Evis do
Attributes []
* attribute generation guided by LLM */
for e; in C do
for Catt, Eatt do Stown_classes classes do } building mapping: S3.2 #/ (sim(evisu Catt,) ) ((Cile
for a in A do
for Cj in Known
/6 fill in the template (eqn. 1) */ 8e e.g. Penguin Size,
Template(Penguin, Size) am using language-vision model to identify (Penguin]. List the (Size) attributes of [Penguin]; J. which will be used for detection. */
Cosine Similarity
eqn.3
c4n.4 [[
end end end end
Template(C; Qj) + LLM generate attributes, e.g, blue */ LLM + attribute
eeq. 5R/ for x in Range(-1, gap) do for eatt, in Eatl do (aa__vll==__
collect attributes and predictions corresponding to LLM* (aj, attribute) + Attributes
Mar val for C, in
end end
K Knownn_classes do for val; in I [[C,][eatt,] do if val, in (x, x + gap/ then maar(vall Mar sur__all -
encoding textual attribute to embedding */
Eatt
for 0; attributee in Attributes do
Max_ver
1a fill in the template (eqn. 2) * 1at e.g. blue, Prompt(color, blue) object which (is/has/etc) <color> is <blue>*/ Prompl (a; allribules) + _ec___ecoder-raa + Eatt
end end end
(x, Max Mar_val) f[Cu][eau.
end
end
1* encoding image to visual embedding */
end
Evie =
8* eqn. here we set gap 0.0001 */ for in Range(-1,,, gap) do
for patch_img in image do
dision_encoder(( (patch - Evis
for Catt, Eat do if (z. 0) in f[Cul[eeat. then f[Culleau,] del (z, 0)
end
1 attribute similarity #/
Linear Interpolation +
Sims = []
(x,, estimate) - [[Call
for
Cuisc in Evis do simss; [
end
for Calt, in Eate do 1* sim denotes the cosine similarity */ + sims,
end end
/* eqn. 7 #/ for x in R gap) do for Catt, Eau do f(Culleatel + Sliding W Window fitte
end
sims; + Sims
end
* known class prediction */ output= [
end end
for idz, sims; in Sims do 1e trained attribute weight /*/
for Cutt; in Eutt do
W [idx] sims; + output equ.3
PMM, M, (S(Cu]lear.)) trasning
end
end
return output
return [PMM1, PMM2, .... PMM,]
13

---

In this section, we supplement the details omitted in the main text.
A.1 Task Formulation
In the context of OWOD, the detection task is divided into a series of subtasks T (Ti, T2, Tir and their corresponding categories K (K1, K2,..., pry} }. T; includes all known categories from previous tasks and introduces new categories on this basis: K; = (U) K2) Knew: where K Knea denotes introduced new categories. When the model is trained on T; our expectation is that the model should be able detect all categories it has encountered so far (i.e., K2), well discover those unlabelled but interesting categories. For the purpose of evaluation, the interest object is defined as those that belong K but not K; (i.e., K K:).
A.2 Datasets
The OWOD benchmark is a combination of COCO 30 and V datasets. In the era of foundation models, the zero-shot capabilities of detectors OWOD benchmark have even reached their limits. Therefore, consistent with 20MO we will switch the evaluation benchmark to RWD. The RWD benchmark consists of five typical application scenarios for object detection, including underwater scenes, representing visual blurring caused by the environment (Aquatic[33) aerial scenes, where the targets are small and difficult distinguish (Aerial (33 ); scenarios using synthetic data when data is lacking (Game[34) medical X-ray scenes, where it is difficult to distinguish between categories and professional knowledge is required (35]); and human surgery scenes, where the field of view is blurred by blood (Surgery (36)). The detailed division of the RWD benchmark is shown in T We divide RWD into two subtasks according to a 50% category ratio. When training Task 1. all categories in the test set that belong to Task 2 are treated as unknown classes, and when training Task 2, the categories of Task are considered as previously seen classes.
A.3 Metric
For known categories, we adopt the widely used mean Average Precision (mAP) as the evaluation metric for object detection. For unknown classes, previous OWOD methods 5 used the recall rate of unknown classes as the evaluation metric. However, such a metric leads to models greedily treating all background objects as potential samples. Therefore, we adopt mAP, consistent with the evaluation metric for known classes, which simultaneously assesses the detector's 's recall ability for unknown classes and the precision of the predictions.
A.4 Details
All experiments were conducted using single NVIDIA GeForce RTX 4090 GPU. Following FOMO, we initialized with the frozen OWL-ViT(19 (L/14 and B/16), which was trained on a mixed dataset composed of Object 365 and Visual Genome 33)), demonstrating strong generalization capabilities. The large language model used for attribute generation GPT-3.5. These attributes were matched with all predictions in the dataset, and the corresponding visual embeddings were collected the IOU exceeded the threshold (0.8). The average of these visual embeddings was calculated to obtain the average embedding of the attributes. Following FOMO, we adopted attribute selection, attribute adaptation, and attribute refinement to train the linear combination weight.
All optimizers used Adam W. During the attribute selection phase, BCE was the loss function, and the learning rate remained constant without decreasing with iterations. The attribute selection phase reduced the number of attributes. Based on the ranking of weights after training, only the top 25 attributes per attribute type were retained. Attribute adaptation was used to narrow the distance between the text attributes and the average embedding of the dataset. This phase used MSE as the loss function, with a maximum of 1000 iterations. Attribute refinement took the text embedding as the parameter to be optimized, with BCE as the loss function. Attribute refinement narrowed the distance between the text embedding and the visual embedding. During the attribute selection and attribute refinement phases, the learning rate and maximum number of iterations for training were set to three values (lle-,, 5e-5, le-4), [1, 10, 100]), iterating over these settings during each training to select the optimal setting.
14

---

Dataset Taskl known And Task2 Previous known
Task2 known And TaskI unknown
Fish(100, 100, 1372), Jellyfish(93, 93, 398)
Puffin(100, 172), Stingray(68, 85),
Aquatic
Shark(100, 100, 179). Penguin( 100, 100, 306)
Starfish(43,57 57)
Expressway-Sene Service-areaa11ee 1000 464)
Vehicle(100, 100, 8350), Storagetank((10) 1000, 100, 6229)
Expressway-toll-staiee (100, 302)
Stadium(100, 100, 277), Ship(100, 100, 12420).
Baschallfield((100 1192), Windmill( 1078)
Aerial
Groundtrackfeeld((000, 100, 641), (olffield(100, 100, 222)
Bridge(100, 809). Chimney(100, 334)
Dam((00, 100, 225), Basketballcour(100, 100, 617)
1000, 1072), Overpass(100, 684)
Airpon(100, 100, 287), Airplane(100, 100, 2423)
Tenniscourt(10) (00, 2583), Trainstation( (100 207)
Ga 31, 30), Poppy(21, 21, 32)
Talon[17, 22), Lux(23, 18),
Blitzcrank(28, 28, 28), IIlaoi(24, 24, 23)
Seraphine 17), Jhin(15, 19)
Si5ged(35,, 35, 37), 7ac(25, 25, 27)
Taric(16, 22), Leona(23, 19)
Janna(39, 33), Ezreal(38, 38, 32)
Viktor(15, 18), Lissandra( 25)
Twitch(25, 25, 25), Camille(29, 29, 17)
Yuumi( 18, 28), Akali(17, 17)
Twisted Fate(18, 18, 31), Jayce(29, 29, 24)
Ekko 21). Samira( 19, 20)
Swain(33, 33, 24), Caitlyn(22, 22, 24)
Kai-Sa(24, 23), Dr- Mundo(18, 23)
Game
Lulu(21, 21, 28), Trundle(25, 25, 33)
Fiora(14, 20), Orianna(19) 19, 22)
Warwick(29, 29, 28), Z30ean(30, 30, 25)
Jinx(16, 19), Yone(19, 20)
Katarina(25, 25, 26). Vex(23, 23, 32)
Quinn(22, 18), Miss Fortune(23, 21)
Z2ggs(29, 29, 29). Braum((6, 26, 25)
Sion(22. 15). Kog-Maw(23, 22)
Darius(16, 16, 37), Cho-Gath(22, 22, 29)
Garen(21, 20). Gruves(17, 19)
Tristana(28, 28, 36), Kassadin(22, 22, 23)
Urgot(23, 24), Galio(24, 18)
Malzahar(23, 23, 24), Heimerdinger(26, 26, 30)
Shaco 14, 28). Zyra(18, 20)
Vi(32, 32, 37), Veigar(21, 21, 23)
Tahm Kench(23, 14)
Bipelarforrrossllsssseettttttt 100, 361)
Suction/TubeSkeletom(nn,, 100, 812)
Curette(59, 57), Hook(100, 123)
Retractors(100, 100, 259)
PliersDown(92, 94), Scissors(38, 30)
Surgery
BipolafFooreepsspp(10,,, 100, 508)
Scalpel(95, 90), PliersUp(98, 98)
BipolafForcepsDDow((10,, 496)
BipolafForopsDOwwit..
TuctionTube(1100, 100, 872)
Second metacarpal hone(86, 86, 96)
Fifth metacarpal bone(82, 82, 95)
Soft tissue calcination(38, 50), Ulna(78, 90)
Distal (aallaness(100, 100, 481)
Medical
Fourth metacarpal bone(83, 95), Artefact(2, 3)
Third metacarpal b8ne(80, 80, 95)
First metacarpal bone(80, 94), Radius(76, 92)
Proximal phalanges(100) 1000, 100, 475)
Intermediate phalanges(96, 96, 376)
Table 3: Detailed explanation of the dataset split. Each dataset is split into subtasks, each maintaining a category proportion of 50%. When training on Task 1, the categories of Task are treated unknown classes. During training in Task 2, the classes from Task are labeled as previously seen categories, and new classes divided into Task 2 are introduced. The numbers (numl, num2, num3) following each category in Task represent the number of training instances Task Task 2, and the test set, respectively. The numbers (numl, num2) in Task 2 represent the number of training and test instances for this category Task 2.
15

---

Upon completion of training, the detector could detect known classes To detect unknowns, we established the empirical probability for each attribute. the distribution optimization phase, we set the window value to 10. In the distribution fitting phase, we used two different probability models (Gaussian and Weibull). During training, Adam was used the optimizer, the learning rate was set to 0.01, the maximum number of iterations was 10000, and the maximum number of probability models was set to 5. Since the purpose fitting was to capture the shape of the empirical probability distribution and establish a continuous probability distribution,
set an interval of 0.0001 between -I and This setting was used to sparsify the data the original distribution. Then, distribution optimization and fitting were performed in the sparsified distribution.
A.5
Comprehensive Ablation Experiments
Aqutaic
35.1
30.7
50.38
111
18.6
18:6
11.1
11.3
Knows
(f- 0.5)
(P-00
(f-m 0:4g -a- Knowz
Unksows
(0.2)
5.0
6.3
6.8
0,5
8.0
0.3
0.6
0.9
0.0
0.2
0.6
00 Abounnal Stine
Moa I Mear 1.555
Medical
Surgery
333393
06.3663
8.6
24.5
Wis () (0.1)
Wh-ff-0.0
Gm (8- 0.3)
3.6
8.0
Figure 5: Subfigures through 5 present the performance variations of the model under different settings of (eqn. 15). For clarity, only display the performance changes corresponding the optimal / (eqn. 4. Subfigure 6 provides the statistics of the recall rates corresponding to these five figures varies. Here, Mean represents the average, while SE denotes the standard deviation.
Figure 5 iillustrates the model's performance under different a settings with the optimal 3. addition. subfigure provides the statistics of the recall rates corresponding to subfigures through 5. In the Aquatic and Game datasets, the performance of UMB is not sensitive to the changes in a, showing minor performance differences. This is because Aquatic is consistent with the training data of OWL-VIT, and Game is synthetic data, neither of which poses additional challenges (such as target size). However, in the remaining datasets, the performance of UMB shows significant changes. For example, the performance of the Aerial dataset drops from 11. mAP (wb) 3.6 mAP. These datasets have significant differences from the OWI--VT training dataset, and their environmental characteristics (such as small objects in Acrial, similar objects in Medical, and blood-contaminated backgrounds in Surgery) pose additional challenges to the model. These constraints make UMB sensitive to changes in a. Nevertheless, a reasonable a can balance in-distribution and empirical probability contributions to achieve better detection performance.
For the recall rate of unknown categories, except for the Aerial and Medical datasets, UMB maintains a high value (93.3% in Aquatic) and remains stable in the remaining datasets. This indicates that our method does not predict more objects under different a settings but predicts potential objects more accurately. In the Aerial and Medical datasets, the model's recall rate is almost half that of other datasets, and these recall rates show significant fluctuations with the change of a. Therefore, we infer that when the model's 's recall rate for specific dataset is high, the balance between intra-division
16

---

Aquetic
111
1.0 0220
4.24
1.00
0.34
002
0000
66000
4035
8.30
033
0322
075
-1.
1.5
106
41.
Comtee
Wib
Come
Catt
2.15
2005
0150
Molica
1-
-3.58
-35
000
1.3
0.5
9.91
Sungry
100.5
0.003
8.04
3.072
3.500
0.317
6.22
0.590
01/52
Figure 6: Training result for the Probabilistic Mixture Model. The training results for Aquatic, Aerial, Game, Medical, and Surgery are represented from top to bottom. We only display the fitting process corresponding to the optimal 3 setting for clarity. The settings for can be found in figure 5 Each original distribution is first filtered for noise using distribution optimization (linear interpolation and sliding window) and then fitted with the basic probability model (Weibull Gaussian) the optimized distribution. The first and third images in each row show the MSE loss distribution corresponding convergence, while the second and fourth images present the best-fitting results across all attributes. In the best-fitting result, the yellow line represents the original distribution, and the green line represents the result after fitting the probability model.
probability and empirical probability has a relatively small impact on the recall rate; when the recall rate is low, this balance has more significant impact on the recall rate.
A.6 Probabilistic Mixture Model
Figure o presents the fitting results of the probabilistic mixture model. When the original probability distribution exhibits multiple peaks, the MSE loss of the probability fitting may stabilize at relatively high value. For instance, in the first column of the second row, the original probability distribution is composed of more than three elemental probability distributions, Upon completion of training, the MSE value stabilizes around 7.3, indicating that the model faces challenges when fitting distributions
17

---

with multiple peaks. However, when the original probability distribution exhibits single peak, the MSE loss approaches zero, demonstrating the model's advantage in fitting unimodal distributions. For example, in the Game dataset, the MSE value of the Gaussian model stabilizes around 0.015. Despite the possibility of multiple peaks the original distribution, both the Weibull and Gaussian distribution can capture its essential shape characteristics. For instance, in the Aerial dataset, the Gaussian model can fit two smooth peaks to represent the original distribution. Similarly, the Weibull model can fit a smooth curve when facing distributions with multiple peak features. However, in the case of unimodal distribution, both the Gaussian and Weibull models can capture the original distribution's unimodal feature and fit smooth eurve with single peak. For example, in the Surgery dataset, the Weibull and Gaussian models in the Game datasets demonstrate this ability.
A.7 Similarity Evaluation
In order to understand the behavior of the model, we provide the known class most similar to the un-
Mediall Game
Wh motchal
Gin nilo
Wh
1991
mot_mathhed
0,52096
0.950.94
Repeat
0,9
0,000
0.80181 0,33
0.300
known object and the at-
0.70
tribute with the most sig-
nificant impact on decision- making. Through visualiza- tion, we observe that many known predictions overlap with unknown objects in the detector's predictions. These known predictions represent that the detector considers the current ob- ject to belong to a known category. Therefore, we estimate the model's accu- racy in similar inference by whether the inferred known class is consistent with these overlapping known predictions. Specifically, if their IoU exceeds 0.95, we consider these two predictions overlapping. Then, if the known class inferred by the detector is consistent with the known prediction, we consider the model inference correct. Otherwise, it indicates an inference error. As shown in Figure 7 (left), the detector has many duplicate predictions for unknown and known classes, Among these duplicate predictions, the proportion of correctly inferred predictions is large, more than 80% (Figure 7). Compared with Wb, Gm's accuracy rate is always lower. In the Aquatic dataset, Gm lags behind Wb by 0. percentage points in accuracy, and in Medical, it lags by 0.6 percentage points. Overall, both Gm and Wb show high inference accuracy (about 90%), and compared with Gm, Wb shows a higher accuracy rate (about 2%).
Aguats Actil
m9
0.0
6300
12600
18900
63000
69300
75600
Agquatcc
Aanal
Gaanz
Malical
Clamc
Ovendl
Figure 7: Evaluation of similarity. The left figure represents the number of predictions where unknown and known categories overlap. The right figure shows the accuracy of the model's inference among these numbers.
A.8 Attribute Study
Figure X and Figure P present the results of attribute analysis. For clarity, we have selected only the top three categories with the highest prediction counts from the detector and the top five attributes that have the most significant impact on them. In each row, the top part shows the prediction counts for these categories. For instance, Puffin (252) in Aquatic indicates that the detector gave 252 predictions for the Puffin category. The left side shows the number of times these attributes have been identified as having the most significant impact. Moreover, the right side represents the attributes with the highest average inference scores from the model.
UMB demonstrates strong ability to capture object attributes. In the Surgery dataset, the attribute (Behavior is repositioning) dominated UMB- Wb's 127 predictions (total 274) for the Hook category, and this number rose to 134 (total 273) in UMB-Gm, accounting for nearly half. This implies that when an object exhibits such an attribute, exhibits precise attribute discrimination capabilities. Since the right side of each row presents the average score of the attribute's 's impact on all detector predictions, their differences are insignificant. Nevertheless, in the Aquatic dataset, UMB-Gm still distinguished these attributes. For instance, the
detector will likely predict it as the Hook category. UMB
18

---

Aquatic: Puffin(252), Stingray(192), Starfish(134)
0.20
Context is tropical
Environment is bird 0.17
0.18
0.19
0.16
Context is seagulls
54
0
Shape is flexible 0.00
0.20
0.17
32
0.12
Size is asymmetric
16
31
9
Context is sand-colored 0.00
0.18
0.00
22
0.08
Texture is feathers
34
27
4
Material is collagen 0.00
0.20
0.1
11
0.04
Texture is feathers
20
46
30
Context is deep sca 0.19
0.20
0.00
0.00
Aerial: (nseballfieel(1122)), Chimney(635), Tenniscourt(606)
Appearance is pillars
Appearance is pillars
0.24
8
0.24
0.24
0.24
Shape is round
Shape is paper
0.24
52
20
0.24
0.24
0.24
Shape is paper
Shape chunky
0.24
239
0.24
0.24
Context is yachts
Behavior is changing direction
0.24
83
113
0.24
Appearance is presence benches
298
83
273
62
Appearance is presence of benches
0.24
0.24
0.24
0.24
Game: Taric(81), Urgot(74), Samira(71)
Context is connective tissue
21
Texture is coarse
0.19
5
8
0.00
0.00
0.00
17
Material is endosteum
0.15
Shape cuboid
6
13
0.00
0.00
13
Features is horns
0.11
Shape is bent
12
0.00
0.15
Context is wrist bones
0.07
Context is elbows
18
21
0.00
0.00
Shape is smooth
13
Context is connective tissue
0.18
0.18
0.18
Surgery: (20k(274), Sca)pel(210), PliersDown((04)
Environment is medical simulation center
Texture is coarse
0.09
22
12
0.09
0.09
0.09
108
Material is endosteum 0.09
0.07
Context is
2
22
15
0.09
0.09
cautery device
81
Features is horns
0.05
Behavior is operating
134
4
233
0.00
0.09
0.09
55
0.04
Context iswrist bones
Shape spherical
29
10.5
67
0.09
0.09
0.09
28
0.02
Shape is rounded
84
33
61
Behavior is operating
0.09
0.09
0.09
2
0.00
Medical: Fourth metacarpal bone(183). First metacarpal Mone(161) Ulna(10
Texture fissured
0.60
29
Features is proximity
0.59
0.00
0.00
Material is cortical shell
0.48
28
Features scaphoid
0.60
0.00
0.59
Appearance is Visible joints
0.36
Appearance visible
43
0.60
0.00
0.00
joints between bones
Features swollen joints
0.24
48
Material is brittle
0.00
0.60
0.00
Features cortical bone
5
41
Features paws
0.60
0.00
0.00
0.12
0.00
Figure 8: Cross-category attribute analysis (Gm). Each row from top to bottom presents the results for Aquatic, Aerial, Game, Surgery, and Medical. The left each row represents the number times the attribute influences the decision, while the right side indicates the average score of the attribute.
19

---

Aquatic: Puffin(252), Stingray(192), Starfish(134)
51
0.16
Context is tropical
34
0
Material is collagen
0.00
0.15
0.13
41
0.13
Features nostril
20
19
Size is large
0.14
0.00
0.00
31
0.10
Context is seagulls
4.5
0
Shape swimming
0.14
0.00
0.00
20
0.06
Size is asymmetric
15
30
Appearance suckers
0.15
0.14
0.15
10
0.03
Texture feathers
155
51
40
Shape is flexible
0.00
0.00
0.16
0.00
Aerial: Bascballfield 1003), Chimney(657), Tenniscour((6422
274
0.24
Shape is round
Context is yachts
37
77
44
0.24
0.24
0.24
Appearance is pillars
223
Behavior is changing
0.24
197
18
71
direction
0.24
0.24
0.24
Behavior is changing direction
172
0.24
108
185
42
Shape is sail
0.24
0.23
0.24
120
0.24
Appcarance is presence of benches
225
60
159
Shape is paper
0.24
0.24
0.24
Appcarance is presence of benches
0.24
Context yachts
274
235
0.24
0.24
0.24
0.23
Game: Taric(81), Urgot(74), Samira(71)
0.16
Shape irregular
0
Shape is cuboid
0.15
0.14
0.00
Shape is bent
Shape is spiky
0.13
3
0.16
0.15
0.16
Shape is smooth
Context metacarpal bones
0.10
6
0.16
0.00
0.16
Context is
9
Context is connective tissue
0.06
connective tissue
12
9
0.16
0.16
0.16
5
0.03
Context elbows
23
Context is wrist bones
0.00
0.16
0.00
0.00
Surgery: Hook(274), Scalpel(210) 10), PlicrsDown(203)
127
0.07
Features is teeth
0
1
Features is teeth
0.00
0.00
0.07
Shape is rounded edges
102
Context cautery device
0.06
37
24
19
0.07
0.07
0.07
Shape is thin
76
Shape rounded edges
0.04
20
71
48
0.07
0.07
0.07
Context is cautery device
51
0.03
90
13
49
Behavior is repositioning
0.07
0.07
0.07
Behavior is repositioning
25
0.01
127
102
86
Shape is thin
0.07
0.07
0.07
0.00
Medical: Fourth metacarpal bone (183), First metacarpal bone( (33), Ulna(16)
Appearance is visible joints between bones
50
Features is triquetrum
0.67
0
0.67
0.00
Features is scaphoid
0.54
Texture fissured
28
0.67
0.00
0.00
Material is cortical shell
Appearance visible
0.40
30
joints between bones
0.67
0.00
0.00
Features is swollen joints
20
0.27
50
0
Matcrial brittle
0.00
0.67
0.00
10
0.13
Featurcs is cortical bone
Appcarance is transparency
39
5
0.67
0.00
0.00
0.00
Figure 9: Cross-category attribute analysis (Wb). Each row from top to bottom presents the results for Aquatic, Aerial, Game, Surgery, and Medical. The left in each represents the number times the attribute influences the decision, while the right side indicates the average score of the attribute.
20

---

attribute (Context is the deep sea) had an average score of 0.2 in Puff but dropped to 0 in Starfish. A similar scenario occurred in UMB-W- Wb.. where the attribute (Shape is flexible) had an impact of 0.16 on Starfish, but it dropped to O for Puffin. Overall, both UMB-Gm and UMBBWW can capture and distinguish object attributes and make corresponding predictions based on these attributes.
We conducted additional comparative experiments on Linear Interpolation (LI) and Sliding Window (SW), as detailed in Table of the attached document. The results indicate that employing LI or SW independently does not lead to significant perfor- mance improvements. Both methods exhibit only marginal enhancements compared to the original approach, suggesting that LI and SW, when used in isolation, are insufficient for aceu- rately modeling the data distribution.
Setting
Aquatic
Aerial
Game
Medical
Surgery
ME+OOD
4.2
4.8
22.4
0.3
17.0
+ID
17.7
3.8
32.2
7.4
16.3
PMM(OD)
18.0
4.1
32.3
7.5
18.3
PMM(LI)
18.0
4.1
32.3
7.7
18.4
PMM(SW)
17.8
5.0
32.3
7.1
18.4
PMM(Gm)
18.6
11.2
35.1
13.2
24.5
PMM(Wb)
18.6
11.1
32.7
8.6
25.6
Table 4: Ablation Studies. ME is Mean Embedding, OOD is Out-of- Distribtttii....nn. PMM(OD) uses original distribution, II, SW use Linear Interpolation, Sliding Window. Gm, Wb are filtered models, denoting Gaussian, Weibull distribution.
A.9 Broad Impact
In this paper, we focus on the perfor- mance of detectors in open-world ob- ject detection and attempt to under-
stand the model's behavior when predicting unknown categories. Our approach can help annotators gain a deep understanding of the model's decision-making process, thereby guiding subsequent optimization work and improving the overall performance of the detector. At the same time, under- standing the model's behavior may expose potential flaws malicious actors could exploit for illegal activities. For this reason, choose to open- our code, both to promote the development of the current field and to identify and prevent these potential issues through the power of the community.
A.10
Limitations
Our focus is on understanding the behaviour of model predictions. Hence, we attempt to migrate the OVC detector to the OWOD task. As our method does not directly train the weights of the OVC detector but merely processes its output, the performance ceiling of our method will be constrained by the inherent performance of the OVC detector itself. Furthermore, due to the visual-text alignment relationships of the OVC requiring extensive data training, fine-tuning actual application datasets could lead to additional annotation costs.
A.11 Failure Cases
We present typical cases of detection failures from each dataset, focusing on recall capability and detection accuracy. For instance, the Aquatic dataset, UMB failed to detect small orange fish, while in the Aerial dataset, it did not successfully recall vehicles. These instances reveal the detector's shortcomings in recall capability. Moreover, the Game and Surgery datasets, UMB displayed occurrences of repeated predictions. Nevertheless, UMB still outperformed FOMO in overall performance. Specifically, in the Aquatic dataset, UMB accurately located the contours of the fish, whereas FOMO showed deviations in contour localization and even missed similar objects. Furthermore, FOMO incorrectly identified the reflection of the photographer's shoes the glass as an unknown object, demonstrating lower precision. Similar issues were observed in the Aerial and Game datasets, where FOMO often confused objects with the background, resulting in new erroneous predictions, such misidentifying rooftops did not commit the same errors in these cases. In summary, although UMB also exhibited some false detections in certain scenarios, it outperformed FOMO in both detection accuracy and the ability to recall potential objects.
single object in the Aerial dataset. However, UMB
21

---

FOMO
UMB-Wb
UMB-Gm
1/3
1/5
11 1/5
Figure Qualitative Results (Failure Cases). From top to bottom, datasets are Aquatic, Aerial, Game, Medical, and Surgery. To ensure clarity and faimess in comparison, we only display predictions for unknown, selecting those with confidence score greater than 0.5 and ranked within the top K unknown category predictions. The results indicate that the UMB method demonstrates higher precision and recall addressing the FOMO problem.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the introduction, we discuss the problems that existing research focuses on (first paragraph of the introduction), well the emphasis and shortcomings of current studies (second paragraph of the introduction). We also detail our methods and the results have achieved (third and fourth paragraphs of the introduction).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In section A.10 provide the limitation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: section 3.2 we provide the full set of assumption. section 4.4 we discuss the impact of empirical probability model performance. Additionally, in section A.6 present the detailed analysis and visualization.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the supplemental material, we provide detailed experimental configuration (A.4) and pseudocode 2). environments, and all other settings. Furthermore, will release all the training code and weights.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our code is available https//aaaayoooo.ooppuuuccuu///a
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
gublic/guidess httae//guuiee//eeebbbbmeaeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co cc/ pttppppllllllppieeee http://ooooooooosssoooooooooeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the supplemental material, provide all the training and test details, including: experimental configuration (A.4). data split (A.2) and hyperparameters (A.3).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the experiments that support the main claims of the paper (sec 4.3 and detailed ablation study (sec A.5)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the supplemental material (sec A.4 we provide the experiment compute resource consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps///eorrrp....ppppbbbttbbb
Answer: [Yes]
Justification: We affirm that all research presented in this paper adheres the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In section A.9 we provide the societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets in the paper have been properly credited, and we adhere to the corresponding licenses and terms of use.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is available at hats://gittbbbcom//xxy/////M
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "jC84VKaNCw", "reviewer": "Reviewer_N6Np", "summary": "This paper introduces a new open-world object detection model (UMB) aimed at understanding the model's behavior when predicting unknown categories. By modeling text attributes and positive sample probability, the paper proposes a joint decision-making method based on empirical probability, in-distribution probability, and out-of-distribution probability to infer the similarity of unknown objects to known classes and identify the attributes that have the most significant impact on the decision-making process. The evaluation results on the Real-World Object Detection (RWD) benchmark show that this method surpasses the previous state-of-the-art (SOTA) with an absolute gain of 5.3 mAP for unknown classes, reaching 20.5 mAP.", "questions": "What are the computational resource requirements for this method in practical applications? Is the method feasible in resource-constrained environments?\nCan this method maintain the same performance improvements on other datasets and in other domains? Further validation in more real-world environments is needed to ensure its broad applicability.\nHow robust is the use of the most influencing attributes in inferring the similarity between unknown and known objects? Could noise or outlier data affect the decision-making process?", "limitations": "The method is quite complex and may require substantial computational resources and time for training and inference. This could limit its use in practical applications with limited computational resources.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "First to focus on understanding the model's behavior in unknown category predictions.\nProposes a new framework (UMB) capable of detecting unknown categories and understanding model behavior using textual descriptions.\nShows significant performance improvements on the Real-World Object Detection (RWD) benchmark, especially in unknown categories.", "weaknesses": "1.\tThe method relies on high-quality text attributes and positive sample probability modeling. Performance may be affected if data quality is poor or attributes are incomplete.\n2.\tWhile performing well on the RWD dataset, its generalization ability to other domains and datasets needs further verification, possibly requiring more testing in different environments\n3.\tThe robustness of using the most influencing attributes in the decision process to infer the similarity between unknown and known objects needs further validation.\n4.\tFor large models, the performance and quality requirements for samples and attributes are high. The categories, quantity, and quality of samples and attributes significantly affect network performance and need further analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your valuable comments on our work, particularly your concerns about similarity verification, resource consumption, and deployment. Here are our detailed responses:\n\n**Similarity Verification**. Ideally, we would like to annotate every unknown object in the RWD dataset and identify its most similar known class. However, this approach presents several challenges: 1) Workload issue: A single image may contain multiple unknown objects, such as stones, making it impractical to annotate each unknown object and find the most similar known class in practice. 2) Subjectivity and inconsistency: When manually annotating the similarity between unknown objects and known classes, subjective judgments are inevitable. These can lead to differing opinions among annotators regarding the similarity of the same unknown object to known classes, resulting in inconsistent annotations. Moreover, the diversity and complexity of unknown objects make it difficult to establish a unified annotation standard. Therefore, manual evaluation of similarity is not only unfeasible in workload but also difficult to ensure the objectivity and consistency. \n\nIn addition, our evaluation focuses on how the detector determines the similarity between unknown objects and known classes during the decision-making process, which reflects the detector’s understanding of the current object. In subsequent improvements, we can use this to customize the distribution of the dataset. For example, when the detector considers that unknown object A is similar to known class B, and we want to intervene in this result, we can add some samples that are not similar to the known class B but belong to unknown object A through incremental learning to urge the detector to distinguish them. This is why we propose the concept of the unknown object most similar to the known class.\n\nDespite the challenges of manual annotation, we provide an approximate evaluation in Appendix (A.7). We found that the detector may produce duplicate predictions when predicting unknown objects. These duplicate predictions reflect the detector’s confusion between the current unknown object and known classes. Therefore, we collected all duplicate predictions and calculated the overlap between these duplicate predictions and the known classes given by the detector to indirectly evaluate the accuracy. Specifically, we calculated the IOU of all known and unknown predictions. If the IOU is greater than 0.95 (the highest value of the COCO standard), the current known and unknown prediction are considered overlapping. In the left figure, we show the number of these duplicate predictions, ending with _Repeat. Subsequently, we calculated the number of matches in these duplicate predictions, ending with _matched, indicating the cases where the most similar known class predicted by the detector is consistent with the overlapping class. In the right figure, we show the corresponding accuracy. From the experimental results, the accuracy reached about 90%. This indirect evaluation proves that UMB has high accuracy in predicting the unknown object most similar to the known class.\n\n**Resource consumption**.  We provided them in Appendix (A.4). All experiments were conducted a single NVIDIA GeForce RTX 4090 GPU. Since OWL-Vit has made all weights public, we can download these weights as a starting point and freeze them. Subsequently, we use text encoder to encode all sentences into text embeddings, and training these text embeddings for known class detection. To detect unknown objects, we only introduce a module (TAM) that needs to be trained. The training of TAM is lightweight, and it only needs to train up to 5 groups of probability models. For example, using the Gaussian model, the number of parameters to be trained is 10, including 5 variances and 5 means. This is why we can complete all experiments on a single NVIDIA GeForce RTX 4090. For the inference, compared to the original OWL-Vit, we only introduced the calculation of OOD (Out-of-Distribution), ID (In-Distribution), and Empirical probability. However, these three parts of the calculation can be completed through a few matrix operations. Compared with the inference computation of OWL-Vit, the increase in computation is negligible. Therefore, our inference resource consumption is almost consistent with OWL-Vit.\n\n**Deployment**. The deployment of object detection is a very complex issue, which needs to consider the restrictions of different devices and environments. At present, the most commonly used deployment scheme is to convert the original PyTorch model into a TensorRT model, and use the optimization acceleration of the TensorRT model (such as FP16 or INT8 inference) to achieve real-time inference. When deploying the UMB model, all texts can be preprocessed into text embeddings, and the text encoder can be discarded, and only the visual encoder is finally deployed on the device. This makes the deployment scheme consistent with OWL-Vit. The newly introduced probability calculation part follows the standard PyTorch implementation, so it can also be converted into a TensorRT model. As we mentioned in inference resource consumption, the computational burden brought by our method is negligible. Therefore, our method can run normally on all devices where OWL-Vit models are applicable. In addition, our method can also serve as a pluggable module, migrated to other OVC-style detectors (e.g. YOLO-World), giving them the ability to detect unknown objects and understand the decision-making process.\n\nThe above are our responses to similarity verification, resource consumption, and deployment. We hope that these explanations can clearly answer your questions and help you better understand the strategies and decisions we adopted in our research. If you have further suggestions or need more information support, please feel free to contact us. Thank you again for your recognition and valuable feedback on our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EO1EeYlCAe", "reviewer": "Reviewer_bGxY", "summary": "This paper proposes a new solution for the challenging task of Open World Object Detection (OWOD) by exploring the reasons behind non-classification and then using the textual attributes of unknown objects to infer the most similar known category. Evaluation results on multiple real-world application datasets show significant performance improvements.", "questions": "(1)The limitation discussion is a bit insufficient. It could be better to add some visualizations of the failure examples\n(2)Some mathematical variable symbols(e.g., Eq.6 and Eq.7 ) are not standard. It is recommended that more common and standard notations be used.", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A new approach to solving classification problems in Open World Object Detection, and verifying its effectiveness.", "weaknesses": "The writing of some mathematical symbols needs to be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, which will help us improve the quality of our manuscript. Below, we address each of the points raised.\n\n**Limitation Discussion**: We appreciate the reviewer's suggestion to enhance the limitation discussion with visualizations of failure examples. We agree that visual representations can provide clearer insights into the nature of the limitations. In the revised manuscript, we will include several visualizations illustrating common failure modes encountered during our experiments. These visual examples will be included in the Appendix of the paper, as illustrated in the attached PDF (Figure 1). We present typical cases of detection failures from each dataset, focusing on recall capability and detection accuracy.\n\nFor instance, in the Aquatic dataset, UMB failed to detect small orange fish, while in the Aerial dataset, it did not successfully recall vehicles. These instances reveal the detector’s shortcomings in recall capability. Moreover, in the Game and Surgery datasets, UMB displayed occurrences of repeated predictions. Nevertheless, UMB still outperformed FOMO in overall performance. Specifically, in the Aquatic dataset, UMB accurately located the contours of the fish, whereas FOMO showed deviations in contour localization and even missed similar objects. Furthermore, FOMO incorrectly identified the reflection of the photographer’s shoes in the glass as an unknown object, demonstrating lower precision. Similar issues were observed in the Aerial and Game datasets, where FOMO often confused objects with the background, resulting in new erroneous predictions, such as misidentifying rooftops as a single object in the Aerial dataset. However, UMB did not commit the same errors in these cases. In summary, although UMB also exhibited some false detections in certain scenarios, it outperformed FOMO in both detection accuracy and the ability to recall potential objects.\n\nThese visualizations will highlight specific instances where the proposed method did not perform as expected, along with a brief analysis of each case. By doing so, we aim to provide a more comprehensive understanding of the limitations and the contexts in which they occur.\n\n**Mathematical Variable Symbols:** We appreciate the reviewer's attention to the mathematical notation used in our paper. To improve clarity and consistency, we will revise the mathematical symbols in Equations 6 and 7 to align with more commonly accepted and standard notations. This will ensure that the equations are easily understood by a broader audience and adhere to conventional mathematical standards.\n\nThank you once again for your valuable feedback. If you have further suggestions or need more information support, please feel free to contact us. Thank you again for your recognition and valuable feedback on our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SImRzIYXeq", "reviewer": "Reviewer_jKzy", "summary": "This paper aims to understand the model’s behavior in predicting the unknown category.  First, the authors model the text attribute and the positive sample probability, obtaining their empirical probability, which can be seen as the detector’s estimation of the likelihood of the target with certain known attributes being predicted as the foreground. Then, they jointly decide whether the current object should be categorized in the unknown category based on the empirical, the in-distribution, and the out-of-distribution probability. Finally, based on the decision making process, the authors can infer the similarity of an unknown object to known classes and identify the attribute with the most significant impact on the decision-making process.", "questions": "see above", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written well and is easy to understand.\n\n2. The studied problem is very important.\n\n3. The results seem to outperform state-of-the-art.", "weaknesses": "1. Is it able to utilize more effective OOD score for equations 13-14?\n\n2. It might be more comprehensive to report more metrics, such as AUROC and FPR.\n\n3. It might be useful to include ablation results on prompt templates and LLM used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback on our work, particularly regarding the concerns related to the detailed technical design. Below is our detailed response:\n\n**OOD Score**: To detect unknown objects, we proposed a method that models the probability of an object being predicted as foreground. By modeling attribute similarity and the positive sample probability of the object, we generate the empirical probability (Empirical Prob). By combining the Empirical Prob with in-distribution probability (ID Prob), we can ensure the possibility that each sample will be recognized as a foreground object. Based on this, UMB can extract potential objects of interest from the background and label them as unknown. However, there is a challenge in distinguishing between known and unknown categories. The ID Prob is used for linear combination to determine the probability of known categories, while the Empirical Prob is also derived based on the confidence of known categories. As a result, both the ID and Empirical Prob are higher for a known object. This is why we introduce the out-of-distribution probability (OOD Prob). However, in our paper, we implemented a simple OOD Prob measure that assesses the uncertainty of known categories, specifically the probability that it belongs to the background. Admittedly, a more complex and representative probability score could be developed to calculate the current object's background probability. However, this would introduce additional computational overhead and complicate the UMB pipline. Therefore, we opted for the simplest method to measure the background probability, which can be accomplished with two matrix operations, ensuring the algorithm's simplicity and efficiency. These settings allowed UMB to achieve significant performance improvements in detecting both known and unknown categories, with a 5.3 mAP increase, establishing a new SOTA.\n\n**More Metrics**: In Table 1 of the attached PDF, we provide a comparison of additional metrics. We included the early OWOD (Open-World Object Detection) metrics used to evaluate model performance in unknown categories (WI, A-OSE, and Recall). WI assesses the difference in recognition accuracy between known and unknown categories, while A-OSE measures the number of errors where the model mistakenly classifies unknown objects as known categories. Our method still achieved leading performance, such as a consistently lower A-OSE compared to FOMO, while also achieving a higher recall.  The AUROC and FPR you mentioned are more commonly used to evaluate classification tasks. Object detection, however, consists of both classification and localization tasks.  Typically, a detector's post-processing filters out all regions predicted as negative samples, meaning that true negatives (TN) are not present in the prediction results. Therefore, metrics that rely on TN, such as FPR, cannot be computed. Thus, the evaluation of detection tasks generally relies on the results predicted as positive samples by the detector, such as recall and precision. However, both early OWOD evaluation metrics and individual recall or precision metrics have their limitations. Relying solely on recall can lead to a detector greedily predicting all possible regions as positive samples, while relying solely on precision can make the detector overly conservative. The early OWOD metrics emerged because detectors performed poorly on unknown objects, focusing on evaluating specific aspects of performance in unknown categories, such as A-OSE. However, with the introduction of open-vocabulary object detection models, detectors' performance on unknown categories has improved. Therefore, we adopted mAP, the most widely accepted metric for object detection, as it balances precision and recall and evaluates the detector's performance across different thresholds. This is also why mAP is typically used to evaluate detector performance in closed-set object detection. We understand that specific scenarios might require additional metrics, so we provided TP for each dataset, which, combined with other metrics, can be used to calculate other values, such as FP. We will include this table as part of the appendix to provide additional information. Additionally, we will release the training and validation settings and final weights to support the application and development of the OWOD.\n\n**Additional Experiments**: Prompt templates and the LLM used are crucial in our detector. For instance, using more templates to generate visual embeddings and utilizing different LLMs could achieve higher performance; however, this would lead to an unfair comparison with FOMO or OWL-Vit. Therefore, to prevent such disparities, we adopted consistent configurations during the training and inference of UMB to ensure fairness. If you are interested in further experiments, we have provided more detailed information on related experiments in the appendix. \n\nThe above is our response to the concerns you raised regarding the detailed technical design. We hope that these explanations can clearly answer your questions and help you better understand the strategies and decisions we adopted in our research. We again express our gratitude for your valuable feedback. If you have any further suggestions or require additional information, please feel free to contact us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zq9qMRuDAD", "reviewer": "Reviewer_sXkA", "summary": "The paper presents Understanding Model Behavior (UMB), a framework for Open-World Object Detection (OWOD) that not only detects unknown objects but also analyzes the decision-making with text attributes. UMB first prompts large language models to generate text attributes. Then, it models the empirical, in-distribution, and out-of-distribution probabilities to estimate the likelihood of an object being predicted as a foreground. Based on this, UMB can infer the similarity of unknown objects to known classes, thereby identifying the most important attributes. It significantly improves over previous state-of-the-art methods on the Real-World Object Detection (RWD) benchmark, demonstrating its effectiveness in identifying unknown objects.", "questions": "1. The performance comparison in Table 1 lacks some advanced open-world object detectors [1-3]\n\n2. It is suggested that more ablative studies be conducted to justify fine-grained designs, including the linear interpolation in PMM, different texture attribute designs, and shift & scale in Eqn.11. The corresponding experimental analysis should also be given.\n\n3. There are some mismatches between Fig. 2 and the method descriptions. I cannot find Class-agnostic Text Embedding in Sec 4.1, and I am confused about the Fig.3 caption \"To establish To establish.\"\n\n4. The literature review should cover more recently published works on open-set learning for object detection [1-4].\n\n[1] Zohar, O., Wang, K. C., & Yeung, S. (2023). Prob: Probabilistic objectness for open-world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11444-11453).\n\n[2] Wang, Z., Li, Y., Chen, X., Lim, S. N., Torralba, A., Zhao, H., & Wang, S. (2023). Detecting everything in the open world: Towards universal object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11433-11443).\n\n[3] Ma, S., Wang, Y., Wei, Y., Fan, J., Li, T. H., Liu, H., & Lv, F. (2023). Cat: Localization and identification cascade detection transformer for open-world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19681-19690).\n\n[4] Li, W., Guo, X., & Yuan, Y. (2023). Novel Scenes & Classes: Towards Adaptive Open-set Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15780-15790).", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation for modeling fine-grained text attributes in open-world learning is reasonable and interesting.\n2. The performance gains on benchmarks are significant in both known and unknown classes.\n3. The method can predict attributes for unknown classes, benefitting practical usage.", "weaknesses": "1. The compared methods are not convincing, lacking some more advanced open-world object detection works.\n2. The ablative studies are not sufficient.\n3. There are some typos leading to the reading difficulty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable feedback you provided on our work, particularly regarding the comparisons with recent works, the ablation experiments, and the content discrepancies. Below is our detailed response to each of your comments:\n\n**Comparison with Recent Works**:  The references [1]-[4] you mentioned include three distinct areas: open-set object detection [4], open-vocabulary object detection (OVC) [2], and open-world object detection (OWOD) [1, 3].  Open-set object detection (OSOD) requires the detector to be trained on a closed set without any annotations of unknown objects. During testing, the detector is required to detect both known and unknown objects. OVC (Section 2.1) narrows the distance between regions and text during training, leveraging pre-trained models on large-scale datasets and using pre-trained encoders (e.g., CLIP) to enhance generalization. In testing, the matching between customized text and regions determines the class to which a region belongs.  OWOD imposes stricter requirements, involving two components: detecting unknown objects of interest and incremental learning. The former, similar to OSOD, demands the detector to identify unknown objects during inference, while the latter requires the detector to classify some unknown objects as known classes and fine-tune the model to accommodate the new classes. These two processes simulate real-world applications where unknown objects are continuously labeled, and annotators select interesting targets for incremental learning, thereby enhancing the detector's performance. We have placed the problem definitions in Appendix (A.1), and a diagram illustrating the annotator's involvement in the entire process is provided in Figure 1.\n\nAmong these references, those relevant to our work are [1, 3]. However, these methods do not utilize the foundation model, training from randomly initialized weights, resulting in poor performance. For example, in the COCO and VOC mixed M-OWODB, OWL-Vit achieves  79% unknown recall with the assistance of LLM, while PROB only achieves 19.4%. This result was obtained under the zero-shot setting of OWL-Vit, and if OWL-Vit were fine-tuned, the gap would further widen. Therefore, we follow FOMO and convert the evaluation dataset to the RWD benchmark. The RWD is more complex and adheres to the few-shot setting, where using detectors like PROB and CAT, which are trained from random weights, is less appropriate. In fact, when we simply implemented PROB on the RWD, we found it could not even detect unknown classes. Thus, we did not compare PROB and CAT because they rely on large amounts of training data, which is not compatible with the few-shot setting of RWD. We have discussed more relevant and recent methods, such as FOMO, and foundation models with the same supervision in the paper, with technical details provided in the appendix. If you have further questions, please refer to the appendix, where we explain the problem definition, benchmark setup, and additional experiments in more detail. Furthermore, we have cited PROB and CAT in our paper, as seen in references [20] and [5].\n\n**Ablation Studies**: In the ablation study, we provide incremental comparisons. Initially, we used averaged text embeddings and OOD probability as the baseline. We then added ID probability and PMM (Gaussian and Weibull distributions) to demonstrate the contribution of each component. This ablation study structure is consistent with that of PROB and CAT.\n\nWe have included additional comparison experiments involving linear interpolation (LI) and sliding window (SW), as detailed in Table 2 of the attached document. The results indicate that the use of LI or SW individually does not lead to significant performance improvements. Both methods demonstrate only marginal enhancements over the original approach, indicating that neither LI nor SW can accurately model the data distribution when used in isolation. This is further substantiated by the fitting performance illustrated in Figure 2 (the attached document). The combined use of LI and SW yields the most substantial performance gains and achieves a more accurate fit to the original data distribution.\n\nRegarding the different text attribute designs you mentioned, using different LLMs could lead to unfair comparisons. Although we could utilize the latest LLMs like GPT-4o to generate text attributes or even combine multiple LLMs to generate richer text attributes, doing so would create an imbalance when comparing with models like FOMO and OWL-Vit. Therefore, we did not modify other settings to ensure fairness. Additionally, shift & scale is the default setting of the benchmark model. To align the output of OWL-Vit, we adopted the pre-trained shift & scale to rescale the similarity.\n\n**Content Discrepancies**: We are grateful for your observation regarding the mismatches between Figure 2 and the method descriptions, as well as the presence of typos. We sincerely apologize for these errors, which occurred due to inconsistencies between the final version of the manuscript and the initial structure. In the original version, we placed the problem definitions in Appendix A.1 as a separate section following the related work. However, as the detailed method descriptions took up more space than anticipated, we moved this section to the appendix, resulting in misaligned section indices in Figure 2. We have since conducted a thorough review of the entire manuscript, correcting all mismatches and typing errors. Regarding the algorithm's process, we have provided explanations in both Section 3 and Figure 2. Additionally, pseudocode implementation is provided at the beginning of the appendix, which we hope will help clarify any remaining questions.\n\nWe hope our explanations help address your concerns and clarify our research strategies and decisions. Please reach out if you have further suggestions or need more information. Thank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate your valuable feedback, which has been instrumental in guiding us to improve our work. In the attached document, we have provided additional experiments, including examples of detection failures (Figure 1), a comparison of more metrics (Table 1), further ablation studies (Table 2), and the corresponding fitting visualizations (Figure 2). We hope this additional information will help you better understand the content conveyed and the technical design implemented in the paper. Thank you once again for your feedback. If you have any further questions or require additional clarification, please do not hesitate to contact us.

Meta-review: This work attempts to understand the model’s prediction when identifying unlabeled objects and establish connections between unknown and known categories. All reviewers rate positively. During the rebuttal & discussion process, concerns, like evaluation metrics and lack of some ablation studies, have been solved effectively. After careful consideration, The AC is inclined to accept this paper.
Decision: Accept (poster)