Title: UMB: Understanding Model Behavior for Open-World Object Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Computer Vision

Object Detection

Open-World Object Detection
Abstract: Open-World Object Detection (OWOD) is a challenging task that requires the detector to identify unlabeled objects and continuously demands the detector to learn new knowledge based on existing ones. Existing methods primarily focus on recalling unknown objects, neglecting to explore the reasons behind them. This paper aims to understand the model's behavior in predicting the unknown category. First, we model the text attribute and the positive sample probability, obtaining their empirical probability, which can be seen as the detector's estimation of the likelihood of the target with certain known attributes being predicted as the foreground. Then, we jointly decide whether the current object should be categorized in the unknown category based on the empirical, the in-distribution, and the out-of-distribution probability. Finally, based on the decision-making process, we can infer the similarity of an unknown object to known classes and identify the attribute with the most significant impact on the decision-making process. This additional information can help us understand the behavior of the model's prediction in the unknown class. The evaluation results on the Real-World Object Detection (RWD) benchmark, which consists of five real-world application datasets, show that we surpassed the previous state-of-the-art (SOTA) with an absolute gain of 5.3 mAP for unknown classes, reaching 20.5 mAP. Our code is available at https://github.com/xxyzll/UMB.

UMB: Understanding Model Behavior for Open-World Object Detection

Abstract

Open-World Object Detection (OWOD) is a challenging task that requires the detector to identify unlabeled objects and continuously demands the detector to learn new knowledge based on existing ones. Existing methods primarily focus on recalling unknown objects, neglecting to explore the reasons behind them. This paper aims to understand the model's behavior in predicting the unknown category. First, we model the text attribute and the positive sample probability, obtaining their empirical probability, which can be seen as the detector's estimation of the likelihood of the target with certain known attributes being predicted as the foreground. Then, we jointly decide whether the current object should be categorized in the unknown category based on the empirical, the in-distribution, and the out-of-distribution probability. Finally, based on the decision-making process, we can infer the similarity of an unknown object to known classes and identify the attribute with the most significant impact on the decision-making process. This additional information can help us understand the behavior of the model's prediction in the unknown class. The evaluation results on the Real-World Object Detection (RWD) benchmark, which consists of five real-world application datasets, show that we surpassed the previous state-of-the-art (SOTA) with an absolute gain of 5.3 mAP for unknown classes, reaching 20.5 mAP. Our code is available at https://github.com/xxyzll/UMB.

1 Introduction

As a fundamental task in computer vision, object detection has always been the focus of extensive attention[1, 2, 3]. Traditional object detection methods are trained on closed datasets, assuming all detected objects have already been annotated in the training set. However, the real-world environment's complexity means it is impossible to annotate all objects. As a result, the application of traditional detection methods is limited. Open World Object Detection (OWOD) has been introduced to address the issue. OWOD can be divided into two subtasks: mining potential objects and incremental learning. The former requires the model to detect categories in the test set that have not been annotated in the training set. These newly discovered objects are then handed over to annotators, who select the categories of interest. Subsequently, the model is required to fine-tune its existing knowledge to detect these newly added categories (incremental learning).

Existing works primarily focus on generating pseudo-labels for potential objects in the training set, treating these pseudo-labels as annotations for unknown categories. For instance, ORE[4] labels samples with high objectness predicted as background as potential objects. CAT[5] and RE-OWOD[6] utilize selective search to provide annotations for unknown categories. OW-DETR[7] proposes an attention-driven pseudo-label strategy to mine potential positive samples. However, despite these

The object was predicted as an

unknown class due to its similarity

with the fish. The color was the

most influential factor in the

model's decision-making process.

The model has predicted

an unknown object, but

what is the reason behind

this? How is it related to

the known class?

Others:  Merely identifying unknown obje-

cts and providing them to the annotator.

unknown

known

unknown: Fish Color

known

Ours:  Identifying unknown targets and c-

omprehending the behavior of the model.

Annotator

heuristic methods being able to recall potential objects, they share a common flaw. As shown in Figure 1 (left), existing methods can only detect unknown objects and then provide these labels to annotators. However, the reason the model would predict these objects remains unknown to the annotators. Therefore, this paper attempts to understand the model's prediction of potential objects, establish connections between unknown and known categories, and then provide this additional information to annotators.

To achieve this, we propose a novel model (UMB) that uses textual attributes to mine potential unknown objects. Specifically, we first define targets that share similar attributes with known categories but are predicted as background as potential objects. Then, to find these potential objects, we build a distribution that associates attribute similarity with the probability of positive samples, which can be seen as the empirical probability of an object possessing a particular attribute being classified as a positive sample. If a sample predicted as background has a high empirical probability and attribute similarity, we regard it as an unknown object. Finally, based on the decision-making process, we infer the most similar known classes with the unknown object and calculate the most significant impact attribute. As shown in Figure 1 (right), our method can identify the unknown and provide information about their connections with known categories and the influence of attributes on decision-making.

We evaluated our method on the Real-World Object Detection (RWD) benchmark composed of datasets from five practical applications, and our method achieved significant improvements. We improved almost all datasets, surpassing the OVC (GT) that uses real class names. Significantly, in the Surgery[36] dataset, we achieved the 213% performance in unknown category. The main contributions of this paper are as follows:

To the best of our knowledge, we are the first to notice the limitations of models on unknown predictions and attempt to understand the predictive behaviour of models.

To achieve this, we propose a new model framework (UMB) that can detect unknown categories and understand model behavior utilizing the textual description of known categories.

Wemodel the textual attributes and the probability of positive samples to obtain the empirical probability. By combining the empirical probability, the in-distribution probability, and the out-of-distribution probability, we are able to discover unknown categories.

The evaluation results on the RWD benchmark show that our method achieved significant performance improvements, establishing a new state-of-the-art (SOTA) with 5.3 mAP advantages in both known and unknown category performance.

2 Related Works

2.1 Open Vocabulary Object Detection

Open Vocabulary Object Detection (OVD), as a subset of open-world perception, was initially introduced by OVR-CNN[8]. OVD employs the text encoder to transform classes needing detection

into text embeddings, determining the current object's class by calculating the similarity between all text and visual embeddings. Subsequent works further have enhanced OVD's performance, including knowledge transfer from pre-trained models through distillation[9, 10, 11], the addition of highquality object candidates[12, 13], and alignment of text-visual regions[14, 15, 16, 17, 18]. However, in the setting of OWOD, these methods fail to detect the unknown class due to the uncertainty of object categories. Our approach, based on OWL-ViT[19], broadens OVD to OWOD by modelling the correlation between text attributes and the probability of positive samples.

2.2 Open-World Object Detection

Open World Object Detection (OWOD), distinct from OVD, presents stricter settings and is a more challenging task, as proposed by ORE[4]. OWOD requires the detector not only to detect potential unknown objects without any information of unknown classes (including category names) but also to fine-tune the detector on newly introduced classes for continuous learning of new knowledge. Existing research focuses on heuristic assumptions for potential targets. ORE considers background samples with high objectness in RPN as potential unknowns, OW-DETR[7] calculates the average score of feature regions to determine positive samples, and PROB[20] proposes the use of Mahalanobis distance to discover the potential positive samples. Some other methods use additional pseudo-label generation mechanisms to generate annotations for potential objects, including selective search[5, 21, 22], random sample generation[23], and large model knowledge transfer [24, 25, 26]. However, these OWODmethods focus on detecting potential objects and ignore investigation into underlying reasons. Our method attempts to understand the behaviour of the model's unknown prediction, establishing a relationship between unknown objects and known classes.

3 Our Approach

Our method, named UMB, is built upon OWL-ViT[19], with the overall process illustrated in Figure2. First, what characteristics should of an unknown target possess? We posit that if an object is predicted as background but exhibits attributes of known classes, it should be considered an unknown target. Therefore, we model the attributes of known classes and the probability of positive samples to build distribution of the empirical probability (Sec. 3.2). The distribution represents the detector's empirical confidence in predicting objects with known class attributes as positive samples. If a background sample's empirical confidence and similarity to known class attributes (In-Distribution Probability Sec. 3.3) are both high, we consider it a potential object.

Then, since predictions for known and unknown classes are based on text attributes, we can infer the most similar known class based on the attribute similarity of the unknown object (eqn. 16). Finally, we can calculate the contribution of each attribute based on the decision-making process of unknown predictions, thereby identifying the attributes that have the greatest impact on decision-making (eqn. 17). This additional information can aid in understanding the model's behaviour in unknown classes.

3.1 Background

To obtain text that describe objects, we use the following template[28, 29] to request the Large Language Model (LLM) to list all attributes related to known classes:

Where C and Z denote the class name and predefined attribute type (e.g., shape), respectively. These attributes are filled into the prompt template[27]:

where A is the attribute text generated in eqn. 1, e.g., blue. Then, those prompts are fed into the trained text encoder to generate text attribute embeddings E att = [ e att 1 , e att 2 , ..., e att n ] ⊤ ∈ R n × d , where d is the hidden dimension, e.g, 512. n denotes the number of the text attributes. The image is fed into the trained visual encoder to generate visual embeddings E vis = [ e vis 1 , e vis 2 , ..., e vis m ] ⊤ ∈ R m × d , where m represents the number of patch. In order to establish a connection between these classagnostic attributes and known categories, we use additional weights W ∈ R m × n trained in known

Zohar O, Lozano A, Goel S, et al. Open World Object Detection in the Era of Foundation Models[J]. arXiv preprint arXiv:2312.05745, 2023.

（

FOMO

）

Eqn. 15

Class-agnostic Text Embedding

Linear Projection

Trained

Text

Transformer

Encoder

Eatt

Evis

Trained Attribute Weight (W)

set

train

score

Known Categories

fit

Multimodal

Distribution

Probabilistic

Mixture Model

PMM

Attribute Generation Guided

List of cla-

ss names

Text Attribute Modeling

Unknown Inference

Trained

Visual

Transformer

Encoder

Eatt

Evis

(

x1

, y

1

, w

1

, h

1

)

box

1

Class Head

sim

(

Eatt

,

Evis

)

Eqn.12: shift

&

scale

PMM

Eqn. 12

Empirical Prob

ID Prob

Eqn. 13

OOD Prob

Eqn. 14

Unknown

Prob

(

x2

, y

2

, w

2

, h

2

)

box

2

(

, y

3

, w

3

, h

3

)

box

3

(

x4

, y

4

, w

4

, h

4

)

box4

p

(C  |

)

e vis k

u

p

(

)

e vis 1

p

(

)

e vis 2

p

(

)

e vis 3

p

(

)

e vis 4

e att u

Eatt

p

(

)

e vis k

ID: in-distribution

OOD: out-of-distribution

Prob: probability

Trained

Visual

Transformer

Encoder

sim

(

Eatt

,

Evis

)

W

Eqn.

13

Behavior, App-

earance, ... ...

Sec. 3.1

W

categories to linearly combine similarities. Therefore, given a visual embedding e vis i , the probability of its corresponding known category j is:

where Sigmoid is the Sigmoid activation function, and sim denotes the Cosine Similarity. The pseudocode for known class prediction and attribute generation can be found in Algorithm 1.

3.2 Text Attribute Modeling (TAM)

3.2.1 Attribute Modeling

We model the attribute similarities in the training set with category confidence as the positive sample probability to build an empirical probability distribution. However, as shown in eqn. 3, the score is the linear combination of all attribute similarities, so it is influenced by all attributes simultaneously. Thus, we weigh confidence with linear combination weights ( W ∈ R m × n ) to balance the contributions of different attributes. Specifically, given visual embedding e vis k , positive sample probability of attribute i for category j can be represented as:

where β is a hyperparameter used to balance the contributions of weights and scores. For simplicity, we use the geometric weighted average. We incorporate all similarities in the training set and their corresponding probabilities of positive samples, establishing a mapping f i,j : sim ( e att i , e vis k ) → ˜ p ( e att i , C j | e vis k ) . However, during training, the model cannot utilize the annotations of any unknown classes. Therefore, we define the positive sample probability of attribute i for the unknown class as the maximum of its probability to all known classes. Specifically, the probability of attribute i for the unknown class C u can be represented as:

Linear

Interpolation

&

Sliding Window

Original

Distribution

Probabilistic

Mixture Model

0. 12

0. 12

0. 12

0.10

0. 10

0.10

80 °0

0.08

0. 08

0.06

0.06

0. 06

0. 04

0. 04

0. 04

0. 02

0. 02

0. 02

0.00

0.00

0.00

-1.00.80.60.40.20.00.20.40.60.8

1.0

018'0900200070-0-9'0-80-0'1-

0'18'090'02'00070-'0-9'0-80-0'1-

3.2.2 Distribution Optimization and Fitting

To establish a continuous probability distribution, we need to optimize and fit the original distribution. First, contrary to the OWOD benchmark that heavily relies on extensive COCO[30] and VOC[31] data, RWD pays more attention to real-world application and is specifically designed for the few-shot setting. This results in the model not having sufficient samples to establish the probability distribution. Consequently, there are some undefined points in the mapping function f i,j . To mitigate this, we employ the linear interpolation to estimate the values of these missing points x :

where, x l and x r respectively represent the points to the left and right of x that are closest in the mapping f i,j .

Then, we utilize the sliding window to filter the noise present in the distribution. With a predetermined window size, we calculate the maximum positive sample probability across the entire window to substitute the current value:

where W sz denotes the window size. As depicted in Figure 3, the employment of linear interpolation and the sliding window ensures the original shape of the probability distribution remains intact, concurrently minimizing the noise inherent in the distribution.

Finally, as shown in Figure 3 (middle), the optimized probability distribution f i,u demonstrates the multi-peak characteristic. Consequently, we postulate that the original distribution is composed of multiple basic probability distributions (e.g., Gaussian Distribution). As a result, we employ the mixture probability distribution to fit the initial distribution. Specifically, we construct the model using the linear combination of multiple Gaussian distributions:

where A is the number of the Gaussian distribution. Additionally, we observed that certain attributes demonstrate the skewed distribution, suggesting that fitting with the Gaussian model may not be the optimal choice. Consequently, we utilize the asymmetric Weibull distribution as a substitute for the Gaussian distribution:

In order to ascertain the parameters of these distributions, we designate them as learnable parameters and employ Mean Squared Error (MSE) as the loss function for optimization. The pseudocode for text attribute modeling can be found in Algorithm 2.

3.3 Unknown Inference

Following FOMO[27], we calculate the weighted mean of all attribute embeddings as the embedding for the unknown class:

Following this, we utilize the pre-trained scale layer ( R d → R 1 ) and shift layer ( R d → R 1 ) for the purpose of scaling the similarity[19]:

Finally, we adjust the similarity of the average embedding. This adjustment is segmented into three components: empirical probability, in-distribution probability, and out-of-distribution probability.

Empirical Probability (Empirical Prob) . For known categories, each attribute contributes unevenly to the category score (eqn. 6). Hence, for the unknown class, merely using the summation of the empirical probability to ascertain category confidence is suboptimal. We utilize the maximum weight from the known class to balance the contributions from various attributes of the unknown class. Specifically, for the visual embedding e vis k , its corresponding empirical probability is:

Herein, f i,u denotes the positive sample probability of attribute i towards the unknown class, as established earlier.

In-Distribution Probability (ID Prob) . We aspire for the model to observe the known attribute of the current object. Consequently, we incorporate the weighted sum of the scaled attribute similarities:

Out-of-Distribution Probability (OOD Prob) . Both empirical probability and in-distribution probability are based on the model's prediction on known classes. Therefore, inevitably, the model predicts high empirical probabilities and in-distribution probabilities for known categories. To counteract this, we employ out-of-distribution probability to offset their influence:

Ultimately, given the visual embedding e vis k , the corresponding confidence for the unknown class is denoted as:

where α is used to balance the contribution form in-distribution and empirical probability.

3.4 Additional Information

Similarity between known and unknown classes . Predictions for both unknown and known classes are determined by attribute similarity. Hence, we can compute its similarity with known classes based on the visual embedding of objects classified as unknown. Similar to unknown inference, we take into account the empirical probability of the current object and its confidence in being predicted as a known class. Specifically, for the visual embedding e vis k of objects predicted as unknown, the corresponding similarity to known classes is:

Maximal attribute contribution . Attributes are used to compute the similarity with visual embeddings, and then the model makes predictions based on this similarity. Therefore, the contribution of each similarity can be calculated to determine the impact of a particular attribute in the decisionmaking process. For a visual embedding e vis k that is predicted as an unknown class, the influence of attribute i on the current decision is denoted as:

4 Experiments

4.1 More Details and Experiments

In our supplemental material, we provide detailed information about our experiments, including: comprehensive descriptions of the datasets (sec A.2), definition of OWOD (sec A.1), evaluation metrics (sec A.3), details (sec A.4), more extensive ablation studies (sec A.5), analysis and visualization of PMM training (sec A.6), similarity evaluation(sec A.7), attribute study (sec A.8), discussion of the limitations (sec A.10) and broad impact(sec A.9).

4.2 Datasets

The OWOD benchmark is established on the VOC[31] and COCO[30] datasets. In the era of foundation models, the zero-shot capability of detectors on such datasets has reached its limit, for instance, OWL-ViT[19] unknown recall is 79.0. Therefore, following FOMO, we have shifted the benchmark for evaluating detector performance to the more practically applicable RWD benchmark.

4.3 Comparison with Other State-of-the-art Models

Table 1 presents the comparison of our UMB method and previous SOTA methods established on the RWD benchmark. Overall, our method achieved comprehensive leadership, surpassing previous methods with the unknown performance advantage of 4.4 mAP (Wb-B) and 5.3 mAP (Gm-L), demonstrating the effectiveness of our method. In addition, although Base+GT uses the name of unknown categories, it performs poorly in the Aerial, Game, Medical, and Surgery datasets. Our method does not rely on unknown class names and significantly outperforms Base+GT (e.g., Surgery: 2.4 (Base+GT-L) vs 25.6 (UMB-Wb-L)). When compared with Base-FS, which received the same

supervision, FOMO did not achieve comprehensive leadership and even lagged behind by 3.6 mAP in Aquatic. Our method leads whether compared with Base-FS or FOMO, and in the Surgery dataset, we doubled the performance of FOMO (12.0 (FOMO-L) vs 25.6 (UMB-Wb-L)). Gm and Wb exhibit different strengths in various OWL-ViT models. UMB-Wb shows an advantage on the B/16 model (+1.1 mAP), while the trend is reversed on the L/14 base (-1.2 mAP). Therefore, we provide two different types of probability distributions (Gm and Wb) as interchangeable options.

4.4 Ablation Study

Table 2 provides the incremental results of our UMB. The initial performance uses the average category embedding (eqn. 10) and out-of-distribution probability (eqn. 14). When the in-distribution probability is introduced, which is used to capture the known attributes of the current target, the performance improves by 5.8 mAP. However, in the Aerial dataset, U AP only reaches 3.8 mAP, reducing by 1 mAP, replaced by a significant increase in recall rate (+44.1), which means that the detector erroneously treats many background samples as unknown objects. Such a result also proves the limitations of the unknown recall as the detection metric previously used in the OWOD benchmark. Finally, by adding the empirical distribution, our UMB achieves a comprehensive lead (Gm +5.0, Wb +3.8). In addition, the effect of balance parameter α on the recall rate is not obvious. In fact, in the Aquatic and Surgery datasets, the variance of the recall rate distribution reaches 0, which means that alpha correctly suppresses the background samples erroneously predicted by the detector. Overall, our UMB can provide a higher unknown recall rate (UMB-Gm 73.4, UMB-Wb 73.8) while ensuring detection accuracy.

4.5 Visualization

In figure 4, we provide the qualitative analysis of FOMO and our UMB. The visual analysis is divided into three parts: the recall ability of the unknown category, recall precision, and analysis of additional information. UMB shows superior performance in recalling unknown objects. UMB successfully recalled tools in the Surgery dataset (fifth row, Wb ID 3) and accurately recalled playgrounds and roofs in the Aerial dataset (second row, Wb ID 1, 2), while FOMO failed to recall these objects. Regarding recall precision, FOMO predicts multiple results to an object and incorrectly classifies unknown classified objects as known classes, such as the hero characters in the Game dataset (third row) and the misclassification of four objects in the Aquatic dataset (first row). In contrast, our UMB shows higher precision. Regarding additional information, the misclassification of FOMO in the Aquatic dataset reflects the defects of OWL-ViT in classifying these objects. With the help of formula n, UMB can infer the category most similar to the current object, and these categories correspond to the categories misclassified by FOMO. In addition, for the same object (ID 3,4), UMB identifies that the attribute with the greatest impact on the entire decision is consistent. These results prove the accuracy of our method in inferring the connection between unknown and known and discovering the attributes that have the greatest impact on decision-making.

5 Conclusion

This paper attempts to understand the detector's behaviour in predicting unknown objects. To achieve this, we propose a novel detection framework, UMB, which employs class-agnostic textual attributes to unearth potential objects in the background. Given that the model's detection process for known

and unknown classes hinges on textual attributes, our UMB can use the textual attributes of unknown objects to infer the most similar known category. In addition, we can calculate the attributes that have the most significant impact on the entire decision-making process. This supplementary information aids annotators in understanding the model's behaviour in predicting unknowns. We hope that UMB can promote the application of Open-World Object Detection in real-world scenarios. 1: 88.3%

1: 88.3%

ID

Category

Attribute Text

1

Shark

(30.6 %)

Texture is feathers.

(16.3 %)

2

Shark

(35.1 %)

Appearance is suckers.

(17.1 %)

3

Penguin

(34.3 %)

Context is sand-colored.

(16.7 %)

4

Penguin

(36.1 %)

Context is sand-colored.

(17.5 %)

2: 87.7%

3: 71.5%

4: 67.8%

ID

Category

Attribute Text

1

Shark

(30.5 %)

Texture is feathers.

(18.2 %)

2

Shark

(35.0 %)

Appearance is suckers.

(20.5 %)

3

Penguin

(34.2 %)

Context is sand-colored.

(19.5 %)

4

Penguin

(36.1 %)

Context is sand-colored.

(20.4 %)

1: 88.3%

2: 88.0%

3: 72.0%

4: 68.4%

ID

Category

Attribute Text

1

Chimney

(14.0 %)

Shape is round.

(23.6 %)

2

Chimney

(14.6 %)

Shape is round.

(22.7 %)

3

Expressway-

toll-station

(36.1 %)

Context is concessio

n stands.

(23.0 %)

1: 57.9%

2: 56.8%

3: 54.8%

1: 56.7%

ID

Category

Attribute Text

1

Chimney

(14.0 %)

Shape is round.

(23.5 %)

2

Expressway-

toll-station

(12.2 %)

Shape is round.

(22.9 %)

3

chimney

(36.1 %)

Shape is round.

(23.2 %)

2: 55.2%

3: 55.1%

ID

Category

Attribute Text

1

Jhin

(4.4 %)

object which (is/has/

etc) Features is horns.

(14.8 %)

2

(4.2 %)

object which (is/has/

etc) Shape is cuboid.

(14.8 %)

3

Jhin

(4.6 %)

object which (is/has/

etc) Texture is fine.

(15.1 %)

UMB-Wb

1: 82.3%

2: 81.6%

3: 80.3%

UMB-Gm

ID

Category

Attribute Text

1

Jhin

(4.5 %)

object which (is/has/

etc) Features is horns.

(17.1 %)

2

(4.3 %)

object which (is/has/

etc) Shape is cuboid.

(16.9 %)

3

Jhin

(4.6 %)

object which (is/has/

etc) Texture is fine.

(17.5 %)

1: 82.5%

2: 81.7%

1: 82.5%

FOMO

59.1%

55.7%

55.2%

ID

Category

Attribute Text

1

Soft Tissue

Calcination

(19.1 %)

Material is cortical

shell.

(67.8 %)

2

Distal

Phalanges

(12.2 %)

Features is

triquetrum.

(69.5 %)

3

Third Meta-

carpal Bone

(36.1 %)

Context is torso.

(68.9 %)

1: 92.8%

2: 92.7%

3: 92.5%

ID

Category

Attribute Text

1

Soft Tissue

Calcination

(19.1 %)

Material is cortical

shell.

(61.0 %)

2

Distal

Phalanges

(20.1 %)

Features is

triquetrum.

(62.2 %)

3

Distal Phal-

anges

(17.7 %)

Context is torso.

(57.4 %)

1: 92.6%

2: 92.5%

3: 92.3%

78.2%

77.3%

75.8%

ID

Category

Attribute Text

1

Scalpel

(20.0 %)

Behavior is

repositi-oning.

(6.8 %)

2

Scalpel

(20.0 %)

Shape is thin.

(6.8 %)

3

Hook

(22.0 %)

Shape is thin.

(6.9 %)

1:84.5%

ID

Category

Attribute Text

1

Scalpel

(20.0 %)

Behavior is

operating.

(9.0 %)

2

Scalpel

(19.8  %)

Shape is spherical.

(9.0 %)

3

Hook

(21.8 %)

Shape is spherical.

(9.0 %)

2:84.5%

3:69.5%

1:85.0%

68.8%

61.4%

77.5%(Shark)

67.1%

(Penguin)

65.7%(Shark)

61.9%

(Penguin)

3: 80.5%

2:84.9%

3:69.4%

57.6%

53.5%

Aquatic

Aerial

Medical

Surgery

1/3

Acknowledgments

This work was supported in part by TCL Science and Technology Innovation Fund (Project No. 20231752).

References

Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: A simple and strong anchor-free object detector. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(4):1922-1933, 2020.

Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao. Yolov9: Learning what you want to learn using programmable gradient information. arXiv preprint arXiv:2402.13616 , 2024.

Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, and Yi Liu. Detrs beat yolos on real-time object detection. arXiv preprint arXiv:2304.08069 , 2023.

KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5830-5840, 2021.

Shuailei Ma, Yuefeng Wang, Ying Wei, Jiaqi Fan, Thomas H Li, Hongli Liu, and Fanbing Lv. Cat: Localization and identification cascade detection transformer for open-world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19681-19690, 2023.

Xiaowei Zhao, Yuqing Ma, Duorui Wang, Yifan Shen, Yixuan Qiao, and Xianglong Liu. Revisiting open world object detection. IEEE Transactions on Circuits and Systems for Video Technology , 2023.

Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: Open-world detection transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9235-9244, 2022.

Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14393-14402, 2021.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 , 2021.

Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11186-11196, 2023.

Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14074-14083, 2022.

Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15254-15264, 2023.

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference on Computer Vision , pages 106-122. Springer, 2022.

Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14084-14093, 2022.

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16793-16803, 2022.

Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23497-23506, 2023.

Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for openvocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11144-11154, 2023.

Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7031-7040, 2023.

Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision , pages 728-755. Springer, 2022.

Orr Zohar, Kuan-Chieh Wang, and Serena Yeung. Prob: Probabilistic objectness for open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11444-11453, 2023.

Zhiheng Wu, Yue Lu, Xingyu Chen, Zhengxing Wu, Liwen Kang, and Junzhi Yu. Uc-owod: Unknown-classified open world object detection. In European Conference on Computer Vision , pages 193-210. Springer, 2022.

Xiaowei Zhao, Yuqing Ma, Duorui Wang, Yifan Shen, Yixuan Qiao, and Xianglong Liu. Revisiting open world object detection. IEEE Transactions on Circuits and Systems for Video Technology , 2023.

Yanghao Wang, Zhongqi Yue, Xian-Sheng Hua, and Hanwang Zhang. Random boxes are open-world object detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6233-6243, 2023.

Yulin He, Wei Chen, Yusong Tan, and Siqi Wang. Usd: Unknown sensitive detector empowered by decoupled objectness and segment anything model. arXiv preprint arXiv:2306.02275 , 2023.

Ruohuan Fang, Guansong Pang, Lei Zhou, Xiao Bai, and Jin Zheng. Unsupervised recognition of unknown objects for open-world object detection. arXiv preprint arXiv:2308.16527 , 2023.

Shuailei Ma, Yuefeng Wang, Ying Wei, Peihao Chen, Zhixiang Ye, Jiaqi Fan, Enming Zhang, and Thomas H Li. Detecting the open-world objects with the help of the brain. arXiv preprint arXiv:2303.11623 , 2023.

Orr Zohar, Alejandro Lozano, Shelly Goel, Serena Yeung, and Kuan-Chieh Wang. Open world object detection in the era of foundation models. arXiv preprint arXiv:2312.05745 , 2023.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 , 2022.

Orr Zohar, Shih-Cheng Huang, Kuan-Chieh Wang, and Serena Yeung. Lovm: Language-only vision model selection. Advances in Neural Information Processing Systems , 36, 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision , 111:98-136, 2015.

Roboflow 100. aquarium dataset. https://universe.roboflow.com/roboflow-100/ aquarium-qlnqy , may 2023. visited on 2024-04-09.

Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS journal of photogrammetry and remote sensing , 159:296-307, 2020.

Roboflow 100. team fight tactics dataset. https://universe.roboflow.com/ roboflow-100/team-fight-tactics , may 2023. visited on 2024-04-09.

Roboflow 100. x-ray rheumatology dataset. https://universe.roboflow.com/ roboflow-100/x-ray-rheumatology , may 2023. visited on 2024-04-09.

David Bouget, Rodrigo Benenson, Mohamed Omran, Laurent Riffaud, Bernt Schiele, and Pierre Jannin. Detecting surgical tools by modelling local appearance and global shape. IEEE transactions on medical imaging , 34(12):2603-2617, 2015.

Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8430-8439, 2019.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123:32-73, 2017.

A Appendix / supplemental material

In this section, we supplement the details omitted in the main text.

A.1 Task Formulation

In the context of OWOD, the detection task is divided into a series of subtasks T = { T 1 , T 2 , ..., T | T | } and their corresponding categories K = { K 1 , K 2 , ..., K | T | } . T i includes all known categories from previous tasks and introduces new categories on this basis: K i = ( ⋃ i -1 j =1 K j ) ∪ K new , where K new denotes introduced new categories. When the model is trained on T i , our expectation is that the model should be able to detect all categories it has encountered so far (i.e., K i ), as well as discover those unlabelled but interesting categories. For the purpose of evaluation, the interest object is defined as those that belong to K but not to K i (i.e., K -K i ).

A.2 Datasets

The OWODbenchmark is a combination of COCO[30] and VOC[31] datasets. In the era of foundation models, the zero-shot capabilities of detectors on OWOD benchmark have even reached their limits. Therefore, consistent with FOMO[27], we will switch the evaluation benchmark to RWD. The RWD benchmark consists of five typical application scenarios for object detection, including underwater scenes, representing visual blurring caused by the environment (Aquatic[32]); aerial scenes, where the targets are small and difficult to distinguish (Aerial[33]); scenarios using synthetic data when data is lacking (Game[34]); medical X-ray scenes, where it is difficult to distinguish between categories and professional knowledge is required(Medical[35]); and human surgery scenes, where the field of view is blurred by blood (Surgery[36]). The detailed division of the RWD benchmark is shown in Table 3. We divide RWD into two subtasks according to a 50% category ratio. When training in Task 1, all categories in the test set that belong to Task 2 are treated as unknown classes, and when training in Task 2, the categories of Task 1 are considered as previously seen classes.

A.3 Metric

For known categories, we adopt the widely used mean Average Precision (mAP) as the evaluation metric for object detection. For unknown classes, previous OWOD methods[5, 4] used the recall rate of unknown classes as the evaluation metric. However, such a metric leads to models greedily treating all background objects as potential samples. Therefore, we adopt mAP, consistent with the evaluation metric for known classes, which simultaneously assesses the detector's recall ability for unknown classes and the precision of the predictions.

A.4 Details

All experiments were conducted using a single NVIDIA GeForce RTX 4090 GPU. Following FOMO, we initialized with the frozen OWL-ViT[19] (L/14 and B/16), which was trained on a mixed dataset composed of Object 365[37] and Visual Genome[38], demonstrating strong generalization capabilities. The large language model used for attribute generation is GPT-3.5. These attributes were matched with all predictions in the dataset, and the corresponding visual embeddings were collected if the IOU exceeded the threshold (0.8). The average of these visual embeddings was calculated to obtain the average embedding of the attributes. Following FOMO, we adopted attribute selection, attribute adaptation, and attribute refinement to train the linear combination weight.

All optimizers used AdamW. During the attribute selection phase, BCE was the loss function, and the learning rate remained constant without decreasing with iterations. The attribute selection phase reduced the number of attributes. Based on the ranking of weights after training, only the top 25 attributes per attribute type were retained. Attribute adaptation was used to narrow the distance between the text attributes and the average embedding of the dataset. This phase used MSE as the loss function, with a maximum of 1000 iterations. Attribute refinement took the text embedding as the parameter to be optimized, with BCE as the loss function. Attribute refinement narrowed the distance between the text embedding and the visual embedding. During the attribute selection and attribute refinement phases, the learning rate and maximum number of iterations for training were set to three values ([1e-5, 5e-5, 1e-4], [1, 10, 100]), iterating over these settings during each training to select the optimal setting.

Upon completion of training, the detector could detect known classes. To detect unknowns, we established the empirical probability for each attribute. In the distribution optimization phase, we set the window value to 10. In the distribution fitting phase, we used two different probability models (Gaussian and Weibull). During training, Adam was used as the optimizer, the learning rate was set to 0.01, the maximum number of iterations was 10000, and the maximum number of probability models was set to 5. Since the purpose of fitting was to capture the shape of the empirical probability distribution and establish a continuous probability distribution, we set an interval of 0.0001 between -1 and 1. This setting was used to sparsify the data of the original distribution. Then, distribution optimization and fitting were performed in the sparsified distribution. Zohar O, Lozano A, Goel S, et al. Open World Object Detection in the Era of Foundation Models[J]. arXiv preprint arXiv:2312.05745, 2023. （ FOMO ） Gm-Known Wb ( β = 0.1): Known Unknown Gm ( β = 0.2): Known Unknown Aerial

A.5 Comprehensive Ablation Experiments

Wb (

β

= 0.1)

β

= 0.2)

Game

Wb (

β

= 0.1)

β

= 0.3)

Medical

Surgery

Wb (

β

= 0.1)

β

= 0.2)

Aqutaic

Wb (

β

= 0.1):

Known

Unknown

β

= 0.4):

Known

Unknown

Aerial

Wb (

β

= 0.8)

β

= 0.9)

Mean

Abnormal Value

Mean +/- 1.5 SE

35.1

50.7

50.8

42.7

42.8

32.7

-11.1

18.6

18.6

11.1

0.0

0.3

0.6

0.9

0. 0

0.3

0.6

0.9

0. 0

0.3

0.6

0. 9

10093.393.3

96.396.3

22.2

36.6

90.890.8

22.3

36.6

13.2

50

42.842.2

25.6

8.6

24.5

0

gmwb

gmwb

gmwb

gmmwb

gm

wb

Aquatic

Aerial

Game

Medical

Surgery

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

Figure 5 illustrates the model's performance under different α settings with the optimal β . In addition, subfigure 6 provides the statistics of the recall rates corresponding to subfigures 1 through 5. In the Aquatic and Game datasets, the performance of UMB is not sensitive to the changes in α , showing minor performance differences. This is because Aquatic is consistent with the training data of OWL-ViT, and Game is synthetic data, neither of which poses additional challenges (such as target size). However, in the remaining datasets, the performance of UMB shows significant changes. For example, the performance of the Aerial dataset drops from 11.1 mAP (wb) to 3.6 mAP. These datasets have significant differences from the OWL-ViT training dataset, and their environmental characteristics (such as small objects in Aerial, similar objects in Medical, and blood-contaminated backgrounds in Surgery) pose additional challenges to the model. These constraints make UMB sensitive to changes in α . Nevertheless, a reasonable α can balance in-distribution and empirical probability contributions to achieve better detection performance.

For the recall rate of unknown categories, except for the Aerial and Medical datasets, UMB maintains a high value (93.3% in Aquatic) and remains stable in the remaining datasets. This indicates that our method does not predict more objects under different α settings but predicts potential objects more accurately. In the Aerial and Medical datasets, the model's recall rate is almost half that of other datasets, and these recall rates show significant fluctuations with the change of α . Therefore, we infer that when the model's recall rate for a specific dataset is high, the balance between intra-division

Aquatic

Aquatic

Wb

Gm

Aerial

Aerial

Wb

Gm

Game

Game

Wb

Gm

Wb

Gm

Medical

Medical

Gm

Surgery

Wb

Surgery

13.0

10

10.4

7.8

5.2

2.6

0.0

0.207 0.276 0.345 0.414 0.483 0.552 0.621 0.690

-0.24

0.00

0.24

0.48

0.72

0.010

0.020

0.030

0.040

-0.26

0.00

0.26

0.52

0.78

28

21

40

30

20

10

7.3

14.6

21.9

29.2

-0.2

0.0

0.2

0.4

0.6

0.8

0

10

20

30

0

-0.2

0.0

0.2

0.4

0.6

8'0

32

0

24

30

16

20

10

0.015

0.030

0.045

970-

0.00

0.26

0.52

0.78

0.075

0.150

0.225

0.300

-0.26

0.00

0.26

0.52

0.78

25

20

20

1s

15

10

10

0. 02

0. 04

0. 06

0. 08

-0.28

0.00

0.28

0.56

80

0.088

0.176

970

0.352

0+0

-0.28

0.00

0.28

0.56

80

25

30

20

24

15

18

10

12

0.018

0.036

0.054

0.072

0.090

-0.28

0.00

0.28

0.56

8°0

0.039

0.052

-0.28

0.00

0.28

0.56

0.84

0.013

0.026

probability and empirical probability has a relatively small impact on the recall rate; when the recall rate is low, this balance has a more significant impact on the recall rate.

A.6 Probabilistic Mixture Model

Figure 6 presents the fitting results of the probabilistic mixture model. When the original probability distribution exhibits multiple peaks, the MSE loss of the probability fitting may stabilize at a relatively high value. For instance, in the first column of the second row, the original probability distribution is composed of more than three elemental probability distributions. Upon completion of training, the MSE value stabilizes around 7.3, indicating that the model faces challenges when fitting distributions

with multiple peaks. However, when the original probability distribution exhibits a single peak, the MSE loss approaches zero, demonstrating the model's advantage in fitting unimodal distributions. For example, in the Game dataset, the MSE value of the Gaussian model stabilizes around 0.015. Despite the possibility of multiple peaks in the original distribution, both the Weibull and Gaussian distribution can capture its essential shape characteristics. For instance, in the Aerial dataset, the Gaussian model can fit two smooth peaks to represent the original distribution. Similarly, the Weibull model can fit a smooth curve when facing distributions with multiple peak features. However, in the case of unimodal distribution, both the Gaussian and Weibull models can capture the original distribution's unimodal feature and fit a smooth curve with a single peak. For example, in the Surgery dataset, the Weibull and Gaussian models in the Game datasets demonstrate this ability. Zohar O, Lozano A, Goel S, et al. Open World Object Detection in the Era of Foundation Models[J]. arXiv preprint arXiv:2312.05745, 2023. （ FOMO ）

A.7 Similarity Evaluation

In order to understand the behavior of the model, we provide the known class most similar to the unknown object and the attribute with the most significant impact on decisionmaking. Through visualization, we observe that many known predictions overlap with unknown objects in the detector's predictions. These known predictions represent that the detector considers the current object to belong to a known category. Therefore, we estimate the model's accuracy in similar inference by

Game

17979

17754

]18949

Wb_matched

Wb_Repeat

Gm_rate

Wb_rate

Medical

1635

18971

Gm_Repeat

Gm_matched

0.9

0.970.96

0.950.94

0.900.88

1971

1495

1970

0.76

Game

6296

16158

]7299

0.6

7300

AquaticAerial

61580

59477

67107

69419

0.3

8512

8797

8421

8781

0

6300

12600

18900

63000

69300

75600

0.0

AquaticAerial

Game Medical

1Game

Overall

whether the inferred known class is consistent with these overlapping known predictions. Specifically, if their IoU exceeds 0.95, we consider these two predictions overlapping. Then, if the known class inferred by the detector is consistent with the known prediction, we consider the model inference correct. Otherwise, it indicates an inference error. As shown in Figure 7 (left), the detector has many duplicate predictions for unknown and known classes. Among these duplicate predictions, the proportion of correctly inferred predictions is large, more than 80% (Figure 7). Compared with Wb, Gm's accuracy rate is always lower. In the Aquatic dataset, Gm lags behind Wb by 0.1 percentage points in accuracy, and in Medical, it lags by 0.6 percentage points. Overall, both Gm and Wb show high inference accuracy (about 90%), and compared with Gm, Wb shows a higher accuracy rate (about 2%).

A.8 Attribute Study

Figure 8 and Figure 9 present the results of attribute analysis. For clarity, we have selected only the top three categories with the highest prediction counts from the detector and the top five attributes that have the most significant impact on them. In each row, the top part shows the prediction counts for these categories. For instance, Puffin (252) in Aquatic indicates that the detector gave 252 predictions for the Puffin category. The left side shows the number of times these attributes have been identified as having the most significant impact. Moreover, the right side represents the attributes with the highest average inference scores from the model.

UMBdemonstrates a strong ability to capture object attributes. In the Surgery dataset, the attribute (Behavior is repositioning) dominated UMB-Wb's 127 predictions (total 274) for the Hook category, and this number rose to 134 (total 273) in UMB-Gm, accounting for nearly half. This implies that when an object exhibits such an attribute, the detector will likely predict it as the Hook category. UMB exhibits precise attribute discrimination capabilities. Since the right side of each row presents the average score of the attribute's impact on all detector predictions, their differences are insignificant. Nevertheless, in the Aquatic dataset, UMB-Gm still distinguished these attributes. For instance, the

Aquatic: Puffin(252), Stingray(192), Starfish(134)

Texture is feathers

Texture is feathers

Size is asymmetric

Context is seagulls

Context is tropical

Context is deep sea

Material is collagen

Context is sand-colored

Shape is flexible

Environment is bird

Appearance is

presence of benches

Context is yachts

Shape is paper

Appearance is pillars

Shape is round

Appearance is

presence of benches

Behavior is

changing direction

Shape is chunky

Shape is paper

Appearance is pillars

Aerial: Baseballfield(1020),  Chimney(635), Tenniscourt(606)

Shape is smooth

Context is elbows

Shape is bent

Context is connective

tissue

Shape is cuboid

Context is

connective tissue

Context is

wrist bones

Features is horns

Material is endosteum

Texture is coarse

Game: Taric(81),  Urgot(74), Samira(71)

Shape is rounded

Shape is spherical

Behavior is operating

Context is

cautery device

Environment is medical

simulation center

Behavior is operating

Context iswrist bones

Features is horns

Material is endosteum

Texture is coarse

Surgery: Hook(274),  Scalpel(210), PliersDown(204)

Features is

cortical bone

Features is swollen

joints

Texture is fissured

Material is

cortical shell

Appearance is

visible joints

Features is paws

Material is brittle

Features is proximity

Appearance is visible

joints between bones

Features is scaphoid

Medical: Fourth metacarpal bone(183),  First metacarpal bone(161), Ulna(16)

54

45

0

0

43

54

0

0

32

16

31

9

22

34

27

4

11

20

46

30

0

0.20

0.17

0.18

0.19

0.16

0.00

0.20

0.17

0.12

0.00

0.18

0.00

0.08

0.00

0.20

0.17

0.04

0.19

0.20

0.00

0.00

311

8

8

0

249

33

52

20

187

49

239

145

124

311

83

113

62

298

83

273

0

21

5

8

0

17

6

1

13

13

12

9

6

8

18

21

0

4

13

6

6

0

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.19

0.00

0.00

0.00

0.15

0.00

0.00

0.00

0.11

0.00

0.15

0.18

0.07

0.00

0.18

0.00

0.04

0.18

0.18

0.18

0.00

134

7

22

12

108

2

22

15

81

134

4

23

55

29

105

67

28

84

33

61

2

48

1

29

0

38

0

28

6

29

43

0

0

19

48

0

0

10

5

41

4

0

0.09

0.09

0.09

0.09

0.07

0.09

0.09

0.09

0.05

0.00

0.09

0.09

0.04

0.09

0.09

0.09

0.02

0.09

0.09

0.09

0.00

0.60

0.59

0.00

0.00

0.48

0.60

0.00

0.59

0.36

0.60

0.00

0.00

0.24

0.00

0.60

0.00

0.12

0.60

0.00

0.00

0.00

Texture is feathers

Size is asymmetric

Context is seagulls

Features is nostril

Context is tropical

Shape is flexible

Appearance is suckers

Shape is swimming

Size is large

Material is collagen

Aquatic: Puffin(252), Stingray(192), Starfish(134)

Aerial: Baseballfield(1043),  Chimney(657), Tenniscourt(642)

Context is yachts

Appearance is

presence of benches

Behavior is

changing direction

Appearance is pillars

Shape is round

Appearance is

presence of benches

Shape is paper

Shape is sail

Behavior is changing

direction

Context is yachts

Game: Taric(81),  Urgot(74), Samira(71)

Context is elbows

Context is

connective tissue

Shape is smooth

Shape is bent

Shape is irregular

Context is wrist bones

Context is

connective tissue

Context is

metacarpal bones

Shape is spiky

Shape is cuboid

Surgery: Hook(274),  Scalpel(210), PliersDown(203)

Behavior is

repositioning

Context is

cautery device

Shape is thin

Shape is

rounded edges

Features is teeth

Shape is thin

Behavior is

repositioning

Shape is

rounded edges

Context is

cautery device

Features is teeth

Medical: Fourth metacarpal bone(183),  First metacarpal bone(163), Ulna(16)

Features is

cortical bone

Features is

swollen joints

Material is

cortical shell

Texture is fissured

Appearance is visible

joints between bones

Appearance is

transparency

Material is brittle

Appearance is visible

joints between bones

Features is scaphoid

Features is triquetrum

51

34

0

0

41

20

19

3

31

45

0

1

20

15

30

5

10

15

51

40

0

274

37

77

44

223

197

18

71

172

108

185

42

120

225

60

159

69

274

47

235

18

0.16

0.00

0.15

0.13

0.13

0.14

0.00

0.00

0.10

0.14

0.00

0.00

0.06

0.15

0.14

0.15

0.03

0.00

0.00

0.16

0.00

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.23

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.23

23

8

0

7

18

6

3

14

14

12

6

6

9

12

9

6

5

20

23

0

0

0.16

0.15

0.14

0.00

0.13

0.16

0.15

0.16

0.10

0.16

0.00

0.16

0.06

0.16

0.16

0.16

0.03

0.00

0.16

0.00

0.00

127

0

0

1

102

37

24

19

76

20

71

48

51

90

13

49

25

127

102

86

0

0.07

0.00

0.00

0.07

0.06

0.07

0.07

0.07

0.04

0.07

0.07

0.07

0.03

0.07

0.07

0.07

0.01

0.07

0.07

0.07

0.00

50

28

0

0

40

1

28

0

30

0

30

4

20

50

0

0

10

7

39

5

0

0.67

0.67

0.00

0.00

0.54

0.67

0.00

0.00

0.40

0.67

0.00

0.00

0.27

0.00

0.67

0.00

0.13

0.67

0.00

0.00

0.00

attribute (Context is the deep sea) had an average score of 0.2 in Puff but dropped to 0 in Starfish. A similar scenario occurred in UMB-Wb, where the attribute (Shape is flexible) had an impact of 0.16 on Starfish, but it dropped to 0 for Puffin. Overall, both UMB-Gm and UMB-Wb can capture and distinguish object attributes and make corresponding predictions based on these attributes.

We conducted additional comparative experiments on Linear Interpolation (LI) and Sliding Window (SW), as detailed in Table 4 of the attached document. The results indicate that employing LI or SW independently does not lead to significant performance improvements. Both methods exhibit only marginal enhancements compared to the original approach, suggesting that LI and SW, when used in isolation, are insufficient for accurately modeling the data distribution.

A.9 Broad Impact

In this paper, we focus on the performance of detectors in open-world object detection and attempt to under- stand the model's behavior when predicting unknown categories. Our approach can help annotators gain a deep understanding of the model's decision-making process, thereby guiding subsequent optimization work and improving the overall performance of the detector. At the same time, understanding the model's behavior may expose potential flaws malicious actors could exploit for illegal activities. For this reason, we choose to open-source our code, both to promote the development of the current field and to identify and prevent these potential issues through the power of the community.

A.10 Limitations

Our focus is on understanding the behaviour of model predictions. Hence, we attempt to migrate the OVC detector to the OWOD task. As our method does not directly train the weights of the OVC detector but merely processes its output, the performance ceiling of our method will be constrained by the inherent performance of the OVC detector itself. Furthermore, due to the visual-text alignment relationships of the OVC requiring extensive data training, fine-tuning on actual application datasets could lead to additional annotation costs.

A.11 Failure Cases

We present typical cases of detection failures from each dataset, focusing on recall capability and detection accuracy. For instance, in the Aquatic dataset, UMB failed to detect small orange fish, while in the Aerial dataset, it did not successfully recall vehicles. These instances reveal the detector's shortcomings in recall capability. Moreover, in the Game and Surgery datasets, UMB displayed occurrences of repeated predictions. Nevertheless, UMB still outperformed FOMO in overall performance. Specifically, in the Aquatic dataset, UMB accurately located the contours of the fish, whereas FOMO showed deviations in contour localization and even missed similar objects. Furthermore, FOMO incorrectly identified the reflection of the photographer's shoes in the glass as an unknown object, demonstrating lower precision. Similar issues were observed in the Aerial and Game datasets, where FOMO often confused objects with the background, resulting in new erroneous predictions, such as misidentifying rooftops as a single object in the Aerial dataset. However, UMB did not commit the same errors in these cases. In summary, although UMB also exhibited some false detections in certain scenarios, it outperformed FOMO in both detection accuracy and the ability to recall potential objects.

FOMO

UMB-Wb

UMB-Gm

M1/3

M1/3

M1/3

BICHL

BICHL

BICHLA

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the introduction, we discuss the problems that existing research focuses on (first paragraph of the introduction), as well as the emphasis and shortcomings of current studies (second paragraph of the introduction). We also detail our methods and the results we have achieved (third and fourth paragraphs of the introduction).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In section A.10, we provide the limitation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In section 3.2, we provide the full set of assumption. In section 4.4, we discuss the impact of empirical probability on model performance. Additionally, in section A.6, we present the detailed analysis and visualization.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the supplemental material, we provide detailed experimental configuration (A.4) and pseudocode (1, 2), environments, and all other settings. Furthermore, we will release all the training code and weights.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is available at https://anonymous.4open.science/r/UMB-B61C/.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In the supplemental material, we provide all the training and test details, including: experimental configuration (A.4), data split (A.2) and hyperparameters (A.5).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the experiments that support the main claims of the paper (sec 4.3) and detailed ablation study (sec A.5).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the supplemental material (sec A.4), we provide the experiment compute resource consumption.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We affirm that all research presented in this paper adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In section A.9, we provide the societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets in the paper have been properly credited, and we adhere to the corresponding licenses and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is available at https://github.com/xxyzll/UMB.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Understanding Model Behavior (UMB), a framework for Open-World Object Detection (OWOD) that not only detects unknown objects but also analyzes the decision-making with text attributes. UMB first prompts large language models to generate text attributes. Then, it models the empirical, in-distribution, and out-of-distribution probabilities to estimate the likelihood of an object being predicted as a foreground. Based on this, UMB can infer the similarity of unknown objects to known classes, thereby identifying the most important attributes. It significantly improves over previous state-of-the-art methods on the Real-World Object Detection (RWD) benchmark, demonstrating its effectiveness in identifying unknown objects.\n\nSTRENGTHS:\n1. The motivation for modeling fine-grained text attributes in open-world learning is reasonable and interesting.\n2. The performance gains on benchmarks are significant in both known and unknown classes.\n3. The method can predict attributes for unknown classes, benefitting practical usage.\n\nWEAKNESSES:\n1. The compared methods are not convincing, lacking some more advanced open-world object detection works.\n2. The ablative studies are not sufficient.\n3. There are some typos leading to the reading difficulty.\n\nQUESTIONS:\n1. The performance comparison in Table 1 lacks some advanced open-world object detectors [1-3]\n\n2. It is suggested that more ablative studies be conducted to justify fine-grained designs, including the linear interpolation in PMM, different texture attribute designs, and shift & scale in Eqn.11. The corresponding experimental analysis should also be given.\n\n3. There are some mismatches between Fig. 2 and the method descriptions. I cannot find Class-agnostic Text Embedding in Sec 4.1, and I am confused about the Fig.3 caption \"To establish To establish.\"\n\n4. The literature review should cover more recently published works on open-set learning for object detection [1-4].\n\n[1] Zohar, O., Wang, K. C., & Yeung, S. (2023). Prob: Probabilistic objectness for open-world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11444-11453).\n\n[2] Wang, Z., Li, Y., Chen, X., Lim, S. N., Torralba, A., Zhao, H., & Wang, S. (2023). Detecting everything in the open world: Towards universal object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11433-11443).\n\n[3] Ma, S., Wang, Y., Wei, Y., Fan, J., Li, T. H., Liu, H., & Lv, F. (2023). Cat: Localization and identification cascade detection transformer for open-world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19681-19690).\n\n[4] Li, W., Guo, X., & Yuan, Y. (2023). Novel Scenes & Classes: Towards Adaptive Open-set Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15780-15790).\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

comments: SUMMARY:
This paper aims to understand the model’s behavior in predicting the unknown category.  First, the authors model the text attribute and the positive sample probability, obtaining their empirical probability, which can be seen as the detector’s estimation of the likelihood of the target with certain known attributes being predicted as the foreground. Then, they jointly decide whether the current object should be categorized in the unknown category based on the empirical, the in-distribution, and the out-of-distribution probability. Finally, based on the decision making process, the authors can infer the similarity of an unknown object to known classes and identify the attribute with the most significant impact on the decision-making process.

STRENGTHS:
1. The paper is written well and is easy to understand.

2. The studied problem is very important.

3. The results seem to outperform state-of-the-art.

WEAKNESSES:
1. Is it able to utilize more effective OOD score for equations 13-14?

2. It might be more comprehensive to report more metrics, such as AUROC and FPR.

3. It might be useful to include ablation results on prompt templates and LLM used.

QUESTIONS:
see above

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
This paper proposes a new solution for the challenging task of Open World Object Detection (OWOD) by exploring the reasons behind non-classification and then using the textual attributes of unknown objects to infer the most similar known category. Evaluation results on multiple real-world application datasets show significant performance improvements.

STRENGTHS:
A new approach to solving classification problems in Open World Object Detection, and verifying its effectiveness.

WEAKNESSES:
The writing of some mathematical symbols needs to be improved.

QUESTIONS:
(1)The limitation discussion is a bit insufficient. It could be better to add some visualizations of the failure examples
(2)Some mathematical variable symbols(e.g., Eq.6 and Eq.7 ) are not standard. It is recommended that more common and standard notations be used.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new open-world object detection model (UMB) aimed at understanding the model's behavior when predicting unknown categories. By modeling text attributes and positive sample probability, the paper proposes a joint decision-making method based on empirical probability, in-distribution probability, and out-of-distribution probability to infer the similarity of unknown objects to known classes and identify the attributes that have the most significant impact on the decision-making process. The evaluation results on the Real-World Object Detection (RWD) benchmark show that this method surpasses the previous state-of-the-art (SOTA) with an absolute gain of 5.3 mAP for unknown classes, reaching 20.5 mAP.\n\nSTRENGTHS:\nFirst to focus on understanding the model's behavior in unknown category predictions.\nProposes a new framework (UMB) capable of detecting unknown categories and understanding model behavior using textual descriptions.\nShows significant performance improvements on the Real-World Object Detection (RWD) benchmark, especially in unknown categories.\n\nWEAKNESSES:\n1.\tThe method relies on high-quality text attributes and positive sample probability modeling. Performance may be affected if data quality is poor or attributes are incomplete.\n2.\tWhile performing well on the RWD dataset, its generalization ability to other domains and datasets needs further verification, possibly requiring more testing in different environments\n3.\tThe robustness of using the most influencing attributes in the decision process to infer the similarity between unknown and known objects needs further validation.\n4.\tFor large models, the performance and quality requirements for samples and attributes are high. The categories, quantity, and quality of samples and attributes significantly affect network performance and need further analysis.\n\nQUESTIONS:\nWhat are the computational resource requirements for this method in practical applications? Is the method feasible in resource-constrained environments?\nCan this method maintain the same performance improvements on other datasets and in other domains? Further validation in more real-world environments is needed to ensure its broad applicability.\nHow robust is the use of the most influencing attributes in inferring the similarity between unknown and known objects? Could noise or outlier data affect the decision-making process?\n\nLIMITATIONS:\nThe method is quite complex and may require substantial computational resources and time for training and inference. This could limit its use in practical applications with limited computational resources.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}