Sniffing Threatening Open-World Objects in Autonomous Driving by Open-Vocabulary Models

Authors: siqi  wang
yusong  tan
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-04 03:03:54

Abstract: Autonomous driving (AD) is a typical application that requires effectively exploiting multimedia information. For AD, it is critical to ensure safety by detecting unknown objects in an open world, driving the demand for open world object detection (OWOD). However, existing OWOD methods treat generic objects beyond known classes in the training set as unknown objects and prioritize recall in evaluation. This encourages excessive false positives and endangers safety of AD. To address this issue, we restrict the definition of unknown objects to threatening objects in AD, and introduce a new evaluation protocol, which is built upon a new metric named U-ARecall, to alleviate biased evaluation caused by neglecting false positives. Under the new evaluation protocol, we re-evaluate existing OWOD methods and discover that they typically perform poorly in AD. Then, we propose a novel OWOD paradigm for AD based on fine-tuning foundational open-vocabulary models (OVMs), as they can exploit rich linguistic and visual prior knowledge for OWOD. Following this new paradigm, we propose a brand-new OWOD solution, which effectively addresses two core challenges of fine-tuning OVMs via two novel techniques: 1) the maintenance of open-world generic knowledge by a dual-branch architecture; 2) the acquisition of scenario-specific knowledge by the visual-oriented contrastive learning scheme. Besides, a dual-branch prediction fusion module is proposed to avoid post-processing and hand-crafted heuristics. Extensive experiments show that our proposed method not only surpasses classic OWOD methods in unknown object detection by a large margin ($\sim$3$\times$ U-ARecall), but also notably outperforms OVMs without fine-tuning in known object detection ($\sim$ 20\% K-mAP). Our codes are available at https://github.com/harrylin-hyl/AD-OWOD.

Main paper: 
 introduction: 
 Autonomous driving (AD) is a typical application that demands effective exploitation of multimedia information such as visual and linguistic modalities [13, 23, 24] . For AD, ensuring driving safety stands as a fundamental requirement, with one critical aspect being the detection of potential unknown objects in an open world. Recently, open world object detection (OWOD) task has been proposed to address this problem, aiming to bridge the gap between classic object detection (OD) [2, 34, 43] and practical OD in the open world. OWOD methods are expected to detect unknown objects beyond the known classes in the train set, which poses a challenge to evaluating their performance due to the extreme difficulty of defining "unknown objects" and labeling them all. To handle this issue, existing OWOD methods [8, 16] adopt a compromised way by re-organizing the original dataset, where a subset of classes is treated as known and the remainder as unknown. Meanwhile, recent works [30, 44] utilize the mean average precision of known classes (K-mAP) and the recall of the unknown class (U-Recall) to evaluate the performance of detecting known and unknown objects, respectively. However, U-Recall encourages the generation of high-confidence predictions to cover as many unknown objects as possible, regardless of whether many of them are false alarms (see Fig. 2 ). As a result, existing methods cannot satisfactorily address OWOD in AD. To this end, this paper focuses on this challenge and makes three-level contributions: K-mAP and U-ARecall (details in Sec. 4) are used to evaluate the performance of known and unknown object detection, respectively. Classic OWOD methods exhibit a notable performance gap in U-ARecall. The OVM without fine-tuning excels in U-ARecall but falls short in K-mAP. Other OWOD methods based on fine-tuning OVMs improve K-mAP but show an apparent decline in U-ARecall. Compared to them all, our method achieves state-of-the-art overall performance in both U-ARecall and K-mAP. (a) Ground-truth, classic OWOD methods: (b) OW-DETR [8] and (c) PROB [44] , along with (d) Our method. As can be seen, an abundance of false positives are notable in (b) and (c).(1) On the level of evaluation, we argue that unknown objects should be strictly restricted to those that likely emerge on the road and pose a threat to AD (i.e., threatening objects), such as roadblocks and wild animals, rather than all generic objects in previous OWOD works. By this definition, we are able to avoid introducing irrelevant objects and significantly reduce the difficulty of labeling. Meanwhile, we devise a new evaluation metric, termed average recall of the unknown class (U-ARecall) detailed in Sec. 4, to amplify the penalty for false positives including irrelevant objects to AD. This prevents the misguidance in original OWOD evaluation, which overemphasizes the recall of all unknown objects while neglecting the problem of false alarms due to incorrect recall. With the above definition and the proposed U-ARecall, we establish a new evaluation protocol for AD-oriented OWOD. Based on this evaluation protocol, we re-evaluate existing OWOD methods (see Fig. 1 ), and unveil that they typically yield low U-ARecall values, indicating their poor ability to accurately detect unknown objects.(2) On the level of overall detection paradigm, we propose a novel OWOD paradigm based on fine-tuning foundational openvocabulary models (OVMs), e.g., GroundingDINO [28] . OVMs possess the capability to comprehend high-level language semantics and recognize diverse visual patterns, which evidently contributes to detecting threatening objects in the open world. To further assist OVMs in comprehending the semantics of "threatening objects", we propose to concretize the concept of "threatening objects" by constructing a general vocabulary bag (see Sec. 5.1.2). This vocabulary bag is generally applicable to the AD scenario and contains vocabularies of possible threatening objects. Notably, we avoid using the unknown class vocabularies from the test set during the generation process, thus ensuring the generality of vocabulary bag and preventing information leakage. Meanwhile, we notice that OVMs tend to be "generalists" in generic scenarios rather than "specialists" in a specific scenario, which is reflected by the poor K-mAP of GroundingDINO in Fig. 1 . Thus, OVMs need to be further fine-tuned in our paradigm to adapt the AD scenario.(3) On the level of specific solution, we first identify two core challenges when fine-tuning OVMs for AD-oriented OWOD: 1) The risk of forgetting open-world generic knowledge, e.g., the OVM's performance in U-ARecall is degraded by 36% and 7.5% after being fine-tuned by fully fine-tuning (FFT) and LoRA [12] (see Fig. 1 ). 2) The effectiveness of acquiring scenario-specific knowledge, i.e., how to improve the K-mAP of OVMs while preserving a satisfactory U-ARecall performance. To address the two challenges above, we propose a brand-new solution that includes the following novel techniques: 1) A dual-branch architecture (DBA) that contains a frozen and a trainable branch to preserve general open-world knowledge by freezing the parameters of OVMs. 2) A visual-oriented contrastive learning scheme to enhance the effectiveness of acquiring scenario-specific knowledge by minimizing the contrastive loss between training samples and high-confidence visual exemplars. Besides, we propose a prediction fusion module to integrate predictions of two branches in DBA by aligning their query positions, thereby avoiding hand-crafted post-processing.In summary, our contributions are four-fold: (1) We devise a more suitable evaluation protocol for AD-oriented OWOD, which includes restricting unknown objects to threatening objects in AD and introducing a new evaluation metric to alleviate the biased evaluation in OWOD. Based on this evaluation protocol, we re-evaluate existing OWOD methods and establish the AD-oriented OWOD benchmark. (2) We propose a new OWOD paradigm for AD based on fine-tuning OVMs, exploiting rich prior knowledge, including high-level language semantic and diverse visual patterns, to distinguish between threatening objects and false positives. (3) We identify and address two core challenges when fine-tuning OVMs: preserving open-world generic knowledge by dual-branch architecture and acquiring scenario-specific knowledge by visual-oriented contrastive learning. (4) We propose a prediction fusion module that can integrate predictions from multiple branches without the need of post-processing and hand-crafted heuristics, serving as a generalized method applicable to transformer-based detectors. 
related works: 
 Open World Object Detection. Open world object detection (OWOD) aims to adapt classic object detection to the open-world setting. In this setting, methods are required to detect unknown objects beyond the known classes in the train set. OWOD task initially proposed by Joseph et al. [16] has garnered significant attention [8, 30, 31, 36, 39, 44] . ORE [16] adapted Faster-RCNN [34] detector with an energy based unknown identifier for the OWOD objective. Subsequently, OW-DETR [8] adapted Transformer-based detector (D-DETR [43] ) into OWOD, employing a pseudo-labeling scheme to guide the detection of unknown objects. PROB [44] then introduced a probabilistic objectness head to identify unknown objects by estimating their likelihood. VOS [7] modeled a decision boundary to distinguish between unknown objects by adaptively synthesizing virtual outliers. UnSniffer [25] established a generalized object confidence score to separate non-object and object classes. Despite these efforts, existing classic OWOD methods tend to yield an intolerably high false alarm rate for AD. Meanwhile, OWOD research for the AD scenario is also underway. ORDER [36] is a pioneering work in extending OWOD to AD, introducing a feature-mix method to enhance the semantics of unknown objects. However, it followed the original evaluation in classic OWOD, and its code is not available. SalienDet [6] utilized the saliency map to improve the capabilities of unknown object detection, but the saliency map is not always available in practical application. In [31] , they explored the zero-shot object detection in AD by exploiting the language model BERT [5] , but did not consider detecting openworld unknown objects. Besides, its code is also not available. Therefore, there is an urgent demand for a comprehensive AD-oriented OWOD benchmark to advance the research of this field.Open-vocabulary Model. Open-vocabulary model (OVM) aims to expand the recognizable classes of models by constructing a open vocabulary bag. Recently, vision-language pretraining [14, 33] has become a popular paradigm for OVMs to learn rich prior knowledge from large amounts of raw image-text pairs. CLIP [33] is a representative OVM work, which has shown strong zero-shot recognition ability in open vocabulary classification (OVC). In addition to OVC, OVMs have also been adapted into various tasks, e.g., open vocabulary detection (OVD) [21, 28] . In OVD, OVR-CNN [41] first adopted BERT [5] to pre-train the detector on image-caption pairs and then fine-tuned the model for the downstream detection task. Following it, significant efforts [18, 35, 38] have been made to enhance the OVD benchmark. Visual grounding is another related research direction, first proposed by GLIP [21] , which reformulated object detection as a phrase grounding problem. GroundingDINO then extended the advanced transformer-based detector DINO [42] to OVD by performing vision-language modality fusion. While OVMs have achieved high accuracy on generic scenarios [19, 27] , they are more like "generalists" rather than "specialists", and perform poorly in some specific scenarios. Therefore, it is valuable to fine-tune OVMs to adapt the specific scenarios, e.g., autonomous driving.Parameter-efficient Fine-tuning. Fine-tuning is a common strategy to adapt models to specific scenarios. Fully fine-tuning (FFT) is widely used to make the entire network trainable, but it is timeconsuming and prone to overfitting when trained on small datasets. Efficient transfer leaning (ETL) attempts to address this issue by updating or adding a small set of trainable parameters, limiting the dimension of the optimization problem to prevent catastrophic forgetting [32] . For methods without introducing new parameters, linear probing (LP) learned a linear probe on top of frozen embeddings. LP-FT [17] further adopted a two-stage fine-tuning, first performing LP and then FFT with the weight of classifier initialized in the first stage. Alternatively, some methods introduced new parameters within the network [11, 12, 15, 22] . Adapter [11] , as an additionbased method, incorporated a bottleneck adapter structure into the transformer blocks. While LoRA [12] , as a reparameterizationbased method, optimized two low-rank matrices, and then merged them into the weight matrices. Currently, most ETL methods are centered around natural language processing (NLP), with relatively limited research on OVMs. 
preliminaries of owod: 
 Referring to [16, 25] , the problem of OWOD is formulated as follows: a model is trained by a dataset with known objects from a set of known classes C 𝑘 = {1, 2, ..., 𝐶}. During inference, the model is required to detect both known objects from C 𝑘 and unknown objects, which are classified into an unknown class C 𝑢 = {𝐶 + 1}. This formulation also applies to the AD scenario in this paper. Theoretically, "unknown objects" is a vague concept that refers to all generic objects outside known classes C 𝑘 , so it is extremely difficult to precisely define unknown objects and label them all. To bypass this difficulty, previous methods typically perform OWOD by re-organizing existing multi-class object detection datasets (e.g., Pascal VOC [10] and MS COCO [27] ), i.e., treating a subset of classes as known and the remainder as unknown, which avoids defining and labeling unknown objects by themselves. However, such reorganization does not overcome the difficulty of defining unknown objects from the root, and many of detected unknown objects are irrelevant to the specific application scenario like AD.As to the evaluation of OWOD, recent works like [8, 30, 44] tend to adopt mAP of known classes (K-mAP) and recall of the unknown class (U-Recall) to assess the performance of known and unknown object detection, respectively. Specifically, AP is equal to the area under precision-recall (PR) curve for one class, and K-mAP denotes the mean of AP across all known classes. As to the detection of unknown objects, its U-Recall is yielded by computing the recall with 100 predictions, i.e., U-Recall 100 . However, U-Recall 100 does not penalize false positives within predictions. Since 100 is often much larger than the number of objects in most cases, existing OWOD methods tend to output as many high-confidence predictions as possible to cover more unknown objects, regardless of whether many of them are false positives (see Fig. 2 ). This will incur higher false alarm rate and pose a grave threat to the safety of AD. 
the proposed evaluation protocol: 
 To enable a properer evaluation of OWOD in AD, we propose a new evaluation protocol, which mainly includes redefining unknown objects and designing a more valid evaluation metric. Based on this new evaluation protocol, we re-evaluate existing OWOD methods and establish the AD-oriented OWOD benchmark.To tackle the vagueness of "unknown objects", we propose to strictly restrict them to threatening objects, i.e., objects that likely emerge on the road and pose a threat to AD. By this definition, we can mitigate the ambiguity when labeling unknown objects for OWOD in AD, as the concept of "threatening objects" is more definite than that of "unknown objects". Besides, we can also ensure that the detected unknown objects are tightly relevant to the AD scenario. In this way, our definition enables more meaningful OWOD for AD than previous research like [8, 16, 30, 36, 44] .Having redefined "unknown objects" for OWOD in AD, we intend to devise a new evaluation metric to amplify the penalty for false positives (e.g., distracting background and irrelevant objects to AD). As stated in Sec. 3, we observe that the number of predictions used in evaluation (𝑁 ) usually poses a significant impact on the value of recall, i.e., allowing more predictions for evaluation often leads to a higher recall, but it may also introduce more false positives. Previous methods typically use U-Recall 100 that simply sets 𝑁 to a relatively large number (100), which overemphasizes the recall but ignores false positives. To address this problem, given the maximum number of unknown objects 𝑁 𝑚𝑎𝑥 in the test set (𝑁 𝑚𝑎𝑥 is available in evaluation), we first sample 𝑟 values of 𝑁 from 0 to 𝑁 𝑚𝑎𝑥 : 0 < 𝑁 1 < ... < 𝑁 𝑟 < 𝑁 𝑚𝑎𝑥 at an equal interval. Then, we propose a new evaluation metric named U-ARecall to evaluate the performance of unknown object detection:EQUATIONwhere U-Recall 𝑁 𝑖 is the recall value with 𝑁 𝑖 predictions. By virtue of U-ARecall, when only a small number of predictions are allowed for evaluation, we can penalize those high-confidence false positives by a low recall in this case. Meanwhile, we are also able to yield a more comprehensive assessment of recall by varying the value of 𝑁 , so as to handle images with different number of unknown objects. In this way, U-ARecall amplifies the penalty for excessive false positives, thereby providing a more reasonable measurement for OWOD in AD compared to frequently-used U-Recall 100 , simplified as U-Recall (see Fig. 4 ). In addition, we introduce a metric named UK-Mean to assess the overall performance of both known and unknown object detection:EQUATION)where 𝛽 is a parameter to trade off the importance of K-mAP and U-ARecall. In our setup, 𝛽 is set to 0.5. We also carefully select appropriate data sets from the AD scenario (details in Sec. 6.1), which are combined with the aforementioned redefinition and new metric to construct a new benchmark for AD-oriented OWOD. Figure 6 : The Inference architecture of our method. The general vocabulary bag, including known and unknown class texts, is used to generate text features. Meanwhile, the input image is used to produce image features. With these features above, the known and unknown branches generate respective predictions. Finally, the PPF module fuses the predictions of two branches. semantics of "unknown objects", so as to correctly distinguish between unknown objects and false positives. Specifically, we define three coarse classes according to the common sense in the traffic scenario, i.e., {"obstacle", "vehicle", "roadblock"}. Despite the simplicity, those coarse classes have already covered most unknown objects on the road, as they are relatively abstract and each of them semantically includes many sub-classes. Meanwhile, our vocabulary bag avoids using any unknown class text from the test set, thereby avoiding information leakage. Therefore, the vocabulary bag is highly generalizable and not specific to a certain dataset. Pleasantly surprising, this general vocabulary bag is simple yet effective, which exhibits excellent performance in our later experiments. 
fine-tuning: 
 OVMs. pre-training on massive image-text data, OVMs enjoy a strong ability to recognize diverse objects in the open world. However, OVMs tend to be "generalists" rather than "specialists". In the specific scenario like AD, OVMs suffer from a notable performance gap in detecting known objects (see K-mAP in Fig. 1 ). Thus, our OWOD paradigm fine-tunes OVMs to adapt the current AD scenario. We select the recently proposed GroundingDINO [28] as the default OVM, as it achieves state-of-theart performance across various zero-shot OD benchmarks [19, 27] .Following the proposed OWOD paradigm, we customize some classic fine-tuning techniques like linear probe (LP), as well as some recent fine-tuning methods like LoRA [12] and Adapter [11] , specifically for AD-orientated OWOD. However, as shown in Fig. 1 , they either suffer from catastrophically forgetting open-world generic knowledge (e.g. LP-FT), or fail to effectively learn scenario-specific knowledge (e.g., LoRA). Therefore, we identify those challenges and tackle them by corresponding techniques: 1) the maintenance of open-world generic knowledge by a dual-branch architecture (DBA) in Sec. 5.2.2. 2) the acquisition of scenario-specific knowledge by a visual-oriented contrastive learning (VorCL) scheme in Sec. 5.2.3. Besides, a post-processingfree prediction fusion (PPF) module is introduced in Sec. 5.2.4 to integrate the dual-branch predictions. Based on those novel techniques, we propose a brand-new OWOD solution, the training and inference procedures of which are elucidated in Fig. 5 and Fig. 6 .To mitigate catastrophic forgetting, the proposed DBA first uses the frozen vision/text backbone to extract image/text features, which are then fed into a dual-branch architecture that consists of a frozen unknown class branch and a trainable known class branch. With the frozen backbones and the frozen unknown class branch, DBA can effectively preserve the generic knowledge within OVMs to handle the unknown objects in the open world. Meanwhile, the trainable known class branch enables DBA to acquire the specific knowledge of the AD scenario for detecting known objects. Since the number of parameters in the known class branch is much less than the OVM, the computational cost of DBA is affordable. Besides, DBA is parallelizable and can be optimized to reduce inference time during deployment.Let us define 𝐼 as the input image, 𝑇 as the input text, B 𝑣 as the vision backbone, B 𝑡 as the text backbone. Since the components of both the known class branch and the unknown class branch are identical, we use the same notation for their corresponding components for simplicity. Let E and D be the encoder and decoder of the branch. F denotes a query selection function that selects top-k confident query embeddings from the output of E, and p is the query positions. The forward pass of a branch starts by: F, r 𝑡 = E (B 𝑣 (𝐼 ), (B 𝑡 (𝑇 ))(3)EQUATIONwhere F and r 𝑡 are the enhanced image and text features, respectively. r 𝑏 is the region-level visual features at positions p. Then, the detection scores are calculated by the classification head H 𝑐𝑙𝑠 :EQUATIONwhere ⟨•⟩ is the similarity function that computes the inner product. 𝑇 is the temperature to re-scale the value, set to 1 as [28] . Bounding boxes can be obtained by the regression head H 𝑟𝑒𝑔 : b = H 𝑟𝑒𝑔 (r 𝑏 ).Learning. We begin by reviewing the standard contrastive learning [33] in OVMs, which minimizes the distance between visual features and textual features. It uses the detection scores s in Eq. 5 to calculate the contrastive loss. For example, the Focal loss [26] for contrastive learning is: where 𝑦 denotes the labels of known classes, and 𝛾 is a hyperparameter that is set to 2 [28] .EQUATIONTo further enhance the effectiveness of fine-tuning via contrastive learning, we propose the VorCL scheme that aims to minimize the distance between visual features from training samples and high-confidence exemplar visual features. These exemplar visual features serve as semantic centers to optimize hard samples, as illustrated in Fig. 7 . The known class branch is the only trainable part of our solution, while its enhanced text features r 𝑡 , region-level visual features r 𝑏 and corresponding scores s, can be obtained by Eq. 3, Eq. 4 and Eq. 5, respectively. Let 𝑁 𝑞 be the number of queries per image and 𝐵 be the training batch size. The 𝑐-th class exemplar visual feature (r 𝑣 ) 𝑐 can be obtained as follows:(r 𝑣 ) 𝑐 = 𝑁 𝑞 𝐵 𝑖=1 1(max((s 𝑖 ) 𝑐 ) ≥ 𝜏)(r 𝑏 𝑖 ) 𝑐 • (s 𝑖 ) 𝑐 𝑁 𝑞 𝐵 𝑖=1 1(max((s 𝑖 ) 𝑐 ) ≥ 𝜏)(s 𝑖 ) 𝑐 (7)where (s 𝑖 ) 𝑐 and (r 𝑏 𝑖 ) 𝑐 are s and r 𝑏 for 𝑖-th query and 𝑐-th class, respectively. 𝜏 is a selection threshold used to select r 𝑣 . As described in Eq. 7, the r 𝑏 with the same class will be weighted by their corresponding scores s and summed, producing r 𝑣 in the current training batch. Then, we adopt a memory bank with the length of known classes to store and update r 𝑣 . The memory bank is initialized by r 𝑡 and continuously updated with the input training batch. In the next training batch, the (r 𝑣 ) 𝑐 at the 𝑡-th iteration is updated by exponential moving average (EMA):EQUATIONwhere 𝜆 is an EMA hyper-parameter. Then, the visual-oriented detection scores s 𝑣 are computed by: s 𝑣 = H 𝑐𝑙𝑠 (r 𝑣 , r 𝑏 ). The calculation process of VorCL loss is analogous to Eq. 6 but uses s 𝑣 :EQUATIONBased on Eq. 6 and Eq. 9, the total loss of our OWOD solution is:EQUATIONwhere L 𝑟𝑒𝑔 is the regression loss as in [28] .Post-processing-free Prediction Fusion. PPF module aims to integrate the predictions of the known class branch and the unknown class branch to produce the final predictions. In general, non-maximum suppression (NMS) is a prevalent module to fuse two distinct sets of predictions in OD. However, it introduces handcrafted heuristics, which contradicts the core idea of transformerbased detectors (GroundingDINO also falls to this category), i.e., minimizing the manual design. This motivates us to propose a new fusion technique without the need of hand-crafted post-processing. By inspecting the positional variation of proposals and bounding boxes, we found that the misalignment between predictions of the known and the unknown branch arises from the differences in query positions, i.e., p in Eq. 4. Query positions represent the positions of learnable proposals, which are continuously updated and changed during training, thereby leading this misalignment problem. By applying the same query position for both branches, their predictions can be well aligned as a one-to-one pairing, so as to achieve one-to-one fusion. Specifically, we use the query positions of the unknown branch p 𝑢 for both branches. This is because p 𝑢 is class-agnostic and can already cover the majority of objects. As mentioned in Sec. 5.2.2, we can obtain the detection scores and bounding boxes of the known branch (s 𝑘 , b 𝑘 ) and the unknown branch (s 𝑢 , b 𝑢 ). Then, we apply the geometric mean to combine s 𝑘 and s 𝑢 . The 𝑐-th class final detection scores (s 𝑓 ) 𝑐 are calculated by:EQUATIONwhere 𝛼 is the hyper-parameter to control the contributions of (𝑠 𝑘 ) 𝑐 and (𝑠 𝑢 ) 𝑐 . Subsequently, we adopt a class-wise box selection operation to fuse b 𝑘 and b 𝑢 . The 𝑐-th class final bounding boxes (b 𝑓 ) 𝑐 are calculated as follows:EQUATION6 Experiment 
datasets: 
 For the evaluation of AD-oriented OWOD, we survey existing AD datasets [1, 4, 9, 20, 37, 40] and discover that, to our best knowledge, only the labeling criteria of the CODA dataset [20] align well with our definition of threatening objects. However, CODA is originally designed for corner case detection, and corner cases in validation set are accessible, which contradicts to the problem setting of OWOD that open-world objects are strictly unknown during training. Therefore, in our evaluation protocol, we treat the common classes of CODA as known classes and its corner-case classes as unknown classes. We leverage the released validation set of CODA as the test set of our evaluation protocol. It consists of 4,484 road driving images with 29 representative classes, including 6 common classes and 23 corner-case classes.We fine-tune OVMs by the train set of the SODA [9] or the BDD100K [40] dataset. The two datasets are used to simulate the case with small or large domain differences from CODA. In this way, we can inspect the generalization abilities of methods. Specifically, SODA includes 10,000 images with 6 classes, while BDD100K contains 70,000 images with 10 classes. All classes in the train set of SODA are viewed as known classes. We view 5 of the 10 classes of BDD100k as known classes (i.e., person, rider, car, bus, and truck), as only those 5 classes appear in common classes of CODA.As to the setting of U-ARecall, we get the maximum value of labeled unknown objects in CODA (𝑁 𝑚𝑎𝑥 = 43), and adopt three sampling prediction numbers (𝑁 1 = 10, 𝑁 2 = 20, 𝑁 3 = 30) to handle images with different numbers of unknown objects. Due to the superior open-world capabilities of GroundingDino [28] , we choose it as the OVM for further fine-tuning, which is implemented by MMDetection [3] . We train the model over 12 epochs with the adamW optimizer [29] , and its weight decay is set to 1 × 10 -4 . The learning rate is initialized by 1 × 10 -4 and reduced by a factor of 10 after reaching the 11-th epoch. Without careful selection, we set the hyper-parameters in VorCL and PPF based on our intuition and experience: 𝜏 = 0.3 in Eq.7, 𝜆 = 0.99 in Eq.8, and 𝛼 = 0.2 in Eq. 11.In our AD-oriented OWOD benchmark, as shown in Tab. 1 and Tab. 2, methods under comparison are categorized into five types: classic OD methods (the first part), classic OWOD methods (the second part), the raw OVM [28] without fine-tuning (the third part), the OVM directly fine-tuned by standard fine-tuning methods (the fourth part) and our method (the last part). We list results of classic OD methods to show their inability to handle OWOD, and results of classic OWOD methods to demonstrate the greater difficulty of AD-orientated OWOD than classic OWOD. As our method is based on OVM fine-tuning, we also include the raw OVM and OVM finetuned by standard methods to present a fairer comparison. Methods employed for fine-tuning the OVM include fully fine-tuning (FFT), linear probing (LP), LP-FT [17] , Adapter [11] , and LoRA [12] . The experimental results using SODA dataset are shown in Tab. 1, which reveal that our method achieves state-of-the-art performance for AD-oriented OWOD. As for the overall performance (UK-Mean), our method significantly improves UK-Mean by more than 33 % compared to the best classic OWOD method (UnSniffer), which justifies the necessity of exploiting the OVM's rich open-world knowledge. Even compared to the second-best method (LoRA), our method also notably surpasses it by 7.4% in UK-Mean, which indicates the effectiveness of our proposed techniques. When we look at the performance of unknown object detection (U-ARecall) alone, our method has achieved an remarkable performance advantage over classic OWOD methods, and it is even slightly better than the raw OVM by 0.8%. In particular, our method evidently outperforms the OVM fine-tuned by standard methods by at least 8.3% U-ARecall, which demonstrates that our method can effectively mitigate the problem of catastrophic forgetting during fine-tuning, thereby preserving the open-world generic knowledge within the OVM. As for the performance of known object detection (K-mAP), our method notably improves K-mAP by 23.5% when compared with the raw OVM, which unveils the necessity of fine-tuning OVMs to adapt the AD scenario. More importantly, our method performs comparably to OVM fine-tuned by standard methods (≤ 0.2% K-mAP). Such results suggest that our method can be as effective as standard fine-tuning methods in acquiring scenario-specific knowledge, without being severely influenced by catastrophic forgetting like those methods. As a consequence, our method achieves fairly satisfactory performance in detecting both unknown and known objects. In addition, the experimental results using BDD100K (see Tab. 2) exhibit a very similar trend to that of SODA. This proves the effectiveness of our method again, and verifies the fine generalization ability of our method across different domain gaps.Fig. 8 depicts the qualitative results of Raw OVM, FFT and our method. We set a threshold of 0.3 to determine the visualization of known and unknown objects. As can be seen, the raw OVM tends to overlook or misidentify certain known objects, e.g., pedestrian, and bus. FFT typically fails to detect some unknown objects, e.g., traffic cone, stroller, and dog. In contrast, our method exhibits a strong ability to accurately detect both known and unknown objects, demonstrating its robust open-world recognition capabilities.Ablation Studies. We analyze the contributions of our proposed DBA, VorCL and PPF, with the experimental results presented in Tab 3. When we apply the DBA, U-ARecall improves by a large margin (27.7%), which unveils its advantage in alleviating knowledge forgetting. Replacing NMS with PPF leads to notable improvements in both U-ARecall and K-mAP by 9.4% and 6.4%, respectively, demonstrating the strength of avoiding hand-crafted heuristics in NMS. Additionally, incorporating the VorCL scheme into our solution, results in a 1.1% improvement in K-mAP, proving its effectiveness in acquiring scenario-specific knowledge during fine-tuning. Sensitivity Analysis. As illustrated in Fig. 9 , we conducted experiments to analyze the sensitivity of hyper-parameters in our method. It is evident that our method is not sensitive to hyper-parameters. Discussion. In Sec. 5.1.2, we define a general vocabulary bag with three coarse unknown classes for AD-oriented OWOD. An intuitive question is whether refining the class vocabularies in the vocabulary bag can improve performance. To investigate this issue, we also design a more delicate method to generate a fine vocabulary bag, which contains texts of 59 finer classes in total, and compare its performance with our default vocabulary bag. As shown in Tab. 4, it is obvious that our default vocabulary bag is much better than the refined vocabulary bag. This indicates that having more vocabularies is not necessarily better. It increases the risk of more false positives, as the OVM may not be able to correctly recognize all classes in the vocabulary bag. 
conclusions: 
 This paper address a core challenge that impacts the safety of AD, i.e., detecting threatening open-world objects in AD. To address this challenge, we make three-level contributions: 1) On the level of evaluation, we mitigate the ambiguity of labeling unknown objects by restricting them to threatening objects, and prevent the misleading evaluation in original OWOD by introducing a new evaluation metric named U-ARecall. 2) On the level of overall OWOD paradigm, we alleviate the difficulty of distinguishing between unknown objects and false positives by exploiting the rich multimodal knowledge of OVMs, and adapt them to the AD scenario by fine-tuning. 3) On the level of specific solution, we identify two challenges when fine-tuning OVMs: catastrophic forgetting and scenario-specific knowledge acquisition. By addressing these challenges, our method achieves state-of-the-art performance for AD-oriented OWOD. 
 
 Appendix:  


Decision: This paper aims to address the challenge of detecting unknown, potentially threatening objects in autonomous driving. Its pros including the introduction of U-ARecall metric, innovative use of a dual-branch architecture, efficient parameter migration, thorough experiments, good results, and etc. The reviewers raised concerns regarding model complexity, results relying on several hyperparameters, etc. The authors well address these concerns in the rebuttal. Reviewer 249B gave borderline reject but had some vital misunderstandings. Given the strong recommendation from Reviewer Jhu7 and borderline accept recommendation from Reviewer 7JWU, the AC recommends this paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper aims to address the challenge of detecting unknown, potentially threatening objects in autonomous driving (AD). Existing methods for open-world object detection (OWOD) often result in excessive false positives, which pose a safety risk for AD. To tackle this, the authors propose a new evaluation protocol focused on threatening objects specifically related to AD, introducing the U-ARecall metric to account for false positives more effectively. The paper also presents a novel OWOD paradigm based on fine-tuning foundational open-vocabulary models (OVMs). The proposed solution includes a dual-branch architecture to maintain open-world knowledge, a visual-oriented contrastive learning scheme to acquire scenario-specific knowledge, and a prediction fusion module to avoid manual post-processing. The authors demonstrate through extensive experiments that their method significantly outperforms existing OWOD methods in terms of both U-ARecall and K-mAP.
strength_weakness: strength_weakness: 1. The introduction of the U-ARecall metric addresses the critical issue of false positives in OWOD, which is particularly important for the safety of AD.

2. The innovative use of a dual-branch architecture helps in retaining general open-world knowledge while fine-tuning for specific scenarios, mitigating the problem of catastrophic forgetting.

3. This technique enhances the model's ability to acquire scenario-specific knowledge, improving the detection of both known and unknown objects.

4. The authors conduct thorough experiments on different datasets, demonstrating the robustness and effectiveness of their approach across various domain gaps.

5. The proposed method significantly outperforms existing methods, achieving approximately three times the U-ARecall and a notable improvement in K-mAP.
limitations: limitations: 1. The proposed dual-branch architecture and visual-oriented contrastive learning add complexity to the model, which may impact the practicality of deployment in real-world scenarios.

2. While the method performs well on selected datasets, further validation on a wider range of datasets and real-world conditions would strengthen the generalizability claims.

3. Although the sensitivity analysis shows robustness to hyperparameter changes, the method still relies on several hyperparameters, which might require fine-tuning for different applications.
paper_summary: paper_summary: This is a paper that fully considers open-world object detection in autonomous driving scenarios. First of all, based on the characteristics of autonomous driving scenarios, the article proposes a new indicator u-arecall to alleviate the evaluation bias caused by ignoring false alarms. Afterwards, this paper provides rich prior information for the OWOD task by introducing an open vocabulary model and fine-tuning it. Finally, the paper optimizes the fine-tuning process and achieves good results.
strength_weakness: strength_weakness: (1) This paper will propose new indicators adapted to autonomous driving scenarios.
(2) This paper introduces efficient parameter migration to make LLM better adapt to the current scenario. Compared with overall fine-tuning, the method in this paper avoids both overfitting and large-scale parameter updates.
limitations: limitations: In the experimental part, Visual Prompt Tuning should be introduced to further illustrate the effectiveness of the method in this paper.
paper_summary: paper_summary: The authors proposed a novel framework for detecting threatening open-world objects by introducing an open vocabulary model.  This framework addresses two core challenges of fine-tuning open vocabulary models (OVMs) through two innovative techniques: 1) the preservation of open-world generic knowledge using a dual-branch architecture;  and 2) the acquisition of scenario-specific knowledge via a visually-oriented contrastive learning scheme.  Extensive experiments and analyses on benchmark datasets demonstrate that the proposed methods achieve state-of-the-art performance.
strength_weakness: strength_weakness: 1. The paper's motivation is highly novel and addresses a fundamental challenge that significantly impacts the safety of autonomous driving.
2. Numerous experiments have been conducted to demonstrate the efficacy of the method.
3. Open vocabulary model offers a novel perspective for sniffing threatening open-world objects.
4. The writing style and structure of the paper make it easy for readers to comprehend the methodology and significance of the study.
limitations: limitations: 1. The confidence level of the car and person in Figure 8 is too low. A proficient object detection model designed for detecting hazards should consistently uphold a high level of accuracy across the annotated categories.
2. The paper fails to provide a reasonable explanation of how the model prevents misclassifying obstacles as background during the fine-tuning process in the training phase.
3. The paper lacks a reasonable explanation of how the model can differentiate obstacles from the background during the inference phase, and instead solely relies on frozen OVMs.
4. Whether the specific classification of obstacles depends on the existing knowledge in OVM？ 
5. The visualization results presented in the supplementary material appear to be the outcome of a direct test of the 59 finer classes using the existing OVM.
6. The experiment lacks unknown class mAP.


Further Comments

1. I hope the author takes the rebuttal seriously, and I will revise the review as appropriate based on the response provided in the rebuttal.
2. The paper need cite more papers related to multimedia.