Zero-Shot Open-Vocabulary OOD Object Detection and Grounding using Vision Language Models

Venue: NLDL_org_2025_Conference
Published: 2024-09-09 00:58:43

Abstract: Automated driving involves complex perception tasks that require a precise understanding of diverse traffic scenarios and confident navigation. Traditional data-driven algorithms trained on closed-set data often fail to generalize upon out-of-distribution (OOD) and edge cases. Recently, Large Vision Language Models (LVLMs) have shown potential in integrating the reasoning capabilities of language models to understand and reason about complex driving scenes, aiding generalization to OOD scenarios. 
However, grounding such OOD objects still remains a challenging task. In this work, we propose an automated zero-shot framework for promptable open vocabulary OOD object detection, segmentation, and grounding in autonomous driving. 
We leverage LVLMs with visual grounding capabilities, eliminating the need for lengthy text communication and providing precise indications of OOD objects in the scene or on the track of the ego-centric vehicle.  
We evaluate our approach on OOD datasets from existing road anomaly segmentation benchmarks such as SMIYC and Fishyscapes. Our zero-shot approach shows superior performance on RoadAnomaly and RoadObstacle and competitive results on the Fishyscapes subset as compared to supervised models and acts a baseline for future zero-shot methods based on open vocabulary OOD detection.

Main paper: 
 introduction: 
 With the emergence of large-scale pre-trained models or foundation models, the domain of Artificial Intelligence has experienced a huge paradigm shift. These foundation models are trained on large amounts of data across multiple different domains consisting of millions or billions of parameters over several weeks or months. The immense training scale allows such models to capture sufficient generic knowledge about the world, and thus serve as a 'foundation' to be effectively utilized for downstream tasks, often in a zero-shot manner.The recent advancements of foundation models in the form of Large Language Models (LLMs) such as BERT [1] , , Llama [3] etc have sparked development of Multimodal Large Language Models.They aim to merge the reasoning capabilities of LLMs with the rich and diverse feature space of other data modalities such as image, video, audio, and point cloud data. Different modalities provide 048 a diverse range of tasks to be solved, which in turn 049 enhances the overall performance in categorizing the 050 data.Autonomous vehicles (AV) are tasked with com-052 prehending one of the most complex perception tasks 053 in the form of complicated traffic scenes. Traffic 054 scenes can be very unpredictable with a high de-055 gree of variance. Automated driving (AD) system 056 needs to precisely perceive its surroundings, follow 057 intricate rules, make proper decisions, and navigate 058 with certainty. Traditional data-driven algorithms 059 follow a modular approach that relies on three key 060 components: perception, prediction, and planning. 061 However, since they are trained with limited data 062 in closed scenarios, they have been found to be in-063 adequate in generalizing towards open domain like 064 perceiving open vocabulary objects and dealing with 065 corner cases [4] . LLMs have shown quite a promise 066 in tackling such issues by incorporating linguistic 067 reasoning into complex cases. However, their appli-068 cation has been limited to prediction and planning 069 systems, due to their inability to process visual data. 070 Now, with cutting-edge LVLMs such as GPT-4V 071 [5] , newer possibilities have cropped up in handling 072 the perception component. Preliminary experiments 073 [6] with GPT-4V on a subset of CODA dataset [4] 074 indicate a strong grasp of traffic understanding and 075 generalization abilities for OOD scenarios but strug-076 gle with vision grounding tasks such as localizing 077 entities and specifying pixel-level coordinates. More-078 over, often verbose prompt-based communication 079 makes it difficult to derive precise instructions.In this work, we demonstrate an automated zero-081 shot approach to detect and ground OOD objects 082 in the scene using LVLMs. We leverage LVLMS 083 communication, allowing users to express intentions for motion planning in everyday language. E.g., the GPT-driver [14] explains vehicle action recommendations, [15] assesses lane occupancy and safety, and "Drive as You Speak" [16] integrates advanced reasoning and language capabilities for personalized planning.Large Vision Language Models: Incorporating the reasoning abilities of LLMs into vision data has led to the emergence of several LVLMs for diverse critical vision tasks such as Segment Anything Model [17] for prompt-based segmentation, DALL-E [18] for prompt-based image generation, Grounding DINO [8, 19] for visual grounding and open-vocabulary detection, and so on. In AD, SAM3D [20] combines bird-eye-view images from lidar point clouds with SAM for 3D object detection, however, the performance lags compared to SOTA so far. GTP-4V [5] also shows great potential for complex scene understanding and OOD generalization capabilities but at the same time struggles with the visual grounding of such entities. Thus, the integration of LVLMs with visual grounding capabilities seems paramount for OOD detection tasks. 
method: 
 In this section, we provide an overview of our framework, as illustrated in Fig. 2 . We first present the generic framework for detecting all the OOD objects in the scene and subsequently, present a refinement module for automatically detecting OOD objects appearing only on the track of the ego-centric AVs.LVLMs: A LVLM is said to be capable of universal visual perception when it can perform the following tasks: (a) Unified detection and grounding, i.e. merging object detection and visual grounding in the form of instance level region-sentence matching, such that models can easily scale to query thousands of vocabulary concepts at a time, (b) Unified image segmentation, i.e. a universal model that can handle queries based on the semantic, instance and panoptic segmentation without causing mutual interference between things (foreground) and stuff (background) categories within each query. APE [7] trained on diverse detection, segmentation, and grounding datasets and benchmarked on over 160 datasets has proven to be robust with one suite of weights for all tasks. Thus, APE shows outstanding potential to serve as a frozen detector backbone for the task of OOD object detection due to its training with instance-level region-sentence matching in the OVOD setting.Framework for detecting all OOD objects: Our framework is based on frozen pre-trained LVLMs with zero-shot detection capabilities in the OVOD setting. Our method depends on a few inference steps without requiring any fine-tuning on the Figure 2 . Overview of our proposed framework. Firstly, an inference with no prompts is run on frozen LVLM to detect all foreground instances(MT ) and then a second inference is run using ODD class list as prompt to obtain in-domain instances (MI ), which are then combined into a single binary mask. The remaining non-matching instances (MZ ) together with inverted difference mask (MD) are fed into our proposed Algorithms 1 and 2, to obtain final mask with OOD objects (MOOD). A refinement module is applied to detect OOD objects on the ego-track of an AV. Detected OOD objects are shown in 'red' with corresponding LVLM predictions in 'white'. Input: OOD List L = ['car', 'person', . . .], N , M Z , M D Output: Mask with OOD objects M OOD i ← 1, iou ← 0, ioumax ← 0 ; M OOD ← 0 ; while i < N do for 1 ≤ inst ≤ Z do iou ← IoU(Minst, M D ) ; if iou > ioumax then ioumax ← iou ; if pred(inst) / ∈ L then i ← i + 1 ; M OOD ← M OOD + Minst ; end M D ← M D -Minst ; end end endclasses. One can query LVLMs to retrieve known 231 objects. However, LVLMs trained on a variety of 232 data, tend to predict multiple object names even for 233 similar object instances, such as a non-distinct bird 234 has multiple predictions as duck, seagull etc. Also, 235 fine-grained object instances are detected such as 236 shoes which is part of person or wheel part of car 237 object class. There can be several object instances 238 that are not linguistically aligned with ODD object 239 prompts and are falsely detected as OOD.We provide two algorithms to determine the plau-241 sible OOD instances from the binary difference mask 242 M D , as well as the remaining mask list(M Z ). In 243 Algorithm 1, we assume a pre-defined maximum 244 number of OOD objects (N ) to be detected in the 245  
implementation details: 
 Experimental Setup: Our approach is completely based on zero-shot inference on frozen features of any LVLM capable of OVOD. The LVLM used in this work, is APE (D) [7] having VIT-L backbone. Using a validation set, we calculated an optimum number of OOD instances that should be detected for each dataset using Algo 1 as shown in Fig A .2 in Suppl. Similarly, for Algo 2, a generic NIS threshold for all datasets is set at 0.8 to determine plausible objects from M OOD , as described in Sec. 3.We evaluate OOD object detection for AD on existing OOD detection benchmarks for road anomaly segmentation. The 19 semantic classes given in the urban road driving dataset -Cityscapes [21] , have been used as the ODD or in-domain classes for AD on road. SegmentMeIfYouCan (SMIYC) benchmark [11] provides two OOD datasets with real OOD objects: RoadAnomaly21 (RA) and Road-Obstacle21 (RO). We also evaluate on a subset, FS Static, provided by Fishyscapes [12] (FS) benchmark It consists of generic objects taken from PASCAL VOC [22] synthetically overlayed on Cityscapes.Evaluation Metric: While dealing with zero-shot inference on frozen foundation models, we need to detect every foreground object for OOD detection which requires the detector threshold to be fixed at a low value. The zero-shot setup is different from traditional supervised methods which depended upon logits predicted over closed-set categories based on varying detector thresholds and thus the corresponding benchmark metrics in the SMIYC [11] , Fishyscapes [12] are no longer relevant. Hence, we report Intersection over Union (IoU) and mean F1 metrics with a fixed minimum detector threshold and compare with methods that report these metrics. More details on datasets and evaluation in Sec. A.1 in Suppl. 
results and discussion: 
 Here, we present experiments to show our proposed zero-shot promptable framework using LVLM for detecting and grounding OOD objects for AD.Table 1 shows comparison of our zero-shot method with SOTA-supervised methods reporting IoU and mean F1 metrics by [23] , and qualitative samples for comparison with SOTA are presented in Fig. A .3 in Suppl. To the best of our knowledge, there exists no prior work using LVLMs for zero-shot OOD object detection, segmentation or grounding in AD benchmarks or otherwise. Thus, there exists no direct baselines or established evaluation metrics for such zero-shot methods. It should be noted that all other methods are explicitly trained on domain data (ODD) object classes for pixel-wise segmentation. We present results with our approach for both Algorithms: Algo 1 for a deterministic number of OOD objects (N ), and Algo 2 for all possible OOD OOD is probably susceptible to change in texture, however using LVLMs they are correctly grounded to domain class, although leading to reduced performance as no OOD is detected. Thus, we also report results by excluding such dubious images.In Fig. 3 , we show examples with our algorithms across OOD datasets. For detecting multiple OOD objects, we find optimal results with N = opt and Algo 2 such as all the birds in Col.1 or the very small objects like watering can, water jug in Col.3. Since Algo 2 is independent of N , we find in Col.5, it detects all possible OOD objects in the scene beyond the annotated OOD in GT, such as dumpster, postbox etc. However, often multiple boxes with closely matching semantic object categories are predicted for instances in Algo 2, based on multiple predictions generated by the LVLM. In such cases, the maximum IoU matching provided by Algo 1 is able to provide the most confident prediction for each instance. 
conclusion: 
 In this work, we proposed a novel automated zeroshot promptable method to detect open-vocabulary OOD objects using frozen LVLMs capable of visual grounding. It aims to detect and ground OOD objects in AD application, which is often difficult for supervised methods trained on closed-set categories. We proposed two algorithms, one based on detecting the pre-determined number of OOD objects and the other for detecting all possible OOD for practical use-cases. In future, this approach can also be used for creating datasets with real OOD objects from unlabelled data, for future benchmarks without much added annotation effort. Lastly, we believe this framework can serve as a useful first baseline for future zero-shot methods on foundation models in the open vocabulary setting for OOD detection. Dataset: SegmentMeIfYouCan (SMIYC) bench-570 mark [11] provides two OOD datasets with real 571 OOD objects: RoadAnomaly21 (RA) and Road-572 Obstacle21 (RO). RA consists of 100 test and 10 573 validation images with real objects or animals as 574 OOD appearing anywhere in the scene. In contrast, 575 RO has OOD objects appearing on the road/ ego-576 track. SMIYC benchmark withholds the GT for 577 the test set, where the scores are only accessible by 578 submitting the method to the official benchmark. 
 
 Appendix: We also evaluate on a subset, FS Static, provided 580 by Fishyscapes [12] (FS) . As pointed out in Sec. 5, 581 we note that 10 images from the FS Static test set 582 have falsely annotated OOD objects whose classes 583 overlap with In-domain classes given in Cityscapes 584 [21] such as bus, truck, car, person, etc as shown 585 in Fig. A.1. Surprisingly, all the SOTA supervised 586 methods continue to detect these domain objects as 587 OOD as probably they detect the change of texture 588 for the pasted synthetic OOD object, rather than 589 object itself. However, LVLMs such as APE are able 590 to correctly detect and ground them to respective 591 ODD classes. Thus, this leads to reduced perfor-592 mance as there are no OOD objects detected where 593 there are some annotated. We verify this by observ-594 ing an increase in performance with our approach 595 when such dubious images are eliminated.Evaluation: SMIYC and Fishyscapes are road 597 anomaly segmentation benchmarks that support su-598 pervised methods and metrics explicitly trained on 599 domain data. These benchmarks require logits var-600 ied over confidence-based detector thresholds over 601 closed-set classes and thus, are incompatible with 602 methods such as ours based on zero-shot inference 603 on foundational models trained on a large number 604 of classes. Thus, the evaluation metrics on these 605 benchmarks such as Average Precision (AP), False 606 Positive Rate (FPR), etc which require variation over 607 confidence thresholds, are no longer relevant with 608 zero-shot methods which do not require choosing 609 any threshold for every dataset. We show evaluation 610 on all predicted instances from the frozen LVLMS us-611 ing Intersection over Union (IoU) and F1. Thus, we 612 are unable to make benchmark submissions. Since 613 the SMIYC test setup withholds the GT for test 614 data, we are forced to show these evaluations on 615 the validation split (where GT is given) for RA and 616 RO and show visibly impressive qualitative result 617 images from the test data in Fig. 1 and A.3.Ablation to determine the optimum number 620 of OOD objects: Mapping similar objects to their categorical object groups via language alignment can be potential future direction of research. Secondly, for datasets with small objects such as RoadObstacles, although our method reports reasonable results compared to many SOTA supervised methods that do not report results, however very small objects that match with background often fail to get detected such as the small obstacles in the  


Decision: This paper suggests to rely on large vision-language model (LVLM) to perform zero-shot out-of-domain (OOD) object detection.

All reviewers agreed about the importance of the problem, the relevance of the techniques involved, the clarity of the paper, and the significance (to some extent) of the experimental results. However, a few negative aspects were also identified, lack of technical contributions, overstated results, lack of technical details. The authors provide a systematic rebuttal to reviewers' comments and their replies were judged valuable. 

Overall, the paper brings an intuitive and simple, but effective contribution and the work is presented in a clear manner. The authors should, however, be reminded to add a description of the newly added baseline to the manuscript.

All reviewers suggest to accept the paper. Given the strength of the contribution, a poster presentation seems more appropriate.

Reviews: review: title: title: Official Review
paper_summary: paper_summary: The work proposes a zero-shot framework for OOD object detection and grounding in an autonomous vehicle setting. The framework leverages the capabilities of frozen large vision language models (LVLM) to first detect all objects in the image. In-domain objects are then detected via prompting the LVLM with a list of in-domain classes and the list of potential OOD objects is then retrieved by subtracting the masks of in-domain objects from the mask containing all objects. As the resulting OOD mask will contain noise/artifacts, two algorithms are introduced to retrieve the final list of OOD masks. These two algorithms take as input the inverse in-domain mask and the noisy OOD mask and either rely on the assumption that the number of OOD objects is known or that a threshold has been selected that considers the overlap between instances in the two masks. Finally, the LVLM is prompted to return the road mask and object outside the convex hull of the road are removed.
main_review: main_review: After going over the rebuttal and changes that the authors have made to the document, I believe that the authors have addressed the majority of the concerns and recommend acceptance of the manuscript. The paper makes an intuitive and simple, but effective contribution and the work is presented in a clear manner. The authors should, however, add a description of the newly added baseline to the manuscript.
strength_weakness: strength_weakness: - In the evaluation, the authors claim that the model outperforms prior approaches on the RoadAnomaly and RoadObstacle datasets while achieving competitive results on Fishyscapes (FS). However, as no results are included for RoadObstacle for the baseline methods and the difference between the best baseline approach and the proposed method on the FS dataset is quite significant, results appear overstated. Given that the experiments compare the zero-shot approach with supervised models, the reviewer believes that the significant differences on the FS dataset are not a significant problem.

- The authors mention several places that a very low fixed LVLM detector threshold is critical, however, it is not specified what this threshold is set to.

- It would have been beneficial to include a discussion of limitations and/or failure cases.

Minor: 
L298 Figure reference missing.
title: title: Good paper which presents a zero-shot framework for OOD object detection
paper_summary: paper_summary: SUMMARY: This paper presents two zero-shot algorithms for out-of-distribution (OOD) object detection using Large Vision Language Models (LVLMs). Experiment shows that the proposed method outperforms its competitors.
main_review: main_review: I would like to keep my rating
strength_weakness: strength_weakness: 1. The proposed methods are still simple