AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving

Authors: dirk  neumann
alexey  weissenborn
  dosovitskiy
  aravindh
anurag  mahendran
mostafa  arnab
  dehghani
  zhuoran
cheng  liu
chih-yao  ma
zijian  he
chia-wen  kuo
kan  chen
peizhao  zhang
bichen  wu
zsolt  kira
  unbiased
dongdong  yu
yaqian  li
yandong  guo
yuchen  guo
matthias  minderer
alexey a gritsenko
neil  houlsby
li  chen
penghao  wu
kashyap  chitta
bernhard  jaeger
andreas  geiger
zhaowei  cai
gukyeong  kwon
avinash  ravichandran
er- 675  han bas
zhuowen  tu
rahul  bhotika
stefano  soatto
chuang  lin
peize  sun
yi  jiang
ping  luo
lizhen  qu
gho- 680  lamreza haffari
zehuan  yuan
jianfei cai learning
]  kj
salman  khan
shahbaz  khan
zhipeng  bao
pavel  tokmakov
allan  jabri
yu-xiong  wang
sagar  vaze
kai  han
andrea  vedaldi
andrew  zisser
moin  zhong
elisa ricci nabi
xiuye  gu
tsung-yi  lin
weicheng  kuo
yin  cui
harold  liunian
pengchuan  li
haotian  zhang
  zhang
mingfei  gao
zizhao  zhang
guo  yu
sercan ö arık
jiacheng  zhang
xiangru  lin
wei  zhang
kuo  wang
wayne  zhang
xiaoliang  dai
junjiao  tian
pengzhen  ren
yun  xiao
xiaojun  chang
po-yao  huang
zhihui  li
b  gupta
xiaojiang  chen
xin  wang
sean  segal
nishanth  kumar
sergio  casas
wenyuan  zeng
max  jiang
mahyar  najibi
charles r qi
yin  zhou
peng  ning
xun  zhao
yu  li
sheng-jun  huang
Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-03-28 17:43:14

Abstract: Autonomous vehicle (AV) systems rely on robust perception models as a cornerstone of safety assurance. However, objects encountered on the road exhibit a long-tailed distribution, with rare or unseen categories posing challenges to a deployed perception model. This necessitates an expensive process of continuously curating and annotating data with significant human effort. We propose to leverage recent advances in vision-language and large language models to design an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. This process operates iteratively, allowing for continuous self-improvement of the model. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.

Main paper: 
 introduction: 
 Autonomous vehicles (AVs) operate in an ever-changing world, encountering diverse objects and scenarios in a longtailed distribution. This open-world nature poses a significant challenge for AV systems since it is a safety-critical application where reliable and well-trained models must be deployed. The need for continuous model improvement becomes apparent as the environment evolves, demanding adaptability to handle unexpected events. Despite the wealth of data collected on the road every minute, its effective utilization remains low due to challenges in discerning which data to leverage. While solutions exist for this in industry [1, 2], they are often trade secrets and presumably require significant human effort. Hence, developing a comprehensive automated data engine can lower entry barriers for the AV industry.Designing automated data engines can be challenging, but the existence of Vision-Language Models (VLMs) and Large Language Models (LLMs) allows new avenues to these hard problems. A traditional data engine can be bro-  
related works: 
 Data Engine for Autonomous Vehicles (AV) Exploiting large-scale data collected by AV is crucial to speed up the iterative development of the AV system [13] . Existing literature mostly focuses on developing general [14, 15] learning engines or specific [16] data engines, and most of them [17, 18] mainly focus on the model training part.However, a fully functional AV data engine requires issue identification, data curation, model retraining, verification, etc. A thorough examination reveals a lack of systematic research papers or literature that delves deeply into AV data engines in academia, where a recent survey [13] also underscores the lack of study in this context. On the other hand, existing solutions [1, 2] for AV data systems mainly rely on the design of data infrastructure and still need lots amount of human effort and intervention, thus limiting their maintenance simplicity, affordability, and scalability. In contrast, the present paper exploits the burgeoning progress of vision language models (VLMs) [19] [20] [21] to design our data engine, where their strong open-world perception capability largely improves our engine's extendability and makes it more affordable to scale up our AVs on detecting novel categories. To our best knowledge, this paper is also the first work that provides a systematic design of data engines for AVs with the integration of VLMs.Novel Object Detection Conventional 2D object detection has made enormous progress [22, 23] On the other hand, limited research has focused on novel object detection for AVs. This is especially crucial because a false-negative detection of unseen objects may result in fatal consequences for AVs. Existing OVOD methods mostly benchmark on datasets of general objects [42, 44] while putting little attention on AV datasets [45] [46] [47] [48] [49] [50] . Different from the pursuit of generality in OVOD, perception in AVs has its domain concerns oriented from the image-capturing process by on-car cameras and the object categories due to the scene prior (e.g., road/street objects), which demands task-specific design to enable efficient and scalable system to iteratively enhance AVs on detecting novel objects during its lifecycle. To strike a better trade-off between specificity and generality, our proposed AIDE iteratively extends the closed-set detector's label space so that we can retain decent performance on both novel and known categories for better detection.Semi-Supervised Learning (Semi-SL) and Active Learning (AL) As AVs keep collecting data in operation, a native solution to enable novel category detection is to manually identify the novel category over a collected unlabeled data pool, label them, and then train the detector. Semi-SL [5, 6, 9, 51-54] and AL [8, 10, 18, 55-58] seem to help as they require only a small amount of labeled data to initialize the training. However, labeling even a small amount of data for novel categories will be challenging and costly when given a vast amount of unlabeled data [8, 56, 59-61] by AVs. Moreover, both Semi-SL and AL assume that the labeled and unlabeled data come from the same distribution [51, 62, 63] and share the same label space. However, this assumption does not hold when new categories emerge, inevitably leading to changes in the label space. Naive fine-tuning of the detector only on the novel categories will lead to catastrophic forgetting [64] [65] [66] of known categories learned previously. However, Semi-SL methods for object detection do not consider continual learning, while existing continual semi-supervised learning methods [67] [68] [69] [70] are also specific to image classification, which is not applicable for object detection. Tab. 4). Instead, our Data Feeder utilizes the VLMs to perform text-guided image retrieval on the image pool to query for relevant images related to the novel categories. 
method: 
 We consider BLIP-2 [21] given its strong open-vocabulary text-guided retrieval capability. Precisely, given an image and a specific text input, we measure the cosine similarity between their embeddings from BLIP-2 and only retrieve the top-k images for further labeling in our Model Updater.For the text prompt, we experiment with common prompt engineering practice [71] and find that a template like "An image containing {}" can readily provide good precision and recall for the novel categories in practice. Fig. 4 shows some examples of retrieved images.The goal of our Model Updater is to make our detector learn to detect novel objects without human annotations. To this end, we perform pseudo-labeling on the images queried by the Data Feeder and then use them to train our detector.Motivated by the previous success in pseudo-labeling for object detection [41] , we designed our pseudo-labeling procedure with two parts: box and label generation. Such a two-stage framework can help us better dissect the issue of pseudo-label generation and improve the label generation quality. Box generation aims to identify as many object proposals in the image as possible, i.e., high recall for localizing novel categories, to guarantee a sufficient number of candidates for label generation. To this end, region proposal networks (RPN) pretrained with closed-set label space [41] and the open vocabulary detectors (OVD) [11] can be considered, where the former can localize generic objects while the latter can perform text-guided localization. We observe that the SOTA OVD, i.e., OWL-v2 [11] that has been selftrained on web-scale datasets [43] , exhibits a higher recall to localize novel categories compared to the RPN. We conjecture that proposals of RPN may be readily biased toward the pre-trained categories.Thus, we choose OWL-v2 as our zero-shot detector to get the box proposal. Specifically, we append the novel category name provided by Issue Finder to our existing label space and create the text prompts, then we prompt the OWL-v2 to inference on an image. Note that we only retain the box proposals and remove the labels from the OWL-v2's predictions. This is because we empirically find that OWL-v2 can not achieve reliable precision on the novel categories presented in AV datasets, e.g., less than 10% AP averaging over the novel categories in AV datasets [45, 50] , while it can get >40% AP on novel categories of LVIS [42] datasets. We conjecture that this performance degradation may come from the domain shift of the images collected in the AV scenario. For instance, the pretraining data of OWL-v2 mainly comes from the daily image captured by humans from a close distance. However, the street objects are always small in the image due to their long distance from the on-car camera, and the aspect ratio of the image presented in AV datasets is relatively large, making OWL-v2 hard to classify the correct label of the object proposals.Motivated by this insight, we consider conducting another round of label filtering with CLIP [71] to purify the predictions of the OWL-v2 and generate the pseudo labels. Specifically, we pass the box prediction by OWL-v2 to the original CLIP model [71] for zero-shot classification (ZSC), as shown in Fig. 5 . To mitigate the potential issue of the aspect ratio mentioned above, we increase the box size to crop the image and then send the cropped image patch to CLIP for ZSC. This can involve more scene contextual information to help the CLIP better differentiate between the novel and known categories. Regarding the label space for CLIP to do zero-shot classification, we first create a base label space, which is a combination of the label space from datasets we have pre-trained and COCO [44] , to ensure that we can mostly cover daily objects that would probably be present in the street. The base label space will automatically extend when the Issue Finder identifies novel categories not in the base label space.Directly training our existing detector on the pseudo-labels of novel categories presents a challenge, as these labels may lead the detector to overfit and catastrophically forget the known categories. The issue arises because the unlabeled data can contain both novel and known categories that the detector has previously learned. Without labels for those known categories and only having labels for novel categories, the model may incorrectly suppress predictions for known categories, focusing solely on predicting novel categories. As training progresses, the known categories gradually fade from memory. To address this issue, we draw inspiration from existing self-training strategies and include the pseudo-labels of the known categories that have been trained on. Consequently, our existing detector is updated with the pseudo-labels of both novel and known categories.To obtain pseudo-labels for the known categories, we first Methods for Comparison To our knowledge, there is little work about the systematic design for automatic data engines tailored to the novel object detection for AV systems.Thus, it is hard to identify a comparable counterpart for our AIDE. To this end, we dissect our evaluation into two parts:(1) compare to alternative detection methods and learning paradigms on the performance of novel object detection;(2) ablation study and analysis of each step of the automatic data engine. For (1), as our AIDE can enable the detector to detect novel categories without any labels, we first compare our method with the zero-shot OVOD methods on novel categories' performance. Moreover, to show the efficiency and effectiveness of our AIDE in reducing label cost, we further compare with semi-supervised learning (Semi-SL) and fully supervised learning that trains the detector with different ratios of ground-truth labels. Specifically, we compare system, we are interested in how our engine can strike a balance between the cost of searching and labeling images and the performance on novel object detection. We measure the human labeling costs [72] and also the GPU infer- Experimental Details Given the real-time requirement for inference, we choose the Fast-RCNN [22] as our detector instead of OVOD methods like OWL-ViT [4] as the FPS for OWL-ViT is only 3. We run our AIDE to iteratively scale up its capability of detecting novel objects. For multi-dataset training, we follow the same recipe from [74] . For each novel category, we train for 3000 iterations with the learning rate of 5e-4, and we use the same hyperparameter for all the comparison methods if they require training. We attach our full experimental details in the supplementary material. 
overall performance: 
 In this section, we provide the overall performance of novel object detection after running our AIDE for a complete cycle. Our results are shown in Fig. 1 In the following subsections, we will dissect each part of our AIDE to validate our design choice.As mentioned in Section 3.1, the main goal of our IssueFinder is to automatically identify categories that do not exist in our label space. To this end, we evaluate the success If the prediction is incorrect, we can ask annotators to label the images, which are used to further improve the detector. To this end, we randomly select 10 LLM-generated descriptions, for which top-1 retrieved image (based on BLIP-2 cosine similarity) was predicted incorrectly, and labeled these 10 images to update our detector by Model Updater.As shown in Fig. 7 , after updating the model with a few human supervisions, our model can successfully predict the object, e.g., the motorcyclist in the figure, which was missdetected before. For the overall performance, we achieve 14.2% AP on novel categories, which improves our zeroshot performance by 2.2% AP, while the total cost only increases to $1.59. This is still less than $2.1 of semisupervised learning, and our AP for known categories remains 26.6% after Verification. 
conclusion: 
 We proposed an Automatic Data Engine (AIDE) that can automatically identify the issues, efficiently curate data, improve the model using auto-labeling, and verify the model through generated diverse scenarios. By leveraging VLMs and LLMs, our pipeline reduces labeling and training costs while achieving better accuracies on novel object detection.The process operates iteratively which allows continuous improvement of the model, which is critical for autonomous driving systems to handle expected events. We also establish a benchmark for open-world detection on AV datasets, demonstrating our method's better performance at a reduced cost. One of the limitations of AIDE is that VLM and LLM can hallucinate in issue finder and verification. Despite the effectiveness of AIDE, for a safety-critical system, some human oversight is always recommended. 
 
 Appendix: return a synonym to the sought label of the novel category 216 



Reviews: review: title: title: Review
main_review: main_review: This paper proposes an impressive and comprehensive Automatic Data Engine (AIDE) that leverages vision-language models (VLMs) and large language models (LLMs) to automate the entire data engine pipeline for open-world object detection in autonomous driving. The authors have put together a thoughtful and well-designed system that addresses the key challenges in this domain.

Strengths:

The authors have identified a crucial problem in autonomous driving - the need for continuous model improvement to handle the long-tailed distribution of objects encountered on the road. Their proposed AIDE system addresses this in a systematic and automated way.
The use of VLMs and LLMs to power the various components of the data engine (Issue Finder, Data Feeder, Model Updater, Verification) is a clever and effective application of these powerful models.
The detailed experiments and analysis demonstrate the effectiveness of AIDE, showing significant improvements over baselines like open-vocabulary object detection and semi-supervised learning, while reducing labeling costs.
The proposed benchmark for open-world detection on AV datasets is a valuable contribution that can enable further research in this direction.

Weaknesses:

The paper could benefit from a more thorough discussion of the limitations and potential failure modes of the AIDE system, especially around the reliability of the VLM and LLM components.
While the authors mention the need for some human oversight for a safety-critical system like autonomous driving, more details on the human-in-the-loop aspects could strengthen the paper.
The connection between the individual components of AIDE and how they work together cohesively could be explained in a more intuitive way.

Recommendation: Overall, this is a strong and well-executed paper that makes a significant contribution to the field of autonomous driving. I recommend acceptance, subject to addressing the minor points raised above in the camera-ready version.