DriveLM: Driving with Graph Visual Question Answering

Authors: venice erin liong
qiang  xu
anush  krishnan
yuxin  pan
li  chen
penghao  wu
kashyap  chitta
bernhard  jaeger
andreas  geiger
aditya  prakash
zehao  yu
ing  ieee t-pami
daniel  dauner
marcel  hallgarten
minghua  ma
wei  zhang
si  qin
saravan  rajmohan
qing- 313 wei  lin
dongmei  zhang
danny  driess
fei  xia
mehdi s m sajjadi
  corey
edward j hu
yelong  shen
phillip  wallis
allen- 323  zhu
yuanzhi  li
shean  wang
lu  wang
weizhu  chen
  corl
shengchao  hu
hongyang  li
  junchi
dacheng tao yan
  eccv
yihan  hu
jiazhi  yang
keyu  li
chonghao  sima
Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-03-28 09:15:39

Abstract: We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems to boost generalization and enable interactivity with human users. While recent approaches adapt VLMs to driving via single-round visual question answering (VQA), human drivers reason about decisions in multiple steps. Starting from the localization of key objects, humans estimate object interactions before taking actions. The key insight is that with our proposed task, Graph VQA, where we model graph-structured reasoning through perception, prediction and planning question-answer pairs, we obtain a suitable proxy task to mimic the human reasoning process. We instantiate datasets (DriveLM-Data) built upon nuScenes and CARLA, and propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving. The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides a challenging benchmark for this task. Our DriveLM-Agent baseline performs end-to-end autonomous driving competitively in comparison to state-of-the-art driving-specific architectures. Notably, its benefits are pronounced when it is evaluated zero-shot on unseen objects or sensor configurations. We hope this work can be the starting point to shed new light on how to apply VLMs for autonomous driving. To facilitate future research, all code, data, and models will be available to the public.

Main paper: 
 introduction: 
 Current Autonomous Driving (AD) stacks are still lacking crucial capabilities [4, 5] . One key requirement is generalization, which involves the ability to handle unseen scenarios or unfamiliar objects. A secondary requirement pertains to the interaction of these models with humans, highlighted for example by EU regulations that mandate explainability in deployment [1] . Furthermore, unlike today's AD models, humans do not navigate based on geometrically precise bird's-eye view (BEV) representations [6, 13, 16] . Instead, humans implicitly perform object-centric perception, prediction, and planning (which we refer to as P 1-3 ): a rough identification and localization of key objects, followed by reasoning about their possible movement and aggregation of this information into a driving action [22, 27] .Simultaneously, another field has been forging ahead:Vision-Language Models (VLMs) [17, 19, 30, 34] . These models have several strengths. First, they hold a base understanding of the world from internet-scale data that could potentially facilitate generalization for planning in AD. In fact, this sort of generalization has already been achieved by VLMs for simpler robotics tasks [9, 35] . Second, the use of language representations as an input and output offers a platform for human-friendly interaction with these models, unlike bounding boxes or trajectories that are more common to current methods [7, 12, 18, 25] . Finally, VLMs are able to make decisions in multiple steps linked by logical reasoning [2, 8, [31] [32] [33] 35] . Importantly, even though they reason in multiple separate steps, VLMs are end-to-end differentiable architectures, a characteristic that is highly desirable for autonomous driving [4] .to AD systems falls into two categories: scene-level or single object-level Visual Question Answering (VQA). Scenelevel VQA refers to the task of describing the driving behavior by one or two supporting reasons, e.g., "The car is moving into the right lane because it is safe to do so." [14, 15] . Single object-level VQA formulates the understanding of the ego vehicle's response to a single object by a chain of QAs in the form of "what-which-wherehow-why", e.g., "The ego vehicle stops because there is a pedestrian in a white shirt crossing the intersection in front of the ego vehicle and it does not want to crash into the pedestrian." [21, 24, 26] . Unfortunately, neither of these paradigms provides a suitable proxy task to mimic the P 1-3 reasoning process in humans, who consider multiple objects and reason about each in multiple steps. Therefore, in this paper, we propose a new task, along with corresponding datasets and a baseline model architecture (Fig. 1 ).Task. Graph Visual Question Answering (GVQA) involves formulating P 1-3 reasoning as a series of questionanswer pairs (QAs) in a directed graph. Its key difference to the aforementioned VQA tasks for AD is the availability of logical dependencies between QAs which can be used to guide the answering process. GVQA also encompasses questions regarding behavior and motion planning, with dedicated metrics (details in Section 2).Datasets. DriveLM-nuScenes consist of annotated QAs, arranged in a graph, linking images with driving behavior through logical reasoning. In comparison to existing bench-marks, they provide significantly more text annotations per 089 frame (Fig. 2 ). We pair these training datasets with chal-090 lenging test data for evaluating zero-shot generalization.Model. DriveLM-Agent employs a trajectory tokenizer 092 that can be applied to any general VLM [17, 19, 23, 34] , 093 coupled with a graph prompting scheme that models logi- Moving state?What to do?Always Brake.Focus on diversity and avoid fixed formulaic answers.What to do?Human AnnotatorConsistent with answers of previous and next frames? The combination of the speed and steering categories for a trajectory form its behavior category as B = (B sp , B st ).While we use a simple definition of B as a starting point for research on driving with VLMs, we note that our formulation supports the incorporation of more abstract behaviors such as a lane changes or overtaking. 
drivelm-data: 
 We tions with visual question answering. arXiv preprint 
 
 Appendix:  



Reviews: review: title: title: Generally good-quality paper with some minor concerns.
main_review: main_review: Summary:

This work introduces a new framework which integrates vision-language models into end-to-end autonomous driving systems. Through a task called Graph Visual Question Answering, this task mimics human decision-making processes in driving by using a series of interconnected question-answer pairs that logically depend on each other across object and task levels. These levels encompass perception, prediction, planning, behavior, and motion.

Strengths:

The introduction of GVQA as a task for autonomous driving systems is innovative. Graph decomposition allows sufficient explainability in the context of end-to-end autonomous driving.

The DriveLM-Data contains rich annotations and provides a robust platform for the training and validation of similar models.

The proposed framework has better zero-shot generalization capability across different sensor configurations.

Weaknesses:

The framework is single-framed, however, many SOTA methods on NuScene datasets, including UniAD, show that multi-frame methods are generally better since they could take account into the historical information. The complexity of the proposed framework (graph-based VQA) might grow exponentially in multi-frame context, make it challenging to scale up to video-based framework.

"Interestingly, in the experimental setting of Table 1 which does not involve generalization, the Chain and Graph versions of DriveLM-Agent do not provide any further advantage over no context." Does the paper have any explanation for this?

Suggestions:

It will be great to have some discussion on how the proposed work's method is different comparing with other existing LLM-powered driving models (e.g., DriveGPT4, GPT-Driver, etc.).