DriveLM:
Driving with Graph Visual Question Answering
Chonghao Sima4,1∗ Katrin Renz2,3∗ Kashyap Chitta2,3 Li Chen4,1
Hanxue Zhang1 Chengen Xie1 Jens Beißwenger2,3 Ping Luo4
Andreas Geiger2,3† Hongyang Li1†
OpenDriveLab, Shanghai AI Lab, China
2
University of Tübingen, Germany
3
Tübingen AI Center, Germany
4
University of Hong Kong, China
simachonghao@pjlab.org.cn
katrin.renz@uni-tuebingen.de
1

Graph Visual Question Answering

Q – Question; A – Answer;

DriveLM-Data

Object Level

𝑪: Context

𝑷𝟏 - Perception

𝑷𝟏 - Perception

Q: What are the objects worth noting in
the current scenario?

𝑷𝟐 - Prediction

A: There is a van, a sedan and apedestrian in front of the ego car.

𝑷𝟑 - Planning

𝑪: 𝑷𝟏

Task Level

𝑷𝟐 - Prediction

𝑩 - Behavior
𝑴 - Motion

Q: Where might the van, the sedan andthe pedestrian move in the future? 𝑪: 𝑷𝟏

QA pairs

A: Those objects may move to the rightlane, cross the ped-crossing, etc..
𝑪: 𝑷𝟐
𝑷𝟑 - Planning

Vision Language Models• Answer

Q: What are the safe actions of the ego
car considering those objects? 𝑪: 𝑷𝟐
A: The ego car should wait for thepedestrian and the sedan, then turn right.

• Behavior• Motion

Question with

Context

DriveLM-Agent
Graph Visual QA
Q: What are the important objects in the current scene?
Predicted Answer
There is a stopped
pickup to the front of
the ego car.

DriveLM-Metrics

Generalization

Trajectory Prediction

Ground Truth Answer
GPT
Score

nuScenes

Predicted Trajectory

There is one moving
truck to the front of
the ego car.

Ground Truth Trajectory
Displacement
Error

Waymo
Unseen scenes&
sensor setting

Prompt: Predict the motion of the ego vehicle.
w/o pedestrianw/ pedestrian
Unseen objects

Fig. 1: We present DriveLM: A new task, dataset, metrics, and baseline for end-toend autonomous driving. Inspired by [10], DriveLM considers Graph Visual Question Answering (GVQA), where QA pairs are interconnected via logical dependencies at the object-level, i.e., interactions between object pairs, and the task-level,
e.g., perception → prediction → planning → behavior (discretized action described innatural language) → motion (continuous trajectory). We propose DriveLM-Data fortraining DriveLM-Agent, a baseline for GVQA. We validate its effectiveness using
the DriveLM-Metrics on challenging settings requiring zero-shot generalization.
Abstract. We study how vision-language models (VLMs) trained on
web-scale data can be integrated into end-to-end driving systems to∗Equal contribution. † Project lead.

2

C. Sima et al.
boost generalization and enable interactivity with human users. While
recent approaches adapt VLMs to driving via single-round visual question answering (VQA), human drivers reason about decisions in multiple
steps. Starting from the localization of key objects, humans estimate object interactions before taking actions. The key insight is that with our
proposed task, Graph VQA, where we model graph-structured reasoning through perception, prediction and planning question-answer pairs,
we obtain a suitable proxy task to mimic the human reasoning process. We instantiate datasets (DriveLM-Data) built upon nuScenes and
CARLA, and propose a VLM-based baseline approach (DriveLM-Agent)
for jointly performing Graph VQA and end-to-end driving. The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides achallenging benchmark for this task. Our DriveLM-Agent baseline performs end-to-end autonomous driving competitively in comparison tostate-of-the-art driving-specific architectures. Notably, its benefits arepronounced when it is evaluated zero-shot on unseen sensor configurations. Our question-wise ablation study shows that the performance gain
comes from the rich annotation of prediction and planning QA pairs in
the graph structure. All data, models and an official evaluation server
are available at https://github.com/OpenDriveLab/DriveLM.
Keywords: Vision Language Model · End-to-end Autonomous Driving

1

Introduction

Current Autonomous Driving (AD) stacks are still lacking crucial capabilities [10,15]. One key requirement is generalization, which involves the ability to handle
unseen scenarios or unfamiliar sensor configurations. A secondary requirement
pertains to the interaction of these models with humans, highlighted for example
by EU regulations that mandate explainability in deployment [3]. Furthermore,unlike today’s AD models, humans do not navigate based on geometrically precise bird’s-eye view (BEV) representations [17,32,46]. Instead, humans implicitly
perform object-centric perception, prediction, and planning (which we refer to
as P1−3 ): a rough identification and localization of key objects, followed by reasoning about their possible movement and aggregation of this information intoa driving action [60, 76].
Simultaneously, another field has been forging ahead: Vision-Language Models (VLMs) [47,54,90,106]. These models have several strengths. First, they holda base understanding of the world from internet-scale data that could potentiallyfacilitate generalization for planning in AD. In fact, this sort of generalization
has already been achieved by VLMs for simpler robotics tasks [23, 108]. Second, the use of language representations as an input and output offers a platform for human-friendly interaction with these models, unlike bounding boxes
or trajectories that are more common to current methods [19, 31, 49, 70]. Finally, VLMs are able to make decisions in multiple steps linked by logical rea-DriveLM: Driving with Graph Visual Question Answering

3soning [5, 21, 93, 95, 103, 108]. Importantly, even though they reason in multiple
separate steps, VLMs are end-to-end differentiable architectures, a characteristicthat is highly desirable for autonomous driving [10].
Recent work towards enabling the application of VLMs to AD systems falls
into two categories: scene-level or single object-level Visual Question Answering(VQA). Scene-level VQA refers to the task of describing the driving behavior by
one or two supporting reasons, e.g., “The car is moving into the right lane because
it is safe to do so.” [41,42]. Single object-level VQA formulates the understandingof the ego vehicle’s response to a single object by a chain of QAs in the formof “what-which-where-how-why”, e.g., “The ego vehicle stops because there is apedestrian in a white shirt crossing the intersection in front of the ego vehicle
and it does not want to crash into the pedestrian.” [56, 67, 71]. Unfortunately,
neither of these paradigms provides a suitable proxy task to mimic the P1−3
reasoning process in humans, who consider multiple objects and reason about
each in multiple steps. Therefore, in this paper, we propose a new task, alongwith corresponding datasets and a baseline model architecture (Fig. 1).
Task. Graph Visual Question Answering (GVQA) involves formulating
P1−3 reasoning as a series of question-answer pairs (QAs) in a directed graph.
Its key difference to the aforementioned VQA tasks for AD is the availabilityof logical dependencies between QAs which can be used to guide the answering process. GVQA also encompasses questions regarding behavior and motion
planning, with dedicated metrics (details in Section 2).
Datasets. DriveLM-nuScenes and DriveLM-CARLA consist of annotated
QAs, arranged in a graph, linking images with driving behavior through logicalreasoning. In comparison to existing benchmarks, they provide significantly more
text annotations per frame (Table 1 and Fig. 2). Furthermore, as an integral
component of DriveLM-CARLA, we build PDM-Lite [4], the first working rulebased expert algorithm for CARLA Leaderboard 2.0. We pair these training
datasets with challenging test data for evaluating zero-shot generalization.
Baseline. DriveLM-Agent employs a trajectory tokenizer that can be applied
to any general VLM [47,54,65,106], coupled with a graph prompting scheme thatmodels logical dependencies as context inputs for VLMs. The result is a simple
methodology to effectively repurpose VLMs for end-to-end AD (Section 3).
Our experiments provide encouraging results. We find that GVQA on DriveLM is a challenging task, where current methods obtain moderate scores andbetter modeling of logical dependencies is likely necessary to achieve strong
QA performance. Even so, DriveLM-Agent already performs competitively tostate-of-the-art driving-specific models [32] when tested in the open-loop planning setting, despite its task-agnostic architecture. Furthermore, employing agraph structure improves zero-shot generalization, enabling DriveLM-Agent tocorrectly handle unseen deployment on Waymo data [77] after training only
on nuScenes [6] data. Finally, with a question-wise analysis we find that the
QA pairs from the prediction and planning stages help the final driving decision
most. From these results, we believe that improving GVQA holds great potential
towards building autonomous driving agents with strong generalization.

4

C. Sima et al.
DriveLM-nuScenes

DriveLM-CARLA
Selected Key Frames & Key Objects

Town Settings

Configure

Execute Expert

Simulator
Generate

A Batch of Videos

Checklist• Factual Questions

Human
Annotator

QuestionsMoving State/Type…

vSample 10%

Error Feedback
What to do?

Refer

Always Brake.

Quality Check· Sensor Data
· Object Info
· Environment

What to do?

Human annotatedFocus on diversity and avoid
fixed formulaic answers.

Rule-based QA Generation

Quality Check

Moving state?

Object Interaction

Qualified Data
Rule-base generationfrom GT of nuScenes

Parked.

Objects?

Where?

Maroon car

Right to ego

Action?
Follow slowly

DriveLM: Task, Data, Metrics

Human drivers usually decompose their decision-making process into distinct
stages that follow a logical progression which encompasses the identificationand localization of key objects, their possible future action and interaction, andego planning based on all this information [27, 55]. This inspires us to propose
the GVQA as the critical ingredient of DriveLM, which serves as a suitableproxy task to mimic the human reasoning process. Within this section, we illustrate the formulation of the GVQA task (Section 2.1), introduce DriveLM-Data(Section 2.2) to exemplify the instantiation of GVQA using prominent driving
datasets, and overview the DriveLM-Metrics used for evaluation (Section 2.3).
To encourage further research in this direction, an official evaluation server (witha public leaderboard) will be set up to benchmark different methods and discover
more insights about combining language models with autonomous driving.
2.1…
Possible
Attention

Graph Building

Fig. 2: Annotation Pipeline: In DriveLM-nuScenes, we adopt a semi-rule-based QAlabeling pipeline, where both the ground truth annotation in nuScenes/OpenLane-V2
and feedback from human annotators are used. A critical part of our pipeline is themulti-round quality check, which guarantees high data quality at reasonable costs. In
DriveLM-CARLA, we meet the same standards while exploiting a fully rule-based QAlabeling pipeline instead, using a new expert algorithm called PDM-Lite.

2

Distributionof Questions

Planning/Importance …

Factual Questions

Go Straight

ImportanceRanking

Key Frames
Selection

Possible
Collision

Qualified Data
Open-ended Questions

Safe / Unsafe
Action List

Example Image

Consistent with answers ofprevious and next frames?
• Open-ended Questions

ObjectPo

Planning and Reasoning

DriveLM-Task: GVQA

We organize all the question-answer pairs (QAs) for an image frame into a graph.
We use the terminology “graph-structured” to refer to a directed acyclic graph(DAG), e.g., the current question can get context from (multiple) parent andgrandparent nodes. The graph G = (V, E) contains a set of vertices V , where
each vertex represents a QA pair v = (q, a) associated with one or more key
objects in the scenario. The key difference between GVQA and ordinary VQA
is that the QAs in GVQA have logical dependencies, which we formulate as theedges between the vertices. E ⊆ V × V , is a set of directed edges, where eachedge e = (vp , vc ) connects the parent QA and the child QA. We formulate thePerception

Vis…
Prediction

PlaDriveLM: Driving with Graph Visual Question Answering

5

Table 1: Comparison of DriveLM-nuScenes & -CARLA with Existing
Datasets. DriveLM-Data significant advances annotation quantity, comprehensiveness (covering perception, prediction and planning), and logic (chain to graph).
† full dataset, ‡ keyframe dataset, ∗ semi-rule-based labeling (w/ human annotators),∗∗
fully-rule-based (no human annotators). - means publicly unavailable.
DatasetAvg. QA
SourceDataset # Frames per Frame Perception Prediction Planning Logic

nuScenes-QA [67]nuPrompt [98]
HAD [41]
BDD-X [42]LingoQA [59]
DRAMA [56]Rank2Tell [71]nuScenes
nuScenes
HDD
BDD
LingoQA
DRAMA
Rank2Tell

34,14934,14925,54926,22828,00017,785
5,800

13.5
1.0
1.8
1
15.3
5.8
-

460k∗∗
35k∗
25k
26k
85k
-

✗
✗
✗
✗
✗
✗

✗
✗
20k
✗
17k
-

None
None
None
None
None
Chain
Chain

DriveLM-nuScenes nuScenes
DriveLM-CARLA† CARLA
DriveLM-CARLA‡ CARLA

4,871
64,285
5,721

91.4
24.4
24.8

144k∗
697k∗∗
63k∗∗

153k
311k∗∗
28k∗∗

146k
558k∗∗
51k∗∗

Graph
Graph
Graph

edge set E by incorporating two dimensions: object-level and task-level edges.
At the object level, we construct the logical edges e ∈ E to represent the impact
of interactions between different objects. For example, the planning QA node
for the sedan is influenced by the perception QA node of the pedestrian in theillustration from Fig. 1 (center). At the task-level, we establish the logical edgese ∈ E to capture the logical chain of different reasoning stages:– Perception (P1 ): identification, description, and localization of key objects
in the current scene.
– Prediction (P2 ): estimation of possible action/interaction of key objects
based on perception results.
– Planning (P3 ): possible safe actions of the ego vehicle.
– Behavior (B): classification of driving decision.
– Motion (M ): waypoints of ego vehicle future trajectory.
The concepts of perception, prediction, and planning (P1−3 ) are similar to
those in end-to-end AD [10], while the concepts of motion and behavior are basedon the ego vehicle future trajectory. Specifically, we define the motion M as theego vehicle future trajectory, which is a set of N points with coordinates (x, y)in bird’s-eye view (BEV), denoted as M = {(x0 , y0 ), (x1 , y1 ), ..., (xN , yN )}. Eachpoint is the offset between the future position and the current position by a fixed
time interval. Then, the distance for x, y at each time interval is computed as:  {\{x,y\}}_{\text {dist}} = \{ (\delta _{x,1},\delta _{y,1}),...,(\delta _{x,N},\delta _{y,N}) \}, \label {eq:xy_dist} (1)

where δx,i = xi − xi−1 and δy,i = yi − yi−1 , for i = 1, 2, . . . , N. The goal of the
behavior representation is to serve as an interface from P1−3 to M . To obtaina behavior representation, we map the mean of xdist and ydist to one of thepredefined bins, where each bin corresponds to a category in either speed or
steering. These are denoted as Bsp and Bst respectively. In this work, we consider
5 bins:  {\{x,y\}}_{dist} = (({\{x,y\}}_1 - {\{x,y\}}_0), ..., ({\{x,y\}}_N - {\{x,y\}}_{N-1})) (2)

The mean of xdist and ydist are mapped to one of the predefined bins, where
each bin corresponds to a category in either speed or steering, noted as Bspeed
and Bsteer respectively. Finally, the speed and steering categories for this trajectory form the behavior category as (Bspeed , Bsteer ). We compare them with the
behaviors of our DriveLM-Agent outputs and calculate the related accuracy.

DriveLM: Driving with Graph Visual Question Answering

D.3

41

Motion Task Metrics

For measuring the performance of the motion stage, we use standard metrics fromthe nuScenes and Waymo benchmarks: average and final displacement error,(ADE, FDE), and the collision rate of the predicted trajectory.
ADE stands for Average Displacement Error, indicating the average L2 distance between the predicted trajectory and the ground truth trajectory over all
predicted horizons. It is the average of the errors at 1st , 2nd and 3rd second.
FDE stands for Final Displacement Error, which measures the Euclidean distance between the predicted endpoint and the true endpoint at the last predicted
step (the 3rd second).
Collision Rate accounts for the ratio of how many test frames the predictedtrajectory collides with objects in over all test frames. The number reported inTable 2 of the main paper is the average of the collision rate at 1st , 2nd and 3rd
second.
Note that the calculation of ADE, FDE and Collision Rate follows thesetting used in UniAD [32] but not ST-P3 [31]. For example, in terms of the
FDE and the collision rate at 3rd second, the UniAD setting will consider theerror/collision rate at only this timestep, while the ST-P3 setting will consider
the error/collision rate as an average over 0.5, 1, 1.5, 2, 2.5, 3 seconds. For moredetails, please refer to the UniAD repo discussion. Additionally, please note thaterrors reported on the full nuScenes validation dataset (in prior work) is notdirectly comparable to results reported on the DriveLM-nuScenes val split, achallenging subset of this consisting of only keyframes with intention changes.

E

DriveLM-AgentIn this section, we introduce the details of DriveLM-Agent, including the graphprompting scheme and the trajectory tokenization process.
E.1

Prompting with Context

In terms of the implementation, the content of context differs during the training and inference of DriveLM-Agent, following the teacher-forcing setting [44,82]
generally adopted in recurrent networks. During training, for each edge e ∈ E in
the frame, we pick the child QA. The child questions in the edges are appended
with the ground truth parent QA as the context. All QA pairs are used during training, including those without context. The objective used is next token
prediction, the standard approach for language modeling. During inference, themodel is applied interactively in multiple rounds to get the required context
predictions as inputs for each child question. Specifically, the model is promptedwith the five stages of questions in the sequential order of P1 , P2 , P3 , B, M . In
this order, the model can only infer the questions in the succeeding stages aftergetting the predicted answer from the preceding stages.

42

C. Sima et al.

Fig. 12: Detailed architecture of DriveLM-Agent. An inverse mapping is designed to embed the real-world ego-vehicle trajectory into the token space of blip-2flan-t5-xl.

E.2

Trajectory Tokenization Details

To generate action sequences (i.e., ego future trajectories) directly with thelanguage model we use for building the graph, we adopt the method of RT2 [108]. This process entails the discretization and tokenization of the continuous
trajectory.
Initially, we analyze the distribution of the future trajectories within the
nuScenes dataset. To effectively convert the continuous (x, y) coordinate space
into a discrete set of actions, we partition each coordinate axis into 256 discreteintervals. This granularity ensures a sufficient level of detail while maintaining
a manageable number of tokens for the language model.
Each discretized bin corresponds to a unique token within the vocabulary of
the language model. We extract the token identifiers (IDs) for numeric tokens
within the vocabulary. To ensure coherence and preserve the ability to express
numerical values, we omit single-digit tokens from this mapping process. Out of
the remaining numeric tokens, a subset of 256 token IDs is selected to represent the trajectory data. In addition to these, we introduce two special tokens
designated for marking the start and end of a trajectory sequence – the startof-trajectory (SOT) token and the end-of-trajectory (EOT) token, respectively.
This tokenization scheme enables us to encode complex trajectory information asa sequence of tokens that a language model can process. Using this mapped vocabulary, the language model can generate predicted future trajectory sequencesby outputting a series of tokens, which are then translated back into the coordinate space.

F

Experiments

In this section, we introduce the details of experiments, including implementation
details of each subsection in Section 4 in the main paper, more metrics on the
VQA part, more ablation and comparison on computational complexity.

DriveLM: Driving with Graph Visual Question Answering

F.1

43

Implementation Details

Here we explain the implementation details for the training and validation settings used in our experiments of each of each subsections in Section 4 in themain paper.
Fine-tuning Details. We configure the learning rate as 0.0001, no learningrate scheduler, random seed as 1234, and other settings following the default
LoRA [30] configuration. For the BLIP-2 model, we use a maximal sequencelength of 400, and other hyperparameters remain the same as the official BLIP2 implementation.
Implementation Details for Experiment in Section 4.1 & 4.3 & 4.4.
During training, we utilize all QAs as input per frame, with a subset of themhaving contexts (questions from P2,3 , B, and M ). The contexts are extracted
from ground truth, following the teacher-forcing setting [44,82] generally adopted
in recurrent networks. As for inference, due to the variant complexity of the scenarios, the count of P1−3 QA per frame is highly imbalanced across the dataset,with a variance of over 260 on DriveLM-nuScenes. To balance the impact of
this, we compute the GVQA Scores on only a subset of QA associated witheach frame. To extract the QA subset, we design a set of QA patterns for eachstage based on the questions generally associated with that stage. We ensurethat for all our validation frames, each stage has at least one question matchedwith the designed pattern. In this process, except for the questions in stage P1 ,all questions in other stages have context from the previous stage’s QA, where
the answers are derived from the prediction in the preceding steps. Two specificgraph structure examples can be found in Fig. 13 and Fig. 14.
Implementation Details for Experiment in Section 4.2. The model istrained in the same scheme as in Section 4.1 & 4.3 & 4.4, and the training
set is the DriveLM-nuScenes train split. During the inference, as there are noannotation on the Waymo (not even questions), we devise the question in arule-based manner. Specifically, we re-use the general-purpose questions in theperception stage from DriveLM-nuScenes for Waymo as the starting questions.
Then we try to find if there is any objects in the answer that is matched in the
DriveLM-nuScenes annotation, such as “pedestrians”, “cars”, “trucks” and so on.
Then, we generate the questions based on those matched objects automatically,
which serve as the following questions in the prediction and the planning stages.
A specific graph structure example can be found in Fig. 15.
Implementation Details for Experiment in Section F.2. For the generalization experiment in Section F.2 we add two new questions to the graph: (1) Is
there a person in the scene? and (2) if the answer to the first question is yes we
ask What should the vehicle do based on the pedestrian that is crossing the road?
if the answer is no we ask What should the ego vehicle do?. The answer to thesecond question gets concatenated to the context of the final behavior question.
Three examples can be found in Fig. 16.

44

C. Sima et al.

Table 11: Generalization in DriveLM-CARLA. All methods underperform on
DriveLM-CARLA-ped with the novel pedestrian object, but DriveLM-Agent can besignificantly improved by including a pedestrian-specific question in its GVQA graph.
move this to supp??
MethodF.2

DriveLM-CARLA (B) DriveLM-CARLA-ped (B)Acc. ↑ Spd. ↑Str. ↑ Acc. ↑ Spd. ↑Str. ↑

TransFuser++ [36]70.19 73.29

90.68

8.72

8.72

100.00

DriveLM-Agent+ Pedestrian QA

59.63 61.50
52.17 55.28

78.26 4.59 4.59
77.64 27.04 27.04

100.00100.00

DriveLM-Agent (GT) 60.25 65.22+ Pedestrian QA
60.25 65.22

80.12 20.92 20.92
80.12 92.35 92.35

100.00100.00Generalization to Unseen Objects

Next, we evaluate zero-shot generalization to novel objects. DriveLM-CARLA
is collected without any pedestrians in the training or validation splits. We now
generate a new test set called DriveLM-CARLA-ped, which only consists offrames where a pedestrian is present in the scene. The correct behavior is tostop for the pedestrian.
Baselines. For this experiment, we compare DriveLM-Agent to TransFuser++ [36],the state-of-the-art for CARLA. It uses a larger input image, an additional
LiDAR sensor, and several driving-specific annotations (depth, semantics, 3D
bounding boxes, HD map) in comparison to DriveLM-Agent. However, becauseof these task-specific inputs and outputs, TransFuser++ can only be trainedon the base DriveLM-CARLA dataset and cannot incorporate general computer
vision data during training, which makes generalization more challenging.
DriveLM-Agent. Taking advantage of the more general architecture of a VLM,
we include samples from COCO [53] and GQA [34] along with DriveLM-CARLA
during training for DriveLM-Agent. We compare several versions: (1) we investigate the addition of a new P1 question during inference, ‘Is there a person
crossing the road?’ (‘+ Pedestrian QA’). (2) As an upper bound, we directly
input the ground truth P1−3 graph to the model during inference, instead of themodel’s predictions. For more details, please refer to the supplementary material.
Results. We present our findings in Table 11. We observe that TransFuser++struggles on DriveLM-CARLA-ped relative to DriveLM-CARLA, with a dropin accuracy from 70.19 to just 8.72. DriveLM-Agent experiences a similar dropfrom 59.63 to 4.59. However, adding the pedestrian QA significantly boosts performance on the generalization setting to 27.04, albeit with slightly reduced
accuracy on regular scenes. This is mainly attributed to the fact that the VLM
is not able to detect all pedestrians correctly. This indicates that the large performance gains of recently published VLMs [26,65] can support even better generalization ability in the domain of driving. Additionally, when the pedestrian
QA is provided in the privileged setting that assumes access to perfect context
for each question in the graph, DriveLM-Agent achieves a near-perfect score

DriveLM: Driving with Graph Visual Question Answering

45

Table 12: Graph-structured reasoning facilitates improved VQA with
VLMs. Completeness measures how many questions are correctly answered in one
frame of data. The improvement trends are not consisten across different conventional
metrics, thus we need the GPT score as the main metrics as it evaluates the performance more comprehensively.
DriveLM-nuScenes
Context

Off-the-Shelf BLIP-2

DriveLM-Agent

BLEU-4 ↑ METEOR ↑ CIDEr ↑ ROUGE_L ↑ SPICE ↑ GPT ↑ Comp. ↑ BLEU-4 ↑ METEOR ↑ CIDEr ↑ ROUGE_L ↑ SPICE ↑ GPT ↑ Comp. ↑
None
Graph

0.022
0.022

3.317
3.882

0.11850.0771

7.205
7.397

4.336
7.710

42.97
45.21

1.064
0.859

51.89
53.09

35.81
36.19

2.463
2.786

66.79
65.58

42.56
49.54

71.39
72.51

30.04
31.66

GT

0.022

4.397

0.0758

8.033

8.192

41.10

1.315

53.06

36.64

3.069

66.69

50.29

72.94

32.41

(20.92 → 92.35) on the frames with pedestrians Note that DriveLM-CARLAped only contains pedestrians crossing a straight road, so all models obtain a
100% accuracy on the steering class (which is always straight).
F.3

Results with More Metrics in VQA

In Table 12, we provide the performance under BLEU-4 [64], METEOR [45],CIDEr [86] and ROUGE-L [52] of the Table 4 in the main paper. One key
observation is that different metrics reflect different trends in the performance,
and the improvement is not consistent across all these metrics. This brings us
the motivation to use the GPT Score as the main metric in the VQA evaluation
part.
F.4

Stage-wise Ablation on Zero-shot Generalization across Sensor
Configurations

In Table 14, we provide more settings of context in the zero-shot generalization
across sensor configurations as in Table 2 in the main paper. One key observationis that the higher the accuracy of the behavior task, the better the performanceof the motion task. With more context in the behavior task, the accuracy improvement mainly originates from the improvement of the speeding accuracy,
which finally affects the FDE score.
F.5

More VLMs evaluated on DriveLM-nuScenes

Table 13: More VLMs evaluation on DriveLM-nuScenes. On the base model of
LLaMA-Adapter V2, we observe a slight improvement by using multi-frame as input.
Table A.
Base VLM
blip-2-flan-t5-xl
LLaMA-Adapter V2
LLaMA-Adapter V2∗

Behavior Motion
Behavior (B)Motion (M )Context Context Acc. ↑ Speed ↑ Steer ↑ ADE ↓ Col. ↓
Graph
Graph
Graph

B
B
B

57.49 69.89 80.63
55.31 61.97 81.43
57.83 67.21 83.49

1.74
1.86
1.75

1.89
2.03
1.69

46

C. Sima et al.

Table 14: Zero-shot Generalization across Sensor Configurations. B* denotes
using ground truth behavior QA as context for motion task. Results on 1k randomly
sampled frames from the Waymo val set after training on DriveLM-nuScenes. Our
key observation is that the higher the accuracy of the behavior task, the better the
performance of the motion task.
MethodBehavior Motion
Behavior (B)Motion (M )Context Context Acc. ↑ Speed ↑ Steer ↑ ADE ↓ FDE ↓Command Mean
UniAD-Single
BLIP-RT-2---

DriveLM-Agent

None
P1
P2
P3
P1−3

B
B
B
B
B

35.70
38.20
39.52
34.62
39.73

-

B*

100.0

BLIP-RT-2

-

-

7.98
4.16
2.78

11.41
9.31
6.47

43.90 65.20
43.67 70.74
44.20 78.67
41.28 64.55
54.29 70.35

2.76
2.67
2.62
2.85
2.63

6.59
6.41
6.19
6.89
6.17

100.0

2.41

5.79

100.0

In terms of validating the generality of the proposed DriveLM-Agent andexploring if the video input helps the model’s performance in this task, we select
LLaMA-Adapter V2 (7B) [I] as it is compatible with single- and multi-frame
inputs. The results in Table 13 (within 9 training epochs) show performance
similar to BLIP-2.
Note that due to the large variety of question types in DriveLM-nuScenes,
analyzing each question would be prohibitively expensive.
Instead, we provided an ablation of the effectiveness of each stage (P1−3 ) in
Table 14, and a representative-question-wise analysis.
We have to admit that it is non-trivial in improving the QA performance(P1-3) under current question setting and we found that it had little affect on
the final task performance (behavior and motion).
This inspired us that maybe the question setup worth more effort to explore, and with well-set question, the model could be “prompted correctly” tounderstand the scene and perform the downstream task better.
F.6

Computational Complexity

In Table 5 of the main paper, we provide a comparison of the computational
complexity of DriveLM-Agent to UniAD-Single. A future direction would becaching the vision tokens and batching the different question patterns, which
can speed up the inference time fundamentally.

G

Qualitative Results

In this section, we show the qualitative results of the experiments, including VQA
on our DriveLM-nuScenes, generalization results across sensor data on Waymo,
and generalization results to unseen objects on DriveLM-CARLA.

Answer: There are two cars, one truck and one
pedestrian to the front of ego car.

SPICE: 1.0

GPT score: xxx.

Question: What are important objects tothe front of the ego car?

DriveLM: Driving with Graph Visual Question Answering
Perception

47

Scene Image

Question: What are objects to the front of the ego car?

Answer: There are many cars, one truck and one
pedestrian to the front of ego car.

GT: There are many pedestrians, one trailer, three cars, one

SPICE: 1.0
GPT score:truck to the front of the ego car.

SPICE: 0.783

GPT score: 40

Prediction
Question with pred-context: Perception Q:

Question with pred-context: Perception Q:
What are objects to the front of the ego car?
Perception A: There are many cars, one
truck and one pedestrian to the front of egocar. Q: What is the status of the truck that isto the front of the ego car?

Question with pred-context: Perception Q:

What are objects to the front of the ego car?
Perception A:There are many cars, one truck
and one pedestrian to the front of ego car. Q:
What is the status of the cars that are to thefront of the ego car?

Answer: Many cars are moving and one is

Answer: One truck is moving.

Answer: One pedestrian is moving and one

GT: Three cars are moving.

GT: One truck is parked.

GT: Many pedestrians are moving.

SPICE: 0.667

SPICE: 0.667

parkedGPT score: 40

GT: Accelerate and go ahead.

What are objects to the front of the ego car?
Perception A: There are many cars, one truck
and one pedestrian to the front of ego car. Q:
What is the status of the pedestrian that is tothe front of the ego car?

SPICE: 1.0
GPT score: xxx.

is standing.

GPT score: 0

SPICE: 0.667

GPT score: 10

Planning
Question with pred-context: Perception Q: What are objects to the front of the ego car?

Perception A: There are many cars, one truck and one pedestrian to the front of ego car.
Prediction Q: What is the status of the cars that are to the front of the ego car? Prediction A:
Many cars are moving and one is parked. Prediction Q: What is the status of the truck that is tothe front of the ego car? Prediction A: One truck is moving. Prediction Q: What is the status of
the pedestrian that is to the front of the ego car? Prediction A: One pedestrian is moving and
one is standing. Q: In this scenario, what are safe actions to take for the ego car?

Answer: Keep going at the same speed,
accelerate and go ahead.

GT: Keep going at the same speed, decelerate
gradually without braking.

SPICE: 1.0

GPT score: 50

Behavior
Question with pred-context: Perception Q: What are objects to the front of the ego car?

Perception A: There are many cars, one truck and one pedestrian to the front of ego car.
Prediction Q: What is the status of the cars that are to the front of the ego car? Prediction A:
Many cars are moving and one is parked. Prediction Q: What is the status of the truck that is tothe front of the ego car? Prediction A: One truck is moving. Prediction Q: What is the status of
the pedestrian that is to the front of the ego car? Prediction A: One pedestrian is moving and
one is standing. Planning Q: In this scenario, what are safe actions to take for the ego car?
Planning A: Keep going at the same speed, accelerate and go ahead. Q: Predict the action
template of the ego car.

Behavior

Answer: The ego vehicle is going straight. The
ego vehicle is driving fast.

GT: The ego vehicle is going straight. The ego
vehicle is driving fast.

SPICE: 1.0

GPT score: 95

Fig. 13: Detailed qualitative results on DriveLM-nuScenes. The graph prompting process can be divided into different tasks, and different QAs in each task revolve
PlanningaroundQuestionwith nodifferent
context: In this objects.
scenario,
Question with pred-context: This is previous
Question with gt-context: This is previous
what are dangerous actions to take for theego vehicle?

Answer:G.1

question: What are important objects to thefront of the ego car? This is previous
answer: There is one bus to the front of theego car. In this scenario, what are
dangerous actions to take for the ego
vehicle?

DriveLM-nuScenes

SPICE: 1.0
GPT score: xxx.

Answer: Accelerate

SPICE: 1.0

question: What are important objects to thefront of the ego car? This is previous
answer: There is one bus to the front of theego car. In this scenario, what are
dangerous actions to take for the ego
vehicle?

Answer:SPICE: 1.0

GPT score: xxx.
and go ahead.
GPT score:xxx.
This section shows the qualitative
examplesfor the DriveLM-nuScenes.
In Fig.
13,
we showcase a detailed example of GVQA reasoning process on DriveLM-nuScenes,
encompassing P1−3 QA and the behavior task. We compare the predicted answers with ground truth and provide SPICE scores and GPT scores. In thisfigure, the second question in the prediction stage represents a typical error.
Due to the input of single-frame images, our model often struggles to accurately
determine the correct movement status of objects. This judgment is indeed challenging even for humans. Furthermore, in Fig. 14, we present additional qualitative results to showcase our model’s performance.

GT: The ego vehicle is going straight. The
ego vehicle is driving fast.

SPICE: 1.0
GPT score: xxx.

48

C. Sima et al.
Perception VQAQuestion with no context:

What areobjects to the front of the ego car?

GT: There are three cars, one motorcycle tothe front of the ego car.

SPICE:0.5333

Answer: There are two cars, one truck,

GPT score: 30

one pedestrian to the front of the ego car.

Prediction VQAQuestion with pred-context: Perception
results + What is the status of the cars thatare to the front of the ego car?

GT: Two of the cars are moving and one
is parked.

SPICE: 1.0

Answer: Two cars are moving and one

GPT score: 100

is parked.

Planning VQAQuestion with pred-context: Prediction
results + In this scenario, what are safe
actions to take for the ego vehicle?

GT: Brake gently to a stop.
Answer: Keep going at the same speed,
accelerate and go ahead.

SPICE: 0.0
GPT score: 0

Behavior VQAQuestion with pred-context: Perception
results + Prediction results + Planning
results + Predict the action template of
the ego vehicle.

GT: The ego vehicle is going straight. The
ego vehicle is driving fast.

SPICE: 1.0

Answer: The ego vehicle is going

GPT score: 50

straight. The ego vehicle is driving slowly.

Perception VQAQuestion with no context:

What areobjects to the front of the ego car?

GT: There are pedestrians, three traffic

cones, three cars to the front of the ego car.

SPICE: 0.5556

Answer: There are two cars, one

GPT score: 60

pedestrian to the front of the ego car.

Prediction VQAQuestion with pred-context: Perception
results + What is the status of the carsthat are to the front of the ego car?

GT: Two of the cars are moving and one
is parked.

SPICE: 0.6666

Answer: Three cars are moving and one

GPT score: 90

is parked.

Planning VQAQuestion with pred-context: Prediction
results + In this scenario, what are safe
actions to take for the ego vehicle?

GT: Decelerate gradually without braking,brake gently to a stop.

SPICE: 0.0

Answer: Keep going at the same speed,

GPT score: 0

accelerate and go ahead.

Behavior VQAQuestion with pred-context: Perception
results + Prediction results + Planning
results + Predict the action template of
the ego vehicle.

GT: The ego vehicle is going straight. The
ego vehicle is not moving.

SPICE: 0.8571

Answer: The ego vehicle is going

GPT score: 70

straight. The ego vehicle is driving slowly.

Fig. 14: More qualitative results on DriveLM-nuScenes. The examples in the
figure illustrate the robust ability of our DriveLM-Agent to perform VQA tasks indriving scenarios.

G.2

Waymo

This section demonstrates the generalizability of our model across sensor configurations. Fig. 15 illustrates the results of our model, trained on DriveLMnuScenes, when applied to inference on Waymo. As we do not annotate data on
Waymo, the questions are manually defined, and ground truth is not provided.
These results showcase the robust generalization capability of our model.
G.3

DriveLM-CARLAIn this section, we provide qualitative examples for the CARLA dataset.
Generalization to the unseen pedestrian scenario. Fig. 16 shows the generated behaviors for the generalization test set on the unseen pedestrian scenario.
The first example, illustrated on the top of Fig. 16, demonstrates a successful
situation where DriveLM-Agent accurately recognizes a pedestrian. It subsequently infers the appropriate action to undertake, which in this case, is to stopthe vehicle. The behavior generation is able to interpret this context, resulting in
the correct behavior pattern, as evidenced by the ego vehicle coming to a complete stop. The other two examples represent the predominant failure modesof DriveLM-Agent in scenarios involving pedestrians. The middle example ofFig. 16 shows the case where the model still detects the pedestrian. However,it fails to translate this detection into the correct behavior. The final example, shown at the bottom, highlights a more severe limitation where the modelDriveLM: Driving with Graph Visual Question Answering

49

Perception VQAQuestion with no context: What are objects to the front of
the ego car?

Answer: There are two cars, one truck,

one pedestrian to the front of the ego car.

Prediction VQAQuestion with pred-context: Perception results + What is
the status of the cars that are to the front of the ego car?

Answer: Two cars are parked and one ismoving.

Planning VQAQuestion with pred-context:

Prediction results + In this
scenario, what are safe actions to take for the ego vehicle?

Answer: Keep going at the same speed,
accelerate and go ahead.

Behavior VQAQuestion with pred-context: Perception results + Prediction
results + Planning results + Predict the action template of theego vehicle.

Answer: "The ego vehicle is going straight.
The ego vehicle is driving fast..

Perception VQAQuestion with no context: What are objects to the front of
the ego car?

Answer: There are many cars, one pedestrian
to the front of the ego car。

Prediction VQAQuestion with pred-context: Perception results + What is thestatus of the pedestrians that are to the front of the ego car?

Answer: One pedestrian is moving and one
is standing.

Planning VQAQuestion with pred-context:

Prediction results + In this

scenario, what are safe actions to take for the ego vehicle?

Answer: Keep going at the same speed,
decelerate gradually without braking..

Behavior VQAQuestion with pred-context: Perception results + Prediction
results + Planning results + Predict the action template of theego vehicle.

Answer: The ego vehicle is going straight.
The ego vehicle is driving slowly.

Fig. 15: Qualitative results on Waymo. We present two examples showcasing thegenerability of DriveLM-Agent to new sensor configurations, demonstrating the strong
generalization capability of DriveLM-Agent.

completely overlooks the pedestrian. In such instances, DriveLM-Agent acts as if
the pedestrian is non-existent, which consequently results in it not executing any
evasive or stopping maneuvers, posing a significant risk in a real-world scenario.
Graph Visual Question Answering. This section presents two examples of
the graph visual question answering tasks using the CARLA dataset to evaluate
the performance of DriveLM-Agent (Fig. 17). We only show a subset of theevaluated questions. In the first example the ego vehicle drives behind anothervehicle. The primary task is to follow the road and adjust the speed in accordancewith the leading vehicle. Our results indicate that DriveLM-Agent demonstrates
proficient scene understanding by accurately identifying all important objects
in the scene. Despite the ground truth data indicating occasional inaccuracies
in object color labeling by the CARLA simulator, DriveLM-Agent maintains
reliable performance in object recognition. Additionally, the model can identify
the vehicle in front and reason about what to do based on the leading vehicle.
The second example takes place at an intersection regulated by a stop sign.
DriveLM-Agent identifies all objects and can reason about the situation. It correctly identified that it needs to stop not simply due to the stop sign, but primarily because of a motorcycle positioned ahead. This implies that DriveLM-Agentis capable of prioritizing dynamic obstacles over traffic control devices undercertain circumstances.

50

C. Sima et al.

Perception

Scene Image

Question: Is there a person in the scene?
Answer: Yes, there is a person in the scene.

Prediction
Context: Yes, there is a person in the scene. Question: Whatshould the ego vehicle do based on the pedestrian that iscrossing the road?

Answer: The ego vehicle should stop and give the pedestrianspace to cross.

Behavior
Context: The ego vehicle should stop and give the pedestrianspace to cross. Question: Predict the behavior of the ego
vehicle.

Answer: The ego vehicle is going straight. The ego vehicle is
not moving.

Perception

Scene Image

Question: Is there a person in the scene?
Answer: Yes, there is a person in the scene.

Prediction
Context: Yes, there is a person in the scene. Question: Whatshould the ego vehicle do based on the pedestrian that iscrossing the road?

Answer: The ego vehicle should stop and give the pedestrian
the right of way.

Behavior
Context: The ego vehicle should stop and give the pedestrian
the right of way. Question: Predict the behavior of the ego
vehicle.

Answer: The ego vehicle is going straight. The ego vehicle isdriving slowly.

Perception

Scene Image

Question: Is there a person in the scene?
Answer: No, there is no person in the scene.

Prediction
Context: No, there is no person in the scene. Question: Whatshould the ego vehicle do?

Answer: The ego vehicle should follow the road to the next
junction.

Behavior
Context: The ego vehicle should follow the road to the next
junction. Question: Predict the behavior of the ego vehicle.
Answer: The ego vehicle is going straight. The ego vehicle isdriving fast.

Fig. 16: Qualitative results on the generalization test set in DriveLMCARLA. We show three examples of DriveLM-Agent handling the pedestrian scenario. The first example shows a success case and the second and third show two
common failure cases of the model.

DriveLM: Driving with Graph Visual Question Answering

Perception

51

Scene Image

Question: What are important objects to the front of the ego
car?

Answer: The important objects in the scene are the maroon

car to the front left of the ego vehicle and the maroon van to
the front of the ego vehicle.

GT: The important objects in the scene are the maroon car to

the front left of the ego vehicle and the olive van to the front of
the ego vehicle.

Perception
Context: The important objects in the scene

are a junction, the red traffic light, themaroon car to the front left of the ego
vehicle and the black car to the front of theego vehicle. Question: Is there anothervehicle right in front of the ego vehicle?

Answer: Yes, there is another car in front.

GT: Yes, there is another car in front

Prediction

Planning

Context: Yes, there is another car in front.
Question: What should the ego vehicle do
based on the vehicle in front?

Context: The ego vehicle should drive withthe same speed as the car in front. As they are
currently driving with the same speed the ego
vehicle can maintain its speed. Question:
Does the ego vehicle need to stop? Why?

Answer: The ego vehicle should drive with thesame speed as the car in front. As they are
currently driving with the same speed the ego
vehicle can maintain its speed

Answer: There is no reason for the ego
vehicle to stop.

GT: The ego vehicle should drive with thesame speed as the car in front. As they are
currently driving with the same speed the ego
vehicle can maintain its speed.

Perception

GT: There is no reason for the ego vehicle tostop.

Scene Image

Question: What are important objects to the front of the ego
car?

Answer: The important objects in the scene are a junction, astop sign, the black car to the front left of the ego vehicle andthe green motorcycle to the front of the ego vehicle.

GT: The important objects in the scene are a junction, a stop

sign and the green motorcycle to the front of the ego vehicle.

Prediction

Planning

Context: Yes, there is another car in front.
Question: What should the ego vehicle doContext: The important objects in the scene

Answer: The ego vehicle should brake toAnswer: The ego vehicle should remain

based on the vehicle in front?

maintain a safe distance to the vehicle infront.

GT: The ego vehicle should brake to maintaina safe distance to the vehicle in front.

are a junction, a stop sign, the black car to thefront left of the ego vehicle and the green
motorcycle to the front of the ego vehicle.
Question: What should the ego vehicle do
based on the stop sign?

stopped and stay behind other vehicles thatare standing at the stop sign.

GT: The ego vehicle should remain stopped
and stay behind other vehicles that are
standing at the stop sign.

Context: The ego vehicle should brake tomaintain a safe distance to the vehicle in front.
The ego vehicle should remain stopped andstay behind other vehicles that are standing at
the stop sign. Question: Does the ego vehicle
need to stop? Why?

Answer: The ego vehicle should stopbecause of the green motorcycle that is in
front of it.

GT: The ego vehicle should stop because of
the green motorcycle that is in front of it.

Fig. 17: Qualitative VQA results in DriveLM-CARLA. The first example shows
how DriveLM-Agent deals with a situation with a vehicle directly in front of the ego
vehicle. The second example shows an intersection with a stop sign and other traffic
participants waiting in front of the ego vehicle at the stop sign.

52

H

C. Sima et al.

More Related WorkIn this section, we supplement the related work mentioned in the main paper.
Reasoning Over Graph Structures. Reasoning is one of the basic forms ofsimulated human thinking, enabling the derivation of new judgments from one orseveral existing judgements [12]. Many prior reasoning works have been groundedin graph-based approaches [5, 13, 75, 92]. XNMs [75] employs scene graphs for
explainable and explicit reasoning with structured knowledge. KPRN [92] utilizes
knowledge graph for reasoning and applies it to recommender systems. GoT [5]models LLM-generated information as an arbitrary graph and brings the LLM
reasoning closer to brain mechanisms. Inspired by these successful attempts, we
try to link the stages of perception, prediction, and planning in autonomous
driving through a graph, enabling the model to grasp the reasoning process anddeduce unseen scenarios based on learned graph structure.
Embodied Planning with LLMs Recent work endeavors to leverage theformidable reasoning and generalization capacity of LLMs [25,43,83] for embodied AI systems [23, 33, 35, 40, 51, 69, 108]. PaLM-E [23] trains an LLM for various
embodied tasks including sequential robotic manipulation planning. CaP [51]provides a robot-centric formulation of language model generated programs executed on real systems. RT-2 [108] represents robot actions as language tokens,training vision-language models to output robot policies. These methods showcase the capabilities of LLMs in embodied planning tasks, inspiring us to apply
them to address the current shortcomings in generalization in AD, which is far
less explored.
Vision-language Benchmarks for Driving. An increasing number of visionlanguage datasets have been proposed for AD systems [20, 41, 42, 56, 59, 67, 71,
97, 98, 101, 107]. NuScenes-QA [67] and NuPrompt [98] provide perceptual information as text by describing the positions and states of surrounding objects.
BDD-X [42] provides reasons for the ego vehicle’s actions in natural language descriptions. DRAMA [56] and Rank2Tell [71] identify crucial objects and provide
corresponding driving suggestions. However, these datasets focus on scene-level
context or individual objects. DriveLM fills this gap in the literature by organizing language annotations from object-level and task-level with a graph structure.

I

Broader Impact

We believe that this approach can accelerate progress in the field of autonomous
driving by enabling it to directly benefit from better VLMs. Our goal is tomake progress towards autonomous driving, which will have profound impact
if successful. We recognize that by bringing VLMs into this area, we accept
their ethical implications, such as hallucinations and high resource use. Yet, byimproving the interactivity between humans and autonomous driving systems,
we can build confidence in the technology. This could hasten its acceptance andlead to safer transportation in the long term.