DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
Authors: Xiaoyu Tian
Junru Gu
Bailin Li
Yicheng Liu
Yang Wang
Zhiyong Zhao
Kun Zhan
Peng Jia
XianPeng Lang
Hang Zhao
Venue: CoRL 2024 (2024)
Keywords: Autonomous Driving
Vision Language Model
Dual System

Abstract: A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments.

DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
Xiaoyu Tian Junru Gu
Bailin Li
Yicheng Liu
Yang Wang²
Zhiyong Zhao
Kun Zhan
Peng Jia?
Xianpeng Lang² Hang Zhao
IIS, Tsinghua University Li Auto
Abstract: A primary hurdle of autonomous driving in urban environments un- derstanding complex and long-tail scemarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene under- standing and planning capabilities. DriveVLM integrates unique combination of reasoning modules for scene description, scene analysis, and hierarchical plan- ning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose Drive VL--uual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driv- ing pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy VLM and Drive in handling complex and unpredictable driving conditions. Finally, we deploy the Drive on production vehicle, verifying it is effective in real-world autonomous driving environments.
Keywords: Autonomous Driving, Vision Language Model, Dual System
1 Introduction
Autonomous driving, with its great promise to revolutionize transportation, has been an active re- search area over the past two decades. A primary hurdle to fully autonomous driving system is scene understanding [I], which involves navigating complex, unpredictable scenarios such as ad- verse weather, intricate road layouts, and unforeseen human behaviors.
Existing autonomous driving systems, typically comprising 3D perception, motion prediction, and planning, struggle with these scene understanding challenges. Specifically, 3D perception [2, 3, 4, 5] is limited to detecting and tracking familiar objects, omitting rare objects and their unique attributes; motion prediction [6. 7, 8, 9, 10] and planning [11, 12, 13] focus on trajectory-Ievel actions, often neglecting the decision-level interactions between objects and vehicles.
We introduce RLiv novel autonomous driving system that aims these scene understanding challenges, capitalizing on the recent Vision-Language Models (VLMs) [14, 15, 16, 17] which have demonstrated exceptional prowess in visual comprehension and reasoning. Specifically, DriveVLM contains a Chain-of- Though (CoT) process with three key modules: scene description, scene anal- ysis, and hierarchical planning. The scene description module linguistically depicts the driving environment and identifies critical objects the scene; the scene analysis module delves into the characteristics of the critical objects and their influence on the ego vehicle; the hierarchical planning module formulates plans step-by-step. from meta-actions and decision descriptions to waypoints. senceptiio--rredccti
These modules respectively correspond the components of the traditional planning pipeline, but the difference is that these modules tackle object perception, intention-level prediction and task-level planning, which were extremely challenging to cope with in the past.
"Equal contribution. Listing order is random. *Corresponding to: hangzhao @ mail, tsinglittnghhhhee edu.cn Project Page: https//tsinghuu-maas labs-lab www.tt........ieeeeeeeeeee
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

While VLMs excel in visual understanding, they have limitations in spatial grounding and reasoning, and their computational intensity poses challenges for onboard inference speed. Therefore we fur- ther propose traditional systems. DriveVLM-Durs optionally integrates DLM with traditional 3D percep- tion and planning modules, such as 3D object detectors, occupancy networks, and motion planners, enabling the system to achieve 3D grounding and high-frequency planning abilities. This dual sys- tem design, akin to the human brain's 's slow and fast thinking processes, adapts efficiently to varying complexity in driving scenarios.
a hybrid system that combines the strengths of both DLM and
Meanwhile, we formally define the scene understanding and planning (SUP) task, and propose new evaluation metrics to assess the scene analysis and meta-action planning capabilities of DLiveVLM and DriveVL.--Dual. We carry out a comprehensive data mining and annotation pipeline struct an in-house SUP-AD dataset for the SUP task. Extensive experiments both the nuScenes dataset and our own dataset demonstrate the superior performance of DriveVLM, particularly in few- shot scenarios. Furthermore, DLiveVL.M-uua exceeds state-of-the-ae end-to-end motion planning methods. We have also deployed the model on a production vehicle, confirming that Drive VLM- Dual is effective in real-world autonomous driving environments. Additionally, we have included a demo the supplementary materials.
In summary, the contribution of this paper is three-fold:
We introduce VLM, a novel autonomous driving system that leverages VLMs for effective scene understanding and planning. We further introduce DriveVLM-Dual, hybrid system that incorporates VLM and traditional autonomous pipeline, which achieves improved spatial reasoning and real-time planning capabilitiis.
2. We present a comprehensive data mining and annotation pipeline to construct a scene under- standing and planning dataset (SUP-AD), together with metrics for evaluation.
3. We have successfully deployed Drive PLiveVLM-Dual system in production vehicle and test various effective strategies for accelerating VLM deployment in real driving scenarios.
2 Related Works
Vision-Language Models (VLMs). Recently, there has been surge in research on large Vision-
Language Models (VLMs), exemplified by works such MiniGPT-4 [16), LLaVA [17], Qwen- VL [18], and others [19, 14, 20, 21J. VLMs can be used in various scenarios, especially robotics [22, 23, 24, 25, 26], where VLMs output corresponding actions that can be high-level instructions [22] or low-level robot actions [24). DriveVI.M focuses on utilizing VL.Ms to assist in autonomous driving, thereby establishing a novel framework. A Concurrent work [15] shares a similar motivation.
Learning-based Planning. The integration of learning frameworks into motion planning has been an active area of research since Pomerleau [11] pioneering contributions. One promising line of work is Reinforcement learning and imitation learning [27, 28, 29). These methods can learn an end-to-end planning policy that directly maps raw sensory inputs to control actions [29]. Several works [30, 31, 32, 33] improve interpretability by explicitly building dense cost maps derived from learning-based modules. recent trend involves training multiple blocks in an end-to-end fash- ion [32, 33, 34, 35]. These methods enhance overall performance, but rely on backpropagation from future trajectory predictions loss in a less interpretable decision-making process [36).
Driving Caption Datasets. Recent works [37, 15, 38] argue that language captions are an impor- tant medium to connect human knowledge with the driving objective, helping to inform decisions and actions. Refer-KITTI [39] annotates objects in the KITTI dataset [40] with language prompts that can reference a collection of objects. Talk [41], NuPrompt [42] and nuScenes-QA [43] introduce free-form captions and QA annotation to the nuScenes dataset [44]. BDD-X [45] and BDD-OIA [46] offer datasets with language explanations for the ego vehicle's actions traffic sce- narios [47] [48]. These datasets offer scenes for natural language use, but lack sufficient data on critical safety scenarios in self-driving systems.

---

DriveVLM
Large Vision Language Model
Scene Description
Scene Analysis
Hierarchical Planning
<SYSTEMS Maxched ofjecn:
<SYSTIM Ego upae
codtiies.
police cit, luakey uxjodsey sou.
antoriad majccuss
OHIVLEVLM-
Descile citiial oeeess fhe egeseliik..
Weather
3 CIRICLINUM
Read type: wharhen.
? -DRIVLEVLM-
Matt actions (eirs donn, dightly to right cooigihh at spool)
Soqunee of Images
Theer Daytine
Charativrition Phrriing the
Lass con coniition right lace impo gaoohite
ede of mad.
lafleane: Blodkng the laze indianin ponennaa for foraccidents otter onerinceerrss
banic
stniall at
Summuroed Anatyste...
coneast spoad
Warpsinte (141) yo .o. yraj
Low Frequency
Trajectory Refinement
Matching
Prompling
DriveVLM-Dual
3D Perception
Motion Prediction
Trajectory Planning
Traditional Pipeline
High Frequency
Figure 1: DriveVLM and DriveVLM--Dul model pipelines. DriveVI.M takes images as input and, through Chain-of-Thought (CoT) mechanism, outputs scene description, scene analysis, and hierarchical planning results. DriveVLM--ual further incorporates traditional 3D perception and trajectory planning modules to achieve spatial reasoning capability and real-time trajectory planning.
DriveVLM
VLM takes images as input and outputs planning results through a CoT reasoning process, as illustrated in Figure 1. The reasoning process can be divided into three modules: scene descrip- tion (Section 3.1), scene analysis (Section 3.2), and hierarchical planning (Section 3.3). For real- world deployment, we propose hybrid system, DriveVLM-Dual, in Section 3.4. which combines DriveVLM, and the traditional pipeline, leveraging the strengths of both approaches.
3.1 Scene Description
The scene description module identifies driving environment description and critical objects.
Environment Description. Driving environments, such as weather and road conditions, have a non-negliiible impact on driving difficulty. Therefore, the model is first prompted to out- put a linguistic description of the driving environment, including several conditions: E = Eweather: Etiee Exoad, E, each representing a crucial aspect of the driving environment. The weather component, Eweather spans conditions from sunny to snowy, affecting visibility and trac- tion. The time component, Eume- distinguishes between daytime and nighttime, impacting driving strategies due to visibility changes. Road types, Fmud such urban or highway, introduce different challenges, while lane conditions, Lane. focus current lane positioning and possible maneuvers, crucial for safe driving decisions.
Critical Object Identification. Unlike traditional autonomous driving perception modules, which detect all objects within a specific range, we solely focus on identifying critical objects that are most likely to influence the current scenario, inspired by human cognitive processes during driving. Each critical object, denoted as O,,, contains two attributes: the object category and its approximate bounding box coordinates b(z1, (f11,y1,4 .2, y2) on the image. Taking advantage of the pre-trained vision encoder, Drive DriveVLM can identify long-tail tail critical objects that may elude typical 3D object detectors, such as road debris or unusual animals.
3.2
Scene Analysis
In the traditional autonomous driving pipeline, the prediction module typically concentrates on fore- casting the future trajectories of objects. The emergence of advanced vision-language models has provided with the ability to perform scene-level analysis summarizes all the critical objects together with the environmental description.
comprehensive analysis of the current scene. The
3

---

This summary gives a comprehensive understanding of the scene, and is fed into the following plan- ning module.
Critical Object Analysis. DriveVLM characterizes critical objects three aspects: static tributes C., motion states C.. and particular behaviors C₁. Static attributes C. describe inherent properties of objects, such as roadside billboard's visual cues truck's oversized cargo, which are critical in preempting and navigating potential hazards. Motion states Cm describe an object's dynamics over a period, including position, direction, and action characteristics that are vital in predicting the object's ss future trajectory and potential interactions with the ego vehicle. Particular behaviors C. refer to special actions or gestures of an object that could directly influence the ego vehicle's 's next driving decisions. We do not require the model analyze all three characteristics for all objects. In practice, only one characteristics apply critical object. Upon analyzing these characteristics, DrM then predicts the potential influence of each critical object on the ego vehicle.
3.3 Hierarchical Planning
The scene-level summary then combined with the route, ego pose and velocity to form prompt for planning. Finally, DLM progressively generates driving plans, in three stages: meta- actions, decision description, and trajectory waypoints.
Meta-actions A. A meta-action, denoted as ag, represents short-term decision of the driving strategy. These actions fall into 17 categories, including but not limited to acceleration, deceleration, turning left, changing lanes, minor positional adjustments, and waiting. To plan the ego vehicle's future maneuver over a certain period, we generate a sequence of meta-actions.
Decision Description D. Decision description D articulates the more fine-grained driving strategy the ego vehicle should adopt. It contains three elements: Action A, Subject S, and Duration D. Action pertains to meta actions such as 'turn', 'wait', or 'accelerate'. Subject refers to the interacting object, such as pedestrian, a traffic signal, or a specific lane. Duration indicates the temporal aspect of the action, specifying how long it should be carried out or when it should start.
Trajectory Waypoints W. Upon establishing the decision description D, our next phase in- volves the generation of corresponding trajectory waypoints. These waypoints, denoted by W {Wi, 12, .... J. wi- (I,, vi). depict the vehicle's ' path over a certain future period with prede- termined intervals At. We map these numerical waypoints into language tokens for auto-regressive generation.
3.4 LLi-eaL
To mitigate the challenges of high latency and imprecise spatial and motion understanding in VLMs, we propose DriveVLM-Dual collaboration between DLM and the traditional autonomous driving system. This novel approach involves two key strategies: incorporating 3D perception for critical object analysis, and high-frequency trajectory refinement.
Integrating 3D Perception. We represent objects detected by a 3D detector as OzD = {cáD D/D ). where dip denotes the i-th bounding box and cip denotes its category. These 3D bounding boxes are then back-projected onto 2D images to derive corresponding 2D bounding boxes byp- We conduct IoU matching between these 2D bounding boxes byp and bg- by are the bounding boxes of previously identified critical objects Ouiitical {ce,, 6g). We classify critical objects that meet certain approx- imate loU threshold and belong to the same category matched critical objects manched defined as
S Sygnes Søg.
matchhdd
(cc. b2), if cl cip and alo bio) > T, where aloU (íD)
Those critical objects without a corresponding match in the 3D data are noted as Oummatched.
In the scene analysis module, for mancched the center coordinates, orientations, and historical tra- jectories of the corresponding 3D objects are used as language prompts for the model, assisting in

---

Sooe Summery
Thes vadiilo moking conotent spead ilong the axmert line, with orgiing redd constracioon work abead; (hem am these cirstrationn workers working the iiit of the lase medbile.
Suzzy
Road Condition:
Corstruction
Deptim
Lane Condition
Lane
Otgect
Class
Thme Constrution
Oritcal Oltenn
Wodsers
Class
Constuctor Zone
Chandtriittc
Constaatiion work on the side the lane to the jet of the bost vabicio
Chandlatiste:
Road repair is frent the bast vubicle
laze
Inflommoe
fiects normel
Influmon
Affects
speed of the hoet
whide driee
strught semmaly
Meta
(SSoo donne, CCaagge
stright slowly
Dedsion Desctption.
Docderiae and thange lanes the left, keeping a saft distance from the
corstruction wathers on the trazt-sdd
Figure 2: An annotated sample of the SUP-AD dataset.
object analysis. Conversely, for Oummatched, analysis relies solely on the language tokens derived from the image. This design enables DLiveVLM-Dual understand the locations and motions of critical objects more accurately, enhancing the overall performance.
High-frequency Trajectory Refinement. To achieve real-time, high-frequency inference capa- bilities, we integrate it with conventional planner to form a slow-fast dual system, combining the advanced capabilities of VLM with the efficiency of traditional planning methods. After obtaining trajectory from Dri low frequency, denoted as Wsoow. we take it as refer- ence trajectory for classical planner for high-frequency trajectory refinement. the case of an optimization-based planner, W stow serves the initial solution for the optimization solver. For neural network- based planner, Wslow is used input query, combined with additional input fea- tures f, and then decoded into new planning trajectory denoted as W The formulation of this process can be described as:
W fast = (Tanner([Waun slows fl).
(1)
This refinement step ensures that the trajectory produced by DriveVLM--Dul (1) achieves higher trajectory quality, and (2) meets real-time requirements. In practice, the two branches operate asyn- chronously in slow-fast manner, where the planner module in the traditional autonomous driving branch can selectively receive trajectory from the VLM branch as additional input.
Task and Dataset
To fully exploit the potential of VLM and DriveVLM-Dual in handling complex and long-tail driving scenarios, we formally define task called Scene Understanding for Planning (Section 4.1), together with a set of evaluation metrics (Section 4.2). Furthermore, we propose a data mining and annotation protocol to curate a scene understanding and planning dataset (Section 4.3).
4.1 Task Definition
The input of Scene Understanding for Planning task comprises multi-view videos V from surround- ing cameras and optionally 3D perception results P from a perception module. And its output in- cludes the following components: Scene Description s: Composed of different conditions, object- level analysis and scene-level summary; Meta Actions A: A sequence of actions representing task- level maneuvers; Decision Description D: A detailed account of the driving decisions; Trajectory Waypoints W: The waypoints outlining the planned trajectory of the ego vehicle.
4.2 Evaluation Metrics
To comprehensively evaluate model's performance, we care about its interpretation of the driving scene and the decisions made. Therefore, evaluation has two aspects: description/amalysi evaluation and meta-action evaluation. In the Scene description/analyse evaluation, we leverage
5

---

Long-tiil Otject Mining
Driving Scenario
Manual Filtering
Video Keyframe
Scane Annotation
Verification
Video Dutabase
Challenging Scenario Mining
The proposed data mining and annotation pipeline for constructing the scene under- standing and planning dataset. Scenario examples randomly sampled from the dataset (below) demonstrate the diversity and complexity of the dataset.
Figure 3:
a structured evaluation approach by prompting the GPT-4 to extract individual pieces of information from each scene description. Subsequently, we ask the GPT-4 to score and aggregate the results based on the matching status of each extracted piece of information. For Meta-action evaluation, we focus on the sequence of decision-making actions defined as meta-actions, comparing these sequences generated by the model against a ground truth using dynamic programming. To ensure the robustness of our evaluation, the ground truth, with the highest similarity score determining the final decision-making accuracy. Further details on these metrics are elaborated in Appendix B.
GPT-4 generate semantically equivalent alternatives to
4.3 Dataset Construction
We propose a comprehensive data mining and annotation pipeline, shown in Figure 3, to construct Scene Understanding for Planning (SUP-AD) Dataset for the proposed task. This process begins with long-tail object mining and challenging scenario mining from large database collect special objects and scenes like weird-shaped vehicles and road debris. For each scene, Keyframe selection is performed to identify crucial moments for decision-making, which are then annotated using a specialized Scene Annotation tool. Details of the construction process and dataset statistics are available in the Appendix A.
5
Experiments
5.1 Settings
We test DriveVLM and DriveVLM-Dual on our proposed SUP-AD dataset and nuScenes dataset [44]. The setup details of our model and training details are listed in Appendix C.
SUP-AD Dataset. The SUP-AD dataset dataset built by our proposed data mining and anno- tation pipeline. It is divided into train, validation, and test splits with ratio of 7.5 1 1.5. We train models on the training split and use our proposed scene description and meta-action metrics to evaluate model performance on the test split. We also employ co-tuning with additional datasets to ensure the generalization of the LLM is not compromised. For details, see Appendix C.
nuScenes Dataset. The nuScenes dataset large-scale driving dataset of urban scenarios with 1000 scenes, where each scene lasts about 20 seconds. Following previous works [34, 49], adopt Displacement Error (DE) and Collision Rate (CR) as metrics to evaluate models' performance.
5.2 Main Results
SUP-AD. We present the performance of our proposed DriveVLM with several large vision- language models and compare them with GPT-4V, as shown in Table 1. Drive utilizing Qwen- VI. as its backbone, achieves the best performance due to its strong capabilities in question answer-

---

Table 1: Results on test of our proposed SUP-AD dataset. : Using the official API of GPT-4V. For Lynx and CogVLM, utilize the training split for fine-tume sine-tuning purposes. contrast, for GPT- 4V, we employ in-context learning.
Table Planning results on nuScenes validation dataset. Drive achieves the best performance. denotes coop- erating with VAD [49).
Method
1a (m)
Collision (%)
3x
NMP [30)
2,51
1.92
SA-MMP [30)
Metlod
Scene Description
Mets-action
FF [32)
0.55
1.20
2.54
1.43
0.06
0.17
1.07
0.43
Fine-tuning wi Eynx [14)
0.46
0.15
ED
2.78
0.09
0.88
uning CogVLM
ST:P3 [52]
1.33
2.11
2,90
2,11
0.23
0.62
1.27
0.71
GPT-4V
0:38
0.19
DEneVLM w/ Queen
0.7
0,37
VAD Bave (49)
0.17
0.34
0.60
0.37
0.07
0.10
0.24
0.14
DriveLMM
DrieeLLM-DDal
0.15
0.29
0.48
0.31
0.05
0.08
0.17
0.10
ing and flexible interaction compared to the other open-source VLMs. Although GPT-4V exhibits robust capabilities in vision and language processing, its inability to undergo fine-tuning, restricting it solely in-context learning, often results in the generation of extraneous information during scene description tasks. Under our evaluation metric, the additional information is frequently classified as hallucination, consequently leading to lower scores.
nuScenes. As shown in Table 2, LLM-eal achieves state-of-the-art performance on the nuScenes planning task when cooperating with VAD. demonstrates that our method, although tailored for understanding complex scenes, also excels ordinary scenarios.
5.3
Ablation Study
Model Design. To better understand the significance of our designed modules in Dri,, we conduct ablations on different combinations of modules, as shown in Table 3. The inclusion of criti- cal object analysis enables model to identify and prioritize important elements in the driving vironment, enhancing the ppcision-making accuracy for safer navigation. Integrating 3D perception data, our model gains refined understanding of the surroundings and achieves precise predictions.
Traditional AD Pipeline. demonstrate the generalization of our dual system design, we test VLve with different traditional autonomous driving pipelines on the validation set of nuScenes. As illustrated in Table our proposed DriveVLM-Dus adapts well to different tradi- tional AD pipelines. While standalone MLP method shows notable performance gap compared to VAD, both variants of DriveVL.M-Dual achieve nearly identical performance, underscoring the efficacy and robustness of our dual system design.
5.4 Qualitative Results
Qualitative results of DriveVLM are shown in Figure In Figure 4a, Dr. accurately pre- dicts the current scene conditions and incorporates well-considered planning decisions regarding the cyclist approaching us. In Figure 4b, DriveVLM effectively comprehends the gesture of the traffic police ahead, signaling the ego vehicle to proceed, and also considers the person riding tricycle on the right side, thereby making sensible driving decisions. These qualitative results demonstrate our model's s exceptional ability to understand complex scenarios and make suitable driving plans. More visualization of our model's output shown the Appendix E.
Table Ablations of design choices on the validation set of muScenes, "Base" refers to only indicating the hierarchical planning sults without our pruposed CoT inference. "CO" represents the addition perception
Table Ablations of traditional autonomous driving pipeline in Drive VriveVLL- MLP stands for methods similar to MLP (53).
critical object analysis. "3D" denotes the inclusion results auxiliary language prompt.
L2 tnai
Calisio (91)
Metel
Avg.
3s
L2 (m)
Collisios (%)
URAD (3]]
048
0.66
1.65
0.15
0.17
0.71
0.31
ID
Base CO 3D
Ang.
Aug.
MEP
0.5
0.66
0.67
0.66
011
0.18
0.6
0.70
8.41
Dea
0.20
0.38
066
0.35
VAD (49)
D.2
0.34
0.00
0.37
037
0.16
014
018
0.34
0.68
0.40
0.10
0.22
0.45
0.27
M.
0129
0.88
018
7

---

right electric sigrae entertir
Envirerent:: Croursed.
Aiterattioe Lane: Sntersctioon.
the Interectinn
Sime
Dectatio Sescraption:
Jricane cado dictive from Vittes wille forvand cledly
guilis pas
(a)
(b)
Figure 4: Qualitative results of DriveVLM. The orange curves represent the model's planned
future trajectories for the next seconds.
6 Onboard Deployment and Testing
autonomous vehicle equipped with two OrinX processors, with the fast end-to-end driving system on OrinX-1 and the slower DriMeVLM OrinX-2. The total computational power is 508 TOPS (including both sparse and DLA computational power). Given the vehicle's 's hardware constraints, we select models with fewer than 4 billion parameters for real- time inference. The Qwen series [18] demonstrate superior inference speed due to its wide and shallow architecture, meeting the vehicle's speed requirements with an average total time for prefill and decode within 400ms. For the visual encoder, we choose the SigLIP-L-384 [54] model with PE interpolation fine-tuning, optimizing for high resolution and fine-grained understanding. To handle the computational load from high-resolution images, use LDPNetv2 [55], reducing image tokens by 75% without compromising performance, and cropped non-informative regions from images. To better assess object motion, we employe memory bank strategy [56] for storing visual features from historical frames, enhancing temporal context with SE [57] blocks for weighted fusion. We utilize the Eagle [58] speculative sampling technique, achieving a 2.7x speedup in decode latency with our custom inference framework on the Orin chip, ensuring feasible real-time deployment for autonomous driving. Through these designs and optimizations, the DriveVLM can achieve real-time inference speed of approximately 400ms per inference. The end-to-end fast system has real-time speed of 130 ms per inference, and since the two systems run asynchronously, the overall system latency is about 130 ms. For detailed deployment techniques and model performance, please refer to Appendix D.
We deploy DriveVL.M-Dual
7 Limitations
Our VLM model is obtained by continuing pretraining and fine-tuning an existing pretrained VLM. Consequently, the upper bound of VLM's capabilities is inherently limited by that of the original pretrained VLM. Moreover, current VLMs exhibit certain deficiencies in spatial understanding, such position and distance, which is why we integrate 3D perception results into VLM's input. Ad- ditionally, current VLMs are susceptible to "hallucination" issues, which can potentially lead to safety concerns. Completely eliminating such hallucinations remains a challenge, though this is an inherent limitation VLMs and not specific to our method.
8 Conclusion
In summary, we introduce Dr. and DriveVL.M-Dual. DriveVI.M leverages VLMs, signifi- cantly progressing in interpreting complex driving environments. DriveVLM-Dual further enhances these capabilities by synergizing existing 3D perception and planning approaches, effectively ad- dressing the spatial reasoning and computational challenges inherent in VLMs. Moreover, we define a scene understanding for planning task for autonomous driving, together with evaluation metrics and dataset construction protocol. DriveVLM and Drive LiiveVLM-Duall have surpassed the state-of-the art methods on the public and our benchmarks, especially in handling intricate and dynamic scenar- ios. Finally, we have verified the effectiveness of Drive VLM-Dua through onboard deployment and testing on production vehicle.

---

Acknowledgments
We thank Chenxu Hu Tsinghua University for the help on paper writing. We thank Xu Bian, Hongkun Chen, Sui Cong, Chengze Guan, Mingyu Guo, Yue Jiang, Qi Jiang, Pengfei Ji, Wei Xiao, Dafeng Wei, Zijian Wang, Zhao Yang, Chenglong Zhao, Simeng Zhao, and Jian Zhou at Li Auto for their efforts in the experiments related to onboard deployment.
References
[1] I. Barabas, A. Todorut, N. Cordos, and A. Molea. Current challenges in autonomous driving. In IOP conference series: materials science and engineering, volume 252, page 012096. IOP Publishing, 2017.
[2] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[3] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast en- coders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697-12705, 2019.
[4] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang. H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-191. PMLR, 2022.
[5] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai. Bevformer: Learning bird's s-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
[6] Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[7] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp. B. Varadarajan, Y. Shen, Y. Shen, Chai, C. Schmid, C. Li, and D. Anguelov. Tnt: Target-driven trajectory prediction. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 895-904. PMLR, 16-18 Nov 2021.
[8] Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7577-7586, 2021.
[9] Gu, Sun, and H. Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15303- 15312, 2021.
[10] N. Nayakanti, R. AI-Rfou, A. Zhou, K. Goel, S. Refaat, and B. Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2980 2987. IEEE, 2023.
[11] D. A. Pomerleau. Alvinm: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[12] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv: 18/2.0007, 2018.
[13] Li. F. Nie Q Sun, F. Da, and H. Zhao. Uncertainty decision transformer for stochastic driving environments. arXiv preprint (xXiv:2309.16399) 2023.

---

[14] Y. Zeng, H. Zhang, J. Zheng, J. Xia, G. Wei, Y. Wei, Y. Zhang, and T. Kong. What matters in training a gpt4-style language model with multimodal inputs? arXiv preprint 0rXiv23377024660 2023.
[15] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K. K. Wong, Z. Li, and H. Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. arXiv preprint arXiv:2310.01412, 2023.
[16] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4; Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304 2023.
[17] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. 2023.
[18] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-languee language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 22966, 2023.
[19] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, Fung, and S. Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning, 2023.
[20] Zhang, X. Dong, B. Wang, Y. Cao, C. Xu, L. Ouyang, Zhao, S. Ding. S. Zhang, H. Duan, W. Zhang. H. Yan, X. Zhang, W. Li, Li, K. Chen, C. He, Zhang, Y. Qiao, D. Lin, and J. Wang- Internlm-xcomposer A vision-language large model for advanced text-image com- prehension and composition, 2023.
[21] W. Wang, Q. Lv, W. Yu, W. Hong. J. Qi, Y. Wang, Ji, Z. Yang, L. Zhao, X. Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0309, 2023.
[22] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang. Y. Chebotar, P. Sermanet, D. Duckworth, Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng. Mordatch, and P. Florence. Palm-e: embodied multimodal language model, 2023.
[23] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.066 17, 2022.
[24] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski. T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, Levine, Lu, H. Michalewski, Mordatch, K. Pertsch, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-2: Vision- lasion-anguage-actionn models transfer web knowledge robotic control. In arXiv preprint arXiv:2307. 15818, 2023.
Rao,
[25] W. Huang, Wang. Zhang. Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[26] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.0886. 2023.
[27] R. Chekroun, M. Toromanoff, S. Hornauer, and F. Moutarde. Gri: General reinforced imitation and its application to vision-based autonomous driving. Robotics, 12(5): 127, 2023.
[28] D. Chen, V. Koltun, and P. Krähenbühl. Learning to drive from world rails. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15590-15599, 2021.
10

---

[29] M. Toromanoff, E. Wirbel, and F. Moutarde. End-to-end model- free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7153-7162, 2020.
[30] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang. S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8660-8669, 2019.
[31] B. Wei, M. Ren, W. Zeng. M. Liang, B. Yang, and R. Urtasun. Perceive, attend, and drive: Learning spatial attention for safe self-driving. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4875-4881. IEEE, 2021.
[32] Hu, A. Huang, J. Dolan, D. Held, and D. Ramanan. Safe local motion planning with self- supervised freespace forecasting. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12732-12741, 2021.
[33] S. Casas, A. Sadat, and R. Urtasun. Mp3: A unified model to map, perceive, predict and plan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14403-14412, 14412, 2021.
[34] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 17853-17868 2023.
[35] K. Renz, K. Chitta, O.-B. Mercea, A. Koepke, 7. Akata, and A. Geiger. Plant: Explain- able planning transformers via object-level representations, arXiv preprint arXiv:2210.14222, 2022.
[36] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li. End-to-end autonomous driving: Challenges and frontiers. arXiv preprint urXiv:2306. 16927, 2023.
[37] A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309. /7080, 2023.
[38] Z. Yang. X. Jia, H. Li, and J. Yan. A survey of large language models for autonomous driving. arXiv preprint arXiv:23/1.0104, 2023.
[39] D. Wu, W. Han, T. Wang, X. Dong, X. Zhang, and J. Shen. Referring multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14633-14642. 2023.
[40] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(113:22111133, 2013.
[41] Deruyttere, S. Vandenhende, D. Grujicic, L. Van Gool, and M.-F. Moens. Talk2car: Taking control of your self-driving car. arXiv preprint 19Xiv:1909,1088 2019.
[42] D. Wu, W. Han, T. Wang. Y. Liu, X. Zhang, and J. Shen. Language prompt for autonomous driving. arXiv preprint arXiv:2309.04379 2023.
[43] T. Qian, J. Chen, L. Zhuo, Y. Jiao, and Y.-G. Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario. arXiv preprint 133v:22305.18336 2023.
[44] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. F. Liong, Q Xu, A. Krishnan, Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
11

---

[45] J. Kim, A. Rohrbach, T. Darrell, J. Canny, and Z. Akata. Textual explanations for self-driving vehicles. In Proceedings of the European conference on computer vision (ECCV), pages 563- 578, 2018.
[46] Y. Xu, X. Yang, L. Gong, H.-C. Lin, T.-Y. Wu, Li, and N. Vasconcelos. Explainable object- induced action decision for autonomous vehicles. In Proceedings the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9523-9532, 2020.
[47] E. Sachdeva, N. Agarwal, S. Chundi, S. Roelofs, J. Li, B. Dariush, C. Choi, and M. Kochen- derfer. Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. arXiv preprint Xiw:22399006997, 2023.
[48] Malla, C. Choi, I. Dwivedi, J. Choi, and Li. Drama: Joint risk localization and cap- tioning in driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2023.
[49] B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang. Vad: Vectorized scene representation for efficient autonomous driving. arXiv preprint arXiv:2303. 12077, 2023.
[50] OpenAl. Gpt-4 technical report, 2023.
[51] Khurana, Hu, Dave, J. Ziglar, D. Held, and D. Ramanan. Differentiable raycasting for self- supeevupere occupancy forecasting. In European Conference Computer Vision, pages 353-369. Springer, 2022.
[52] S. Hu, L. Chen, Wu, H. J. Yan, and D. Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. European Conference on Computer Vision, pages 533-549. Springer, 2022.
[53] J.-T. Zhai, Z. Feng, Du, Mao, J.-J. Liu, Z. Tan, Zhang, X. Ye, and Wang. Rethink- ing the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint 205.5 2023.
[54] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[55] X. Chu, L. Qiao, X. Zhang. Xu, Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, Zhang, al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 2024.
[56] B. He, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S.-N. Lim. Ma- Imm: Memory-augmente large multimodal model for long-term video understanding. arXiv preprint arXivv2404.00576, 2024.
[57] Hu, L. Shen, and G. Sun. Squeeze-and-eecitatie networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[58] Y. F. Wei, C. Zhang, and H. Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024.
[59] T. Deruyttere, Vandenhende, D. Grujicic, L. Van Gool, and M.-F. Moens. Talk2car: Taking control of your self-driving car. arXiv preprint arXiv: 1909. 1088, 2019.
[60] Kim, A. Rohrbach, T. Darrell, J. Canny, and Akata. Textual explanations for self-driving vehicles. In Proceedings the European conference computer vision (ECCV), pages 563- 578, 2018.
12

---

[61] S. Malla, C. Choi, I. Dwivedi, J. H. Choi, and J. Li. Drama: Joint risk localization and cap- tioning in driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1043-1052, 2023.
[62] L. Xu, H. Huang, and J. Liu. Sutd-trafficqa: A question answering benchmark and efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9878-9888, 2021.
[63] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction wning. Advances in neural information processing systems, 36, 2024.
[64] Q. Team, Introducing qwenl.5. 15, February 2024. URL qwen1.5/.
[65] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint urXiv:2403.088 088005, 2024.
[66] S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024.
[67] M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 2024.
[68] H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16296-26306 2024.
[69] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- for image recognition at scale. arXiv preprint arXiv:2010.119999 2020.
[70] X. Chu, L. Qiao, X. Zhang, Xu, F. Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, B. Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766. 2024.
[71] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Mil- lican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[72] T. Cai, Y. Li, Z. Geng, H. Peng, and T. Dao. Medusa: Simple framework for accelerating llm generation with multiple decoding heads, 2023.
13

---

A SUP-AD Dataset
A.1 Meta-actions
Meta-action statistics. We use the meta-action sequence to formally represent the driving strategy. Meta actions are classified into 17 categories. We show the distribution of each meta-action being the firstsscond/third place in the meta-action sequence, as shown in Figure 5. indicates that the meta-actions are quite diverse the SUP-AD dataset. We also show the distribution of the length of meta-actions per scene Figure 6. Most scenes contain two or three meta-actions, and a few scenes with complex driving strategies contain four more meta-actions.
Annotation of meta-actions. The meta-action sequence for each driving scene is manually anno- tated based on the actual driving strategy in the future frames. These meta-actions are designed to encompass a complete driving strategy and are structured to be consistent with the future trajectory of the ego vehicle. They can be divided into three primary classes:
1. Speed-control actions. Discerned from acceleration and braking signals within the ego state data, these actions include These actions can be discerned from acceleration and brak- ing signals within the ego state data. They include speed up, slow down, slow down rapidly, straight slowly, go straight at constant speed, stop, wait, and reverse.
2. Turning actions. Deduced from steering wheel signals, these actions consist of turn left, turn right, and turn around.
3. Lane-control actions. Encompassing lane selection decisions, these actions are derived from a combination of steering wheel signals and either map or perception data. They involve change lane the left, change lane to the right, shift slightly the left. and shift slightly the right.
A.2 Scenario Categories
The SUP-AD dataset is comprised of 1,000 video clips of driving scenarios. As illustrated in Fig- ure 7, it encompasses wide range of driving scenarios, spanning over 40 categories. Below are explanations for some of the scenarios:
AEB Data: Automatic Emergency Braking (AEB) data.
Road Construction: A temporary work zone with caution signs, barriers, and construction equip- ment ahead.
Close-range Cut-ins: A sudden intrusion into the lane of the ego vehicle by another vehicle.
Roundabout: A type of traffic intersection where vehicles travel in a continuous loop.
Animals Crossing Road: Animals crossing the road front of the ego vehicle.
Braking: Brake is pressed by human driver of the ego vehicle.
Traffic Police Officers: Traffic police officers managing and guiding traffic.
Blocking Traffic Lights: A massive vehicle obscuring the visibility of the traffic signal.
Cutting into Other Vehiele: Intruding into the lane of another vehicle ahead.
Ramp: A curved roadway that connects the main road to the branch road in highway.
Debris on the Road: Road with different kinds of debris.
Narrow Roads: Narrow roads that require cautious navigation.
Pedestrians Popping Out: Pedestrians popping out in front the ego vehicle, requiring slowing down braking.
People on Bus Posters: Buses with posters, which may interfere the perception system.
14

---

400
First place
Second place
Third place
300
200
100
Change lane to the left
Change lane the right
Go straight at constant speed
Go straight slowly
Reverse
Shift slightly the left
Shift slightly the right
Slow down
Speed up
Speed up rapidly
Stop
Turn around
Turn left
Turn right
Wait
Figure 5: Distribution of each meta action being the first, second, and third place of the meta action sequence, respectively.
One meta action
26.3%
Two meta actions
37.4%
5.5%
Four or more meta actions
30.8%
Three meta actions
Figure 6: Distribution of the length of meta actions per scene.
15

---

Figure 7: Diverse driving scenarios in the SUP-AD dataset.
Merging into High Speed: Driving from low-speed road into high-speed road, requiring speed- ing up.
Barrier Gate: Barrier gate that can be raised obstructing the road.
Fallen Trees: Fallen trees on the road, requiring cautious navigation avoid potential hazards.
Complex Environments: Complex driving environments that requiring cautious navigation.
Mixed Traffic: A congested scenario where cars, pedestrians, and bicycles appear on the same or adjacent roadway.
Crossing Rivers: Crossing rivers by driving on the bridge.
16

---

Sursnny:
changes laxo waergeeay riglt-nay lasc, witho licyde
Closdy
Uniicall Object
Chonetematic
Taller tiver
volile
Inflscece:
Bloking vchicke
Options Line
Soac palk
Road Ensireement:
Liben
Meta Action:
Sam dona"" "Changg lax right.. "Ga araish slondo"1
Decuica Desciptian:
Slow and make therd's vehide ceeting froez the nghh nar brforz
charging lines right
Figure 8: An example of overturned bicycles and motorcycles in the SUP-AD dataset. A bicycle has fallen in front of the ego vehicle, requiring the ego vehicle to change lanes.
Screen: Roads with screens on one side, which may interfere the perception system.
Herds of Cattle and Sheep: A rural road with herds of cattle and sheep, requiring careful driving avoid causing distress to these animals.
Vulnerable Road Users: Road users which are more susceptible to injuries while using roads, such
as pedestrians, cyclists, and motorcyclists.
Road with Gallet: A dusty road with gallet scattered across the surface.
The remaining scenario categories are: Motorcycles and Trikes, Intersection, People carrying Um- brella, Vehicles Carrying Cars, Vehicles Carrying Branches, Vehicles with Pipes, Strollers, Children, Tunnel, Down Ramp, Sidewalk Stalls, Rainy Day, Crossing Train Tracks, Unprotected U-turns, Snowfall, Large Vehicles Invading, Falling Leaves, Fireworks, Water Sprinklers, Potholes, Over- turned Motorcycles, Self-ignition and Fire, Kites, Agricultural Machinery.
A.3 Annotation Examples
We provide more examples of annotation contents Figure 8, 9, 10, 11, 12, and 13. The scenario categories of these examples are overturned bicycles and motorcycles, herds of cattle and sheep, collapsed trees, crossing rivers, barrier gate, and snowfall respectively.
Scene Summmary
The vahide mnvding diright curcnt last, tarre of slon-maning aboad lanz.
Watter
Clocaly
Tine:
Daytinz
Critial Olsice:
Clame
hund of cown
Chameserstos
Slowlyy moving
aooal age
haflazccng
Lase Optone
Own Laze
drvving
Boad Enstroment: Monstan
Meta Aotiot:
['llww dona", "Go slowly
Decbiue Descijtion:
Reduce speed and proceed slowly wiilee mriitining s daaaoe fioes le 5r40 comn.
Figure 9: An example of herds of cattle and sheep in the SUP-AD dataset. A group of cattle move slowly in front of the ego vehicle, requiring the ego vehicle to proceed slowly and maintain safe distance from the cattle.
17

---

Oriical Object
Weadon
Cloaly
Uber
Thoe
Daytter
Chancteistis::
Leaning towads vibidee the Ic3 fore
our
mosing
Xita Action:
["Show down ruuidyy. "Sop". Wai'')
Decisio
belonn
Figure 10: An example of collapsed trees in the SUP-AD dataset. A tree suddenly falls towards the ego vehicle, requiring the ego vehicle decelerate immediately.
Sooe Summnay
The veliclo trvc consarr sped slong fhe curcet real towarts bikgge cheed. The hridge widh alows only single velide
Weather
Cloady
Time:
Daytime
Crikial Object:
Clas:
Narow bridge
Clanateristo:: Paralii widit
only a single
vebicle
sopplig
Land Optinee
No Lane Marking
Road Envionment: Namow Brddge
Mita Action
('Skon dowa", "Go smsight slowly
Desiton Descigtion
Bakk and doedenne, dive skonly twand dhe bridge without stopping
Figure 11: An example of crossing rivers in the SUP-AD dataset. The ego vehicle going across bridge of which width allows only single vehicle to pass, requiring the ego vehicle to drive without stopping.
vihidee
bocisocial ba cleed.
Wardan
Crtikal Ofjec:.
Time:
[lytine
Class
Croober
Clamstnisiis
Are estoneecshi
shead
Tane Optione: Marking
Metu Action:
[SSov donal" "Slup". "Wak""
Deckkas Decrtpior:
"Sow
harooral
onttinus
Figure 12: An example of barrier gate the SUP-AD dataset. A horizontal barrier blocks the entrance of park, requiring the ego vehicle stop and wait for permission to continue.
18

---

Summang:
ogo vehicel cunootly driving simnly slong following
Weateen
Sapwy
Objerr
Time
Daylim::
Optioss:
Macking
Road Eavinonenn:: Soon-soverro
Road
Maa Astion
Go anight slowly1
Decisor
forward
Figure 13: An example snowfall in the SUP-AD dataset. Most the road is covered by snow, requiring the ego vehicle move forward cautiously by following the snow-free tire tracks.
B Evaluation Method
The ability of an autonomous driving system accurately interpret driving scenes and make log- ical, suitable decisions is of paramount importance. As presented in this paper, the evaluation of VLMs in autonomous driving concentrates on two primary components: the evaluation of scene description/anallsii and the evaluation meta-actions.
B.1 Scene Description/Anallsi Evaluation
In terms of scene description/analysi evaluation, the process of interpreting and articulating driv- ing scenes is subject inherent subjectivity, there are numerous valid ways express similar descriptions textually, which makes it difficult to effectively evaluate the scene description using fixed metric. To overcome this challenge, we utilize GPT-4 to evaluate the similarity between the scene descriptions generated by the model and the manually annotated ground truth. Initially, we prompt GPT-4 to extract individual pieces of information from each scene description. Subse- quently, we score and aggregate the results based on the matching status each extracted piece of information.
The ground truth labels for scene descriptions encompass both environment descriptions and event summaries. Environmental condition description includes weather conditions, time conditions, road environment, and lane conditions. Event summaries are the characteristics and influence of critical objects. We employ GPT-4 to extract unique key information from both environment descriptions and event summaries. The extracted information then compared and quantified. Each matched pair is assigned a score, which is estimated based on the extent of the matching, whether complete, partial, absent. Instances of hallucinated information incur penalty, detracting from the overall score. The aggregate of these scores constitutes the scene description score.
1.0 Ammtcched + 0.5 X Oportial
Score
Rgt
0.25 x Ylhallucination
Rgt
The prompt for GPT-4 in evaluating scene descriptions is carefully designed, as shown in Table 5. Initially, a role prompt employed to establish as an intelligent and logical evaluator, possessing comprehensive understanding of appropriate driving styles. This is followed by specifying the input format, which informs GPT-4 that its task involves comparing output description with ground truth description. This comparison is based on the extraction and analysis of key information from
19

---

System Prompt:
You are smart and logical evaluator with extensive driving experience, you try your to give reasonable and logical evaluation result.
Input Prompt:
Given two driving scenario descriptions, one is the reference description, the other is the outpur deseription from a model. Please extract key information from reference description use comparison standards compure proximity evalunte the completeness of each piece of information. Key information extracted can be classified into environmental conditions and critical events. Environmental condition information includes weather, time, road environment, and lanc conditions. critical event deseribes the event related ego vehicles driving decisioon, which usually given the format of critical object verb'adj influence ego vehicle. An
output descriptions. Additionally.
example of critical event like this: A consiruction zone is blocking the ego lane and the right lame in front, so the car needs lo go around to avoid it. Now I will give you the reference deseription and the curput description.
The reference description:
Weather: Sunny.
Time: Day.
Road Environment: Urban. Lane Options: Left Lane, Own Lane, Right Lane.
Ego lane
Position: Middle Lane.
Critical Events: There are police officers ahead inspecting passing vehicles. The police offeer front of your lane is signaling you to stop for inspection. There also vehicles queuing the left lame. The right lame is merging lame and is currenily separated, you cannot change
lanes the right.
The ourput description:
Weather: Sunny.
Time: Day.
There is vehicle driving
distance front of you. are traflic police on the and right, the traflic police signal stop.
Compare the two descriptions using extracted key information and ourput similarities score on the scale from 0 to 1.0 and an explanation describe the logic of caleulating the similarit scale. Each matched key information scores 1.0, and each partially matched key information 0.5, Iallucinaations will penalized as well. any key information exists in the oulput but not the reference, ussigned. The final score is the total score divided by the number of key information extracted in the reference description. For example, the total is 1.5, and total number key information reference description then the final would be 1.5/3 0.5. Please outpur the detailed extraction and ovaluation process.
of 0.25 will be
Response:
Reference description key informatiin:
Weather: Sunny (envirommental conditions)
2. Time: Day (environmental conditions)
Road Envizonment: Urban (environmental conditions)
4. Lane Options: Left Lane, Own Lane, Right Lane (envirommental conditions)
5. Ego Lane Position: Middle Lane (envirommental conditions)
6. Police officers ahead inspocting vehicles (critical event)
7. The police officer in froat ego vehicle signalling to stop (critical event)
8. Vehicles queuing
left lane (critical 9. Cannot change to the right lane as its an merging lane and currently separated (critical event)
Output description key information:
1. Weather: Sunny.
2. Time: Day.
3. Vehicle driving in the distance in front of driver
4. Trallic police on the lefl and right
Traffic police signal stop
By comparing the output to the reference:
Key information
1.0 points(weather and time, police officers their signal to stop) Key information in the ourput not mentioned in reference, deduction of -0.25 points.
are matched,
No scores for:
Key information 3, 5, 8, and
Total score (1.0 1.0 1.0 1.0- 0925)/9=0003
Table 5: Evaluation process of scene description by GPT-4, Our proposed evaluation method is not only capable of extracting and evaluating information from structured scene descriptions but is also applicable unstructured descriptions.
both descriptions. Lastly, the prompt outlines the criteria for scoring, as well as the format for the evaluation output, ensuring a structured and systematic approach to the evaluation process.
B.2 Meta-action Evaluation
The evaluation process for the meta-action sequence must consider both the quantity and the sequen- tial arrangement of the matched meta-actions. We employ dynamic programming to compare the model's output and the annotated ground truth. Our dynamic programming approach is similar to the method utilized in identifying the longest common subsequence, albeit with two supplementary considerations.
The first consideration acknowledges the unequal weighting of different meta-actions. For instance, certain meta actions such as "Slow Down", "Wait", and "Go Straight Slowly" exhibit a greater em- phasis on attitude rather than action. The presence or absence of these actions from meta-action sequence does not alter the hasic semantic essence of driving decisions but rather modifies the driv-
20

---

ing strategy to be either more assertive or more cautious. For example, a meta action sequence of "Slow Down -> Stop -> Waif" conveys a similar driving decision as a sequence with only the meta action "Stop". Consequently, these sequences should not incur penalty comparable to other meta actions such as "Turn Left" or "Change Lane to the Right". Therefore, these are designated as "conservative actions". and a reduced penalty is applied when they do not match during sequence evaluation.
The second consideration addresses the potential semantic equality among different meta-action sequences. For example, the sequences "Change Lane to the Left -> Speed Up -> Go Straight At a Constant Speed -> Change Lane to the Right" and "Change Lane to the Left -> Speed Up Rapidly Go Straight At a Constant Speed -> Change Lane the Right" might both represent valid approaches overtaking slow-speed vehicle ahead. Recognizing that different meta-action sequences might convey similar meanings, we initially use GPT-4 generate variant sequences that have comparable semantic meanings, in addition to the unique ground truth meta-action sequence, as shown in Table 6. In the subsequent sequence- sequence-matching phase of the evaluation, all these variations, together with the manually annotated ground truth, are taken into consideration. The highest-scoring matching is then adopted as the definitive score for the final decision evaluation.
Prompt:
Given a predefined set of action tokens and sequence of reference driving actions, find all alterna- tive sequences of actions that have similar meanings but choose slightly different action tokens from the predefined action tokens.
Predefined action tokens: [Speed up. Slow down, Speed up rapidly, Slow down rapidly, Go straight slowly, Go straight at a constant speed, Turn left, Turn right, Change lane to the left, Change lane to the right, Shift slightly the left, Shift slightly to the right, Stop, Wait, Turn around, Reverse]. For example, if the reference driving action sequence is: Slow down -> Shift slightly to the right -> Go straight at constant speed, alternative action sequences with similar meanings could be:
1. Slow down -> Change lane to the right -> Go straight at a constant speed;
Slow down rapidly Shift slightly to the right -> Go straight at a constant speed:
3. Slow down -> Change lane to the right -> Go straight slowly;
Slow down -> Shift slightly to the right -> Go straight slowly.
Now the reference action sequence is: Slow down -> Shift slightly to the right -> Go straight at constant speed. What are the alternative action sequences?
Response:
Given the reference sequence: "Slow down -> Shift slightly the right -> Go straight constant
speed." " Alternative action sequences with similar meanings based on the provided action tokens are:
1. Slow down -> Change lane the right -> Go straight constant speed.
2. Slow down rapidly -> Shift slightly to the right -> Go straight at a constant speed.
3. Slow down -> Change lane to the right Go straight slowly.
4. Slow down -> Shift slightly to the right -> Go straight slowly. You' 've provided this example in
your question.
Based on the tokens and the context, these are indeed appropriate alternative sequences.
Table 6: Example of generating alternative action sequences by GPT-4.
The state of dynamic programming saved in a 2D matrix, wherein each row corresponds to meta action in the ground truth action sequence, and each column corresponds meta action in the model output action sequence, noted as r. The dynamic programming initiates recursive calcu- lations beginning from the first meta action of both sequences. Each element of the 2D matrix en- compasses the optimal total score at the current matching position, as well as the preceding matching condition that yielded the optimal matching. In our dynamic programming algorithm, three transi- tion equations govern distinct cases: Smissing for missing matching, Spodundant for redundant matching, and Smatching for successful matching. Successful matching occurs when the meta action identical the th position in the reference sequence and the th position in the model-generated sequence. In the case of missing matching, the meta action the th position in the reference sequence is unmatched, prompting a comparison with the r t position in the reference sequence and the ch
21

---

position in the action at the el position in the model-generated sequence is unmatched, leading to further exami- nation of the nd position the reference and the c th position in the model-generall sequence. The transformation equations for these cases are as follows:
sequence. Conversely, redundant matching implies that the meta
missiing
Pmissing
T.C redundant
T TT,
Predundant
(3)
S matching
-r-1,c-- c,,-1 Smatching
S max ( S. S
missinng* redunddant
matching!
where Smatching = 1.0 represents the reward score after successful matching. If an action considered missing or redundant is classified as conservative action, the penalties Punissing and Predundant are quantified as half of Smatching i.c., 0.5. Conversely, if an action is not conservative, both penalties are assigned the same magnitude as Smatching i.e., 1.0. This approach is based on the premise that omitting a crucial meta action or inaccurately introducing a non-existent one equally hampers the effectiveness of the action sequence. The final score Scoreactoo should be divided by the length of the selected reference meta-action sequence, formulated as follow:
S 5%.0
Score action action
(4)
N₂
C Co-tuning
To preserve the LLM's generalization capabilities during the fine-tuning process, we employed co- tuning with several additional datasets. These include the Talk2Car [59], BDDX [60], Drama [61], SUTD [62], and LLAVA [63] datasets. For each dataset, conducted random sampling in 1:1 ratio corresponding to the data volume of the SUP-AD and nuScenes datasets. Following this co- tuning approach, we found that the scores on the SUP-AD dataset, under the evaluation metrics of Scene Description and Meta Action, remained virtually unchanged, simultaneously ensuring the preservation the LLM's original capabilities and its generalization capacity.
D DriveVLM-Durd Onboard Deployment
SEEDBeuch
ReCCOCO 15%
SUP-AD 15%
Drivelm-QA 7.5%
Drivelm- Grounding
Driveln-Grounding
Realworld-VOa
Base
Avg.
5%
20%
7.5%
MobileLLaMAI. M.M.4 [55]
0.457
0.331
0.59
0.421
0,520
01.86
0.735
0.501
Owen-1.8B 164]
0.622
Gemma-28 [65]
0.439
0.345
0.571
0.330
0.510
0.680
0.721
0.507
MiniCPM-2244B [66]
0.482
0.379
0.64
0.444
0.539
016676
0.717
0,553
MobileLLaMA2.78 [55]
Phi3-3.8B [67]
0.538
0.435
0,688
0.608
0.604
0.697
0.743
0.592
Qwen-48 [64]
0.511
0.366
0.671
0.601
0,515
(1681
0,735
0,562
Qwan-1B
Table Performance of different LLMs on various datasets using the LLAVA-1.5 [68] architecture with ViT-L-336 [69] as the image encoder. Note that indicates using SigL.P----38 [54] as the image encoder.
Base LLM Due to the limited memory and bandwidth the vehicle's hardware, we cannot use overly large LLMs to maintain real-time inference. Therefore, we chose models with fewer than billion parameters. As shown in Table and 8, our experiments revealed that on the Orin architec- ture. the "wide and shallow" Qwen series (wider and fewer layers) models outperform "narrow and deep" models (narrower and more layers) in inference speed.
22

---

LLM
Presi Lauphoooo
Pratt lacky 10
Prefit Opkio
Dioak: (aib)
Ougus (sto))
Boook Bacacy (o
Monid Sine (CBB)
Lase Nam
Head Soz
Vocih Stac
40.9
255
Pa3-4, (67)
1945
1.3
195.3
-3
1.3
1.
53
0v:
5306
Mahrod LaMd 155)
A.T
Qnesd8 (64)
1073
0.57
1882
44.
1.39
135
40
128
153336
Qveel (64)
1075
0.3
4700
706
0.24
377
24
125
150936
Table 8: Inference performance of different LLMs after quantization and deployment OrinX
chip. The Qwen series achieved the best performance.
Bacidwa
Volee Lengih Natiol
Micibbaad Tond
XIMMMI
.52
2055
155
SLP-AD
15%
7.5%
DESE Desens-menccy
7.55
Reileoed VIA
155
155
WTTE
0.166
0.555
01543
VIE---336
Plod 152 CA-256
B413
0.383
0.82
0.327
0.343
0,001
03333
11330
1.503
NT14-445
Piad 1s2 CA-250
03335
0.391
1340
0154
0/00)
0715
0.4
0362
VIE---355
Plod 152 32
1398
0.333
RA11
0108
0.60
02001
0711
0491
0314
NTL--300
Dil a EA--20
D480)
03377
038
0.54
0735
0325
0354
VITL-336
DASIG CA-256
0.981
0.881
0.5
0160
075
0.56
0.56
VITL-330
DAdid SM LT
0413
0.408
0617
0.90
0.55
0/90
0737
0.57
0,30
DAS-
038
0023
022
BAT
VTTL-306
6515
570
0.300
1631
B615
0424
630
0338
11556
036
SYLUPLL-758
04538
6888
6212
1:50
576
0.563
Sy9-194-244--54
576
D454
0.367
B.51
0351
0776
0793
SgLIP-L5::2900
Table 9: Performance of different methods for scaling ViT's original input resolution to higher reso- lution. "CA" stands for cross-attention, "DM" the abbreviation for Dynamic Max, "PT" indicates the use of patch end token, "SM" stands for Spatial Merge, "LT" indicates the use of line end token, "AAP" the abbreviation for Adaptive Average Pooling, and "CD" stands for Convolutional Down- sampling. "PE" represents Position Embedding Interpolation. Among these methods, applying PE interpolation to SigLIP-L-384, transforming it to accept 768 resolution images as input, achieves a good trade-off between inference speed and performance.
Visual Encoder High-resolution images essential for fine-grained understanding in tonomous driving. As shown in Table 9, compared to the basic ViT model used visual encoder, we explored several options, including different GridPatch strategies and PE (Position Embedding) interpolation. Ultimately, for real-time inference, we selected the simpler SigLIP-L-384 model with PE interpolation, achieving high resolution input through original 384 resolution PE interpolation and fine-tuning parameters with additional conv layers.
Visual Token Compression To address the increased computational load from high-resolution images, we implemented LDPNetv2 [70] to reduce the number of image tokens by 75% without compromising performance, as shown in Table 10. Additionally, we enhanced performance by replacing avg- ayg-pooling layer with convolutional layer in LDPNetv2.
Speculative Sampling Speculative Sampling is used to accelerate inference by preemptively gen- erating likely outputs. This approach reduces the latency of generating predictions, achieving significant speedup without substantial loss in accuracy. As shown in Table 11, we test two specu- lative sampling method Medusa and Eagle with our inference framework designed specifically for OrinX chip. Eagle achieved a 2.7x speedup in decode latency compared to Medusa's 2.17x, making real-time vehicle deployment feasible.
Projaki
Orgin
Guseesol
Ospolino)
Prefliun)
MMDII
SE3DV]
BDD
Dristugpetnnng Realeaal-VOA RefOOCO
SUP-AD
LDeSav22 (oo
576
661.30
1U7.43 42
50.33
32.53
56.11
53.35
66.90
73.31
48.10
21.29
54.10
Pasiner Roasple:[P]] [21]
S76
576
637.20
655.20
40:35
50.93
58.84
12.31
08.37
71.00
40.51
10.00
+3.10
606 11
79
11.11
38.51
62
41.11
73.75
90.19
90:10
Fod stalfi
576
144
504.38
943
40.40
38.35
61.53
36/55
73.18
51.8
30.37
58.00
61
616.38
30 60
66.60
73.00
$1.00
10
Table 10: Performance different methods for visual token compression. Using LDPNetV2 to
compress the original tokens to 75% of the original token count, achieves the best trade-off between performance and speed.
23

---

Eugle 44f16.B
Base 44116
Eugle [58] + gifl6
Medusa [721 + qi116
Eagle qi116
Shrink Vocab Size(1024)
Quant Type
q4f16_1 (384, 900)
q4f16_1 (384, 960)
q4f16_ft (384, 960)
q4f16_ft (384, 960)
q4f16_ft
Input Size
(384, 960)
Prefill Tokens
604
604
613
604
613
Ourput
Prefill Speed (tok/s)
1818
1793
2160.64
2230
2175.35
Decade Speed (tok()
109
232.34
295.04
518.3
Prefill Latency (8)
0.332
0.340
0.284
0.276
0.274
Decode Latency
0.328
0.216
0.176
0.130
0.071
Table II: Performance of different speculative sampling method. "Shrink Vocab Size" means reduce the vocabulary to the 1024 most frequently used words. "q4f16_1" is 4-bit quanti- zation method using a 16-bit Moating-poin representation for efficient model compression, while "q4f16.ft" includes subsequent tinettnnin enhance performance post-quantization.
E Qualitative Results
To further demonstrate the effectiveness and robustness of our DLM, we provide additional visualization results in Figure 14, 15, 16, 17, and 18. In Figure 14, DriveVLM recognizes the slowly moving vehicle ahead and provides a driving decision to change lanes for overtaking. In Figures and 16, DriveVLM accurately identifies the type of unconventional vehicles and fallen tree, demonstrating its capability in recognizing long-tail objects. In Figure 17, the traffic police signaling proceed with hand gestures has been accurately captured by DL.. In Figure 18, DLi successfully recognizes the road environment of roundabout and generates planned trajectory with a curved path.
24

---

T-3
T-2 T-1
These few frames taken intervels from driving védeo<img><//mg. <imp>sfing> himp://mmmmmmp...mmm////m/ Suppose driving scenario,
scene
Scene Condition:
Sunny
Tigh:Night.
EnvironmenttUrren
Lane Own Lane, Right Lane.
Pasition: Leftmast
Please fist the critical objects the front view that affect ego velicle's is nexi, driving decisions, and box them
There are critical object in the front view. The objects include [SSdann]. Leceee//edrrr//eeeoooooe/--e (48 (08,)3)//oo
Matching Objects: Sedan at img region [404, 518, 633).
Matching information: Object in the SFV view (18.85, 1.77), history trajectory (113.9113), (9,,0.8), (4.3,0.4), (-01,00.1p. Descr loe cheracter stics the object and indicte its influence besed the metching information
Characteristics: Driving ahead
Infiuence:
vehicie.
Ego States:
Historical Trajectory (last seconds]: ( 5.32,0.04), (11047,0.16), (-15 38,038,) 20.44,00655
Navigation: MAINTAIN.
Please make a summary the current scene based on the information provided and your previous analysis.
Scene Condition:
Weather:Sunny
Time:Day Day.
Road Environmenttrban.
OptionsiDan Lane, Left
Ego Besttion.Rightmos Lane.
Cribial Events: The ego vehicle maintains constant speed along the current lane, with slow moving sedan ahead and oncomiin veticles the lane.
Based above analysis and current scenario, please determine the driving mete actions and decisions you will take next.
Meta Actions:
I'Change lame to the right, 'Go straight et constent speed). Dects Maintain safe distance from vehicle ahead, lanes the right, and
Besed on the above decisions and anelysis, please plan the 3s future trajectory of the ego vehicle.
DrieVEM
DriveVLM
System Prompi
Trajecory
Figure 14: Visualization of Drive output. DLM recognizes the slowly moving vehicle ahead and provides a driving decision for changing lanes to overtake.
25

---

T-3
T-2
T-1
regular from drivigg vido:xim-s//pp...omm <impp////mmm///m/........./ Suppose you driving this scenario, please describe the scene condition.
Condition:
Weather:Sunng Sunny.
Time:Day Day
Road Enwrammentt-rbam.
Options:Own Lane,
Ego Lane Righton:gghtmmtt Lane.
Pease list the critical oojects the front view that affect ego vehicle's next driving decisions, step step.
There are critical object in the front view The onjects Include WWater spraying vehicle).. Letrefsaater spraying whhidees/nnt-ccaoo///44444,
Matching
Water-spraying vehicle
596).
Matching Information: Object Describe the characteristics the object and indicte its influence based on the matching information
4.05), history trajectory: (/26.3441), (27.0,4.0), (282, (28.6,4))..
Choracteristics: Water-sgeeyin vehicle ahead on the current road.
Infueece: No influence.
Ego States:
Historical Trajectory (las: 2 seconds):((44. 0 51, (22,7 0.0.7770000 00,000 MAINTAIN. summary current information provided and your previous analysis.
Condition
Suntyer-Snnyy
Time:Day:
Road Enwrenment:ürban
Options:Own
Ego Lane Pos Righttttttt Lane.
Critical Events: The vehicle is driving straight the current road, and there is the
water-sprayin
ego
analysis actirs and decisions you will take next.
straight speed"). Decaions: Maintain safe distance from the vehicle ahead and continue driving.
Based on the above decisions and analysis, please plan the às future trajectory of the ego vehicle.
DriveVLM
System Promue
Plasned Trajectory
Figure 15: Visualization of Drive output. DLM has identified the type of the uncon- ventional vehicle ahead.
26

---

T-3
T-2
T-1
These are few Fremes taken at regular intervals from driwing vddeo<iigx</ings <ing>s/ings. Suppose you are driving this describe the scene condition.
Condition:
Sunny
Time:Day:
Road Erwranment:Urban
Lare Options:Own Lane, Ego Lane Rightion-ighhmmst Lane.
Please crttical the front affect vehicle's driving dectsions, box stap by st82.
object Object 1xtrefaFallen (eee://eeesbooomsss../////800000
objects
Unmatching Objects: img region
Describe the cs of objectt and indicte influence besed the matching information
Character iicks: in the lane on the front side the ego vehicle. Influence: Blocking the future path the ego vehicle.
Fge- Sta-States:
Historical Trajectory (lest seconds):l--6 21,-0
43,662 44,0.671]
Hevigation:
Please make a summary the current scene based the informetion provided and your previous analyis..
Scene Condition:
Wonther:Sunny.
Time-Day.
Road Envitonmentt.raan.
Options-Own
Ego Lane Risttion:iignmmost Lame.
Critical Events: The ego vehicle driving streight aheed constant speed
the current fallen blocking the lane shead.
Based the above analysis and the current scenarto, please determine the driving meta decisions take
Meta 'Shitt slightly straight Slow down, right fallen
driving.
dedisians analysis,
trajectory vahicle.
Trojecoory
Figure 16: Visualization of LLM's'LM's output. Drive precisely detect the fallen tree and its position, subsequently planning an appropriate detour trajectory.
27

---

T-2
T-1
These are at regular intervals driving wwwp:ciing/iipp..immpp... m/p.. Suppose you are driving in this scenarin, please describe the scane condition.
Condition:
Wlaaher:Cloudy
TimerDay.
Road Unironmmettriimm
Lane Opt Lane.
Lane PashionnONA.
objects that ago driving box stop by stop.
objects
objects (TTrfffi police". "Trffiic police')
Objectt 2i<ie0-Traffic /ttittec//fbccoooo11444444444444 535 741)</bmcc Object Traffi-Traffic polttee/ee-cccoei885555.... 736)</bax.
Unmatching 20 Objects:
img region (134, 523, 350, 741).
Describe the characteristics the object and indicte its Ind uence based the matching Information
Characteristics Traffic police the left side of road shead Influence: giaaling that the ego veh can proceed
Traffic
805, 793).
obyect
metching
Characteristics: Traffic police the right dide of the road ahead Influence: Signaling that the ego vehicle can proceed
Ego States:
Hisooiccll Trajectory (ast seconds)(]11 1.777.000, 65.0 o5,) 5.61...088.... wwt.:/.9......oooo
Navigation: MAINTAIN
Please make
of the besed the information provided and previous analysi.
Scone Condition:
WeathercCluddy
Road Enwronment
Lane Options Own Lane,
Ego Lane Position DNA.
Events: The sides road ahead signaliing proceed.
driving straight, traffic police
Based above analysis current scenario, please determine driving actions and decitoos you will take next.
Meta Actions: lo straight at constant speed").
Dections: Maintain constant speed and continue driving straight.
Based the above decisio is and analysis, please plan the future trajectory the vehicle.
XOX
818.00 0l5.0008.0001sc00040000000000000
DriseVLM
System Prompe
Plance Trajoctey
Figure 17: Visualization of Dri'e output. The traffic police signaling proceed with hand gestures has been accurately captured by DLi.
28

---

T-3
T-2
These are few frames taken at regular intervals from a driving videoxiips///ppp.com htmp:///mm...mmmmmmmmmm.. Suppose you are driving in this scenario, please describe the scene condition.
Scene Condition:
WeatherSunny ght
Road Environment Entrommenttouuddabouut
Lane Options inninnteeree intersection.
Posttion:Do not
Please list the crit cal objects in the front view that affect ago verici's next driving decisions, and box them step step.
There are critical object the front view objects include [SUV]] Object 1-tref-Water spraying wekccbee/eeSdonp/////0000 (ttp://9900.00095//////////
Matching Cijjcts: img region
Matching Information: Ooject view 116.23, 2.841, history trajectory: (14.7,.33,1133,,,7,, (33.0,9.)) (128.124 Describb ibe the characteristiss the object and indicte influence based on the matching Information
Characteristics: Driving around the roundsbout in front of the vehicle. influence: No influence.
Historical Trajectory (lest (esoodd//11--1 11-00.00 1111- 1- 84, 42)6-8 22.0 133) 20.11.0211..0000 -91) Navigation: MAINTAIN.
Please make
the current the information provided
Scene Condition:
SweathrrSunny
Time:Night.
Envroonment Roundabout.
Lane Options intersaciion.
Lane Positio:DDo not apply.
Oittical Events: The
roundabeut shead, impact ago
driving
scenario, please driing meta actions and decsions you will take next.
analyss
speed', right") Decisions: Maintain distance continue driving.
Basod on the above dedisions and analysis, plezse pian the future trajectory of the ago vehicle.
DaveVLM
Syste Prompt
Planned Trajedivry
Figure 18: Visualization of DriveVLM's output. DriveVI.M successfully recognizes the road environment of roundabout and generates a planned trajectory with curved path.
29


Reviews:
[{"review_id": "XWWPJ4jix2", "reviewer": "Reviewer_cCjc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the use of Vision-Language Models (VLM) to enhance Autonomous Vehicles’ (AV) scene understanding, reasoning, and planning performance. The proposed VLM uses Vision Transformers (ViT) as the image tokenizer and Qwen as the LLM backbone. The Visual Question Answering (VQA) datasets are custom-built using a combination of rules and human labor. The paper proposes two frameworks: DriveVLM - utilizes the proposed VLM for various AV tasks; DriveVLM Dual - Injecting perception information (critical object analysis) from DriveLLM into the traditional AV 3D perception module and using the low-frequency motion plan from DriveVLM as the initial plan in the traditional AV planning module. The DriveVLM Dual framework is validated using onboard experiments.", "summary_of_recommendation": "The major contribution is the annotated dataset (if the authors plan to release the annotated Nuscenes dataset) and the validation of driveVLM-dual in real AV systems. Additional planning evaluation results in long-tail scenarios could further improve the paper.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "See Q1 and Q2 in the main review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6MmXkwpHQ", "reviewer": "Reviewer_fMiT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors introduce DriveVLM, an autonomous driving system leveraging VLMs for enhanced scene understanding and planning capabilities. The proposed method integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, the authors propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline.", "summary_of_recommendation": "More experimental results and discussions are required to demonstrate the effectiveness of the proposed framework, which are elaborated in the \"Review\" section.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Please address the comments and questions in the \"Review\" section during the rebuttal phase. In particular, it would be great to provide more experimental results mentioned in the comments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uw8m7z5T7X", "reviewer": "Reviewer_1XCj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces DriveVLM, an autonomous driving system that leverages Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. The authors also propose DriveVLM-Dual, a hybrid system that combines the strengths of DriveVLM and the traditional autonomous driving pipeline.", "summary_of_recommendation": "The paper is well organized with sufficient hardware demonstration. So I recommend a weak accept for this paper.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "* How to handle the situation where one of the systems fails or makes significantly different decisions?\n* In the environment description, whether the extreme weather conditions will be taken into consideration, which might be beneficial for the end-to-end driving system.\n* In Appendix part, you mentioned meta actions are classified into 17 categories, but why there is only 15 categories in Figure 1 ?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In brief, the strengths of the paper are, the clarity of presentation, good experimental results, and technically sound framework. The Reviewers appreciate the authors effort on adequately addressing the reviewers' comments during the rebuttal stage. The paper has significantly improved its quality.
Decision: Accept