DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

Authors: xiaoyu  tian
junru  gu
bailin  li
yicheng  liu
yang  wang
zhiyong  zhao
kun  zhan
peng  jia
xianpeng  lang
hang  zhao
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:27:22

Abstract: A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments.

Main paper: 
 introduction: 
 Autonomous driving, with its great promise to revolutionize transportation, has been an active research area over the past two decades. A primary hurdle to a fully autonomous driving system is scene understanding [1] , which involves navigating complex, unpredictable scenarios such as adverse weather, intricate road layouts, and unforeseen human behaviors.Existing autonomous driving systems, typically comprising 3D perception, motion prediction, and planning, struggle with these scene understanding challenges. Specifically, 3D perception [2, 3, 4, 5] is limited to detecting and tracking familiar objects, omitting rare objects and their unique attributes; motion prediction [6, 7, 8, 9, 10] and planning [11, 12, 13] focus on trajectory-level actions, often neglecting the decision-level interactions between objects and vehicles.We introduce DriveVLM, a novel autonomous driving system that aims at these scene understanding challenges, capitalizing on the recent Vision-Language Models (VLMs) [14, 15, 16, 17] which have demonstrated exceptional prowess in visual comprehension and reasoning. Specifically, DriveVLM contains a Chain-of-Though (CoT) process with three key modules: scene description, scene analysis, and hierarchical planning. The scene description module linguistically depicts the driving environment and identifies critical objects in the scene; the scene analysis module delves into the characteristics of the critical objects and their influence on the ego vehicle; the hierarchical planning module formulates plans step-by-step, from meta-actions and decision descriptions to waypoints. These modules respectively correspond to the components of the traditional perception-predictionplanning pipeline, but the difference is that these modules tackle object perception, intention-level prediction and task-level planning, which were extremely challenging to cope with in the past.  
drivevlm: 
 DriveVLM takes images as input and outputs planning results through a CoT reasoning process, as illustrated in Figure 1 . The reasoning process can be divided into three modules: scene description (Section 3.1), scene analysis (Section 3.2), and hierarchical planning (Section 3.3). For realworld deployment, we propose a hybrid system, DriveVLM-Dual, in Section 3.4, which combines DriveVLM and the traditional pipeline, leveraging the strengths of both approaches.The scene description module identifies driving environment description and critical objects.Environment Description. Driving environments, such as weather and road conditions, have a non-negligible impact on driving difficulty. Therefore, the model is first prompted to output a linguistic description E of the driving environment, including several conditions: E = {E weather , E time , E road , E lane }, each representing a crucial aspect of the driving environment. The weather component, E weather , spans conditions from sunny to snowy, affecting visibility and traction. The time component, E time , distinguishes between daytime and nighttime, impacting driving strategies due to visibility changes. Road types, E road , such as urban or highway, introduce different challenges, while lane conditions, E lane , focus on current lane positioning and possible maneuvers, crucial for safe driving decisions.Critical Object Identification. Unlike traditional autonomous driving perception modules, which detect all objects within a specific range, we solely focus on identifying critical objects that are most likely to influence the current scenario, inspired by human cognitive processes during driving. Each critical object, denoted as O c , contains two attributes: the object category c and its approximate bounding box coordinates b(x1, y1, x2, y2) on the image. Taking advantage of the pre-trained vision encoder, DriveVLM can identify long-tail critical objects that may elude typical 3D object detectors, such as road debris or unusual animals.In the traditional autonomous driving pipeline, the prediction module typically concentrates on forecasting the future trajectories of objects. The emergence of advanced vision-language models has provided us with the ability to perform a more comprehensive analysis of the current scene. The scene-level analysis summarizes all the critical objects together with the environmental description. This summary gives a comprehensive understanding of the scene, and is fed into the following planning module.Critical Object Analysis. DriveVLM characterizes critical objects in three aspects: static attributes C s , motion states C m , and particular behaviors C b . Static attributes C s describe inherent properties of objects, such as a roadside billboard's visual cues or a truck's oversized cargo, which are critical in preempting and navigating potential hazards. Motion states C m describe an object's dynamics over a period, including position, direction, and action-characteristics that are vital in predicting the object's future trajectory and potential interactions with the ego vehicle. Particular behaviors C b refer to special actions or gestures of an object that could directly influence the ego vehicle's next driving decisions. We do not require the model to analyze all three characteristics for all objects. In practice, only one or two characteristics apply to a critical object. Upon analyzing these characteristics, DriveVLM then predicts the potential influence I of each critical object on the ego vehicle.The scene-level summary is then combined with the route, ego pose and velocity to form a prompt for planning. Finally, DriveVLM progressively generates driving plans, in three stages: metaactions, decision description, and trajectory waypoints.Meta-actions A. A meta-action, denoted as a i , represents a short-term decision of the driving strategy. These actions fall into 17 categories, including but not limited to acceleration, deceleration, turning left, changing lanes, minor positional adjustments, and waiting. To plan the ego vehicle's future maneuver over a certain period, we generate a sequence of meta-actions. Action pertains to meta actions such as 'turn', 'wait', or 'accelerate'. Subject refers to the interacting object, such as a pedestrian, a traffic signal, or a specific lane. Duration indicates the temporal aspect of the action, specifying how long it should be carried out or when it should start.Trajectory Waypoints W . Upon establishing the decision description D, our next phase involves the generation of corresponding trajectory waypoints. These waypoints, denoted by W = {w 1 , w 2 , ..., w n }, w i = (x i , y i ), depict the vehicle's path over a certain future period with predetermined intervals ∆t. We map these numerical waypoints into language tokens for auto-regressive generation.To mitigate the challenges of high latency and imprecise spatial and motion understanding in VLMs, we propose DriveVLM-Dual, a collaboration between DriveVLM and the traditional autonomous driving system. This novel approach involves two key strategies: incorporating 3D perception for critical object analysis, and high-frequency trajectory refinement.Integrating 3D Perception. We represent objects detected by a 3D detector asO 3D = {c i 3D , b i 3D }, where b i3D denotes the i-th bounding box and c i 3D denotes its category. These 3D bounding boxes are then back-projected onto 2D images to derive corresponding 2D bounding boxes b i 2D . We conduct IoU matching between these 2D bounding boxes b i 2D and b j c . b j c are the bounding boxes of previously identified critical objects O critical = {c j c , b j c }. We classify critical objects that meet a certain approximate IoU threshold and belong to the same category as matched critical objects O matched c , defined asO matched c = {c j c , b j c }, if c j c = c i 2D and aIoU(b j c , b i 2D ) > τ, where aIoU(b j c , b i 2D ) = S b j c ∩b i 2D S b i 2D ,Those critical objects without a corresponding match in the 3D data are noted as O unmatched c .In the scene analysis module, for O matched c , the center coordinates, orientations, and historical trajectories of the corresponding 3D objects are used as language prompts for the model, assisting in , analysis relies solely on the language tokens derived from the image. This design enables DriveVLM-Dual to understand the locations and motions of critical objects more accurately, enhancing the overall performance.High-frequency Trajectory Refinement. To achieve real-time, high-frequency inference capabilities, we integrate it with a conventional planner to form a slow-fast dual system, combining the advanced capabilities of DriveVLM with the efficiency of traditional planning methods. After obtaining a trajectory from DriveVLM at low frequency, denoted as W slow , we take it as a reference trajectory for a classical planner for high-frequency trajectory refinement. In the case of an optimization-based planner, W slow serves as the initial solution for the optimization solver. For a neural network-based planner, W slow is used as an input query, combined with additional input features f , and then decoded into a new planning trajectory denoted as W fast . The formulation of this process can be described as:EQUATIONThis refinement step ensures that the trajectory produced by DriveVLM-Dual (1) achieves higher trajectory quality, and (2) meets real-time requirements. In practice, the two branches operate asynchronously in a slow-fast manner, where the planner module in the traditional autonomous driving branch can selectively receive trajectory from the VLM branch as additional input. 
task and dataset: 
 To fully exploit the potential of DriveVLM and DriveVLM-Dual in handling complex and long-tail driving scenarios, we formally define a task called Scene Understanding for Planning (Section 4.1), together with a set of evaluation metrics (Section 4.2). Furthermore, we propose a data mining and annotation protocol to curate a scene understanding and planning dataset (Section 4.3).The input of Scene Understanding for Planning task comprises multi-view videos V from surrounding cameras and optionally 3D perception results P from a perception module. And its output includes the following components: Scene Description s: Composed of different conditions, objectlevel analysis and scene-level summary; Meta Actions A: A sequence of actions representing tasklevel maneuvers; Decision Description D: A detailed account of the driving decisions; Trajectory Waypoints W : The waypoints outlining the planned trajectory of the ego vehicle.To comprehensively evaluate a model's performance, we care about its interpretation of the driving scene and the decisions made. Therefore, our evaluation has two aspects: scene description/analysis evaluation and meta-action evaluation. In the Scene description/analysis evaluation, we leverage a structured evaluation approach by prompting the GPT-4 to extract individual pieces of information from each scene description. Subsequently, we ask the GPT-4 to score and aggregate the results based on the matching status of each extracted piece of information. For Meta-action evaluation, we focus on the sequence of decision-making actions defined as meta-actions, comparing these sequences generated by the model against a ground truth using dynamic programming. To ensure the robustness of our evaluation, we use GPT-4 to generate semantically equivalent alternatives to the ground truth, with the highest similarity score determining the final decision-making accuracy. Further details on these metrics are elaborated in Appendix B.We propose a comprehensive data mining and annotation pipeline, shown in Figure 3 , to construct a Scene Understanding for Planning (SUP-AD) Dataset for the proposed task. This process begins with long-tail object mining and challenging scenario mining from a large database to collect special objects and scenes like weird-shaped vehicles and road debris. For each scene, Keyframe selection is performed to identify crucial moments for decision-making, which are then annotated using a specialized Scene Annotation tool. Details of the construction process and dataset statistics are available in the Appendix A. 
settings: 
 We test DriveVLM and DriveVLM-Dual on our proposed SUP-AD dataset and nuScenes dataset [44] . The setup details of our model and training details are listed in Appendix C.SUP-AD Dataset. The SUP-AD dataset is a dataset built by our proposed data mining and annotation pipeline. It is divided into train, validation, and test splits with a ratio of 7.5 : 1 : 1.5. We train models on the training split and use our proposed scene description and meta-action metrics to evaluate model performance on the test split. We also employ co-tuning with additional datasets to ensure the generalization of the LLM is not compromised. For details, see Appendix C.nuScenes Dataset. The nuScenes dataset is a large-scale driving dataset of urban scenarios with 1000 scenes, where each scene lasts about 20 seconds. Following previous works [34, 49] , we adopt Displacement Error (DE) and Collision Rate (CR) as metrics to evaluate models' performance.We present the performance of our proposed DriveVLM with several large visionlanguage models and compare them with GPT-4V, as shown in Table 1 . DriveVLM, utilizing Qwen-VL as its backbone, achieves the best performance due to its strong capabilities in question answer- ing and flexible interaction compared to the other open-source VLMs. Although GPT-4V exhibits robust capabilities in vision and language processing, its inability to undergo fine-tuning, restricting it solely to in-context learning, often results in the generation of extraneous information during scene description tasks. Under our evaluation metric, the additional information is frequently classified as hallucination, consequently leading to lower scores.nuScenes. As shown in Table 2 , DriveVLM-Dual achieves state-of-the-art performance on the nuScenes planning task when cooperating with VAD. It demonstrates that our method, although tailored for understanding complex scenes, also excels in ordinary scenarios.Model Design. To better understand the significance of our designed modules in DriveVLM, we conduct ablations on different combinations of modules, as shown in Table 3 . The inclusion of critical object analysis enables our model to identify and prioritize important elements in the driving environment, enhancing the decision-making accuracy for safer navigation. Integrating 3D perception data, our model gains a refined understanding of the surroundings and achieves precise predictions.Traditional AD Pipeline. To demonstrate the generalization of our dual system design, we test DriveVLM-Dual with different traditional autonomous driving pipelines on the validation set of nuScenes. As illustrated in Table 4 , our proposed DriveVLM-Dual adapts well to different traditional AD pipelines. While a standalone MLP method shows a notable performance gap compared to VAD, both variants of DriveVLM-Dual achieve nearly identical performance, underscoring the efficacy and robustness of our dual system design.Qualitative results of DriveVLM are shown in Figure 4 . In Figure 4a , DriveVLM accurately predicts the current scene conditions and incorporates well-considered planning decisions regarding the cyclist approaching us. In Figure 4b , DriveVLM effectively comprehends the gesture of the traffic police ahead, signaling the ego vehicle to proceed, and also considers the person riding a tricycle on the right side, thereby making sensible driving decisions. These qualitative results demonstrate our model's exceptional ability to understand complex scenarios and make suitable driving plans. More visualization of our model's output is shown in the Appendix E.  
onboard deployment and testing: 
 We deploy DriveVLM-Dual on an autonomous vehicle equipped with two OrinX processors, with the fast end-to-end driving system on OrinX-1 and the slower DriveVLM on OrinX-2. The total computational power is 508 TOPS (including both sparse and DLA computational power). Given the vehicle's hardware constraints, we select models with fewer than 4 billion parameters for realtime inference. The Qwen series [18] demonstrate superior inference speed due to its wide and shallow architecture, meeting the vehicle's speed requirements with an average total time for prefill and decode within 400ms. For the visual encoder, we choose the SigLIP-L-384 [54] model with PE interpolation fine-tuning, optimizing for high resolution and fine-grained understanding. To handle the computational load from high-resolution images, we use LDPNetv2 [55] , reducing image tokens by 75% without compromising performance, and cropped non-informative regions from images. To better assess object motion, we employe a memory bank strategy [56] for storing visual features from historical frames, enhancing temporal context with SE [57] blocks for weighted fusion. We utilize the Eagle [58] speculative sampling technique, achieving a 2.7x speedup in decode latency with our custom inference framework on the Orin chip, ensuring feasible real-time deployment for autonomous driving. Through these designs and optimizations, the DriveVLM can achieve a real-time inference speed of approximately 400ms per inference. The end-to-end fast system has a real-time speed of 130 ms per inference, and since the two systems run asynchronously, the overall system latency is about 130 ms. For detailed deployment techniques and model performance, please refer to Appendix D. 
limitations: 
 Our VLM model is obtained by continuing pretraining and fine-tuning an existing pretrained VLM. Consequently, the upper bound of VLM's capabilities is inherently limited by that of the original pretrained VLM. Moreover, current VLMs exhibit certain deficiencies in spatial understanding, such as position and distance, which is why we integrate 3D perception results into VLM's input. Additionally, current VLMs are susceptible to "hallucination" issues, which can potentially lead to safety concerns. Completely eliminating such hallucinations remains a challenge, though this is an inherent limitation of VLMs and not specific to our method. 
conclusion: 
 In summary, we introduce DriveVLM and DriveVLM-Dual. DriveVLM leverages VLMs, significantly progressing in interpreting complex driving environments. DriveVLM-Dual further enhances these capabilities by synergizing existing 3D perception and planning approaches, effectively addressing the spatial reasoning and computational challenges inherent in VLMs. Moreover, we define a scene understanding for planning task for autonomous driving, together with evaluation metrics and dataset construction protocol. DriveVLM and DriveVLM-Dual have surpassed the state-of-theart methods on the public and our benchmarks, especially in handling intricate and dynamic scenarios. Finally, we have verified the effectiveness of DriveVLM-Dual through onboard deployment and testing on a production vehicle. 
 
 Appendix: A.1 Meta-actions Meta-action statistics. We use the meta-action sequence to formally represent the driving strategy. Meta actions are classified into 17 categories. We show the distribution of each meta-action being the first/second/third place in the meta-action sequence, as shown in Figure 5 . It indicates that the meta-actions are quite diverse in the SUP-AD dataset. We also show the distribution of the length of meta-actions per scene in Figure 6 . Most scenes contain two or three meta-actions, and a few scenes with complex driving strategies contain four or more meta-actions.Annotation of meta-actions. The meta-action sequence for each driving scene is manually annotated based on the actual driving strategy in the future frames. These meta-actions are designed to encompass a complete driving strategy and are structured to be consistent with the future trajectory of the ego vehicle. They can be divided into three primary classes: The SUP-AD dataset is comprised of 1,000 video clips of driving scenarios. As illustrated in Figure 7 , it encompasses a wide range of driving scenarios, spanning over 40 categories. Below are explanations for some of the scenarios: Merging into High Speed: Driving from a low-speed road into a high-speed road, requiring speeding up.Barrier Gate: Barrier gate that can be raised obstructing the road.Fallen Trees: Fallen trees on the road, requiring cautious navigation to avoid potential hazards.Complex Environments: Complex driving environments that requiring cautious navigation.Mixed Traffic: A congested scenario where cars, pedestrians, and bicycles appear on the same or adjacent roadway.Crossing Rivers: Crossing rivers by driving on the bridge.The ego vehicle is currently driving slowly along a snow-covered road, following the snow-free tire tracks on the road surface. The ability of an autonomous driving system to accurately interpret driving scenes and make logical, suitable decisions is of paramount importance. As presented in this paper, the evaluation of VLMs in autonomous driving concentrates on two primary components: the evaluation of scene description/analysis and the evaluation of meta-actions.In terms of scene description/analysis evaluation, the process of interpreting and articulating driving scenes is subject to inherent subjectivity, as there are numerous valid ways to express similar descriptions textually, which makes it difficult to effectively evaluate the scene description using a fixed metric. To overcome this challenge, we utilize GPT-4 to evaluate the similarity between the scene descriptions generated by the model and the manually annotated ground truth. Initially, we prompt GPT-4 to extract individual pieces of information from each scene description. Subsequently, we score and aggregate the results based on the matching status of each extracted piece of information.The ground truth labels for scene descriptions encompass both environment descriptions and event summaries. Environmental condition description includes weather conditions, time conditions, road environment, and lane conditions. Event summaries are the characteristics and influence of critical objects. We employ GPT-4 to extract unique key information from both environment descriptions and event summaries. The extracted information is then compared and quantified. Each matched pair is assigned a score, which is estimated based on the extent of the matching, whether complete, partial, or absent. Instances of hallucinated information incur a penalty, detracting from the overall score. The aggregate of these scores constitutes the scene description score.EQUATIONThe prompt for GPT-4 in evaluating scene descriptions is carefully designed, as shown in Table 5 .Initially, a role prompt is employed to establish as an intelligent and logical evaluator, possessing a comprehensive understanding of appropriate driving styles. This is followed by specifying the input format, which informs GPT-4 that its task involves comparing an output description with a ground truth description. This comparison is based on the extraction and analysis of key information from System Prompt: You are a smart and logical evaluator with extensive driving experience, and you will try your best to give reasonable and logical evaluation result.Given two driving scenario descriptions, one is the reference description, the other is the output description from a model. Please extract key information from the reference description and use it as comparison standards to compare the proximity of the output descriptions. Additionally, evaluate the completeness of each piece of information. Key information extracted can be classified into environmental conditions and critical events. Environmental condition information includes weather, time, road environment, and lane conditions. A critical event describes the event related to ego vehicleś driving decisions, which is usually given in the format of critical object + verb/adj + influence on ego vehicle. An example of a critical event is like this: A construction zone is blocking the ego lane and the right lane in front, so the car needs to go around to avoid it. Now I will give you the reference description and the output description. Compare the two descriptions using extracted key information and output a similarities score on the scale from 0 to 1.0 and an explanation to describe the logic of calculating the similarity scale. Each matched key information scores 1.0, and each partially matched key information scores 0.5. Hallucinations will be penalized as well. If any key information exists in the output but not in the reference, a score of -0.25 will be assigned. The final score is the total score divided by the number of key information extracted in the reference description. For example, if the total score is 1.5, and the total number of key information in reference description is 3, then the final score would be 1.5/3 = 0.5. Please output the detailed extraction and evaluation process.Reference description key information: Table 5 : Evaluation process of scene description by GPT-4. Our proposed evaluation method is not only capable of extracting and evaluating information from structured scene descriptions but is also applicable to unstructured descriptions.both descriptions. Lastly, the prompt outlines the criteria for scoring, as well as the format for the evaluation output, ensuring a structured and systematic approach to the evaluation process.The evaluation process for the meta-action sequence must consider both the quantity and the sequential arrangement of the matched meta-actions. We employ dynamic programming to compare the model's output and the annotated ground truth. Our dynamic programming approach is similar to the method utilized in identifying the longest common subsequence, albeit with two supplementary considerations.The first consideration acknowledges the unequal weighting of different meta-actions. For instance, certain meta actions such as "Slow Down", "Wait", and "Go Straight Slowly" exhibit a greater emphasis on attitude rather than action. The presence or absence of these actions from a meta-action sequence does not alter the basic semantic essence of driving decisions but rather modifies the driv-ing strategy to be either more assertive or more cautious. For example, a meta action sequence of "Slow Down -> Stop -> Wait" conveys a similar driving decision as a sequence with only the meta action "Stop". Consequently, these sequences should not incur a penalty comparable to other meta actions such as "Turn Left" or "Change Lane to the Right". Therefore, these are designated as "conservative actions", and a reduced penalty is applied when they do not match during sequence evaluation.The second consideration addresses the potential semantic equality among different meta-action sequences. For example, the sequences "Change Lane to the Left -> Speed Up -> Go Straight At a Constant Speed -> Change Lane to the Right" and "Change Lane to the Left -> Speed Up Rapidly -> Go Straight At a Constant Speed -> Change Lane to the Right" might both represent valid approaches to overtaking a slow-speed vehicle ahead. Recognizing that different meta-action sequences might convey similar meanings, we initially use GPT-4 to generate variant sequences that have comparable semantic meanings, in addition to the unique ground truth meta-action sequence, as shown in Table 6 . In the subsequent sequence-matching phase of the evaluation, all these variations, together with the manually annotated ground truth, are taken into consideration. The highest-scoring matching is then adopted as the definitive score for the final decision evaluation.Given a predefined set of action tokens and a sequence of reference driving actions, find all alternative sequences of actions that have similar meanings but choose slightly different action tokens from the predefined action tokens. The state of dynamic programming is saved in a 2D matrix, wherein each row corresponds to a meta action in the ground truth action sequence, and each column corresponds to a meta action in the model output action sequence, noted as S r, c . The dynamic programming initiates recursive calculations beginning from the first meta action of both sequences. Each element of the 2D matrix encompasses the optimal total score at the current matching position, as well as the preceding matching condition that yielded the optimal matching. In our dynamic programming algorithm, three transition equations govern distinct cases: S missing for missing matching, S redundant for redundant matching, and S matching for successful matching. Successful matching occurs when the meta action is identical at the r th position in the reference sequence and the c th position in the model-generated sequence.In the case of missing matching, the meta action at the r th position in the reference sequence is unmatched, prompting a comparison with the r -1 th position in the reference sequence and the c th position in the model-generated sequence. Conversely, redundant matching implies that the meta action at the c th position in the model-generated sequence is unmatched, leading to further examination of the r th position in the reference and the c -1 th position in the model-generated sequence.The transformation equations for these cases are as follows:S r, c missing = S r-1, c -p missing , S r, c redundant = S r, c-1 -p redundant , S r, c matching = S r-1, c-1 + s matching , S r, c = max(S r, c missing , S r, c redundant , S r, c matching ),where s matching = 1.0 represents the reward score after a successful matching. If an action considered missing or redundant is classified as a conservative action, the penalties p missing and p redundant are quantified as half of s matching , i.e., 0.5. Conversely, if an action is not conservative, both penalties are assigned the same magnitude as s matching , i.e., 1.0. This approach is based on the premise that omitting a crucial meta action or inaccurately introducing a non-existent one equally hampers the effectiveness of the action sequence. The final score Score action should be divided by the length of the selected reference meta-action sequence, formulated as follow:EQUATIONTo preserve the LLM's generalization capabilities during the fine-tuning process, we employed cotuning with several additional datasets. These include the Talk2Car [59] , BDDX [60] , Drama [61] , SUTD [62] , and LLAVA [63] datasets. For each dataset, we conducted random sampling in a 1:1 ratio corresponding to the data volume of the SUP-AD and nuScenes datasets. Following this cotuning approach, we found that the scores on the SUP-AD dataset, under the evaluation metrics of Scene Description and Meta Action, remained virtually unchanged, simultaneously ensuring the preservation of the LLM's original capabilities and its generalization capacity. Table 7 : Performance of different LLMs on various datasets using the LLAVA-1.5 [68] architecture with ViT-L-336 [69] as the image encoder. Note that * indicates using SigLIP-L-384 [54] as the image encoder.Base LLM Due to the limited memory and bandwidth of the vehicle's hardware, we cannot use overly large LLMs to maintain real-time inference. Therefore, we chose models with fewer than 4 billion parameters. As shown in Table 7 and 8 , our experiments revealed that on the Orin architecture, the "wide and shallow" Qwen series (wider and fewer layers) models outperform "narrow and deep" models (narrower and more layers) in inference speed. Visual Token Compression To address the increased computational load from high-resolution images, we implemented LDPNetv2 [70] to reduce the number of image tokens by 75% without compromising performance, as shown in Table 10 . Additionally, we enhanced performance by replacing avg-pooling layer with convolutional layer in LDPNetv2.Speculative Sampling Speculative Sampling is used to accelerate inference by preemptively generating likely outputs. This approach reduces the latency of generating predictions, achieving a significant speedup without substantial loss in accuracy. As shown in To further demonstrate the effectiveness and robustness of our DriveVLM, we provide additional visualization results in Figure 14 , 15, 16, 17, and 18. In Figure 14 , DriveVLM recognizes the slowly moving vehicle ahead and provides a driving decision to change lanes for overtaking. In Figures 15 and 16 , DriveVLM accurately identifies the type of unconventional vehicles and a fallen tree, demonstrating its capability in recognizing long-tail objects. In Figure 17 , the traffic police signaling to proceed with hand gestures has been accurately captured by DriveVLM. In Figure 18 , DriveVLM successfully recognizes the road environment of a roundabout and generates a planned trajectory with a curved path.  


Decision: In brief, the strengths of the paper are, the clarity of presentation, good experimental results, and technically sound framework. The Reviewers appreciate the authors effort on adequately addressing the reviewers' comments during the rebuttal stage. The paper has significantly improved its quality.

Reviews: review: title: title: Additional planning evaluation on long-tail scenarios could further improve the paper.
paper_summary: paper_summary: This paper studies the use of Vision-Language Models (VLM) to enhance Autonomous Vehicles’ (AV) scene understanding, reasoning, and planning performance. The proposed VLM uses Vision Transformers (ViT) as the image tokenizer and Qwen as the LLM backbone. The Visual Question Answering (VQA) datasets are custom-built using a combination of rules and human labor. The paper proposes two frameworks: DriveVLM - utilizes the proposed VLM for various AV tasks; DriveVLM Dual - Injecting perception information (critical object analysis) from DriveLLM into the traditional AV 3D perception module and using the low-frequency motion plan from DriveVLM as the initial plan in the traditional AV planning module. The DriveVLM Dual framework is validated using onboard experiments.
main_review: main_review: Strengths:

1. This paper is well written, and the logic is clear.

2. The annotated dataset could greatly help the VLM for AV community if the authors plan to release the Nuscenes dataset.

3. The proposed DriveVLM dual is validated using onboard experiment

Weakness:

Q1. It has been shown that using ego state along can achieve SOTA performance in the Nuscenes dataset. I am wondering how the driveVLM performs without the ego state information.

Q2. I think the main reason for using LLM for driving is that it contains reasonably good knowledge about how to drive in complex long-tail settings that traditional ML-based planner may struggle with. Even though the authors spent effort on annotating long-tail scenarios,  the planning evaluation seems to only focuses on the entire validation dataset. This makes it difficult to evaluate can LLM truly help planning in long-tail settings. Since the authors already annotated and identified these critical driving scenarios, it would be great if the L2 and collision metrics on these long-tail settings could be reported.
questions: questions: See Q1 and Q2 in the main review.
limitations: limitations: 1
review_summary: review_summary: The major contribution is the annotated dataset (if the authors plan to release the annotated Nuscenes dataset) and the validation of driveVLM-dual in real AV systems. Additional planning evaluation results in long-tail scenarios could further improve the paper.
title: title: Paper Review
paper_summary: paper_summary: The authors introduce DriveVLM, an autonomous driving system leveraging VLMs for enhanced scene understanding and planning capabilities. The proposed method integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, the authors propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline.
main_review: main_review: Strengths:
1. The paper is generally well-written and easy to follow.
2. The paper provides thorough experimental results on both the nuScenes dataset and a custom SUP-AD dataset, showcasing the systems' capabilities.
3. The demo video provides interesting real-world demonstrations of the proposed method that were deployed on a real vehicle.

Weaknesses:
1. The proposed method involves a large VLM, which requires a huge amount of computational resources, which may lead to latency in practical deployment. However, there seems no explicit discussion on the running time of the model and the effect of inference latency on the end-to-end framework performance. How much does the frequency of the "low-frequency" component influence the performance? It would be great to provide more discussions on the computational efficiency and the required computational resources onboard autonomous vehicle of the proposed framework.
2. The authors claim that DriveVLM can identify long-tail critical objects. It would be better to provide a more concrete definition of the long-tail critical objects. How you decide which objects are long-tail ones? It would be better to provide quantitative and qualitative results on the evaluation of long-tail cases?
3. There is no discussion on the limitations of the proposed framework.
questions: questions: Please address the comments and questions in the "Review" section during the rebuttal phase. In particular, it would be great to provide more experimental results mentioned in the comments.
limitations: limitations: 1
review_summary: review_summary: More experimental results and discussions are required to demonstrate the effectiveness of the proposed framework, which are elaborated in the "Review" section.
title: title: Paper review of submission 592
paper_summary: paper_summary: This paper introduces DriveVLM, an autonomous driving system that leverages Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. The authors also propose DriveVLM-Dual, a hybrid system that combines the strengths of DriveVLM and the traditional autonomous driving pipeline.
main_review: main_review: Strengths:
* Introduces DriveVLM and DriveVLM-Dual, which leverage Vision - Language Models (VLMs) to enhance scene understanding and planning in autonomous driving, providing a novel solution to address the long tail challenges in this field.
* Proposes the DriveVLM - Dual hybrid system that combines the strengths of DriveVLM and the traditional autonomous pipeline, improving spatial reasoning and real-time planning capabilities.
* Defines a scene understanding for planning task, along with evaluation metrics, and constructs a dataset (SUP-AD) through a data mining and annotation pipeline, which is valuable for research in this area.
* Demonstrates the superior performance of DriveVLM and DriveVLM-Dual in experiments on both the nuScenes dataset and the SUP-AD dataset, and verifies their effectiveness through onboard deployment and testing.

Weaknesses:
* It would be more desirable if System 1 could acquire additional information from the VLM. For an optimization-based planner, only a reference trajectory is sufficient; however, for a neural-based planner, more information could be shared through the feature space.
* While the paper acknowledges the limitations of VLMs in spatial grounding and reasoning, and the high latency and computational intensity, a more detailed discussion on the potential limitations and challenges of the proposed system could be beneficial.
questions: questions: * How to handle the situation where one of the systems fails or makes significantly different decisions?
* In the environment description, whether the extreme weather conditions will be taken into consideration, which might be beneficial for the end-to-end driving system.
* In Appendix part, you mentioned meta actions are classified into 17 categories, but why there is only 15 categories in Figure 1 ?
limitations: limitations: 1
review_summary: review_summary: The paper is well organized with sufficient hardware demonstration. So I recommend a weak accept for this paper.