Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving
Authors: Jianbiao Mei
Yukai Ma
Xuemeng Yang
Licheng Wen
Xinyu Cai
Xin Li
Daocheng Fu
Bo Zhang
Pinlong Cai
Min Dou
Botian Shi
Liang He
Yong Liu
Yu Qiao
Venue: NeurIPS 2024 poster (2024)
Keywords: Autonomous Driving
Dual-process System
Knowledge-Driven
Vision Language Model
Primary area: robotics
TL;DR: LeapAD, a new autonomous driving paradigm inspired by human cognition, improves adaptability and interpretability in complex scenarios through dual-process decision-making and continuous learning from past experiences.

Abstract: Autonomous driving has advanced significantly due to sensors, machine learning, and artificial intelligence improvements. However, prevailing methods struggle with intricate scenarios and causal relationships, hindering adaptability and interpretability in varied environments. To address the above problems, we introduce LeapAD, a novel paradigm for autonomous driving inspired by the human cognitive process. Specifically, LeapAD emulates human attention by selecting critical objects relevant to driving decisions, simplifying environmental interpretation, and mitigating decision-making complexities. Additionally, LeapAD incorporates an innovative dual-process decision-making module, which consists of an Analytic Process (System-II) for thorough analysis and reasoning, along with a Heuristic Process (System-I) for swift and empirical processing. The Analytic Process leverages its logical reasoning to accumulate linguistic driving experience, which is then transferred to the Heuristic Process by supervised fine-tuning. Through reflection mechanisms and a growing memory bank, LeapAD continuously improves itself from past mistakes in a closed-loop environment. Closed-loop testing in CARLA shows that LeapAD outperforms all methods relying solely on camera input, requiring 1-2 orders of magnitude less labeled data. Experiments also demonstrate that as the memory bank expands, the Heuristic Process with only 1.8B parameters can inherit the knowledge from a GPT-4 powered Analytic Process and achieve continuous performance improvement. Project page: https://pjlab-adg.github.io/LeapAD

Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving
Jianbiao Mei
Yukai M2
122
Xuemeng Yang² Licheng Wen
Xinyu Cai²
Xin L
Daocheng Fu²
Bo Zhang"
Pinlong Cai Min Dou²
Botian Shi2.t
Liang He
Yong Liu Yu Qiao²
Zhejiang University Shanghai Artificial Intelligence Laboratory
East China Normal University Shanghai Jiao Tong University
Abstract
Autonomous driving has advanced significantly due to sensors, machine learning, and artificial intelligence improvements. However, prevailing methods struggle with intricate scenarios and causal relationships, hindering adaptability and inter- pretability in varied environments. To address the above problems, introduce LeapAD, a novel paradigm for autonomous driving inspired by the human cogni- tive process. Specifically, LeapAD emulates human attention by selecting critical objects relevant to driving decisions, simplifying environmental interpretation, and mitigating decision-making complexities. Additionally, LeapAD incorpo- rates an innovative dual-process decision-making module, which consists of an Analytic Process (System-Il) for thorough analysis and reasoning, along with Heuristic Process (System-I) for swift and empirical processing. The Analytic Process leverages its logical reasoning to accumulate linguistic driving experi- ence, which is then transferred to the Heuristic Process by supervised fine-tuning. Through reflection mechanisms and a growing memory bank, LeapAD continu- ously improves itself from past mistakes in a closed- loop environment. Closed-loop testing CARLA shows that LeapAD outperforms all methods relying solely on camera input, requiring 1-2 orders of magnitude less labeled data. Experiments also demonstrate that as the memory bank expands, the Heuristic Process with only 1.8B parameters can inherit the knowledge from GPT-4 powered Ana- lytic Process and achieve continuous performance improvement. Project page: attps://pjlabt adg. gi github.... io/LeapAD/
Introduction
Since the early 21st century, starting with the DARPA Grand Challenge [1], humanity has explored re- placing human drivers with computer algorithms. Over the past two decades, advancements in sensor technology, machine Icarning, and artificial intelligence have propelled the evolution of self-driving technology. Recent data-driven approaches achieved considerable success, evidenced by new vehicle models featuring intelligent driving assistance and the commercial operation of L4 robotaxis in several cities [2-4]. However, these methods depend heavily on diverse training data distributions, resulting in a superficial understanding of underlying semantics and potential misconceptions in complex situations. This is because data-driven approaches primarily perform induction observed patterns without the capability for deduction, thus constraining their performance to the coverage of the annotated data. Therefore, there urgent need for system capable of reasoning about unseen scenarios and utilizing knowledge in a human cognition manner.
equal contribution, + corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The latest advancements in Large Language Models (LLMs) and Vision Language Models (VLMs), noted for their embedded world knowledge and robust explanatory and reasoning capabilities, have captured the interest of researchers [5-8]. For example, in the autonomous driving field, some knowledge-bald methods [9-12] employ L.MSs and VI.Ms as the driving agents, However, these methods perform open-loop testing, which merely evaluates errors between model output and the ground truth from datasets, failing to reflect the dynamic interactions between ego car and the real-world environment [13]. Consequently, they are often inadequate to effectively assess the responsiveness and adaptability of driving agents.
In fact, human learning to drive involves a continuous interaction and exploration process within closed-loop environments, where drivers make decisions based on the surroundings and receive feedback accordingly. As per the levels: 1) Heuristic Process (System-)), which is automatic, quick, empirical, and domain-specifie; and 2) Analytic Process (System-If), which is rational, slow, and excels in logical reasoning and creativity across various domains. This dual-process thinking evident in the progression from novice to experienced driver. Initially, individuals rely heavily on common sense due to their lack of driving experience. Through training, they develop driving skills via a closed-loop learning process involving continuous trial and error, along with rational analysis (Analytic Process) to evaluate their behavior. These skills become internalized over time, forming muscle memory that enables quick, instinctive reactions in familiar driving scenarios (Heuristic Process). Even after obtaining driver's 's license, individuals continue to gain experience and learn from accidents to enhance driving skills.
theory [14-16], human intelligence operates on two
To this end, we develop dual-process closed-loop autonomous driving system that is continuously learning, adapting and improving, named LeapAD. Similar to the human attention mechanism, the scene understanding module in LeapAD mainly focuses on critical objects that may affect driving decisions, simplifying the environmental description and the decision-making process. Following such scene understanding, we develop a dual-process decision-making module that emulates human cognitive processes, featuring a Heuristic Process and an Analytic Process [17, 15]. Through a closed-loop setup, Analytic Process accumulates experience and builds transferable memory bank of high-quality driving decisions. The knowledge can be adapted to various scenarios and then transferred to the lightweight model in Heuristic Process through supervised fine-tuning (SFT). The Heuristic Process is employed for closed-loop decision-makee traffic accidents occur, the Analytic Process intervenes to analyze these incidents and update the memory bank, enabling the system to continuously improve through self-reflection. The main contributions of our work are summarized as follows:
using a few-shot strategy. When
We develop an innovative closed-loop autonomous driving approach that emulates the critical object attention mechanisms and the learning processes observed in human driving behavior.
We propose a dual-process prcision-making module inspired by human cognition theory. In the absence of human involvement, our approach enables the fast, empirical Heuristic Process to inherit the capabilities of the slow, rational Analytic Process a self-supervised manner.
LeapAD utilizes the Analytic Process and reflection mechanism to accumulate transferable memory bank, enabling the system to achieve continuous learning and generalization capabilities in a closed-loop driving environment.
Extensive experiments in CARLA show that LeapAD not only outperforms all other methods relying solely on camera input, but also achieves this with 1-2 orders of magnitude less annotated data.
2 Related Works
2.1 Large Vision Language Models
Inspired by the successful deployment of Large Language Models (LLMs) like LLaMAs [18, 19] and Vicuna [20], plethora of Vision Language Models (VLMs) [21-29] has emerged to broaden their applicability to multi-modal understanding. Various models, such as BLIP2 [30] which utilizes the Q-former, Flamingo [21] leveraging a perceiver resampler, and LLaVA [31] alongside MiniGPT- 4 [32] that incorporate instruction tuning, have been innovated to enhance feature alignment, seo-shot learning, and create versatile visual agents. Moreover, models like Qwen-VL [28] with its three-stage training, and Intern [29]'s image-text alignment method, assist in achieving advanced multi-lin lintillinggul and fine-graines visual comprehension. The rise of VLM and risual-language action (VLA) models has injected the vitality of autonomous driving, presenting researchers with new opportunities.

---

2.2 Empowering Autonomous Driving with Foundation Models
Recent work [33, 34, 9-11, 35] explores the use of large foundation models in autonomous driving, leveraging their embedded world knowledge and powerful interpretation and reasoning capabilities. For understanding driving scenarios, a series of datasets and benchmarks [36, 37, 6, 38] have been proposed. To improve the interpretability of autonomous driving, LMDrive [9] and DriveMLM [39] use LLMs to generate human-instructed decisions in the simulated environment, which is data- dependent and hard adapt to the real world. Agent-Driver [10] adopts LLM agent for planning, which is less efficient due to excess environmental data. EL.M [8] introduces a vision-language model tailored for embodied understanding within driving scenarios. RAG-Driver [11] improves driving interpretation and signal prediction by integrating retrieval augmentation and in-context learning. Recent DriveVLM-Dual [12] integrates VLM and data-driven planning pipelines, providing solutions for deployment. Contrary to the mentioned techniques, our LeapAD draws from attention mechanisms and observational learning and Decision-making in human driving. It utilizes memory bank for experience storage and replay in a closed-loop scenario, enabling continuous learning via memory and reflection mechanisms.
2.3 From Data-Driven to Knowledge-driven Autonomous Driving
While the prevailing data-driven approaches [2-4, 40-49] have led to success in both academia and industry in past decades, allowing autonomous driving technology to be used in people's daily lives. However, these methods are limited to the distribution of training data and frequently encounter adaptability issues and long-tail challenges when expanding across different areas [50, 51J. On the other hand, human drivers possess a deep common sense of understanding the world, which enables them to adapt to unexpected scenarios. This highlights the need for shift to the knowledge-driven approaches, which involve using empirical reasoning and induction to learn from the environment [52, 53], and updating insights develop specialized skills [54, 55]. Knowledge- driven methods acquire general knowledge rather than merely implementing predefined human rules abstracting characteristics from collected data in specific domains [7]. These approaches enhance performance, interpretability, and safety by integrating human-like logic into Al systems, particularly in managing complex traffic scenarios. In the era of foundation models, the advanced reasoning and knowledge application capabilities exhibited by LLMS and VLMs have proven highly effective for complex tasks such as understanding, reasoning, and recision-making within the domain of autonomous driving [33, 56, 57]. These foundation models have embedded world knowledge and robust explanatory and reasoning capabilities through extensive training on diverse datasets and captured the researchers' interest [5-8].
3 Methodology
3.1 Overview
In this section, we introduce how we design our anthropomorphic closed-loop autonomous driving system, LeapAD. Figure 1 illustrates that LeapAD consists of three main components: the VLM for scene understanding (Section 3.2), the dual-process decision-making module comprising the Analytic Process (Section 3.3) and the Heuristic Process (Section 3.4), along with the action executor for low-level control (Appendix A). In the CARLA simulator, LeapAD utilizes VLM to process the surrounding images and generate descriptions of critical objects. These scene descriptions are then fed into the dual-process decision-making module in order to derive scene reasoning and driving decisions. Finally, these high-level decisions are forwarded to the action executor, translated into control signals, and interact with the simulator.
In closed-loop driving environments, the fine-tuned lightweight model Heuristic Process used to perform quick, empirical decisions with the transferable experience in the memory bank. And when the Heuristic Process encounters accidents, the Analytic Process intervenes. The Analytic Process exploits L.MMs to analyze traffic accidents, leveraging its embedded world knowledge, particularly its understanding of traffic nules. It then generates corrected, high-quality driving experiences, enriching the memory bank and enabling continuous learning for the entire system.
3

---

CARLA Simulator
Prompte for Traffic
Reflection
Dual-Process Decision-making
Anolytic Process and Rational
Scene Understanding
Scene Description
Critical object Wenttiy
Heuristic Process Fost and Empiniaal
Action Executor
Mulli-nand conversution
SFT on emall dofeect
Fine-tuningi
Update
RAG
Fine-tuning
Fine-huning
HRII
Memory Accumulating
Memory Bank
Closed-loop Driving
Figure 1: The detailed architecture of our proposed LeapAD. The scene understanding module analyzes surrounding images and provides descriptions of critical objects that may influence driving decisions. These scenario descriptions making. The generated decisions are then transmitted to action executors, where they are converted into control signals for interaction with the simulator. The Analytic Process then uses an LLM to accumulate experience in driving analysis and decision-make conducting reflections accidents. The experience is stored in the memory bank and transferred lightweight language model, forming Hcuristic Process for quick responses and continuous lcarning.
then fed into the dual-process decision module, which drives reasoning and decision-
3.2 Scene Understanding with VLM
Human drivers typically focus on critical elements surrounding the vehicle to prevent information overload, enhance reaction time, and minimize cognitive load. This approach helps improve driving concentration and reduces accident probabilities. Inspired by such mechanism, the scene understand- ing module in LeapAD is designed to selectively identify critical objects, simplifying the description of the surrounding environment and reducing the load on decision-making processes.
Specifically, since off-the-shelf foundation VLMs lack domain-specifee knowledge in the driving domain, we perform SFT and prompt the VLMs to output the linguistic descriptions the objects that may influence subsequent driving decisions. The description of these critical objects includes their semantic, spatial, motion attributes, and behavioral reasoning. Integrating these aspects promotes comprehensive understanding of the environment, which can ensure safety and adaptability in complex and dynamic driving environments. For a specific driving scene, the descriptions generated by VLM can be expressed as D = {Aa,ss Ar, Am,is Cr.),= - where N denotes the number of the critical objects. For each critical object O,, the description contains: i) the semantic attribute A, describes its semantic category, usually important traffic participants (e.g.. vehicles and cyclists) and infrastructure (e.g., traffic lights and stop signs). ii) The spatial attribute A, indicates its bounding box, the lane it locates, and the distance from the ego car, which are important for safety and collision avoidance. iii) The motion attribute Am refers the motion direction of the object. iv) Behavioral reasoning C, describes why the object is critical and how it influences the driving decision of the ego car. For example, when the ego car goes straight, the stop sign the right side of high importance because it indicates the need to stop at the intersection. We provide an example to further illustrate the descriptions of critical objects in the driving scene, as shown in Figure 9 in Appendix B. Notably, the VLM not only excels in simulated environments but also demonstrates robust performance in real-world scenarios.
3.3 Analytic Process
Based on the scene descriptions provided by the VLM, we design the Analytic Process to imitate the rational thinking of human driver. The Analytic Process relies on logical reasoning, employ- ing rational thinking to analyze complex situations and make safe driving decisions. The LLMs, through their extensive pre-training on diverse datasets, have encapsulated vast amounts of world knowledge, equipping them with the ability to handle intricate problems with nuanced understanding

---

ilistory queue
Frame 0
Collision!
Frame -9
Reflection Mechaniam
somple
Frame -2
Reflection System Prompt
Reasoning Thes moment had the best charce of preventinng the collaoon IDIF with
maintairing diose lention to the slualion.. However the importont object the view only 6988 meters to the front of the ego con driving in the aome direction. This diatonce is below the 15 me-eer threshald where a STOP mandated to avoid colltion.
the
made
Avtolytcc
cornected
Memory
Process
samples
Deciaion
STOP
Detailed procedure of the reflection mechanism. When Heuristic Process encounters traffic accidents. the Analytic Process intervenes, analyzing historical frames pinpoint errors and provide corrected samples. These corrected samples are then integrated into the memory bank facilitate continuous learning.
Figure 2:
and reasoning [34]. This capability aligns with the requirements of the Analytic Process in driving scenarios, where decisions must be made based on deep analysis and contextual understanding of the environment, Our Analytic Process harnesses the power of LLMs, leveraging its world knowledge to understand the scene descriptions and perform high-quality driving analysis and decisions. We empirically found that prompting LLMS with specific traffic rules provided in Appendix C further improves safety and is more reliable for on-road scenarios.
Furthermore, we integrate the VLM and the Analytic Process to run closed-loop experiments and collect the high-quality decision-making processes and results generated by the Analytic Process as "experience" in a memory bank. The accumulated experience can be seamlessly transferred to the Heuristic Process, facilitating it to react quickly based on experience when handling similar situations, as described in Section 3.4.
We also employ the Analytic Process reflect on traffic accidents, shown in Figure 2. Specifically, when VI.M and Heuristic Process run in a closed-loop driving scenario, any accident will trigger the reflection mechanism. During this procedure, the scene description D, reasoning R, and decision S of the preceding frames before the accident are forwarded to Analytic Process. It is then required to meticulously analyze the cause of the event, locate the error, and provide corrected reasoning and decisions. The insights gained from the reflection procedure are further integrated into the memory bank, allowing the LeapAD continuously learn from failures and progressively lead to more informed and accurate docision-making in future driving scenarios. Importantly, the experience in the memory bank has good transferability and generalization. It can be directly utilized by other lightweight models and easily generalized to different scenarios, as demonstrated in section 4.4.
Reflection mechanism.
3.4 Heuristic Process
While the Analytic Process can offer more precise driving reasoning and decisions due to its detailed analysis and careful consideration, the inherent slow processing causes duplicated and redundant effort, limiting its application in practical driving scenarios. In contrast, human drivers form muscle memory through repeated practice and experience, requiring less effort over time. To reflect this quick and empirical thinking pattern and facilitate practical application, LeapAD incorporating lightweight language model. Specifically, we perform supervised fine-tuning (SFT) using the samples stored in the accumulated memory bank mentioned in Section 3.3 to distill knowledge into the lightweight language model. By this means, the Heuristic Process achieves behavior adaption to various scenarios and runs much faster than Analytic Process (about times faster in our experiments). We empirically found that the lightweight model without SFT is unable to produce appropriate driving decisions.
craft Heuristic Process in
Few-shot Prompting. Moreover, we perform few-shot prompting [34] to enhance the Heuristic Pro- cess's generalization ability for unseen scenes and mitigate hallucinations for more robust decisions.
5

---

Through such mechanism, the Heuristic Pro- cess can effectively leverage the experience and deep insights from the existing memory bank, improving the accuracy future driving decisions. To facilitate the retrieval of similar driving scenes from the memory bank for few- shot prompting, primarily rely on the em- bedding similarity between the current scene's s descriptions and those stored in the memory bank. However, if directly calculating text sim- ilarity based on the original descriptions, the presence of redundant linguistic information in descriptions complicate the differen- tiation between scenes. Thus, we propose a novel scene encoding method to extract and encode compressed captions that comprise key elements such the critical object's ' category, lane, and distance from the ego car. This ap- proach streamlines the procedure of querying similar scenarios, enhancing retrieval efficiency and accuracy by prioritizing the most influential aspects for driving decisions of each scenario. Afterward, the compressed captions are sent a text encoder to encode the embedding vectors, which are expressed as follows:
Scene Descriptions
Recsoning & Decision
@WenLM
Learnabie Query Embe
Qwen-1.5 ,
A
CrossAttm
System
Prompt
Scene Descriptions
DRIVE
The illustration of the fine- tneettning process. We fine-tune the VLM (Qwen-VL-7B) using 11K instruction- following data for scene understanding (left). Also, utilize the collected samples in the memory bank fine- tune Qwen- used in Hcuristic Process, as illustrated in the right part.
Figure 3:
e e (F.
where D is the scene description, F, denotes the compressing process. and T. indicates the text encoder. Subsequently, the cosine similarity between the query embedding eg for the current scene and the embedding {er} M-1 for the memory bank with the size of M is computed by:
C. C;
s(e,, e,)
| ||:
We select top-k samples with the highest similarity scores queried scenes. The scene descriptions (D.) reasoning {R, )==)))) and decisions (S [5= of k samples and the scene descriptions D. of the current scene are both fed into the Heuristic Process for the final reasoning R. and decision S..
Experiments
4.1 Data preparation
Data for VLM. We construct the instruct-following datasets for supervised fine-tuning of our VLM by integrating Rank2Tell [58], DriveLM [36], and data collected within CARLA [59]. To maintain consistency across all the datasets, we adopt a uniform standard reference format for critical objects <ref>In In {camera view), (properties) </refiessrrrrdiiiiee dataset, specific Q& pairs are created to suit their unique structures and contents. The conversations are structured in summary-elaboration manner, with the first question for the Rank2Tell and DriveLM datasets focusing on determining the number, semantic, and spatial attributes, such as the bounding box coordinates of key objects. For the Rank2Tell dataset, follow up by inquiring about the moving state, importance, and corresponding reasoning for each critical object. For the DriveLM dataset, we retain most of the original questions but eliminate redundant ones. We extract only 6K frames of data from Rank2tell and DriveLM and organize them in the standardized format. The data gathered from the CARLA simulator is exclusively dedicated to the closed-loop experiments detailed in Section 4.3. A comprehensive training dataset of 5K frames is collected from Town 01-04, 06, 07, and 10. To identify key objects within the scene, we design several automatic annotation rules, as detailed in Appendix B. For clarity, we provide the data illustration in Figure 9 in Appendix B.
(coordinates)< </bo>>, For each
leverage the integration of Analytic Process and VLM to accu- mulate experience within the closed-loop setup and save it in the memory bank for subsequent SFT and few-shot prompting of Heuristic Process. Moreover, our approach incorporates dynamic updates to address errors encountered by the Heuristic Process, mentioned in the reflection mechanism
Data for Heuristic Process.

---

Table 1: Comparison of our LeapAD with competitive methods on Town05 Short benchmark. Notably, LeapAD demonstrated superior performance with smaller data footprint, outperforming other approaches. "DD" "KD" denote data-driven and knowledge-drivee driven, respectively. "L" "C" indicate LiDAR and camera modalities.
Method
Modality
Type
Annotations
DS
RC f
InterFuser [41]
L+C
DD
3M
94.95+1.91
95.19+12.57
TransFuser [42]
L+C
DD
228K
54.52:14.29
78.41:13.75
VAD [49]
DD
228K
64.30
87.30
NEAT [45]
DD
130K
58.70::4.11
77.32:t4.91
Roach [46]
DD
65.26413.63
88.2445.16
WOR [62]
C
DD
1M
64.79+5..3
87.47+44.68
LBC [48]
DD
157K
30.97.1..17
55.0111..14
CILRS [47]
DD
720K
7.471:2.51
13.40.11.09
VIM + GPT-4
KD
11K
81.31+2.37
94.22++3.18
LeapAD (w/o Town05)
KD 11K
75.73+1.36
12.10+1.44
LeapAD
KD
11K
83.11
94.98:10.54
outlined in Section 3.3. The memory bank in our approach is configured to default size of 9.0K, including samples collected from various towns (01-04, 06, 07, and 10). It is worth noting that samples are obtained in closed-loop environment without human involvement. Each sample consists of the scene descriptions D depicted in Section 3.2, reasoning R, and decisions S. The reasoning R explains the process of decision- maasing based on the scene descriptions D and chain of thought.
4.2 Implementation Details
We employ Qwen-VL-Th VL-7B [28] as the VLM for scene understanding. GPT-4 the Analytic Process for rational and logic thinking, and 1.-en1.5-1.88 [60] as Heuristic Process for automatic and quick thinking LeapAD. We use the OpenAl embedding model as the text encoder T. extract text embedding. To fully excite our VLM's capabilities in autonomous driving, we perform SFT with the instructtionffolloinng data discussed in Section 4.1. We utilize the AdamW optimizer [61] with 31 = 0.9 and 82 = 0.95, coupled with cosine decay of the learning rate, initially set to le-5, The batch size set to 16, and the model is trained for 5 epochs on 8 A100 GPUs, requiring about 26 hours. The input image resolution is set at 448 x 448 pixels. For Heuristic Process, we conduct SFT on Qwen1.5-1...
for 5 epochs using samples stored in the memory bank, taking about hours. The training parameraeeserss are consistent with the training procedure of VLM. The detailed fine- tuning process shown in Figure The dual-pron proccss decision module outputs meta-actions (e.g., "AC", "DC", "IDLE", "STOP") at frequency of 2 H7, which are further refined to control signals, as detailed in the Appendix A. Please refer to Appendix D for more details about the reflection mechanism and Appendix B for the performance of VLM on both simulated and real datascts.
4.3 Evaluation in Closed-Loop Driving
We conduct closed-loop experiments in CARLA, a popular and realistic open-source simulator, to evaluate the performance of our LeapAD. To validate the effectiveness, we conduct a comprehensive assessment in closed-loop driving scenario on the Town05 benchmark. Our evaluation metrics include Driving Score (DS), Route Completion (RC), and Infraction Score (IS). RC signifies the proportion of the route successfully navigated by the agent, while IS indicates penalties incurred from accidents. By multiplying RC by IS, we obtain the final metric DS for evaluating our method's driving performance on a route. Table compares our method with competitive methods on the Town05 Short benchmark. Specifically, we provide three different configurations to evaluate methods comprehensively: i) VLM + GPT-4 represents directly using non-fine cnnfine-tuned GPT-4 as the decision module along with the VLM (Qwen-Vi VL [28]); ii) LeapAD (w/o Town05) represents the dual-process system with the memory bank of 9K samples accumulated from various towns (01-04, 06, 07, and 10) except Town05; dii) LeapAD denotes the dual- qual-process system with the memory bank of 18K samples accumulated from various towns (01-07, and 10) and 0.1K reflection data in Town05.
As shown in Table our LeapAD outperforms all other methods that rely solely camera sensor input. Besides, our method surpasses TransFuser [42], which additionally utilizes LiDAR sensor inputs. It is worth noting that in all experiments, only used total of 11K data to fine-tune the
7

---

Table 2: Generalization of accumulated knowledge in the memory bank. We evaluate the performance of our LeapAD various (Town05, Town01, and Town04) by leveraging memory banks accumulated from diverse sources. Lavg denotes the average lengths routes within each town.
Memory (Town01-O4 06)
Memory (Town05)
Test town
(m)
DS f
RC f
IS
Few-shot
SFT
Few-shot
SFT
66.40
90,40
73,81
75.73
92.10
82.66
Town05
70.1
69.90
91.79
76.64
78.07
91.69
85.89
v
83.11
94.98
87.78
Town01
129.1
V
68.68
100.0
68.68
Town04
119.3
V 95.08
97.96
96.56
VLM, while all other methods employ tens to hundreds of times more data. Moreover, our dual- process decision module does not involve human annotations, demonstrating the labeling efficiency of LeapAD. For instance, although InterFuser [41] achieves higher performance than our LeapAD, relies on 3 million camera and LiDAR annotations, approximately 272 times more than As shown in the results, we observe that even without prior driving experience in Town05, LeapAD can surpass other camera-input methods. However, there remains gap compared to VLM + GPT- 4, which integrates the scene understanding module and the Analytic Process, achieving a DS of 81.31. This demonstrates that GPT-4's understanding of world knowledge and common sense aids in performing driving-spen -ppeciicp tasks. But using GPT-4 directly as the decision -aaiing module is both time-consuming and expensive, making it impractical for deployment in vehicles. On the other hand, with the continuous accumulation and adaptation of experience in the test town, our LeapAD) surpasses VI.M + GPT-4, while only using 1.8B model in Heuristic Process. By leveraging enriched memory bank, it achieves a final DS of 1. This result fully validates the effectiveness of our dual-process decision-making module. Moreover, we provide the evaluation results on the Town05 Long benchmark and visualizations in Appendix E and F.
method.
4.4 Ablation Study
We conduct extensive ablation studies about the number of few shots, size of the memory bank, reflection mechanism, and accumulated experience in closed-loop driving setup to demonstrate the generalization and continuous learning capabilities of our LeapAD.
Ablation on the number of few-shot. We highlight the importance of few-shot prompting in using accumulated experience to guide current decision-makiin iments are tested on the Town05 Short benchmark with
* With Different Shots
With Different Memory Sizes
The exper-
80
72:35
Driving Score (DS)
the memory bank consisting of 9K samples automatically collected by Analytic Process in Town05. The results are presented in Figure 4, showing that our approach surpasses several methods (e.g., CLIRS [63], LBC [48] in Table 1) even with a zero-shot setting. Furthermore, there is a no- table performance improvement when moving from zero- shot to one-shot scenarios. Moreover, there is consistent increase in closed-loop experiment results as the number of shots is increased to three, which experimentally demon- strates the value of the experience the memory bank and the effectiveness of the few-shot strategy.
64.95
72:33
80
50
51,77
0/0
1/
2/900 3/9000
Shots Memory Sizes
Figure 4: The illustration for ablation stud- ies of few-shot and memory size. See Ap- pendix E for detailed data.
The impact of memory sizes. The memory bank contains accumulated experiences crucial for improving the performance of our approach. Therefore, we conduct additional ablation studies to explore the impact of the size of the memory bank, which is equipped with the few-shot strategy (defaulting to 3-shot). We conduct closed-loop evaluations with memory banks of different sizes: i) base memory bank with 9K samples. ii) compressed memory banks with 900 and 90 samples evenly sampled from the base memory bank. îūj no memory bank in system. The quantitative results are presented in Figure and illustrate gradual performance increase as the memory size grows. This
8

---

further demonstrates the continuous learning capability of our proposed LeapAD, indicating that our model's performance can improve with accumulated experience.
Effectiveness of the reflection mechanism. Reflection plays a crucial role in the continuous learn- ing capability of proposed LeapAD. It includes reflecting mistakes and incorporating correct experiences into the memory bank, fostering self-improvement through proactive summarization and accumulation of experiences in unfamiliar scenes, notably corner cases. To enhance experimental efficiency, we employ configuration of three shots and memory bank with 900 stored memories in the Town05 environment. Reflective experiments are conducted by selecting routes with scores below 50. incorporated, and the post-reflection experiences were added to the memory bank. Figure indicates that reflection significantly enhances out method's performance, alheit with some instances where individual sequence scores temporarily decrease, potentially due to the inherent randomness of the simulation environment and the limitations of the VLM model. Addressing these limitations will be the focus of our future work.
Generalization of accumulated knowledge. Finally, we conduct a series of experiments to demonstrate the generalization and transferabil- ity of the experience in the memory bank. The results are illustrated in Table 2, which shows the robustness of our proposed LeapAD on dif- ferent towns. As shown in Row 1 and Row 3, solely employing experience adaption (SFT) from different towns with zero-shot, our Lea- pAD has achieved commendable performance on the Town05 Short benchmark, surpassing many methods in Table 1. Implementing the few-shot strategy yields substantial enhance- ments (Row 2 & 4), further highlighting the effectiveness of the experience in the memory perform cross-vall voss-validation
200
88.89
81.35
20
72.23
Driving Soore (DS)
47/91
29
23
20
route 07
route 26
mude 27
30
route 19
avergge score
bank. Furthermore, to demonstrate the generalization of the accu- mulated experience, focusing on two setups: i) testing on Town05 with the memory bank col- lected from other towns (Row 1 & 2); testing on the other towns (Town01, 04) with the mem- ory bank accumulated only from Town05 (Row 6 & 7). Comparing Row 2 with Row 4, our LeapAD utilizing different memory sources achieves comparable performance in the same town (Town05). Additionally, our proposed LeapAD demonstrates good performance across different towns when using the same memory bank (Row & 7). In this section, several low-quality samples in the memory bank on routes with low driving scores are withdrawn when accumulating experience, and no special memory processing is performed on those experiments in Figure 4.
Reflection rounds
Figure Effectiveness the reflection mechanism. The z-axis represents the rounds of reflection, while the y- axis denotes the resulting driving score. The dashed line illustrates performances different routes after multi- round reflection, and the red "average score" denotes the mean performance across all routes.
Conclusion
In this paper, introduce LcapAD, dual-process closed-loop autonomous driving system with continuous learning, adapting, and improving capabilities. Similar to human attention, our approach selectively prioritizes critical objects that can influence driving decisions, simplifying the scene description and reducing decision- decision-making complexity. Furthermore, the dual-process decision- making module mimics human cognitive processes through fast, empirical Heuristie Process and slow, rational Analytic Process. Through reflection mechanisms and a transferable memory bank, LeapAD continuously improves from past experiences in closed-loop environment, demonstrating continuous learning capabilities and strong adaptability to various driving scenarios. Moreover, LeapAD can be seamlessly integrated with the mainstream cloud-edge architectures employed in intelligent vehicles. The Heuristic Process operates at the edge, enabling instant decision-making within the vehicle, while the Analytic Process handles more complex scenarios in the cloud.

---

Limitations and Broader Impacts
Currently, LeapAD relies solely on single-frame camera inputs, without any temporal input. Another bottleneck of our approach is the VLM's inability to participate in the reflection mechanism, which hinders further system improvements. Additionally, there is notable gap between predefined agent behaviors in the CARLA benchmark and those in real-world scenarios, underscoring the need for a high-fidelity world simulator. For impacts, autonomous driving systems gather extensive data on driving behaviors, routes, and passenger movements, raising concerns about data privacy and legal implications. Nonetheless, advancements in technology and regulatory frameworks can help address these issues, paving the way for safer, more efficient, and accessible driving systems.
7 Acknowledgments
This research was supported by the NSFC 6208810 101 Autonomous Intelligent Unmanned Systems. Ad- ditionally, it received support from the Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China (Grant No. 20222D0160104 601004) and the Science and Technology Commission of Shanghai Municipality (Grant No. 22DZ1
References
[1] S. Thrun, M. Montemerlo, H. Dahlkamp. D. Stavens, A. Aron, J. Diebel, P. Fong. J. Gale, M. Halpeany, G. Hoffmmann, et al. "Stanley: The robot that won the darpa grand challenge." Journal of field Robotics. vol. 23, no. pp. 661-692, 2006.
[2] T. Yin, X. Zhou, and P. Krahenbuhl, "Center- based 3d object detection and tracking." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11784 11793, 2021.
Z. Li, W. Wang, Li, E. Xic, C. Sima, T. Lu, Y. Qiao, and Dai, "Bevformer: Learning bird's-eyc-vier representation from multi-camera images via spatiotemporal transformers," in European conference on computer vision, pp. 1-18, Springer, 2022.
[4] Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. L. Rus, and S. Han, "Bevfusion: Multi-task multi-sensor fusion with unified bird' s-eye view representation," in 2023 IEEE international conference on robotics and automation (ICRA). pp. 2774-2781, IEEE, 2023.
[5] Wayve, "Lingo-1: 1: Exploring natural language for autonomous driving." " 2023.
[6] Y. Ma. Y. Cao, J. Sun, M. Pavone, and C. Xiao, "Dolphins: Multimodal language model for driving." arXiv preprint arXiv:2312.00443: 2023.
X. Li, Y. Bai, P. Cai, Wen, Fu, Zhang, X. Yang, X. Cai, Ma. Guo, er al., "Towards knowledge-drive autonomous driving," arXiv preprint wwXiv:2312.043116 2023.
[8] Zhou, Huang, Q. Bu, Zeng, Li, H. Qiu, Zhu, M. Guo, Y. Qiao, and H. Li, "Embodied understanding of driving scenarios," arXiv preprint arXiv:2403.04500 2024.
[9] H. Shao, Y. Hu, L. Wang. S. L. Waslander, Y. Liu, and H. Li, "Lmdrive: Closed-loop end-to-end driving with large language models," arXiv preprint arXiv:2312.074588 2023.
[10] J. Mao, J. Ye, Y. Qian, M. Pavone, and Y. Wang, "A language agent for autonomous driving." arXiv preprint arXiv:2311.10811 2023.
[11] J. Yuan, S. Sun, D. Omeiza, B. Zhao, P. Newman, L. Kunze, and M. Gadd, "Rag-driver, Generalisable driving explanations with sarrival--uggeette in-context learning in multi-modal large language model," arXiv preprint arXiv:2402.10822, 2024.
[12] X. Tian, Gu, B. Li, Liu, Hu, Y. Wang. K. Zhan, P. Jia, X. Lang, and Zhao, "Drivevlm: The convergence of autonomous driving and large vasion-language models," arXiv preprint arXiv:2402.12288 2024.
[13] Z. Li, Z. Yu, S. Lan, J. Li, Kautz, T. Lu, and J. M. Alvarez, "Is ego status all you need for open-loop end-to-end autonomous driving?." arXiv preprint arXiv:2312.0303 2023.
[14] D. Kahneman, "Fast and slow thinking." Allen Lane and Penguin Books, New York, 2011.
10

---

[15] J. S. B. Evans and K. E. Stanovich, "Dual-process theories of higher cognition: Advancing the debate." Perspectives on psychological science, vol. 8, no. pp. 223-241, 2013.
[16] P. C. Wason and J. 5, B. Evans, "Dual processes in reasoning?," Cognition, vol. 3. no. 2. pp. 144, 1974.
[17] S. Epstein. "Cognitive-experietti self-theory of personality," Comprehensive handbook of psychology, vol. 5, pp. 159 184, 2003.
[18] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B, Rozière, N. Goyal, E. Hambro, F. Azhar, et al., "Llama: Open and efficient foundation language models," arXiv preprint 2rXiv:2302.13971, 2023.
[19] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhar- gava, Bhosale, et al.. "Llama Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09226 2023.
[20] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, Stoica, and E, P. Xing, "Vicuna: An open-source chatbot impressing gpt-4 with 90%* chhttgtt quality," March 2023.
[21] J.-B. Alayrac, J. Donahue, P. Luc, A. Micch, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, R. Ring. E. Rutherford, S. Cabi, T. Han, Z. Gong. S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisser- man, and K. Simonyan, "Flamingo: visual language model for few-shot learning," Advances in neural information processing systems, vol. 35, pp. 23716 23736, 2022.
[22] X. Chen, X. Wang. S. Changpinyo, A. Piergiovanni. P. Padlewski, D. Salz, S. Goodman, A. Gryener, B. Mustafa, Beyer, el al., "Pali: A jointy-scaled multilingual language-image model," arXiv preprint arXiv:2209.067 067944 2022.
[23] B. Li, Zhang, L. Chen, Wang, Pu, J. Yang. C. Li, and Liu, "Mimic-it: instruction tuning." arXiv preprint 2rXiv:2306.05425 2023.
in-context
[24] K. Li, Y. Hc, Y. Wang, Y. Li, Wang, P. Luo, Y. Wang. L. Wang, and Qiao, "Videochat: Chat-centric video understanding," arXiv preprint ar 2i3: 06355, 2023.
[25] 7. Li, B. Yang, Q. Liu, 7. Ma, S. Zhang, J. Yang, Y. Sun, Y. Liu, and X. Bai, "Monkey: Image resolution and text label are important things for large multi-modal models," arXiv preprint arXiv:2311.0666 2023.
[26] H. Zhang, Li, and L. Bing, "Video-Ilama: An instruction tuned audio-visua language model for video understanding." arXiv preprint arXiv:2306.0285. 2023.
[27] W. Wang, Q Lv, W. Yu, W. Hong, Qi, Y. Wang, Ji, Yang. L. Zhao, Song, et al., "Cogvlm: Visual expert for pretrained language models," arXiv preprint arXiv:231 1..00231 2023.
[28] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. 7hou, and Zhou, "Qwen-vl: A frontier large vision-language model with versatile abilities," arXiv preprint arXiv:2308. 1...0088299666 12966, 2023.
[29] Z. Chen, Wu, W. Wang, W. Su, Chen, Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, al., "Intervl:: Scaling up vision foundation models and aligning for generic visual-linguistic tasks," arXiv preprint arXiv:2312. 14238, 2023.
[30] Li, D. Li. S. Savarese, and S. Hoi, "BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," " arXiv preprint urXiv:2301.12599 2023.
[31] H. Liu, C. Li, Q. Wu, and Y. Lee, "Visual instruction tuning," Advances in neural information processing systems, vol. 36, 2024.
[32] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing laioun-language understanding with advanced large language models." arXiv preprint arXiv:2304.1083 2023.
[33] D. Fu. X. Li, L. Wen, M. Dou. P. Cai, Shi. and Y. Qiao, "Drive like human: Rethinking autonomous driving with large language models," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 919- 2024.
[34] L. Wen, D. Fu, X. X. approach to autonomous driving with large language models," in The Twelfth International Conference on Learning Representations, 2024.
X. Cai, T. MA, P. Cai, M. Dou, B. Shi, L. He, and Y. Qiao, "Dilu: A knowledge-driven

---

[35] L. Wen, X. Yang. D. Fu, X. Wang, P. Cai, X. I.. MA, Y. L. XU. Shang. Z. Zhu, S. Sun, Y. BAI. X. Cai, M. Dou, S. Hu, B. Shi, and Y. Qiao, "On the road with GPT-4v(ision): Explorations of utilizing visual-language model as autonomous driving agent," ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
[36] C. Sima, K. Renz, K. Chitta, I.. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li, "Drivelm: Driving with graph visual question answering," arXiv preprint arXiv: 14150, 2023.
[37] A.-.. Marcu, L. Chen, Hünermann, A. Karnsund, B. Hamotte, P. Chidananda, S. Nair, V. Badrinarayanan,
A. Kendall, J. Shotton, E. Arani, and O. Sinavski, "Lingoqa: Video question answering for autonomous driving," arXiv preprint ar3iv:2312. 15, 2023.
[38] Nie, R. Peng. C. Wang, X. Cai, J. Han, H. Xu, and Zhang, "Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving," " arXiv preprini arXiv:2312.0366. 2023.
[39] W. Wang, Xie. Hu, Zou, J. Fan, W. Tong, Y. Wen, Wu, H. Deng. Z. Li, et al., "Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving." arXiv preprint 23Xi::3312.0924: 2023.
[40] X. Jia, P. Wu, L. Chen, J. Xie, C. He, J. Yan, and H. Li, "Think twice before driving: Towards scalable decoders for end-to-end autonomous driving." in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21983-21994, 2023.
[41] H. Shao, L. Wang, R. Chen, H. Li, and Y. Liu, "Safety-enhanced autonomous driving using interpretable sensor fusion transformer," in Conference on Robot Learning, pp. 726-737, PMLR, 2023.
[42] K. Chitta, A. Prakash, B. Jaeger. Z. Yu, K. Renz, and A. Geiger, "Transfuser: Imitation with transformer- based sensor fusion for autonomous driving." IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[43] P. Wu, X. Jia, L. Chen, J. Yan. H. Li, and Y. Qiao, "Trajectory- control prediction for end-to-end autonomous driving: A simple yet strong baseline," Advances in Neural Information Processing Systems, vol. 35, pp. 6119-6132, 2022.
[44] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al., "Pllaning-orieented autonomous driving," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 17853-17862, 2023.
[45] K. Chitta, A. Prakash, and A. Geiger, "Neat: Neural attention fields for end-to-end autonomous driving," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15793-15803, 2021.
[46] Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. Van Gool, "End-to-end urban driving by imitating rein- forcement learning coach," in Proceedings of the IEEE/CVF international conference on computer vision. pp. 15222-15232, 2021.
[47] F. Codevilla, E. Santana, A. López, and A. Gaidon, "Exploring the limitations behavior cloning for autonomous driving," in Proceedings of the IEEE/CVF international conference on computer vision, pp. 9329-9338, 2019.
[48] D. Chen, B. Zhou, Koltun, and P. Kräähenbühl, "Learning by cheating," " in Conference on Robot Learning. pp. 66-75, PMLR, 2020.
[49] B. Jiang, S. Chen, Q. Xu. B. Liao, J. Chen, H. Zhou, Q. Zhang, Liu, C. Huang, and X. Wang. "Vad: Vectorized scene representation for efficient autonomous driving," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8350, 2023.
[50] Y. Peng. J. Han, Z. Zhang, L. Fan, T. Liu, S. Qi. X. Feng. Y. Ma, Y. Wang. and S.-C. Zhu, "The tong test: Evaluating artificial general intelligence through dynamic embodied physical and social interactions," " Engineeringg 2023.
[51] S. Gildert and G. Rose, "Building and testing a general intelligence embodied in a humanoid robot," arXiv preprint 2307 16770, 2023.
[52] Z. Liu, H. Jiang. H. Tan, and F. Zhao, "An overview of the latest progress and core challenge of autonomous vehicle technologies." in MATEC Web of Conferences, vol. 308, 06002, EDP Sciences, 2020.
[53] F. Dou, J. Ye, G. Yuan, Q. Lu, W. Niu, H. Sun, L. Guan, G. Lu, G. Mai, N. Liu, et al., "Towards artificial general intelligence (agi) in the internet of things (iot): Opportunities and challenges." arXiv preprint arXiv:2309. 07438, 2023,
12

---

[54] B. Zhang, J. Zhu, and H. Su, "Toward the third generation artificial intelligence," Science China Information Sciences, vol. 66, 2. p. 121101, 2023.
[55] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang. S. Jin, E. Zhou, et al. "The rise and potential of large language model based agents: A survey," arXiv preprint arXiv:2309.07866 2023.
[56] Y. Liu, F. Wu, Z. Liu, K. Wang. F. Wang, and X. Qu "Can language models be used for real-world urban-delivery route optimization?, " The Innovation, vol. 4, 6, 2023.
[57] C. Cui, Y. Ma, X. Cao, W. Ye, Y. Zhou, K. Liang, J. Chen, J. Lu, 7. Yang, K.-D. Liao, al., "A survey multimodal large language models for autonomous driving." in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 258-979, 2024.
[58] E. Sachdeva, N. Agarwal, S. Chundi, S. Roelofs, J. Li, M. Kochenderfer, C. Choi, and B. Dariush, "Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7513-7522, 2024.
[59] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, "Carla: An open urban driving simulator," in Conference on robot learning, pp. 1-16, PMLR, 2017.
[60] Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng. Fan, W. Ge, Han, F. Huang, Hui, L. Ji, M. Li. Lin, R. Lin. D. Liu, G. Liu. C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang. S. Wang, W. Wang. S. Wu, B. Xu, J. Xu, A. Yang. H. Yang. J. Yang, S. Yang. Y. Yao. B. Yu, H. Yuan.
Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu, "Qwen technical report," arXiv preprint arXiv:2309. 16609, 2023.
[61] Loshehilov and F. Hutter, "Decoupled weight decay regularization." arXiv preprint arXiv:1111.05101. 2017.
[62] D. Chen, V. Koltun, and P. Krähenbühl, "Learnin to drive from world rails," in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 155999 2021.
[63] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with unified text-to-text transformer," The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485-5551, 2020.
[64] R. C. Coulter et al. Implementation of pure pursuit path tracking algorithm. Camegie Mellon University, The Robotics Institute, 1992.
[65] T. Wang, E. Xie, R. Chu, Z. Li, and Luo, "Drivecot: Integrating chain-of-thought reasoning with end-to-end driving," arXiv preprint arXiv:2403.1 16996, 2024.
13

---

Appendix
A Low-level Control
As mentioned, our dual-process makiing module outputs meta-actions (e.g., "AC", "DC", "IDLE", "STOP"), which are further refined to control signals such as steering, acceleration and brake. For the sake simplicity, we define "AC" mean acceleration Im/s, and "DC" to mean deceleration of -mm/s
A.1 Planned Waypoints
The default route waypoints provided by CARLA are sparse, with distances between consecutive waypoints reaching up several dozen meters. This makes it difficult for our lower-level controller to decompose meta-actions into control signals. To address this issue, we leverage the high-definition map to densify these sparse waypoints into 1-meter interval path-points, which form the reference path for the ego vehicle. Subsequently, our controller employs the Pure Pursuit algorithm [64] to track the reference path, ensuring that the ego vehicle remains on the correct road. The selection of the target path-point for tracking is adaptive, depending on the vehicle's s speed, with the controller choosing one of the third seventh path points ahead.
is worth noting that high-definition maps are not a necessary requirement for our approach. Alternative methods, such those proposed in DriveCot [65] TransFuser [42], which utilize separate neural networks to predict the future reference path based on camera images with sparse navigation information, also compatible with controller design, without affecting our methodology.
A.2 PID Controller
Given the target path-point and the target vehicle speed, ego vehicle's control method employs two independent PID controllers, building upon previous works [45, 42, 43, 40, 65]. Specifically, the control system contains two separate PID controllers: one longitudinal for throttle and brake and one lateral for steering. The longitudinal PID controller, tuned with gains Kp 5.0, K2 0.5, and K 1.0, takes the current speed and desired target speed as inputs and utilizes a 40-frame buffer compute the throttle and brake values. Meanwhile, the lateral PID controller, tuned with gains Kp = 1.0, K2 0.5, and KD = 0.2, receives the angle difference between the ego vehicle's 's heading and the vector pointing to the chosen future waypoint as input, and employs 20-frame buffer to calculate the steering value.
B Dataset Annotations
In this section, present examples of datasets utilized to train the VLM, including the self-collected CARLA dataset (Figure 6), the DriveLM dataset (Figure 7), and the Rank2Tell dataset (Figure 8). The data annotations principally categorize the critical objects based four attributes (semantic, spatial, motion, and risk ranking), as outlined in Figure and Section 3.2. is worth noting that not every dataset comprehensively covers questions and answers pertaining these attributes. For instance, the DriveLM dataset lacks the selection of important objects; the Rank2Tell dataset merges semantic and spatial information into single overview.
To conduct closed-loop experiments, we specifically define questions about the semantic and spatial details of key objects in the CARLA dataset and design several automatic annotation rules. Specifi- cally, the critical objects are defined as those situated in the front, left, and right fields of view: i) Vehicles and cyclists positioned within 20 meters of the ego or less than 60 meters in the ego lane. ii) Pedestrians within 40 meters of the ego car. iii) Traffic lights that control the vehicle's travel direction and stop signs.
In addition to presenting the datasets, we further elaborate on description generated by the VLM. The examples described in Figure 6, 7, demonstrate VLM's proficiency in accurately identifying key objects that influence driving decisions and effectively describing the related attributes. We observe that VLM not only excels in simulated environments but also demonstrates robust performance
14

---

Images
Critical Objects
Quasiiin:
Gurnenl soeme CAM FRON' socjes GAM WWW.RRONTLLE LEFT: socjjr a city street and this image Pleoze deseribe detai the otjects
CAML PRONT, RIGHT RIGHTT moejjg, you driing
shows
aftect sethrriing vehicle agest Intormation thouls trottle lights that emeggey. the otyert's postton relette the oor sell lone, objerts store motion, and the objects bourding box, os well az the opanosmaat diftnnce from the self cor
your drivng particuar trotlic ligta
ogjects
Arewor Thire are 5 important osjects ir the soars. The information these key objects ore cref> In "CAALFRONT; green troffic light is locoted asgrodmatey 42.050 s/rds cooo (652. 4161 (961, 449) </buoo and In </hise
CAML RON
fromt right the coproumately 1.109
"CAM_FRONT vhhicle locaned
c/refs choses (612,
ago car in drive in the come direction cirefs <boo (0, 01 (884, 1000) <thowo crefs In FRDNT, whiicle located coprocmately 9.705 the froet of the ago oon is orhic in the some direction cires dbso (404, 424). (592. c80) c/bowo crefy In "CAALLEFT in kor is localed </ret> cosoo (364, 204) (1P9, 4E3) c/boo.
"CM, RIGHT, veticle located approdmenely 474
foot right
17.099
back left
CANUFRONT_
GT: Them ore importont objects the acoe The informen on of trese key chjects creh 'CMM, FROST, grean troffic light located app appormmttely 42,006 (ee),
crefs the right the ego car is drive the come direction ciress choso (612. 427) (885, eo6) <iboo crefs In "CAMLRG -T sehick is locatedd opprocmintly 4745 in 1o the front right the age car is drive in the come direction <irets <bow (o, 01 (504, 1000) </bowo crets In "CAMFRRN"" whick lacated ooproemetely 9.20s
night the ego ocr
cbolo (eog
'CAM, FRONT) whick eosted appradinctly 11.050
the froet the egn is CCALLEEFT direction c/nets coors (411, (592, c/ooio craty of agu
oree
approsenulidy 17
CAM FRONT RIGHT
</reh> <boo (364, 294), (379, 4933 </bowo.
Figure 6: Annotation format and example of scene description generated by VLM on a self-collected CARLA simulation dataset
Row Images
Motion
Guaoier Whht the slolca of the truck that lo the back righ of the ego cor?
Anover: One truck is corked
GT One
FRONTLIEFT
DAMU RONT
CAN WWW.FRAT..OOO ROSHHT
Pelattiin while
CANLFRONT citets (877, sN) (59), 820 c/boo??
Arover: Moving
CAM_SACTURIGIT
GT. Moving
Critical Objacts
Cursher: "Whot moving iimus stject enrty Mowin, ihakattiin while and afhrris CANLFRONT srrefs cbows (er7, an4), (59) E24 x/bew??
Currert
FRONT,
GML FRONT RIGHT
CARLIAAE
CAM EACUIITTT
CAM JAAK
urinng urtan afnec and tha image indiaale stame thai Puzze descrios
in detill the key objects ir the ocone that may offect your driving The cbject Information should
Arawa: Ertsiin, the crssomrkk
condain the otgesta conegory the otgecia bounding sox, and the okjas's boutin reluive Iee ego cor"
GT: Canny shoaa.
Anower There are importore ageets in the acene. The informatio or of these key objects ore arefs
Somantic
'CAM FRONT wrrts copo. (000, 402). f616, 631) ciboo, end' Stutonary seden CCM BACC cinels cieo cooo 4903. 1720, 1000) <boo.
croctin
Oh2 092) c/horo
Guestion: the visual cesoriatir Moving
cref Stahoonrry Btock secon CAM
Pedertrinn
"DAIL
CAM FRONN cind's chooo (677, 404) (596 524) c/tos?
Blad
chriss (625. 323) (546, 400) </bov.
CAML FROMT
"he descrictio of cnd' and shome IAAM J FRONT cirds <boo 1477 444) (895, 804) cyboro pesiattion weating o whih top and onay shorts.
Resittion
There ore importont odjects The scone The informction of these key edts are arc
Kloiing Redernion weating o withe top and gray shom "AAL crrb <bno (477, knnd ports 7271 c/boo, Moving sedon Surtowrr </boso neatoonrry Red TAMLRRONT
Dioa, 100) cisso Moving iestemmon weaning white
CANUFRONT_.OO knehs <boos
GT: "The stoad descrption of ents Moding Redeetton ahorr CaM FRONT
c/nets (60s 5030 Casl Bace
wearing whll and cireb <bao (477 4ne), 1896, 5241 o/boo pedeertion wea while up and pray ahorts"
tAN JANOT cieet (a) 437), (722. (ooo)
ein''s chaos (300, 301) (500, 404) citoooo
Positiin
Question Which lones are erch imporont object on in the scone?
Aroven end's Moving Palciiiinaa crosiin the reed YAA, FRoN' c/nds daso (660, 462). (ets 83t) c/toso in the ego lone, aref Slalooury Buok aoton
BACC cinet <boo (392. 490) (112, 922) <itoo is in the ago lone, cott Shotonory ooc redion CAM JURONLLLTT LLEFT c/ret doso 478), (220, 2000) <lose the lone eabb aaroryy White sedan in CAN FaoNT <inet dooeo (454, 1755, 650) cisoe is the ego lane cress toving cooo 3201, (545, 400) c/ooo
secon 'CMLFROOT c/rety (0. 5E4), (250, </boo
left lone. crets light "CANLFRONT
ego lors Stmtonory
CAMLFRONT, LSFFT cirds
Figure 7: Annotation format and examplc of scene description generated by VLM on DriveLM datasct.
15

---

Images
Critical Objects
Questton: Current scene: FRONT Joouing CMA FRONT LEFT. WWUNT_RNNTAAAAA seoijng You driing
urfenn you Sheose describe your criing The object should the object's conegoy. objec's bounding box, the objects locatin relatve to ego the ego lane.
streel and intees the
detail
There
information cbjects the right ago oor dijects crefs stop
crboro, creh cao ond neighboring lone, "AAILFRNNT cirets dooo (415, 463), (437, 546) </b00, crcb wecring other botom and wecring other too bloycist on the ego lone of the ago cur, FRONT stnds <boo (560, 4240. (598, 596) </boio, <rets weoring other top and wearing black bottom bicyckist the len of the ego con negriboring lone, CAM, FRONT <trefs eboso 0126. 4072. (849, 8530 </boo, ands weering bloyclis the
"CAMFFRONT <frets cooo (559, 394),
bontom wecoing
pedestrion
CAVLFRONT
solfom neighboring ego cor, FRANTRRNNT </rets
"GAM_FRONT" strefs sign on d000 (396, (412, 438) 4/sso.
There important objects the right crboro, erets wearing ego cot FRONT
Information
chjects cnd's rnd sigr (594,
CANUFRONT_LETT
CANLFRONT FOROOONTRIS RIGHT
FAANTRRNT ctrets
other bottom pecesttion on ego lone c/rel choso (529, 454), (540, 592) c/soo, ents wecring other actiom and weoring other top b cyclist on the ego lone of the ego por. CAM FRONT c/ne's doow (566, 481). (599, 596) crhoos crefs wearing other bottom and weoring other top pedestros on the left or the ago cor neighbbring lone. CAM, FRONT </rehs <boio (418, 466), (443, 550) other too bioyslot other botfom and 'CA,,FRONT </rets
Mobiow
Quertion: What the motion state of werring other bothom pedectrion on on the ego lane of the ego par. CAM, FRONT (529, 454), (549, #22)7 you should describe the utject's stalus, and the motion direction relutive the ego cur
</sor, crefs
neighooring 1255, </bos,, wearing book bicycket the FRONT c/ret> choo (299, 436), (352, 693) </boio.
Crossing
away ego
top wecring ago neightoring lone.
GT: Creasing the crocowalk, away from ego vchicle
Risk Ranking
importonce should ego
stop sign right ego DAALFRONTYS54 386), (616. 426) belong?
three actional impomonne ronk high. medium, low You should elso give decision reasoning
Answer: High, the stop sign of high importonce becouse it on the right side of the ego oor and it on indication to ego car to stoo of the intersection while ago oor infends to turn left.
High stop sign of high importance becouse signals traffic stop, the ego slowing cown intends to turn the inforsection.
Questior: To what level of importance should ego car conaider wearing other botfor and weoring ofher 'opp pedestrion on the left of the egp can neighboring
lone. "CHALF RONT1418 466), (403, 550) to befong? There are three optional importonce ronk: high. medium and low You should also give the decidoo reasoring
Answen Medium, the pecesttion is of medium importance becouse they ore croesing the creaswalk the left neighbering lone, moving owly from the can inrends left the intersection
GT Medium, podestiion corrying backspokk medium importance becouse crossing crosswalk along neighboring ego lane moving away from ego vihicle de the ego vericle is moving.
Figure 8: Annotation format and example of scene description generated by VLM on Rank2Tell dataset. It demonstrates the robust performance of our VLM in real-world scenarios
in real-world scenarios. The versatility of our scene understanding module enables the proposed LeapAD with the promise of practical applicability across real-world scenarios.
C Prompt Details
We outline the specifies of the system prompt (Figure 10) utilized by our Analytic Process during the accumulation of experience within a closed-loop environment. The prompt consists of task definitions, meta-actions, adherence to traffic rules, and the desired output format. Furthermore, the figure illustrates the system prompt (Figure 11) utilized during the reflection procedure, which contains fundamental descriptions from the previous prompt alongside criteria for identifying potential errors within historical frames. We also detail the prompts (Figure 12) of the VLM utilized for identifying the critical objects in the traffic scenes.
16

---

Objects
Eloborstion
Semantic Attobata:
Spatial Attritute:
DOO
BAA
Cotegony, Cobs Appwosrcs, Age
Lone, Direction
CAN FROMI
What Is vouu cescriation crehsitoiin White fox
Which lones are each ontiical ozject on the sorne?
Infrastructure
"CMM FRONT_LEF LETTTLLFFT//ieee -itFse//s>
01000 1000)</bors?
cnety Moving
GWw.ROnT...r .IIT
and's CAALFRONT
White rad.
ICA,, FRONT LEFT EFFic/reb--oos (491, #66). (10D, //
</ref> dtoo (488, 394)
</boo>
C: What is the vioud descriation
cref-Groen
the lone, and ane Statiorrryyyy White muck ISAM_FRONT RIGHT
"CMM_ Www.__RONTTcfee (68
<crets chowo(G, 31 (ta6o, 1000)
394), (567. 473)</bo00?
CAel TRON" SIGHT
Groen lightt
<c/boso
right lome
Mation Attributt:: Intortion, Action
Risk Ranking & Rassoning: High, Mealum, Low
Cyclist
Pedestrion
"CAM_""OR WMML/RRONTLE LLOT </refs <boro (60: 456101000 1000)
Moving
22 What moving stotue
object Noxts- Siver sedon in 'CMM, BACIC chrets (450,
should ego car consicer Green
what leve importonce
crehs Stationary White truck
500), (613, 799te/hoos?
light in "AM,, FRNNT c/rets
cooos1688 1967, chooss?
CAN 127T
'CM, FRONT RIGHT <trets (0. 3), (1000, 1000; </bei
What is the acserved status of object Moxing Silver sedon
Going shead
High, the traffic light high Importance bocouse noe ago cor go stroignt
crefs Moving Sihee sedon
'CaM BLBBACC (1r66 5000, 1613, 722/erboo>
"CAMLBBAC c/refs eboo> (465. 500), 1638, /22) c/boi
intersection and the green color Indicoting to
CAN WWW.MOX.JOO 20SHT
Mavinng towards ego cor.
Figure 9: Detailed data format for the descriptions of critical objects.
SYSTEM PROMPT
trained OpenAL
occurate and odvice humon driver complex
urban driving scenaris receive some desoriation from view onboord camera, You' Information At each decision frome you receve information cbout the current scone and a collection of octions You will perform reasoning bosed on the the desoriation frant-vew image. Eventuclly you will select the oppropriate aztion outpur from the oction set Make sure trat of your recsioing outsul the "## Recsoring" seation, and the * Decision" actiion should only ourpul the nome of the octior, 'AC.
onivng Inferences decitions based on
"IOLE"
at Avoilable Actions
AC: increasing your speed
DC slon down your speed.
STDP zrop, your curment spead shord be nern
You must obey these important ruka below:
moving fowords should pay attertion right lone. you don't need dlo down. You need to they change to ago lore and come close, then you
on
Vehictes
patential
ore awaye moving on
down or sropi keep your distnce and avild me potentisl call sion
When your steering value negottoe you should pay ottertion onjecre the left lone: when your stearing vouue ie pos tive. you should pay attention onjects the
night
should always
dialance from
expacially the object
lone. You should 'bd' STOP object very
the ogo
2f you see red light on the ego lane from the voew you should DC or 'STO imm ecictely You should shop just before the white line (perpendicular to the direction you ore nunning) front of you and do not cross it than 25m away from vericle, should slow down ('C11 and distonce less meters should "SOO objects kinge 25m
there
immediitelly
lights stop signa
you, you sponeds Oeg
lowar ther tm/hd you shoule sodect irelaand of "IDLE"
If there is any stop sign that is less than 10m away, you ahould DC and if less thon 5m dway from the vericle, you should 'STOP immediately.
onswer should
format
Recsoning
receuning based on the descriaption of the front-view
Decison
one oct action det/SHOULD BE exccty some and no other words()
You must obey the rules above. Ann
Figure 10: System prompt for Analytic Process
17

---

(EE SYSTTN..SSTE..... PROMET-
You are per of outemntve solety you teem Your taak recvcluste rececring and dediaooo bosed on currert veticle ccoident informat and deseripione of
Iistrriall fremes You srould iderrity porirntill inference errors in hustoria fromes and propose rersed mirenecs end decolers bosed on thi recnilaition hote that you
stau sdlect one hittrroon Inaim that haw the best chante meniing the oument aoscont to augpoe your mesilcstion chout the moseeing and dezitoon
Hole sure mot your reasonting ourpuf in the the Reasoning" section ond in the "ew Decidon section you shouls only citpur the nomr the action eg NC., TOOE
44 Auciicblle actiors
Ac incressing
DC slow
your
obey impo
ape movinn ego bringg potenttal or right lone, you don't need slow down need to notice champe lone and mop keep your dironce and averd the ponertial col lson
should pay ottention otject ogo Vericles
ahways moving
you
When your smeening sooringwww vo negette you should pay omeentin objechs the left lonz when your steering volue poetiie, you should pey dttertion orjents the
right lome.
you should clways kees your dusonce from the oljecta around you, capesiclly the object the ego lone. You shoull TDC or even 'STDP when on oiject la very cleae on
the tome
1f you rod lgte on the ago lone from the view you shoul DC "SOP" Immesieney You chould rtop just before the white line (perpendicu
direction you
are running) in fromr off you ond do not crose
there otject in the view that is lesa than 25m ovay from the verice, should slow down (DCC), and the daltnnce los then i5 mikers you shoul STOP
immasdioolly owost laton. If the disennes of otjects
then 25m, there reftis ligtt signs front
than lmvs), should there
insloand of TDUE ehould IDC
10m
the vehicle should STOP immediately
storical
Becenning
resscring about cecidion based on the sescrioption oelected frome Pay aftertion to modttving pronouns, such using the trome" or momerf Instoor using the Hatorical frome" or for fime of"
a Decision
ore the octone in the oction setud-DDUUD DE exccty some ad no other wordel
You muat oaey the rules dbove:
Figure 11: System prompt in the reflection mechanism
NUMLSYSTEMLBR WALLPROOPLLTTT TENPLATE
alredt mage inalssies you scone that may aftect your driving The otject information should conton the objects omegory abjects location mistie to ego car ago lone the otjects motion stone and the objects bounding box. os well os the approdincte distncee from the ego oor Expecially there are any trofts iggts the information should contaln the Inoi: lptt's color end bounding box.
"Carmnt Delm--
shiing
any skpeds
Figure 12: VLM system prompt
Table 3: Comparison of our LeapAD with competitive methods in Town05 Long benchmark. "DD" and "KD" denote data-driven and knowledge-driven, respectively.
Method
Modality
Type
Annotations
DS (%) f
RC (%) 1
IS (%) 1
DriveMLM [39]
L+C
DD
2M
76.1
98.1
78.0
ThinkTwice [40]
L+C
DD
2M
70.9
95.5
75.0
InterFuser [41]
L+C
DD
3M
68.3
95.0
72.0
TransFuser [42]
L+C
DD
228K
31.0
47.5
77.0
VAD [49]
C
DD
228K
30.3
75.2
TCP [43]
C
DD
420K
$7.2
80.4
73.0
NEAT [45]
DD
130K
37.7
62.1
61.0
Roach [46]
C
DD
43.6
80.4
54.4
WOR [62]
C
DD
1M
44.8
82.4
54.0
LBC [48]
C
DD
157K
7.1
32.1
22.1
CIL.RS [47]
C
DD
720K
3.7
7.2
51.4
LeapAD (ours)
C
KD
IIK
51.7
100
51.7
Table 4: Ablation on the number shots.
Table Affect memory size few-shot learning.
Shots
DS (%)
RC (%) f
IS (%)
Mem Sizes
DS (%)
RC (%)
IS (%)
51.77
99.72
52.45
51.77
99.72
52.45
64.95
100.0
64.95
90
60.72
98.30
62.30
2 71.20
94.86
73.60
900
66,68
97.93
68.10
3
72,33
100.0
72.33
9000
72.33
100.0
72.33
18

---

D Reflection mechanism
As mentioned in Section 3.3, we employ the Analytic Process reflect on traffic accidents, facilitating the system's s capabilities to continuously improve. Specifically, in the reflection mechanism, we maintain memory queue Q {(D,, Rg S,) storing the previous 10 samples from Heuristic Process at a frequency of Hz. When an accident occurs, samples in Q are fed to Analytic Process reflection. This prompts system identify keyframes that caused
accident and provide correct reasoning and decisions, which are then added to the memory bank. The detailed prompt used the reflection procedure depicted in Figure 11. In addition, Figure 16 depicts specific reflection procedure.
E
Other experiments
Moreover, we provide the evaluation results on the Town05 Long benchmark in Table 3. The results show that our LeapAD) can still achieve competitive results compared to those methods that take images as input, despite using annotated data that is much smaller. Our study delves into the nuanced effects of few-shot learning and memory size through rigorous ablation analyses, as detailed in Tables and 5. The baseline performance the SFT model without memory presented in the first row, revealing clear positive correlation with increased shots and expanded memory capacity in closed-loop experiments. Table 4 substantiates the efficacy of the few-shot heuristic process and highlights the success of our method in identifying similar memories. Table 5 demonstrates the incremental performance gains as memory accumulates, exemplifying the principle of continuous learning.
## Scene Description
There are importort objects the scere The information of these key objects ore crefs In CAM_FRONTI rod tramcc light appraximahely
right ego CAM_FRONT: vehicle locoted opproximately 10.226 the front right of the ego con la oway from ago the right </ref> eboo (468, 444), (805, 644) </bowo.
(494, 330). 1502, </bose
Ego State
Your current 2225993368795865 m/ss Your
Reasoning
The description indicates that there red troffic light approxinately 3L712 meters front of the which immediole
the light to slow down avoid croesing white line intersection and not entering intersection. current speed ego cor 2 225903398796665 m/s which is low and acceplable under urbon driving concitions. Fowever the presence of mainianinn speed
the
necessory
overnides the
#W Decision
Figure 13: Case study at intersection.
19

---

#W Scene Description
There ure 2 imporiani objects in the scere The nformation of these key ubjects ore <rel In CANLTRROT vehidle located coorowimutely 101 the front opprodmatally 7673 direct </refx-boxx (681, 465), (913, 687) </boso-crat CAMJ EFT, opposite direction </rets-baxc (197, 476), (509, 671) </bors.
ago driving
Your current speed is 6.3816738999881 m/s. Your current steering value is -0:877666001027788
Reasoning
right ego wurrunt of feniion us is within the some lone or close The relelive csordinales show that vehicle moelly right lone, not directly the ego lone. Given the current steering volue negative, should also be aware of objects the mimediolee right lone reletvelyy
approdmately
enough
the scenan Since the veh any suiden lann change into the ago lone could present risk, would be prudent to slow down to maintain safe following distance and ready react necessory. There red troffic lights stop mentioned those rues do
side,
vehicle the left lone
direction
other details
ottention, does not influence dedalon-making
Figure 14: A case where bicycle suddenly appears.
20

---

F Visualization Cases
Representative cases. We also provide several examples show the qualitative performance of our proposed LeapAD, as illustrated in Figure 13, 14. These cases present the zero-shot results of our proposed system: The VLMs take the images of the current scene and generate the scene descriptions, which are fed into the decision module (Heuristic Process) along with the ego state (i.e., speed and steering values) for reasoning and decision-making As shown in Figure 13, VLM correctly perceives the important traffic participants at the intersection, such as the red lights and passing vehicles, prompting the decision module to make a deceleration decision. Figure 14 showcases challenging scenario where cyclists suddenly appears from the curb while the vehicle is moving. The system adeptly detects the risk and promptly responds with a timely decelcration.
Cases of few-shot prompting. Figure 15 presents case to show the results using the few-shot strategy. As mentioned Section 3.4, several memory samples are queried based on similarities of the scene descriptions' embedding between the current scene and those in the memory bank. From this case, we can see that the queried samples are highly related to the current scene. And our system LeapAD observes there is red traffic light and several vehicles in the front. According to the previous experience and current ego status, the system suggests that should obey the red traffic light and prepare stop.
Cases of reflection mechanism. We also provide an example to illustrate the reflection mechanism in Figure 16. When an incident occurs, the historical descriptions, reasoning, and decisions fed into the Analytic Process to analyze the sequence of events meticulously, identify potential errors, and provide the reflected reasoning and decision. As the case shows, after the collision occurs the current frame (frame 0), the Analytic Process found there exist reasoning errors at the previous frame (frame -2). For instance, when a vehicle is dangerously close (6.98m) ahead of the ego car, the initial analysis by the Heuristic Process incorrectly identifies the crucial object and misinterprets its movement status, necessitating immediate attention. Furthermore, the Heuristic Process struggles to comprehend the relationship between the ego car's speed and the safe distance from the vehicle in front, resulting in poor decision-maie making. Conversely, during the reflection procedure, the Analytic Process accurately identifies the vehicle 6.98 meters ahead as the primary concern. It questions the prior decision to maintain an "IDLE" status, suggesting may have been an inference error, as stopping would have been necessary to avoid closing in too rapidly on the vehicle ahead in this scenario.
Failure Cases. We have included two typical failure cases in the uploaded PDF. (1) "Run red light" as shown in Figure 17. this scenario, the system lacks temporal information regarding the yellow light's remaining duration, making it difficult to determine whether to accelerate through or stop. When the light is yellow, the system cautiously issues a "DC" command, causing the vehicle to cross the stop line slowly. When the light turned red, CARLA interpreted this as running a red light, even though a "STOP" command was issued at this time. (2) "Collision" as shown in Figure 18. In this case, the VLM did not detect the at the left rear edge of the field of view due to the camera's field of view limitation. Furthermore, in the CARLA setting, other vehicles will not proactively yield to the ego vehicle, leading to collisions caused by other vehicles.
21

---

Current frame
Queried memory
Description
## Scene Description
There are Importont objects in the scene The information of
There are 3 important objects the scene, The information of
these key objects ure: cref red traffic light is located
these key objecis are <in's red trofile light is located
oparaxinately 20845
the front the ego car c/refs choo
approximarey 24985
the front of the
</ref> <bowo
(520, 342). (535, 370) </boio <refs vehicle lccted osprodmmately
(504, 248). (314, 280) </boes <refs vehicle located appnowmmately 434466 m to the tront right at the ago can is arive in the some direction </rofs choso (503, 191). (173, 520) </hows crefs whirle localed upprossrualeyy 573
678688 m to the tront righr at the ego car ln driving in the same
direction c/rets chme (5MM, 490), (578, 518) c/hoos crefs vehicle is loculed approximalely 36.513 lo the
of the eyo our oway from eqo the left </ref> eboo (391, 494). (423. 523) </box> crefs vehicle located approximatdy 32:154 m to the front left of the ego furn left c/refs ebsio (292, 427). (395, 542) e/box..
from lell away from ego the left </ref> <ooo (379, 489). (431. 520) </boro.
ego cur
#* Ego state
Your current speed is 0.093986 m/s. Your current steering value la 0.0.
#* Ego state
Your current spead 6,744 m/a Your current snearing volue is -0.0088
## Reasoning
There is red traffic light in both the front view of approedmately 24:955 meters ahead of the eqo cur According to the scenorio description and rukes, must obey the rules of the traffic lights the view. we appreaching a traffic ineerediion proceed of a low speed. Our current speed is already low (0.09328600002222400
#al Receoning
There red troffi light approsimately 20.45 meters the front
the ego cur which requires stoping vehicle comply with trffic reguictions Even though our current speed is relctiveyy low (6.244 m/s), we must dezalernte to come
must
complete stop due to red traffic light Addmoonally, observing traffic light confirms is red and we must abide by it. which still warranta stopping the vehicle to some extent.
m/s). but as we are appreaching the intoraection with light ahead, should prepore stop before the white line and ensure we do not enter the intereection.
#W Decision STOP
#hl Decision STOP
Description
#W Description
There are 4 Importont objects in the scene. The information of
There are 4 important objects in the scene. The information of these key objects are, crefs red trattic light is located front right
these key objects are. crafs red traffic light is locoted oparaximately 29,475
the tront
ago c/rets choss
approsdmeraly 3725
ego c/rets choso
(504, 294). (514. 326) </boi eret> vehiole is ocated asprosdmately 30.57t m to the frant right of the ego our is away from ego to the right </ref> <box> (739. 186). (866. 537) </boso <ret> vehidle la located approsomaeely 38,217 to the corne closer from the left c/ref <amo (378, 483). (664, 527) </box crefs vehicle incated opproeeentlly 17223 m to the front left of the ego furn right </refs <boss (0, 4781, (166. 606) </boc.
(594 393), (602, 429) </boo <ref> whillle located approdmatelyy 47954 m to the fron right of the ego cor is oriving in the some
direction </ref> ebooo (554. 541), (583, 571) </boro <ref> vehicle upproxemately 34.74
of the ego can is
ego car away from ego to the left </re>> <boso (456, 556), (500, 589) </boso crefs vehicls is loonted opproximately 36 791 m to the front right of closer from the right </ref> <boo (3c0, 535). (617, 570) </hoxo.
front left
ego
#* Ego state
Your current speed Is 382230174854917 m/a Your current steering
#W Ego state
Your current speed 0CC411076008114999 m/s current steering value 0.O.
#* Reasoning
multile inpurs conside:
## Reasoning
mrffiic light ahead observed approoomataly 29m, which takes precedence over the from view.
There
From front wew thare's a red traffic light oparaometely 37.2 meters to the front right of the ego car, and since we are approaching traffic intersection we should obey the red light There 4795 meters
2. Our curment speed above the reconmeeded low apeed for oporoaching o traffic infersection, which
vehicle some direction. o vehicle about 36.79 meters to the front right coming closer from the night, and anolher vehicle approomatieyy 3474 meters the front left moving oway. Given that the clesest object the red trattic light and it is obeying the traffic rule whan the stsering volue is at G.O. indicating we are in the ego lone We need to prepore to stop for the red light.
front right driving
3. There are two vehicles in prosomity one on the front-right at about Simn which posas no immediaae threal us itls moving oway, und and 4 Consderrin the ciraance to the red light, we read to stort slowing
onother on the front-left at about 17m which turning
may cross into ego lone, pasing o potanticl
down to stop time and noiid nunning the red light
Our steering focus on one side versus the other of this moment
currentyy neutral (0.0).
don't have
Our current speed is extremely low 0.04 m/s but since the rules suggest that we must obey the red troffic light and prepore to stop
Given the proximity the red light and its preceseence, together with recommended speed for oparsaching a traffic infrsectiin, we must ignore. the speed rule for this scenaria and focus on
when are approoching
infersection, increasing speed is not
current speed and approach traffic light cauttously, As the red troffic light is not imminently close
option. Thus, will maintain
obeying the traffic, light and preparing to stop.
(nat under 25 merens), we are nat required to stop immedistely but should prepored do
Decision
DC
2
#* Deciaion IDLE
3
Figure 15: Case study for few-shot strategy.
22

---

Frome 0
History queue
Frome
Collision!
Frome -9
Memory Bank
Reflection Mechonism
You are a part an autornctive safely and ystssss team, Your tack is reevaluere recsoring
Reflection System Prompt
decisions based current vehicle accident Information and descriptions
Isertify patertial inference erorss in historical fromes and propese revised inferences and based this recviluatio Nate that frome that hos the best chonce of avolding the current accident to suggest your moditcations about the reasoning and decision.
frames.
Historical
Amalytie
Process
Corrected somple
Frame -2
#tst Reasoning
mace was close attentior to the situation. However, the important object in the FAONT view is only 6988 meters to the front of the ego can driiing the some direction. This simonce below the J3-meter "theeholdd where STOP is mandoted aveld co collison. The speed
the
prevenling
becouse
emphaaii
3.338556/807388 m/i, there was sufficient speed to warrent lone) elevated its risk potenifial The historical decision to IDLE was a poteriiaa inference error since in this frome the ego car should closing too quickly vehicle ahead.
to ensure safety. vehicle seing an the
(ego
have stopped
Decision
STOP
Historical mistaken somple
#-8 Frome -2
Scene Description
There ore importontt objeste in the scene. The information of these key objects one cref> In 'GAM_ RIGHT vehicle located appraximately 16.50 to the front right of the car is driving in the same direct onc/rad--bbooo111 484), (590, coo). crefovehicle located </boxo. approxdmately 16.313 m to the front left of the ego can is drive in the same direction </refs <horo. 495), (279, 806). crefs vehicle is located approdmmattyy 6.98 the front of the ego car driiing the same direction </ret> <oo>> [164, 509, 537, 100) 4/box.
oppnodmattll 46.445 to the
right
ego
right
490).
Ego state
Your current speed 3.36880 167892288 m/s Your current steering value is -0.0002922329841333
Reasoning
The description provides severol key points consider:
The vehicle in CAM HIGH!! in to 6o awyy and driving in the some direction; this does not pose immedctt nakk The vahicle direction: this requres ottention becouse
CAM
698 away also driing in the porentially be risk due to the snort distonce. The vahicies in "CAMLLEFT and FAOLF are of sate distance relatielyy for from the (16.313 and 46.666 respestiveyy. thus they don't require immediate action. The current speed the 333685167842367 m/s which
the egp
is curment steering volue
excessively 8 ow. there's room for occelerotion if stuation ollws, negative 01602292332284-335 which meons we should pay attention to objects in the left lane, but the object in the left lane is not and nearby object extremely
There mantion of traffic lights the imunedicle vicinily that would recuire oclion The pr many focas should be or mainlcining a safe distonce from the vehicle direclly ahead in the ego lane while not slowing down unrocessarlyy the
signs
observations
ahead
or speed ego were increase signif contly.
appropriate in mesa ares would be to DDC on even SSOP However, given the current speed only slightly above m/A, significart deceleration not required. Therefore, appropriate action would be maintain Decision
and monitor the siluction closely.
IDLE
Figure 16: Case study reflection mechanism.
23

---

light
After 5 seconds, light turns red !!!
Time: 28232418
28232453
#* Scene Description
There important objects the The information these key objects <ref> "CAN,FRONT, yellow traffic light located opproxdmately 14015 the front right ego </ret> <boko (320. 156). (379, 356) </box>, FRON,RONT, vehicle locared approvimately 19.784 m the front left of the <boro (245, 469), 543) </box.
from to the left</ref>
#el Ego State
speed 01340855006024415566 current stsering 0112698765937922699
# Reasoning
Additionally, the ego speeds should be considered. However: presence of yellow Iraffic light aponoximaleey meters the front rightl indicates that oction will soon be necessory at the intersection Glven the proximity the yellow light and expectation will furn soon, precoutton required The red light takes priority: the vehlole must prepore stop, Keeping speed low precautton at this specific moment apprapriate.. The vehicle on the front left moving away, does pase on immediale
speed very (0.144 m/s), which the threshold where time westing at
Decision
Figure 17: Failure case that running the red light.
Collision!
Scene Description
There are 2 important objects in the scene. The information of these key objects are crefs In 'CAM_FRON FRONT, vehicle located cpprodimaelly 15.975 to the front left of the car driving some direction </reh> <booo (283, 432) (396. 584) </box <ref> In CAM_FRONT, vehicle looted upproximately 13.945 in the front of the ego con is drive in the some direction </ref> <boo> (441, 476). (528, 587) </booe
Ego
Your current speed is 1217683110116860 m/s Your current steering volue O.C.
Reasoning
Information. understond the some direction, and tis vehicle within the bounds of the ego lane, which suggests our direct path. current speed
directly
driving
there's object close prosmmity (under 25m) lone, our priority to adjust our speed accordingly to ensure safety. Our steering value 0 indicating that we are moving stroight chead, and thue the vehiele directly in froit thon metera s0 don't need to come complete stop yet, we should decrease apeed
(12176832130196866 m/s),
ego
evant lone. The distance to The vehidle ahead greater
Decision
Figure 18: Failure case of collisions caused by other vehicles.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The motivations and contributions are well depicted and summarized in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
No theoretical results are included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The data preparation and implementation details are clearly and fully presented
in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [No]
Justification: We provide open access to the project page, which includes the GitHub repository for subsequent releases.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/uuides https///ooosuuiisiiinnne...e
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httpps.cccpubiic//eecc /ttoe/uuudee ssi www.inssssooiiiiiiio oniioiiyl) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details are presented in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided the mean value and standard deviation of several runs for the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources illustrated in the implementation
details of Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https /neurisu.....e/ cc cctpublic//eicccccccccc ctwp:i//iieiittieeeeeeieeeeeeeee ines?
Answer: [Yes]
Justification:
We make sure that the research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: have discussed the potential societal impacts in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "nDXBRpltDy", "reviewer": "Reviewer_SMxu", "summary": "The paper \"LeapAD\" introduces a new approach to autonomous driving that addresses key challenges in adaptability and interpretability. It draws inspiration from human cognition to enhance decision-making processes in complex environments.\nThe system incorporates two complementary processes:\n- Analytic Process: Provides thorough analysis and reasoning, accumulating driving experience through logical reasoning.\n- Heuristic Process: Employs swift, empirical processing and learns from the Analytic Process through supervised fine-tuning. This dual-process setup enhances adaptability and performance.\n\nClosed loop testing in the CARLA simulator demonstrates that LeapAD outperforms methods relying solely on camera input. The Heuristic Process can inherit knowledge from an Analytic Process powered by GPT-4, leading to continuous performance improvements as the memory bank expands.", "questions": "- The authors should detail the data generation process for complex driving scenarios like intersections, lane changes, and overtaking. Based on my understanding, the current model primarily focuses on simpler scenarios involving a single-lane and limited interaction with other actors. \n-  I recommend evaluations in more dynamic settings such as intersections and scenarios involving lane changes and overtaking, where multiple actors interact and cooperate for safety. \n- A comparison with traditional vectorized or feature-based planning systems, such as Wayformer or Precog, would be beneficial. These systems process scenes as images and convert data into vectors instead of text, which might offer insights into efficiency and performance. \n- I see DriveLM also has good performance in the case of VLM based driving. Is there any reason why that has not been put as a baseline, considering the similarity in dataset generation processes.\n\nI look forward to seeing how these suggestions might be incorporated to further enhance the robustness and applicability of the proposed approach in more complex driving scenarios.\n\nRef: \n- http://arxiv.org/abs/2207.05844 \n- https://openaccess.thecvf.com/content_ICCV_2019/papers/Rhinehart_PRECOG_PREdiction_Conditioned_on_Goals_in_Visual_Multi-Agent_Settings_ICCV_2019_paper.pdf\n- https://arxiv.org/abs/2312.14150", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is generally well-written and but easy to follow. Good motivation for the model design in the introduction. \n2.\tI like the problem setup: how can we design AV systems that continually learn from its mistakes. \n3.\tThe experimental results seem to support the authors' claims.", "weaknesses": "I overall liked the idea of closed-loop autonomous driving approach that could emulate the critical attention mechanisms required for smooth driving environment in safety critical scenarios. The notion of heuristic and analytical processes for executing actions in robotics seems a novel approach.\nHowever, my primary concern lies in the setup of data and models for generating scene descriptions into text to identify critical objects. Operating within the text domain, which requires subsequent interpretation and tokenization by the analytical and heuristic modules, seems less efficient than using a direct vectorized representation. For instance, representing an object with parameters such as {v = 0.2 m/s, s = 3m, class = Car} is likely more efficient and robust than the text output \"The car is 3 m away from the ego vehicle and is moving at 0.2 m/s.\" This textual method could lead to inefficiencies, especially in scenarios with multiple dynamic actors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your constructive comments. We provide discussions and explanations about your concerns as follows.\n\n**Q1:** My primary concern lies in the setup of data and models for generating scene descriptions into text to identify critical objects. Operating within the text domain, which requires subsequent interpretation and tokenization by the analytical and heuristic modules, seems less efficient than using a direct vectorized representation. \n\n**A1**: Indeed, adopting vectorized representations can significantly compress data, thereby accelerating inference speed. However, by logically integrating target attributes (such as orientation, distance, and category) into coherent natural language statements, we not only clarify the meaning of these physical quantities but also better align with the data distribution of pre-trained large language models. This enables us to leverage the embedded world knowledge in these models to enhance their understanding of autonomous driving environments while maintaining their generalization capabilities.\n\nIt is also worth noting that, as shown in Table 1 of our paper, compared to traditional vectorized or feature-based methods such as VAD, InterFuser, and TransFuser, the amount of labeled data we use is typically one to two orders of magnitude smaller. Furthermore, our dual-process decision-making process requires no human intervention and is capable of continuous improvement, clearly demonstrating the effectiveness of knowledge-driven methods.\n\n**Q2**: Evaluations in more dynamic settings as intersections and scenarios involving lane changes and overtaking, where multiple actors interact and cooperate for safety.\n\n**A2**: In fact, the closed-loop benchmark in CARLA Town05 involves numerous dynamic settings, such as intersections, traffic lights, STOP signs, and sudden appearances of pedestrians or cyclists from the side. Our method has achieved strong performance across these diverse scenarios, demonstrating its adaptability to dynamic environments. Additionally, it is important to emphasize that our approach can handle complex situations such as intersections and some corner cases. As shown in Figures 13 and 14 in the appendix, our method is able to handle intersections with multiple actors (vehicles and traffic lights) interacting (Figure 13), and react appropriately to unexpected events, such as a cyclist suddenly appearing from the side (Figure 14) by making timely decisions like slowing down.\n\nMoreover, the project page linked in our paper's abstract includes several demos that further illustrate our method's adaptability in dynamic environments. For example, the first video shows our method slowing down and coming to a stop when a cyclist suddenly appears at an intersection between the 5-15 second and 35-40 second marks. The second video, between the 25-45 second marks, showcases our method of navigating a complex intersection with multiple participants, and the third video demonstrates our approach stopping briefly at an intersection with a STOP sign.\n\n**Q3:** A comparison with traditional vectorized or feature-based planning systems, such as Wayformer or Precog, would be beneficial. These systems process scenes as images and convert data into vectors instead of text, which might offer insights into efficiency and performance. \n\n**A3:** It is important to note that our primary focus is on exploring continuous learning in closed-loop autonomous driving based on a dual-process approach, using image inputs and a knowledge-driven method. Unlike our approach, Wayformer does not use image inputs but rather relies on ground truth sparse abstract state descriptions of the world, while Precog requires LiDAR input. Both of these methods are evaluated in an open-loop setting, which has certain inherent limitations: (1) the current decisions do not influence subsequent navigation, making it impossible to assess cumulative errors; (2) there is no interaction between agents, leading to a lack of dynamic behavior; (3) and there is no global closed-loop evaluation metric. In fact, we have already compared our method with traditional vectorized or feature-based systems. As shown in Table 1, methods like InterFuser, TransFuser, and VAD use neural networks to represent scenes as implicit vectors. Our method outperforms most of these approaches while only requiring one to two orders of magnitude less labeled data, and it also has the capability for continuous learning. This further demonstrates the effectiveness of leveraging knowledge from large models.\n\n**Q4:** I see DriveLM also has good performance in the case of VLM based driving. Is there any reason why that has not been put as a baseline, considering the similarity in dataset generation processes. \n\n**A4:** Our LeapAD features a dual-process approach for closed-loop autonomous driving, whereas DriveLM only reports open-loop performance without providing closed-loop metrics. Additionally, DriveLM has not released the CARLA dataset, the fine-tuned network weights on CARLA, and the inference code on CARLA, making it difficult to conduct closed-loop experiments using DriveLM. Furthermore, DriveLM is based on the BLIP-2 architecture, while Qwen-VL, which we use, supports higher resolutions and has demonstrated better grounding and vision-language understanding capabilities across multiple benchmarks, even with a similar parameter count. Finally, the output from the original DriveLM is quite verbose. For our dual-process decision method, the scene understanding-related data in DriveLM is more important. Therefore, we refined DriveLM's data and combined it with Rank2Tell and collected CARLA data to fine-tune Qwen-VL as our scene understanding module.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1qkVsvJU37", "reviewer": "Reviewer_eSfn", "summary": "This paper introduces a paradigm to design an annotation-efficient end-to-end autonomous driving system that harnesses the power and generalizability of open-source LLM models. It proves that critical frame/instance selection are critical to a decision-making module training. This method is evaluated by closed-loop testing in CARLA and achieves the SOTA performance among camera-based methods.", "questions": "1. Why decouple into 2 separated modules, scene understanding, and decision making?\n2. The scene understanding section mentions that the motion direction is one of the outputs. However, since the input sensor data is single-frame based, how does the model know the motion direction?\n 3. It is not clear how the interaction with GPT4 completes in the reflection mechanism. It would be better to provide more details.", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The core idea is straightforward.\n2. Achieves the SOTA result.\n3. Complete adequate ablation studies to support its claim.", "weaknesses": "1. No quantitative benchmark on its VLM module on simulation and the real world. Only some samples are listed in the paper.\n2. The paper only presents an overall benchmark on the system but no failure case analysis.\n3. The result relies on the foundation model performance and the paper does not show a way to fill the gap between the simulation and the real world, which limits its impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your constructive comments. We will discuss and explain your concerns as follows.\n\n**Q1:**  No quantitative benchmark on its VLM module on simulation and the real world. Only some samples are listed in the paper. \n\n**A1**: Thank you for your valuable suggestions. We have added evaluation results of VLMs on Rank2tell (real) and Carla (simulated) datasets. We evaluated the performance of Qwen-VL-7B on these two datasets, and also evaluated LLaVA-1.5-7B for comparison. We provided Grounded scores, including precision, recall, and F1 score, to assess the models' grounding performance, as well as Chat Scores, including language score (ROUGE) and GPT score (GPT-4-turbo), to evaluate the models' reasoning and question-answering capabilities. Our experiments found that Qwen-VL-7b demonstrates stronger grounding abilities.\n\n（1）Evaluations on the collected Carla data.\n|  VLMs| Precision | Recall | F1 Score | ROUGE | GPT Score  |\n| --- | --- | --- | --- | --- | --- |\n| LLaVA-1.5-7B | 34.95 | 32.00 | 33.41 | 78.04 | 58.09 |\n| Qwen-VL-7B | 51.41 | 47.14 | 49.18 | 83.24 | 63.01 |\n\n（2）Evaluations on Rank2Tell test dataset.\n| VLMs | Precision | Recall | F1 Score | ROUGE | GPT Score |\n| --- | --- | --- | --- | --- | --- |\n|LLaVA-1.5-7B| 28.49 | 25.22 | 26.75 | 69.26 | 65.20 |\n| Qwen-VL-7B | 46.70 | 37.37 | 41.52 | 70.23 | 66.59 |\n\n**Q2:** The paper only presents an overall benchmark on the system but no failure case analysis. \n\n**A2:** Thanks for your advice. We have included two typical failure cases in the uploaded PDF.  \n\n(1) “Run a red light” as shown in Figure 1 in the uploaded PDF. In this scenario, the system lacks temporal information regarding the yellow light's remaining duration, making it difficult to determine whether to accelerate through or stop. When the light is yellow, the system cautiously issues a “DC” command, causing the vehicle to cross the stop line slowly.  When the light turned red, CARLA interpreted this as running a red light, even though a “STOP” command was issued at this time.\n\n(2) “Collision” as shown in Figure 2 in the uploaded PDF. In this case, the VLM did not detect the car at the left rear edge of the field of view due to the camera’s field of view limitation. Furthermore, in the CARLA setting, other vehicles will not proactively yield to the ego vehicle, leading to collisions caused by other vehicles. \n\n**Q3:** The result relies on the foundation model performance and the paper does not show a way to fill the gap between the simulation and the real world, which limits its impact. \n\n**A3:**  We use the VLM to observe and interpret the driving environment and provide scene descriptions. The LLM then performs driving reasoning and makes decisions based on these descriptions. Consequently, the domain gap between simulated and real-world scenarios primarily affects the scene descriptions generated by the VLM. Notably, the VLM demonstrates strong generalization capabilities. The data used for fine-tuning includes both simulated and real-world data, which enables our VLM to generate accurate scene descriptions in both contexts. This is illustrated by the cases shown in Figures 6 and 7 of the paper, and further supported by the quantitative experiments (A1) we have included.\n\nWhile there is an inherent gap between simulation experiments and real-world closed-loop scenarios, we believe the main limitation is the current lack of a high-fidelity simulator in the industry, as highlighted in the limitations.\n\n**Q4:** Why decouple into 2 separated modules, scene understanding, and decision making? \n\n**A4:** We have divided the system into separate modules for the following reasons:\n\n(1) The modular design enables easy replacement and upgrading of individual components, particularly the scene understanding module and the Analytic Process.\n\n(2) We adopt the scene understanding module to generate scene descriptions that effectively encode the environment.  The dual-process decision-making module generates the driving reasoning and decision. These historical scene descriptions and reasoning can be conveniently encoded into a vector database (memory bank) for rapid retrieval of similar scenes and to guide the Heuristic Process in making accurate decisions through a few-shot approach.\n\n(3) As noted in the limitations section of our paper, the VLM’s inability to participate in the reflection mechanism hinders further system improvements. Addressing this limitation will be a key focus for future development.\n\n**Q5:** Since the input sensor data is single-frame based, how does the model know the motion direction? \n\n**A5:** Indeed, accurately assessing vehicle motion based on a single image alone is challenging. Interestingly, we have found that, due to the remarkable generalization abilities of large vision language models, it is possible to infer the motion direction of vehicles simply based on the vehicle’s heading. As noted in the limitations section of our paper, the current version relies solely on single-frame input and lacks temporal information. Incorporating temporal cues to more accurately assess the motion of surrounding objects will be a focus for future work.\n\n**Q6**: It is not clear how the interaction with GPT4 completes in the reflection mechanism. It would be better to provide more details.\n\n**A6**: In the appendix (Figure 16), we provide a detailed example of the reflection mechanism and explain it thoroughly in Section F. Specifically, when a traffic incident occurs during the Heuristic Process, the reflection mechanism is triggered. During reflection, information from historical frames is fed into the Analytic Process to identify and correct any potentially erroneous reasoning decisions. These corrections are then added to the memory bank to further enhance the accuracy of the Heuristic Process. For more detailed information, please refer to the relevant sections in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofWaRSxcCh", "reviewer": "Reviewer_zRJR", "summary": "This paper presents LeapAD, a dual-process closed-loop autonomous driving system.\n\nLeapAD first uses a VLM to analyze the scene by selecting and locating critical objects in the scene, and then it uses a dual-process learning approach to learn driving behaviors.\n\nThe dual-process learning system contains an Analytical Process and a Heuristic Process. The Analytical Process is strong but expensive to run. It is used to summarize the driving experience into the Memory Bank. The Heuristic Process is more lightweight and is used to generate controls to control the vehicle. The Heuristic Process is trained with data in the Memory Bank.\n\nThe Analytical Process can also reflect from collision events in previous simulation runs. It will analyze the cause of the collisions and save the knowledge in the Memory Bank.\n\nThe authors evaluated the LeapAD method in closed-loop simulation with the CARLA simulator. They used the Qwen models as the VLMs and GPT-4 for the Analytical Process.\n\nThe evaluation result shows that LeapAD surpasses the performance of the other camera-only models on the CARLA Town05 benchmark.", "questions": "N/A", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The dual-process idea is neat and thought-provoking. It equips the autonomous driving system with the ability to learn from past experiences.\n\n* The method achieves stronger performance than state-of-the-art methods in CARLA closed-loop simulation.\n\n* This paper is well-written and provides sufficient details for reproducing their approach.", "weaknesses": "* The performance improvement is not very significant compared to the baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThanks a lot for your acknowledgement, and we appreciate the time and effort you dedicated to enhancing the quality and clarity of our manuscript.\n\n**Q:** The performance improvement is not very significant compared to the baseline.\n\n**A:** Thanks for your feedback. As you mentioned in summary, LeapAD proposes a new paradigm for autonomous driving that addresses the limitations of current data-driven methods in complex scenarios. We focus on verifying the superiority of this dual-process system. At the same time, we also mentioned in the limitations section that we are actively researching several areas, such as integrating time inputs, enabling VLM to participate in reflection processes, and developing real-world simulations to further improve the performance of our system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mcWlz9XlRk", "reviewer": "Reviewer_kf7r", "summary": "The paper introduces **LeapAD**, an interesting paradigm for autonomous driving inspired by human cognitive processes, addressing the limitations of prevailing data-driven methods in complex scenarios. LeapAD incorporates a dual-process decision-making module consisting of an Analytic Process (System-II) for logical reasoning and experience accumulation, and a Heuristic Process (System-I) for quick, empirical decision-making based on the learned knowledge from System-II. By emulating human attention to focus on critical objects, LeapAD simplifies environmental interpretation and mitigates decision-making complexities. The system is tested in the CARLA simulator, demonstrating superior performance over camera-only methods with less labeled data. The Heuristic Process shows continuous improvement through a reflection mechanism and a growing memory bank, indicating the effectiveness of the dual-process approach.", "questions": "There are areas where improvements could be made: \n\n \n\n### Methodological Concerns: \n\n \n\nThe paper should clearly distinguish between the Analytic Process and Heuristic Process? How are these processes defined and why is the Heuristic Process called such if it uses a lightweight language model? \n\n  \n\n### Experimental Limitations: \n\n \n\nCan results be reported to evaluate the performance of LeapAD using a real-world dataset like nuScenes? \n\n  \n\n### Clarity and Presentation:  \n\n \n\nIt is not clear whether the performance improvement is due to Qwen VLM or the two-system design. Can you report ablation studies to explore the influence of VLMs, for example, also try LLaVa? \n\n  \n\nBy addressing these questions, the authors can provide a more thorough and robust evaluation of LeapAD.", "limitations": "Yes, the limitations are discussed in Section 6.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents several notable strengths across dimensions of originality, quality, clarity, and significance: \n\n  \n\n### Originality \n\n1. **Dual-Process Decision-Making**: The combination of an Analytic Process (System-II) and a Heuristic Process (System-I) emulates human cognitive functions, offering a biologically inspired framework for autonomous driving. \n\n  \n\n### Quality \n\nI think the quality is good. \n\n1. **Continuous Learning**: The reflection mechanism and growing memory bank enable continuous learning and improvement, showcasing the adaptability of the proposed system. \n\n  \n\n### Clarity \n\nThe paper is well-written, with clear and concise explanations of complex concepts. The dual-process framework and its components are described in detail, making the methodology accessible to a broad audience. \n\n  \n\n### Significance \n\n**Advancing the Field**: By introducing a dual-process decision-making framework, the paper opens avenues for research in autonomous driving and artificial intelligence, potentially influencing future developments in the field.", "weaknesses": "While the paper presents some interesting contributions, there are areas where improvements could be made: \n\n  \n\n### Methodological Concerns \n\n  \n\nWhile I appreciate the design of the Analytic Process and Heuristic Process, does the paper clearly distinguish between the two? My understanding is that the Analytic Process uses LLMs, while the Heuristic Process uses a lightweight language model. Why can it be called the Heuristic Process? It would be better to clearly state why can it be called the Heuristic Process and the Analytic Process. \n\n \n\n### Experimental Limitations \n\n  \n\n1. **Quantitative Metrics**: \nThe paper's experimental results are primarily based on the CARLA simulator, lacking real-world experiments. CARLA scenarios are still too simple. It would be better to report results that can comprehensively evaluate the performance of LeapAD, such as using the real-world dataset nuScenes. \n\n  \n\n### Clarity and Presentation \n\n  \n\n1. **Technical Details**: \nThis paper is based on Qwen VLM. It is not clear whether the performance improvement is due to this Qwen VLM or the two-system design. It would be better to include more ablation studies to explore the influence of VLMs, such as LLaVa. \n\n  \n\nBy addressing these weaknesses, the authors can provide a more thorough and robust evaluation of LeapAD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your constructive comments. We provide discussions and explanations about your concerns as follows.\n\n**Q1:** The paper should clearly distinguish between the Analytic Process and Heuristic Process? How are these processes defined and why is the Heuristic Process called such if it uses a lightweight language model? \n\n**A1**: As explained in the introduction, our approach is inspired by the dual-process theory of human intelligence. The distinction between the Analytical Process and Heuristic Process lies primarily in their roles within our system. The Analytical Process is rational, slow, and excels in logical reasoning and creativity across various domains, while the Heuristic Process is quicker, empirical, and domain-specific. Although we use a lightweight language model, it performs a function in our dual-process decision module that aligns with the role of the Heuristic Process.\n\n**Q2:** The paper's experimental results are primarily based on the CARLA simulator, lacking real-world experiments. CARLA scenarios are still too simple. It would be better to report results that can comprehensively evaluate the performance of LeapAD, such as using the real-world dataset nuScenes. \n\n**A2:** It is important to note that our primary focus is on exploring continuous learning in **closed-loop autonomous driving** based on a dual-process approach. Closed-loop autonomous driving involves interactions between agents and the environment, while publicly available datasets like nuScenes only provide open-loop evaluations. The open-loop evaluations show certain inherent limitations: (1) the current decisions do not influence subsequent navigation, making it impossible to assess cumulative errors; (2) there is no interaction between agents, leading to a lack of dynamic behavior; (3) and there is no global closed-loop evaluation metric. We believe that close-loop experiments in real-world environments are primarily constrained by the availability of high-fidelity simulators. Currently, there is no well-established high-fidelity simulator, but developing such a simulator is an area we are actively working on.\n\n**Q3:** This paper is based on Qwen VLM. It is not clear whether the performance improvement is due to this Qwen VLM or the two-system design. It would be better to include more ablation studies to explore the influence of VLMs, such as LLaVa.\n\n**A3:** Our scene understanding module (Qwen-VL) and decision-making module (dual-process design) are relatively independent components. Qwen-VL provides scene descriptions, which are fed into our dual-process decision-making module for driving reasoning and decision. Directly applying Qwen-VL for the decision-making process does not work well. We experimentally found its output can not align well with the data format required for decision reasoning. \n\nEmpirically, accurate scene understanding positively correlates with the effectiveness of subsequent decision-making. However, since our focus is on exploring the dual-process approach for autonomous driving rather than scene understanding per se, we have not extensively investigated the network architecture of the VLM. We selected Qwen-VL as the scene understanding module due to its demonstrated strengths in visual understanding and grounding on various public benchmarks, aligning well with our requirements.\n\nAs suggested by the review, we also added additional ablation studies to explore the influence of Qwen-VL and LLaVA. This includes scene understanding evaluation results on the Rank2tell (real) and Carla (simulated) datasets, as well as the closed-loop performance, detailed below. We provided Grounded scores, including precision, recall, and F1 score, to assess the models' grounding performance, as well as Chat Scores, including language score (ROUGE) and GPT score (GPT-4-turbo), to evaluate the models' reasoning and question-answering capabilities. For the close-loop experiments, we test the performance of the first eight routes of the Town05 Short benchmark. From these results, it is evident that Qwen-VL exhibits superior grounding abilities and better DS scores in closed-loop settings.\n\n（1）Evaluations on the collected CARLA data.\n|  VLMs| Precision | Recall | F1 Score | ROUGE | GPT Score  |\n| --- | --- | --- | --- | --- | --- |\n| LLaVA-1.5-7B | 34.95 | 32.00 | 33.41 | 78.04 | 58.09 |\n| Qwen-VL-7B | 51.41 | 47.14 | 49.18 | 83.24 | 63.01 |\n\n（2）Evaluations on Rank2Tell test dataset.\n| VLMs | Precision | Recall | F1 Score | ROUGE | GPT Score |\n| --- | --- | --- | --- | --- | --- |\n|LLaVA-1.5-7B| 28.49 | 25.22 | 26.75 | 69.26 | 65.20 |\n| Qwen-VL-7B | 46.70 | 37.37 | 41.52 | 70.23 | 66.59 |\n\n(3) Closed-loop experiments on Town05 Short benchmark\n| VLMs | DS | RC | IS |\n| --- | --- | --- | --- |\n| LLaVA-1.5-7B | 78.87 | 86.12 | 92.75 |\n| Qwen-VL-7B | 88.25 | 100 | 88.25 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you very much for taking the time to review this manuscript and for helping to improve our work. I greatly appreciate all your comments and suggestions. Please find my detailed responses below.

As suggested by Reviewer eSfn (Q2), we have included visualizations of the failure cases in the uploaded document. Additionally, due to the page limitation, please refer to A2 for detailed failure case analyses.

Meta-review: The paper "LeapAD" introduces a novel approach to autonomous driving, drawing inspiration from human cognitive processes to enhance decision-making in complex environments. The system integrates two complementary processes: an Analytic Process (System-II) and a Heuristic Process (System-I). The Analytic Process provides thorough, logical reasoning and accumulates driving experience, while the Heuristic Process enables quick, empirical decision-making by leveraging knowledge learned from the Analytic Process. LeapAD's design is centered on the efficient use of labeled data, leveraging open-source large language models (LLMs) to analyze and interpret scenes by focusing on critical objects. The system's dual-process framework allows it to balance the thoroughness of the Analytic Process with the speed of the Heuristic Process, enhancing both adaptability and performance. Evaluation of LeapAD in closed-loop testing using the CARLA simulator shows that it outperforms existing camera-only methods, achieving state-of-the-art performance. The Heuristic Process benefits from continuous improvement through a reflection mechanism that allows the Analytic Process to analyze collision events and update the system's memory bank, further refining decision-making capabilities over time. This approach highlights LeapAD's potential to address the challenges of adaptability and interpretability in autonomous driving.

The paper is well-written and easy to follow. The authors and reviewers engaged into a fruitful discussion where most of the issues could be resolved. The authors provided additional quantitative experimental evidence (e.g. on failure cases and on the VLM module).

There was some discussion about the fact the LeapAD uses text to encode the scene (rather than a vectorized representation), which is still left open. However, in summary all reviewers voted for acceptance. I, therefore, recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version, and to add some discussion on the open point of the textual representation.
Decision: Accept (poster)