A Language Agent for Autonomous Driving
Authors: Jiageng Mao
Junjie Ye
Yuxi Qian
Marco Pavone
Yue Wang
Venue: COLM (2024)
Keywords: Language Agent, Autonomous Driving
TL;DR: Agent-Driver transforms the conventional perception-prediction-planning framework by introducing LLMs as an agent for autonomous driving.

Abstract: Human-level driving is an ultimate goal of autonomous driving. Conventional approaches formulate autonomous driving as a perception-prediction-planning framework, yet their systems do not capitalize on the inherent reasoning ability and experiential knowledge of humans. In this paper, we propose a fundamental paradigm shift from current pipelines, exploiting Large Language Models (LLMs) as a cognitive agent to integrate human-like intelligence into autonomous driving systems. Our system, termed Agent-Driver, transforms the traditional autonomous driving pipeline by introducing a versatile tool library accessible via function calls, a cognitive memory of common sense and experiential knowledge for decision-making, and a reasoning engine capable of chain-of-thought reasoning, task planning, motion planning, and self-reflection. Powered by LLMs, our Agent-Driver is endowed with intuitive common sense and robust reasoning capabilities, thus enabling a more nuanced, human-like approach to autonomous driving. We evaluate our system on both open-loop and close-loop driving challenges, and extensive experiments substantiate that our Agent-Driver significantly outperforms the state-of-the-art driving methods by a large margin. Our approach also demonstrates superior interpretability and few-shot learning ability to these methods.

Published as a conference paper at COLM 2024
A Language Agent for Autonomous Driving
Jiageng Mao Junjie Ye! Yuxi Qian! Marco Pavone
Yue Wang
'University of Southern California Stanffrd University NNVIDI {jiagengm, yejunjie, yuxiqian, yue. wtw.wweeweee.eeoo edu, pavone@stanford.ed
Abstract
Human-level driving ultimate goal of autonomous driving. Con- ventional approaches formulate autonomous driving as perception- prediction-planning framework, yet their systems do not capitalize on the inherent reasoning ability and experiential knowledge of humans. In this paper, we propose fundamental paradigm shift from current pipelines, exploiting Large Language Models (LLMs) as cognitive agent to inte- grate human-like intelligence into autonomous driving systems. Our sys- tem, termed Agent-Driven transforms the traditional autonomous driv- ing pipeline by introducing a versatile tool library accessible via function calls, a cognitive memory of common sense and experiential knowledge for makinng and a reasoning engine capable of chain-of-thought reasoning, task planning, motion planning, and self-reflection. Powered by LLMs, our agent-Drives is endowed with intuitive common sense and robust reasoning capabilities, thus enabling more nuanced, human-like ap- proach to autonomous driving. We evaluate our system on both open-loop and close-loop driving challenges, and extensive experiments substantiate that our Agent-Driver significantly outperforms the ofate-of-the-ar driving methods by large margin (more than 30% on the nuScenes dataset). Our approach also demonstrates superior interpretability and few-shot learning ability to these methods. Please visit our webpage for more details.
1 Introduction
Imagine car navigating quiet suburban neighborhood. Suddenly, ball bounces onto the road. A Inputs
Sunsury
Perception
Prodiction
Planning
Action
human driver, leveraging extensive experiential knowledge, would not
Peroetion-Prediction: wwee---rrtonn-aaaaiaeee Pipeline.
only perceive the immediate pres- ence of the ball, but also instinc- tively anticipate the possibility of
Tool Library
C Cognitive Memory
chasing child and consequently decide to decelerate. In contrast, an autonomous vehicle, devoid of such reasoning and experiential an- ticipation, might continue driving until sensors detect the child, only allowing for a narrower margin of safety. The importance of human prior knowledge in driving systems becomes clear: driving is not merely about reacting to the visible, but also to the conceivable scenarios where the system needs to reason and respond even in their absence.
Sensory
Inputs
Neural Modules
Agent-Driver
Reasoning Engine
Action
(b) A Agent-Driver:: LLL am Agent for utonomous Driving.
Figure 1: Comparison between (a) the conven- tional driving system and (b) the proposed Agent- Driver. Our approach transforms the conventional perception-prediicion-lllal
framework by intro- ducing LLMs an agent for autonomous driving.
To integrate human prior knowledge into autonomous driving systems, previous ap- proaches (Sadat et al., 2020; Casas et al., 2021; Hu et al., 2022; 2023; Jiang et al., 2023)
indicates equal contribution.

---

Published as a conference paper at COLM 2024
Cognitive
Text Interfice
Reesoning Engine
Conssencene Nenery
Experinnc Minuary
Neracy lata
Chule-ll-haaghh Beueedaa
(1) duakeripptt
sibofretireer
Tost Library
-ep Tirar Theeray
Hoae tolk luage learoighh etteatii peviee plar
Oncepency
Mapping
Menors & Tast Use
Remsoning & Plansing
Marice Planiigg
Neural Module
12 1339.73311
Saf-Btallction
Deluntas Medale
Prodictio Medal:
Ocopuner Medak
Magping Madde
Large Language Models an Agent
2211 BCBA 1.00....10000 (8396421))
Oiservations
Enstrenmeunal Updade
Actions
Figure 2: Architecture of Agent-Driver. Our system dynamically collects necessary environ- mental information from the output of neural modules via the tool library. The collected information is further utilized to query the cognitive memory. Consequently, the reasoning engine takes collected environmental information and retrieved memory data input, and traceably derives a safe and comfortable trajectory for driving through chain-of-thought reasoning, task planning, motion planning, and self-reflection.
deconstruct the human driving process into three systematic steps following Figure (a). Perception: they interpret the human perceptual process as object detection (Mao et al., 2023b) or occupancy estimation (Peng et al., 2020). Prediction: they abstract human drivers' foresight of upcoming scenarios as the prediction of future object motions (Casas et al., 2018). Planning: they emulate the human decision-making process by planning a collision-free trajectory, either using hand-crafted rules (Treiber et al., 2000) or by learning from data (Zeng et al., 2019). Despite its efficacy, this perception pprception-predcctiion-- peeed-ition-lllamiiiii fies the human decisionnmmkinn process and cannot fully model the complexity of driving. For instance, perception modules in these methods are notably redundant, necessitating the detection of all objects in a vast perception range, whereas human drivers can main- tain safety by only attending to a few key objects. Moreover, prediction and planning are designed for collision avoidance with detected objects. Nevertheless, they lack deeper rea- soning ability inherent to humans, e.g. deducing the connection between visible ball and a potentially unseen child. Furthermore, it remains challenging to incorporate long-term driving experiences and common sense into existing autonomous driving systems.
framework overly simpli-
In addressing these challenges, we found the major obstacle of integrating human priors into autonomous driving lies in the incompatibility of human knowledge and neural-network- based driving systems. Human knowledge is inherently encoded and utilized as language representations, and their reasoning process can also be interpreted by language. However, conventional driving systems rely on deep neural networks that are designed to process numerical data inputs, such sensory signals, bounding boxes, and trajectories. The discrepancy between language and numerical representations poses a significant challenge to incorporating human experiential knowledge and reasoning capability into existing driving systems, thereby widening the chasm from genuine human driving performance.
Taking step towards more human-like autonomous driving, we propose Agent-Driver, cognitive agent empowered by Large Language Models (LLMs). The fundamental insight of our approach lies in the utilization of natural language as a unified interface, seamlessly bridging language-based human knowledge and reasoning ability with neural-network- based systems. Our approach fundamentally transforms the conventional perception- prediction-planiing framework by leveraging LLMs an interactive scheduler among system components. As depicted in Figure 1 (b), on top of the LLMs, we introduce: 1) versatile tool library that interfaces with neural modules via dynamic function calls,
2

---

Published as a conference paper at COLM 2024
streamlining perception with less redundancy, 2) a configurable cognitive memory that explicitly stores common sense and driving experiences, infusing the system with human experiential knowledge, and 3) reasoning engine that processes perception results and memory data to emulate human-like decision-making Specifically, the reasoning engine performs chain-of-thougie to derive a high-level driving plan, motion planning to generate driving trajectory, and self-reflection ensure the safety of the planned trajectory. These components, coordinated by LLMs, culminate in an anthropomorphic driving process. To conclude, we summarize our contributions follows:
reasoning to recognize key objects and events, task planning
We present Agent-Driven an LLM-powered agent that revolutionizes the traditional penception-predictio---
framework, establishing a powerful yet flexible paradigm for human-like autonomous driving.
agent-Driver integrates tool library for dynamic perception and prediction, cognitive memory for human knowledge, and a reasoning engine that emulates hu- man decision-making, all orchestrated by LLMs to enable a more anthropomorphic autonomous driving process,
Agent-Drived significantly outperforms the state-of-the-art autonomous driving sys- tems by large margin, with over 30% collision improvements in motion planning. Our approach also demonstrates strong few-shot learning ability and interpretabil- ity on the nuScenes benchmark.
We provide ablation studies to dissect the proposed architecture and understand the efficacy of each module, to facilitate future research in this direction.
Related Works
Plannnnnn--aaniiinnniin in Driving Systems. Modern autonomous driving systems paradigm to make driving decisions based on sensory inputs. Perception modules aim to recognize and localize objects in driving scene, typically in a format of object detection (Mao et al., 2021b; Wang et al., 2022; Mao et al., 2021a; 2023b) or object occupancy prediction (Peng et al., 2020; Tong et al., 2023). Prediction modules aim to estimate the future motions of objects, normally represented as predicted trajectories (Casas et al., 2018; Ivanovic & Pavone, 2019; Shi et al., 2022) or occupancy flows (Agro et al., 2023; Casas et al., 2021). Planning modules aim to derive safe and comfortable trajectory, using rules (Urmson et al., 2008; Fan et al., 2018; Sauer al., 2018; Chen et al., 2015; Bacha et al., 2008; Leonard et al., 2008; Thrun et al., 2006; Treiber al., 2000) or learning from human driving trajectories (Dauner et al., 2023; Chitta al., 2022; Mao et al., 2023a). These three modules are generally performed sequentially, either trained separately or in an end-to-end manner (Liang et al., 2020; Sadat et al., 2020; Casas et al., 2021; Hu et al., 2022; 2023). This prrrepttoonpreeiccion--llee simplifies the human driving process and cannot effectively incorporate human priors such as common sense and past driving experiences. By contrast, our Agent-Driver transforms the conventional perception -ereettiiioeeedtiiooo..
rely on Preptttion-eeiiiiioonnn
framework overly
framework by introducing LLMs as an agent bring human-like intelligence into the autonomous driving system.
LLMs in Autonomous Driving. Trained on Internet-scale data, LLMs (Brown et al., 2020; APenAI, 2023; Touvron et al., 2023a;b) have demonstrated remarkable capabilities in com- monsense reasoning and natural language understanding. How to leverage the power of LLMs to tackle the problem of autonomous driving remains an open challenge. GPT- Driver (Mao et al., 2023a) handled the planning problem in autonomous driving by reformu- lating motion planning as language modeling problem and introducing fine-tuned LLMs as a motion planner. DriveGPT4 (Xu et al., 2023) proposed an end-to-end driving approach that that leverages Vision-Language Models to directly map sensory inputs to actions. DiLu (Wen et al., 2023) introduced a knowledge-driver approach with Large Language Models. These methods mainly focus on an individual component in conventional driving systems, e.g. muestion-answeriin (Xu et al., 2023), planning (Mao et al., 2023a), or control (Sha et al., 2023). Some approaches (Fu et al., 2023; Wen et al., 2023) are implemented and evaluated in simple simulated driving environments. By contrast, Agent-Driver presents systematic approach
3

---

Published as a conference paper at COLM 2024
that leverages LLMs an agent to schedule the whole driving system, leading to strong performance on the real-world driving benchmark.
3 Agent-Driver
In this section, we present Agent-Driver, an LLM-based intelligent agent for autonomous driving. We first introduce the overall architecture of our Agent-Driver in Section 3.1. Then, we introduce the three key components of our method: tool library (Section 3.2), cognitive memory (Section 3.3), and reasoning engine (Section 3.4).
3.1 Overall Architecture
pipelines leverage a series of neural networks as basic modules for different tasks. However, these neural-network- systems lack direct compatibiliity with human prior knowledge, constraining their potential for leverag- ing such priors to enhance driving performance. To handle this challenge, we propose novel framework that leverages text representations as a unified interface to connect neural networks and human knowledge. The overall architecture of Agent- Deret-- is shown in Figure 2. Our approach takes sensory data as input and introduces neural modules for processing these sensory data and extracting environmental information about detection, prediction, occupancy, and map. On top of the neural modules, we propose a tool library where set of functions are designed to further abstract the neural outputs and return text-based messages. For each driving scenario, an LLM selectively activates the required neural modules by invoking specific functions from the tool library, ensuring the collec- tion of necessary environmental information with less redundancy. Upon gathering the necessary environmental information, the LLM leverages this data as a query to search in a cognitive memory for pertinent traffic regulations and the most similar past driving experience. Finally, the retrieved traffic rules and driving experience, together with the formerly collected environmental information, are utilized as inputs to an LLM-based rea- soning engine. The reasoning engine performs multi-round reasoning based on the inputs and eventually devises safe and comfortable trajectory for driving. Our Agent-Driver architecture harnesses dynamic perception and prediction capability brought by the tool library, human knowledge from the cognitive memory, and the strong decision-makes ability of the reasoning engine. This synergistic integration results a more human-like driving system with enhanced decision-making capability.
Conventional
3.2 Tool Library
The profound challenge of incorporating human knowledge into neural-nettork-basee driving systems is reconciling the incompatibility between text-based human priors and the numerical representations from neural networks. While prior works (Kuo et al., 2022) have attempted to translate text-based priors into semantic features or regularization terms for integration with neural modules, their performances are still constrained by the inherent cross-modal discrepancy. By contrast, we leverage text as unified interface to connect neural modules and propose tool library built upon the neural modules to dynamically collect text-based environmental information.
The cornerstones of the tool library are four neural modules, i.e., detection, prediction, occupancy, and map modules, which process sensory data from observations and generate detected bounding boxes, future trajectories, occupancy grids, and maps respectively. The neural modules cover various tasks in perception and prediction and extract environmental information from observations. However, this information is largely redundant, with significant portion insignificant to mecision-making To dynamically extract necessary information from the neural module outputs, we propose tool library- a set of functions designed to summarize the neural outputs into text-based messages, and the information collection process can be established by dynamic function calls. An illustration of this process is shown in Figure 3.
Functions. We devised various functions for detection, prediction, occupancy, and mapping, in order to extract useful information from the neural module's outputs respectively. Our
4

---

Published as a conference paper at COLM 2024
tool library contains more than 20 functions covering diverse usages. Here are some exam- returns a text description of the object in front of the ego-vehicle the same lane. For prediction, get pred trajs for object returns a text- based predicted future trajectory for a specified object. For occupancy, get occ at loc time returns the probability that specific location is occupied by other objects at a given timestep. For map, get_ genelanes returns the information of the left and right lanes to the ego-vehicle, and ret_shoulders returns the information of the left and right road shoulders to the ego-vehicle. Detailed descriptions of all functions are in Appendix A.1.
ples. For detection, get_leading.objer
Tool Use. With the functions in the tool library, an LLM is instructed to collect necessary environmental information through dynamic function calls. Specif- ically, the L.M is first provided with initial information such the current state for its subsequent decision-making Then, the LLM will be asked whether it is necessary to activate a specific neural module, f.edetection, prediction, occu- pancy, and map. If the LLM decides to activate a neural module, the functions related to this module will be provided to the LLM, and the LLM chooses to call one or some of these functions to col- lect the desired information. Through multiple rounds of conversations, the LLM eventually collects all necessary information about the current environ- ment. Compared to directly utilizing the outputs of the neural modules, our approach reduces the redundancy in current systems by leveraging the rea- soning power of the LLM to determine what environmental information is of real importance to the decision-making process. Furthermore, the neural mod- ules are only activated when the LLM decides to call the relevant functions, which brings flexibility to the system. A detailed example of LLM leveraging tool in Appendix A.2.
Agent
System Prompix
The tou bbuddt ope tnteraatio (Denutioes traditiona Mass
YES
ouscan nor the Bilaytad tanciaos object de Farcia cutara the lnddng otbuu
Cal
pit www.eeeeeett.oooooooooooooo datcckine partarann)
ao tie Buadion
Troir disc Aeteckie
Detection Mochite
popocceed to couoe tnjurary and otoe Thase ancver
con were frrum tralecteryl
tndedarin specia chacte parranire Edlpect [2 2.00
Rigustty by alpectd Ein
De meed bopee pancy tomrrrdee?
Do jou seddt get rap inbrraccal Faaue saoeer cr
Bram cos Hirry...
Cuatre www.eeeeeeeiiccc. wotorbiticbodebe........ therighchhede
Figure 3; Illustration of function calls in the tool library. Agent-Driver can effectively collect nec- essary environmental information from neural modules through dynamic function calls.
functions for environment perceiving is shown
3.3 Cognitive Memory
Human drivers navigate using their common sense, such as adherence to local traffic regulations, and draw upon driving experiences in similar situations. However, it is non- trivial to adapt this ability to the conventional By contrast, our approach tackles this problem through interactions with a cognitive memory. Specifically, the cognitive memory stores text-based common sense and driving experiences. For every scenario, we utilize the collected environmental information the cognitive memory for similar past experiences to assist decision-making The cognitive memory contains two sub-memories: commonsense memory and experience memory.
framework.
query to search in
The commonsense memory encapsulates the essential knowledge a driver typically needs for driving safely on the road, such as traffic regulations and knowledge about risky behaviors. is worth noting that the commonsense memory is purely text-based and fully configurable, that is, users can customize their own commonsense memory for different driving conditions by simply writing different types of knowledge into the memory.
Commonsense Memory.
5

---

Published as a conference paper at COLM 2024
Experience Memory, The experience memory contains series of past driving scenarios, where each scenario is composed of the environmental information and the subsequent driving decision at that time. By retrieving the most similar experiences and referencing their driving decisions, our system enhances its capacity for making more informed and resilient driving decisions.
Memory Search. As exhibited in Figure 4, we present an innovative two-stage search algorithm to effectively search for the most similar past driving scenario in the expe- rience memory. The first stage of our algo- rithm is inspired by vector databases (Lewis et al., 2020; Wang et al., 2021), where we encode the input query and each record in the memory into embeddings and then retrieve the top-K similar records via K- nearest neighbors (K-NN) search in the em- bedding space. Since the driving scenar- ios are quite diverse, the embedding-based search is inherently limited by the encod- ing methods employed, resulting in insuf- ficient generalization capabilities. To over- come this challenge, the second stage incor- porates an LLM--baeed fuzzy search, where the LLM is tasked to rank these records ac- cording to their relevance to the query. This ranking is based on the implicit similarity assessment by the LLM, leveraging its capabilities in generalization and reasoning.
E Quary
Cagalthe Moneey
Ersinomental
K-NN Search
Phan Scunario to [3)-
bul Prodicion -rrsppncy
inveramentaa lido E3
Thujecsyy 12)
Paas Scanar to (N) Envernncital libl (N) Divving Trijemory (N)
Excode
Eie Query Scotaris
Pael Sceaarta ())
Paur Scoarra ())
(oveariny
Stage-1: Vector Search
ylu surrent
the sitile patthiess
desase makinn
Rarress
(Jurry Sccearis...
Past Scouuna [Il...
Sexrarte
Part Semard
Themss
LLM Based Fuzzy Seurch
Figure 4: Illustration of memory search. The proposed two-stage search algorithm effec- tively retrieves the most similar driving expe- rience and facilitates the subsequent decision- making process.
The cognitive memory equips our system with human knowledge and past driving experi- ences. The retrieved most similar experiences, together with commonsense and environ- mental information, collectively form the inputs to the reasoning engine. Text unified interface aligns the environmental information with human knowledge, thereby enhancing our system's 's compatibility. Please refer to Appendix B for more details.
3.4 Reasoning Engine
Reasoning, a fundamental ability of humans, is critical to the decision-making process. Conventional methods directly plan driving trajectory based on perception and predic- tion results, while they lack the reasoning ability inherent to human drivers, resulting in insufficient capability to handle complicated driving scenarios. Conversely, shown in Figure 5, we propose a reasoning engine that effectively incorporates reasoning ability into the driving decision-making process. Given the environmental information and retrieved memory, our reasoning engine performs multi-round reasoning to plan safe and comfort- able driving trajectory. The proposed reasoning engine consists of four core components: chain-of-thought reasoning, task planning, motion planning, and self-reflection.
Chain-of-Thought Reasoning. Human drivers are able identify the key objects and their potential effects on driving decisions, while this important capability is typically absent in conventional autonomous driving approaches. To embrace this reasoning ability in our system, we propose novel chain-of-thought reasoning module, where we instruct an LLM to reason on the input environmental information and output a list of key objects and their potential effects in text. To guide this reasoning process, we instruct the LLM via in-context learning of a few human-annotated examples. We found this strategy succesffully aligns the reasoning power of the LLM with the context of autonomous driving, leading to improved reasoning accuracy.
Task Planning. High-level driving plans provide essential guidance to low-level mo- tion planning. Nevertheless, traditional methods directly perform motion planning with- out relying this high-level guidance, leading to sub-optimal planning results. In our approach, we define high-level driving plans as a combination of discrete driving be- haviors and velocity estimations. For instance, the combination of driving behavior

---

Published as a conference paper at COLM 2024
laanege.aaneto.oleee and velocity estimation deceleration results in a high-level driving plan lagee....lt th.deceleration We instruct an LLM via in-context learning to devise high-level driving plan based on environmental information, memory data, and mhain-of-thoughe reasoning results. The devised high-level driving plan characterizes the ego-vehicle's s coarse locomotion and serves strong prior to guide the subsequent motion planning process.
Motion Planning. Motion planning aims to devise a safe and comfortable trajectory
""Teak-dsccippiicc" descriptionne*
You're austonns vehide- beats. Plan Loueearry avail chraalo...
"Eovivaannntal
for driving, and each trajectory repre-
deutt Jeteciato Putare inseciarics for spealla vigettor.. madiiee......
"Coumen sessert
sented as sequence of waypoints. Fol-
Masisas sal doasnse fium
of
reforence**
lowing (Mao et al., 2023a), we re-formulate motion planning as a language modeling problem. Specifically, we leverage environ- mental information, memory data, reason- ing results, and high-level driving plans col- lectively inputs to an LLM, and we in- struct the LLM to generate text-based driv- ing trajectories by reasoning on the inputs. By fine-tuning with human driving trajecto- ries, the LLM can generate trajectories that closely emulate human driving patterns. Fi- nally, we transform the text-based trajecto- ries back into real trajectories for system execution.
Sccurrio informaion: Reforescal plansingroulte...
Caati
reasoning-
Netablo obgats polist (082.1851 mowing (35520090 at aemoed. Petezttal clecs tne cdlidd continue devrag Mhropaok Netablk obictt
"Taak plansing
8dsenue nave fawwad Spead dockeatiin Driwigg platz more keooed deccleration
Visrol Demumetration
"Notion planning
cillainnn change
Figure 5: Illustration of reasoning engine. Agent-Driver makes driving decisions like hu- man in a step-by-step procedure.
Self-Reflection. Self-reflection is a crucial ability in humans' macision-making process, aim- ing to re-assess the former decisions and adjust them accordingly. To model this capability in our system, we propose a collision check and optimization approach. Specifically, for planned trajectory from the motion planning module, the collision_chf _cccceccckkkkkc function in the tool library is first invoked to check its collision. If collision detected, we refine the trajectory into a new trajectory I* by optimizing the cost function C:
T min CT, f) min A1//T f|l2 A2F (T).
(1)
In this manner, the safety of the planned trajectory is greatly improved.
Our reasoning engine models the human decision-making process in driving as step-by- step procedure involving reasoning, hierarchical planning, and self-reflection. Compared to prior works, our approach effectively emulates the human decision- process, leading to enhanced decision-making capability and superior planning performance. More details of the reasoning engine can be found in Appendix C.
4 Experiments
In this section, we demonstrate the effectiveness, few-shot learning ability, and other char- acteristics of Agent-Driver through extensive experiments. First, we introduce the experi- mental settings in Section 4.1. Next, we evaluate the planning performance of our approach on both open-loop and closed-loop settings (Section 4.2). Subsequently, we investigate the few-shot learning ability (Section 4.3), interpretability (Section 4.4), compatibility (Section 4.5), and stability (Section 4.6) of Agent-Driver. Finally, we discuss the choices of in-context learning and fine-tuning in Section 4.7. More experiments can be found in Appendix D.
4.1 Experimental Setup
Benchmarks. For open-loop autonomous driving, we conduct experiments on the large- scale nuScenes dataset (Caesar et al., 2020). The nuScenes dataset is a real-world autonomous driving dataset that contains 1, 000 driving scenarios and approximately 34, 000 key frames encompassing diverse range of locations and weather conditions. We follow the general practice and split the whole dataset into training and validation sets. We utilize the training set to train the neural modules and instruct the L.LMs, and we utilize the validation set to
7

---

Published as a conference paper at COLM 2024
I.2 (m)
4
Collision (%)
Method
1s
2s
3s
Avg.
1s
2s
3s
Avg.
ST-P3 (Hu et al., 2022)
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0171
VAD (Jiang et al., 2023)
0.17
0.34
0.60
0.37
0.07
0.10
0.24
0.14
ST-P3 metrics
GPI-Driver (Mao al., 2023a)
0.20
0.40
0.70
0.44
0.04
0.12
0.36
0.17
Agent-Driver (ours)
0.16
0.34
0.61
0.37
0.02
0.07
0.18
0.09
NMP (Zeng et al., 2019)
2.31
1.92
SA-NMP (Zeng et al., 2019)
2.05
1.59
FF (Hu et al., 2021)
0.55
1.20
2.54
1.43
0.06
0.17
1.07
01.4
UniAD metrics
EO (Khurana et al., 2022)
0.67
1.36
2.78
1.60
0.04
0.09
0.88
0.33
UniAD (Huu et al., 2023)
0.48
0.96
1.65
1.03
0.05
0.17
0.71
0.31
GPT-Driver (Mao et al., 2023a)
0.27
0.74
1.52
0.84
0.07
0.15
1.10
0.44
Agent-Driver (ours)
0.22
0.65
1.34
0.74
0.02
0.13
0.48
0.21
Table 2: Open-loop planning performance compared to the state-of-theearts. Agent-Driver significantly outperforms prior works in terms of L2 and collision rate. Our approach attains more than 30% performance gains in collisions compared to the state-of-the-art methods.
evaluate the performance of our approach, ensuring a fair comparison with prior works. Following prior works (Hu et al., 2022; 2023; Jiang al., 2023), the L2 error and collision rate are reported to evaluate the planning performance. For the closed-loop autonomous driving, we adopt the Town05-Short benchmark (Prakash et al., 2021) powered by the CARLA simulator (Dosovitskiy et al., 2017) for evaluation. The Town05-Short benchmark consists of 10 challenging driving routes with 3 intersections each, including a high density of dynamic agents. We adopt the widely-used route completion and driving scores to evaluate the planning performance. Route Completion denotes the progress of driving on route, and the Driving Score additionally takes comfort and safety into calculation. More details of the open-loop and closed-loop evaluation metrics can be found in Appendix D.1.
Implementation Details. We utilize 1rt-3.5-turbo-0613 as the foundation LLM for different components in our system. For motion planning, we follow (Mao et al., 2023a) and fine-tune the LLM with human driving trajectories in the nuScenes training set for one epoch. For neural modules, we adopted the modules in (Hu et al., 2023). For closed-loop experiments, we leveraged the perception modules in LAV (Chen Krähenbühl, 2022) and kept the other parts of our system the same. We also followed the same training setting and evaluation protocols (Chen & Krähenbühl, 2022) for a fair comparison. More implementation details can be found in the appendix D.2.
4.2 Comparison with State-of-the-art Methods
Open-Loop Results. As shown in Table 2, Agent-Driver surpasses state-of-the-art methods in both metrics and decreases the collision rate of the second-best performance by large margin. Specifically, under ST-P3 metrics, Agent-Driver realizes the lowest average L2 error and greatly reduces the average collision rates by 35.7% compared to the second- best performance. Under UniAD metrics, Agent-Drived achieves an I.2 error of 0.74 and a collision rate of 0.21%, which are 11.9% and 32.3% better than the second-bee methods GPT-Driver (Mao et al., 2023a) and UniAD (Hu al., 2023), respectively. The promising performance on the collision rate verifies the effectiveness of the reasoning ability of Agent- Driver, which considerably increases the safety of the proposed autonomous driving system.
Closed-Loop Results. To analyze the per- formance of our approach in closed-loop settings, we evaluate Agent-Driver against other state-of-the-an methods on the au- thoritative CARLA simulator. The results on the Town05-Short benchmark (Chitta et al., 2022) are shown in Table 1. Agent- Driver achieves the highest route comple- tion, surpassing the second-best VAD (Jiang et al., 2023) by 4.1%. In terms of driving score, Agent-Driver also yields performance of 57.33%, which is par with the prior arts.
Methods
Driving Score
Route Completion 13.0l
CILRSTCodese et 2019]
7.47
1BC (Cui et 2022)
30.97
55.01
Transluser (Prakash et al,, 2021)
54.52
76.41
ST-P3 (Hu etal. 2022)
55.14
86 74
VAD (fang at. 2025)
64.29
87
Agent Driver (Uus)) (Ours)
57.33
91.77
Table 1: Closed-loop planning performance compared to the state-of-the-arts. Agent- Driver yields the best route completion and an on-par driving score compared to prior arts
8

---

Published as a conference paper at COLM 2024
geeennnnnrt tndermations-
Enrieneenntaa lnfor mastion
breat olpert dern styet pedestris, olyeet pocttion (+4 taag dian (6 913
Prontoieet deteted, dewetedddyyeeerree dyyentape cejeet 18 posttios (18.1. 16993 dhe 674)
jwoatiss stac 3.09)
object posttion
Pazare for speciia chiectc
Futtre injectaria for upedilcc cbedar
rngerig ryple gedeet aijert kinure wagg der courde to [1-4-4, 14.50r..
mijert tye pedestiiss, olpest it fattre sucpeier moor tn sij
oogectypp: podotziaa exject
coundi [L17%, 200ha) -.
Orject pedestrias obgect fnture exeedaaiten 11522, 32,710 Mau talerusken (lasa))
Sap informatie aien (toalattll
ega
right cloolder
rightiion
CeenCamemnn seese
seeerpoonno oense
you.....
eeer dehttigg
refercnac
esseppat desstag experience refurance:
Stas autlie dthing axpericnes fran muesay with stmullarig acorcd 2.77
Most saudl-a fut experiens frure awinary with strtlarrt more 0.42:
The plaxaed
nomarta reformnce (10042449) 20.12220
phamed injuctery this nastari for entenneee (22000)) -.. 65100.2777
ceanoslag
Nutillle snjeets podestrinn anaa (6.00, 1011) mooinn ti su200ssjur seroid
lolilleeleee car (1.11 (. mowingos 1- ma Stj seond
Prteonal offecto eiliste contiee srining this ipeed
Pooeettll effers indide the sifeyy tone egoredddde 1.5
eeerraak plansing:
Taskraaa plansing wee.e
sastenn birw speod deveiration, trining plan nowe donwor deviinattion
induaaioon nure lett Sy develoaatinn wing pia tum left develecation
plansing
"hepcsors d.0s, 2,07 (-2.00 4.84) 64.19 7.00, 1-22% 2.25) 12.2% (-049) eveesaff rellectonst
Trajecion:: (1:8.12.3 (2.099) (4.32 a01 (-3 (-2422
(1.4400)) (220558.29))
Nechung nothereatio y plaviing ressing reckl.
Nochange
plansing yoadt
Figure 7: Interpretability of Agent-Driver. In the referenced images, planned trajectories of our system and human driving trajectories are in red and green respectively. Agent-Drive extracts meaningful objects (in yellow) from all detected objects (in blue) via the tool library. The reasoning engine further identifies notable objects (in red). Messages from the tool library, cognitive memory, and reasoning engine are recorded in colored text boxes. Every message is documented and our system is conducted in an interpretable and traceable way.
4.3 Few-shot Learning
To assess the generalization ability of mo- tion planning in our approach, we conduct few-shot learning experiment, where we keep other components the same and fine- tuned the core motion planning LLM with 0.1%, 1%, 10,, 50% and 100% of the train- ing data for one epoch. For comparison, we adopted the motion planner in UniAD (Hu et al., 2023) trained with 100% data as the baseline. The results are shown in Figure 6. Notably, with only 0.1% of the full training data, i.e, 23 training samples, Agent-Driver realizes a promising performance. When exposed to 1% of training scenarios, the pro- posed method surpasses the baseline by large margin, especially under the average collision rate. Furthermore, with increased training data, Agent-Driver stably achieves better motion planning performance.
0.40
1.20
Avg. 1.2 (m)
Avg. Collision
Baseline trained with 100% data
1.03
0.31
0.95
28 2%
3,20 300
0.80
0.24
0.10
0.26
0.20
0.1%
1%
10%
50%
100%
Percentage of training samples
Figure 6: Few-shot learning. The motion plan- ner in Agent- fine-tuned with 1% data exceeds the state-of-the-art (Hu et al., 2023) trained on full data, verifying its few-shot learning ability.
4.4 Interpretability
Unlike conventional driving systems that rely on black-box neural networks to perform different tasks, the proposed Agent-Driver inherits favorable interpretability from LLMs. As shown in Figure 7, the output messages of LLMs from the tool library, cognitive memory, and reasoning engine are recorded during system execution. Hence the whole driving decision-making process is transparent and interpretable.

---

Published as a conference paper at COLM 2024
4.5 Compatibility with Different LLMs
We tried leveraging the Llama-2-7B (Tou- vron et al., 2023b), 3pt-3.5-turbo-1166 and 3pt-3.5-turbo-0633 models as the founda- tion LLMs in our system. Table demon- strates that Agent-Driver powered by dif- ferent LLMs can yield satisfactory perfor- mances, verifying the compatibility of our system with diverse LLM architectures.
(m)
Collision
Aug.
Avg.
0.78
0.35
8pb-3.5 tarbo-1106 8p--3--turbo-0113
000
0.22
0.74
002
0.48
Table 3: Compatibility to different LLMs.
Agent-Driver realizes satisfactory motion planning performance utilizing different types of LLMs agents.
4.6 Stability
LLMs typically suffer from arbitrary predictions- they might produce invalid outputs (eg., hallucination or invalid formats)- which is detrimental to driving systems. To investigate this effect, we con- ducted stability test of our Agent-Driver. Specifically, we used different amounts of training data to instruct the LLMs in system, and we tested the number of invalid outputs during inference on the validation set, where we define "invalid" as waypoint outputs containing non-numerical values. As shown in Table Agent-Driver exposed to only 1% of the training data sees zero invalid output during inference of 6,019 validation scenarios, suggesting that our system attains high output stability with proper instructions.
Percentage of training samples 0.300%
1%
10%
50%
100%
Number of invalid outputs
Table 4: Stability of Agent-Driver exposed to different amounts of training samples. With only 1% training samples (~~ 230 samples), proriuces
Agent-Driver produces zero invalid output.
4.7 In-Context Learning vs. Fine-Tuning
Two prevalent strategies instruct an LLM for novel tasks are in-context learning and fine-tuning. To determine which is the most effective strategy, we apply these two strate- gies to the LLMs of the chain-of-thougit reasoning, task planning, and motion plan- ning modules respectively, benchmarking them on the downstream motion planning performance. As indicated in Table 5, in- context learning performs slightly better than fine-tuning in collision rates for reasoning and task planning, suggesting that in-context learning is a favorable choice in these modules. In motion planning, the fine-tuning strategy significantly outperforms in-context learning, demonstrating the necessity of fine-tuning LLMs in motion planning.
Resson, Plan.
Modules
Plan.
Avg. 1.2 (m)
Avg. Col. (%)
funing
contect learning
In-oontiost learning
coeeeett learning
Fine-tuning
Fine-tuning
learning
Fine-turingg
Table 5: In-context learning vs. fine-tuning. In-context learning performs slightly better in reasoning and task planning. Fine-tuning is indispensable for motion planning.
5 Conclusion
This work introduces Agent-Driver, novel human-like paradigm that fundamentally transforms autonomous driving pipelines. Our key insight is leverage LLMs as an agent to schedule different modules in autonomous driving, On top of the LLMs, we propose tool library, a cognitive memory, and a reasoning engine bring human-like intelligence into driving systems. Extensive experiments on the real-world driving dataset substantiate the effectiveness, few-shot learning ability, and interpretability of Agent-Driver These findings shed light on the potential of LLMs as an agent in human-level intelligent driving systems. For future works, we plan to optimize the LLMs for real-time inference.
10

---

Published as a conference paper at COLM 2024
Ethics Statement
This paper presents Agent-Driver, a novel method that leverages large language models as an agent for autonomous driving. While Agent-Driver uses language generation as part of its reasoning and planning process, the goal is to improve the quality of representations and decision-making for autonomous vehicles, not text generation itself. Therefore, the potential negative impact of Agent-Driver on areas like misinformation or deepfakes is minimal. On the privacy protection side, Eggent-Drives uses driving data that does not include any personal or location information. In summary, we believe Sgent-Driver does not lead to any ethics concerns.
Reproduciblity Statement
We provide as many implementation details as possible in the paper submission and in the appendix. We also include the code for Agent-Driver, including the large language model, neural modules, tool library, cognitive memory, and reasoning engine components in the supplementary material. This will allow others to faithfully reproduce our results and build upon our proposed approach.
Acknowledgments
We'd like to acknowledge Xinshuo Weng for fruitful discussions. We also acknowledge gift from Google Research.
References
Ben Agro, Quinlan Sykora, Sergio Casas, and Raquel Urtasun. Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving, In CVPR, 2023.
Andrew Bacha, Cheryl Bauman, Ruel Faruque, Michael Fleming, Chris Terwelp, Charles Reinholtz, Dennis Hong, Al Wicks, Thomas Alberi, David Anderson, et al. Odin: Team VictorTango's Entry in the DARPA Urban Challenge. JFR, 25(8), 2008
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. NeurlPS, volume 33, 2020.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. In CVPR, 2020.
Sergio Casas, Wenjie Luo, and Raquel Urtasun. IntentNet: Learning to Predict Intention from Raw Sensor Data. In CoRL, 2018.
Sergio Casas, Abbas Sadat, and Raquel Urtasun. MP3: A Unified Model to Map, Perceive, Predict and Plan. CVPR, 2021.
Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving. In ICCV, 2015.
Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17222-17231, 2022.
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. TransFuser: Imitation With Transformer-Basee Sensor Fusion for Autonomous Driving. JEEE TPAMI, 2022.
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In ICCV, 2019.
11

---

Published as a conference paper at COLM 2024
Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, and Raquel Urtasun. LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving. In ICCV, 2021.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention Fast and Meeorry-ffficient Exact Attention with I0-Awareness. NeurIPS, 35, 2022.
Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with Misconceptions about Learning-based Vehicle Motion Planning. In CoRL, 2023.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An Open Urban Driving Simulator. In CoRL, volume 78, 2017.
Haoyang Fan, Fan Zhu, Changchun Liu, Liangliang Zhang, Li Zhuang, Dong Li, Weicheng Zhu, Jiangtao Hu, Hongye Li, and Qi Kong. Baidu Apollo EM Motion Planner. arXiv preprint ar 180: 007.08848 2018.
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive Like Human: Rethinking Autonomous Driving with Large Language Models. arXio preprint ar xxxe:2307.071662 2023.
Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe Local Motion Planning with Self-Supervised Freespace Forecasting. In CVPR, 2021.
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. ST-P3: End-to-End Aision-Based Autonomous Driving via Opetial-Temporal Feature Learning. In ECCV, 2022.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-orienti autonomous driving. In CVPR, 2023.
Boris Ivanovic and Marco Pavone. The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs. In ICCV, 2019.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. VAD: Vectorized Scene Representation for Efficient Autonomous Driving. In ICCV, 2023.
Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu. Adapt: Action-aware driving caption transformer. In 2023 IÊEE International Conference on Robotics and Automation (ICRA), pp. 7554-7561. IEEE, 2023.
Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differentiable Raycasting for Self-supervised Occupancy Forecasting, In ECCV, 2022.
Yen-Ling Kuo, Xin Huang, Andrei Barbu, Stephen G. McGill, Boris Katz, John J. Leonard, and Guy Rosman. Trajectory Prediction with Linguistic Representations. ICRA, 2022.
John Leonard, Jonathan How, Seth Teller, Mitch Berger, Stefan Campbell, Gaston Fiore, Luke Fletcher, Emilio Frazzoli, Albert Huang, Sertac Karaman, et al. Perreption-Driven Autonomous Urban Vehicle. JFR, 25(10), 2008
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- Augmented Generation for Knowledge- NLP Tasks. NeurlPS, 33, 2020.
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? arxiv preprint arXiv:2312.03031, 2023.
Ming Liang, Bin Yang, Wenyuan Zeng, Yun Chen, Rui Hu, Sergio Casas, and Raquel Urtasun. PnPNet: End-to-BB ton Perception and Prediction With Tracking in the Loop. In CVPR, 2020.
12

---

Published as a conference paper at COLM 2024
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. In ICML, 2023.
Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu. Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection. In ICCV, 2021a.
Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel Transformer for 3D Object Detection. In ICCV, 2021b.
Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang- GPT-Driver: Learning to Drive with GPT. arviv preprint arXiv:2310.01411 2023a.
Jiageng Mao, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. 3D Object Detection for Autonomous Driving: A Comprehensive Survey. 1JCV, 2023b.
OpenAl. GPT-4 Technical Report. arXio preprint arXio:2303.08774, 2023.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional Occupancy Networks. In ECCV, 2020.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. In CVPR, 2021.
Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urta- sun. Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations. In ECCV, 2020.
Axel Sauer, Nikolay Savinov, and Andreas Geiger. Conditional Affordance Learning for Driving in Urban Environments. In CoRL, 2018.
Hao Sha, Yao Mu, Yuxuan Jiang, Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving. arXiv preprint ar XrXi::2310.30066, 2023.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion Transformer with Global Intention Localization and Local Movement Refinement. NeurlPS, 35, 2022.
Sebastian Thrun, Mike Montemerlo, Hendrik Dahlkamp, David Stavens, Andrei Aron, James Diebel, Philip Fong, John Gale, Morgan Halpenny,, Gabriel Hoffmann, al. Stanley: The Robot that Won the DARPA Grand Challenge. JFR, 23(9), 2006.
Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as Occupancy. In ICCV, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arviv preprint arXiv:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. Xiv preprint XrXi::2307 0092888, 2023b.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested Traffic States in Empirical Observations and Microscopic Simulations. Physical Review E, 62(2), 2000.
Chris Urmson, Joshua Anhalt, Drew Bagnell, Christopher Baker, Robert Bittner, MN Clark, John Dolan, Dave Duggins, Tugrul Galatali, Chris Geyer, et al. Autonomous Driving in Urban Environments: Boss and the Urban Challenge. /FR, 25(8), 2008.
13

---

Published as a conference paper at COLM 2024
Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al. Milvus: A Purpose-Buile Vector Data Management System. In SIGMOD, 2021.
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries. In CoRL, 2022.
Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. DiLu: A Knowledge-Driver Approach to Autonomous Driving with Large Language Models. arXiv preprint ar Xrxi:22309.16: 16292, 2023.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient Stream- ing Language Models with Attention Sinks. Xrviv preprint Xiv:2309.17453, 2023.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth KY Wong, Zhenguo Li, and Hengshuang Zhao. DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model. arXive preprint arXtv:2310.01412, 2023.
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-To-End Interpretable Neural Motion Planner. in CVPR, 2019.
Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes. ar Xiv preprint ar XrX:::2055.04330 2023.
14

---

Published as a conference paper at COLM 2024
Appendix
A Tool Library
16
A.1 Functions
16
A.2 Tool Use
16
Cognitive Memory
16
B.1 Memory Data
16
B.2 Memory Search
16
C Reasoning Engine
17
Chain-of-Thought Reasoning
17
C.2 Task Planning
17
C33 Motion Planning
18
C44 Self-Reflection
18
C.5 Comparison with GPT-Driver
19
D Experiments
19
D.1 Evaluation Metrics
19
D.1.1 Open-Loop Metrics
19
D.1.2 Closed-Loop Metrics
20
D.2
Implementation Details
20
D33 Ablation Study
D4 Impact of Ego-States on Open-Loop Motion Planning
21
D.5 Compatibility with Different Neural Modules.
D.6
Language Justification on the BDD-X Dataset
22
Qualitative Analysis
23
E.1
Qualitative Ablation
23
E.2
Qualitative Results
24
E.3
Failure Cases
24
Limitations
24
15

---

Published as a conference paper at COLM 2024
A Tool Library
In this section, we will first introduce the detailed descriptions of all functions in the tool library (Section A.1). Next, we will provide a detailed example of how the agent interacts with the tool library (Section A.2).
A.1 Functions
We include all function definitions in the tool library in Tables 6 and 7. The proposed functions cover detection, prediction, occupancy, and mapping, and enable flexible and versatile environmental information collection.
A.2 Tool Use
detailed example of how the LLM leverages tools to collect environmental information is shown in Figures 4 and 5. System prompts shown in blue, the response of LLMs is shown in green, and the collected data is shown in orange. System prompts provide sufficient context and guidance for instructing the LLM dynamically invoke the functions in the tool library to collect necessary environmental information.
B Cognitive Memory
In this section, we detail the data format and retrieving process of the commonsense and experience memory.
B.1 Memory Data
As shown in Figure of the main text, the commonsense memory consists of essential knowledge for safe driving, which is cached in a text-based format and is fully configurable.
We build the experience memory by caching the environmental information of driving scenarios and corresponding driving trajectories in the training set. Please note that this experience memory can be editable online, meaning that expert demonstrations conducted by human drivers can be easily inserted into the memory and benefit the subsequent decision-making of Agent-Driver.
B.2 Memory Search
We propose two-stage searching strategy for retrieving the most similar past driving scenario to the query scenario.
In the first stage, we generate vectorized key k; [I (ex((netnn" (ng for each past scenario i by vectorizing its ego-states e; E Rl mission goals gi E R1xx,, , and historical trajectories h; E RIx%, The N past scenarios in the experience memory collectively construct key tensor K IR Nx(e
K = (i. (1,2,..,N)..
(1)
Similarly, we can vectorize the query scenario into Q (e, hi] E ine
Subsequently, we compute the similarity scores 5 E IR between the querying scenario Q and the past scenarios K:
= QAKT, where A = diag(Aer Ap) E ( (M.+ng+ indicates the weights of different components.
Finally, top-K samples with the K highest similarity scores are selected as candidates for the second-stage search.
16

---

Published as a conference paper at COLM 2024
In the second stage, propose LLM-based fuzzy search. The top-K past driving scenarios selected in the first stage are provided to the LLM. Then, the LLM is tasked to understand the text descriptions of these scenarios and determine the most similar past driving scenario to the query scenario. The driving trajectory corresponding to the selected scenario is also retrieved for reference.
With the proposed vector search and LLM-based fuzzy search, Agent-Driver can effectively retrieve the most similar past driving experience. The past experience and driving decision could help the current decision-making process.
C Reasoning Engine
In this section, provide detailed information the workflow of the reasoning engine. The reasoning engine takes environmental information and memory data as inputs, per- forms chain-of-thought reasoning, task planning, motion planning, and self-reflection, and eventually generates a driving trajectory for execution. Figures 9, 10, and 11 show an exam- ple of how the reasoning engine works. We denote the input environmental information as O and the retrieved memory data as M. Please note that O and M are in the text format.
C.1 Chain-of-Though Reasoning
Chain-of-thoughh reasoning aims to emulate the human reasoning process and generate text-based reasoning results R, which can be formulated as:
R. OO,
(3)
where FLIM is a LLM. To avoid arbitrary reasoning outputs of LLMs which might lead to hallucination and results not relevant to planning, we constrain R contain two essential parts: notable objects and potential effects. Specifically, we first instruct the LLM to identify those notable objects that have critical impacts on decision-making from the input environ- mental information. Then, we instruct the LLM to assess how these notable objects will influence the subsequent decision-making process. The instruction can be established by two strategies: in-context learning and fine-tuning.
For in-context learning, each time we leverage four human-annotated examples of notable objects and potential effects in addition to O and M collectively as inputs to the LLM:
R = (O,,(O,,,E))
(4)
For fine-tuning, auto-generate the reasoning targets R. leveraging the technique pro- posed in (Mao et al., 2023a). Then we fine-tune the LLM to make its reasoning outputs R. approaching the targets R.
Both in-context learning and fine-tuning effectively reduce invalid outputs. As shown in Table 6 of the main paper, compared to the fine-tuning strategy, in-context learning enables the LLMs to generate more diverse reasoning outputs, and results in better motion planning performance.
C.2 Task Planning
Task planning aims to generate high-level driving plans P for autonomous vehicles, taking the reasoning results R as well as the environmental information O and memory data M as inputs. The process can be formulated as
P = FLLM (0,,)),
5
We define the driving plan as a combination of discrete driving behaviors and speed estimations. In this paper, we proposed discrete driving behaviors: fove_forward, change lane to left, change lane to right, turn left, turn right, and stop. We also propose 6 speed estimations: constant speed, deceleration, quick deceleration,
17

---

Published as a conference paper at COLM 2024
peceleeration.t.zeee acceleration, quick.acceleration. The combinations of driving behaviors and speed estimations result in 31 different driving plans (stop has no speed estimation). These driving plans can cover most driving scenarios and they are fully con- figurable, which means that we can add more behavior and speed types to cover those long-tailed scenarios. See Table 1 for details.
Similar to the reasoning module, in- context learning and fine-tuning can also be applied to instruct the LLM to generate driving plans. As shown in Ta- ble 6 of the main paper, in-context learn- ing is more appropriate for instructing the LLM for task planning.
Driving behavior forwafd change_ Lanee wane_to.lle tel change.lane.to.. waa.ntto.ight turn.left turn.right stop
Speed estimation constant.speed deceleration deceleration wwweeerratin.to.zzee zero acceleration quick_ quick.acceleratio
C.3 Motion Planning
Table Driving behaviors and speed estimations.
Motion planning aims to plan a safe and comfortable driving trajectory T, with the driving plan P, reasoning results R, environmental information O, and memory data M as inputs. The process can be formulated as
T (OM,,
(6)
The planned trajectory т can be represented as waypoint coordinates in 3 seconds: [(x1,))),... (X6, A recent finding (Mao et al., 2023a) suggests a fine-tuned LLMs can generate text-based coordinates quite accurately. That is, the LLM can generate text string "(1.23, 0.32)" representing a coordinate, and this can be easily transformed back into its numerical format (1.23, 0.32) for subsequent execution. In particular, given a trajectory t, we first transform it into sequence of language tokens w using tokenizer T:
((1 y1),, (x6, }6) f
Wn
With these language tokens, we then reformulate motion planning as language modeling problem:
N Σ
LI
log [10 ... ,,,,1,,)
(8)
where w and @o are the language tokens of the planned trajectory T from the LLM and the human driving trajectory ê respectively. By learning to maximize the occurrence probability P of the tokens æ derived from the human driving trajectory f, the LLM can generate human-like driving trajectories. We suggest readers refer to (Mao et al., 2023a) for more details.
With proper fine-tuning, the LLM is able to generate text-based trajectory that can be further transformed into its numerical format. This step maps natural-languagee perception, memory, and reasoning into executable driving trajectories, enabling our agent to perform low-level actions.
C.4 Self-Reflection
Self-reflection is designed to reassess and rectify the driving trajectory planned by the LIM. Specifically, the collision check function is first invoked to check the collision of t utilizing the estimated occupancy map. Specifically, we place the ego-vehicle at each waypoint in a trajectory, and then we will mark the trajectory as collision if there an obstacle within a safe margin to the ego-vehicle in the occupancy map. If a trajectory is not marked with collision, we directly use this trajectory as output without further rectification. Otherwise, for those trajectories that have collisions, we leverage an optimization approach to rectify the trajectory 7 into a collision-fred one 7*, Following (Hu et al., 2023), we sample the obstacle points O near each waypoint at timestep t in the occupancy map. Then,
18

---

Published as a conference paper at COLM 2024
optimization problem is formulated and solved through the Newton iteration method:
T arg min( -- 1/r--111122 +2
A
(-11T || - y2
(9)
Σ Σ T (xy)EO, 0V2 OV2T
exp
20
where À and a are hyperparameters. The first term regulates the optimized trajectory r" to be similar to the original T, and the second term pushes the waypoint Ti in the trajectory away from the obstacle points O; for each timestep f.
Attributing to self-reflection, those unreliable decisions made by the LLM can further be corrected, and collisions in the planned trajectories can be effectively mitigated.
C.5 Comparison with GPT-Driver
The most relevant implementation is GPT-Driver (Mao et al., 2023a), which handles the planning problem in autonomous driving by reformulating motion planning as language modeling problem and introducing fine-tuned LLMs as a motion planner. Compared to (Mao et al., 2023a), our Agent-Driver introduce additional modules such as tool libraries and cognitive memories that GPT-Driver does not use. Also, it decouples the thain-of-thoughe reasoning, task planning, and motion planning as separate steps to the LLM. Compared to generating both reasoning and planning in single pass as in GPT-Driver, we empirically found that such separation eases the learning and prediction process, and yields better result. In addition, we use in-context learning to instruct the LLM to perform reasoning and task planning, which encourages better diversity and generalization ability. While in GPT-Driver, it uses fine-tuning, Finally, we additionally introduce a self-reflection stage to further rectify the motion planning results, which leads to fewer collisions and better safety.
D Experiments
D.1 Evaluation Metrics
D.1.1 Open-Loop Metrics
As argued in (Hu et al., 2023), autonomous driving systems should be optimized in pursuit of the ultimate goal, ie., planning of the self-driving car. Hence, we focus on the motion planning performances to evaluate the effectiveness of our system. There are two com- monly adopted metrics for motion planning on the nuScenes dataset: I.2 error (in meters) and collision rate (in percentage). The average L2 error is computed by measuring each waypoint's distance in the planned and ground-truth trajectories, reflecting the proximity of a planned trajectory to a human driving trajectory. The collision rate is calculated by placing an ego-vehicle box on each waypoint of the planned trajectory and then checking for collisions with the ground truth bounding boxes of other objects, reflecting the safety of a planned trajectory. We follow the common practice and evaluate the motion planning result in a 3-second time horizon.
We further note that in different papers there are subtle discrepancies in computing these two metrics. For instance, in UniAD (Hu et al., 2023) both metrics at k-th second are measured as the error or collision rate at this certain timestep, while in ST-P3 (Hu et al., 2022) and following works (Jiang et al., 2023; Mao et al., 2023a), these metrics at k-th second is average over seconds. There are also differences in ground truth objects for collision calculation in different papers. We detail the two different metric implementations as follows.
The output trajectory 7 is formatted as 6 waypoints in 3-second horizon, i.e., T = (xx,y)), (x2, 12),, (x3, (x4, ys), (x6, y6)]. For in each driving scenario, the L2 error is computed as:
1 = V ()2-))2= -x (y; - 9,)2 i=1
(10)
19

---

Published as a conference paper at COLM 2024
where 12 R6x1 and & denotes human driving trajectory. Then, the average L2 enror 12 E I6611 can be computed by averaging /2 for each sample in the test set.
In the UniAD metric (Hu et al., 2023), the L2 error at the k-th second (k =12,,3) is reported as the error at this timestep:
L 2,6
uniad -2[[2k].
(11)
The average L2 error is then computed by averaging 1.2x of the timesteps.
In the ST-P3 metric (Hu et al., 2022) and following works (Jiang et al., 2023; Mao et al., 2023a),
the L2 error at the k-th second is reported the average error from to k second:
2k
24
stp3
(12)
2
2k
The average L.2 error is computed by averaging the 12k of the 3 timesteps again (average over average in other words).
stp
In terms of the collision, is computed by counting the number of times planned trajectory collides with other objects in the ground truth occupancy map for all scenarios in the test set. We denote C E N6x1 as the total collision times at each timestep.
Similarly, UniAD reports the collision Cuniad at the k-th second (k = 1,2,3) as C(2k), while
ST-P3 reports cstp3 as the average from to second:
26 C
Slp3 K
(13)
2k
In addition to the differences in calculation methods, there is also a difference in how the ground truth occupancy maps are generated in the two metrics. Specifically, UniAD only considers the vehicle category when generating ground truth occupancy, while ST- P3 considers both the vehicle and pedestrian categories. This difference leads to varying collision rates for identical planned trajectories when measured by these two metrics, yet it does not impact the L2 error.
In this paper, we faithfully evaluated our approach and the baseline methods using the e
officially implemented evaluation metrics in the two papers (Hu et al., 2023; 2022), ensuring completely fair comparison with other methods.
D.1.2 Closed-Loop Metrics
Within the CARLA simulator (Dosovitskiy et al., 2017), the metrics of route completion and driving score are generally used to evaluate the planning performance autonomous driving systems.
Route completion a straightforward metric, measuring the percentage of a predefined route that the autonomous vehicle successfully completes before the simulation ends. This metric is crucial for understanding the basic capability of an autonomous vehicle to navigate from point A to point B.
The driving score is calculated by adjusting the route completion rate with a penalty factor that considers various infractions, including collisions with pedestrians, other vehicles, and stationary objects, deviations from the planned route, lane violations, ignoring red traffic lights, and failing to stop at stop signs.
D.2 Implementation Details
We employ gpt-3.5-turbo-06133 as the foundation LLM in Agent-Driver. We leverage the same LLM for every task except motion planning, and we fine-tuned another LLM specially for motion planning.
20

---

Published as a conference paper at COLM 2024
ID
Tool Library
Common. Memory
Exp. Memory
Col Reason.
Task
Self- Reflect.
(m) 4
Collision (%)
+
Plan.
1s
2s
Avg.
1s
3s
Avg.
X
/
X
0.24
0.71
1.44
0.90
0.03
0.27
0.91
0.40
/
0.24
0.69
0.03
0.83
x
v
0.24
0.72
1.46
0.81
0.07
0.23
0.96
0.42
/
0.24
/
/
/
/
x
0.25
0.72
1.47
0.81
0.05
0.23
0.93
0.40
/
0.24
0.70
0.79
0.03
0.20
0.81
0,25
0.71
1.43
0.80
0.03
0.08
0.56
0.23
Table 2: Ablation of components in Agent-Driver. The removal of any component can influence the planning efficacy of our system, indicating the importance of all components in our system.
For tool use and memory search, the LLM is guided by system prompts without fine-tuning or exemplar-based in-context learning. In chain-of-thought reasoning and task planning, the LLM is instructed by two randomly selected exemplars derived from the training set. This approach encourages the models to develop various insightful chain-of-thought processes and detailed plans for tasks independently. Please note that we utilize the original pre- trained LLM with different system prompts and user input for the above tasks. In motion planning, we fine-tune an LLM with human driving trajectories for only one epoch.
D.3 Ablation Study
Table 2 shows the results of ablating different components in Agent-Driver. All variants utilize 10% training data for instructing the LLMs. From ID 1 to ID 5, we ablate the main components in Agent-Driver, respectively. We deactivate the self-reflection module and directly evaluate the trajectories output from LLMs to better assess the contribution of each other module. When the tool library is disabled, all perception results form the input to Agent-Driver without selection, which yields ~22 times more input tokens and harms the system's efficiency. The removal of the tool library also increases the collision rate, indicating the effectiveness of this component. In addition, the collision rate gets worse by removing the commonsense and experience memory, reasoning, and task planning modules, demonstrating the necessity of these components. Besides, we further note that self-reflection also greatly reduces the collision rate.
D.4 Impact of Ego-States on Open-Loop Motion Planning
As pointed out by Zhai et al. (2023) and Li et al. (2023), ego status information can pro- vide strong heuristic for open-loop mo- tion planning. To investigate the impact of ego states on Agent-Driver, we conduct experiments of Agent-Driver using purely ego-status and without ego-status. The re- sults are shown in Table 3, where all set- tings are trained with 10% data. We can see that: (1) The ego-only baseline is worse than Agent-Driver, which indicates Agent- does not merely rely on ego-states but also benefits from the proposed components, and (2) After dropping the ego-states, Agent-Driver still works well and performs on-par with UniAD, which means Agent-Driver can still work decently well without ego-states.
Settings
Avg. L.2 (m)
Avg. Col. (%)
Ego-only
0.93
0.32
Agent- w/o /o/cgg
1.00
0.32
UniAD
1.03
0.31
Agent- Driver
Table 3: Impact of ego-states on Agent-Driver
We further note that most prior works that Agent-Driver compared with in Table of the main text use ego-states for planning. Specifically, VAD explicitly inputs ego-states side information for the planner, and UniAD implicitly includes ego-states via its BEV module (Li et al., 2023).
D.5 Compatibility with Different Neural Modules
21

---

Published as a conference paper at COLM 2024
As shown in Table 4, Agent-Driver stantly maintains a favorable performance with combinations of variable neural mod- ules. We argue that discrepancy in per- ception and prediction performance can be compensated by strong reasoning systems and is no longer the bottleneck of our sys- tem. Notably, unlike conventional frame- works which need retraining upon any module change, attributed to the flexibility of Agent-Driver, all neural modules in our system can be displaced in plug-and-play manner, indicating our system's 's compatibility.
Delection
Prodiction
Neuna Modultss
Occupancy
Mapping
Aug. (2
Col.
VAD
VAD
ST43
ST.F3
0.3
0.24
VAD
VAD
UnIAD
UnIAD
0/22
022
UniAD
UnIAD
5T:P
ST.P3
0.7
024
UnIAD
UnIDD
UnIAD
UnIDD
0.74
021
Table 4: Compatibility to different perception modules.
D.6 Language Justification on the BDD-X Dataset
Our method is compatible with diverse functional calls in the tool library. By incorporating the state-of-the-art vision-language model (Jin et al., 2023) as a tool, our Agent-Driver can provide language justification actions based on visual inputs. We demonstrate this capability on the BDD-X dataset, which is a commonly used language benchmark for autonomous driving. Both quantitative results in Table and qualitative results in Figure have demonstrated the language justification ability of Agent-Driver.
Narration
Reasoning
Method
BLRU4
METEOR
ROUGE-I
CIDERr
BLEU4
METHOR
ROUGE-L
CIDEr
Agent-Driver
34.55
30.58
62.75
246.79
11.65
15.36
32.00
103.16
Table Quantitative performance of Agent-Driver on the BDD-X dataset.
Action description: the car is driving forward. Action justification: because the path is clear.
Action description: the car is driving slowly down the road. Action justification: because traffic ahead moving slowly.
Action description: the car slows down.
Action justification: because the car in front of has stopped.
22

---

Published as a conference paper at COLM 2024
Action description: the car is driving forward. Action justification: because there are no other cars on the road.
Action description: the car is driving forward. Action justification: because there are other cars on the road.
Action description: the car is driving forward. Action justification: because there is no traffic in that lane.
Action description: the car merges left. Action justification: because the lane is clear.
Action description: the car is slowing down complete stop. Action justification: because the traffic light at the intersection has turned red.
Figure 1: Visualization of Agent-Driver language justification capability on BDD-X.
E Qualitative Analysis
E.1 Qualitative Ablation
A qualitative visualization of ablating the system components in Agent-Driver is shown in Figure 2. The result shows that when ablating the memory module of Agent-Driver, the planned trajectory (in purple) has larger discrepancy to the ground truth trajectory. When deactivating the thain-of-thoughh reasoning and task-level planning, the planned trajectory (in yellow) deviates more from the ground truth, suggesting that reasoning is a critical component for accurate planning of the system. The qualitative ablation further verifies the effectiveness of the proposed components in our Agent-Driver.
23

---

Published as a conference paper at COLM 2024
E.2 Qualitative Results
Figure 12 provides more examples to show the inter- pretablility of Agent-Driver. Figures 13, 14 and 15 visualize critical objects identified by Agent-Driver and the planned driving trajectories, from which we can see our system progressively identifies critical objects via tool use and reasoning, and eventually plans a safe trajectory for driving. Driving videos are shown Figures 16, 17, and 18. The qualitative results verify the effectiveness and interpretability of our Agent-Driver.
Agure-Drive Doive
we CST placting
5/0 Nonzty
Groas Treds
Figure 2: Qualitative ablation of sys- tem components.
E.3 Failure Cases
We provide failure cases Agent-Driver in Figure 3. We observed that heading errors of large objects, e-g., buses, have a critical impact on motion planning, which indicates the importance of accurate heading prediction in detection networks.
Failure Case 2:
Fallure Case
heading estimations large objects, buses, result in different driving behaviors from deceleration to acceleration.
Inaucurate heading estimations for large ohjects, such as conservative turning trajectory.
Figure 3: Failure cases in Agent-Driver.
F Limitations
Due to the limitations of the OpenAI APIs,
unable to obtain the accurate inference time of our Agent-Driver. Thus it remains uncertain whether our approach can meet the real- argue that recent advances
time demands of commercial driving applications. However, in accelerating LLM inference (Dao et al., 2022; Liu et al., 2023; Xiao et al., 2023) shed light promising direction to enable LLMss for real-time applications. In the meantime, due to hardware development, autonomous vehicles' onboard computational power also evolves rapidly. Given these two factors, believe the inference restriction will be resolved in the near future.
24

---

Published as a conference paper at COLM 2024
Task
Function Name
Function Descriptions
Parameters
Get the defection of the leading object, the function will return the leading object id and its position and size. there is no leading object, return None.
get leading object detection
Null
Detection
Get the detections of the surrounding objects in 20mr 20m range,
surrouuung wwweooc.eeeecttttttttt
function will return list of surrounding object ids and their positions and sizes. If there no surrounding object, return None.
Null
Get the detections of the objects in front of you 20m "40m range, the function will return list of front object ids and their positions and sizes. there no
Null
object, return None.
Get the detections of the objects in customized range (x. start,, x end)"((y.stan start, end)m",
["x start", "xeend" "y-start" "y_end"]
get object detections in range
the function will return list
objecc ids and their positions and sizes. If there is no object, return None.
Get the detections of all objects in the whole scene, the function will return list object ids and their positions and sizes.
get.all.objcct..o aallloobict.eeetettooss detections
Null
Always avoid using this function there are other choices.
Get the predicted future trajectory of the leading object, the function will return trajectory contaliting series of waypoints. If there no leading vehicle, return None.
get leading object future trajectory
Null
Prediction
Get the future trajectories of specific objects of object ids), the function will return trajectories for each object. there no object, return None.
(specified by
get future trajectories for specific objects
["object ids"]
Get the future trajectories where any waypoint this trajectory falls into a given range (x.start, x.endy start, y.end)mm, the function will return each trajectory that satisfies the condition. there no trajectory satisfied, return None
start"" "x end". "y start", "yoend]
get future trajectories rragee
Get the future waypoints of specific objects specific timestep, the function will return a list of waypoints. there is nu object or the object does not have waypoint at the given timestep, return None.
get future waypoint of specific objects at timestep
['object ids", "timestep")
Get the predicted future trujecturies of all objects in the whole scene, the function will return list of object ids and their future trajeclories. Always avoid using this function there are other chaices.
get.all get.all.future.eet trajere.taaecctories
Null
Table 6: Function definitions in the tool library.
25

---

Published as a conference paper at COLM 2024
Task
Function Name
Function Descriptions
Parameters
Get the drivability the locations [xx yi Yr]]
get.drivable.a..loattios
["locations"]
location the map scope, return Nore.
check settivable.of.plannne
trajecc trajectory
Check the drivability at the planned trajectory.
""trajectory""
Map
Get the lane category the locations ((1a. ) ... (xn.9t].
["locations, "ret prob"]
wallaneccatgory.aaloooaiio
the location is out of the map scope, return None.
Get the distance both sides road shoulders at the locations [[),, (1) ... (Xa, Vnl]. the location is out the map scope,
get.distance.to.shollt location
["locations"]
None.
Get the distance to both sides of road shoulders for the
get current shoulder
Null
ego-vehicle location.
Get the distance to buth sides of road lane dividers locations (t),,, (Jn,ya))
get.distance, to htw..eoodddddidiiiraaaaaaaaaaaae dividdii.ddr..a.oaaiios
["locations"]
the location is out of the scope, return None.
get.cureet.lane.ddidee
Get the distance to both sides road lane dividers for the current ego-vehicle location.
Null
nearest pedestrian crossing to the ego-vehicle. If there is no such pedestrian crossing, return None.
Get location
get.reares.pedesttiin..sssss
Null
Get the probability whether list of locations (B).)))... [Xn, yo.] is occupied at the timestep t. If the location is out of the occupancy prediction scope, zeturn None.
["locations" "timestep"l
get occupancy at locations for timestep
Occupancy
Check the probability of whethera plansed trajectury (1x,91)... (Xn.)..( collides with other objects.
check.collision for: planned practrnd.ttajecttrry
["trajectory""
Table 7: Function definitions in the tool library. Cont'd.
26

---

Published as a conference paper at COLM 2024
Language Agent
Driving**
Role: You are the brain autonomous vehicle (alka. ago-vehicle). this need extract necessary information driving scenario.
you extracted useful the next step mation planning.
Necessary
might include following:
Detections: The detected objects that you need to pay attention to.
Prodictions: The estimated future motions the detected objerts.
Maps: Map information indudes traffic lanes and road boundaries.
Occunpancy: Occupancy implies whether location has been occapied by other objects.
Task
You should dhink about what types of Information (Detections, Prediction,, Maps, Occupancy) you need extract from
driving sconario.
Detections Predictions quite important motion planning. should one nemssary.
Maps information are also importast. You should more attention road shoulder and lane divider information to ego-vehicle location.
-I will guide you through the thinking process step step.
wwee"Contett Information:
Current State:
Velocity (vx,ry): (-0.01.0.92)
Heading Angular Velocity (v_yaw): (0.00)
Acceleration (axаa)]: (-0.00,4.500
Can Bus: (-0.74.0.14)
Heading Speed: (0.95)
Steering (-0.02)
Historical Trajectory (last seconds):
Mission Goal: FORWARD
Do need perform detections in this driving scemario? Please answer
YES
You
of the following functions object detection results (don't execute functions that have used
before):
leading object, detection[] #Get the detection of the leading object, the function will return the leading object id and its leading object, return Xend, start, start, end) #Get the detections of objects in given range (x_start, object
pasition
objeoobjettde detections range(x start,
ondj"[ty start, y end return None
function return
pasitions and
object,
range, return list of surroundind object and their positions and sizes. If there no sur surrounding object, return None 10m rango, function will return list front object ids and their positions a sizes. If there is no front object, return Nome the function will return a list of objecc
get ret_surrouuding sarreunding_ojjet object_ [etectionss]
surrounding objects
front object detections() #Gitt the detections the objects front of you
get_alllon wwwpalllbeeeeeeeeeeeeeeeeeeeeeee [etections] #Get the detections all objects the whole and their positions and sizes. Always avoid using this function ffthere are other choices.
("name": "gett front object detections", "arguments":
Front object detections:
Pront ohject detected, object type: car, object id: 2, position: (4.36, 9.56), sixe: 186, 4.72)
object object type: car, object position: (:3.70, 13.08), size: (2.01, 4.92)
Do need to perform future trajectory predictions for the detected objects? Please answer NO.
YES
Figure 4: An example of the tool use process.
27

---

Published as a conference paper at COLM 2024
You of the following functions object future trajestory predictions (don't exccute functions that have been used before):
leading object, future, trajectory() #het the predicted future trajectory the leading objert, function will trajectory containing series of waypoints tfeeere is no leading vehicle, return None
furfuuuuree, trajectories, for.speeiiccobbjesssceef @bjects(cbject_ide #Get the future trajectories of specifie objects (specified List of object ids), function will return trajectories for object. there no object, return None
future, trajectories in, (xnge(x (t.ppttttttt start, cnnd, start,y end) #Get the fature trajectories where waypoint in this trajectory into given range start end) ststat, tyyyyystttyyytttete. y_endJm the condition. If there is trajectory satisfied, return None get, future, waypoint, sppcccic_ppppcecccccceee
return trajectory that
waypoints specific objects specißi timestep, the function will retura a list of waypoists, If ththe is no object the object does have a waypoint at given timestep,,
timeste timestep)
get all future. trajectories() ffiett the prodicted future trajectories of all objects in the whole scene, the funation will return list object ids and their future trajectories Always avoid using this fanction there other choices.
("name": "atee"""""""uuuee fet_uuuure_eeee www.uuuttttteeettteeeeee www.eectoriisseooe. http:/oo__pccfioobjjeeeecc "arguments": "object_ids" (2)
Future trajectories for specific objects: Object object kl: 2, future waypoint coordinates 3s: [(4.6, 9.56), (4.36, 9.56). (4.36, 9.57). (4.36, 9.57), (4.36, 9.56). (4.36, 9.56]]
Object object 3, future waypoint coordinates 36: [(-2.66, 13.82), (-1.99, 14.79), (-0.99, 16.13) (-025, 17.7). (0.19, 19442). (0.57, 21.35]]
you need get occupancy Please answer YES or NO.
driving scenario?
need get information for this driving scenario? Please answer YES
YES
You can exncute one of the following functions to get map Information (don't execute functions that have been used before]:
get, drivablle locations(loration) #Grt the drivabiiity the lorations [(x (, y.1).. (..)....(eee (..n. y.n]] the location out of
the map scope, return None
June category. Jocations(location. [sturesscore) #Get the lane of map scope, retur None
the locations y_.), (K...y.))) Ir
location
get distance, to shoulder at locations(Incatttn y_1)... (SLA y.n]]
#Get the distance to both sides of road shoulders at the locations [(x. the map scope, return #Get the distance both sides of road shoulders for the current ego-vehicle location. distance both
location
locations (locatoon((catioon)) yBLl,y_1).. v_. (RLA, y_t)] If the location is out of the map scope, return None
get_distance,ue to lane_divides
lane, dividers
ggot current Jame divider()
dividers current ego-vehicle location
get nearest podestrian crosingg] #isn the location of the nearest podestrian crossing to the ego-vehicle. If t'there is no
such pedestrran crossing, return None
("name": "gett current shoulder" "arguments": '("")
shoulders current ego vehicle location: Current ego-velicle's distance to left shoukder 7.5m and right shoulder is 4.0m
Distance
Figure 5: An example of the tool use process. Cont'd.
28

---

Published as a conference paper at COLM 2024
Language Agent Rale: You are the brain of autonomous vehicle (akka, egu-vehicle). this step, need retrieve the most simllar past driving expertence help deciston making
Driving""
Task
You will receive the current driving scmmorio.
receive past driving experiences.
You should decide ONLY ONE experience that is mast similar to the current soemarto based on the information provided.
Please answer ONLY the index (eg, 2) of the most similar experience.
La nguage Agent Autonomous Driving:"
Rale: You are the brain of autonomous vehicle (akaa egu-vehicie). In this step, you need retrieve the most simlaar past driving expertence deciston
Task
You will recetre the driving
You will also receive several past driving experiences.
You should decide ONLY ONE experience that mast similar to the current scenario based on the information provided. similar experience.
Please
index (eg,
#* Current Scenario:
***** Bavirommental formatiion.**
Current State:
Velocity (vx,y]: (-0.01,0:92)
Headdnn Angular Velucity (v_jaw): (0.00)
Acceleration (axay)): (.0.00-0.50)
Can Rus: (-0.74.0.14)
Heeading
Stoeting: (-0.2)
HEsturiaa Trajectory (last seconds): (,00000669..0000133330000222,,00 (00000)))))))))))))00
Front objject detections:
object object type: car,
position: 9.56), (1.86, 4.22]
Front object delected, object type car, object id: 3, position: 13.08), size: (2.01, 4.92]
trajectortes spectfic objects:
Object type: car, object id: 2, future waypoint courdinates in 3s: [[4.6, 9.56), (4.36, 9.56), (4336, (453) (4336, 9.56). (1.36, 9.56]]
Objecct type: car, object 19.42), (0.57, 21.35)
waypoint coordinates ([:2,6, 13.82), (-1.99, 14.79),(...9))) (-0.2) 17.73) (0.19.
Distarcce both sides road shoulders current egu-vebiie locatioon: Current ego wehucle's 's distance left shoulder 7 S.m and right sh shouldes 4.0m
Found leveevnt experiences:
Driving Experience
a****Past Environmental Information:
Current State:
Velocity (vs.y]: (0.00.1.07)
lleading Angular Velucity (v_jaw): (-0.00)
Accoleration (axaay]: (-0.02-0044)
Can (-0.6700033)
Heading Speed: (1.00)
Steering: (0.13)
Historical Trajectury
Mission Goal: FORWARD
trajectortes spectfic objects:
Objectt type: car, object id: 2, future waypoint coordinates in [[-1.13..1133 (1. (-0.9,.9.2))...
-7.96), (-0.93, (393366671]
Object type: car, object
waypoint coordinates [[:25.19, 11.77),, (:55.19, 17.79) (:25,,)(.22.11 (:25.18. 11.79) (:25.18, 11.78), (-25.1R, 1R. -1.778) (-25.17, 17.78]
Figure 6: An example of the memory search process.
29

---

Published as a conference paper at COLM 2024
Past Briving Experience
PPtt Environmental Information
Current State:
Velocity (vxzy): (0001,097)
Heading Angular Velocity (9 yau): (0.00)
Acceleration (ax,ay]: (-0.010046)
(us:(-0.6800.1]
-Headinng Speed: (1.16)
-Steertag: (0.04)
Historical Trajectory (last
Mission Gool: FORWARD
object
Front object detected, tyjecttype: car, object id: positin: [-0.8, 9.54), size: (1.75, 3.99)
Front object detected, object object in positiio:: (3.57, 7.27), size: (188, 4.49) object lype:
object
positions (-0.06, 19,97), sizer (2.93, 9.04)
Future trajectortes for specific objects:
Object
object id: 2, future waypoint coordinates 3s: [33.9, 12.47). (3.61, 17.78). 23.26), (3.70, 28.79). 34.46), (3.65, 40.07)
Object type: tuck, object id: 3, future waypoint coordinatessnn in 3e: [[-0,0, 19.96), (-0.08, 19.98), (-0.09, 19.93), (-0.09, 19.92), (-0.88, 19.92). (-0.09, 19.91)]
aa Past Briving Experience
****ePast Enstronmental Information: r<e.*
VVelocty (vazy]: (-0.01.1066)
Heading Angular Velocity (rayaw): (-0.0)
(ax,ay]: (0.00)0.32)
Can (-0.77077)))04
HHadding Speed: (1.06)
Steerring: (-0.01)
Histocical Trajectory (last seconds):
(200:44411220000,2240000222221.0 [-002-106]]
Mission Goal: FORWARD
Front object detections:
ubject lype: object position: (-3.06, 2.56), 4.5]]
Future trajectortes for specific objects:
Object type: car, objectid: 2, fature waypoint coordinates in 3s: 18.54, -24.3)) (:1854, 24430) (-1854, 24.30), (:1854, 24.30),
(-18.54, -24.31), (-18.54,222
Object type: car, objecttd: future waypoint goordinates [(33.26 -31.78) (:343, 22.27) (:136, -2.09)) (3345, 2214).
3.46, -1 19622, (3351, (3351,177.45) 11.445]
Please return the Index 1-3 of the most similar experience:
2
*****past Driving Experience for Referena::**.
similar driving experience memury with confidlnnce The planned trajestury in this experience for your reference: [0.01,09))) (001,1887), (0.02,2.40), (0.02,2.44), (0.022.7))..00222..88)))) (0.02.266))
Figure 7: An example of the memory search process. Cont'd.
wev"Common Sersse:
Avaid millison with other objects.
Always drive drivable regions.
Avoid driving on occupled regions.
Pay attention
ego and historical trajectory when planning
Maintain a safe distance from the objects in front of you.
Figure 8: Retrieved commonsense memory.
30

---

Published as a conference paper at COLM 2024
**A Language Agent for Autonnmous Driving"
Role: You are the brain of an automonnus vehicle (akaa, ego-veliicee) In this siep, you need to first deterriize notable ubjects and identily their pocential offects drising route, and then derive a high-level driving, plan.
Contest:
perpendicula,
parallel to direction you're 're facing. You're at point (0,0).
Input
You will receive your current ego states. You will also recetve current perception results.
Task
egu-states, Notable objects objecte the abjects that your driving roste. you should always pay atteation objects front (with positive y) of you, (within meters] you.
need determine the notable objects
perception
have patential
Yoo need to describe the potential eflects of thuse notable ebjects on your driving route.
You need to derive a high-level driving plan based on the former information and reasoming results,
The driving plan should be combination of meta action from (" ("STOP". "MOVE FORWARD" "TURN LEFT", "CHANGE LANE TO LEFT. "TURN RIGHT", "CHANELANE TO RIGHT") and speed description from I"A CONSTANT SPEED", "A D DECHI ERATION", "A QUICK DECELERATION "A DECELERATION ZERO", ACCELERATION", QUICK ACCELERATION the "STOP"
** "Stritlyy follow output
Output
-w*"Choii-d--Thon Reasoning" t-rey
Notable Objects:
Potental Elfects:
Notable Objects:
Potential
.*. "***"Task Planning:
Behavior: Speed::
Driving plan:
Here are examples for your referemce:
Example
#t Input:
meviiooemennall Information: **..e
Current State:
Velodity [vxuy]: (-0.02,2.66)
Heading Angular Velacity (v.y))): [0001)
- Acceleration [mxay]: (0.00.0.00)
Can Bus: (-1.72,0.95)
Heading Speed: (2.83)
Steering: (1.12)
Historical Trajectory (last secands): [[-1.6, 1043), (1.8,,,)9)
Mission Goal: RIGHT
Pront objest detected, object type: bicycle, object id: 0, position: (-1.02, 7.49) size: (0.49, 1.67)
Front object detected object lype object id: position (8.71, 18.66), size: (1.92, 4.55)
Puture trajecturtes specific objects:
Object type: bicycle, object future waypotot coordinates 3s: [(1.02, 7,5 (-1.22, 7.52), (-1.22 7.54), 7.55] (1102, 7.59), (-1,02, 7.61]]
Objeett typer tar, object id: 1, future waypoint coordimates in 3s: [(8.71, 18666), (8.70, 18.65,, (8 1865), (8.69, 1864), (8.69,
18.63), (8.69, 18.65]]
Distance to both sides of road shoulders of current vehich location: ego-vehicle's distance shoulder 1.0m right
## Expocted Output
<*e**glaatn Thoughts Reasoning: *....
Notable Objects: bicydl at (- (-1.02,7.49), moving to (-102,,7.51] at 0.5 second
Potential Effects: within the safe the theeeeeeeiide 0.5 second
-re*Task Planning
Behavior: TURN RIGHT, Speed: A CONSTANT SPEED
Driving plan: TURN RIGHT WITH CONSTANT SPEED
Figure 9: An example of thain-of-thought reasoning and task planning.
31

---

Published as a conference paper at COLM 2024
Exampte
#B Input:
"**** "Environmental Information
Velocity (vgvy]: (-0.10,5.42)
Heading Angular Velacity (::))::: (-0.00)
Acceleration [axay]]: [0.02,1.14)
Can Bus: (0.92,0.25)
-Heuding Speed: (4.53)
Steering: (0.03)
Historical Trajectory (last seconds]: [0tp::/:wwwam...00000000000/0010
Pront object detections:
objecc object type: pedestrian, objjert 4, positiio: (6.49, 16.88), (0.66, 0.72)
Puture trajectories spectfl objects:
Object type: pedestrian, object kd: future waypoint coordinates 3s: [[6.46, 17.53), (644, 18.20), (6:42, 18.89), (6.38.19.57) 19.57). (6.37, 2026). (6.34, 20.91))
Distance both sides of road shoulders of selected lacations: Location (6.4, 16889) distance left shoulder 2.5m and distance shou kdee uncertain
## Espected Output
.**** "*"ssin-of-TTouue Thouughts Reasoning: .*..*
Objects:
(244,44.97)
Putential Effects: within the safe zone of the ego-velide at second
.** =****aask Planning exee*
Behavior: FORWARD, Speed: A DECHLERATION
Driving plan: MOVE PORWARD WITH DECELERATION
-.. Envirunmmentall Information:"
Velocity (va-yy): (-0.01.0.2))
Heading Angular Velacity (v_yaw): (0.00)
Acceleration (as,ay]: (-0.00-0.50)
(an Bus: (00.4.00.4))
Headdin Speed: (0.95)
-Steering: (.0.22)
Historica Trajectory (last seconds]; http:://g.........../////0//44/0
Mission Goal: FORWARD
Pront object detections:
object detected, type: object id: (4.36, 9.56),
Pront object detected, object type: car, object position: (:3.70, 13.08), size: (2.01, 4.92)
trajecturies specific objects:
Object type: car, abject id: 2, future waypoint mordinatees in 3s: [(4366 9.56), (4.36, 9.56), (4.36, 9.57), (4.36, 9.57), (4.36 9.66). (4.36, 9.56]]
type:
waypoint
3s: [[: [(22.66, 13.82), (-1.9,, 14.799, (0.999.6.11)))) 16.13), (-0.55, 12.73), (0.19,
19 19.42), (0.57, 21.35]]
Distance Location (4.36, 9.56] distance to lell shoulder is 16.5m and right shoulder is 0.5m Location
locations:
distance shoulder 13.0m right shoulder 8.5m
*****Chain Thoughts Reasoning:****.
Objects: (4.36,9.66)
Potential Riffects within the safe zone of the ego-vehicle at 0.5 second
Notable Objerts:
(33.7,11308)
Potential Ellects within safe zone the ego-veliclled 0.5 second
as<enTaskd Plamning:****
plan: MOVE FORWARD DECELERATION
Figure 10: An example of chain-of-thought reasoning and task planning. Cont'd.
32

---

Published as a conference paper at COLM 2024
"A "Attomomous Driving Plannes** You're an autonomous
trajectory
Context:
Courdinates: perpendicular,
facing You're point (0.0). meters.
Goal: Plan 3-secund route using & waypotnts (0.5s intervals).
Inputs:
1. Ego States (Important]:: e: Current stats (voladity, acmoerattonn pa trajectory, goal direction.
2. Perception Resullss
Expertences (important::
expertences with confidence scores referenced planned trajectory.
4. Thaffic: Rules.
5. Ressoning (important] Notable ubjects ailestinn plan and top-level driving plan.
Tasle
Based imputs, safe, feastble s-secnnd trajectory of waypoints.
Output:
Planned Trajectory:
[(x1y1], (x2,2).... -.. (X6y6]
***** Envisomentaal Information: **re.
Current State:
Vellcity (vxuy]]: (-..0
Heading Angular Velocity yaw))((0000
Acuellration (oxay): (-0.00.0.50)
Can (us: (-0.74.0.14)
Heading Speed: (0.95)
Steeriing: (-0.02) Historical Trajectory (last secunds): [[-0.07-6643]) (4000777.66.0000000::0000,,,0000
Frunt object detections:
Front object detected, object type: car, object id: position: (4.36, 9.56), size: [1.86, 4.72]
Frunt object detected, object type: objectid: positioos: (-3.70, 13.06), size:: (2.01,4:92)
Future trajectories spocific objocts: Object type car, objectt id: 2, future waypoint courdinates in 3st [[4.66 9.56), (4.36, 9.56), (4.36, 9.57) (436, 9.57), (4.36, 9.56). (9.36, 9.56]] Object type: car, abjeett 3, waypoint coordinates in [[-2,66, 13.82), (-169, 14.79) (-0.99, 16.13), (-025, 17.7), (0.19, 19.42), (0.57, 21.35)]
Distance to both sidee of road shoulders of selected lorations:
Location (4.36, 9.56) distance to left shoulderts is 16.5m and right shouldee is 0.5m Location (33.70,
distance
shoulder 13.0m and right shoul
*****Past Driving Experie experience for Refereace:**eee
similar driving experience from memory with confidencee
The planned trajectory experience for your reference:
[(0.01.0.90). (0.01,187). (0.02,2.40) (0.02.2.69), (0.02.2.66) (0.02.286))
--m-Coommnn
Avoid colliston with other objects.
Always drive drivable regions.
Avoid driving un occupted regions.
Pay attention to your ego states and historical trajectory when planning.
Maintain distance from
front you.
e*ereChatt of Thouuhhs Reasoning:**...
Thoughts::
Notable Objects: cur at (4.36.956) moving (4.36.9.57) 57) at second
Postnntial Pifects: within the safe zone of the tgo ago-vehicle at 0.5 second
Objects:
moving
second
Potential lffects: within the safe zone of the ego-vehucle at second
*.exeTack Planning: ****.
plan: MOVE FORWARD DECELERATION
Planned Trujectory: [000100 (0.01.144), (0.02.199)) (0.02.2.7), (0.02244). (0.02257]
Figure 11: An example of motion planning.
33

---

Published as a conference paper at COLM 2024
coneennnnnnaat informatioee
deseccioon
Sammending devaiisse
Faus: dajud olossal, obass pppccun de pestiare (240) 2iztlaee see (V/1, 466)
Sammendngg box diteded ubast opec are obbou.ie pootber (455 sabtuce (1,5k, 437)
sijuct disacial eypecio st of pestiare 222e
Pature sdtetoo forapat whovaar
auss inbouurk fur spediic abedde
Used tops oa db sdld kourcweppen cronilaaa in delll 0200
(2--4, [.3].-]
Maar adferation (mve doouidenj.
Cijec ypu coc akjer Carmes caenebiccl derane abouadtur mumaaal sluulderte 15ah
Curmiee agenenengeeee aggeeiiiiee dessace chatiee
ceeteCaeeeen weteeennnneeeeeeeee denap.
Arad ou duaa with oiber abtacas..
Avold all lasas vitlriler diject...
diivinn
freiner craita
dialtr perieee
dialivity
Scatur inkenzatad:
Six recassillemmaiait
redormses (1) 13.21 (9,53/207
artw serond
Pounntal Eéleea wabli anloaal poie witheeae--eiee voliiid at seased
Petarca ofteca wabli raki wheeee---eddeeeeee sened
seserTak plaratag.
esserTadk planstng seess
Bebankr fumand Socdlt ao Brivin plas: move Reoaar ivtta acoileatios
Dubader zum raat:Speed caustais Briving hlaee sammiaat whth cracamassed
Tra outtott-- (0022 1Ex cess 0-0270003 22.22.200 32.322221 60,6423)
Traautery (102222:7] (0.5, (Las, 1022) 3r0u1244m (5.44:4.15)
reflection
snmpastovenvettl informotionnne
-e Renirneeeeed infernation
Lasting ohieer deramann
Faniee coger milie opedidir dynte
Puturs tmaociooiis apeciiiettecc
tope oa abjes
vappasa courdinatas 3s 2.45..
eiperadi
1821)
May deeikiee)
Conneragen wwweeenggeeeiiiik.oo dicone to cloaldevis Simond cividdericcee
Map inpermeemataa
ieennsernvve ceass
exempust arrving raper pepeiio rnfornaee *tese
severtust diiving experience for infrresppu-teee squattee from mumuryy sindarty uoveco.25
Nostata driding snperemns comaary with more 0.8s.
glavend toge in somsattoir deeece [1003) (100119:3)
TTe pli bncr withis senaa reto refivenr (10 (001..5554)
enerr thougitto reasontegr
sesenghate of diihhegght resseniin Serzik abjects 12.79.626) 20]
Nueada Miject carza (1.22
dadd planning
ewwwya Maston plareing peee
eserrattion plaseing
Trajecors (1601.24 weteeteeeeeeeeeeeee meee
su F. (1a2a..9) (0028
(00 (022340)) (0.91137)) 18300, 29,2222.00 (38002232) eeergetieettn No calirie. to caage ae the oustie planeegeet
No odliduu Wo dhongs doareettonn pannggaeese
Figure 12: Interpretability of Agent-Driver.
34

---

Published as a conference paper at COLM 2024
Sensory Data
Sensory
Perception
Perception Neural Nets
Meaningful Objects from Tool Library
Meaningful Objects Tool Library
Notable Objects from CoT. Reasoning
Notable Objects Reasoning
Trajectory Planning
Trajectory Motion Planning
Figure 13: Visualization of how Agent-Driver progressively identifies critical objects and performs motion planning.
35

---

Published as a conference paper at COLM 2024
Sensory Data
Sensory
Perception Neural
Perception from Neural Nets
Meaningful Objects from Library
Meaningful Objects Library
Notable Objects from CoT. Reasoning
Notable Objects Reasoning
Trajectory Motion Planning
Trajectory Motion Planning
Figure 14: Visualization of how Agent-Driver progressively identifies critical objects and performs motion planning.
36

---

Published as a conference paper at COLM 2024
Sensory Data
Sensory
Perception
Perception Neural Nets
Meaningful Objects from Tool Library
Meaningful Objects Library
Notable Objects from CoT. Reasoning
Notable Objects Reasoning
Trajectory Planning
Trajectory Motion Planning
Figure 15: Visualization of how Agent-Driver progressively identifies critical objects and performs motion planning.
37

---

Published as a conference paper at COLM 2024
Figure 16: Driving video of Agent-Driver.
38

---

Published as a conference paper at COLM 2024
FF
Figure 17: Driving video of Agent-Driver.
39

---

Published as a conference paper at COLM 2024
Figure 18: Driving video of Agent-Driver.
40


Reviews:
[{"review_id": "udPnc0hKqv", "reviewer": "Reviewer_sSRF", "summary": "The paper presents a novel framework using LLMs for autonomous driving planning tasks. The proposed method integrates 1) a tool library for perception and prediction, 2) a memory module of commonsense and experience, and 3) LLM-based reasoning module that performs chain-of-thoughts reasoning, and motion planning. The paper conducts comprehensive experiments to demonstrate 1) the effectiveness of the methods in terms of L2 distance to ground truth and collision rates, 2) few shot learning ability by training on a small percentage of data, and 3) performance difference for in-context learning and fine-tuning. The paper also includes comprehensive details of the method and implementation in the appendix (20+ pages), which is laudable. The paper is well written, and easy to follow. The diagrams are illustrative. Experiments are reasonable. \n\nOverall, it's a forward-outlook novel paper that showcases the effectiveness of LLMs applied to autonomous driving. This work is a valuable addition to the field. The results are worth sharing. I hesitate between top 5% and top 15% accept papers, as this paper is already pretty well-known in the field of autonomous driving, and is considered one of the early notable papers that apply end-to-end LLM agents to autonomous driving.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions!\n\n- Costs: inference time using API calls takes 500 ms on average, which also includes the network communication latency. We note that many papers are actively working on accelerating LLM inference, and vehicles' onboard computational power also evolves rapidly. Therefore, we believe this limitation will be resolved soon. We will add more explanations to the limitation section. \n\n- Where does the method struggle: similar to other driving methods, our method is limited by the perception results from the tool library. If the tool library provides noisy results, the reasoning engine might not make appropriate decisions. Figure 2 in Appendix F.3 better illustrates this issue. Future work to address this limitation includes leveraging multiple perception models to provide increased robustness.  We will add more explanations to this section. \n\n- How much does CoT help in long-tail / challenging driving scenarios: we provided Figure 1 in Appendix F.1 as a qualitative study showing the effects of CoT in challenging scenarios. The figure shows that when driving near a construction area, CoT helps generate more suitable driving trajectories that effectively avoid approaching this construction area.\n\n- Progress and comfort as evaluation metrics: in addition to the L2 and collision metrics, we have also included Route Completion and Driving Scores as metrics for driving progress and comfort in Carla experiment (please refer to Table 1 and section 3.2 of the main paper). The Route Completion denotes the progress of driving on a route, and the Driving Score additionally takes comfort and safety into calculation. \n\n- What does \"invalid\" mean in sec3.6: as the reasoning engine needs to generate waypoint coordinates for motion planning, we denote \"invalid\" as the waypoint outputs contain non-numerical values. We note that fine-tuning enables robust alignments and our reasoning engine can always generate valid waypoints with a few fine-tuning data. We will add more explanations to this section. \n\n- In-context learning: we will add more clarifications in the main paper regarding the concept of in-context learning and fine-tuning. For fine-tuning, we can use more samples, as it is not restricted by the context window like in-context learning.\n\nWe hope these responses properly address your concerns. Thank you again for your valuable suggestions! We are committed to providing additional responses should you have any further comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a novel framework using LLMs for autonomous driving planning tasks. The proposed method integrates 1) a tool library for perception and prediction, 2) a memory module of commonsense and experience, and 3) LLM-based reasoning module that performs chain-of-thoughts reasoning, and motion planning. \n- The paper conducts comprehensive experiments to demonstrate 1) the effectiveness of the methods in terms of L2 distance to ground truth and collision rates, 2) few shot learning ability by training on a small percentage of data, and 3) performance difference for in-context learning and fine-tuning.\n- The paper also includes comprehensive details of the method and implementation in the appendix.\n- The paper is well written, and easy to follow. The diagrams are illustrative.", "reasons_to_reject": "Some minor comments and questions (not necessarily reasons to reject the paper):\n\nI would be interested in learning \n- Training and inference costs. The paper only mentions it in Appendix G as \"unable to obtain the inference time\". It'd be helpful to provide a rough estimate of at least the magnitude of inference time compared to non-LLM based state-of-the-art-ish baselines. \n- Where does the method struggle? The paper provides a couple of failure examples but does not provide a description on where the LLM-based driver likely to fail. The limitation paragraph in the appendix only mentions inference time.\n- How much does CoT help in long-tail / challenging driving scenarios? The paper does not demonstrate it despite the opening example in the Intro section.\n\nExperiments\n- The paper reports L2 distance to ground truth and the collision metrics. Showing additional metrics such as progress, and comfort would strength the results.\n- What does \"invalid\" mean in sec3.6?\n- The main text does not specify the \"in-context learning\" method. From the appendix, in-context learning learns from \"four human-annotated examples\". Does fine-tuning also use such small number of samples? Would be better to clarify in either the main text or appendix to establish fair comparisons.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2cWrjAZsH", "reviewer": "Reviewer_zg5k", "summary": "The paper presents Agent-Driver, a system utilizing an LLM as a driving agent. The system consists of several modules: tool library for detection, prediction, occupancy, and map modules, cognitive memory which allows search in past driving scenarios, reasoning engine which prompts an LLM for chain-of-thought reasoning, task planning, motion planning and self-reflection (which are all steps of giving prompts in a specified format). \nThe system is evaluated with experiments on open-loop driivng on the nuScenes dataset and closed-loop driving in the CARLA simulator.\nThe system is compared to several other recent works with LLMs for autonomous driving. The results show improvement in route completion for closed-loop driving and improvement on most metrics for open-loop driving.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\n- What is the difference between the reasoning engine and the implementation in GPT-Driver? There are several key differences between our reasoning engine and the one in GPT-Driver: \n\n(1) We decoupled the chain-of-thought reasoning, task planning, and motion planning as separate steps to the LLM. Compared to generating both reasoning and planning in a single pass as in GPT-Driver, we empirically found that such separation eases the learning and prediction process, and yields a better result.\n\n(2) We use in-context learning for instructing the LLM to perform reasoning and task planning, which encourages better diversity and generalization ability. While in GPT-Driver, it uses fine-tuning. \n\n(3) We additionally introduce a self-reflection stage to further rectify the motion planning results, which leads to fewer collisions and better safety. \n\n- Information flow in the current method: We employed multiple rounds of conversations and used separate messages at each stage to the LLM. Compared to sending all messages as a whole to the LLM, we empirically found that separating the messages makes the LLM focus on one task at each stage, which eases the learning process and yields a better result. \n\nWe hope these responses properly address your concerns. We are committed to addressing any further concerns. Thank you again for your valuable comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The presented system builds on top of recent line of research that uses LLMs for autonomous driving. In contrast to some recent work, the system uses components which specialize indifferent aspects of the driving task, and according to the reported results, this setup seems to be beneficial. \nThe components are described in a good detail and the architecture of the system is mostly understandable.", "reasons_to_reject": "Not really a reason to reject, but maybe something that should be explained. The text prompts to the LLM in the reasoning engine resemble the one in GPT-Driver. Maybe this could be commented. What is the difference between the reasoning engine and the implementation in GPT-Driver? Is the current paper an extension that adds memory and tool library (this does not mean this is bad, but just that maybe it could be explained for clarity)?\nIn the current paper, the prompt is given step-by-step, how is this information passed to the LLM? As separate messages? Does it make any difference in comparison to having one big message, or does this separation make it easier for the model to learn?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZMNyiffKH", "reviewer": "Reviewer_THFV", "summary": "This paper presents Agent-Driver, an LLM-based agent designed for autonomous driving. Agent-Driver operates other perceptual neural models as tools and is equipped with a reasoning engine that facilitates chain-of-thought reasoning, task planning, motion planning, and self-reflection. The model's promising performance is evidenced by both open-loop and closed-loop evaluations. Overall, this paper is well-written and features robust experimental work, with some minor concerns over the setups and reproducibility.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\n- Multimodal Domain Integration: Thanks for your comments! While leveraging a unified multi-modal LLM to handle all driving challenges is a promising research direction, a multi-modal LLM that directly maps sensory inputs to actions generally lacks sufficient robustness, as it cannot fully utilize important intermediate representations like detections, predictions, and maps to avoid collision. On the other hand, we choose a fundamentally different approach, where we still leverage those modular perception modules as tools and introduce LLM as a scheduler and reasoning engine, which could benefit from both these accurate perception results and the reasoning and planning ability of LLM. Which approach is better remains an open question, and we're open to insightful discussions.\n\n- Language-related Benchmarks: Thanks for your suggestions! Compared to the multi-modal driving agents that focus more on language-based driving scene understanding or driving question-answering, our Agent-Driver has a different focus: we aim to address the planning and decision-making problems in real-world driving. Hence, we choose to evaluate our approach on the broadly used driving challenges like nuScenes and Carla. Language benchmarks, such as BDD-X for VLM driving agents, are out of the scope of the current paper. Also, our system does not have a VLM available for BDD-X evaluation. However, we agree that scene description and understanding are very important to autonomous driving, and we plan to integrate a multi-modal LLM as a tool for scene understanding in a follow-up work. Thanks for your suggestion.\n\n- Environmental setup: Thanks for your suggestions! For closed-loop experiments on Carla, we leveraged the perception modules in LAV [1] and kept the other parts of our system the same. We also followed the same training setting and evaluation protocols in [1] for a fair comparison. We will add more details about the closed-loop experiments to our paper. Thanks for your suggestions!\n\n- Commonsense memory and experience memory. Sorry for the confusion. We will change the terminology to semantic and episodic memories and provide further explanations in our paper. Thanks for pointing this out!\n\nWe hope these responses properly address your concerns. Feel free to let us know if you have further concerns and we would be thrilled to address them. Thank you again for your valuable comments!\n\n[1] Chen et al. Learning from All Vehicles.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The Agent-Driver: The proposed Agent-Driver framework applies an LLM agent to the autonomous driving domain. This agent operates other perceptual neural models as tools and includes a sophisticated reasoning engine. The technical execution regarding the LLM agent is thoroughly solid.\n\n2. Experiments: The experimentation is comprehensive. It includes both closed-loop and open-loop evaluations, alongside various ablation studies that test different LLM variants and neural module variants, providing a thorough examination of the system's capabilities.", "reasons_to_reject": "1. Multimodal Domain Integration: Given that autonomous driving is a highly multimodal domain, this paper utilizes perception modules from prior work, such as UniAD. However, considering the availability of multimodal LLM agents like DriveGPT-4, the paper should justify why it opts to use other modality processors as tools instead of leveraging a unified multimodal approach.\n\n2. Language-related Benchmarks: It would be beneficial for the experiments to be conducted using the BDD-X dataset, which focuses on the language capabilities and interpretability of driving agents. This would provide a clearer assessment of how well the Agent-Driver can handle real-world language and driving scenarios.\n\n3. Clarity in Experimental Setup: The descriptions of the closed-loop experiments lack clarity. It is not specified whether the same neural networks are employed across different test setups, which is crucial for assessing the consistency and reliability of the results.", "questions_to_authors": "Questions:\n1. Are the same neural networks employed in the CARLA evaluation?\n\n2. As an LLM agent, is Agent-Driver capable of giving justifications to actions, e.g., as defined in the BDD-X benchmark? It might be interesting (though not required) to compare with effective baselines on BDD-X (ADAPT, DriveGPT-4, DriVLMe, etc).\n\nSuggestions:\n1. I was confused by the terms “commonsense memory and experience memory”, I think in the memory modeling community these are close to semantic and episodic memories.", "extra_scores": {}}, {"review_id": "HczItXQYxy", "reviewer": "Reviewer_1NQx", "summary": "This paper introduces an LLM-supported agent named Agent-Driver, and the proposed system consists of a multifunctional tool library, a cognitive memory, and a reasoning engine. The cognitive memory contains commonsense memory and experience memory. The reasoning engine contains four parts: chain-of-thought reasoning, task planning, motion planning, and self-reflection. It conducted experiments on open-loop and close-loop driving challenges, and achieved SoTA results on the nuScenes benchmark.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\n- Comparison with DiLu: Thanks for pointing this out! We have discussed the comparisons with DiLu in the related works (please refer to the last paragraph of appendix A). Our method is fundamentally different from DiLu in many aspects:\n\n(1) Architecture: Compared to DiLu, Agent-Driver has a more comprehensive architecture and has a better capability to handle realistic driving challenges. Our tool library includes a comprehensive perception and prediction toolkit which covers detection, prediction, occupancy, and mapping. Our memory is divided into a commonsense memory that stores driving knowledge and an experience memory that stores past driving experiences. Our reasoning engine is more structured and contains reasoning, task and motion planning, and self-reflection. All the above-mentioned components were not proposed or included in DiLu. \n\n(2) Experimental evaluation: Compared to DiLu which was evaluated in a virtual and simple  highway environment, Agent-Driver is evaluated on a real-world dataset nuScenes, and a more challenging close-loop simulator Carla. Evaluation in these realistic and challenging driving scenarios better reflects the effectiveness of our method.\n\nWe will add more discussions of DiLu in the related works. Thanks for your comments!\n\n- Latency issue: In addition to the limitations we have discussed in the paper, we'd also like to point out that recently more and more papers [1-2] have been actively working on accelerating the inference of LLMs, which greatly reduces the latency of LLMs. In the meantime, due to hardware development, autonomous vehicles' onboard computational power also evolves rapidly. Given these two factors, we believe the inference restriction will be resolved in the near future. We will add more explanations to the limitation section. Thanks for your advice!\n\n- Loadable LLM model instead of API calls: We have provided experiments using Llama2-7b as a core LLM in Section 3.5 and Table 3. The results show that using a loadable LLM like Llama2 can lead to a decent performance. \n\nWe hope these responses properly address your concerns. We remain fully committed to addressing any additional concerns that may arise. Thank you again for your valuable comments!\n\n[1] Liu et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time.\n[2] Lin et al. Awq: Activation-aware weight quantization for llm compression and acceleration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tAgent-Drivers are redefining the traditional framework of autonomous driving by embedding human-like reasoning and decision-making capabilities. The system uniquely integrates a dynamic perception and prediction tool library and cognitive memory to achieve a more humane and anthropomorphic driving process.\n-\tThis paper is well structured and highly readable.\n-\tIt surpasses existing autonomous driving methods, showing over 30% collision improvements in motion planning, and excels in few-shot learning on the nuScenes benchmark.", "reasons_to_reject": "-\tUsing LLMs as agents to achieve reasoning and memory functions is not a unique design of this work. Previous research, such as the DiLu [1] framework, has also implemented similar concepts. \n[1] Wen L, Fu D, Li X, et al. Dilu: A knowledge-driven approach to autonomous driving with large language models. ICLR, 2024.\n-\tAlthough the paper proposes a novel paradigm for autonomous driving, the inherent latency issue due to reliance on API calls to LLM is still a concern. The author also mentioned this in the limitation, which is worthy of recognition. The delay issue is directly related to the real-time applicability of this method in actual driving scenarios.", "questions_to_authors": "Have the authors considered choosing a loadable LLM model such as Vicuna instead of calling the LLM Api?", "extra_scores": {}}]

Author rebuttal:

Meta-review: A well-done application paper that applies end-to-end LLM agents to autonomous driving, which all the reviewers unanimously liked. The paper is well-written and easy to follow, with clear diagrams. The technical execution regarding the LLM agent is solid, and the experimentation is comprehensive. Agent-Driver's unique integration of a dynamic tool library, cognitive memory, and a sophisticated reasoning engine distinguishes it from prior works. This would be a great addition to the conference.

Pros:
- Novel framework integrating LLM with traditional autonomous driving components.
- Strong performance on open-loop and closed-loop driving challenges including the challenging nuScenes benchmark

Cons:
- Lack of evaluation on language-related benchmarks like BDD-X.
- Need for clearer explanation of the differences between Agent-Driver's reasoning engine and other implementations (e.g., GPT-Driver).
Decision: Accept