Trajectory-LLM: A Language-based Data Generator for Trajectory Prediction in Autonomous Driving
Authors: Kairui Yang
Zihao Guo
Gengjie Lin
Haotian Dong
Zhao Huang
Yipeng Wu
Die Zuo
Jibin Peng
Ziyuan Zhong
Xin WANG
Qing Guo
Xiaosong Jia
Junchi Yan
Di Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Trajectory Prediction
Large Language Model
Autonomous Driving
Primary area: generative models

Abstract: Vehicle trajectory prediction is a crucial aspect of autonomous driving, which requires extensive trajectory data to train prediction models to understand the complex, varied, and unpredictable patterns of vehicular interactions. However, acquiring real-world data is expensive, so we advocate using Large Language Models (LLMs) to generate abundant and realistic trajectories of interacting vehicles efficiently. These models rely on textual descriptions of vehicle-to-vehicle interactions on a map to produce the trajectories. We introduce Trajectory-LLM (Traj-LLM), a new approach that takes brief descriptions of vehicular interactions as input and generates corresponding trajectories. Unlike language-based approaches that translate text directly to trajectories, Traj-LLM uses reasonable driving behaviors to align the vehicle trajectories with the text. This results in an "interaction-behavior-trajectory" translation process. We have also created a new dataset, Language-to-Trajectory (L2T), which includes 240K textual descriptions of vehicle interactions and behaviors, each paired with corresponding map topologies and vehicle trajectory segments. By leveraging the L2T dataset, Traj-LLM can adapt interactive trajectories to diverse map topologies. Furthermore, Traj-LLM generates additional data that enhances downstream prediction models, leading to consistent performance improvements across public benchmarks. The source code is released at https://github.com/TJU-IDVLab/Traj-LLM.

Published as a conference paper at ICLR 2025
TRAJECTORY-LLM: A LANGUAGE-BASED DATA GEN- ERATOR FOR TRAJECTORY PREDICTION IN AU- TONOMOUS DRIVING
Kairui Yang! Zihao Guo Gengjie Lin²
Haotian Dong!
Zhao Huang!
Yipeng Wu Die Zuo
Jibin Peng'
Ziyuan Zhong
Xin Wang*
Qing Guo
Xiaosong Jia? Junchi Yan?
Di Lin
"Tianjin University *Shanghai Jiaotong University
*Columbia University
"The Hong Kong Polytechnic University IIHPC and CFAR, Agency for Science, Technology and Research, Singapore
ABSTRACT
Vehicle trajectory prediction is crucial aspect of autonomous driving, which re- quires extensive trajectory data to train prediction models to understand the com- plex, varied, and unpredictable patterns of vehicular interactions. However, quiring real-world data expensive, so we advocate using Large Language Mod- els (LLMs) to generate abundant and realistic trajectories of interacting vehicles efficiently. These models rely on textual descriptions of vehicle-to-vede vehicll inter- actions map produce the trajectories. We introduce Trajectory-LLMM (Traj- LLM), a new approach that takes brief descriptions of vehicular interactions as in- put and generates corresponding trajectories. Unlike language- that translate text directly to trajectories, Traj-LLM uses reasonable driving behav- iors to align the vehicle trajectories with the text. This results an "interaction- behavior-trajectory" translation process. We have also created new dataset, Language-to- teeguagge--Trajectorry (L2T), which includes 240K textual descriptions of ve- hicle interactions and behaviors, each paired with corresponding map topologies and vehicle trajectory segments. By leveraging the 1,T dataset, Traj-LLMM can adapt interactive trajectories to diverse map topologies. Furthermore, Traj-LLM generates additional data that enhances downstream prediction models, leading consistent performance improvements across public benchmarks. The source /ttpp:://githbb.co
approaches
code is released at httpt:ts/////mm /gggghitth..oooom com JU- IDVLab TailLab/Traj-aane LLM
INTRODUCTION
Accurately predicting the trajectory of vehicles is crucial aspect of autonomous driving systems. A vast amount data regarding vehicle trajectories is required to train the trajectory prediction models. However, collecting such data from real-world scenarios requires considerable manual effort and resources. This results in significant costs associated with capturing vehicle trajectories that exhibit intense interactions like overtaking, yielding, and bypassing.
The autonomous driving industry has developed a solution of using traffic-flow generators to syn- thesize trajectories and enhance training and testing data. An excellent generator should generate much synthetic data with minimal human operation. Previous works (Li et al. 20233 Feng et al.) 2024) have shown the effectiveness of combining real-world and synthetic data for training pre- diction models. Popular generators like LGSVL (Rong et al., 2020) have graphical interfaces that enable users to create and control vehicle trajectories and interaction relationships on a map through and-dradddropp operations. However, even single vehicle's 's precise editing often requires tens of drag-and-drop operations. Other generators like Scenic (Fremont et al.. (023) provide pro- grammable interfaces that allow users to control vehicle motion using functional parameters such as position, velocity, and yaw. However, they require users to have excellent programming skills to utilize codes to depict complex vehicle interactions.
The recent literature advocates a more user-friendly generator with language interfaces built on top
of the large language models (LLMs) (Zhong et al. 2023a Tan al. 2023 Ding al. 2024
"Co-first authors.
'Co-corresponding authors.

---

Published as a conference paper at ICLR 2025
20km/h
20km/h
Cur A can
Car can
130km/h
40km/h
only change lanes from the left side.
Change lanes fo-
om both sides,
(a) Dangerous Speed
(b) Safe Speed
(c) Narrow Road
(d) Wide Road
Figure I: (a) Dangerous speeding of Car A for overtaking a slow Car B is unusual, which may lead to a collision with Car C. (b) A reasonable speed of Car for overtaking a slow Car B follows a safer driving logic. Without the guidance of driving logic, language interface that has only seen trajectories of left turns in narrow road scenes like (c) may find it difficult to generate trajectories such as left and right turns in wide road scenes like (d).
Mao al. 2023 Zhao al. 2024), which translates the brief text of vehiele interactions to the translation). The brevity of the text precludes
trajectories in the map (a.k.a., the specifie parameters regarding vehicle motions, rendering it highly abstract description of ve- hicle interactions. Therefore, users can efficiently generate many vehicle trajectories that align with the interaction described by the text. This elficiency motivates us to study the vehicle trajectory generator based on the large language model for saving data collection efforts. The vehicle trajectories can be viewed as a collection of waypoints, with each waypoint associated with the spe- cific motion parameters of the vehicle it passes through. These trajectories concretely describe the vehicle interaction. Language interfaces utilize the map's 's environmental information and vehi- cles" partial trajectories guidance generate the complete trajectories aligned with the text. This approach is similar to trajectory prediction models, but these two tasks fundamentally differ. While trajectory prediction models aim to predict relatively deterministic vehicle paths, using language interfaces may generate many trajectories that align with the interaction described in the short text.
However, when delving deeper into the vehicle trajectories generated by the language interface, many of them show unreasonable driving behaviors (see Figure []]. In contrast to the brief text of vehicle interaction and the concrete parameters of vehicle motion, the driving behaviors (e.g.. change lane, cruise, accelerate/dbcellratee and stop) reflect the inherent logic of human beings or autonomous driving systems that dynamically adjust the vehicle motions based on the changing environment. In Figure [a-b),, given the text description of "Car A overtakes a very slow Car B", the trajectory of Car in (a) represents a dangerous behavior of speeding, while in (b) Car A overtakes B at a safe speed. This leads to a lack of realism in the generated data. Furthermore, without reasonable driving behaviors, the language interface may become accustomed to generating trajectories like those in the known scenarios rather than the novel ones in the unseen scenarios, thus lowering the data diversity.
Figure [cgure(cdd), given the text of "Car bypasses a stationary Car B", Car A in (c) can only change lanes from the left side to bypass B a narrow road after extensive training. but failing to generate a legal right-hand bypassing on an unseen wide road in (d). Thus, the primary objective of this paper is to involve the human driving logic to guide the translation between the text description of vehicle interactions and trajectories.
This paper reformulates the "interaction-trapeee trajectorry translation by the language interface to the translation by adding the driving behaviors in-between to guide the generation of vehicle trajectories. We propose a Trajectory-LLM (Traj-LLM), language- based generator for producing vehicle trajectories. As illustrated in Figure 2 Traj-LLM divides the Setructioonbbhvviio "rajecoortr translation into two stages. At the first stage (see Figure 2(a)), Traj-LLM takes input as the text of interaction between multiple vehicles and the map with lanes, outputting the driving behaviors of each vehicle. Chronologically. we organize the driving behaviors of each vehicle into text sequence. Each driving behavior is followed by text that describes the logic behind this behavior, in the format like "Change Lane: There are approximately meters of road width on the left, while the road width on the right is insufficient, therefore choosing to change to the left lane" and "Change Lane: Both the left and right sides have ample road widths, allowing for a lane change to either the left the right". At the second stage (see Figure 2())), Traj -LM fuses the vehicle interactions and behaviors in the text with the map information to output specific motion parameters for each vehicle's trajectory.
2

---

Published as a conference paper at ICLR 2025
We introduce dataset, named Language-to-Trajette (L2T), to aid in the training of Traj-LLM LLM and associated research efforts. L2T comprises vehicle trajectories derived from 240K traffic sce- marios, encompassing six diverse road topologies such straightway, bend, roundabout, cross/T- shaped/Y-shaped intersection. Skilled drivers carefully craft these intricate trajectories and utilize textual segments to capture the vehicle interactions and behaviors within each scenario. We engage our drivers to provide annotations regarding vehicle interactions and behaviors for these scenarios.
Traj-LLM can generate realistic, diverse, and interactive vehicle trajectories based brief texts. These generated trajectories can be utilized for training trajectory prediction models, significantly improving their performances. We summarize our contributions as follows:
We advocate a new paradigm of the language interface with the "interaction- trajectory" translation generate trajectories coherent with the human driving logic.
We propose novel language interface named Traj-LLM, vehicle trajectory generator based on the large language model. Furthermore, we collect a new L2T dataset containing 240K traffic scenarios with vehicles' interactive trajectories. This dataset also contains rich text descriptions of vehicle behaviors and interactions for training Traj-LLMMss "interaction- behavior-trajectory" translation.
Traj- Mraj-LLM trained on the L2T dataset generate vehicle trajectories additional data for training trajectory prediction models, whose performances are effectively improved on the public Wayme and Argoverse datasets. These results can further inspire the relevant research on the language-basee trajectory generator.
2 RELATED WORK
We survey three groups of generators, which are equipped with the graphical, programmable, and language interfaces to control the vehicle interactions and generate the trajectories.
Graphical Interface Generators with graphical interfaces can be divided into Logsim and World- sim. Logsim uses the real scenes provide complex traffic situations. The scenes provided by Logsim are unchangeable, thus it fails to test the object interactions specified by different users. Worldsim allows users to design the scenes for testing the autonomous vehicles. These genera- tors (Shah et al.) 2018: Samak et al. 2023] Dosovitskiy et al., 2017 Rong et al. [020 (iilvera et al.) 2022) employ the game engines like Unity 3D that have the graphic renders of the realistic scenes obeying the real-world rules. Nevertheless, Worldsim costs expensive labor to operate the graphical interface to control every vehicle to create interactions. In contrast, the language interface relies on the short text to efficiently create the object interactions.
Programmable Interface Compared the graphical interface that partially omits the details of ve- hicle motions to simplify the generator, the programmable interfaces (Zhao et al. 2024 Yang et al.) 2023) offer an extensive collection of controlling functions of the vehicle interactions. Scenic (Fre- mont et al.) 2023) is domain-specifec languages for describing the vehicle behaviors. Yet, they need significant amount of code to build a road scene with complex vehicle interactions. The
Reinforcement learning (Rempe al., 2023
Janner et al.
2022 Li al. 2019 Shah et al.) 2018 2020 Amini et al.) 2020; Zhong et al,
Dosovitskiy et al., 2017, Zhang et al.,
2021 Rong et al.
2023b) allows the traffic rules, which also belong to the formal language. guide the generation of the specific vehicle interactions without requiring intricate coding. The traffic rules help to construct the reward function that drives reinforcement learning. The limited number of traffic rules not only constrain the driving behavior of each vehicle, but fail to express the logic of making any behavior. This logic important for teaching vehicles to navigate different maps, executing maneuvers such as lane changes and merges safely.
In this work, we introduce driving logic applicable to different map topologies through text. These logics can guide LLM to generate reasonable vehicle trajectories in various map topologies, thereby increasing the diversity of generated data. Diverse data aids in training downstream trajectory pre- diction models, enhancing prediction accuracy in complex scenarios.
hutss//unity.com
3

---

Published as a conference paper at ICLR 2025
(3)
Interaction mecraction-Behavior
(b) Behavior- Bchavior-Trajcetory
Initial
Fx
RLA
RLA
F
Trajectory T
Trajectories X
Behaviors B
RLA
CCar
34,5
FM
Change Lane: There are
speed: 12 mls yaw: 0"
approximately 5 meters..
F
Polyline
[Car B]
Map M
Encoder P
Stop: Prepare for chang- ing lames the right... Keep Stil: Wait for the the right pass...
D
H
Cur bypusses Car Car A overtakes Car C.
[Car C]
Cruise: There vehi- ele shead,allowed to...
Interactions
LLM
LIM T
F Initnaaaal Trajectory Feature
F ar Map Feature
F,/F, Interaction Feature
F Fu/FF Behavior Fcature
Car
Frozen
(e: Text Embedding
D
Decoder
Random Locality Attention
- Regression Hcad
The two stages of the "interaction- heweiiion--ehviio----eee -iraavvviioorrjcoorr"" translation. (a) We employ LLM with the random locality attention to translate the textual description of vehicle interactions into the behavior of each vehicle. Each behavior is associated with the underlying logic. (b) Given the vehicle interactions and behaviors, LLM translates them the sequential motion parameters that represent the trajectory of each vehicle. We illustrate the random locality attention in Figure
Figure 2:
Language Interface The recent LLMs (Devlin al.) 2019, Stiemnon et al. 2020 Brown et al.
2020
Touvron et al. 2023 Bai et al.
2023) enable the language interface to create virtual scenes
with complicated vehicle interactions. Though people can use the text-to-vides methods (Blattmann)
et al.
2023
Wu et al. [023 Ho et al.
2022 Voleti et al., 2022) and traffic generation methods (Park
ct al.
2023
Zhong ct al. 2023a b) Achiam et al. 2023] Vemprala et al. 2024] Cui et al. 2024 Wen
al.
2023
Chen al. 2024 Mao al.) 2023 Sha al.) 2023 Jin al. 2023)
design the
object interactions, they may fail to produce the usual vehicle motions like those in the real world.
CTG (Zhong et al. al.] (023b) and CTG++ (Zhong al. 2023a) depend the traffic rules generate realistic motions for the individual objects, yet lack a practical scheme for producing the complex vehicle interactions.
In contrast to the existing language interfaces, we propose a two-stage translation from vehicle interactions and behaviors, trajectories. Real driving logic guides the entire translation process, resulting in a high realism of the generated vehicle trajectories. Additionally, we propose random locality attention that effectively utilizes information from vehicle interactions to guide the trajectory generation, thus enabling the generated trajectories to exhibit complex interactions.
TRAJECTORY-LLM
We present the pipeline of using Traj-LLM to conduct the "interaction serrriio-bbhhaiittrraettttto transla- tion. providess an overview of Traj-LLLM, which is divided into stages (see Interaction- Behavior Translation in Figure a) and Behavior- Dehavior-Trraeccory Translation in Figure bb))
Interaction- DiteractioonBehavior Translation At the first stage illustrated in Figure Traj-LLM translates the text of vehicle interactions to behaviors. We denote the text of vehicle interactions as I, which records O pairs of vehicle-to-vehicle interaction between N vehicles. Each pair of interactions is documented in text like "Car A overtakes Car B" and "Car B bypasses Car C". We keep these texts short to enhance the convenience of using Traj LLM. We feed K text segments of into LLM T, which outputs the text of the behaviors B for N vehicles. We document the behaviors of each vehicle like "[Car A] Change Lane: Both the left and right sides have ample road widths..."
We feed the text interactions I into the text embedding layer & of LLM T to compute the interaction feature F, E OxC in Eq. (T). C' counts the feature channels. The map M E EDxxU in the traffic scenario is represented by E polylines with points in each polyline, and U denotes the number of attributes for each point. We input map M and the initial trajectories X (Sxx))
4

---

Published as a conference paper at ICLR 2025
of N vehicles to polyline encoder P in (Shi al. 2022 trajectories Fx E RNCC and the map feature F F'M E
Gao et al. 2020 to compute the initial
pxx as:
F E(), Fx- - P(X), FMr P Each initial trajectory includes S waypoints, and each waypoint is associated with four motion parameters (i.e., s/yy-corddinates, heading, and speed). E' denotes the number of polylines in the map M. The initial trajectories and the map together represent the spatial configuration of vehicles.
To ensure the consistency between vehicle interaction and map for preventing invalid situations such vehicles moving beyond the map boundaries, propose random locality attention between the 3). the initial trajectories X, and the specific map M as:
interaction I (see Figure
FJ.EX
F7.EX VC FE E
x sofftoo
S -AXI-.... So.k = =top (S
VC
F7-EM M
VC M )
So.1 IIM
So.K J-M
= =opp
S/M (2)
E
VC
S1+X E HOXN and S E ROXE represent the correlation scores between O pairs of vehicle interaction, N initial trajectories and E' polylines in the map M. In Eq. ( topx operation selects the top-K largest scores from the set of correlation scores. The selected scores used to weight the initial trajectory feature Fx and map features F Fas, which are added to the feature F, as:
K So.k Cat
F = F7
F + af STEM E Fir,
afmxx ~ NNO, 1). (3)
k=1
By setting K < N and E, we let the fea- ture F₁ E ROxC capture the locality corre- lation between the interaction and the spatial configuration of the map. This locality enables each pair of vehicle interactions to occur at the most reasonable locations on the map. Besides, at ate,, M are random variables obeying the normal distribution, which jitters the corre- 11+x:$I++M feature F7 E R represent the tth interaction, whose location has some jitter within a range.
gal
8 FeuX
topx
Softmax
5 /exX
S Pe..
output
feature
F
S Pe
F&
lation scores 11+x; I++M E R. They let the
FV
~ Mitter~N(00
Figure 3: Illustration of the random locality atten- tion. Here, the attention between O inter- action features {F) 0 = 1, I.... O} and initial trajectory features (F%
Given the interaction feature F,, LLM T uses the regression head to output N text seg- ments in B. We denote this regression head as H (query, key, value), which has cross- attention architecture with tunable parameters denoted as:
= 1, .... N)
ample. This illustration applies to the attention formulated in Eqs. (2) and (6).
B H(B o
LA = CCreC.rBBB B).
(4)
In Eq. means the fulle-connected layers with non-linear activation. The regression head fol- lows the autoregressive style, utilizing the interaction feature F, and the already output behavioral text B to update the final behavioral text B. Here, the already output text B queries the contex- tual information within the interaction feature F,, which helps to output vehicle behaviors aligned with the interactions. During the training phase, we use the cross-entropy loss Lg to penalize the difference between the regressed segments in B and the ground-truth segments in B.
Behavior- Trajectoy Translation At the second stage illustrated in Figure 2bb), Traj-LLM trans- lates the textual description of vehicle behaviors in to trajectories T N XXXxx4) for N vehicles. We convert numerical data of trajectories into text for training and inference of L.M.. Again, each trajectory includes S waypoints with four motion parameters.
Figure (bb illustrates the behavior-trajedt trajectory translation. We input the text segment of the behaviors B into LLM T, and text embedding layer & computes the behavior feature FB E IR XC as:
FB &(B).
(5)
5

---

Published as a conference paper at ICLR 2025
+
Dicycle
Vipp
Cross Enterscation
Band
Y-huped Imerrection
T-lleed Entersetioon
Stnightwy
Roundebont
(h) Road Topologies
Pedestrian
Trattic Cone
R: 20.5 123
sped a0 3at
(a) Obstracle
waypoinn
Keep Still
Cruise
Change Lane
Stop
Obsinele waypoint
Thrme
These other vahile iiusi off soad, and soad is wide streght ned. the ovenal ansing condition sre good, suitable for spealing up.
There appentinaall 5 neten of med with on the left, while the nexl eith the right inerfficent ther- fixe cleosing charge to the lose fhere are to other cars rroud.
vetridiee ciricing amees!
Shortest interactoon
polotiiass Sust, st besserm waz for while.
the intosectios 20 meten alead.
distance less than 10 m
there so other lane diive is,
nocoary stop and muke way fir all vchidea the antemection
Waypoutt parameters
(c) Vehicle Behaviors
y:
and wait St
Bypass: : Car A bypasses Trafiic Cone B.
Overtake: Car overtakes Car C.
Yield Car yields Pedestrian D.
(d) Vehicle Interactions
Figure 4: (a) We prepare four kinds of objects (i.e., vehicle, pedestrian, bicycle, and traffic cone). (b) There are six kinds of road topologies, where the road shapes are different. (c) Each object can take the behaviors of changing lanes, cruising, stopping, and keeping still. (d) Bypassing the static object, overtaking and yielding the dynamic object are the interactions in the L2T dataset.
We employ the random locality attention to capture the correlation between the behavior feature FB and the interaction feature Fy. computing the correlation scores $ S E NxO as:
F F1 VC
FR. PP VC
S K = =top
B so ftmar
(
(6)
where K <. We use the correlation scores SBet to weight the interaction feature F, as:
K
F = F
Σ
a B
Sp Bost
Fl a ~ N(O.
where we sample the random variable a from the normal distribution. In Eq. 7 we compute the behavior feature Fy RC for the th vehicle. The random locality attention focuses the behavior feature Fn on the most relevant interaction features and the map information contained within them. The random variable ate also jitters the behavior feature F allowing Traj-LLM generate more diverse trajectories aligned with the interactions and the map.
We feed the behavior feature F NxC N vehicles to the tunable head H(qurey, key, value) of LLM T to generate the trajectories in T as:
T (FFT, (FB)), = (T.
where is the already-outputeed trajectories. It queries the behavior information from Fn during the generation of the trajectories in T. We minimize the cross-entropy loss Ly to penalize the difference between the generated trajectories T and the ground-truth trajectories T.
LANGUAGE-TO-
DATASET
Considering the high risk of traffic accidents associated with collecting vehicle trajectories with high-intensity interactions on real roads, we opt to simulate the interactive trajectories in the virtual world. We construct simulated environment. The L2T dataset contains 240K road scenes, where the object classes, road topologies, and interactions illustrated in Figure 10. We divide the L.2T dataset into training and
simulator to increase the realism of vehicle trajectories collected in the

---

Published as a conference paper at ICLR 2025
testing sets, which contain 200K and 40K scenes, respectively. Below we briefty provide the basic information of the L2T dataset. More information can be found in the App A.
Object Classes Each object can be taken from the classes of vehicle, pedestrian, bicycle, and traffic cone (see Figure 10(a),, which are widely seen in reality. Expect traffic cones to always be static; other objects can be stationary moving. These objects may be of different sizes in road scenes. In this paper, we only use the bird's- bird's-eye view projection of objects to determine their sizes.
Road Topologies We prepare six typical kinds of road topology, including straightway, bend, roundabout, www.T-ssaaed---saappeeeee intersection (see Figure 10(b)). Each kind of road topology has variant shapes and lanes in the training and testing sets. A set of 2D coordinates represents the boundary of each lane. We use the XODR file to store each road topology with multiple lanes.
Vehicle Behaviors Each vehicle has the behaviors of changing lanes, cruising, keeping still, and stopping. Cruising or changing lanes means moving along the same lane or across lanes. We limit the velocity during vehicle cruising to 120km/h. Keeping still/stopping temporary/permanen. As illustrated in Figure 10(c), we document a vehicle's 's behaviors sequentially. We recruit group of drivers with rich experience to explain and document the logic behind each behavior.
Vehicle Interactions A vehicle may interact with 1~5 static/moving objects. In contrast to the behavior of a single object, the interaction defined here captures the relationship between vehicle and an object. The possible interactions present in the L,T dataset are illustrated in Figure 10(d). The vehicle can bypass a static object to avoid collision. It can also overtake or yield the moving object. During overtaking or yielding, the vehicle occupies lane sooner later than the interactive object. We focus on the short-range interactions between objects. These high- high-intensity interactions should be completed in a short range of less than 10 meters, where the vehicle must take prompt action to avoid collision with the objects. The recruited drivers document each interaction in text. We employ two-level check evaluate the quality of annotations. Firstly, rule-based script automatically examines whether the interactions match the annotated content. Secondly, the results of the automated check are randomly grouped, and a fixed-proportion random check is conducted by drivers. More details can be found in the App A.
Vehicle Trajectories Every moving object has a trajectory, which consists of series of waypoints arranged chronologically. As illustrated in Figure 10(c-d), we associate each waypoint with four motion parameters (i.e., x/y-coordinates, heading, and speed). The duration of all the trajectories we provide is within 20 seconds and the sampling frequency is 20 Hz.
5
EXPERIMENTAL RESULTS
5.1
REALISM, DIVERSITY, AND CONTROLLABILITY
We use 7lama-7B (Touvron et al. 2023) as the language model used in Traj-LLM, as it is among the most popular open-source large language models and demonstrates superior performance in understanding traffic scenarios. For more implementation details of Traj-LLL,, see App C. Below, we evaluate the realism and diversity of the trajectories generated by various methods. We also compare the control capability of different methods for trajectories to verify if these methods can generate the expected trajectories according to the interactions specified in the text.
The compared methods differ in their control conditions for generating vehicle trajectories. For fair comparison. all methods utilize the L2T for training and testing. SNet (Bergamini et al.. 202 and TSim (Suo et al. 202I) the BEV road topologies generate trajectories. We train BITS (Xu et al. 2023) and CTG (Zhong et al., 2023b) on the trajectories from 1.2T, relying on rule-based conditions for trajectory generation as in (Xu al. 2023 Zhong et al. 20235). CTG++ (Zhong ct al. 2023a), LGen (Tan et al. 2023). and Traj-LLM utilize text of interaction as the condition.
we report the agent- and scene-level (Zhong et 2023b realism of the generated trajectories. The agent-level realism measures the distribution distances of longi- tudinal acceleration magnitude (LO), lateral acceleration magnitude (LA), jerk (JE), and yaw rate (YR), and their average (AVG), between each agent's ''s generated and ground-truth trajectories. The scene-level realism is assessed by the relative scores of LO, LA, JE, YR, and AVG. More details about calculating these metries can be found in App B. To compute relative score, we gauge the distribution distance between each pair of vehicles' generated/ground evaluate the difference between the above two distances the relative score.
Realism In
trajectories. Then, we
7

---

Published as a conference paper at ICLR 2025
We compare Traj-LLM with state-of-the-art methods on the L2T dataset. The results measure the realism and diversity of the trajectories generated by various methods. The realism score has been multiplied by 100. J/f means smaller/larger value represents better performance.
Table 1:
Realism
Diversity
Method
Agent
Scene
Agent
Scene
LO
LA
JEL
YR, AVG
LOL
LA1
JEL
YR!
AVG
MD1
AD1
FD1
WD1
SNet
14.20
9.84
8.38
7.92
10.09
11.28
4.78
2.76 8.72
6.88
0.00 0.00
0.00
0.00
Image
TSim
25.36
21.78
5.33
3.65
14.03
15.94
4.60
10.67
7.91
9.78
0.52
0.25 0.30
2.92
BITS
10.16
5.82
9.23
4.30
7.38
10.57
1.70
3.54
4.75
5.14
6.80
3.22
6.98
7.27
Rule
CTG
8.26
7.33
19.92
1.68
9.30
10.40
6.14
10.10
3.64
7.57
6.49
3.44
6.03
5.70
LGen
7,55
9.73
13.09
1.91
8.07
9.59
7.89
10.21
4.10
7.95
3,38
1.74
2.19
4.02
Text
CTG++
6.13
7.13
21.19
1.13
8.90
7.94
4.16
10.37
2.22
6.17
6.38
3.55
6.08
5.79
Traj-LLM
2.56
3.17
0.74
0.28
1.69
0.73
2.68 0.52 0.54
1.12
14.91
6.81
13.81
Traj-LLMM surpasses other methods terms of realism at both the agent and scene levels. Traj-LLM introduces the interaction-behavia translation. Through learning from the real data, the short text of the vehicle interactions is translated into the behaviors with driving logic. makes each vehicle's trajectory consistent with the human driving habits, generating more natural trajectories. Since each scene contains multiple vehicles, the entire scene also made more realistic.
0-0-17s
12
t=0-17s
1-
1- 8
== 12s
Car
time [s]
17
Figure 5: Diverse trajectories generated by Traj-LLM from the same initial positions.
Diversity In Table I "Diversity", measure the agent-level diversity Zhang et al. 2022) by reporting the map-aware average self distance (MD), average self distance (AD), and final self dis- tance (FD). We measure the scene-level diversity (Xu al. 2023) by reporting Wasserstein distance (WD). To evaluate the agent-level diversity of trajectories, we conduct multiple experiments with the same initial conditions (history trajectories. rules, images) and evaluate metrics for the same vehicle in every two experiments. MD/AD denotes the average L2 distance between the most di- vergent/similar trajectories. FD assesses the average L.2 distance among the final position of the most similar trajectories. To evaluate the scene-level diversity, we compute the generated trajectory distribution on the map. Then, we calculate the density histogram of the distribution as the density profile. We measure the average WD between pairwise density profiles across multiple experiments.
Traj-LLM achieves better trajectory diversity, as it employs the random locality attention to the behavior-trajectory translation. It dynamically focuses the road topology, the driving status of neighbor vehicles, and the history trajectories. We jitter the parameters of the driving trajectories, further enriching the diversity of generated trajectories. We visualize the generated trajectories in Figure demonstrating that Traj-LLM generates more diverse trajectories.
Controllability In Table 2 we compare Traj- -LM with state-of-the-an methods that also rely on text to generate interactive vehicle trajectories, in terms of the scene-level controllability. We conduct multiple experiments for each vehicle and record the success rate of achieving the specified
8

---

Published as a conference paper at ICLR 2025
0-175
CIG
1-0-17s
(-63 1-83
10s
Traj-LLM
Car
time [s]
17
Figure 6: Controllability of CTG++ and LLaj-LLM. In this example, Traj-LLM successfully control Car to overtake Car I, while the state-of-the-art CTG++ fails in this case.
interaction and adhering to traffic rules. We calculate the average success rate of different vehicles having various interactions
multiple experiments.
Traj-LLM achieves a higher success rate be- cause it introduces the two-stage translation that generates realistic driving logic. This trans- lation incorporates the random locality atten- tion to enhance the realism and diversity of the generated trajectories. Solely relying on the short text, Traj-LLM presents stronger control over the interactions than other methods (see Figure 6). Traj-LLM hopefully shows stronger controllability with more detailed control in- structions.
Table 2: Comparison of interaction success rates %) with text-based generation methods.
Method
CTG++
LCTGen
Traj-LLM
Overtake
65.2
36.9
80.1
Bypass
91.8
54.5
81.3
Yield
77.7
45.5
83.5
Avg
78.2
45.6
81.6
5.2 ABLATION STUDY
In Table 3 we conduct the ablation study to measure the realism and diversity of the trajectories and Behavior- Trajectory" translation. We use the AVG. WD and success rate for measuring the scene-level re- alism, diversity, and controllability. Without the driving behaviors that reflect the inherent logic of human driving, I-T produces inferior results (see the first row of Table 3 compared to I-B-T.
the controllability LLM. We denote I-T/I-B-T as "Interaction- (rajectory"/IIntraattti "Interaction-Traeccttr"""
Table 3: I-T/IBB-T means "Interaction- iitteraatioo-Trrjetttrrrr// Ben
translation.
The results measure the realism, diversity and controllability (%).
Translation
Logic
Random
Locality
Realism
Diversityt
Control
I-T
14.52
14.39
36.7
3.08
22.64
48.2
4.81
9.66
62.1
I-B-T
1.45
17.26
58.3
2.60
12.55
67.9
1.12
13.81
81.6
In Table "I-B-T,, we list the critical modules of I-B-T translation, i.e., the driving-logic learning (Logic), random jitter (Random), and locality attention (Locality). Without Logic in the second row, LLM to generate the temporal sequence of behaviors only for each vehicle. This de- grades realism, diversity, and controllability, demonstrating the importance of driving-logic learn- ing. Based on Logic, remove Random Locality (see the third to fifth rows). Especially,

---

Published as a conference paper at ICLR 2025
the alternative without Random and Locality degrades the random locality attention to a vanilla cross-attention (the third row). Compared to these alternatives, the complete Traj-LLM shows bet- ter performance. This is because Locality enables the vehicles to generate reasonable trajectories aligned with the interactions and the map, while Random makes the generated trajectories more diverse. Note that Locality also offers efficient constraints and affects diversity. The fifth row moves Random but diversity doesn't 't drop much compared with the last row, while in the third and fourth rows, when Locality is removed, Random significantly impacts diversity.
5.3 GENERATED DATA FOR TRAJECTORY PREDICTION
In Table 4 we evaluate the effectiveness of Traj-LLMM, by using the generated vehicle trajectories to train downstream trajectory prediction models, i.c., MTR (Shi et al. 2022) and HDGT (Jia et al.) 2023). Apart from the model training and validating on the same dataset, we follow the protocol of the cross coos--ataseet validation in (Torralba & Efros] 2011), where we train and test each prediction model on different datasets. The cross-datasee validation helps to evaluate the robustness of the pre- diction model trained and tested on various data distributions. We utilize two parts of trajectory data from the training set of WOMD (Ettinger et al., (02I) (WOMD (train)) and the trajectories gen- erated by Traj-LLe MtMj-LLM (Traj-LLM (L2T)), where Traj-LLM is trained on 200K scenarios of the L2T dataset. Each part of the training data contains 400K scenes. Each scene includes 2-8 vehicles with various interactions. The trained models are evaluated on the validation sets of WOMD (WOMD (val)) and L,2T (L22 (val)), each containing about 40K scenes. We report the trajectory prediction results using mAP, minADE, and Miss Rate, as defined in WOMD.
Table Results of trajectory prediction.
Train
Test (mAP 1, minADE J Miss Rate L)
WOMD (train)
Traj-LLM (L2T))
WOMD (val)
L2T (val)
MTR
HDGT
MTR
HDGT
0.405, 0.650, 0. 0.1
0.272, 0.604, 0.151
0.211, 0.921, 0.267
0.134, 1.674, 0.491
0.106, 1.583, 0.459
0. 041 1.290, 0.434
0.340, 0.662, 0.154
0. 196, 0.968, 0.280
0.416, 0.633, 0.139
0.294, 0.578, 0.148
0.384, 0.620, 0.145
0.243, 0.775, 0.194
By training MTR/HDGT on Traj-LLM (L2T) and validating it across on WOMD (val) (see blue cells in the second row), we find performance degradation compared to the prediction models trained on WOMD (Train) (see the blue cells in the first row). This is because different trajectory distributions exit in WOMD (Train) and the generated data of Traj-LLM (L2T). A large portion of the trajectories in Traj-LLM (L22T) represent the dense interactions between vehicles, whereas WOMD (Train) provides many vehicles without interaction. We also find similar degradation in the purple cells of the first and second rows when training the prediction models on WOMD (Train) and validating it on L2T (test).
By integrating the data of WOMD (Train) and the generated trajectories in Traj-LLM (L2T), MTR and HDGT outperform (see the last row of Table their counterparts trained on a single dataset. The quantitative and qualitative results demonstrate that the trajectories generated by Traj-LLM exhibit remarkable realism and diversity, thus benefiting downstream prediction models. Furthermore, we conduct ablation study of generated data and L2T under fixed / non-fixed total dataset size in App D.
6 CONCLUSION
The precision prediction of vehicle trajectories holds significance in autonomous driving. To train accurate trajectory prediction models, substantial quantity of data of vehicle trajectories impera- tive. The recent development of generative artificial intelligence makes it possible to produce a large amount of trajectory data for training the trajectory prediction models at the cost of less human ef- fort. This paper proposes a trajectory generator, Traj-LLM, which relies on the short text description of vehicle interaction to produce the interactive trajectories efficiently. We utilize the large language model to build Traj-LLM. In contrast to the language-basee based generators for "interactionn-rraectoryy translation, Traj-LLM conducts the erful capability in understanding the interactions between vehicles and outputting their behaviors and driving logics. This capability allows Traj-LLM to produce realistic and diverse trajectory data for training downstream trajectory prediction models. We also contribute the L2T dataset, which contains the interactive trajectories associated with the text descriptions of vehicle interactions, be- haviors, and driving logic, to train Traj-LLM and promote relevant research in the future.
where it possesses a pow-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their constructive suggestions. This work is supported by the National Natural Science Foundation of China (No. 62476192), Natural Science Foundation of Tianjin (No. 23ICQNIC02010), the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative (No. DTC-RGC-04). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.
REFERENCES
Josh Achiam, Steven Adler. Sandhimi Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman. Shyamal Anadkat, et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.06774, 2023.
Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Karaman, and Daniela Rus. Learning Robust Control Policies End-to-End Autonomous Driving From Data-Driven Simulation. IEEE Robotics and Automation Letters, 5(2): 144--110,11,,, 2020.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen Technical Report, arXiv preprint arXiv:2309. /6609, 2023.
Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blažej Osiński, Hugo Grim- mett, and Peter Ondruska. SimNet: Learning Reactive Self-driving Simulations from Real-world Observa- tions. In 202/ IEEE International Conference on Robotics and Automation, pp. 51 19-5552.. IEEE, 2021.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis. Align Your Latents: High- High-Resolution Video Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 22563-22575, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nec- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33: 33:18777-2 877-1901, 2020.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund. and Jamie Shotton. Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving. 2024 IEEE International Conference on Robotics and Automation, pp. 14093- 14100. IEEE, 2024.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Drive You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vchicles. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 902-909, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-411444 1771-4166 2019.
Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, and Marco Pavone. RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios. In European Conference Computer Vision, pp. 93-110. Springer. 2024.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An Open Urban Driving Simulator. In Conference on Robot Learning, pp. 1-16. PMLR, 2017.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R Qi. Yin Zhou, et al. Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9710-9719, 2021.
Lan Feng, Mohammadhossein Bahari, Kaouther Messaoud Ben Amor, Éloi Zablocki, Matthieu Cord, and Alexandre Alahi. UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction. In European Conference on Computer Vision, pp. 106-123. Springer, 2024.

---

Published as a conference paper at ICLR 2025
Daniel Fremont, Edward Kim, Tommaso Dreossi, Shromona Ghosh, Xiangyu Yue, Alberto L Sangiovanni- Vincentelli, and Sanjit A Seshia. Scenic: A language scenario specification and data generation. Machine Learning 11 (12 -804349930 2023.
Jiyang Gao, Chen Sun. Hang Zhao, Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vec- torNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 11525- 11925-11533 11533, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video Diffusion Models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations, 1(2):3 2022.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with Diffusion for Flexible Behavior Synthesis. International Conference on Machine Learning, 162:9902 01099902-9911.... 2022.
Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. HDGT: Heterogeneous Driv- ing Graph Transformer Multi-Agent Trajectory Prediction via Scene Encoding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1 18:10380-13875 2023.
Ye Jin, Xiaoxi Shen, Huiling Peng. Xiaoan Liu, Jingli Qin, Jiayang Li, Jintao Xie. Peizhong Gao, Guyue Zhou, and Jiangtao Gong. SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model. arXiv preprint arXiv: 13/93, 5(7):8, 2023.
Quanyi Li, Zhenghao Mark Peng, Lan Feng, Zhizheng Liu, Chenda Duan, Wenjie Mo, and Bolci Zhou. See- narioNet: Open-Source Source Platform for Large- Scale Traffic Scenario Simulation and Modeling, Advances in Neural Information Processing Systems, 36:3894 3920, 2023.
Weizi Li, David Wolinski, and Ming C ADAPS: Autonomous Driving Via Principled Simulations. In 2019 International Conference on Robotics and Automation, pp. 7625-7631. IEEE, 2019.
Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang. GPT-Driver: Learning to Drive with GPT, arXiv preprint 2023.
Joon Sung Park. Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bern- stcin. Generative agents: Interactive simulacra of human behavior, In Proceedings of the 36th Annual Acm Symposiun on User Interface Software and Technology, pp. 1-22, 2023.
Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or I Litany. Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13756-13766 2023.
Guodong Rong, Byung Hyun Shin, Hadi Tabatabace, Qiang Lu, Steve Lemke, Märtinš Možeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, al. LGSVL Simulator: A High Fidelity Simulator for Au- tonomous Driving. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems, pp. 1-6. IEEE, 2020.
Tanmay Samak, Chinmay Samak, Sivanathan Kandhasamy, Venkat Krovi, and Ming Xic. AutoDRIVE: A Comprchensive, Flexible and Integrated Digital Twin Ecosystem for Autonomous Driving Research & Edu- cation. Robotics, 12(3):77, 2023.
Hao Sha, Yao Mu. Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving. arXiv preprint 23 2023.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. In Field and Service Robotics: Results of the 11th International Conference, pp. 621-666666666 635. Springer, 2018.
Shaoshuai Shi, Jiang, Dengxin Dai, and Bernt Schiele. Motion Transformer with Global Intention Localiza- tion and Local Movement Refinement. Advances in Neural Information Processing Systems, 35:6531-6543, 2022.
Gustavo Silvera, Abhijat Biswas, and Henny Admoni. DReyeVR: Democratizing Virtual Reality Driving Sim- ulation for Behavioural & Interaction Research. In 2022 17th ACM/IEEF International Conference on Human- Human-Robot Robot Interaction, pp. 639-643. IEEE, 2022.
12

---

Published as a conference paper at ICLR 2025
Nisan Stiemnon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Infor- mation Processing Systems, 1-33088-32111 2020.
Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. TrafficSim: Learning To Simulate Realistic Multi-Agent Behaviors. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10400-10409, 2021.
Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Krähenbühl. Language Conditioned Traffic Generation. In Conference on Robot Learning, volume 229, pp. 2714 2752, 2023.
Antonio Torralba and Alcxci A Efros. Unbiased Look at Dataset Bias, In Proceedings of the 1EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1521-1528. IEEE, 2011.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. 1..00.....99 13971, 2023.
Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. ChatGPT for Robotics: Design Principles and Model Abilities. IEEE Access, 12:55682 55696, 2024.
Vikram Voleti, Alexia aaicceur-Martineau. and Chris Pal. MCVD Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation. Advances in Neural Information Processing Systems, 35:23371- 23385, 2022.
Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai. Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. DiLar A Knowledge-Drivve Driven Approach Autonomous Driving with Large Language Models. International Conference on Learning Representations, 2023.
Jay Zhangjic Wo, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufci Shi. Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune- A- Video: One-Shor Tuning of Image Diffusion Models for Text- to- Video Generation. In Proceedings of the IEEEJCVF International Conference on Computer Vision, pp-
7589-7599, 2023.
Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. BITS: Bi-level Imitation for Traffic Simulation. In 2023 IEEE International Conference on Robotics and Automation, pp. 2929-2936. IEEE, 2023.
Kairui Yang, Zihao Guo, Gengjie Lin, Haotian Dong, Die 7no, Jibin Peng, Zhao Huang, Zhecheng Xu, Fupeng Li, Ziyun Bai, et al. Natural-language-dee Object Interactions in Virtual Road Scenes. arXiv preprint 231.v2331..00000 2023.
Simulation Benchmark and Copilot for Efficient Production of
Qichao Zhang, Yinfeng Gao, Yikang Zhang, Youtian Guo, Dawei Ding, Yunpeng Wang, Peng Sun, and Dong- bin Zhao. TrajGen: Generating Realistic and Diverse Trajectories With Reactive and Feasible Agent Be- haviors for Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems, (3(1):244474- 24487, 2022.
Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. End-to-End Urban Driving by Imitating a Reinforcement Learning Coach. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15222-15232, 2021.
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation. arXiv preprint arXiv:2403.06845 2024.
Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic. Yulong Cao. Danfei Xu. Marco Pavone, and Guided Traffic Simulation via Scene- Diffusion. In Conference Robot Learning, pp. 144 177, PMLR, 2023a.
Baishakhi Ray. Language- Language-Guided
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided Conditional Diffusion for Controllable Traffic Simulation. In 2023 IEEE International Conference on Robotics and Automation, pp. 3560 05600 IEEE, 2023b.
Zikang Zhou, Jianping Wang, Yung- Hui Li. Yu-Kai Huang. Query-Centte Trajectory Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17863-17873, 2023.
13


Reviews:
[{"review_id": "3YDoJhv3DA", "reviewer": "Reviewer_baVM", "summary": "This paper leverages large language models (LLMs) to generate trajectories for autonomous driving by introducing an interaction-behavior-trajectory translation approach, rather than the commonly used interaction-trajectory translation in other LLM-based methods. Their approach consists of two stages: in the first, vehicle interaction descriptions are translated into behaviors, and in the second, these behaviors are converted into specific motion parameters to create trajectories. Additionally, the authors developed a new Language-to-Trajectory (L2T) dataset. Their method demonstrates promising performance on both the L2T and Waymo Motion Prediction datasets, outperforming state-of-the-art methods.", "questions": "The clarity of the paper can be improved. In the introduction, the motivation and objectives should be clearly stated, along with a summary of their accomplishments.\n\nIt is unclear how the trajectories are generated. Are they based on real data, generated through a simulator, or did the authors develop their own simulator?\n\nThe methodology section is difficult to follow and needs rewriting for better clarity.\n\nThe paper does not specify what kind of input was used to train the LLM model. Did they use generated prompts, or did they provide feature embeddings?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. They have published a dataset , Language-to-Trajectory (L2T), which includes 240K textual descriptions of vehicle interactions and behaviours.", "weaknesses": "The clarity of the paper can be improved. In the introduction, the motivation and objectives should be clearly stated, along with a summary of their accomplishments.\n\nIt is unclear how the trajectories are generated. Are they based on real data, generated through a simulator, or did the authors develop their own simulator?\n\nThe methodology section is difficult to follow and needs rewriting for better clarity.\n\nThe paper does not specify what kind of input was used to train the LLM model. Did they use generated prompts, or did they provide feature embeddings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ImVmBhR7Ku", "reviewer": "Reviewer_9AVa", "summary": "This paper adopts large language models (LLMs) to generate the trajectories for autonomous driving. In this regard, they explore the interaction-behavior-trajectory translation process in contrast to only interaction-trajectory, which is widely used for other LLMs-based approaches. Their method comprises two stages, where in the first stage, the textual descriptions of vehicle interaction are translated to behaviors, and in the second stage, those behaviors are converted into specific motion parameters for trajectories. Based on this, the authors have also created a new Language-to-Trajectory (L2T) dataset. The proposed method has shown promising results on L2T and the Waymo Motion Prediction datasets compared to state-of-the-art methods.", "questions": "However, the paper has shown better performances in the evaluation, yet some comments need to be addressed:\n1. I acknowledge that the authors have put great effort into creating the dataset, but it was unclear whether they have used existing datasets, for instance, Waymo and Nuscenes, to build their dataset. If this is not the case, and they adopt some manual feedback from the driver, how can they evaluate those behavior's annotations? \n2. The authors have experimented with the proposed method in open-loop settings; it would be good to see how the proposed method performs in the closed-loop evaluation. \n3. It is good to see that authors have also dedicated a section in supplementary material for the limitation of their work, but it would be good to see what the future directions will be based on their work. \n4. In the paper, the author has used random locality attention; it would be good to do an ablation study on its effect on driving behavior and also the trajectories. Also, if another form of attention, for instance, cross attention, is used, what will affect behavior and trajectories?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Here are the strengths of this paper:\n1. The inclusion of driving behaviors for trajectory prediction is promising.\n2. Another promising aspect of this paper is the creation of a new benchmark dataset, L2T.\n3. Overall, the presentation quality of the paper is good. However, some grammatical mistakes can be corrected afterwards.", "weaknesses": "However, the paper has shown better performances in the evaluation, yet some comments need to be addressed:\n1. I acknowledge that the authors have put great effort into creating the dataset, but it was unclear whether they have used existing datasets, for instance, Waymo and Nuscenes, to build their dataset. If this is not the case, and they adopt some manual feedback from the driver, how can they evaluate those behavior's annotations? \n2. The authors have experimented with the proposed method in open-loop settings; it would be good to see how the proposed method performs in the closed-loop evaluation. \n3. It is good to see that authors have also dedicated a section in supplementary material for the limitation of their work, but it would be good to see what the future directions will be based on their work. \n4. In the paper, the author has used random locality attention; it would be good to do an ablation study on its effect on driving behavior and also the trajectories. Also, if another form of attention, for instance, cross attention, is used, what will affect behavior and trajectories?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EQojPQKrBV", "reviewer": "Reviewer_v7Vc", "summary": "This paper introduces Trajectory-LLM (Traj-LLM), which leverages Large Language Models to generate realistic vehicle trajectories from textual descriptions. The method employs an \"interaction-behavior-trajectory\" process to translate vehicle interactions into trajectories, ensuring the realism and controllability of the trajectories.", "questions": "Please see above sections.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Well-written paper. Easy to follow.\n* The proposed \"interaction-behavior-trajectory\" process to guide the trajectory generation can help ensure better realism and controllability. \n* The author contributed a new dataset L2T.", "weaknesses": "* The proposed dataset L2T is collected in simulation platform and manually use the wheels to drive and collect the data. The realism of the proposed dataset is questionable. Further justification and motivations are needed for creating a new dataset with good realism instead of curating existing open real-world dataset to create such paired dataset. \n* The main benchmark is performed on the proposed L2T and open dataset WOMD is only used in ablation study while  existing literature mainly compared on other open datasets. Without benchmarking on the same open dataset, it would be hard to justify the technical contribution of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hDgTuJurR4", "reviewer": "Reviewer_qEvw", "summary": "This paper presents a method for generating vehicle trajectories in various traffic scenarios using large language models, employing a hierarchical generation strategy of 'interaction-behavior-trajectory.' Additionally, this work creates the L2T dataset to train the proposed model. Experimental results show that the motion prediction model trained with the generated data performs better on the mAP metric compared to the model trained solely on WOMD, providing preliminary validation of the method's effectiveness.", "questions": "1. Can the proposed model be used for simulating traffic scene? I suggest the authors consider this application area, rather than limiting the focus solely to trajectory prediction tasks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organized and easy to read.\n2. The proposed method may help alleviate the reliance of existing motion prediction models on large amounts of real data, facilitating the development of these models.", "weaknesses": "1. I did not see which specific large language model was used; the only information provided in the system diagram shows the LLM has an encoder-decoder structure, which does not seem to align with GPT-like models. I urge the authors to elaborate on the design of the large language model.\n2. As a generative model designed for trajectory prediction tasks, the experimental section is insufficiently thorough. Firstly, it only selects the WOMD dataset and does not test on other representative datasets, such as Argoverse. Secondly, it only uses mAP as the metric, neglecting other common metrics such as distance error and miss rate. Therefore, the generalizability of the proposed method remains questionable.\n3. Training a generative model on a third-party dataset and then using the generated data for model training—does this yield effects equivalent to training on the original third-party dataset? I recommend the authors include an ablation study comparing the experimental results of training with the original dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)