Language-Driven Interactive Traffic Trajectory Generation
Authors: Junkai XIA
Chenxin Xu
Qingyao Xu
Yanfeng Wang
Siheng Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Trajectory generation
Interaction
Language control
Primary area: generative_models
TL;DR: In this work, we propose InteractTraj, the first language-driven traffic trajectory generator that can generate interactive traffic trajectories.

Abstract: Realistic trajectory generation with natural language control is pivotal for advancing autonomous vehicle technology. However, previous methods focus on individual traffic participant trajectory generation, thus failing to account for the complexity of interactive traffic dynamics. In this work, we propose InteractTraj, the first language-driven traffic trajectory generator that can generate interactive traffic trajectories. InteractTraj interprets abstract trajectory descriptions into concrete formatted interaction-aware numerical codes and learns a mapping between these formatted codes and the final interactive trajectories. To interpret language descriptions, we propose a language-to-code encoder with a novel interaction-aware encoding strategy. To produce interactive traffic trajectories, we propose a code-to-trajectory decoder with interaction-aware feature aggregation that synergizes vehicle interactions with the environmental map and the vehicle moves. Extensive experiments show our method demonstrates superior performance over previous SoTA methods, offering a more realistic generation of interactive traffic trajectories with high controllability via diverse natural language commands.

Language-Driven Interactive Traffic Trajectory Generation
Junkai 3.3+
Chenxin Xu
Qingyao Xu 1.3
Yanfeng Wang
Siheng Chen 1,2,3
Shanghai Jiao Tong University SShaggaii AI Laboratory
Multi-Agent Governance & Intelligence Crew (MAGIC)
Abstract
Realistic trajectory generation with natural language control pivotal for advancing autonomous vehicle technology. However, previous methods focus on individual traffic participant trajectory generation, thus failing to account for the complexity of interactive traffic dynamics. In this work, we propose InteractTrrj. the first language-driven traffic trajectory generator that can generate interactive traffic trajectories. InteractTrra interprets abstract trajectory descriptions into concrete formatted interaction-awant numerical codes and Iearns a mapping between these formatted codes and the final interactive trajectories. To interpret language descriptions, we propose language-to-code encoder with a novel interaction-awar encoding strategy. To produce interactive traffic trajectories, we propose code- to-tra trojerejccrry decoder with interaction-awan feature aggregation that synergizes vehicle interactions with the environmental map and the vehicle moves. Extensive experiments show our method demonstrates superior performance over previous SoTA methods, offering more realistic generation of interactive traffic trajectories with high controllability via diverse natural language commands. Our code is available
com X1a- wtt/:/ooo///i//ieniiiiee jtaikk/Inntttttt
Introduction
Driving simulations are increasingly vital in the development of autonomous driving 22 5 By projecting real-world scenarios into virtual environments, driving simulation enables the generation of driving data in diverse conditions at significantly reduced cost, especially in safety-critical scenarios. Trajectory data, representing the driving behaviors of traffic vehicles, serves as a key part of the driving simulation. This paper focuses on the generation of traffic trajectories.
One of the most critical aspects of trajectory generation is controllability, which involves generating highly realistic trajectory data tailored to specific user needs. Several works have been proposed to prompt controllable traffic trajectory generation. TrafficGen [ enables the generation of traffic scenarios conditioned on blank map with controllable vehicle numbers. CTG [6] allows users to control desired properties of trajectories using signal temporal logic test time like reaching goal or following a speed limit by a guide sampling in the diffusion process. However, given that these control signals are pre-defined, their fiexibility is inherently limited.
With the rise large language models, researchers have begun to use human natural language to achieve a more flexible and user-friendly control. A representative work is LCTGen 77] which leverages a large language model to transform text descriptions into structured representations, followed by transformer-basee decoder to generate corresponding scenarios. However, this work exclusively focuses on individual traffic participant trajectory generation, disregarding the interactions between multiple trajectories. Such interactions, crucial for replicating the dynamic, involve the
"These authors contributed equally to this work. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Language description
InteractTraj
trajectories
valialo cwethiis the sificle weitdls tim #dtnesttinn
Overtaking
were
Inceraction Code
Speed Orienationn
Language- Coco
Jchicle.Cods
Code-to-
Traeetory
Enooder
Map Code
Decoder
Intersection
Figure I: Overview of InteractTraj. InteractTraj uses series of semantic interaction aware numerical codes to depict interactive trajectories. An LLM-based language-to codeee encoder converts language descriptions into numerical codes, which are then transformed into interactive trajectories by code-to-trajectory decoder.
complex interplay between various participants' movements and decisions. The absence of interaction modeling causes limited controllability in trajectory generation. For instance, traffic jams, involving many vehicles, cannot be accurately generated. Yet, generating these scenarios is essential as they highlight the vehicles' capabilities to respond to real-world challenges.
To achieve more realistic and controllable trajectory generations, in this work, we propose InteractTraj. novel generator that generates interactive traffic trajectories from natural language descriptions. The key design rationale InteractTraj is to interpret abstract trajectory descriptions into concrete formatted interaction-awar numerical codes and to learn mapping between these formatted codes and the final interactive trajectories. Specifically, InteractTraa consists two modules: an LLM- based language-to to-eeeeetoeeeeeeeeeeeeeeeeeeeeee encoder and code-to- -otajeettrr decoder. 1) To interpret user language commands, the language-to-con encoder utilizes a novel interaction-awart encoding strategy, which LLM with interaction-aware prompts to convert language commands into three types of numerical codes, including interaction, vehicle and map codes. As the core to model interactive relationships of vehicles, the interaction codes consist of key factors, including relative position and relative distance. These factors are designed to be discrete to have semantic meanings that correspond LLM and are formed in series model the temporal continuity of interactions. 2) To produce interactive traffic trajectories, the cote-to-trajectory decoder employs a novel two-step interaction-awarr aggregation strategy that integrates code information. This approach synergizes vehicle interactions with environmental map data, thereby using these interactions to enrich the realism and coherence of vehicle trajectories. Compared to previous work [7]. which generates independent traffic trajectories, InteractTraj is capable of generating realistic and interactive traffic trajectories with enhanced controllability through language commands.
We conduct extensive experiments on the Waymo Open Motion Dataset (WOM)) and nuPlan and show that Interac(Traj can generate realistic interactive traffic trajectories with high controllability through various natural languages, Our method achieves SoTA performance with an improvement of 15.4%/18.7%% on average ADE/FDE over previous methods WOMD, and 17.1 %...0.4%% on average ADE/FDE on nuPlan. Our method also achieves a more realistic generation under different user commands including vehicle interactive actions of overtaking, merging, yielding and following. We conduct user studies showing our method has 47.5% higher average user preference compared to the baseline method. We summarize our contributions as follows:
We propose InteractTraj, the first language-driven traffic trajectory generator that can generate interactive traffic trajectories. The core idea of Interac(Traj is to bridge abstract trajectory descriptions and generated trajectories with formatted interaction aware numerical codes.
We design a novel interaction interpretation mechanism with LLM in the language-to-cne -o-cede encoder and two-step feature aggregation to fuse interaction information for more coherent generation in the code-to-trajectory decoder.
We conduct extensive experiments and show that InteractTraj is capable of generating realistic interactive traffic trajectories with high controllability through various natural languages.
2
Related Work
2.1
Traffic Trajectory Generation
Traffic trajectory generation is crucial in intelligent transportation systems, producing all agents' trajectories scene from provided maps or historical data. Traditionally, rule-based methods [8] [
2
employed heuristic models to encode traffic rules like lane-keeping and following the leading vehicle but lack diversity and realism due to fixed rule patterns. Recently, learning-based have emerged to generate more realistic traffic trajectories by learning from
10]
3
methods 12
I3
[4] [5]

---

real-world data. However, these methods usually face challenges with controllability, unable to fulfill specific requirements like instructing vehicle turn left, and they rely on past trajectories that are expensive and difficult to obtain. There is growing interest in controllable trajectory generation [4]
16 7], focusing on customizing trajectories to meet diverse user requirements. TrafficGen generates specific number of vehicles and their trajectories on blank map. CTG [6] uses loss function to guide trajectory generation according to user controls. influenced by LLMs, language- driven traffic scenario generation is emerging. CTG++ [16] employs LLM to convert user queries into a loss function for realistic, controllable generation. While CTG and CTG++ require costly past trajectories, limiting their practical deployment, LCTGen (7] generates scenarios purely from language descriptions using an LLM-based interpreter and a transformer based generator. However, these methods lack interaction awareness and struggle with complex text descriptions. Our approach addresses these issues with interaction-aware code representation and refined vehicle behavior control.
2.2 Motion Prediction
Motion prediction and trajectory generation are closely related concepts in the field of autonomous systems and robotics since both approaches aim to anticipate the future state of agents. Motion prediction models are often used as backbones to convert latent states into agent trajectories as part of the generated scenarios. allowing for the simulation of realistic traffic scenarios. Early methods [17 19] utilize various physics-based kinematic models for modeling agent behaviors and predicting trajectories. With the development of deep learning and neural networks, RNN and CeTT-bbbsed structures are applied for trajectory prediction [20 21
due to their proficiency in processing sequential data. To handle more complex trajectory prediction tasks where multiple agents are involved, models in recent years have also incorporated methods such as diffusion
22
23
24
32 33 achieve better results multi-agent motion prediction tasks by focusing on interaction and relational reasoning. Trajectory generation facilitates the creation of realistic scenarios, acting as supplementary data for the development and evaluation of prediction models.
or transformer [25 26, (7 28 to achieve more accurate results. [29 30 3]
2.3 Large Language Models and Their Multimodal Applications
Recent years have seen dramatic advancement in the development of Large Language Models (LLMs) 34
35
36
37
38
39
40] such ChatGPT [41 and GPT-4 (38] The success of LLMs
triggers a boom in multimodal tasks that require comprehensive understanding across multiple modalities, including text 42 so on. Notable examples including DALL-E [56] and Sora [57]. DALL-E (56) treats text and image tokens as unified data stream, generating realistic images from text input. Sora [57] demonstrates the ability to create long. realistic, and imaginative videos from text descriptions. In this work, we
43
44
45
46), audio 47
48 49 50,, motion [51 53] 54 55 and
focus on language-driven trajectory generation. Inspired by (7). our method uses GPT-4 (38) as the language encoder to leverage the deep traffic scene understanding and reasoning capabilities of LLMs.. We design an interaction-awarc code and prompt GPT-4 to convert language input into these codes, which contain detailed information about interactions, vehicles and map.
3 Problem Statement
Language- -rriven traffic trajectory generation aims to create realistic trajectories of traffic participants over period of time according to language descriptions. Given language description L, our goal is
to propose scenario generation model (-)) so that the generated corresponding traffic trajectories
S = G(L) are realistic and match with the language description. Here S [S,, S SN E
N. xxxxxx represents the trajectory of N vehicles over T timesteps, where S, = [s] s
E
RTX Vi (I,...,NN) NJ. and of E R² denotes the 2D positions of vehicle i the (-th timestep.
4 Methodology
4.1
Architecture Overview
InteractTraj is language-gun guided interactive traffic trajectory generation framework that generates realistic vehicle trajectories based on natural language descriptions. The core idea of Interact is to use a series of semantic numerical codes to depict interactive trajectories and learn a transformation between these codes and the interactive trajectories, see Figure ] for a sketch. InteractTrra consists of two parts: an M-beed language-to-code encoder and a code-to-trajectory decoder. The language- to-code encoder is designed to interpret language commands and turn the commands into three types
3

---

Tock Rode conner AnsAe with desorit
Query: one gence ihw vori
Summary
emegeency [V7)
the (V1) Grleme
Meriing
code lend buses
Explanation:
[aaea]] intercd folloing belber
vistiwe ditance dredien
II lego ai) lo all zero since itss ago oor
tivent eoplavation)
ovantiec andther vhiii cirecty thee the variide novingtten slow
Sentlle
V1 (ego vehidde) i the ownter and moves fonwen.
12 jomergeree velided to y..
invthwe can rakingg tii occute
Meaning Webice Code stitbue
LLM
Output:
[Tuine] Term
Encure the code each vericle has a lorth of
likidee Code atricty hillowing rules:
-m: 0.0 0.0.0 (8.0 0. 0. C" I. 3 MI [3
Code:
Merriag the Map Ooode at buses
Maa prompt
VT. 11 0.O.O.4.4 4 4.4.4 5.
127 D.
[Rake]] Generas the Mao Codetal oving the rulrs balowe
th weeeee Laniing lette wyg.
14u: 3.
Figure 2: Sketch of interaction -ware prompt and numerical codes.
of numerical codes, including interaction codes, vehicle codes and map codes. The code-to-trajectory decoder then transforms these codes to produce interactive traffic trajectories.
Mathematically, given the language description I., InteractTraa generates the traffic trajectories by
m, V, I = E(L), $ = D(m, V.I),
(1)
where E() represents the language- -oddee encoder and D(-) represents the tode-to-trajectoyy
decoder. m is the map codes representing the environment map information, V is the vehicle codes representing the vehicle's individual driving information and I is the interaction codes representing the vehicle interaction information. We illustrate the detailed structures of these codes in the following.
4.2 Language-to-Code Encoder
The language-to-code module, E()). distills essential information from input natural language descriptions and transforms this information into interaction-awarr numerical codes. This transformation leverages large language models, such as GPT-4 (38). The whole language-to- code encoder incorporates two key designs: the structure of the interaction-awarc numerical codes and the tailored prompts for the large language model. The numerical codes are comprised of three components: interaction codes, vehicle codes, and map codes. To depict vehicle interactions concretely, we design the format of the interaction codes with the relative factors modeling. To descriptions into the code format, we design prompts with
interpret the abstract interaction- interaction descriptions to assist the LLM.
Interaction codes I. Interaction codes encapsulate vehicle interactive relationships. The core idea is that spatial relationships and changes between vehicles significantly affect their perception and reactions, revealing their interactions. To capture high-level actions and interaction tendencies, we resample the vehicle attributes at regular intervals across T' timesteps. We denote T as the set of timesteps of the resampling process. To effectively model these interactive relationships, the designed interaction codes consist of two key factors, relative distance and relative direction, motivated by the representation of polar coordinates. Formally, the interaction codes are denoted (i)j) de)) J.ET, where /d,, is the relative direction/distance of jth vehicle with the ego interacted vehicle at the 1-th sampling timestep. To enrich the interaction code with semantic meanings, we discretize the relative direction/distance. Specifically, we divide the surrounding space centered by ego vehicle into six regions: front, rear, left front, left rear, right front and right rear. The relative direction p of agent can thus be represented by the index of the region in which agent is located time t. The relative distance d' is also discretized by dividing with fixed interval and then rounding down. This discrete code representation with semantic meanings facilitates the use of LLM to link code values with corresponding language commands.
by I= [(p)
Vehicle codes V. The vehicle codes contain the information of vehicle individual driving states. To describe the vehicle driving states, the vehicle codes consist of two components diffferent trajectory scales: the global trend and the detailed movement. Formally, the vehicle codes are denoted by V = [r:; (1..e( where r; is the trajectory type of agent i modeling trajectory global trend and a; is the vehicle states of agent modeling detailed movement. Specifically, we categorize the trajectory type into stop, straight ahead, left turn, right turn, left change lane, and right change lane. r, is the category index. Trajectory states a; = [o;, q: [o]]er]]. contain the initial orientation o,, initial position gi, and the discrete speeds [us]eET of agent at sampled timesteps.
Map codes m. The map codes contain the information on key map features. We adopt the representation m 2 similar to 77), which represents the number of lanes in each of the four

---

Feature Extraction
Feature Aggregation
Generation Head
MILP
Inseraction Coces
Interciionn
Fouture
E
Inteactioon
M
Map Codes
Mop Lane Festure
Riy
Votidle
Fecture
weiile Fasture
Genersted Tejectory
-MIP
Velide Codes
VATE
Foutre
Iraectary
Figure 3: The architecture of code-to -ot-rreecttorr decoder. The decoder generates vehicle trajectories by fusing and decoding information between vehicles and interactions.
directions, the distance between ego vehicle and the intersection, and the lane index of the ego vehicle, respectively.
LLM prompts. Given language description L, we utilize carefully designed interaction- prompts to help the large language model analyze the descriptions and extract interact-related information, enabling to generate corresponding interaction-aware numerical codes that align with the language description. The prompt mainly incorporates three key components: 1) [interaction prompt] interaction prompt defines the format of interaction code, explains some key interaction events to help with the better understanding of the scenarios and informs LLM to interpret the possible interaction behaviors inferred in the descriptions and output the corresponding interaction codes / of all the vehicles involved by analyzing the vehicles" relative distance and position relationships. 2) [vehicle prompt] vehicle prompt defines the format of vehicle code and explains driving rules to make the generated scenarios more realistic, such as the need to slow down when turning, etc. 3) [map prompt] map prompt defines the format of map code and make LLM analyze whether intersections roundabouts are involved and decides the number of lanes of all directions according to the number vehicles and their orientations to fill out the map codes. Figure 2 presents a sketch of LLM prompts and an example of numerical codes and the appendix for full prompt.
4.3 Code-to-Trajectory Decoder
With the interaction-aware codes from the encoder, the code-to-trajectory decoder D(-) generates vehicle trajectories by aggregating and decoding information between vehicles and interactions. In the decoder, we propose key design of two-step interaction- feature aggregation, which synergizes vehicle interactions with environmental map data, thereby using these interactions to enrich the realism and coherence of vehicle trajectories.
Given the map codes m, the vehicle codes V and interaction codes I from the encoder, the overall decoding process can be formulated as
EM, Ev, Ej Fext (m, V,I), Ev (EM, Ev, E1), 3 Fread (Ev),
where Fext() (.) denotes a feature extraction module, Fage() denotes an attention-based feature aggregation module, Fhead (-) denotes the generation head module for obtaining vehicle attributes and trajectories, and EM, Ev, E, are the map lane features, vehicle features and the interaction features, respectively. Ey denotes the fused features for vehicles and S is the generated trajectory.
Feature extraction (.). The feature extraction module F-xi(-) transforms the numerical codes into initial embeddings for subsequent calculation. For map codes, we retrieve a map M that best fits map code from the pre-defined map dataset, which is the same as [7]. The map M E TR N consisting of N2 lanes with their NA attributes, is then passed to the multi-context gating(MCG) blocks (58) obtaining map features EM E ND by aggregating neighboring lane information, where Dr is the dimension of each lane feature. For the vehicle codes and interaction codes, we apply MLPs with position encoding layer for each to obtain their higher-dimensional latent features, that is, Ev = MLPv (PE(V)) E ND Ej = MLP, (PE(I)) E N x Dr, where PE(.) is the position encoding function, Dy and Dj the dimensions of extracted vehicle and interaction features.
Feature aggregation Fagg (.). The feature aggregation module aims to fuse the map features and interaction features into vehicle features for subsequent trajectory generation. Based the intuition that the vehicle interactions are constrained by the road structure and the vehicle states are affected by both the road structure and vehicle interactions, we apply a two-step feature aggregation strategy. First,
5

---

we fuse the map feature into the interaction feature and the vehicle feature respectively by multi-head cross-attention operations, that is. Ej MHATT, EMM EM). where MHATT( k, v)) denotes the multi-head cross-attention functionn with query q, key k, value v, Ej and Ey are the interaction features and the vehicle features after aggregation. Second, we fuse the interaction feature into the vehicle feature to obtain the final vehicle feature, that is, Ey MHATTy v (EVy, Ej, Ej). The final vehicle feature contains both the interaction and map information, which can be manipulated for further trajectory generation.
E M (Ev, E EM)-
Generation head (-). The generation head aims to generate vehicle" states and trajectories based on vehicle features. For agent with agent feature Ev and the trajectory type r; in the vehicle codes, we generate its trajectory positions S, through a series of MLP heads. For different trajectory types, we assign different MIP heads. Formally, the generation process is formulated
S; he.dne,,,r; (Ev.i).
where MLPPeed, denotes the assigned r,th heading MLP, We finally assemble all the trajectories S S1 S Sx] together with the map M as output for the scenario generation.
4.4 Training
Generating training samples. Due to the lack of data directly matching linguistic descriptions with traffic scenarios, we cannot directly optimize the model under ground-truth trajectory supervision using language inputs. As an alternative, we extract map, vehicle, and interaction codes directly from ground-truth trajectories to train the decoder's s scenario reduction capability, During the training process, for a ground-truth scenario S derived from real-world datasets, we re-generate the scene by
m, V,I= (S), $ = D(m, V.I).
(4)
where (-) extracts information from the ground-truth vehicle trajectories to fulfill the codes, including the obtainment and discretization of vehicle speeds, their positions and distances relative to the ego vehicle, and the classification of their trajectory types. The specific computational rules will be mentioned in the appendix. We thus train the decoder D(.) by minimizing the gap between S and S.
LOss. We apply MSE loss L-aa(-) minimize differences between generated and ground-truth vehicle trajectories. Furthermore, to enhance the network's 's sensitivity to trajectory interactions, we additionally supervise the relative distances with the ego vehicle among vehicle trajectories with another MSE loss Creta()) (.). For the ith vehicle, the relative distance at last timestep is d, = s?" - ss. Formally, the final loss of InteractTraj is presented
N
N
C
Σ
Cira((S S.) + Crata (di, di)
Σ
(5)
N
4.5 Discussion
Compared to previous representative traffic trajectory generation method, including CTG 6 CTG++ [16, TrafficGen [4] and LCTGen [.]. our method is the first language-conditee conditioned interactive trajectory generation method. (1) At the task level, CTG and CTG++ generate traffic trajectories within the need of vehicles' past trajectory observations. The necessity of collecting past trajectories significantly increases the data generation costs, imposing extra burden. TrafficGen generates traffic trajectories by only taking a map as input to produce a scenario, resulting in a lack of controllability over the generated trajectories. LCTGen and our methods are specifically designed to generate traffic trajectories based on language conditions, which not only achieves high controllability but also reduces dependency on extensive data sets. (2) Under the same task, compared LCTGen, our technical novelty comes from two aspects. First and foremost, we propose the interaction codes, corresponding LLM prompts, and interaction-awant feature aggregation which serve as the key to generating interaction-awant aware traffic trajectories. In contrast, LCTGen does not account for vehicle interactions during trajectory generation. Second, within the vehicle codes, we incorporate mixed- scale design that both addresses the global type and the detailed movement of vehicle trajectory, which allows the generated trajectories to align with high-level intentions as well as precise positional changes. Conversely, LCTGen only considers the local detailed movement, leading to potential discrepancies between the language descriptions and generation at the high-level trend, such as receiving descriptions to turn left but generating right turn.

---

Table 1: Evaluation on trajectory generation realism under WOMD and nuPlan datasets. indicates lower better. Interac(Traj significantly improves trajectory realism.
Dataset
Method
mADE
minADE
mFDE +
minFDE
SCR
HDD L
TrafficGen
9.531
1.440
20.106
3.690
0,086
5.733
LCTGen
1.262
0.224
2.696
0.463
0.072
1.295
WOMD
Inae I)
1.205
0.207
2.479
0.346
0.090
1,210
InteractTraa
1.067
0.181
2.190
0.320
0.070
1.076
TrafficGen
9.418
1.416
19.686
3.627
0,082
5,874
LCTGen
1.161
0.218
2.497
0.448
0.074
1,301
nuPlan
InteraceTraj(w/o I)
1.108
0.181
2.277
0.323
0.070
1.150
Interact'rraj
0.962
0.160
1.987
0.321
0.067
1.129
AGe tiltenn roz Irtanechive
3.6
136 LCTion vntencin)
of
2.a
Performancen Sranncemm mm
Performancemn
2.4
2.8
2.4
2.0
2.0
1.6
1.6
1.2
0.8
0.8
0.4
0.0
follow
merge Interaction Type
vleld
overtake
0.0
LC
AVG
Trajectory Type
(a) Performances under various interaction types.
(b) Performances under various trajectory types.
Figure 4: Comparison of model performances under different settings on WOMD. Lower better. InteractTraj generates more realistic interactive trajectories for different types. ST: straight forward, LT; left turn, RT: right turn, LC: left lane change, RC: right lane change and AVG: average performance.
5
Experiments
5.1 Dataset and Baseline
We two datasets, Waymo Open Motion Dataset (WOMD) [59 60 and nuPlan 661. which both provide real-world vehicle trajectories and corresponding lane maps. We compare our method against two state-of-the-art controllable trajectory generation baselines, TrafficGen [ and LCTGen [ Please refer to the appendix for details on the datasets and the choice of baselines.
5.2 Experimental Setup
In the language-to-code encoder, we sample the vehicles' trajectories at I-second (10 timesteps) intervals to get |T| = timesteps set. In the code-to- tot-eee-trrjrryyy decoder, the vehicle features Dy and interaction features Dj set to 256. During the training process, train the decoder using the AdamW optimizer [62] with an initial learning rate 3e See more details in the appendix.
5.3 Evaluation Metric
Given ground-truth trajectories, quantify the realism of generated trajectories with 6 metrics: 1) mean average displacement error(mADE); 2) minimum average displacement crror(minADE); 3) mean final displacement error(mFDE); 4) minimum final displacement error(minFDE): 5) scenario collision rate(SCR); 6) Hausdorff distance(HD). Detailed formulations of these metrics are provided in the appendix.
5.4 Reconstruction-bast Evaluation
Since the dataset contains only trajectories and not language-trajectee trajectrrrcrryyoe pairs, we evaluate our methods and baselines guantitatively through reconstruction approach. For all methods, generate conditional codes or inputs directly from the ground-truth trajectory instead of the LLM, and then reconstruct the trajectories to assess alignment with the input conditions.
Quantitative results on all scenarios. We first evaluate our generated trajectories by comparing them to ground-trum truth trajectories on the whole dataset. compares the performance of InteractTraj with two baseline methods on reconstruction. Since previous methods lack interaction-awar input design, we add one more ablated version of InteractTraj without the interaction code, to have comparison with the same input information, noted as InteractTraj(//o I). We see that i) our method significantly outperforms previous methods across all the metrics, indicating it generates more realistic

---

Overtaking
Merging
Yielding
Following
The sedan accelerates, overtaking the slower-moving truck on the right. as it navigates the curvy highwwa.
The school bus initates lsne change merge traffi.
The eyclist yiolds the furning the intersection.
One car turned right at intersectio, followed by another.
LCTGen
Ours
Figure 5: Comparison of model performances under different interaction types. InteractTraj generates
trajectories that better align with language descriptions by performing the right vehicle interactions.
Sconario type
average
Percentage
0.6
overtake
merge
ylold
others
merge 3048 overtate average Interaction Type
Percentage
(b) Percentage of users considering the generated scenarios the interaction types.
(a) Percentage of users' preference of generated trajectories description of different methods.
Figure 6: Results of the user study for overall and interaction performances.
scenarios with vehicle interactions. Specifically, our method reduces the mADE/mFDE/HD by 1..%%/1..7%/166.. compared to SoTA methods; ii) The ablated InteractTraj(wfoo I) still outperforms previous models, showing the effectiveness of our vehicle code design.
Quantitative results on scenarios on different interaction types. model generation on scenarios with trajectory interaction, we test the model on the representative interactive scenarios. The scenarios mainly categorized into four types according to vehicle interaction: overtaking, converging, yielding and following. Figure 44| shows the method comparison on all types of interactive scenarios with LCTGen. We see that for all types of interactions, InteractTrai significantly reduces the mADE and mFDE, showing a powerful capability to generate realistic interactive trajectories by interaction-awant coding design.
To evaluate the performance of
Quantitative results on scenarios on different trajectory types. To evaluate the performance of model generation on different individual driving trajectories, we categorize trajectories of the test set into six types according to individual actions: straight, stop, turn left, turn right, left change lanes, left, right change lanes, see detailed rules in the appendix. According to individual action types, we divide the test set and report the average reconstruction performance on every set. Figure [4b] shows that InteractTraj's 's generation results are closer to the ground truth than LCTGen, reflecting we generate trajectories more aligned with language descriptions across different individual driving actions.
It is important to note that the ultimate goal of our method is to generate traffic trajectories from language commands. Previous methods 7 4 exhibit limitations that they fail to address the interaction information encoded within languages. Consequently, in the reconstruction-basee evaluation, these methods inherently lack interaction information in their inputs. contrast, our method is capable of incorporating interaction during the generation process, which represents significant advantage. As a result, our inputs for reconstruction- based evaluation contain more comprchensive information, enabling more realistic and effective trajectory generation.
5.5 Language-Conditiones Evaluation
In this section, we compare end-to-end our method with previous methods by analyzing trajectories generated from language descriptions. Given the absence of specific ground-truth trajectories for certain language commands, we employ qualitative evaluation and user studies for assessment. Qualitative results. We compare our model to LCTGen, which also transforms language input into traffic scenarios. We evaluate our methods with that of baseline methods qualitatively given
8

---

(a) Three cars
(b) Several cars drive parallel move platoon each other.
(c) Surrounding vehicles pull over as the ambulance right turn at the approaches.
(d) Three cars
(c) Three cars
(f) A car is
lined up. The car
lined up. The turning left and car behind ego
formation.
behind makes a
moving
a
car makes a right turn at the intersection.
straight line.
intersection.
Figure 7: Visual analysis for model's performance when dealing with less common interaction types and language ambiguities.
language commands containing different interaction descriptions. Figure 5 provides visualizations of representative user language commands including four types of interaction: vehicle overtaking, vehicle merging, vehicle yielding, and vehicle following. We that compared to previous work, the scenarios generated by InteractTraj better align with language descriptions with the help of interaction-aware code representation, while the previous method not perform the corresponding interactive action since it generates trajectories of each agent independently.
Generalisation capability analysis. Our model demonstrates the ability to generate compliant scenarios even when handling less common or emergent interaction types as shown in the left part of Figure 7 We test interaction scenarios not mentioned explicitly in the prompt, including uncommon cases of parallel driving (Figure 7a), platooning (Figure 7b), and pulling over (Figure (c). The results show that method effectively interpret and translate these less common interaction types, producing scenarios that align with expected behaviors without retraining. This is primarily due to the robust generalization capacity of the LLM used in encoding process. The decoding process is also equipped with strong generalization abilities due to extensive training with massive numerical codes, which would translate these codes into trajectories and generate compliant scenarios.
In addressing the potential ambiguities in language descriptions, our approach leverages the reasoning capabilities of LLM without introducing additional prioritized proximity. Ambiguities typically arise in two main forms: unclear references to objects and contradictions within the language instructions.
1) For cases where the object reference is ambiguous, the LLM interprets the linguistic input and converts it into numerical codes that correspond to one of the plausible meanings. As illustrated in Figure 7d the LLM assigns the label "the car behind" to randomly selected vehicle. However, when the description is more specific, such as "the car behind ego car" (Figure 7e). the LLM accurately resolves the reference and appropriately handles the description.
2) For cases where there are self-contradictrry language requirements, the LLM generates scenarios that partially align with the instructions. For instance, as shown in Figure 7 the instruction presents an inherent contradiction. The LLM resolves this by prioritizing one part of the instruction while disregarding the other. To better tackle language ambiguity, potential solution involves introducing LLM-human interaction to iteratively verify language descriptions.
Controllability analysis. In Figure g it is evident that each generated scenario adheres to the provided linguistic descriptions when there are variations the details. This highlights the strong controllability of Interac(Traj in accurately manipulating vehicle behaviors, ensuring that the generated scenarios align with user-specified control commands. The precision is achieved through the LLM's robustness in understanding and applying detailed instructions based on different linguistic inputs. User study. We conduct two user studies on WOMD to qualitatively assess the language-condition traffic scenario generation capabilities of InteractTraj from two perspectives: 1) overall generation performance and 2) vehicle interaction performance. GPT-4 used to generate descriptions of different interaction types input language descriptions, see appendix for the details.
In the first user study, each user is given forty language commands and corresponding trajectories generated by models and is asked to choose the one trajectory that better fits the language description. Figure (6a shows the results of users' preferences for scenarios generated by LCTGen and InteractTraj. We see that i) for each interaction type, significantly more users prefer scenarios generated by InteractTraj over those produced by I.CTGen; ii) on average, 73.7% responses are more favorable to the scenarios generated by InteractTraj. and model achieves at least 66.9% support all

---

(a) Generate complex scenario on a two-way highway, while ego car driving straight forward.
(b) Generate more complex scenario on a two-way highway, while ego car making left- lane change..
Generate more complex scenario on two-way highway, while ego car is making right- lane change.
(d) Generate
complex scenario on a two-way highway. while ego car being overtaken.
Figure 8: Visualization results for model's controllability
Table 2: Ablation study proposed code and network designs evaluated on WOMD. All designs are beneficial.
Table 3: Ablation study on the granularity of the discretization of the interaction codes on WOMD.
Model IC RD RP
Lea
mADE J mFDE 4 SCR
Model Gap Areas
mADE 4 mFDE 4 SCR
(a)
V
1.205
2.479
0.346
(e)
10
6
1.087
2.228
0.074
(b)
V
1.167
2.442
0.084
(f)
20
6
1.117
2.299
0,070
(c)
1.194
2.475
0.080
(g)
15
4
1.237
2.810
0.071
(d)
1.165
2.446
0.080
(h)
15
8
1.069
2.190
0.071
Ours
V
1.067
2.190
0.070
Ours
15
6
1.067
2.190
0.070
sub-categories.
This reflects that InteractTraj has stronger capability at generating interactive trajectories than LCTGen, and excels in representing interaction aspects of language descriptions.
The second user study contains fifty questions covering different interaction types, and users are asked answer whether the scenarios generated fulfill the corresponding textual descriptions. The results are shown in Figure [bb We see that i) for each interaction type, significantly more users consider the scenarios generated by InteractTraj to fulfill the requirements given by language descriptions; ii) on average, 72.4% positive responses consider that InteractTraj generates scenarios with required interactions, while LCTGen only have 31.5% positive responses in average. This reflects that InteractTraj effectively extracts the interaction information in the descriptions, and generates sufficiently satisfying traffic scenarios.
5.6
Ablation Study
Effect of proposed code and network design. We conduct the ablation study based on the reconstruction evaluation to evaluate the effectiveness of proposed designs, including a) the addition of whole interaction codes (IC); b) the relative distance in interaction codes (RD); e) the relative position in interaction codes (RP); d) the relative distance loss Lra. Table 2 presents the results. We see that all designs are beneficial to a more realistic trajectory generation.
Effect of the setting of hyper- parameters. We conduct ablation study of the granularity of the discretization of the relative distances and relative positions, specifically including c), f) the interval gap used for discretizing relative distances; g). h) the number of areas used for discretizing relative distances, as shown in Table 3 We see that our current parameter choices achieve the best results.
Conclusion
We propose InteractTraj. a novel interaction-awarr soaguage-guides traffic scenario generation model. Our core idea is to convert language descriptions into multi-level codes and generate trajectories by attention- based information aggregation. Experiments show that InteractTraj effectively reproduces real-life scenario distribution and generates scenarios aligned with language description.
Limitations and future work. This work focuses on generating trajectories only vehicles and generation of maps is limited by the map library. In the future, plan extend the work more types traffic participants and more flexible map generation. We also plan to apply the generated traffic scenarios to the training of autonomous driving systems by expanding the motion dataset.
7 Acknowledgment
This research is supported by NSFC under Grant 62171276 and the Science and Technology Commission of Shanghai Municipal under Grant 21511 100900, 2251 106101, and 22D72229005. Special thanks to Lightwheel AI for the support.
10

---

References
[1] Joaqun Maroto, Eduardo Delso, Jess Felez, and Jose Ma Cabanellas. Real-time traffic simulation with microscopic model. IEEE Transactions Intelligent Transportation Systems, 7(4):513- 527, 2006.
[2] Jordi Casas, Jaime L Ferrer, David Garcia, Josep Perarnau, and Alex Torday. Traffic simulation with aimsun. Fundamentals of traffic simulation, pages 173-232, 2010.
[3] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[4] Lan Feng, Quanyi Li, Zhenghao Peng. Shuhan Tan, and Bolei Zhou. Trafficgen: Learning generate diverse and realistic traffic scenarios. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3567-3575. IEEE, 2023.
[5] Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu. Changxing Liu. Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative Ilm-agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
[6] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560-3566. IEEE, 2023.
[7] Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Krachenbuchl. Language conditioned traffic generation, arXiv preprint (tt::2307.00947,, 2023.
[8] Lcontios G Papalcondiou and Marios D Dikaiakos. Trafficmodeler: A graphical tool for programming microscopie traffic simulators through high-level abstractions. In VTC Spring 2009-IEEE 69th Vehicular Technology Conference, pages 1-5. IEEE, 2009.
[9] Jakob Erdmann. Sumo's 's lane-changing model. In Modeling Mobility with Open Data: 2nd SUMO Conference 2014 Berlin, Germany, May 15-16, 2014, pages 105-123. Springer, 2015.
[10] Martin Fellendorf and Peter Vortisc.. Microscopic traffic flow simulator vissim. Fundamentals of traffic simulation, pages 63-93, 2010.
[11] Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker- Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie WieBner. Microscopic traffic simulation using sumo. In 2018 2 international conference on intelligent transportation systems (ITSC), pages 2575-2582. IEEE, 2018.
[12] Shuhan Tan, Kelvin Wong, Shenlong Wang. Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 892-901 2021.
[13] Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10400-10409, 2021.
[14] Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: Bi-level imitation for traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2929-2936 2936. IEEE, 2023.
[15] Qiao Sun, Xin Huang, Brian C Williams, and Hang Zhao. Intersim: Interactive traffic simulation via explicit relation modeling. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 11416-1 1423. IEEE, 2022.
[16] Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovie, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guided traffic simulation via scene-level diffusion. arXiv preprint arXiv: 2306.064444 2023.

---

[17] Alexander Barth and Uwe Franke. Where will the oncoming vehicle be the next second? In 2008 IEEE Intelligent Vehicles Symposium, pages 1068-1073. IEEE, 2008.
[18) Panagiotis Lytrivis, George Thomaidis, and Angelos Amditis. Cooperative path prediction in vehicular environments. In 2008 1 11th International IEEE Conference on Intelligent Transportation Systems, pages 803-808. IEEF, 2008.
[19] Rajesh Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
[20] Chaofan Tao, Qinhong Jiang, Lixin Duan, and Ping Luo. Dynamic and static context-aware Istm for multi-agent motion prediction. In European Conference on Computer Vision, pages 547-563. Springer, 2020.
[21] Yonghwan Jeong, Seonwook Kim, and Kyongsu Yi. Surround vehicle motion prediction using Istm-rnn for motion planning of autonomous vehicles at multi-lane turn intersections. IEEE Open Journal of Intelligent Transportation Systems, 1:2-14, 2020.
[22] Florent Altché and Arnaud de La Fortelle. An lstm network for highway trajectory prediction, 2018.
[23] Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction using recurrent neural networks, 2017.
[24] Moritz Wolter and Angela Yao. Complex gated recurrent neural networks, 2018.
[25] Zikang Zhou, Luyao Yc, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8813 8823, 2022.
[26] Chiyu Max Jiang, Andre Cornman, Cheolho Park, Ben Sapp. Yin Zhou, and Dragomir Anguelov. Motiondiffuser: Controllable multi-agent motion prediction using diffusion, 2023.
[27] Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Multi-level motion attention for human motion prediction, 2021.
[28] German Barquero, Sergio Escalera, and Cristina Palmero. Belfusion: Latent diffusion for
behavior-driven human motion prediction, 2023.
[29] Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 6488 6498, 2022.
[30] Chenxin Xu, Yuxi Wei, Bohan Tang, Sheng Yin, Ya Zhang, and Siheng Chen. Dynamic-group- aware networks for multi-agent trajectory prediction with relational reasoning, 2022.
[31] Chenxin Xu, Robby Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, and Yanfeng Wang. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning, 2023.
[32] Qingyao Xu, Wei Mao, Jingze Gong, Chenxin Xu, Siheng Chen, Weidi Xie, Ya Zhang, and Yanfeng Wang. Joint-relation transformer for multi-person motion prediction. 2023 IEEF/CVF International Conference on Computer Vision (ICCV, pages 9782-9792, 2023.
[33] Sirui Xu, Yu-Xiong Wang, and Liang-Yar Yan Gui. Stochastic multi-person 3d motion forecasting, 2023.
[34] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1800: 04805, 2018.
Ming-Wei
[35] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al.
Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
12

---

[36] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[37] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1(555555-555,, 2020.
[38] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint (oXXii:2302.3997 2023.
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini urKiv: 2023.
[41] OpenAl. chatgpt (mar 14 version).
wtt.coom..cc.... 2023. 2023.
[42] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint urXiv:2301.1259) 2023.
[44] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-225 220000, 10, 2023.
[45] Nikita Pavlichenko and Dmitry Ustalov. Best prompts for text-to-image models and how to find them. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2067-2071, 2023.
[46] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. Promptify: Text-to-image generation through interactive prompt exploration with large language models. In Proceedings the 36th Annual ACM Symposium User Interface Software and Technology, pages 1-14, 2023.
[47] Qingqing Huang, Daniel S Park. Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et Noise2musie: Text-conditioned music generation with diffusion models. arXiv preprint 1xXiv:2302.009117 2023.
[48] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301. 2023.
[49] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to- audio generation using instruction-tuned Imm and latent diffusion model. arXiv preprint ar04 13731, 2023.
[50] Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. 2023.
arXiv preprint arXiv:23
[51] Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Gül Varol. Sine: Spatial composition of 3d human motions for simultaneous action generation. arXiv preprint arXiv:2304. 10417,
2023.
13

---

[52] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36, 2024.
[53] Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are general-purpose motion generators. arXiv preprint arXiv: 10900, 2023.
[54] Jinpeng Liu, Wenxun Dai, Chunyu Wang. Yiji Cheng, Yansong Tang, and Xin Tong. Plan, posture and go: Towards open-world text-to-motion generation. arXiv preprint arXiv:2312.14822 14828, 2023.
[55] Takahide Yoshida, Atsushi Masumori, and Takashi Ikegami. From text to motion: Grounding gpt-4 in a humanoid robot" alter3". arXiv preprint drXIv:2312.06571, 2023.
[56) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. sero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[57] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman. Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[58] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi-Pang Lam, Dragomir Anguelov, and Benjamin Sapp. Multipath++:: Efficient information fusion and trajectory aggregation for behavior prediction. CoRR, abs/2111.14973, 2021.
[59] Kan Chen, Runzhou Ge, Hang Qiu. Rami Ai-Rfou, Charles Qi, Xuanyu Zhou, Zoey Yang, Scott Ettinger, Pei Sun, Zhaoqi Leng, Mustafa Mustafa, Ivan Bogun, Weiyue Wang, Mingxing Tan, and Dragomir Anguelov. Womd-lidar: Raw sensor dataset benchmark for motion forecasting. arXiv preprint arXiv:2304.03834, April 2023.
[60] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R. Qi, Yin Zhou, Zoey Yang. Aur' elien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley. Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 9710-9719, October 2021.
[61] K. Tan et al. H. Caesar, J. Kabzan. Nuplan: A closed-loop ml- ml-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop. 2021.
[62] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711 2017.
14

---

Appendix
In the appendix, we further explain the experiment settings, the definition of metrics used for evaluation, complete prompts used to transform language input into codes, the design of the trajectory analyzing module, and some examples and illustrations extracted from our user studies.
A Additional Experiment Settings
Dataset. We use WOMD and nuPlan as the datasets for our experiments. For WOMD, we adopt 68, 000 scenarios for training and 2500 scenarios for testing, and for nuPlan, we selected 82, 122 scenarios for training and 20, 756 scenarios for testing from the whole dataset. Following the setting in previous works 7. for each scenario, we keep maximum number of 384 lanes and 32 vehicles. We generate 5-ceonnd trajectory at 10 fps. that is, T=50 timesteps.
Baseline. We consider two existing controllable trajectory generation baselines, TraffieGen [4] and LCTGen 77] TrafficGen is the first approach that generates vehicle trajectories purely from empty maps without relying on vehicle past states. I.CTGen is the latest open-sourced language- guided vehicle trajectory generation approach. Both models represent state-of-the-an performance in trajectory generation. We make some adjustments to the baseline models to standardize the number of vehicles compared in the scene and the length of the predicted trajectories, ensuring a fair comparison.
Experimental setup. In the language-tor to-cce encoder, we discretize inter-vehicle distances at 15-meter intervals and discretize speeds at 2.5- 1..5-metr--peep peeter-per-seoond intervals. We sample the vehicles' trajectories at 1-second (10 timesteps) intervals to get [7] = timesteps. We use MCG blocks with 5 layers for lane feature extraction, 2-layer transformer with 4 heads is used for decoding vehicle and interaction queries, and a cross-attention module with 8 heads is used for fusing interaction features into vehicle features. An MIP with latent dimension of 512 is finally used for trajectory generation. Only the code-tto-trjeectory decoder needs to he trained as the LLM in our model encoder does not need training. It takes about 12 hours for 100 epochs on 4 NVIDIA GeForce RTX040 GPUs for the decoder training process. During the inference phase, it takes about 30 seconds to generate scenario, which includes about mainly 30 seconds receive the response from the LLM and about 0.2 seconds to generate corresponding trajectories.
B Evaluation Metric Details
For each trajectory prediction and the corresponding ground truth trajectory, we measure their similarity using the following six metrics. Mean average displacement error(mADE), minimum average displacement error(minADE), mean final displacement error, minimum final displacement error(minFDE), and Hausdorff distance(HD) measure the gap between the projected paths and the actual paths, while the collision rate reflects the rationalization of the generated scenarios. For all metrics, smaller values mean that the generated trajectories are closer to the ground truth values, and therefore the corresponding model can be considered to generate scenarios closer to the real-life data distribution.
Symbol Definition
Let N be the number of agents in the scenario.
Let be the set of scenarios used for testing, let NG = ||G|| be the number of scenarios tested.
Let T be the overall timesteps.
Let Sy 1 1S91,Sy2. Syz.... Syn] E NxT'22 be the set of ground-truth trajectories contained in a scenario E G. where Sgi = | 22
s7 E Rxx2 Vi E (1,..., N) denotes the ground-truth trajectory of agent over Tth timesteps.
Let [ S Sg............ SyNj NxTx2 be the set of predicted trajectories contained in the scenario g E G. where Syz 1 sg. .... syil R Vi f1...., N} denotes the predicted trajectory of agent i over T timesteps.
15

---

Mean Average Displacement Error(mADE): mADE refers to the average mean square error (MSE) between all predicted points and the ground-truth points over the trajectories scenario. Its average value over the entire test set is defined as:
NG
N
MADB(C) NG
Σ | t=1
N i=1
Mean Final Displacement Frror(mFDE): mFDE refers to the average distance between the predicted final destination and the ground-true final destination over all the trajectories in a scenario. Its average value over the entire test set is defined as:
1 NG
mFDE G
o 11
N
Minimum Average Displacement Error(minADE)) minADE refers to the minimum mean square error (MSE) between all predicted and ground-truth trajectories in scenario. Its average value over the entire test set is defined as:
Nc
I Σ Is gi of 11
minADE (G)
Σ
min TEN
NG g=1
Minimum Final Displacement Error(minFDE): minFDE refers to the minimum distance between the predicted final destination and the ground-true final destination over all the trajectories in scenario. average value over the entire test set defined as:
Nic
minFDE(G) NG
min IEN
I gi
2 11
9=1
Hausdorff distance(HD): HD refers to the Hausdorff distance between all predicted and ground-truth trajectories, which measures how far two trajectories of metric space from each other when viewed as point sets in a scenario. Its average value over the entire test set is defined as:
HD(G)
1 NG
NG
Σ
N
dh (S 5go))
N
1=1
where the Hausdorff distance is defined
dh (X, Y) max MSE(2, ( Y), supyer MSE(X,
Scenario collision rate(SCR): Among the predicted trajectories, We note the bounding box of agent i as box; at time For pre-defined threshold 8, agent and agent j are considered to be collided Vi T, loU (box,, box;)) 6, where IoU(..-) denotes the intersection over union between two objects. We thus define scenario collision rate
No
N
N
2
SCR(G) NG
1 E, (box,, box;) > >6)
N
(N
C LLM Prompt Details
Here we show our prompt used to generate vehicle codes, map codes, and interaction codes.
You are faithful format converter that translates natural language descriptions to fixed-form format to appropriately describe the scenario with motion action. You also need to output appropriate map description that supports this scenario. Your ultimate goal is to generate
16

---

realistic traffic scenarios that faithfully represent natural language descriptions and normal scenes that follow the traffic rules. Answer with list of codes describing the attributes of each of the vehicles and the interactions within the events in the scenario.
Desired format:
Summary: summarize the scenario in short sentences, including the number of vehicles. Also, explain the underlying map description.
Explanation: If there are interaction behaviors concluded in the requirements, first explain the meaning of these terms such as the behaviors of the agent involved. Then explain for each group of vehicles why they are put into the scenario and how they fulfiil the requirement in the description.
Vehicle Code: A list of codes of length ten, describing the attributes of each of the vehicles in the scenario, only output the values without any text:
-'VI':
-'22:
.'V3': [
Map Code: code of length six describing the map attributes, only output the values without
any text:
'Map': [mn]]
Interaction Code: For each agent, generate two codes, each of length five. The first code represents the relative distance of the vehicle with respect to the ego car, and the second one represents the relative position of the ego car. Only output the values without any text:
-'I'' [...]] [...]
-'12': [..]] [...]] Meaning of the vehicle code attribute:
dim 0: ppss": [-1,5] The relative position of the vehicle with respect to ego car in the order 'front right', 2- 'back right', "back', 4 'back left', 5 'front left']. J. -1 the vehicle is the ego vehicle.
of [0 'front',
dim 1: 'distance": [0.3] the distance range index the vehicle towards the ego vehicle; the range from 0 to 60 meters with 15 meters intervals. 0 if the vehicle is the ego vehicle, For example, if the distance value is less than 15 meters, then the distance range index is O. dim 2: 'direction': [0, 3] the direction of the vehicle relative the ego vehicle, in the order of 0- '-'parallel same', 1- parallll opposite', 2- perpendicular up". 3- perpendicular down'. 0 if the vehicle is the ego vehicle.
dim 3-8: 'speed trend': [0,8] speed of the agent in future 5 seconds(including initial speed and final speed) with consecutive dimensions having a time interval of 1s. Velocity is categorized into nine grades from 0-8, with smaller grades resulting in higher speeds. The range is from
m/s with a 2.5 m/s interval. For example, 20m/s is the range of therefore the speed value is 8.
dim 9: "action": [0,5] category of vehicle behavior during this time. The vehicle's action is divided into six types: [0 'stop', 1- 'straight", 2 'left-turn', 3 'right-turn' 4- 'left-change-lane,, 5 "right-change-anne]]
Mcaning of the Map code attributes:
dim 0-1: 'parallel lane count': 2-dim. The first dim is the number of parallel dime-direction lanes of the ego lane, and the second dim is the number of parallel opposite direction lanes of the ego lane.
apposite-direction
dim 2-3: 'perpendicular lane count': 2-dim. The first dim is the number of perpendicular upstream-ddirettion lanes, and the second dim is the number of perpendicular downstream- direction lanes.
dim 4: 'dist to intersection': 1-dim. the distance range index of the ego vehicle to the intersection center in the x direction, range is from 0 60 meters with 15 meters intervals. -1 there is no intersection in the scenario.
dim 5: "lane id': 1-dim. the lane ID of the ego vehicle, counting from the rightmost lane of the same-dires dire--iroction lanes, starting from 1. For example, if the ego vehicle is in the rightmost lane, then the lane id is if the ego vehiele in the leftmost lane, then the lane id is the number of the same same-directo direction lanes. Meaning of the interaction code attributes:
17

---

Each code has a length of 5, representing the relative distance and direction of the vehicle with ego car respectively. Two neighboring values in each code have an interval of one second, so it can be used to represent the tendency of the trajectory relative the ego car.
The values of the first code represents the distance this car relative to the ego car. It is divided into five bins from 0-5, with each bin has an interval of five meters. When the value is O, it means that the car is very close to the ego car. The farther away this vehicle is from the ego car, the larger the value is. When the distance is more than 25 meters, the value is always set to 5.
The values of the second code pair indicates the distance of the relative the ego For each dimension, O means that the vehicle is in front of the ego car directly, represents the right front, represents the right rear. represents the rear, represents the left brear and 5 represents the left front.
Traffic rules that you should obey when creating representation for traflic scenes:
When the car drives to an intersection, it should slow down whether it is turning or going straight ahead.
If another vehicle is close to the ego vehicle and passes in front of the ego vehicle, for example, driving from the left front to the right front, the vehicle should stop and wait for the other car's passing by.
Wheen changing lanes, pay attention to whether there are other vehicles on the near left or near right side of the car, and if there are, keep driving in the current direction.
When the vehicle turns left, it should pay attention to the left rear and make sure that there other vehicles, keep straight ahead otherwise.
When the vehicle turns right, should pay attention to the left rear and make sure that there
other vehicles, keep straight ahead otherwise.
The car should not change lanes to the left when it is in the far left lane.
The should not change lanes to the right when it is in the far right lane.
Some nomenclature so you can better understand how vehicles interact with each other to represent their trajectories and movement trends:
overtake: When another vehicle directly in front of the vehicle is moving too slowly, ego vehicle can overtake the vehicle in front of by changing lanes the left or right and accelerating. Generally, that car is directly in front of the ego car at the beginning and drives slower than the ego car. In the end, it will be on the rear side of the ego car.
merge: There are more than two cars on the road. One vehicle is first selected as the ego car and is kept in a straight line and keeps its speed throughout the process. The other cars are in the lane adjacent to the ego car, keeping a relatively close distance at first. These cars in turn merge into the ego car's lane by changing lanes. For example, the vehicle to the left of the ego car changes lanes to the right, and the vehicle to the right of the ego car changes lanes to the left. If the car wants to change lanes to the left and there is another vehicle in the left lane, it should decelerate and wait for that vehicle to pass before making the left lane change. The same is true when a vehicle changing lanes to the right. During the process, you should keep a safe distance with other cars in order not to collide. rear-ending/rear-eee
collision: When the first car (for example, to avoid someone crossing the street) makes sudden deceleration, and the car behind collides with it. Generally, the second car will be behind the ego car at the very beginning. some distance away and faster But at the end, that car's position will almost overlap with the ego car and
than the ego come to a stop.
Transform the query sentence the vehicle codes strictly following the rules below:
Ensure the code of each vehicle has a length of 10.
Focus the interactions between the vehicles in the scenario.
Focus on realistic action generation of the motion to reconstruct the query scenario.
First determine the action type of the agent fill index 9.
Pay particular attention to the type of trajectories generated, and the corresponding trajectories are generated according to the inferred trajectory class.
Follow traffic rules to form fundamental principle in most road traffic systems to ensure the safety and smooth operation of traffic. You should incorporate this rule into the behavior of our virtual agents (vehicles).
18

---

For speed and distance, convert the unit to m/s and meter, and then find the interval index in the given range.
Describe the initialization status of the scenario.
During generation, the number of vehicles is within the range of [1, 32].
The maximum distance should not exceed 60m (index 1).
The maximum speed should not exceed 20m/s (index 3-8).
Always generate the ego vehicle first (V1).
Always assume the ego car is in the center of the scene and is driving in the positive x direction.
In the input descriptions, regard VI, Vehicle 1 as the ego vehicle. All the other vehicles are the surrounding vehicles. For example, for "Vehicle was traveling southbound", the ego car is Vehicle 1.
If the vehicle is stopping, its speed should be Om/s (index 3-8). Also, if the first action is 'stop', then the speed should be Om/s (index 3-8).
If vehicle move slow speed, the speed should be less than 2.5m/s (index 1) or 5m/s (index 2).
-Try to increase the variation of placement and motion of the vehicles under the constraints of the description.
Both turning and lane-changing processes need to reflect changes in speed. For example, during turning and lane changing the vehicle needs to undergo deceleration process to ensure safety, and after the maneuver is completed the vehicle will reaccelerate. The change in speed is reflected in the 'speed trend' dimensions in the vehicle code.
Generate the map code following the rules below:
If there is vehicle turning left right, there must be intersection ahead.
If the car was going to change lanes to the left, he couldn't have been in the far left lane. the car going to change lanes to the right, he couldn' 't have been in the far right lane. Should at least have one lane with the same direction as the ego lane; i.e., the first dim of Map should be least 1. For example, if this is one-way two-lane road, then the first dim of the Map should be 2.
Regaad the lane at the center of the scene as the ego lane.
Consider the ego car's direction as the positive x direction. For example, for "VI was traveling northbound in lane five of five-lane controlled access roadway". there should be lanes in the same direction as the ego lane.
The generated map should strictly follow the map descriptions the query text. For example, for "Vehicle was traveling southbound", the ego car should be in the southbound lane. there an intersection, there should be least one lane in either the upstream or downstream direction.
If there is no intersection, the distance to the intersection should There should be a vehicle driving vertically to the ego vehicle in the scene only when there an intersection in the scene. For example, when the road is just two-way, there should not be any vehicle driving vertically to the ego vehicle.
If no intersection mentioned, generate intersection scenario randomly with real-world statistics.
Generate the interaction codes following the rules below:
Ensure the interaction code has length of 10.
Generate an interaction code for each of the vehicles, with the first code remaining all zero since the ego car always overlaps itself.
vehicle as a trajectory over a future period of time. Determine their mutual speed and direction from the relative positions of the cars. If the relative position crosses the vehicle, such as driving from the left rear to the right front of the vehicle, or from the right rear the left front, there is possibility of collision, and vice versa.
Interpreting the relative position
If another vehicle drives from the left front of the ego vehicle all the way to the right front, or the right front to the left front, it is possible for the ego vehicle to remain stopped waiting for the other vehicle go first.
19

---

Reasoning about the relative distance and position of two cars based the information of the map. For example, if there is only one lane, there are only other cars that may be in front of and behind the car, and there cannot be any other cars the left or right side.
D Trajectory Analyzing Module
During the training process, for a ground-truth scenario S derived from real-world datasets, we re-generate the scene by
V.I (S), S = D(m, V.I,
(6)
where (-) denotes an analyzing module that discretizes ground-truth trajectories into vehicle, map and interaction codes. The length of each code and the meaning of each dimension are the same as the codes generated by the language-to-code module, as described in the previous sections and the detailed prompts.
Map code: (r()) calculates the number of lanes included in the map both vertically and horizontally, the distance of the ego car from the intersection, and the lane on which the ego car is located on to fill up the map code.
Interaction code: For each vehicle other than the ego car, (-) samples its ground -tuut trajectory at regular intervals, using these sampled time points to compute the relative position and relative distance to represent its trend of movement relative to the ego vehicle. In practice, we sample the trajectory points every second and remove outliers concluded. For each sampled point, (-(-) discretizes the distance and position to fill the corresponding part of the interaction code. The relative distance is divided by a fixed interval the relative distance code; for the relative position, we use the ego car as the origin and the heading of the ego car as the positive direction of the x-axis, and the whole space is divided into several squal-partitioned regions. The relative position code can be thus represented by the serial number of the grid in which the vehicle is located. For example, when we select the number parts, with code representing the
of areas be 6, then the space is divided into six area directly in front, code representing the front-right area, code 2 representing the front- back area, code 3 representing the rear area, code representing the back-left area, and code 5 representing the front-left area. The length of the interval and the number of areas chosen are discussed in the ablation study.
The computation of relative position and the discretization of speed are similar to the previous parts, and we also sample fixed intervals to represent the trend of the vehicle's speed change. In addition, we categorize vehicles into six types based on their trajectory: stop, straight ahead, left turn, right turn, left lane change, and right lane change. Likewise, we note S, = [ s s.. []] E xx2 Vi E [1, be the ground-truth trajectory of agent i over T timesteps, V [m] uf ... i E RT,, Vi E (1....,A) .N be the ground-truth velocity of agent i over T timesteps and H, = [M] h..
Vehicle code:
.. h2] E x2x R2x2, Vi NN be the ground-truth heading
of agent over T timesteps.
We categorize the trajectory types according to the following rules. Stop: Given pre-defined distance threshold d and velocity threshold 8,,, Agent i is considered to be stopping if v(t,t') t') T I|s; sf S &a and v(t, E T llof of || s 5,. In practice, 8g is set to be 1 meter, and 6, is set to be 0.2 meters per second.
Left turn: Given pre-defined lane-width tnp and angle threshold 6, 0, agent i is considered to be making a left turn it's heading sharply towards its left and its displacement in the direction
perpendicular to its heading greater than the lane-width: ||(s x.) (ss -.) h.)) b 2 Lw-
t(t,t') T2, (N. Ky ) 2 2 x da. In practice, 8g set to be -
Right turn: Given pre-defined lane-width Ln and angle threshold 5, > 0. agent is considered to be making a right turn if it's heading sharply towards its right and its displacement in the direction perpendicular to its heading is greater than the lane-width:
| xg) ((of <?>hę) nol 2 wwo
v(t,t') T2, (A ht) )₂ 2 2 x da. In practice, 8, set to be x
12.
20

---

Left lane change: Given pre-defined lane-width tu and angle threshold 6g > 0, agent is considered to be making a left lane change if it's heading slightly towards its left and its displacement in the direction perpendicular to its heading is greater than a certain multiple of lane-width:
(o7 - s
{s} s h 186 2 Lw. v(t,t)) E T (1, ) E 16a. 2 x 8g]- In practice,
on is set to be
12'
Right lane changel. Given pre-defined lane-width iy and angle threshold on > 0, agent is considered be making right lane change if it's heading slightly towards its right and its displacement in the direction perpendicular to its heading is within a certain multiple of lane-width:
(of b) (of - - k)) ng 2 Iw. ((L, E T2, ( x nt 1n) E [-22x 6a óg]. In practice, 5a is set to be 12-
Straight Most of the remaining agent trajectories remain within the lane width, and the variation in the direction of the agent is within proper range, and thus these agents considered be driving straight forward. There remains a small percentage of agents that have missing values or outliers in their trajectories, which are given default trajectory type and masked out in the following processes.
E
Additional User Study Details
We use GPT-4 to generate series natural language descriptions of traffic scenes containing vehicles, e.g., "The car speeds up pass the vehicle ahead of it.", which later InteractTraa and LCTGen are used generate scenarios respectively. Users are invited evaluate the generated scenarios depend different requirements. We balance the frequency of each type of interaction event these descriptions as much as possible.
The first user study contains forty language commands, and for each command, the two models generate the corresponding trajectory. Each user is asked to choose the trajectory that better fits the language description. A total of 31 interviewees participated in the research, with a total of 1240 samples. In the second user study, we have fifty language descriptions that cover and emphasize the most representative interaction types. For each description, users are asked to answer whether the scenarios generated fulfill the corresponding textual descriptions from their perspectives. The answer can be simultaneously positive or negative for either of the questions. A total of 28 users participated in the study, with a total of 1400 samples.
In this section, we give two examples for each user study as an illustration.
User study 1: overall generation performance The first user study contains forty language commands, and for each command, LCTGen and InteractTraj are used to generate corresponding trajectories respectively. Each user is asked to choose from either of them that better fits the language description. Figures Jand 10 illustrate two language descriptions and the corresponding scenarios generated.
User study 2: vehicle interaction performance The second user study contains fifty language commands and each command represents representative type of interaction previously mentioned. For each command, LCTGen and InteractTrraj are used to generate corresponding trajectories respectively and each user is asked to evaluate whether the scenarios fit the interaction behaviors given by the linguistic description. TT and 12 illustrate two language descriptions and the corresponding scenarios generated.
21

---

(a) Option
(b) Option h.
Figure 9: Description: The vehicle slows down and turns left at the intersection.
(a) Option
(b) Option b.
Figure 10: Description: The lead vehicle signals lane change, prompting the following cars to adjust their speeds and positions accordingly
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We present the main problem addressed by the article, its contributions, and the current results in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
22

---

(a) Option
(b) Option h.
Figure 11: Yielding: The sedan yields to the oncoming ambulance.
(a) Option
(b) Option
Figure 12: Merging: As the car approached the intersection, it slowed down, allowing the motorcycle to merge into the space.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have included a discussion of the current limitations and directions for future work in section O
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
23

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We described the model structure as well as the training setup in section 5.2 and we will release the codes afterward.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed descriptions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
24

---

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., . a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: we will release the codes afterward.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides: (eggli/cuuddas//deeebbbiissssss Poodeeubmisssiooolliieo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips ctttlps.cc.pubbcc/// hwwbbl/ccggiides//oooooo
wtt.eeeSuuummsssisooooooolee for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We mentioned this part in section 5.2 and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
25

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We mentioned this part in section 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httts::/neuurps. http://cccc.ppp..pttttttacsaaaaa
Answer: [Yes]
Justification:
We don't 't have such problem in paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
26

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: For now, this work does not have much social impact. Current work lies mainly in the simulation phase, which is still far from practical application.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
27

---

Answer: [Yes]
Justification:
We eite the articles, models, and datasets mentioned in paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's ' creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We describe the experimental setup in section 5.4 and the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
28

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The subjects in our experiment had no such risk.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "9XuY5GOQ1I", "reviewer": "Reviewer_YJzW", "summary": "The paper introduces a novel method called InteractTraj, which is the first language-driven traffic trajectory generator capable of producing interactive traffic trajectories. This method is critical for advancing autonomous driving technology because it can generate realistic and controllable vehicle interaction trajectories based on natural language instructions, addressing the limitations of previous approaches that focused solely on generating trajectories for individual traffic participants without considering complex traffic dynamics.\n\nInteractTraj features an innovative language-to-code encoder with an interaction-aware encoding strategy that translates abstract trajectory descriptions into concrete, interaction-aware numerical codes. Additionally, it includes a code-to-trajectory decoder that employs interaction-aware feature aggregation, integrating vehicle interactions, environmental map data, and vehicle movements to generate interactive traffic trajectories.\n\nExperimental results show that InteractTraj outperforms SOTA methods in generating interactive traffic trajectories under diverse natural language commands, offering more realistic simulations of interactive traffic scenarios. This research is significant for driving simulation, particularly in reducing the costs of safety-critical scenarios by recreating real-world situations in virtual environments, thereby accelerating the development of autonomous driving technologies.", "questions": "1. **Scope of Interaction Types**: The study focuses on a set of predefined interaction types. Could the authors clarify how the model performs when dealing with emergent or less common interaction types that might not have been explicitly trained on, like parallel driving or platooning? Would the model generalize well to such scenarios, or would it require retraining?\n\n2. **Language Ambiguity Handling**: How does InteractTraj handle ambiguity in natural language descriptions? For instance, phrases like \"the car behind\" can be ambiguous in dense traffic scenarios. Does the model prioritize proximity or other factors when resolving such ambiguities?", "limitations": "The authors acknowledge some limitations of their work and outline future directions for addressing these constraints. However, there is room for improvement in terms of comprehensively addressing the limitations and discussing potential negative societal impacts. Here are some areas that could benefit from further exploration and discussion:\n\n1. **Limited Scope of Traffic Participants**: Currently, the focus is on generating trajectories for vehicles only. The system does not account for other traffic participants such as pedestrians, cyclists, or other non-motorized vehicles. This limitation could be addressed by extending the model to encompass a broader range of entities in the traffic ecosystem.\n\n2. **Map Generation Constraints**: The authors admit that map generation is restricted by the available map library. To enhance flexibility and realism, the system could be expanded to generate more diverse and flexible maps, possibly through generative models that create new environments based on learned patterns.\n\n3. **Complex Interaction Handling**: While the model performs well with common interaction types, its ability to handle more complex or rare interactions has not been thoroughly evaluated. The authors could consider testing the model's performance in scenarios involving intricate multi-agent interactions to understand its limitations better.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**Originality** \n\nThe paper introduces a significant innovation in the field of autonomous vehicle technology and driving simulation by proposing InteractTraj, the first language-driven traffic trajectory generator capable of producing interactive traffic trajectories. This concept stands out due to its novelty in interpreting abstract trajectory descriptions into concrete, interaction-aware numerical codes, which then guide the generation of realistic traffic scenarios. The method's originality is further highlighted by its ability to encapsulate complex interactive dynamics that were previously unaddressed by focusing solely on individual traffic participant trajectories.\n\n**Quality** \n\nThe research demonstrates high-quality methodology and rigorous experimental validation. The authors propose a language-to-code encoder with an interaction-aware encoding strategy, followed by a code-to-trajectory decoder that synergizes vehicle interactions with the environment. The comprehensive experiments conducted show superior performance over state-of-the-art (SoTA) methods, substantiated by quantitative metrics and qualitative assessments through user studies. The results indicate that InteractTraj not only generates more realistic and controllable interactive traffic trajectories but also effectively captures the essence of diverse natural language commands.\n\n**Clarity**\n\nThe paper is well-structured and clearly articulated, making it accessible to reseachers in the field. The abstract succinctly summarizes the contributions and the significance of the work. The introduction sets the context by highlighting the importance of driving simulations in autonomous vehicle development and identifies the gap addressed by the proposed solution. The methodology is explained with sufficient detail to allow for replication, supported by figures and tables that aid in understanding the findings.\n\n**Significance**\n\nThe impact of this work is substantial, particularly in the realm of autonomous driving and traffic simulation. By enabling the generation of interactive traffic scenarios through natural language commands, InteractTraj opens up new possibilities for creating diverse and realistic driving conditions, which is critical for testing and validating autonomous vehicle algorithms. This could potentially lead to safer and more reliable autonomous systems by exposing them to a wide range of driving situations that might be difficult or dangerous to replicate in real life. \n\nIn summary, the paper excels in its original contribution to the field, the high quality of its research, the clarity of its presentation, and the significant implications of its findings for autonomous vehicle technology and driving simulation.", "weaknesses": "Despite the significant contributions and innovative approach outlined in the paper, there is a weaknesses where the work could be improved to better achieve its stated goals:\n\n1.**Robustness to Ambiguous or Complex Language Descriptions**: The paper does not extensively explore how the system handles ambiguous or complex linguistic instructions. Since natural language can be nuanced and subject to interpretation, evaluating the system's ability to generate accurate trajectories when faced with unclear or highly detailed descriptions would provide a clearer picture of its robustness and flexibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer YJzW**\n\n$\\textbf{Question1:}$  Clarify how the model performs when dealing with emergent or less common interaction types, like parallel driving or platooning? Would the model generalize well to such scenarios, or would it require retraining?\n\n$\\textbf{Answer1:}$ Our model can generate compliant scenarios when dealing with less common interaction types. The left part of Figure S1 in the rebuttal pdf shows the experimental results. We present the results for rare interaction types including mentioned uncommon cases of parallel driving (Figure S1-1), platooning (Figure S1-2), and another uncommon case involving pulling over (Figure S1-3). We see that for less common interaction types, our method effectively translates the relevant behaviors to generate compliant scenarios.\nOur model is able to generalize scenarios with emergent or less common interactions without requiring retraining. The main reason is that the LLM used in our encoding process possesses strong generalization abilities. The large language model can understand linguistic descriptions of uncommon interaction scenarios and convert them into appropriate numerical codes for decoding. Then the decoding process, which also has generalization ability due to training with massive numerical codes, would translate these codes into trajectories and generate compliant scenarios.\n\n$\\textbf{Question2:}$ How does InteractTraj handle ambiguity in language descriptions, like phrases like \"the car behind\" in dense traffic scenarios? Does the model prioritize proximity or other factors when resolving such ambiguities?\n\n$\\textbf{Answer2:}$ We leverage the reasoning ability of the large language model to manipulate linguistic inputs to solve possible ambiguity problems. We do not introduce additional prioritized proximity or other factors. Ambiguous situations mainly fall into two categories: those where the reference to the object is unclear, and those where there is a contradiction within the language instruction itself.\n- For the first type, LLM would understand language descriptions and convert them into numerical codes to satisfy one of the meanings of phrases. We present the result of the mentioned phrase \"the car behind\", see the right part of Figure S1 in the rebuttal pdf. We see that when the reference to \"the car behind\" is ambiguous (Figure S1-4), the LLM would randomly designate one vehicle as the \"behind car\". However, with clearer descriptions like \"the car behind ego car\" (Figure S1-5), the LLM would accurately handle the descriptions.  \n- For the second type involving self-contradictory requirements, the scenarios generated would partially meet the instructions' criteria. Figure S1-6 illustrates this with an example of language instruction \"A car is turning left and moving in a straight line.\" LLM may opt to generate an intersection in the scene to fulfill the left-turning requirement while disregarding the requirement for straight-line movement.\nTo better tackle language ambiguity, an optimal solution involves introducing LLM-human interaction to iteratively verify language descriptions. This approach will be explored in our future work.\n\n$\\textbf{Question3:}$ Currently, the focus is on generating trajectories for vehicles only. This limitation could be addressed by extending the model to encompass a broader range of entities.\n\n$\\textbf{Answer3:}$ At present, our work concentrates on trajectory generation for vehicles since vehicles are the most common subject, specifically creating interactive scenarios by modeling vehicle interactions.  To incorporate various types of traffic participants, such as pedestrians and cyclists, we could utilize specialized heterogeneous networks as the backbone for the decoder network. Additionally, we plan to design specialized dynamic models, such as the bicycle kinematic model, tailored for different subjects. This comprehensive systematic design will be pursued in future work.\n\n$\\textbf{Question4:}$ The authors admit that map generation is restricted by the available map library. The system could be expanded to generate more diverse and flexible maps, possibly through generative models.\n\n$\\textbf{Answer4:}$ We use the current map library method since i) it ensures the obtained map is totally realistic and ii) the built map library could already support plenty of trajectory generation cases.  Although a map generation method could generate more flexible maps, it may encounter the problem of being unrealistic. Meanwhile, in this work, our main solving problem is how to generate traffic trajectories with interactions. Thus, we hope the map is ensured to be realistic to better evaluate our trajectory generation design. In future work, we will switch the focus to map generation. To expand our method with more efficient map generation methods to obtain maps that conform more closely to linguistic inputs, we plan to use masked prediction for map expansion given partial map observations or use diffusion models for map generation. \n\n$\\textbf{Question5:}$ While the model performs well with common interaction types, its ability to handle more complex or rare interactions has not been thoroughly evaluated. The authors could consider testing the model's performance in scenarios involving intricate multi-agent interactions.\n\n$\\textbf{Answer5:}$ We have analyzed and shown some qualitative visualizations of complex or rare interaction cases in the responses A1 and A2. We will include more results of complex and rare interactions in the revised version. The quantitative evaluation of intricate multi-agent interactions is restricted by the existing benchmarks since there are no direct datasets full of intricate interactions labeled with language descriptions. In the future, when direct language-to-scenario datasets or specific datasets focusing on intricate interactions are available, we will conduct a more thorough evaluation of intricate multi-agent interaction cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F1YYtxJe2H", "reviewer": "Reviewer_xSeP", "summary": "This work proposes a novel framework for traffic trajectory generation with natural language description of interactions, called InteractTraj. Specifically, the proposed framework consists of two main components: a language-to-code encoder and a code-to-trajectory decoder. The language-to-code encoder is designed to generate the code representation of the natural language description, and the code-to-trajectory decoder is used to generate the trajectory based on the code representation. The proposed framework is evaluated on two dataset and shows promising results compared to the state-of-the-art methods.", "questions": "1. Could the proposed model handle the different length of the generated trajectories? or it only generates the fixed-length trajectories?", "limitations": "This paper addresses the problem of using language for controllable trajectory generation and discusses the limitations of the model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper use language description as the conditional information for controlling the generation of traffic trajectories, which is a interesting idea.\n2. This paper is well-written and easy to follow. Each section and each component of the model is clearly described.\n3. The proposed framework is evaluated on two real-world datasets and shows promising results compared to the state-of-the-art methods. The authors also do a series of experiments like ablation study and user study to further validate the effectiveness of the proposed framework.", "weaknesses": "1. It seems the proposed framework is computationally expensive; the authors should provide the details about the efficiency of the model.\n2. Is the language description encoder used in this paper GPT-4? There is no comparison of the effects of different large language models as the encoder.\n3. Controllability is an important contribution of this paper, which should be validated more intuitively (visual) or quantitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "##  **Response to Reviewer xSeP**\n\n$\\textbf{Question1:}$ It seems the proposed framework is computationally expensive; the authors should provide the details about the efficiency of the model.\n\n$\\textbf{Answer1:}$  The overall resources needed for model training and inference are not expensive. Only the code-to-trajectory decoder needs to be trained as the LLM in our model encoder does not need training. It takes about 12 hours for 100 epochs on 4 NVIDIA GeForce RTX040 GPUs for the decoder training process. During the inference phase, it takes about 30 seconds to generate a scenario,  which includes about mainly 30 seconds to receive the response from the LLM and about 0.2 seconds to generate corresponding trajectories. \n\n\n$\\textbf{Question2:}$ Is the language description encoder used in this paper GPT-4? There is no comparison of the effects of different large language models as the encoder.\n\n$\\textbf{Answer2:}$ We use GPT-4 as the language description encoder in our paper, but many other LLMs also can be incorporated into our work. Here we further show the result of applying two other representative LLMs as encoders, including the Llama3-3.1-70b and Mistral-large. See Figure S2, which illustrates the visualization results provided by both Llama3 and Mistral-large for different types of interaction. We see that applying other LLMs also achieves compliant scenario generation.\n\n\n$\\textbf{Question3: }$ Controllability is an important contribution of this paper, which should be validated more intuitively (visual) or quantitatively.\n\n$\\textbf{Answer3:}$  Our method can control the specified vehicle trajectory generation in the scene. As controllability relies on the language input, here we validated more intuitively through visualization. Figure S3 shows the generation results under four different control descriptions applied to ego vehicles.  We observe that our method can control specific vehicle trajectories through different linguistic descriptions. \n\n\n$\\textbf{Question4:}$ Could the proposed model handle the different lengths of the generated trajectories? or it only generates the fixed-length trajectories?\n\n$\\textbf{Answer4:}$ The length of the generated trajectories is pre-defined before the model is trained. In our paper, we generate 5-second trajectories at 10 fps in experiments. The setting of fixed-length trajectories is common in related tasks like trajectory generation [1,2] and prediction [3,4]. \n[1] Learning to generate diverse and realistic traffic scenarios, ICRA 2023\n[2] Language-conditioned traffic generation, CoRL 2023\n[3] Motiondiffuser: Controllable multi-agent motion prediction using diffusion, CVPR 2023\n[4] MotionLM: Multi-Agent Motion Forecasting as Language Modeling, CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZEeETFiPO1", "reviewer": "Reviewer_awjY", "summary": "This paper proposes a large language model-based traffic trajectory generation method. Due to the designed interaction interpretation mechanism, the proposed method can generate a better corresponding trajectory. In addition, to improve the generation quality, the authors also proposed a well-designed prompt and a two-step feature aggregation. Finally, the authors also considered a variety of experiments to verify the effectiveness of the method.", "questions": "i) Since the training of the proposed method also still relies on pre-collected ground truth data, I think the authors need to compare it with CTG and CTG++. Another reason is these two methods are also based on language conditions.\n\nii) How diverse are the trajectories generated by the proposed method? Will they be similar to the training data?\n\niii) In some visualization results, there are some collisions in the generated vehicle trajectories, such as the red vehicle in the overtaking in Figure 5, and the orange vehicles in Figures 7(b) and 8(b). What is the reason for this?\n\niv) What is the size of the trajectory data generated by the proposed method? The visualization results show that the number of vehicles is not large. Can a more complex trajectory be generated?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "i) The LLM-based traffic trajectory generator can generate interactive traffic trajectories.\n\nii) A well-designed interaction-aware prompt and code-to-trajectory decoder.\n\niii) In addition to quantitative analysis, a visual case study is also given in the experiment.", "weaknesses": "i) The author did not compare with the two related baseline methods.\n\nii) There are some collisions in the generated trajectory.\n\niii) The authors did not evaluate the diversity and scalability of the proposed generation method.\n\nMinors:\n\nLine 43: \"a LLM-based\" -> an\n\nLine 631: “7 and 8 illustrate” -> “Figures 7 and 8”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "##  **Response to Reviewer awjY**\n$\\textbf{Question1:}$ Since the training of the proposed method also still relies on pre-collected ground truth data, I think the authors need to compare it with CTG and CTG++. Another reason is these two methods are also based on language conditions.\n\n$\\textbf{Answer1:}$ The setting of CTG and CTG++ is different from our method setting of language-conditioned trajectory generation. CTG and CTG++ require past trajectory observations to generate traffic trajectories, as outlined in their problem formulation (Sec.III-A in CTG and Sec 3.1 in CTG++). Without past trajectory observations, these models are unable to produce plausible trajectories. This is the primary reason why these two models were not selected when comparing our model to the baseline models. \n\n$\\textbf{Question2:}$ How diverse are the trajectories generated by the proposed method? Will they be similar to the training data?\n\n$\\textbf{Answer2:}$  Because of the generalization ability and multiple possibilities of the LLM's response,  diversified scenarios can be generated from the same language input. Figure S4-1, S4-2 and S4-3 in the rebuttal pdf demonstrate one such case. For the same language input, the LLM will generate satisfying responses with variations, reflecting the realism and diversity of the generated scenarios by our model. Additionally, because of the LLM's generalization ability and multiple possibilities of LLM's response, the generated trajectories may differ significantly from the training data.\n\n$\\textbf{Question3:}$ In some visualization results, there are some collisions in the generated vehicle trajectories, such as the red vehicle in the overtaking in Figure 5, and the orange vehicles in Figures 7(b) and 8(b). What is the reason for this?\n\n$\\textbf{Answer3:}$ The vehicles involved in collisions in these figures mentioned are from scenarios generated by baseline models (LCTGen). This is due to two main reasons:  i) the baseline method does not consider vehicle interaction and generates vehicle trajectory independently, which can lead to conflicts or collisions between different vehicle trajectories. ii) the intermediate code design in the baseline method does not account for the global movement trend, thus there may be conflicts between the vehicle end state and map boundary. In comparison, our method i) considers interaction-aware numerical codes to jointly generate interaction-aware vehicle trajectories to avoid unreasonable collisions; ii) introduces the global trend in the vehicle code V to provide a more reasonable overall trajectory moving states, thereby avoid the collision between the vehicle end state and map boundary.\n\n$\\textbf{Question4:}$ What is the size of the trajectory data generated by the proposed method? The visualization results show that the number of vehicles is not large. Can a more complex trajectory be generated?\n\n$\\textbf{Answer4:}$ We generate a 5-second trajectory at 10 fps for each vehicle.  For the requirement of generating more complex traffic scenarios, LLM can smoothly assign appropriate vehicle codes to each vehicle and analyze inter-agent interactions to produce reasonable traffic scenarios. When the number of agents increases, our model is still able to generate scenarios that conform to linguistic descriptions. Figure S5 shows our model's ability to generate complex scenarios with more than 5 vehicles. Even as the number of vehicles in the scene increases, our model maintains its performance in scenario generation, handling cases with up to ten vehicles well, whereas the baseline method often results in significant vehicle collisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your valuable feedback! 
We have responded to questions based on the opinions of each reviewer. The attached PDF contains various figures. Please review it.

Meta-review: This paper introduces InteractTraj, a novel language-driven traffic trajectory generator capable of producing interactive traffic trajectories. The method uses a language-to-code encoder with an interaction-aware encoding strategy and a code-to-trajectory decoder employing interaction-aware feature aggregation. The authors evaluate their approach on two real-world datasets and compare it with state-of-the-art methods.

Reviewers highlight several strengths of the paper:
1. Originality: InteractTraj is noted as the first of its kind in addressing interactive traffic trajectory generation from natural language descriptions.
2. Methodology: The paper presents a well-structured and clearly articulated methodology.
3. Experimental Validation: Comprehensive experiments include quantitative metrics, qualitative assessments, and user studies.
4. Potential Impact: The work shows significant implications for autonomous vehicle and driving simulation.

Overall, reviewers view the paper positively, with two recommending acceptance and one giving a borderline accept.

Recommendation:

Based on the reviews, the recommendation is to accept the paper. The proposed approach and potential impact outweigh the noted limitations. However, the authors should address the following in the camera-ready version:
1. Include comparisons with language-based baseline methods (CTG and CTG++).
2. Provide analysis of the model's efficiency and computational requirements.
3. Address collision issues in generated trajectories.
4. Expand on the model's ability to handle diverse and scalable scenarios.
5. Discuss how the model handles ambiguous language descriptions and complex interaction types.
Decision: Accept (poster)