Language Conditioned Traffic Generation
Authors: Shuhan Tan
Boris Ivanovic
Xinshuo Weng
Marco Pavone
Philipp Kraehenbuehl
Venue: CoRL 2023 Poster (2023)
Keywords: Self-driving
Content generation
Large language model

Abstract: Simulation forms the backbone of modern self-driving development. Simulators help develop, test, and improve driving systems without putting humans, vehicles, or their environment at risk. However, simulators face a major challenge: They rely on realistic, scalable, yet interesting content. While recent advances in rendering and scene reconstruction make great strides in creating static scene assets, modeling their layout, dynamics, and behaviors remains challenging. In this work, we turn to language as a source of supervision for dynamic traffic scene generation. Our model, LCTGen, combines a large language model with a transformer-based decoder architecture that selects likely map locations from a dataset of maps, and produces an initial traffic distribution, as well as the dynamics of each vehicle. LCTGen outperforms prior work in both unconditional and conditional traffic scene generation in terms of realism and fidelity.

Language Conditioned Traffic Generation
Shuhan Tan'
Boris Ivanovic Xinshuo Weng"
Marco Pavone2
Philipp Krähenbühl
UT Austin 2VIDDA
Simulation forms the backbone of modern self-driving development. Simulators help develop, test, and improve driving systems without putting hu- mans, vehicles, or their environment at risk. However, simulators face a major challenge: They rely on realistic, scalable, yet interesting content. While re- cent advances in rendering and scene reconstruction make great strides in cre- ating static scene assets, modeling their layout, dynamics, and behaviors remains challenging- In this work, turn language source of supervision for dy- namic traffic scene generation. Our model, LCTGen, combines a large language model with a transformer- based decoder architecture that selects likely map loca- tions from a dataset of maps, produces an initial traffic distribution, as well as the dynamics of each vehicle. LCTGen outperforms prior work both unconditional and conditional traffic scene generation in-terms of realism and fidelity. Code and demo available https://ariostt:: www.pp...............g www.ggi......ttteeeeeeeeeeee iot/en
Abstract:
Keywords: Self-driving, Content generation, Large language model
1 Introduction
Driving simulators stand comerstone in self-driving development. They aim to offer trolled environment to mimic real-world conditions and produce critical scenarios at scale. Towards this end, they need to be highly realistic (to capture the complexity of real-world environments), scalable (to produce diverse range of scenarios without excessive manual effort), and able to cre- ate interesting traffic scenarios (to test self-driving agents under different situations).
In this paper, we easily articulate interesting and complex traffic scenarios through high-level descriptions. Instead of meticulously crafting the details of each individual scenario, language allows for a seamless conversion of semantic ideas into simulation scenarios at scale. To harness the capacity of natural language, we propose LCTGen. LCTGen takes as input a natural language description of a traffic scenario, and outputs traffic actors' initial states and motions on compatible map. As we will show in Section 5, LCTGen generates realistic traffic scenarios that closely adhere diverse range of natural language descriptions, including detailed crash reports [1].
natural language
solution. Natural language allows practitioners to
The major challenge of language- condutiino traffic generation is the absence of shared repre- sentation between language and traffic scenarios. Furthermore, there are no paired language-traffic datasets to support learning such a representation. To address these challenges, LCTGen (see Fig- ure 1) uses scenario-only dataset and a Large Language Model (LLM). LCTGen has three modules: Interpreter, Generator and Encoder. Given any user-specified natural language query, the LLM- powered Interpreter converts the query into compact, structured representation. Interpreter also retrieves appropriate map that matches the described scenario from real-world map library. Then, the Generator takes the structured representation and map to generate realistic traffic scenar- ios that accurately follow the user's specifications. Also, we design the Generator as a query-based Transformer model [2], which efficiently generates the full traffic scenario in a single pass.
This paper presents three main contributions:
1. We introduce LCTGen, a first-of-its-kind model for language-conditional traffic generation.
2. We devise method to harness LLMs to tackle the absence of language-seut scene paired data.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Map
Map
Nutrieval
21p522116.21
"ego vehicle turns right
Intersection
Interpretar(LUM)
V212A954444I VS (CA224AAA)
Vit 2.62:11
Generator
Text Description
Structured Representation
Bcenario
Figure I: Overview of our LCTGen model.
3. LCTGen exhibits superior realism and controllability over prior work. We also show LCTGen can be applied to instructional traffic editing and controllable self-driving policy evaluation.
Related Work
Traffic scenario generation traditionally rely on rules defined by human experts [3], e.g. rules that
enforce vehicles to stay in lanes, follow the lead vehicles [4, 5, 6] change lanes [7]. This approach is used in most virtual driving datasets [8, 9, 10, 11] and simulators [12, 3, 13]. However, traffic scenarios generated in this way often lack realism as they do not necessarily match the distribution of real-world traffic scenarios. Moreover, creating interesting traffic scenarios in this way requires non-trivial human efforts from experts, making it difficult to scale. In contrast, LCTGen learns the real-world traffic distribution for realistic traffic generation. Also, LCTGen can generate interesting scenarios with language descriptions, largely reducing the requirement of human experts.
Prior work in learning-based traffic generation is more related to work. SceneGen [14] uses toregressive models to generate traffic scenarios conditioned on ego-vehicle states and maps. Traf- ficGen [15] applies two separate modules for agent initialization and motion prediction. BITS [16] learns to simulate agent motions with a bi-level imitation learning method. Similar to LCTGen, these methods learn to generate realistic traffic scenarios from real-world data. However, they lack the ability to control traffic generation towards users' preferences. In contrast, LCTGen achieves such controllability via natural languages and the same time can generate highly realistic traffic sce- narios. Moreover, we will show in the experiments that LCTGen also outperforms prior work in the setting of unconditional traffic reconstruction, due to our query-based end-to-end architecture.
Text-conditioned generative models have recently shown strong capabilities for controllable con- tent creation for image [17], audio [18], motion [19], 3D object [20] and more. DALL-E [17] uses transformer model text and image tokens single stream of data. Noise2Music [18] uses conditioned diffusion models to generate music clips from text prompts. MotionCLIIP [19] achieves texr-oohhmman-motinn generation by aligning the latent space of human motion with pre-trained CLIP [21] embedding. These methods typically require large-scale pairs of content-text data for training. Inspired by prior work, LCTGen is the first-of-its-kine model for text-conditioned traf- fic generation. Also, due to the use of LLM and our design of structured representation, LCTGen achieves text-conditioned generation without any text-traffic paired data.
Large language models have become increasingly popular in natural language processing and lated fields due to their ability to generate high-quality text and perform language-related tasks. GPT-2 [22] is a transformer-based language model that is pre-trained on vast amounts of text data. Following this trend, GPT-3 [23] shows strong in-contett-learning capacity, and InstructGPT [24] improves the instruction following capacity by fine-tuning with human feedback to better align the models with their users. More recently, GPT-4 [25] demonstrates strong performance in both in- context learning and instruction following. In work, adapt the GPT-4 model [25] with chain- of of-thought [26] prompting method as Interpreter.
Preliminaries
Let m be map region, and s, be the state of all vehicles in scene at time t. A traffic scenario (m, S1.7) is the combination of map region and T timesteps of vehicle states $1:7 [s1, ... ST).

---

As Vehicie 1 approached the Intersection, driver did not notice the vehicles stopped ahead the traffic light. The traffic signal green Vehicle begen slowly move forward. The frontal plane Vehicle struck the rear plane of Vehicle 2...
Summary: V1 approaches an Intersection and does not notice v2 ahead...
Explanation: [V1) Because V1 moving, we assume VT's initial speed
mis (Index 4). V1 keeps going straight, so its actions are (keep speed),
GPT4
[V2] moving straight hits from behind,
front
Oulput:
VV: 01, 4, 4,4,, (3: 8, 4, 4) Mept: 12, 2, 2, 8,
Figure 2: Example Interpreter input and output. We only show partial texts for brevity. Map. We represent each map region m by set of S lane segments denoted by m = {(1...... US }. Each lane segment includes the start point and endpoint of the lane, the lane type (center lane, edge lane, or boundary lane), and the state of the traffic light control.
Vehicle states. The vehicle states s; = {s!,,... .e.,...... sNN } time consist of N vehicle. For each vehicle, model the vehicle's position, heading, velocity, and size. Following prior work [14, 15], choose the vehicle at the center of the scenario in the first frame as the ego-vehicle. It represents the self-driving vehicle in simulation platforms.
LCTGen: Language-Conditionel Traffic Generation
Our goal is to train
model ~ (, that produces traffic scenar- ios from a text description L and a dataset of maps M. Our model consists of three main compo- nents: A language Interpreter (Section 4.1) that encodes a text description into structured repre- sentation z. Map Retrieval m ~ Retrieval MM) that samples matching map regions m from dataset of maps M. A Generator (Section 4.3) that produces scenario T Generator m) from the map and structured representation 2. All components
stochastic, allowing to sample multiple scenes from single text description L and map dataset M. We train the Generator with real-world scenario-only driving dataset (Section 4.4).
4.1 Interpreter
The Interpreter takes a text description L as input and produces a structured representation 2 = Interpreter(L). After defining the representation 2, we show how to produce via GPT-4 [25].
Structured representation 2 [>" 2f..... 2f contains both map-specific 2nh and agent-specific components 2! For each scenario, we use 6--dimeniooaa vector 2nth describing the local map. It measures the number of lanes in each direction (north, south, east, west), the distance of the map center to an intersection, and the lane the ego-vehicle finds itself in. This compact abstract allows a language model to describe the important properties m and interact with map dataset M. For each agent i, = an d-dimentional integer vector describing the agent relative to the ego vehicle. contains agent's 's quadrant position index (1-4), distance range (0-20m, 20-40m...., orientation in- dex (north, south, east, west), speed range (0-2.5m/s, 2.5-5m/s, ...). and action description (turn left, accelerate, ...). Please refer to Supp.A. for a complete definition of z. Note that the representation does not have fixed length, as it depends on the number of agents in scene.
Language interpretation. To obtain the structured representation, we use a large language model (LLM) and formulate the problem into a text-to-text transformation. Specifically, we ask GPT-4 [25] translate the textual description of traffic scene into YAML-like description through in-context learning [27]. To enhance the quality of the output,
Chain-of-Thought [26] prompting to let GPT4 summarize the scenario in short sentences and plan agent-by-agent how to generate See Figure for an example input and output. Refer to Supp. A for the full prompt and Supp. D.4 for more complete examples.
4.2 Retrieval
The Retrieval module takes a map representation z and map dataset M, and samples map re- gions m ~ Retrieval(=mM .M). Specifically, we preprocess the map dataset M into potentially overlapping map regions {m1, m.,...). We sample map regions, such that their center aligns with the locations of an automated vehicle an offline driving trace. This ensures that the map region is both driveable and follows natural distribution of vehicle locations. For each map my, pre- compute its map representation 2/a. This is possible, as the map representation is designed to be
3

---

Representation
Motion Pred
Agent Query
Generative Transformer
Agent Feature Q"
Attribute Prod
Attribute
Map m
Position
Agent Mask
Map Mask MILP
Encoder
Map Feature F
Lane Masi gn2
Figure 3: Architecture of our Generator model.
both easy to produce programmatically and by a language model. Given 2"h the Retrieval ranks each map region my based its feature distance ||="! =/ Finally, Retrieval randomly samples m from the top-K K closest map regions.
4.3 Generator
Given a structured representation and map m, the Generator produces a traffic scenario т = Generator(z, m)). We design Generator as query-based transformer model to efficiently capture the interactions between different agents and between agents and the map. places all the agents in single forward pass and supports end-to-end training. The Generator has four modules (Figure 3): 1) map encoder that extracts per-lane map features F; 2) an agent query generator that converts structured representation 2; to agent query qú 3) generative transformer that models agent-agent and agent-map interactions; 4) scene decoder to output the scenario
Map encoder processes a map region m {ui,..., vs) with lane segments v; into a map feature F {fi..... fs}. and meanwhile fuse information across different lanes. Because S could be very large, multi-context gafing (MCG) blocks [28] for efficient information fusion. MCG blocks approximate transformer's cross-attention, but only attend single global context vector in each layer. Specifically, MCG block takes set of features f1.s as input, computes context vector then combines features and context in the output vi.s. Formally, each block is implemented via
MLP((:) @ MLP(c) where MaxPool(v1.s)
where @ is the element-wise product. The encoder combines 5 MCG blocks with 2-layer MLPs.
Agent query generator transforms the structured representation z of each agent into an agent query 4 d We implement this module as an MLP of positional embeddings of the structured representation yi MLP(PE(2{)) MLP(x;). We use a sinusoidal position encoding PE(-). We also add learnable query vector z; inputs, inspired by the object query in DETR [29].
Generative transformer. To model agent-agent and agent-map interactions, we use F' = (fi....,fst fs} and Q = (q....., 4N } as inputs and pass them through multiple transformer lay- ers. Each layer follows Q' = MHCA(MHSA(Q)) (Q), F), where MHCA, MHSA denote that multi-head cross- attention and multi-head self-attentin respectively [2]. The output of Q' in each layer used as the query for the next layer cross-attend to F. The outputs of the last-layer Q' are the agent features.
Scene decoder. For each agent feature q,' the scene decoder produces the agents position, attributes, and motion using an ML.. To decode the position, we draw inspiration from MaskFormer [30], placing each actor on a lane segment in the map. This allows us to explicitly model the positional relationship between each actor and the road map. Specifically, we employ an MLP to turn q into an actor mask embedding eggent E Rd Likewise, we transform each lane feature fy into per- lane map mask embedding d)me E Rd The position prediction P; E R for the i-th agent is then ú = softmax(e, agent x [elane ,...,,.lmmoy ),
For each agent query, we predict its attributes, namely heading, velocity, size, and position shift from the lane segment center, following Feng et al. [15]. The attribute distribution of potential agent is modeled with Gaussian mixture model (GMM). The parameters of K K-way GMM for

---

each attribute of agent are predicted as [1s T. = MLP(g)), ) where Mss E; and T. denote the mean, diagonal covariance matrix, and the categorical weights of the K-way GMM model.
We further predict the future T step motion of each agent, by outputting K' potential future MLP(((), where pos represents the k-th
trajectories for each agent: {pos prob,&J trajectory states for T future steps, and prob,, 4.k is its probability. Specifically, for each timestamp t, fo = (I, y, 0) contains the agent's position (ær y) and heading at t.
During inference, we randomly sample values from the predicted position, attribute, and motion distributions of each agent query to generate an output agent status through T time stamps <i.-- For categorical distributions, we select the category with the highest probability. For GMMs, we randomly sample value from the model. Compiling the output for all agents, derive the vehicle statuses $1:7. In conjunction with m, the Generator outputs the final traffic scenario (m, $1.T).
4.4 Training
The Generator is the only component of LCTGen that needs to be trained. We use real-world self- driving datasets, composed of D traffic scenarios {T, { For each traffic scene, we use an Encoder to produce the latent representation 2, then train the Generator to reconstruct the scenario.
Encoder. The Encoder takes traffic scenario and outputs structured agent representation: 2" Encoder(T). As mentioned in Section 4.1, x" contains compact abstract vectors of each agent {2f, }- For each agent i, the Encoder extracts from its position, heading, speed, and trajectory from the ground truth scene measurements si. in т, and converts into 2 following a set of predefined rules. For example, it obtains the quadrant position index with the signs of (ar, y) position. In this way, we can use Encoder to automatically convert any scenario latent codes z. This allows to obtain paired dataset (m, S1:N, : from scenario-only driving dataset.
Training objective. For each data sample (m, SI,N, <f.N), we generate a prediction p Generator(2, m). The objective to reconstruct the real scenario T. We compute the loss as:
[(P,T) (position(P (p, 7) + Laur T) + Lmottoo (p,T),
where Cposiitonn Latr; Lmation are losses for each of the different predictions. We pair each agent with ground-truth agent in based on the sequential ordering of the structured agent representation 2". We then calculate loss values for each component. For Cposition. we use cross-entropy loss between the categorical output P and the ground- truth lane segment id. For Laurs log-likelihood loss, computed using the predicted GMM on the ground-truth attribute values. For Lmotion. we use MSE loss for the predicted trajectory closest to the ground-truth trajectory. The training objective is the expected loss L over the dataset. We refer readers to Supp. B for more detailed formulations of the loss functions.
negative
Experiments
Datasets. We use the large-scale real-world Waymo Open Dataset [31], partitioning it into 68k traffic scenarios for training and 2.5k for testing. For each scene, we limit the maximum number of lanes to S = 384, and set the maximum number of vehicles to N = 32. We simulate T' = 50 timesteps at 10 fps, making each 7 represent a 5-second traffic scenario. We collect all the map segments the training split to obtain the map dataset M with 68K map regions.
Implementation. We query GPT-4 [25] (with a temperature of 0.2) through the OpenAl API for Interpreter. For Generator, we set the latent dimension d 256. We use 5-layer MCG block for the map encoder. For the generative transformer, we use a 2-layer transformer with 4 heads. We use a dropout layer after each transformer layer with dropout rate of 0.1. For each attribute prediction network, we use a 2-layer MLP with a latent dimension of 512. For attribute GMMs, we use K components. For motion prediction, we use K" 12 prediction modes. We train Generator with AdamW [32] for 100 epochs, with learning rate of 3e-4 and batch size of 64.
5

---

Initialization
Motion mFDE
Method
Pos
Heading
Speed
Size
mADE
SCR
TrafficGen [15]
0.2002
0.1524
0.2379
0.0951
10.448
20.646
5.690
MotionCLIP [19]
0.1236
0.1446
0.1958
0.
6.683
13.421
8.842
LCTGen (w/o 2)
0.1319
0.1418
0.1948
0.1092
6.315
12.260
8.383
LCTGen
0.0616
0.1154
0.0719
0.1203
1.329
2.838
6.700
Table 1: Traffic scenario generation realism evaluation (lower the better).
5.1 Scene Reconstruction Evaluation
We evaluate the quality of LCTGen's generated scenarios by comparing them real scenarios from the driving dataset. For each scenario sample (7, m) in the test dataset, generate scenario with f = Generator(z, m) and then compute different metrics with and f.
Metrics. To measure the realism of scene initialization, we follow [14, 15] and compute the maxi- mum mean discrepancy (MMD [33]) score for actors' positions, headings, speed and sizes. Specif- ically, MMD measures the distance between two distributions q and p. For each pair of real and generated data (T, f), we compute the distribution difference between them per attribute. To mea- sure the realism of generated motion behavior, employ the standard mean average distance error (mADE) and mean final distance error (mFDE). For each pair of real and generated scenarios (T, +), we first use the Hungarian algorithm to compute a matching based on agents' initial locations with their ground-truth location. We then transform the trajectory for each agent based on its initial po- sition and heading to the origin of its coordinate frame, to obtain its relative trajectory. Finally, we compute mADE and mFDE using these relative trajectories. We also compute the scenario collision rate (SCR), which is the average proportion of vehicles involved in collisions per scene.
Baselines. We compare against a state-of-the-arr traffic generation method, TrafficGen [15]. As TrafficGen only takes a map as input to produce a scenario we train a version of LCTGen that also only uses m as input for a fair comparison, referred to as LCTGen (w/o 2). We also compare against MotionCLIP [19], which takes both a map m and text L as input to generate a scenario т. Please refer to Supp. C for the implementation details of each baseline.
Results. The results in Table indicate the superior performance of LCTGen. In terms of scene initialization, LCTGen (w/o 2) outperforms TrafficGen in terms of MMD values for the Position, Heading, and Speed attributes. Importantly, when conditioned on the language input L, LCTGen significantly improves its prediction of Position, Heading, and Speed attributes, significantly out- performing both TrafficGen and MotionCI.IP on MMD (> 2x). LCTGen also achieves 7-8x smaller mADE and mFDE than baselines when comparing generated motions. The unconditional version of LCTGen, without 2, also outpaces TraffieGen in most metrics, demonstrating the effectiveness of Generator's query-based, end-to-end transformer design. We note that LCTGen (w/o) has an on-par Mize-MMD score with TrafficGen, which is lower than LCTGen. We conjecture that this is because our model learns spurious correlations of size and other conditions in z in the real data.
5.2 Language-conditionee Simulation Evaluation
LCTGen aims to generate a scenario 7 that accurately represents the traffic description from the input text L. Since no existing real-world text-scenario datasets are available, we carry out our experiment using text T. from a text-only traffic scenario dataset. To evaluate the degree of alignment between each scenario and the input text, we conduct human study. We visualize the output scenario generated by LCTGen or the baselines, and ask humans to assess how well it matches the input text.
Datasets. We use a challenging real-world dataset, the Crash Report dataset [1], provided by the NHTSA. Each entry in this dataset comprises a comprehensive text description of a crash scenario, including the vehicle's condition, driver status, road condition, vehicle motion, interactions, and more. Given the complexity and intricate nature of the traffic scenarios and their text descriptions, this dataset presents significant challenge (see Figure 2 for an example). We selected 38 cases from this dataset for the purposes of our study. For more controllable evaluation, also use an Attribute Description dataset. This dataset comprises text descriptions that highlight various

---

Crash Report
Attribute Description Ours Prefered (%)
Method
Ours Prefered (%)
Score (1-5)
Score (1-5)
TrafficGen [15] MotionCLIP [19]
92.35
1.58
90.48 95.60
2.43
95.29
1.65
2.10
LCTGen
3.86
4.29
Table 2: Human study results the Inaguage-condttions simulation.
"VI treveling oust left turn and attempts tumn left when it rraveing lift through lane."
"V' and V2 callide at an intersetionn of urban trafflwwyy, with striking the leltt side VI"
"the scene is sparse. more ony ego vahiclk cars are moving in fast speed. the turn left."
"the ego car tumns right, most cars are mosing spaedd"
Figure 4: Qualitative results on cext-conditioned generation.
attributes a traffic scenario. These include aspects like sparsity ("the scenario is dense"), position ("there are vehicles on the left"), speed ("most cars are driving fast"), and the ego vehicle's motion ("the ego vehicle turns left"). We create more complex descriptions by combining 2, 3, and attributes. This dataset includes 40 such cases. Refer Supp. C for more dataset details.
Baselines. We compare with TrafficGen and MotionCLIP. For each text input L, LCTGen outputs a scenario 7 = (m, $1.)). To ensure fairness, we feed data m to both TrafficGen and MotionCLII to generate scenarios on the same map. As TrafficGen does not take language condition as input, we only feed T. to MotionCLII. addition, TrafficGen can't automatically decide the number of agents, therefore uses the same number of agents as our output T.
Human study protocol. For each dataset, we conduct a human A/B test. We present the evaluators with a text input, along with pair of scenarios generated by two different methods using the same text input, displayed a random order. The evaluators are then asked to decide which scenario they think better matches the text input. Additionally, evaluators are requested to assign score between and 5 to each generated scenario, indicating alignment with the text description; a higher score indicates better match. A total of 12 evaluators participated in this study, collectively contributing 1872 scores for each model.
We provide preference score, reflecting the frequency with which LCTGen's 's output is chosen as a better match than each baseline. We also provide the average matching score, indicating the extent to which evaluators believe the generated scenario matches the text input. With LCTGen often chosen as the preferred model by human evalua- tors (at least 90% of the time), and consistently achieving higher scores compared other methods, these results underline its superior performance in terms of texl-controllabiliiy over previous works. The high matching score also signifies LCTGen's 's exceptional ability to generate scenarios that faith- fully follow the input text. We include more analysis human study result in Supp. D.3.
Quantitative Results. We show the results in Table 2.
Qualitative Results. We show examples of LCTGen output given texts from the Crash Report (left two) and Attribute Description (right two) datasets in Figure 4. Each example pair of input text and the generated scenario. Because texts in Crash Report excessively long, only show the output summary of our Interpreter for each example (Full texts in Supp. C). Please refer to Supp. video for the animated version of the examples here. We show more examples in Supp. D.2.
5.3 Application: Instructional Traffic Scenario Editing
Besides language-conditioned scenario generation, LCTGen can also be applied to instructional traf- fic scenario editing. Given either real generated traffic scenario along with editing instruc- tion text T, LCTGen can produce an edited scenario è that follows 7. First, we acquire the structured

---

Input
"make the car front turn left" "remove all the horizontal cars" "add more cars on the left" "speed up same-direction cars"
Figure 5: Instructional editing on real-world scenario. Refer to Supp. for full prompts.
Method
Pos
Hcading
Speed
Size
Method
mADE
mFDE
SCR
w/o Quad.
0.092
0.122
0.076
0.124
w/o Speed
2.611
5.188
7.150
Dist.
0.071
0.124
0.073
0.121
w/o Action
2.188
5.099
7.416
w/o Ori. 0.067
0.132
0.082
0.122
LCTGen init. [15) motion
2.467
5.682
5,210
LCTGen
0.062
0.115
0.072
0.120
LCTGen
1,329
2.838
6.700
(a) Ablation study for scene initialization.
(b) Ablation study motion behavior generation.
Table 3: Scene reconstruction ablation study the Waymo Open Dataset.
representation of the scenario using = Encoder(r). Next, we compose unique prompt that structs Interpreter alter in accordance with 1, resulting generate the edited scenario f Generator(2, m.), where m is the same map used in the input.
Interpreter(2, I). Finally,
We show an example of consecutive instructional editing of a real-world scenario in Figure 5. We can see that LCTGen supports high-level editing instructions (vehicle removal, addition and action change). produces realistic output following the instruction. This experiment highlights LCTGen's potential for efficient instruction based traffic scenario editing. As another application of LCTGen, also show how LCTGen be utilized to generate interesting scenarios for controllable self- driving policy evaluation. Please refer Supp. D.1 for this application.
5.4 Ablation study
Scene initialization. Table summarizes the results, where the last row corresponds to our full method. To validate the performance of LCTGen for scene initialization, we mask out the quadrant index, distance, and orientation in the structure representation z for each agent, respectively. As result, we observed significant performance drop, especially in the prediction of Position and Heading attributes, as shown in the left side of Table 3. This suggests that including quadrant index, distance, and orientation in structured representation is effeetive.
Motion behavior generation. We summarized the results in Table 3 (right). By masking out the speed range and action description in the structured representation for each agent, we observed significant performance drop in the metrics for motion behavior. Moreover, we initialize the scene with LCTGen while generating agents' motion behavior using TrafficGen's 's [15], we also observed significantly worse performance than using LCTGen to generate the traffic scenario in one shot. The results suggest that the end-to-end design of scene initialization and motion behavior generation by our LCTGen can lead to better performance. We show more ablation study results in Supp. D.5.
Conclusion
In this work, we present LCTGen, a first-of-its-king method for language-conditiong traffic scene generation. By harnessing the expressive power of natural language, LCTGen can generate realistic and interesting traffic scenarios. The realism of our generated traffic scenes notably exceeds previ- ous tfate-of-the-ar methods. We further show that LCTGen can be applied to applications such as instructional traffic scenario editing and controllable driving policy evaluation.
The primary constraint LCTGen lies in the Interpreter module's 's inability to output perfect agent placements and trajectories, as it lacks direct access detailed lane information from the map. Our future work aims overcome these issues by equipping the Interpreter with map and math APIs, enabling to fetch precise map data and output more comprehensive traffic scenarios.
Limitations.

---

Acknowledgement. We thank Yuxiao Chen, Yulong Cao, and Danfei Xu for their insightful discus- sions. We thank all the human evaluation participants for their time and effort our experiments. We also appreciate the constructive comments from the anonymous reviewers. This material is supported by the National Science Foundation under Grant No. IIS-1845485.
References
[1] National Highway Traffic Safety Administration. Crash injury research engineering network. https://crashvieeer www.thaddoo.... wtt./ggoo////////ee rearchIndex, 2016.
[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and
I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/ppoooeeediiinnn nneurr http:///////////////o///////////
Paper. .ddf.
[3] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Llicknn, Rummel, Wagner, and E. Wießner. Microscopic traffic simulation using sumo. In The s IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018. URL http:://elib.ddr..de/100
[4] L. Papaleondiou and M. Dikaiakos. Trafficmodeler: graphical tool for programming micro- scopic traffic simulators through high-level abstractions. In VETECS, pages 1 5, 05 2009. doi: 100310000000 2009.503881.
[5] J. Maroto, E. Delso, J. Félez, and J. Cabanellas. Real-time traffic simulation with microscopic model. Intelligent Transportation Systems, IEEE Transactions on, 7:513 527, 01 2007. doi: T..109//ITS.2006.380
[6] E. Gunawan. Two-vehicle dynamics of the car-following models on realistic driving con- dition. Journal of Transportation Systems Engineering and Information Technology. 12(2): 67 75, 2012. ISSN 1570-6672. ddcchttss///kkkooo////ooo.166665
1661177117777111119993 URL
http://ww,, http://www..ccceeeeeeccccc.co/// wtw.:c/ccccccctccccccc.///ttttta
[7] J. Erdmann. Lane-changing model in sumo. In Proceedings of the SUM02014 Modeling Mobility with Open Data, volume 24, 05 2014.
[8] M. Wrenninge and J. Unger. Synscapes: A photorealistic synthetic dataset for street scene parsing. Arxiv, Oct 2018. URL
wtt..or0///b//110000/.... 08705.
[9] G. Ros, L. Sellart, J. Materzynska, Vazquez, and M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3234-3243, 2016.
[10] M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan. Driv- ing in the matrix: Can virtual worlds replace human- menaareeeeertedd annotations for real world tasks? In IEEE International Conference Robotics and Automation, pages 1-8, 2017.
[11] S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
[12] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. Proceedings of the st Annual Conference on Robot Learning, pages 1-16, 2017.
[13] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield. Structured domain randomization: Bridging the reality gap by context-aware synthetic data. In ICRA, pages 7249-7255, 05 2019. 200100.109//CRA2211999.990

---

[14] S. Tan, K. Wong, S. Wang. S. Manivasagam, M. Ren, and R. Urtasun. Scenegen: Learning to generate realistic traffic seenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 892-901, 2021.
[15] L. Feng, Q. Li, Z. Peng, S. Tan, and B. Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In IEEE International Conference on Robotics and Automation, London, United Kingdom, May 2023.
[16] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone. BITS: Bi-level imitation for traffic simulation. In IEEE International Conference on Robotics and Automation (ICRA), London, UK, May 2023. URL.
[17] A. Ramesh, M. Paviov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[18] Q. Huang, D. S. Park, T. Wang. T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang. Yu, C. Frank, al. Noise2musíc: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917, 2023.
[19] G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or. Motionclip: Exposing human motion generation to clip space. In Computer Vision ECCnn 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 358-374. Springer, 2022.
[20] J. Gao, T. Shen, Z. Wang, W. Chen, K. Yin, D. Li, O. Litany, Z. Gojeic, and S. Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural Information Processing Systems, 2022.
[21] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748 8763. PMLR, 2021.
[22] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[23] T. Brown, B. Mann, N. Ryder, M. Subbiah. J. D. Kaplan, P. Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. verbert-Vosss, G. Krueger, Henighan, R. Child, A. Ramesh, D. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Rad- ford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Asso- ciates, Inc., 2020. URL https::/prooceedings wtw.ripp.ccpappeeffiiii. filess/ppeer//22// #ite/457700001105574448866666662 Papedr.---pp
[24] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[25] OpenAl. Gpt-4 technical report, 2023.
[26] Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and Zhou. Chain of thought prompting elicits reasoning in large language models, arXiv preprint arXiv:2201.11903, 2022.
[27] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Re- thinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
10

---

[28] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cormman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov, and B. Sapp. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction, 2021.
[29] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Computer Vision ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, page 213-229, Berlin, Heidel- berg, 2020. Springer- Verlag. ISBN 978-3-030-58451--1 URL https:://dii.org///////// 1607/978 3-000--3-------0-0 58452-8.12
[30] B. Cheng. A. G. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021.
[3]] Waymo LLC. Waymo open dataset: An autonomous driving dataset, 2019.
[32] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017.
[33] A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[34] Q. Z. Peng. L. Feng, Q. Zhang, Z. Xue, and Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[35] M. Treiber, Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical Review E, 6252);1805-1824, aug 2000. doi:10.1103/ physteve.62.1805. URL https://doi. org/10. 110342Fphys 62. 1805.
[36] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, Bhosale, Bikel, L.. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

---

Appendix
In the appendix, we provide implementation and experiment details of our method as well as ad- ditional results. In Section A and Section B, we show details of Interpreter and Generator re- spectively, In Section C we present implementation details of our experiments. In Section D, show more results on applications ablation study, well additional qualitative results. Finally, in Section E, present analysis on LCTGen with different LLMs.
Interpreter
A.1 Structured representation details
The map specific TM a 6-dim integer vector. Its first four dimensions denote the number lanes each direction (set as north for the ego vehicle). The fifth dimension represents discretized distance in 5-meter intervals from the map center the nearest intersection (0-5, 5-10.... The sixth dimension indicates the ego vehicle's lane id, starting from for the rightmost lane.
For agent i, the agent-specific z, is an 8-dim integer vector describing this agent relative to the ego vehicle. The first dimension denotes the quadrant index (1-4), where quadrant represents the front- right of the ego vehicle. The second dimension is the discretized distance to the ego vehicle with a 20m interval, and the third denotes orientation (north, south, east, west). The fourth dimension indicates discretized speed, in 2.5m/s intervals. The last four dimensions describe actions the next four seconds (one per second) chosen from discretized set of seven possible actions: lane changes (left/right), turns (left/right), moving forward, accelerating, decelerating, and stopping.
A.2 Generation prompts
The scenario generation prompt used for Interpreter consists several sections:
1. Task description:
simple description of task of scenario generation and output formats.
2. Chain-of-thought prompting [26]: For example, "summarize the scenario in short sen- tences", "explain for each group of vehicles why they are put into the scenario".
3. Description of structured representation: detailed description for each dimension of the structured representation. We separately inform the model Map and Actor formats.
4. Guidelines: several generation instructions. For example, "Focus on realistic action gener- ation of the motion reconstruct the query scenario".
5. Few-shot examples: A few input-output examples. We provide Crash Report example.
We show the full prompt below:
Prompt 1:
Full prompt for Interpreter scenario generation.
You are a very faithful format converter that translate natrual language traffic scenario descriptions to a fix-form format to appropriately describe the scenario with motion action. You also need to output an appropriate map description that is able to support this scenario. Your ultimate goal to generate realistic traffic scenarios that faithfully represents natural language descriptions normal scenes that follows the traffic rule.
Answer with list of vectors describing the attributes of each of the vehicles in the scenario.
Desired format:
Summary: summarize the scenario in short sentences, including the number of vehicles. Also explain the underlying map description.
Explaination: explain for each group of vehicles why they are put into the scenario and how they fullfill the requirement in the description.
Actor Vector:
attributes each vehicles in the scenario, only output the values without any text:
of vectors describing
'Vl': Lunrn)
12

---

10 'V2':
'V3': [......
12 Map Vector: A vector describing the map attributes, only output the values without any text:
13 'Map': [.....]
15 Meaning of the Actor vector attribute:
16 dim 0: 'pos'; [-1,3] whether vehicle
four quadrant ego vechile
order [0 'front left',
'back left', 2- 'back right', 'front right
the vehicle is the ego vehicle.
dim 1: 'distance': [0.3] the distance range index of the vehicle towards the ego vehicle range is from 0 to 72 meters with 20 meters interval. if the vehicle is the ego vehicle. For example, if distance value is 15 meters, then the distance range index is 0.
18 dim 2: 'direction': [0,3] the direction of the vehicle relative to the ego vehicle, in the order of [e- 'parallel same', 1-'paratlel opposite', 2- 2-'perpendiculat up', 3- perpendicular.dowe
the vehicle the ego vehicle.
19 dim 3: 'speed': : [0.20] the speed range index of the vehicle; range from with 2.5 interval. For example, 20m/s in range 8, therefore the speed value
m/s
8.
dim 4-7: 'action': [0,7] 4-dim, generate actions into the future 4 second with each two actions have time interval of is (4 actions in total), the action ids are [0 'stop 1. "turn left', 2 'left lane change', 3- 'decelerate', 4- keep_sspedd', 5- accelerate', 6- G-'right lane change", 7- turu right'l.
21
Meaning of the Map attributes:
23
dim 0-1: "paralletLlane_cnt' 2-dim. The first dim ego lane, and second dim number of parallel opposite-
number of parallel same-
direction lanes direction lanes the ego lane.
dim 2-3: perpendicular_Lanee
2-dim. The first dim is the number of perpendicular gpstream-direction lanes, and the second dim is the number of perpendicular downstrean direction lanes
25 dim 4: 'dist_ intersection' 1-dim. the distance range index of the ego vehicle to the intersection center in the x direction, range is from 0 to 72 meters with meters interval. -1 there is no intersection
the scenario.
26
5: 'lane id': 1-dim. the lane id of the ego vehicle, counting from the rightmost lane 1. For example, the ego vehicle in the the leftmost lane,
the same same- direction lanes, starting if the ego vehicle rightmost lane, then the lane then the lane id is the number of the dime-dirctiin lanes.
Transform the query sentence to the Actor Vector strictly following the rules below: Focus an realistic action generation of the motion to reconstruct the query scenario. Follow traffic rules to form fundamental principle in most road traffic systems to ensure safety and smooth operation of traffic. You should incorporate this rule into the behavior of our virtual agents (vehicles).
29
30
31
Traffic rule: in an intersection, when the vehicles on one side the intersection are crossing, the vehicles on the other
of the intersection should be waiting. For example, if v1 is crossing the intersection and V2 is the perpendicular lane, then V2 should be waiting.
For speed and distance, convert the unit to m/s and meter, and then find the interval index in the given range.
33 Make sure the position and direction of the generated vehicles are correct.
34 Describe the initialization status of the scenario.
35
During generation, the number of the vehicles within the range of [1, 32).
36 Always generate the ego vehicle first (V1).
37 Always assume the ego car in the center the scene and is driving the positive x direction.
the input descriptions, regard other vehicles are the surrounding vehicles. For example, for "Vehicle southbound", the ego car is Vehicle 1.
Vehicle 1 Unit
the ego vehicle. All the traveling
39
If the vehicle is stopping, its speed should be 0m/s (index 0). Also, if the first action is 'stop', then the speed should be 0m/s (index 0).
40
Focus on the interactions between the vehicles in the scenario.
41
Regard the last time stamp as the time stamp of second into the future.
42
43 Generate the Map Vector following the rules below:
there is vehicle turning left right, there must be an intersection ahead. lane with the direction the ego lane; 1.e., the first dim of Map should be at least 1. For example, if this is one way two lane road, then the first dim of Map should be 2.
Should at least have
16 Regard the lane at the center of the scene as the ego lane.
13

---

47 Consider the ego car's direction the positive x direction. for example, for *VI
traveling northbound in lane five of a five lane controlled access roadway", there should be lanes in the same direction as the ego lane.
48 The generated map should strictly follow the map descriptions in the query text. For example, for "Vehicle was traveling southbound", the ego car should be in the southbound lane.
89
there is an intersection, there should be at least one lane in either the upstream or downstream direction.
If there is intersection, the distance to the intersection should be -1.
There should be vehicle driving vertical to the ego vehicle in the scene only when there is intersection in the scene. For example, when the road is just two-way, there should not be any vehicle driving vertical to the ego vehicle.
52 If no intersection is mentioned, generate intersection scenario randomly with real-world statistics.
53
55 Query: crash occurred during daylight hours a dry, bituminous, two- two-lane roadway under clear skies. There was northbound travel lane and one southbound travel lane with speed limit of 40 km/h (25 mph). The northbound lane had a -3.6 percent grade and the southbound lane had a +3.6 percent grade. Both travel lanes were divided by double yellow line. A 2016 Mazda CX-3 3 (V1) was in a parking lot attempting to execute a left turn to travel south. A 2011 Dodge Charger (V2/police car) was traveling north responding to an emergency call with lights sirens activated. V1 was in a parking lot ( facing west) and attempted to enter the roadway intending to turn left. As v1 entered then
the roadway it was impacted on the left side by the front of (Event 1). rotated counterclockwise and traveled off the west road edge and impacted an embankment with front left bumper (Event 2). After initial impact v2 continued on northern direction and traveling final rest approximately 40 meters north of impact area facing north in the middle of the roadway. V1 and V2 towed from the scene due to damage.
56
57 Summary: v1 attempts to turn left from a parking lot onto a two- roadway and is struck by V2, a police car traveling north with lights and sirens activated, There are 2 vehicles in this scenario. This happens on parking lot to a two-lane two-way road with intersection.
58
Explanation:
59
(ego vehicle) is attempting find vI's speed in the query. Because tries to turn left, its initial speed should be set low. We set V1's speed as m/s, which has the index of 2. V1 turns left, so its actions are all 1 (turn left).
left from parking lot the roadway. We cannot
60
V2 is a police car traveling north with lights and sirens activated. As V1 is turning left 5 seconds before the crash, V1 is facing west and V2 is coming from northbound, crossing the path of V1. In the coordinates of V1 (which is facing west initially), V2 comes from the front and is on the left side. Hence, V2's position "front left" (3).
V1 is facing west and V2 facing north, V2 is moving in the perpendicular down cannot find
direction with VI. Therefore speed in the query. Because V2 assume V2's init speed is m/s (index 4), Given this speed, V2's distance to v1 is m/s 5s 50m (index 10). V2 keeps going straight, speed).
direction is (perpendicular down)_ police car responding to emergency call,
its actions are all 4 (keep
61 Map: v1 tries to turn left from partking lot onto a two-lane roadway. There are a one- way exit lane from parking lot (one same- direction parallel) and the ego vehicle is in the left turn lane with lane 1. On the perpendicular side there is a two-land roadway. V1 is about turn left, so the distance to the intersection is set to be 10m (index 2).
62 Actor Vector:
'Vi': [-1, 0, 2,
'V2': [0, 10, 3, 4, 4, 4,
Map Vector:
'Map': [1, 8, 1, 1,
2,
1J
67
68 Query: INSERT_ QUERY HERE
20
Output:
A.3 Instructional editing prompts
We also provide Interpreter another prompt for instructional scenario editing. This prompt follow similar structure to the generation prompt. We mainly adopt the task description, guidelines,
14

---

and examples to scenario editing tasks. Note that for the instructional editing task, we change the distance interval (second dimension) of agent-specific =f from 20 meters to meters. This is to ensure the unedited agents stay the same region before and after editing.
We show the full prompt below:
Prompt 2: Full prompt for Interpreter instructional scenario editing.
You are traffic scenario editor that edit fix-form traffic scenario descriptions according to user's natural language instructions.
The user will input fix-form traffic scenario description well the map description.
The also natural language instruction to modify the scenario. You need to output fix-form traffic scenario that is modified according to the instruction.
Input format:
V1:
[.....
V2:
V3:
Map: L.....]
10
Instruction: natural language instruction modify the scenario.
Output format:
Summary: summarize the scenario in short sentences. summarize the user instruction, and indicate which part of the scenario should be modified.
14 Explaination: explain step-by-sted
how each part of the scenario is modified.
15
Actor Vector: list of vectors describing the attributes of each of the vehicles. Only the vehicles that are modified should be included in the output. 16 V2: [.......]
17
Meaning of the Actor vector attribute:
dim 0: "pos'; [-1,3] whether the vehicle is the four quadrant of ego vechile in the order of [0 'front left', 1 'back left', 2- 'back right', 'front right']. -1 if the vehicle is the ego vehicle.
20 dim 1: 'distance': [0, 14] the distance range index of the vehicle towards the
vehicle; range is from to 72 meters with 5 meters interval. 0 if the vehicle is the ego vehicle.
21 dim 2: 'direction': [0,3] the direction of the vehicle relative to the ego vehicle, in the order of [0- 'parallel same', 1. 1-'parallel opposite', 2 2-'perpendicular up', vehicle 22 dim 3: 'speed': [0.8] the speed range index the vehicle; range is from with 2.5 interval. For example, 20m/s is in range 8, therefore the speed value
perpendicular
ego vehicle.
m/s
8.
dim 4-7: 'action': [8,7] 4-dim, generate actions into the future 4 second with each two actions have time interval of is (4 actions in total), the action ids are [0 'stop 1. 'turn left', 2 'left lane change', 3- 'decelerate', 4- 'keep speed', , 5. accelerate', 6- 'right lane change', 7- turu right'].
Meaning of the Map attributes:
26
dim 0-1: "paralleL_lane_cnt
2-dim. The first dim direction lanes the ego lane, and the second dim the number of parallel opposite-
number of parallel same-
direction lanes of the ego lane.
dim 2-3: "perpendicular_ l_n_llaeecct_ 2-dim. The first dim is the number of perpendicular upstrean-direction lanes, and the second dim is the number of perpendicular downstream direction lanes.
28 dim 4: 'dist_ intersection': 1-dim. the distance range index of the ego vehicle to the intersection center in the x direction, range is from 0 to 72 meters with meters interval. -1 there is no intersection in the scenario.
29
5: 'lane id': 1-dim. the lane id of the ego vehicle, counting from the rightmost lane the same same-di direction lanes, starting from 1. For example, the ego vehicle in the rightmost lane, then the lane id
if the ego vehicle then the lane id is the number of the same-direction lanes.
the leftmost lane,
31
Follow the instructions below:
32
"V1' is the ego vehicle, and the other vehicles are the surrounding vehicles. The user will input a fix-form traffic scenario description as well as the map description
33
The user also an natural language instruction to modify the scenario. You need to output fix- form traffic scenario that is modified according the instruction.
34 First figure out which part the scenario should modified according to the should turn vehicle front
instruction. For example, left", then the vehicle in front the ego vehicle should be modified.
instruction
15

---

36 Input:
37 Actor vector:
38 v1: [-1, 8, 0, 0, 4,
4, 41
39 V2:
2, 1, 0, 1,
4,
4,
4,
40 V3:
3, 3, 0, 1, 4,
4,
4, 01
41 V4: 3, 4, 0, 8, 4, 4, 2,
42 V5:
[0, 9, 1, 8, 1,
5, -11
43 V6: 3.
0,
0,
V7: [ 9, 3, 0, 0,
0, 0,
v8: 3, 10, 3, 3, 4, 5 1, 8]
46 V9: 0,, 10,
3, 0,
0
0
0, -11 0,
47 V10: 3,, 10,
2,
0
0
0, -1]
48 v11: [ 3, 11,
2,
8
8 0] 0
0
8
49 V12: [ 3, 11,
2,
0,
7,
0, 0]
50 Map: [4, 3, 2, 3,
6,
4]
51
52 Instruction: move vehicle behind the ego vehicle opposite lane move faster.
Output:
Summary: The instruction is to move the vehicle behind the ego vehicle to the opposite lane and accelerate. First find which vehicle is behind the ego vehicle. There are only 1 vechile behind the ego vehicle, that is V2 (with position=2, indicating on the right back side of the ego vehicle). Therefore, the vehicle v2 should be modified.
56
Explainations The vehicle V2 is modified to move to the opposite lane and accelerate. The in the right back side the ego vehicle, and the ego vehicle
vehicle
rightmost lane of the same-direction lanes. Therefore, the leftmost lane opposite to the ego vehicle, changed to (parallel..ppositee). In this lane, should be moved to the left back of the ego car, its position should be changed to 1. V2 should move faster, its speed should be changed to 10 (25 m/s). Actor vector:
vehicle should move to directioo lanes. Therefore, V2's direction should be
opposite-direction
57
58
V2: 1, 1, 1, 10, 4, 4, 4, 4]
59
60
Instrontion: remove all the vehicles on the front of the ego car and moving the same direction.
Output:
Summary:
The instruction to moving in the same direction. First find which vehicles are the front of the ego vehicle. V3-V12 are all the front of the ego vehicle. Then, only V3, V4 and V6 has the same direction as the ego vehicle (G). Therefore, V3, V4 and V6 should be removed.
all the vehicles the front of the ego and
64
Explaination: V3, V4, V6 are on the front of the ego vehicle and moving in the same direction. V3, V4 and V6 are removed from the scenario.
65
66 Actor vector:
V3:
removed.
68 V4: removed.
v6:
removed.
Input: ESSERT_QUERY_HEEE
23 Output:
B Generator
B.1 Training objectives
In the main paper, show the full training objective of Generator
L(p,T) = Lpositiion T) + Laue (p, r) + Lmostoo (p, T).
(2)
In this section, we provide details of each loss function. We first pair each agent à; in p with ground-truth agent a; in based on the sequential ordering of the structured agent representation 2a Assume there are in total N agents in the scenario.
16

---

For Cposition, we use eross- loss between the per-lane categorical output p and the ground- truth lane segment id I. Specifically, we compute it as
N Σ
Pposition (p, 7)
log (4;),
(3)
where I; is the index of the lane segment that the i-th ground-truth agent is on.
For Latrs we use a negative log- leeliiklioood loss, computed using the predicted GMM on the ground- truth attribute values. Recall that for each attribute of agent we use an MLP to predict the param-
eters of GMM model [ny, =.] Here, we these parameters construct GMM model and compute the likelihood of ground-truth attribute values. Specifically, we have
Cas (p. T)
log GMMpesding (h;) log GMM
(4)
log ( log GMMpuxi(poss)).
where GMMnedinggi GMMyeli- GMMaize,, GMM represent the likelihood function of the pre- dicted GMM models of agent i's heading, velocity, size and position shift. These likelihood values are computed using the predicted GMM parameters. Meanwhile, h, vel, bboz; and pos; represent the heading, velocity, size and position shift of the ground-truth agent a; respectively.
For Limotionn we use MSE loss for the predicted trajectory closest to the ground-truth trajectory fol- lowing the multi-path motion prediction idea [28]. Recall that for each agent â, we predict K" different future trajectories and their probabilities as {pos? prob, 1K" = MLP(4 For each timestamp t, pos contains the agent's 's position and heading, We assume the trajectory of ground- truth agent ag is pos?:T. We can compute the index k* of the closest trajectory from the K' pre- dictions as ht* = arg ming 2 pos )2 Then, we compute the motion loss for agent i as:
Lmution, log prob, k + E(poss post,.)
(5)
where encourage the model to have a higher probability for the cloest trajectory K- and reduce the distance between this trajectory with the ground truth. The full motion loss is simply:
Emotion (p, T)
Σ
mottion.
(6)
where we over all the motion losses for each predicted agent in p.
C Experiment Details
C.1 Baseline implementation
TrafficGen [15]. We use the official implementation'. For fair comparison, we train its Initial- ization and Trajectory Generation modules on our dataset for 100 epochs with batch size 64. We modify T = 50 the Trajectory Generation to align with our setting. We use the default values for all the other hyper- parammerrseters..... During inference, we enforce TrafficGen to generate vehicles by using the result of the first N autoregressive steps of the Initialization module.
Motion [19]. The core idea of MotionCLIP is to learn shared space for the interested modality embedding (traffic scenario in our case) and text embedding. Formally, this model contains a scenario encoder E, text encoder É, and scenario decoder D. For each example of scene- text paired data (7, I, m). we encode scenario and text separately with their encoders z = E(T),
17

---

2 = E(L). Then, the decoder takes and m and output scenario p D(z, m). MotionCLIP
trains the network with faee to reconstruct the scenario from the latent code:
Frec = posiition (p, 7) (p, 7) Lmotion (p,
where we use the same set of loss functions as ours (Equation 2). On the other hand, MotionCLII aligns the embedding space of the scenario and text with:
Calign 1 (z, 2),
(8)
which encourages the alignment of scenario embedding z and text embedding 2. The final loss function is therefore
C = L100 + Aaalign
where we set À = 100.
During inference, given an input text L and a map m, we can directly use the text encoder to obtain latent code and decode scenario from it, formally == ((É(L),
For the scenario encoder E, we use the same scenario encoder in [15], which is 5-layer multi- context gating (MCG) block [28] to encode the scene input т and outputs E IR 1024 with the context vector output e of the final MCG block. For text encoder É, we use the sentence embedding of the fixed GPT-2 model. For the scenario decoder D, we modify our Generator to take in latent representation with a dimension of 1024 instead of our own structured representation. Because D does not receive the number agents as imput, we modify Generator to produce the 32 agents for every input and additionally add MLP decoder predict the objectiveness score of each output agent. Here objectiveness score is a binary probability score indicating whether we should put each predicted agent onto the final scenario or not. During training, for computation of Cree, we use Hungarian algorithm to pair ground-truth agents with the predicted ones. We then supervise the objectiveness score in a similar way as in DETR.
Note that need text-scenario paired data train MotionCLIP. To this end, method to convert real dataset scenario with language. Similar to our Attribute Description dataset, in each text, we enumerate the scenario properties 1) sparsity; 2) position; 3) speed and 4) ego vehicle's motion. Here is one example: "the scene is very dense; there exist cars on the front left of ego car; there is no car on the back left of ego car; there is no car on the back right of ego car; there exist cars on the front right of ego car; most cars are moving in fast speed; the ego car stops".
rule-based text L. This is done by describing different attributes of the
We transform every scenario in dataset into text with the format above. We then train MotionCLIP on our dataset with the same batch size and number of iterations as LCTGen.
C.2 Metric
We show how compute MMD in this section. Specifically, MMD measures the distance between two distributions g and p.
MMD (p. 4) = (x,,, (x,,x'))) x')] Eyy.'' (Ky.
(10)
2E B. k<<,y)))
where k is the kernel function (a Gaussian kernel in this work). We use Gaussian kernel in this work. For each pair of real and generated data (r, 7), we compute the distribution difference between them per attribute.
C.3 Dataset
Crash Report. We use 38 cases from the CIREN dataset [1] from the NHTSA crash report search engine. Each case contains long text description of the scenario well PDF diagram showing
18

---

the scenario. Because the texts are very long and require a long time for humans to comprehend, in our human study, along with each text input, we will also show the diagram of the scenario as a reference. We show example crash reports in Section D.4. We also refer the reader to the NHTSA website" to view some examples of the crash report.
We create text descriptions that highlight various attributes of a traffic scenario. Specifically, we use the following attributes and values:
Attribute Description.
1. Sparsity:
"the scenario is [nearly empty/sparse/wiih medium density/very dense)".
2. Position: "there are only vehicles on the [lettrighh/fron/back] side(s) of the center car" or "there are vehicles on different sides of the center car".
3. Speed:
"most cars are moving in (slow/mediium/fast) speed" or "most cars are stopping".
4. Ego-vehicle motion: "the center car (stops/moves straight/turns left/turns right)".
(case_0.ppp 2aase_
weelday (ayygins, during uphil
ccodlenti
wislfoourd
wosfooadden
fronr vehicle
bent After velaie indiected northweet
drodoviee
soadidde Velide
enoulder the werthronn anne, fesing thanrs impart Webicde
rointing diccioiis facing sortheant Wehlde towerd damage suttal in craah.
nett left nortiouundann
Creah repart (ax neleweec)
(case_0_pair_2) Which generated scenario above match the text description better?
2
Left
Right
(case. wwt...pppp.r_1_ 2) LEFT scenario matches the text description.
Strongly Disagree
Strongly Agroe
(aase_0_pair_2) RIGHT scenario matches the text description.
NHTSA
Strongly Disagree
Strongly Agree
Figure A1: Human study user interface.
We create sentences describing each of the single attributes with all the possible values. We also compose more complex sentences by combining 2,3 or 4 attributes together with random values for each of them. In total, we created 40 cases for human evaluation. Please refer to Section D,4 for some example input texts from this dataset.
19

---

C.4 Human study
We conduct the human study to access how well the generated scenario matches the input text. We showcase the user interface of our human study in Figure A1. We compose the output of two models with the same text input in random order and ask the human evaluator to judge which one matches the text description better. Then, we also ask them to give each output a 1-5 score. We allow the user to select "unsure" for the first question.
We invite 12 human evaluators for this study, and each of them evaluated all the 78 cases we pro- vided. We ensure the human evaluators do not have prior knowledge of how different model works on these two datasets. On average, the human study takes about 80 minutes for each evaluator.
C.5 Qualitative result full texts
In Figure 4 and Figure A2, we show examples of the output of our model on Crash Report data. Recall that the texts we show in the figures are the summary from our Interpreter due to space limitations. We show the full input text for each example in this section.
Text 1: Full texts of examples in Figure 4.
Figure 4 Column (CIREN ID 594):
"This crash occurred during daylight hours on a dry, bituminous divided trafficway (median strip without positive barrier) under clear skies. There were four east travel lanes (two through lanes, one left turn and one right turn) and four west travel lanes (two through lanes, one left and one right). The east lanes have a slight right curve and the west lanes curve slightly to the left. Both east/west travel lanes were level grade point of impact and divided by grass median. The speed limit this location is 80km/h north travel lane and three south travel lanes (one through lanes, one left and one right). These travel lanes were divided by raised concrete median on the northern side of the intersection. This intersection is controlled by overhead traffic signals. A 2017 Dodge Grand Caravan (V1) was traveling east in the left turn lane and a 2006 Nissan Sentra (V2) was traveling west in the left through lane. As v1 was traveling east it temptted to execute a left turn to travel north when its front bumper impacted the front bumper of V2 (Event 1). After initial impact, v1 rotated counterclockwise approximately 80 degrees before traveling to its final resting position of the intersection facing north. V2 was traveling west in the left through lane and attempting to travel through the intersection when its front bumper impacted the front bumper of v1. After initial impact V2 rotated clockwise approximately 20 degrees before traveling to its final resting position in the middle of the intersection facing northwest. V1 and V2 were towed from the scene due to damage sustained in the crash."
mph). The intersecting north/south roadway consisted of one
the middle
Figure 4 Column (CIREN ID 31):
"A 2016 Kia Sedona minivan (V1) was traveling southwest in the right lane of three. A 2015 Chevrolet Silverado cab chassis pickup (V2) was ahead of in the right lane. was working vehicle picking up debris on the highway in construction zone, The driver of stopped his vehicle in the travel lane. The driver of recognized an impending collision and applied the brakes while steering left in the last moment before impact. V1 slid approximately three meters before the front of V1 struck the back plane of V2 in rear-end collision with full engagement across the striking planes (Event 1). Both vehicles came to rest approximately two meters from impact. v1 was towed due to damage while V2 continued in service."
Figure A2 Row (CIREN ID 33):
"This teo-vehicle collision occurred during the pre-dawn hours (dark, street lights present) of a fall weekday at the intersection of two urban roadways. The crash only involved the eastern leg of of four westbound lanes that included right turn lane, two through lanes, and a left turn lane. The three eastbound lanes of the eastern leg consisted of a merge lane from the intersecting road and two through- lanes. The roadway was straight with speed limit of 89 kmph (55 mph), and the intersection was controlled by overhead, standard electric, tri-colored traffic signals. At the time of the crash, the weather was clear and the roadway surfaces were dry. As Vehicle approached the intersection, its driver did not notice the vehicles stopped ahead at the traffic light. The traffic signal turned green and Vehicle began to slowly move forward. The frontal plane of Vehicle 1 struck the rear plane of Vehicle 2 (Event 1). Both vehicles came to rest in the left through-lame of the westbound lane facing in westerly direction. Vehicle 1 was towed from the scene due to damage sustained in the crash. Vehicle not towed nor
intersection. The westbound lanes of the eastern leg consisted
20

---

disabled. The driver of Vehicle 2 was transported by land to local trauma center and was treated and released."
10
Figure A2 Row (CIREN ID 77):
I1 "A 2017 Chevrolet Malibu LS sedan (V1) was traveling southeast in the right lane cresting hill. A 1992 Chevrolet C1500 pickup (V2) was traveling northwest the second lane cresting
same hill. Vehicle 2 crossed left across the center turn lane, an oncoming lane, then into V1Au20195 oncoming collided in head-on, offset-frontal configuration (Event 1). Vehicle attempted to
of travel. Vehicle and Vehicle 2
steer left just before impact, focusing the damage to the middle-right of its frant plane. Both vehicles rotated a few degrees clockwise before coming to rest in the
roadway, where they were towed from the scene due to damage.
12
13 Figure A2 Row (CIREN ID 56):
14
"A 2013 Honda CR-V utility vehicle (V1) was traveling west in the right lane approaching an intersection. A 2803 Chevrolet Silverado 1580 pickup (V2) was stopped facing north stop sign. Vehicle proceeded north across the intersection and was struck on the right plane by the frant plane of (Event 1). The impact caused both vehicles travel off the northwest corner of the intersection, where they came to rest. Both vehicles were towed due to damage."
Additional Results
D.1 Controllable self-driving policy evaluation
show how LCTGen can be utilized to generate interesting scenarios for controllable self-driving policy evaluation. Specifically, we leverage LCTGen generate traffic scenario datasets possessing diverse properties, which then
assess self-driving policies under various situations. For this purpose, input different text types into LCTGen: Crash Report, the data from CIREN; 2) Traffic density specification, text that describes the scenario "sparse", "medium dense", or "very dense". For each type of text, we generate 500 traffic scenarios for testing. Additionally, we use 500 real-world scenarios from the Waymo Open dataset.
crash report
We import all these scenarios into interactive driving simulation, MetaDrive [34]. We evaluate the performance of the IDM [35] policy and PPO policy provided in MetaDrive. In each scenario, the self-driving policy replaces the ego-vehicle in the scenario and aims to reach the original end-point of the ego vehicle, while all other agents follow the trajectory set out in the original scenario. We show the success rate and collision rate of both policies in Table A1. Note that both policies experi- ence significant challenges with the Crash Report scenarios, indicating that these scenarios present complex situations for driving policies. Furthermore, both policies exhibit decreased performance in denser traffic scenarios, which involve more intricate vehicle interactions. These observations give better insight about the drawbacks of each self-driving policy. This experiment showcases LCTGen as valuable tool for generating traffic scenarios with varying high-level properties, enabling more controlled evaluation of self-driving policies.
IDM [35] Collision (%)
PPO (MetaDrive) [34]
Test Data
Success (%)
Success (%)
Collision (%)
Real
93.60
3.80
69.32
14.67
LCTGen + Crash Report [1]
52.35
39.89
25.78
27.98
LCTGen "Sparse"
91.03
8.21
41.03
21.06
LCTGen "Medium"
84.47
12.36
43.50
26.67
LCTGen + "Dense"
68.12
19.26
38.89
32.41
Table Al: Controllable self-driving policy evaluation.
21

---

Ours
Trafticoon
MotionCLIP
"Vl is travaling cast in the loft turn lane and attempts turn left when coliess with V2 travoling west the ieft through lane."
"V1 approaches an intersection and does not notice stopped vehicles ahead. Vn strikes the rear of V22"
colidde Intersection urban traffiwway, V2 striking side V1"
"Vl maveiing southeast and V2 travaiig northwest, both cresting hV. V1 crosses into V2s lano, resulting in head on coltsion.
traveling right lane approaching an Intersection, stopped facing north stop sign. VZ proceeds struck by
Figure A2: Qualitative result comparison text-conditioned generation on Crash Report.
22

---

Ours
TrafficOon
MotionCLIP
The scene three are vehicias differant sides of the center car, most cars are moving in medium speed. the center car moves straight"
sparse. only behind ego moving
"the ago car tams right, most cars are maving in slow speed."
"he scene is vary dense. there are only vehicles on the lort sizie of the center car, most cars ane moviig in fast speed. the ego vehicll turns right"
(
moving
Figure A3: Qualitative result comparison text-conditioned generation Attribute Description.
23

---

60
LCTGen
50
50
MotionCLIP
MotionCLIP
Proportion (96)
TrafficGen
40
TrafficGen
40
Propertion
30
20
10-
3
4
3 Score
4
Score
(a) Crash Report
(b) Attribute Description
Figure A4: Human study score distribution.
Ours TrafficGen
Ours MotionCLIP
MotionCLIP vs. TraffiGGen
80
Proportion 1S6
60
Proportion (96)
Proportion (S5)
80
20
Ours botter
Neutral Others better
Ours better
Noutral
Others better
MC Better
Neutral
TG better
(a) Crash Report
vs.
MotionCLIP
MationCLIP vs. TrafficGen
40
60
60-
Proportion 196)
Proportion (96)
Proportion (96)
30
20
20
Ours better
Neutral Others better
Ours better
Neutral Otheers better
MC Better
Neutral
TG better
(b) Attribute Description
Figure A5: Human study A/B test distribution.
D.2 Text-conditioned simulation qualitative results
We show the more qualitative results of text-conditioned simulation in Figure A2 (Crash Report) and Figure A3 (Attribute Description). Here, we also compare the output of LCTGen with Motion- CLIP [19] and TrafficGin [15].
D.3 Human study statistics
Score distribution. We show the human evaluation scores of the two datasets in Figure A4. We observe that our method is able to reach significantly better scores from human evaluators.
A/B test distribution. We show the distribution of A/B test result for each pair of methods in Fig- ure A5. Note that our method is chosen significantly more frequenly as the better model compared with other models. We also observe that TrafficGen is slightly better than MotionCLII in Attribute Description dataset, while the two models achieve similar results in Crash Report.

---

Crash Report
Attribute Description
Method
Avg. Score
Human Std.
Avg. Score
Human Std.
TraffiGenn [15]
1.58
0.64
2.43
0.72
MotionCLIP [19]
1.65
0.67
2.10
0.64
LCTGen
3.86
0.87
4.29
0.65
Table A2: Human study average score and variance.
Human score variance. We show the variance of quality score across all human evaluators in Table A2. Specifically, for each case, we compute the standard deviation across all the human eval- uators for this case. Then, average all the standard deviation values across all the cases and show in the table "Human Std.". This value measures the variance of score due to human evaluators' subjective judgement differences. According to the average score and human variance shown in the table, we conclude that our model outperforms the compared methods with high confidence levels.
Age Distribution
Gender Distribution
Drive Experience Distribution
60
50
Proportion (%)
40
Proportion
30
30
30
20
10
10:
10-
20-25
>25
Female
Mali
Never drive
Rarely drive
aften drive
(a) Age
(b) Gender
(c) Driving experience
Figure A6: Human study population distributions.
We show the population distributions the human evaluators involved in our experiment. Specifically, we include statistics of the human evaluators' age, gender and driving experience in Figure A6.
Human evaluator population distribution.
For driving experience, we use the following classifications:
"Never drive": does not have a driver's license.
"Rarely drive": has driver's license but does not often drive.
"Often drive": drives routinely.
D.4 Interpreter input-output examples
Here we show the full-text input and output of Interpreter for four examples Figure 4. Specifi- cally, show examples from Crash Report and two examples from Attribute Descriptions.
Text 2: Input-output examples of Interpreter.
Figure 4 Column (CIREN ID 594):
Input:
"This crash occurred during daylight hours on a dry, bituminous divided trafficway (median
strip without positive barrier) under clear skies. There were four east travel lanes (twa through lanes, one left turn and one right turn) and four west travel lanes (two through lanes, one left
one right). The east lanes have a slight right curve the west lanes curve slightly to the left. Both east/west travel lanes were level grade point of impact and divided by grass median. The speed limit at this location is 80km/h mph). The intersecting north/south roadway consisted of
25

---

north travel lane and three south travel lanes (one through lanes, one left and one right). These travel lanes were divided by a raised concrete median on the northern side of the intersection. This intersection is controlled by overhead traffic signals. A 2017 Dodge Grand Caravan (V1) was traveling east in the left turn lane and a 2006 Nissan Sentra (V2) was traveling west in the left through lane. As V1 was traveling east it attempted to execute a left turn to travel north when its front bumper impacted the front bumper of (Event 1). After initial impact, v1 rotated counterclockwis final resting position
approximately 80 degrees before traveling
the middle of the intersection facing north. V2 was traveling west in the left through lane and attempting to travel through the intersection when its front bumper impacted the front bumper of v1. After initial impact V2 rotated clockwise approximately 20 degrees before traveling to its final resting pasition in the middle of the intersection facing northwest. v1 and V2 were towed from the scene due to damage sustained in the crash."
Output:
Summary: V1 traveling east the left turn lane and attempts to turn left when collides with V2 traveling west the left through lane. There are vehicles in this scenario. This happens at the intersection of eight-lane divided trafficway and four-lane north/south roadway, controlled by traffic signals. Explanation:
V1 (ego vehicle) is traveling east in the left turn lane and attempts to turn left. We cannot find Vi's speed in the query. Because V1 tries to turn left, its initial speed should be set low. We set vl's speed as 5 m/s, which has the index of 2. V1 turns left, so its actions are all 1 (turn left).
is traveling west in the left through lane. As V1 is turning left, seconds before the coming from the opposite direction (westbound), crossing the path V1. v1 (which is facing east initially),
crash, the coordinates the left side. Hence, V2's position is \"front left\" (0). v1 is facing east and V2 facing west, V2 is moving in the parallel opposite direction with v1. Therefore its direction is 1 (parallel opposite). We cannot find V2's speed in the query. Because V2 is traveling west and hit by V1 5 seconds later, we assume V2's s init speed is m/s (index 3). Given this speed, V2's distance to V1 is 8m/s + 5s = 40m (index 8). V2 keeps going straight, so its actions are all 4 (keep speed).
comes from
front and
10
Map: This happens at the intersection of an eight- divided trafficway (4 lanes for eastbound and westbound traffic) and a four-lane north/south roadway (1 lane for northbound and lanes for southbound traffic). The ego vehicle is the left turn- only lane with lane id 4. There an intersection with lane north/south road ahead. v1 is about to cross the interaction, therefore the distance to the intersection is with 15 meter index
11 Actor Vector:
12
'V1': [-1, 0, 0, 2, 1, 1, 1,
13 'V2': [0, 8, 1, 3, 4, 4, 4, 4]
14 Map Vector:
15 'Map': [4, 4, 1, 3, 3, 41
16
17 Figure
(CIREN ID 33):
18 Input:
"This two-veh collision occurred during the pre- dawn hours (dark, street lights present) of fall weekday at the intersection of two urban roadways. The crash only involved the eastern leg of the intersection. The westbound lanes of the eastern leg consisted of four westbound lanes that included right turn lane, two through lanes, and a left turn lane. The three eastbound lanes of the eastern leg consisted of a merge lane from the intersecting road and two trrough-lanees. The roadway was straight with a speed limit 89 kmph (55 mph), and the intersection was controlled by overhead, standard electric, tri-colored traffic signals. and the roadway surfaces were dry. As Vehicle approached the intersection, its driver did not notice the vehicles stopped ahead at the traffic light. The traffic signal turned green and Vehicle 2 began to slowly move forward. The frontal plane of Vehicle 1 struck the rear plane of Vehicle 2 (Event 1). Both vehicles came to rest in the left through- lane of the westbound lane facing in a westerly direction. Vehicle 1 was towed from the scene due to damage sustained in the crash. Vehicle 2 was not towed nor disabled. The driver of Vehicle 2 was transported by land to a local trauma center and was treated and released."
time of the crash, the weather was clear
20
Output:
Summary:
v1 approaches an intersection and does notice stopped vehicles ahead. The traffic light turns green, and V2 starts moving forward slowly. v1 strikes the rear of v2. There are 2 vehicles in this scenario. This happens at the intersection of two
urban roadways with multiple lanes and traffic signals.
23
Explanation:
24 v1 (ego vehicle) is approaching the intersection and does not notice the stopped vehicles ahead. We cannot find Vl's speed in the query. Because V1 is maving and does not stop,
26

---

assume vI's initial speed is 10 n/s (index 4). V1 keeps going straight, its actions are all 4 (keep speed).
25 V2 is stopped at the traffic light and starts moving forward slowly when the light turns green. As V1 is moving straight and hits V2 from behind, 5 seconds before the crash, V2 is in front of v1, Hence, V2's position is \"front right\ (3). As V1 and V2 are both moving westbound,
moving the parallel same direction with v1. Therefore its direction is (parallel_same)
V2's speed is slow, s0 we set 2.5m/s 5s 37.5m (index 7). - its actions are all 4 (keep speed).
2.5 m/s (index 1).
this speed, V2's distance v1 10m/s keeps going sträight,
Map: This happens at the intersection of two urban roadways with multiple lanes and traffic signals. The ego vehicle is in the left through- lane of the westbound lane with Lane id 3. There are 4 westbound lanes and 3 eastbound lanes. The distance of the ego vehicle to the intersection is set to be 10m/s * 5s = 50m (index 10).
27
Actor Vector:
28
'V1': [-1, 8, 0, 4, 4, 4, 4,
29 'V2': 13, 7, 1, 4, 4, 4,
30
Map Vector:
'Map': [4, 0, 0, 10, 31
33
34
Figure A3 Row 1:
35 Input:
36
"the scene is with sparse density. there are vehicles on different sides of the center car. most cars are moving in medium speed. the center car moves straight"
37
38
Output:
39
Summary:
The ego vehicle (V1) moves straight a medium speed a sparse traffic scenario. There vehicles different sides the ego vehicle, corresponding to the sparse density. We generate 4 vehicles in the scene six lane two-way road. The ego is placed the third lane from the right.
40
Explanation:
41
V1 (ego vehicle) is in the center and moves straight. Because V1 moves with a medium speed its speed is set to 4 (medium speed) and action indices are 4 (keep speed).
42
is in the closest distance range (index 0). It is moving in the same direction as the located in the back right (index 2) of the ego vehicle. its actions are all to keep speed (index
ego vehicle (index moving with a medium speed (index 4),
4).
is in further distance range (index 1). It is moving in the opposite direction of the ego vehicle (index 1), therefore it is on the left side of the ego vehicle, locating in the back left (index 1) of the ego vehicle. It is moving with medium speed (index
4), and its actions are all to keep speed (index 4).
44 V4 is in the farthest distance range (index 2). It is moving in the sane direction as the
ego vehicle (index 0) and is located in the front right (index 3) of the ego vehicle. moving with medium speed (index 4), and its actions are all to keep speed index 4).
45
in the farthest distance range (index 2). the ego vehicle (index 1), therefore it is on the left side of the ego vehicle, locating in the front left (index 0) of the ego vehicle. is moving with medium speed (index 4), and its actions are all to keep speed (index 4).
moving
opposite direction
46 Map: Given the sparse density and the presence of vehicles different sides of the ego vehicle, we assume this happens on a 6-lane two-way road. The ego vehicle is in the third lane from the right, with lane id 3. There is no intersection nearby, indicated by -1.
47 Actor Vector:
48 'v1': [-1, 0, 0, 4, 4, 4, 4, 4]
'V2': [2, 0,
4 4,
50 'V3':
4,
4,
'V4': [3, 2,
4,
4,
4, 4]
'V5': [0, 2, 4, 4, 4, 4, 4]
53 Map Vector:
54 'Map': : [3, 3, 0, 0, -1, 3]
55
56 Figure 4 Column 3:
57 Input:
58
"the scene is sparse density. there are only vehicles on the back side center most are moving fast speed. the center turns left"
Output:
Summary: The ego vehicle (V1) turns left at fast speed in sparse traffic scenario. All other vehicles are located behind the ego vehicle, which corresponds to the scenario's 's
27

---

sparse density. We generate vehicles in the scene a six-lams two-way road with an intersection in front. The ego car is placed on the left-turn lane.
63
Explanation:
64
V1 (ego vehicle) is in the center and turns left. Because v1 moves with a fast speed, its speed is set to 6 (fast speed) and action indices are 1 (turn left).
65
V2-V3 are the closest distance range (index 0). They are moving in the same direction as the ego vehicle (index 0). Because the ego car on the left-turn lane, they are both located fast speed (index or 7), and their actions
the
right (index 2) of the ego vehicle. They all to keep speed (index 4).
both moving with
V4 is in further distance range (index 1). It is moving in the opposite direction of the vehicle (index 1), therefore it is on the left side of the ego vehicle, locating in the back left (index 1) of the ego vehicle. It is moving with a fast speed (index 6) in the opposite direction as the ego vehicle (index 1). Its actions are all to keep speed (index 4).
67
Map: Given the sparse density and the presence of vehicles behind the ego vehicle, we assume this happens on l-nane two-way, road. The ego vehicle in the left- turn lane, with lane
There is intersection meters ahead (index as the ego vehicle turning left.
Actor Vector:
'V1': [-1, 8, 0, 6, 1, 1, 1]
70 'V2': [2, 0, 0, 6, 4, 4, 4, 4]
71
'V3': [2, 8, 8, 7, 4, 4, 4, 4]
22
'V4': [1, 1, 1, 6, 4, 4, 4, 4]
73
Map Vector:
34
'Map': [3, 3, 2, 2, 2, 31
D.5 Attribute Description Result Split
Method
Density
Position
Speed
Ego-car Motion
TrafficGen [15]
2.75
2.03
2.34
2.27
MotionCLIP [19]
1.89
2.24
1.91
1.78
LCTGen
4.24
4.28
4.38
4.40
Table A3: Human study result split analysis on Attribute Description scores.
We generate the Attribute Description dataset with different attributes. this section, we split the matching score result for the full dataset into different attributes. We show the result in Table A3. We observe our method has nearly identical performance over all the attributes. TraffieGen the best results with Density, while MotionCLIP performs the best with Position.
D.6 Full Ablation Study
Initialization
Motion mFDE
Method
Pos
Heading
Speed
Size
mADE
SCR
w/o Quad.
0.092
0.122
0.076
0.124
2.400
4.927
8.087
w/o Dist.
0.071
0.124
0.073
0.121
1.433
3.041
6.362
w/o Ori. 0.067
0.132
0.082
0.122
1.630
3.446
7.300
w/o Speed
0.063
0.120
0.104
0.122
2.611
5.188
7.150
w/o Action
0.067
0.128
0.173
0.128
2.188
5.099
7.146
w/o I,
0.067
0.133
0.076
0.124
1.864
3.908
5,929
w/o GMM
0.064
0.128
0.078
0.178
1.606
3.452
8.216
LCTGen
0.062
0.115
0.072
0.120
1.329
2.838
6.700
Table A4: Ablation study of LCTGen
In our main paper, we split the ablation study into different groups. Here show the full results of all the ablated methods in Table A4. We additionally show the effect of 1) using the learnable query 2, and 2) using the GMM prediction for attributes.
28

---

"add the left"
Tremov right" "lt change" "make and speed up"
Figure A7: Instructional editing on real-world scenario
D.7 Instructional traffic scenario editing
We show another example of instructional traffic scenario editing in Figure A7. Different from the compound editing in Figure 5 in the main paper, here every example is edited from the input scenario.
E LCTGen with different LLMs
To study the effects of different LLMs, provide the outputs of LCTGen using various state-of- the-art commercial and open-source LLMs, namely GPT-4 [25], GPT-3.5 [24], Llama2-7B [36], and Llama2-70B [36].
We provide four input texts from Crash Reports and Attribute Descriptions to each LLM. Then, we show the LCTGen output with different LLM output Figures A8- All and the LLM outputs below each figure. We then provide an intuitive evaluation of using ICCTGen with different L.LMs in the following sections.
E.1 Analysis on LLM outputs
For an intuitive comparison of the outputs from different LLMs, we evaluate different LLMs with following criteria on these examples:
Q1: Do the summary and explanation match the input text?
Q2: Does the output vector match the input text?
Q3: Does the L.M correctly chain-of-thought (i.e., applies correct logic which matches the final answer)?
* Q4: Does the LLM follow the formatting guidelines?
We compute the average rate that these criteria are met for different LLMs with the 4 examples show:
Table A5: Evaluation of different LLM's 's result
Q1
Q2
Q3
Q4
GPT-4
100% (1/1/1/1)
100% (1/1/1/1)
100% (1/1/1/1)
100% (1/1/1/1)
GPT--355
100% (1/1/1/1)
50% (0/1/1/0)
75% (1/1/0/1)
100% (1/1/1/1)
Llama2-7B
75% (0/1/1/1)
25% (1/0/0/0)
50% (0/0/1/1)
50% (0/0/1/1)
Llamm2-70B
25% (0/1/0/0)
25% (1/0/0/0)
0% (0/0/0/0)
0% (0/0/0/0)
We have the following observations:
GPT-4 does well in chain-of-thought (CoT) inference and outputs vectors consistent with the CoT logic. See example Text 3 L9-17, where the inference process matches well with the output. It also produces correct vectors most of the time.

---

GPT-3.5 sometimes makes commonsense mistakes. For example, on Text 6 L33, it says V1 turns right, however, on 1.39 it indicates there is no intersection in the map. This commonsense error leads to the strange scenario in Figure A11.
Llama 2 models often produce outputs that do not follow guidelines. For example, on Text 4 L36, the model first outputs vectors and then outputs an explanation. This issue makes its thain-of-though prompting ineffective and leads to incorrect results (c.g., Figure A9).
Llama 2 models often make incorrect chain-of-thoughe inferences. For example, on Text 5 L54, it indicates that VI should turn left, but set VI's action to 7 (turn right).
Llama2-70B model refuses to answer two queries due to "safety concerns". For example, on Text 6 L88,, it indicates the query may produce "potential safety hazard" and refuse to answer. This is problematic as in the prompt we already indicate this is for traffic scenario generation, which is not harmful.
E.2 Discussion
GPT- does well in terms of all the aspects most of the time, which aligns with the strong perfor- mance of LCTGen in our paper. We recommend using GPT-4 for LCTGen.
GPT-3.5 works stably across different cases and produces reasonable outputs. However, it some- times produces incorrect results with flawed chain-of-thought processes. One can use GPT-3.5 for LCTGen, but it might lead inferior performance compared to GPT-4.
Llama 2 models do not reliably work for LCTGen, they often output incorrect results and do not follow our guidelines (e.g.. no chain-of-thoughh refuse to generate). For the current version of LCTGen prompts, we do not recommend using Llama 2. Note that one might be able to engineer a prompt for Llama 2 and other open-source LLMs to work stably for LCTGen going forward.
GPT-4
GPT-3.5
Lllama2-78
Lllama2-70B
traveling east the left turn lane and attempts to turn left when collides traveling west in the left through lane."
Figure A8:
Qualitative comparison of result from different LLMs for Figure Column 1. The red car is the ego vehicle (V1).
Text Output from different LLMs for Figure 4 Column 1.
Input:
"This crash occurred during daylight hours on a dry, bituminous divided trafficway (median strip without positive barrier) under clear skies. There were four east travel lanes (two through lanes, one left turn and one right turn) and four west travel lanes (two through lanes, one left and one right). The east lanes have a slight right curve and the west lanes curve slightly to the left. Both east/west travel lanes were level grade at point of impact and divided by grass median. The speed limit at this location is 80km/h north travel lane and three south travel lanes (one through lanes, one left and one right). These travel lanes were divided by raised concrete median on the northern side of the intersection. This intersection is controlled by overhead traffic signals. A 2017 Dodge Grand Caravan (V1) was traveling east in the left turn lane and a 2006 Nissan Sentra (V2) was traveling west in the left through lane. As v1 was traveling east it attempted to execute a left turn to travel north when its front bumper impacted the front bumper of (Event 1). After initial impact, v1 rotated counterclockwise approximately 80 degrees before traveling to its final resting position the middle of the intersection facing north. V2 was traveling west
mph). The intersecting north/south roadway consisted of
left through lane and attempting travel through the intersection when its front bumper impacted the front bumper of vI. After initial impact v2 rotated clockwise approximately 20 degrees
30

---

before traveling to its final resting position in the middle of the intersection facing northwest. V1 and V2 were towed from the scene due to damage sustained in the crash."
GPT
Output:
Summary: ; traveling east left turn lane and attempts to turn left when
collides with traveling west in the left through lane. There vehicles in this scenario. This happens at the intersection of eight- lane divided trafficway and a four-lane north/south roadway, controlled by traffic signals.
Explanation:
V1 (ego vehicle) is traveling east in the left turn lane and attempts to turn left. We cannot find V1's speed in the query. Because V1 tries to turn left, its initial speed should be set low. We set v1's speed as 5 m/s, which has the index of 2. V1 turns left, so actions are all 1 (turn left). crash, In the coordinates is the left side. Hence, V2's position is \"front lefti" (0). As V1 is facing east and V2 facing west, V2 is moving in the parallel opposite direction with V1. Therefore its direction is 1 peeraalee_oopositte V2 is traveling west and hit by V1 seconds later, we assume V2 init speed is m/s (index 3). Given this speed, V2's distance to V1 is 8m/s + 5s 40m (index 8). V2 keeps going straight, so its actions are all 4 (keep speed)..
10
traveling west in the left through lane. As v1 turning left, seconds before the coming from the opposite direction (westbound), crossing from the front and
path V1.
v1 (which facing east initially),
We cannot find V2's speed in the query. Because
n
Map: This happens at the intersection
an eight-lane divided trafficway lanes for eastbound and westbound traffic) and a Tour-lane north/south roadway (1 lane for northbound and lanes for southbound traffic). The ego vehicle intersection with four lane north/south road
left turn-
only lane with lane id 4. There ahead. V1 is about to cross the interaction, therefore the distance to the intersection is with 15 meter index 3.
12
13 Actor Vector:
14 'v1': [-1, 8, 0, 2, 1, 1, 1, 11
15 'V2': [0, 8, 1, 3, 4, 4, 4, 4]
16 Map Vector:
17
'Map': 14, 4, 1, 3, 3,
19
GPT-3.5
Output:
Summary: V1 is traveling east in the left turn lane and attempts to turn left to travel north, colliding with V2 traveling west in the left through lane. There are 2 vehicles in this scenario. This happens at the intersection of divided trafficway and north/ south roadway, controlled by overhead traffic signals.
23
Explanation:
24
query, so we v1's speed 20 (index 8) which is the maximum speed. attempts to turn lett, its actions
traveling east
left turn lane, speed
mentioned
all 1 (turn left).
V2 is traveling west in the left through lane. As V1 is turning left, 5 seconds before the crash, VI is facing north and V2 is coming from the opposite direction (eastbound), crossing the path of v1. In the coordinates of v1 (which is facing north initially), v2 cones from the front and is on the right side. Hence, V2's position is \"front right\" (3). V1 is facing north and V2 facing west, V2 is moving in the parallel opposite direction is (parallel [parallel_opposite)) We cannot through lane, we assume (index 6). Given this speed, V2's distance all 4 (keep speed). Map: This happens the intersection of a divided trafficway (4 east travel lanes and 4 west travel lanes) and north/south roadway (1 north travel lane and 3 south travel overhead traffic signals. The ego vehicle is
direction with V1. Therefore
speed the query. Because V2 is traveling west the opposite).
V2's init speed
15m/s
(index 15). V2 keeps going straight, so its actions
lanes). The intersection is controlled in the left turn lane with lane id 3. There is a raised concrete median dividing the north/south travel lanes. The distance of the ego vehicle to the intersection is set to be 20m (index 4).
27
28
Actor Vector:
'vi': [-1, 0, 8,
'V2': [3, 15, 1, 6, 4,
Map Vector:
'Nap': [4, 4, 1, 3, 4, 31
33
34
Llama2 Sesee
35 Output:
31

---

36
leg consisted Thank you for providing the query sentence! I'll be happy to help you generate the Actor Vector and Map Vector for this scenario.
37
Actor Vector:
38
'V1': 10, 0, 1, 1, 1, 11
39
'V2': [0, 0, 8, 3, 4, 4, 4]
40
41 Map Vector:
'Map': [4, 4, 2, 2, 3,
Explanation:
For V1, its position is [0, -1], indicating that it is traveling eastbound and attempting to turn left to northbound. Its speed is 0 m/s, which means it is not moving yet. Its direction is [1, 0], indicating that it is turning left. v1's actions are [1, 1, 1, 1],
which means it is attempting to turn left and then keep moving straight for 5 seconds.
45 For V2, its position is [0, 0), indicating that it is traveling westbound. Its speed is m/
5, which means it moving at higher speed than V1. direction is [0, 1],
indicating that
traveling straight. V2's actions are [4, 4, 4, 41, which means keeping speed direction for seconds before the impact.
Llamal- 708
49 Output:
50 Summary:
This crash occurred during daylight hours on dry, bituminous divided trafficway with four east travel lanes (two through lanes, one left turn, and one right turn) and four west travel lanes (two through lanes, one left, and one right). The east lanes have slight right curve, and the west lanes curve slightly to the left. The speed limit this location is km/h (50 mph) The intersecting
rravel north/south roadway (one through lanes, left, and right). The travel lanes divided by raised concrete median the northern side of the intersection, which is controlled by overhead traffic signals.
consists of
lane and three
52
Actor Vector:
53
'v1': 10, 8, 8, 2, 1, 1, 1, 11
54
'V2': [0, 8, 1, 3, 4, 4, 4, 4]
55
Map Vector:
56
'Map': 14, 4, 2, 2, 3, 41
GPT-4
GPT-3.5
Lllama2-7B
Lllama2-70B
"V1 approaches an intersection and does not notice stopped vehicles ahead. V1 strikes the rear of V2.
Figure A9: Qualitative comparison of result from different L.MMs for Figure Column 2. The red car is the ego vehicle (VI).
Text 4: Output from different LLMs for Figure 4 Column 2.
Figure A2 Row 1 Column 2 (CIREN ID 33):
Input:
"This two-vehicle collision occurred during the pre-dawn hours (dark, street lights present) urban roadways. The crash only involved fall weekday at intersection westbound lanes
the eastern leg of four westbound lanes that included turn lane. The three eastbound lanes of the eastern leg consisted of a merge lane from the intersecting road and two through- lanes. The roadway was straight with speed limit of 89 kmph (55 mph), and the intersection was controlled by overhead, standard electric, tri-colored traffic signals. At the time of the crash, the weather was clear and the roadway surfaces were dry. As Vehicle approached the intersection, its driver not notice the vehicles stopped ahead at the traffic light. The traffic signal turned green and Vehicle began to slowly move forward. The frontal plane Vehicle 1 struck the rear plane Vehicle (Event 1). Both vehicles came to rest the left through-lame of the westbound lane facing westerly direction. Vehicle 1 towed from the scene due to damage sustained the crash. Vehicle not towed nor
intersection. right turn lane, two through lanes, and left
32

---

disabled. The driver of Vehicle 2 was transported by land to local trauma center and was treated and released."
GPT
Output:
Summary: approaches an intersection and does not notice stopped vehicles ahead. The moving forward slowly. strikes the of V2. There are 2 vehicles in this scenario. This happens at the intersection of two urban roadways with multiple lanes and traffic signals.
traffic light
green,
Explanation:
10
V1 (ego vehicle) is approaching the intersection and does not notice the stopped vehicles ahead. We cannot find v1's speed in the query. Because V1 is maving and does not stop, we assume V1's initial speed is 10 m/s (index 4), V1 keeps going straight, so its actions are is stopped at the traffic light and starts moving forward slowly when the Light turns green. As V1 moving straight
(keep speed).
11
from behind, 5 seconds before the crash, v1 and
both moving westbound, V2 is moving in the parallel same di direction with V1. Therefore its direction is 0 (parallel_sst same).
in front of V1. Hence, V2's position is \"front right\" " (3).
same). V2's speed is slow, Given this speed, V2's distance to V1 is 10m/s * 5s 2.5m/s 5s 37.5m (index 7). v2
set it to 2.5 m/s (index 1). keeps going straight, so its actions are all (keep speed).
12 Map: This happens at the intersection of two urban roadways with multiple lanes and traffic signals. The ego vehicle is in the left through- lane of the westbound lane with lane 3. There are westhound lanes and eastbound lanes. distance of the ego vehicle the intersection is set to be 10m/s
50m (index 10).
13 Actor Vector:
'v1': [-1, 0, 4, 4,
'V2': [3, 7,
4, 4, 4,
16 Map Vector:
17
'Map': [4, 3, 0, 0, 10, 3]
18
19
GPT -.5-3.5-
20
Output:
Summary:
comes V2 enter intersection the same time collide. V1 rolls over and top, while V2 comes rest off the west edge the roadway. There vehicles this scenario. This happens the intersection of two urban trafficways, controlled by tri-colored traffic signals.
Explamation:
24
V1: V1 enters the intersection at the same time as V2. We cannot find V1's speed in the query. Because V1 collides with V2, its initial speed should be set high. We set vi's speed as 18 m/s, which has the index of 7, V1 keeps going straight, so its actions are all (keep speed).
25
V2: V2 traveling westbound the far left lane. As V1 and V2 enter the intersection at same time, seconds before the crash, Hence, V2's position is \*front left\" (0). v2 is moving northbound, We cannot find V2's speed in the query. Because V2 is approaching the intersection, assume v2's init speed is 15 m/s (index 6). Given this speed, v2's distance to VI is 15m/s 5s 75m (index 15). V2 keeps going straight, so its actions are all 4 (keep speed).
approaching
intersection from the
east and is the left side of moving wesuddund and
V2's direction is (perpendicularr_up)
26
Map: This happens at the intersection two urban trafficways. The north/south trafficway consists of two roadways divided by a raised concrete median. The southbound roadway has three lanes, including a right turn lane. The way westbound channelized roadway has travel lanes for through traffic intersection. The distance of the ego vehicle the intersection is set to be 18m/s 5s 90m (index 18).
dedicated right lane the
Actor Vector:
28
'vi': [-1, 8, 0, 7, 4, 4, 4, 4]
29
'V2': [0, 15, 2, 6, 4, 4, 4, 4]
30
Map Vector:
31
'Map': [3, 0, 2, 1, 18, 21
32
33
Lam.llmaa27 10ma2-78-
Output:
Summary: In this scenario, there is head-on collision between vehicles, Vehicle (ego
vehicle) and Vehicle 2, at an intersection during pre-dawn hours on weekday. The intersection is between two urban roadways, with total of six lanes (four in each direction). The eastern leg of the intersection has four westbound lanes. including right turn lane, two through lanes, and a left turn lane. The three eastbound lanes
33

---

consist of merge lane from the intersecting road and two through lanes. The roadway is straight, and the speed limit is 89 kmph (55 mph). The intersection is controlled by overhead, standard electric, tri-colored traffic signals. The weather is clear, and the roadway surfaces are dry.
37
38
Actor Vector:
39
'V1': 1-1,3,00001,111.11
'V2': [0,8,1,3,4,4,4,4]
Map Vector:
'Map' [4,4,2,2,3,4]
43
44 Explanation:
45 V1 is the ego vehicle, traveling northbound in the left turn-only lane. As it approaches the intersection, it does not notice the vehicles stopped ahead at the traffic light. When the traffic light turns green, Vehicle begins to move slowly forward, frontal plane strikes the rear plane of Vehicle 1. Both vehicles come rest the left through- lane the westbound lane, facing in westerly direction. is towed from the scene due damage sustained disabled. The driver of Vehicle 2 is transported to local trauma center and is treated and released.
their
crash, while Vehicle not towed
46 The actions of V1 are as follows:
47 0: Initialize pasition, direction, and speed to the initial values.
48 1: Turn left, moving straight.
49 2-5: Move straight for 5 seconds, maintaining a speed of 5 m/s.
50 Stop.
51 actions V2 are as follows:
52 Initialize position, direction, and speed the initial values.
1-5: Move slowly forward for seconds, maintaining speed of m/s.
Come to rest.
55
The position, direction, and speed of the vehicles at each action are as follows:
56 V1:
57
Initial position:
58
59
aamaalmmaaaa..mmm 708
60
Output:
Summary:
This two-vehicle collision that occurred roadways during pre-dawn hours. The crash involved the eastern leg the intersection, and the ego vehicle (V1) failed to notice the stopped vehicles ahead and struck Vehicle 2 (V2) as it began to move forward after the traffic signal turned green. Both vehicles came to rest in the left through- Lane of the westbound lane, facing westerly. v1 was towed from the scene due to damage, while v2 was not towed nor disabled. The driver of V2 was transported to a local trauma center and was treated and released.
intersection of urban
63
68
Actor Vector:
65
'V1': [0,
1,
'V2': [0,
3, 4, 4,
Map Vector:
20
'Map': [4, 4, 2, 2, 3, 4]
21
Explanation:
73
38
V1: The ego vehicle,
GPT-4
GPT-3.5
Lllama2-7B
Lllama2-708
N/A
"the scene sparse. there are only vehicles behind ego-vehicle. most cars are moving fast speed. center car turns
Figure A10: Qualitative comparison result from different LLMs for Figure 4 Column 3. The red car the ego vehicle (V1). Note that L.lamaa-770 does not output any vector.
34

---

Text 5: Output from different LLMs for Figure 4 Column 3.
Input:
"the is with sparse density. there only vehicles the back side the center most moving in fast speed. the center turns left."
GPT
Output:
Summary:
The ego vehicle (V1) turns left at fast speed in sparse traffic scenario. other vehicles are located behind the ego vehicle, which corresponds to the scenario scene six-lane two-way with an intersection front. The ego is placed the left-turn lane.
sparse density. generate vehicles
10 Explanation:
v1 (ego vehicle) is in the center and turns left. Because V1 moves with fast speed, its speed is set to 6 (fast speed) and action indices are (turn left).
12
V2-V3 are in the closest distance range (index 0). They are moving in the same direction as the ego vehicle (index 0). Because the ego car is on the left-turn lane, they are both located the back right (index 2) of the ego vehicle. They are both moving with fast speed (index or 7), and their actions are all to keep speed (index 4).
13
further distance range (index 1). It moving in the opposite direction the the ego vehicle, locating
is the left side in the back left (index of the ego vehicle. It is moving with fast speed (index 6) in the opposite direction the ego vehicle (index 1). Its actions are all to keep speed (index 4).
ego vehicle (index 1), therefore
14
Map: Given the sparse density and the presence of vehicles behind the ego vehicle, we assume this happens on 6-lane two-way road. The ego vehicle is in the left-turn lane, with lane id 3. There is an intersection 10 meters ahead (index 2) as the ego vehicle turning left.
15
16 Actor Vector:
'v1': [-1, 0, 6, 1,
'V2': [2, 0, 0, 6,
4,
4,
4, 4]
'V3': 12, 8
8,
7,
4,
4,
4, 4]
20 'V4': [1, 1, 1, 6,
4, 4,
4, 4]
21
Map Vector:
22
'Map': : [3, 3, 2, 2, 2, 31
23
24
GPT
25
Output:
Summary:
The ego vehicle (V1) turns left in sparse traffic scenario. Because the scene is sparse, there are only few vehicles located behind the ego vehicle. We generate 4 vehicles in the scene on four-lane two-way road. The ego car is placed on the second lane from the right. Since V1 turns left, there should be an intersection ahead in this scenario.
29
30 Explanation:
(ego vehicle) to 5m/s (index 2) and its actions
center turns left. Because turns left, its speed is set all 1 (turn left).
V2-V3 are in the closest distance range (index 0). They are moving in the same direction as the ego vehicle (index 8) and are located in the back right (index 2) of the ego vehicle. They are moving at fast speed (index 7) and their actions are all to keep speed (index 4).
33 V4 is in further distance range (index 1). It is moving in the opposite direction of the ego vehicle (index 1) and is located in the back left (index 1) of the ego vehicle. It is moving a fast speed (index 7)
its actions are all to keep speed (index 4). 34 Map: Given the sparse density and the presence of vehicles behind the ego vehicle, we assume this happens on 4-lane two-way road. The ego vehicle
second lane from the right, with lane id There is intersection around 10 meters (index 2)
ahead, as the ego vehicle is turning left.
35
36
Actor Vector:
37
'Vi': [-1, 8, 0, 2, 1,
1, 1,
1]
38
'V2': 12, 0, 8,
7,
4,
4,
4,
4]
39
'V3': [2, 0, 8,
7,
4,
4,
4, 4]
40
'V4': [1, 1,
4,
4,
4, 41
41
Map Vector:
'Map': [2, 0,
0,
2,
35

---

43
46
Lamaz 76 -
45
46 Output:
47
48
Sure, here's output for the updated query:
69
50
Summary:
The ego vehicle (V1) turns left fast speed in sparse traffic scenario. All other vehicles are located behind the ego vehicle, which corresponds to the scenario 's sparse density. We generate vehicles in the scene four-lane two-way road. The ego car is placed on the second lane from the right.
52
53
Explanation:
54
(ego vehicle) is the center and turns left. Because V1 moves with a fast speed, its speed set 7 (fast speed) and action indices are (turn left) and (decelerate). closest distance ranges (index and 1). They are moving
55
V2-V3 are
same lane, they all located in the back right (index 2) of the ego vehicle. They are all moving with fast speed (index 3 4), in the same direction actions are all to keep speed (index 4) and then decelerate (index 7) after the turn left action.
direction the ego vehicle (index 0). Because the ego car on the
the ego vehicle (index 0). Their
56
V4 is in further distance range (index 1). It is moving in the opposite direction of the ego vehicle (index 1), therefore is on the left side the ego vehicle, locating the back left (index of the ego vehicle. or 4), in the same direction as the ego vehicle (index 0). Its action is keep left action.
is moving with a medium speed (index
speed (index 4) and then decelerate (index 7) after
Map: Given the sparse density and the presence of vehicles behind the ego vehicle, we assume this happens
4- lane way road. The ego vehicle is in the second lane from the right, with lane id 2. There is no intersection nearby, indicated -1.
58
59
Actor Vector:
60
'V1': [-1, 0, 0, 7, 7, 7, 7, 71
61
'V2': [2, 0, 8, 4, 4, 4, 4, 4]
62 'V3': [2, 1, 3, 4, 4, 4, 41
63
'V4': [2,
4,
4, 4,
Vector:
'Map': : [2, 2, 0, 0, -1 21
68
69
I hope this updated output meets your requirements! Let me know if you have any further questions or if you'd like me to generate a different scenario."
70
71
lama2
72
73 Output:
I understand that want generate scenario based the given query, but must point out that the query contains some harmful and unethical content. The query mentions "the scene is with sparse density," which could be interpreted as a harmful and toxic metaphor. Additionally, the query mentions "most cars are moving in fast speed," which could encourage dangerous and reckless behavior.
76
77 As helpful, respectful,
honest assistant, I must refuse to answer this question as violates my policy commit, advise, or give information on illegal activities. programmed provide socially unbiased and positive responses, and I cannot provide answers that promote harmful unethical behavior.
Instead, I suggest rephrasing the query in a safe and respectful manner. For example, "the scene is with a low volume of traffic* instead of "the scene is with sparse density." Additionally, "most cars are moving at a moderate speed" instead of "most cars are moving in fast speed."
80
Once query rephrased, would happy to help generate scenario that socially unbiased positive in nature. Please let know
Text 6: Output from different LLMs for Figure Column 4.
Input:
"most moving in slow speed. the center turns right."
36

---

GPT-4
GPT-3.5
Lllama2-7B
Lilama2-708
N/A
ego turns right, most are moving slow speed."
Figure All: Qualitative comparison result from different LLMs for Figure 4 Column 4. The red car is the ego vehicle (V1). Note that Lllama2-70B does not output any vector.
GPT
Output:
Summary: The ego vehicle (V1) turns right in slow traffic scenario. Most cars are moving at a slow speed. We generate 8 vehicles in the scene. Because V1 takes right turn, we should have an intersection ahead of V1 in this scenario.
10
Explanation
11
center turns right. turns right, so speed slow 5m/s (index 2). It turns right, its actions all (turn right). and the closest distance range (index V2-V4 ego (index 0). They are all moving in slow speed (index 1 or 2) and all moving straight (index 4) with low speed (index or 2). V5-V6 are in the farer distance range (index 1), with perpendicular-uu directions (index 3). Because the other vehicles are crossing the intersection, with traffic rules they are waiting at the intersection. They are on the front left of ego car (index 8). They stop to wait for traffic light with action index (0).
(ego vehicle)
V2-V4
the right back ego car
(index 2), with the
direction
13
14
V7-V8 are the farer distance range (index 2), with opposite direction as the ego car index 1). They are
2). Their actions front left of ego car (index 0), moving in slow speed (index moving straight, whether keeping speed (index 4), accelerate (index 5) decelerate (index 3).
Map: This happens in 6- s-lane two-way road, the ego vehicle is the right-turn lane with lane id 3. Because V1 is turning, there is intersection with 4-way perpendicular road around 10 meters (index 2) ahead as ego vehicle is turning right.
16
17
Actor Vector:
18
'v1': [-1, 0, 0,
2,
7,
7,
7,
7]
19
'V2': [2, 0,
0,
2,
4,
4,
3,
31
20
'V3': 12, 8
1,
4,
4,
3,
3]
'V4':
3,
3,
'VS': [0,
0,
0,
'V6': [0, 1,
3,
0,
0,
8
0,
0]
'V7': 10,
2,
2,
4,
5,
3, 3]
25
'V8': [0, 2,
1,
2,
4, 4, 5, 5]
26
Map Vector:
27
'Map'' [3, 3,
2,
2, 1, 3]
28
29
-3.5
30
Output:
Summary:
The ego vehicle (V1) turns right in scenario where most cars moving at slow speed. To fulfill the requirement of slow speed, we set the speed of all vehicles to be 2 (slow speed). The ego car turns right, so its actions are all 7 (turn right).
34
35
Explanation:
36
(ego vehicle) is the center turns right. Because turns right, its speed is set to (slow speed) its actions are all (turn right).
37
V2-V4 the ego (index with the direction the ego vehicle (index 0). They
closest distance range (index 0). They are located the back right of
all moving slow speed (index 2) and their actions all (keep speed).
37

---

38
V5-V7 are in the farther distance range (index 1). They are located in the back left of the ego car (index 1) with the opposite direction as the ego vehicle (index 1). They are all moving at slow speed (index or 2) and their actions are all (keep speed). 39 Map: Because most cars are moving at a slow speed, we assume this happens on a 4- lane two- way road. The ego vehicle is in the second lane from the right, with lane id 2, There is no intersection nearby, indicated by -1.
40
Actor Vector:
'V1': [-1, 0, 0, 2, 7,
7,
7
71
'V2': [2, 8, 8, 2, 4,
4,
4, 4]
44 'V3': [2, 0, 8
1, 4, 4, 4,
4]
45 'V4': [2, 0,
0,
1, 4, 4, 4,
4]
46 'V5': [1, 1,
1,
2,
4, 4, 4
41
47 'V6': [1, 1, 1,
1,
4, 4,
4,
41
48 'V7': [1, 1, 1,
1,
4, 4, 4, 41
49 Map Vector:
50
MMap: 12, 2, 0, 0, 1, 21
Lamal-
54
55
Output:
56
57
Sure, here's s the output for the updated query:
58
59 Summary: The ego vehicle (v1) moves straight at a slow speed meddium-densitee traffic
scenario. Most cars are moving slow speed, and the ego vehicle turns right. We generate 6 vehicles in the scene four-land two-way road. The ego is placed on the second lane from the right.
60
61
Explanation:
62
V1 (ego vehicle) is in the center and moves straight. Because V1 moves with slow speed, its speed is set to (slow speed) and action indices are (stop).
63
V2-V4 are the closest distance ranges (index 0). They are moving in the same direction as the ego vehicle (index 0). Because the ego car on the second lane, they all located the back right (index 2) of the ego vehicle. They are
moving with slow
speed (index 0 1), and their actions further distance range (index 1). They
all stop (index 0).
V5-V6
moving in the opposite direction of the ego vehicle (index 1), therefore they are all the left side of ego vehicle, locating in the back left (index 1) of the ego vehicle. They are all moving with slow speed (index 1), in the same direction as the ego vehicle (index 0). Their actions are all to stop (index 0).
65 Map: Given the slow speed and the presence of vehicles behind the ego vehicle, we assume this happens on a 4-lane two-way road. The ego vehicle is in the second lane from the right, with lane id 2. There no intersection nearby, indicated by -1.
66 Actor Vector:
'v1': [-1,
0,
0, 8,
'V2': [2,
0, 0,
'V3': [2,
1,
8,
8 8, 0]
'V4': [2,
1,
8,
8
8
8
0]
71
'V5': [1,
1,
1,
0
8
8,
0 0]
72
'V5': [1,
1,
1,
0,
0,
0, 0, 0]
73
74 Map Vector:
75 'Map': [2, 2, 0, 0, 11 21
76
Note that the output for the updated query slightly different from the previous output, the ego vehicle moves slow speed and turns right. The other vehicles in the scene are also updated to match these changes.
78
79
Lareeeslllaaaaa Llama 70B
80
81 Output:
82
83 understand that you want generate traffic scenario based on the given query.
However, must point that the query contains harmful moving in slow speed, which could be potential safety hazard. Additionally, the query mentions that the center car turns right, which could be violation of traffic rules and potentially cause accident.
unethical content.
The query implies that the
84
85 As responsible and ethical assistant, I must refuse to answer this query as it goes against my programming rules rules rules rules to provide information that promotes or
38

---

encourages harmful or unethical behavior. Instead, suggest rephrasing the query in safer and more ethical manner, such as "What is safe and efficient way to turn right in traffic scemario?" This revised query facuses on promoting safe and responsible driving practices, rather than potentially harmful or illegal actions.
86
87
Furthermore, must emphasize that
important to prioritize safety ethical considerations when generating traffic scenarios. It is not appropriate encourage or promote dangerous illegal activities, such violating traffic rules driving slow speeds. It is crucial to ensure that all
39


Reviews:
[{"review_id": "hOhYZ_lU3u", "reviewer": "Reviewer_ixPH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a system called LCTGen for generating traffic scenarios from language descriptions. The system first uses an Interpreter module to convert a textual description of a traffic scenario into a compact, structured format. This is used by a Retrieval module to sample a map region that fits the structured scenario representation. The structured representation and map are given to a Generator module, which is a transformer model that outputs a scenario. The authors use the Waymo Open Dataset to evaluate scene reconstruction quality of the Generator module and find that it outperforms two baselines - TrafficGen and MotionCLIP. The authors then evaluate the language-conditioning aspect on two small datasets (NHTSA crash reports and Attribute Description dataset) using a human study. Finally, the authors show that LCTGen can also be used to edit traffic scenarios via commands expressed in natural language.", "summary_of_recommendation": "This paper proposes an interesting approach for language conditioned traffic scenario generation for testing self-driving in simulation. Experiments show strong performance of the proposed approach compared to baselines. However, additional analysis is needed for LLM task in the Interpreter module, as well as qualitative description of what information language adds that can explain the improved performance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The main issue I see is lack of evaluation of the LLM task. LLMs are known to ignore instructions, hallucinate facts in responses, or make mistakes. They may also make up reasons to justify what it outputted previously in order to stay consistent. As the use of the LLM is a key component of the proposed approach, it would be informative to evaluate how reliably the LLM actually accomplishes the text interpretation task.\n\nSome ideas:\n- Does the output (summary, explanation, and/or vectors) actually match what the input description says?\n- Does the chain of thought have logical errors? Does the final answer match the intermediate outputs in the chain of thought?\n- Does the LLM output actually follow all of the many guidelines specified?\n\nSome other clarification questions:\n- In the Waymo Open Dataset results, where does the language input L come from, if there is no language-scene paired data? Is the Interpreter not used in these experiments, or did someone write a description of each scenario?\n- In baselines that don't use the Interpreter, where does the structured representation (the output of the Interpreter) come from?\n- Intuitively, what kind of information does the language description add that allows LCTGen to perform better? Some analysis on this would be appreciated.\n\nMinor comments:\n- Sec. 4.1: 8-dimentional -> 8-dimensional\n- Table 2: Ours Prefered -> Ours Preferred\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The proposed approach enables use of real-world traffic incident datasets, which might be stored in textual format rather than geometric format. This can allow generation of simulated traffic scenarios that better reflect the type of incidents that actually occur in real life.\n- The approach leverages the abilities of LLMs such as GPT-4 which excel at textual conversion tasks.\n- The proposed traffic scenario generative model outperforms baselines.\n- The human study shows that users find scenarios generated by LCTGen to better match the input text descriptions.\n\nWeaknesses:\n- The evaluation datasets for language-conditioned simulation appear to be small scale (40 cases for one, 38 cases for the other).\n- The authors propose use of an LLM to interpret textual scenarios as a key component of their system. However, there appears to be no evaluation of that component in isolation.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "YDq-QO03Odq", "reviewer": "Reviewer_5bXt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel method to generate traffic scenes conditioned on languages. Their method contains three components, a GPT-4 based Interpreter that converts language into features, a Retrieval that retrieves suitable maps based on feature similarity and a Generator network that generates traffic agents based on maps and previous features. Experimental results show that their method outperforms previous baselines, and generates traffic scenes that are both more realistic and aligned with the language. ", "summary_of_recommendation": "I think the method shows nice novelty and performance. In addition, I like the usage of language-conditioned traffic scene editing, it is quite novel. I think the method is potentially important for autonomous driving and deserves a presentation at the conference.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1.\tFig.3 is not referred to in the paper.\n2.\tThe Large Language Model paragraph in the related work section should also cite GPT3 (when referring to in-context learning) and InstructGPT (when referring finetune with human feedback) \n3.\tIs the covariance matrix in L159 diagonal?\n4.\tIn L164, what do you mean ‘sample’ the most probable values, is it greedy search?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength\n\n1.\tThe method that uses GPT-4 to generate traffic scenes is interesting and novel.\n2.\tThe performance improvement is decent.\n3.\tThe method can be used for language-conditioned traffic scene editing, which is never shown by previous methods before. \n4.\tThe paper writes well\n\nWeakness\n\n1.\tIt would be interesting to include an ablation experiment to study whether open-sourced LLMs can replace GPT-4 in the system.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "CfxKbtrWuY", "reviewer": "Reviewer_iJXw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces LCTGen, a generative model combining LLMs and transformer based decoder to generate controllable and realistic traffic scenarios. It primarily consists of an interpreter and a generator, where interpreter converts the query to a structured representation of the scene consisting of map and agents scecific information using GPT-4. It follows a Chain of Thought promoting to summarize the scenario using GPT-4 and plan agent by agent to generate structured representation of the scene. Generator uses structured representation and sampled map to generate realistic traffic scenarios that matches the user's specifications. Results demonstrate that the proposed method outperforms the existing baselines (TrafficGen and MotionCLIP) for language conditioned generations and simulations, and is evaluated up with human studies.", "summary_of_recommendation": "I would be happy to increase my rating, if the issues are addressed in the updated draft.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Need more details on the following to understand this work:\n\n- How the map dataset is generated from waymo dataset?\n- Most importantly, it is not clear how the input text is associated with the map. What if language says : \"ego car overtakes a truck from the left neighboring lane\", but if there  isn't any lane on the left of the ego vehicle in the map. Do you visualize each scenario\nand then generate text corresponding to that based on feasibility of the scenario? How does it make sure that the structural representation obtained from GPT-4 is actually feasible with any map in the map dataset. And is the map sampled, to make sure that the scenario sampled from the dataset actually corresponds to the language description, before using MSE loss for prediction of agents' motion. \n- How does the mapencoder work, does it take lane information as polylines, and learns graph adjacency matrix? \n- Providing some statistics of the users involved in human study, for instance- their experience of driving, or their age, would be helpful. \n- Further, Fig 4 and Fig 5 don't provide any qualitative results of the other 2 baselines: MotionCLIP and TrafficGen.\n- I recently found this paper on similar domain, would be interesting to see how this methods is different, by including it in the related work [1].\n\n[1]. https://arxiv.org/abs/2306.06344", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1. Most of the paper is well written, and is easy to read. \n2. The idea of leveraging LLMs for realizing, and controlling complex traffic scenarios is quite novel, and interesting.\nIt has immense potential to generate and augment dataset with rare and safety critical scenarios. \n3. The baselines are evaluated on scene reconstruction, and language conditioned simulation, and the results are \nbacked up with human studies.\n\nWeaknesses:\n\n- While overall the paper is easy to read, there are certain sections which aren't easy to comprehend as follows:\n - Abstract and Introduction: These sections don't holistically provide the motivation, and future scope and applications of this work, be it out of distribution generalization or reducing sim to real gap. \n- Few modules aren't easy to understand from the current version. For instance: the map dataset generation, association and feasibility of input text with the map in the map dataset.\n- there are no details of the input format of map encoder, and human study involved.\n- No qualitative results shown for other 2 baselines: MOTIONCLIP, TrafficGen\n\nFor language conditioned generation, While the current method doesn't seem to capture the infrastructure information like traffic signs, it would be interesting to see if the traffic light turns red, how does the behavior of the agents change at the intersection.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "2odlb3iy61", "reviewer": "Reviewer_zVxc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method LCTGen that can leverage LLM to generate traffic scenes. It has 3 components: interpreter, generator and encoder. The interpreter convert natural language into a structured representation and then generator takes the structured representation and map to generate realistic traffic. Experimental results show that LCTGen can generate realistic and interesting traffic scenarios and this technique can be applied to applications such as instructional traffic scenario editing and controllable driving policy evaluation.", "summary_of_recommendation": "Using language to generate traffic scene is an interesting idea. It is much more efficient than collecting real traffic pattern or manually generate traffic scenes. I would recommend accepting this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "It will be interesting to show results w/ different LLMs and study how LLM can influence the results.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well written and the idea is novel. Experimental results show much better results than baseline methods.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces LCTGen, a novel system for generating realistic traffic scenarios from natural language descriptions. It includes an Interpreter module for converting text into structured representations, a Generator module utilizing both structured input and map data to create coherent scenarios. The system outperforms existing benchmarks in language-conditioned scenario generation and scene quality, as demonstrated by evaluations on various datasets including Waymo Open Dataset and human studies. LCTGen's versatility extends to scenario editing through natural language commands and controllable driving policy evaluation, showcasing its potential for practical applications.

The reviewers pointed out the paper's strengths in its innovative utilization of LLMs for scenario generation, demonstrating performance improvements over baselines, and the exploration of novel applications such as language-conditioned editing. Additionally, the paper's high-quality writing contributes to its overall impact and accessibility.

The main weaknesses of the paper pointed out by the reviewers include that the evaluation datasets are small, the LLM component is not evaluated in isolation, and there is unclear or missing information in certain sections of the paper.

During the rebuttal phase, the authors provided informative new ablation study on different LLMs. 

We recommend accepting this paper as poster. The authors should include the ablation study on different LLMs and address other weaknesses in the camera ready version.
Decision: Accept (Poster)