Language-Guided Traffic Simulation via Scene-Level Diffusion
Authors: Ziyuan Zhong
Davis Rempe
Yuxiao Chen
Boris Ivanovic
Yulong Cao
Danfei Xu
Marco Pavone
Baishakhi Ray
Venue: CoRL 2023 Oral (2023)
Keywords: Traffic Simulation
Multi-Agent Diffusion
Large Language Model

Abstract: Realistic and controllable traffic simulation is a core capability that is necessary to accelerate autonomous vehicle (AV) development. However, current approaches for controlling learning-based traffic models require significant domain expertise and are difficult for practitioners to use. To remedy this, we present CTG++, a scene-level conditional diffusion model that can be guided by language instructions. Developing this requires tackling two challenges: the need for a realistic and controllable traffic model backbone, and an effective method to interface with a traffic model using language. To address these challenges, we first propose a scene-level diffusion model equipped with a spatio-temporal transformer backbone, which generates realistic and controllable traffic. We then harness a large language model (LLM) to convert a user's query into a loss function, guiding the diffusion model towards query-compliant generation. Through comprehensive evaluation, we demonstrate the effectiveness of our proposed method in generating realistic, query-compliant traffic simulations.

Language-Guided Traffic Simulation via Scene-Level Diffusion
Ziyuan Zhong',
Davis Rempe Yuxiao Chen', Boris Ivanovic Danfei Xu Marco Pavone Baishakhi Ray'
Yulong Cao",
CColumbia University,
NVIDIA Research,
Stanfordd University, *Georgia Tech
Abstract: Realistic and controllable traffic simulation core capability that is necessary to accelerate autonomous vehicle (AV) development. However, cur- rent approaches for controlling learning-based traffic models require significant domain expertise and are difficult for practitioners use. To remedy this, we present CTG++ a scene-level conditional diffusion model that can be guided by language instructions. Developing this requires tackling two challenges: the need for a realistic and controllable traffic model backbone, and an effective method to interface with a traffic model using language. To address these challenges, we first propose scene-level diffusion model equipped with a spatio-temporal trans- former backbone, which generates realistic and controllable traffic. We then har- ness a large language model (LLM) to convert user's query into a loss function, guiding the diffusion model towards query-compliant generation. Through com- prehensive evaluation, we demonstrate the effectiveness of our proposed method in generating realistic, query-compliant traffic simulations.
Keywords: Traffic Simulation, Multi-Agent Diffusion, Large Language Model
Introduction
Given the high costs and risks large-scale real-world autonomous vehicle (AV) testing [1, 2}. AV developers increasingly rely on simulations for developing robust systems [3]. For maximum efficacy, simulators must offer realistic and controllable traffic behaviors, complemented by user- friendly interface. The realism traffic patterns ensures that development and testing conducted in simulation environments generation of relevant traffic scenarios scrutinize specific AV behaviors. For example, controlling vehicle to collide with the AV to check how it reacts in dangerous situations. A user-friendly interface simplifies how desired behaviors can be specified. However, generating realistic [4. 5] and controllable [6] traffic poses considerable challenges, and the exploration user- frienndyy interfaces traffic generation has been limited. This work strives to develop an expressive scene-centric traffic model that can be controlled through a user-friendly text-based interface. Such an interface has the potential to connect simulation previously unusable text-based data, such governmental and insurance collision reports. It also facilitates new simulation capabilities, such reconstructing real-world collision scenarios [7]-
be transferred to real-world scenarios. Controllability permits the
Building a traffic simulation model with a language interface presents two challenges. First, the traffic model must generate realistic trajectories both agent and scene levels, and provide con- trollability over its generated trajectories. Current simulators [8, 9, 10], whether replaying logs using heuristic controllers for agent behavior, lack realism and expressiveness. Data-driven ap- proaches [4, 5] merely reflect training data distribution, lacking control over generated traffic. Re- cently, CTG [6] applies a diffusion model, which has demonstrated promising results across various conditional generation tasks [11, 12, 13, 14, 15], to traffic generation. CTG shows that diffusion is well-suited for controllable traffic simulation through guidance, which allows test-time adaptability user controls. However, CTG models agents independently, leading to unrealistic interactions. 7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

For example, two vehicles modeled separately might collide if the leading vehicle slows without the following vehicle responding. The second challenge is grounding language in a powerful traffic sim- ulation backbone, since language conveys more abstract patterns (e.g., "traffic jam" or "following") while traffic models operate on low-level trajectories.
address similar issues, recent research on Large Language Models (LLMs) for robotic behaviors [16, 17] designs suite of high-level func- tions (e.g.. "pick up" and "use item") that an LLM can employ to control the robot in order to achieve user-specified task (e.g.. "make an omelette"). Essentially, these high-level functions bridge tex- tual instructions and robotic behaviors. Unfortunately, this approach cannot be directly used for realistic traffic simulation. It is infeasible for an LLM to only use a few high-level functions (c.g., "go to location") ") to generate the entire low-level human-like like trajectories.
In this work, we propose model called CTG++ (see Figure 1) to overcome the aforementioned chal- lenges. To achieve realistic and adaptable traffic model, our approach harnesses the strengths of diffusion and significantly enhances capa- bility cater to multi-agent sce- narios. This is achieved through newly proposed scene-level condi- tional diffusion model, underpinned by spatial-temporal transformer ar- chitecture. This architecture applies alternating temporal and spatial at- tention modules, effectively capturing the dynamics of multi-agent interactions over time. To create natural language interface with the traffic model, we leverage the proven capacity of LLMs to generate code from natural language [18]. Instead of a direct translation from text to traffic, introduce an intermediate representation: differentiable loss function, which encodes a user's in- tention from the language command. The loss function guides the diffusion model to generate command-compplant trajectories. With this two-step translation, the LLM and diffusion model effi- ciently bridge the gap between user intent and traffic simulation.
rod vettcle
colkites
Gumry
det guidance, hudtance_oossttttoooorreeo Iossttrgecctriiss....
retur ccllisoon Joss difforentaile
Pre-oetred APte
and Poweppio
GPT4
code format
coliision
Guide
CTG++ Beanb- Leritl Conditgnal Difusión Modol
Genersto
Context
Trijectroy
Figure 1:
Overview of CTG++ A user query. predefined APIs, and examples passed GPT4, which generates differentiable loss to guide CTG++ query-compliant trajectories.
We evaluate CTG++ the nuScenes dataset [19], showing its ability to follow user-specified lan- guage commands and generate realistic trajectories. In summary, our contributions are: (1) scene- level conditional diffusion model, leveraging spatial-temporal transformer backbone, designed for the generation of realistic and controllable traffic, (2) language interface adept at generating trajec- tories that align with user-defined rules in language, and (3) extensive evaluation comparing CTG++ to state-of-the arr baselines, highlighting its superiority generating high-quality scenarios.
2 Related Work
Traffic simulation. Traffic simulation methods can be divided into rule-based and learning-based. Rule-based strategies often feature an interface allowing users to specify the vehicles' routes, where the motion is governed by analytical models like intelligent driver model [20]. Although they deliver user-friendly controllability, their behavioral expressiveness limited, resulting in trajectories far from human driving patterns. To improve realism, learning-based approaches utilize deep generative models trained on trajectory datasets, aiming to emulate authentic driving behaviors [21, 22, 23, 4, 5]. However, they trade off eser-controllabiliiyy for increased realism, users cannot customize the properties of generated trajectories. In contrast, our scene-level diffusion model and LLM-based language interface allow the generation of realistic and language command-compliant traffic.
Diffusion models for conditional generation in roboties. As diffusion models have shown strong performance and test-time adaptability, they have been recently used for various conditional gener- ation tasks in robotics and traffic. Existing works use trained classifiers [13, 14] or expert-designed

---

loss functions [6, 24, 25] to guide the denoising process to achieve user-desired properties. For ex- ample, CTG [6], the closest work to ours, uses a manually designed loss function based on Signal Temporal Logic (STL) to guide denoising. However, both training a classifier and manually crafting a loss function for each new property require domain expertise. contrast, our approach allows user to easily specify desired properties natural language. which then converted into relevant loss function by LLM. Moreover, most works, including CTG, model each agent independently [26, 13, 14, 6], resulting in unrealistic agent interactions. In contrast, our scene-level diffusion model consider all agents in scene jointly, resulting in realistic modelling of interactions.
Large language models for robotics. Recent breakthroughs in LLMs have motivated series of works applying LLMs to robotic tasks. One approach is train a multi-modal LLM that takes in embodied data in addition to text data [27, 28, 29]. Unfortunately, available. Other works directly prompt a pre-trained LLM with high-level function library along with user query. This lets the LLM plan a robot's 's behaviors via the provided functions to achieve the goals in the query [16, 17]. This approach does not directly apply to traffic simulation as existing data-driven approaches cannot be controlled via high-level functions. To tackle this challenge, we leverage pre-trained LLM to translate a user query into differentiable loss function in code format and use it to guide scene-level conditional diffusion model for traffic generation.
such text-traffic data is
3 Methodology
After formulating the problem of controllable traffic generation (Section 3.1), we provide the details of our approach, CTG++. The training stage involves training a scene-level conditional diffusion model to capture diverse behaviors from real-world driving data (Section 3.2), utilizing scene- level
transformer architecture (Section 3.3). During the inference stage, CTG++ generates query- compliant behaviors via the guidance of a user query derived loss (Section 3.4).
3.1 Problem Formulation
Similar to CTG [6], we formulate the traffic simulation as an imitation learning problem. For the M vehicles in scene we want to simulate, let their state at timestep be 8 [s] s M where of = (aiz v ot, 0f) (2D location, speed, and yaw) and the action (i.e., control) be a [a] at where a: (tt, 0.) (acceleration and yaw rate). We denote
(T, St-T to be decision-relevant context, which consists of local semantic maps for all the agents / = {1 {I".... ...., IM] and their current and Thist previous states St-Taueit = {St-Thuees St). To obtain state $141 at time t 1, we assume a transition function (e.g., a unicycle dynamics model) ʃ that computes St+1 = -'(,, ax) given the previous state 8 and control 0₃. Our goal is to generate realistic and query-satisfying traffic behavior for the agents given (1) the decision context с and (2) a function r : RR x 2T R derived from a user query to measure rule satisfaction of the state and action trajectories. A model should generate future trajectories for the agents Str1++ the next T timesteps. Ideally, these trajectories maximize satisfaction r ((nmrr-1, 81-144) ) to avoid violating the given rule.
3.2 Scene-Level Conditional Diffusion for Traffic Modeling
Diffusion models [30, 31, 13, 14] generate new samples through an iterative denoising process by learning to reverse a diffusion process. As a traffic scene involves multiple traffic participants, single-agent diffusion model [13, 14, 6] may generate sub-optimal samples when a scene involves significant interactions among multiple agents. To tackle this issue, we propose a scene-level diffu- sion model that jointly models all traffic participants in a scene. Unlike CTG, which models each agent's 'ss future trajectory independently, our model operates on the past and future trajectories of all the agents in a scene jointly (see Figure 2) and thus captures the interactions among agents both spatially and temporally. Starting from Gaussian noise, the diffusion model is applied iteratively to predict a clean, denoised trajectory of states and actions for all agents in scene.
3

---

Ageni--genn Mirsotion
per Sincoop
Kap
Suite sclion
Encoder
du
Temporal Atteiton
Spaza Atartoon
SANT
(Teat
using
That
Tux
Thst +
derebin
Papaat
Figure 2: A denoising step using scene-level spatial-temporal transformer.
Trajectory Representation. We denote the futurre trajectory that the model operates on as:
T= [==[ab ---
T [ s7].
Additionally, we represent the historical trajectory in the context с as Thist In accordance with [6], model predicts solely the action trajectory т.. employing the known dynamics to deduce the states T via a rollout from the initial state so (which forms part of the context c). To maintain the physical feasibility of the state trajectory throughout the denoising process, we consistently denote T as state trajectory resulting from actions: = s( T.).
Formulation. Consider Th as the action trajectory at the k-th diffusion step, with marking the original clean trajectory. The forward diffusion process that starts from T is defined as:
K
K
((=, l:K (T )======== == (I
(1)
k=1
k=1
Here, 3, for all i 1, 1,.,, K are pre-determined variance schedule, controlling the magnitude of noise added at each diffusion step. As the noise incrementally accumulates, the signal transforms into an approximately isotropic Gaussian distribution N IO, For trajectory generation, our goal is to reverse this diffusion process through a learned conditional denoising model (Figure 2) that is iteratively applied starting from sampled noise. The reverse diffusion process, is as follows:
po (c) (T I1 po c):=
NI ()) p(Tx "o(T" k. c), Ee( k. c)), (2)
where p(T,)) (O, and 0 represents the parameters of the diffusion model. At each step, the model takes in actions Th and the resulting states -. f T.) input. As per [31, 26], the variance term of the transition is a fixed schedule such that (a k k c) = Σ = IfI. The training in our approach mirrors that in [6], but with a key difference trajectories are sampled at the scene level instead of the agent level, as our model simultaneously predicts outcomes for all agents a scene. Detailed information is available Appendix A.I.
3.3 Model Architecture: Scene-Level
Transformer
Unlike previous works which use U-Net [13, 6] single-agent transformer [26] to model the denoising process for a single agent, we design an architecture that models multiple agents jointly. Inspired by recent works on transformer-basee motion prediction [32, 33, 34, 35, 36), we propose a spatial-temporal transformer architecture to model multiple agents jointly. Unlike most previous work [32. 33], which employs scene-centric coordinates to capture interactions, we adopt agent- centric coordinates. As traffic scene configuration combinatorial, it is easy for scene end up in nut-of-distribution configuration if is modeled in ccene-centric coordinates, since errors compound over time. In contrast, agent-centric coordinates are invariant to translation and rotation of the scene, and therefore more robust during closed-loop simulation. However, agent-centric coor- dinates discard relative information among agents, which is important for interactions. To avoid this, we introduce a spatial attention module that enables the exchange of relative information between agents. Inspired by previous work [36], to avoid the combinatoric explosion of attention pairs, we alternate between temporal attention, spatial attention, and map attention module fully capture the interactions among agents and the map. We next introduce the details of proposed architecture by showing the data flow of denoising step (Figure 2; see Figure Al for more details).

---

Input and Temporal Attention. We first concatenate the ground-truth agent history trajectories with the predicted future trajectories along the temporal dimension and apply a row-wise feed- forward network (rFFN) to project each element (each agent per timestep) from the attribute dimen- sion d, + dh to the hidden dimension dn. The denoising step & is injected into the encoded trajectory using sinusoidal positional encoding function [37]. We next capture the temporal information in the encoded trajectory by feeding it into the temporal attention block, standard transformer encoder [37] that captures the temporal-wise relation of each agent.
Spatial Attention. The encoded trajectory is then fed into a spatial attention block which is a customized transformer decoder block with key and value designed to capture the relative geometric relationships among agents. Similar to [35], we extend a regular attention layer to be aware of the relative information e," between agents and j timestep I:
of' -0=((R (Arde AVol)) cos( (.),)) sin, (i of cos(AO%)) - vg. 2g sin( (A di
where o, is a feed-forward network, Axd, := x/-- Avo, == y. vo. and AOY. 02 0² 08 represent the position and orientation differences from j at timestep t to i at time step (the current timestep), dy is the relative distance between and j at timestep t, and Ri is the rotation matrix associated with agent i at timestep 0. For future timesteps at the training stage and history timesteps, we use the ground-trute truth relative information. For future timesteps at the inference stage, since we do not have the ground-trutt truth information, we use a constant velocity model (which assumes the agents to keep their current velocity for the future timesteps in [38]) estimate the state of all the agents in the future and thus their relative information. The pair-wis, relative information is then incorporated into the transformation of the encoded trajectory via keys and values of the decoder layer:
== W hi, ky W [b]
W Vichiall
[h] e
(4)
where hū and hi are the slices of the encoded trajectories corresponding to the agent i and j at timestep t, and WQaen Wkise Wa learnable matrices.
Map Attention and Output. The map attention block is a multi-head attention layer with keys and values from the encoded agent-centric vectorized map (as in [34], we encode the map via an attention layer which transforms waypoints associated with each lane into lane vector) and captures the interaction between agents and map. The map attention is applied to each agent independently as the map is agent-centric. The output encoded trajectory is projected back to the input dimension d, + d, and results in the predicted clean action trajectory 4 At test time, we additionally apply iterative guidance with differentiable loss function J (see Section 3.4 and Appendix A.2) on the predicted action trajectory. Finally, we apply dynamics to get the predicted state trajectory.
3.4 Guided Generation with Language
A language interface for the powerful diffusion model would enable the user to easily control trajec- tories with minimum domain knowledge. However, the absence of paired text-to-traffi data renders direct training of such a model infeasible. Hence, we explore using an intermediary representation to bridge the two. Recent advancements in LLMs facilitate high-quality conversion from natural language commands into code. Meanwhile, the diffusion model exhibits control over its generation through guidance from a loss function. Thus, we suggest utilizing a loss function implemented in code to bridge the two. Since the guidance loss function must operate on trajectories, we provide helper functions for coordinate transformations and handful of text-loss function paired exam- ples alongside the user's query to the LLM. We then utilize the returned loss function to guide the diffusion model, as discussed in Section 3.2, for generating query-compliant traffic simulation.
Guidance Formulation. Building upon prior work [13, 6], we apply guidance to sampled trajec- tories from our diffusion model at each denoising step satisfy a predefined objective. Guidance uses the gradient of the loss J to perturb the model's predicted mean such that each denoising step (in Equation (2)) becomes: pe po(T +*,c) x N k-1, - E"> I (4), Σ²) (see Appendix A.2).
5

---

generate loss class such that vahicla should collide with vehicle 2.
Exampie1 query: generate loss class such that vehicle should always drive with accelaration
Response:
dass Collsion Loss:
limit
fonwardjsal.
Examplet_ response:
pas pred exl...
Class Acci postlimlll.os
pos, prad _____rrdd_ofddc transform pooord In wortssposspredt datn batch)
det forwandisaff, X):
SSeecct the relevant agent
pred 1 world select agent indipos pred world pos_prad___ve Loord palect_ggent indipos__pete pred worls, Computo the
User Quarry
X_ wewlestteeeeeeeeeeeeee Saceless__ege________ ind[x.
#elect the acodioration
GPT4
belween the vehicles
The function transforms posilions from
x_il...
pos _eee_reed_______ee poo_pred___wordd
egent coordinates world coordinate.
Eetimate the acosleration deviation
dim--1)
del traneform saorrffommoodddd__ee lo_ wwoldlppeeeeeo.. agent);
acc. dav torch aba(scs) - seff.acc Imit
Compute the collision loss by penalizing the distance greater than the collision redius
Ciip the negalive values to
return world
loss a clpfhe min-D)
koss torch dip(dist self collision racius, min-O) loes
Helpor Functions
Paired Examples
Diftarnntable Loss Function in Pytnon
Figure 3:
Example of prompting and querying LLM for loss function promoting two vehicles to collide.
Language Guidance. Rather than training classifier reward function [13, 12] defining an analytical reward function [6] for J, use GPT4 [39] to translate the intention in a user language query into the corresponding guidance function. In particular, we pass few helper functions used to manipulate trajectory coordinates, a couple of (query - loss function code) paired examples, and the user query into GPT4, and extract an implemented loss function from the returned message. Figure 4 shows an example of querying for a loss function that causes a vehicle to collide with another one. The input paired example shows GPT4 how apply a sequence of differentiable operations to generate loss with respect to the predicted trajectory ("x"). We also provide list of helper functions for trajectory manipulation (such coordinate transformations) for GPT4 to leverage as we have found these can help to avoid minor mistakes on common operations (see Appendix F.2 for additional helper functions and pair examples). GPT4 returns loss function that penalizes trajectories for not having a collision between the specified vehicles, which will guide the diffusion model to generate trajectories having such a collision. The guided sampling is performed jointly for all the agents in a scene.
Traffic Simulation. We perform closed-loop traffic simulation of each scene for 10 seconds. In particular, we apply our model for all the agents in standard control loop: at each step, the model generates a guided trajectory and takes the first few actions before re-planning at 2 Hz.
Experiments
Following the setup (Section 4.1), we conduct experiments to affirm: CTG++ can effectively pro- duce realistic and query-complias traffic behaviors (Section 4.2), and compared with strong base- lines, CTG++ yields superior trade-offs among stability, rule satisfaction, and realism (Section 4.3).
4.1 Experimental Setup
Dataset. nuScenes [19] is an extensive real-world driving dataset encompassing 5.5 hours of precise trajectories from two cities, featuring diverse scenarios and heavy traffic. We train all models using the training split and evaluate them on 100 scenes randomly selected from the validation split. Our focus is exclusively on simulating moving vehicles, as they are the most control-relevant entities.
Metrics. Following [6. 5]. we evaluate stability (i.e., avoiding collisions and off-road driving), controllability, and realism of generated trajectories. We evaluate stability by reporting the failure rate (fail), measured the percentage of agents encountering collision road departure in a scene. We evaluate controllability using rule-specific violation metrics (rule) (see Appendix F.2). To assess realism, we compare data statistics between generated trajectories and ground truth trajectories from the dataset by calculating the Wasserstein distance between their normalized histograms of driving profiles. We measure realism using realism deviation (real), which is the average of the distribution distance for the longitudinal acceleration magnitude, lateral acceleration magnitude, and jerk. We introduce a new scene-level realism metric (rel real) which the average of the distribution distance for relative (averaged over every pair of vehicles in scene) longitudinal acceleration magnitude, relative lateral acceleration magnitude, and relative jerk.

---

(a) CTG++ keep distance (h) CTG keep distance
(c) CTG++ collision
(d) CTG collision
Figure 4: Generated trajectories for query "vehicle A should always keep within 10-30m from vehicle B" and "vehicle A should collide with vehicle B", respectively. The collision and offroad locations marked in * and A. For both CTG++ and CTG, our language interface generate query compliant trajectories. However, CTG++ docs not sacrifice other aspects like keeping on on-road and smoothness while CTG docs.
Traffic Model Baselines. The closest related work on rule-compliant traffic generation is CTG [6], a traffic model based on conditional diffusion. We also consider BITS [5], a bi-level imitation learn- ing model, and adapt its sampling ranking function use our loss function. Its variant, BITS+opt, applies optimization to the output action trajectory for controllability. To ensure fair comparison, this optimization employs the same loss function the one used for guidance in CTG++.
Large Language Model. We use OpenAl's GPT-4 [39] (accessed through the OpenAl API) for evaluating the language interface. We do not train the LLM and use only few-shot prompting.
4.2 Case Study of Language Interface
We conduct two case studies on queries for common traffic behaviors demonstrating CTG++ can effectively generate trajectories satisfying user's queries. Both queries involve the interaction of two vehicles. The first query "vehicle A should always keep within 10-30m from vehicle B" (GPT keep distance) is a common scene in the real world and the generated scene can be used test vehicle following. Figure 4a shows the simulation generated by CTG++: vehicle A (in red) follows vehicle B (in blue) to straight and slightly turn right with safe distance as specified. Both vehicles along with the background vehicles have smooth motion without critical failures during the rollout. The second query "vehicle A should collide with vehicle B" (GPT collision) makes it possible to potentially generate scenarios like those from a crash report [7] and test vehicle's reaction in dangerous situations. Figure 4c shows the simulation generated by CTG++: vehicle A collides with vehicle B while turning right, desired. The motion of all vehicles is smooth and does not have collisions other than the one requested. The generated query-compliant trajectories for both cases show that our proposed language interface in CTG++ enables effective text-to-traffic generation.
4.3
Evaluation of Traffic Model
Table 1: Quantitative results of CTG++ and the baselines under GPT-generate rules and STI. rules.
GPT keep distance
GPT collisios
20 collision
speed linit
fail
rile
real
rel real
fail
rale
real
rel real
full
rile
real
rel rell
fail
rale
real
rel real
BITS
0.183
2,615
0.116
0.362
0.176
0.660
0.107
0.359
0.992
0.065
0.099
0.352
0.111
0.550
0.104
0.352
BITS+0ps
0.340
0,000
0.007
0.360
01277
0.130
0.668
0.362
0.109
0.041
0.670
0.353
0.162
0.130
0.058
0.353
CTG
0.343
0,000
0,877
0.342
0.356
0
0.074
0.349
0.142
0.052
0.044
0,346
0.128
0.029
0.075
0.350
CTG+
0.173
0.000
0.677
0,331
0.264
0.006
0.085
0.331
0.084
0.836
0.840
0,312
0.083
0.828
0.843
0,344
Lagel spoed
D0 odoooaa
20al wwgpeiniiiiiicomm Mapcctppol
stogeggin otiocad
rale
mal
6x1
nol
ratmal
ndal
rile?
rral
Sel
miel
nike
T041
8111
1.575
0.114
D/018
0.355
5111
2.761
0.315
0.005
0,088
0.006
0,379
6/42
0.345
0118
0U053
0.345
0,602
0.090
0.224
0.02
0.710
0.99
0.004
ILESS
0.528
8.111
2
0.545
6.836
0,386
0
ELORS
0.411
0.776
0.134
We assess the traffic model component of CTG++ under the two GPT-generated rules (as described in Section 4.2) and six STL rules from [6] (see Appendix E.I for details of each rule). The quantita- tive results in Table underscore CTG++'s superiority over baselines with a good balance between stability, rule satisfaction, and realism. Specifically, CTG++ secures the lowest failure rate and scene-level realism deviation in 7 out of 8 settings, reflecting its effective scene-level modeling and
7

---

enhanced interaction dynamics. Furthermore, CTG++ also achieves the lowest rule violation and agent-level realism deviation for the majority of settings, demonstrating that enhanced interaction modeling does not compromise agent-level realism or rule adherence.
Qualitatively, CTG++ generates rule-compliant trajec- tories featuring more realistic motion with fewer in- stances of collisions or off-road incidents than the base-
Table 2: Ablation study of CTG++ features.
Tail
rulc
real
rel real
lines. We provide examples of CTG-generated trajec- tories (when using the same language generated loss functions) for the same scenes as those previously shown for CTG++ (Figure 4b and Figure 4d). Specifically, in Figure 4b, CTG's trajectories display off-road instances (indicated by Δ) involving vehicles and background vehicle. Similarly, in Figure 4d, for CTG's trajectories, the collision between vehicles A and B (in x) occurs off-road, and the background vehicle has curvy, unrealistic path. See Appendix C for qualitative comparisons under STL rules.
CTG++
0.173
0.000
0.077
0.331
CTG+ no odge
0.227
0.000
0.077
0,341
CTG+ scone
0.392
Ablation Study. We evaluate the efficacy of our spatial module and the utilization of agent coordinates. As demonstrated in Ta- ble 2, CTG++ surpasses CTG++ edge (i.e., CTG++ but having c²² in Equation (3) replaced by zeros) and CTG++ scene (i.e., CTG++ but using scene-centric coordinates) under the GPT Keep Distance rule. The absence of edge in- formation leads increased failure rates due to decreased awareness of interactions. Moreover, the use of scene-centric coordinates notably in- flates failure rates and realism deviations, as the traffic rapidly deviates from the training distri- bution during the rollout. To visualize the ef- fectiveness of spatial attention, we display the attention maps for vehicle of interest (highlighted in blue) for both CTG++ and CTG++ no edge when guidance not applied. As depicted in Figure 5, CTG++ guides the vehicle to pay attention relevant neighboring agents, while CTG++ no edge results in arbitrary attention and consequent collision (marked +)) with the vehicle ahead.
(a) CTG++
(h) CTG++ edge
Figure 5: Darker red means higher attention by the blue vehicle. Without edge information, CTG++ no edge results in collision (in st).
5 Conclusion
Summary. In this paper, present, CTG++, language- scene-level conditional diffusion model for realistic, query-compliant traffic simulation. In particular, we leverage I.I.M for trans- lating user query into differentiable loss function and propose a scene-level conditional diffusion model (with a spatial-temporal transformer architecture) to translate the loss function into realistic, query compliant trajectories. Extensive evaluation demonstrates the effectiveness of our approach.
Limitations and Future Work. CTG++ currently does not support complex commands involving many interactions with map (see Appendix F.4.. Our framework can be extended by using a multi- modal LLM that takes in vision data (e.g.. bird's-eye view map) for finer control of the traffic and thus supports more complex commands. Second, our framework does not support automatic error detection and fixing for the GPT4 generated loss function. As the loss function (in code format) can have wrong semantics, it is necessary instruct GPT4 to detect and repair it. Our framework can be potentially extended provide the simulation running results to GPT4 to iteratively instruct it to fix the generated loss function. Third, current trajectory generation is relatively slow and take about 1 minute to generate each simulated scenario. Recent work which uses distillation [40] to greatly speed up the generation process can be leveraged to reduce the time cost. Our work opens up many possibilities including adapting our architecture to general multi-agent robotic tasks and using our proposed two-step approach for other tasks with no paired text-behavior data.

---

Acknowledgment
The authors want to thank Sushant Veer and Shuhan Tan for valuable discussions. This work started when Ziyuan Zhong interned at NVIDIA Research. He is also supported by NSF CCF 1845893 and IIS 2221943.
References
[I] CARSURANCE. 24 self-driving car statistics & facts, 2022. URL https://carruurace.. net/insightss/elffdrrrr warriingccr-statiistie
[2] N. Kalra and M. Paddock. Driving to Safery: How Many Miles of Driving Would Take Demonstrate Autonomous Vehicle Reliability? RAND Corporation, 2016. URL http:
//www/ jstorsttr.oo org/stable/10. 7249/ j. ettlbtcoxx.
[3] Waymo. wplass/ille/doemeensssstttt22222 -0aa------aaaa---a---eeerrrrrrre
Waymo safety report.
February 2021.
[4] S. Suo, S. Regalado, S. Casas, and R. Urtasun. Trafficsim: Learning to simulate realistic multi- agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10400-10409, June 2021.
[5] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone. Bits: Bi-level imitation for traffic simulation, 2022.
[6] 7. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided conditional diffusion for controllable traffic simulation, 2022.
[7] NHTSA. Nhtsa crash viewer. https://crashvieeet wwt.a.tttaa..mm dot gov/, 2023. Accessed: 2023-05-03.
[8] A. Lopez, M. Behrisch, Bieker-Walz. J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wiessner. Microscopie traffic simulation using sumo. In 2018 21 International Conference on Intelligent Transportation Systems (ITSC), pages 2575-2582, 2018. doi: 1TL.L1//SS...2188859999999
[9] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. volume 78 of Proceedings of Machine Learning Research, pages 1-16, 13-15 Nov 2017. URL http: http:///roceeddnggs mlr press//v78//oooovttsii.... html.
[10] LG Electronics. SVL Simulator: An Autonomous Vehicle Simulator, A ROS/ROS2 Multi- 2021.
robot Simulator for Autonomous Vehicles.
[11] Dhariwal and A. Nichol. Diffusson models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[12] X. L. Li, J. Thickstun, I. Gulrajani, Liang, and T. B. Hashimoto. Diffusion-Im improves controllable text generation, 2022.
[13] M. Janner, Y. Du, J. B. Tenenbaum, and Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning, 2022.
[14] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making?, 2022.
[15] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models, 2022.
[16] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions feasible plans, 2023,

---

[17] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor. Chatgpt for robotics: De- sign principles and model abilities. Technical Report MSR-TR-2023-8, Microsoft, February 2023. URL http://ssww...acatct..o.o//////a
[18] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. March 2023.
[19] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
[20] E. Brockfeld, R. D. Kühne, A. Skabardonis, and P. Wagner. Toward benchmarking of micro- scopic traffic flow models. Transportation Research Record, 1852(1):124-125) 129, 2003.
[21] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning (CoRI.), pages 86-99. PMLR, 2020.
[22] Y. Chen, B. Ivanovic, and M. Pavone. Scept: Scene-consistent, policy-based trajectory pre- dictions for planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17103-17112 June 2022.
[23] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-fessible trajectory forecasting with heterogeneous data. Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[24] D. Rempe, Z. Luo, X. B. Peng. Y. Yuan, K. Kitani, K. Kreis, S. Fidler, and O. Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion, 2023.
[25] C. ". Jiang, A. Cornman, C. Park, Sapp, Y. Zhou, and D. Anguelov. Motiondiffuser: Con- trollable multi-agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pages 9644-9653, June 2023,
[26] T. Gu, G. Chen, J. Li, C. Lin, Y. Rao, J. Zhou, and J. Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17113-17122, June 2022.
[27] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03372 2023.
[28] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Ju- lian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, Parada, Peralta, E. Perez, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, Singh, S. Sontakke,
I. Mordatch, O. Nachum,
Pertsch, Quiambao, K. Rao,
Stone, Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, Xiao, P. Xu, Xu, T. Yu, and B. Zitkovich. Ri- 1: Robotics transformer for real-world control at scale. In arXiv preprint arX 2022.
[29] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, Ho, J. Hsu, Ibarz, Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao. K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu,
10

---

M. Yan, and A. Zeng. Do as i can, not as say: Grounding language in robotic affordances, 2022.
[30] J. Sohh-DDicsss,nn, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium theroodynamics. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.
[31] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6851. Curran Associates, Inc., 2020.
[32] Y. Yuan, X. Weng, Y. Ou, and K. Kitani. Agentformer: Agent-aware transformers for socio- temporal multi-agent forecasting. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[33] Ngiam, V. Vasudevan, B. Caine, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. J. Weiss, B. Sapp. Z. Chen, and J. Shlens. Scene transformer: unified architecture for predicting future trajectories of multiple agents. In International Conference Learning Representations, 2022. URL https://openrevaet wwt.:.ereetrrm id=Wm3EA501HsG.
[34] R. Girgis, F. Golemo, F. Codevilla, M. Weiss, D'Souza, S. E. Kahou, Heide, and C. Pal. Latent variable sequential transformers for joint multi-agent motion prediction. In International Conference Learning Representations, 2022. URL htpps:////oopereeeeee net/forum?idee
[35] Z. Zhou, L. Wang, K. Wu, and K. Lu. Hivt: Hierarchical vector transformer for multi- agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8823 8833,, June 2022.
[36] N. Nayakanti, R. Al-Rfou, A. Zhou, Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting via simple & efficient attention networks, 2022.
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceeddng..tt www.:eedddnnnnn.rrr.s.o.m.mm pppaper_files/paper///ee (aaee44444raeeltt0000ooooaaaaaa.
pdf..
[38] Y. Chen, B. Ivanovic, and M. Pavone. Scept: Scene-consistent, policy-based trajectory pre- dictions for planning. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recog- nition (CVPR), pages 17082-17091, Los Alamitos, CA, USA, jun 2022. IEEE Computer So- URL https://d//do ieeecompatersocietty
ciety. doi:10.1 (U99/CVPR52688.0222.000 org/10. 1 1ww111101///// /CV/R/22R222.. 2022.01659.
[39] OpenAl. Gpt-4 technical report, 2023.
[40] T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models, 2022.
[41] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[42] OpenAl. Openai guides gpt chat completions api, 2023. URL https://platforn. openai. wwwp//oco///ccccocccccccccc..ccc /gtia///////e//eeeeeeeeeeesssse/ hptp/chtt-comlletioon

---

A Algorithm of Training and Sampling in Details
We mostly follow the training and sampling procedures from [6] and show the detailed algorithms for training and sampling in the following.
A.1
Training
Algorithm Training
1:
Require
driving conditional diffiusion train fig. transition function f, denoising steps
2:
while vonverge
3:
c.
4:
k- (I......
5:
c=N((,,))
6:
Cormupt action trajectory Th =vanT. vT with Iio
7:
Get the corresponding state trijectory T f(so.
8:
model predict uncorrupted trajectory fg (ug(t",kke) ck
9: Get predicted trajectury 7" [F2; f(so. F2)
10:
gradient step Voli=" -+ #9112 end while
11:
Contrary to [6], which samples trajectories the agent level, opt for scene-level trajectory sam- pling, allowing the model to make joint predictions all scene agents. The process is detailed in Algorithm 1. During each training iteration, the context с and the ground-truth trajectory 70 are sampled from a real-world driving dataset, and the denoising step k: is uniformly selected from (1....,K) K.. We derive the noisy input 7* from 7" by initially corrupting the action trajectory via Ta VO + vT öggt with e (0,11 I) and a II 1 3r. Subsequently, the corresponding state computed as T f(o, "...T)) =²). The diffusion model indirectly parameterizes μ4 in eq. (2) by predicting the uncorrupted trajectory 40 [+0: fSo, +.)], where + 1 (T k. c)) is the net- work's direct output (see [12, 13, 41)). We then use a simplified loss function to train the model as
follows:
L(O)= E,,,,r -|
(5)
cosine variance schedule [13, 41] utilized in the diffusion process, employing 100 diffusion steps.
A.2 Sampling
We show the guided sampling algorithm in Algorithm 2 which is directly from [6] as the notations and procedure remain the same. The key difference is that our diffusion model formulation and backbone models are all scene-level rather than agent-level as [6]. The scene-level formulation helps improve scene-level realism and decrease failure rates as the agents' interactions can be captured by the model inherently.
Algorithm 2 Guided Sampling
1: Require conditional diffusion model transition function f. guide J. scale covariances x diffusion steps K, inner gradient descent steps number of actions hefore re-planning! I.
2: while not donc do
3: Observe state so and cpntext e
4:
Initialize trajactory 7 (O, =(.., - [T]]
5:
for k K K do
6:
(T(T" k.c) 1a:- +
7:
(0)
8:
9:
( ((ا-1 oV (N (1()))
Aμ= (ق) (0)
11:
Ap clip(1 Ap. -n,
12:
(1) (0) Au
13:
end for
14:
(M) x : A-1 f(50: xb11): 1-1
15::
for
16:
Exccute first i actions of trajectory 70
17:
end while
12

---

Following [6, 12], the predicted mean is a weighted sum between the predicted clean action trajec- tory and the input action trajectory from last denoising step:
1BL-1BR + a Vas (1-11111100 -xxx 4 k-1 1 k.
Ta
1-
The process of perturbing the predicted means from the diffusion model using gradients of spec- ified objective is summarized in algorithm 2. Following [6], we use an iterative projected gradient descent with the Adam optimizer and filtration, i.c., we guide several samples from the diffusion model and choose the one with the best rule satisfaction based on J.
B
More Details on Architecture
B.1 Detailed Architecture
We show the detailed data flow of our proposed architecture in Figure A1. Its main difference with the simplified architecture shown in Figure 2 is that we show position encoding, rFFN, and the details of the guidance module explicitly.
AgentiAgent Intrtcctio Emeelee
Thcee
Atarniin
Alterton
Adaition
d2
Zlue
V
Thaa -
Thut T
Thew +TT
Rownetta Forward
Panwordd
Crooking
Test Time Only Guidance Step
action
dg
d
d
wewiigne and
r
d
Test time denoising step using scene-level spatial-temporal transformer. de das and dy represent the dimensions of action, state, and latent for each vehicle per timestep.
Figure A1:
B.2 Gated Attention
Following [35], we use variant of the original scaled dot-product attention block. In particular, we use a gating function to fuse the environmental features m with the central agent's features ht. enabling the block to have more control over the feature update. The resulting query, key, and value vectors of the social attention layer are taken as inputs to the block:
a softmax
q
[k
[(K))
sevil)
Vdx
m
Σ JaN,
at
g sigmoid (Waate [hi. mi))
h gi o w h gi) @ m
where Ng is the set of agent i's neighbors (all the agents except the agent itself within a certain social radius), Waae and wyeelf are learnable matrices, and denotes element-wise product @.
13

---

C Qualitative Comparison under STL rules
In this section, show few qualitative examples (Figure A2 Figure A7) comparing CTG++ and the strongest baseline (in terms of rule satisfaction) under the STL rules. Overall, CTG++ generates realistic, rule-satisfying trajectories. The baseline method can usually also satisfy the rule. However, their trajectories usually sacrifice one or more of the following aspects: (1) the trajectories are curvy, unrealistic, (2) the trajectories involve off-road accidents, and (3) the agent interaction is sub-optimal leading to collision(s).
(a) CTG++ speed limit (0.037)
(b) CTG speed limit (0.041)
Figure A2: Qualitative comparison between CTG++ and CTG under speed limit STL rule (the numbers in parentheses represent rule violations). CTG++ achieves lower rule violation than CTG. Besides, CTG involves collision between the blue vehicle and the green vehicle.
(a) CTG++ target speed (0.213)
(b) CTG target speed (0.163)
Figure A3: Qualitative comparison between CTG++ CTG under target speed STL rule (the numbers in parentheses represent rule violations). Although CTG achieves bit better target speed rule satisfaction, involves vehicle collides with crossing vehicles and then goes off-road.
14

---

(a) CTG++ no collision (0)
(b) BITS+ collision (0)
Figure A4: Qualitative comparison between CTG++ and BITS+opt under no collision STL rule (the numbers in parentheses represent rule violations). Both methods satisfies the rule perfectly as no collision happens. However, BITS+opt have highly curvy, unrealistic trajectories as the cost of satisfying the rule.
CTG++ off-road (0)
CTG off-road (0)
Figure A5: Qualitative comparison between CTG++ and CTG under no off-road STL rule (the numbers in parentheses represent rule violations). Both methods satisfies the rule perfectly no off-road happens. However, CTG lead multiple collisions among the pink vehicle and vehicles that stationary.
D Hyperparameters
D.1 Training Hyperparameters
CTG++ is trained on a machine with Intel 19 12900 and NVIDIA GeForce RTX 3090. It takes approximately 10 hours to train CTG++ for 50K iterations. use Adam optimizer with learning rate of le-4.
D.2 Pair Selection Criteria for GPT query based rules
We choose two vehicles A and B in each scene such that they satisfy the following criteria:
15

---

(a) CTG++ stop sign off-road (0, 0)
(b) CTG stop sign off-road (0.732, 0)
Figure A6: Qualitative comparison between CTG++ and CTG under stop sign and no off-road STL rule (the numbers in parentheses represent rules violations). Vchicles are supposed to stop within the marked bounding boxes without going off-road. CTG++ satisfies both rules while CTG only satisfies the no off-road rule. Bc- sides, CTG involves collision between the grey vehicle and the blue vehicle.
CTG++ goal waypoint target speed (0.991, (b) CTG goal waypoint + target speed (1.24, 0) 0.296)
Figure A7: Qualitative comparison hetween CTG+ and CTG under goal waypoint target speed STL rule (the numbers parentheses represent rules violations). Vehicles are supposed reach marked waypoints with target speed (same speed as in the dataset). CTG++ satisfies both rules better than CTG. Besides, CTG involves collision between the two orange vehicles in the end.
Both have current speed larger than 2m/s.
At and 2s, the distance between and B is within the range 10m 30m.
At Os and 2s, the orientation difference between and b is smaller than 108 degrees (for GPT collision) and 36 degrees (for GPT keep distance).
The criteria a coarse-grained filtration for those pairs that more likely to have keep distance / collision interactions in the original training dataset. If more than pair in the scene satisfy the following criteria, select the pair with smallest distance. If none of pairs in scene satisfy the
16

---

following criteria, we skip the scene. After the filtrations, out of the 100 validation scenes, we have 50 scenes remained for GPT collision and 40 scenes for GPT keep distance.
E Experiment Details
E.1 Quantitative Rules
We assess the traffic model component of CTG++ under both GPT-generated rules and STL rules from [6]: (1) GPT Keep Distance and (2) GPT Collision, as described in Section 4.2 (refer to Appendix D.2 for pair selection); (3) No Collision dictates that vehicles must avoid collisions with each other; (4) Speed Limit ensures vehicles do not exceed a set speed limit threshold (75% quantile given scene); (5) Target Speed requires vehicles to maintain specified speed (50% of their speed in the ground truth scene) each time step; (6) No Offroad prohibits vehicles from leaving the drivable area; (7) Goal Waypoint and Target Speed instructs vehicles to reach their designated goal while adhering to the specified target speeds; (8) Stop Sign and No Off- rff-road requires vehicles to halt upon entering a stop sign region and avoid straying off the road.
of all moving vehicles
E.2 Metrics of Rule Violation
We provide the details for the metrics we use for measuring rule violation in this section. For all the metrics of rule violation, we average the metrics over all validation scenes. Besides, they are designed such that the smaller the better (i.e., rules are better satisfied).
GPT Keep Distance. the following vehicle's (in the chosen pair) average L.2 distance deviation from the specified range.
GPT Collision. if a collision happens between the two vehicles in the chosen pair.
No Collision. collision rate of all vehicles in scene.
Speed Limit. average deviation from the speed limit of all vehicles in a scene.
Target Speed. average deviation from the target speed of all vehicles in scene.
No Offread. off-road rate of all vehicles scene. We consider vehicle going off-road if its center goes off-road.
Goal Waypoint. average vehicle's smallest 12 distance deviation from the specified corresponding goal waypoints of all vehicles in scene.
Stop Sign. average smallest speed within the stop sign region of all vehicles in scene.
F Details of Language Interface
In this section, we provide more details and limitation of our proposed language interface for traffic simulation.
F.1 Details of Vehicle Indexing
In practice, instead using color for vehicles which is used in Figure for better illustration pur- pose, we use indices according to the context from the driving dataset. The user can tell GPT4 the vehicles to control via their indices, e.g., "vehicle should collide with vehicle 2".
F.2 Details of Prompting
In Figure 4, provide example of pre-defined API function and query-loss function pair. In experiments, additionally provide the following API functions:
17

---

etc.) and construct transform coord world to agent i. this function transform the predicted position and yaw from world coordinate to the agent coordinate.
select.agent ind. this function returns the slice of with index
get_current.ee wwwrrentaaneeeeeeeee Jane_ sane_projection, this function returns the projection of each vehicle predicted trajec- tory on its current lane in agent-centric coordinate.
lellleff_aanee projection. this function is similar to get.current lane except it returns the left lane waypoints. If there no left lane, the original trajectory will be returned.
get_right_aane. BeL_right_anee-proecttooo this function is similar to Let_curenttlaae except it returns the right lane waypoints. If there is no right lane, the original trajectory will be returned.
In addition to the acceleration loss paired example shown Figure 4, we provide another query- loss function pair example. "Generate loss class such that, vehicle should always stay on the left side of vehicle 2." The corresponding function penalize the cases when vehicle on the left size of vehicle 2. This function provides GPT4 sense of the relationship between direction and the trajectories.
We additionally specify the dimension of the input trajectory and the input and output the expected loss function wrapped in a loss class such that GPT4 know which dimension of the trajectory to operate on when needed: "The generated loss class should have a function: forward(x, datc_, agt_mask). is tensor representing the current trajectory with shape (B, N, T, 6) where B is the number of vehicles (consisting of vehicle with index 0 to B-1). N is the number of samples for each vechile, T is the number of timesteps (each represents 0s1s), and 6 represents the (x, vel, yaw, acc, yawvel) in corresponding agent coordinate of each vehicle. data_batch is dictionary that can be used as parameter for relevant APIs. The function should return a loss for every sample of every vehicle with the shape (B, N) or return loss for every sample with the shape (N)."
F.3 Success Examples with Complete Query and Response
In this section, provide five generated success example programs and their queries. For all the examples, in addition to the user query, GPT4 is provided the APIs for the desired loss function and the helper functions as well as two paired examples: acceleration limit and stay on left. The examples help to elucidate two key aspects:
the standard format of a loss class generate.
how to manipulate different trajectory dimensions using various helper functions (e.g., wood:m/doooddwwddd..oo
transform transform_coord_agee _coood_ageettt..ss sttsstttoordddr differentiable loss the trajectory.
We first show the complete query (including system messages and user query sent to GPT4 API) and response for GPT Collision in Appendix F.3.1. Next, in Appendix F.3.2-Appendix F.3.5, for four other success exmples, we only show the user query and response as they use the same system messages as GPT Collision in Appendix F.3.1).
F.3.1 Success Example: GPT Collision
The OpenAl GPT4 API allows one specify multiple system messages (which includes the API of the loss function to generation, the APIs of the helper functions, and two paired prompted examples), and one user message (which is essentially the user query) (see [42] for more details). Thus, we list each system message and the user message in the following. The qualitative example of using this returned loss function has been shown in Figure 4c.
System Message Loss Function API and Helper Functions API:
"The generated loss class should have a function \ forward (x, datch agt_mask), is a tensor representing the current trajectory with shape
18

---

be used as parameter for relevant APIs. B, R, т, 6) where is the number of vehicles (consisting vehicle to B-1), number of samples for each vechile, is the number of timesteps (each represents 0.1s ), and representa the (x, y, vel, yau, acc, yawvel) corresponding agent coordinate of each vehicle. data_batch is dictionary that
the
The function should return a loss for every sample of every vehicle with the shape (B, N) return loss for every sample with the shape (N). \ You can use PyTorch and the following APIs if needed amd you should not use other unseen functions: \
transfor..coorr.egsttooooooodd (pos_pre, yaw_pred, data_batch) pos_pred is the predicted position trajectory in agent coordinate with shape (B, N, T, 2) and 2 correspond to (x, y). yaw_pred is the predicted yav trajectory in agent coordinate with shape (B, N, T, 1). The function transform the predicted position and yau from their agent coordinates into the world coordinate. The function returns position and yaw in the world coordinate with shape (B, N, T, 2) (B, N, T, 1). 2. trraaffor_ccord_ agentaagen____. i prod__rred_ooddd yav.pred_vorld
ind_k). pos_pred is the predicted position trajectory world coordinate with shape (B, N, т, 2) and 2 represents (x, y). yav_pred is the predicted yav trajectory in world coordinate with shape N, т, 1). data_batch is the dictionary mentioned before.
data_
ind_k is the index whose agent coordinate will be converted The function transform the predicted position and yaw from world coordinate to the agent coordinate. The function returns position and yaw in the agent coordinate with the shape (B, N, T, 2) and (B, N, T
1). \ select select_agent_ing aagnttanenttttde (x, i). has shape (B, N, т, k) where any positive integer and i a non non-tigative integer representing the selected index This function returns the slice
be
with inder with shape (N, T, k).
4. eisccurennt_lanee projectctooootionn (pos_pred, yav_pred, data_batch). pos_pred and yav_pred have
19

---

its shape (B, N, т, 2) and (B, N, T 1). They all in agent coordinate. data_batch dictionary mentioned earlier. This function returns the projection of each vehicle predicted trajectory current lane in agent coordinate with shape (B, N, т, 3) where 3 represents (x, yaw).
a
yaw- data_batch), It similar to lane except it returns the left lane waypoints. If there is no left lane, the original trajectory will returned.
6. get_right_lane projection (pos_pred, yav_pred, data_batch). It is similar to cot_currntt_lan except it returns the right there is
lane waypoints. left lane, the original trajectory will be returned."
System Message 2 Paired Example Query:
"Generate
loss class such that vehicle 1 should always drive with acceleration below acc_limit."
System Message 2 Paired Example 1 Response:
class AccLimitLoss (GuidanceLoss):
Keep accelerations below certain limit.
def __init__ (self, acc_limit):
,.,
accc_immit accelrrtion limit.
super () . init__(
self acc_limit acc_limit
11
12
def forl, x, datc_ agt_mask=None).
13
the current trajectory (B, !1 T, 6) where the number samples and (x, y, vel, yau, acc, yawvel)
loss: (B, !I) ,,>
17
18
if agt_mask is not None:
= x [agttmaskk] x(.... [4]]
21
acv_dev torch abs (acc) self acc_ limit
22
acc_loss torch. clip (acc_dev, min=0)
23
loss torch. mean (acc_ dim dim=[-2, -1])
return loss
System Message Paired Example 2 Query:
"Generate loss class such that vehicle 20 should always stay the
left side of vehicle 13."
20

---

System Message 3 Paired Example 2 Response:
class StayOnLeftLoss (GuidanceLoss))
Vehicle with index target_ ind should always keep on the left side vehicle with index ref_ind.
def ___innt__ (self, langet_ind- 20, ref_ind-13, decay_rate-0.9): super Di.__ini___0 self target_ ind target_ind ref_ine ref_ind self delaydecay_rate decey_rate
def forward x, data_batch, agt_mask=None)
B, N, I, shape
13
if agt_mask is not None:
14
x x [[agt_mask]
# Select positions
(B,N,T,6) -> (B,N,T, 2,
17
pos_pred x(..., :2]
18
# Select yaus
# (6),,T,6) -> (B,
yaw_pred x(..., 3:4]
21
convert prediction from the respective agent coordinates to the world coordinate
22
# (B,,,T,2), (B, dict >> (B,,,T,2), (B,H,T,1)
pos_ pre_lde__word,, world, yaw_pred_workd world transform_coord to (pos.pred, yaw_pred, data_batch
25
# convert prediction from the world coordinate to the agent rolf.refiimg coordinate (B, 2), (B,,,T,1), dict, int -> (B,N,T,2), (B,N,T,1) pos_pred ind coanston
27
pos_pred_world, yaw.pred_world, data_batch, self.ref_ind
29
select the relevant agents with index self. target_ing in the agent self ref_ind coordinate
# (B, H. T, 2), int (N, T, 2) pos irw.eeeiii_ree_iiiiii select agent
31
select_agentiinee ind (ptd_ppped.nn___ee.
self target_ind)
32
# select the relevant agents with index selg.target_ft ind in the agent relf.ref_im ind coordinate
33
(B, T, 2), int (N, T, pos jos_pred__jine in select agent ind (pospppod_edd pred_ireedrrrre indd aelf ref ref_ind)
34
# Since only care about the y-axis, only need compute the y-axis (rather than both x and y axis) deviation.
# (H, T, 2), (N, T, 2) -> (N, T) pos_ dev pos_pred.j.in..eff [.... pos_pred.i.in_rett [...,]]
39
# Clip the position deviation to so that only penalize the positive deviation
41
(11, (H1 T) pos_loss torch. clip ppss_dee min=0)
42
43
penalize early steps than later steps
21

---

exp_veights = torch. tensor ([self decay_rate *t for in
range (T)], device=x. device) device)
46
exp_ /= rep_weights.c sum pos_loss pos_loss exi_htights [None, :]
47
48
# Take the mean over time
# (N, T) (N)
pos_loss po mean (-1)
51
52
return pos_ loss
User Message User Query:
"Generate loss class such that vehicle should collide with vehicle
Received Message (from GPT4):
class Collisionloss (GuidancsLoas):
Vehicle with index target_ing should collide with vehicle with index ref ind.
def __init__ (self, target_ind-1 ref_ind-2, collisiion.adius-- 0): super Q._ init__ self target_ind ind target_ind fil.ref_inne ind_ind self caffccollisionrradio
collision_radius
11
forward x, dath_batch, agt_mask=None)) B, N, T, shape if agt_mask is not None:
12
13
x[agt_mask]
Select positions
# (B,N,T,O -> (B,N,T, 222)
17
pos_pred x(..., :2]
# Select yaws (B,N,T,6) -> (B,
20
yaw_pred x(..., 3:4]
21
# convert prediction from the respective agent coordinates to the world coordinate
(B,N,T,2), (B,,,T,1), dict -> (B,N.T,2), (B,M,T,1) pos_pred_world. yaw_pred_ transform coneform_ccordd.oom agents__gentsttate to to world spos_preee yaw_pred, data_
24
25
select the relevant agents with index self. target_ind in the world coordinate
27
# (B, H, T, 2), int -> (N, T, 2) pos i_.ppeddd_voriii. = select_ ind (pspppeeeeedddo self target_ind)
28
select the relevant agents with index self ind the world coordinate
30
# (B, N, T, 2), int -> (N, T, 2) pos_pred_j.worie = ienectt_ggnt_inpppppeeeeeee
31
ind ref_ind)
self
33
# Compute the distance between the two vehicles # (1i, т, 2), (N, T, 2) -> (N, T)
34
dist = torch norm ppod_ - pos.pred.j.vordd, dim=-1)
37
Compute the collision
by penalizing the distance
greater than the collision radius
# (II, T) (H1 T)
22

---

collision_loss torch clip (self collision_radius dist, min=
0)
40
41
# Take the mean over time
42
# (N, T) -> (N)
loss follision_lose mean (-1)
45
return loss
The resulted loss function from GPT4 showcases its capabilities:
It adeptly follows the template provided in the examples, producing syntactically correct program.
It comprehends the "collision" concept from the query and knows how integrate this concept with the input trajectory, as the examples used for prompting (acceleration limit and stay on left) are not related to collisions at all. This understanding results in loss that encourages trajectory collision between two vehicles, achieved through the helper functions.
F33.2 Success Example: GPT KeepDistance
In this success example, the returned program indicates that GPT4 understands the meaning of "keep within" and generates a function that promotes vehicle 1 to stay within certain distance of vehicle 2.
User Message User Query:
"Generate loss class such that vehicle 1 should always keep within
10-30m from vehicle 2.
Received Message (from GPT4):
class XeepDistanceL.ss (GuidanceLoss))
Vehicle with index target_ ind should always keep within din_distance and max pex_distance
from vehicle with index ref_ind
def __init__ (self, target_ind-l, ref ind=2, dintaien,,,--10, max.distance=30)
self telfftarget_ind ind target_ind ref_ine = ref_ind
self meldsainndistance min_distance self max_daannisteeee max_distance
11
12
def forvard x, datch agt_mask=None))
B, N, T, _ shape if agt_mask is not None:
15
x xlagt_mask]
16
# Select positions
# (B,N,T,6) -> (,,N) pos_pred x(..., :2] Select yaws
20
# (B,M,T,6) -> (B,
21
yav_pred I[..., 3:4]
# convert prediction from the respective agent coordinates to the world coordinate
23
(B,,,T,2), (B, diet -> (B,,,T,2), (B,M,T,1) pos_pred_world, vorld cransform_coor_ tor (pos_pred) yaw_pred, data_batch
24
25
23

---

select the relevant agents with index self. target_ind in the world coordinate
28
(B, 11, T, 2), int -> (N, T, 2) pos_pred_i_wordd select_agens aleett_geent_ie and ppos_pred_world, self target_ind)
29
select the relevant agents with index self.red ind the world coordinate
31
# (B, H1 7, 2), int -> (N, T, 2) jos_pred_..world = select_ ind prod_pped_world, self ref_ind)
32
33
Compute the distance between the two vehicles # (11, т, 2), (N, Т, 2) -> (N, T) distance torch. norm ppos.pred
35
pos.pred.j.world, dim =-1)
37
38
# Compute the deviation from the desired distance range # (11, T) -> (N, T)
39
distance_dey_min self distancee distance distance_dev_max distance self max_distance
41
42
43
# Clip the negative values to 0 # (Ii, T) -> (Ii, T)
histance__oss__r distance_loss_met
torch. clip cealsttaneedev_miii min=0)
46
torch. clip ddiitanneeeee.nnnt. min=0)
47
48
# Combine the losses # (N1 T) (N, T)
distance_
distance_loss_mitt
Distance_loss.mt
51
52
# Take the mean over time # (NI, T) -> (N)
loss torch. (distance__oss, dim=-1)
55
56
return loss
F.3.3 Success Example: GPT SameDirection
In this success example, the returned program indicates that GPT4 understands the meaning of "same direction" and generates a function that promotes vehicle and vehicle to move with the same orientation.
User Message User Query:
"Generate loss class such that vehicle should move along the same direction as vehicle 2."
Received Message (from GPT4):
class SameDirectionLoss (GuidanceLoas):
Vehicle with index target_ind should move along the same direction vehicle with index ref ind_
def inint target_ind- ref_ind-2):
super in._.init__0 aelf target_ing ind target_ind relfiref_ind = ref_ind
def fsrv,rd x, datch agt_mask-Non.
11
B, N, T, shape if agt_mask is not None:
[agt_mask]]

---

# Select yaus
# (B, N,,, -> (B,N,I,1)
16
yaw_pred x(..., 3:4]
17
# convert prediction from the respective agent coordinates to the world coordinate # (B,,,T,2), (B, dict - (B,N.I,2), (B,M,T,1) yaw_pred_world
wransform_coord_ageeee to_.tttstttoooodddtt.. yav_pred, bata_batch) # select the relevant agents with index self target_r # (B, 11, T, 1) -> (N, T, 1) yaw_pred_i
:2]
30
21
self target_ind)
23
# select the relevant agents with index self ref relf.ref_im # (B, Ni т, 1) -> (!i, т, 1) yaw.pred_j = select_agent. iygget..iddaawappreeeee
24
25
self ref ind)
Compute the yaw deviation (11 T, 1), (N, T, 1) -> (N, yaw_dev torch. abs (yav_pred yav_pred_j)) squeeze (-1)
29
30
Normalize the yas deviation to [0, pi] (N, T) (N, T) yaw_dev torch. fnod yaw_dev mach.min(yav_dde 2 math. pi - yaw_dev)
32
math.pi.
34
# Penalize deviations larger than small threshold (e.g., 0.1 radian)
37
threshold 0.1 yaw_loss torch.
38
- threshold, min=0)
Take mean over time # (N1, T) (N) loss = torch. mean (yaw_loas, dim=-1)
41
42
return loss
F.3.4 Success Example: GPT Collision From Behind
In this success example, the returned program indicates that GPT4 understands the meaning of "collide" and "from behind" and create the respective losses (see collision_los: in line 49 and not behind loss in line 53), respectively. It finally combines the two losses by taking their product as the final loss (line 57).
User Message User Query:
"Generate loss class such that vehicle should collide with vehicle
2 from behind."
Received Message (from GPT4):
class CollideFreeBehhndLooss (arieaadddooosiinaaeeaeaeaaea (GuidanceLoss):
Vehicle with index target_ind should collide with vehicle with index ref ind from behind.
def .nint target_ind- ind=i collision.threholle.. 0):
6
(
Solf.target_ing = target_ind
ind ref_ind
self. colf.coollssin.ttreeeee
collision_threshold
10
def forvardseeff x, data_ agt_mask-Nonee B, N, T, x.shape
25

---

if agt_mask is not None:
x[aat_maak]
15
Select positions # (, -> (B,M,T,
16
pos_pred x(..., :2] Select yaus (B,N,T,6) -> (B,N,T,1)
20
yaw_pred x(..., 3:4]
21
# convert prediction from the respective agent coordinates to the world coordinate (B,,,T,2), (B,N,I,1), dict -> (B,N,I,2), (B,M,T,1) pos_pred_world, yaw_pred transform_comm coord ctwpsffrrmccoordaeeettttoo agents to world (poe_pred, yav.pred, data_bater
23
24
# convert prediction from the world coordinate to the agent self.ref_ind coordinate (B,,,T,2), (B,,,T,1), dict, int -> (B,N, (B,N,I,1) pos_pred_in_reff in_ref_ind,
26
27
pos_pred_vorid, yaw.pred_world, data_batch, self.ref_ind)
29
# select the relevant agents with index self targetarett_nn ind in the agent self ref_ing ind coordinate
30
(B, T, 2), int (N, T, 2) pos innsprrrddiiinrfffiiiii select_ages agent iegpt_iinppo__.rddiii..e.. indd__in_e__idd
31
self target_irdd
# select the relevant agents with index self target_nn in the agent self
coordinate
33
# (B, H, 7, 2), int -> (N, т, 2) po ie__iiii____ia_eee ind = select_ agent _nddttinddiie
pred_in_refs _idd self relfidef_ind)
35
36
# Compute the distance between the two vehicles along the x- axis (Iongitudinal)
# (II, T, 2), (N, T, 2) -> (N, T) dist_x po ind [..., 0] pos_pred.i.in_ree wewiineeffiieeeeee ind
[...,
39
# Compute the distance between the two vehicles along the axis (lateral)
(II, T, 2), (N, T, 2) -> (N, T) dist_y po ind ...., [..., 1]
41
pos_pred_i.in_.ef..
42
Compute the squared distance between the two vehicles (11, T), (N, T) -> (N, dist dist_equared dist_x ** dist_y 2
45
46
# Compute the collision loss by checking if the squared distance is less than the collision threshold
48
# (11, T) (I, T) collision_ loss Qdist_squared < wow.colsiiionttreehooooo
49
float
51
Penalize the cases where the target vehicle is not behind the reference vehicle
52
# (II, T) -> (II T) not not_behind_loss torch, clip (-dist_x, min-0)
55
Combine the collision loss and not not_behind_les # (1I, T), (N, 7) -> (N, T)
56
total_ loss = lollision_loss not_behind.loss
26

---

Take the time # (N, T) (N)
00
61
loss torch. mean (total_loss, dim=-1)
62
return loss
F.3.5
Success Example: Lane Following
In this success example, the returned program indicates that GPT4 can extropolate the number of vehicles of a scenario (although both provided examples only two vehicles are involved, GPT4 can generalize the loss function involving three vehicles). Besides, GPT4 knows how to use functions described in helper function APIs but not even used the provided paired examples. In particular, it knows
wwt.ecuretttaaeeeee lane-projection to get the lane information in order to promote vehicles lane following behavior when it was asked to let the vehicles to "follow their current lanes".
User Message User Query:
"Generate loss class such that vehicle 1, vehicle 2, and vehicle all follow their current lanes."
Received Message (from GPT4):
class LaneFollovingLoss (Guidancelosss)
Vehicles with indices target_ inds should follow their current lanes
def __init__
2, 31):
super init___ self target_inds target_inds
def forward x, data_batch, agt_mask=None))
B, N, T, T, x shape
if agt_mask is not None:
x[agt_mask]
Select positions
14
# (,,,, -> (B,N,T, 22 pos_ x(..., :2]
15
# Select yaws
(,,,,T,6) -> (B, (B,,,T,1) yav_pred I(..., 3:4]
18
19
# Get the current lane projection in agent coordinate # (B,,,T,2), (B,M,T,1), dict -> (B,N,T,3) www.aaan.ppooeeioopppppppoooc lane_proj get.ourrent.lanoprroo data_batch)
22
yas_pred,
23
# Compute the deviation from the lane projection 2,2), (B,H,T,3) -> (B, (,2)) pos_dev pos_pred lane_proj [..., :2] yaw_dev yaw_pred lane_proj [..., 2:3]
26
27
# Compute the squared loss for position and yaw deviation (B, -> (B,N,T) T)
31
pos_loss torch. (pos_ded 2, dim=-1)
32
# 1, -> (B,N,
yaw_loss forch.squeess (yav_dev ** 2, dim=-1)
35
Combine position yaw loss
36
# (B,N,T) (B,N,T) -> (T total_loss pos_loss + yav voss
37
27

---

# Select the loss for the target vehicles
# (B, list -> (tan(tagett_inds)) N, T)
41
target_losses [select_agent_ (total_loss, ind) for ind
self targ.target
42
Stack the losses for target vehicles list -> (len (target_ target_losses torch. strch (targett dim=0)
7)
45
46
# Take the mean over time # (len (target__idds) N, T) -> (targgot___dds)) N) target_losses torch. mean (target_ dim=-1)
49
50
51
Take the mean over target vehicles # (len (target_ inds), N) -> (N)
loss torch.meat (target_losses dim=0)
54
55
return loss
F.4
Failure Cases
The main limitation of the current language interface is on the complex interactions between the vehicles and the map. As we don't I explicitly pass the map information into the language interface, cannot handle commands involving heavy interaction with the map. For example, "vehicle A and B move the rightmost lane one by one and then both turn right at the next intersection". However, believe if one provides more helper functions (especially those interacting with the map) and more relevant examples to the language interface, LLM can handle such more complicated commands. Additionally, we observe GPT4 can also fail to understand some concepts properly (c.g., "cut in"") As the current work aims to provide early promising results on the feasibility text-to-traffic, we leave a deeper exploration for future work.
F4.1 Failure Example Move Rightmost and Right Turn
We provide a failure example below. The system messages remain the same as in Appendix F.3.1. 1. The generated loss function fails because it fails to consider "turn right at next intersection". This can be potentially fixed by providing a couple more helper functions (e.g., a function to get the next intersection and corresponding lane's information) and an additional example that interacts with intersection. In other words, our framework can potentially support such queries via extensions on helper functions and additional examples. As in the current work, we mainly want to show framework of using LLM on language- guided realistic traffic generation, we leave comprehensive study covering all kinds of complex queries for traffic generation as future work.
User Message User Query:
"vehicle 1 and move the rightmost lane by and then both turn right at the next intersection
Received Message (from GPT4):
class RightTurnLoss (GuidanceLoss)) Vehicle and should move to the rightmost lane one by one and then both turn right at the next intersection.
"
def _sen target_inds- 2), decay_rate-0.9):
super in.__init__6 self. target_inds = target_inde
self. decay_rate decay_rate
28

---

def forward (self, x, data_batch, agt_mask=None)
11
B, N, T, shape
12
if agt_mask is not None:
13
x -x- x[agt_mask]
# Select positions
(B,N,T,6) -> (B,N,T,2)
16
poa_pred xx..., :2]
# Select yaus
# (B,,,T,6) -> (B,NTT, yaw_pred x(...; 3:4]
20
21
Get the right lane projection
22
# (B, H1 T, 2), (B, N, т, 1), dict -> (B, N, Т, 3) right_land 1ate_______pp] get_right_ perps_llee_ppoeeteeeee (pos_pred, yaw_pred lata_batch)
24
25
# Compute the deviation between the predicted position and the right lane projection
# (B, N1 7, 2), (B, N, T, 3) -> (B, N, T, 2) pos dev pos_pred right_lane_prof [..., :2]
27
28
29
# Compute the squared distance
# (B, H, т, 2) -> (B, N, T) pos_dev_sq torch. sum (pos_dev 2, dim=-1)
32
33
# Penalize early steps more than later steps exp_veights torch. tensor ([self decay_ ** t for t in (T)], device)
34
exp_ xei_heettts /- exp_weights sum pos_dev_ag pos_dev_sq
36
weights None, :]
37
# Take the mean over time # (B, Ni T) -> (B, N1) pos_loss pos_dev_sq mea._ss..meem... mean (-1)
40
41
42
# Select the loss for target vehicles # (B, n1), list -> (len (Jen (target_inds). N)
target_loss torch. stack (tacl((seeect,,.... ([select_ aaaettt__genntiiiii ind (pos_ loss, i) for self
45
# Take the mean over target vehicles # (len (target_ inds), N) -> (N) loss larget_loss.meande
48
49
return loss
F.4.2 Failure Example: Cut In if Behind and On The Left
In this failure example, the returned program indicates that GPT4 fails understand "cut in" only promotes vehicle to be ahead of vehicle when it is behind and the left. However, cut-in requires vehicle to also move horizontally (i.e., along the axis).
User Message User Query:
"Generate loss class such that vehicle 1 should cut in ahead of vehicle 2 if it is behind vehicle 2 and its left lane."
Received Message (from GPT4):
class CutInLoss (GuidanceLoss))
29

---

Vehicle with index target_ind should cut in ahead of vehicle with index ref_ind if it is behind vehicle ref_ind and on its left lane.
def __.nnit__sslle target_ind-1, ref_ind-2, decay.rate-0.9);
super() (nniiit...it_ self target_ind target_ind self.ref_ind ref_ind self self.decay_rate date
11
def forward x, data_ agt_mask=None)
12
B, N, T, shape if agt_mask is not None:
13
x[agt_mask]
Select positions
# (B,M,T,6) -> (B,N,T,2)
poa_pred x(..., :2]
# Select yaus
# (B,N,T,6) -> 1,
20
yaw-pred x(..., 3:4]
21
# convert prediction from the respective agent coordinates to the world coordinate
# (B,,,T,2), (B, dict -> (B,N,T,2), (B,H,T,1)
pos_pred_world, waw-pred_world haapssfommccorrdaagntssss
24
(pos_pred, yaw_pred, data_batch
25
# convert prediction from the world coordiitat to the agent
(B,,,T,1), dict, self.ref_ ind coordinate (B,N,T,2), (B,N,T,1) pos_pred_in in_ref_ind sranfforn.corr_ioolll..o.oeee
27
pos_pred_vorid, yaw.pred_worid. data_batch, self.ref_ind)
28
29
# select the relevant agents with index self. target ind in the
agent self ref ind coordinate
# (B, T, 2), int (N, T, select i iagpecc.ooo..ddpppppr...........
31
pos_
self target_ind)
# select the relevant agents with index self target_inde in the agent self ref_ coordinate
33
# (B, H1 T, 2), int (N, T, 2) pos pos_predd_j.n.rrff
34
select agent (ttp__indpoooppd___11__1 pred_in_ref_iile self ref red_ind)
36
Compute the x-axis and y-axis deviation # (11, T, 2), (N, т, 2) -> (N, T) pos_dev_x pos_pred_j.in_et inf_reff__iii ind [.... 0]
37
38
itn_predd.ii_rroffiddee
0]
39
pos_ pos_dev__ pos_pred__ inn_ [...,
1]
ind [...,
1]
41
Check the target vehicle is behind and on the left lane of the reference vehicle behind and left (pos_dev_x 0) & (pos_dev_y' 0)
42
Penalize the target vehicle for not cutting in ahead of the reference vehicle
45
cut in lo88 torch. where (behiebb where(behind,and_11e pos_dev_x, torch. zeros like (p)s_de)))
30

---

Penalize early steps than later steps exp_veights torch, tensor ([self. decay_rate for
48
range (T)), prrice=x.devieen
49
exp_veights /- exp_weights. sum ()
cut_in_loss
Take the cut_in_loss exp_weights None, :] (I1, T) (N)
time
52
53
cut_in_loss = cut out_in_lee cut_in_loss..a mean (-1)
54
return cut_in_loss
F55
Qualitative comparison between GPT4 and GPT3.5
We next show two examples where we send the same message but use GPT3.5 rather than GPT4 as the LLM. keep the queries to be exactly the same as in Appendix F.3.1 and Appendix F.3.2, respectively. The received messages are shown in the following. Both loss functions are correct but
similar the provided examples by containing code applying decay loss over time dimension (e.g., see line 45-48 in Received Message (from GPT3.5)). These lines of code almost the exactly ones (except the loss name) in provided example (stay on left) but these code are not always necessary to be included in the loss functions.
F.5.1
GPT Collision (GPT3.5)
User Message User Query:
"Generate loss class such that vehicle should collide with vehicle
Received Message (from GPT3.5):
class CollisionLoss (GuidanceLoas):
Vehicle with index target_ ind should collide with vehicle with index rnd_
,.
def __init__ target_ind-1, ref_ind=2, decay_rate=0.9, collision.distancee distance=2.00
(u.err))..nni_.. self target_ ind target_ind
sel_inef ref_ind
daffydecyy_rate = decay_rate
celf.coolision = collision_distanu
def forvard x, dath_ st__maskNoone
13
B, N, T, shape if agt_mask is not None:
14
= [[agt_mask] Select positions
17
(B,M,T,6) -> (B,,, pos_pred x(..., :2]
18
# Select yaws
# (B,N,T,6) -> (B,N,T, 1)
21
yaw.pred x(..., 3:4]
22
# convert prediction from the respective agent coordinates to the world coordinate
23
# (B,M,T,2), (B,,,T,1), dict -> (B,,,T,2), (B,H,T,1) pos_pred_worle yow- transform_ wow. (pos_pred yaw_pred, data_bateh
25
26
# convert prediction from the world coordinatee to the agent self.ref ind coordinate
31

---

# (B, (,,,,,,2), (B,,,I,1), dict, int -> (8,N,T,2), (B,M,T,1) poe_pred_ in_ref_ind, craaffor..corttooodddb...aattt
pos_pred_worid,
yaw.pred_world; data_batch, self.ref_ind)
select the relevant agents with index self target_ ind the agent relf.ref_ime coordinate
31
# (B, H, T, 2), int -> (N, T, 2) salect_agents
pos_ in
self target_ind) # select the relevant agents with index self. targeta ind in the agent self ief_ind coordinate
33
(B, 7, 2), int (N, T, pos_ peep-pedddd__eeeeeee_ee____ee
35
lahect_agentt
leeeect_aaeetiinngeee wpw..pppeddiinnee._...o self.ref ref_ind)
36
# Compute the distance between the tho agents (N, T, 2), (N, T, 2) -> (N, dist torch norm ipw.e_eedddiiin___ppepppp
39
pred_i_in. ref ind
pt.pprddj..inref_... dim= -1)
41
Clip the distance the collision distance # (II, T) -> (I, T) dist_loss torch. clip(self collissoondistaanee - dist, min=0)
42
43
penalize early steps more than later steps exp_veights torch, tensor ([self decay_ for in range(T)] device=x. device)
46
exp_weighss /= exp_weights.. sum ()
dist_loss dist_loss exp_weights [None, :]
49
Take the mean over time # (II, T) (N)
50
51
dist_loss dist_lose mean (-1)
dist loss
F,5.2
GPT Keep Distance (GPT3.5)
User Message User Query:
"Generate loss class such that vehicle 1 should always keep within 10m30m from vehicle 2."
Received Mcssage (from GPT3.5):
class KeepDistanceloss (GuidanceLLoss):
Vehicle with index target_ind should always keep within min_dist and max dist from vehicle with index ref_ind.
def ..init_ target_ind-l
ind=1 min_dist=10, max_dist- 30, decay_rate-0.9)
6
(uper()...inii_...
self. talgetarget_ind target_ind
self.ref_ind ref_ind
self.min.dist min_dist
10
max_dist
self felf.decay_rate = decay_rate
32

---

def forward (self, data_batch, agt_mask=None))
N, I, shape
15
if agt_mask is not None:
16
x x[agt_mask]
# Select positions
# 6, -> (B,N,T,2)
poa_pred x[..., :2]
20
# Select yaws
21
# (B,M,T,6) -> (,1
22
yaw-pred x(..., 3:4]
convert prediction from the respective agent coordinates to the world coordinate (B,,,T,2), (B, -> (B,,,T,2), (B,N,T,1) pes_pred_wolld wowld coo.
24
25
poss_pred yaw.pred, data_batch
27
convert prediction from the world coordinatee to the agent self. coordinate # 2,2) (B,,,T,1), dict, int -> (B,N,T,2), (B,N,T,1) pos_pred in rn_ craasfor..oordd..oddd.......
28
pos_pred_world, yav.pred_world, data_batch, self.ref_ind)
select the relevant agents with index self. target ind in the agent self.ref_ind coordinate
32
# (B, N1 7, 2), int -> (N, T, 2) pos pos_
select agent_ ipwp__indppos.peddiin... self target_ind
34
select the relevant agents with index self.targes in the agent self ref_irffind coordinate
35
# (B, 1. T, 2), int (N, T, 2) pos_ pre__pee___.o re
36
select select_agent_teee hoop.__npppppppdd.n...r.... self ref_nd)
37
# Compute the distance between the two agents # (11, 2), (N, 2) -> (N, dist torch, norm (pod_ppeed_ii_ppppp in
40
bee_pred_i_in_ree dim= -1)
42
Compute the deviation from the desired distance range # (II, T) (N1, T) divt_dev torch.where (dist < milfsmin_iist, aal.minndee
43
44
dist, torch. where (dist > self max_ dist self max_dist, torch zeros like dist)))
45
# penalize early steps than later steps exp_heights torch. tensor ([self decay_ for (T)] device=x, device))
48
exp_veights /= wep_weights.sum ()
dist_dev dist_dev exp_weights [Nione, :]
Take the # (II, T) (N)
time
51
52
dist_loss dist_der. mean (-1)
53
return dist_loss
33

---

G Quantitative Evaluation Results with Additional Runs
To substantiate the superior performance of our method, we conduct additional experiments, the results of which are presented in Table 3. these experiments, we compare CTG++ against the strongest baseline, CTG, across three distinct runs with varying random seeds. The settings are the same those for Table and we take the average and standard deviation of the three runs. We highlight the better value only when it is significantly better than the other (i.e., if the values of the two methods differ by at least the sum of their standard deviations). In all eight settings, CTG++ significantly performs better than CTG in terms of failure rate and scene-level realism. CTG++ also tends to perform better than CTG in terms of rule satisfaction (winning 4 and tied on 6). In terms of realism, CTG++ is comparable to CTG (winning 3, losing 2, and tied on 3). Thus, the results suggest that CTG++ significantly performs better than the strongest baseline CTG++.
Table 3: Quantitative results (mean with standard deviation of three runs) of CTG++ and the strongest baselines CTG under GPT gen-reene rules and STL rules. We highlight the winning method that is significantly better than the other (i.c., if the values of the methods differ by least the sum of their standard deviations).
distance
GPT collision
fail
nile
real
nel real
fail
rule
reil
rel real 0.349 0,002
CTG CiG.+
0.327 a 0.02
0
0.07 0.006 0.071
0.343 a 0.003 0.334 0.003
0.346 - 0.018
0
0.071 0.004
0.171
0.006
0.264
0.084 0.004
0.006
collionn
speed limit
fail
rale 0.003
nell
rel real
full
nile 0.029 0
noil
nel real 0.363 0.003
CTG
0,187 a 0.01
0.048
0.048 a 0.005
0.346 0.002
0.129 2 0. 00002
0.077 & 0.002
CTG
0.085 0.002
0.045 0.001
0.047 a 0.007
0.326
0.006
0.087 =: 0 004
O. 0.028 a
042 0. 0.003
O 0.34 0.004
larget spesi
affira
mie 0 0.281 = 0.001
nal
nclacal
CTG
0.083 0.007
0.108 0.003
0.379 0.002
0.367 008
0.003 0.008 O
0.041 0 002
0.343 0.003
CTG
0.062 0.001
0.272 a 0.002
0.0 0.068 0.004
0.571 0.004
0.104 00.005
0.044 a 0.006
0.523 a 0.005
goall koppokeeagggr quod
CTT
6. 6.1155 0.15
2.00 107 0:066
6.59 8.50
0.005 (07
0.3 0.00
0. 158 e 0.0
0.000 0.001
5. THE 0.17
0.066 0 006
0
CTO++
0.002
2.361 0.021
0.394 0.300
001
0.336 0.004
0.08
0,008 2
0.44 0.051
0.075 0.00L
9.323 0.000
34


Reviews:
[{"review_id": "oMSXROObdtX", "reviewer": "Reviewer_QJon", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to use language to generate different traffic configurations. The authors use a LLM to generate a guidance function that is used to steer a diffusion model for traffic synthesis. The authors further propose architectures that enable effective learning of the diffusion model such as a agent-centric coordinate system and a series of agent pairwise and temporal attention blocks. The authors show the efficacy of the approach across many different tasks.", "summary_of_recommendation": "I enjoyed reading this paper and think it should be accepted.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "It might be interesting to show some more examples of synthesized programs -- to what extent does the prompted example program need to roughly reflect the cost function we want to use for guidance?\n\nIt might be interesting to try to the ULA sampler from [1] to combine the guidance function with diffusion sampling. Directly adding gradients during sampling has some theoretical issues that are addressed by running additional MCMC steps to incoperate the constraint.\n\n[1] Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I enjoyed reading this paper and thought the paper was well written. Each of the proposed improvement are natural, well motivated, and clearly described\n\nThe paper demonstrates strong emperical results in Table 1 and outperforms relevant baselines.\n\nIt Figure clearly illustrates both the generated programs and the effect of each design choice in the paper.\n\nThis paper tackles a timely problem that is of interest to many people at CoRL.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "iKRCruiDNyG", "reviewer": "Reviewer_eWu2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a natural-language-conditional diffusion model for generating traffic scenes. It presents a sophisticated workflow where a transformer is used as the diffusion mode while GPT4 is used to generate a loss function conditional on the language instruction. This loss function is then used to guide a transformer-based diffusion model sampling by adding a score vector point to the gradient direction of this language-conditional loss.", "summary_of_recommendation": "The significance of this paper appears unclear to me (therefore leaning toward rejection). This may be due to my lack of background in this topic (therefore low confidence).", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "\nIt seems the language-based guidance is only sensible for an actual traffic state. Applying this loss function to the denoising process where t is not 0 seems strange to me. Can the authors comment on this?\n\nI don’t have a good grasp of the absolute values of the metrics in Table 1. Can the authors give error bars to the results in Table 1?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not very familiar with the task setting of traffic simulation. My review is mainly on the technical method and assumes the experimental benchmark and baselines are standard and competitive.\n\nThe transformer-based neural model seems sensible. While it is interesting to see GPT4 can generate loss functions for guiding diffusion models, it seems the authors are evaluating 8 template-based different tasks. It is not entirely clear to me to what extent the tasks are diverse in terms of natural language instructions. It is also unclear how GPT4 would perform in scenarios not tested by the authors. It is hard for me to assess the significance of the proposed framework because it is hard to assess the model’s capability outside the 8 scenarios that the authors constructed.\n\nIs the main claim of the paper on the superiority of its transformer-based diffusion model or language-conditional generation capability? If former, it seems to me a more comprehensive benchmarking (potentially without language) will be convincing. If latter, it seems a certain level of diversity in the language instruction is needed to establish evidence for GPT4’s capability to generate good loss function for diffusion guidance.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "GI8xe1YeN4", "reviewer": "Reviewer_qk9Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes CTG++, a new method to generate trajectories for traffic simulation. It presents the following novel elements: \n\n- Scene-level conditional diffusion model, with multi-agent trajectories as output \n- Guided sampling of trajectories from natural language \n\nCTG++ builds upon Controllable Traffic Generation (CTG), presenting an architecture that models multi-agent trajectories using a spatio-temporal diffusion model. After denoising, it subsequently produces a set of trajectories for each agent. The paper leverages prompts from GPT4 to guide trajectory sampling. To overcome the lack of text-to-traffic data, they devised a method to inject language by prompting differentiable loss functions for different scenarios.", "summary_of_recommendation": "The approach is interesting, but the experimental results are underwhelming vs baselines. Currently, the results do not support the usefulness of the presented method.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- How does this method compare when evaluated more widely? \n- How often is the output from the LLM incorrect? Does the method have a means for detecting this?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n- Model architectures that take inspiration from several subfields. Captures the multi-agent aspects of traffic simulation well. \n- Novel approach of generating differentiable loss functions with LLM. This allows for more controllability compared to user-defined rules. \n\n## Weaknesses\n\n- The results presented in Table 1 shows that performance of CTG++ does not always perform better than baselines, and when it does, performance is only marginally better. is only marginally better.  \n- Evaluation is performed only on 1 dataset. How does this method compare when evaluated more widely? \n- Exploiting LLMs is a core contribution of this work; in order to get a deeper understanding of the approach, it would be useful to investigate the effect of a weaker/stringer LLM. Would this lead to better results? \n- The page formatting has been abused in order to try and fit into the 8 page limit which made it difficult to parse captions.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "FlvWFIWX5j", "reviewer": "Reviewer_J6rY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "CTG++ is a very interesting and meaningful work for the realistic and controllable traffic simulation area. Different from CTG, it uses a scene-level conditional diffusion model with agent-centric coordinates, leveraging a spatial-temporal transformer backbone. In addition, a language interface is adept at generating trajectories that align with user-defined rules in language, which can improve scenario controllability and the user's interactivity. The experimental results prove the effectiveness of the proposed CTG++.", "summary_of_recommendation": "CTG++ uses a conditional diffusion model at the scene level with agent-centered coordinates and a spatial-temporal transformer backbone. In addition, a language interface is adept at generating trajectories that align with user-defined rules in language, which can improve the scenario controllability and the user's interactivity. This work is well organized and has a certain novelty.  It is beneficial for the autonomous driving community if this work is open-sourced. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "What is the real query of the LLM? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is well organized and well written. The proposed CTG++ is interesting and novel.\n\nI am curious about how to choose two interesting vehicles. In Fig. 3, it is listed in the response of LLM based on the select_agent_ind. If the query is shown as the one in Fig. 3, how does the LLM know the distance of two vehicles without any scenario context information? I suggest the authors provide a whole example of the LLM guidance in detail.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes CTG++, a natural language-conditioned diffusion model for generate traffic configurations. CTG++ uses an LLM to generate a guidance function that then steers a diffusion model for traffic synthesis. The paper additionally proposes architectures that enable effective learning of the diffusion model such as an agent-centric coordinate system and a series of agent pairwise and temporal attention blocks. Experiments across different tasks show the efficacy of the approach.

Reviewers recognize that the paper is well-written (QJon, qk9Q), presents a novel approach (qk9Q, J6rY) that tackles a timely problem that is of interest to many people at CoRL (QJon). Post-rebuttal (on request by the reviewers), the paper has also been improved to include more examples and analysis of synthesized programs (QJon) and additional experiment details and discussions (qk9Q, J6rY, eWu2). Several reviewers have acknowledged these changes and upgraded their scores. There are still some concerns on that the evaluations could be made stronger (in particular, it's relevance beyond language-to-traffic, qk9Q, eWu2) – nevertheless, within scope of the applications presented, reviewer sentiment is overall positive, leaning "strong accept" with extra points to novelty.

I agree with reviewers that the ideas presented in this work (while predominantly evaluated in the context of traffic simulation) are novel and well-positioned to inspire others in the community to extend these ideas across other pertinent areas in robotics.
Decision: Accept (Oral)