Promptable Closed-loop Traffic Simulation
Authors: Shuhan Tan
Boris Ivanovic
Yuxiao Chen
Boyi Li
Xinshuo Weng
Yulong Cao
Philipp Kraehenbuehl
Marco Pavone
Venue: CoRL 2024 (2024)
Keywords: Autonomous Driving
Scenario Generation
Traffic Simulation
TL;DR: 1) We propose ProSim, a multimodal promptable closed-loop traffic simulation framework. 2) We create ProSim-Instruct-520k, a large-scale multimodal prompt-scenario paired driving dataset.

Abstract: Simulation stands as a cornerstone for safe and efficient autonomous driving development. At its core a simulation system ought to produce realistic, reactive, and controllable traffic patterns. In this paper, we propose ProSim, a multimodal promptable closed-loop traffic simulation framework. ProSim allows the user to give a complex set of numerical, categorical or textual prompts to instruct each agent’s behavior and intention. ProSim then rolls out a traffic scenario in a closed-loop manner, modeling each agent’s interaction with other traffic participants. Our experiments show that ProSim achieves high prompt controllability given different user prompts, while reaching competitive performance on the Waymo Sim Agents Challenge when no prompt is given. To support research on promptable traffic simulation, we create ProSim-Instruct-520k, a multimodal prompt-scenario paired driving dataset with over 10M text prompts for over 520k real-world driving scenarios. We will release data, benchmark, and labeling tools of ProSim-Instruct-520k upon publication.

Promptable Closed-loop Traffic Simulation
Shuhan Tan 1+ Boris Ivanovic
Yuxiao Chen'
Boyi Li Marco Pavone
Xinshuo Weng?
Yulong Cao
Philipp Krähenbühl UT Austin "NVIDIA
Abstract:
Simulation stands as cornerstone for safe and efficient autonomous driving development. At its simulation system ought to produce realis- tic, reactive, and controllable traffic patterns. In this paper, we propose ProSim, a multimodal promptable closed-loop traffic simulation framework. ProSim al- lows the user to give a complex set of numerical, categorical or textual prompts to instruct each agent's behavior and intention. ProSim then rolls out a traffic sce- nario in closed-loop manner, modeling each agent's 's interaction with other traffic participants. Our experiments show that ProSim achieves high prompt control- lability given different user prompts, while reaching competitive performance on the Waymo Sim Agents Challenge when no prompt is given. To support research on promptable traffic simulation, we create modal prompt-scenario paired driving dataset with over 10M text prompts for over 520k real-world driving scenarios. Code, data and labeling tools avaliable http:://arrisstgxgittuue
a multi-
Keywords: Autonomous Driving, Scenario Generation, Traffic Simulation
1 Introduction
Simulation provides an efficient way to evaluate AV systems in realistic environments. It avoids the high costs and risks associated with real-world testing, and forms the first step of any deployment cycle of an autonomous driving policy. A simulator is only able to fulfill its role it is capable of mimicking real-world driving conditions. Chief among these conditions is reahly-r sim- ulation of interactive traffic agents and their behavior patterns. Further, the simulation must be controllable enable the creation of interesting traffic scenarios by editing and customizing each agent's 's behaviors and intentions via user-friendly inputs prompts.
To capture these needs, we propose a new task: promptable closed-loop traffic simulation (Sec- tion 3). Aside from simulating realistic traffic agent interactions in loosed traffic models should also generate agent motions that satisfy complex set of user-specified prompts. Our baseline algo- rithm ProSim (Figure takes scene initialization and multimodal prompts as inputs, and produces set of agent policies that yield closed- reactive scenario rollouts. As show in Section 6, ProSim achieves high realism and controllability with variety complex user prompts. ProSim even achieves competitive performance on the Waymo Open Sim Agent Challenge [1] without prompts.
To further enable research into promptable closed-loop traffic simulation task, we created ProSim-Instruct-S5
multimodal paired prompt-scenario dataset (Section 5). Leveraging the real-world AV dataset [2], we provide rich prompts for all agents. Specifically, for each sce- mario we label agents' Goal Points (their destinations). Action Tag A.c.lerraee,..eee LefiTurm), Route Sketch (a noisy sketch of an agent's 's route), and Text instructions (e.g., "Instruct <A0> to decelerate before turning right." "), visualized in Figure 1. These prompts simulate some of the diverse ways users may wish to prompt scenarios. We ensure the labeled prompts accurately depict the agent's ground- truth (GT) behavior with careful human quality assurance stages. ProSim-Inee Instruct----ttt 520k
Work done during an internship at NVIDIA.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Scene initilization o
Agent Querios
Goal
Sketch
Action
Text
Policy Tokers
Closed-loop Rollout
A
R
W1
Policy
A
W2
Policcy
Env
Encoder
Generator
LLM + LoRA
Policc
(a)
(b)
(c)
(d)
Goal A7, AB
Sketch A29
Goal A3
Sketch AB
Goal AB, A14
(A14, Stopping!
(A7, Decelerate)
Action (A29,
Unconditional Rollout
Text: "aate <A3> turn right smoothly"
Text "Instruct </29>> turnleet"
Text: LLet <AB> stop"
Figure I:
Overview ProSim and promptable closed-loop traffic simulation. All agents are con- trolled by ProSim: green ones are unconditioned and others are prompted with multimodal prompts.
contains 520K prompt-scenario pairs of real-world driving data, including over 10M sentences of text prompts for 10M unique agents and 575 hours of driving. We will release the data, a benchmark, and labeling tools of ProSim Instruct -ettrrrccttktkkkkk to facilitate future research upon publication.
Our key contributions are threefold:
1. We introduce ProSim, a first-of-its-kine promptable closed-loop traffic simulation framework.
2. We create ProSim-Instruce Instruct 520k, a large-scale multimodal prompt-scenario driving dataset. It is the first driving dataset with semantic-rich agent motion labels and text captions.
3. We will release code and checkpoints of ProSim. We will also release data, a benchmark, and the labeling tools of Seetim-Innntuttt2221 to facilitate agent motion simulation research.
2 Related Work
Unconditional closed-loop traffic simulation. Recent advancements in generative models for closed-loop traffic simulation have garnered significant attention. These works aim to learn realistic multi-agent interactions from real-world data. Early work TrafficSim [3] learns multi-agent interac- tions with GNNs. More recent works like BITS [4] and TrafficBots [5] learn agent behaviors with a bi-level imitation learning design, while RTR [6] combines reinforcement learning and imitation learning objectives. Most recently, works like Trajeglish [7] and SMART [8] showed the effective- ness of formulating the task as a next-token prediction problem. For this research direction, Waymo Open Sim Agent Challenge [1] provides a common evaluation methodology and benchmark. The objective of these works is to replicate real-world agent interactions without much control over the simulation process. In contrast, ProSim focuses on providing users with different ways to prompt the agents, allowing efficient creation of interesting traffic scenarios.
Controllable open-loop traffic generation. Another line of work focuses on generating traffic sce- narios with user-input controls. Early works like CTG [9] and MotionDiffuser [10] generate agent trajectories with diffusion models that imposes control signals with cost functions. CTG++[1I] re- places the gradient guidance process in CTG with cost functions generated by an LLM,, supporting text conditions as input. Different from these works that assume existing agent locations as input, SceneGen [12] and TrafficGen [13] automatically initialize agent positions conditioned on an empty map. LCTGen [14] further supports both agent placement and motion generation with text condi- tions as input. RealGen [15] incorporates retrievals of examples to generate desired scenarios given the query. While these approaches can generate realistic scenarios controlled by user inputs, they only produce fixed-length agent trajectories in open loop. These trajectory outputs do not allow agents to react to each other the tested driving policy in simulator, limiting their usefulness in

---

closed-loop simulation. In contrast, ProSim is explicitly designed for closed-loop traffic simulation, yielding natural agent interactions and complex prompt adherence.
3 Problem Formulation
In the task of traffic modeling, we are given initial traffic scene (M, A) comprised of map elements M = {vi...,, (s} and the initial states of N agents A = aN } (i.e. their sizes, types, and past trajectory). Each map element e is a polyline segmentation of lane into its centerline and edges. Agent states s; = -s.,.....s? 8; are agent positions and headings at time t.
To simulate traffic scenario for steps, the task of traffic modeling is to obtain the distribution p(St.r o). We denote rollout p($1:r(o) modeling [1] factorizes the rollout distribution autoregressively, p($1.T(o) II p(se|st:e-1, a), independently for each agent, p(s) $1:-1, o) = IT, p(s) $1=111 σ). Thus, the task of uncondi- tional traffic simulation [1] reduces to predicting p(s σ) II. II p(s) $14-1.0))
sample from this distribution. Interactive traffic
In this work, we aim to incorporate user-specified prompts = (pI...., PN. L}, which is a set of prompts for each agent in the scenario as well as an optional natural language prompt L. For agent i, the prompt Ps contains its static properties as(i] and optionally a set of user prompts ((11, wI),,,, (IIp.1:()p where x is the prompt input and w is the prompt type (e.g., goal point, action). The text prompt can include multiple sentences or be empty. Our goal to generate scenarios following user prompts, yielding the promptable closed-loop traffic simulation task:
T N
p(SITlO, = s1).
(1)
When T' is large, modeling the above equation becomes computationally hard. We thus dis- cretize T into smaller k-step chunks, and factorize the probability accordingly: p(si. ( p) =
II 11 f o, p). where T" T-k In this way, we balance computation cost and simulation accuracy by selecting different values for k.
&
4 ProSim
ProSim(rlo,p) p) is a prompt-conditioned model that produces interactive traffic rollouts from scene initialization a and a set user prompts p. ProSim has three main components: first, a scene Encoder (Section 4.1) efficiently encodes an scene initialization a to shared set of scene tokens F Encoder(ar). Secondly, a policy token Generator (Section 4.2) takes the scene tokens and agent prompts to generate policy tokens for all agents { x Generator(FF p). Thirdly, for agent i timestep f, Policy network (Section 4.3) takes the policy token n,, agent states S1:--1, and current observation F to predict the next & agent states si + = Policy (olicy(TT,,1..111 $1.-11 F). Finally, we obtain the sample rollout f ~ ProSim(o, p) by iterating between sampling s from Policy for each agent and updating the observation Sy from f to with k-step strides. We train ProSim with efficient closed-loo loop training (Section 4.4) on ProSim-Instrutt.
ProSim prioritizes three key properties. First, it is highly controllable. ProSim allows users to give a flexible combination of numerical, categorical, and textual prompts for each agent, or prompt for unconditional driving. Users may also mix and match different prompting strategies. Second, ProSim allows interactive closed-loop traffic simulation. For each agent, the Generator outputs a policy token instead of the full trajectory. This allows each agent to interact with the environment and other agents, simulating real-world reactive behaviors. Third, ProSim allows efficient closed- loop training. During rollout, ProSim only runs the heavy Encoder and Generator once, while the Policy runs for all agents in parallel at each step. This design enables to train ProSim closed-loop with full rollout of all the agents and get complete training signal through all simulation steps.
3

---

4.1 Encoder
The Encoder translates the initial scene a = A) into set of scene tokens F = F Fal, where Fm = [fm[l))---mm(ss } is the set of S map features and Fa = [fati)----,a } is the set of N agent features. To enable closed-loop inference for all agents parallel, we design to be symmetric across all agents and directly usable by the Generator and Policy modules for any agent.
We follow Shi et al. [16] and use symmetric Encoder design. We normalize the map M and actors A to their local coordinate systems, and use global positions to reason about positional relation- ships between features with position-aware attention module. The output features are then both symmetric and position-aware.
Symmetric Feature Encoding. Each map element i, is a set of polyline vertices representing lane segment. These points are originally in global coordinate system. To transform these points into an element-centered local coordinate system, we compute the geometric center and tangent direction of U; in global coordinates as Pmili and Ammjs] and transform all points in U; to the local coordinate system with [(1), /mmaa) where I is the coordinate transform function. Similarly, for each historical agent state a4(4] in A, we obtain its last-step position Paji] and heading ha(e] in the global frame. We then transform all historical states in aN(j to this local coordinate system with [(ani) Pujs], hatg)). These scene features are encoded with PointNet-like encoder [17]:2 fmit (MLLP(T<)))) fala = (MML((((ang)))) where é denotes max-pooling and f Sall] E RD are the agent-symmetric input tokens for e and a, respectively, with feature dimension D. All map and agent input tokens are concatenated together to obtain input scene tokens Fmn.
Position-aware Attention. Since each token in F is normalized its local coordinate system, vanilla MHSA cannot infer the relative positional relationship between tokens. Instead, explic- itly model the relative positions between tokens with a position-aware attention mechanism. The resulting tokens we denote as F. Please refer to the Appendix A.1 for additional details.
4.2 Generator
The policy token Generator takes scene tokens F and user prompts and outputs policy tokens for all the agents (गı,... J. contains multiple layers of information: agent static properties ax(i]. multimodal agent-centric prompts P. and a scene-level text prompt L. Therefore, we generate the policy token n. with three steps. We first obtain agent policy query [a[i] with Oz(e] and F. We then condition (a[j] with agent-centric prompts Pe to obtain prompt-come conditioned policy query 9p(])- Finally, we embed {plj and T. together into text space and query LLM output the final policy token T.. We explain these steps in details below.
Policy Query from Scenes. We use an MLP to encode the agent static properties ax(i] into an agent policy query q, E R... To model agent-agent and agent-scene interactions, we use Q = {41, qN and scene tokens F as inputs and pass them through multiple transformer layers. Each layer follows Q' MHCA (MHSA (Q), F)), where MHCA', MHSA' denote position-aware attention modules as in Section 4.1. We take last-layer outputs the policy queries Qu (9a[1)-...44[[T])
Multimodal Agent-centric Prompting. For each agent, its user-input prompt contains |p:| multi- modal prompts { ((11,W1) wi...., (I]p.) [e]]: [[)])) where x is the prompt raw input and w is the prompt type. We encode each prompt in /i into D-dimensional feature ej = Encw, (2y), where Encwy is the feature encoder for prompt type wy. We then aggregate all the prompts of the same agent pound prompt condition feature: 401l = d'((e), ..,..., }x where ø' can be any feature aggrega- tion function (max pooling, average pooling or learned self-attention module). We add this prompt condition feature into policy queries to get prompt-conditioned policy queries 4p|4 [a[j] + %6g
com-
Text Prompting with LLMs. For each scene, we have an optional user-input text prompt that con- tains multiple sentences that could describe agent behaviors, interactions, and scenario properties. To process the scene-level text prompt L and condition all the agents, we use LLM comprehend
We omit the l parameters in I' for simplicity.

---

the natural language prompt and policy queries, and generate Language-conditoone policy queries for each agent [LM)- In particular, we employ a LLaMA3-8B [18] model finetuned with LoRA [19] as backbone, with two adaptors to bridge the latent spaces of the LLM and the policy tokens. Please refer to the Figure Al and Appendix A.2 for details. Finally, we obtain each agent's poliey token x; by adding the prompt and text conditional policy queries together: n, = 4pld + q213]-
4.3 Policy
For rollout, at each timestep t each agent independently gets its next l states from Policy net- work: $ Po $1.--1, Policy first updates the dynamic tokens in F' with the new state observations $1.-11, then it outputs agent actions for the next lo steps conditioned the policy token πy. We describe these two steps below.
Dynamic Observation Update. Recall that [Fm, Fa]. where Fn is the map feature token set and Fg is the agent history token set. At timestep 0, F'm is unchanged while the agent trajectory feature F. obtained at t 0 is outdated. To update agent tokens with new observations, we encode S1:-1 with symmetric feature encoding described in Section 4.1. Specifically, for each agent i in S1æ-1. we have its history states si: 1- last-step position pi-1 and heading hi-1 in the global coordinate system. We encode it with: fi+t-1 =
-1 12--,,,;11 k)),))), where and & are the coordinate transform and feature pooling functions defined in Section 4.1. With this operation we obtain the new agent token set Fa.t = [f ..... f. /
with symmetric features.
Prompt-conditioned Action Prediction. For each agent policy token mi we let it attend to all the map and agent history tokens through multiple cross-attention layers. At each layer we have Ty = [HCA'([TT]] [m, F, where MHCA is the position-awan multi-head cross attention module mentioned in Section 4.1. We extract the last-layer output ,i, as the action prediction feature for agent at step f. Then, we
MLP to predict the state changes for the next k steps (As}, lt'st
MLP(TT,e). Finally, for each timestep l' E [1, ktj, we simply add the state changes to the previous step state: st. Ast. + s This leads the next k-step agent states s {s/ .., $ We run this process for all the agents independently in parallel, supporting efficient scenario rollout.
4.4 Training
We train ProSim in a closed-loop manner. In this section we first describe the closed-loop rollout process, and then introduce the training objectives of ProSim. We further discuss the training details of the Generator I.LM in Appendix A.3.
Closed-loop Rollout. Training a reactive traffic simulator requires strong signals of 1) how an agent's current action influences its future; 2) how each agent's 's action influences other agent's ac- tions in real time. This leads to training ProSim in closed-loop manner. Specifically, given input scene and prompt (a, we run Encoder and Generator once to get the policy tokens for all the agents. Then, at each timestep, we sample new states from Policy for all the agents indepen- dently. We update all the agent's 's states with states generated by Policy and feed them to Policy in the next iteration as new observations. Finally, when & T we obtain the full rollout & and compute the loss with the full trajectories of all agents.
The fully differentiable design of ProSim makes this theoretically possible we can directly opti- mize the loss with pack-propagation through time in simulation. We also design ProSim to make it practically feasible by only running the heavy Encoder and Generator once in the beginning, while running the
Policy autoregressively for all agents in parallel through time.
Learning Objective. We supervise the output ProSim(a, p) against GT scenario with imita- tion learning. Specifically, output contains N agent trajectories through T steps i = [6] (......... s J. We compute the imitation learning loss with Ln. 1 Σ Er d(s5, sf). where is distance N.
function (e.g., Huber) and s² is the GT state from 7. In addition to LI., we also add a collision loss Looll and offroad loss Lor encourage agents to avoid collisions and stay on drivable areas. For Looll. we compute the intersection area between each pair of agents' bounding boxes across time
5

---

according to their sizes and trajectories in f. For Lott we compute the intersection area between each agent's bounding boxes and the non-drivable area'. We denote the weight Loll and fott as A and Ag. Formally, the overall learning loss of ProSim is = LIL + A2oll + A2hont
5 ProSim-Instruct-55
To provide realistic and diverse agent motion data with multimodal prompts for promptable closed- loop traffic simulation, we propose ProSim-Instruct-222 scenario dataset with 520K real-world driving scenarios from the Waymo Open Motion Dataset (WOMD) [20]. It includes multimodal prompts for more than 10M unique agents, representing over 575 hours of driving data. For each real-world scenario, we label a comprehensive and diverse set of prompts (action tags, text descriptions, etc.) for all agents. We ensure that these descrip- tions accurately correspond to agent motions in the scenario with carefully-designed labeling tools and meticulous quality assurance by human labelers. Further, the labeling tools we developed can be directly transferred other driving data, making easy to expand the scale of our dataset. In the remainder of this section, describe our labeling process and metrics to evaluate promptable closed-loop traffic simulation. Details about quality assurance can be found in Appendix B.5.
high-quality paired prompt-
5.1 Multimodal Prompt Labeling
Given a scenario (a, T) containing N agents, we aim to obtain a set multi-modal prompts for each agent (pI...., "at....,... PN) } well as a natural language prompt for the whole scenario. For each type of prompt, carefully design labeling tool, explained below with further details in Appendix B.
Goal Point is a a 3-dimensional prompt. It allows the user to prompt agent to reach a target location with a single click on the map. For an agent in the scenario, its goal point (Ir, y, 4) represents a 2D) location (I, y) to which this agent intends to reach at time t. We extract Goal Point from GT rollouts.
Route Sketch is point-set prompt. It allows the user to draw sketch the map with the mouse route that they instruct agent to follow. Route Sketches enable the users to have fine yet convenient control of the agents' routes. We randomly subsample and add Gaussian noise to the GT trajectories and view them as Route Sketch.
Action Tag a categorical prompt. It allows the user to give high-level semantic behavior instruc- tion agent by choosing from a of action tags. Each action tag consists of an action type and its temporal duration [t,. te]. We define 8 different action types: Speed (Accelerate, Decelerate, KeepSpeed, Stopping. Parked) and Turning (LeftTurn, RightTurn, Straight). For each action type, we carefully design an automatic labeling function that outputs the duration of an agent satisfying this action in the scene given GT trajectories. After obtaining tags from all action types, we conduct a postprocessing step to smooth temporal labels and remove conflicts.
Text is natural language prompt. It allows the the scenario properties with free-form natural language. To ensure that we have diverse language expressions while keeping the motion descriptions accurate, we prompt the Llama3-70B [18] model to generate diverse sentences given structured agent behavior descriptions (Action Tags). Please refer to Appendix B.3 for details. total, the text prompts contain around 10M sentences.
describe multiple agents' behaviors and
5.2 Metrics
We measure the performance of promptable closed-loop traffic simulation via realism and control- lability. Here we implement metrics for them. Given GT data (T,a, we evaluate a model that outputs rollout given initial scene and prompts: f = Model(a,,)) p). We present formal formulation for the metrics in Appendix B.4.
We refer readers Appendix A.3 for detailed formulations of Lcall and Log.

---

Prompt
Goal Point
Route Sketch
Action Tag
Text
All w/o Text
All Types
ADE (m) % Gain
0,3882
0,5137 45.91% %
0.6718 29.26%
0.6937 26.96%
0,3635 61.72 %
0.2877
59.12%
69.71%
Table 1: Realism and controllability evaluation of ProSim.
% Gain prompt quantity
80%
Method
Composite
Kinematic
Interactive
Map-based
ConstVel
0.399
0.225
01433
01.553
60%
TraffiBBoss[5]
0.699
0,430
0.711
0.836
MIR+++(21]
0,700
0.329
0.713
(1.692
40%
VPD-Prior
0.728
MIR-E[21]
0.716
0.426
0.747
0,341
VIDD[22]
0.720
0.417
0.782
0.814
20%
Trajeglish[7]
0.735
0.479
0.783
0,820
SMART-lurge[8]
0.756
0.477
0.799
0.862
ProSim
0.718
0.401
0.778
0.822
Goal
Sketch Action
Text
All
Figure 2: Prompt quantity analysis.
Table 2: WOMD Sim Agents Challenge 2024
Realism. Following prior works [2], quantify the realism of the model by measuring the Average Displacement Error (ADE) between i and GT rollout: ADE(f, r), which is the mean 1.2 distance between agent positions and their GT across all timesteps. Lower ADE indicates better realism.
Controllability. We quantify controllability by comparing the relative improvement (% Gain) in realism of the model's output with and without prompts. We compute % Gain by comparing rollout ADE with and without prompt conditioning.
Experiments
6.1
Implementation Details
Dataset. We train ProSim on the training split of ProSim- ProSim-Instruct- Instruct-5208 which contains around 480K paired prompt-scenario data. Each scenario contains at most 128 agents, including 1.1s history and future with FPS 10 (T 80). We use k as the steps of each policy action segment Inwtim-IIntuuttttteeeeee 520k the testing set.
length. We the remaining 48K scenes in
Training. We train ProSim in two stages. In the first stage, we train ProSim without any prompt for 10 epoches with a batch size of 64. We use the Adam W optimizer with an initial learning rate of le- 3 and a CosineAnnealing scheduler with a 2500-step warm start. In the second stage, we fine-tune ProSim using types prompts. For each scenario, ensure ProSim retains unconditional rollout capacity, we randomly mask out 50% of the prompts. In this stage, we fine-tune full ProSim for epoches with batch size of 16 and learning rate of le-4. For LLM, fine-tune Llama3-8B [18] using a LoRA module with R = 16, 0.1. For the loss function, we set A 50.0, X2 5.0.
6.2 Promptable Closed-loop Traffic Simulation
Realism and Controllability. Table 1 evaluates the realism and controllability of ProSim when different types of prompts are given separately, and when all types of prompts are given together, during evaluation. To simulate real-world user inputs, for each scenario, we provide 50% of all possible prompts model input.
As can be seen, ProSim achieves high realism and controllability under all types of prompt. For example, given Goal Point as input, ProSim achieves an ADE of only 0.39m over an 8-second tra- jectory, which is 59.12% better than without prompt conditioning. Even the most high-level and abstract Text prompt leads ProSim to obtain a 26.96% Gain over the unconditional rollout. These results indicate that ProSim creates realistic traffic rollouts while closely following user prompts. Further, passing all prompt types as input achieves the best result of 0.28m ADE (69.7% better than without prompts), showing that ProSim efficiently combines and follows complex sets of multi- modal prompts.

---

(a)
(b)
(c)
(d)
A14
A14
AB Action (A2, Accelerste)
Skeechh
Skatch
Action (A32, Parkeod
acceerates then turns infi, Make <A7> and stop: <A25> forward"
Unconditional Roilout
Text: "<A17> accolorates
Figure 3: ProSim condition variety of prompt types. All agents controlled by ProSim.
Qualitative Results. We visualize the prompting capabilities of ProSim in two scenarios in Fig- ures and 3. Green agents are controlled by ProSim without prompts, while agents with other colors are controlled by the corresponding type of prompt (e.g., blue for Goal Point). The Goal Point is shown with crosses, Route Sketch with dots, while Action Tag and Text written under figures.
Visible in Figures! and 3 are ProSim's two core properties: It is promptable and closed-loop. Figure (d) in particular shows agents being prompted by a single sentence with complex high- level information, with other agents simultancously prompted by low-level goal points. Figure also demonstrates how agents react to each other closed loop. As can be seen, the unconditioned right-tuming agent (A7) in (a) changes its behavior to yield to A24 in (c). Also, the two agents following A7 brake A7 yields in (c). Please refer to our video for animated demos.
Prompt quantity analysis. Table I shows results of ProSim when we sample 50% of the prompts for evaluation. We ablate this sampling ratio in Figure 2 by setting the sampling ratio to [25%, 50%, 75%, 100%]. We show Gain of each sampling ratio from left to right for each prompt type. We can see for most prompt types there is a clear trend that more prompts lead higher realism gains. These results indicate that ProSim adapts varying number of prompts, and the behavior generation realism scales with the number of prompts.
6.3 Unconditional Rollout Evaluation
As ProSim can run without any prompts, we can compare its performance on unconditional traffic simulation on the Waymo Sim Agent Challange [1]. Table 2, we show the result of ProSim on the 2024 challenge. We compare with the SOTA method and several representative methods on the leaderboard. Overall, ProSim shows competitive performance. In particular, ProSim has a high Interactive score, showing the effectiveness of ProSim's closed-loop training. On the other hand, ProSim has a relatively low Kinematic score (measures the diversity of agents' speed and acceleration), indicating that we can further improve ProSim's output diversity during unconditional rollouts. As the focus this work is to make ProSim support promptable and closed-loop traffic simulation, we leave output diversity as future research.
7 Conclusion
We present ProSim, multimodal promptable closed-loop traffic simulation framework. Given com- plex sets of multimodal prompts from the users, ProSim simulates a traffic scenario in a closed-looy manner while instructing agents follow the prompts. ProSim shows high realism and controlla- bility given different complex user prompts. We also developed multimodal prompt-scenario paired driving dataset with over 520K scenarios and 10M+ prompts. We believe the ProSim model and dataset suite will open gates for future research on promptable human behavior simulation, within and beyond driving scenarios.
Instruct- 520k, the first
Limitations. ProSim does not yet support arbitrary prompts. Complex agent interactions (e.g., "<A0> overtakes <AI> from the left lane") or more complex modalities (e.g., prompt <AO> with its front-view image) are left future work.
8

---

Acknowledgments
We thank Yue Zhao, Vincent Cho, Jeffrey Ouyang-Zhang and Brady Zhou for their insightful dis- cussions.
References
[1] N. Montali, J. Lambert, P. Mougin, A. Kuefler, Rhinchart, M. Li, C. Gulino, Emrich, Z. Yang. Whiteson, B. A. White, and D. Anguelov. The waymo open sim agents chal- lenge. ArXiv, abs/2305.12032, 2023. URL sttp:////i.semmniiccchlllaaa hag.raiiicccooooooorrrrroooSo org/CorpusID: 258832394.
[2] Waymo LLC. Waymo open dataset: An autonomous driving dataset, 2019.
[3] S. Suo, Regalado, S. Casas, and R. Urtasun. Trafficsim: Learning to simulate realistic multi- agent behaviors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10400-10409, 2021.
[4] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone. BITS: Bi-level imitation for traffic simulation. In IEEE International Conference on Robotics and Automation (ICRA), London, UK, May 2023. URL
wtt.o///abb///////// 200bb//2228.12033 12403.
[5] Z. Zhang, A. Liniger, Dai, F. Yu, and L. Van Gool. TrafficBots: Towards world models for autonomous driving simulation and motion prediction. International Conference on Robotics and Automation (ICRA), 2023.
[6] C. Zhang. J. Tu, L. Zhang, K. Wong. S. Suo, and R. Urtasun. Learning realistic traffic agents in closed-loop. In 7th Annual Conference on Robot Learning, 2023.
[7] J. Philion, X. B. Peng, and S. Fidler. Trajeglish: Traffic modeling as next-token prediction.
The Twelfth International Conference on Learning Representations, 2024. URL https:
/ //openreview.. hwt.eeeiwwwwwweeeeeeeeeeeeeee pett/forummdddz2obbprps
[8] W. Wu, X. Feng, 7. Gao, and Y. Kan. Smart: Scalable multi-agent real-time simulation via next-token prediction, 2024.
[9] Z. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided condi- tional diffusion for controllable traffic simulation. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560-3566. IEEE, 2023.
[10] C. M. Jiang, A. Cormman, C. Park, B. Sapp. Y. Zhou, and D. Anguelov. Motiondiffuser: Controllable multi-agent motion prediction using diffusion, 2023.
[11] Z. Zhong, D. Rempe, Y. Chen, B. Ivanovic, Y. Cao, D. Xu, M. Pavone, and B. Ray. Language- guided traffic simulation via scene-level diffusion. In Conference on Robot Learning, pages 144-177. PMLR, 2023.
[12] Tan, K. Wong, Wang. Manivasagam, M. Ren, and R. Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 892-8092890999 992-901, 2021.
[13] L. Feng, Q. Li, Z. Peng, S. Tan, and B. Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. IEEE International Conference on Robotics and Automation (ICRA), 2023.
[14] S. Tan, B. Ivanovic, X. Weng, M. Pavone, and P. Krachenbuehl. Language conditioned traffic generation. In 7th Annual Conference Robot Learning (CoRL), 2023. URL https://

---

[15] W. Ding, Y. Cao. D. Zhao, C. Xiao, and M. Pavone. Realgen: Retrieval augmented generation for controllable traffic scenarios. arXiv preprint urXiv:2312.13300, 2023.
[16] Shi, I.. Jiang, D. Dai, and B. Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(5):3955-3971, May 2024. ISSN 1939-3539. doi: 1ett10.11190 10091......
2024.3352811. URL http:http://// wwtpddx.di...ogg1/./ 1109/TPAMI. 2024. 2 3352811.
[17] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point for 3d classification and segmentation. arXiv preprint arXiv:1612.00593, 2016.
[18] H. Touvron, Lavril, G. Izacard, Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, Hambro, Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023.
[19] E. J. Hu, Shen, Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation large language models, International Conference on Learning Rep- resentations, 2022. URL https://openrevieee
net//forum//dnnneWWeFYYy9
[20] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp. C. R. Qi, Y. Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710-9719, 2021.
[21] C. Qian, D. Xiu, and M. Tian. The 2nd place solution for 2023 waymo open sim agents challenge, 2023.
[22] Z. Huang, Z. Zhang, A. Vaidya, Y. Chen, C. Lv, and J. F. Fisac. Versatile scene-consistent traffic scenario generation as optimization with diffusion, 2024.
10

---

Appendix
In the appendix, we provide implementation and experiment details of our method ProSim, our dataset ProSim-Instruct-2222 as well as additional experiment results. In Section A, we present details of Encoder and Generator well the training process. In Section B, we present details of labeling, metric and quality assurance processes of our dataset ProSim-Instruce Inwt-nncttttttttt Finally, in Section C, we show additional quantitative and qualitative results of ProSim.
A ProSim
A.1 Encoder: Position-aware Attention Details
To model relationships between scene tokens and aggregate features, we pass F through multiple Transformer layers. In most existing methods, each layer follows F = MHSA(Fina ), where MHSA denotes multi-head self-attention and / is the layer index. However, since each token F is normalized to its local coordinate system, basic MHSA cannot infer the relative positional relationship between tokens. Instead, we explicitly model the relative positions between tokens with position- aware attention mechanism. For scene token i, we compute its relative positional relationship with scene token j with:
Pmali,51 = Rot(Pmalj] - Pmalf' --maliis-mm(i))) Ammaji.j]
hmaly]
hmali
where Pmale.j] E B² and lmaait E R are the relative position and heading of token j in token i's coordinate system, Rot(-) is the vector rotation function. We denote this paired relative position as Tmali.j] )ma((,j] /majt,ji] Then, we perform position-aware attention for token i with:
Smo = MHSA(Q:
[Tieat] ma][) PE(Tmof(.s))
PE(rma (t)
(3)
V: ([ + PE(r PE(rmada)))bbea)))
where PE denotes positional encoding and ??(1) is the scene token index of the neighboring tokens of In our experiments, we set (2(1) contain the nearest 32 tokens of i according to their positions. Note that the above result remains the same regardless of which global coordinate system for the scene input
(M, A). With this formulation, we model the relative position relationship between different scene tokens symmetrically. At each layer, we apply Equation 3 to all scene tokens in parallel. We denote this position-aw -ware attention module as:
F me MHSA' (F Pmmx Hma)
Note that this position-aware modification can be similarly applied to multi-headed cross-attention MHCA', ' which we will use later in the Generator and Policy modules. Finally, we obtain the last- layer token features as scene tokens
[Fms Fa).
A.2 Generator: Language Prompting Details
For each scene, we have optional user-input text prompt L that contains multiple sentences that could describe agent behaviors, interactions and scenario properties. To make it easy to refer to different agents the scene, we ask the user to use specific format "<A[i]>" when mentioning the i-th agent. For example, to instruct agent a to stop, the user would say "lel <AI> stop".
To process the levnl-level text prompt L and condition all the agents, the natural language prompt and policy features, and generate ianguage-conditionn policy features for all agents. To do this, we use a LLaMA3-8B model finetuned with LoRA as backbone, as well as two adaptors to bridge the latent spaces of LLM and policy tokens. We show an overview of our model Figure AI.
LLM comprehend
two-layer MLP adaptor to convert all the policy features to in the LLM's feature space [ta[1],..., ] MLP((g,(u).... ... (.tr))).. This operation enables the LLM to take
Specifically,
11

---

Text Tokens
Agent Token Replacement
Les
9p
and
4p12]
stop
there
Pollcy Queries
Policy
π
4p(2]
Adaptor
Llama3-8B
Adeptor
2
4p(3]
T
Figure Al: Language condition encoder for Generator.
and comprehend agent policy features in the text space. Next, LLM's text tokenizer and input embedding obtain text tokens from the natural language prompt L (tz]1))----([]] ) where O is the number of text tokens in L.
Note that some tokens L specifically mentions indexed agents (c.g., "<Al>"). To help LLM un- derstanding the correspondence between agent reference and their policy tokens, we directly replace these reference text tokens with agent policy tokens. Specifically, for an agent-reference text token [bbj) we replace it with the corresponding agent policy token (Lis] (a(i] given [ا()] corresponds to "<A[i]>" To make sure each "<A[i]>" is tokenized to single token, we add them new tokens to the text tokenizer. We call this step "Agent Token Replacement" in Figure A1.
After getting all the text and agent token features, we concatenate them to create the complete input sequence (tr,u):-..,,A1 ke[1) [a[N]] and feed it to the LLM. We then extract the hidden features for the last N tokens from the last LLM layer {Ka11 .... (inj) J. which contains text-contextualized policy features for each agent. Next, the text-conditional policy features {IL[I)= ... (L[N] MLP ((E (=
MLP adaptor to convert these features back to get
Finally, for each agent, we obtain its policy token n; by adding the prompt and text conditional policy tokens together, π, 4piz] + [2(3)-
A.3 Training
LLM Pretraining. described above, we train the L.M Generator with LoRA to compre- hend and generate policy token features. However, found that directly training the LLM with the closed-loop imitation loss leads to inferior text-prompting performance. We conjecture this is because the early training stage the LLM is not prepared interact with the policy tokens. Mean- while, the rollout loss can be decently optimized without using the LLM's output, giving little signal for LLM to learn.
To deal with this issue, we propose to pretrain the LLM have the capacity to interact with policy token features. To this end, first train ProSim model without using text prompts and the LLM with the rollout loss L. Next, add simple MLP layer after the Generator to predict the goal point of each agent given its policy token x. We supervise this task with an MSE loss L.gaal using GT goal points. This task is much simpler than the full task while enforces the LLM to interact with policy tokens. To pretrain the LLM, we fix all other modules and only train the LLM and this new MLP with Egoat- Here we only use text as the prompt in inputs. After pretraining, the LLM learss to predict the goal intention of each agent from text prompt and add that information to π, making already useful for the Policy. Finally, module with all types of prompts and the complete loss L. In experiments, pretrain the LLM on ProSim- Inie-Insttrttt 520k for epoches before using it in the full Generator module.
discard the goal-prediction MLP and train the full ProSim im
12

---

Collision loss. For collision loss Looll we aim to compute the overlapping area between each pair of agent using their bounding boxes through the full rollout trajectory. To this end, at each timestep, for each agent we compute their occupation polygon using their size, position and heading at this timestep. Then, we measure the overlapping area of each agent polygon pair by computing the signed distance between these polygons, where positive distance indicate no collision while negative distance collision indicate collision. Specifically, we compute the signed distance between polygons A and B with the distance between the origin point and the Minkowski sum A (-B). For all the signed distance, we compute the loss value by first setting the positive distance to 0 (no collision), and then taking the negative of the rest of signed distances to penalize collisions. We compute the average over all agents through all timesteps to obtain the final Leoll- We implement this loss function by referring to the Waymo Open Sim Agent Challenge collision metric.
Offread loss. For offroad loss Coll, aim compute the overlapping area of each agent and the offroad areas through the full rollout trajectory. To this end, we use the similar strategy in the collision loss. Specifically, at each timestep we obtain all the agent polygons in the same way as in the collision loss. Then, we compute the signed distance between the four corners of each polygones to a densely sampled set of road edges. Similarly, positive distance indicate no offroad while negative distance indicate being offroad. Finally, We average over all negative distances for all agents through all timesteps to obtain the final offroad loss. We implement this loss function by referring to the Waymo Open Sim Agent Challenge offroad metric.
B ProSim-Instruct-5200
B.1 Route Sketch Labeling Details
Compared with the real trajectory. points in the route sketch are sparse, noisy, and incomplete. We simulate these effects with the following steps. First, we extract each agent's complete trajectory from т. Then, we process this point set by 1) uniformly subsampling the points for sparsity; 2) adding random noise to each point: 3) randomly sampling a consecutive subset. For each agent, its route sketch a set of ordered 2D points representing sketch points on the map. The number of points route sketch could be different across agents. In our experiment, we use a uniform subsample rate of 5. We than add random noise with standard variation of 0.1 meter. Finally, ensure the randomly sampled consecutive subset contains least 5 points.
B.2 Action Tag Labeling Details
For each action type, we carefully design a heuristic function that takes an agent's trajectory Sty from step f to f2, and outputs a binary label whether St,ff, satisfies the condition of this action. Then,
this function across the full rollout with sliding window and temporal aggregation to obtain the full duration [t,, te] that this motion tag is valid. For each agent, run all the motion tags with this method we obtain set of motion tags. Finally, we post-process the labels to remove conflicting motion tags and temporal noises. Please refer to our codebase for the heuristic function implementation details upon release.
B.3 Natural Language Labeling Details
For each scenario, we provide the LLM model with agent properties (name and type) as well as all of their agent tags (action type and duration). We then prompt L.M to output 20 different sentences, each describing the agent behavior or scenario properties in natural language. To obtain interesting and diverse language description of the scenario, in the system prompt we instruct the LLM to 1) describe temporal transition of agent behavior (e.g., "Let <AI> change to the left lane and then make a left turn."); "); 2) describe scenario properties (e.g.. "This is busy scene with most agents accelerating") )); 3) describe relationships of different agents (e.g., "Let <Al>, <A2>, <A3> keep their own lanes simultaneously"). We concatenate these sentences together to form the prompt L for each scenario.
13

---

Here we show full prompt we used for the LLM labeling:
Prompt 1: Full prompt for LLM labeling
Example input:
Vehicle Agents:
[' I'<ego=', '<71flc=', '<df6al>', '<dad99>')
Pedestrian Agents:
['<a261a>', '<191e8=']
Motorcycle Agents:
['<d3ddc>', '<8cc93>', '<73c13=' 'dd69ees']
Agent to Agent:
ParallelDriving Agent (Left):<d6a9e Agent :dight);< Start:50, End:80
ByPassingRight Agent (Right, Faster, Overtaking) =<ggo>, Agent (Left, Slower, Overtaken
:<dad99>>, Start:30, End:65
Agent Behavior:
Decelerate =gent:<<3dde>, Start:0, End:5
Decelerate Agent:<ego>, Start:30, End:55
Decelerate =dent:<dad99>, Start:35, End:45
Decelerate Agent: Start:45, End:80
KeepLane Agent:<d3dde>, Start:0, End:60
KeepLane Agent:<dad99>, Start:30, End:80
KeepLane Agent:<d6a9e>, Start:49, End:80
KeepSpeed Agent:<191e8>, Start:0, End:20
KeepSpeed Agent:<ego>, Start:0, End:30
KeepSpeed Agent:<d6age>, Start:40, End:45
Parked Agent:< Start:0, End:25
Parked Agent: Start:0, End:80
RightLaneChange Agent: Start:0, End:
Straight Agent:=191e8>, Start:0, End:20
Straight Agent:<d3ddc>, Start:0, End:70
Straight Agen,:<ego>, Start:0, End:80
Straight =dad9;,, Start:30, End:80
rarigght <deate, Start:40, End:80
Stopping sdddt=< Start:5, End:70
Stopping <dad99,,, Start:45, End: 80
LeftLaneChange Agent:<ego>, Start:15, End:65
Example output:
Here the 20 commands for the simulation:
"<ego> bypasses <dad99> from the right lane side when <dad99> is driving slower finally stopping."
"Do right lane change <ego>
start "Motorcycle <d3ddc> decelerates early on."
simulation."
"After finishing the initial maneuver, redirect <ego> to occupy the left lane." "Make <dad99> and <d6a9e> to drive parallel, with <d6a9e> on the left." "<ego> slows down following the sequence of lane changes.."
"ALl vehicles, except for <ego>, <dad99>, and <d6a9e>, remain parked." "Let <ego> maintain steady speed after decelerating." "Command <d3ddc> cyclist come a complete stop after its initial slowdown." "Instruct <dad99> decelerate and then stop towards the end of the scemario." "Keep <d6a9e> in its lane after it finishes driving parallel."
"Walking person <191e8> should keep a steady pace before stopping." "<d3ddc> resumes moving straight after stopping."
"Keep all parked vehicles stationary to represent low-activity scene." "Direct <dad99>
travel straight for extended period after deceterating." straight throughout the simulation."
"Ensure <ego>
"After slowing, instruct <d6a9e> continue on direct trajectory." "Let the pedestrian <a26la> standstill in the scene. "Emphasize the limited activity within the scene, highlighting agents either stopping or staying within their lanes." "Pedestrian <191e8> walks straight."
We also show two LLM output examples for two scenarios:
14

---

Prompt 2: Text prompt labeling example 1
Here are the 20 commands for the simulation:
1. "Have <43> accelerate and make left turn throughout the entire simulation."
2. "Initially, <58> accelerates, but then slows down and comes to a stop."
3. "Keep <10> moving straight while it decelerates early on."
"Make <48> decelerate and then turn left towards the middle of the simulation." "Command <39> accelerate
drive straight after seconds." "Make <7> accelerate rapidly towards the end
simulation."
"<ego> remains parked and stationary throughout
entire scenario."
8. "Direct <11> to stay parked for the entire duration of the simulation."
9. "Most vehicles, except for a few, are parked and stationary at the start of the simulation."
10. "After 40 seconds, <14> starts moving after being parked initially."
11. "Make <4> change direction with right turn after being parked for while."
12. "Let <48> decelerate and
stop before making left turn."
13.
<58> accelerate initially, but then slow down stop."
14. "Command <45> <56> stop after 40 seconds."
15. "Instruct <32> to stop after 15 seconds and then stay stationary.""
16. "Make <15> stop and stay stationary throughout the entire simulation."
17. "Have <50>, <52>, and <58> accelerate and drive straight throughout the simulation."
18. "Keep <24> moving straight after 65 seconds."
19. "Make <39> accelerate and drive straight after being parked initially."
20. "Emphasize the dynamic nature of the scene, with agents accelerating, decelerating, and changing directions."
Prompt 3: Text prompt labeling example 2
Here the commands the simulation:
"<653> accelerates while making a left turn throughout the entire simulation."
2. "Have <654> maintain steady acceleration from start to finish."
3. "<661> gradually slows down and drives straight throughout the scemario."
4. "<666> decelerates and moves straight without any turns or stops."
5. "Keep <1465> cycling in straight line for the entire 8 seconds."
"Make <651> drive straight without any changes speed direction.*
"Ensure <659>, <660>, <662>, <664>, and <ego> remain parked and stationary throughout the simulation."
"After accelerating, have <653> continue moving in straight line."
9. "Instruct <661> to decelerate and then maintain a steady speed."
10. "Make <654> overtake <661> from the left lane.""
11. "<663> follows <661> at a steady pace, maintaining safe distance."
12. "Command <653> to merge into the lane where <661> is driving."
13. "Direct <1465> to
<651> the right side."
14. "Have <661> change lanes the left and then continue driving straight."
15. "Make <654> drive parallel to
on right side."
16. "<ego> remains stationary, observing the surrounding traffic."
17. "After accelerating, have <653> change lanes to the right."
18. "Ensure <660> and <664> remain parked, blocking the left and right lanes respectively."
19. "Inttruct <663> to decelerate and then stop behind <661>."
20. "The bicycle <1465> cycles past the parked vehicles, maintaining steady pace."
B.4 Metric Formulation
In our paper, we provide metrics of promptable closed-loop traffic simulation measure realism and controllability Here we link these metrics to the problem formulation in Section 3. Recall that we formulate promptable closed-loop traffic simulation as
N II
p(SI.Tl,, =
p str-
(5)
1=1
15

---

Metric
ADE 4
Gain f
ADE 1
Gain
ADE 4
Gain f
ADE
Gain t
Prompt
Goal Sketch
Goal Text
Sketch Action
Sketch
Text
ProSim
0.4845
48.98%
0,5983
37.00%
0,5588
41.16%
0,5698
40.00%
Prompt
Goal Action + Sketch
Goal + Action + Text
Text + Sketch + Action
All Types
ProSim
0.3635
61.72%
0.5663
40.37%
0.5311
44.08%
0.2877
69.71%
Table Al: Controllability evaluation of ProSim
Given GT data (T, o, p), realism measures the probability of the real rollout under the model distri- bution p(r|o, p). In our main paper, we implement this metric with ADE(f,
On the other hand, controllability measures how well the model follows the prompt p. We quantify this by comparing the model's realism gain against the unconditional model rollout p(T|o, p(T(a). In our main paper, we implement this metric with relative improvement (% Gain) in realism of the model's output with and without prompts. We compute % Gain by comparing rollout ADE with and without prompt conditioning. % Gain ADE(T, (IB((T7)-ADA ADB(f.T) 100%.
ADE
B.5 Quality Assurance
To ensure the prompts we generate faithfully reflect the agent behaviors in the scenario, we conduct a careful quality assurance process with human effort. As Goal Point and Route Sketch are directly modified from real trajectories, there is no need to check their accuracy. On the other hand, the accuracy of Text largely dependent on the accuracy of the Action Tag as it stems from Action Tags of all agents action types, we ask human labelers manually check whether the labeled action tag accurate both semantically and temporally by viewing the rollout videos. At each round, we ask human labelers check 100 motion tag examples for each action type. If the qualification rate of a certain action type is below 85%, we rewrite the heuristic function of this action type according to the human feedback, relabel the motion tags of this type and ask for another round of human checking. We continue this process until all the action types pass the quality threshold.
scene. Therefore, we focus on checking the quality of Action Tag. For each
We show the interface we developed for human labelers in Figure A2. This interface allow human labelers to go through and rewind the scenario easily with the interactive progress bar. For each scenario with multiple action tags, the interface let the labeler to through all the action tags all together. This allows the human labeler to QA multiple motion tags of the same scenario very efficiently. In average, we found human labelers take around 10 seconds to give the QA output for each motion tag. Additionally, we ask human labelers to give QA output for each tag, choosen from Correct, Wrong Action, Wrong Time, Wrong Agent, and Need Attention (unsure or other types). These different types of QA error tags provide useful feedback improve heuristic functions.
C Additional results
C.1 Controllability analysis
In our paper, we show benchmark results of 5 different prompt combinations. Aside from these combinations shown in the paper, we also show the Gain results of other prompt combinations in Table A1. We can see from the Table Al that ProSim achieves consistent gain with different kinds of prompt modality combinations. These results show that ProSim allows users to freely combine different prompt modalities with high controllability.
16

---

Agent D. (rahille) at 1392 66 in (157/02 heading -2.33 (-13.31 deg)
Unknown
Vehicle
20
ac76a
Pedestrian
c9341
33
153317777777
Bicycle
Motorcycle
Past Molion
50b4f
Future Motion
Lane Center
8414c
14215
2007
19247
Road Area
Road Lanes
99071
9789d
Crosswalks
Sidewalks
-00
-00 Bimmecape (1.00 into the Jutune:
40
Trrwetpe to Render o.. 40
Fletyon: sep4
Moten
QA:
Follningg pomp se ca 1133 Miccelded eggn 51-21
Connet
Follwing wwwppeeee-lteeeeeeeeoo 11w.00000011100000000000 www.:1000000tc00ooooooo
Figure A2: Interface used by human labeler for Quality Assurance.
Sample batch size
4
8
16
Unconditional
248 64
79.89
53.08
43.09
All Prompting w/o Text
266.31
82.65
54.82
44.02
All Prompting with Text
365.98
105.94
66.54
49.99
Table A2: Average inference time of ProSim in milliseconds for scenario with 64 agents given different sample batch sizes.
C.2 Runtime analysis
In this section, provide thorough rollout time analysis of ProSim. Specifically, single NVIDIA A100 GPU to measure the runtime of ProSim under three different settings: I) sample batch size; 2) number of simulated agents per scenario and 3) number of prompts used per scenario. In each experiment, ProSim is tasked with generating 8-second long trajectories for to 128 simulated agents per scenario.
# Simulated agents
16
32
64
128
Unconditional
239.51
240.76
247.69
267.06
All Prompting w/o Text
249.31
251.56
262,54
283,54
All Prompting with Text
324.71
332.43
353.52
394.34
Table A3: Average inference time ProSim in milliseconds for scenario with batch size given different numbers of simulated agents.
17

---

Quantity of prompts
25%
50%
75%
100%
All Prompting w/o Text
258.01
258.80 858866
260.67
259.96
Text Prompting Only
327.53
341.97
346.39
352.99
All Prompting with Text
337.94
355.03
363.55
372.58
Table A4: Average inference time of ProSim in milliseconds for scenario with batch size 1 and 64 agents given different prompt quantities.
Firstly, in Table A2 conduct ablation study on inference batch size (number of scenarios per batch). prompt 50% of all agents random (also randomly selecting prompt type, i... goal, sketch, action, text). This table shows that ProSim is able to achieve very high inference efficiency, especially when we use batched scenario inference. Notably, when sample batch size is 16, ProSim can simulate an 8-second scenario with 64 agents and all kinds of prompts (including text) within 50ms on average. This table shows that ProSim can be highly efficient with parallel computation.
Next, in Table A3 do ablation study on the number of simulated agents per sceanrio, where set batch size equal to 1. This table shows ProSim is able to easily scale to large number of agents (up to 128) with minimal increases in inference time. Specifically, inference time only grows by 12% from 16 agents to 128 agents. Finally, in Table A4, we vary the number of prompts used per scenario. This table shows ProSim can maintain efficiency even with large number of prompts.
We achieve the high efficiency of ProSim with careful designs. Firstly, ProSim only run Encoder and Generator
per scenario, regardless of the number of agents in the scenario. For Encoder, the main bottleneck is from the many scene tokens, so agent tokens don't 't't affect runtime much. For the Generator, the agent-centric prompt encoders are very lightweight and not problem. As for the LLM module, the main bottleneck again is from text tokens, not agent tokens. Overall, the runtime of the Encoder and Generator doesn't 't increase much with more agents.
On the other hand, ProSim the Policy for all agents in parallel. We make this possible by letting each agent act individually given only its scene and policy tokens (Section 4.3). This design not only realistically reflects how agents interact in the real world, but also allows for batched policy inference at each timestep for all agents (even across different scenarios). For example, given M scenarios, each with N simulated agents, at each timestep we conduct M x N policy inferences parallel within a single batch. This operation allows us to achieve very high inference throughput on GPUs.
C.3 Conflicting prompting analysis
Recall that when prompt one agent with prompts with multiple modalities, we always provide prompts with the similar semantics, this is the expected way users will use ProSim for. We also train ProSim with this manner. However, it is interesting to see what will happen if for single agent we provide multiple prompts with different or conflicting semantics simultancously. This experiment could show which types of prompt ProSim is more sensitive to.
Therefore, in this section, we investigate ProSim's 's behavior if multiple prompts with different se- mantics given simultaneously single agent. We provide case study showing how ProSim deals with prompts with different semantics in Figure A3.
In this experiment, we prompt agent A8 with four different pnompts,one modality at a time, as in Figure A3 (1)-(4) Note how ProSim able to make A8 follow different modalities of prompts closely. To investigate the case when we provide multiple prompts with different semantics simulta- neously, in (5)-(12) we prompt agent A8 with 2-3 prompts with conflicting semantics and modalities together. For example, in (7) prompt A8 with Goal point to turn left and with Text turn right. Here observations:
18

---

Goal prompts dominate over other prompt types. Whenever Goal prompt is provided, agents will try to follow it first and tend to ignore other prompt types, as can be seen from (5) and (7).
Similar to Goal, Sketch also dominates over Action and Text, as shown in (8) and (9). Note that in (6) when Sketch and Goal appear together, the agent does not strietly follow Goal, but rather tries interpolate between the Goal and Sketch points. This indicates that Sketch has similar controllability as Goal.
Action and Text provide more abstract instructions and are therefore dominated by more explicit Goal and Sketch prompts. When Action and Text appear together in (10). we can the agent also tries to interpolate between Action and Text (turning right but also stopping, mix between (3) and (4)). This also indicates that Text has similar controllability as Action.
(1)
(4)
Goal- AB
Sketch AB
Action - (AB, Stoppingt
Text: "Let <AB> accelorato and turn right"
(5)
(6)
(8)
Goal AB
Goal All
Goal AB
Sketch AB
Action - (AB, Shopping)
Sleotch- AB
Text: "Let <AB> accelerate and turn right"
Action (AB, Stopping)
(10)
(11)
(12)
Sketthh AB
Action (AB, Stoppingt
Goal AB Action (AB, Stopping)
Sketch AB Action (AB, Stoppingl
Text:: "Lat <AB> accelerate
Text: "Le <AB> accelenate
Text: "Let <AB> acce/erate
Text: "Let KAB> accelerate
right
right"
right"
right"
Figure A3: Prompting the same agent with prompts with different semantics. All agents are trolled by ProSim. Green agents are unconditional agents. On the first row, in each column agent A8 is prompted by single prompt modality that has distinct semantics. On the last two rows, agent A8 is simultaneously prompted by more than prompt modalities with conflicting semanties.
Overall, found that, in terms of controllability, when multiple semantically-conficcii prompts appear, Goal > Sketch > Action ~ Text. This rank resonates with the controllability scores in Ta- ble 1. We conjecture that this is because Goal and Sketch prompts contain more explicit information while Action and Text prompts are more abstract. Therefore, during training, ProSim learns to focus more on Goal and Sketch prompts more (which helps it reduce losses more effectively).
19


Reviews:
[{"review_id": "wpaVxAVw8y", "reviewer": "Reviewer_M9no", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "ProSim is a first-of-its-kind promptable closed-loop traffic simulation framework. The ProSim-Instruct-520k, a large-scale multimodal prompt-scenario driving dataset, is the first driving dataset with semantic-rich agent motion labels and text captions.", "summary_of_recommendation": "ProSim is a first-of-its-kind promptable closed-loop traffic simulation.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "How to make the representation alignment (encoder) for the LLM backbone to understand the complex map and agent information without QAs?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M593bLmFzl", "reviewer": "Reviewer_Vo5Q", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces ProSim, a promptable closed-loop traffic simulation framework that effectively models interactivity while maintaining realism and controllability. The paper is well-written and easy to follow. However, the prompting method can be seen as a straightforward extension of common multi-agent prediction or simulation frameworks. The improvement in ADE results with the proposed prompts, such as GT goal points, is expected. Deeper analysis and more results on long-tail scenarios are desired.", "summary_of_recommendation": "This paper propose a promptable closed-loop traffic simulation framework. The content is complete, but the novelty is limited.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "Please refer to cons.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZsOE6VPts", "reviewer": "Reviewer_w2z7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "A closed-loop traffic simulation framework supporting multiple prompts, including coordinates and text. A hierarchal motion generator is used. New dataset proposed.", "summary_of_recommendation": "The paper is sound and well-written, with sufficient novelty and contribution to the community.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 1, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Are the LORA of LLM trained together with the whole pipeline in closed-loop manner?\n2. Is it possible to use text prompt to prompt the behaviors of multiple agents? I think it's possible. Have you every tested this: as the number of agents being prompted increase, the controllability saturates. \n3. In limitation you said complex agent interaction is not promptable right now. Can I understand this as the autolabeling mechanism when you creating the dataset limits the diversity of text prompt. The text prompt is actually the action tag in text format. As in Figure 3, we can see the text prompts are basically \"some agents make a left turn / move forward\". So the so call text prompt is not that useful as you claim.\n4. As I said in Weakness 1, why the WOSAC result is not presented properly?\n5. In MTR++, they use 64 queries to increase the diversity of the output. I wonder whether the \"unconditional generation\", and as well as the WOSAC result, can be improved if we discover say 64 \"common prompts\". (Note that 64 is for motion prediction, in WOSAC the required modes are 32).\n6. How many agent are simulated?\n7. In WOSAC they require you to simulate all 128 agents. How you address the scalability issue if you are simulating 128 agents?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- Novel promptable closed-loop traffic simulation framework supporting multiple prompt types (coordinates, text, etc.)
- Creation of a large-scale multimodal prompt-scenario driving dataset (ProSim-Instruct-520k) 
- Well-written paper that is easy to follow
- Competitive performance on Waymo Sim Agents Challenge, with high prompt controllability

Weaknesses:
- Limited analysis of prompt effects on agent interactions and corner cases
- Scalability concerns as number of agents increases
- Questions about usefulness/diversity of text prompts beyond basic action tags
- Lack of proper SOTA baselines and composite metric results for Waymo Open Sim Agents Challenge
Decision: Accept