Promptable Closed-loop Traffic Simulation

Authors: shuhan  tan
boris  ivanovic
yuxiao  chen
boyi  li
xinshuo  weng
yulong  cao
philipp  krähenbühl
marco  pavone
u t austin
  nvidia
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-02 04:27:41

Abstract: Simulation stands as a cornerstone for safe and efficient autonomous driving development. At its core a simulation system ought to produce realistic, reactive, and controllable traffic patterns. In this paper, we propose ProSim, a multimodal promptable closed-loop traffic simulation framework. ProSim allows the user to give a complex set of numerical, categorical or textual prompts to instruct each agent’s behavior and intention. ProSim then rolls out a traffic scenario in a closed-loop manner, modeling each agent’s interaction with other traffic participants. Our experiments show that ProSim achieves high prompt controllability given different user prompts, while reaching competitive performance on the Waymo Sim Agents Challenge when no prompt is given. To support research on promptable traffic simulation, we create ProSim-Instruct-520k, a multimodal prompt-scenario paired driving dataset with over 10M text prompts for over 520k real-world driving scenarios. We will release data, benchmark, and labeling tools of ProSim-Instruct-520k upon publication.

Main paper: 
 introduction: 
 Simulation provides an efficient way to evaluate AV systems in realistic environments. It avoids the high costs and risks associated with real-world testing, and forms the first step of any deployment cycle of an autonomous driving policy. A simulator is only able to fulfill its role if it is capable of mimicking real-world driving conditions. Chief among these conditions is highly-realistic simulation of interactive traffic agents and their behavior patterns. Further, the simulation must be controllable to enable the creation of interesting traffic scenarios by editing and customizing each agent's behaviors and intentions via user-friendly inputs or prompts.To capture these needs, we propose a new task: promptable closed-loop traffic simulation (Section 3). Aside from simulating realistic traffic agent interactions in closed-loop, traffic models should also generate agent motions that satisfy a complex set of user-specified prompts. Our baseline algorithm ProSim (Figure 1 ) takes a scene initialization and multimodal prompts as inputs, and produces a set of agent policies that yield closed-loop, reactive scenario rollouts. As we show in Section 6, ProSim achieves high realism and controllability with a variety complex user prompts. ProSim even achieves competitive performance on the Waymo Open Sim Agent Challenge [1] without prompts.To further enable research into promptable closed-loop traffic simulation task, we created ProSim-Instruct-520k, a multimodal paired prompt-scenario dataset (Section 5). Leveraging the real-world AV dataset [2] , we provide a rich set of prompts for all agents. Specifically, for each scenario we label agents' Goal Points (their destinations), Action Tag (e.g., Accelerate, LeftTurn), Route Sketch (a noisy sketch of an agent's route), and Text instructions (e.g., "Instruct <A0> to decelerate before turning right."), visualized in Figure 1 . These prompts simulate some of the diverse ways users may wish to prompt scenarios. We ensure the labeled prompts accurately depict the agent's ground-truth (GT) behavior with careful human quality assurance stages. ProSim-Instruct-520k contains 520K prompt-scenario pairs of real-world driving data, including over 10M sentences of text prompts for 10M unique agents and 575 hours of driving. We will release the data, a benchmark, and labeling tools of ProSim-Instruct-520k to facilitate future research upon publication.Our key contributions are threefold:1. We introduce ProSim, a first-of-its-kind promptable closed-loop traffic simulation framework.2. We create ProSim-Instruct-520k, a large-scale multimodal prompt-scenario driving dataset. It is the first driving dataset with semantic-rich agent motion labels and text captions. 3. We will release code and checkpoints of ProSim. We will also release data, a benchmark, and the labeling tools of ProSim-Instruct-520k to facilitate agent motion simulation research. 
related work: 
 Unconditional closed-loop traffic simulation. Recent advancements in generative models for closed-loop traffic simulation have garnered significant attention. These works aim to learn realistic multi-agent interactions from real-world data. Early work TrafficSim [3] learns multi-agent interactions with GNNs. More recent works like BITS [4] and TrafficBots [5] learn agent behaviors with a bi-level imitation learning design, while RTR [6] combines reinforcement learning and imitation learning objectives. Most recently, works like Trajeglish [7] and SMART [8] showed the effectiveness of formulating the task as a next-token prediction problem. For this research direction, Waymo Open Sim Agent Challenge [1] provides a common evaluation methodology and benchmark. The objective of these works is to replicate real-world agent interactions without much control over the simulation process. In contrast, ProSim focuses on providing users with different ways to prompt the agents, allowing efficient creation of interesting traffic scenarios.Controllable open-loop traffic generation. Another line of work focuses on generating traffic scenarios with user-input controls. Early works like CTG [9] and MotionDiffuser [10] generate agent trajectories with diffusion models that imposes control signals with cost functions. CTG++ [11] replaces the gradient guidance process in CTG with cost functions generated by an LLM, supporting text conditions as input. Different from these works that assume existing agent locations as input, SceneGen [12] and TrafficGen [13] automatically initialize agent positions conditioned on an empty map. LCTGen [14] further supports both agent placement and motion generation with text conditions as input. RealGen [15] incorporates retrievals of examples to generate desired scenarios given the query. While these approaches can generate realistic scenarios controlled by user inputs, they only produce fixed-length agent trajectories in open loop. These trajectory outputs do not allow agents to react to each other or the tested driving policy in a simulator, limiting their usefulness in closed-loop simulation. In contrast, ProSim is explicitly designed for closed-loop traffic simulation, yielding natural agent interactions and complex prompt adherence. 
problem formulation: 
 In the task of traffic modeling, we are given an initial traffic scene σ = (M, A) comprised of map elements M = {v 1 , ..., v S } and the initial states of N agents A = {a 1 , ..., a N } (i.e., their sizes, types, and past trajectory). Each map element v is a polyline segmentation of a lane into its centerline and edges. Agent states s t = {s 1 t , ..., s N t } are agent positions and headings at time t. To simulate a traffic scenario for T steps, the task of traffic modeling is to obtain the distribution p(s 1:T |σ). We denote rollout τ ∼ p(s 1:T |σ) as a sample from this distribution. Interactive traffic modeling [1] factorizes the rollout distribution autoregressively, p(s1:T |σ) = T t=1 p(s t |s 1:t-1 , σ), independently for each agent, p(s t |s 1:t-1 , σ) = N i=1 p(s i t |s 1:t-1 , σ).Thus, the task of unconditional traffic simulation [1] reduces to predicting p(s1:T |σ) = T t=1 N i=1 p(s i t |s 1:t-1 , σ).In this work, we aim to incorporate user-specified prompts ρ = {ρ 1 , ..., ρ N , L}, which is a set of prompts for each agent in the scenario as well as an optional natural language prompt L. For agent i, the prompt ρ i contains its static properties a s[i] and optionally a set of user prompts {(x 1 , ω 1 ), ..., (x |ρi| , ω |ρi| )}, where x is the prompt input and ω is the prompt type (e.g., goal point, action). The text prompt L can include multiple sentences or be empty. Our goal is to generate scenarios following user prompts, yielding the promptable closed-loop traffic simulation task:EQUATIONWhen T is large, modeling the above equation becomes computationally hard. We thus discretize T into smaller k-step chunks, and factorize the probability accordingly: p(s1:T |σ, ρ) = T ′ t=1 N i=1 p(s i t•k:t•k+k |s 1:t•k-1 , σ, ρ), where T ′ = T -k k .In this way, we balance computation cost and simulation accuracy by selecting different values for k. 
prosim: 
 ProSim(τ |σ, ρ) is a prompt-conditioned model that produces interactive traffic rollouts τ from a scene initialization σ and a set of user prompts ρ. ProSim has three main components: first, a scene Encoder (Section 4.1) efficiently encodes an scene initialization σ to a shared set of scene tokens F = Encoder(σ). Secondly, a policy token Generator (Section 4.2) takes the scene tokens and agent prompts to generate policy tokens for all agents {π 1 , ..., π N } = Generator(F, ρ). Thirdly, for agent i at timestep t, a Policy network (Section 4.3) takes the policy token π i , agent states s 1:t-1 , and current observation F to predict the next k agent states s i t:t+k = Policy(π i , s 1:t-1 , F ). Finally, we obtain the sample rollout τ ∼ ProSim(σ, ρ) by iterating between sampling s i t:t+k from Policy for each agent and updating the observation s t from t = 1 to T with k-step strides. We train ProSim with efficient closed-loop training (Section 4.4) on ProSim-Instruct-520k.ProSim prioritizes three key properties. First, it is highly controllable. ProSim allows users to give a flexible combination of numerical, categorical, and textual prompts for each agent, or no prompt for unconditional driving. Users may also mix and match different prompting strategies. Second, ProSim allows interactive closed-loop traffic simulation. For each agent, the Generator outputs a policy token instead of the full trajectory. This allows each agent to interact with the environment and other agents, simulating real-world reactive behaviors. Third, ProSim allows efficient closedloop training. During rollout, ProSim only runs the heavy Encoder and Generator once, while the Policy runs for all agents in parallel at each step. This design enables us to train ProSim closed-loop with full rollout of all the agents and get a complete training signal through all simulation steps.The Encoder translates the initial scene σ = (M, A) into a set of scene tokens F = [F m , F a ], where F m = {f m[1] , ..., f m[S] } is the set of S map features and F a = {f a [1] , ..., f a[N ] } is the set of N agent features. To enable closed-loop inference for all agents in parallel, we design F to be symmetric across all agents and directly usable by the Generator and Policy modules for any agent.We follow Shi et al. [16] and use a symmetric Encoder design. We normalize the map M and actors A to their local coordinate systems, and use global positions to reason about positional relationships between features with a position-aware attention module. The output features are then both symmetric and position-aware. Symmetric Feature Encoding. Each map element v i is a set of polyline vertices representing a lane segment. These points are originally in a global coordinate system. To transform these points into an element-centered local coordinate system, we compute the geometric center and tangent direction of v i in global coordinates as p m[i] and h m[i] and transform all points in v i to the local coordinate system withΓ(v i , p m[i] , h m[i]), where Γ is the coordinate transform function. Similarly, for each historical agent state a h[i] in A, we obtain its last-step position p a[i] and heading h a[i] in the global frame. We then transform all historical states in a h[i] to this local coordinate system withΓ(a h[i] , p a[i] , h a[i]). These scene features are encoded with a PointNet-like encoder [17] :2 f 0 m[i] = ϕ(MLP(Γ(v i ))), f 0 a[i] = ϕ(MLP(Γ(a h[i] ))), where ϕ denotes max-pooling and f 0 m[i] , f 0 a[i] ∈ R D are the agent-symmetric input tokens for v i and a i , respectively, with feature dimension D. All map and agent input tokens are concatenated together to obtain input scene tokens F 0 ma . Position-aware Attention. Since each token in F 0 ma is normalized to its local coordinate system, vanilla MHSA cannot infer the relative positional relationship between tokens. Instead, we explicitly model the relative positions between tokens with a position-aware attention mechanism. The resulting tokens we denote as F . Please refer to the Appendix A.1 for additional details.The policy token Generator takes scene tokens F and user prompts ρ and outputs policy tokens for all the agents {π 1 , ..., π N }. ρ contains multiple layers of information: agent static properties a s[i] , multimodal agent-centric prompts ρ i and a scene-level text prompt L. Therefore, we generate the policy token π i with three steps. We first obtain agent policy query q a[i] with a s[i] and F . We then condition q a[i] with agent-centric prompts ρ i to obtain prompt-conditioned policy query q ρ[i] . Finally, we embed q ρ[i] and L together into text space and query an LLM to output the final policy token π i . We explain these steps in details below.Policy Query from Scenes. We use an MLP to encode the agent static properties a s[i] into an agent policy query q i ∈ R D . To model agent-agent and agent-scene interactions, we use Q = {q 1 , ..., q N } and scene tokens F as inputs and pass them through multiple transformer layers. Each layer followsQ ′ = MHCA'(MHSA'(Q), F )), where MHCA', MHSA' denote position-aware attention modules as in Section 4.1. We take last-layer outputs as the policy queriesQ u = {q a[1] , ..., q a[N ] }.Multimodal Agent-centric Prompting. For each agent, its user-input prompt contains |ρ i | multimodal prompts {(x 1 , ω 1 ), ..., (x |ρi| , ω |ρi| )}, where x is the prompt raw input and ω is the prompt type. We encode each prompt in ρ i into a D-dimensional feature e j = Enc ωj (x j ), where Enc ωj is the feature encoder for prompt type ω j . We then aggregate all the prompts of the same agent to a compound prompt condition feature:q ′ ρ[i] = ϕ ′ ({e 1 , e 2 , . . . , e k }), where ϕ ′ can be any feature aggregation function (max pooling, average pooling or a learned self-attention module). We add this prompt condition feature into policy queries to get prompt-conditioned policy queries q ρ[i] = q a[i] + q ′ ρ[i] .Text Prompting with LLMs. For each scene, we have an optional user-input text prompt L that contains multiple sentences that could describe agent behaviors, interactions, and scenario properties. To process the scene-level text prompt L and condition all the agents, we use an LLM to comprehend the natural language prompt and policy queries, and generate language-conditioned policy queries for each agent q L[i] . In particular, we employ a LLaMA3-8B [18] model finetuned with LoRA [19] as backbone, with two adaptors to bridge the latent spaces of the LLM and the policy tokens. Please refer to the Figure A1 and Appendix A.2 for details. Finally, we obtain each agent's policy token π i by adding the prompt and text conditional policy queries together:π i = q ρ[i] + q L[i] .For rollout, at each timestep t each agent i independently gets its next k states from a Policy network: s i t:t+k = Policy(π i , s 1:t-1 , F ). Policy first updates the dynamic tokens in F with the new state observations s 1:t-1 , then it outputs agent actions for the next k steps conditioned the policy token π i . We describe these two steps below.Dynamic Observation Update. Recall that F = [F m , F a ],where F m is the map feature token set and F a is the agent history token set. At timestep t > 0, F m is unchanged while the agent trajectory feature F a obtained at t = 0 is outdated. To update agent tokens with new observations, we encode s 1:t-1 with symmetric feature encoding described in Section 4.1. Specifically, for each agent i in s 1:t-1 , we have its history states s i 1:t-1 , last-step position p i t-1 and heading h i t-1 in the global coordinate system. We encode it with:f i 1:t-1 = ϕ(MLP(Γ(s i 1:t-1 , p i t-1 , h i t-1 ))), where Γ and ϕ are the coordinate transform and feature pooling functions defined in Section 4.1. With this operation we obtain the new agent token set F a,t = {f 1 1:t-1 , ..., f N 1:t-1 } with symmetric features. Prompt-conditioned Action Prediction. For each agent policy token π i , we let it attend to all the map and agent history tokens through multiple cross-attention layers. At each layer we haveπ ′ i = MHCA ′ ([π i ], {F m , F a,t }),where MHCA is the position-aware multi-head cross attention module mentioned in Section 4.1. We extract the last-layer output π * i,t as the action prediction feature for agent i at step t. Then, we use an MLP to predict the state changes for the next k steps{∆s i t ′ } t+k t ′ =t = MLP(π * i,t ). Finally, for each timestep t ′ ∈ [t, t + k],we simply add the state changes to the previous step state:s i t ′ = ∆s i t ′ + s i t ′ -1. This leads to the next k-step agent states s i t:t+k = {s i t , ..., s i t+k }. We run this process for all the agents independently in parallel, supporting efficient scenario rollout.We train ProSim in a closed-loop manner. In this section we first describe the closed-loop rollout process, and then introduce the training objectives of ProSim. We further discuss the training details of the Generator LLM in Appendix A.3.Closed-loop Rollout. Training a reactive traffic simulator requires strong signals of 1) how an agent's current action influences its future; 2) how each agent's action influences other agent's actions in real time. This leads us to training ProSim in a closed-loop manner. Specifically, given input scene and prompt (σ, ρ), we run Encoder and Generator once to get the policy tokens for all the agents. Then, at each timestep, we sample new states from Policy for all the agents independently. We update all the agent's states with states generated by Policy and feed them to Policy in the next iteration as new observations. Finally, when t = T we obtain the full rollout τ and compute the loss with the full trajectories of all agents.The fully differentiable design of ProSim makes this theoretically possible as we can directly optimize the loss with back-propagation through time in simulation. We also design ProSim to make it practically feasible by only running the heavy Encoder and Generator once in the beginning, while running the light-weight Policy autoregressively for all agents in parallel through time.Learning Objective. We supervise the output τ = ProSim(σ, ρ) against GT scenario τ with imitation learning. Specifically, an output contains N agent trajectories through T steps τ = {ŝ 1 1 , ..., ŝN T }. We compute the imitation learning loss withL IL = 1 N •T N i=1 T t=1 d(ŝ t i , s t i ), where d is a distance function (e.g., Huber) and s t i is the GT state from τ . In addition to L IL , we also add a collision loss L coll and offroad loss L off to encourage agents to avoid collisions and stay on drivable areas. For L coll , we compute the intersection area between each pair of agents' bounding boxes across time according to their sizes and trajectories in τ . For L off we compute the intersection area between each agent's bounding boxes and the non-drivable area 3 . We denote the weight of L coll and L off as λ 1 and λ 2 . Formally, the overall learning loss of ProSim isL = L IL + λ 1 L coll + λ 2 L off . 
prosim-instruct-520k: 
 To provide realistic and diverse agent motion data with multimodal prompts for promptable closedloop traffic simulation, we propose ProSim-Instruct-520k: a high-quality paired promptscenario dataset with 520K real-world driving scenarios from the Waymo Open Motion Dataset (WOMD) [20] . It includes multimodal prompts for more than 10M unique agents, representing over 575 hours of driving data. For each real-world scenario, we label a comprehensive and diverse set of prompts (action tags, text descriptions, etc.) for all agents. We ensure that these descriptions accurately correspond to agent motions in the scenario with carefully-designed labeling tools and meticulous quality assurance by human labelers. Further, the labeling tools we developed can be directly transferred to other driving data, making it easy to expand the scale of our dataset. In the remainder of this section, we describe our labeling process and metrics to evaluate promptable closed-loop traffic simulation. Details about quality assurance can be found in Appendix B.5.Given a GT scenario (σ, τ ) containing N agents, we aim to obtain a set of multi-modal prompts for each agent {ρ 1 , ..., ρ N } as well as a natural language prompt L for the whole scenario. For each type of prompt, we carefully design a labeling tool, explained below with further details in Appendix B.Goal Point is a 3-dimensional prompt. It allows the user to prompt an agent to reach a target location with a single click on the map. For an agent in the scenario, its goal point (x, y, t) represents a 2D location (x, y) to which this agent intends to reach at time t. We extract Goal Point from GT rollouts.Route Sketch is a point-set prompt. It allows the user to draw a sketch on the map with the mouse as a route that they instruct an agent to follow. Route Sketches enable the users to have fine yet convenient control of the agents' routes. We randomly subsample and add Gaussian noise to the GT trajectories and view them as Route Sketch.Action Tag is a categorical prompt. It allows the user to give high-level semantic behavior instruction to an agent by choosing from a set of action tags. Each action tag consists of an action type and its temporal duration [t s , t e ]. We define 8 different action types: Speed (Accelerate, Decelerate, KeepSpeed, Stopping, Parked) and Turning (LeftTurn, RightTurn, Straight). For each action type, we carefully design an automatic labeling function that outputs the duration of an agent satisfying this action in the scene given GT trajectories. After obtaining tags from all action types, we conduct a postprocessing step to smooth temporal labels and remove conflicts.Text is a natural language prompt. It allows the user to describe multiple agents' behaviors and the scenario properties with free-form natural language. To ensure that we have diverse language expressions while keeping the motion descriptions accurate, we prompt the Llama3-70B [18] model to generate diverse sentences given structured agent behavior descriptions (Action Tags). Please refer to Appendix B.3 for details. In total, the text prompts contain around 10M sentences.We measure the performance of promptable closed-loop traffic simulation via realism and controllability. Here we implement metrics for them. Given GT data (τ, σ, ρ), we evaluate a model that outputs a rollout given initial scene and prompts: τ = Model(σ, ρ). We present a formal formulation for the metrics in Appendix B.4. Realism. Following prior works [2] , we quantify the realism of the model by measuring the Average Displacement Error (ADE) between τ and GT rollout: ADE(τ , τ ), which is the mean L2 distance between agent positions and their GT across all timesteps. Lower ADE indicates better realism.Controllability. We quantify controllability by comparing the relative improvement (% Gain) in realism of the model's output with and without prompts. We compute % Gain by comparing rollout ADE with and without prompt conditioning.6 Experiments 
implementation details: 
 Dataset. We train ProSim on the training split of ProSim-Instruct-520k, which contains around 480K paired prompt-scenario data. Each scenario contains at most 128 agents, including 1.1s history and 8s future with FPS = 10 (T = 80). We use k = 10 as the steps of each policy action segment length. We use the remaining 48K scenes in ProSim-Instruct-520k as the testing set.Training. We train ProSim in two stages. In the first stage, we train ProSim without any prompt for 10 epoches with a batch size of 64. We use the AdamW optimizer with an initial learning rate of 1e-3 and a CosineAnnealing scheduler with a 2500-step warm start. In the second stage, we fine-tune ProSim using all types of prompts. For each scenario, to ensure ProSim retains unconditional rollout capacity, we randomly mask out 50% of the prompts. In this stage, we fine-tune full ProSim for 5 epoches with a batch size of 16 and a learning rate of 1e-4. For LLM, we fine-tune Llama3-8B [18] using a LoRA module with R = 16, α = 0.1. For the loss function, we set λ 1 = 50.0, λ 2 = 5.0.Realism and Controllability. Table 1 evaluates the realism and controllability of ProSim when different types of prompts are given separately, and when all types of prompts are given together, during evaluation. To simulate real-world user inputs, for each scenario, we provide 50% of all possible prompts as model input.As can be seen, ProSim achieves high realism and controllability under all types of prompt. For example, given Goal Point as input, ProSim achieves an ADE of only 0.39m over an 8-second trajectory, which is 59.12% better than without prompt conditioning. Even the most high-level and abstract Text prompt leads ProSim to obtain a 26.96% Gain over the unconditional rollout. These results indicate that ProSim creates realistic traffic rollouts while closely following user prompts. Further, passing all prompt types as input achieves the best result of 0.28m ADE (69.7% better than without prompts), showing that ProSim efficiently combines and follows complex sets of multimodal prompts. As can be seen, the unconditioned right-turning agent (A7) in (a) changes its behavior to yield to A24 in (c). Also, the two agents following A7 brake as A7 yields in (c). Please refer to our video for animated demos.Prompt quantity analysis. Table 1 shows results of ProSim when we sample 50% of the prompts for evaluation. We ablate this sampling ratio in Figure 2 by setting the sampling ratio to [25%, 50%, 75%, 100%]. We show % Gain of each sampling ratio from left to right for each prompt type. We can see for most prompt types there is a clear trend that more prompts lead to higher realism gains. These results indicate that ProSim adapts to varying number of prompts, and the behavior generation realism scales with the number of prompts.As ProSim can run without any prompts, we can compare its performance on unconditional traffic simulation on the Waymo Sim Agent Challange [1] . In Table 2 , we show the result of ProSim on the 2024 challenge. We compare with the SOTA method and several representative methods on the leaderboard. Overall, ProSim shows competitive performance. In particular, ProSim has a high Interactive score, showing the effectiveness of ProSim's closed-loop training. On the other hand, ProSim has a relatively low Kinematic score (measures the diversity of agents' speed and acceleration), indicating that we can further improve ProSim's output diversity during unconditional rollouts. As the focus of this work is to make ProSim support promptable and closed-loop traffic simulation, we leave output diversity as future research. 
conclusion: 
 We present ProSim, a multimodal promptable closed-loop traffic simulation framework. Given complex sets of multimodal prompts from the users, ProSim simulates a traffic scenario in a closed-loop manner while instructing agents to follow the prompts. ProSim shows high realism and controllability given different complex user prompts. We also developed ProSim-Instruct-520k, the first multimodal prompt-scenario paired driving dataset with over 520K scenarios and 10M+ prompts.We believe the ProSim model and dataset suite will open gates for future research on promptable human behavior simulation, within and beyond driving scenarios. Note that some tokens of L specifically mentions indexed agents (e.g., "<A1>"). To help LLM understanding the correspondence between agent reference and their policy tokens, we directly replace these reference text tokens with agent policy tokens. Specifically, for an agent-reference text token t L[j] , we replace it with the corresponding agent policy token t L 
 
 Appendix: [j] ← -t a[i] given t L[j] corresponds to "<A[i]>"To make sure each "<A[i]>" is tokenized to a single token, we add them as new tokens to the text tokenizer. We call this step "Agent Token Replacement" in Figure A1 .After getting all the text and agent token features, we concatenate them to create the complete input sequence {t L [1] , ..., t L[O] , t a[1] , ..., t a[N ] } and feed it to the LLM. We then extract the hidden features for the last N tokens from the last LLM layer {t ′ a [1] , ..., t ′ a[N ] }, which contains text-contextualized policy features for each agent. Next, we use an MLP adaptor to convert these features back to get the text-conditional policy features {q L [1] , ..., q L[N ] } = MLP({t ′ a [1] , ..., t ′ a[N ] }). Finally, for each agent, we obtain its policy token π i by adding the prompt and text conditional policy tokens together,π i = q ρ[i] + q L[i] .A.3 Training LLM Pretraining. As described above, we train the LLM in Generator with LoRA to comprehend and generate policy token features. However, we found that directly training the LLM with the closed-loop imitation loss leads to inferior text-prompting performance. We conjecture this is because at the early training stage the LLM is not prepared to interact with the policy tokens. Meanwhile, the rollout loss can be decently optimized without using the LLM's output, giving little signal for LLM to learn.To deal with this issue, we propose to pretrain the LLM to have the capacity to interact with policy token features. To this end, we first train a ProSim model without using text prompts and the LLM with the rollout loss L. Next, we add a simple MLP layer after the Generator to predict the goal point of each agent given its policy token π. We supervise this task with an MSE loss L goal using GT goal points. This task is much simpler than the full task while enforces the LLM to interact with policy tokens. To pretrain the LLM, we fix all other modules and only train the LLM and this new MLP with L goal . Here we only use text as the prompt in inputs. After pretraining, the LLM learns to predict the goal intention of each agent from text prompt and add that information to π, making it already useful for the Policy. Finally, we discard the goal-prediction MLP and train the full ProSim module with all types of prompts and the complete loss L. In our experiments, we pretrain the LLM on ProSim-Instruct-520k for 5 epoches before using it in the full Generator module.Collision loss. For collision loss L coll , we aim to compute the overlapping area between each pair of agent using their bounding boxes through the full rollout trajectory. To this end, at each timestep, for each agent we compute their occupation polygon using their size, position and heading at this timestep. Then, we measure the overlapping area of each agent polygon pair by computing the signed distance between these polygons, where positive distance indicate no collision while negative distance collision indicate a collision. Specifically, we compute the signed distance between polygons A and B with the distance between the origin point and the Minkowski sum A + (-B). For all the signed distance, we compute the loss value by first setting the positive distance to 0 (no collision), and then taking the negative of the rest of signed distances to penalize collisions. We compute the average over all agents through all timesteps to obtain the final L coll . We implement this loss function by referring to the Waymo Open Sim Agent Challenge collision metric.Offroad loss. For offroad loss L off , we aim to compute the overlapping area of each agent and the offroad areas through the full rollout trajectory. To this end, we use the similar strategy as in the collision loss. Specifically, at each timestep we obtain all the agent polygons in the same way as in the collision loss. Then, we compute the signed distance between the four corners of each polygones to a densely sampled set of road edges. Similarly, positive distance indicate no offroad while negative distance indicate being offroad. Finally, We average over all negative distances for all agents through all timesteps to obtain the final offroad loss. We implement this loss function by referring to the Waymo Open Sim Agent Challenge offroad metric.B ProSim-Instruct-520kCompared with the real trajectory, points in the route sketch are sparse, noisy, and incomplete. We simulate these effects with the following steps. First, we extract each agent's complete trajectory from τ . Then, we process this point set by 1) uniformly subsampling the points for sparsity; 2) adding random noise to each point; 3) randomly sampling a consecutive subset. For each agent, its route sketch is a set of ordered 2D points representing sketch points on the map. The number of points in route sketch could be different across agents. In our experiment, we use a uniform subsample rate of 5. We than add random noise with standard variation of 0.1 meter. Finally, we ensure the randomly sampled consecutive subset contains at least 5 points.For each action type, we carefully design a heuristic function that takes an agent's trajectory s t1:t2 from step t 1 to t 2 , and outputs a binary label whether s t1:t2 satisfies the condition of this action. Then, we can run this function across the full rollout with sliding window and temporal aggregation to obtain the full duration [t s , t e ] that this motion tag is valid. For each agent, we run all the motion tags with this method we obtain a set of motion tags. Finally, we post-process the labels to remove conflicting motion tags and temporal noises. Please refer to our codebase for the heuristic function implementation details upon release.For each scenario, we provide the LLM model with agent properties (name and type) as well as all of their agent tags (action type and duration). We then prompt LLM to output 20 different sentences, each describing the agent behavior or scenario properties in natural language. To obtain interesting and diverse language description of the scenario, in the system prompt we instruct the LLM to 1) describe temporal transition of agent behavior (e.g., "Let <A1> change to the left lane and then make a left turn."); 2) describe scenario properties (e.g., "This is a busy scene with most agents accelerating"); 3) describe relationships of different agents (e.g., "Let <A1>, <A2>, <A3> keep their own lanes simultaneously"). We concatenate these sentences together to form the prompt L for each scenario.In our paper, we provide two metrics of promptable closed-loop traffic simulation to measure realism and controllability. Here we link these metrics to the problem formulation in Section 3. Recall that we formulate promptable closed-loop traffic simulation as Given GT data (τ, σ, ρ), realism measures the probability of the real rollout under the model distribution p(τ |σ, ρ). In our main paper, we implement this metric with ADE(τ , τ ).EQUATIONOn the other hand, controllability measures how well the model follows the prompt ρ. We quantify this by comparing the model's realism gain against the unconditional model rollout p(τ |σ, ρ)p(τ |σ). In our main paper, we implement this metric with relative improvement (% Gain) in realism of the model's output with and without prompts. We compute % Gain by comparing rollout ADE with and without prompt conditioning. : % Gain = ADE(τ ,τ )-ADE(τ ,τ ) ADE(τ ,τ ) × 100%.To ensure the prompts we generate faithfully reflect the agent behaviors in the scenario, we conduct a careful quality assurance process with human effort. As Goal Point and Route Sketch are directly modified from real trajectories, there is no need to check their accuracy. On the other hand, the accuracy of Text is largely dependent on the accuracy of the Action Tag as it stems from Action Tags of all agents in a scene. Therefore, we focus on checking the quality of Action Tag. For each action types, we ask human labelers to manually check whether the labeled action tag is accurate both semantically and temporally by viewing the rollout videos. At each round, we ask human labelers to check 100 motion tag examples for each action type. If the qualification rate of a certain action type is below 85%, we rewrite the heuristic function of this action type according to the human feedback, relabel the motion tags of this type and ask for another round of human checking. We continue this process until all the action types pass the quality threshold.We show the interface we developed for human labelers in Figure A2 . This interface allow human labelers to go through and rewind the scenario easily with the interactive progress bar. For each scenario with multiple action tags, the interface let the labeler to go through all the action tags all together. This allows the human labeler to QA multiple motion tags of the same scenario very efficiently. In average, we found human labelers take around 10 seconds to give the QA output for each motion tag. Additionally, we ask human labelers to give a QA output for each tag, choosen from Correct, Wrong Action, Wrong Time, Wrong Agent, and Need Attention (unsure or other types). These different types of QA error tags provide us useful feedback to improve our heuristic functions.In our paper, we show benchmark results of 5 different prompt combinations. Aside from these combinations shown in the paper, we also show the % Gain results of other prompt combinations in Table A1 . We can see from the Firstly, in Table A2 we conduct ablation study on inference batch size (number of scenarios per batch). we prompt 50% of all agents at random (also randomly selecting a prompt type, i.e., goal, sketch, action, text). This table shows that ProSim is able to achieve very high inference efficiency, especially when we use batched scenario inference. Notably, when sample batch size is 16, ProSim can simulate an 8-second scenario with 64 agents and all kinds of prompts (including text) within 50ms on average. This table shows that ProSim can be highly efficient with parallel computation.Next, in Table A3 we do ablation study on the number of simulated agents per sceanrio, where we set batch size equal to 1. This table shows ProSim is able to easily scale to a large number of agents (up to 128) with minimal increases in inference time. Specifically, inference time only grows by 12% from 16 agents to 128 agents. Finally, in Table A4 , we vary the number of prompts used per scenario. This table shows ProSim can maintain efficiency even with a large number of prompts.We achieve the high efficiency of ProSim with careful designs. Firstly, ProSim only run Encoder and Generator once per scenario, regardless of the number of agents in the scenario. For Encoder, the main bottleneck is from the many scene tokens, so agent tokens don't affect runtime much. For the Generator, the agent-centric prompt encoders are very lightweight and not a problem. As for the LLM module, the main bottleneck again is from text tokens, not agent tokens. Overall, the runtime of the Encoder and Generator doesn't increase much with more agents.On the other hand, ProSim runs the Policy for all agents in parallel. We make this possible by letting each agent act individually given only its own scene and policy tokens (Section 4.3). This design not only realistically reflects how agents interact in the real world, but also allows for batched policy inference at each timestep for all agents (even across different scenarios). For example, given M scenarios, each with N simulated agents, at each timestep we conduct M × N policy inferences in parallel within a single batch. This operation allows us to achieve very high inference throughput on GPUs.Recall that when we prompt one agent with prompts with multiple modalities, we always provide prompts with the similar semantics, as this is the expected way users will use ProSim for. We also train ProSim with this manner. However, it is interesting to see what will happen if for a single agent we provide multiple prompts with different or conflicting semantics simultaneously. This experiment could show which types of prompt ProSim is more sensitive to. Therefore, in this section, we investigate ProSim's behavior if multiple prompts with different semantics are given simultaneously to a single agent. We provide a case study showing how ProSim deals with prompts with different semantics in Figure A3 .In this experiment, we prompt agent A8 with four different prompts,one modality at a time, as in Figure A3 (1)-(4). Note how ProSim is able to make A8 follow different modalities of prompts closely. To investigate the case when we provide multiple prompts with different semantics simultaneously, in ( 5)- (12) we prompt agent A8 with 2-3 prompts with conflicting semantics and modalities together. For example, in (7) we prompt A8 with a Goal point to turn left and with Text to turn right. Here are our observations:• Goal prompts dominate over other prompt types. Whenever a Goal prompt is provided, agents will try to follow it first and tend to ignore other prompt types, as can be seen from ( 5) and (7) .• Similar to Goal, Sketch also dominates over Action and Text, as shown in ( 8) and ( 9). Note that in (6) when Sketch and Goal appear together, the agent does not strictly follow Goal, but rather tries to interpolate between the Goal and Sketch points. This indicates that Sketch has similar controllability as Goal.• Action and Text provide more abstract instructions and are therefore dominated by more explicit Goal and Sketch prompts. When Action and Text appear together in (10) , we can see the agent also tries to interpolate between Action and Text (turning right but also stopping, a mix between (3) and ( 4)). This also indicates that Text has similar controllability as Action. Overall, we found that, in terms of controllability, when multiple semantically-conflicting prompts appear, Goal > Sketch > Action ≃ Text. This rank resonates with the controllability scores in Table 1. We conjecture that this is because Goal and Sketch prompts contain more explicit information while Action and Text prompts are more abstract. Therefore, during training, ProSim learns to focus more on Goal and Sketch prompts more (which helps it reduce losses more effectively).We omit the p, h parameters in Γ for simplicity.We refer readers to Appendix A.3 for detailed formulations of Lcoll and Loff. 


Decision: Strengths:
- Novel promptable closed-loop traffic simulation framework supporting multiple prompt types (coordinates, text, etc.)
- Creation of a large-scale multimodal prompt-scenario driving dataset (ProSim-Instruct-520k) 
- Well-written paper that is easy to follow
- Competitive performance on Waymo Sim Agents Challenge, with high prompt controllability

Weaknesses:
- Limited analysis of prompt effects on agent interactions and corner cases
- Scalability concerns as number of agents increases
- Questions about usefulness/diversity of text prompts beyond basic action tags
- Lack of proper SOTA baselines and composite metric results for Waymo Open Sim Agents Challenge

Reviews: review: title: title: Promptable traffic simulation is an important topic for autonomous driving simulation. ProSim provides a LLM-based method. The dataset will be helpful for the sim agent community.
paper_summary: paper_summary: ProSim is a first-of-its-kind promptable closed-loop traffic simulation framework. The ProSim-Instruct-520k, a large-scale multimodal prompt-scenario driving dataset, is the first driving dataset with semantic-rich agent motion labels and text captions.
main_review: main_review: 1. ProSim is a first-of-its-kind promptable closed-loop traffic simulation framework. The ProSim-Instruct-520k, a large-scale multimodal prompt-scenario driving dataset, is the first driving dataset with semantic-rich agent motion labels and text captions. 
2. This development including ProSim and the dataset will be helpful for the sim agent and traffic simulation community.
3. This paper is easy to follow and well written. 
4. The experimental results prove the effectiveness of the proposed ProSim. ProSim achieves high prompt controllability given different user prompts, while reaching competitive performance on the Waymo Sim Agents Challenge when no prompt is given.  

There are some minor questions for Prosim.
5. How to make the representation alignment (encoder) for the LLM backbone to understand the complex map and agent information without QAs? 
6. I suggest that the author can provide the inference time analysis for the traffic simulation. 
7. How to make a scene initialization for the Waymo Open Motion Dataset?
questions: questions: How to make the representation alignment (encoder) for the LLM backbone to understand the complex map and agent information without QAs?
limitations: limitations: 3
review_summary: review_summary: ProSim is a first-of-its-kind promptable closed-loop traffic simulation.
title: title: This paper propose a promptable closed-loop traffic simulation framework. The content is comprehensive, but the novelty is limited.
paper_summary: paper_summary: This paper introduces ProSim, a promptable closed-loop traffic simulation framework that effectively models interactivity while maintaining realism and controllability. The paper is well-written and easy to follow. However, the prompting method can be seen as a straightforward extension of common multi-agent prediction or simulation frameworks. The improvement in ADE results with the proposed prompts, such as GT goal points, is expected. Deeper analysis and more results on long-tail scenarios are desired.
main_review: main_review: Pros:
1. The paper introduces a novel promptable closed-loop traffic simulator.
2. The simulator supports four types of prompts, enhancing realism. Combined prompts lead to higher realism results.
3. Qualitative results demonstrate the simulator's controllability and the effectiveness of its interactivity modeling.
4. The simulator achieves competitive results in the Waymo Sim Agent Challenge.
5. The authors have built a dataset, which is claimed to be released.

Cons:
1. The controllability aspect needs further exploration. The impact of different prompts on agent behavior should be analyzed in more depth. For example, what are the effects of different prompts on agent interactions? What happens if multiple prompts with different semantics are given simultaneously? Can the simulator generalize to corner cases where agent behaviors deviate from the conventions recorded in the training dataset?
2. In Figure 3(d), agents seem to be represented with unique indexes in the text prompt. How to encoder this information in the network and bridge the text prompt and numeric features?
3. In Figure 2, the prompt effectiveness of sketch and action is not monotonically increasing, which lacks detailed analysis.
4. The diversity of the simulation results is limited.
5. The simulation efficiency should be demonstrated and compared.
questions: questions: Please refer to cons.
limitations: limitations: 2
review_summary: review_summary: This paper propose a promptable closed-loop traffic simulation framework. The content is complete, but the novelty is limited.
title: title: Nice paper
paper_summary: paper_summary: A closed-loop traffic simulation framework supporting multiple prompts, including coordinates and text. A hierarchal motion generator is used. New dataset proposed.
main_review: main_review: # Clarity

The paper is well-written and easy to follow.

# Originality

The framework is novel. To my knowledge there is no such a method that can take multiple prompts to control the scene generation. The hierarchal motion generator is also very interesting.

# Strengths

1. The paper covers different forms of prompts and align them into the same framework.
2. The dataset is a good contribution to the community.
3. The paper is well-written.
4.  The idea of closed-loop training, though not novel in motion prediction, integrates well with other part of the model and is promising.


# Weaknesses

1. Table 2 does not have SOTA baselines, and it doesn't present the composite metric. This is very confusing as the golden standard of WOSAC is the composite metric.
2. The method doesn't seem to be scale as the number of agents increase.
3. I have concern on the usefulness of the text prompt. It seems that the text prompt is just the action tags (left / forward / right) in the textual format.
questions: questions: 1. Are the LORA of LLM trained together with the whole pipeline in closed-loop manner?
2. Is it possible to use text prompt to prompt the behaviors of multiple agents? I think it's possible. Have you every tested this: as the number of agents being prompted increase, the controllability saturates. 
3. In limitation you said complex agent interaction is not promptable right now. Can I understand this as the autolabeling mechanism when you creating the dataset limits the diversity of text prompt. The text prompt is actually the action tag in text format. As in Figure 3, we can see the text prompts are basically "some agents make a left turn / move forward". So the so call text prompt is not that useful as you claim.
4. As I said in Weakness 1, why the WOSAC result is not presented properly?
5. In MTR++, they use 64 queries to increase the diversity of the output. I wonder whether the "unconditional generation", and as well as the WOSAC result, can be improved if we discover say 64 "common prompts". (Note that 64 is for motion prediction, in WOSAC the required modes are 32).
6. How many agent are simulated?
7. In WOSAC they require you to simulate all 128 agents. How you address the scalability issue if you are simulating 128 agents?
limitations: limitations: 2
review_summary: review_summary: The paper is sound and well-written, with sufficient novelty and contribution to the community.