SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout
Authors: Chiyu Max Jiang
Yijing Bai
Andre Cornman
Christopher Davis
Xiukun Huang
Hong Jeon
Sakshum Kulshrestha
John Wheatley Lambert
Shuangyu Li
Xuanyu Zhou
Carlos Fuertes
Chang Yuan
Mingxing Tan
Yin Zhou
Dragomir Anguelov
Venue: NeurIPS 2024 poster (2024)
Keywords: Autonomous Vehicles
Simulation
Diffusion Models
Scene Generation
Close Loop Simulation
Primary area: diffusion_based_models

Abstract: Simulation with realistic and interactive agents represents a key task for autonomous vehicle (AV) software development in order to test AV performance in prescribed, often long-tail scenarios. In this work, we propose SceneDiffuser, a scene-level diffusion prior for traffic simulation. We present a singular framework that unifies two key stages of simulation: scene initialization and scene rollout. Scene initialization refers to generating the initial layout for the traffic in a scene, and scene rollout refers to closed-loop simulation for the behaviors of the agents. While diffusion has been demonstrated to be effective in learning realistic, multimodal agent distributions, two open challenges remain: controllability and closed-loop inference efficiency and realism. To this end, to address controllability challenges, we propose generalized hard constraints, a generalized inference-time constraint mechanism that is simple yet effective. To improve closed-loop inference quality and efficiency, we propose amortized diffusion, a novel diffusion denoising paradigm that amortizes the physical cost of denoising over future simulation rollout steps, reducing the cost of per physical rollout step to a single denoising function evaluation, while dramatically reducing closed-loop errors. We demonstrate the effectiveness of our approach on the Waymo Open Dataset, where we are able to generate distributionally realistic scenes, while obtaining competitive performance in the Sim Agents Challenge, surpassing the state-of-the-art in many realism attributes.

SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout
Chiyu Max Jiang Yijing Bai Andre Cornman"
Christopher Davis"
Xiukun Huang" Xuanyu Zhou'
Hong Jeon"
Sakshum Kulshrestha" John Lambert"
Shuangyu Li
Carlos Fuertes
Chang Yuan
Mingxing Tan
Yin Zhou
Dragomir Anguelov
Waymo LLC
Abstract
Realistic and interactive scene simulation is a key prerequisite for autonomous vehicle (AV) development. In this work, we present SceneDiffuser, scene-level diffusion prior designed for traffic simulation. It offers unified framework that addresses two key stages of simulation: scene initialization, which involves gener- ating initial traffic layouts, and scene rollout, which encompasses the closed-loop simulation of agent behaviors. While diffusion models have been proven effec- tive in learning realistic and multimodal agent distributions, several challenges remain, including controllability, maintaining realism in closed-loop simulations, and ensuring inference efficiency. To address these issues, introduce amortized diffusion for simulation. This novel diffusion denoising paradign garadigm amortizes the computational cost of denoising over future simulation steps, significantly reducing the cost per rollout step (16x less inference steps) while also mitigating closed-loop errors. We further enhance controllability through the introduction of generalized hard constraints, a simple yet effective inference-time constraint mechanism, as well language-based constrained scene generation via few-shot prompting of a large language model (LLM). Our investigations into model scaling reveal that increased computational resources significantly improve overall simulation real- ism. We demonstrate the effectiveness of our approach on the Waymo Open Sim Agents Challenge, achieving top open-loop performance and the best closed-loop performance among diffusion models.
Introduction
Simulation environments allow efficient and safe evaluation of autonomous driving systems [
S
22 31 32 46 52 54 Simulation involves initialization (determining starting conditions for
agents) and rollout (simulating agent behavior over time), typically treated as separate problems [4] Inspired by diffusion models" success in generative media, such as video generation [2 10] and video editing (inpainting (2 24 28), extension, uncropping etc.), we propose SceneDiffuser, a unified spatiotemporal diffusion model that addresses both initialization and rollout for autonomous driving, trained end-to-end on logged driving scenes. To our knowledge, SceneDiffuser the first model to jointly enable scene generation, controllable editing, and efficient learned closed-loop rollout (Fig. T
One challenge in simulation evaluating long-tail safety-critical scenarios data mining can help, such scenarios are often rare. We address this by learning generative scene realism prior that allows editing logged scenes or generating diverse scenarios. Our model supports scene perturbation (modifying scene while retaining similarity) and agent injection (adding agents to create challenging scenarios), We also enable synthetic scene generation on roadgraphs with realistic
46 While
'Equal contribution core technical contributors (alphabetically ordered).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Scene Initialization (Generation and Editing)
Log Perturbation Scene Rollout (Closed-loop with Amortized Diffusion)
Log Scenario
Agent Injection
Fully Synthetic Scenario
Rollout Step = 80 (final) Figure 1: SceneDiffuser: generative prior for simulation initialization via log perturbation, agent injection, and synthetic scene generation, and for efficient closed-loop simulation at 10Hz via amortized diffusion. progressively refines initial trajectories throughout rollout. Environment sim agents are in green-blue gradient (temporal progression), AV agent in orange-yellow, and synthetic agents in red-purple.
Rollout Step= 2
Rollout Step = 30
Rollout Step = 60
layouts. We design protocol for specifying scenario constraints, enabling scalable generation, and demonstrate how a few-shot prompted LLM can generate constraints from natural language.
Given scene, realistically simulating agents and AV behavior is challenging 15
50 52
38 41 48 where entire future trajectories are jointly predicted in a single inference, simulator predictions are iteratively fed back into the model, requiring realism at each step. This poses challenges:
Unlike motion prediction tasks [18] 25 26
distributional drift from compounding errors, high computational cost for models like diffusion, and the need to simulate various perception attributes realistically.
We propose Amortized Diffusion for simulation rollout generation, a novel approach for amortizing the cost of the denoising inference over a span of physical steps that effectively addresses the challenges of simulation realism due to closed-loop drift and inference efficiency. Amortized diffusion iteratively carries over prior predictions and refines them over the course of future physical steps (see Sec. 3.2 and Fig. 4.. This allows our model to produce stable, consistent, and realistic simulated trajectories, while requiring only a single denoising function evaluation at each physical step while jointly simulating all perception attributes each step. Experiments show that Amortized Diffusion not only requires 16x less model inferences per step, but is also significantly more realistic.
In summary, ScfneDiffuser's 's main contributions are:
A unified generative model for scene initialization and rollout, jointly learning distributions for agents, timesteps, and perception features including pose, size and type.
A novel amortized diffusion method for efficient and realistic rollout generation, significantly improving trajectory consistency and reducing closed-loop error.
Controllable scene initialization methods, including log perturbation, agent injection, and synthetic generation with a novel hard constraint framework and LLM.
Investigation of model scaling, showing increased compute effectively improves realism.
Demonstration of effectiveness on the Waymo Open Sim Agents Challenge, achieving top open-loop performance and the best closed-loop performance among diffusion models.

---

# Time Step
Past Current Future
Agents
InPaint
Agent
length,
Behavior
Intent Cond
Conditional
Uncenditional
Full Scene
Agent
Features
Prediction Behavior Prodiction
SceneCien
The Scene Tensor
Multi-task Inpainting
Figure 2: We formulate various different tasks, including behavior prediction, conditional scenegen and unconditional scenegen as inpainting tasks the scene tensor. We represent the scene tensor as a normalized tensor x E RAXTxD, for the number of agents, timesteps and feature dimensions.
Related Work
2.1 Data-driven Agent Simulation
variety generative models have been explored for scene initialization and simulation, including
autoregressive models 8 22 46 cVAEs 455, cGANs [[] and Gaussian Mixture Models (GMMs)
8 477. For closed-loop rollouts, these models have been extended with GMMs [ST], GANs [S]
AR models over discrete motion vocabularies [31], cVAE [54], and deterministic policies [0 52
Open-loop rollouts have also been explored using cVAE 35
2.2 Diffusion Models for Agent Simulation
Open-loop Sim Open-loop simulation generates behavior for agents that all lie within one's control, iee does not receive any external inputs between steps. Open-loop simulation thus cannot respond to an external planner stack (AV), the evaluation of which is the purpose of simulation. Diffusion models have recently gained traction in multi-agent simulation, particularly in open-loop scenarios (multi- agent trajectory forecasting) [31 [39], using either single-shot autoregressive (AR) generation. Single-shot approaches employ spatiotemporal transformers in ego-centric [6] [8] or scene-centric
frames with motion/velocity deltas O [53] Soft guidance techniques enhance controllability [17 56)
DJINN 27
uses 2d condition masks for flexible generation.
Closed-loop Sim Closed-loop simulation with diffusion remains challenging due compounding errors and efficiency concerns. Chang et al. explore route and collision avoidance guidance in closed-loop diffusion, while VBD [4] combines denoising and behavior prediction losses with
query-centric Transformer encoder 42 VBD found computationally infeasible to replan at HHz frequency in receding horizon fashion over the full WOSAC test split due to the high diffusion inference cost, therefore testing in open-loop except over 500 selected scenarios.
Initial Condition Generation Diffusion-based initial condition generation has also been studied 20 Pronovost al. [32 33 adapt the L.DM framework rendered scene images, while SLEDGE and DriveSceneGen diffuse initial lane polylines, agent box locations, and AV velocity.
2.3
Diffusion for Temporal World Modeling and Planning
Outside of the autonomous driving domain, diffusion models have proven effective for world sim- ulation through video and for planning. Various diffusion models for 4d data have been proposed, often involving spatiotemporal convolutions and attention mechanisms [T] 12 43 In robotics, diffusion-based temporal models leverage Model Predictive Control (MPC) for closed-loop control
4 and have shown state-of-the-a performance for imitation learning 29
Similar to our Amortized Diffusion approach, TEDi [55] proposes to entangle the physical timestep and diffusion steps human animation, thereby reducing O(T T) complexity for T physical timesteps and 7 denoising steps to O(T). However, are the first work to demonstrate the effectiveness of this approach for reducing closed-loop simulation errors, and the first to extend it to multi-agent simulation setting.

---

Roudgraph P.
Temsor 4
(K. V)
X-4mm
Fused Context
Project [N, +N, .1]
[4. D)
Global Contect
Adhuptive I.N
[4. T.
1A. F. H]
Triliee Sgal, IN, DJ
Concal
Project
Project
Global Context
Concar
Latent Tokens [X,
X-dtin
[N,
Repeot
[4. T. D)
Context
PosEnee r PosEnc
Self-Attn (Time)
Seff-AAiu
Denoise
(0)
[4
(Agents)
Scene Tensor:
Global Context Encoder
Transfommrr Denoiser Rackhone
Figure 3: SceneDiffuser architecture. Global scene context is encoded into a fixed number of No. tokens via Perceiver IO [T6] encoder. The noisy scene tokens are fused with local and global context, then used to condition a spatiotemporal transformer-base backbone [49] via Adaptive I LayerNorm (AdaLN) 30]. Input/output tensor are green, context tensors in blue, and ops in italics.
3
Method
3.1 Scene Diffusion Setup
We denote the scene tensor as E RAXTxD where A is the number of agents jointly modeled in the scene, T is the total number of modeled physical timesteps, and D is the dimensionality of all the features that are jointly modeled. We learn to predict the following attributes for each agent: positional coordinates y, heading bounding box dimensions h, w, and object type (AV, car, pedestrian, cyclist). We model all tasks considered SceneDiffuser as multi-task inpainting this scene tensor. Given an inpainting mask m E BAXTxD the corresponding inpainting context values x == m. ( ar, set of global context e (such as roadgraph and traffic signals), and validity mask for given agent at given timestep
BAT (to account for there being < A agents in the scene or for occlusion), we train a diffusion model to learn the conditional probability p(x) where C. == {m, &, c, 0). See Fig. 2 for an illustration of the scene tensor.
Feature Normalization To simplify the diffusion model's learning task, we normalize all feature channels before concatenating them along D to form the scene tensor. We first encode the entire scene in centric coordinate system, namely the AV's coordinate frame just before the simulation commences. We then scale y. = by fixed constants, I, h, w by their standard deviation, and one-hot encode h. See Appendix for more details. This simple yet generalizable process allows us to jointly predict float, boolean, and even categorical attributes by converting into normalized space of floats. After generating a scene tensor æ, we apply a reverse process to obtain the generated features.
Diffusion Preliminaries We adopt the notation and setup for diffusion models from [3], The forward diffusion process gradually adds Gaussian noise æ. The noisy scene tensor diffusion step t can be expressed as q(z, [x) = - N(zz I),, where ag and 0, are parameters which control the magnitude and variances of the noise schedule under a variance- peeenncviigeeeigggg.... model. Therefore z a,æ + JyC,, where E, _N0,, I). One major departure from the classic diffusion setup in our amortized diffusion regime is that we not assume a uniform noise level E RR for the entire scene tensor ar Instead, we have E TR? where i. can be relaxed to have different value per physical timestep in the scene tensor as described in Sec. S.2. We utilize the commonly used a-cosine schedule where a (Ttt22 and a. sin(nt/ 22). At the highest noise level of t 1, the forward diffusion process completely destroys the initial scene tensor æ resulting in 2, = Eg ~ (O, I). Assuming a Markovian transition process, we have the transition distributions q(zr zz.)
oi I). where at = az/a, and 0² = of - of and t > s. In the denoising process, conditioned on a single datapoint æ, the denoising process can be written
g(=o(ze, x)) NZ (e of
(I)
where 14r-s a..0. of 2, + a.o. x and 0t-oa L. In the denoising process, æ is approximated
using learned denoiser é. Following [13 and [37. we adopt the commonly used v prediction, defined as V2 (E:, x) = arE; - R, We trained a model parameterized by to predict v, given &t. t and context C: 6. = 0p(#,, C). The predicted &, can be recovered via &; a. a, The model end-to-end trained with a single loss:
E
ve x)
(2)

---

Appead
Noise
Level
Pop
1. Warm up
2. Denoise I step
3. Step Forward
4. Next Step
Repeat (Loop)
Figure 4:
Amortized diffusion rollout procedure. The warm up step initializes the future predictions for the entire future horizon, which is then perturbed by monotonic noise schedule f. The trajectory is iteratively denoised by one step at each simulation step.
D = C, = 1,2. | is the dataset containing paired agents and scene context data, t is probabilistically either sampled from a uniform distribution, or sampled a monotonically increasing temporal schedule f, where f,- max (0, (T Thistory) (aasary)/TTuunea /Tuuuee) facilitate amortized rollout which will be discussed in Sec. 3.2 Each is sampled with 50% probability. .M {mas O
mecuiroll Mennggen O mcmmrrl) } is the set of inpainting masks for the varied tasks.
Scene Diffusion Tasks Different tasks are fomulated as inpainting problems (Fig. 2
Scene Generation (SceneGen): Given the full trajectory of some agents, generate the full trajectory of other agents. We have Myyonngee E RR1.1.1 (broadcastable to T timesteps and D features), where macenggen. a Pr(X- = AAseloct/Avalid Avalid). where Ageleet [0, Avalia) the number of agents sampled to be selected as inpainting conditions out of Avalid valid agents in the scene.
Behavior Prediction (BP): Given past and current data for all agents, predict the future for all agents. We have map E R..TT.L (broadcastable to agents and D features), where map.r = I(T < Thistory)-
Conditional SceneGen and Behavior Prediction: Both scenegen and behavior prediction masks are multiplied by control mask at training time to enable controllable scenegen and controllable behavior prediction at inference time. We have Mccntrol E RR4.7. .D, where conttol, (a, Io I, Ia. La ~ Pr(X- = Acomtrol/ AAatia) 1- ~ Pr(X- = Teontrol/ 'T) Ia ~ Pr(X = Pa) where Pa of the corresponding feature channel. This allows us to condition on certain channels, such as positions x. y with or without specifying other features such as type and heading.
Architecture We present schematic for the SceneDiffuser architecture in Fig. 3 consisting of two end-to-end trained models: a global context encoder and a transformer denosier backbone. Validity ō used transformer attention mask within the transformer denoiser backbone.
Diffusion Sampler We use DPM++ 19 with a Heun solver. We utilize 16 denoising steps for our one-shot experiments and for our amortized diffusion warmup process.
3.2 Scene Rollout
Future prediction with replanning ('One-Shot') is not used in simulation due to its non-reactivity. and forward scene inference, under the standard diffusion paradigm ('Full AR'), is computationally intensive due to the double for-loop over both physical rollout steps and denoising diffusion steps [55]. Moreover, executing only the first step while discarding the remainder leads to inconsistent plans that result in compounding errors. We adopt an amortized autoregressive ('Amortized AR') rollout, aligning the diffusion steps with physical timesteps to amortize diffusion steps over physical time, requiring single diffusion step at each simulation step while reusing previous plans.
We illustrate the three algorithms in Algorithm 3 using the same model trained with a noise mixture t (0, 1): f) (Eqn. 2). We also illustrate Algorithm 3 in Fig. 4 We denote the total number of timesteps T = H + F., where H, F' denote the number of past and future steps. We denote x : x -H:FI to be the temporal slicing operator where [0] is the final history step.
Input: Global context (roadgraph and traffic signals), history states xl-H:0),, validity U.
Output: Simulated observations for unobserved futures a [IIF))
2 We omit L/2 due training collapse.
5

---

Algorithm 1 One-Shot (Open-Loop)
Algorithm 2 Full AR (Closed-Loop)
OneShos(al-W:o)) 21 ())
FallAR(al-/oo) ml-1:0)
2:
3: fort .o.
For each diffusion timestep
3:
Tor 0, do
For each physical timestep
b V-prediction 4: */++1r+47)
OneShot(a (al- firl, c) Updase bulfer indices
di + + (=4 C)
@: +
a a1-10 @ map + de @ (~ may) Apply inpuinting
5: return veturaalt-y)
Sx (
return (1:2)))
Algorithm Amortized AR (Closed-Loop)
1: AmerizedARR((a ()):
2: à OneShot(zl--Ro0)
Wamn-Up Add noise
3: a(0:0) +- --ggal::] ag-
4: for r- =1 T
For each physical timestep Recover solutoon from prediction Add noise br Apply inpainting
5: alrirtP] agallir+e - og (alrir+11, 6)
6: alri++F] aegaliit m
7:
alr-Mirt21 +- alr- Hir) @ mip alr- Hi++F) (~ mip))
8: return (1.FF)
WOSAC Composite Metric
Composite Metric
1/1
8.54
2
3:32
MI
4.1
A4s
20+6
9:33
TesG
68%
8:50
10/00m
Replanning Interval/Periot (ms)
2.5 3008 4.MM
Parums
204 2014 -014
IC3M
Figure 5: We compare the influence of replan rate performance for our Full AR and Amor- tized AR models. Circle radius x # inference calls over the simulation. At 10Hz, Amortized AR requires 16x less model inference per step and is more realistic compared to Full AR.
Figure Scene generation realism with model parameter and resolution scaling². Decreased temporal patch sizes (i.e. increased temporal resolution) and increased parameters both effective for improving realism via compute seal- ing. Circle radius compute GFLOPs.
3.3 Controllable Scene Generation
To simulate long-tail scenarios such as rare behavior of other agents, it is important effectively insert controls into the scene generation process. To do so, we input an inpainting context scene tensor æ where some pixels are pre-filled. Through pre-filled feature values in T, we can specify particular agent of specified type to be appear at a specific position at a specific timestamp.
Data Augmentation via Log Perturbation The diffusion framework makes it straightforward to produce additional perturbed examples of existing ground truth (log) scenes. Instead of starting from pure noise 2, (0, I) and diffusing backwards from 0, we take original log scene x' and add noise to such that our initial z; = a,æ' + E, where E; ~ NO I,I). Starting the diffusion process at f yields the original data, while t produces purely synthetic data. For E (0. 1), higher values increase diversity and decrease resemblance to the log. See Figs. [ and 12 (Appendix).
Language-based Few-shot Scene Generation The diffusion model inpaint constraints can be defined through structured data such as Protocol Buffer" ('proto"). Protos can be converted into inpainting values, and we leverage the off-the-shelf generalization capabilities of a publicly accessible
httpp::/ https://protobuf.dee

---

Full AR 0.1 HZ
Full AR @ Hz
No Hard Constraints
Full AR @ 10 Hz
Amortized AR @ 10 Hz
(b) Post-Diffusion GHC
(c) In-Diffusion GHC
Figure 7: Full AR quality deteriorates at in- creasing replan rates due to compounding errors. Amortized AR retains a high level of realism even 10 Hz while being more efficient.
Figure 8: Applying no-collision constraints pre- vents collisions (red-purple) in generated scenes (b, c). Iteratively applying constraints with every diffusion step further enhances realism (c b).
chat app powered by a large language model (LLM)*, generate new Scene Diffusion constraints protos solely using natural language via few-shot prompt engineering. We show example results generated by the LLM in Fig. 10 Details in the Appendix (A.7)
3.4 Generalized Hard Constraints
Users of simulation often require agents to have specific behaviors while maintaining realistic trajectories. However, diffusion soft constraints [27] [56 57 require differentiable cost for the constraint and do not guarantee constraint satisfaction. Diffusion hard constraints [21] are modeled inpainting values and are limited in their expressivity.
Inspired by dynamic thresholding [36 in the image generation domain, where intermediate images are dynamically clipped to range at every denoising step, we introduce generalized hard constraints (GHC), where generalized clipping function is iteratively applied at each denoising step. We modify
Equ. such that at each denoising step M4++ or + Ln clip(x). where clip(-) denotes the
of
of
Specificciic clipping operator. See more details on constraints in Appendix A.9
We qualitatively demonstrate the effect of hard constraints for unconditional scene generation in Fig. g Applying hard constraints post-diffusion removes overlapping agents but results in unrealistic layouts, while applying the hard constraints after each diffusion step both removes the overlapping agents and takes advantage of the prior improve the realism of the trajectories. We find that the basis on which the hard constraints operate is important: good constraint will modify significant fraction of the scene tensor (for example, shifting an agent's s entire trajectory rather than just the overlapping waypoints), or else the model "rejects" the constraint on the next denoising step.
4 Experimental Results
Dataset We use the Waymo Open Motion Dataset (WOMD) for both agent simulation experiments. WOMD includes tracks of all agents and corresponding vectorized maps in each scenario, and offers large quantity of high-fidelity object behaviors and shapes produced by a state-of-the-art offboard perception system.
generation and
4.1 Simulation Rollout
Benchmark We evaluate closed-loop simulation models on the Waymo Open Sim Agent
Challenge (WOSAC) (33 metries (see Appendix(A.1] a popular sim agent benchmark used in many
recent works 9 4 31 51 53). Challenge submissions consist of x/y/z/y trajectories representing centroid coordinates and heading of the objects' boxes that must be generated in closed-loop and
The chat app is available at filggeinnn.. gemini.google.... com powered by Gemini V1.0 Ultra the time of access.
7

---

METRICS
M/1
M/1+GHC
L/1
LoG
COMPOSITE METRIC
0.516
0.558
0.549
0.593
12
Agcal
Conomree Agents
LINEAR SPEED
0.326
0.327
0.331
0.339
10
Luggsl
LINEAR ACCEL.
0.387
0.383
0.378
0.445
ANGULAR SPEED
0.529
0.562
0.534
0.572
ANGULAR ACCEL.
0.595
0.608
0.588
0.625
landi
DIST. TO OBI.
0.154
0.176
0.174
0.192
COLLISION
0.692
0.841
0.794
0.875
TIME TO COLLISION
0.826
0.827
0.827
0.842
DIST. TO RD. EDGE
0.164
0.165
0.176
0.204
OFFROAD
0.546
0.549
0.566
0.605
0.5
widls 310
Table I: Realism metrics on WOMD val. for scene gener- ation for the M/1 model, M/1 with hard constraints, L/I, and oracle log distribution matching (see Sec. 4.3). Real- ism can be improved through hard constraints scaling.
Figure 9: Generated vs logged distri- bution. SceneDiffuser learns realistic joint distributions across modeled fea- tures such as length and width.
COMPOSIIE (1)
RATE
OFFA0AD
H2)
Fix)
COMPOSITE SCORE COMPOSITE SCORE PS
18 1280
0.700
0.673
7.90% -MULITAAS
TypE
SEXTALL
ADoLN Zeno
-2.04%
-14.5%
0,68%
+11.8% +209%
6.35%
2%
42.90%
+29.3%
11.24%
Table 2:
Distrib. realism metries on WOSAC. L/1 denotes the Large model of patch size 1.
Table Design analysis and ablation studies.
with factorized AV vs. agent models. WOSAC uses the test data from the Waymo Open Motion Dataset (WOMD) Up to 128 agents (one of which must represent the AV) must be simulated in each scenario for the 8 second future (comprising 80 steps of simulation), producing 32 rollouts per scenario for evaluation. In a small departure from the official setting, we utilize the logged validity mask as input to our transformer and unify the AV and agents' rollout step for simplicity.
Evaluation In Tab. we show results on WOSAC. We show that Amortized AR (10 Hz) not only requires 16x fewer model inference calls, but is also significantly more realistic than Full AR at 10Hz replan rate. In Amortized AR, re-use the plan from the previous step, leading to increased efficiency and consistency. The one-shot inference setting is equivalent to Full AR with no replanning (0.125 Hz) and achieves comparably higher realism, though as it is not executed in closed-loop, not reactive to external input in simulation, and thus not valid WOSAC entry.
In Figs. and 7 investigate the effects of varied replan rates to simulation realism. While high replan frequency leads to significant degredation in realism under the Full AR rollout paradigm, Amortized AR significantly reduces error accumulation while being 16x more efficient.
we compare against the WOSAC leaderboard with the aforementioned modifications. We achieve top open-loop performance and the best closed-loop performance among diffusion models.
Tab. 4
4.2 Scene Generation
Unconstrained Scene Generation We use the unconditional scene generation task as a means to quantitatively measure the distributional realism of model. We condition the scene using the same logged road graph and traffic signals, as well as the logged agent validity to control for the same number of agents generated per scene. All agent attributes are generated by the model.
Due to lack of public benchmarks for this task, we adopt slightly modified version the WOSAC 23 metrics, where different metrics buckets are aggregated per-scene instead of per-agent, due to the lack of one-to-one correspondence between agents in the generated scene versus the logged scene (see Appendix [.2 for more details). Metrics aggregated over all agents that are valid in the 9 second trajectory.
We show our model's realism metrics in Tab. Even compared to the oracle performance (comparing logged versus logged distributions), our model achieves comparable realism scores in every realism bucket. Introducing hard constraints collisions can significantly improve the composite metric
8

---

AGENT POLICY
RULANN
LINEAR
LINEAR
ANG
ANGL
DOST.
COLLISION
TTC
DISC: TO
OOFROND
COurOSTER
ADE
SINADE
OBFRBADD
Rasioa Anaker
0.166
1140
2A13
0043
0.51
1.03
0:221
0:307
0:228
0.503
03330
1330
0.003
0:981
0.001
2335
1.69
0180
B1335
0:335
0333
0.381
0.546
0.363
6.70
0811
0.61
0.735
0.68
6.158
2.44
0:17
Ortns Do Stroty
0:335
0.39
0.35
0.510
0.605
0.401
0.944
0.337
0.886
0.993
0.736
2.456
105
0.00
D172
LiKoaio DeAcca
0448
0.77
0.578
0.864
11477
3J881
10863
083333
020V
16000
(148)
4100
20111
Table 4: Per-component WOSAC metric results the test split of WOMD, representing likelihoods. Methods are ranked by composite metric on the 2024 Challenge scores; Closed-loop results within 1 % of the best are in bold (for models with 10Hz replan). Diffusion-based methods marked in blue.
8
2
S
(M) Cut-In
(M) Tailgater
M
L
(L) S-shape
Surrounding Traffic
Figure 10: Long-tail synthetic scenes gen- erated via control points either explicitly defined by manually defined (M) or LLM-generated (L) config. Magenta in- dicates generated motorcyclists/car agents.
Fully synthetic scene generation quality com- parison via scaling model parameters (5 + L) and in- creasing temporal resolution (patch size 8 1). Increas- ing compute by scaling either the model size or temporal resolution improves the overall realism.
Figure 11:
by preventing collisions, while scaling the model without hard constraints improves most realism metrics as the model learns to generate more realistic trajectories. The realism metrics only apply to trajectories and do not account for generated agent type and size distributions. We compare the gener- ated size distributions versus log distributions in Fig. 9 and find the marginal and joint distributions both closely track the logged distribution. We show more examples of diverse, unconstrained scene generation when conditioning the same global context in Appendix A.8 Fig. B
Constrained Scene Generation and Augmentation The controllability we possess in the scene generation process as product of our diffusion model design can be useful for targeted generation and augmentation of scenes. Fig. [0] we show qualitative results scenes with constrained agents generated either via manually defined configs or by few-shot prompted LLM. Extended qualitative results are listed Appendix A.7.3
4.3 Model Design Analysis and Ablation Studies
Scaling Analysis Given two options of scaling model compute, either by increasing transformer temporal resolution by decreasing temporal patch sizes, or increasing the number of model parameters, we investigate the performance of multiple transformer backbones: [Model Size X (Temporal Patch Size] = (L. M, S} x (8, 4, 2, 1}. We vary model size by jointly scaling the number of transformer layers, hidden dimensions, and attention heads (see Sec. A.6 of Appendix for details). We show quantitative results from this model scaling in Fig. Bland qualitative comparisons in Fig. 11 Increasing both temporal resolution and number of model parameters improves realism of the simulation.
Multi-task Compatibility We find that multitask co-training across BP, SceneGen and with random control masks improves performance compared to single-task, BP only model on the sim agent rollout task, notably reducing collision and offroad rates. We find that jointly learning multiple agent features (x, y, 2, 7, size, type) achieves on-par performance with pose-only (z, y, 2, 7) model.

---

Model Architecture Ablation As shown in Tab. replacing AdaLN-Zero conditioning with cross attention leads 7.99% decrease in realism performance, largely due to significantly higher collision rates and offroad rates. Removing the agent-wise spatial attention layer very significantly increases collision rate, as removes the mechanism for agents to learn a joint distribution.
Conclusion
We have introduced SceneDiffuser, scene-level diffusion prior designed for traffic simulation. SceneDiffuser combines scene initialization with scene rollout to provide a diffusion-basees approach closed-loop agent simulation that is efficient (through amortized autoregression) and controllable (through generalized hard constraints). We performed scaling and ablation studies and demonstrated model improvements with computational resources. On WOSAC, demonstrate competitive results with the leaderboard and state-of-the-art performance among diffusion methods.
Limitations While our amortized diffusion approach is, to our knowledge, the only and best performing closed-loop diffusion-basee agent model with competitive performance, we do not exceed current SOTA performance for other autoregressive models. We do not explicitly model validity masks and resort to logged validity this work. Future work looks to also model the validity mask.
Broader Impact This paper aims to improve AV technologies. With our work we aim to make AVs safer by providing more realistic and controllable simulations. The generative scene modeling techniques developed in this work could have broader social implications regarding generative media and content generation, which poses known social benefits as well as risks of misinformation.
Acknowledgments and Disclosure of Funding
No third-party funding received in direct support of this work. We thank Shimon Whiteson for his detailed for detailed feedback and also the anonymous reviewers. We would like to thank Reza Mahjourian, Rongbing Mu, Paul Mougin, and Nico Montali for offering consultation and feedback on evaluation metrics. We thank Kevin Murphy for his assistance developing the mathematical notation for the likelihood metrics. We thank Zhaoyi Wei and Carlton Downey for helpful discussions about the project. All the authors are employees Waymo LLC.
10

---

References
[1] Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blažej Osiński, Hugo Grimmet, and Peter Ondruska. SimNet: Learning reactive self-driving simulations from real-world observations. In ICRA, 2021.
[2] Tim Brooks, Bill Peebles, Connor Homes, Will DePue. Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman. Eric Luhman. Clarence Wing Yin Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL /ttpp://openai,.com wtt.://o//eeeeeeeeee resarchhs video www-eegeeeeaaenooeeee models as as-aaaswwoolo world- simulators
[3] Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. Con- trollable safety-critical closed-loop traffic simulation via guided diffusion, 2023.
[4] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[5] Kashyap Chitta, Daniel Dauner, and Andreas Geiger. Sledge: Synthesizing simulation environments for driving agents with generative models, 2024.
[6] Younwoo Choi, Ray Coden Mercurius, Soheil Mohamad Alizadeh Shabestary, and Amir Rasouli. Dice: Diverse diffusion model with scoring for trajectory prediction, 2023.
[7] Scott Etinger,, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R. Qi, Yin Zhou, Zoey Yang, Aurélien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV, 2021.
[8] Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolci Zhou, Traffiegen: Learning to generate diverse and realistic traffic scenarios. In ICRA, 2023.
[9] Zhiming Guo, Xing Gao, Jianlan Zhou, Xinyu Cai, and Botian Shi. SceneDM: Scene-level multi-agent trajectory generation with consistent diffusion models, 2023.
[10] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023.
[11] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022.
[12] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Floct. Video diffusion models. In NewrlPS, volume 35, pages (633-8646, 2022.
[13] Emiel Hoogehoom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. JCML, pages 13212-1322.. PMLR, 2023.
[14] Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Feráándzz Fisac. Versatile scene-consistent traffic scenario generation as optimization with diffusion, 2024.
[15] Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In ICRA, 2022.
[16] Andrew Jaegle, Sebastian Borgeaud, Jean- -aattttt Alayrac, Carl Doersch, Catalin Ionescu, David Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, for structured inputs & outputs. arXiv preprint arXiv:2 107,99955, 2021.
Perceiver io: A general architecture
[17] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. ICML, 2022.
[18] Chiyu "Max" Jiang, Andre Comman, Cheolho Park, Benjamin Sapp, Yin Zhou, and Dragomir Anguclov. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In CVPR, 2023.
[19] Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li. and Jun Zhu. Dpm-solverr+: Fast solver for guided sampling of diffuusin probabilistic models. arXiv preprint arXiv:2211.01095. 2022.
[20] Jack Lu, Kelvin Wong, Chris Zhang, Simon Suo, and Raquel Urtasun. Scenecontrol: Diffusion for controllable traffic scene generation. In ICRA, 2024.

---

[21] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 11461-11471, June 2022.
[22] Reza Mahjourian, Rongbing Mu, Valerii Likhosherstov, Paul Mougin, Xiukun Huang, Joao Messias, and Shimon Whiteson. Unigen: Unified modeling of initial agent states and trajectories for generating autonomous driving scenarios. In ICRA, 2024.
[23] Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nick Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, Brandyn White, and Dragomir Anguelov. The waymo open sim agents challenge. In Advances in Neural Information Processing Systems Track on Datasets and Benchmurks, 2023.
[24] Jiteng Mu, Michaël Gharbi, Richard Zhang, Eli Shechtman, Nuno Vasconcelos, Xiaolong Wang, and Taesung Park. Editable image elements for controllable synthesis. arXiv preprint arXiv:2404. 16029, 2024.
[25] Nigamaa Nayakanti, Rami AI-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.05844, 2022.
[26] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling. Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David J Weiss, Benjamin Sapp. Zhifeng Chen, and Jonathon Shlens. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In ICLR, 2022.
[27] Matthew Niedoba, Jonathan Lavington, Yunpeng Liu, Vasileios Lioutas, Justice Sefas, Xiaoxuan Liang, Dylan Green, Setarch Dabiri, Berend Zwartsenherg, Adam Scibior, and Frank Wond. diffusion-model of joint interactive navigation. In NeurlPS, 2023.
[28] Yotam Nitzan, Zongze Wu, Richard Zhang, Eli Shechtman, Daniel Cohen-Or, Taesung Park, and Michaël Gharbi. Lazy diffusion transformer for interactive image editing, 2024.
[29] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Val- carcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In ICLR, 2023.
[30] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195-4205. October 2023.
[31] Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Learning the language of driving scenarios. In ICLR, 2024.
[32] Ethan Pronovost, Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang, Andres Morales, Kai Wang. and Nick Roy. Scenario diffusion: Controllable driving scenario generation with diffusion. Advances in Neural Information Processing Systems, 2023.
[33] Ethan Pronovost, Kai Wang, and Nick Roy. Generating driving scenes with diffusion. In ICRA Workshop Scalable Autonomous Driving, June 2023.
[34] Cheng Qian, Di Xiu, and Minghao Tian. A simple yet effective method for simulating realistic multi-agent behaviors. Technical report, 2023.
[35] Davis Rempe, Jonah Philion, Leonidas J Guibas, Sanja Fidler, and Or Litany. Generating useful accident- prone driving scenarios via learned traffic prior. In CVPR, June 2022.
[36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffussio models with deep language understanding. Advances in neural information processing systems, 35:36479-36494 2022.
[37] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In The Tenth International Conference on Learning Representations, ICLR. OpenReview.net, 2022.
[38] Benjamin Sapp. Yuning Chai, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning, pages 86 99. PMLR, 2020.
12

---

[39] Ari Seff, Brian Cera, Dian Chen, Mason Ng. Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8579-8590, October 2023.
[40] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML, 2018
[41] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mur-: 1st place solution for 2022 waymo open dataset challenge motion prediction. arXiv preprint arXiv:2209. 10033, 2022.
[42] Shaoshuai Shi. Jiang. Dengxin Dai, Bernt Schiele. Mi+++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 46(5):3955-399 53751,9112220 2024.
[43] Uriel Singer. Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang. Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video- Text-to-video generation without text-video data. In ICLR, 2023.
[44] Shuo Sun, Zekai Gu, Tianchen Sun, Jiawei Sun, Chengran Yuan, Yuhang Han, Dongen Li, and Marcelo H Ang. Drivescenegen: Generating diverse and realistic driving scenarios from scratch. IEEE Robotics and Automation Letters, 2024.
[45] Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In CVPR, 2021.
[46] Shuhan Tan, Kelvin Wong. Shenlong Wang. Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traflic scenes. CVPR, June 2021.
[47] Shuhan Tan, Boris Ivanovie, Xinshuo Weng, Marco Pavone, and Philipp Krähenbühl. Language conditioned traffic generation. 7th Annual Conference on Robot Learning (CoRL), 2023.
[48] Balakrishnan Varadarajan, Ahmed Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Comman, Kan Chen, Bertrand Douillard, Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. arXiv preprint arXiv:2111.14973, 2021.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[50] Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. Nocturne: a scalable driving benchmark for bringing multi-agent learning step closer to the real world. In NewrlPS Datasets and Benchmarks Track, 2022.
[51] Yu Wang, Tiebiao Zhao, and Fan Yi. Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023. Technical report, Pegasus, 2023.
[52] Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: Bi-level imitation for traffic simulation. In ICRA, 2023.
[53] Chen Yang, Aaron Xuxiang Tian, Dong Chen, Tianyu Shi, and Arsalan Heydarian. Wedt: World-centric diffusion transformer for traffic scene generation, 2024.
[54] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. Trafficbots: Towards world models for autonomous driving simulation and motion prediction. In ICRA, 2023.
[55] Zihan Zhang, Richard Liu, Kfir Aberman, and Rana Hanocka. Tedi: cumporally-eentnghee diffusion for long-ter motion synthesis, 2023.
[56] Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language- guided traffic simulation via scene-level diffusion. In CoRL, 2023.
[57] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In ICRA, 2023.
13

---

A Appendix supplemental material
A.1 WOSAC Metrics
Suppose there are N 500k scenarios, each of length 80 steps, each containing S 128 agents (objects). For each scenario, we generate K 32 samples (conditioned on the true initial state), which is a of trajectories for each object for each time step. where each point in the trajectory is a D 4-dim vector recording location (ar, y, =) and orientation 0. Let all this generated data be denoted by x(1 : N, 1: A, K, T, D). Let the ground truth data be denoted x x1(1 N, 1 A', 1 T, 1 D). Below we discuss how to evaluate the likelihood of the true (test) dataset x under the distribution induced by the simulated dataset r.
(Note that we may have A' > A, since the ground truth (GT) can contain cars that enter the scene after the initial prefix used by the simulator; this is handled by defining a validity mask, v(1 N, : T, A'), which is set to 0 if we want to exclude a GT car from the evaluation, and is set to I otherwise.)
Rather than evaluating the realism of the full trajectories in the raw (x, y, z, 0) state space, WOSAC defines M 9 statisties (scalar quantities of interest) from each trajectory. Let Fy(x(i, :)) represent the set of statistics/features (of type derived from xxi, o, 1 : 7) by pooling over T, K. This is used compute histogram Pije(-) for the empirical distribution of Fy for scenario i. Let F2 (2=(( (i, t)) be the value of this statistic from the true trajectory for vehicle a at time t. Then we define the negative log likelihood to be
NLL a,,,,,,t))) = log Pija (F, (z" ai,
(3)
The j'th metric for scenario i is defined as
m(a, i.j) = cxp (-l (((,a)
1E oli, (cl.a,.))LL(.ttt..))
E
m(i.j) Σ m(a,i,j)
(4)
a(i,a) Σ v(i, a, t) is the number of valid points.
Finally an aggregated metric, used to rank entries, is computed as
M
score
Σ Σ (im((,
(5)
N'
1=
where 0 S wy <
The component metrics are defined as linear speed, linear acceleration magnitude, angular speed, an- gular acceleration magnitude, distance to nearest object, collisions, time-to-collision (TTC), distance to road edge, and road departures.
A.2 SceneGen Metries
We instead let (a(i.:)) represent the set of statistics/features (of type j) derived from x(1, 1 A K--H: T') by pooling over T,, A', K. This is used to compute histogram (.ç(.) (.) for the empirical distribution of F, for scenario i.
A.3 Additional Evaluation Details
Simulation step validity Due to the requirement of validity masks during inference, which is applied as an attention padding mask within the transformer, the model does not generate valid values for invalid steps. As the WOSAC challenge evaluates simulation agents for all steps, regardless of the step's 's validity, we use linear interpolation extrapolation to impute values for all invalid steps in our simulations for the final evaluation.
14

---

A.4 Additional Dataset Information
WOSAC uses the v1.2.0 release of WOMD, and we treat WOMD as a set D of scenarios where each scenario is history-future futtre pair. This dataset offers a large quantity of and shapes produced by state-of- of-the-att offboard perception system. We use WOMD's 9 second 10 Hz sequences (comprising H 11 observations from 1.1 seconds of history and 80 observations from seconds of future data), which contain object tracks at Hz and map data for the area covered by the sequence. Across the dataset splits, there exists 486,995 scenarios in train, 44,097 in validation, and 44,920 in test.
object behaviors
A.5 Additional Amortized Diffusion Algorithm Details
Warm up: At inference time, the rollout process is preceded by warm up step. The warm up step necessary for initializing& a buffer of future timesteps before any diffusion iterations take place. The warm up entails single iteration of a one-shot prediction process described in Algorithm Bai This process samples pure noise for some future steps and conditions the denoising process on the set of past steps.
Amortized autoregressive rollout: In Fig. we provide visual illustration of our amortized autoregressive rollout procedure. We operate the rollout procedure using buffer to track future steps in the trajectory. After the warm up, the future buffer contains T predicted steps with increasing noise level. Note that step T has very little noise applied. The future buffer in this state is denoised for single iteration using past steps condition the process. After a single iteration, the clean step at = is popped off of the buffer, and it is added to the past steps. Before the next iteration, step T = + is sampled from a pure noise distribution and is appended the end of the future buffer. The described rollout process can be repeated to generate trajectories of arbitrary length as clean steps are popped off the buffer.
A.6 Additional Implementation Details
Architecture Details: : For our base model, our scene encoder architecture uses 256 latent queries. Each scene token is 256-dimensional, with 4 transformer layers and 4 transformer heads, with transformer model dimension of 256. We train and run inference with all 128 agents.
Scaling Hyperparameters:
Small Model: Scene token dimension 128, Transformer layers, 128 Transformer model dimensions, 2 Transformer Heads.
Medium Model:
Scene token dimension 256, Transformer layers, 256 Transformer model dimen- sions, 4 Transformer Heads.
Scene token dimension 512, 8 Transformer layers, 512 Transformer model dimensions, 8 Transformer Heads.
Large Model:
Optimizer: We use the Adafactor optimizer [40]. with EMA (exponential moving average). We decay using Adam, with 3 = 0.9, decay adam 0.9999, weight decay of 0.01, and clip gradient norms to 1.0.
Training details Train batch size of 1024, and train for 1.2M steps. We select the most competitive model based on validation set performance, for which we perform final evaluation using the test set. We use an initial learning rate of 3 x 10-4. We use 16 diffusion sampling steps. When training. we mix the behavior prediction (BP) task with the scene generation task, with probability 0.5. The randomized control mask is applied both tasks.
Additional Hyperparameters To preprocess features, we use scaling constants of 1 for features I, y, z, and compute mean μ and standard deviation of features I, w, h.
30
We preprocess each agent feature f to produce normalized feature I' via l 2,0,
1-HL where:
p 4.5, Pw 2.0, Ph = 1.75, 14 = 0.5.
(6)
and
ou 0.8, 2.5,
Th = 0.6, 0x 0.5.
15

---

Perturb Step = 0/ (Log) Perturb Step 16 32 Perturb Step 24 Perturb Step 32 32
Figure 12: pog-perturbation via noising and denoising the original logs to different noise levels. An increasing level of noise added and then removed results in scenes more and more dissimilar from the original log, yet increasingly diverse. The scenes are realistic regardless of the perturbation noise level.
We scale by twice the std a values to allow sufficient dynamic range for high feature values for some channels.
A.7 Prompts used in Language-based few-shot Scene Generation
A.7.1 Prompt:
Prompt 1: Prompts used in Language-based few-shot Scene Generation.
You writing proto generate and control an agent 's behavior for autonomous vehicle (AV) simulation. I will give the follow 2 examples input and the generated proto Muttilgenthhh which will constrain the agent's position in either past, current, or future timestep. 5 timesteps equals 1 second, so for example 15 steps would equal 3 seconds. Given natural language description of the agent 's desired behavior, please generate the corresponding huttiAggetthuttiioosrraain
Very Important limitations:
valid for PAST time step 2. Only time_step_idx values in [0] are valid for CURRENT time step.
1. Only time_step_ide values in [0,8]
3. Only time_step_idx values in [9] are valid for FUTURE time step.
4. You may only use types PAT_CAR, POT NOT_MOTORCYCLIST, POT PEDESTRIAN to generate these examples
5. No agents should overlap each other the same step in the same time frame.
The following are 2 examples of natural language input and the output is text file that creates the corresponding constraint.
Example 1
Input: Generate constraints where motorcycle agent cuts in front of the AV coming from the right side at some time in the future. Output: rcr
agent_ cgent_constraints {
step_constraints {
relative_step conettainnta
time_frame: CURRENT
sime_step_idx:
lat_distance: 3.7 # Slightly right of ADV
long-distance: 0.0
agent_type: POT MOT_ROTTRTT
step coo__constrainss { calative_step_constree
16

---

time_frame: FUTURE
_tme_stepiidx: 30
lat_distance: 0.0
long_ dongadistanc: 3.0 # ahead of AV
agent_type: POT_MOTORCYCLIST
}
agent_ 'motorcycle_0-
scene_name, isinglo.notoreyyle..ctu..
Example 2
Input: Generate constraints where car agent is tailgating the AV by following behind it closely. Output: .c.
agent_constraing {
step_constraints {
relative_step_coute constrain
time_frame: PAST
time_step_idx:
lat_distance: 0.0
long_distance: -15.0
agent_type: PAT_CAR
}
step_constraints {
time_frame: CURRENT
time_step.idx:
lat_distance: 0.0
long-distance: -10.0
agent_type: POT_CAR
}
step constraits {
selatiivessee_constree
time_frame: FUTURE
time_step_idx: 30
lat_distance: 0.0
long_distance: -2.0
agent_type: PAT_CAR
agent_name: 'car_0'
tcene_namer
.cc
Please output just the MulbiAgen:hlllloonssttuaa in the response and leave any explanation the comments. Please double check the important limitations described before are met.
Now take deep breath and lets think step by step to write the proto given the following input:
S Shaped Maneuver: Create a motorcyclist agent that is doing a S shaped maneuver around the AV.
with 8 agents that surrounding the and traveling along for a few seconds. There are agents on the left lane of AV in a straight line and 4 agenta on the lane of AV in a straight line. 2 of the front agents at motorcyclists while the
Surrounding Traffic:
Create
17

---

rest are cars.
A.7.2 Scene Diffusion Constraint Protos Result:
Prompt 2: (Manual) Cut-in Constraint
agent_constraintt.
step_ constraints {
relacive.step_conttteee constraint {
time_frame: PAST
time_step_idx:
lat_distance: 3.0
long.distance: 0.0
agent_type: POT_CYCLIST
step.constaaints {
selative__tep_consee constraint {
time_frame: FUTURE
time_step_idx:
lat_distance: 0.0
long.distance: 0.0
agent_type: POT_CYCLIST
step_ constraits {
seatiive_step_conttrateee {
rime_frame. FUTURE
time_step_idx: 40
lat_distance: -3
long_distance: 10
agent_type: POC_CTCLIST
agent_name: ccut_in_00
Prompt 3: (Manual) Tailgate Constraint
agent_constraints {
conttrannss {
constraint {
time_frame: PAST
time_step_idx:
lat_distance:
long_distance: -20
agent_type: POT_CYCLIST
step constraits {
htteevvveeeee__cnattrriit {
time_frame FUTURE
time_step_idx: 20
lat_distance: 0.0
dingadistance: -10
agent_type: POT_CYCLIST }
step_constraints {
http:veee_tte_costtaaiii {
time_frame: FUTURE
time_step_idx: 40
lat lat_distance:
long.distance: -3
18

---

agent_type: POT_CYCLIST
agent_namm name: 'tailgater_agent_0"
Prompt 4: (LLM Generated) S-shaped Constraint
agent_constraing
step_ cospcoonttaints {
# Initial position slightly to the right of the AV
time_ teme_stte_ide idx 0)
{
time_frame: CURRENT
time_step_idx: 0
lat_distance: 2.0 Adjust this value as needed for initial lateral offset
distadistncee: 0.0
agent_type: POT_CYCLIST
constrantss
Move the motorcycle diagonally the left the future around 15 steps)
seletive__tep_coonrriin {
rime_frame: FUTURE
time_step_ide 15
lat_distance: -3.0 Adjust this value for desired leftward
movement
long.distance: 5.0 # Adjust this value for desired forward
movement
agent_type: POT_CYCLIST
step_ constraints {
Move the motorcycle diagonally back the right the future (around 30 steps)
relative_atep_com wtwpppcosttaanttttteoomm {
time_frame: FUTURE
time_step_idz 30
lat_distance: 2.0 Adjust this value for desired rightward
movement
long.distance: 10.0 # Adjust this value for desired forward
movement
agent_type: POT_CYCLIST
agent_name: 'motorcycle_0'
Prompt 5: (LLM Generated) Surrounding Traffic Constraint
agent_ const_const {
step_constraints {
relative_step_co constraint
itnnttrr {
time_frame: CURRENT
time_step_ids
lat_distance: -3.7
long.distance: 10.0
agent_type: POT
step_ constraints {
relative_step.cm colattieestto_coostraatt {
19

---

time_frame: FUTURE
rime_step_iet 25
lat_distance: -3.7
long.distance, 30.0
agent_type: POT_MOTORCYCLIST
agent_name:
'motorcycle_left_00
agent_constraints { costtranssss {
rolac:ve__tep__oostrraatt {
time_ CURRENT
time_step.idx: 0
lat_distance: -3.7
long_distance: 5.0
agent_type: PAT_CAR
constrannss {
sesativv_steepcoostraiie {
time_frame: FUTURE
sime_step_idet 25
lat_distance: -3.7
long_distance: 25.0
agent_type: PAT_CAR
}
agent_name: 'car_left_0"
agent_constraints
conptaonsss {
seneiive_steepcoosttaaee {
time_frame: CURRENT
time_stop_idx:
lat_distance: -3.7
long_distance: -5.0
agent_type, PAT_CAR
step_constraints {
selatiive_ste__oosttaa constraint {
time_ frame: FUTURE
time_ sime_step_idx: 25
lat_distance: -3.7
long long.distance: -1.0
agent_type: POT_CAR
agent_name: 'car_left_1'
}
agent_constraints
step_ constrannsssi {
senetive_stee_coostseeeee {
time,frame: CURRENT
time_step_idx: 0
lat_distance: -3.7
long_distance: -10.0
agent_type: PAT_CAR
Step_constraints
http:vve__tee__onssaaiiii {
sime.frame: FUTURE
time_stop_idx: 25
20

---

lat_distance: -3.7
Long-distanel -6.0
agent_type: PAT_CAR
agent_name: 'car 'car_eeft_2"
agent_constraints
cteptaconss {
satative_step_constte {
time_frame: CURRENT
time_step_idx:
lat_distance: 3.7
long,distance: 10.0
agent_type: POT MOT_MOTORC
step_ constrainss {
stteelvvesstepccostraa {
time_ fiamef FUTURE
time_step_idx: lat_distance: 3.7
long_distance: 30.0
agent_type: POT_MOTORCYCLIST
agent_name: 'motorcycleerigat_00
agent_constrainta
constrainssss {
relattiee_see__costraaae {
time_frame: CURRENT
time_step_idx: 0
lat_distance: 3.7
long_distance: 5.0
agent_type: PAT_CAR
}
telativv__ssppcoostraatt {
time_frame: FUTURE
time_step_idx: 25
lat_distance: 3.7
long.distance 25.0
agent_type: PAT_CAR
agent_name: 'car 'car_right_''
agent_constraints {
coop_constraints {
velative__teepccostreee {
time,frame: CURRENT
time_step_ide
lat_distance: 3.7
long_distance: -5.0
agent_type: PAT_CAR
s step_constraints {
cet.iveesstep.costtaiii {
time,frame: FUTURE
time_step_idx: 25
lat_anstance: 3.7
long_distance: -1.0
21

---

agent_type: POT_CAR
agent_name. 'car_right_1'
agent_constraints
step_constraing {
relattie__seep.onttraaiit {
time_frame: CURRENT
time_step_idx: 0
lat_distance: 3.7
long_distance: -10.0
agent_type: PAT_CAR
step_constraints
selaaire__teppcoosttraitt {
Lime_frame: FUTURE
time_step_idx: 25
lat_distance: 3.7
long_distance: -6.0
agent_type: PAT_CAR
}
agent_name 'car_ 'cah_right_2
A.7.3 Controllable Scenegen Results
Qualitative results showing one successful and one failed example of applying the control points to scene generation task with the protos are listed Appendix A.7.2 For measuring the success and failures of this scene generation task, we randomly selected 25 examples that were generated with each of the control protos and qualitatively determined success on 1) the new object does not overlap with any existing objects in the scene and 2) if the new object semantically behaves in the way intended by the control points. Otherwise, we considered it failure. Overall, we measured
success rate of 40/100.
22

---

Cut-in (left: success, right: failure)
Tailgate (left: success, right: failure)
23

---

S-shape (left:: success, right: failure)
Surrounding Traffic (left: success, right: failure)
C20
A.8
Scene Generation
We show additional unconditioned scene generation results Fig. 13
A.9 Generalized Hard Constraint Definitions
Non-collision Constraints ensure the boxes of generated agents do not overlap. We
define the potential field of agent a be rounded square potential da (I, y) (1- if [|( Na.y - yallla < 1.5 else O. We define clipcollision (x) arg min_ (EaeA Eo En' (a + iWa. Ya + j(a)) that minimizes the po- tential of each agent's 's corners against all other agents. (x. y) is defined in the normalized space.
Range Constraints limit certain feature aa within the range of dmii and dmax. In the context of Scene Generation for example, this can be used limit the length vehicle to an arbitrary range, e.g. between 7-9 meters. We have clip (xa) min(max a dmin), dmax).
24

---

Figure 13: Results of unconditioned scene generation for randomly selected road locations. For each example, show the ground truth log along with generated scenes.
Onroad Constraints ensure that the bounding boxes of specified generated agents stay road. We define the offroad potential of road graph polyline i to be ox(x, y) = (x - xx.2 + (y )² Wa(r, 0 else 0, where (x,, yi) is the closest point the road graph with respect (ar, y) and Wa(x, y) is the winding number of position (z, y) to polyline i, such that we only penalize a trajectory for going offroad. We only consider the closest road graph segment and only consider trajectories that are more than > 20% onroad. We define clip onioa (x) = arg min minneRG (1 8)).
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
accurately reflects the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
No theoretical results included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: Yes]
Justification: We describe in detail the model architecture, training details, and algorithmic pseudocode for our inference procedures.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes] and [No].
Justification: We do not plan to release code in the near future. However our methods are clearly described and our dataset is based on the Waymo Open Motion Dataset, which is already publicly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis..c)
http://guuides/oeeuubbmi.i.eeee oosssssoisssiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. httpsscccpbblcc/cccccccooo getggudddee//oeummiiiii SoreddeubmissionPPee Policy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [No]
Justification: not affordable train multiple models for error bars in our experiments, especially the Large models.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high computational cost it is infeasible to run multiple training runs of the large models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide number of model parameters as well as inference FLOPS count for our family of models in Figure 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttss:/neeurs httppeewwws....ccueeeeeeeeeeeeee http://p........eoeeeeaaaceeeaea
Answer: [Yes]
Justification: guidelines.
We have reviewed the Code of Ethics and confirm that we conform to the
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed Section 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We not releasing data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We accessed and utilized public datasets (the Waymo Open Dataset) in compliance with the licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA].
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "twOdBV2mYB", "reviewer": "Reviewer_XARz", "summary": "The paper introduces SceneDiffuser, a novel scene-level diffusion model designed to enhance traffic simulation for autonomous vehicle (AV) development. It presents a unified framework that addresses scene initialization, involving the generation of initial traffic layouts, and scene rollout, which includes the closed-loop simulation of agent behaviors. SceneDiffuser leverages diffusion models to learn realistic and multimodal agent distributions, focusing on controllability, realism maintenance in simulations, and inference efficiency. The model introduces amortized diffusion for simulation, reducing computational costs and mitigating closed-loop errors. Additionally, it enhances controllability through generalized hard constraints and language-based constrained scene generation using large language models (LLMs). The paper demonstrates SceneDiffuser's effectiveness in the Waymo Open Sim Agents Challenge, achieving top performance among diffusion models.", "questions": "How does SceneDiffuser handle scenarios with a high number of agents or complex traffic situations not seen in the training data?\n\nHow does the model ensure the diversity and representativeness of the generated traffic scenarios?", "limitations": "- **Model Validity:** The paper does not explicitly model validity masks, relying instead on logged validity, which could be a limitation for scenarios not covered in the logs.\n- **SOTA Comparison:** While the model performs well among diffusion models, it does not exceed the current state-of-the-art performance for other autoregressive models, suggesting a need for comparison and potential integration.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Innovative Approach:** SceneDiffuser's use of amortized diffusion for simulation rollout generation is a creative solution that significantly reduces the computational cost per step.\n- **Unified Framework:** The model's ability to handle both scene initialization and rollout in a unified framework is a notable strength, simplifying the simulation process.\n- **Controllability:** The introduction of generalized hard constraints and the use of LLMs for constraint-based scene generation offer a high degree of control over simulation scenarios.\n- **Performance:** Achieving top performance in the Waymo Open Sim Agents Challenge indicates that SceneDiffuser is effective in real-world applications.\n- **Scalability:** The model's performance improves with increased computational resources, showing that it can scale with available hardware.", "weaknesses": "Despite mitigation efforts, the paper acknowledges that closed-loop errors remain a challenge, indicating room for further improvement.\n\nWhile the model performs well among diffusion models, it does not exceed the current state-of-the-art performance for other autoregressive models, suggesting a need for comparison and potential integration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We address the comments below:\n\n> While the model performs well among diffusion models, it does not exceed the current state-of-the-art performance for other autoregressive models, suggesting a need for comparison and potential integration.\n\nPotential integration of diffusion models with autoregressive approaches is an interesting and promising direction. In fact, amortized diffusion also follows an autoregressive rollout schedule, making it possible to integrate with AR models. One interesting recent work ([Autoregressive Image Generation without Vector Quantization](https://arxiv.org/pdf/2406.11838) from Kaiming He’s team) also hints at this direction in combining the autoregressive models with diffusion models that operate in the continuous vector space. We acknowledge that this is an interesting and promising frontier, where we hope our work serves as one of the first explorations in this space.\n\n> How does SceneDiffuser handle scenarios with a high number of agents or complex traffic situations not seen in the training data? \n\nThe Waymo Open Motion Dataset (WOMD) is specifically mined for a high number of agents in the scene, with up to 128 agents per scene, accompanied by complex scenarios. By designing our transformer backbone to iterate through axial attention separately across agents and time, it reduces the complexity of scaling to more agents. See Figure 13 (appendix) for examples from the held-out validation set, containing some very dense and complex traffic scenarios generated by our model.\n\n> How does the model ensure the diversity and representativeness of the generated traffic scenarios?\n\nThanks for this question. We can try to answer from three angles:\n1. Diffusion models by design are known for being able to learn complex, diverse and multimodal distributions, which is also observed in other works (e.g., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion). Diffusion models sample from a bias-free Gaussian prior that enables generation of diverse outputs.\n\n2. We implicitly measure and capture the diversity and representativeness of the generated scenes in our measured metrics. Since the unconditional scenegen and sim agent metrics based on the WOSAC challenge measure for distributional realism (measured from 32 samples), precision and recall from the distributional realism metrics reward “representativeness” and “diversity” respectively.\n\n3. For a quantitative assessment of diversity and representativeness, see Figure 13 (appendix) for generated examples from our model containing some very dense and complex traffic scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "apP7zXLBba", "reviewer": "Reviewer_xCJf", "summary": "The paper proposes a method called SceneDiffuser, to generate multi-agent scenarios for autonomous driving and rollout the scenarios. The two tasks are unified in a model by formulating them as an inpainting task for the scene tensor. The diffusion model uses an amortized diffusion technique to align the diffusion steps with the physical prediction timestamp, enhancing the efficiency. Additionally, language-based agents can be inserted in the scene tensor and rule-based constraints are applied in the diffusion procedure. The scene rollout task is evaluated on the Waymo Sim Agent task and achieves comparable results with SOTA auto-regressive methods, and the scene generation task is validated on the Waymo Motion dataset to realize great realism.", "questions": "- Would amortized diffusion strategy have problems (such as different diffusion steps) on the boundary of the prediction window?\n- Is the predicted size for one specific agent not consistent during the prediction horizon?", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unifying the scene generation and simulation rollout in a single model is great and relatively new. As far as I know, some models can use post-processing to realize these two closely related tasks while there are no unified models previously.\n- The figures are very illustrative and can help understanding. The paper is easy to understand in an overall sense.\n- SceneDiffuser adopts the amortized diffusion strategy from the human animation literature. It eases the efficiency problem in simulation rollout.\n- The overall performance is great. Scalability is validated as well. Hard constraints are injected during diffusion, to solve the problem of realism.", "weaknesses": "I do not see critical problems, but there are a few things worth discussion and improvement.\n\n- Metrics. It lacks descriptions or some simple introductions about the metrics for those experiments. For example, the statement in Line 234 (\"different metrics buckets are aggregated per-scene instead of per-agent\") could be further elaborated for better readability. The scene generation task uses some different metrics, compared to previous scene generation methods, eg [8,a]. Detailed introductions and discussions on these issues would be ideal.\n- The layout and the order of the tables and figures, can be further improved. Figures and tables are not ordered by the reference order and are placed close to where they are referenced. I acknowledge that it could be difficult to perfectly place them considering the amount and different sizes. However, the current layout and order pose difficulties in reading the paper, making the reading process not smooth.\n- Though the paper focuses on the scene generation and simulation rollout, similar to previous related literature. If the generated scenes can be utilized to help motion prediction or even other downstream tasks, the impact of the paper would be broader.\n- There are some related works about scene generation not discussed, eg [a-c]. I am also wondering if those using WMOD could be compared quantitatively. This question also relates to the metrics.\n- The proposed method still can have realism problems, even though constraints are injected in the diffusion procedure. This internal drawback of diffusion methods could not be fully relieved.\n- Typos:\n  - Line 18: We demonstrate\n  - Line 112: Appendix A.2 should be Appendix A.4.\n \n[a] Language Conditioned Traffic Generation. CoRL 2023.  \n[b] DriveSceneGen: Generating Diverse and Realistic Driving Scenarios from Scratch. RA-L, 2024 (arXiv 2309).  \n[c] CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectories. arXiv 2403.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We address the comments below:\n\n>  Figures and tables are not ordered by the reference order and are placed close to where they are referenced.\n\nThank you for the suggestion, we will improve the layout in the camera-ready version.\n\n> Missing / additional citations.\n\nWe do cite Language Conditioned Traffic Generation, but the year was malformatted in the BibTeX entry, which we have addressed now. Thank you for suggesting the two other works, we have now added a reference in our Related Work.\n\n> Would amortized diffusion strategy have problems (such as different diffusion steps) on the boundary of the prediction window?\n\nGreat question. Due to the diffusion noise schedule being a linear ramp from 0 to 1 going from the current step to the boundary of the prediction window, the step at the prediction window is associated with maximum (full) noise. Therefore simply appending a random gaussian vector is sufficient for that step. As the rollout proceeds, we iteratively append a random gaussian vector at the end of each step to extend the simulation.\n\n> Is the predicted size for one specific agent not consistent during the prediction horizon?\n\nObject size is treated similarly as all other features such as position and yaw, therefore there is no hard constraint that enforces it to be constant. However, even in logged data the object sizes for each agent are in fact not strictly constant due to the existence of perception / detection noise. We believe that learning this small fluctuation of perception object features due to perception noise further improves the realism of the simulation.\n\n> Metrics. It lacks descriptions or some simple introductions about the metrics for those experiments. For example, the statement in Line 234 (\"different metrics buckets are aggregated per-scene instead of per-agent\") could be further elaborated for better readability. \n\nWe apologize that the metrics descriptions are not sufficiently detailed due to the submission page limit. We have now added a comprehensive technical description of the metrics to the appendix and **will add this to the camera-ready version** given the additional page allowance.\n\nThe metrics used in the unconditional scenegen task are minor variants for the metrics in the Waymo Open Sim Agents Challenge (WOSAC). The core idea of the WOSAC metrics is to measure the negative log likelihood (NLL) of the ground truth logged scene under the distribution from the generated samples. The NLL is computed over 9 measurements: kinematic metrics (linear speed, linear acceleration, angular speed, angular acceleration magnitude), object interaction metrics (distance to nearest object, collisions, time-to-collision), and map-based metrics (distance to road edge, and road departures). A weighted average over the NLLs across the 9 measurements is then computed as the final composite score. However, the NLLs in WOSAC are computed on a per-agent granularity. That means that each logged agent’s log likelihood is measured under the distribution of 32 samples in the predictions for the same agent. This can be done because there exists a one-to-one correspondence between each simulated agent and each logged agent (since they share the same history). However in unconditional scenegen, there does not exist a one-to-one mapping between each logged agent and each generated agent. Therefore when measuring the NLL, we flatten (num_agents, num_steps * num_samples) into (num_agents * num_steps * num_samples,) and compute the histograms per-scene by scrambling all agents and all timesteps into the same histogram.\n\n**WOSAC Metrics**: Suppose there are $N \\approx 500k$ scenarios, each of length $T=80$ steps, each containing $A \\leq 128$ agents (objects). For each scenario, we generate $K=32$ samples (conditioned on the true initial state), which is a set of  trajectories for each object for each time step, where each point in the trajectory is a $D=4$-dim vector recording  location $(x,y,z)$ and orientation $\\theta$. Let all this generated data be denoted by $x(1:N, 1:A, 1:K, 1:T, 1:D$). Let the ground truth data be denoted $x^*(1:N, 1:A', 1:T, 1:D$). Below we discuss how to evaluate the likelihood of the true (test) dataset $x^*$ under the  distribution induced by the simulated dataset $x$.\n\n(Note that we may have $A' > A$, since the ground truth (GT) can contain cars that enter the scene after the initial prefix used by the simulator; this is handled by defining a validity mask, $v(1:N, 1:T, 1:A')$, which is set to 0 if we want to exclude a GT car from the evaluation, and is set to 1 otherwise.)\n\nRather than  evaluating the realism of the full trajectories in the raw $(x,y,z,\\theta)$ state space, WOSAC defines M=9 statistics (scalar quantities of interest) from each trajectory. Let $F_j(x(i,a,:))$ represent the set of statistics/features (of type j) derived from  $x(i, a, 1:K, 1:T)$ by pooling over $T,K$. This is used to compute a histogram $p_{ija}(.)$ for the empirical distribution of $F_j$ for scenario i. Let $F_j(x^*(i,a,t))$ be the value of this statistic from the true trajectory $i$ for vehicle $a$ at time $t$ . Then we define the negative log likelihood to be\n$$\n NLL(i,a,t,j) = -\\log p_{ija}(F_j(x^*(i,a,t))\n$$\nThe j'th metric for scenario i is defined as \n\n$$\n\\begin{aligned}\n  m(a,i,j) &= \\exp\\Big(- [\\frac{1}{N(i,a)}] \\sum_t v(i,a,t) NLL(i,a,t,j) \\Big) \\\\\\\\\n   m(i,j) &= \\frac{1}{A} \\sum_a m(a,i,j) \\\\\\\\\n   N(i,a) &= \\sum_t v(i,a,t) \\text{ is the num. valid points}.\n\\end{aligned}\n$$\n\nFinally an aggregated metric to rank entries is computed as \n$$\n  score =\\frac{1}{N'} \\frac{1}{M} \\sum_{i=1}^{N'} \\sum_{j=1}^M w_j m(i,j)\n$$\nwhere $0 \\leq w_j \\leq 1$.\n\n**SceneGen Metrics**: We instead let $F_j(x(i,:))$ represent the set of statistics/features (of type j) derived from  $x(i, 1:A', 1:K, -H:T)$ by pooling over $T,A',K$. This is used to compute a histogram $p_{ij}(.)$ for the empirical distribution of $F_j$ for scenario i.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3eM419JJ9", "reviewer": "Reviewer_j5Jn", "summary": "The paper proposes a diffusion model data driven simulation of driving scenes. The model is able to both generate driving scenes and to perform closed loop simulation of these driving scenes. The problem is posed as an inpainting task in the scene tensor, which contains all the agents, time steps (past and future) and road user attributes/states. To avoid excessive compute when using the diffusion model in a receding horizon fashion (autoregressive rollout) the paper proposes to reuse the solution of the last iteration (similar to nonlinear MPC) and use a custom noiseing strategy that adds more noise to time steps further in the future. The resulting solution is competitive with other diffusion based methods at a fraction of the compute and reusing the previous trajectory as a warm start allows for more consistent trajectories over time.", "questions": "- How is it enforced that the size and type of a vehicle do not change over time?\n- Did you investigate taking several denoising steps in the rollout phase. In nonlinear MPC this can often improve the results if the computation load allows it and reduce the dependency on the previous trajectory.\n- Why is the offroad rate not lower, is it not possible to constrain the motion to onroad predictions?\n- What is the run time of the model in the rollout phase, time for one denoising step?\n- The WOMD setup has a horizon of 8s, in the rollout phase do you always predict the full 8 seconds or do you reduce the horizon length. If you keep the 8s horizon how do you deal with the horizon potentially leaving the map. Similarly if you would reduce the prediction horizon to about 4s how would this change the results?\n\nTypo:\n- L261: notedly - notebly\n- L18: We demonstrate of effectiveness - We demonstrate the effectiveness", "limitations": "Limitations have been adressed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The computational burden of diffusion policies is in my opinion one of their main drawbacks. The proposed amortized diffusion allows to reduce this burden while resulting in more consistent trajectories over time. \n\nBy formulating the problem as an inpainting problem the method is general and can tackle several interesting tasks.", "weaknesses": "- By relying on the previous solution, there seems to be a risk of getting stuck in local minima. Did you notice such situations?\n- Implementing hard constraints by projection seems risky, especially in the amortized rollout phase where only one denoising step is taken. I can imagine that for some types of constraints this works well but others will be problematic. Similar to optimization problems where bounds can be handled with clipping but more general constraints need more elaborate implementations (e.g. barrier functions in interior point methods). Did you notice that generalized hard constraints worked better or worse for some types of constraints?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We address the comments below:\n\n> By relying on the previous solution, there seems to be a risk of getting stuck in local minima. Did you notice such situations?\n\nThank you for this question. Even though we utilize the previous timestep’s solution, the previous predictions are progressively noised, with a higher noise the further out the time step is. This allows the trajectories to still vary to a great extent. While the immediate next step is only noised by 1/T (where T is the total length of the future horizon), it is reasonable to assume that the object cannot veer too far from its current position in one step to begin with. Our competitive simulation realism metrics is another indirect evidence that this is not a common occurrence.\n\n> Implementing hard constraints by projection seems risky, especially in the amortized rollout phase where only one denoising step is taken. I can imagine that for some types of constraints this works well but others will be problematic. Similar to optimization problems where bounds can be handled with clipping but more general constraints need more elaborate implementations (e.g. barrier functions in interior point methods). Did you notice that generalized hard constraints worked better or worse for some types of constraints? \n\nGreat thoughts. In our experiments we applied hard constraints in the unconditional SceneGen experiment which jointly denoises all past and future steps from the same noise level in a one-shot fashion, therefore these clipping constraints are applied across up to 32 steps of denoising. This would be interesting to try in amortized rollouts, and based on this comment it would make sense to start applying clipping constraints only on steps k-steps in the future, since future steps go through more iterations of denoising before being finalized. We found that the basis on which the hard constraints operate is important: a good constraint will modify a significant fraction of the scene tensor, or else the model effectively \"rejects\" the constraint on the next denoising step. For example, to correct for collisions we shift an agent's entire trajectory rather than shifting just the overlapping waypoints so as to maintain a more realistic (and non-colliding) trajectory for the diffusion process. We will further clarify these details in the final paper.\n\n> How is it enforced that the size and type of a vehicle do not change over time?\n\nObject size is treated similarly as all other features such as position and yaw, therefore there is no hard constraint that enforces it to be constant. However, even in logged data object sizes for each agent are in fact not strictly constant due to the existence of perception / detection noise. We believe that learning this small fluctuation of perception object features due to perception noise further improves the realism of the simulation.\n\n> Did you investigate taking several denoising steps in the rollout phase. In nonlinear MPC this can often improve the results if the computation load allows it and reduce the dependency on the previous trajectory.\n\nWe did not look into different denoising schedules across future timesteps and only looked into a linear noise schedule from 0 to 1 from the current to final time step to reduce simulation cost (single denoiser evaluation per simulation step). However, this is a good idea and very recent work ([Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://arxiv.org/pdf/2407.01392)) has looked into arbitrary per-step noise schedules with encouraging results.\n\n> Why is the offroad rate not lower, is it not possible to constrain the motion to onroad predictions?\n\nIn the WOMD dataset there are many vehicles that are located \"offroad\", such as in parking lots or driveways. Consequently the offroad metric measures a model's ability to produce such agents at the appropriate rate. SceneDiffuser learns to produce both on-road and offroad agents, but much headroom remains. We explored using an onroad constraint (see Appendix A.7) to force individual agents to be on-road or off-road, but we found this did not significantly improve the offroad metric. The problem is not that an on-road agent goes offroad; the problem is that SceneDiffuser produces offroad agents at the wrong rate relative to logs, which we have been successful at using more generic methods such as architecture scaling (see Table 1).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "khx58qOUJY", "reviewer": "Reviewer_bzy1", "summary": "This paper provides a diffusion-based method for traffic simulation, called SceneDiffuser. SceneDiffuser unifies simulation, including scene initialization and scene rollout for generating initial scene layouts and simulating closed-loop agent behavior. To accelerate the time-cost diffusion process, this paper introduces an amortized diffusion technique to accelerate the process with 16x fewer inference steps. The paper further introduces LLM for few-shot prompt engineering. For verification, the SceneDiffuser achieves good performance on Waymo Open Sim Agent Challenge, top performance on both open-loop and close-loop setting.", "questions": "Please see weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. I think it is good introducing diffusion mechanism to the topic of traffic simulation. Previous methods, like MTR++ or TRAJEGLISH typically use transformer architecture and formulate it as a regression or auto-regressive regression problem. \n\n2. The extension to LLM looks interesting to me.", "weaknesses": "1. I think the writing can be improved. \n\n    a. It is hard to understand Algorithm 1-3 given so many details and notations without explanation. \n\n    b. The role of Figure 4 is to illustrate the auto-regressive process, or not? It confuses me when reading the figure.\n\n    c. The amortized diffusion part is hard to understand.\n\n2. The motivation of the LLM extension is unclear to me. For me, it seems like an application of the SceneDiffuser, and has nothing to do with the improvements of SceneDiffuser to existing traffic simulators. \n\n    a. It would be interesting to see how LLM + SceneDiffuser helps autonomous driving.\n\n3. Missing experiments:\n\n    a. How about the proposed method compared to SceneTransformer [1], especially in terms of performance. For me, I think the two papers share some insights and should be compared fairly.\n\n    [1]. Scene Transformer: A unified architecture for predicting multiple agent trajectories", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We address the comments below:\n\n> I think the writing can be improved.\na. It is hard to understand Algorithm 1-3 given so many details and notations without explanation.\nb. The role of Figure 4 is to illustrate the auto-regressive process, or not? It confuses me when reading the figure.\nc. The amortized diffusion part is hard to understand.\n\nThank you for your feedback. With the additional 1 page of space in the camera-ready, we have added additional clarifying details regarding the algorithms by adding additional comments in the algorithm pseudocode and by correcting a typo in the notation. We note that we do have positive feedback from other reviewers about the figures such as “The figures are very illustrative and can help understanding” and “The paper is easy to understand in an overall sense” (Reviewer *xCJf*). If there is more detailed feedback for a specific part of Figure 4 or the algorithms that is difficult to understand, we would be happy to add more detail and clarifications.\n\nRegarding notation, the notations in Algorithm 1-3 are consistent with the notations defined in Sec 3.1 (Scene Diffusion Setup) as well as Sec 3.2 (Scene Rollout), lines 164 - 169. Each of the Algorithms 1-3 are kept in under 8 lines of pseudo-code. Regarding Figure 4 -- that is correct that it illustrates the improved autoregressive process in diffusion models, which we refer to as Amortized Diffusion, as stated in the figure caption. We created the schematic in Figure 4 to make it easier to understand Algorithm 3 which details the amortized diffusion process.\n\n> The motivation of the LLM extension is unclear to me. For me, it seems like an application of the SceneDiffuser, and has nothing to do with the improvements of SceneDiffuser to existing traffic simulators.\na. It would be interesting to see how LLM + SceneDiffuser helps autonomous driving.\n\nLLMs can be leveraged to simplify the scenario creating process using SceneDiffuser. While SceneDiffuser allows controllable inference by specifying the location, type, size etc of certain agents at certain waypoint steps and generating the scene in an inpainting-style, defining the controllable waypoints itself is non-trivial. Connecting the common sense world knowledge of LLMs to the SceneDiffuser via the proto interface in a few-shot approach, and experimentally validating its feasibility, is a non-trivial finding that is worth sharing in the paper, and helps with integrating the model with LLM interfaces.\n\n> Missing experiments:\na. How about the proposed method compared to SceneTransformer [1], especially in terms of performance. For me, I think the two papers share some insights and should be compared fairly.\n[1]. Scene Transformer: A unified architecture for predicting multiple agent trajectories\n\nThanks for the suggestion and reference. We have added a citation now to SceneTransformer in the introduction where we discuss behavior prediction methods. SceneDiffuser shares some similarities with SceneTransformer regarding using masking to direct the same model towards different tasks. However, SceneTransformer is only applied to the behavior prediction task (predicting motion at all future timesteps in a single model inference) and is not applied to either scene generation or closed-loop simulation, so we do not believe it is applicable for a direct task-level comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thorough and thoughtful comments. We are pleased to see that all reviewers are overall positive about the work, finding our proposed amortized diffusion rollout to be a **“creative solution”** (*XARz*) that alleviates **“one of the major drawbacks (of diffusion policies)”** (*j5Jn*). Reviewers also appreciate our unified framework that **“simplifies the simulation process”** (*XARz*) given that **“there are no unified models previously”** (*xCJf*). We are happy that reviewers find that our **“figures are very illustrative”** (*xCJf*) and the paper is **“easy to understand”** (*xCJf*).

We will address all technical questions in the per-reviewer rebuttal section. For comments regarding typos, figure / table / context layout arrangements and more detailed clarifications, we will address them in the final camera-ready paper if the work is accepted to the conference.

Meta-review: This work introduces SceneDiffuser, an efficient and controllable paradigm for generating simualtion in autonomous driving. The paper points out two critical challenges (controllability and inference efficiency) in the field and addressed them with solid experiments and novel methods (generalized hard constraints, amortized diffusion).

All reviewers reach concensus that the manuscript is ready for publication. The rebuttal did a good job addressing reviewers concerns. Good job! Please incorporate all comments and revise the draft for camera-ready.
Decision: Accept (poster)