Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion
Authors: Ethan Pronovost
Meghana Reddy Ganesina
Noureldin Hendy
Zeyu Wang
Andres Morales
Kai Wang
Nicholas Roy
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
(Other) Applications
(Other) Machine Learning Topics
TL;DR: We adapt latent diffusion to generate driving scenarios based on a map and scenario descriptors.

Abstract: Automated creation of synthetic traffic scenarios is a key part of scaling the safety validation of autonomous vehicles (AVs). In this paper, we propose Scenario Diffusion, a novel diffusion-based architecture for generating traffic scenarios that enables controllable scenario generation. We combine latent diffusion, object detection and trajectory regression to generate distributions of synthetic agent poses, orientations and trajectories simultaneously. This distribution is conditioned on the map and sets of tokens describing the desired scenario to provide additional control over the generated scenario. We show that our approach has sufficient expressive capacity to model diverse traffic patterns and generalizes to different geographical regions.

Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion
Ethan Pronovost" Meghana Reddy Ganesina
Noureldin Hendy Nicholas Roy
Zeyu Wang
Andres Morales
Kai Wang
Zoox
{epronovost, L nhendy, zewang, andres, kai, wnwoyrroooooo..o com
Abstract
Automated creation of synthetic traffic scenarios is key part of validating the safety of autonomous vehicles (AVs). In this paper, we propose Scenario Diffusion, a novel diffusion-basse architecture for generating traffic scenarios that enables controllable scenario generation. We combine latent diffusion, object detection and trajectory regression to generate distributions synthetic agent poses, orientations and trajectories simultaneously. To provide additional control over the generated scenario, this distribution is conditioned on a map and sets of tokens describing the desired scenario. We show that our approach has sufficient expressive capacity model diverse traffic patterns and generalizes to different geographical regions.
Introduction
Automated creation of synthetic traffic scenarios is key part of validating the safety of autonomous vehicles (AVs). To efficiently target rare and safety critical scenarios we would like to direct scenario generation to produce specific types of events. Prior methods for heuristically created scenarios tend to be of limited complexity and miss large number of possible real-world situations [6] 17). Recent works using deep learning models 6. are able to produce complex scenarios conditioned on a map region, but do not offer additional controls over the generation process. In this paper we propose Scenario Diffusion: novel architecture for generating realistic traffic scenarios that enables controllable scenario generation.
Our problem setting is generate set of bounding boxes with associated trajectories that describe the location and behavior of agents driving scenario. We accomplish this scenario generation using denoising diffusion generative model 212 27 29 We condition model both map image and a set of tokens describing the scenario. We leverage these tokens to provide a variable rate of control over the generated scenario, that the diffusion model generalize to complex scenes conditioned only on small set of tokens that control both a subset of the individual agents and global scene properties. Further, this variable rate of control allows to learn the model from partially tokenized training instances, substantially reducing the labelling requirements.
Motivated by the insight that the instantaneous position of each agent is inextricably linked to their behaviors, we combine latent diffusion [24), object detection, and trajectory regression to simultancously generate both oriented bounding boxes and trajectories, providing generative model of both the static placement of agents and their behaviors. We evaluate Scenario Diffusion at generating driving scenarios conditioned as well. Finally, we provide an analysis of the generalization capabilities of our model across geographical regions, showing that our diffusion-based approach has sufficient expressive capacity to model diverse traffic patterns.
only the map, and with additional conditioning tokens
Coorrepponding author
Work done during an internship at Zoox
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Lane Change
U-turn
hai
Futro
Scenario
Scenario 2
Scenario
Scenario 2
Driving scenarios generated using the map location and token features describing the two cireled vehicles. Vehicle bounding boxes are shown in green with their trajectories colored from pink in the past to blue in the future. The vehicle circled in red represents the AV, and the vehicle circled in orange a key interacting vehicle. The token features describe desired position, heading, size, speed, and trajectory curvature of the two circled agents, Using this description, the model generates scenarios that include two such agents and adds additional agents scenario. The two scenarios in each example are generated using the same conditional inputs and different initial diffusion noise samples. By using tokens, we do not need to fully define the circled vehicles' trajectories, and the model produces different possible scenarios that conform to the abstract description encoded by the token.
Figure I:
realistic manner fill the
2 Problem Setting
We aim to generate traffic scenarios conditioned on a map and an optional description of the scenario. Similar to prior works 12. 6 (32 we represent traffic scenarios abstractly with oriented bounding boxes and trajectories AV software is typically divided into perception and motion planning. The interface between these components describes agents with bounding boxes, historical trajectories. and additional attributes. In simulation this abstract representation is synthesized to evaluate the motion planning component in isolation, alleviating the need for expensive photorealistic simulation.
Fig. demonstrates the type controllability we aim to achieve. We condition the generative model by describing the state of two vehicles: the AV (circled in red) and one key interacting agent (circled orange). The model is able to generate diverse set of scenarios that include these two agents and additional ones. Notably, describing the two key agents does not require having observed such agents in the real world (e.g. by copying these two agents from an existing log). Instead, the model is able to generalize to generating novel agents based on the given description.
2.1 Scenario Representation
In this work we use topdown birds' eye view (BEV) representations similar to 2 32 for the inputs convolutional networks and sparse representations of oriented boxes and trajectories the final output of the decoder.
Topdown Representations The map context is represented multi-channel BEV image of shape Cm W that contains information about road geometry, regions of interest (e.g. driveways, crosswalks), and traffic control devices (e.g. stop signs, traffic lights). Agents are also represented in multi-channel BEV image of shape C₂ H x W that encodes each agent's bounding box, heading, and trajectory. Sec. 4.1 descrribs two different datasets with different rendering styles for m and æ that both yield good results with our model.
*Our simulations are abstract representations that are produced by perception systems. While photorealistic simulation would also allow the perception system to be tested in the loop. in this work we focus on scenario generation that tests planning and decision making for computational tractability.

---

3
Cer
Las
Fooue
One
2
AM New
up
m
Em
-20
171
Tokens
MLP
Takens
MLP
(a) Autoencoder training.
(b) Diffusion training.
(c) Inference.
Figure Model architectures for training and inference. (a) The autoencoder takes as input birds' eye view renderings of the map and entities and outputs sparse bounding box detections with trajectories. (b) The diffusion model is trained to denoise latent embeddings from the autoencoder using a birds' eye view rendering of the map and tokens describing the scene. (c) To perform inference, initial random noise is iteratively denoised and then decoded to generate bounding boxes and trajectories.
Sparse Representations We also represent the agents in the scene as a of oriented bounding boxes, with associated trajectories included as an additional feature vector per box. The oriented bounding box is defined by a center position, heading, length, and width and the agent trajectory as sequence of future and past poses spaced at uniform intervals in time, is common in motion with the bounding box pose representing the "current timestep". This sparse representation used the labels for the autoencoder's detection task and to compute tokens describing agents in the scenario.
forecasting 1 g 22 23 25
33
Method
As in [24], our diffusion model consists of two parts: autoencoder transform between the data space and a latent space and a diffusion model that operates in this latent space.
3.1 Scenario Autoencoder
The scene autoencoder is a variational autoencoder (VAR) [1] that learns to encode and decode sets of agents as shown in Fig. 2a The VAE trained with combination on the decoder output and KL divergence loss LKL on the latent embedding. The architecture is based on the autoencoder of [24), but we train the model to perform anchor-free one-to-one detection instead of pixelwise image reconstruction.
reconstruction loss fee
encoder E takes the BEV image of the agents in the scene x and outputs a latent embedding 2 {zz) of shape C2 x H' x W', where H' H/2/ and W" = W/2 for a downsampling factor capture the placement of agents, there was empirically no benefit to providing the map image input to the encoder. This latent embedding is given to the decoder DD along with the BEV map image m obtain the detection output y D(z, of shape Cy x H" x W" Because we output detections, H" and W" do not need to equal H and W. the original input shapes. In practice, having lower output resolution provides significant memory savings during training. The decoder downsamples the map image m, concatenates the latent embedding z, and then upsamples to produce the final output: D(2, m. = Dup (z, Daown (m)).
E N. Since the encoder designed
The decoder output y represents oriented bounding box detections and trajecto- ries. Each pixel of y represents one box proposal, with different channels representing the probability, position, heading, length, width, and trajectory of the box. Concretely, the first channel is sigmoid logit for the box probability. The second and third channels represent the cosine and sine of the box orientation. The next four channels represent the distances from the pixel center to the front, left, back, and right edges of the box in log scale. These seven channels define the oriented bounding box for the agent.
Detection Outputs
The trajectory describes the agent's pose from T seconds in the past to seconds in future at one second intervals. For each timestep E (1,..., T} of the future trajectory three channels of represent the delta between the pose and the pose / (with channels for x, y, and heading). These deltas are accumulated to produce the final trajectory starting from the center pose of the

---

oriented bounding box. The past trajectory is handled symmetrically (i.e. deltas between t and t+ for t {-T..... ..... -)). In total channel encodes the box probability, 6 channels encode the oriented bounding box, and 67 channels encode the trajectory.
Reconstruction Loss The reconstruction loss used to train the autoencoder adapts the one-to- one detection loss of [30] to accommodate oriented boxes. Given the simplicity of our detection problem (perfectly rendered rectangles with disambiguated orientations) we find that replacing the IoU loss (which is -on-differentiable with oriented boxes) with an L2 loss on the box vertices yields detection performance sufficient for training the diffusion model. Following [30,, each ground truth box is matched with one predicted box by selecting the predicted box that minimizes matching score between it and the ground truth box. This matching score is a weighted combination of classification score, L.1 loss on the box parameters, and an 1.2 loss on the box vertices. A classification loss encourages matched predicted boxes to have high probability and unmatched predicted boxes to have low probability, and regression losses encourage matched predicted boxes and trajectories to equal their corresponding ground truth box. We use a weighted combination of an LI loss on box parameters, L2 loss on box vertices, L2 loss on trajectory positions, and cosine loss on the trajectory headings. Further details on the reconstruction loss can be found Appendix B.2
3.2 Diffusion Fundamentals
Once the autoencoder is trained, a diffusion model is trained in the latent space of the autoencoder. This means that the diffusion model does not operate on bounding boxes and trajectories directly. We use the EDM diffusion algorithm [18 to train our model and generate novel scenarios.
Training Let å E(x) be the latent embedding obtained from trained and frozen encoder. Each example can also include associated conditional information (e.g., map information, tokens). We discuss options for this conditional data in following sections.
For each training example we sample a according to log-normal distribution (i.e. log ~ A (Pμ, P?) with hyperparameters Pj and Po). We then create a noisy sample = 2 + ac, where has the same shape as & and is sampled from the standard normal distribution.
We train the denoising model M(2; c, o by minimizing the reconstruction loss
8 - Cskip (o) 2
L= -oo (o) M(an(o)- ( -
(I)
Cout(ar)
where Cin (o), Cout Skip (o), and X(o) are all scalar functions of a as defined in [88 In practice
this model M is conditional Unet as in B. 12 28 This process is depicted in Fig. 2b
Given
noisy sample z, the denoised version can be estimated as
M M(z; 0) Cskip(o)
Cou((o) M((n(())
(2)
Inference To generate novel samples during inference, we start with an initial noisy sample z ~ N (0, o oax I). This sample is iteratively refined according to the reverse process ODE as described in [18. The gradient of the log probability distribution can be approximated as V, log R (M( - 2) /02. This simplifies the reverse process ODE to
dz -V logglogpp c, 2 M(2;c,
(3)
do
After integrating the sample = from a = 4max to a = 0 using Fuler integration, is passed through the decoder to obtain the generated boxes and trajectories D(z, m)) This procedure is depicted in Fig. [2c] During inference we keep the generated boxes with probability above a fixed threshold chosen by optimizing a metric that compares generated and ground truth boxes (described in Sec. [4.]).
We then filter any overlapping boxes, keeping the highest probability box of those that overlap.
3.3 Diffusion Conditioning
Conditional Unet models have two primary ways to provide the conditional information e: concate- nating to the noisy sample that is input to the denoising model and applying cross attention over set of tokens inside the denoising model. We explore both options in this work.

---

3.3.1 Map Conditioning
One method include conditional data in the denoising model is via conditional image that gets concatenated with the noisy sample imput to the denoising model [24]. In this work we use the BEV map image described in Sec. [a1 as the conditional image. The map image m is first processed Se1.2211aa by separate encoder &m to downsample it to the same pixel dimension as z. The encoded map Em (m) is concatenated with in the channel dimension before being input the denoising Unet M.
3.3.2 Token Conditioning
We can also condition diffusion models on a set of tokens via cross-attention mechanism + within the denoising model [24]. In image generation these tokens often come from text embeddings of a caption for the image [26]. While we do not have datasets of driving scenarios fully annotated with text captions, we can still use the concept of conditional tokens that describe aspects of the scenario. It is exactly these tokens that provide controllability over scenario generation. In this work we explore two types of tokens: agent tokens and global scene tokens.
Agent Tokens Agent tokens are tokens that describe single agent. In this work use procedurally generated tokens (i.e. tokens that can be computed from available data in the dataset), so are able to provide token for every agent. However that is not requirement for this approach; if using human-labeled tokens it would be possible to train a diffusion model while only having partially labeled scenarios. The token feature vector for each agent expresses various quantities that might want associated with that agent, such as the agent's 's position, speed, length, or motion profile. In this work the meaning of the token feature vector fixed for given experiment. One potential direction for future work is to have a single model support multiple types of agent token feature embeddings (e.g. allow the user specify the length for one agent and the speed for another).
To enable the model to generate agents not described by tokens we use a partial tokenization strategy. During training we achieve this by sampling token mask probability Pmask and masking out agent tokens with this probability. More information on this can be found in Sec. 4.3.T
Global Scene Tokens Global scene tokens encode scalar quantities that allow us to control global properties of the entire scenario. In this work consider single global scene token to represent Pmask- which encodes information about the number of agents to be generated relative to the number of tokens provided. During inference the value represented by this global scene token be modulated to control the number of agents in the generated scenarios. By providing no agent tokens and setting Pmaak = 1 in the global scene token we are able generate scenarios conditioned on only the map from the same model.
Processing Each type of token is processed by separate MLP to get final embeddings in a common latent space, to be used keys and values during cross-attention inside the denoising model. At different levels of the Unet each pixel from the intermediate image representation is used as query, and the agent and global scene tokens are used as the keys and values. This architecture can support any number of tokens. We refer the reader to [24] for more details on the cross-attention mechanism.
Experiments
4.1 Experimental Setup
Datasets We use two datasets in this work. The Argoverse 2 motion forecasting dataset [36 contains 250,000 driving scenarios over 6 geographical regions representing 763 hours of total driving. We use the provided training and validation splits. More information about this dataset can be found in Appendix A.I
The second dataset is an internal dataset containing 6 million real world driving scenarios from Las Vegas (LV), Seattle (SEA). San Francisco (SF), and the campus of the Stanford Linear Accelerator Center (SLAC). Training and validation data are not separated by location, so given validation example may have training examples from the same location at other times. More information about this dataset can be found in Appendix A.2
5

---

Table 1: Quality metrics for generated scenarios on Argoverse.
MMD Positions (1)
MMD
MMD Velocity (U)
Traj On
Lane Heading
Method
Headings (U)
Drivable
Difference
Ground Truth Log
0.900 (0.000)
0.203 (0.000)
Random Log Selection
(0.001)
0.386 (0.001)
0.104 (0.001)
U.407 (0.001)
1.497 (0.04)
TraflicGen [6] Placement Model
0.113 (0.002)
0.124 (0.003)
0.054 (0.002)
Scenario Diffusion (No Map)
0.142 (0.006)
0.352 (0.012)
0.091 (0.007)
0.411 (0.009)
1,504 (0.004)
Scenario Diffusion
0.093 (0.006)
0.108 (0.007)
0.055 (0.003)
0.895 (0.008)
0.271 (0.017)
For both datasets, the scenes are centered the autonomous vehicle. As the autonomous vehicle is treated as just another vehicle during training, the model almost always places a vehicle in the center of the scene during inference. We use second time horizon for both the future and the past trajectory (i.e. the bounding box represents t = O and the trajectory goes from t = -2 to f = 2). This time window is sufficient to populate the agent history for many motion forecasting methods
P 22
31 38 40).
Training See Appendix B and for details of training the autoencoder and Appendix C for details of training the diffusion model.
Inference We use 80% as the threshold for the generated box probabilities. See Appendix D.2 for the ablation experiment used select this threshold. We note that the performance of Scenario Diffusion is relatively robust to choices of probability threshold between 50% and 90%.
Metrics We consider metries to compare the generated and data distributions and measure the quality of generated scenarios. One popular metric for comparing two distributions using samples the maximum mean discrepancy (MMD) (9] While this metric is typically defined in terms of two distributions, in practice we compute this metric using samples drawn from the two distributions. Given two sets A and B generated by sampling from two distributions and some kernel k, the maximum mean discrepancy is defined as
MMD BM
Σ A1.99E1
k(a1, (2)
Σ bg.
k(b1,,22 b2)
2
Σ
k(a, (4)
A
BB
||
aEA,BEB
Similar 16 32], we use a Gaussian kernel l and apply this metric to the sets of agent center positions, heading unit vectors, and velocities, all in IR?
To measure the quality of generated trajectories compute the fraction of waypoints that fall within the drivable area, averaged over all generated agents and all 5 trajectory waypoints. For each predicted pose along the trajectory we also compute the minimum angle difference (in radians) between the pose heading and the heading of all lanes at that location. These metrics do not have a clear optimal value (trajectories in the Argoverse dataset sometimes leave the drivable area and aren't perfectly aligned with the lane tangent); we compare the metric between the dataset and generated scenarios.
Baselines As simple heuristic baseline implement Random Log Selection, which takes the agents from a random sample the training dataset (ignoring map information). We also report the trajectory metrics the ground truth validation samples themselves.
To compare our method previous autoregressive approaches we adapt TrafficGen 6 to the Argoverse 2 [36] dataset. See Appendix E for more details on the modifications needed.
4.2 Map-Conditioned Scenario Generation
We first evaluate Scenario Diffusion in generating scenarios conditioned only map data, the problem setting explored in prior works 2 6 32]. In Tab. we report MMD metrics that compare generated and ground truth scenes and trajectory metrics that measure how well the trajectories conform to the map. We evaluate Scenario Diffusion models with and without the map context m, and compare against the baselines described above.
Scenario Diffusion outperforms other methods in the MMD metric for position and heading, and performs equally with TrafficGen in velocity. In the trajectory metrics, Scenario Diffusion produces similar values as those of the ground truth logs. As expected, Scenario Diffusion without map context performs similarly to random log selection, both methods ignore the map context.

---

Global Secne Token 0%
Global Scene Taken = 50%
Global Scene Token 90%
Figure 3: Generated Argoverse scenarios conditioned on the same map image and three agent tokens with different values for the global scene token. The generated agents corresponding to the three agent tokens are circled in orange. When the global scene token is set to non-zero values, the model reconstructs the three agents described by tokens and adds additional agents to fill in the scene. Trajectories are omitted from this figure for clarity.
Table 2: Agent token matching metrics on Argoverse
Token Mask Prohability Pazsk
0%
10%
30%
50%
70%
90%
Agent token match rate
0.97 (0.01)
0.97 (0.01)
0.97 (0.01)
0.97 (0.01)
(0.01)
0.96 (0.02)
Number of additional generated agents
0.4 (0.3)
3.0 (0.8)
5.5 (0.9)
(0.9)
10.4 (0.8)
12.6 (U.7)
4.3
Token Conditioning
We next explore applications of token conditioning in Scenario Diffusion. Fig. demonstrates how tokens can be used to generate scenarios with a specific type of interaction between the AV and another agent. Our approach is able to generate these interactions without having to fully specify the box and trajectory for the agent (e.g. by hand or by copying an agent from an existing log). By using more abstract features in the agent tokens we allow the model to generate different scenarios that meet the given description (e.g. we do not prescribe the specific trajectory of the u-turning agent).
The flexibility of tokens provides variable degrees of control over the generated scenarios. In this work we demonstrate models that use a variety of features in the agent tokens, such as position, heading, length, and speed. More details on the exact token features can be found in Appendix C.3
4.3.1 Partial Tokenization
It is crucial that the learned model be robust the presence or absence of user-provided tokens. To ensure this, during training we omit or "mask" supervision tokens so that the model learns to fill in additional agents in the scenario. During training we select Pmask such that 40% of the time we keep all tokens (i.e. Pmask = 0) and the other 60% of the time sample Pmask Beta 1) For each sample, the chosen masking probability is bucketed into one of ten bins ([O, 0. 10.1, 0.2), etc.) and embedded one-hot vector define the global scene token. This global scene token and the agent tokens that remain after masking are used to condition the diffusion model as described in Sec. 3.3.2
During inference can modulate the value represented by the global scene token to control how many additional agents the model generates beyond those described by agent tokens. Fig. B shows scenarios generated by the model using fixed map, three agent tokens, and the global scene token. All scenarios contain the three agents described by the tokens. As the value set in the global scene token increases, the model infers that there additional agents not described by tokens. This approach allows a user to specify certain agents while having the model fill in the remainder of the scenario. Using this formulation by setting the global scene token to 100% and providing no additional agent tokens.
are able emulate a model conditioned only the map image
We first evaluate the model's ability to use the token information by matching generated agents to tokens. We take samples from the Argoverse validation dataset, compute agent tokens from the ground truth scene, and apply fixed amount of agent token masking. We generate scenes conditioned on these tokens and compute a one-to-one matching between generated agents and agent tokens using the Hungarian algorithm position into bins of width 1.56m, and 1.56 2 2.2) and 0.2 radians. In Tab. 2 we report the
(20). We only consider pairs that are within 2.2 meters (we discretize
"Simpler choices such as sampling Pmask from uniform distribution over [0. 1J yields similar results when fully trained. Empirically find the chosen mixture distribution yields faster convergence.
7

---

Final Speed: m/s
Final Speed: m/s
Patt
Figure 4: Generated Argoverse scenarios conditioned the same map image and one agent token with different values for the final speed (2 seconds in the future). The agent corresponding to the agent token is circled in orange. When the final speed is zero the model generates cross traffic agents, and when the final speed is non-zero the horizontal traffic has right of way. In both cases the current pose and speed of the agent token are the same; the only difference is the future speed. Scenario Diffusion is able to infer how the future behavior described in the conditional inputs should impact the placement of additional agents.
Table Agent token speed metrics on Argoverse
Agent token features
Reconstruction metrics
Curren pose
Current speed
Final speed
Agent token match rate (t)
Current speed MAE (L)
Final speed MAE (U)
Extents
0.971 (0.006)
1.43 (0.17)
1.57 (0.17)
(0.003)
0.49 (0.01)
(0.02)
0.989 (0.007)
0.49 (0.02)
0.58 (0.03)
agent token match rate (what percentage of agent tokens were matched to corresponding generated agent) and the number of additional generated agents per scene (those unmatched to an agent token). As the value of Pmask increases the agent token match rate stays at 96-97% while the average number of additional agents increases roughly linearly, showing that the model is able to follow both agent and global scene tokens.
4.3.2 Token Controllability
To evaluate how well the model uses other information in the agent tokens we compare models that use different agent token features. We run inference conditioned on agent tokens computed from the ground truth sample as in Sec. 4.3.1 For these experiments we use no token dropout (i.e. Pmask 0). We perform the same matching between generated and ground truth agents and report the agent token match rate in the previous section. For each matched pair, compute the error of the current speed and final (i.e. = 2) speed. In Tab. 3 we report the mean absolute error (MAE) over all matched pairs, again showing the mean and standard deviation of this metric across multiple models trained with different initial random seeds. As described in Appendix C.3) the speed features are discretized into bins of width 2 m/s, these models are not expected to achieve perfect reconstruction.
Adding current speed to the agent tokens significantly decreases the MAE for current speed and somewhat decreases it for final speed (as the current and final speed are correlated). Adding final speed feature to the agent tokens further reduces the MAE for final speed. Adding these additional agent features does not significantly impact the agent token match rate.
4.3.3 Relationship Between Agents
One of the key contributions of our work is to generate bounding boxes and trajectories simultaneously for all agents. Doing so allows the generated outputs to capture not just static snapshot of the scene but also the behavior various agents. This joint inference of the placement and behavior of agents is particularly important for controlled scenario generation using agent tokens. The behavior described by agent tokens influences where other agents can be placed. Approaches that separate initial placement and behavior into two models not able to perform this joint reasoning.
Fig. shows scenarios generated using one agent token, with features for the position, heading, extents, current speed, and final speed (i.e. at +2s). Both cases use the same position, heading, extents, and current speed (set to m/s); the only difference the final speed. For this vehicle stopped at an intersection, its future behavior implies which lanes have right-of-way through the intersection. When
8

---

Evaluated on:
SEA Scene
SF Scene
Trained on:
LV Model
SEA Model
SF Model
LV Model
SEA Model
SF Model
Poll
Figure 5: Comparison of scenarios for the internal dataset generated by models trained on only one urban region. While best results are obtained by using a model trained on data from the region of interest, many scenarios generated by models trained in other regions are plausible for validation.
Evaluated
LV Scene
SLAC Scene
Trained on:
LV Model
SLAC Model
Full Model
LV Model
SLAC Model
Full Model
Figure 6: Comparison of scenarios for the internal dataset generated by models trained on data from distinct regions. SLAC a suburban research campus, notably different from the other three regions. The model trained only on SLAC produces notably worse results when applied to other regions, and the model trained without data from SLAC fails to pick up on unique aspects of the campus road network. The model trained on all regions is able to generate high quality scenarios in both regions.
the future speed is 0 m/s (resp., 8 m/s) the model generates additional agents traveling vertically (resp., horizontally).
4.4 Generalization Across Geographic Region
To evaluate the ability to generalize across regions, we train models on data from each region of the internal dataset separately. We then evaluate these models and models trained on the full dataset on validation splits for each region. Results are shown in Tab. Each entry is average across 5 experiments with different random seeds, with the standard deviation reported in parentheses.
In each region, models trained on that region tend to outperform models trained on other regions. The models trained on the full dataset come close to the region-specialized models. This suggests that there are unique aspects of each region, and that the full model has sufficient capacity to capture this diversity. SLAC is the most distinct from the other regions and shows the largest generalization gap. Additional metrics can be found in Appendix D33 they follow the same pattern as in Tab. 4
Fig. 5 shows examples of models generalizing across the three metropolitan regions (LV, SEA, and SF). The model trained on San Francisco data is able to produce reasonable scenes in Seattle, and vice versa. This suggests the potential to produce validation data for new regions before large volumes of driving logs in the new regions have been collected.
Generalizing dissimilar regions is significantly harder. Fig. shows examples of generalization between LV (a major metropolitan area) and SLAC (a suburban academic campus). Not surprisingly, models trained each of these two regions performs worse when applied to the other. For example, the LV model does not generate realistic parked vehicles in the parking lots from SLAC and the
Table 4: Generalization across regions
MMD" Positions (L)
Traj On Drivable (1)
Model
LV Secnes
SEA Scenes
SF Seenes
SLAC Scenes
LV Scenes
SEA Socnes
SP Scenes
SLAC Scenos
LV
0.077 (0.004)
(0.013)
0.126 (0.013)
0.204
0.984 (0.001)
0.959
0.961 (0.005)
(0.009)
SEA
0.108 (0.007)
0.071 (0.044)
0.080 (0.005)
0,141 (0.007)
0.965 (0.004)
0.972 (0.002)
0.976 (0.004)
0.928 (0.010)
0.1 (0.006)
(0.006)
(0.002)
0.141 (0.008)
0.978 (0.006)
0.978 (0.005)
(0.001)
(0.005)
SLAC
0.209 (0.025)
0.225 (0.020)
0.193 (0.019)
0.053 (0.002)
0.893 (0.031)
0.936 (0.017)
0.950 (0.011)
0.969 (0.001)
Full
0.083 (0.006)
(0.013)
0.055 (0.008)
(0.009)
0.983 (0.003)
0.971 (0.006)
(0.003)
(0.003)

---

SLAC model generates vehicles facing against the lane direction at locations in LV. Fortunately, the model trained on the full dataset is able to produce high quality scenarios both regions.
Related Works
A number of prior works have used heuristics fixed grammars to generate driving scenes [3 7 [6 Such approaches are limited their ability to generate complex and diverse driving scenes. SceneGen 321 and TrafficGen [6] use autoregressive approach, adding agents one time. TrafficGen adds a second stage where motion forecasting model [33] is used to generate trajectories for the boxes. SimNet [2 uses conditional generative adversarial network [13] to generate BEV occupancy image. Heuristic post-processing is applied to identify connected components and fit bounding boxes to them. A second agent-centric model is then used generate trajectories for these agents over time. In contrast, our method directly produces bounding boxes and trajectories simultaneously for all agents in end-to-end differentiable architecture. Our approach also does constrain vehicles to be in lane, as in TrafficGen and several heuristic approaches.
A number of works generate future trajectories for already-existing agents in simulation [14 31 37 39 [40]. Some focus on specifically producing adversarial future trajectories given agents' initial states 4 10 35]. These approaches complementary to our work, and can be used to extend trajectories beyond what is output by Scenario Diffusion.
Conclusion
In this paper, we demonstrate a novel technique for using diffusion learn to generate scenarios of dynamic agents moving through an environment for the purposes of testing an autonomous vehicle's ability to navigate through that environment and plan according to those agents. We have shown that our technique leads to models that not only appropriately capture the desired distributions of scenarios and agent trajectories, but allow scenario generation be controlled target specific types of scenarios. The fact that diffusion allows declarative information such as tokens to be combined with models learned from data creates the potential for new models that combine other forms of symbolic knowledge such as traffic rules, physical constraints, and common sense reasoning.
Limitations: More work is required show broader generalization. The training data assumes specific model of perception in the form of bounding boxes and trajectory models. Additionally, we restricted the agent models on-road vehicle models. While we do not anticipate significant chal- lenges in applying this model to other forms of agents such as pedestrians, we have not incorporated those agents in the research described here. simulation this approach may need be extended to iteratively generate agents over a larger region as the AV navigates through the environment.
This paper focuses developing models to improve self-driving There are many positive and negative aspects to the development of self-driving cars that depend as much the system-wide design and regulatory aspects these aspects depend on the technical capabilities. However, the focus on simulation in this paper should partially reduce the risks of deploying self-driving cars by providing more effective and systematic coverage of testing scenarios.
Broader Impact:
technologies.
Acknowledgements: We would like to thank Gary Linscott, Jake Ware, and Yan Chang for helpful feedback on the paper; Allan Zelener and Chris Song for discussions object detection; Peter Schleede for discussions on diffusion. We also thank the NeurlPS anonymous reviewers, area chair, and program chairs.
10

---

References
[1] Elmira Amirloo, Amir Rasouli, Peter Lakner, Mohsen Rohani, and Jun Luo. LatentForeer: Ageti- Transformer- -Bscd Interaction Modeling and Trajectory Prediction. 2022. A.Rlhhttss///rxiv.rrr
abs/2203. 01880
[2] Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blazej Osinski, Hugo Grimmett, and Peter Ondruska. SimNet: Learning Reactive deliidriving Simulations from Real-world Ohservations. ICRA, 2021. uttp/ttt:://rrttt org/ abaaaaaal555 2200..10... 10.322
[3] Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. NeurIPS, 2021. URL http://///// org/ abs/2105. 05233
[4] Wenhao Ding, Baiming Chen, Minjun Xu, and Ding Zhao. Learning to Collide: An Adaptive Safety-Critd Critical Scenarios Generating Method. IROS, 2020. URL U htttps://aaxiii org/ abs 2003. 01197
[5] Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: An Open Urban Driving Simulator. CoRL, 2017. URL http:///arxd..orrrab///111.... wttp 03938
[6] Lan Feng, Quanyi Li, Zhenghao Peng. Shuhan Tan, and Bolei Zhou. TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios. ICRA, 2023. http:////// org/ hwwgg//b///// 2t1///21.0000.0 006009066699966
[7] Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example- Synthesis of 3d Object Arrangements. ACM Transactions on Graphics (TOG), 31,
[8] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling. ICLR, 2022. URL
attps://arxie /ttpss///arxx..or///b///212/ www.orgg///b//aooooeee 06607
[9] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch. Bernhard Schölkopf. and Alexander Smola. A Kernel Two-Sample Test. Journal Machine Learning Research, 13/25):723-773, 2012. URL
http:://jmlr.or//ppppp. wttp:o/pppp.p1///////o http://////eetetaaaee// htmll
[10] Niklas Hanselmann, Katrin Renz, Kashyap Chitta, Apratim Bhattacharyya, and Andreas Geiger. King: Generating Safety-Critice Driving Scenarios for Robust Imitation via Kinematics Gradients. ECCV, 2022.
URL (tRL/htps::///tt //axxiv org/ abs/ abs//244. 2204. 13683
[11] Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta- VAE: Learning Basic Visual Concepts with Constrained Variational Framework. ICLR, 2017. UtLlttttss://opeerreeeo wwwpeeewwweeetoooo/dd//////rrr
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS, 2020. UtL/tttss:/ //arxiv...ri////aa org/ 200g//bb//006... 2006. 11239
[13] Joey Hong, Beajamin Sapp, and James Philbin. Rules of the Road: Predicting Driving Behavior with Convolutional Model of Semantic Interactions. CVPR, 2019. URL http://arxiv, org/ abs/a 08945
[14] Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci. Brandyn White, and Shimon Whiteson. Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation. ICRA, 2022. URL https://///:: org/ abs/ 22055. 03195
[15] Phillip Isola, Lun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-Image Translation with Condi- tional Adversarial Networks. CVPR, 2017. URL
07004
[16] Gregory Izatt and Russ Tedrake. Generative Modeling of Environments with Scene Grammars and Varia- tional Inference. JCRA, 2020. URL htttttttt///oooo www.rroopss.s.....oooaoeeeaaeeee wwt.c.alaa..aiiii.u.. http://oppp...allaaaaaaaadddrrtt wtwiccsseeeettereeeeeeee cetppp/pcccceeeeeepp....c.c publicc
papers/Izatt20 .dff
[17] Amlan Kar, Aayush Prakash, Ming- Yu Liu, Eric Cameracei, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-Sim: Learning to Generate Synthetic Datasets. ICCV, 2019.
http:://aa org/ abs/ 1/bs/19900 1904. 11621
[18] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion- Beruuddon-BBaddd Generative Models. NeurlPS, 2022. URL http: http:://arxx //arxiv org hbtp://ba./2...... 00364
[19] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR, 2015. URL https: / //trxivvvvv////////////e////rtl org/ hbs///b///et /oss//111,.. 6980

---

[20] H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2 (1-2):8399,, 1955.
[21] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. ICLR, 2019. URL http: //arxiv. www.orrr//////a/eeeeeeeeeeeeeaee 05101
[22]
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, and Benjamin Sapp. Wayformer: Motion Forecasting via Simple & Efffcientt Attention Networks, ICRA, 2023. URL
httpa: attpa://arxile orgg httg//ba/22000 05844
[23] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan. Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roclofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp, Zhifeng Chen, and Jonathon Shlens. Scene Transformer: A unified architecture for predicting multiple agent trajectories. ICLR, 2022. URLhttp:
08417
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. Resolueso Image Synthesis with Latent Diffusion Models. CVPR, 2022. https:/::://ant http://wxxxvvi.g////// aba/ 2212
10752
[25] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Mueti-Agent Generative Trajectory Forecasting With Heterogenenus Data for Control. RCCV, 2020. URL. http
//arxiv. wwwpi//.rrg/b/bbebeeeeee 2001 .00093
[26] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image- Text Pairs. Data Centric Al NeurlPS Workshop, 2021. URL
abs/2111 /hss/111.02114
[27] Jascha Sohl-Dickstein Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Noncquilibrium Thermodynamies. ICML, 2015. URL http://arxiv org/ abs 03585
[28]
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. ICLR, 2021.
URL https://arxiy, www.aiiiw...gaaaae.eoo 2010 02502
[29] Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR, 2021. URL.
attp://aaritt http:///aaiiii.ooaaaaa//aaaa//// 2011, 13456
[30] Peize Sun, Yi Jiang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. What Makes for End-to-End Object Detection? ICML,, 2021. http:/sss//aar....rrr/a///////// 05780
[31] Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors. CVPR, 2021. URL https://aaax owg. 2201 06557
[32] Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. SceneGen: Learning Generate Realistie Traffic Scenes. CVPR, 2021. URL http::ttt/txa...oooooo org/ abs/2101.0650 065441
[33] Balakrishnan Varadarajan, Ahmed Hefnny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Comman, Kan Chen, Bertrand Douillard, Chi-Pang Lam, Dragomir Anguelov, and Benjamin Sapp. MultiPath++:: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction. ICRA,
2022. htttss:tts://// h/arxivv////x/v.///e///a org/ abs///1....11111 14973
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. NeurIPS, 2017. URL http:://axxii.... www.axwww.....reeeoo org/abs/
03762)
[35] Jingkang Wang, Ava Pun, James Tu. Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun, AdvSim: Generating Safety- Critical Scenarios for Self-Driving Vehicles. CVPR, 2021. URL htttss://axxii org/ /t1. 2ttp//61110000999
[36] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lamhert, Jagject Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnttt, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next Generation Datasets for Self-driving Perception and Forecasting. NeurIPS, 2021. URL httpss///axii
00493
[37] Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone, Bits: Bi-level Imitation for Traffic Simulation. ICRA, 2023. httph:tt/:/eee http:///////xvee/aa http///rrrr.rrrra//////// wttp:r//gaabb...aaa//////// 12403
12

---

[38] Ye Yuan, Xinshuo Weng. Yanglan Ou, and Kris Kitani. Agentformer: Awent- Transformers for Socio-Temporaa Multi-Agent Forecasting. ICCV, 2021. URLhtts:://ann http:://axivvvv.. www.a//.og/bb///.///.00O 14023
[39] Qichao Zhang, Yinfeng Gao, Yikang Zhang, Youtian Guo. Dawei Ding, Yunpeng Wang. Peng Sun, and Dongbin Zhao. TrajGen: Generating Realistic and Diverse Trajectories With Reactive and Feasible Agent Behaviors Autonomous Driving. IEEE Transactions Intelligent Transportation Systems. 23(12): 2022. URL https:://aaaat org/
16792
[40] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided Conditional Diffusion for Controllable Traffic Simulation. ICRA, 2023, URL
http://arxiv. org/ abs/
wttpaa/s.../.e/////////aa.a..eea 2210 17366
13

---

A
B
D
E
F
G
A
B
"
E
n
F
Figure 7: Two examples from Argoverse 2. Images A and B are concatenated in the channel dimension to obtain the 6 channel map image m. Image C shows the current location of entity bounding boxes colored by object type with a line to indicate box orientation. Images D-G show the time delta renderings A-2, A-1- A1, A2, respectively. C-G are concatenated in the channel dimension to obtain the 15 channel entity topdown image Image shows the same example using the plotting style used to show generated scenarios.
A Datasets
A.1 Argoverse 2
The Argoverse motion forecasting dataset (36) contains 250,000 driving scenarios, each 11 seconds long. These scenarios cover 6 geographical regions and represent 763 total hours of driving.
To generate inputs for Scenario Diffusion we take all 4s windows (for seconds render an input example for each. To do this rendering we use the provided visualization library to plot a BEV image of the map and add an additional image showing the lane center lines colored by the lane direction. For the entity layers we use the provided visualization library to plot vehicles and buses as rectangles of two different shades of blue. To disambiguate the orientation of the boxes we also render line from the center of each box to show the heading.
seconds) and
As the argoverse dataset does include information on agent dimensions, we use constants for each object type (i.e. vehicle and bus) as in the argoverse visualization library. These constants are also used define the ground truth boxes used to train the autoencoder.
To represent agents' trajectories in the topdown image, we render images for each time delta along the trajectory. For E 2- 1,,0,1,2) let p. for whether the agent was observed at the given timestep. We compute four position deltas and corresponding masks:
(=) p! be the agent's position and m, mask pp
Q- = m-1 A m- A-2= - -P2
A_ Pa P-1
mg A m-11
(5)
A = P1 - Po
an = m1 A mo
A₂ = Pi
02 = m2 A mi
14

---

Figure 8: Example topdown inputs for the internal dataset. The left image is the first three channels of the map image m. The middle image is the first three channels of the topdown entity image x which fill each bounding box with ones, the cosine of the heading, and the sine of the heading, respectively. The right image shows the corresponding figure using the ground truth boxes and trajectories from the example.
For each i {-2, 1, we add channels where each agent's bounding box from 0 is filled with ag, clip (a /30 + 0.5, 0, ).. and clip | (v) /30 0.5, 0,1)
Scenes centered the autonomous vehicle, which is treated as just another vehicle, and cover area of 100m x 100m. Examples of the topdown inputs m and x for this dataset can be seen in Fig. 7
A.2 Internal Dataset
The internal dataset contains 6 million real world driving scenarios from Las Vegas (LV), Seattle (SEA), San Francisco (SF), and the campus of the Stanford Linear Accelerator Center (SLAC). Note that Las Vegas, Seattle, and San Francisco represent dense urban settings, while SLAC a suburban academic campus. Training and validation data are separated by location, so a given validation example may have training examples from the same location at other times. Scenarios are sampled every 0.5 seconds from real-world driving logs.
Map information comes from a pre-defined high definition map and vehicle tracks are detected by an on-vehicle perception system. We filter these tracks to only include bounding boxes that overlap with drivable areas in the map (e.g. roads, public parking lots, driveways).
Scenes are centered on the autonomous vehicle, which is treated as just another vehicle during training, and cover an area of 100m x 100m. Examples of scenes from this dataset can be seen in figure Fig. g
15

---

Legend
Downssmple
Dewnsmaple
block
Downsample
32 block
256 po
129 px
E
D
Dowmample
Downsample
block
Dovnsample
block
Upsample
32 pa
64 px
Figure 9: Model architecture for the scenario autoencoder. The architecture used by [24] is modified include additional conditional image input (the map image m) the decoder D.
26
Conor
pp Mack
Dowmanpk
16 Mack
Donssmpp
s back
Upaanppc
pé Meck
Upaample
pi Mock
Legend
Em M
Tokens
Figure 10: Model architecture for the scenario diffusion model. The denoising Unet M combines the noisy latent sample z, the encoded map information & (m²) and the tokens. During training is generated by running the frozen encoder obtain {x) and adding noise (see Fig. 2b).
B Scenario Autoencoder Details
B.1
Architecture Details
The architecture for the scenario autoencoder is based 24 and is shown in Fig. 9 We modify the architecture to add the conditional image m as an input to the decoder D(2, m). The map image is encoded using the same architecture as & (but different weights) to obtain latent embedding with the same pixel dimension. We also truncate the upsampling stage of the decoder to output box proposals 64 pixel resolution. The same architecture is used for both the Argoverse and internal datasets.
The 256 pixel and 128 pixel blocks use 64 channels, and the 64 pixel and 32 pixel blocks use 128 channels. We use one Resnet block per resolution.
B.2 Reconstruction Loss
To compute the reconstruction loss we first match predicted box to each ground truth box and then apply loss terms between matched pairs. The losses and matching the same cost functions with different weights.
The classification loss is binary cross-entropy loss between the predicted box's logit and the probability of the ground truth box (defined to be 1). The coordinate loss applies an L1 loss on the 6 parameters that define the box shape (the cosine and sine of the box heading and the log of distance the front, left, back, and right of the box). The vertex loss applies an L2 loss on the positions of the four corners of the box. The trajectory position loss applies an L2 loss on the positions the trajectory. The trajectory heading loss applies a cosine loss on the headings of the trajectory.
16

---

To match a predicted box to given ground truth box we select the predicted box that minimizes weighted combination the classification, coordinate. and vertex losses. The classification term is up-weighted by a factor of 4 over the other two terms.
We apply the five loss functions the matched pairs. For this loss the classification term is up-weighted by a factor of 20 over the other four terms.
To balance between different motion profiles we categorize each ground truth trajectory stationary, straight, or turning. To better balance between these three categories we weight the trajectory losses for each category by 0.1, 0.3, and 4, respectively.
We also use a binary cross- enssopy loss on the unmatched predicted boxes have probability As there are many more such negative predictions, we down- down-weight this loss term. Negative boxes within 3 meters of a ground truth box are down-weighted by factor 0.2, and negative boxes more than 3 meters from ground truth box are down-weighted by factor of 0.002.
B.3 Training Details
The scenario autoencoder trained using the Adam optimizer [9),, initial learning rate of le-4, and weight decay of le-4. The learning rate is decreased by factor of 10 when the validation loss plateaus. Gradients are clipped to have norm less than or equal to 1. The KL regularization loss 4KL weighted by 0.1 compared to the reconstruction loss described above. We use 2 RTX A6000 GPUs with batch size of 16 per GPU.
C Scenario Diffusion Details
C.1 Architecture Details
The architecture for the scenario diffusion model is also based on [24] and shown in Fig. TO An encoder En, with the same architecture as with different weights process the conditional map image to the same latent dimension. The noisy latent embedding and the encoded map information &m (m) are concatenated form the input to the denoising Unet M. In the lower resolution blocks of M, the cross attention layers attend to the conditional tokens. The same architecture is used for both the Argoverse and internal datasets.
The 32 pixel, 16 pixel, and 8 pixel blocks of M use 64. 128, and 256 channels, respectively. Each resolution block features a single Resnet block. Cross attention is performed with 8 attention heads. The embedded map information Em(m) uses 64 channels.
C.2 Training Details
We use the EDM training algorithm Pa. 0.5,, and Pa = 1. See Table of 18 for more information (note that P. and Pa are referred as Pimaan and Std that table).
18
We set min 0.02, 0max 20, data 0.5, p=
The diffusion model is trained with an AdamW optimizer (21), an initial learning rate of 3e-4, and weight decay of le-5. The learning rate is decreased by factor of 10 when the validation loss plateaus. We use 4 RTX A6000 GPUs with batch size of 64 per GPU.
C.3
Token Details
In this work we consider the following possible agent token features:
Current position relative to the scene center
Current position relative to the scene center
Current heading
Box length
Box width
Current speed
17

---

Table Agent token features used for figures.
Figure
Current Pose
Extents
Current Speed
Final Speed
Heading to Final Position
Fig
Fig.
5
Fig.
V
Fig
Fig.
13
Fig.
Final speed (at +2s)
Relative heading from the position the f position
given model will use subset of these features for its agent tokens. For brevity we use "current pose" to refer to the trio of current position, current y position, and current heading and "extents" to refer to the pair of box length and box width.
Each agent token feature is processed by per-feature embedding function; the embeddings of all features are concatenated to get the final token feature vector.
The x and y positions are each embedded using a one-hot vector for 64 bins over the extent of the scene (100m). The current heading is encoded with two channels containing the cosine and sine of the heading. The length and width are encoded with one channel containing the loglp of the value. Current and final speed encoded using a one-hot vector for 10 bins over the interval [0, 20] in m/s. Speeds above 20 m/s are clipped to 20 m/s. The relative heading is encoded using a one-hot vector for 10 bins over the interval 0.7.7.077
Tab. lists the agent token features used for the various figures shown this paper.
D Additional Experiments
D.1 Autoencoder Ablation Experiments
One of the key hyperparameters for Scenario Diffusion is the latent embedding dimension, the number of channels in 2. Tab. shows the results of ablation experiments training diffusion models with different autoencoders using different latent dimensions. We find the latent embedding dimension used for experiments in the main paper (4) performs slightly better on MMD metrics and achieves trajectory metrics slighly closer to those of ground truth logs. However, the difference is not statistically significant given the standard deviation across diffusion models trained with different random initial seeds.
D.2 Diffusion Ablation Experiments
We begin by measuring the effect of changing the probability threshold used during inference. To do so, we repeat the procedure described in Sec. [.3.1 of sampling scenarios from the Argoverse validation dataset and computing agent tokens from the ground truth agents in the scene. Using Pmask = O, we expect the generated agents to match the ground truth agents. We perform the same matching algorithm described in Sec. 4.3.1 Treating the matching binary outcome,
Table 6: Ablation experiments on the autoencoder latent embedding dimension.
Latent Embedding
MMD
MMD?
MMD
Traj On
Lane Heading
Dimension
(L)
Headings (1)
Velocity (L)
Drivable
Difference
2
0.096 (0.01 l)
0.111 (0.009)
0.058 (0.007)
0.886 (0.019)
0.283 (0.032)
4
0.093 (0.006)
0.108 (0.007)
0.055 (0.003)
0.895 (0.008)
0.271 (0.017)
8
0.099 (0.009)
0.116 (0.010)
0.059 (0.005)
0.892 (0.010)
0.282 (0.018)
16
(0.007)
(0.009)
(0.004)
(0.016)
(0.035)
Ground Truth Log
0.900 (0.000)
0.203 (0.000)
18

---

Table 7: Ablation experiments measuring the effect of changing the box probability threshold.
Probahility Threshold
Precision
Recall
F1
0.05
0.695 (0,086)
0.981 (0.005)
0.811 (0.067)
0.10
0.886 (0.013)
0.981 (0.005)
0.931 (0.009)
0.15
0,906 (0.012)
0,980 (0.005)
0.942 (0.009)
0.20
0.917 (0.01))
0,980 (0.005)
0.948 (0.008)
0.25
(0.011)
0.980 (0.005)
0.952 (0.008)
0.30
0,932 (0.011)
0.979 (0.005)
0.955 (0.008)
0.35
(U.011)
(0.016)
0,958 (0.008)
0.40
(0.011)
0.979 (0.006)
0,960 (0.008)
0.45
0.947 (0.011)
0.978 (0.006)
0.962 (0.008)
0.50
0,951 (0.010)
0.978 (0.006)
0.964 (0.008)
0.55
0,955 (0.010)
0.977 (0.006)
0.966 (0.008)
0.60
0,959 (0.010)
0976 (0.006)
0.968 (0.008)
0.65
0,963 (0.010)
0.975 (0.006)
0.969 (0.008)
(0.010)
(0.006)
0,970 (0.008)
0.75
0,971 (0:010)
0.972 (0.006)
0.971 (0:007)
0.80
0.974 (U.010)
0.968 (0.006)
0.971 (0.007)
0.85
0,977 (0.010)
0.963 (0.006)
0.970 (0.007)
0.90
0.979 (0.010)
0.954 (0.006)
0.966 (0.007)
0.95
0,981 (0.009)
0.940 (0.006)
0.960 (0.007)
Table 8: Ablation experiments on the diffusion architecture. The values used in the main experiments
are bolded.
Ablation Variable
Ablation Value
MMD
MMD Headings (1)
Num Agents EMD (4)
Traj Off
Positions (4)
Drivable (t)
0.076 (0.009)
0.091 (0.007)
(.0
0.025 (0.003)
Number Rosnet Blocks
2
0.075 (0.006)
0.089 (0.006)
(1.2)
0.022 (0.002)
0.074 (0.004)
0.090 (0.005)
(0.9)
0.022 (0.003)
0.072 (0.005)
0.086 (0.004)
(..((1.2)
0.024 (0.002)
S. Output
Dimension
16
0.071 (0.006)
0.085 (0.006)
(1.4)
0.023 (0.003)
32
0.076 (0.009)
0.090 (0.008)
(.19(
0.023 (0.003)
0.073 (0.003)
0.089 (0.004)
(0.8)
0.026 (0.002)
128
0,077 (0.007)
0 (0.006)
5136(
0.024 (0.004)
Attention
32px, 16px, Spx
0,078 (0.007)
0.097 (0.005)
$138
0.025 (0.005)
16px, 8 px
0.076 (0.006)
0.090 (0.005)
(.14
0.023 (0.002)
Levels
0.078 (0.007)
0.093 (0.007)
(1..).
0.025 (0.004)
can define each matched pair of ground truth and generated agent to be "true positive" (TP), ground truth agents that are not matched to generated agents as "false negatives" (FN), and generated agents that are matched to ground truth agents as "false positives" (FP). From these numbers, TP/ (TP + FP), recall = TP/ (TP + FN), and F1 = 2 precision recall/ (precision + recall). Note that this definition of "recall" is the "Agent token match rate" and "false positives" are the "additional generated agents" from Tab. 2 Precision makes sense to measure in this context because we fix Pmask 0, so the number of additional generated agents should be close to O.
we can compute precision
These results are shown in Tab. 7 reporting the mean and standard deviation across multiple models trained with different random initial seeds. We observe that precision increases and recall decreases as the probability threshold increases. The mean F1 score is maximized with a probability threshold of 75% or 80%. From this, we select 80% as the threshold to use for most experiments. However, we note that the differences in Fl score are relatively small for probability thresholds between 40% and 95%.
Tab. shows the results of ablation experiments several key hyperparameters of the diffusion model architecture using the internal dataset. The values used in the experiments for the main paper are bolded. In general, we find that increasing the model size does not significantly improve the available metrics.
D.3 Cross Region Generalization
Tab. shows additional metrics from the evaluation described in Sec. 4.4 These metrics follow the same pattern as Tab. 4 This table introduces a new metrie: the earth movers' distance (EMD) on the number of generated agents. This metric measures how much the probability mass needs to move to
19

---

Table 9: Generalization across regions
MMD' Heading (1)
EMD for Nurbeer of Agents (L)
Model
LV Scenes
SEA Scenes
SF Scenes
SLAC Scenes
LV Scenes
SEA Scenes
SP Scenes
SLAC Scenes
LV
0.091 (0.003)
(0.008)
(0.012)
0.265 (0.028)
4.2
(1.4).4)
(3.5
12.3 (1.5)
SEA
0.177 (0.01 0.075 (0.002)
0.137 (0.007)
0.222 (0.010)
(2.1)
1.9 (0.6)
(1.())4)
9.3 (0.9)
0.160 (0.007) 0.144 (0.004) 0.048 (0.001)
0.197 (0.005)
9.2
2.6)
3.1 (0.8)
(0.9)
SLAC
0.375 (0.035) 0.360 (0.029)
0.341 (0.023)
0,073 (0.003)
(0.8)
11.3 (0.5)
18.0 (0.6)
28.77.7
(0.009)
(0.023)
(0.013)
(0.012)
(12.1.8)
1.6 (1.3)
(1.9)
31.7).
Global Scene Token 0%
Global Scene Token
2 Agent
Tokens
Agent
Tokens
Figure 11: Generated scenarios for the internal dataset. All scenarios are conditioned on two agent tokens; the bottom row of scenarios also uses third agent token. Agents described by the tokens are circled, with the agent circled in orange representing the AV and other agents circled in red. The model generates variety of trajectories how agent tokens can be used to generate driving scenarios to test specific category of interactions (e.g. vehicles pulling front of the AV) while providing diversity within that category.
the agents described by the tokens. This demonstrates
transform from one probability distribution another. In this context, the distributions in question are ID histograms of the number of agents per scene, measured over the ground truth validation scenes and generated scenes using the same map locations.
D.4 Qualitative Results
Fig. IT shows scenarios generated using different numbers of agent tokens and different values for the global scene token. Fig. 12 shows adversarial scenarios in junction created using two agent tokens. These examples illustrate how token controllability allows users to create specific types of scenarios, while letting the model figure out most of the details.
Fig. 13 and Fig. [4] show two more examples of modifying the final speed of agent tokens while keeping all other input features the same. This can be performed with any number of agent tokens.
Fig. 15 takes a ground truth scene from the Argoverse validation dataset, computes agent tokens from two vehicles of interest, and generates novel scenes using these tokens. This process allows users to sample modified versions of a given log to see how the placement of other agents impacts the AV behavior.
Figure 12: Generated scenarios for the internal dataset showing an agent nudging into the AV's lane while junction. All scenarios are conditioned on the same two agent tokens; the corresponding agents are circled, with the agent representing the AV in orange and the other agent in red. In these examples the global scene token is set to 50%. By testing many such scenarios we can find the conditions that result in undesirable driving behavior.
20

---

Final Speed:
Final Speed: 7
V Fure
Pot
Figure 13: Generated Argoverse scenes conditioned on one agent token, changing only the final speed token feature. The behavior described by the agent token impacts nearby generated agents.
Final Speed:
Final Spoed:
Pbc Pulazz
Figure 14: Generated Argoverse scenes conditioned on two agent tokens, changing only their final speed. Both agent tokens are set to use the same final speed. The behavior described by the agent tokens impact nearby generated agents.
E
TrafficGen
Several modifications were made TrafficGen 66 accommodate the Argoverse 2 dataset [36]. Argoverse does not contain traffic light information, so such features are not used in the map context. Argoverse also does not include the length and width of agent boxes. In the provided plotting code fixed constants are used for different agent types. We include the "car" and "bus" agent types for training TrafficGen, and update the size prediction head perform 2-way classification instead of outputting a Gaussian mixture model. To match Scenario Diffusion, we set the model to only generate vehicles in the 100m region around the AV.
Fig. 16 shows scenes generated by our implementation of TrafficGen.
Real World
Log
Generated Argoverse scenes using two key agents from real world log. The top left scene is the original log; the other nine scenes are generated using the same map location and two agent tokens corresponding the two agents circled in orange. The model is able produce diverse driving scenes incorporating the two key agents.
Figure 15:
21

---

Figure 16: Generated Argoverse scenes from implementation of TrafficGen. The blue lines show the velocity extended second into the future.
22


Reviews:
[{"review_id": "wTzUFOr7p0", "reviewer": "Reviewer_cvm4", "summary": "This paper presents a conditional latent diffusion model (LDM) for generating oriented BEV vehicle bounding boxes, each associated with a 4-second trajectory (2s past, 2s future) at a 1s temporal resolution. The autoencoder component of this LDM uses an object detection architecture similar to CenterNet, with rasterized agent inputs. The diffusion model is conditionable on the local map, a global traffic density parameter, and an optional per-agent attribute descriptor. The model is validated on an internal proprietary dataset and Argoverse 2 via a series of ablation studies.", "questions": "1. Would it be possible to include one heuristic and one autoregressive approach (as mentioned in Section 5) for the Argoverse dataset as additional columns in Table 2? This is the main limitation of the paper from my perspective, and I am happy to raise my score if it is addressed, or a convincing reason for why such baselines are not useful is provided.\n2. Since the conditioning (controlability) is pitched as a key contribution, this should be clearly understandable from Fig. 2b. Would it be possible to expand “M” to show both its encoder and decoder in Fig. 2, with more clarity on how the different inputs/conditions are passed into the diffusion model architecture, instead of a single input arrow?\n\nMinor:\n\n1. I couldn’t understand the caption in Table 3, specifically the text enclosed in brackets\n2. [34] and [37] have incorrect citation “year” fields (2022 → 2023)\n\nUpdate:\n\nThanks a lot for your clarifications and the efforts made to update the draft.\n\nWhile I no longer have any issues regarding the method/presentation, I believe the rebuttal does not fully address the concern regarding limited baselines in the analysis of the proposed model. The other reviewers have also commented on the insufficiency of the current experiments to demonstrate a high degree of realism and controllability with the proposed model, and I concur. Therefore, I am maintaining my rating of borderline reject.", "limitations": "The key limitations (such as the currently limited generalization capability and simple task definition) are discussed in Section 6.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "What stands out in this work is the overall simplicity and elegance of the framework. It draws on ideas from different communities (generative modeling, object detection, traffic simulation). It also includes interesting original ideas that are potentially applicable to other generative modeling tasks in robotics and self-driving, such as (1) the imbalanced VAE (raster input, vector output); and (2) the masking idea to make agent-level conditioning optional. The writing is clear and mostly easy to follow.", "weaknesses": "The key weakness of this work is the lack of any baselines in the experimental analysis. Note that this is not a completely novel task (Section 5 discusses prior techniques). While it is true that not all prior work may be conditionable on per-agent attributes like the proposed model, I believe a comparison to prior work (potentially in the setting without agent token conditioning, as in the last column of Table 2) is warranted to meaningfully understand how well diffusion models perform this task. Secondly, the diffusion model architecture is hard to understand from the main paper in its current form, and a clear description is only available in the supplementary. For example, L_{KL} is marked in Fig. 2a but not described at any point in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review!\n\nWe have tried to address the question about comparison to existing techniques in the global response, but we agree that we did not provide sufficient detail for the comparison and the advantages of the diffusion model, and will add these to the final paper if accepted. We can definitely clarify the diffusion architecture – the $L_{KL}$ regularization loss in Fig. 2a was unfortunately only described in the supplementary material – we apologise for that oversight. \n\nQuestions: \n* The suggestion of a comparison to an autoregressive model and a heuristic as additional columns in Table 2 is extremely helpful – thank you for that suggestion. However, we hope that our explanation in the global response provides insight as to why we did not provide the explicit comparison to an autoregressive model in the paper. \n* The suggestion to expand “M” to show both its encoder and decoder in Fig. 2 is also very helpful and we will make that change to the final paper if accepted. It may be helpful to imagine the denoising module “M” being expanded to show something similar to Figure 3 from [Latent Diffusion](http://arxiv.org/abs/2112.10752).\n\nThank you for the corrections – the text in the parentheses in Table 3 was meant to be an internal comment in the source latex. \n"}, {"review_id": "3ewo9lhSGX", "reviewer": "Reviewer_uQ2d", "summary": "This paper tackles the problem of generating both the initial configuration and vehicle trajectories for driving scenarios given an input top-down map. It proposes a latent diffusion model that operates in the latent space of a learned autoencoder trained to decode vehicle boxes and trajectories. The model can be directly conditioned to encourage a subset of agents to adhere to user specifications. Experiments show that once trained on a large real-world driving dataset, the model can produce reasonable scenarios at multiple scales and allows for user control over a subset of agents.", "questions": "Overall, I am borderline on this paper. While the latent diffusion formulation is novel and technically sound and the model nicely generates both starting positions and motion for the scenario, the primary contribution of “controllability” is not sufficiently evaluated, and additional evaluations/baselines would strengthen the paper. In the rebuttal, I would really like the authors to clarify my confusion on the agent tokens and ideally evaluate the controllability of the model by showing how closely it follows user specifications. \n\nThe method Sec 3 at times is rather high level and leaves some questions:\n* Sec 3.1: why is the encoder not conditioned on the map like in the typical CVAE formulation?\n* Sec 3.3: A lot of info about tokens are relegated to experiments in Sec 4.3.1 or the appendix but may be better to include in the methods. I’m still left wondering how exactly the cross attention works, e.g. how are the tokens associated with agents in the scene since there is no canonical agent IDs or ordering? Also, if tokens are specifying the properties of agent trajectories, why do multiple samples of the same scenario (e.g. Figs 1, 7, 8) exhibit varying trajectories for the specified agents?\n* How are the number of agents in the scene determined by the diffusion model? Through the agent probability? Why is the global scene token represented as the masking probability and not directly the number of agents that are desired in the scene?\n* The methods mentions 2 agent tokens are used, but then 3 are used in Fig 3 and N are used in Table 2, so it’s not clear what is the problem formulation and what’s used in practice.\n\nAnother other small suggestion: since explicit trajectory diffusion models are becoming more prevalent [37] [Trace and Pace, Rempe et al., CVPR 2023] [Stochastic trajectory prediction via motion indeterminancy diffusion, Gu et al., CVPR 2022] [Planning with Diffusion for Flexible Behavior Synthesis, Janner et al., ICML 2022], it would be good to discuss them in related work and compare to the proposed latent diffusion. \n\n===================== After Rebuttal ======================\n\nAfter considering other reviews and discussion with the authors and reviewers, I have decided to keep my rating leaning towards reject. \n\nGiven that the problem is relatively new and the latent diffusion approach is novel, I am willing to be lenient on the lack of a baseline comparison. Especially since TrafficGen (ICRA 2023) is concurrent work and SceneGen (CVPR 2021) did not release code. However, without a baseline, the experiments and metrics need to be comprehensive and convincing to show generated scenes are realistic and match user specifications; I don’t believe this is the case in the current draft. The rebuttal Table 1 adds some new map-based metrics, but important realism evaluations like collision rate, plausibility of trajectories (smoothness and comfort), and diversity are still missing. Moreover, the added evaluation of controllability in rebuttal Figure 3 is rather coarse (considered “correct” if within 2.2 m) and does not consider other user specifications like length, width, speed, and curvature. \n\nI also think it would be a strong addition to show results on higher-level semantic conditioning of the model such as vehicle type and maneuver type (e.g. lane change, turning, or u-turn) to justify the claim that the model can handle “abstract” specifications rather than just partial explicit specifications.", "limitations": "Limitations are sufficiently discussed in Sec 6.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The problem is important for AV testing and training.\n\nThe proposed method simultaneously generates both an initial scene configuration and vehicle trajectories, instead of in two stages as in prior work.\n\nThe proposed latent diffusion model is novel for vehicle trajectories – all prior work I have seen is explicit trajectory diffusion and the design of the autoencoder for such a model is not trivial.\n\nThe model allows directly (optionally) conditioning on attributes of one or more agents which gives nice flexibility and allows generating variations of scenarios directly taken from e.g. a log or specified by a user.\n\nResults of generating short 4 sec scenarios look reasonable. Experiments show the ability to generate scenarios of various sizes on the sizable Argoverse 2 dataset. Fig 4 is a neat demo showing that the model is picking up on the semantics of the relationships between vehicles.\n\nThe appendix is extensive and provides most of the necessary details to enable reproducibility.", "weaknesses": "There are a couple claims that seem over-stated and should be toned down or clarified:\n* Conditioning tokens are “more abstract features” (L208) that does not require “fully specifying box and trajectory for the agent” (L206) or “having observed such agents in the real world” (L43) – if I understand correctly, these tokens require a user to give the model the desired agent position, heading, length, width, speed, and sometimes curvature over time (L211),  which is quite a low-level specification of the trajectory and likely will need to be taken from a log. Or do tokens describe the state only at a single $t=0$? If so, this should be made more clear, and I still would not say this is “abstract”.\n* The model “generalizes to different geographical regions” (L9) – Table 3 shows seemingly limited ability to generalize across regions with models trained specifically on each region far outperforming other single-region models. The model trained on all regions does reasonably well on each, but this is not generalizing across regions. Moreover, it would be helpful to add a metric that indicates similarity between regions (e.g. MMD between ground truth scenes in A and B) to put the numbers in context. While this experiment is informative about performance across region, it may be a stretch to say it demonstrates generalization across regions. \n\nSome limitations of the technical approach may make the proposed method less practically useful:\n* The agent tokens are not hard constraints and in results like Fig 3 the specified agents move around between samples and don’t exactly meet user specifications. A natural solution for diffusion would be to use some kind of test-time guidance as in [37] and [Trace and Pace, Rempe et al., CVPR 2023].\n* The 4 sec time horizon (2 sec past + 2 sec future) seems quite short and limits the possible scenarios it can capture. \n* The conditioning requires a user to specify all attributes of an agent. But it seems with the masking strategy it should be possible to condition on partial information (e.g. specify the past trajectory but generate the future, or only specifying speed) which would greatly enhance the flexibility.\n\nSome additional evaluation metrics would help to evaluate realism and diversity. For example, MMD on speeds, collision rate between vehicles, and a diversity metric (like average pairwise distance between multiple samples). Additionally, some key experiments are missing:\n* The accuracy of the model’s controllability is never evaluated, which is the main claim of the paper. It would be good to report how closely generated agents follow the agent tokens specified by the user (e.g. L2 distance between positions, headings, speed, etc..). The 0% model in Table 2 kind of does this, but the reported MMD metric is not very intuitive.\n* A comparison to a baseline to justify the advantages a diffusion model to generate both static states and dynamic trajectories jointly. E.g. a comparison to TrafficGen [6] or a simpler baseline that only generates static states and then uses a trajectory prediction or traffic simulation model to roll out motion, e.g. [BITS: Bi-level Imitation for Traffic Simulation, Xu et al., ICRA 2023]. Or evaluating if the trajectories generated by the diffusion model align with those from a SOTA trajectory forecasting model?\n* It would be good to see an example of how these scenarios can be useful. For example, that training a driving policy on a dataset augmented with generated scenarios does better than just a real-world dataset. \n* More qualitative results (ideally videos, but additional figures in the supplement would also be helpful).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review!\n\nWe tried to address the question about additional metrics in the global response, but we agree that we did not provide sufficient detail for the comparison and the advantages of the diffusion model, and will add these to the final paper if accepted. We appreciate the guidance to be more precise about the claims of the paper.\n\n* It is correct that the tokens encode agent position, heading, length, width, and speed, and that these are only initial conditions. However, even with the curvature feature, we do not prescribe the exact path of the trajectory — the model produces variations from the partial description provided by agent tokens. The same architecture can be used with higher level descriptions. For example, we can give position using relatively coarse specifications, and use “bus” vs “car”  tokens instead of the vehicle’s size, and the type of motion (e.g. lane change, turning, or u-turn). We have experiments with these higher level descriptions.\n* We would be grateful for clarification on the comments that our techniques do not generalize. While performance on each training set in Table 3 is definitely better than performance on a different test set, the differences are not significant in every case, e.g., Train/Test on B vs Train-on-B/Test-on-C and we believe this demonstrate some generalization across environments, The largest difference involves environment D, and we are careful in the paper to say that we do not generalize to D, and give specific examples of where and how the models are not generalizing.  \n* The agent tokens are not hard constraints by design. We want the model to produce different variations following a partial description via tokens. As you suggest, we have used guidance to refine parts of the generated scenario (e.g. stay close to lane centers, avoid overlapping boxes) but those results were out of the scope of this paper. \n* We used 1-4 seconds of agent history as input and trajectory prediction to be consistent with the literature. \n* We have provided a precision-recall graph of the controllability in the attached PDF, see Figure 3 in the attachment. Our model achieves precision and recall well above 90%. \n* We have tried to address the question about comparison to existing techniques in the global response, but we agree that we did not provide sufficient detail for the comparison and the advantages of the diffusion model, and will add these to the final paper if accepted. \n* We agree that evaluation of the performance of downstream tasks using the generated scenarios is the ultimate purpose of our model but our hope was that our experimental results describing the performance under partial tokenization (Figures 3 and 4, Table 2) and generalization across environments (Figure 5, Table 3) were more informative of the detailed performance of our model. \nWe have added additional qualitative figures in the PDF attachment, see Figures 1 and 2 in the PDF. \n\nQuestions \n* In Sec 3.1, the map information is provided to the decoder to help the output trajectories be consistent with the map. Since the only outputs of the scene autoencoder are boxes and trajectories we do not need the latent embedding to include information about the map explicitly. Empirically when we tested concatenating the map image to the encoder inputs we saw no change in performance for the scene autoencoder.\n* In Sec 3.3, the cross-attention inside the denoising model is as described in the Latent Diffusion paper. Each pixel from the intermediate representation of the Unet is a query, and the keys/values are agent tokens. When agent tokens include position information the model learns to associate them with queries from the corresponding region of the image.\n* As you correctly point out, there is no fixed agent ID or ordering required. While we do not demonstrate it in this paper, the tokens can be flexible (e.g. multiple tokens describing different aspects of a single agent, a token describing multiple agents, etc) The agent tokens as we define them indicate “there’s an agent in this region with these properties” and the cross-attention updates the latent embeddings in that region to reflect this. \n* Multiple samples have different trajectories because the tokens provide only a partial description of agents. We do not want to exactly dictate the trajectory to allow the model to generate slight variations, hence the term “abstract features”, which we will replace. \n* Regarding the number of agents, if no tokens are used the model learns the distribution of scenes (including number of agents) conditioned on the map. This is measured by the Earth Mover’s Distance metric reported in Table 5. If agent tokens are used, the model still learns the distribution of scenes (including number of agents), but now is conditioned on the map, the number of agent tokens (since every agent token describes a unique agent), and the global scene token which provides information on how many additional agents there are. We experimented with various ways to represent the scene density as a token (e.g. masking probability, total number of agents, total number of agents divided by the drivable surface area) but found no clear advantages in terms of model performance.\n* Regarding the number of tokens, Section 3.3.2 says “we explore two types of tokens”. There can be any number of agent tokens. These tokens are used via cross-attention, which supports an arbitrary number of keys and values. We intentionally demonstrate this capability by showing examples with 2 and 3 agent tokens, and as you point out Table 3 requires using an arbitrary number of tokens. For demonstration purposes we found it better to limit the number of agent tokens to 2-3 so that the model had flexibility to fill in the rest of the scene."}, {"review_id": "WVhdo4akfC", "reviewer": "Reviewer_VGNk", "summary": "This paper proposes a novel approach to use latent diffusion for driving scenario generation. The generated driving scenario is defined as bounding boxes with associated trajectories. The proposed diffusion model is comprised of an autoencoder to encode the BEV image into latent space and a conditional UNet for diffusion modeling. To increase the controllability of the generated scenario, map conditioning and token conditioning are incorporated into the process. The experiments are conducted both on the large-scale internal dataset and the Argoverse 2 dataset, which validates the effectiveness of the proposed algorithm and its flexibility for controllable generation.", "questions": "1. During the sampling process, would it be helpful to use techniques similar to classifier-free guidance to make the sample ground on the conditioning token more faithfully?\n2. Currently the agent token still needs very specific values of different aspects to define its behavior, is it possible to use some more abstract behavior tokens to only specify its overall behavior like overtaking, speed up, right turn, etc. to promote better diversity without much engineering effort?", "limitations": "The authors have well discussed the limitations and the potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of applying latent diffusion to driving scenario generation is novel and interesting, as diffusion methods have shown great capability in generative tasks.\n2. The proposed token conditioning mechanism brings more controllability and the possibility of customization for this task.\n4. The overall writing and presentation of the paper are clear and easy to follow.", "weaknesses": "1. For the metrics to evaluate the generated scenarios, it is very important to evaluate the validity of the driving scenario, but the paper only measures the rate of off-road trajectories. Other aspects like the rate of collisions between agents, smoothness and plausibility of the trajectories, and possible traffic rules violations should also be measured.\n2. Although in Sec 4.3.2 some examples have shown that the proposed method is aware of the relationship between agents, using BEV encoded image to represent the scene without specific designs for agents interactions could not well capture the relationships between agents compared with those GNN-based methods shown in motion-forecasting. Having specific designs to explicitly model the possible interactions is important for the complex driving scenario generation.\n4. There is no direct quantitative measurement of how the generated sample faithfully follows the conditioning agent and global token.\n3. The goal of scenario generation is to facilitate the development of other autonomous driving tasks like motion forecasting or planning. Therefore, a better way to show the effectiveness and diversity of the generated scenario is by testing them on downstream tasks. Some simple experiments showing the performances of a motion forecasting model training on real and generated data would greatly improve the value of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! \n\nWe have tried to address the comment about metrics in the global response. We respectfully disagree with the comment that “the paper only measures the rate of off-road trajectories”. We believe that the statistics reported in the different tables comparing the training and generated distributions are appropriate metrics that capture the performance of the model but we appreciate the suggestion for additional metrics. \n* We have tried to address the comment about agent interactions in the global response – we certainly agree that GNN-based methods are a very good way to model agent interactions, but we believe that the diffusion model has shown that it can capture interactions in a scalable manner.  \n* We respectfully disagree with the statement that “there is no direct quantitative measurement of how the generated sample faithfully follows the conditioning agent and global token.” We believe that the MMD and EMD statistics exactly show those quantitative metrics, and Table 2 in particular shows the effect of removing the tokens: when every agent has a token, the paper states that the MMD approaches 0, and we see the results when the tokens are masked completely. \n* We agree that showing “the effectiveness and diversity of the generated scenario is by testing them on downstream tasks.” and downstream tasks are the ultimate purpose of our model but have not included these results. Our hope was that our experimental results describing the performance under partial tokenization (Figures 3 and 4, Table 2) and generalization across environments (Figure 5, Table 3) were more informative of the value of our model. \n\nQuestions:\n* We very much agree that classifier-free guidance would be helpful, and we have explored using CFG to increase the strength of both map and token conditioning. We have also explored heuristic guidance functions to improve both agent box placement (to avoid overlapping boxes) and to improve trajectories (e.g. to avoid collisions, stay close to lane centers). However, we opted not to include that work in this paper to give more space to describe the essential parts of the approach (scene autoencoder, diffusion, token conditioning).\n* It is definitely possible to use more abstract tokens. We described explicit tokens as they were more obvious demonstrations and were easier to evaluate. The final position heading token feature described in Appendix C.3 is more abstract in that it describes the curvature of the future trajectory but does not directly describe the past trajectory or exact trajectory shape. As seen in Figures 1, 7, and 8 this allows the model to generate different trajectories that fit within the constraints described by the tokens. We have also explored using abstract motion categories (e.g. “lane change”, “turning”, “u-turn”, “stopping”) as token features.\n"}, {"review_id": "AMffT0JUlS", "reviewer": "Reviewer_9X4d", "summary": "This work adopts a conditional diffusion model to generate the abstract driving scene.", "questions": "See weakness for details.\n\nI am mostly concerned that this work only decribe their method but do not compare with rich existing literatures at all.  It is hard to tell whether the proposed method really works and is better than existing works.", "limitations": "No concern.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "It has great potentials to generate future scenes with diffusion model.", "weaknesses": "(W1) No comparsion with existing works at all. It is significant to compare with previous methods to demonstrate the superiorty of the proposed method in terms of diversity, generalizability or other aspects. Quantitative comparisons with methods such as SceneGen [1], TrafficGen [2], CTG [3], and Scene Diffusion [4] would help to make the improvement more convincing and easier to recognize. Could the authors provide related results or briefly discuss why these methods were not included in the comparison?\n\n(W2) The proposed method aims to generate all agents simultaneously so that the relationships between them can be learned. Nevertheless, auto-regressive manner also embodies such functionality as it can condition on the generated agents. Have the authors explored the use of auto-regressive ways within the proposed framework?\n\n(W3) As a generation work, a great potential would be producing additional data for training, customizing safety critical scenes for evaluation. However, this part is missing from the current version, which means the claimed benefit is unjustified at all. Could the authors complement some experiments to support that?\n\n[1] SceneGen: Learning to Generate Realistic Traffic Scenes\n[2] TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios\n[3] Guided Conditional Diffusion for Controllable Traffic Simulation\n[4] Generating Driving Scenes with Diffusion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! \n* (W1) We have tried to address in the global response why we did not provide a comparison to autoregressive models, and the difficulty of a metric that highlights the need for joint inference, rather than conditional inference. Our hope is that the additional figures in the attached PDF capture this property of our model. \n* (W2) We do not have a substantial and rigorous evaluation of auto-regressive models. Due to the factored nature of auto-regressive models, joint inference of vehicle states would be a challenge for these models. \n* (W3) We agree that experiments showing the applicability of our scenarios in, for instance, control validation, is the long term goal. However, it is not clear how we could add an adequate explanation of such experiments and the results within the space constraints.  \n\nQuestions:\n* In response to the comment that “it is hard to tell whether the proposed method really works”, we note that our MMD and other metrics are also used by the existing work in the literature such as TrafficGen and we believe are an appropriate basis for at least assessment of competence. \n"}]

Author rebuttal:
We appreciate the reviewers thoughtful and detailed comments, and agree with the majority of the comments and suggestions. In terms of the overall identified weaknesses, the reviewers’ concerns can be roughly grouped into:
* Absence of comparisons to previous work, especially previous autoregressive models 
* Limited quantitative evaluations or metrics. 
(along with additional comments from individual reviewers). We believe that these two concerns are linked, and get to the heart of what we were attempting to show in this paper. 

One of the challenges of urban driving is that the correlations between the agents are often tightly coupled and cannot be easily factored in scenarios where the agents are forced to interact with each other. For instance, creating a scenario where another vehicle is pulling out in front of our AV cannot easily be generated by an autoregressive conditional model — both our AV and the other vehicle must be placed simultaneously with respect to each other in order to force the interaction. We show our ability to generate this scenario in Figure 1 of our attached PDF. 

This is not to say that an autoregressive model cannot be coerced into a limited set of pairwise or joint interactions by regressing on combinations of vehicles, but learning such a model is non-trivial and will not scale as easily as the joint model produced by diffusion. It is worth noting that the previous results such as TrafficGen do not report scenarios that require *specific* forms of interaction — for instance, the interaction results in the TrafficGen paper report interaction metrics as collisions, rather than rate of inter-vehicle responses. 

Rather than show this difference experimentally, we attempted to articulate within the text (in the introduction and in section 4.3.2) a principled reason for a diffusion model, and that autoregressive models are by design limited to locally factored representations. We agree that our metrics and evaluations do not necessarily highlight this capability relative to autoregressive models, but it is also not clear how convincing a metric comparison would be. The advantage of diffusion is the ability to learn a high-dimensional joint distribution, capturing many agent correlations as in Figure 4 of our paper. Our best attempt at a metric was the distributional comparisons in tables 1, 2 and 3. We will add additional metrics proposed by the reviewers such as rate of collisions between agents and traffic rules violations, and have added additional metrics in Table 1 of the attached PDF. 

We agree that the paper was not clear on the advantages of a joint distribution inferred by diffusion, and that such a model can capture correlations that an autoregressive model does not easily. We will revise the text to better reflect the purpose of diffusion, and show clearer examples. 


Meta-review: The paper proposes a driving scenario generator that is controllable using diffusion models. The proposed pipeline combines perception and motion prediction/planning simultaneously. Authors show the approach has sufficient capacity to model diverse patterns and generalizes well to different reigions.

The review score got 4,4,,4 and 6. All reviewers agree that the method / presentation is well stated and easy to follow. The technical novelty is non-trivial, given the problem setting is new and the diffusion approach is novel. The **main concern** is lack of baseline comparison and how to design appropriate evaluation metric. 

There are good discussions back and forth among reviewers / authors. AC read the paper carefully, all the rebuttal discussions, and decide to accept the paper nonetheless, based on the following reasons.

- The paper indeed proposes an important probem setup, to generate controllable driving scenarios that can be manipulated with multiple stacks. The proposed method is novel using diffusion. As far as AC's knowledge, there are **few work** on autonomous driving that could utilize diffusion methods to work well. This could **trigger future research** in the community, eg, motion prediction, data augmentation in the sense of foundation model, simulation and scene generation. 

- Despite the lack of baselines and evaluation metric is questionable, we should **encourage** novel contributions to be accepted for NeurIPS. Authors did a good rebuttal to address most concerns / technical details, and add addtional experiments to explain.

- The main concern (experimental comparison) raised by reviewers are indeed very important. However, these are fixable and AC trust authors would address them (based on rebuttal and the additional results provided, though not perfect).

As such, it is strongly recommended for authors to revise the paper as sugggested by reviewers (eg. show results on semantic conditioning level by Reviewer uQ2d, etc.) and improve the manuscript to great extent. It is great if the codebase / datasets are open-sourced and results are reproducible as well.
Decision: Accept (poster)