Title: Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

(Other) Applications

(Other) Machine Learning Topics
Abstract: Automated creation of synthetic traffic scenarios is a key part of scaling the safety validation of autonomous vehicles (AVs). In this paper, we propose Scenario Diffusion, a novel diffusion-based architecture for generating traffic scenarios that enables controllable scenario generation. We combine latent diffusion, object detection and trajectory regression to generate distributions of synthetic agent poses, orientations and trajectories simultaneously. This distribution is conditioned on the map and sets of tokens describing the desired scenario to provide additional control over the generated scenario. We show that our approach has sufficient expressive capacity to model diverse traffic patterns and generalizes to different geographical regions.

Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion

Abstract

Automated creation of synthetic traffic scenarios is a key part of validating the safety of autonomous vehicles (AVs). In this paper, we propose Scenario Diffusion, a novel diffusion-based architecture for generating traffic scenarios that enables controllable scenario generation. We combine latent diffusion, object detection and trajectory regression to generate distributions of synthetic agent poses, orientations and trajectories simultaneously. To provide additional control over the generated scenario, this distribution is conditioned on a map and sets of tokens describing the desired scenario. We show that our approach has sufficient expressive capacity to model diverse traffic patterns and generalizes to different geographical regions.

1 Introduction

Automated creation of synthetic traffic scenarios is a key part of validating the safety of autonomous vehicles (AVs). To efficiently target rare and safety critical scenarios we would like to direct scenario generation to produce specific types of events. Prior methods for heuristically created scenarios tend to be of limited complexity and miss a large number of possible real-world situations [5, 7, 16, 17]. Recent works using deep learning models [2, 6, 32] are able to produce complex scenarios conditioned on a map region, but do not offer additional controls over the generation process. In this paper we propose Scenario Diffusion: a novel architecture for generating realistic traffic scenarios that enables controllable scenario generation.

Our problem setting is to generate a set of bounding boxes with associated trajectories that describe the location and behavior of agents in a driving scenario. We accomplish this scenario generation using a denoising diffusion generative model [12, 27, 29]. We condition our model on both a map image and a set of tokens describing the scenario. We leverage these tokens to provide a variable rate of control over the generated scenario, so that the diffusion model can generalize to complex scenes conditioned only on a small set of tokens that control both a subset of the individual agents and global scene properties. Further, this variable rate of control allows us to learn the model from partially tokenized training instances, substantially reducing the labelling requirements.

Motivated by the insight that the instantaneous position of each agent is inextricably linked to their behaviors, we combine latent diffusion [24], object detection, and trajectory regression to simultaneously generate both oriented bounding boxes and trajectories, providing a generative model of both the static placement of agents and their behaviors. We evaluate Scenario Diffusion at generating driving scenarios conditioned on only the map, and with additional conditioning tokens as well. Finally, we provide an analysis of the generalization capabilities of our model across geographical regions, showing that our diffusion-based approach has sufficient expressive capacity to model diverse traffic patterns.

U-turn

Past

Future

Scenario 1

Scenario 2

Scenario 1

Scenario 2

2 Problem Setting

We aim to generate traffic scenarios conditioned on a map and an optional description of the scenario. Similar to prior works [2, 6, 32] we represent traffic scenarios abstractly with oriented bounding boxes and trajectories 3 . AV software is typically divided into perception and motion planning . The interface between these components describes agents with bounding boxes, historical trajectories, and additional attributes. In simulation this abstract representation is synthesized to evaluate the motion planning component in isolation, alleviating the need for expensive photorealistic simulation.

Fig. 1 demonstrates the type of controllability we aim to achieve. We condition the generative model by describing the state of two vehicles: the A V (circled in red) and one key interacting agent (circled in orange). The model is able to generate a diverse set of scenarios that include these two agents and additional ones. Notably, describing the two key agents does not require having observed such agents in the real world (e.g. by copying these two agents from an existing log). Instead, the model is able to generalize to generating novel agents based on the given description.

2.1 Scenario Representation

In this work we use topdown birds' eye view (BEV) representations similar to [1, 2, 13, 32] for the inputs to convolutional networks and sparse representations of oriented boxes and trajectories as the final output of the decoder.

Topdown Representations The map context is represented as a multi-channel BEV image m of shape C m × H × W that contains information about road geometry, regions of interest (e.g. driveways, crosswalks), and traffic control devices (e.g. stop signs, traffic lights). Agents are also represented in a multi-channel BEV image x of shape C x × H × W that encodes each agent's bounding box, heading, and trajectory. Sec. 4.1 describes two different datasets with different rendering styles for m and x that both yield good results with our model.

(a) Autoencoder training.

(b) Diffusion training.

(c) Inference.

Conv

Net

MLP

Function

Loss

Module Attn

Frozen

Cross

Concat

Add Noise

Z

Ddown

串

Dup

Ddown

Dup

J

N(0, ²max)

Repeat x100

Sparse

m

Sparse

M

M

Cm

8

8.

CKL

m

m

C

Tokens

MLP

Tokens

MLP

Sparse Representations We also represent the agents in the scene as a set of oriented bounding boxes, with associated trajectories included as an additional feature vector per box. The oriented bounding box is defined by a center position, heading, length, and width and the agent trajectory as a sequence of future and past poses spaced at uniform intervals in time, as is common in motion forecasting [1, 8, 22, 23, 25, 31, 33, 38, 40], with the bounding box pose representing the 'current timestep'. This sparse representation is used as the labels for the autoencoder's detection task and to compute tokens describing agents in the scenario.

3 Method

As in [24], our diffusion model consists of two parts: an autoencoder to transform between the data space and a latent space and a diffusion model that operates in this latent space.

3.1 Scenario Autoencoder

The scene autoencoder is a variational autoencoder (V AE) [11] that learns to encode and decode sets of agents as shown in Fig. 2a. The VAE is trained with a combination of a reconstruction loss L rec on the decoder output and a KL divergence loss L KL on the latent embedding. The architecture is based on the autoencoder of [24], but we train the model to perform anchor-free one-to-one detection instead of pixelwise image reconstruction.

An encoder E takes the BEV image of the agents in the scene x and outputs a latent embedding z = E ( x ) of shape C z × H ′ × W ′ , where H ′ = H/ 2 f and W ′ = W/ 2 f for a downsampling factor f ∈ N . Since the encoder is designed to capture the placement of agents, there was empirically no benefit to providing the map image as input to the encoder. This latent embedding is given to the decoder D along with the BEV map image m to obtain the detection output y = D ( z, m ) of shape C y × H ′′ × W ′′ . Because we output detections, H ′′ and W ′′ do not need to equal H and W , the original input shapes. In practice, having a lower output resolution provides significant memory savings during training. The decoder downsamples the map image m , concatenates the latent embedding z , and then upsamples to produce the final output: D ( z, m ) = D up ( z, D down ( m )) .

Detection Outputs The decoder output y represents oriented bounding box detections and trajectories. Each pixel of y represents one box proposal, with different channels representing the probability, position, heading, length, width, and trajectory of the box. Concretely, the first channel is a sigmoid logit for the box probability. The second and third channels represent the cosine and sine of the box orientation. The next four channels represent the distances from the pixel center to the front, left, back, and right edges of the box in log scale. These seven channels define the oriented bounding box for the agent.

The trajectory describes the agent's pose from T seconds in the past to T seconds in the future at one second intervals. For each timestep t ∈ { 1 , . . . , T } of the future trajectory three channels of y represent the delta between the pose at t and the pose at t -1 (with channels for x, y, and heading). These deltas are accumulated to produce the final trajectory starting from the center pose of the

oriented bounding box. The past trajectory is handled symmetrically (i.e. deltas between t and t +1 for t ∈ {-T, . . . , -1 } ). In total 1 channel encodes the box probability, 6 channels encode the oriented bounding box, and 6 T channels encode the trajectory.

Reconstruction Loss The reconstruction loss used to train the autoencoder adapts the one-toone detection loss of [30] to accommodate oriented boxes. Given the simplicity of our detection problem (perfectly rendered rectangles with disambiguated orientations) we find that replacing the IoU loss (which is non-differentiable with oriented boxes) with an L2 loss on the box vertices yields detection performance sufficient for training the diffusion model. Following [30], each ground truth box is matched with one predicted box by selecting the predicted box that minimizes a matching score between it and the ground truth box. This matching score is a weighted combination of a classification score, L1 loss on the box parameters, and an L2 loss on the box vertices. A classification loss encourages matched predicted boxes to have high probability and unmatched predicted boxes to have low probability, and regression losses encourage matched predicted boxes and trajectories to equal their corresponding ground truth box. We use a weighted combination of an L1 loss on box parameters, L2 loss on box vertices, L2 loss on trajectory positions, and cosine loss on the trajectory headings. Further details on the reconstruction loss can be found in Appendix B.2.

3.2 Diffusion Fundamentals

Once the autoencoder is trained, a diffusion model is trained in the latent space of the autoencoder. This means that the diffusion model does not operate on bounding boxes and trajectories directly. We use the EDM diffusion algorithm [18] to train our model and generate novel scenarios.

Training Let ˆ z = E ( x ) be the latent embedding obtained from a trained and frozen encoder. Each example can also include associated conditional information c (e.g., map information, tokens). We discuss options for this conditional data in following sections.

For each training example we sample σ according to a log-normal distribution (i.e. log σ ∼ N ( P µ , P 2 σ ) with hyperparameters P µ and P σ ). We then create a noisy sample z = ˆ z + σϵ , where ϵ has the same shape as ˆ z and is sampled from the standard normal distribution.

We train the denoising model M ( z ; c, σ ) by minimizing the reconstruction loss

where c in ( σ ) , c out ( σ ) , c skip ( σ ) , and λ ( σ ) are all scalar functions of σ as defined in [18]. In practice this model M is a conditional Unet as in [3, 12, 28]. This process is depicted in Fig. 2b. Given a noisy sample z , the denoised version ˆ z can be estimated as

Inference To generate novel samples during inference, we start with an initial noisy sample z ∼ N ( 0 , σ 2 max I ) . This sample is iteratively refined according to the reverse process ODE as described in [18]. The gradient of the log probability distribution can be approximated as ∇ z log p ( z ; c, σ ) ≈ ( M ( z ; c, σ ) -z ) /σ 2 . This simplifies the reverse process ODE to

After integrating the sample z from σ = σ max to σ = 0 using Euler integration, it is passed through the decoder to obtain the generated boxes and trajectories D ( z, m ) . This procedure is depicted in Fig. 2c. During inference we keep the generated boxes with probability above a fixed threshold chosen by optimizing a metric that compares generated and ground truth boxes (described in Sec. 4.1). We then filter any overlapping boxes, keeping the highest probability box of those that overlap.

3.3 Diffusion Conditioning

Conditional Unet models have two primary ways to provide the conditional information c : concatenating to the noisy sample that is input to the denoising model and applying cross attention over a set of tokens inside the denoising model. We explore both options in this work.

3.3.1 Map Conditioning

One method to include conditional data in the denoising model is via a conditional image that gets concatenated with the noisy sample input to the denoising model [24]. In this work we use the BEV map image m as described in Sec. 2.1 as the conditional image. The map image m is first processed by a separate encoder E m to downsample it to the same pixel dimension as z . The encoded map E m ( m ) is concatenated with z in the channel dimension before being input to the denoising Unet M .

3.3.2 Token Conditioning

We can also condition diffusion models on a set of tokens via a cross-attention mechanism [34] within the denoising model [24]. In image generation these tokens often come from text embeddings of a caption for the image [26]. While we do not have datasets of driving scenarios fully annotated with text captions, we can still use the concept of conditional tokens that describe aspects of the scenario. It is exactly these tokens that provide controllability over scenario generation. In this work we explore two types of tokens: agent tokens and global scene tokens .

Agent Tokens Agent tokens are tokens that describe a single agent. In this work we use procedurally generated tokens (i.e. tokens that can be computed from available data in the dataset), so are able to provide a token for every agent. However that is not a requirement for this approach; if using human-labeled tokens it would be possible to train a diffusion model while only having partially labeled scenarios. The token feature vector for each agent expresses various quantities that we might want associated with that agent, such as the agent's position, speed, length, or motion profile. In this work the meaning of the token feature vector is fixed for a given experiment. One potential direction for future work is to have a single model support multiple types of agent token feature embeddings (e.g. to allow the user to specify the length for one agent and the speed for another).

To enable the model to generate agents not described by tokens we use a partial tokenization strategy. During training we achieve this by sampling a token mask probability p mask and masking out agent tokens with this probability. More information on this can be found in Sec. 4.3.1.

Global Scene Tokens Global scene tokens encode scalar quantities that allow us to control global properties of the entire scenario. In this work we consider a single global scene token to represent p mask , which encodes information about the number of agents to be generated relative to the number of tokens provided. During inference the value represented by this global scene token can be modulated to control the number of agents in the generated scenarios. By providing no agent tokens and setting p mask = 1 in the global scene token we are able to generate scenarios conditioned on only the map from the same model.

Processing Each type of token is processed by a separate MLP to get final embeddings in a common latent space, to be used as keys and values during cross-attention inside the denoising model. At different levels of the Unet each pixel from the intermediate image representation is used as a query, and the agent and global scene tokens are used as the keys and values. This architecture can support any number of tokens. We refer the reader to [24] for more details on the cross-attention mechanism.

4 Experiments

4.1 Experimental Setup

Datasets We use two datasets in this work. The Argoverse 2 motion forecasting dataset [36] contains 250,000 driving scenarios over 6 geographical regions representing 763 hours of total driving. We use the provided training and validation splits. More information about this dataset can be found in Appendix A.1.

The second dataset is an internal dataset containing 6 million real world driving scenarios from Las Vegas (LV), Seattle (SEA), San Francisco (SF), and the campus of the Stanford Linear Accelerator Center (SLAC). Training and validation data are not separated by location, so a given validation example may have training examples from the same location at other times. More information about this dataset can be found in Appendix A.2.

For both datasets, the scenes are centered on the autonomous vehicle. As the autonomous vehicle is treated as just another vehicle during training, the model almost always places a vehicle in the center of the scene during inference. We use a 2 second time horizon for both the future and the past trajectory (i.e. the bounding box represents t = 0 and the trajectory goes from t = -2 to t = 2 ). This time window is sufficient to populate the agent history for many motion forecasting methods [1, 8, 22, 23, 25, 31, 33, 38, 40].

Training See Appendix B and for details of training the autoencoder and Appendix C for details of training the diffusion model.

Inference We use 80% as the threshold for the generated box probabilities. See Appendix D.2 for the ablation experiment used to select this threshold. We note that the performance of Scenario Diffusion is relatively robust to choices of probability threshold between 50% and 90%.

Metrics We consider metrics to compare the generated and data distributions and measure the quality of generated scenarios. One popular metric for comparing two distributions using samples is the maximum mean discrepancy (MMD) [9]. While this metric is typically defined in terms of two distributions, in practice we compute this metric using samples drawn from the two distributions. Given two sets A and B generated by sampling from two distributions and some kernel k , the maximum mean discrepancy is defined as

Similar to [6, 32], we use a Gaussian kernel k and apply this metric to the sets of agent center positions, heading unit vectors, and velocities, all in R 2 .

To measure the quality of generated trajectories we compute the fraction of waypoints that fall within the drivable area, averaged over all generated agents and all 5 trajectory waypoints. For each predicted pose along the trajectory we also compute the minimum angle difference (in radians) between the pose heading and the heading of all lanes at that location. These metrics do not have a clear optimal value (trajectories in the Argoverse dataset sometimes leave the drivable area and aren't perfectly aligned with the lane tangent); we compare the metric between the dataset and generated scenarios.

Baselines As a simple heuristic baseline we implement Random Log Selection, which takes the agents from a random sample in the training dataset (ignoring map information). We also report the trajectory metrics on the ground truth validation samples themselves.

To compare our method to previous autoregressive approaches we adapt TrafficGen [6] to the Argoverse 2 [36] dataset. See Appendix E for more details on the modifications needed.

4.2 Map-Conditioned Scenario Generation

We first evaluate Scenario Diffusion in generating scenarios conditioned on only map data, the problem setting explored in prior works [2, 6, 32]. In Tab. 1 we report MMD metrics that compare generated and ground truth scenes and trajectory metrics that measure how well the trajectories conform to the map. We evaluate Scenario Diffusion models with and without the map context m , and compare against the baselines described above.

Scenario Diffusion outperforms all other methods in the MMD metric for position and heading, and performs equally with TrafficGen in velocity. In the trajectory metrics, Scenario Diffusion produces similar values as those of the ground truth logs. As expected, Scenario Diffusion without map context performs similarly to random log selection, as both methods ignore the map context.

Global SceneToken=0%

Global Scene Token=50%

Global Scene Token =90%

4.3 Token Conditioning

We next explore applications of token conditioning in Scenario Diffusion. Fig. 1 demonstrates how tokens can be used to generate scenarios with a specific type of interaction between the AV and another agent. Our approach is able to generate these interactions without having to fully specify the box and trajectory for the agent (e.g. by hand or by copying an agent from an existing log). By using more abstract features in the agent tokens we allow the model to generate different scenarios that meet the given description (e.g. we do not prescribe the specific trajectory of the u-turning agent).

The flexibility of tokens provides variable degrees of control over the generated scenarios. In this work we demonstrate models that use a variety of features in the agent tokens, such as position, heading, length, and speed. More details on the exact token features can be found in Appendix C.3.

4.3.1 Partial Tokenization

It is crucial that the learned model be robust to the presence or absence of user-provided tokens. To ensure this, during training we omit or 'mask' supervision tokens so that the model learns to fill in additional agents in the scenario. During training we select p mask such that 40% of the time we keep all tokens (i.e. p mask = 0 ) and the other 60% of the time we sample p mask ∼ Beta (2 , 1) . 4 For each sample, the chosen masking probability is bucketed into one of ten bins ([0, 0.1), [0.1, 0.2), etc.) and embedded as a one-hot vector to define the global scene token. This global scene token and the agent tokens that remain after masking are used to condition the diffusion model as described in Sec. 3.3.2.

During inference we can modulate the value represented by the global scene token to control how many additional agents the model generates beyond those described by agent tokens. Fig. 3 shows scenarios generated by the model using a fixed map, three agent tokens, and the global scene token. All scenarios contain the three agents described by the tokens. As the value set in the global scene token increases, the model infers that there are additional agents not described by tokens. This approach allows a user to specify certain agents while having the model fill in the remainder of the scenario. Using this formulation we are able to emulate a model conditioned on only the map image by setting the global scene token to 100% and providing no additional agent tokens.

We first evaluate the model's ability to use the token information by matching generated agents to tokens. We take samples from the Argoverse validation dataset, compute agent tokens from the ground truth scene, and apply a fixed amount of agent token masking. We generate scenes conditioned on these tokens and compute a one-to-one matching between generated agents and agent tokens using the Hungarian algorithm [20]. We only consider pairs that are within 2.2 meters (we discretize position into bins of width 1.56m, and 1 . 56 ∗ √ 2 ≈ 2 . 2 ) and 0.2 radians. In Tab. 2 we report the

Final Speed: 0 m/s

Final Speed: 8 m/s

Past

Future

Figure 4: Generated Argoverse scenarios conditioned on the same map image and one agent token with different values for the final speed (2 seconds in the future). The agent corresponding to the agent token is circled in orange. When the final speed is zero the model generates cross traffic agents, and when the final speed is non-zero the horizontal traffic has right of way. In both cases the current pose and speed of the agent token are the same; the only difference is the future speed. Scenario Diffusion is able to infer how the future behavior described in the conditional inputs should impact the placement of additional agents.

agent token match rate (what percentage of agent tokens were matched to a corresponding generated agent) and the number of additional generated agents per scene (those unmatched to an agent token). As the value of p mask increases the agent token match rate stays at 96-97% while the average number of additional agents increases roughly linearly, showing that the model is able to follow both agent and global scene tokens.

4.3.2 Token Controllability

To evaluate how well the model uses other information in the agent tokens we compare models that use different agent token features. We run inference conditioned on agent tokens computed from the ground truth sample as in Sec. 4.3.1. For these experiments we use no token dropout (i.e. p mask = 0 ). We perform the same matching between generated and ground truth agents and report the agent token match rate as in the previous section. For each matched pair, we compute the error of the current speed and final (i.e. t = 2 ) speed. In Tab. 3 we report the mean absolute error (MAE) over all matched pairs, again showing the mean and standard deviation of this metric across multiple models trained with different initial random seeds. As described in Appendix C.3, the speed features are discretized into bins of width 2 m/s, so these models are not expected to achieve perfect reconstruction.

Adding current speed to the agent tokens significantly decreases the MAE for current speed and somewhat decreases it for final speed (as the current and final speed are correlated). Adding a final speed feature to the agent tokens further reduces the MAE for final speed. Adding these additional agent features does not significantly impact the agent token match rate.

4.3.3 Relationship Between Agents

One of the key contributions of our work is to generate bounding boxes and trajectories simultaneously for all agents. Doing so allows the generated outputs to capture not just a static snapshot of the scene but also the behavior of various agents. This joint inference of the placement and behavior of agents is particularly important for controlled scenario generation using agent tokens. The behavior described by agent tokens influences where other agents can be placed. Approaches that separate initial placement and behavior into two models are not able to perform this joint reasoning.

Fig. 4 shows scenarios generated using one agent token, with features for the position, heading, extents, current speed, and final speed (i.e. at +2s). Both cases use the same position, heading, extents, and current speed (set to 0 m/s); the only difference is the final speed. For this vehicle stopped at an intersection, its future behavior implies which lanes have right-of-way through the intersection. When

Evaluated on:

SEAScene

SFScene

Trained on:

SEAModel

SF Model

SEAModel

SF Model

Past

Future

Evaluated on:

SLACScene

Trained on:

SLACModel

LVModel

SLACModel

Past

Future

the future speed is 0 m/s (resp., 8 m/s) the model generates additional agents traveling vertically (resp., horizontally).

4.4 Generalization Across Geographic Region

To evaluate the ability to generalize across regions, we train models on data from each region of the internal dataset separately. We then evaluate these models and models trained on the full dataset on validation splits for each region. Results are shown in Tab. 4. Each entry is an average across 5 experiments with different random seeds, with the standard deviation reported in parentheses.

In each region, models trained on that region tend to outperform models trained on other regions. The models trained on the full dataset come close to the region-specialized models. This suggests that there are unique aspects of each region, and that the full model has sufficient capacity to capture this diversity. SLAC is the most distinct from the other regions and shows the largest generalization gap. Additional metrics can be found in Appendix D.3; they follow the same pattern as in Tab. 4.

Fig. 5 shows examples of models generalizing across the three metropolitan regions (LV, SEA, and SF). The model trained on San Francisco data is able to produce reasonable scenes in Seattle, and vice versa. This suggests the potential to produce validation data for new regions before large volumes of driving logs in the new regions have been collected.

Generalizing to dissimilar regions is significantly harder. Fig. 6 shows examples of generalization between LV (a major metropolitan area) and SLAC (a suburban academic campus). Not surprisingly, models trained on each of these two regions performs worse when applied to the other. For example, the LV model does not generate realistic parked vehicles in the parking lots from SLAC and the

SLAC model generates vehicles facing against the lane direction at locations in LV. Fortunately, the model trained on the full dataset is able to produce high quality scenarios in both regions.

5 Related Works

A number of prior works have used heuristics or fixed grammars to generate driving scenes [5, 7, 16, 17]. Such approaches are limited in their ability to generate complex and diverse driving scenes. SceneGen [32] and TrafficGen [6] use an autoregressive approach, adding agents one at a time. TrafficGen adds a second stage where a motion forecasting model [33] is used to generate trajectories for the boxes. SimNet [2] uses a conditional generative adversarial network [15] to generate a BEV occupancy image. Heuristic post-processing is applied to identify connected components and fit bounding boxes to them. A second agent-centric model is then used to generate trajectories for these agents over time. In contrast, our method directly produces bounding boxes and trajectories simultaneously for all agents in an end-to-end differentiable architecture. Our approach also does not constrain vehicles to be in lane, as in TrafficGen and several heuristic approaches.

A number of works generate future trajectories for already-existing agents in simulation [14, 31, 37, 39, 40]. Some focus on specifically producing adversarial future trajectories given agents' initial states [4, 10, 35]. These approaches are complementary to our work, and can be used to extend trajectories beyond what is output by Scenario Diffusion.

6 Conclusion

In this paper, we demonstrate a novel technique for using diffusion to learn to generate scenarios of dynamic agents moving through an environment for the purposes of testing an autonomous vehicle's ability to navigate through that environment and plan according to those agents. We have shown that our technique leads to models that not only appropriately capture the desired distributions of scenarios and agent trajectories, but allow scenario generation to be controlled to target specific types of scenarios. The fact that diffusion allows declarative information such as tokens to be combined with models learned from data creates the potential for new models that combine other forms of symbolic knowledge such as traffic rules, physical constraints, and common sense reasoning.

Limitations: More work is required to show broader generalization. The training data assumes a specific model of perception in the form of bounding boxes and trajectory models. Additionally, we restricted the agent models to on-road vehicle models. While we do not anticipate significant challenges in applying this model to other forms of agents such as pedestrians, we have not incorporated those agents in the research described here. In simulation this approach may need to be extended to iteratively generate agents over a larger region as the A V navigates through the environment.

Broader Impact: This paper focuses on developing models to improve self-driving car technologies. There are many positive and negative aspects to the development of self-driving cars that depend as much on the system-wide design and regulatory aspects as these aspects depend on the technical capabilities. However, the focus on simulation in this paper should partially reduce the risks of deploying self-driving cars by providing more effective and systematic coverage of testing scenarios.

Acknowledgements: We would like to thank Gary Linscott, Jake Ware, and Yan Chang for helpful feedback on the paper; Allan Zelener and Chris Song for discussions on object detection; Peter Schleede for discussions on diffusion. We also thank the NeurIPS anonymous reviewers, area chair, and program chairs.

References

Elmira Amirloo, Amir Rasouli, Peter Lakner, Mohsen Rohani, and Jun Luo. LatentFormer: Multi-Agent Transformer-Based Interaction Modeling and Trajectory Prediction. 2022. URL https://arxiv.org/ abs/2203.01880 .

Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blazej Osinski, Hugo Grimmett, and Peter Ondruska. SimNet: Learning Reactive Self-driving Simulations from Real-world Observations. ICRA , 2021. URL http://arxiv.org/abs/2105.12332 .

Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. NeurIPS , 2021. URL http://arxiv.org/abs/2105.05233 .

Wenhao Ding, Baiming Chen, Minjun Xu, and Ding Zhao. Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method. IROS , 2020. URL https://arxiv.org/abs/2003.01197 .

Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: An Open Urban Driving Simulator. CoRL , 2017. URL http://arxiv.org/abs/1711.03938 .

Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios. ICRA , 2023. URL http://arxiv.org/abs/2210.06609 .

Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example-based Synthesis of 3d Object Arrangements. ACM Transactions on Graphics (TOG) , 31(6):1-11, 2012.

Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling. ICLR , 2022. URL https://arxiv.org/abs/2110.06607 .

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research , 13(25):723-773, 2012. URL http://jmlr.org/papers/v13/gretton12a.html .

Niklas Hanselmann, Katrin Renz, Kashyap Chitta, Apratim Bhattacharyya, and Andreas Geiger. King: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients. ECCV , 2022. URL https://arxiv.org/abs/2204.13683 .

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. ICLR , 2017. URL https://openreview.net/forum?id=Sy2fzU9gl .

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS , 2020. URL https://arxiv.org/abs/2006.11239 .

Joey Hong, Benjamin Sapp, and James Philbin. Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions. CVPR , 2019. URL http://arxiv.org/abs/1906. 08945 .

Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation. ICRA , 2022. URL https://arxiv.org/abs/ 2205.03195 .

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-Image Translation with Conditional Adversarial Networks. CVPR , 2017. URL https://arxiv.org/abs/1611.07004 .

Gregory Izatt and Russ Tedrake. Generative Modeling of Environments with Scene Grammars and Variational Inference. ICRA , 2020. URL https://groups.csail.mit.edu/robotics-center/public_ papers/Izatt20.pdf .

Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-Sim: Learning to Generate Synthetic Datasets. ICCV , 2019. URL http://arxiv.org/abs/1904.11621 .

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS , 2022. URL http://arxiv.org/abs/2206.00364 .

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR , 2015. URL https://arxiv.org/abs/1412.6980 .

H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly , 2 (1-2):83-97, 1955.

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. ICLR , 2019. URL http: //arxiv.org/abs/1711.05101 .

Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, and Benjamin Sapp. Wayformer: Motion Forecasting via Simple & Efficient Attention Networks. ICRA , 2023. URL https://arxiv.org/abs/2207.05844 .

Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp, Zhifeng Chen, and Jonathon Shlens. Scene Transformer: A unified architecture for predicting multiple agent trajectories. ICLR , 2022. URL http://arxiv.org/abs/2106.08417 .

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR , 2022. URL https://arxiv.org/abs/2112. 10752 .

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Multi-Agent Generative Trajectory Forecasting With Heterogeneous Data for Control. ECCV , 2020. URL http: //arxiv.org/abs/2001.03093 .

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. Data Centric AI NeurIPS Workshop , 2021. URL https://arxiv.org/ abs/2111.02114 .

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML , 2015. URL http://arxiv.org/abs/1503. 03585 .

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. ICLR , 2021. URL https://arxiv.org/abs/2010.02502 .

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR , 2021. URL http://arxiv.org/abs/2011.13456 .

Peize Sun, Yi Jiang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. What Makes for End-to-End Object Detection? ICML , 2021. URL https://arxiv.org/abs/2012.05780 .

Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors. CVPR , 2021. URL https://arxiv.org/abs/2101.06557 .

Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. SceneGen: Learning to Generate Realistic Traffic Scenes. CVPR , 2021. URL https://arxiv.org/ abs/2101.06541 .

Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi-Pang Lam, Dragomir Anguelov, and Benjamin Sapp. MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction. ICRA , 2022. URL https://arxiv.org/abs/2111.14973 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. NeurIPS , 2017. URL http://arxiv.org/abs/ 1706.03762 .

Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles. CVPR , 2021. URL https://arxiv.org/abs/2101.06549 .

Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next Generation Datasets for Self-driving Perception and Forecasting. NeurIPS , 2021. URL https://arxiv.org/abs/2301.00493 .

Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: Bi-level Imitation for Traffic Simulation. ICRA , 2023. URL https://arxiv.org/abs/2208.12403 .

Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting. ICCV , 2021. URL https://arxiv.org/abs/2103.14023 .

Qichao Zhang, Yinfeng Gao, Yikang Zhang, Youtian Guo, Dawei Ding, Yunpeng Wang, Peng Sun, and Dongbin Zhao. TrajGen: Generating Realistic and Diverse Trajectories With Reactive and Feasible Agent Behaviors for Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems , 23(12): 24474-24487, 2022. URL https://arxiv.org/abs/2203.16792 .

Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided Conditional Diffusion for Controllable Traffic Simulation. ICRA , 2023. URL http://arxiv.org/abs/2210.17366 .

A

B

C

H

1

D

E

F

G

A

B

C

H

日

D

I1

E

I1

F

I1

G

I1

A Datasets

A.1 Argoverse 2

The Argoverse 2 motion forecasting dataset [36] contains 250,000 driving scenarios, each 11 seconds long. These scenarios cover 6 geographical regions and represent 763 total hours of driving.

To generate inputs for Scenario Diffusion we take all 4s windows (for -2 seconds to +2 seconds) and render an input example for each. To do this rendering we use the provided visualization library to plot a BEV image of the map and add an additional image showing the lane center lines colored by the lane direction. For the entity layers we use the provided visualization library to plot vehicles and buses as rectangles of two different shades of blue. To disambiguate the orientation of the boxes we also render a line from the center of each box to show the heading.

As the argoverse dataset does not include information on agent dimensions, we use constants for each object type (i.e. vehicle and bus) as in the argoverse visualization library. These constants are also used to define the ground truth boxes used to train the autoencoder.

To represent agents' trajectories in the topdown image, we render images for each time delta along the trajectory. For t ∈ {-2 , -1 , 0 , 1 , 2 } , let p t = ( p ( x ) t , p ( y ) t ) be the agent's position and m t a mask for whether the agent was observed at the given timestep. We compute four position deltas and corresponding masks:

■

三

"

For each i ∈ {-2 , -1 , 1 , 2 } we add channels where each agent's bounding box from t = 0 is filled with α i , clip ( ∆ ( x ) i / 30 + 0 . 5 , 0 , 1 ) , and clip ( ∆ ( y ) i / 30 + 0 . 5 , 0 , 1 ) .

Scenes are centered on the autonomous vehicle, which is treated as just another vehicle, and cover an area of 100m x 100m. Examples of the topdown inputs m and x for this dataset can be seen in Fig. 7.

A.2 Internal Dataset

The internal dataset contains 6 million real world driving scenarios from Las Vegas (LV), Seattle (SEA), San Francisco (SF), and the campus of the Stanford Linear Accelerator Center (SLAC). Note that Las Vegas, Seattle, and San Francisco represent dense urban settings, while SLAC is a suburban academic campus. Training and validation data are not separated by location, so a given validation example may have training examples from the same location at other times. Scenarios are sampled every 0.5 seconds from real-world driving logs.

Map information comes from a pre-defined high definition map and vehicle tracks are detected by an on-vehicle perception system. We filter these tracks to only include bounding boxes that overlap with drivable areas in the map (e.g. roads, public parking lots, driveways).

Scenes are centered on the autonomous vehicle, which is treated as just another vehicle during training, and cover an area of 100m x 100m. Examples of scenes from this dataset can be seen in figure Fig. 8.

Legend

256 px block

Downsample

128 px block

Downsample

64 px block

Downsample

32 px block

D

256 px block

Downsample

128 px block

Downsample

64 px block

Downsample

32 px block

32 px block

Upsample

64 px block

Concat

256 pxblock

Downsample

128 px block

Downsample

64 px block

Downsample

32 px block

32 px block

Downsample

16pxblock

Downsample

8px block

Upsample

16 px block

Upsample

32 px block

Concat

Legend

M

m

Tokens

B Scenario Autoencoder Details

B.1 Architecture Details

The architecture for the scenario autoencoder is based on [24] and is shown in Fig. 9. We modify the architecture to add the conditional image m as an input to the decoder D ( z, m ) . The map image is encoded using the same architecture as E (but different weights) to obtain a latent embedding with the same pixel dimension. We also truncate the upsampling stage of the decoder to output box proposals at a 64 pixel resolution. The same architecture is used for both the Argoverse and internal datasets.

The 256 pixel and 128 pixel blocks use 64 channels, and the 64 pixel and 32 pixel blocks use 128 channels. We use one Resnet block per resolution.

B.2 Reconstruction Loss

To compute the reconstruction loss we first match a predicted box to each ground truth box and then apply loss terms between matched pairs. The losses and matching use the same cost functions with different weights.

The classification loss is a binary cross-entropy loss between the predicted box's logit and the probability of the ground truth box (defined to be 1). The coordinate loss applies an L1 loss on the 6 parameters that define the box shape (the cosine and sine of the box heading and the log of the distance to the front, left, back, and right of the box). The vertex loss applies an L2 loss on the positions of the four corners of the box. The trajectory position loss applies an L2 loss on the positions of the trajectory. The trajectory heading loss applies a cosine loss on the headings of the trajectory.

To match a predicted box to a given ground truth box we select the predicted box that minimizes a weighted combination of the classification, coordinate, and vertex losses. The classification term is up-weighted by a factor of 4 over the other two terms.

We apply the five loss functions on the matched pairs. For this loss the classification term is up-weighted by a factor of 20 over the other four terms.

To balance between different motion profiles we categorize each ground truth trajectory as stationary, straight, or turning. To better balance between these three categories we weight the trajectory losses for each category by 0.1, 0.3, and 4, respectively.

We also use a binary cross-entropy loss on the unmatched predicted boxes to have probability 0. As there are many more such negative predictions, we down-weight this loss term. Negative boxes within 3 meters of a ground truth box are down-weighted by a factor of 0.2, and negative boxes more than 3 meters from a ground truth box are down-weighted by a factor of 0.002.

B.3 Training Details

The scenario autoencoder is trained using the Adam optimizer [19], initial learning rate of 1e-4, and weight decay of 1e-4. The learning rate is decreased by a factor of 10 when the validation loss plateaus. Gradients are clipped to have norm less than or equal to 1. The KL regularization loss L KL is weighted by 0.1 compared to the reconstruction loss described above. We use 2 RTX A6000 GPUs with a batch size of 16 per GPU.

C Scenario Diffusion Details

C.1 Architecture Details

The architecture for the scenario diffusion model is also based on [24] and shown in Fig. 10. An encoder E m with the same architecture as E with different weights process the conditional map image to the same latent dimension. The noisy latent embedding z and the encoded map information E m ( m ) are concatenated to form the input to the denoising Unet M . In the lower resolution blocks of M , the cross attention layers attend to the conditional tokens. The same architecture is used for both the Argoverse and internal datasets.

The 32 pixel, 16 pixel, and 8 pixel blocks of M use 64, 128, and 256 channels, respectively. Each resolution block features a single Resnet block. Cross attention is performed with 8 attention heads. The embedded map information E m ( m ) uses 64 channels.

C.2 Training Details

We use the EDM training algorithm [18]. We set σ min = 0 . 02 , σ max = 20 , σ data = 0 . 5 , ρ = 7 , P µ = -0 . 5 , and P σ = 1 . See Table 1 of [18] for more information (note that P µ and P σ are referred to as P mean and P std in that table).

The diffusion model is trained with an AdamW optimizer [21], an initial learning rate of 3e-4, and weight decay of 1e-5. The learning rate is decreased by a factor of 10 when the validation loss plateaus. We use 4 RTX A6000 GPUs with a batch size of 64 per GPU.

C.3 Token Details

In this work we consider the following possible agent token features:

Current x position relative to the scene center

Current y position relative to the scene center

Current heading

Box length

Box width

Current speed

Final speed (at +2s)

Relative heading from the t = 0 position to the t = 2 position

A given model will use a subset of these features for its agent tokens. For brevity we use 'current pose' to refer to the trio of current x position, current y position, and current heading and 'extents' to refer to the pair of box length and box width.

Each agent token feature is processed by a per-feature embedding function; the embeddings of all features are concatenated to get the final token feature vector.

The x and y positions are each embedded using a one-hot vector for 64 bins over the extent of the scene (100m). The current heading is encoded with two channels containing the cosine and sine of the heading. The length and width are encoded with one channel containing the log1p of the value. Current and final speed are encoded using a one-hot vector for 10 bins over the interval [0 , 20] in m/s. Speeds above 20 m/s are clipped to 20 m/s. The relative heading is encoded using a one-hot vector for 10 bins over the interval [ -0 . 7 , 0 . 7] .

Tab. 5 lists the agent token features used for the various figures shown in this paper.

D Additional Experiments

D.1 Autoencoder Ablation Experiments

One of the key hyperparameters for Scenario Diffusion is the latent embedding dimension, the number of channels in z . Tab. 6 shows the results of ablation experiments training diffusion models with different autoencoders using different latent dimensions. We find the latent embedding dimension used for experiments in the main paper (4) performs slightly better on MMD metrics and achieves trajectory metrics slighly closer to those of ground truth logs. However, the difference is not statistically significant given the standard deviation across diffusion models trained with different random initial seeds.

D.2 Diffusion Ablation Experiments

We begin by measuring the effect of changing the probability threshold used during inference. To do so, we repeat the procedure described in Sec. 4.3.1 of sampling scenarios from the Argoverse validation dataset and computing agent tokens from the ground truth agents in the scene. Using p mask = 0 , we expect the generated agents to match the ground truth agents. We perform the same matching algorithm described in Sec. 4.3.1. Treating the matching as a binary outcome, we

can define each matched pair of a ground truth and generated agent to be a 'true positive' (TP), ground truth agents that are not matched to generated agents as 'false negatives' (FN), and generated agents that are not matched to ground truth agents as 'false positives' (FP). From these numbers, we can compute precision = TP / ( TP + FP ) , recall = TP / ( TP + FN ) , and F1 = 2 · precision · recall / ( precision + recall ) . Note that this definition of 'recall' is the 'Agent token match rate' and 'false positives' are the 'additional generated agents' from Tab. 2. Precision makes sense to measure in this context because we fix p mask = 0 , so the number of additional generated agents should be close to 0.

These results are shown in Tab. 7, reporting the mean and standard deviation across multiple models trained with different random initial seeds. We observe that precision increases and recall decreases as the probability threshold increases. The mean F1 score is maximized with a probability threshold of 75% or 80%. From this, we select 80% as the threshold to use for most experiments. However, we note that the differences in F1 score are relatively small for probability thresholds between 40% and 95%.

Tab. 8 shows the results of ablation experiments on several key hyperparameters of the diffusion model architecture using the internal dataset. The values used in the experiments for the main paper are bolded. In general, we find that increasing the model size does not significantly improve the available metrics.

D.3 Cross Region Generalization

Tab. 9 shows additional metrics from the evaluation described in Sec. 4.4. These metrics follow the same pattern as Tab. 4. This table introduces a new metric: the earth movers' distance (EMD) on the number of generated agents. This metric measures how much the probability mass needs to move to

Global Scene Token = 0%

Global Scene Token = 50%

2Agent

Tokens

3Agent

Tokens

transform from one probability distribution to another. In this context, the distributions in question are 1D histograms of the number of agents per scene, measured over the ground truth validation scenes and generated scenes using the same map locations.

D.4 Qualitative Results

Fig. 11 shows scenarios generated using different numbers of agent tokens and different values for the global scene token. Fig. 12 shows adversarial scenarios in a junction created using two agent tokens. These examples illustrate how token controllability allows users to create specific types of scenarios, while letting the model figure out most of the details.

Fig. 13 and Fig. 14 show two more examples of modifying the final speed of agent tokens while keeping all other input features the same. This can be performed with any number of agent tokens.

Fig. 15 takes a ground truth scene from the Argoverse validation dataset, computes agent tokens from two vehicles of interest, and generates novel scenes using these tokens. This process allows users to sample modified versions of a given log to see how the placement of other agents impacts the AV behavior.

9

Final Speed: 1 m/s

Final Speed: 7 m/s

0

Past

Future

Final Speed: 1 m/s

Final Speed: 5 m/s

Past

Future

E TrafficGen

Several modifications were made to TrafficGen [6] accommodate the Argoverse 2 dataset [36]. Argoverse 2 does not contain traffic light information, so such features are not used in the map context. Argoverse 2 also does not include the length and width of agent boxes. In the provided plotting code fixed constants are used for different agent types. We include the 'car' and 'bus' agent types for training TrafficGen, and update the size prediction head to perform 2-way classification instead of outputting a Gaussian mixture model. To match Scenario Diffusion, we set the model to only generate vehicles in the 100m region around the AV.

Fig. 16 shows scenes generated by our implementation of TrafficGen.

Real World

Log

Past

Future

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work adopts a conditional diffusion model to generate the abstract driving scene.\n\nSTRENGTHS:\nIt has great potentials to generate future scenes with diffusion model.\n\nWEAKNESSES:\n(W1) No comparsion with existing works at all. It is significant to compare with previous methods to demonstrate the superiorty of the proposed method in terms of diversity, generalizability or other aspects. Quantitative comparisons with methods such as SceneGen [1], TrafficGen [2], CTG [3], and Scene Diffusion [4] would help to make the improvement more convincing and easier to recognize. Could the authors provide related results or briefly discuss why these methods were not included in the comparison?\n\n(W2) The proposed method aims to generate all agents simultaneously so that the relationships between them can be learned. Nevertheless, auto-regressive manner also embodies such functionality as it can condition on the generated agents. Have the authors explored the use of auto-regressive ways within the proposed framework?\n\n(W3) As a generation work, a great potential would be producing additional data for training, customizing safety critical scenes for evaluation. However, this part is missing from the current version, which means the claimed benefit is unjustified at all. Could the authors complement some experiments to support that?\n\n[1] SceneGen: Learning to Generate Realistic Traffic Scenes\n[2] TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios\n[3] Guided Conditional Diffusion for Controllable Traffic Simulation\n[4] Generating Driving Scenes with Diffusion\n\nQUESTIONS:\nSee weakness for details.\n\nI am mostly concerned that this work only decribe their method but do not compare with rich existing literatures at all.  It is hard to tell whether the proposed method really works and is better than existing works.\n\nLIMITATIONS:\nNo concern.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to use latent diffusion for driving scenario generation. The generated driving scenario is defined as bounding boxes with associated trajectories. The proposed diffusion model is comprised of an autoencoder to encode the BEV image into latent space and a conditional UNet for diffusion modeling. To increase the controllability of the generated scenario, map conditioning and token conditioning are incorporated into the process. The experiments are conducted both on the large-scale internal dataset and the Argoverse 2 dataset, which validates the effectiveness of the proposed algorithm and its flexibility for controllable generation.\n\nSTRENGTHS:\n1. The idea of applying latent diffusion to driving scenario generation is novel and interesting, as diffusion methods have shown great capability in generative tasks.\n2. The proposed token conditioning mechanism brings more controllability and the possibility of customization for this task.\n4. The overall writing and presentation of the paper are clear and easy to follow.\n\nWEAKNESSES:\n1. For the metrics to evaluate the generated scenarios, it is very important to evaluate the validity of the driving scenario, but the paper only measures the rate of off-road trajectories. Other aspects like the rate of collisions between agents, smoothness and plausibility of the trajectories, and possible traffic rules violations should also be measured.\n2. Although in Sec 4.3.2 some examples have shown that the proposed method is aware of the relationship between agents, using BEV encoded image to represent the scene without specific designs for agents interactions could not well capture the relationships between agents compared with those GNN-based methods shown in motion-forecasting. Having specific designs to explicitly model the possible interactions is important for the complex driving scenario generation.\n4. There is no direct quantitative measurement of how the generated sample faithfully follows the conditioning agent and global token.\n3. The goal of scenario generation is to facilitate the development of other autonomous driving tasks like motion forecasting or planning. Therefore, a better way to show the effectiveness and diversity of the generated scenario is by testing them on downstream tasks. Some simple experiments showing the performances of a motion forecasting model training on real and generated data would greatly improve the value of the proposed method.\n\nQUESTIONS:\n1. During the sampling process, would it be helpful to use techniques similar to classifier-free guidance to make the sample ground on the conditioning token more faithfully?\n2. Currently the agent token still needs very specific values of different aspects to define its behavior, is it possible to use some more abstract behavior tokens to only specify its overall behavior like overtaking, speed up, right turn, etc. to promote better diversity without much engineering effort?\n\nLIMITATIONS:\nThe authors have well discussed the limitations and the potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of generating both the initial configuration and vehicle trajectories for driving scenarios given an input top-down map. It proposes a latent diffusion model that operates in the latent space of a learned autoencoder trained to decode vehicle boxes and trajectories. The model can be directly conditioned to encourage a subset of agents to adhere to user specifications. Experiments show that once trained on a large real-world driving dataset, the model can produce reasonable scenarios at multiple scales and allows for user control over a subset of agents.\n\nSTRENGTHS:\nThe problem is important for AV testing and training.\n\nThe proposed method simultaneously generates both an initial scene configuration and vehicle trajectories, instead of in two stages as in prior work.\n\nThe proposed latent diffusion model is novel for vehicle trajectories – all prior work I have seen is explicit trajectory diffusion and the design of the autoencoder for such a model is not trivial.\n\nThe model allows directly (optionally) conditioning on attributes of one or more agents which gives nice flexibility and allows generating variations of scenarios directly taken from e.g. a log or specified by a user.\n\nResults of generating short 4 sec scenarios look reasonable. Experiments show the ability to generate scenarios of various sizes on the sizable Argoverse 2 dataset. Fig 4 is a neat demo showing that the model is picking up on the semantics of the relationships between vehicles.\n\nThe appendix is extensive and provides most of the necessary details to enable reproducibility.\n\nWEAKNESSES:\nThere are a couple claims that seem over-stated and should be toned down or clarified:\n* Conditioning tokens are “more abstract features” (L208) that does not require “fully specifying box and trajectory for the agent” (L206) or “having observed such agents in the real world” (L43) – if I understand correctly, these tokens require a user to give the model the desired agent position, heading, length, width, speed, and sometimes curvature over time (L211),  which is quite a low-level specification of the trajectory and likely will need to be taken from a log. Or do tokens describe the state only at a single $t=0$? If so, this should be made more clear, and I still would not say this is “abstract”.\n* The model “generalizes to different geographical regions” (L9) – Table 3 shows seemingly limited ability to generalize across regions with models trained specifically on each region far outperforming other single-region models. The model trained on all regions does reasonably well on each, but this is not generalizing across regions. Moreover, it would be helpful to add a metric that indicates similarity between regions (e.g. MMD between ground truth scenes in A and B) to put the numbers in context. While this experiment is informative about performance across region, it may be a stretch to say it demonstrates generalization across regions. \n\nSome limitations of the technical approach may make the proposed method less practically useful:\n* The agent tokens are not hard constraints and in results like Fig 3 the specified agents move around between samples and don’t exactly meet user specifications. A natural solution for diffusion would be to use some kind of test-time guidance as in [37] and [Trace and Pace, Rempe et al., CVPR 2023].\n* The 4 sec time horizon (2 sec past + 2 sec future) seems quite short and limits the possible scenarios it can capture. \n* The conditioning requires a user to specify all attributes of an agent. But it seems with the masking strategy it should be possible to condition on partial information (e.g. specify the past trajectory but generate the future, or only specifying speed) which would greatly enhance the flexibility.\n\nSome additional evaluation metrics would help to evaluate realism and diversity. For example, MMD on speeds, collision rate between vehicles, and a diversity metric (like average pairwise distance between multiple samples). Additionally, some key experiments are missing:\n* The accuracy of the model’s controllability is never evaluated, which is the main claim of the paper. It would be good to report how closely generated agents follow the agent tokens specified by the user (e.g. L2 distance between positions, headings, speed, etc..). The 0% model in Table 2 kind of does this, but the reported MMD metric is not very intuitive.\n* A comparison to a baseline to justify the advantages a diffusion model to generate both static states and dynamic trajectories jointly. E.g. a comparison to TrafficGen [6] or a simpler baseline that only generates static states and then uses a trajectory prediction or traffic simulation model to roll out motion, e.g. [BITS: Bi-level Imitation for Traffic Simulation, Xu et al., ICRA 2023]. Or evaluating if the trajectories generated by the diffusion model align with those from a SOTA trajectory forecasting model?\n* It would be good to see an example of how these scenarios can be useful. For example, that training a driving policy on a dataset augmented with generated scenarios does better than just a real-world dataset. \n* More qualitative results (ideally videos, but additional figures in the supplement would also be helpful).\n\nQUESTIONS:\nOverall, I am borderline on this paper. While the latent diffusion formulation is novel and technically sound and the model nicely generates both starting positions and motion for the scenario, the primary contribution of “controllability” is not sufficiently evaluated, and additional evaluations/baselines would strengthen the paper. In the rebuttal, I would really like the authors to clarify my confusion on the agent tokens and ideally evaluate the controllability of the model by showing how closely it follows user specifications. \n\nThe method Sec 3 at times is rather high level and leaves some questions:\n* Sec 3.1: why is the encoder not conditioned on the map like in the typical CVAE formulation?\n* Sec 3.3: A lot of info about tokens are relegated to experiments in Sec 4.3.1 or the appendix but may be better to include in the methods. I’m still left wondering how exactly the cross attention works, e.g. how are the tokens associated with agents in the scene since there is no canonical agent IDs or ordering? Also, if tokens are specifying the properties of agent trajectories, why do multiple samples of the same scenario (e.g. Figs 1, 7, 8) exhibit varying trajectories for the specified agents?\n* How are the number of agents in the scene determined by the diffusion model? Through the agent probability? Why is the global scene token represented as the masking probability and not directly the number of agents that are desired in the scene?\n* The methods mentions 2 agent tokens are used, but then 3 are used in Fig 3 and N are used in Table 2, so it’s not clear what is the problem formulation and what’s used in practice.\n\nAnother other small suggestion: since explicit trajectory diffusion models are becoming more prevalent [37] [Trace and Pace, Rempe et al., CVPR 2023] [Stochastic trajectory prediction via motion indeterminancy diffusion, Gu et al., CVPR 2022] [Planning with Diffusion for Flexible Behavior Synthesis, Janner et al., ICML 2022], it would be good to discuss them in related work and compare to the proposed latent diffusion. \n\n===================== After Rebuttal ======================\n\nAfter considering other reviews and discussion with the authors and reviewers, I have decided to keep my rating leaning towards reject. \n\nGiven that the problem is relatively new and the latent diffusion approach is novel, I am willing to be lenient on the lack of a baseline comparison. Especially since TrafficGen (ICRA 2023) is concurrent work and SceneGen (CVPR 2021) did not release code. However, without a baseline, the experiments and metrics need to be comprehensive and convincing to show generated scenes are realistic and match user specifications; I don’t believe this is the case in the current draft. The rebuttal Table 1 adds some new map-based metrics, but important realism evaluations like collision rate, plausibility of trajectories (smoothness and comfort), and diversity are still missing. Moreover, the added evaluation of controllability in rebuttal Figure 3 is rather coarse (considered “correct” if within 2.2 m) and does not consider other user specifications like length, width, speed, and curvature. \n\nI also think it would be a strong addition to show results on higher-level semantic conditioning of the model such as vehicle type and maneuver type (e.g. lane change, turning, or u-turn) to justify the claim that the model can handle “abstract” specifications rather than just partial explicit specifications.\n\nLIMITATIONS:\nLimitations are sufficiently discussed in Sec 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a conditional latent diffusion model (LDM) for generating oriented BEV vehicle bounding boxes, each associated with a 4-second trajectory (2s past, 2s future) at a 1s temporal resolution. The autoencoder component of this LDM uses an object detection architecture similar to CenterNet, with rasterized agent inputs. The diffusion model is conditionable on the local map, a global traffic density parameter, and an optional per-agent attribute descriptor. The model is validated on an internal proprietary dataset and Argoverse 2 via a series of ablation studies.\n\nSTRENGTHS:\nWhat stands out in this work is the overall simplicity and elegance of the framework. It draws on ideas from different communities (generative modeling, object detection, traffic simulation). It also includes interesting original ideas that are potentially applicable to other generative modeling tasks in robotics and self-driving, such as (1) the imbalanced VAE (raster input, vector output); and (2) the masking idea to make agent-level conditioning optional. The writing is clear and mostly easy to follow.\n\nWEAKNESSES:\nThe key weakness of this work is the lack of any baselines in the experimental analysis. Note that this is not a completely novel task (Section 5 discusses prior techniques). While it is true that not all prior work may be conditionable on per-agent attributes like the proposed model, I believe a comparison to prior work (potentially in the setting without agent token conditioning, as in the last column of Table 2) is warranted to meaningfully understand how well diffusion models perform this task. Secondly, the diffusion model architecture is hard to understand from the main paper in its current form, and a clear description is only available in the supplementary. For example, L_{KL} is marked in Fig. 2a but not described at any point in the main text.\n\nQUESTIONS:\n1. Would it be possible to include one heuristic and one autoregressive approach (as mentioned in Section 5) for the Argoverse dataset as additional columns in Table 2? This is the main limitation of the paper from my perspective, and I am happy to raise my score if it is addressed, or a convincing reason for why such baselines are not useful is provided.\n2. Since the conditioning (controlability) is pitched as a key contribution, this should be clearly understandable from Fig. 2b. Would it be possible to expand “M” to show both its encoder and decoder in Fig. 2, with more clarity on how the different inputs/conditions are passed into the diffusion model architecture, instead of a single input arrow?\n\nMinor:\n\n1. I couldn’t understand the caption in Table 3, specifically the text enclosed in brackets\n2. [34] and [37] have incorrect citation “year” fields (2022 → 2023)\n\nUpdate:\n\nThanks a lot for your clarifications and the efforts made to update the draft.\n\nWhile I no longer have any issues regarding the method/presentation, I believe the rebuttal does not fully address the concern regarding limited baselines in the analysis of the proposed model. The other reviewers have also commented on the insufficiency of the current experiments to demonstrate a high degree of realism and controllability with the proposed model, and I concur. Therefore, I am maintaining my rating of borderline reject.\n\nLIMITATIONS:\nThe key limitations (such as the currently limited generalization capability and simple task definition) are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}