SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Global Scene Token  $= 90\%$
Figure 3: Generated Argoverse scenarios conditioned on the same map image and three agent tokens with different values for the global scene token. The generated agents corresponding to the three agent tokens are circled in orange. When the global scene token is set to non-zero values, the model reconstructs the three agents described by tokens and adds additional agents to fill in the scene. Trajectories are omitted from this figure for clarity.
Table 2: Agent token matching metrics on Argoverse
Token Mask Probability pmask	0%	10%	30%	50%	70%	90%
Agent token match rate	0.97 (0.01)	0.97 (0.01)	0.97 (0.01)	0.97 (0.01)	0.96 (0.01)	0.96 (0.02)
Number of additional generated agents	0.4 (0.3)	3.0 (0.8)	5.5 (0.9)	8.0 (0.9)	10.4 (0.8)	12.6 (0.7)
4.3 Token Conditioning
We next explore applications of token conditioning in Scenario Diffusion. Fig. 1 demonstrates how tokens can be used to generate scenarios with a specific type of interaction between the AV and another agent. Our approach is able to generate these interactions without having to fully specify the box and trajectory for the agent (e.g. by hand or by copying an agent from an existing log). By using more abstract features in the agent tokens we allow the model to generate different scenarios that meet the given description (e.g. we do not prescribe the specific trajectory of the u-turning agent).
The flexibility of tokens provides variable degrees of control over the generated scenarios. In this work we demonstrate models that use a variety of features in the agent tokens, such as position, heading, length, and speed. More details on the exact token features can be found in Appendix C.3.
4.3.1 Partial Tokenization
It is crucial that the learned model be robust to the presence or absence of user-provided tokens. To ensure this, during training we omit or "mask" supervision tokens so that the model learns to fill in additional agents in the scenario. During training we select  $p_{\mathrm{mask}}$  such that  $40\%$  of the time we keep all tokens (i.e.  $p_{\mathrm{mask}} = 0$ ) and the other  $60\%$  of the time we sample  $p_{\mathrm{mask}} \sim \mathrm{Beta}(2,1)$ . For each sample, the chosen masking probability is bucketed into one of ten bins ([0, 0.1), [0.1, 0.2), etc.) and embedded as a one-hot vector to define the global scene token. This global scene token and the agent tokens that remain after masking are used to condition the diffusion model as described in Sec. 3.3.2.
During inference we can modulate the value represented by the global scene token to control how many additional agents the model generates beyond those described by agent tokens. Fig. 3 shows scenarios generated by the model using a fixed map, three agent tokens, and the global scene token. All scenarios contain the three agents described by the tokens. As the value set in the global scene token increases, the model infers that there are additional agents not described by tokens. This approach allows a user to specify certain agents while having the model fill in the remainder of the scenario. Using this formulation we are able to emulate a model conditioned on only the map image by setting the global scene token to  $100\%$  and providing no additional agent tokens.
We first evaluate the model's ability to use the token information by matching generated agents to tokens. We take samples from the Argoverse validation dataset, compute agent tokens from the ground truth scene, and apply a fixed amount of agent token masking. We generate scenes conditioned on these tokens and compute a one-to-one matching between generated agents and agent tokens using the Hungarian algorithm [20]. We only consider pairs that are within 2.2 meters (we discretize position into bins of width  $1.56\mathrm{m}$ , and  $1.56*\sqrt{2} \approx 2.2$ ) and 0.2 radians. In Tab. 2 we report the
4. Simpler choices such as sampling  $p_{\mathrm{mask}}$  from a uniform distribution over [0, 1] yields similar results when fully trained. Empirically we find the chosen mixture distribution yields faster convergence.
Final Speed:  $0\mathrm{m / s}$
Figure 4: Generated Argoverse scenarios conditioned on the same map image and one agent token with different values for the final speed (2 seconds in the future). The agent corresponding to the agent token is circled in orange. When the final speed is zero the model generates cross traffic agents, and when the final speed is non-zero the horizontal traffic has right of way. In both cases the current pose and speed of the agent token are the same; the only difference is the future speed. Scenario Diffusion is able to infer how the future behavior described in the conditional inputs should impact the placement of additional agents.
Final Speed:  $8\mathrm{m / s}$
Table 3: Agent token speed metrics on Argoverse
Agent token features	Reconstruction metrics
Current pose	Extents	Current speed	Final speed	Agent token match rate (↑)	Current speed MAE (↓)	Final speed MAE (↓)
✓	✓	0.971 (0.006)	1.43 (0.17)	1.57 (0.17)
✓	✓	✓	0.969 (0.003)	0.49 (0.01)	0.79 (0.02)
✓	✓	✓	✓	0.959 (0.007)	0.49 (0.02)	0.58 (0.03)
agent token match rate (what percentage of agent tokens were matched to a corresponding generated agent) and the number of additional generated agents per scene (those unmatched to an agent token). As the value of  $p_{\mathrm{mask}}$  increases the agent token match rate stays at 96-97% while the average number of additional agents increases roughly linearly, showing that the model is able to follow both agent and global scene tokens.
4.3.2 Token Controllability
To evaluate how well the model uses other information in the agent tokens we compare models that use different agent token features. We run inference conditioned on agent tokens computed from the ground truth sample as in Sec. 4.3.1. For these experiments we use no token dropout (i.e.  $p_{\mathrm{mask}} = 0$ ). We perform the same matching between generated and ground truth agents and report the agent token match rate as in the previous section. For each matched pair, we compute the error of the current speed and final (i.e.  $t = 2$ ) speed. In Tab. 3 we report the mean absolute error (MAE) over all matched pairs, again showing the mean and standard deviation of this metric across multiple models trained with different initial random seeds. As described in Appendix C.3, the speed features are discretized into bins of width  $2 \, \mathrm{m/s}$ , so these models are not expected to achieve perfect reconstruction.
Adding current speed to the agent tokens significantly decreases the MAE for current speed and somewhat decreases it for final speed (as the current and final speed are correlated). Adding a final speed feature to the agent tokens further reduces the MAE for final speed. Adding these additional agent features does not significantly impact the agent token match rate.
4.3.3 Relationship Between Agents
One of the key contributions of our work is to generate bounding boxes and trajectories simultaneously for all agents. Doing so allows the generated outputs to capture not just a static snapshot of the scene but also the behavior of various agents. This joint inference of the placement and behavior of agents is particularly important for controlled scenario generation using agent tokens. The behavior described by agent tokens influences where other agents can be placed. Approaches that separate initial placement and behavior into two models are not able to perform this joint reasoning.
Fig. 4 shows scenarios generated using one agent token, with features for the position, heading, extents, current speed, and final speed (i.e. at  $+2s$ ). Both cases use the same position, heading, extents, and current speed (set to  $0\mathrm{m / s}$ ); the only difference is the final speed. For this vehicle stopped at an intersection, its future behavior implies which lanes have right-of-way through the intersection. When
Figure 5: Comparison of scenarios for the internal dataset generated by models trained on only one urban region. While best results are obtained by using a model trained on data from the region of interest, many scenarios generated by models trained in other regions are plausible for validation.
Figure 6: Comparison of scenarios for the internal dataset generated by models trained on data from distinct regions. SLAC is a suburban research campus, notably different from the other three regions. The model trained only on SLAC produces notably worse results when applied to other regions, and the model trained without data from SLAC fails to pick up on unique aspects of the campus road network. The model trained on all regions is able to generate high quality scenarios in both regions.
the future speed is  $0\mathrm{m / s}$  (resp.,  $8\mathrm{m / s}$ ) the model generates additional agents traveling vertically (resp., horizontally).
4.4 Generalization Across Geographic Region
To evaluate the ability to generalize across regions, we train models on data from each region of the internal dataset separately. We then evaluate these models and models trained on the full dataset on validation splits for each region. Results are shown in Tab. 4. Each entry is an average across 5 experiments with different random seeds, with the standard deviation reported in parentheses.
In each region, models trained on that region tend to outperform models trained on other regions. The models trained on the full dataset come close to the region-specialized models. This suggests that there are unique aspects of each region, and that the full model has sufficient capacity to capture this diversity. SLAC is the most distinct from the other regions and shows the largest generalization gap. Additional metrics can be found in Appendix D.3; they follow the same pattern as in Tab. 4.
Fig. 5 shows examples of models generalizing across the three metropolitan regions (LV, SEA, and SF). The model trained on San Francisco data is able to produce reasonable scenes in Seattle, and vice versa. This suggests the potential to produce validation data for new regions before large volumes of driving logs in the new regions have been collected.
Generalizing to dissimilar regions is significantly harder. Fig. 6 shows examples of generalization between LV (a major metropolitan area) and SLAC (a suburban academic campus). Not surprisingly, models trained on each of these two regions performs worse when applied to the other. For example, the LV model does not generate realistic parked vehicles in the parking lots from SLAC and the
Table 4: Generalization across regions
Model	\(MMD^2\) Positions (↓)	Traj On Drivable (↑)
LV Scenes	SEA Scenes	SF Scenes	SLAC Scenes	LV Scenes	SEA Scenes	SF Scenes	SLAC Scenes
LV	0.077 (0.004)	0.139 (0.013)	0.126 (0.013)	0.204 (0.028)	0.984 (0.001)	0.959 (0.004)	0.961 (0.005)	0.917 (0.009)
SEA	0.108 (0.007)	0.071 (0.004)	0.080 (0.005)	0.141 (0.007)	0.965 (0.004)	0.972 (0.002)	0.976 (0.004)	0.928 (0.010)
SF	0.110 (0.006)	0.102 (0.006)	0.049 (0.002)	0.141 (0.008)	0.978 (0.006)	0.978 (0.005)	0.991 (0.001)	0.965 (0.005)
SLAC	0.209 (0.025)	0.225 (0.020)	0.193 (0.019)	0.053 (0.002)	0.893 (0.031)	0.936 (0.017)	0.950 (0.011)	0.969 (0.001)
Full	0.083 (0.006)	0.084 (0.013)	0.055 (0.008)	0.060 (0.009)	0.983 (0.003)	0.971 (0.006)	0.989 (0.003)	0.963 (0.003)
SLAC model generates vehicles facing against the lane direction at locations in LV. Fortunately, the model trained on the full dataset is able to produce high quality scenarios in both regions.
A number of prior works have used heuristics or fixed grammars to generate driving scenes [5, 7, 16, 17]. Such approaches are limited in their ability to generate complex and diverse driving scenes. SceneGen [32] and TrafficGen [6] use an autoregressive approach, adding agents one at a time. TrafficGen adds a second stage where a motion forecasting model [33] is used to generate trajectories for the boxes. SimNet [2] uses a conditional generative adversarial network [15] to generate a BEV occupancy image. Heuristic post-processing is applied to identify connected components and fit bounding boxes to them. A second agent-centric model is then used to generate trajectories for these agents over time. In contrast, our method directly produces bounding boxes and trajectories simultaneously for all agents in an end-to-end differentiable architecture. Our approach also does not constrain vehicles to be in lane, as in TrafficGen and several heuristic approaches.
A number of works generate future trajectories for already-existing agents in simulation [14, 31, 37, 39, 40]. Some focus on specifically producing adversarial future trajectories given agents' initial states [4, 10, 35]. These approaches are complementary to our work, and can be used to extend trajectories beyond what is output by Scenario Diffusion.
In this paper, we demonstrate a novel technique for using diffusion to learn to generate scenarios of dynamic agents moving through an environment for the purposes of testing an autonomous vehicle's ability to navigate through that environment and plan according to those agents. We have shown that our technique leads to models that not only appropriately capture the desired distributions of scenarios and agent trajectories, but allow scenario generation to be controlled to target specific types of scenarios. The fact that diffusion allows declarative information such as tokens to be combined with models learned from data creates the potential for new models that combine other forms of symbolic knowledge such as traffic rules, physical constraints, and common sense reasoning.
Limitations: More work is required to show broader generalization. The training data assumes a specific model of perception in the form of bounding boxes and trajectory models. Additionally, we restricted the agent models to on-road vehicle models. While we do not anticipate significant challenges in applying this model to other forms of agents such as pedestrians, we have not incorporated those agents in the research described here. In simulation this approach may need to be extended to iteratively generate agents over a larger region as the AV navigates through the environment.
Broader Impact: This paper focuses on developing models to improve self-driving car technologies. There are many positive and negative aspects to the development of self-driving cars that depend as much on the system-wide design and regulatory aspects as these aspects depend on the technical capabilities. However, the focus on simulation in this paper should partially reduce the risks of deploying self-driving cars by providing more effective and systematic coverage of testing scenarios.
Acknowledgements: We would like to thank Gary Linscott, Jake Ware, and Yan Chang for helpful feedback on the paper; Allan Zelener and Chris Song for discussions on object detection; Peter Schleede for discussions on diffusion. We also thank the NeurIPS anonymous reviewers, area chair, and program chairs.
[38] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting. ICCV, 2021. URL https://arxiv.org/abs/2103.14023.
[39] Qichao Zhang, Yinfeng Gao, Yikang Zhang, Youtian Guo, Dawei Ding, Yunpeng Wang, Peng Sun, and Dongbin Zhao. TrajGen: Generating Realistic and Diverse Trajectories With Reactive and Feasible Agent Behaviors for Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems, 23(12): 24474-24487, 2022. URL https://arxiv.org/abs/2203.16792.
[40] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided Conditional Diffusion for Controllable Traffic Simulation. ICRA, 2023. URL http://arxiv.org/abs/2210.17366.
Figure 7: Two examples from Argoverse 2. Images A and B are concatenated in the channel dimension to obtain the 6 channel map image  $m$ . Image C shows the current location of entity bounding boxes colored by object type with a line to indicate box orientation. Images D-G show the time delta renderings  $\Delta_{-2}, \Delta_{-1}, \Delta_1, \Delta_2$ , respectively. C-G are concatenated in the channel dimension to obtain the 15 channel entity topdown image  $x$ . Image H shows the same example using the plotting style used to show generated scenarios.
A.1 Argoverse 2
The Argoverse 2 motion forecasting dataset [36] contains 250,000 driving scenarios, each 11 seconds long. These scenarios cover 6 geographical regions and represent 763 total hours of driving.
To generate inputs for Scenario Diffusion we take all 4s windows (for -2 seconds to +2 seconds) and render an input example for each. To do this rendering we use the provided visualization library to plot a BEV image of the map and add an additional image showing the lane center lines colored by the lane direction. For the entity layers we use the provided visualization library to plot vehicles and buses as rectangles of two different shades of blue. To disambiguate the orientation of the boxes we also render a line from the center of each box to show the heading.
As the argoverse dataset does not include information on agent dimensions, we use constants for each object type (i.e. vehicle and bus) as in the argoverse visualization library. These constants are also used to define the ground truth boxes used to train the autoencoder.
To represent agents' trajectories in the topdown image, we render images for each time delta along the trajectory. For  $t \in \{-2, -1, 0, 1, 2\}$ , let  $p_t = \left(p_t^{(x)}, p_t^{(y)}\right)$  be the agent's position and  $m_t$  a mask for whether the agent was observed at the given timestep. We compute four position deltas and corresponding masks:
$$
\Delta_ {- 2} = p _ {- 1} - p _ {- 2} \quad \alpha_ {- 2} = m _ {- 1} \wedge m _ {- 2}
$$
$$
\Delta_ {- 1} = p _ {0} - p _ {- 1} \quad \alpha_ {- 1} = m _ {0} \wedge m _ {- 1} \tag {5}
$$
$$
\Delta_ {1} = p _ {1} - p _ {0} \quad \alpha_ {1} = m _ {1} \wedge m _ {0}
$$
$$
\Delta_ {2} = p _ {2} - p _ {1} \quad \alpha_ {2} = m _ {2} \wedge m _ {1}
$$
Figure 8: Example topdown inputs for the internal dataset. The left image is the first three channels of the map image  $m$ . The middle image is the first three channels of the topdown entity image  $x$  which fill each bounding box with ones, the cosine of the heading, and the sine of the heading, respectively. The right image shows the corresponding figure using the ground truth boxes and trajectories from the example.
For each  $i \in \{-2, -1, 1, 2\}$  we add channels where each agent's bounding box from  $t = 0$  is filled with  $\alpha_i$ ,  $\operatorname{clip} \left( \Delta_i^{(x)} / 30 + 0.5, 0, 1 \right)$ , and  $\operatorname{clip} \left( \Delta_i^{(y)} / 30 + 0.5, 0, 1 \right)$ .
Scenes are centered on the autonomous vehicle, which is treated as just another vehicle, and cover an area of  $100\mathrm{m}\times 100\mathrm{m}$ . Examples of the topdown inputs  $m$  and  $x$  for this dataset can be seen in Fig. 7.
A.2 Internal Dataset
The internal dataset contains 6 million real world driving scenarios from Las Vegas (LV), Seattle (SEA), San Francisco (SF), and the campus of the Stanford Linear Accelerator Center (SLAC). Note that Las Vegas, Seattle, and San Francisco represent dense urban settings, while SLAC is a suburban academic campus. Training and validation data are not separated by location, so a given validation example may have training examples from the same location at other times. Scenarios are sampled every 0.5 seconds from real-world driving logs.
Map information comes from a pre-defined high definition map and vehicle tracks are detected by an on-vehicle perception system. We filter these tracks to only include bounding boxes that overlap with drivable areas in the map (e.g. roads, public parking lots, driveways).
Scenes are centered on the autonomous vehicle, which is treated as just another vehicle during training, and cover an area of  $100\mathrm{m}\times 100\mathrm{m}$ . Examples of scenes from this dataset can be seen in figure Fig. 8.
Figure 9: Model architecture for the scenario autoencoder. The architecture used by [24] is modified to include an additional conditional image input (the map image  $m$ ) to the decoder  $\mathcal{D}$ .
Figure 10: Model architecture for the scenario diffusion model. The denoising Unet  $\mathcal{M}$  combines the noisy latent sample  $z$ , the encoded map information  $\mathcal{E}_m(m)$ , and the tokens. During training  $z$  is generated by running the frozen encoder to obtain  $\hat{z} = \mathcal{E}(x)$  and adding noise (see Fig. 2b).
B Scenario Autoencoder Details
B.1 Architecture Details
The architecture for the scenario autoencoder is based on [24] and is shown in Fig. 9. We modify the architecture to add the conditional image  $m$  as an input to the decoder  $\mathcal{D}(z, m)$ . The map image is encoded using the same architecture as  $\mathcal{E}$  (but different weights) to obtain a latent embedding with the same pixel dimension. We also truncate the upsampling stage of the decoder to output box proposals at a 64 pixel resolution. The same architecture is used for both the Argoverse and internal datasets.
The 256 pixel and 128 pixel blocks use 64 channels, and the 64 pixel and 32 pixel blocks use 128 channels. We use one Resnet block per resolution.
B.2 Reconstruction Loss
To compute the reconstruction loss we first match a predicted box to each ground truth box and then apply loss terms between matched pairs. The losses and matching use the same cost functions with different weights.
The classification loss is a binary cross-entropy loss between the predicted box's logit and the probability of the ground truth box (defined to be 1). The coordinate loss applies an L1 loss on the 6 parameters that define the box shape (the cosine and sine of the box heading and the log of the distance to the front, left, back, and right of the box). The vertex loss applies an L2 loss on the positions of the four corners of the box. The trajectory position loss applies an L2 loss on the positions of the trajectory. The trajectory heading loss applies a cosine loss on the headings of the trajectory.
To match a predicted box to a given ground truth box we select the predicted box that minimizes a weighted combination of the classification, coordinate, and vertex losses. The classification term is up-weighted by a factor of 4 over the other two terms.
We apply the five loss functions on the matched pairs. For this loss the classification term is up-weighted by a factor of 20 over the other four terms.
To balance between different motion profiles we categorize each ground truth trajectory as stationary, straight, or turning. To better balance between these three categories we weight the trajectory losses for each category by 0.1, 0.3, and 4, respectively.
We also use a binary cross-entropy loss on the unmatched predicted boxes to have probability 0. As there are many more such negative predictions, we down-weight this loss term. Negative boxes within 3 meters of a ground truth box are down-weighted by a factor of 0.2, and negative boxes more than 3 meters from a ground truth box are down-weighted by a factor of 0.002.
The scenario autoencoder is trained using the Adam optimizer [19], initial learning rate of 1e-4, and weight decay of 1e-4. The learning rate is decreased by a factor of 10 when the validation loss plateaus. Gradients are clipped to have norm less than or equal to 1. The KL regularization loss  $\mathcal{L}_{\mathrm{KL}}$  is weighted by 0.1 compared to the reconstruction loss described above. We use 2 RTX A6000 GPUs with a batch size of 16 per GPU.
C Scenario Diffusion Details
The architecture for the scenario diffusion model is also based on [24] and shown in Fig. 10. An encoder  $\mathcal{E}_m$  with the same architecture as  $\mathcal{E}$  with different weights process the conditional map image to the same latent dimension. The noisy latent embedding  $z$  and the encoded map information  $\mathcal{E}_m(m)$  are concatenated to form the input to the denoising Unet  $\mathcal{M}$ . In the lower resolution blocks of  $\mathcal{M}$ , the cross attention layers attend to the conditional tokens. The same architecture is used for both the Argoverse and internal datasets.