SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The 32 pixel, 16 pixel, and 8 pixel blocks of  $\mathcal{M}$  use 64, 128, and 256 channels, respectively. Each resolution block features a single Resnet block. Cross attention is performed with 8 attention heads. The embedded map information  $\mathcal{E}_m(m)$  uses 64 channels.
We use the EDM training algorithm [18]. We set  $\sigma_{\mathrm{min}} = 0.02$ ,  $\sigma_{\mathrm{max}} = 20$ ,  $\sigma_{\mathrm{data}} = 0.5$ ,  $\rho = 7$ ,  $P_{\mu} = -0.5$ , and  $P_{\sigma} = 1$ . See Table 1 of [18] for more information (note that  $P_{\mu}$  and  $P_{\sigma}$  are referred to as  $P_{\mathrm{mean}}$  and  $P_{\mathrm{std}}$  in that table).
The diffusion model is trained with an AdamW optimizer [21], an initial learning rate of 3e-4, and weight decay of 1e-5. The learning rate is decreased by a factor of 10 when the validation loss plateaus. We use 4 RTX A6000 GPUs with a batch size of 64 per GPU.
C.3 Token Details
In this work we consider the following possible agent token features:
- Current x position relative to the scene center
- Current y position relative to the scene center
- Current heading
- Box length
- Box width
- Current speed
Table 5: Agent token features used for figures.
Figure	Current Pose	Extents	Current Speed	Final Speed	Heading to Final Position
Fig. 1	✓	✓	✓	✓
Fig. 3	✓	✓	✓
Fig. 4	✓	✓	✓	✓
Fig. 11	✓	✓	✓	✓
Fig. 12	✓	✓	✓	✓
Fig. 13	✓	✓	✓	✓
Fig. 14	✓	✓	✓	✓
Fig. 15	✓	✓	✓	✓
- Final speed (at +2s)
- Relative heading from the  $t = 0$  position to the  $t = 2$  position
A given model will use a subset of these features for its agent tokens. For brevity we use "current pose" to refer to the trio of current x position, current y position, and current heading and "extents" to refer to the pair of box length and box width.
Each agent token feature is processed by a per-feature embedding function; the embeddings of all features are concatenated to get the final token feature vector.
The x and y positions are each embedded using a one-hot vector for 64 bins over the extent of the scene (100m). The current heading is encoded with two channels containing the cosine and sine of the heading. The length and width are encoded with one channel containing the  $\log 1p$  of the value. Current and final speed are encoded using a one-hot vector for 10 bins over the interval [0, 20] in m/s. Speeds above  $20\mathrm{m / s}$  are clipped to  $20\mathrm{m / s}$ . The relative heading is encoded using a one-hot vector for 10 bins over the interval  $[-0.7, 0.7]$ .
Tab. 5 lists the agent token features used for the various figures shown in this paper.
D.1 Autoencoder Ablation Experiments
One of the key hyperparameters for Scenario Diffusion is the latent embedding dimension, the number of channels in  $z$ . Tab. 6 shows the results of ablation experiments training diffusion models with different autoencoders using different latent dimensions. We find the latent embedding dimension used for experiments in the main paper (4) performs slightly better on MMD metrics and achieves trajectory metrics slightly closer to those of ground truth logs. However, the difference is not statistically significant given the standard deviation across diffusion models trained with different random initial seeds.
D.2 Diffusion Ablation Experiments
We begin by measuring the effect of changing the probability threshold used during inference. To do so, we repeat the procedure described in Sec. 4.3.1 of sampling scenarios from the Argoverse validation dataset and computing agent tokens from the ground truth agents in the scene. Using  $p_{\mathrm{mask}} = 0$ , we expect the generated agents to match the ground truth agents. We perform the same matching algorithm described in Sec. 4.3.1. Treating the matching as a binary outcome, we
Table 6: Ablation experiments on the autoencoder latent embedding dimension.
Latent Embedding Dimension	\( MMD^2 \) Positions (↓)	\( MMD^2 \) Headings (↓)	\( MMD^2 \) Velocity (↓)	Traj On Drivable	Lane Heading Difference
2	0.096 (0.011)	0.111 (0.009)	0.058 (0.007)	0.886 (0.019)	0.283 (0.032)
4	0.093 (0.006)	0.108 (0.007)	0.055 (0.003)	0.895 (0.008)	0.271 (0.017)
8	0.099 (0.009)	0.116 (0.010)	0.059 (0.005)	0.892 (0.010)	0.282 (0.018)
16	0.098 (0.007)	0.117 (0.009)	0.058 (0.004)	0.884 (0.016)	0.313 (0.035)
Ground Truth Log	0.900 (0.000)	0.203 (0.000)
Table 7: Ablation experiments measuring the effect of changing the box probability threshold.
Probability Threshold	Precision	Recall	F1
0.05	0.695 (0.086)	0.981 (0.005)	0.811 (0.067)
0.10	0.886 (0.013)	0.981 (0.005)	0.931 (0.009)
0.15	0.906 (0.012)	0.980 (0.005)	0.942 (0.009)
0.20	0.917 (0.011)	0.980 (0.005)	0.948 (0.008)
0.25	0.925 (0.011)	0.980 (0.005)	0.952 (0.008)
0.30	0.932 (0.011)	0.979 (0.005)	0.955 (0.008)
0.35	0.937 (0.011)	0.979 (0.006)	0.958 (0.008)
0.40	0.942 (0.011)	0.979 (0.006)	0.960 (0.008)
0.45	0.947 (0.011)	0.978 (0.006)	0.962 (0.008)
0.50	0.951 (0.010)	0.978 (0.006)	0.964 (0.008)
0.55	0.955 (0.010)	0.977 (0.006)	0.966 (0.008)
0.60	0.959 (0.010)	0.976 (0.006)	0.968 (0.008)
0.65	0.963 (0.010)	0.975 (0.006)	0.969 (0.008)
0.70	0.967 (0.010)	0.974 (0.006)	0.970 (0.008)
0.75	0.971 (0.010)	0.972 (0.006)	0.971 (0.007)
0.80	0.974 (0.010)	0.968 (0.006)	0.971 (0.007)
0.85	0.977 (0.010)	0.963 (0.006)	0.970 (0.007)
0.90	0.979 (0.010)	0.954 (0.006)	0.966 (0.007)
0.95	0.981 (0.009)	0.940 (0.006)	0.960 (0.007)
Table 8: Ablation experiments on the diffusion architecture. The values used in the main experiments are bolded.
Ablation Variable	Ablation Value	\( MMD^2 \) Positions (↓)	\( MMD^2 \) Headings (↓)	Num Agents EMD (↓)	Traj Off Drivable (↓)
Number of Resnet Blocks	1	0.076 (0.009)	0.091 (0.007)	5.0 (2.1)	0.025 (0.003)
2	0.075 (0.006)	0.089 (0.006)	5.0 (1.2)	0.022 (0.002)
3	0.074 (0.004)	0.090 (0.005)	4.1 (0.9)	0.022 (0.003)
\( \mathcal{E}_m \) Output Dimension	8	0.072 (0.005)	0.086 (0.004)	4.6 (1.2)	0.024 (0.002)
16	0.071 (0.006)	0.085 (0.006)	4.3 (1.4)	0.023 (0.003)
32	0.076 (0.009)	0.090 (0.008)	5.1 (1.9)	0.023 (0.003)
64	0.073 (0.003)	0.089 (0.004)	4.1 (0.8)	0.026 (0.002)
128	0.077 (0.007)	0.093 (0.006)	5.3 (1.6)	0.024 (0.004)
Cross Attention Levels	32px, 16px, 8px	0.078 (0.007)	0.097 (0.005)	5.3 (1.8)	0.025 (0.005)
16px, 8px	0.076 (0.006)	0.090 (0.005)	5.1 (1.4)	0.023 (0.002)
8 px	0.078 (0.007)	0.093 (0.007)	5.5 (1.5)	0.025 (0.004)
can define each matched pair of a ground truth and generated agent to be a "true positive" (TP), ground truth agents that are not matched to generated agents as "false negatives" (FN), and generated agents that are not matched to ground truth agents as "false positives" (FP). From these numbers, we can compute precision  $= \frac{\mathrm{TP}}{\left( {\mathrm{{TP}} + \mathrm{{FP}}}\right) }$  ,recall  $= \frac{\mathrm{{TP}}}{\left( {\mathrm{{TP}} + \mathrm{{FN}}}\right) }$  ,and  $\mathrm{F}1 = 2 \cdot$  precision recall/ (precision + recall). Note that this definition of "recall" is the "Agent token match rate" and "false positives" are the "additional generated agents" from Tab. 2. Precision makes sense to measure in this context because we fix  ${p}_{\text{mask }} = 0$  ,so the number of additional generated agents should be close to 0 .
These results are shown in Tab. 7, reporting the mean and standard deviation across multiple models trained with different random initial seeds. We observe that precision increases and recall decreases as the probability threshold increases. The mean F1 score is maximized with a probability threshold of  $75\%$  or  $80\%$ . From this, we select  $80\%$  as the threshold to use for most experiments. However, we note that the differences in F1 score are relatively small for probability thresholds between  $40\%$  and  $95\%$ .
Tab. 8 shows the results of ablation experiments on several key hyperparameters of the diffusion model architecture using the internal dataset. The values used in the experiments for the main paper are bolded. In general, we find that increasing the model size does not significantly improve the available metrics.
D.3 Cross Region Generalization
Tab. 9 shows additional metrics from the evaluation described in Sec. 4.4. These metrics follow the same pattern as Tab. 4. This table introduces a new metric: the earth movers' distance (EMD) on the number of generated agents. This metric measures how much the probability mass needs to move to
Table 9: Generalization across regions
Model	MMD2Heading (↓)	EMD for Number of Agents (↓)
LV Scenes	SEA Scenes	SF Scenes	SLAC Scenes	LV Scenes	SEA Scenes	SF Scenes	SLAC Scenes
LV	0.091 (0.003)	0.180 (0.008)	0.186 (0.012)	0.265 (0.028)	4.2 (1.2)	6.5 (1.4)	13.3 (1.5)	12.3 (1.5)
SEA	0.177 (0.011)	0.075 (0.002)	0.137 (0.007)	0.222 (0.010)	8.3 (2.1)	1.9 (0.6)	8.8 (1.4)	9.3 (0.9)
SF	0.160 (0.007)	0.144 (0.004)	0.048 (0.001)	0.197 (0.005)	9.2 (1.8)	2.6 (0.9)	3.1 (0.8)	9.1 (0.9)
SLAC	0.375 (0.035)	0.360 (0.029)	0.341 (0.023)	0.073 (0.003)	18.0 (0.8)	11.3 (0.5)	18.0 (0.6)	2.8 (0.7)
Full	0.097 (0.009)	0.103 (0.023)	0.060 (0.013)	0.083 (0.012)	5.2 (1.8)	1.6 (1.3)	4.4 (1.9)	3.1 (1.7)
Figure 11: Generated scenarios for the internal dataset. All scenarios are conditioned on two agent tokens; the bottom row of scenarios also uses a third agent token. Agents described by the tokens are circled, with the agent circled in orange representing the AV and other agents circled in red. The model generates a variety of trajectories for the agents described by the tokens. This demonstrates how agent tokens can be used to generate driving scenarios to test a specific category of interactions (e.g. vehicles pulling in front of the AV) while providing diversity within that category.
transform from one probability distribution to another. In this context, the distributions in question are 1D histograms of the number of agents per scene, measured over the ground truth validation scenes and generated scenes using the same map locations.
D.4 Qualitative Results
Fig. 11 shows scenarios generated using different numbers of agent tokens and different values for the global scene token. Fig. 12 shows adversarial scenarios in a junction created using two agent tokens. These examples illustrate how token controllability allows users to create specific types of scenarios, while letting the model figure out most of the details.
Fig. 13 and Fig. 14 show two more examples of modifying the final speed of agent tokens while keeping all other input features the same. This can be performed with any number of agent tokens.
Fig. 15 takes a ground truth scene from the Argoverse validation dataset, computes agent tokens from two vehicles of interest, and generates novel scenes using these tokens. This process allows users to sample modified versions of a given log to see how the placement of other agents impacts the AV behavior.
Figure 12: Generated scenarios for the internal dataset showing an agent nudging into the AV's lane while in a junction. All scenarios are conditioned on the same two agent tokens; the corresponding agents are circled, with the agent representing the AV in orange and the other agent in red. In these examples the global scene token is set to  $50\%$ . By testing many such scenarios we can find the conditions that result in undesirable driving behavior.
Final Speed:  $1\mathrm{m / s}$
Final Speed:  $7\mathrm{m / s}$
Figure 13: Generated Argoverse scenes conditioned on one agent token, changing only the final speed token feature. The behavior described by the agent token impacts nearby generated agents.
Final Speed:  $1\mathrm{m / s}$
Figure 14: Generated Argoverse scenes conditioned on two agent tokens, changing only their final speed. Both agent tokens are set to use the same final speed. The behavior described by the agent tokens impact nearby generated agents.
E TrafficGen
Several modifications were made to TrafficGen [6] accommodate the Argoverse 2 dataset [36]. Argoverse 2 does not contain traffic light information, so such features are not used in the map context. Argoverse 2 also does not include the length and width of agent boxes. In the provided plotting code fixed constants are used for different agent types. We include the "car" and "bus" agent types for training TrafficGen, and update the size prediction head to perform 2-way classification instead of outputting a Gaussian mixture model. To match Scenario Diffusion, we set the model to only generate vehicles in the  $100\mathrm{m}$  region around the AV.
Fig. 16 shows scenes generated by our implementation of TrafficGen.
Figure 15: Generated Argoverse scenes using two key agents from a real world log. The top left scene is the original log; the other nine scenes are generated using the same map location and two agent tokens corresponding to the two agents circled in orange. The model is able to produce diverse driving scenes incorporating the two key agents.
Figure 16: Generated Argoverse scenes from our implementation of TrafficGen. The blue lines show the velocity extended 1 second into the future.