SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Time Series NLL: Given the time series nature of simulation data, two choices emerge for how to treat samples over multiple timesteps for a given object for a given run segment: to treat them as time-independent or time-dependent samples. In the latter case, users would be expected to not only reconstruct the general behaviors present in the logged data in one rollout, but also recreate those behaviors over the exact same time intervals. To allow more flexibility in agent behavior, we use the former formulation when computing NLLs, defining each component metric  $m$  as an average (in log-space) over the time-axis, masked by validity  $v_{t}$ :  $m = \exp \left(-\frac{1}{\sum_{t}\mathbb{1}\{v_{t}\}}\sum_{t}\mathbb{1}\{v_{t}\} NLL_{t}\right)$ .
Figure 3: Visualizations of simulation results on two separate WOMD scenes (top, bottom). Results for various baseline methods are shown on WOMD's validation split, in 2d. 'Simulation input' represents the context history  $o_{-H - 1:0}$ , whereas all other columns visualize both  $(o_{-H - 1:0}, o_{\geq 1})$ . Two scenes are represented: one where the AV completes the execution of a left turn (top row) and another where the AV remains stopped at a red traffic signal (bottom row). Each rendering in the second and third columns depicts the entire duration of the scene. Trajectories of environment sim agents are drawn in a green-blue gradient, and trajectories of the AV agent are drawn in a red-yellow gradient (each as a sequence of circles in a temporal color gradient).
However, we note that as a result, a logged oracle will not achieve likelihoods of 1.0, whereas in the latter formulation a logged oracle would.
**Definitions** We compute NLLs over 9 measurements: kinematic metrics (linear speed, linear acceleration, angular speed, angular acceleration magnitude), object interaction metrics (distance to nearest object, collisions, time-to-collision), and map-based metrics (distance to road edge, and road departures). Please refer to Section A.6 of the Appendix for a complete description and additional implementation details.
4.2.2 Composite Metric
After obtaining component metrics for each measurement, we aggregate them into a single composite metric  $\mathcal{M}^K$  for evaluating submissions:
$$
\mathcal {M} ^ {K} = \frac {1}{N M} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} w _ {j} m _ {i, j} ^ {K}, \quad \sum_ {j = 1} ^ {M} w _ {j} = 1 \tag {3}
$$
where  $N$  is the number of scenarios and  $M = 9$  is the number of component metrics. The component metrics  $m$  and composite metric  $\mathcal{M}$  are also parameterized by a number of samples  $K = 32$ . The value  $m_{i,j}$  represents the likelihood for the  $j^{th}$  metric on the  $i^{th}$  example. The metric  $\mathcal{M}$  is simply a convex combination (i.e. weighted average) over the component metrics, where the weight  $w_{j}$  for the  $j^{th}$  metric is set manually. In the interest of promoting safety, the weighting for collision and road departure NLLs are set to be  $2\times$  larger than the weight for the other component metrics.
In Figure 4 and Table 3, we present quantitative results for a handful of methods. We describe each method in more detail in the sections below.
Figure 4: Challenge composite metric results of various baselines on the test split of WOMD.
Random Agent: An agent that produces random trajectories  $\{(x_{t},y_{t},\theta_{t})\}_{t = 1}^{T}$ , for  $T = 80$ , with  $x,y,\theta \sim \mathcal{N}(\mu ,\sigma^2)$ , with  $\mu = 1.0$  and  $\sigma = 0.1$ , in the AV's coordinate frame.
Constant Velocity Agent: An agent that extrapolates the trajectory using the last heading and speed recorded in the provided context/history. If no two-step difference can be computed based on the valid measurements (e.g. the object appeared only at the final step of context), we set a zero speed for such agents.
Wayformer (Identical Samples) Agent: An agent that produces a hybrid of open-loop and closed-loop data using a Wayformer [40] motion prediction model, by executing model inference autoregressively at  $2\mathrm{Hz}$ . The agents execute the policy forward for 5 simulation steps, and then replan. Results with a  $10\mathrm{Hz}$  replan rate instead are also shown in Table 3, and an ablation on the replan rate is provided in the Appendix. The maximum-likelihood trajectory for each agent is identically repeated 32 times to produce 32 samples. Each agent is executed by the same policy in an agent-centric frame, batched together for inference, thus complying with the required factorization.
Wayformer (Diverse Samples) Agent: An agent that also utilizes Wayformer [40]-generated trajectories, but samples diverse agent plans, from  $K$  possible trajectories according to their likelihood, instead of selecting the maximum-likelihood choice.
Logged Oracle: Agent that directly copies trajectories from the WOMD test split, with 32 repetitions.
5.2 External Submissions
MultiVerse Transformer for Agent simulation (MVTA) [71]: A method inspired by MTR [55, 56] that is trained and executed in closed-loop. MVTA uses a 'receding horizon' policy with a GMM head, and consumes vector inputs.
MVTE: An enhanced version of MVTA [71] that samples a MVTA model from a pool of model variants to increase simulation diversity across rollouts.
MTR+++ [44]: A hybrid open-loop/closed-loop method with a 0.5Hz replanning rate that is inspired by MTR [55, 56] and searches for the densest subgraph in a graph of non-colliding future trajectories.
For a description of other evaluated external submissions, please refer to Section A.4 of the Appendix.
5.3 Learnings from the 2023 Challenge
During the course of our 2023 WOSAC Challenge (March 16, 2023 to May 23, 2023), associated with the CVPR 2023 Workshop on Autonomous Driving, we received 24 test set submissions, and 16 validation set submissions, from 10 teams. We continue to receive submission queries to our evaluation server for our standing leaderboard as new teams submit new methods.
Trends We observed several trends among submissions. First, the challenge champion, MVTA/MVTE [71], was the only method to utilize and benefit from closed-loop training. Other methods that were trained in open-loop, such as MTR+++ [44] or our Wayformer-derived [40] baseline, found operating
Table 3: Per-component metric results on the test split of WOMD, representing likelihoods. Methods are ranked by composite metric on the V1 Leaderboard, rather than the previous V0 Leaderboard; Numbers within  $1\%$  of the best are in bold (excluding 'logged oracle'). * indicates a method that was received after May 23, 2023, which marked the close of the CVPR 2023 competition.
AGENT POLICY	LINEAR SPEED (↑)	LINEAR ACCEL. (↑)	ANG. SPEED (↑)	ANG. ACCEL. (↑)	DIST. TO OBJ. (↑)	COLLISION (↑)	TTC (↑)	DIST. TO ROAD EDGE (↑)	OFFROAD (↑)	COMPOSITE METRIC (↑)	ADE (↓)	MINADE (↓)
RANDOM AGENT	0.002	0.044	0.074	0.120	0.000	0.000	0.734	0.178	0.287	0.155	50.739	50.706
CONSTANT VELOCITY	0.074	0.058	0.019	0.035	0.208	0.345	0.737	0.454	0.455	0.287	7.923	7.923
CONSTANT VELOCITY (+ GAUSSIAN NOISE)	0.157	0.119	0.019	0.035	0.247	0.411	0.775	0.502	0.463	0.324	7.594	7.237
WAYFORER (IDENTICAL SAMPLES, 10 Hz REPLAN) [40]	0.202	0.144	0.248	0.312	0.192	0.449	0.766	0.379	0.305	0.338	6.823	6.823
SBTA-ADIA [38]	0.317	0.174	0.478	0.463	0.265	0.337	0.770	0.557	0.483	0.420	4.777	3.611
WAYFORER (DIVERSE SAMPLES, 10 Hz REPLAN) [40]	0.233	0.212	0.345	0.330	0.241	0.635	0.797	0.424	0.413	0.421	6.866	5.761
CAD [14]	0.349	0.253	0.432	0.310	0.332	0.568	0.789	0.637	0.834	0.531	3.334	2.308
JOINT-MULTIPATH++*[70]	0.434	0.230	0.515	0.452	0.345	0.567	0.812	0.639	0.682	0.533	5.293	2.049
WAYFORER (IDENTICAL SAMPLES, 2 Hz REPLAN) [40]	0.331	0.098	0.413	0.406	0.297	0.870	0.782	0.592	0.866	0.575	2.498	2.498
MTR+++[44]	0.414	0.107	0.484	0.436	0.347	0.861	0.797	0.654	0.895	0.608	2.125	1.679
MVTA [71]	0.439	0.220	0.533	0.480	0.374	0.875	0.829	0.654	0.893	0.636	3.925	1.866
MVTE [71]	0.445	0.222	0.535	0.481	0.383	0.893	0.832	0.664	0.908	0.645	3.859	1.674
LOGGED ORACLE	0.561	0.330	0.563	0.489	0.485	1.000	0.881	0.713	1.000	0.722	0.000	0.000
at slower replan rates necessary to obtain high composite metric results (See Table 3). Second, almost all submissions used Transformer-based methods [67], except for JointMultiPath++, which used LSTM and MCG blocks [66]. Third, all methods built primarily on top of existing motion prediction works, rather than upon existing motion planning works or sim agent methods from the literature. Only one method, MVTA/MVTE [71], incorporated aspects of an existing sim agent work, TrafficSim [60], as well as motion planning techniques, implementing a receding horizon planning policy. Thus, fourth, we observed the benefit of incorporating planning-based methods into a motion prediction framework. Fifth, most methods (excluding JointMultipath++ [70]) built upon the 2022 CVPR Waymo Open Motion Prediction challenge champion, MTR [55, 56], likely due to the open-source availability of its codebase and SOTA performance. Finally, all submissions operated in an agent-centric coordinate frame, rather than jointly sampling from a scene representation simultaneously.
Likelihood Metrics Reward Diversity We found that our likelihood-based metrics reward models that produce diverse futures. For example, generating 32 diverse rollouts per scene with a Wayformer model performs  $11\%$  better on our evaluation metrics than a Wayformer model that produces 32 identical rollouts per scene (see Figure 4).
Collision Minimization as an Algorithmic Objective Several methods designed algorithmic components to determine futures with a minimal number of collisions, e.g., MTR+++ [44] which used clique-finding in an undirected graph of collision-free future trajectories, and CAD [14], which used rejection sampling on open-loop futures that created collisions. This objective aligns with human preference, but as close calls and collisions do occur in real driving data distributions, optimizing for this objective could be seen as trimming the tail of the distribution; distracted drivers generally do exist in everyday real world driving, and in certain scenarios, one would expect a low-quality planner to perform poorly and produce collisions with sim agents, and so such should be taken into consideration for generating realistic simulations. This suggests a limitation of the WOMD [22], which has few examples from the tail distribution of real driving, and efforts to upsample collision data could prove useful. In addition, open-loop methods such as CAD [14] that prune collisions after the fact could prune collisions caused by the AV rather than by the sim agents, yielding a misleadingly optimistic view of the AV's performance.
Composite Metric vs. (min-)ADE and ADE: We see that among submissions to the test set, rankings by ADE and minADE and ranking by our NLL composite metric disagree. However, methods with lower minADE do tend to achieve higher composite scores; ADE does not exhibit such a trend.
Composite Metric Results The ordinal ranking shown in Figure 4 and Table 3 indicates that learned, stochastic sim agents outperform not only heuristic baselines but also learned, deterministic sim agents. We consider a composite metric score of 0.722 as a practical upper bound on submissions, because it involves access to test data via an oracle.
Component Metric Results In Table 3, we provide a breakdown of the composite metric into component metric results. As expected, the 'logged oracle' baseline achieves the highest likelihood in each of the 9 component metrics. The top performing method, MVTE [71] scored highest on all but one component metric (linear acceleration likelihood), where CAD [14] outperformed MVTE by  $12\%$  (likelihood of 0.253 vs. 0.222). Surprisingly, MVTE [71] has angular acceleration likelihoods within a percentage point of the 'logged oracle' (0.481 vs. 0.489). The gap between the top performing learned method (MVTE) and 'logged oracle' in both collision likelihood (0.893 vs. 1.000) and
distance-to-nearest object likelihood (0.383 vs. 0.485) indicates significant room for improvement in future work on interactive metrics.
In Figure 3, we provide a qualitative comparison of various baselines on two WOMD scenarios. The results indicate that the complexity of behaviors within intersections far exceeds the capability of simple heuristics to predict. Collisions are evident from the constant velocity baselines in both examples. Additional qualitative examples from other sim agent methods are shown in Section A.5 of the Appendix.
Limitations. For our 2023 Challenge, we manually verified the validity of each submission according to factorization and closed-loop requirements discussed in each team's report, and we observed that the technical rules were subtle. Several of the submissions that used open-loop or hybrid open-loop/closed-loop methods may have limited applicability for some simulation applications.
Even if we had instituted a benchmark based on Docker-containerized software submissions instead of uploading output trajectory submissions, enforcing our requirements algorithmically and automatically would still be challenging. Although many properties of function calls to Dockerized software can be measured, e.g. latency, as long as any arbitrary state is maintained by the user, the system could not enforce all details of the closed-loop nature of the function call. As a result, user-submitted simulation agent software would have to adhere to strict stateless input and output data APIs. The ability to do so would assist in removing ambiguity regarding whether methods that prune collisions post-hoc qualify as closed-loop.
If a user provides containerized simulator submissions, one approach to encourage adherence to our requirements and to further incentivize closed-loop behavior would be to provide and interact with an AV policy that the user does not control. In our benchmark, the user was allowed to control the AV, albeit through an independent policy; the ability to evaluate simulator submissions on separate, held-out AV motion planning policies and on new scenarios would allow further valuable analysis.
Future Work Object insertion and deletion are important aspects of the simulation problem, yet we intentionally introduced an assumption of no object insertion or deletion in order to reduce the complexity of the first iteration of the WOSAC challenge for users. Motion planners trained or evaluated in a simulator must have the capability to exercise caution regarding areas of occlusion from which new objects may emerge at any timestep. In a future iteration of the challenge, we plan to introduce realism metrics that reward properly-modeled object insertion and deletion, e.g. distributional metrics on the number of vehicles appearing or disappearing at each frame, or the distance of simulated objects from the autonomous vehicle. The data distribution in the WOMD dataset already includes such object insertion and deletion.
Furthermore, we intentionally introduced an assumption of time-invariant object dimensions in our first iteration of WOSAC to simplify the modeling challenge for users. Time-variant object dimensions can be considered as a type of vehicle attribute, and object dimensions do actually change in the underlying data distribution provided in the WOMD dataset. We hope to include time-variant object dimension prediction as an aspect of the benchmark in future iterations.
As discussed in Section 5.3, given the prevalence of collision minimization algorithmic components among submissions, one may presume that collisions are not heavily represented in WOMD [22] data, or our metrics are limited in some way. Another approach would be to "fatten the tails" of the evaluation data distribution by generating synthetic, challenging initial conditions [3, 21, 23, 47, 61, 69], or mining more close calls and collisions from real driving data.
Conclusion. In this work, we have introduced a new challenge for evaluation of simulation agents, explaining the rationale for the different criteria we require. We invite the research community to continue to participate.
No third-party funding received in direct support of this work. We thank Ben Sapp for his helpful feedback in preparing the challenge. We would like thank Mustafa Mustafa, Kratarth Goel, Rami Al-Rfou for offering consultation, models and infrastructure that accelerated our work. We thank Alexander Gorban for his assistance in developing and maintaining the evaluation server. All the authors are employees of Waymo LLC.
[79] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. Trafficbots: Towards world models for autonomous driving simulation and motion prediction. In ICRA, 2023.
[80] Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guided traffic simulation via scene-level diffusion. In CoRL, 2023.
[81] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In ICRA, 2023.
[82] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, IMAN FADAKAR, Zheng Chen, Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Zhengbang Zhu, Yihan Ni, Nhat Nguyen, Mohamed Elsayed, Haitham Ammar, Alexander Cowen-Rivers, Sanjeevan Ahilan, Zheng Tian, Daniel Palenicek, Kasra Rezaee, Peyman Yadmellat, Kun Shao, dong chen, Baokuan Zhang, Hongbo Zhang, Jianye Hao, Wulong Liu, and Jun Wang. Smarts: An open-source scalable multi-agent rl training school for autonomous driving. In CoRL, 2020.
In this Appendix, we provide ablations investigating the impact of replan rate of Wayformer-derived [40] sim agent baselines on the test split, as well as additional learnings from the 2023 CVPR competition. We also include additional descriptions of methods from external challenge submissions, corresponding qualitative results for such methods, and implementation details of each of the 9 component metrics we use. Finally, we describe the exact details of the dataset splits we use and give leaderboard submission instructions.
Benchmark Versioning In December 2023, we improved the accuracy of the collision and offroad likelihood calculation, which improved most collision likelihood scores, offroad likelihood scores, and composite metric results. This paper describes the updated scores (the V1 version of the benchmark), rather than the previous V0 scores presented at the Workshop on Autonomous Driving at CVPR 2023. Both versions of the leaderboards are available online (V1 Leaderboard, V0 Leaderboard).
A.1 Replanning Rate Ablation Results
As shown in Table 3 of the main paper and as discussed in Section 5, we perform an ablation of the impact of replanning rate on composite metric performance for open-loop trained models on the WOMD test set. We show that a more frequent replan rate negatively impacts Wayformer-based [40] agent, regardless of whether multiple diverse rollouts per scene are sampled or if 32 identical rollouts per scene are produced. For the identical-sample producing agent, we see a relative performance drop of  $41.2\%$  (composite scores of 0.575 vs. 0.338) when transitioning from replanning at  $2\mathrm{Hz}$  to  $10\mathrm{Hz}$ .
In Figure 5, we visualize several additional data points from a replanning interval of  $100\mathrm{ms}$  to 1100 ms for the same identical-sample producing agent.
(a) WOSAC Composite Metric vs. replan rate.
(b) ADE vs. replan rate.
(c) Component likelihood metrics vs. replan rate.
Figure 5: Results at various replanning rates on the WOSAC test set for a Wayformer baseline (producing identical samples for the 32 rollouts). As the replanning rate increases from  $1\mathrm{Hz}$ , then to  $2\mathrm{Hz}$ , and then to  $10\mathrm{Hz}$ , we observe a smooth degradation in performance.
A.2 Additional Learnings from the 2023 CVPR Competition
Generative Modeling While many families of generative models exist, most challenge participants restricted their modeling to a narrow class of such models, namely GMMs. To our knowledge, for the 2023 CVPR challenge, no user submitted sim agent behavior generated by normalizing flow, GAN, or variational autoencoder (VAEs) models, or by denoising diffusion models, although such diffusion techniques have recently become more popular in the simulation agent literature [80, 81]. We expect this may change in the future as more entrants participate in the WOSAC challenge.
Further Quantitative Analysis We note that the top-four performing methods were executed fully in closed-loop (MVTE [71], MVTA) or in a hybrid fashion (MTR+++ [44], Wayformer [40]), outperforming the two open-loop submissions, JointMultiPath++ [70] and CAD [14], as shown in Table 4. This gap is especially clear between non-open loop methods and open-loop methods in the collision likelihood metric.
Among all baselines we evaluated, the constant velocity baseline is weakest when it comes to angular-based likelihoods. For example, it achieves close to zero likelihood on both angular speed (0.02) and angular acceleration (0.04), as opposed to the MVTE method, which achieves 0.54 and 0.38 likelihoods on the same two component metrics, respectively. This result is intuitive, as our constant velocity model does not account for any yaw rate.
A.3 Additional Comparisons with Other Benchmarks for Autonomous Driving Behavior
In Table 4, we compare our WOSAC benchmark with other benchmarks used for evaluation of behavior models for autonomous driving.
BENCHMARK NAME	TASK
Argoverse [10]	Trajectory Forecasting
INTERPRET (INTERACTION) [78]	Trajectory Forecasting
Argoverse2 [72]	Trajectory Forecasting
nuScenes [6]	Trajectory Forecasting
WOMD [22]	Trajectory Forecasting
CARLA [19]	Motion Planning
nuPlan [7]	Motion Planning
WOSAC (Ours)	Multi-Agent Simulation
Table 4: Existing benchmarks for evaluation of behavior models for autonomous driving.
A.4 Additional Information about Methods from External Challenge Submissions
MultiVerse Transformer for Agent simulation (MVTA) [71]: A method inspired by TrafficSim [60] that is trained and executed in closed-loop and adheres to WOSAC's factorization and autoregressive requirements. It uses a 'receding horizon' policy (i.e. predicting 1 sec. of future motion but using only the next  $100\mathrm{ms}$ ). Inspired by MTR [55, 56], MVTA places a Gaussian Mixture Model (GMM) head on top of a transformer-based encoder and decoder (employing the same encoder/decoder layers as implemented in MTR), consuming vector inputs. Rather than utilizing a fixed-length history as context, MVTA uses a variable-length history to potentially use all of the past data. The input agent encoding contains agent history motion state (i.e., position, object size, heading angle, and velocity) and a one-hot category mask of each agent. The prediction heads include a regression head that outputs 5 GMM parameters  $(\mu_x,\mu_y,\sigma_x,\sigma_y,\rho)$ , along with the velocity  $(v_{x},v_{y})$  and heading  $(\sin (\theta),\cos (\theta))$  predictions for a timestep, and a classification head that outputs probability  $p$ . Both heads take the query content features (num_query  $\times$  hidden feature dimension) as input.
MVTE: An enhanced version of MVTA [71] wherein 3 variants of MVTA are trained and randomly selected to generate each of the 32 simulations, increasing simulation diversity across rollouts.
$\mathbf{MTR} + + +$  [44]: A hybrid method with a  $0.5\mathrm{Hz}$  replanning rate and a 2 second prediction horizon. We note that  $\mathbf{MTR} + + +$  does not fully adhere to WOSAC's closed-loop requirement, as it does not replan at a  $10\mathrm{Hz}$  rate.  $\mathbf{MTR} + + +$  also does not adhere to the policy factorization requirements, as world vs. AV policies are not separated. Inspired by MTR [55, 56], the method addresses two key limitations of MTR: inaccurate heading predictions and excessive collisions incurred by marginal predictions alone. To overcome the first issue, the authors estimate headings from  $x / y$  trajectories. Second, in order to minimize collisions, the authors consider  $K = 6$  trajectories predicted per agent