The Waymo Open Sim Agents Challenge
Authors: Nico Montali
John Lambert
Paul Mougin
Alex Kuefler
Nicholas Rhinehart
Michelle Li
Cole Gulino
Tristan Emrich
Zoey Zeyu Yang
Shimon Whiteson
Brandyn White
Dragomir Anguelov
Venue: NeurIPS 2023 Datasets and Benchmarks Spotlight (2023)
Keywords: simulation
autonomous driving
TL;DR: We introduce the Waymo Open Sim Agents Challenge (WOSAC), the first public challenge to tackle this task and propose corresponding metrics.

Abstract: Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.

The Waymo Open Sim Agents Challenge
Nico Montali
John Lambert
Paul Mougin
Alex Kuefler
Nicholas Rhinehart
Michelle Li
Cole Gulino
Tristan Emrich
Zoey Yang
Shimon Whiteson
Brandyn White
Dragomir Anguelov
Waymo LLC
Abstract
Simulation with realistic, interactive agents represents key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for number different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.
Introduction
Simulation environments allow cheap and fast evaluation of autonomous driving behavior systems, while also reducing the need to deploy potentially risky software releases to physical systems. While early goal [19, 43] of simulation, use cases have evolved
generation of synthetic sensor data as perception systems have matured. Today, one the most promising use cases for simulation system safety validation via statistical model checking [1, 16] with Monte Carlo trials involving realistically modeled traffic participants, i.e., simulation agents.
Simulation agents are controlled objects that perform real- istic behaviors in virtual world. In this challenge, order to reduce the computational burden and complexity of sim- ulation, we focus on simulating agent behavior as captured by the outputs of perception system, e.g., mid-level object representations [2, 79] such as object trajectories, rather than simulating the underlying sensor data [13, 37, 62, 76] (see Figure 1).
Onboard
Offboard
Planner
Perception
Simulation
A requirement for modeling realistic behavior in simulation the ability for sim agents to respond to arbitrary behav- ior of the autonomous vehicle (AV). "Pose divergence" or "simulation drift" [3] is defined as the deviation between the AV's behavior in driving logs and its behavior during simu- lation, which may be represented through differing position, heading, speed, acceleration, and more. Directly replaying logged behavior of other objects in the scene [32, 34, 35]
Figure 1: WOSAC models the simu- lation problem as simulation mid- level object representations, rather than as sensor simulation.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table I: A comparison of three autonomous- autonomous--ehicle behavior related tasks which involve generation of a desired future sequence of physical states: trajectory forecasting, planning, and simulation. Note that observations O2 O include simulated agent and environment properties.
MULTIPLK
VEHICLE KINEMATIC
SYSTEM
SYSTEM OBJECTIVES
TASK
ORJECT
OCTPUTS
EVALCATION
CATEGORIES
CONSTRAINIS
Mullo-geot Teajusoay Reecssdiing
X
OPEN-LooP
KENTMATIC ACEURACY AND MODE COVERING
AV Motion Pluaning
concnds
SAFETY, COMPORT, PROGRESS
Agent and Environmnnt Simulation
(0)
CLosED-Loge
DISTRIBUTIONA
under arbitrary AV planning may have limited realism because of this pose divergence. Such log- playback agents tend heavily overestimate the aggressiveness of real actors, as they unwilling to deviate from their planned route under any circumstances. On the other hand, rule-based agents that follow heuristics such the Intelligent Driver Model (IDM) [65] and reactive. We seek to evaluate and encourage the development of sim agents that lie in the middle ground, adhering definition of realism that implies matching the full distribution of human behavior.
overly accommodating
To the best of our knowledge, to date there is existing benchmark for evaluation of simulation agents. Benchmarks have spurred notable innovation in other areas related to autonomous driving research, especially for perception [6, 10, 24, 58], motion forecasting [6, 10, 22, 72, 78], and motion planning [19]. We believe a standardized benchmark can likewise spur dramatic improvements for simulation agent development. Among these benchmarks, those focused motion forecasting perhaps most similar to simulation, but all involve open-loop evaluation, which is clearly deficient compared to closed-loop evaluation. Furthermore, we introduce realism metrics which suitable to evaluating long-term futures, Relevant datasets such as the Waymo Open Motion Dataset (WOMD) [22] exist today that contain real-world agent behavior examples, and we build on top of WOMD to build WOSAC. In this challenge, we focus on a subset of the possible perception outputs, e.g., traffic light states or vehicle attributes are not modeled, but we leave this for future work.
The challenges our benchmark raises are unique, and if we can make real progress on it, we can show that we' ve solved one of the hard problems in self-driving. We have number of open questions: Are there benefits to scene-centric, rather than agent-centric, simulation methods? What is the most useful generative modeling framework for the task? What degree of motion planning is needed for agent policies, and how far can marginal motion prediction take us? How can simulation methods be made more efficient? How can we design a benchmark and enforce various simulator properties? During our first iteration of the WOSAC challenge, user submissions have helped us answer a subset of these questions; for example, we observed that most methods found it most expedient build upon state-of tf- marginal motion prediction methods, i.e. operating in an agent-centric manner.
In this work, describe in detail the Waymo Open Sim Agents Challenge (WOSAC) with the goal of stimulating interest in traffic simulation and world modeling. Our contributions are as follows:
An evaluation framework for autoregressive traffic agents based on the approximate negative log likelihood they assign to logged data.
An evaluation platform, online leaderboard, available for submission at https:// waymo. com/ open/ chachhhleeaaaaa wwwllenness//eee 2023 sim- agents/.
An empirical evaluation and analysis of various baseline methods, as well as several external submissions.
2
Related Work
Multi-Agent Traffic Simulation Simulators have been used to train and evaluate autonomous driving planners for several decades, dating back to ALVINN [43]. While simulators such as CARLA [19], SUMO [33], and Flow [73] provide only a heuristic driving policy for sim agents, they have still enabled progress in the AV motion planning domain [11, 12, 15]. Other recent simulators such as
Nocturne [68] use simplified world representation that consists of a fixed roadgraph and moving
agent boxes.

---

Table 2: Existing evaluation methods for simulation agents. Entries ordered chronologically by Arxiv timestamp. There is limited consensus in the literature regarding how multi-agent simulation should be evaluated.
Evalustion Protocod
ADÉ minADE
Ruse
Offroad
Collision
Rais
Disribution Marching
Instunce Level
Duisee Level
Discibution Matching
Spatial Coverage Diversity
Goal Completion
Coan SocialaalPood]] [17]
Teajestonn (29)
PRECOG (45)
BARK [+]
SMARTS
TrafficSim (9)
SimNet [2]
Symphony [25)
Nocturze [65]
BITS [14]
InterSim (59)
MetaDrive [34
TrafliBBokk [79]
WOSAC (Ours)
Simulation agent modeling is closely related to the problem of trajectory forecasting, sim agent could execute a set of trajectory predictions as its plan [4, 59]. However, as trajectory prediction methods are traditionally trained in open-loop. they have limited capability to recover from out of domain predictions encountered during closed-loop simulation [51]. In addition, few forecasting methods produce consistent joint future samples at the scene level [36, 48]. Sim agent modeling is also related to planning, as each sim agent could execute a replica of planner independently [4]. However, each of these three tasks differ dramatically in objectives, outputs, and constraints (see Table 1).
Learned Sim Agents Learned sim agents in the literature differ widely in assumptions around policy coordination, dynamics model constraints, observability, as well as input modalities. While coordinated scene-centric agent behavior is studied in the open-loop motion forecasting domain [8, 9,57,,, to the best of our knowledge, TrafficSim [60] is the only closed-loop, learned sim agent work to use a joint, scene-centric actor policy; all others operate in decentralized manner without coordination [5, 28, 74], i.c., each agent in the scene is independently controlled by replicas of the same model using agent-centric inference. BITS and TrafficBots [74, 79] use unicycle dynamics model and Nocturne uses a bicycle dynamics model [68] whereas most others do not specify any such constraint; others enforce partial observability constraints, such as Nocturne [68]. Other methods differ in the type of input, whether rasterized [3, 74] provided in a vector format [28, 79J. Some works focus specifically on generating challenging scenarios [47], and others aim for user-based controllability [81]. Some are trained via pure imitation learning [3], while others include closed-loop adversarial losses [28, 60), or multi-agent RL [4, 34, 82] in order to learn to recover from its mistakes [51]. Some works such as InterSim [4, 28, 59, 74, 79, 82] use goal-conditioned problem formulation, while others do not [3].
Evaluation of Generative Models Distribution matching has become a common way to evaluate generative models [18, 20, 27, 30, 31, 45, 46, 49, 52, 77], through the Fréchet Inception Distance (FID) [26]. Previous evaluation methods such as the Inception Score (IS) [53] reason over the entropy of conditional and unconditional distributions, but are not applicable in our case due to the multi- modality of the simulation problem. The FID improves the Inception Score by using statistics of real world samples, measuring the difference between the generated distributions and a data distribution (in the simulation domain, the logged distribution). However, FID has limited sensitivity per example due to aggregation of statistics over entire test datasets into a single mean and covariance.
Evaluating Multi-Agent Simulation There is limited consensus in the literature regarding how multi-agent simulation should be evaluated (see Table 2), and no mainstream existing benchmark exists. Given the importance of safety, almost all existing sim agent works measure some form of collision rate [3, 28, 59, 60, 68, 74], and some multi-object joint trajectory forecasting methods also measure it via trajectory overlap [36]. However, collision rate can be artificially driven to zero by static policies, and thus cannot measure realism. Quantitative evaluation of realism requires comparison with logged data. Such evaluation methods vary widely, from distribution matching of vehicle dynamics [28, 74], to comparison of offroad rates [28, 60, 74], spatial coverage and diversity [60, 74], and progress to goal [59, 68]. However, as goals are not observable, they are thus difficult to extract reliably. Requiring direct reconstruction of logged data through metries such as Average Displacement Error (ADE) has also been proposed [3, 68]. but has limited effectiveness because
3

---

there are generally multiple realistic actions the AV or sim agents could take at any given moment. To overcome this limitation, one option is allow the user provide multiple possible trajectories sim agent, such as TrafficSim, which uses minimum average displacement error (minADE) over 15 simulations. [60].
Recently, generative-model based evaluation has become more popular in the simulation domain, primarily through distribution matching metrics. Symphony [28] uses Vensen-Shamme distances over trajectory curvature. NeuraINDE [75] compares distributions of vehicle speed and inter-vehicle distance, whereas BITS [74] utilizes Wasserstein distances on agent scene occupancy using multiple rollouts per scene, along with Wasserstein distances between simulated and logged speed and jerk two kinematic features which can encapsulate passenger comfort. The latter are computed a distribution to to-distribution comparison on a dataset level, however, this type of metric has shown limited sensitivity our experiments.
Likelihood metries An alternative distribution matching framework is to measure point-to- distribution distances. [17] and [29] introduce a metric defined as the average negative log likelihood (NLL) of the ground truth trajectory, as determined by a kernel density estimate (KDE) [42, 50] over output samples at the same prediction timestep. This metric has found some adoption [54, 64], and we primarily build off of this metric in our work. We note that likelihood baseeoodd--addddddd generative models of simulation such as PRECOG [48] and MFP [63] directly produce likelihoods, meaning that the use of a KDE on sampled trajectories estimate likelihoods not needed for such model classes. Concurrent work [79] also measures the NLL of the GT scene under 6 rollouts.
3 Traffic Simulation as Conditional Generative Modeling
Our goal is encourage the design of traffic simulators by defining a data-driven evaluation framework and instantiating it with publicly accessible data. We focus on simulating agent behavior in setting which an offboard perception system is treated as fixed and given.
We formulate driving as a Hidden Markov Model H = (S,O, ((,((((())) p(s) where S denotes the set of unobservable true world states, O denotes the set observations, p(o,(st) denotes the sampleable emission distribution, and p(se(s--1) denotes the hidden Markovian state dynamics: the probability of the hidden state transitioning from 8 at timestep f. - to 8 at time t. Each 0g E O can be partitioned into AV- and environment-centrés components that vary in time: o [o ofTh). O.t! can in general contain rich set of features, but for the purpose of our challenge, it contains solely the poses of the non-AV agents. We denote the true observation dynamics as p n (o.)s:)),) = (p(oos)
Problem formulation.
The task. The task to build "world model" (ox (220))) of p ( (or|st-1) Ez. = [omap, a --HH-H-H---- -.--- i.e., it denotes context of a static map observation, traffic signal observations, and the observation history, with history length H.
Task constraints:
1. q world must be autoregressive for T steps, i.e., sim agent models must adhere to 10Hz resampling procedure, re-observing the updated scene and consuming their previous outputs.
2. q world must factorize according Eq. 1:
world (oel02.) = (o; (90 (o2)),
(1)
where (o5 is a traffic simulator, and π (o (o2x) is an AV policy'. Any submission that fails satisfy both of these properties will not be considered on WOSAC leaderboards, as determined by
We call this policy because it is similar the typical formulation of a policy in a decision process over actions, although not equivalent, because is defined over next observations rather than current actions. It can be made equivalent to standard policy (a,"" jo2x) by defining the AV's action space A to be equivalent to its component of observation space, and defining an dependde world model g world jo2z. AV BioA" 1000"" (oc.). where 5 denotes the Dirac delta function.

---

%
g
Figure 2: Graphical model of required factorization Bayes net: the two distributions from Eq. 1 are autoregressively interleaved: one represents the AV's "policy" |o and another represents the environmental dynamics ((f the graphical model represent T steps of applying these two distributions. Thick outgoing arrows denote passing inputs from all parent nodes to all children.
challenge submission reports. Requiring them to be generative enables sampling from an arbitrary traffic simulator-AV policy pair. These properties imply the probabilistic graphical model shown in Fig. 2, modified from Fig. 9 of [48]. Algorithms and illustrate valid and invalid submissions.
Much of the challenge of modeling p woord lies in the fact that in many situations 8 E S, p weeld assigns density to multiple outcomes due to uncertainty from agents in the scene, which means that both π (oo and (o(+) often must contain multiple modes in order to perform well. We evaluate distribution-matchin of porrd relative to dataset of logged outcomes. The required factorization into a AV observation-space policy and environment observation dynamics, quorld (or ( (o, () (o, (o))), is fairly flexible. We are agnostic their particular structures. One noteworthy choice for the environment observation dynamics is a "multi-agent" factorization, in which q(otth) ( = II Ta (o = ). i.e., the environment observation dynamics factorizes into sequence A observation-space policies, and the environment observation
itself is partitioned into different components, one for each agent: orth = [og 10me1 cnv.A
Algorithm 1 Valid: Factorized, Closed-Loop. Agent- Centric Simulation
Input: Map "ap and traffic signals dignals, Initial actor states 0-H-1:0 = {0 (0---111 --1. On} where
each ofth = -o. ..... Or (ewv.A) for the A actors in the scene.
Output: Simulated observations 01.7 {01, 02. ",,rr for T simulation timesteps. Simulate for requested number of timesteps
1: for = .... do
2:
f- TAV (ozti map (signals)
3: for ...,, do
> Produce next state for each actor at each timestep
4:
0,
ouv.a t- Ta (o<t; mup odignals)
5:
O1 (o, 00x0 Va 1. A} fop
6: return 01:7 02 ,T]
Algorithm Invalid: Factorized, Open-Loop, Agent-Centric Simulation
Input: Map map and traffic signals diguals Initial actor states o 1-11-:00 (0-H-1.-" ,00 where of SaN.A) for the A actors in the scene.
each og p ,
Output: Simulated observations 01:7 = {0r, 02, ,or] for simulation timesteps.
1: 1.TT TAV (o map osignals)
2: for 1,..., A do
> Produce states at all future timesteps for each actor
3:
onyy Taa (a<1 map oignalls
4: return 01:1 = for
env.a Va (......A) ....A (
4 Benchmark Overview
4.1 Dataset
For WOSAC, we use the test data from the v1.2.0 release of the Waymo Open Motion Dataset (WOMD) [22]. We treat WOMD as set D of scenarios where each scenario is history-future
5

---

pair (0-H-1:0, 021). This dataset offers a large quantity of high-fidelity object behaviors and shapes produced by a state-of-the-ar offboard perception system. We use WOMD's second 10 Hz sequences (comprising H = 11 observations from 1.1 seconds of history and 80 observations from 8 seconds of future data), which contain object tracks at 10 Hz. and map data for the area covered by the sequence. Across the dataset splits, there exists 486,995 scenarios in train, 44,097 in validation, and 44,920 test. These 9.1 second windows have been sampled with varying overlap from 103,354 mined segments of 20 second duration. Up to 128 agents (one of which must represent the AV) must be simulated in each scenario for the second future (comprising 80 steps of simulation).
Agent Definition We require simulation of all agents that have valid measurements at time t = 0. Le. the last step of logged initial conditions before simulation begins. Because the test split data is sequestered, users will not have access to objects that appear after the time of handover, and so therefore could not be expected to simulate them. We require simulation of all three WOMD object types (vehicles, cyclists, and pedestrians). Objects' dimensions stay fixed as per the last step of history (while they do change in the original data).
Submission We do not enforce any motion model (also because we have multiple agent types), which means users need to directly report x/ylz centroid coordinates and heading of the objects' boxes (which could be generated directly or through an appropriate motion model). See the Appendix for additional information on the submission format.
By allowing users to produce the simulations themselves, we reduce the burden on the user by avoiding the need submit containerized software for an evaluation server.
4.2 Evaluation
Agents should generate realistic driving scenarios stochastically. We define "realistic agents" as those that match the actual distribution of scenarios observed during real-world driving. Unfortunately, we do not know the analytic form of the distribution, but we do have samples from it: the examples that make up WOMD. We therefore evaluate submissions using the approximate negative log likelihood (NLL) of real world samples under the distribution induced by the agents.
The NLL we wish to minimize is given by:
D
NLL* = |D]
Σ
log world
(021.10<1,1)
However, there are two problems with trying to minimize Equation 2 exactly in our problem setting. First, 021 is high-dimensional. Instead of trying to parameterize the entire ground truth scenario and compute its NLL under a simulated distribution, we therefore parameterize scenarios with smaller number of component metries (see Section 4.2.1) and aggregate them together into composite NLL metric (see Section 4.2.2). Second, agents may support sampling but not pointwise likelihood estimation [41]. In fact, only require challenge entrants to submit samples from their agents, and therefore have no way of knowing the exact likelihood of logged scenarios under different agent submissions. To avoid this problem, standardize the NLL computation by fitting histograms the 32 submitted samples of agent futures, and compute NLLS under the categorical distribution induced by normalizing the histograms.
4.2.1 Component Metrics
Breaking NLL* * into component metrics has a few benefits. It mitigates the curse of dimensionality described in Section 4.2. It also adds more interpretability to the evaluation, allowing researchers to trade off between different types of errors.
Time Series NLL: Given the time series nature of simulation data, two choices emerge for how to treat samples over multiple timesteps for a given object for a given run segment: to treat them as time-independent or dime-dependent samples. In the latter case, users would be expected to not only reconstruct the general behaviors present the logged data in one rollout, but also recreate those behaviors over the exact same time intervals. To allow more flexibility in agent behavior, we use the former formulation when computing NLLs, defining each component metric m as an average (in log-space) over the time-axis, masked by validity 7z exp
(-ΣI(e) 1T( Σ 1 [v

---

Simulation Input
Log Playback (Logged Oracle)
Wayformer (Diverse Sample)
Constant Velocity
Figure 3: Visualizations of simulation results separate WOMD scenes (top, bottom). Results for various baseline methods are shown on WOMD's validation split, in 2d. 'Simulation input' represents the context history o 1:000 whereas all other columns visualize both (0- 02 1). Two scenes are represented: one where the AV completes the execution of a left turn (top row) and another where the AV remains stopped second and third columns depicts the entire duration of the scene. Trajectories of environment sim agents are drawn in green-blue gradient, and trajectories the AV agent are drawn red-yellow gradient (each as sequence of circles in temporal color gradient).
red traffic signal (boltom row). Each rendering in the
However, we note that as a result, a logged oracle will not achieve likelihoods of 1.0, whereas in the latter formulation logged oracle would.
Definitions We compute NLLs over 9 measurements: kinematic metries (linear speed, linear celeration, angular speed, angular acceleration magnitude), object interaction metrics (distance to nearest object, collisions, time-to-collision), and map-based metrics (distance road edge, and road departures). Please refer to Section A.6 of the Appendix for a complete description and additional implementation details.
4.2.2 Composite Metric
After obtaining component metrics for each measurement, we aggregate them into single composite metric M for evaluating submissions:
M Σ 1=1
M
1
Σ
K my ig
Wy =
NM
i=1
where N the number of scenarios and M 9 the number of component metrics. The component metrics and composite metric M are also parameterized by number of samples K = -2. The value m,y represents the likelihood for the jth metric on the 7th example. The metric M is simply a convex combination (i.e. weighted average) over the component metrics, where the weight wy for the jth metric is set manually. In the interest of promoting safety, the weighting for collision and road departure NLLs are set to 2x larger than weight for the other component metrics.
Experimental Results
In Figure 4 and Table 3, we present quantitative results for handful of methods. We describe each method in more detail in the sections below.
7

---

0.575
WOSAC Camposte Matric
0.533
2,420
0.155
Bandom
Canatam:
Conabant
50 14-A014
CAD
Jont
Wayformer
MTT--44
IVTE
Logged
1+Gasssan
Samplesi
Figure 4: Challenge composite metric results of various baselines on the rest split of WOMD.
5.1 Baselines
Random Agent: An agent that produces random trajectories {(rr, Vt. 0,)) for T = 80, with y. [p, o²), with μ = 1.0 and a = 0.1, in the AV's coordinate frame.
Constant Velocity Agent: An agent that extrapolates the trajectory using the last heading and speed recorded in the provided context/history.
no two-step difference can be computed based on the valid measurements (e.g. the object appeared only at the final step of context), set zero speed for such agents.
Wayformer (Identical Samples) Agent: An agent that produces hybrid of open-loop and closed-loop data using a Wayformer [40] motion prediction model, by executing model inference autoregressively at 2Hz. The agents execute the policy forward for 5 simulation steps, and then replan. Results with a 10 Hz replan rate instead are also shown in Table 3, and an ablation on the replan rate is provided in the Appendix. The maximum maximum-likelihood trajectory for each agent is identically repeated 32 times to produce 32 samples. Each agent is executed by the same policy in an agent-centric frame, batched together for inference, thus complying with the required factorization. Wayformer (Diverse Samples) Agent: An agent that also utilizes Wayformer [40]-generated trajectories, but samples diverse agent plans, from K possible trajectories according their likelihood, instead of selecting the maximum likel-iieo choice.
Logged Oracle: Agent that directly copies trajectories from the WOMD split, with 32 repetitions.
5.2 External Submissions
MultiVerse Transformer for Agent simulation (MVTA) [7]]: A method inspired by MTR [55, 56] that is trained and executed in closed-loop. MVTA uses 'receding horizon' policy with GMM head, and consumes vector inputs.
MVTE: An enhanced version of MVTA [71] that samples a MVTA model from a pool of model variants increase simulation diversity across rollouts.
MTR+++ [44]: hybrid open-looppccloee-loog method with a 0.5Hz replanning rate that is inspired by MTR [55, 56] and searches for the densest subgraph in graph of non-colliding future trajectories.
For a description of other evaluated external submissions, please refer to Section A.4 of the Appendix.
Learnings from the 2023 Challenge
During the course of our 2023 WOSAC Challenge (March 16, 2023 to May 23, 2023), associated with the CVPR 2023 Workshop on Autonomous Driving, we received 24 test set submissions, and 16 validation set submissions, from 10 teams. We continue receive submission queries to evaluation server for our standing leaderboard as new teams submit new methods.
Trends We observed several trends among submissions. First, the challenge champion, MVTA/MVTE [71], was the only method to utilize and benefit from closed-loop training. Other methods that were trained in ooen-loop, such MTR++ [44] or our Wayformer-derived [40] bascline, found operating

---

Table 3: Per-component metric results on the test split of WOMD, representing likelihoods. Methods are ranked by composite metric on the V1 Leaderboard, rather than the previous VO Leaderboard; Numbers within 1% of the best are in bold (excluding 'logged oracle'). ). * indicates method that was received after May 23, 2023, which marked the close of the CVPR 2023 competition.
ASS.K.
SP3323
ALS-3..
110 UM0.
171
(1)
(1)
(1)
(1)
011
RANDOW AppKT
0.007
0:011
0.021
0:001
81000
0.731
0.174
01387
01:155
59.77M
10:391
CONEEAT Vareerty
0.054
0.014
0:761
01:15
0.717
0:651
01287
1-411
Nosp)
Wannnrgpe (Iess TIMAAAA SAMPISS 10 S00
00303
0.081
0.318
0.31
0.107
0.310
0.766
0.774
03308
01338
0.871
0.271
ANTA- [S]
01117
0.171
0:774
0:461
0.364
0:331
0.771
0.59
0.181
(40)
0.349
6.251
0.433
0.305
0.717
0.508
0.799
0.677
0.834
0.531
1.334
1.40
Jone MarerrMrrrrrrree [X]]
I:111
0.733
0.511
0:187
0817
067
01887
REPLANI (4))
MITR [11]
0:11
0.007
0:481
0:134
0.917
0.863
0.797
0991
0.895
0.088
2.135
1.799
NIVTA [J]]
0:19
0.230
6.511
8.493
8-334
0177
8329
04611
0.895
0.866
1866
LooKaRo DEATS
0.61
0.330
0.563
0:484
0:085
1.000
0.811
0.713
1.000
0.755
0.030
01000
at slower replan rates necessary obtain high composite metric results (See Table 3). Second, almost all submissions used Transformer-based methods [67], except for JointMuliiPath++, which used LSTM and MCG blocks [66]. Third, all methods built primarily on top of existing motion prediction works, rather than upon existing motion planning works or sim agent methods from the literature. Only one method, MVTA/MVTE [71]. incorporated aspects of an existing sim agent work, TrafficSim [60], as well as motion planning techniques, implementing a receding horizon planning policy. Thus, fourth, observed the benefit of incorporating planning- into a motion prediction framework. Fifth, most methods (excluding MointMultipahh+++ [70] built upon the 2022 CVPR Waymo Open Motion Prediction challenge champion, MTR [55, 56]. likely due to the open-source availability of its codebase and SOTA performance. Finally, all submissions operated in an agent-centre coordinate frame, rather than jointly sampling from scene representation simultaneously.
methods
Likelihood Metries Reward Diversity We found that our likelihood- metries reward models that produce diverse futures. For example, generating 32 diverse rollouts per scene with a Wayformer model performs I % better our evaluation metries than Wayformer model that produces 32 identical rollouts per scene (see Figure 4).
Collision Minimization an Algorithmic Objective Several methods designed algorithmic com- ponents to determine futures with a minimal number of collisions, e.g., MTR++ [44] which used clique-fiding in an undirected graph of collision-free future trajectories, and CAD [14], which used rejection sampling on open-loop futures that created collisions. This objective aligns with human preference, but close calls and collisions do occur in real driving data distributions, optimizing for this objective could be seen as trimming the tail of the distribution; distracted drivers generally do exist in everyday real world driving, and in certain scenarios, one would expect a low-quality planner to perform poorly and produce collisions with sim agents, and so such should be taken into consideration for generating realistic simulations. This suggests a limitation of the WOMD [22], which has few examples from the tail distribution of real driving, and efforts to upsample collision data could prove useful. In addition, open-loop methods such as CAD [14] that prune collisions after the fact could prune collisions caused by the AV rather than by the sim agents, yielding a mislcadingly optimistic view of the AV's performance.
Composite Metric vs. (min-)ADE and ADE: ings by ADE and minADE and ranking by our NLL composite metric disagree. However, methods with lower minADE do tend to achieve higher composite scores; ADE does not exhibit such a trend.
We see that among submissions to the test set, rank-
Composite Metric Results The ordinal ranking shown in Figure 4 and Table 3 indicates that learned, stochastic sim agents outperform not only heuristic basclines but also Iearned, deterministic sim agents. We consider composite metric score of 0.722 as a practical upper bound submissions, because it involves access to test data via an oracle.
Metric Results In Table 3, we provide breakdown of the composite metric into component metric results. As expected, the 'logged oracle' baseline achieves the highest likelihood in each of the 9 component metrics. The top performing method, MVTE [71] scored highest on all but one component metric (lineur acceleration likelihood), where CAD [14] outperformed MVTE by 12% (likelihood of 0.253 vs. 0.222). Surprisingly, MVTE [71] has angular acceleration likelihoods within a percentage point of the 'logged oracle' (0.481 vs. 0.489). The gap between the top performing learned method (MVTE) and 'logged oracle' in both collision likelihood (0.893 vs. 1.000) and
Component

---

distance-to to-nearest object likelihood (0.383 vs. 0.485) indicates significant room for improvement in future work on interactive metrics.
5.4 Qualitative Results
In Figure 3, provide qualitative comparison of various baselines two WOMD scenarios. The results indicate that the complexity of behaviors within intersections far exceeds the capability of simple heuristics predict. Collisions evident from the constant velocity baselines in both examples. Additional qualitative examples from other sim agent methods are shown in Section A.5 of the Appendix.
6 Discussion
Limitations. For our 2023 Challenge, we manually verified the validity of each submission accord- ing factorization and closed-loop requirements discussed in each team's 's report, and we observed that the technical rules were subtle. Several of the submissions that used open-loop or hybrid open-loop/closedllop methods may have limited applicability for some simulation applications.
Even if we had instituted a benchmark based on conter-containeriiee software submissions instead of uploading output trajectory submissions, enforcing our requirements algorithmically and automati- cally would still be challenging. Although many properties of function calls to Dockerized software can be measured, e.g. latency, long any arbitrary state is maintained by the user, the system could not enforce all details of the closed-loop nature of the function call. As a result, user-submitted simulation agent software would have adhere strict stateless input and output data APIs. The ability to do so would assist in removing ambiguity regarding whether methods that prune collisions post-hoc qualify closed-loop.
a user provides containerized simulator submissions, one approach to encourage adherence to our requirements and to further incentivize closed-loop behavior would be to provide and interact with an AV policy that the user does not control. our benchmark, the user was allowed to control the AV. albeit through an independent policy; the ability to evaluate simulator submissions separate, held-out AV motion planning policies and on new scenarios would allow further valuable analysis.
Future Work Object insertion and deletion are important aspects of the simulation problem, yet intentionally introduced an assumption of no object insertion or deletion in order reduce the complexity of the first iteration of the WOSAC challenge for users. Motion planners trained or evaluated in a simulator must have the capability to exercise caution regarding areas of occlusion from which new objects may emerge at any timestep. In future iteration of the challenge, we plan to introduce realism metries that reward mooeely object insertion and deletion, e.g. distributional metrics on the number of vehicles appearing or disappearing at each frame, or the distance of simulated objects from the autonomous vehicle. The data distribution the WOMD dataset already includes such object insertion and deletion.
Furthermore, we intentionally introduced an assumption of time-invariant object dimensions in our first iteration of WOSAC to simplify the modeling challenge for users. Time-variant object dimensions can be considered as a type of vehicle attribute, and object dimensions do actually change in the underlying data distribution provided in the WOMD dataset. We hope to include time-variant object dimension prediction an aspect of the benchmark in future iterations.
As discussed in Section 5.3, given the prevalence of collision minimization algorithmic components among submissions, data, or our metrics are limited in some way. Another approach would be to "fatten the tails" of the evaluation data distribution by generating synthetic, challenging initial conditions [3, 21, 23, 47, 61, 69], or mining more close calls and collisions from real driving data.
may presume that collisions
heavily represented in WOMD [22]
Conclusion. In this work, we have introduced new challenge for evaluation of simulation agents, explaining the rationale for the different criteria we require. We invite the research community to continue to participate.
10

---

Acknowledgments and Disclosure of Funding
No third-party funding received in direct support of this work. We thank Ben Sapp for his helpful feedback in preparing the challenge. We would like thank Mustafa Mustafa, Kratarth Goel, Rami Al-Rfou for offering consultation, models and infrastructure that accelerated our work. We thank Alexander Gorban for his assistance in developing and maintaining the evaluation server. All the authors are employees of Waymo LLC.
References
[1] Gul Agha and Karl Palmskog, A survey of statistical model checking, ACM Transactions on Modeling and Computer Simulation (TOMACS), 28( 1-139, 2018.
[2] Mayank Bansal, Alex Krizhevsky, and Abhijit S. Ogale. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst. In Robotics: Science and Systems XV, 2019.
[3] Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blažej Osiński, Hugo Grimmet, and Peter Ondruska. SimNet: Learning reactive self-driving simulations from real-world observations. In ICRA, 2021.
[4] Julian Bernhard, Klemens Esterle, Patrick Hart, and Tobias Kessler. BARK: Open behavior benchmarking in multi-agent environments. In IROS, 2020.
[5] Raunak P Bhattacharyya, Derek J Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer. Multi-agent imitation learning for driving simulation. IROS, 2018.
[6] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset autonomous driving. In CVPR, pages 11621-11631, 2020.
[7] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong. Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop, 2021.
[8] Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. SpAGNN: Spatially-awar graph neural networks for relational behavior forecasting from sensor data. In ICRA, pages 9491-9497, 2020.
[9] Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. ECCV, 2020.
[10] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Luccy, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, June 2019.
[11] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference Robot Learning, pages 66-75. PMLR, 2020.
[12] Dian Chen, Vladien Koltun, and Philipp Krähenbühl. Learning to drive from world on rails. In ICCV, pages 15590-15599, 2021.
[13] Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. Geosim: Realistic video simulation via geometry-aware composi- tion for self-driving. In CVPR, pages 7230-7240 June 2021.
[14] Hsu-kuang Chiu and Stephen F. Smith. Collision avoidance detour: A solution for 2023 waymo open dataset challenge- sim agents. Technical report, Carnegie Mellon University, 2023.
[15] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. ICRA, pages 4693-4700 IEEE, 2018
[16] Anthony Corso, Robert Moss, Mark Koren, Ritchie Lee, and Mykel Kochenderfer. A survey algorithms for black-box box safety validation cyber-physical systems. Journal of Artificial Intelligence Research, 72: 377-428, 2021.
[17] Nachiket Deo and Mohan M. Trivedi. Convolutional social pooling for vehicle trajectory prediction. In CVPR Workshops. Computer Vision Foundation IEEE Computer Society, 2018
11

---

[18] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In Near!PS. 2021.
[19] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRI. 2017.
[20] Patrick Esser, Robin Rombach, and Bjorn Ommer: Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[21] Nick Roy Ethan Pronovost, Kai Wang. Generating driving scenes with diffusion. In ICRA Workshop on Scalable Autonomous Driving, June 2023.
[22] Scott Etinger, Shuyang Cheng, Benjamin Caine. Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R. Qi, Yin Zhou, Zoey Yang, Aurélien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV, 2021.
[23] Lan Feng. Quanyi Li, Zhenghao Peng. Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In ICRA, 2023.
[24] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[25] E.G. Gilbert, D. D.W. Johnson, and S.S. Keerthi. fast procedure for computing the distance between complex objects in three-dimensional space. IEEE Journal on Robotics and Automation, 4(2):193-22 203, 1988.
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
[27] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion prohabilistic models. In NeurlPS, 2020.
[28] Maximilian Igl. Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In ICRA, 2022.
[29] Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In ICCV, October 2019.
[30] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
[31] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[32] Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, and Peter Ondruska. Drivergym: Democratising reinforeement learning for autonomous driving. arXiv preprint arXiv:2111.06 2021.
[33] Daniel Krajzewicz. Georg Hertkorn, Christian Rössel, and Peter Wagner. Sumo (simulation of urban mobility)-an open-source traffic simulation. In Proceedings of the 4th middle East Symposium on Simulation and Modelling (MESM20002). pages 183-187, 2002.
[34] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. TPAMI, 2022.
[35] Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Becca Roclofs, Benjamin Sapp. Brandyn White, Aleksandra Faust, Shimon Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In IROS, 2023.
[36] Wenjie Luo, Cheol Park, Andre Commann, Benjamin Sapp. and Dragomir Anguelov. Jfp: Joint future prediction with interactive multi-agent modeling for autonomous driving. In CoRL, 2023.
[37] Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. In CVPR, June 2020.
[38] Xiaoyu Mo. Haochen Liu, Zhiyu Huang, and Chen Lv. Simulating behaviors of traffic agents for au- tonomous driving via interactive autoregression. Technical report, Nanyang Technological University., 2023.
12

---

[39] Xiaoyu Mo, Yang Xing, Haochen Liu, and Chen Lv. Map-adaptive multimodal trajectory prediction using hierarchical graph neural networks. IEEE Roboticy and Automation Letlers, 866):3685-369,, 2023.
[40] Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. ICRA, 2023.
[41] Schastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
[42] Emanuel Parzen, On estimation of probability density function and mode. The annals of mathematical statistics, 33(3):1065-1076, 1962.
[43] Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in Neural Information Processing Syxtems, 1988.
[44] Cheng Qian, Di Xiu, and Minghao Tian. A simple yet effective method for simulating realistic multi behaviors. Technical report, 2023.
[45] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[46] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 220022204400125,, 1(2)3, 2022.
[47] Davis Rempe, Jonah Philion, Leonidas J Guibas, Sanja Fidler, and Litany. Generating useful accident- prone driving scenarios via learned traffic prior. In CVPR, June 2022.
[48] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In ICCV, October 2019.
[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. In CVPR, 2022.
[50] Murray Rosenblatt. Remarks on some nonparametric estimates of density function. The annals of mathematical statistics, pages 832-837, 1956.
[51] Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011.
[52] Chitwan Saharia, William Chan, Saurahh Saxena, Lala Li,, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Flect, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
[53] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
[54] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++:: Dynamically- feasible trajectory forecasting with heterogeneous data. In ECCV, 2020.
[55] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mir-a: 1st place solution for 2022 waymo open dataset challenge- motion prediction, 2022.
[56] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems, 2022.
[57] Dilia Su, Bertrand Douillard, Rami Al-Rfou, Cheolho Park, and Benjamin Sapp. Narrowing the coordinate- frame gap in hehavior prediction models: Distillation for efficient and accurate scene-centric motion forecasting. arXiv:2206.03970 2022.
[58] Pei Sun. Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[59] Qiao Sun, Xin Huang. Brian Williams, and Hang Zhao. InterSim: Interactive traffic simulation via explicit relation modeling. In IROS, 2022.
13

---

[60] Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In CVPR, 2021.
[61] Shuhan Tan, Kelvin Wong. Shenlong Wang. Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In CVPR. June 2021.
[62] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
[63] Charlic Tang and Russ R Salakhutdinov. Multiple futures prediction. In NeurlPS, 2019.
[64] Luca Anthony Thiede and Pratik Prabhanjan Brahma. Analyzing the variety loss in the context of prohabilistic trajectory prediction. In ICCV, October 2019.
[65] Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805, 2000.
[66] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Comman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, and Benjamin Sapp. Multipath++:: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA, 2022.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakoh Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[68] Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang. Brandon Amos, and Jakob Foerster. Nocturne: scalable driving benchmark for bringing multi-agent learning one step closer to the real world. In NeurIPS Datasets and Benchmarks Track, 2022.
[69] Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun, Adysim: Generating safety- scenarios for self-driving vehicles. In CVPR, 2021.
[70] Wenxi Wang and Haotian Zhen. Joint-multipath++ for simulation agents. Technical report, 2023.
[71] Yu Wang, Tiebiao Zhao, and Fan Yi. Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023. Technical report, Pegasus, 2023.
[72] Benjamin Wilson, William Qi. Tammay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self drividiiing perception and forecasting. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021), 2021.
[73] Cathy Wu, Aboudy Kreidich, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen, Flow: Architec- ture and benchmarking for reinforeement learning in traffic control. arXiv preprint 1710.0550 10, 2017.
[74] Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: Bi-level imitation for traffic simulation. In ICRA, 2023.
[75] Xintao Yan, Zhengxia Zou, Shuo Feng. Haojie Zhu, Haowei Sun, and Henry X Liu. Learning naturalistic driving environment with statistical realism. Nature Communications, 14(1):2037, 2023.
[76] Zhenpei Yang. Yuning Chai, Dragomir Anguelov, Yin Zhou, Pei Sun, Dumitru Erhan. Sean Rafferty, and Henrik Kretzschmar. Surfelgan: Synthesizing realistic sensor data for autonomous driving, In CVPR, June 2020.
[77] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang I Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
[78] Wei Zhan, Liting Sun. Wang. Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kümmerle, Hendrik Königshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps. 1910003888 [cs, cess], 2019.
14

---

[79] Zhejun Zhang, Alexander Liniger. Dengxin Dai, Fisher Yu, and Luc Van Gool. Trafficbots: Towards world models for autonomous driving simulation and motion prediction. In ICRA, 2023.
[80] Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guides traffic simulation via scene-level diffusion. In CoRL, 2023.
[81] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che. Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In ICRA, 2023.
[82]
Ming 7hou, Jun I. Luo, Julian Villella, Yaodong Yang. David Rusu, Jiayu Miao, Weinan Zhang. Montgomery Alban, IMAN FADAKAR. Zheng Chen, Chongxi Huang. Ying Wen, Kimia Hassanzadeh, Daniel Graves, Zhengbang Zhu, Yihan Ni. Nhat Nguyen, Mohamed Elsayed. Haitham Ammar, Alexander Cowen-Rivers, Sanjeevan Ahilan, Zheng Tian, Daniel Palenicek, Kasra Rezaee, Peyman Yadmellat, Kun Shao, dong chen, Baokuan Zhang, Hongbo Zhang, Jianye Hao, Wulong Liu, and Jun Wang. Smarts: An open-source scalable multi-agent rl training school for autonomous driving. In CoRL, 2020.
15

---

A Appendix
In this Appendix, we provide ablations investigating the impact of replan rate Wayformer-derived [40] sim agent baselines on the test split, as well as additional learnings from the 2023 CVPR competition. We also include additional descriptions of methods from external challenge submissions, corresponding qualitative results for such methods, and implementation details of each of the 9 component metrics we use. Finally, we describe the exact details the dataset splits we use and give leaderboard submission instructions.
Benchmark Versioning In December 2023, we improved the accuracy of the collision and offroad likelihood calculation, which improved most collision likelihood scores, offroad likelihood scores, and composite metric results. This paper describes the updated scores (the VI version of the benchmark), rather than the previous VO scores presented at the Workshop on Autonomous Driving at CVPR 2023, Both versions of the leaderboards are available online (V1 Leaderboard, VO Leaderboard).
A.1 Replanning Rate Ablation Results
As shown in Table 3 of the main paper and as discussed in Section 5, we perform an ablation of the impact of replanning rate on composite metric performance for open-loop trained models on the WOMD test set. We show that more frequent replan rate negatively impacts Wayformer-biee based [40] agent, irregardless of whether multiple diverse rollouts per scene are sampled or if 32 identical rollouts scene are produced. For the identical-sane -ample producing agent, we see relative performance drop of 41.2% (composite scores of 0.575 vs. 0.338) when transitioning from replanning at 2Hz to 10Hz.
In Figure 5, we visualize several additional data points from a replanning interval of 100 ms to 1100 for the same identical-sample producing agent.
0.5
Wayfomar
Ausrage Disglasemmnt Enar (ADE)
0.00
Composee
0.35
0.30
0.15
0.40
Wayfomar
0.25
750
1300
Replanning
Replanning
(a) WOSAC Composite Metric replan
(h) ADE vs. replan rate.
compos matric
Incor speed hood
0.
menrr accelenation las thoos
[ooihhod
angu speed teathood
05
anpu ar scce niton liee hordd
dictance to nearent ob Icelhood
col on indiration Icelhood
time to oo ission line thood
distance to edge Hldl
offrosd Indiation like thood
02:
01
203
800
10000
Replanning Interval/Period (ms)
Component likelihood metrics vs. replan rate.
Figure 5: Results at various replanning rates on the WOSAC test set for a Wayformer baseline (producing identical samples for the 32 rollouts). the replanning rate increases from Hz, then to 2 Hz, and then to 10 Hz, we observe a smooth degradation in performance.
16

---

A.2 Additional Learnings from the 2023 CVPR Competition
Generative Modeling While many families of generative models exist, most challenge participants restricted their modeling to a narrow class of such models, namely GMMs. To our knowledge, for the 2023 CVPR challenge, no user submitted sim agent behavior generated by normalizing flow, GAN, or variational autoencoder (VAFs) models, or by denoising diffusion models, although such diffusion techniques have recently become more popular in the simulation agent literature [80, 81]. We expect this may change in the future as more entrants participate in the WOSAC challenge.
Further Quantitative Analysis We note that the top-four performing methods were executed fully in closed-loop (MVTE [71], MVTA) or in a hybrid fashion (MTR+++ [44], Wayformer [40]), outperforming the two open-loop submissions, JointMultiPath++ [70] and CAD [14], as shown in Table 4. This gap is especially clear between non-open loop methods and open-loop methods in the collision likelihood metric.
Among all baselines we evaluated, the constant velocity baseline is weakest when it comes to angular- based likelihoods. For example, it achieves close zero likelihood on both angular speed (0.02) and angular accelcration (0.04), as opposed to the MVTE method, which achieves 0.54 and 0.38 likelihoods on the same two component metrics, respectively. This result is intuitive, our constant velocity model does not account for any yaw rate.
A.3 Additional Comparisons with Other Benchmarks for Autonomous Driving Behavior
In Table 4, we compare our WOSAC benchmark with other benchmarks used for evaluation of behavior models for autonomous driving.
BENCHMARK NAME
TASK
Argoverse [10]
Trajectory Forecusting
INTERPRET (INTERACTIO) [78]
Trajectory Forecasting
Argoverse2 [72]
Trajectory Forecasting
nuScenes [6]
Trajectory Forecusting
WOMD [22]
Trajectory Forecasting
CARLA [19]
Motion Planning
nuPlan [?]
Motion Planning
WOSAC (Ours)
Multi-Agent Simulation
Table 4: Existing benchmarks for evaluation of behavior models for autonomous driving.
A.4 Additional Information about Methods from External Challenge Submissions
MultiVerse Transformer for Agent simulation (MVTA) [71]: A method inspired by TrafficSim [60] that is trained and executed in closed-loop and adheres to WOSAC's factorization and autoregres- sive requirements. It uses 'receding horizon' policy (i.c. predicting sec. of future motion but using only the next 100 ms). Inspired by MTR [55, 56]. MVTA places a Gaussian Mixture Model (GMM) head on top of a waanformer-basel encoder and decoder (employing the same encoder/decoder layers as implemented in MTR), consuming vector inputs. Rather than utilizing a fixed-length history context, MVTA agent encoding contains agent history motion state (i.e., position, object size, heading angle, and velocity) and one-hot category mask of each agent. The prediction heads include a regression head that outputs 5 GMM parameters (Jzz, (yy, Oa, yy P), along with the velocity (Uz, Uyy and heading (O), (0) predictions for timestep, and a classification head that outputs probability p. Both heads take the query content features (naum_query x hidden feature dimension) as input.
variable-length history to potentially use all of the past data. The input
MVTE: An enhanced version of MVTA [71] wherein 3 variants of MVTA trained and randomly selected to generate each of the 32 simulations, increasing simulation diversity across rollouts.
MTR++ [44]: A hybrid method with 0.5Hz replanning rate and a 2 second prediction horizon. We note that MTR+++ does not fully adhere to WOSAC's closed-loop requirement, it does not replan at 10 Hz rate. MTR+++ also does not adhere to the policy factorization requirements, as world vs. AV policies are not separated. Inspired by MTR [55, 56), the method addresses two key limitations of MTR: inaccurate heading predictions and excessive collisions incurred by marginal predictions alone. To overcome the first issue, the authors estimate headings from s/y trajectories. Second, in order to minimize collisions, the authors consider K 6 trajectories predicted per agent
17

---

by MTR, and prune the exponential number of futures in a greedy fashion. As brute-fore exhaustive search over the 6N combinations computationally infeasible, MTR+++ searches for the densest subgraph in a graph of non-colliding future trajectories.
First, a 6N by 6N distance matrix D is constructed, where entry Dom+i-1,6atj-1 indicates the minimum L2 distance between the ith-highest trajectories of agent m and the jth-highest one of agent n. Second the distance matrix is binarized by evaluating which distances correspond to collisions according to object extents. Finally, a clique-finding heuristic method finds a dense subgraph of size N. An ensemble of 32 fine-tuned MTR models is employed to create the 32 rollouts, each model producing a single rollout.
Collision Avoidance Detour (CAD) [14]: An open-loop method that builds upon existing motion forecasting method, MTR [55, 56] to produce marginal trajectory predictions, and resamples the entire future if future agent collisions are anticipated, until maximum number of trials is exhausted. While CAD adheres to the factorization requirement, it does not adhere to WOSAC's closed-loop requirement. Factorization of world vs. AV policies is accomplished by using different checkpoints of an MTR motion prediction model for the two agent groups, and motion of non-evaluated agents is simulated using a constant velocity model.
Moint-Multipath++ [70]: An open-loop. scene-centric method that builds off of MultiPath++ [66], producing in single model pass 32 rollouts, each representing entire length-80 trajectory. JointMultiPathh+ does not factorize AV vs. world policies. and thus does not fully adhere to WOSAC"s policy factorization requirements. Agent history information (positions and headings of all agents) are transformed into the AV's coordinate frame, while closest lane information is selected for each agent. In its encoder, JointMultiPPahh+ concatenates the output of 2 LSTMs and one MultiContex(Gatin encode agent history, another LSTM is used to encode per-step differences in agent history, and an MCG block is used to encode agent history with corresponding timesteps. Subsequent MCG blocks fuse per-agent information with road network (polyline) embeddings. In its decoder, a series of MLP blocks transform n per-agent embeddings to rollouts represented as n x 32 x 80 x 3 output tensors. 2
(MCG) [66] block to form per-agent embeddings; one LSTM is used to
SBTA-ADIA Mo et al. [38]: Builds upon an existing motion prediction method [39]. This hierarchi- cal method splits the problem into a first phase of multi-agent goal prediction, based on a GNN scene encoder, and simple planning policy which tries accomplish the goal closed-loop.
A.5 Additional Qualitative Results from External Challenge Submissions
In Figure and 7, we provide additional qualitative examples of simulation results from external challenge submissions.
Simulation Input
Log Playback (Logged Oracle)
MTR++ [44]
MVTE [71]
Figure 6: "wo-dimensiooaal visualization of simulation results on WOMD's test split. MVTE exhibits a collision and MTR+++ produces a near-miss. "Simulation input' represents the context history O-H-1:0. whereas all other columns visualize both (o- 121). One possible future for a single scene is represented, selected from 32 submitted rollouts, where the AV remains stopped at red traffic signal. Each rendering in columns 2,3 and 4 depicts the entire duration of the scene. Trajectories of environment sim agents drawn in a green-blue gradient (each sequence of circles in temporal color gradient). The AV agent is drawn in orange.
"Code available at ht tps: : //gi thub. , con howp//aaggeexd--aaddsoooo
18

---

Simulation Input
Log Playback (Logged Oracle)
MTR++ [44]
MVTE [71]
Figure 7: Two-dimensional visualization of simulation results on single scene from WOMD's test split using various baseline methods. Four possible futures for this single scene are represented (one per row), selected from 32 submitted rollouts. 'Simulation input' represents the context history 0-1:00 whereas all other columns visualize both (0_H-1:0, 021). Each rendering in the second, third, and fourth columns depicts the entire duration of the scene. Trajectories of environment sim agents are drawn in green-blue gradient, and trajectories of the AV agent are drawn in red-yellow gradient (each as a sequence of circles in a temporal color gradient). The AV agent drawn in orange.
19

---

A.6 Component Metrics Implementation Details
1. Linear Speed: Unsigned magnitude of the first derivative || - where x; = [Te, Mr, 2r]. Linear speed in 3D computed the 1-step difference between 3D trajectory points. We employ speed, rather than velocity, as velocity can either be defined w.r.t. the ego-agent's heading, or w.r.t. global coordinate system, where velocity directions may be city-specific, based on orientation of roads w.r.t. North. Although this cannot capture objects moving in reverse, rare behavior, we omit for sake of simplicity.
2. Linear Acceleration Magnitude: Signed magnitude of second derivative, in 3D computed as the 1-step difference between speeds objects. ||v, 1|-lve]
3. Angular Speed: Signed first derivative w = d(0++1.8.) computed as the 1-step difference in f heading, where d(-)) represents the minimal angular difference between two angles the unit circle, i.e., d(-) a distance metric on SO(2) computed as min(10++1 -Or], 2n 00--1 0el with 0,, 0-- [0, 2m).
4. Angular Acceleration Magnitude: Second derivative, computed the 1-step difference in angular speed w, as
d(wr41)w)))
5. Distance to nearest object:
Signed distance (in meters) the nearest object in the scene. We use Minkowski difference of box polygons, according to a simplified version of the Gilbert-Dohnnson-eeethhh (GJK) distance algorithm [25].
6. Collisions: Count indicating objects that collided, any point in time, with any other object, i.e. when the signed distances to nearest objects, as described above, achieves a negative value.
7. Time-to to-oooliioiiinn (TTC): Time (in seconds) before the object collides with the object it is following (if one exists), assuming constant speeds. An object is defined as exhibiting boject-following (tailgating) behavior based on alignment conditions derived from heading and lateral distance.
8. Distance to road edge: Signed distance (in meters) to the nearest road edge the scene.
9. Road departures: Boolean value indicating whether the object went off the road, at any point in time [60].
To prevent undefined scores from histogram bins with zero support, we employ Laplace smoothing with pseudocount of 0.1.
Inserted and Deleted Object Handling In order to prevent object insertion/deletion bias between the logged and simulated data distributions during evaluation, discard any newly spawned objects (appearing after the history interval) in the logged test set when computing the logged data distribution. The data distribution in the WOMD dataset already includes such object insertion and deletion.
A.6.1 Evaluation Source Code References
In this section, we provide pointers to our specific implementations of the 9 metrics discussed in Section 4.2.1 of the main paper:
Kinematic-based features: Linear speed, linear acceleration magnitude, angular speed, and angular acceleration magnitude (metrics 1,2,3,4): [Code]
Interaction- features: TTC and distance nearest object (metrics 5, 6, 7): [Code] and modified GJK algorithm implementation [Code]
Map-based features: Road departures and distance to road edge (metrics 8, 9): [Code]
An implementation of our time-series based NLL computation can be found here: [Code].
A.6.2 Evaluation Code License and Dependencies
The WOMD [22] dataset itself is licensed under a non-commercial license (www.. wa ymo. com/ open/terms) and the evaluation code for our Waymo Open Sim Agents Challenge (WOSAC) is released under BSD+limited patent license. See
20

---

and
Dependencies used include NumPy (numpy), the Waymo Open Dataset repository (waymo-opa) -f
TensorFlow (tensorflow), Tensor- Flow Probability watsoorflow_ppobbbllll Matplotlib (matplotiib) TQDM (tqdm), Protocol Buffers eeoogle.poooblle and Python standard library imports (os, tarfile, dataclasses).
A.7 Additional Information about WOMD Splits Used
We exclude 401 segments from evaluation due discrepancies in object counts across the Scenario proto and tf. Example formats, due to object count truncation arising from fixed- shape t Example tensors with exactly 128 object slots. We also exclude from evaluation 9 test run segments which are missing maps (however, maps are present for each scenario present in the the validation set).²
Table 5: Statisties of WOMD dataset [22] splits used.
DATASET SPLIT VALIDATION
TESTING
ALL SCENARIO COUNTS
44097
44920
EVALUATED SCENARIO COUNTS
44520
A.8 Submission Format
Submissions must be uploaded as serialized SimAgent SerrhhaalleeSSummiii ssion protocol buffer data' ("protos"). Each ScenarioRol loutts proto within the submission must contain 32 8-second rollouts of simulation data from one scenario. A validation or test set submission may be submitted to the evaluation server.
provide Jupyter notebook tutorial with additional instructions and examples on how to generate submission for a dataset split. We recommend storing multiple ScenarioRe lout's in each binary proto file (i.e. in each S SimAgent scha sChallehhcccccc SechallengSSune Submis ssion file) to prevent creating tar.gz file with tens of thousands of files; use of 100 to 150 of such shards is recommended. Please refer to the tutorial notebook for the naming convention of these files. Submission data should be compressed as single tar.gz archive and uploaded as single file.
'WOMD download
instructions
available
https::/wwap wttp:///////t/tts//////ooe/sto/s
down1 oad.
'https:/ /ttp://prooobuott/ooooo dev/
21


Reviews:
[{"review_id": "bBbw9AqMJB", "reviewer": "Reviewer_58Fn", "summary": "", "questions": "", "limitations": "The Discussion provided clearly outlines the limitations of this approach.\n\nThat said, given that this is the first iteration of the WOSAC competition, there is reasonable scope for further improvement.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper introduces the Waymo Open Sim Agents Challenge (WOSAC) for autonomous driving.\n\nThe paper notes that this is the first public competition in this important area.\n\nOverall, I think this will be an important data set for autonomous driving.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Cnb7Gko7o8", "reviewer": "Reviewer_ZL1T", "summary": "", "questions": "", "limitations": "As stated in Section 4.1, the dimensions of objects in the simulator remain fixed based on the last step of their history, while in the original data, these dimensions may vary. Therefore, authors are kindly urged to address the associated caveats and discuss any differences that arise between the simulator and the original data regarding object dimensions.", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "\"Pose divergence\" or \"Simulation shift\" between the AV natural driving logs and its behavior during simulation can be represented through different position, heading, speed, acceleration and etc. The paper evaluate the development of simulation agents to match the full distribution of human behavior. There is no existing benchmark for evaluation the realism of AV's simulation agents. The test data contains a large quantity of high-fidelity object behaviors and shapes produced by the state-of-the-art perception system. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "yUqbzfbCVw", "reviewer": "Reviewer_TuS8", "summary": "", "questions": "", "limitations": "Well discussed", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This platform aims at facilitating the research of a new but important AD problem.\n2. The system is well-developed and already attracts some external users.\n3. The task definition is clear and complete\n4. This challenge is released and maintained by a reliable organization leading AD research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vCKV7MUf5P", "reviewer": "Reviewer_9P4x", "summary": "", "questions": "", "limitations": "This is addressed", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. An evaluation framework for autoregressive traffic agents\n2. An evaluation framework and an online leaderboard\n3. Evaluations and analyses are provided", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "N1ynffrAGw", "reviewer": "Reviewer_xmgp", "summary": "", "questions": "", "limitations": "I have no concern regarding authors' discussions about the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. There already has been one completed challenge in CVPR 2023, which shows the validity of the proposed dataset and benchmark.\n2. Since Waymo is the largest open motion prediction dataset, this benchmark enjoys the benefits of large scale.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Five experts reviewed this paper with all accepted recommendations. The area chairs agree that this work makes a very important contribution by introducing a new traffic simulation for autonomous driving evaluation. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper.
Decision: Accept (Spotlight)