SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nico Montali John Lambert Paul Mougin Alex Kuefler Nicholas Rhinehart
Michelle Li Cole Gulino Tristan Emrich Zoey Yang Shimon Whiteson
Brandyn White Dragomir Anguelov
Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.
Simulation environments allow cheap and fast evaluation of autonomous driving behavior systems, while also reducing the need to deploy potentially risky software releases to physical systems. While generation of synthetic sensor data was an early goal [19, 43] of simulation, use cases have evolved as perception systems have matured. Today, one of the most promising use cases for simulation is system safety validation via statistical model checking [1, 16] with Monte Carlo trials involving realistically modeled traffic participants, i.e., simulation agents.
Simulation agents are controlled objects that perform realistic behaviors in a virtual world. In this challenge, in order to reduce the computational burden and complexity of simulation, we focus on simulating agent behavior as captured by the outputs of a perception system, e.g., mid-level object representations [2, 79] such as object trajectories, rather than simulating the underlying sensor data [13, 37, 62, 76] (see Figure 1).
A requirement for modeling realistic behavior in simulation is the ability for sim agents to respond to arbitrary behavior of the autonomous vehicle (AV). "Pose divergence" or "simulation drift" [3] is defined as the deviation between the AV's behavior in driving logs and its behavior during simulation, which may be represented through differing position, heading, speed, acceleration, and more. Directly replaying logged behavior of all other objects in the scene [32, 34, 35]
Figure 1: WOSAC models the simulation problem as simulation of mid-level object representations, rather than as sensor simulation.
Table 1: A comparison of three autonomous-vehicle behavior related tasks which involve generation of a desired future sequence of physical states: trajectory forecasting, planning, and simulation. Note that observations  $o_{t} \in \mathcal{O}$  include simulated agent and environment properties.
TASK	MULTIPLE
OBJECT
CATEGORIES	OUTPUTS	VEHICLE
KINEMATIC
CONSTRAINTS	SYSTEM
EVALUATION	SYSTEM
OBJECTIVES
Multi-Agent Trajectory Forecasting	✓	((xt,yt,θt,vtx,yt))T=1	✗	OPEN-LOOP	KINEMATIC ACCURACY AND MODE COVERING
AV Motion Planning	✗	((xt,yt,θt)T)t=1 or controls	✓	CLOSED-LOOP	SAFETY, COMFORT, PROGRESS
Agent and Environment Simulation	✓	(ot)T=1; ot ∈ O	✗	CLOSED-LOOP	DISTRIBUTIONAL REALISM
under arbitrary AV planning may have limited realism because of this pose divergence. Such log-playback agents tend to heavily overestimate the aggressiveness of real actors, as they are unwilling to deviate from their planned route under any circumstances. On the other hand, rule-based agents that follow heuristics such as the Intelligent Driver Model (IDM) [65] are overly accommodating and reactive. We seek to evaluate and encourage the development of sim agents that lie in the middle ground, adhering to a definition of realism that implies matching the full distribution of human behavior.
To the best of our knowledge, to date there is no existing benchmark for evaluation of simulation agents. Benchmarks have spurred notable innovation in other areas related to autonomous driving research, especially for perception [6, 10, 24, 58], motion forecasting [6, 10, 22, 72, 78], and motion planning [19]. We believe a standardized benchmark can likewise spur dramatic improvements for simulation agent development. Among these benchmarks, those focused on motion forecasting are perhaps most similar to simulation, but all involve open-loop evaluation, which is clearly deficient compared to our closed-loop evaluation. Furthermore, we introduce realism metrics which are suitable to evaluating long-term futures. Relevant datasets such as the Waymo Open Motion Dataset (WOMD) [22] exist today that contain real-world agent behavior examples, and we build on top of WOMD to build WOSAC. In this challenge, we focus on a subset of the possible perception outputs, e.g., traffic light states or vehicle attributes are not modeled, but we leave this for future work.
The challenges our benchmark raises are unique, and if we can make real progress on it, we can show that we've solved one of the hard problems in self-driving. We have a number of open questions: Are there benefits to scene-centric, rather than agent-centric, simulation methods? What is the most useful generative modeling framework for the task? What degree of motion planning is needed for agent policies, and how far can marginal motion prediction take us? How can simulation methods be made more efficient? How can we design a benchmark and enforce various simulator properties? During our first iteration of the WOSAC challenge, user submissions have helped us answer a subset of these questions; for example, we observed that most methods found it most expedient to build upon state-of-the-art marginal motion prediction methods, i.e. operating in an agent-centric manner.
In this work, we describe in detail the Waymo Open Sim Agents Challenge (WOSAC) with the goal of stimulating interest in traffic simulation and world modeling. Our contributions are as follows:
- An evaluation framework for autoregressive traffic agents based on the approximate negative log likelihood they assign to logged data.
- An evaluation platform, an online leaderboard, available for submission at https:// waymo.com/open/challenges/2023/sim-agents/.
- An empirical evaluation and analysis of various baseline methods, as well as several external submissions.
Multi-Agent Traffic Simulation Simulators have been used to train and evaluate autonomous driving planners for several decades, dating back to ALVINN [43]. While simulators such as CARLA [19], SUMO [33], and Flow [73] provide only a heuristic driving policy for sim agents, they have still enabled progress in the AV motion planning domain [11, 12, 15]. Other recent simulators such as Nocturne [68] use a simplified world representation that consists of a fixed roadgraph and moving agent boxes.
Table 2: Existing evaluation methods for simulation agents. Entries are ordered chronologically by Arxiv timestamp. There is limited consensus in the literature regarding how multi-agent simulation should be evaluated.
Evaluation Protocol	ADE or minADE	Offroad Rate	Collision Rate	Instance-Level Distribution Matching	Dataset-Level Distribution Matching	Spatial Coverage or Diversity	Goal progress or Completion
ConvSocialPool [17]	✓	✓
Trajectory [29]	✓	✓
PRECOG [48]	✓	✓
BARK [4]	✓	✓	✓
SMARTS [82]	✓	✓
TrafficSim [60]	✓	✓	✓	✓
SimNet [3]	✓	✓
Symphony [28]	✓	✓	✓	✓
Nocturne [68]	✓	✓	✓
BITS [74]	✓	✓	✓	✓
InterSim [59]	✓	✓	✓
MetaDrive [34]	✓	✓	✓
TrafficBots [79]	✓	✓	✓
WOSAC (Ours)	✓	✓	✓
Simulation agent modeling is closely related to the problem of trajectory forecasting, as a sim agent could execute a set of trajectory predictions as its plan [4, 59]. However, as trajectory prediction methods are traditionally trained in open-loop, they have limited capability to recover from out of domain predictions encountered during closed-loop simulation [51]. In addition, few forecasting methods produce consistent joint future samples at the scene level [36, 48]. Sim agent modeling is also related to planning, as each sim agent could execute a replica of a planner independently [4]. However, each of these three tasks differ dramatically in objectives, outputs, and constraints (see Table 1).
Learned Sim Agents Learned sim agents in the literature differ widely in assumptions around policy coordination, dynamics model constraints, observability, as well as input modalities. While coordinated scene-centric agent behavior is studied in the open-loop motion forecasting domain [8, 9, 57], to the best of our knowledge, TrafficSim [60] is the only closed-loop, learned sim agent work to use a joint, scene-centric actor policy; all others operate in a decentralized manner without coordination [5, 28, 74], i.e., each agent in the scene is independently controlled by replicas of the same model using agent-centric inference. BITS and TrafficBots [74, 79] use a unicycle dynamics model and Nocturne uses a bicycle dynamics model [68] whereas most others do not specify any such constraint; others enforce partial observability constraints, such as Nocturne [68]. Other methods differ in the type of input, whether rasterized [3, 74] or provided in a vector format [28, 79]. Some works focus specifically on generating challenging scenarios [47], and others aim for user-based controllability [81]. Some are trained via pure imitation learning [3], while others include closed-loop adversarial losses [28, 60], or multi-agent RL [4, 34, 82] in order to learn to recover from its mistakes [51]. Some works such as InterSim [4, 28, 59, 74, 79, 82] use a goal-conditioned problem formulation, while others do not [3].
Evaluation of Generative Models Distribution matching has become a common way to evaluate generative models [18, 20, 27, 30, 31, 45, 46, 49, 52, 77], through the Fréchet Inception Distance (FID) [26]. Previous evaluation methods such as the Inception Score (IS) [53] reason over the entropy of conditional and unconditional distributions, but are not applicable in our case due to the multimodality of the simulation problem. The FID improves the Inception Score by using statistics of real world samples, measuring the difference between the generated distributions and a data distribution (in the simulation domain, the logged distribution). However, FID has limited sensitivity per example due to aggregation of statistics over entire test datasets into a single mean and covariance.
Evaluating Multi-Agent Simulation There is limited consensus in the literature regarding how multi-agent simulation should be evaluated (see Table 2), and no mainstream existing benchmark exists. Given the importance of safety, almost all existing sim agent works measure some form of collision rate [3, 28, 59, 60, 68, 74], and some multi-object joint trajectory forecasting methods also measure it via trajectory overlap [36]. However, collision rate can be artificially driven to zero by static policies, and thus cannot measure realism. Quantitative evaluation of realism requires comparison with logged data. Such evaluation methods vary widely, from distribution matching of vehicle dynamics [28, 74], to comparison of offroad rates [28, 60, 74], spatial coverage and diversity [60, 74], and progress to goal [59, 68]. However, as goals are not observable, they are thus difficult to extract reliably. Requiring direct reconstruction of logged data through metrics such as Average Displacement Error (ADE) has also been proposed [3, 68], but has limited effectiveness because
there are generally multiple realistic actions the AV or sim agents could take at any given moment. To overcome this limitation, one option is to allow the user to provide multiple possible trajectories per sim agent, such as TrafficSim, which uses a minimum average displacement error (minADE) over 15 simulations. [60].
Recently, generative-model based evaluation has become more popular in the simulation domain, primarily through distribution matching metrics. Symphony [28] uses Jensen-Shannon distances over trajectory curvature. NeuralNDE [75] compares distributions of vehicle speed and inter-vehicle distance, whereas BITS [74] utilizes Wasserstein distances on agent scene occupancy using multiple rollouts per scene, along with Wasserstein distances between simulated and logged speed and jerk - two kinematic features which can encapsulate passenger comfort. The latter are computed as a distribution-to-distribution comparison on a dataset level, however, this type of metric has shown limited sensitivity in our experiments.
Likelihood metrics An alternative distribution matching framework is to measure point-to-distribution distances. [17] and [29] introduce a metric defined as the average negative log likelihood (NLL) of the ground truth trajectory, as determined by a kernel density estimate (KDE) [42, 50] over output samples at the same prediction timestep. This metric has found some adoption [54, 64], and we primarily build off of this metric in our work. We note that likelihood-based generative models of simulation such as PRECOG [48] and MFP [63] directly produce likelihoods, meaning that the use of a KDE on sampled trajectories to estimate likelihoods is not needed for such model classes. Concurrent work [79] also measures the NLL of the GT scene under 6 rollouts.
3 Traffic Simulation as Conditional Generative Modeling
Our goal is to encourage the design of traffic simulators by defining a data-driven evaluation framework and instantiating it with publicly accessible data. We focus on simulating agent behavior in a setting in which an offboard perception system is treated as fixed and given.
Problem formulation. We formulate driving as a Hidden Markov Model  $\mathcal{H} = \left(S, \mathcal{O}, p(o_t|s_t), p(s_t|s_{t-1})\right)$ , where  $\mathcal{S}$  denotes the set of unobservable true world states,  $\mathcal{O}$  denotes the set of observations,  $p(o_t|s_t)$  denotes the sampleable emission distribution, and  $p(s_t|s_{t-1})$  denotes the hidden Markovian state dynamics: the probability of the hidden state transitioning from  $s_{t-1}$  at timestep  $t-1$  to  $s_t$  at time  $t$ . Each  $o_t \in O$  can be partitioned into AV- and environment-centric components that vary in time:  $o_t = [o_t^{\mathrm{AV}}, o_t^{\mathrm{env}}]$ .  $\mathcal{O}_t^{\mathrm{env}}$  can in general contain a rich set of features, but for the purpose of our challenge, it contains solely the poses of the non-AV agents. We denote the true observation dynamics as  $p^{\mathrm{world}}(o_t|s_{t-1}) \doteq \mathbb{E}_{p(s_t|s_{t-1})} p(o_t|s_t)$ .
The task. The task to build a "world model"  $q^{\mathrm{world}}(o_t | o_{<t}^c)$  of  $p^{\mathrm{world}}(o_t | s_{t-1})$ ,  $o_{<t}^c \doteq [o_{\mathrm{map}}, o_{\mathrm{signals}}, o_{-H-1}, \ldots, o_{t-1}]$ , i.e., it denotes a context of a static map observation, traffic signal observations, and the observation history, with history length  $H$ .
Task constraints:
1.  $q^{\mathrm{world}}$  must be autoregressive for  $T$  steps, i.e., sim agent models must adhere to a 10Hz resampling procedure, re-observing the updated scene and consuming their previous outputs.
2.  $q^{\mathrm{world}}$  must factorize according to Eq. 1:
$$
q ^ {\text {w o r l d}} \left(o _ {t} \mid o _ {<   t} ^ {c}\right) = \pi \left(o _ {t} ^ {\mathrm {A V}} \mid o _ {<   t} ^ {c}\right) q \left(o _ {t} ^ {\mathrm {e n v}} \mid o _ {<   t} ^ {c}\right), \tag {1}
$$
where  $q(o_{t}^{\mathrm{env}}|o_{<t}^{c})$  is a traffic simulator, and  $\pi (o_t^{\mathrm{AV}}|o_{<t}^c)$  is an AV policy<sup>1</sup>. Any submission that fails to satisfy both of these properties will not be considered on WOSAC leaderboards, as determined by
<sup>1</sup>We call this a policy because it is similar to the typical formulation of a policy in a decision process over actions, although not equivalent, because it is defined over next observations rather than current actions. It can be made equivalent to a standard policy  $\pi (a_{t - 1}^{\mathrm{AV}}|o_{< t}^{c})$  by defining the AV's action space  $\mathcal{A}$  to be equivalent to its component of the observation space, and defining an action-dependent world model  $q^{\mathrm{world}}(o_t|o_{< t}^c,a_{t - 1}^{\mathrm{AV}})\doteq \delta (o_t^{\mathrm{AV}} = a_{t - 1}^{\mathrm{AV}})q(o_t^{\mathrm{env}}|o_{< t}^c)$ , where  $\delta$  denotes the Dirac delta function.
Figure 2: Graphical model of required factorization as a Bayes net: the two distributions from Eq. 1 are autoregressively interleaved: one represents the AV's "policy"  $\pi (o_t^{\mathrm{AV}}|o_{< t}^c)$ , and another represents the environmental dynamics  $q(o_{t}^{\mathrm{env}}|o_{< t}^{c})$ ; the graphical model represent  $T$ -steps of applying these two distributions. Thick outgoing arrows denote passing inputs from all parent nodes to all children.
challenge submission reports. Requiring them to be generative enables sampling from an arbitrary traffic simulator-AV policy pair. These two properties imply the probabilistic graphical model shown in Fig. 2, modified from Fig. 9 of [48]. Algorithms 1 and 2 illustrate valid and invalid submissions.
Much of the challenge of modeling  $p^{\mathrm{world}}$  lies in the fact that in many situations  $s_{t-1} \in S$ ,  $p^{\mathrm{world}}$  assigns density to multiple outcomes due to uncertainty from agents in the scene, which means that both  $\pi(o_t^{\mathrm{AV}}|o_{<t}^c)$  and  $q(o_t^{\mathrm{env}}|o_{<t}^c)$  often must contain multiple modes in order to perform well. We evaluate distribution-matching of  $p^{\mathrm{world}}$  relative to a dataset of logged outcomes. The required factorization into a AV observation-space policy and environment observation dynamics,  $q^{\mathrm{world}}(o_t|o_{<t}^c) = \pi(o_t^{\mathrm{AV}}|o_{<t}^c)q(o_t^{\mathrm{env}}|o_{<t}^c)$ , is fairly flexible. We are agnostic to their particular structures. One noteworthy choice for the environment observation dynamics is a "multi-agent" factorization, in which  $q(o_t^{\mathrm{env}}|o_{<t}^c) = \prod_{a=1}^{A}\pi_a(o_t^{\mathrm{env},a}|o_{<t}^c)$ , i.e., the environment observation dynamics factorizes into a sequence of  $A$  observation-space policies, and the environment observation itself is partitioned into  $A$  different components, one for each agent:  $o_t^{\mathrm{env}} = [o_t^{\mathrm{env},1},\dots,o_t^{\mathrm{env},A}]$ .
Algorithm 1 Valid: Factorized, Closed-Loop, Agent-Centric Simulation
Input: Map  $o^{\mathrm{map}}$  and traffic signals  $o^{\mathrm{signals}}$ . Initial actor states  $o_{-H - 1:0} = \{o_{-H - 1},\dots ,o_0\}$  where each  $o_t^{\mathrm{env}} = \{o_t^{\mathrm{env,1}},\ldots ,o_t^{\mathrm{env},A}\}$  for the  $A$  actors in the scene.
Output: Simulated observations  $o_{1:T} = \{o_1, o_2, \dots, o_T\}$  for  $T$  simulation timesteps.
1: for  $t = 1, \dots, T$  do
$\triangleright$  Simulate for requested number of timesteps
2:  $o_{t}^{\mathrm{AV}} \gets \pi_{AV}(o_{< t}; o^{\mathrm{map}}, o^{\mathrm{signals}})$
3: for  $a = 1,\dots,A$  do
$\triangleright$  Produce next state for each actor at each timestep
4:  $o_t^{\mathrm{env},a}\gets \pi_a(o_{< t};o^{\mathrm{map}},o^{\mathrm{signals}})$
5:  $o_{t} = \{o_{t}^{\mathrm{env},a}:\forall a\in 1\dots A\} \cup \{o_{t}^{\mathrm{AV}}\}$
6: return  $o_{1:T} = \{o_1, o_2, \dots, o_T\}$
Algorithm 2 Invalid: Factorized, Open-Loop, Agent-Centric Simulation
1:  $o_{1:T}^{\mathrm{AV}} \gets \pi_{AV}(o_{< 1}; o^{\mathrm{map}}, o^{\mathrm{signals}})$
2: for  $a = 1, \dots, A$  do
$\triangleright$  Produce states at all future timesteps for each actor
3:  $o_{1:T}^{\mathrm{env},a}\gets \pi_a(o_{< 1};o^{\mathrm{map}},o^{\mathrm{signals}})$
4: return  $o_{1:T} = \{o_{1:T}^{\mathrm{env},a} : \forall a \in 1 \dots A\} \cup \{o_{1:T}^{\mathrm{AV}}\}$
4 Benchmark Overview
For WOSAC, we use the test data from the v1.2.0 release of the Waymo Open Motion Dataset (WOMD) [22]. We treat WOMD as a set  $\mathcal{D}$  of scenarios where each scenario is a history-future
pair  $(o_{-H - 1:0}, o_{\geq 1})$ . This dataset offers a large quantity of high-fidelity object behaviors and shapes produced by a state-of-the-art offboard perception system. We use WOMD's 9 second  $10\mathrm{Hz}$  sequences (comprising  $H = 11$  observations from 1.1 seconds of history and 80 observations from 8 seconds of future data), which contain object tracks at  $10\mathrm{Hz}$  and map data for the area covered by the sequence. Across the dataset splits, there exists 486,995 scenarios in train, 44,097 in validation, and 44,920 in test. These 9.1 second windows have been sampled with varying overlap from 103,354 mined segments of 20 second duration. Up to 128 agents (one of which must represent the AV) must be simulated in each scenario for the 8 second future (comprising 80 steps of simulation).
Agent Definition We require simulation of all agents that have valid measurements at time  $t = 0$ , i.e. the last step of logged initial conditions before simulation begins. Because the test split data is sequestered, users will not have access to objects that appear after the time of handover, and so therefore could not be expected to simulate them. We require simulation of all three WOMD object types (vehicles, cyclists, and pedestrians). Objects' dimensions stay fixed as per the last step of history (while they do change in the original data).
Submission We do not enforce any motion model (also because we have multiple agent types), which means users need to directly report  $x / y / z$  centroid coordinates and heading of the objects' boxes (which could be generated directly or through an appropriate motion model). See the Appendix for additional information on the submission format.
By allowing users to produce the simulations themselves, we reduce the burden on the user by avoiding the need to submit containerized software for an evaluation server.
Agents should generate realistic driving scenarios stochastically. We define "realistic agents" as those that match the actual distribution of scenarios observed during real-world driving. Unfortunately, we do not know the analytic form of the distribution, but we do have samples from it: the examples that make up WOMD. We therefore evaluate submissions using the approximate negative log likelihood (NLL) of real world samples under the distribution induced by the agents.
The NLL we wish to minimize is given by:
$$
\mathrm {N L L} ^ {*} = - \frac {1}{| \mathcal {D} |} \sum_ {i = 1} ^ {| \mathcal {D} |} \log q ^ {\text {w o r l d}} \left(o _ {\geq 1, i} \mid o _ {<   1, i}\right) \tag {2}
$$
However, there are two problems with trying to minimize Equation 2 exactly in our problem setting. First,  $o_{\geq 1}$  is high-dimensional. Instead of trying to parameterize the entire ground truth scenario and compute its NLL under a simulated distribution, we therefore parameterize scenarios with a smaller number of component metrics (see Section 4.2.1) and aggregate them together into a composite NLL metric (see Section 4.2.2). Second, agents may support sampling but not pointwise likelihood estimation [41]. In fact, we only require challenge entrants to submit samples from their agents, and therefore have no way of knowing the exact likelihood of logged scenarios under different agent submissions. To avoid this problem, we standardize the NLL computation by fitting histograms to the 32 submitted samples of agent futures, and compute NLLs under the categorical distribution induced by normalizing the histograms.
4.2.1 Component Metrics
Breaking NLL* into component metrics has a few benefits. It mitigates the curse of dimensionality described in Section 4.2. It also adds more interpretability to the evaluation, allowing researchers to trade off between different types of errors.