SlowGuess/NeurIPS_2023_OCR huggingface dataset:
by MTR, and prune the exponential number of futures in a greedy fashion. As brute-force exhaustive search over the  $6^{N}$  combinations is computationally infeasible,  $\mathrm{MTR} + + +$  searches for the densest subgraph in a graph of non-colliding future trajectories.
First, a  $6N$  by  $6N$  distance matrix  $D$  is constructed, where entry  $D_{6m + i - 1,6n + j - 1}$  indicates the minimum L2 distance between the  $i$ th-highest trajectories of agent  $m$  and the  $j$ th-highest one of agent  $n$ . Second the distance matrix is binarized by evaluating which distances correspond to collisions according to object extents. Finally, a clique-finding heuristic method finds a dense subgraph of size  $N$ . An ensemble of 32 fine-tuned MTR models is employed to create the 32 rollouts, each model producing a single rollout.
Collision Avoidance Detour (CAD) [14]: An open-loop method that builds upon an existing motion forecasting method, MTR [55, 56] to produce marginal trajectory predictions, and resamples the entire future if future agent collisions are anticipated, until a maximum number of trials is exhausted. While CAD adheres to the factorization requirement, it does not adhere to WOSAC's closed-loop requirement. Factorization of world vs. AV policies is accomplished by using different checkpoints of an MTR motion prediction model for the two agent groups, and motion of non-evaluated agents is simulated using a constant velocity model.
Joint-Multipath++ [70]: An open-loop, scene-centric method that builds off of MultiPath++ [66], producing in a single model pass 32 rollouts, each representing an entire length-80 trajectory. JointMultiPath++ does not factorize AV vs. world policies, and thus does not fully adhere to WOSAC"s policy factorization requirements. Agent history information (positions and headings of all agents) are transformed into the AV's coordinate frame, while closest lane information is selected for each agent. In its encoder, JointMultiPath++ concatenates the output of 2 LSTMs and one MultiContextGating (MCG) [66] block to form per-agent embeddings; one LSTM is used to encode agent history, another LSTM is used to encode per-step differences in agent history, and an MCG block is used to encode agent history with corresponding timesteps. Subsequent MCG blocks fuse per-agent information with road network (polyline) embeddings. In its decoder, a series of MLP blocks transform  $n$  per-agent embeddings to rollouts represented as  $n \times 32 \times 80 \times 3$  output tensors.
SBTA-ADIA Mo et al. [38]: Builds upon an existing motion prediction method [39]. This hierarchical method splits the problem into a first phase of multi-agent goal prediction, based on a GNN scene encoder, and a simple planning policy which tries to accomplish the goal closed-loop.
A.5 Additional Qualitative Results from External Challenge Submissions
In Figure 6 and 7, we provide additional qualitative examples of simulation results from external challenge submissions.
Simulation Input
LogPlayback (Logged Oracle)
Figure 6: Two-dimensional visualization of simulation results on WOMD's test split. MVTE exhibits a collision and MTR++ produces a near-miss. 'Simulation input' represents the context history  $o_{-H - 1:0}$ , whereas all other columns visualize both ( $o_{-H - 1:0}, o_{\geq 1}$ ). One possible future for a single scene is represented, selected from 32 submitted rollouts, where the AV remains stopped at a red traffic signal. Each rendering in columns 2,3 and 4 depicts the entire duration of the scene. Trajectories of environment sim agents are drawn in a green-blue gradient (each as a sequence of circles in a temporal color gradient). The AV agent is drawn in orange.
MTR+++ [44]
MVTE [71]
$^{2}$ Code available at https://github.com/wangwenxi-handsome/Joint-Multipathpp.
Figure 7: Two-dimensional visualization of simulation results on a single scene from WOMD's test split using various baseline methods. Four possible futures for this single scene are represented (one per row), selected from 32 submitted rollouts. 'Simulation input' represents the context history  $o_{-H - 1:0}$ , whereas all other columns visualize both  $(o_{-H - 1:0}, o_{\geq 1})$ . Each rendering in the second, third, and fourth columns depicts the entire duration of the scene. Trajectories of environment sim agents are drawn in a green-blue gradient, and trajectories of the AV agent are drawn in a red-yellow gradient (each as a sequence of circles in a temporal color gradient). The AV agent is drawn in orange.
A.6 Component Metrics Implementation Details
1. Linear Speed: Unsigned magnitude of the first derivative  $\| \mathbf{v}\| = \| \frac{\mathbf{x}_{t + 1} - \mathbf{x}_t}{\Delta t}\| _2$  where  $\mathbf{x}_t = [x_t,y_t,z_t]$ , Linear speed in 3D computed as the 1-step difference between 3D trajectory points. We employ speed, rather than velocity, as velocity can either be defined w.r.t. the ego-agent's heading, or w.r.t. a global coordinate system, where velocity directions may be city-specific, based on orientation of roads w.r.t. North. Although this cannot capture objects moving in reverse, a rare behavior, we omit it for sake of simplicity.
2. Linear Acceleration Magnitude: Signed magnitude of second derivative, in 3D computed as the 1-step difference between speeds of objects.  $\frac{\|\mathbf{v}_{t + 1}\| - \|\mathbf{v}_t\|}{\Delta t}$ .
3. Angular Speed: Signed first derivative  $\omega = \frac{d(\theta_{t + 1},\theta_t)}{\Delta t}$ , computed as the 1-step difference in heading, where  $d(\cdot)$  represents the minimal angular difference between two angles on the unit circle, i.e.,  $d(\cdot)$  is a distance metric on  $SO(2)$  computed as  $\min \{| \theta_{t + 1} - \theta_t|, 2\pi - | \theta_{t + 1} - \theta_t | \}$  with  $\theta_t, \theta_{t + 1} \in [0,2\pi)$ .
4. Angular Acceleration Magnitude: Second derivative, computed as the 1-step difference in angular speed  $\omega$ , as  $\frac{d(\omega_{t+1}, \omega_t)}{\Delta t}$ .
5. Distance to nearest object: Signed distance (in meters) to the nearest object in the scene. We use Minkowski difference of box polygons, according to a simplified version of the Gilbert-Johnson-Keerthi (GJK) distance algorithm [25].
6. Collisions: Count indicating objects that collided, at any point in time, with any other object, i.e. when the signed distances to nearest objects, as described above, achieves a negative value.
7. Time-to-collision (TTC): Time (in seconds) before the object collides with the object it is following (if one exists), assuming constant speeds. An object is defined as exhibiting object-following (tailgating) behavior based on alignment conditions derived from heading and lateral distance.
8. Distance to road edge: Signed distance (in meters) to the nearest road edge in the scene.
9. Road departures: Boolean value indicating whether the object went off the road, at any point in time [60].
To prevent undefined scores from histogram bins with zero support, we employ Laplace smoothing with a pseudocount of 0.1.
Inserted and Deleted Object Handling In order to prevent object insertion/deletion bias between the logged and simulated data distributions during evaluation, we discard any newly spawned objects (appearing after the history interval) in the logged test set when computing the logged data distribution. The data distribution in the WOMD dataset already includes such object insertion and deletion.
A.6.1 Evaluation Source Code References
In this section, we provide pointers to our specific implementations of the 9 metrics discussed in Section 4.2.1 of the main paper:
- Kinematic-based features: Linear speed, linear acceleration magnitude, angular speed, and angular acceleration magnitude (metrics 1,2,3,4): [Code]
- Interaction-based features: TTC and distance to nearest object (metrics 5, 6, 7): [Code] and modified GJK algorithm implementation [Code]
- Map-based features: Road departures and distance to road edge (metrics 8, 9): [Code]
An implementation of our time-series based NLL computation can be found here: [Code].
A.6.2 Evaluation Code License and Dependencies
The WOMD [22] dataset itself is licensed under a non-commercial license (www.waymo.com/open/terms) and the evaluation code for our Waymo Open Sim Agents Challenge (WOSAC) is released under a BSD+limited patent license. See
https://github.com/waymo-research/waymo-open-dataset/blob/master/src/waymo_open_dataset/wdl_<sup>1</sup> limited/sim_agent。<sup>2</sup> metrics/PATENTS and https://github.com/waymo-research/waymo-open-dataset/blob/master/src/waymo_open_dataset/wdl_<sup>3</sup> limited/sim_agent。<sup>4</sup> metrics/License.
Dependencies used include NumPy (numpy), the Waymo Open Dataset repository (waymo-open-dataset-tf-2-11-0==1.5.2), TensorFlow (tensorflow), TensorFlow Probability (tensorflow(probability), matplotlib (matplotlib), TQDM (tqdm), Protocol Buffers (google.proto), and Python standard library imports (os, tarfile, dataclasses).
A.7 Additional Information about WOMD Splits Used
We exclude 401 run segments from evaluation due to discrepancies in object counts across the Scenario proto and tf.Example formats, due to object count truncation arising from fixed-shape tf.Example tensors with exactly 128 object slots. We also exclude from evaluation 9 test run segments which are missing maps (however, maps are present for each scenario present in the validation set).
Table 5: Statistics of WOMD dataset [22] splits used.
DATASET SPLIT
VALIDATION	TESTING
ALL SCENARIO COUNTS	44097	44920
EVALUATED SCENARIO COUNTS	43696	44520
A.8 Submission Format
Submissions must be uploaded as serialized SimAgentsChallengeSubmission protocol buffer data $^4$  ("protos"). Each ScenarioRollouts proto within the submission must contain 32 8-second rollouts of simulation data from one scenario. A validation or test set submission may be submitted to the evaluation server.
We provide a Jupyter notebook tutorial with additional instructions and examples on how to generate a submission for a dataset split. We recommend storing multiple ScenarioRollout's in each binary proto file (i.e. in each SimAgentsChallengeSubmission file) to prevent creating a tar.gz file with tens of thousands of files; use of 100 to 150 of such shards is recommended. Please refer to the tutorial notebook for the naming convention of these files. Submission data should be compressed as a single .tar.gz archive and uploaded as a single file.
3WOMD download instructions available at https://waymo.com/intl/en_us/open/download. 4https://protobuf.dev/