SlowGuess/NeurIPS_2023_OCR huggingface dataset:
nuPlan. Though nuPlan [6] has more than 1500 hours of driving data, we only use those recorded in Boston as we want to keep all databases in the experiments to have similar sizes. After filtering out scenarios whose ego car moves  $< 10$  meters, we finally collect 50,261 20-second scenarios.
PG. PG stands for procedural generation which is used to generate infinite maps and traffic according to a set of pre-defined roadblocks and traffic initialization rules. It is the default scenario generation method of MetaDrive simulator [30]. We set the traffic density to 15 vehicles per 100 meters and 2 roadblocks in each scenario to keep scenario length consistent with real-world scenarios. We finally collected 50,000 scenarios to form the PG database.
Table 2 provides statistics about the three datasets. Track length stands for the average moving distance of the ego car across all scenarios, and PG scenarios have the longest moving distance. Waymo scenarios are relatively more complex compared to others, as they have more vehicles and more intersection scenarios, which is reflected in Fig. 4 as well. The Construction Ratio represents the ratio of scenarios containing traffic cones and barriers out of all scenarios. nuPlan-boston database has traffic cones and barriers in all scenarios, while no objects are contained in the Waymo database. More database construction details are in Appendix F.
4.2 Visualization of Scenario Embeddings with Scenario Generation Model
To reveal the differences and gaps across three databases, we use the internal embedding from a scene generation model, TrafficGen [19], to visualize the scenario snapshots from all three databases. The state initializer in TrafficGen utilizes an encoder-decoder structure that encodes the traffic scenario into a latent space and then decodes the initial states distribution of traffic participants, enabling the sampling and generation of a new scenario snapshot. We randomly select 1000 scenarios from each database, feed them into TrafficGen to obtain the embeddings of these scenarios and then employ t-SNE [52] to visualize them in Fig. 4. We can see there is a noticeable domain gap between synthetic and real-world data. Only two synthetic intersection scenarios are close to real-world scenarios and bridge the synthetic scenarios and real scenarios. Additionally, even among real-world scenarios, a minor distribution gap remains, primarily due to different road network topologies in different regions where the data is collected. These findings suggest that solely relying on driving data from one
Figure 4: t-SNE visualization of the feature embeddings of scenarios from three databases. Scenarios from nuPlan and Waymo datasets are much more similar, while synthetic PG data has a clear domain gap with both real-world datasets. We also plot the representative scenarios in clusters. As shown in the left-side figures  $(\square)$ , synthetic scenarios can be neatly categorized into distinct clusters, while real-world scenarios shown on the right side are too complex to identify clear boundaries. Nevertheless, the right-side figures show scenarios from nuPlan  $(\square)$  have simpler map structures and mild traffic, compared to the Waymo scenarios  $(\square)$ . In addition, there are two outliers that confuse the model, causing it to cluster them with real-world scenarios. The visualization of the two synthetic scenarios indicates that the intersection scenario can bridge the sim-to-real gap.
region or one database is insufficient to cover all potential traffic situations. Therefore, it is important to aggregate as many scenarios as possible. Additional showcases from different databases, visualizations of generated scenarios, and training details can be found in Appendix A.
4.3 Cross-dataset RL Generalization
The previous experiment reveals that the domain gap exists between synthetic scenarios and real-world scenarios. We would like to further investigate whether the domain gap affects driving policy learning. To this end, we build a data-driven RL environment where the agent should arrive at the destination of the recorded trajectories in time. Traffic vehicles behind the ego car are controlled by IDMPolicy to achieve closed-loop training and testing. We further split training and test sets for the nuPlan database and the PG database, respectively, and get 4 splits: nuPlan-train, PG-train, nuPlan-test and PG-test.
Each training set contains 40,000 scenarios, while each test set has 5,000 non-overlapping scenarios. The scenarios of two training splits are sorted according to the difficult score, a metric considering length and curvature of trajectory, and constitutes 100 levels for conducting the curriculum training. Each level contains 400 scenarios and a  $75\%$  Success Rate, the ratio of scenarios where the agent can reach the destination out of 400 scenarios, is required for the learning environment to level up. We train PPO [43] agents on PG-train and nuPlan-train splits with the curriculum training and evaluate them on nuPlan-test split to study whether the domain gap harms the performance for the learning-based controller. All re
sults are averaged across 5 random seeds and the error bar shows the standard deviation. As shown in Fig. 5, when testing on held-out nuPlan-test set, the agents trained with nuPlan-train split outperform those trained with PG-train split in terms of Success Rate. This is because the agents trained in synthetic scenarios fail to learn high-speed driving skills and thus can not reach the destination in time, inducing a high Timeout Rate. Thus it is necessary to train learning-based controllers with real-world data to close the sim-to-real gap. We also ablate the curriculum training and the results
Figure 5: Evaluation on nuPlan-test split.
Route Completion	Success Rate	Reward	Cost	Average Distance	Final Distance
TD3	0.669 (±0.018)	0.513 (±0.028)	74.87 (±5.90)	2.01 (±0.24)	12.64 (±0.72)	45.13 (±2.73)
PPO	0.740 (±0.037)	0.579 (±0.054)	85.28 (±8.96)	2.83 (±0.53)	12.79 (±0.73)	36.71 (±6.38)
CoPO	0.745 (±0.024)	0.570 (±0.043)	85.94 (±4.60)	2.88 (±0.32)	12.86 (±0.63)	34.75 (±3.78)
AIRL (Disc.)	0.367 (±0.027)	0.153 (±0.028)	-	0.55 (±0.12)	17.73 (±1.87)	80.57 (±5.26)
AIRL (Cont.)	0.138 (±0.016)	0.015 (±0.009)	-	0.42 (±0.16)	3.63 (±1.07)	111.23 (±7.48)
GAIL (Disc.)	0.487 (±0.021)	0.218 (±0.018)	-	1.64 (±0.35)	28.48 (±2.23)	68.53 (±3.57)
GAIL (Cont.)	0.421 (±0.043)	0.189 (±0.051)	-	1.56 (±0.52)	27.33 (±4.72)	75.98 (±9.02)
Table 3: The multi-agent reinforcement learning and imitation learning results on the Waymo dataset.
suggest that it is critical to adjust the scenario difficulty according to the agent performance in the course of training. Finally, we build a new dataset containing 40,000 scenarios by combining half of the PG-train split and half of the nuPlan-train split. Figure. 5 shows that agents trained on these combined datasets achieve the highest Success Rate when evaluated on nuPlan-test split. This is because almost every synthetic scenario contains bends and curved trajectories, which only exist in approximately  $20\%$  of real-world scenarios. As a result, agents acquire a better ability to follow a curved trajectory, achieving the lowest Out of Road Rate. Thus aggregating synthetic scenarios to the training dataset can benefit policy learning as they improve the diversity of training data. More details of the experiments are in Appendix B.
4.4 Multi-agent Cooperative Learning
ScenarioNet can load scenario data into an interactive multi-agent policy learning environment and generate local sensory observations for each agent. Therefore, ScenarioNet enables the critical research on multi-agent reinforcement learning as the agents are self-interested and the number is varying, wherein cooperative MARL methods can not be applied [39], as well as the research on multi-agent imitation learning with the challenges of learning from observation or say the action-free imitation learning, meaning that the ground-truth trajectory does not contain action but only the sequence of states (observations). ScenarioNet bridges the research of multi-agent decision-making with vast volume of real-world data. We load the ground-truth (GT) trajectory in the dataset and use it to form the termination condition, the reward function, and the evaluation metrics. For multi-agent RL, we use the GT trajectory to define the displacement reward:  $r_t^i =$  the displacement of agent  $i$  at step  $t$  projected into its underlying ground-truth trajectory. We train independent PPO [43] and TD3 [23] agents as well as the Coordinated Policy Optimization (CoPO) agents [39]. For multi-agent IL, to address the action-free issue, we use GAIL [25] in multi-agent setting [46] but the discriminator distinguishes state next state pair, instead of state-action pair [50]. We also train multi-agent Adversarial Inverse RL [65] (AIRL) with an additional inverse dynamics model for estimating the expert actions. The inverse dynamics model is trained concurrently with the AIRL policies and learns the action given state next state pairs from the environment interactions. The ground-truth trajectory can be used to measure the learned behaviors, such as the route completion rate, the ratio between the length of projected agent trajectory and the length of GT trajectory as well as the average and final distance between agent trajectory with GT trajectory. The cost is the number of crashes of an agent in one episode. As shown in Table 3, we find that the displacement reward is strong supervision to learn multi-agent behavior as the RL can explore and exploit the reward function by interacting with the environment, without learning a discriminator as in GAIL or a reward function and an inverse dynamics model in AIRL. More details can be found in Appendix C.
4.5 AD Stack Testing
We released ROS bridge [33], allowing connecting ScenarioNet with open-sourced AD stacks like autoware [20] and planning algorithms developed by the ROS community. As a result, researchers can study self-driving systems in real-world scenarios. In our experiment, we test Openpilot [10], an end-to-end AD system taking only visual information as input. As the navigation module of Openpilot is still a beta version, the test happens in scenarios without diverged roads. As shown in Fig. 6, the Openpilot system is robust enough to operate in common scenarios like lane-keeping and stopping at traffic lights. More scenario-based test videos and details are available in Appendix E.2.
Figure 6: Openpilot can operate in real-world scenarios reconstructed with ScenarioNet. It can cruise on urban roads and smoothly stop at crowded intersections.
This work presents an open-source platform called ScenarioNet for simulating and modeling driving scenarios. A massive number of real-world traffic scenarios extracted from various datasets can be used for testing AD stacks, generating scenarios, and learning both single-agent and multi-agent policies. We hope it can open up many new opportunities for AI and autonomous driving communities.
Limitation & Potential Negative Societal Impacts. One limitation of ScenarioNet at the current development stage is the lack of diverse 3D assets. Currently, the assets are collected from the internet, which may produce unrealistic raw sensor data output. Future work will be to reconstruct the mesh and textures of vehicles, other objects, and background models like streets from driving logs using neural rendering. A detailed plan for improving visual fidelity is in Appendix G. In addition, several potential technical drawbacks exist in ScenarioNet. One drawback is that the simulation is single-thread due to the Python GIL and runs on CPU, while some latest simulators like IsaacGym [34] support GPU-based parallel simulation and thus outperform ScenarioNet in terms of sample efficiency. This problem can be alleviated by using multi-process tools like Ray/Rlib [37, 31] for parallel simulation. Another drawback is that ScenarioNet doesn't support differential simulation which allows training policy with the derivative of reward function directly [36]. In the future, ScenarioNet can address this by introducing a bicycle model for vehicle dynamics simulation, making the system differentiable. One potential negative societal impact is that there is no guarantee of safety for even a data-driven AD stack can pass all the testing scenarios in ScenarioNet. There is also a domain gap for the trained model to be deployed in the real world.
Licenses. ScenarioNet is released under Apache License 2.0. Panda3D, used in MetaDrive for 3D rendering, is under the Modified BSD license. Bullet engine is under the Zlib license. The vehicle models are collected from Sketchfab under CC BY 4.0 or CC BY-NC 4.0. For real-world data used in experiments, Waymo Open Dataset [32, 47] is under license terms available at https://waymo.com/open. Besides, the L5 dataset, nuPlan dataset, nuScenes dataset, and Argoverse dataset are provided free of charge under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public license. The toolkits for parsing these datasets are all under Apache license version 2.0. The Openpilot system used for AD stack testing is under MIT license.
Acknowledgement: This work was supported by the National Science Foundation under Grant No. 2235012 and the Samsung Global Collaboration Award.
In this appendix, we provide more details about the four experiments and some scenario examples from the three databases used in the experiments. Section A is on the scenario generation model; Section B is on the single-agent cross-dataset generalization experiment; Section C is on the multiagent reinforcement learning and imitation learning; Section E shows the interface of ROS bridge and the qualitative results of running Openpilot test. Section F shows more rendered scenario examples from each database.
Codebase, documentation, videos, and a scenario gallery are available at https://metadriverse.github.io/scenarioonet.
A Scenario Generation Model
We use all the databases (nuPlan, Waymo, PG) in scenarioNet for cross-dataset traffic scenario generation experiments. The goal is to show that scenarioNet contains diverse traffic scenario data for training deep neural networks. We conduct our experiments based on TrafficGen [19], a neural generative model previously developed for reactive traffic scenario generation.
A.1 Task Definition
A traffic scenario is denoted as  $\tau = (\mathbf{m},\mathbf{s}_{1:T})$ , which lasts  $T$  time steps and contains the High-Definition (HD) road map  $\mathbf{m}$  and the state series of traffic vehicles  $\mathbf{s}_{1:T} = [s_1,\dots,s_T]$ . Each element  $s_t = \{s_t^1,\ldots s_t^N\}$  is a set of states of  $N$  traffic vehicles at time step  $t$ . Given an existing scenario  $\tau = (\mathbf{m},\mathbf{s}_{1:T})$ , the goal of TrafficGen is to learn to generate new traffic scenarios  $\tau' = (\mathbf{m},\mathbf{s}_{1:T}')$  that have similar distribution with  $\tau$  and different states  $\mathbf{s}'$  and longer time steps  $T'$ . After training, TrafficGen takes  $\mathbf{m}$  as input and generates  $\tau'$  as a totally different scenario.
A.2 Model Architecture
Traffic Scenario Encoder. TrafficGen uses vectorization to encode map and vehicle information, representing lanes as sets of vectors, each vector representing a small region. A vector-based coordinate system is established for each small region, with each vector comprising a start point  $p_i^s$ , endpoint  $p_i^e$ , and information about vehicles in this region. Thus, a traffic snapshot  $\tau$  at time step  $t$  is represented as  $\tau_t = \mathbf{v} = v_{i=1}^I$ . Cross attention mechanism is applied to the unordered set  $\mathbf{v}$  to fuse information from different regions into a single context vector.
Decoder for Scenario Generation. TrafficGen places vehicles by generating a set of weights for all regions, which is then turned into a categorical distribution. The local position of a tentative vehicle is modeled by a mixture of  $K$  bivariate normal distributions, as are heading, speed, and size of the vehicle. Autoregressive sampling is used to create a traffic snapshot. A motion forecasting model is used as the trajectory generator.
A.3 Experiment Setting
We train a scenario generation model TrafficGen with mixed data. Specifically, we randomly sample  $30\%$  data from nuPlan, Waymo, and PG. We filter out the scenarios with less than 8 cars and crop a rectangular area with a side length of  $120\mathrm{m}$  centered on the ego vehicle. Each of the 20s scenarios is split into 10 traffic snapshots with 2s intervals. The training is executed on servers with  $8\times$  Nvidia 2080TI and  $256\mathrm{G}$  memory in Distributed Data-Parallel (ddp) mode. We set the feature size to be 1024, and use 3-layer MLPs with hidden dimensions of [1024, 512, 256] for attribute modeling. The training takes 16 hours for 100 training epochs and the learning rate decays by  $20\%$  at every 30 epochs. The detailed hyperparameters are shown in Table 4.
We plot some generated traffic scenarios in Fig. 7 and the dynamics in Fig. 8. It shows that the models trained on the processed scenario data from ScenarioNet can generate realistic and diverse traffic behaviors and interactions.
Figure 7: Traffic scenarios generated from the model trained on different databases: Waymo (□), nuPlan (□), and PG (□).
Figure 8: Dynamics of the generated traffic scenarios. The first column is the original case. The middle columns show the generated scenarios at different timesteps. The last column shows the corresponding scenarios imported in the simulation. The green and red dashed lines indicate the traffic light status of this lane at the intersection.
Figure 9: t-SNE visualizations of 3000 scenarios. PG scenarios (□) are located mostly on the left side, while the scenarios from nuPlan (□) and the Waymo scenarios (□) are scattered on the right side.
A.4 t-SNE Visualizations
The trained model's encoder is used to extract feature embeddings of a given scenario sample. The embeddings are then visualized with t-SNE method to show the similarities and differences. The detailed hyperparameters of t-SNE are shown in Table 5.
The t-SNE result is plotted in Fig. 9. The clustering results show that there is a large domain gap between real-world scenarios (Waymo and nuPlan) and synthetic scenarios (PG). Besides, domain gap exists even in real-world datasets. One prominent feature of the Waymo dataset is the complex crossroad (shown in the lower right corner), which the nuPlan dataset is lacking. The t-SNE visualization reveals the differences between different traffic datasets.
Table 4: TrafficGen
Hyper-parameter	Value
Batch Size	256
Feature Size	1024
Training epochs	100
Learning Rate	3e-4
Activation Function	“relu”
MCG Layers	5
Table 5: t-SNE
Hyper-parameter	Value
Components Number	2
Init	pca
Learning Rate	auto
Perplexity	30
Early Exaggeration	12
B Single-agent Cross-dataset Generalization Experiment
We use the PG database and nuPlan database for cross-dataset generalization experiments. The goal is to investigate how the sim-to-real gap affects the generalizability of the learning-based vehicle controller. To this end, we train agents on synthetic PG scenarios [30] and nuPlan [6] scenarios respectively, and test them on the same held-out real-world test set.
B.1 Task Setup
To be specific, the task is to follow the trajectory of the data collection car and drive as fast as possible while avoiding collisions.
Observation. The observation of the RL agents is as follows:
1. A 120-dimensional vector denoting the Lidar-like point clouds with  $50m$  maximum detecting distance centering at the target vehicle. Each entry is in  $[0,1]$  with Gaussian noise and represents the relative distance of the nearest obstacle in the specified direction.
2. A vector containing the data that summarizes the target vehicle's state such as the steering, heading, velocity, and relative distance to the trajectory to follow.
3. The navigation information that guides the target vehicle toward the destination. Concretely, it consists of 10 points sampled on the future trajectory and the distance between two consecutive points is  $2m$ . The points will be projected to the vehicle coordinates.
4. A 12-dimensional vector denoting the Lidar-like point clouds with  $50m$  maximum detecting the boundary of the drivable area, like the solid lines or sidewalks. (Optional)
As the vehicle for collecting nuPlan data in Boston sometimes drives out of the drivable area for bypassing the cones or barriers, crossing the drivable area boundary usually happens. Therefore, we didn't use the boundary detector in our experiments, and crossing the boundaries like solid lines won't terminate the episode nor penalize the agent.
Action. The driving policy is a fully end-to-end model and directly controls the low-level throttle and steering angle. The action  $a$  is a continuous two-dimensional vector with entries in  $[-1, 1]$ . By multiplying coefficients and clipping the extreme value, the action will be converted into the engine force and steering angle for changing the vehicle states.
Reward and Cost Scheme. The reward function is composed of four parts as follows:
$$
R = c _ {1} R _ {\text {d i s p}} + c _ {2} P _ {\text {s m o o t h}} + c _ {2} P _ {\text {c o l l i s i o n}} + R _ {\text {t e r m}}. \tag {1}
$$
The displacement reward  $R_{disp} = d_t - d_{t-1}$ , wherein the  $d_t$  and  $d_{t-1}$  denotes the longitudinal movement of the target vehicle in Frenet coordinates of the target trajectory between two consecutive time steps, providing a dense reward to encourage the agent to move forward. The smooth penalty  $P_{smooth} = \min(0, 1 / v_t - |a[0]|)$  incentivizes the agent to drive smoothly and avoid a large steering value change between two timesteps, especially, when the velocity is high.  $v_t$  and  $a[0]$  denote the current velocity and the steering value respectively. In addition, if a collision with a vehicle, human, or object happens at timestep  $t$ , the agent will receive a collision penalty. The penalty is set to  $P_{collision} = 2$  when colliding with a human or vehicle and  $P_{collision} = 0.5$  for colliding with an object like cones and barriers. We also define a sparse terminal reward  $R_{term}$ , which is non-zero only at the last time step. At that step, we set  $R_{disp} = R_{speed} = 0$  and assign  $R_{term}$  according to the terminal state.  $R_{term}$  is set to +10 if the vehicle reaches the destination, -5 for being 2.5m away from the reference trajectory. We set  $c_1 = 2$ ,  $c_2 = 1$  and  $c_3 = 1$ .
Termination Conditions and Evaluation Metrics. The episode will be terminated only when: 1) the agent drives  $2.5m$  away from the reference trajectory 2) the agent arrives at the destination and 3) the agent can not finish the episode in recorded Episode_length + 50 steps. For each trained agent, we evaluate it in the held-out test environments and define the ratio of episodes where the agent arrives at the destination as the success rate. The definition is the same for out of road and Timeout. For evaluating the driving behavior, the speed in each scenario is collected and then averaged across all scenarios. Also, a metric similar to the success rate is measured and called route completion, which is the ratio of moving distance to the length of the whole reference trajectory. Since each agent are trained across 5 random seeds, this evaluation process will be executed for 5 agent which has the same training setting but different random seeds. We report the average and std on the metrics mentioned above.
B.2 Curriculum Training System