SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Quanyi Li\*, Zhenghao Peng\*, Lan Feng\*,
Zhizheng Liu†, Chenda Duan§, Wenjie Mo§, Bolei Zhou§
$^{\ddagger}$ University of Edinburgh,  $^{\dagger}$ ETH Zurich,  $^{\S}$ University of California, Los Angeles
Large-scale driving datasets such as Waymo Open Dataset and nuScenes substantially accelerate autonomous driving research, especially for perception tasks such as 3D detection and trajectory forecasting. Since the driving logs in these datasets contain HD maps and detailed object annotations that accurately reflect the real-world complexity of traffic behaviors, we can harvest a massive number of complex traffic scenarios and recreate their digital twins in simulation. Compared to the handcrafted scenarios often used in existing simulators, data-driven scenarios collected from the real world can facilitate many research opportunities in machine learning and autonomous driving. In this work, we present ScenarioNet, an open-source platform for large-scale traffic scenario modeling and simulation. ScenarioNet defines a unified scenario description format and collects a large-scale repository of real-world traffic scenarios from the heterogeneous data in various driving datasets including Waymo, nuScenes, Lyft L5, Argoverse, and nuPlan datasets. These scenarios can be further replayed and interacted with in multiple views from BirdEye-View layout to realistic 3D rendering in MetaDrive simulator. This provides a benchmark for evaluating the safety of autonomous driving stacks in simulation before their real-world deployment. We further demonstrate the strengths of ScenarioNet on large-scale scenario generation, imitation learning, and reinforcement learning in both single-agent and multi-agent settings. Code, demo videos, and website are available at https://metadriverse.github.io/scenarionet.
Autonomous Driving (AD) is revolutionizing mobility with benefits like reliable transportation, travel comfort, and fuel efficiency. As a safety-critical application, it is important to evaluate the AD stack thoroughly and ensure its reliability and safety before the real-world deployment [58]. Unlike the standardized evaluation of the perception module of AD stack on large-scale real-world annotated data [45], the planning and control modules are often evaluated in simulators with synthetic hand-crafted scenarios [21, 51, 27, 56]. These testing scenarios, often overly simplified, cannot reflect the complexity of real-world conditions like map structures and diverse traffic participant behaviors [7]. Moreover, it is time-consuming and requires domain knowledge to craft such testing scenarios, especially those with many traffic participants [22]. Thus, it remains challenging to scale up the evaluation and the training of the AD's decision-making in a wide range of situations.
Collecting traffic scenarios from real-world driving data can address the data accessibility issue. By replaying the real-world traffic scenarios in simulation, we can further benchmark the decision-making of the AD system and improve the generalizability of the data-driven planning and control components. However, several critical issues need to be addressed when building a large-scale
* Quanyi Li, Zhenghao Peng and Lan Feng contribute equally to this work.
Figure 1: Snapshots of three scenarios extracted from nuScenes and their corresponding interactive environments with multiple views and sensors including RGB camera, depth camera, and semantic camera. The RGB sensor can be used for end-to-end driving systems like Openpilot [10].
data-driven simulation platform: First, annotated driving data is precious and we would like to use as much data as possible. However, it is difficult to achieve this since publicly available driving data are released by different organizations and in various formats. This creates a bottleneck in aggregating the data volume from multiple sources for training and evaluating AD systems. Second, existing datasets are closely coupled with specific simulators or toolkits built for ad hoc applications, which largely limits the possibility of data sharing. For example, nuPlan [6] with  $1500+$  hours driving data aims at single-agent Imitation Learning (IL); Nocturne [54] with  $500+$  hours Waymo motion data is mainly designed for multi-agent Reinforcement Learning (RL) under partial observation; SimNet built on 1000 hours L5 motion dataset targets at scenario generation. Researchers who want to use nuPlan data for multi-agent RL have to spend a significant effort on building a new simulator or bridging Nocturne [54] and nuPlan dataset. This greatly restricts the potential of cross-dataset training. Furthermore, driving datasets like nuPlan [6], nuScenes [5], and Argoverse [8, 59] provide raw sensor data like images and point clouds, but existing 2D data-driven simulators can not make use of them. The lack of z-axis and 3D graphics prevents training visuomotor policies and evaluating end-to-end AD stacks, like Openpilot [10] in simulation.
To tackle the aforementioned challenges, we introduce ScenarioNet, an open-source platform to unify the heterogeneous driving data from various sources for large-scale traffic scenario simulation and modeling. It extracts a massive number of traffic scenarios in a unified format from various datasets, including Waymo motion dataset [18], nuScenes dataset [5], L5 motion prediction dataset [26], Argoverse dataset [8, 59], and nuPlan dataset [6]. These scenarios can be replayed and reacted through the MetaDrive simulator [30] in multiple views from BEV layout to realistic 3D rendering. ScenarioNet enables many machine learning applications like large-scale scenario generation, imitation learning, and reinforcement learning in both single-agent and multi-agent settings. With the built-in ROS bridge, the digital twins of the traffic scenarios can further serve as testbeds for AD stacks, where in our case we evaluate the open-source one like Openpilot [10]. We conduct a range of experiments based on ScenarioNet. First, we train a large scenario generation model on a joint dataset and analyze the embedded space of traffic scenarios to identify the relationship across various datasets. After that, we conduct cross-dataset single-agent training and testing experiments to show the advantage of real-world data for training ML-based planners. In addition, multi-agent RL/IL experiments are conducted on Waymo dataset [32, 18] to show the support for agent-based scenario generation. Lastly, we test Openpilot [10], an end-to-end AD stack in reconstructed scenarios. We hope our open-sourced ScenarioNet with its flexible APIs and toolkits empowers the community with many new research opportunities and propels the development of safe and generalizable AI for autonomous driving.
Synthetic scenario database. The majority of the existing scenario databases are based on synthetic data which are either handcrafted [3, 62, 16, 21, 17, 29, 63], or generated based on rules [7, 30, 24, 53,
Simulator	Supported Dataset	RL	IL	Multi-agent RL & IL	Scenario Generation	AD Stack Testing	3D Rendering
nuPlan-devkit [49]	nP	✓
DriverGym [28]	L5	✓	✓
Nocturne [54]	W	✓	✓	✓
TrafficSim [48]	-	✓
SimNet [48]	L5	✓
VISTA [2, 57]	W, nS	✓	✓	✓	✓
ScenarioNet	W, L5, nP, nS, Ag	✓	✓	✓	✓	✓	✓
Table 1: Comparison of representative data-driven simulators. ScenarioNet supports Waymo (W), L5 Lyft, nuPlan (nP), nuScenes (nS), and Argoverse (Ag) datasets and various applications and tasks.
66, 4] and adversarial attacks [13, 14, 40, 68, 67] which, in particular, aims at building safety-critical cases. Synthetic scenarios are predominantly described using OpenScenario [3, 17, 16], Scenic [21], and Python language [62, 29], allowing the definition of map structures, initial actor displacements, and subsequent behaviors. These scenario descriptions can be parsed for building scenarios in CARLA [15] seamlessly. Representative databases are OSC-ALKS [3], highway-env [29], DDrive [16], and SafeBench [62] which mainly provides safety-critical scenarios.
Real-world scenario database. To the best of our knowledge, CommonRoad [1] and nuPlan [6] are the only real-world datasets curated specifically for the development and testing of the planning module. CommonRoad lacks sensor data and has a limited number of scenarios, while nuPlan [6] has more than  $1500+$  hours of annotated driving data. Though high-quality data is provided by nuPlan, the accompanying nuPlan-devkit is limited to open-loop/close-loop IL. Extra effort is needed for it to support wide applications like RL-based planner training. In this case, ScenarioNet can complement single-agent and multi-agent RL training with nuPlan data. Besides, ScenarioNet has a 3D rendering support which can additionally make use of the sensor data in nuPlan dataset for investigating end-to-end AD stack and some interdisciplinary tasks like street scene reconstruction. Furthermore, ScenarioNet can incorporate more motion prediction datasets like Waymo dataset [44], nuScenes dataset [5], and L5 dataset [26] for cross-dataset training and testing.
Data-driven simulation. We refer to driving simulators capable of replaying real-world scenarios or generating realistic traffics as data-driven simulators. In Table 1, we summarize the properties of existing data-driven simulators in terms of widely supported real-world data sources and applications, and tasks. Each simulator has a different set of supported datasets. The support for RL and IL entails single-agent driving policy learning for the ego vehicle only, while MARL (IL) needs to control additional vehicles present in the scenario. Thus scenario generation represents methods requiring not only actuating objects in a similar way to trajectory forecasting [35, 55] but composing new traffic scenarios by inserting vehicles into an empty map. TrafficGen [19] exemplifies how ScenarioNet enables building the training dataset and loading generated scenarios for RL training. Among all the simulators, only the simulation environment provided by ScenarioNet supports AD stack testing. The last column shows that ScenarioNet and VISTA [2, 57] are the only two platforms that utilize camera and lidar data for 3D rendering support. Though both VISTA and ScenarioNet can train driving policies, there is a distinct difference. VISTA achieves closed-loop training and synthesizing new sensor data by applying relative transformations to the recorded sensor data. In contrast, ScenarioNet reconstructs scenarios with mid-level object representations, such as vehicle positions and velocities. Despite the loss of raw sensor data fidelity, this design enables broader scope of capabilities including not only closed-loop training but also scenario-based AD stack testing and scenario generation.
3 System Design of ScenarioNet
Fig. 3 illustrates the system design of the proposed ScenarioNet. It collects a massive number of real-world traffic scenarios by converting heterogeneous driving data from different sources, like Waymo dataset [47], nuScenes/nuPlan dataset [6, 5], and so on. Meanwhile, it can replay the imported scenarios in the MetaDrive simulator [30] through the unified scenario description, supporting various
machine learning applications and tasks, such as AD stack testing, single-agent learning, multi-agent learning, and scenario generation. We describe each component of ScenarioNet in more detail below.
3.1 Unified Scenario Description
A unified scenario description format is the key for extracting scenario data from different sources. As shown in Fig. 2, we define the unified scenario description format as a nested dictionary with 4 top-level keys. Each scenario in the source dataset will be converted into scenario_id.pkl.
Map Features. Each scenario description file stores map features consisting mainly of lanes and lane lines. Both lanes and lane lines have a polyline field recording the lane centerline or lane line itself. Given an object, we can know how far it moves along the lane and the lateral distance to the lane centerline by converting its position from the global Cartesian coordinate to the lane centerline's Frenet coordinate. On the other hand, contact checks can be executed between a vehicle and a lane line to decide whether the vehicle is driving across the lane line. For lanes, it has two extra fields: polygon, and connectivity. Polygon can be used to query if a given object is on the lane, and the drivable area can be determined by combining polygons of all lanes. Connectivity records the neighbor lanes, next and previous lanes of the given lane.
Objects. In most raw driving data, the road objects are usually stored in a frame-centric way where the user need to visit all time frames before collecting the state sequence of an object. In ScenarioNet, the object representation is object-centric meaning that querying the object dictionary with an object ID will return the state sequence across the episode. This is convenient for downstream applications which are mostly object-centric. Take trajectory prediction as an example. For getting the ground truth of the motion of an object, user can query the description with [object_id]["position"] and [object_id]["velocity"]. The queries will return the whole trajectory and velocity. However, not all objects are valid throughout the whole episode and hence we introduce a [object_id]["valid"] key for indicating which frame the object presents. For loading the objects into the simulator, we synchronize the position, heading, velocity, and rotation rate between the logged data and
the simulated object entity when the [object_id] ["valid"] indicator is True at the given frame. Otherwise, the object will disappear and be destroyed. By doing so, the digital twin scenario can be faithfully reconstructed in MetaDrive simulator.
Traffic Lights. Similar to objects, traffic lights are stored in a dictionary with the ID as the key and properties as the value. The light states can be accessed by [light_id]["state"] throughout one episode, where each element can be Red, Yellow, Green, or Unknown and represent the status of the corresponding lane. As each object can localize which lane it is on, it can easily know if there is a traffic light on the current lane or an oncoming lane.
Metadata. ["metadata"] exists at each level of the nested dictionary, storing miscellaneous data. For example, for an object, the [object_id]["metadata"] can store the instance ID provided by the original dataset. For the top-level dictionary of scenario description, a ["metadata"] stores the information about the source dataset, like the original scenario ID, which original file this scenario comes from, the time interval between two frames, the coordinates system of the data, etc. In addition, some statistics of the scenario are included such as the number of objects, the number of traffic lights, and the moving distance of each object. This information can be used to select or filter scenarios to build new databases as in Sec. 3.2.
To ensure a scenario description can be loaded into the simulator, we provide a tool script to check if necessary keys are present in the specific dictionary and if the data type of the key's value is correct. For example, ["position"] can not be absent from the object property dictionary, and its value should be in shape  $(N,3)$  where  $N$  is the episode length. As long as all necessary keys exist, users can add new fields to the scenario description to log more details about the scenario.
Figure 2: Example of Scenario Description
Single-agent learning
Multi-agent learning
III. Application Layer
AD stack test
Training Set
No Overpass
Test Set
Has Traffic Light
Safety-critical Scenarios
Synthetic Dataset
II. System Layer
Waymo
Convert
nuPlan & nuScenes
Real-world Dataset
More Filters
More Dataset
Figure 3: From bottom to top, ScenarioNet platform consists of the data layer, system layer, and application layer which are connected by two critical data flows, data conversion  $(\rightarrow)$  and simulation  $(\rightarrow)$ . Data conversing unifies various data formats and stores them in an internal scenario description. The system layer then provides a set of tools to operate converted data efficiently, such as filtering  $(\rightarrow)$ , merging, sanity-check, splitting  $(\rightarrow)$  and so on. Once the database is ready, it can be loaded into MetaDrive for large-scale simulation and various applications.
3.2 Scenario Database
The database in ScenarioNet is a folder containing two summary files dataset.summary.pkl and a dataset.mapping.pkl. dataset.summary.pkl stores a dictionary with keys indicating which scenarios are included in this database and values storing the metadata of each scenario. dataset.mapping.pkl records the relative path of the scenario file scenario_id.pkl from the database folder, and hence no matter where the scenario files are we can access them through the relative path recorded in dataset.mapping.pkl.
Database Operations. ScenarioNet provides several operations to create and maintain the databases including raw data conversion, sanity-check, merging, splitting, sampling, and filtering. A new database can be formulated with a series of operations. Conversion is the basic operation for creating a database by processing driving data in various formats to the unified scenario description. This operation produces a database: a folder with summary files dataset.summary.pkl and dataset.mapping.pkl and a number of converted scenario files with filenames like sd_nuscenes Scenes-0061.pkl. Currently, we provide data converters for parsing data from Waymo [47], nuPlan [6], nuScenes [5], L5 [26], and procedurally generated dataset [30]. It is also convenient to write new converters for users' own dataset for exploiting the power of ScenarioNet. After converting and building from raw data, the databases then can be operated to create new databases on demand. For example, Merging operation combines several databases together to get a larger one. Filtering operation creates a new database with scenarios satisfying specific conditions. For example, we can build a database with scenarios where ego car moving distance  $>10$  meters and the number of traffic participants  $>200$ . Sanity-check is a special filter for building a new database by discarding broken scenarios that can not be loaded into the simulator. Splitting and sampling aim to divide training and non-overlapping test sets. Instead of copying the raw data when creating new dataset, we only read the summary files from source dataset and create new dataset.summary.pkl and dataset.mapping.pkl. The copy-free design and the parallel operations ensure efficiency.
New Dataset Support. It is easy to add support for new datasets by following these steps:
1. Download the original data and set up the official codebase for parsing this data.
2. Make a converter_function which takes one scenario recorded in the original format as input and returns the scenario described in a new format. Actually, this process is like filling in the blank of the target scenario description by parsing the original data format with official APIs.
3. Scale up the data conversion with the built-in function, write_to_directory. It takes the convert_function and a list of scenario indices or original scenarios as input and writes the converted scenarios into a target directory. In this process, the multi-process parallel converting, sanity-check and other trivial steps will be finished automatically.
Though step 2 seems time-consuming, most of the work in this step is, actually, calling APIs provided by the new dataset, retrieving the desired data, and putting them in the corresponding field. A good example is convert_waymoscenario where most of the subfunction is called extract_xyz. Since the structure and fields to fill in the scenario description are clear and automatic tools are provided, we believe it would be easy for the community to add support for more datasets.
3.3 Scenario Simulation
We simulate the traffic scenarios as interactive environments through an upgraded version of our MetaDrive simulator [30], which encompasses Bullet [12] physics engine, 2D Pygame rendering backend, and 3D OpenGL rendering backend. The physics engine not only simulates realistic vehicle dynamics but also efficient contact and collision detection among vehicles, objects, and map features such as lane lines. As a result, the physics simulation can run up to 500 FPS for real-world scenarios containing +100 interactive objects including pedestrians, vehicles, cyclists, and traffic barriers/cones. We use synthetic models to represent pedestrians and vehicles, so no personal information is identifiable.
Sensors. ScenarioNet supports collecting the mid-level representations of scenarios through 2D pseudo-lidar and bird's-eye view images. It also supports querying the internal simulator states for precise ground truth information about surrounding objects, such as their positions and velocities. Nevertheless, end-to-end commercial AD stacks like Openpilot [10] are emerging and a recent trend suggests it is important to consider the perception and planning system as a whole [11]. ScenarioNet thus provides RGB-camera, depth-camera, and point-cloud for sensor simulation. All camera outputs are synthesized via the OpenGL rendering backend, which is wrapped by the Panda3D game engine and allows users to add new visual effects or shaders via Python code easily. We optimize the camera efficiency by bridging CUDA and OpenGL so that the images rendered by OpenGL in VRAM can be converted to Pytorch [38] Tensor directly. The entire procedure occurs on the GPU, resulting in an improvement from 11 FPS to 300 FPS when simulating a  $1920 \times 1080$  image. In addition, to better simulate RGB camera output, we upgrade the 3D-rendering effect of the native MetaDrive 3D renderer through deferred rendering which enables simulating photorealistic light, shadow, and material. Some examples of the supported sensors are in Fig. 1.
Control Policies. All objects in the scene are controlled by policies. In a single-agent setting, vehicles can be controlled by either the ReplayPolicy to replay logged trajectories or the IDMPolicy to not only replay but react to other objects. The IDMPolicy enables closed-loop simulation by detecting if there are objects present on its future trajectory and adjusting the target velocity to avoid collision with the leading object. In this way, when the ego car deviated from the recorded trajectory, vehicles behind it can react to this change and show yielding behavior. Also, the ego car can be controlled by the ReplayPolicy to the replay trajectory or the EnvInputPolicy to accept external control commands from env-step(). Moreover, by assigning the EnvInputPolicy to traffic vehicles, the task can be turned into a multi-agent setting. This policy-based design in ScenarioNet allows for mixed-policy simulation and provides high customizability. For instance, one can develop a policy that wraps a ROS bridge to connect open-source AD stacks to the simulation.
4 Experiments and Applications
We conduct a range of experiments and demonstrate various applications of ScenarioNet. We introduce how we built databases containing scenarios from different sources and then use these databases to conduct experiments on single-agent and multi-agent learning and AD stack testing.
Waymo Scenario
nuPlan Scenario
PG Scenario
Dataset	Track Length	No. of Vehicles	No. of Pedestrians	Intersection Ratio	Construction Ratio
Waymo	136.55(± 95.98)	89.93(± 64.51)	11.7(± 22.97)	0.71	0.0
nuPlan	95.48(± 42.32)	53.96(± 25.35)	21.99(± 19.9)	0.57	1.0
PG	226.07(± 70.7)	9.81(± 3.31)	0.0	0.36	0.39
Table 2: The statistics of the Waymo, nuPlan, and PG database.
4.1 Database Construction
Waymo. Waymo database is built with the Waymo Motion dataset [47]. The raw data contains  $500+$  hours of driving logs divided into 70,000+ scenarios with 20 seconds duration for each scenario. We convert all of them into internal scenario descriptions and then filter out the scenarios where the ego vehicle moves less than 10 meters or where overpasses exist.