ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling
Authors: Quanyi Li
Zhenghao Peng
Lan Feng
Zhizheng Liu
Chenda Duan
Wenjie Mo
Bolei Zhou
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Autonomous Driving
Real-world Scenarios
Reinforcement Learning
Traffic Simulation
AD Stack Testing

Abstract: Large-scale driving datasets such as Waymo Open Dataset and nuScenes substantially accelerate autonomous driving research, especially for perception tasks such as 3D detection and trajectory forecasting. Since the driving logs in these datasets contain HD maps and detailed object annotations which accurately reflect the real-world complexity of traffic behaviors, we can harvest a massive number of complex traffic scenarios and recreate their digital twins in simulation. Compared to the hand-crafted scenarios often used in existing simulators, data-driven scenarios collected from the real world can facilitate many research opportunities in machine learning and autonomous driving. In this work, we present ScenarioNet, an open-source platform for large-scale traffic scenario modeling and simulation. ScenarioNet defines a unified scenario description format and collects a large-scale repository of real-world traffic scenarios from the heterogeneous data in various driving datasets including Waymo, nuScenes, Lyft L5, and nuPlan datasets. These scenarios can be further replayed and interacted with in multiple views from Bird-Eye-View layout to realistic 3D rendering in MetaDrive simulator. This provides a benchmark for evaluating the safety of autonomous driving stacks in simulation before their real-world deployment. We further demonstrate the strengths of ScenarioNet on large-scale scenario generation, imitation learning, and reinforcement learning in both single-agent and multi-agent settings. Code, demo videos, and website are available at https://github.com/metadriverse/scenarionet

ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling
Quanyi Life, Zhenghao Peng Lan Fengi", Zhizheng Liut, Chenda Duan Wenjie Mo Bolei Zhou
University of Edinburgh, 1ETH Zurich, Universityy of California, Los Angeles
Abstract
Large-scale driving datasets such as Waymo Open Dataset and nuScenes substan- tially accelerate autonomous driving research, especially for perception tasks such 3D detection and trajectory forecasting. Since the driving logs in these datasets contain HD maps and detailed object annotations that accurately reflect the real- world complexity of traffic behaviors, we harvest massive number of complex traffic scenarios and recreate their digital twins in simulation. Compared to the hand- crafted scenarios often used in existing simulators, data-driven scenarios collected from the real world can facilitate many research opportunities in machine learning and autonomous driving. In this work, we present ScenarioNet, an open-source platform for large-scale traffic scenario modeling and simulation. ScenarioNet defines unified scenario description format and collects a large-scale repository of real-world traffic scenarios from the heterogeneous data in various driving datasets including Waymo, nuScenes, Lyft LS, Argoverse, and nuPlan datasets. These scenarios can be further replayed and interacted with in multiple views from Bird- Eye- layout to realistic 3D rendering in MetaDrive simulator. This provides a benchmark for evaluating the safety of autonomous driving stacks in simulation before their real-world deployment. We further demonstrate the strengths of Sce- narioNet on large-scale scenario generation, imitation learning, and reinforcement learning in both single-agent and multi-agent settings. Code, demo videos, and website are available at https: httpee/mmmmtteeeeeeeeeeeeee gi github. io/ see.eaeenii.....
1 Introduction
Autonomous Driving (AD) is revolutionizing mobility with benefits like reliable transportation, travel comfort, and fuel efficiency. As safety-criticll application,
important evaluate the AD stack thoroughly and ensure its reliability and safety before the real-worke deployment [58]. Unlike the standardized evaluation the perception module of AD stack on large-scale real-world annotated data [45], the planning and control modules are often evaluated in simulators with synthetic hand-crafted scenarios [21, 51, 27, 56]. These testing scenarios, often overly simplified, cannot reflect the complexity of real-world conditions like map structures and diverse traffic participant behaviors [7]. Moreover, it is time-coneeee consumnnn and requires domain knowledge to craft such testing scenarios, especially those with many traffic participants [22]. Thus, it remains challenging to scale up the evaluation and the training of the AD's decision-making a wide range of situations.
Collecting traffic scenarios from real-world driving data can address the data accessibility issue. By replaying the real-world traffic scenarios in simulation, we can further benchmark the decision- making of the AD system and improve the generalizability of the data-driven planning and control components. However, several critical issues need to be addressed when building a large-scale
Quanyi Li, Zhenghao Peng and Lan Feng contribute equally to this work.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Driwing Logs
RGB Cameraappanpiiot
Depth Camera
Semantic Camera
Top down view
nuScones mini scene 0061
Ego Car
n=Scema-rr1 00rmma/--A.0iiii--ee----0000
Figure 1: Snapshots of three scenarios extracted from nuScenes and their corresponding interactive environments with multiple views and sensors including RGB camera, depth camera, and semantic camera. The RGB sensor can be used for end-to-end driving systems like Openpilot [10].
data-driven simulation platform: First, annotated driving data is precious and we would like to use as much data as possible. However, it is difficult to achieve this since publicly available driving data are released by different organizations and in various formats. This creates a bottleneck in aggregating the data volume from multiple sources for training and evaluating AD systems. Second, existing datasets are closely coupled with specific simulators or toolkits built for ad hoc applications, which largely limits the possibility of data sharing. For example, nuPlan [6] with 1500+ hours driving data aims at single-agent Imitation Learning (IL); Nocturne [54] with 500+ hours Waymo motion data is mainly designed for multi-agent Reinforcement Learning (RL) under partial observation; SimNet built on 1000 hours L5 motion dataset targets at scenario generation. Researchers who want to use nuPlan data for multi-agent RL have to spend a significant effort on building new simulator or bridging Nocturne [54] and nuPlan dataset. This greatly restricts the potential of cross crossedataset training. Furthermore, driving datasets like nuPlan [6], nuScenes [5]. and Argoverse [8, 59] provide raw sensor data like images and point clouds, but existing 2D data-driven simulators can not make of them. The lack of 7-axis and 3D graphics prevents training visuomotor policies and evaluating end-to-end AD stacks, like Openpilot [10] in simulation.
To tackle the aforementioned challenges, introduce ScenarioNet, open-source platform to unify the heterogeneous driving data from various sources for large-scale traffic scenario simulation and modeling. It extracts massive number of traffic scenarios in a unified format from various datasets, including Waymo motion datset [18], nuScenes dataset [5], L5 motion prediction dataset [26], Argov- erse dataset [8, 59], and nuPlan dataset [6]. These scenarios
be replayed and reacted through the MetaDrive simulator [30] in multiple views from BEV layout to realistic 3D rendering. ScenarioNer enables many machine learning applications like large-scale scenario generation, imitation learning, and reinforcement learning in both single-agent and multi-agent settings. With the built-in ROS bridge, the digital twins of the traffic scenarios can further serve testheds for AD stacks, where in our case we evaluate the open-source one like Openpilot [10]. We conduct a range of experiments based on ScenarioNet. First, we train large scenario generation model on a joint dataset and analyze the embedded space of traffic scenarios to identify the relationship across various datasets. After that, we conduct cross-dataset single-agent training and testing experiments to show the advantage of real-world data for training based planners. In addition, multi-agen RL/IL experiments are conducted on Waymo dataset [32, 18] to show the support for agent-based scenario generation. Lastly, we test Openpilot [10], an end-to-en end AD stack in reconstructed scenarios. We hope our open- sour-sd ScenarioNef with flexible APIs and toolkits empowers the community with many new research opportunities and propels the development of safe and generalizable AI for autonomous driving.
2 Related Work
Synthetic scenario database. The majority of the existing scenario databases are based on synthetic data which are either handcrafted [3, 62, 16, 21, 17, 29, 63), generated based on rules [7, 30, 24, 53,

---

Supported Dataset
Multi-agent
Scenario & Generation
AD Stack Testing
3D
Simulator
RL
II
Rendering
nuPlan-devkit [49]
nP
DriverGym [28]
L5
Nocturne [54]
TrafficSim [48]
SimNet [48]
1.5
VISTA 57]
W, nS
ScemarioNet
W. L5, np, nS, Ag
Table 1: Comparison of representative data-driven simulators. ScenarioNer supports Waymo (W), L5
Lyft, nuPlan (nP), nuScenes (nS), and Argoverse (Ag) datasets and various applications and tasks.
66, 4] and adversarial attacks [13, 14, 40, 68, 67] which, in particular, aims at building safety-crition cases. Synthetic scenarios are predominantly described using OpenScenario [3. 17, 16], Scenic [21], and Python language [62, 29], allowing the definition of map structures, initial actor displacements, and subsequent behaviors. These scenario descriptions can be parsed for building scenarios in
CARLA [15] seamlessly. Representative databases OSC-ALKS [3], highway-env [29], DI-
drive [16]. and SafeBench [62] which mainly provides safety-critical scenarios.
Real-world scenario database. To the best of our knowledge. CommonRoad [1] and nuPlan [6] are the only real-world datasets curated specifically for the development and testing of the planning module. CommonRoad lacks sensor data and has limited number of scenarios, while nuPlan [6] has more than 1500+ hours of annotated driving data. Though high-quality data is provided by nuPlan, the accompanying nuPlan-devkit is limited to open-loop/close-loop IL. Extra effort is needed for it to support wide applications like RL-based planner training.In In this case, ScenarioNer can complement single-agent and multi-agent RL training with nuPlan data. Besides, ScenarioNet has a 3D rendering support which can additionally make use of the sensor data in nuPlan dataset for investigating end-to-end AD stack and some interdisciplinary tasks like street scene reconstruction. Furthermore, ScenarioNet can incorporate more motion prediction datasets like Waymo dataset [44], nuScenes dataset [5], and L5 dataset [26] for cross-dataset training and testing.
Data-driven simulation. We refer to driving simulators capable of replaying real-world scenarios generating realistic traffics
oata-driven simulators. In Table 1, we summarize the properties of existing data-driven simulators in terms of widely supported weal-world data sources and applications, and tasks. Each simulator has different of supported datasets. The support for RL and IL. entails single-agent driving policy learning for the ego vehicle only, while MARL. (IL. needs to control additional vehicles present in the scenario. Thus scenario generation represents methods requiring not only actuating objects in a similar way to trajectory forecasting [35, 55] but composing new traffic scenarios by inserting vehicles into an empty map. TrafficGen [19] exemplifies how ScenarioNet enables building the training dataset and loading generated scenarios for RL training. Among all the simulators, only the simulation environment provided by ScenarioNet supports AD stack testing. The last column shows that ScenarioNet and VISTA [2, 57] are the only two platforms that utilize camera and lidar data for 3D rendering support. Though both VISTA and ScenarioNet can train driving policies, there is distinct difference. VISTA achieves closed-loop training and synthesizing new sensor data by applying relative transformations to the recorded sensor data. In contrast, ScenarioNer reconstructs scenarios with mid-level object representations, such as vehicle positions and velocities. Despite the loss of raw sensor data fidelity, this design enables broader scope of capabilities including not only closed-loop training but also scenario-based AD stack testing and scenario generation.
3 System Design of ScenarioNet
Fig. illustrates the system design of the proposed ScenarioNet. It collects a massive number of real-world traffic scenarios by converting heterogeneous driving data from different sources, like Waymo dataset [47], nuScenes/nuPlan dataset [6, 5], and so on. Meanwhile, it can replay the imported scenarios in the MetaDrive simulator [30] through the unified scenario description, supporting various
3

---

machine learning applications and tasks, such as AD stack testing, single-agent learning, multi-agent learning, and scenario generation. We describe each component of ScenarioNer in more detail below.
3.1 Unified Scenario Description
A unified scenario description format is the key for extracting scenario data from different sources. As shown in Fig. 2, we define the unified sce- nario description format as nested dictionary with 4 top-level keys. Each scenario in the source dataset will be converted into scenario _ddd pkl.
("map_feature features", [
'map obj id 1":
"cenerriine": [..].
"type: "lane",
"connectivity" "; I...)
"palygon': [-]
Map Features. Each scenario description file stores map features con- sisting mainly of lanes and lane lines. Both lanes and lane lines have a polyline field recording the lane centerline or lane line itself. Given an object, we can know how far it moves along the lane and the lateral distance to the lane centerline by converting its position from the global Cartesian coordinate to the lane centerline's Frenet coordinate. On the other hand, contact checks can be executed between vehicle and lane line to decide whether the vehicle is driving across the lane line. For lanes, it has two extra fields: polygon, and connectivity. Polygon can be used to query if a given object is on the lane, and the drivable area can be determined by combining polygons of all lanes. Connectivity records the neighbor lanes, next and previous lanes of the given lane.
mma obj 2%:
pollllnee: [--..
Type": "white solid line"
"objects": of
object id 11:
position": L..]
"heading": [-.].
"velocity" [-).
'valid": I...
"size": (. w. h),
"type"; "VEHICLE"
"traffic light": (
Objects. In most raw driving data, the road objects are usually stored in a frame-centric way where the user need to visit all time frames be- fore collecting the state sequence of an object. In ScenarioNet, the object representation object-centric meaning that querying the object dictionary with object ID will return the state sequence across the episode. This is convenient for downstream applications which are mostly object-centric. Take trajectory prediction an example. For getting the ground truth of the motion of an object, user can query the description with [object__ [bbjectid]] [" poositionn] and [object_ id] ["velocity"]] The queries will return the whole trajectory and velocity. However, not all objects are valid throughout the whole episode and hence we introduce a [object_ id] [ ["valid"] key for indicating which frame the object presents. For loading the objects into the simulator, we synchronize the position, heading, velocity, and rotation rate between the logged data and the simulated object entity when the [object_ _id] ["valid"] indicator is True at the given frame. Otherwise, the object will disappear and be destroyed. By doing so, the digital twin scenario can be faithfully reconstructed in MetaDrive simulator.
'light 1
"states": [..].
"pasition": (x. y. z). "lane": 'map_ob) }
"metadata": (
"dataset"' "nuscenes",
"time interval": 0.1, "summary': "num objects": 53,
Figure 2: Example of Scenario Description
Traffic Lights. Similar to objects, traffic lights are stored in a dictionary with the ID as the key and properties as the value. The light states can be accessed by [light_ide id] ["state"] throughout one episode, where each element can be Red, Yellow, Green, or Unknown and represent the status of the corresponding lane. As each object can localize which lane it is on, it can easily know if there is traffic light on the current lane or an oncoming lane.
MetaData. ["metadata"] exists at each level of the nested dictionary, storing miscellancous data. For example, for an object, the [object_id] ["metadata"] can store the instance ID provided by the original dataset. For the top-level dictionary of scenario description, a ["metadata") stores the information about the source dataset, like the original scenario ID, which original file this scenario comes from, the time interval between two frames, the coordinates system of the data, etc. In addition, some statistics of the scenario are included such as the number of objects, the number of traffic lights, and the moving distance of each object. This information can be used to select or filter scenarios to build new databases as in Sec. 3.2.
To ensure a scenario description can be loaded into the simulator, we provide a tool script to check necessary keys are present in the specific dictionary and if the data type of the key's value is correct. For example, ["position"] can not be absent from the object property dictionary, and its value should be in shape (N, 3) where N is the episode length. As long as all necessary keys exist, users can add new fields to the scenario description to log more details about the scenario.

---

4
101
Single-agent learning
Multi-agent learning
Scenario
AD
III. Application Layer
Simulation
Split
Split
Training Set
Test Set
Ego Car Moving Distance
Number of Objects
No Overpass
Has Traffic Light
More Filters
Convert
Convert
II. System Layer
METADRIVERSE
W
?
Procedural Generated Scenarios Safety-critical Scenarios
Waymo
nuPlan & nuScenes
Level
More Dataset
1. Data Layer
Synthetic Dataset
Real-world Dataset
Figure 3: From bottom to top, ScenarioNer platform consists of the data layer, system layer, and appli- cation layer which connected by critical data flows, data conversion (-) and simulation (-).
Data conversing unifies various data formats and stores them in an internal scenario description. The
system layer then provides set of tools operate converted data efficiently, such as filtering (-),
merging, sanity-check, splitting (-) and so on. Once the database is ready, it can be loaded into MetaDrive for large-scale simulation and various applications.
3.2 Scenario Database
The database in ScenarioNer is folder containing two summary files dataset sarrett_ssmrrrrreee pkl and a dataset_mape pacpttt__pppnnn___eee pkl. dataset dataset_summary, pk1 stores a dictionary with keys indicating which scenarios dataset dataset_mapping mapping.................a....... pkl records the relative path of the scenario file scenario. id. pkl from the database folder, and hence matter where the scenario files relative path recorded
included in this database and values storing the metadata of each scenario.
access them through the
aataset_mapping pk1.
Database Operations. ScenarioNel provides several operations to create and maintain the databases including raw data conversion, sanity-check, merging, splitting, sampling, and filter- ing. A new database can be formulated with series of operations. Conversion is the ba- sic operation for creating a database by processing driving data in various formats to the uni- fied scenario description. This operation produces database: folder with summary files dataset_summary" pk1 and dataset_ dataset_mapping -alllngg and a number of converted scenario files with filenames like sd_ www.unussse... Lassenns__ccnes-OOne setessseeeeessssse pkl. Currently,
provide data converters for parsing data from Waymo [47], nuPlan [6]. nuScenes [5], L5 [26], and procedurally generated dataset [30]. It is also convenient to write new converters for users' own dataset for exploiting the power of ScenarionNet. After converting and building from raw data, the databases then can be operated to create
databases demand. For example, Merging operation combines several databases together to get a larger one. Filtering operation creates a new database with scenarios satisfying specific conditions. For example, can build database with scenarios where ego car moving distance > 10 meters and the number of traffic participants > 200. Sanity-check is a special filter for building a new database by discarding broken scenarios that can be loaded into the simulator. Splitting and sampling aim to divide training and non non-overlapping test sets. Instead of copying the raw data when creating new dataset, we only read the summary files from source dataset pk1 and dataset_mappi
and create new dataset_sumaet the parallel operations ensure efficiency.
pkl. The copy-free design and
New Dataset Support. is easy to add support for new datasets by following these steps:
1. Download the original data and set up the official codebase for parsing this data.

---

2. Make a anreerttrrfucctii which takes one scenario recorded in the original format as imput and returns the scenario described in a new format. Actually, this process is like filling in the blank of the target scenario description by parsing the original data format with official APIs. Scale up the data conversion with the built-in function, write_ to ttlttt_dirree _o_directory. convert_ snnvert_ffnctiie function and list of scenario indices or original scenarios as input and writes the converted scenarios into a target directory. In this process, the multi-process parallel converting, sanity-check and other trivial steps will be finished automatically.
takes the
Though step seems consumnns most of the work in this step is, actually, calling APIs provided by the new dataset, retrieving the desired data, and putting them in the corresponding field. A good scenario where most of the subfunction is called extract_ _a._xyz.
example is convert_ w_wtt__________ wwwyoossceaaraaa Since the structure and fields to fill in the scenario description are clear and automatic tools are provided, we believe it would be easy for the community to add support for more datasets.
3.3 Scenario Simulation
We simulate the traffic scenarios as interactive environments through an upgraded version of our MetaDrive simulator [30], which encompasses Bullet [12] physics engine, 2D Pygame rendering backend, and 3D OpenGL rendering backend. The physics engine not only simulates realistic vehicle dynamies but also efficient contact and collision detection among vehicles, objects, and map features such as lane lines. As a result, the physics simulation can run up to 500 FPS for real-world scenarios containing + +100 interactive objects including pedestrians, vehicles, eyclists, and traffic barriers/cones. We use synthetic models to represent pedestrians and vehicles, so no personal information identifiable.
Sensors. ScenarioNet supports collecting the mid-level representations of scenarios through 2D pseudo-lidar and bird's-eye view images. It also supports querying the internal simulator states for precise ground truth information about surrounding objects, such as their positions and velocities. Nevertheless, end-to-end commercial AD stacks like Openpilot [10] are emerging and recent trend suggests it is important to consider the perception and planning system as whole [11]. ScenarioNet thus provides RGB-camera, depth-camera and point-cloud for sensor simulation. All camera outputs are synthesized via the OpenGL rendering backend, which is wrapped by the Panda3D game engine and allows users to add new visual effects or shaders via Python code easily. We optimize the camera efficiency by bridging CUDA and OpenGL so that the images rendered by OpenGL. in VRAM can be converted Pytorch [38] Tensor directly. The entire procedure occurs on the GPU, resulting in improvement from 11 FPS to 300 FPS when simulating 1920x 1080 image. In addition, to better simulate RGB camera output, we upgrade the 3D-rendering effect of the native MetaDrive 3D renderer through deferred rendering which enables simulating photorealistic light, shadow, and material. Some examples of the supported sensors are in Fig. 1.
Control Policies, All objects in the scene are controlled by policies. In a single-agent setting, vehicles can be controlled by either the ReplayPoiccy to replay logged trajectories or the IDMPolicy to not only replay but react to other objects. The IDMPolicy enables closed-loop simulation by detecting if there are objects present on its future trajectory and adjusting the target velocity to avoid collision with the leading object. In this way, when the ego car deviated from the recorded trajectory, vehicles behind it can react to this change and show yielding behavior. Also, the ego car can be controlled by the ReplayPolicy to the replay trajectory or the EnvInputPolicy to accept external control commands from env. st Moreover, by assigning the Env InvinpuPoliicy to traffic vehicles, the task can be turned into a multi-agent setting. This policy-based design in ScenarioNer allows for mixed- mixed-policy simulation and provides high customizability. For instance, one can develop a policy that wraps a ROS bridge to connect open-source AD stacks to the simulation.
4 Experiments and Applications
We conduct a range of experiments and demonstrate various applications of ScenarioNet. We introduce how we built databases containing scenarios from different sources and then use these databases to conduct experiments on single-agent and multi-agent learning and AD stack testing.

---

Ego
OET
Waymo Scenario
nuPlan Scenario
PG Scenario
Dataset
Track Length
No. of Vehicles
No. of Pedestrians
Intersection Ratio
Construction Ratio
Waymo
c36.55 95.99)
89.93 6451)
11.7c 22,97
0.71
0.0
nuPlan 95.48 42.30)
53.96 2535)
21.99
0.57
1.0
PG
226.07( 70.7)
9.81 331)
0.0
0.36
0.39
Table 2: The statistics of the Waymo, nuPlan, and PG database.
4.1 Database Construction
Waymo.
Waymo database is built with the Waymo Motion dataset [47]. The data contains 500+ hours of driving logs divided into 70,000+ scenarios with 20 seconds duration for each scenario. We convert all of them into internal scenario descriptions and then filter out the scenarios where the ego vehicle moves less than 10 meters or where overpasses exist.
nuPlan. Though nuPlan [6] has more than 1500 hours of driving data, only use those recorded in Boston as we want to keep all databases in the experiments to have similar sizes. After filtering out scenarios whose ego moves 10 meters, we finally collect 50,261 20-second scenarios.
PG. PG stands for procedural generation which is used to generate infinite maps and traffic according to set of pre-defined roadblocks and traffic initialization rules. It is the default scenario generation method MetaDrive simulator [30]. We the traffic density 15 vehicles per 100 meters and roadblocks in each scenario to keep scenario length consistent with real-world scenarios. We finally collected 50,000 scenarios form the database.
Table provides statistics about the three datasets. Truck length stands for the average moving distance of the ego across all scenarios, and PG scenarios have the longest moving distance. Waymo scenarios are relatively more complex compared to others, as they have more vehicles and more intersection scenarios, which is reflected in Fig. 4 as well. The Construction Ratio represents the ratio of scenarios containing traffic cones and barriers out of all scenarios. muPlan-boston database has traffic cones and barriers in all scenarios, while no objects are contained in the Waymo database. More database construction details are in Appendix F.
4.2 Visualization of Scenario Embeddings with Scenario Generation Model
To reveal the differences and gaps across three databases, we use the internal emhedding from a scene generation model, TrafficGen [19]. visualize the scenario snapshots from all three databases. The state initializer in TrafficGen utilizes an encoder-decoder structure that encodes the traffic scenario into a latent space and then decodes the initial states distribution of traffic participants, enabling the sampling and generation of a new scenario snapshot. We randomly select 1000 scenarios from each database, feed them into TrafficGen obtain the embeddings of these scenarios and then employ t-SNE [52] to visualize them in Fig. 4. We can see there is a noticeable domain gap between synthetic and real-world data. Only two synthetic intersection scenarios are close to real-world scenarios and bridge the synthetic scenarios and real scenarios. Additionally, even among real-world scenarios, minor distribution gap remains, primarily due to different road network topologies different regions where the data is collected. These findings suggest that solely relying driving data from one
7

---

Wayma nuPlan
from nuPlan and Waymo datasets are much more similar, while synthetic PG data has clear domain gap with both real-world datasets. We also plot the representative scenarios in clusters. As shown in the left-side figures (L), synthetic scenarios can be neatly categorized into distinct clusters, while real- world scenarios shown the right side too complex identify clear boundaries. Nevertheless, the right-side figures show scenarios from nuPlan (I) have simpler map structures and mild traffic, compared the Waymo scenarios causing it to cluster them with real-world scenarios. The visualization of the two synthetic scenarios Figure 4: t-SNE visualization of the feature embeddings of scenarios from three databases. Scenarios
(C). In addition, there two outliers that confuse the model,
indicates that the intersection scenario can bridge the sim-to-real gap.
region or one database is insufficient to cover all potential traffic situations. Therefore, it is important aggregate as many scenarios as possible. Additional showcases from different databases, visualizations of generated scenarios, and training details can be found in Appendix A.
4.3 Cross-dataset RL Generalization
The previous experiment reveals that the domain gap exists between synthetic scenarios and real-world scenarios. We would like to further investigate whether the domain gap affects driving policy learning. To this end, we build data-driven RL environment where the agent should arrive at the destination of recorded trajectories in time. Traffic vehicles behind the ego car are controlled by IDMPolicy to achieve closed-loop training and testing. We further split training and test sets for the nuPlan database and the PG database, respectively, and get splits: nuPlan-train, PG-truin, nuPlan-test and PG-test. Each training set contains 40,000 scenarios, while each test set has 5,000 non-overlapping scenarios.
The scenarios of two training splits sorted ac- cording to the difficult score, a metric considering length and curvature of trajectory, and constitutes 100 levels for conducting the curriculum training. Each level contains 400 scenarios and 75% Success Rate,
nuPlen wi curriculum
PG wt curricutum
Combine wi curriculum
0.6
6
0.4
4
the ratio of scenarios where the agent can reach the destination out of 400 scenarios, is required for the learning environment to level up. We train PPO [43] agents on PG-train and nuPlan-train splits with the curriculum training and evaluate them on nuPlan- test split study whether the domain gap harms the performance for the learning- based controller. All re- sults are averaged across 5 random seeds and the error bar shows the standard deviation. As shown in Fig. 5, when testing on held-out nuP nuPlan-ter set, the agents trained with nuPlan-train split outperform those trained with PG-train split in terms of Success Rute. This is because the agents trained in synthetic scenarios fail to learn high-speed driving skills and thus can not reach the destination in time, inducing high Timeoul Rate. Thus it is necessary to train learning-based controllers with real-world data to close the sim-tos fim-to-real gap. We also ablate the curriculum training and the results
2
Success Rate
Out of Road Rate
Timeout Rate
Speed (m/s)
Figure 5: Evaluation on nuPlan-test split.

---

Route Completion
Success Rate
Average Distance
Final Distance
Reward
Cost
TD3
0.669 (+0.11))
0.513 (#0028)
74.87 (#:50)
2.01 (40.24)
12.64 (4072)
45.13 (#:773)
PPO
0.740 (45377
0.579 (=:0054))
85,28 (d:8.6)
2.83 (50.53)
12.79 (40773)
36.71 (d:6.8))
CoPO
0.745 (200024)
0,570 (2:004)
85,94 (2:460)
2.88 (20.32)
12.86 (1:60)
34.75 (1:378)
AIRL (Dise.)
0.367 (#20127)
0.153 (=00028)
0.55 (2)0112)
17.73 (4:187)
80.57 (d:5/26)
AIRL (Comt)
0.138 (#0.11))
0.015 (#00009)
0.42 (#0.16)
3.63 (#100)
111.23 (d:748)
GAIL (Disc)
0,487 (1:00211)
0.218 (1:0.18)
1.64 (2:0.35)
28.48 (1:2223)
68.53 (1:3.57)
GAIL (Cont)
0.421 (2:0043)
0.189 (1:0051)
1.56 (1:0.52)
27.33 (1:4.22)
75.98 (1:9.02)
Table 3: The multi-agent reinforcement learning and imitation learning results on the Waymo dataset.
suggest that it is critical to adjust the scenario difficulty according to the agent performance in the course of training. Finally, we build new dataset containing 40,000 scenarios by combining half of the PG-train split and half of the nuPlan-traie split. Figure. 5 shows that agents trained on these combined datasets achieve the highest Success Rate when evaluated on muPlan-test split. This is because almost every synthetic scenario contains bends and curvated trajectories, which only exist in approximately 20% of real- world scenarios. As a result, agents acquire a better ability to follow a curved trajectory, achieving the lowest Out of Road Rate. Thus aggregating synthetic scenarios to the training dataset can benefit policy learning as they improve the diversity of training data. More details of the experiments are in Appendix B.
4.4 Multi-agent Cooperative Learning
ScenarioNet can load scenario data into an interactive multi-agent policy learning environment and generate local sensory observations for each agent. Therefore, ScenarioNet enables the critical research on multi-agent reinforcement learning as the agents are self-interested and the number is varying, wherein cooperative MARL methods can not be applied [39], as well as the research on multi-agent imitation learning with the challenges of learning from observation or say the action-free imitation learning, meaning that the ground-truth trajectory does not contain action but only the sequence of states (observations). ScenarioNer bridges the research of aueti-agent decision-makiln with vast volume of real-world data. We load the ground-truth (GT) trajectory in the dataset and use it to form the termination condition, the reward function, and the evaluation metrics. For multi-agent the GT' trajectory define the displacement reward: r the displacement of agent i at step t projected into its underlying ground-truth trajectory. We train independent PPO [43] and TD3 [23] agents as well as the Coordinated Policy Optimization (CoPO) agents [39]. For multi-agent IL, to address the action-free issue, we use GAIL [25] in multi-agent setting [46] but the discriminator distinguishes state-next state pair, instead of state-action pair [50]. We also train multi-agent Adversarial Inverse RL [65] (AIRL) with an additional inverse dynamics model for estimating the expert actions. The inverse dynamics model is trained concurrently with the AIRL policies and learns the action given state-next state pairs from the environment interactions. The ground-truth trajectory can be used to measure the learned behaviors, such as the route completion rate, the ratio between the length of projected agent trajectory and the length of GT trajectory as well as the average and final distance between agent trajectory with GT trajectory. The cost the number of crashes of an agent in one episode. As shown in Table 3, we find that the displacement reward is strong supervision to learn multi-agent behavior as the RL can explore and exploit the reward function by interacting with the environment, without learning discriminator as in GAIL or a reward function and an inverse dynamics model in AIRL. More details can be found in Appendix C.
RL,
4.5 AD Stack Testing
We released ROS bridge [33], allowing connecting ScenarioNer with open-sourced AD stacks like
autoware [20] and planning algorithms developed by the ROS community. As result, researchers can study self-driving systems in real-world scenarios. In our experiment, we test Openpilot [10], an end-to-end AD system taking only visual information as input. As the navigation module of Openpilot is still a beta version, the test happens in scenarios without diverged roads. As shown in Fig. 6, the Openpilot system robust enough to operate in common scenarios like lane-keeping and stopping at traffic lights. More scenario-base based test videos and details are available in Appendix E.2.

---

Lane Keeping
24
24
24
24
24
Y
t=0s
F-2s
1-46
F-66
[=86
31
24
31
24
31
t=16s
Stopping at Traffic Light
20
19
16
14
11
T
t=0s
t=
tm4s
t=6s
t=8s
6
xn
t=10s
F-126
F166
t=186
Figure 6: Openpilot can operate in real-world scenarios reconstructed with ScenarioNet. It can eruise on urban roads and smoothly stop at crowded intersections.
Conclusion
This work presents an open-source platform called ScenarioNet for simulating and modeling driving scenarios. massive number of real-world traffic scenarios extracted from various datasets can be used for testing AD stacks, generating scenarios, and learning both single-agent and multi-agent policies. We hope can open up many new opportunities for AI and autonomous driving communities.
Limitation & Potential Negative Societal Impacts. One limitation of ScenarioNet at the current development stage is the lack of diverse 3D assets. Currently, the assets are collected from the internet, which may produce unrealistic raw sensor data output. Future work will be to reconstruct mesh and textures of vehicles, other objects, and background models like streets from driving logs using neural rendering. A detailed plan for improving visual fidelity is in Appendix G. In addition, several potential technical drawbacks exist in ScenarioNet. One drawback is that the simulation is single-thres tereeehrradddd due to the Python GIL and runs on CPU, while some latest simulators like IsaacGym [34] support GPU-based parallel simulation and thus outperform ScenarioNer in terms of sample efficiency. This problem can be alleviated by using multi-process tools like Ray/Rllib [37, 31] for parallel simulation. Another drawback is that ScenarioNer doesn't support differential simulation which allows training policy with the derivative of reward function directly [36]. In the future, ScenarioNer can address this by introducing a bicycle model for vehicle dynamics simulation, making the system differentiable. One potential negative societal impact is that there is no guarantee of safety for even a data-driven AD stack can pass all the testing scenarios in ScenarioNet. There is also domain gap for the trained model be deployed in the real world.
Licenses. ScenarioNer is released under Apache License 2.0. Panda3D, used in MetaDrive for 3D rendering, is under the Modified BSD license. Bullet engine is under the Zlib license. The vehicle models are collected from Sketchfab under CC BY 4.0 or CC BY-NC 4.0. For real-world data used in experiments, Waymo Open Dataset [32, 47] is under license terms available at https: / //waymo. com/ open. Besides, the L5 dataset, nuPlan dataset, nuScenes dataset, and Argoverse dataset are provided free of charge under the Creative Commons Attribution- CerrrionnNooooomeeree ShareAlike 4.0 International Public license. The toolkits for parsing these datasets are all under Apache license version 2.0. The Openpilot system used for AD stack testing under MIT license.
Acknowledgement: This work was supported by the National Science Foundation under Grant No. 2235012 and the Samsung Global Collaboration Award.
10

---

References
[1] M. Althoff, M. Koschi, and S. Manzinger. Commonroad: Composable benchmarks for motion planning on roads. In 2017 TEEE Intelligent Vehicles Symposium (IV), pages 719-726. IEEE, 2017.
[2] A. Amini, T.-H. Wang. Gilitschenski, W. Schwarting, Z. Liu, S. Han. S. Karaman, and D. Rus. Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles. In 2022 International Conference Robotics and Automation (ICRA), pages 2419-2426. IEEE, 2022.
[3] asam oss. Alks scenario interpretation in openscenario.
https: htpp:://gitt //gbt/b.///ttt/...// com/ asam- oss/
OSC- ALKS- scenarios, 2020.
[4] M. Behrisch, Bieker. Erdmann, and D. Krajzewicz. Sumo simulation of urban mobility: an overview.
In Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation. ThinkMind, 2011.
[5] Н. Caesar, Bankiti, A. Н. Lang, Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11621-11631, 2020.
[6] H. Caesar, J. Kabzan, K. S. Tan, K. Fong. E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and
S. Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11 11810, 2021.
[7] P. Cai. Lee, Luo, and D. Hsu, Summit: A simulator for urban driving in massive mixed traffic. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4023-4029 IEEE, 2020.
[8] M.-F. Chang, J. Lambert, P. Sangkloy. J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7744-8857, 2019.
[9] K. Cobbe, C. Hesse, Hilton, and Schulman. Leveraging procedural generation benchmark reinforce- ment learning. arXiv preprint arXiv:1912.01588. 2019.
[10] Comma. Openpilot. https: / /ttgtgtttbb.//////////// hatm:/comaa///pppppaaaca
2020.
[11] A. Corso, S. Katz, C. Innes, X. Du, S. Ramamoorthy, and M. J. Kochenderfer. Risk-driven design of 2022.
perception systems. Advances in Neural Information Processing Systems,
[12] E. Coumans and Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. https://docs.ooolleccom wttp://wwwwoeeee///e////e//////e 2016.
[13] W. Ding, H. Lin, B. Li, and D. Zhao. Causalaf: Causal autoregressive flow for goal-directed safety-critical scenes generation. arXiv preprint arXiv:21 10. 13939, 2021.
[14] W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao. A survey safety-critical driving scenario generation a methodological perspective. IEEE Transactions on Intelligent Transportation Systems, 2023.
[15] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[16] D. drive Contributors. DI-drive: OpenDILab decision intelligence platform for autonomous driving simulation.
com http//pennaabbbD-drve.... 2021.
[17] esmini. Enviromment simulator minimalistic (esmini). httpa: /ttp://gggttttttt cectiaub..cmmeeeee coon//ammni//emee
2019.
[18] S. Futinger, S. Cheng, Caine, C. Liu, H. Zhao, Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710-9719. 2021.
[19] L. Feng, Q. Li, Z. Peng, S. Tan, and B. Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. arXiv preprint arXiv:22 06609, 2022.
[20] T. A. Foundation. Autoware. https: / htap:::://gtttttt /gtthbb. cwm/comm//a/// httpa//toorrounaaaaiooooo www.onn/uuuuuureeeoooo 2016.
[21] D. J. Fremont, T. Dreossi. S. Ghosh, X. Yue, A. L. Sangiovami--Vincatelli, and A. Seshia. Scenie:
language for scenario specification and scene generation. In Proceedings of the 40th ACM SIGPLAN Conference on Progranming Language Design and Implementation, pages 63-78, 2019.
[22] D. J. Fremont, E. Kim, Y. V. Pant, S. A. Seshia, A. Acharya, X. Bruso, P. Wells. S. Lemke, Q. Lu, and S. Mchta. Formal scenario-based testing of autonomous vehicles: From simulation to the real world. In 2020 IEEE 23rd International Conference Intelligent Transportation Systems (ITSC), pages 1-8. IEEE, 2020.
[23] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018.

---

[24] A. Gambi, M. Mueller, and G. Fraser. Automatically testing self-driving cars with search-based procedural content generation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 318-328, 2019.
[25] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[26] J. Houston, G. Zuidhof, I., Bergamini, Y. Ye, I.. Chen, A. Jain, S. Omari, V. Iglovikov, and P. Ondruska. One thousand and one hours: Sclf-driving motion prediction dataset. In Conference on Robot Learning, pages 109-418. PMLR, 2021.
[27] C. Innes and S. Ramamoorthy. Testing rare downstream safety violations via upstream adaptive sampling of perception error models. arXiv preprint arXiv:2209.0900 2022.
[28] P. Kothari, C. Perone, L. Bergamini, A. Alahi, and P. Ondruska. Drivergym: Democratising reinforcement learning for autonomous driving. arXiv preprint arXiv:2111.066 10689999.9699..... 2021.
[29] E. Leurent. An enviromment for autonomous driving decision-making. https://githen com/ eleurent/hipe wwwwhhiggggggnn env, 2018.
[30] Q. Li, Z. Peng. Z. Xue, Q. Zhang, and B. Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. arXiv preprint arXiv:2109.1666 2021.
[31] E. Liang, R. Liaw, R. Nishihara, Moritz, Fox, K. Goldberg, Gonzalez, M. Jordan, and Stoica. Rllib: Abstractions for distributed reinforeement learning. In International Conference on Machine Learning, pages 3053-3062, 2018.
[32] W. LLC. Waymo open dataset: An autonomous driving dataset, 2019.
[33] S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall. Robot operating system 2: Design, architecture, and uses in the wild. Science Robotics, 7(66):eabm6074, 2022.
[34] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin. D. Hoeller, N. Rudin. A. Allshire,
A. Handa, et al. Isaac gym: High performance gpu-basee physics simulation for robot learning. arXiv preprint arXiv:2108, 10470, 2021.
[35] N. Montali, Lambert, Mougin, A. Kuefler, N. Rhinehart, M. Li, C. Gulino, T. Emrich. Z. Yang.
S. Whiteson, et al. The waymo open sim agents challenge. arXiv preprint arXiv:2.00. /2032, 2023.
[36] M. A. Z. Mora, M. Peychev, S. Ha, M. Vechev, and S. Coros. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning, pages 7805-7817. PMLR, 2021.
[37] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang. W. Paul, M. L. Jordan, et al. Ray: A distributed framework for emerging [AT] applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), pages 561-577, 2018.
[38] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[39] Z. Peng, K. M. Hui, C. Liu, B. Zhou, et al. Learning to simulate self-driven particles system with coordinated policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
[40] D. Rempe, Philion, I. J. Guibas, S. Fidler, and O. Litany. Generating useful accident-prone driving scenarios via a learned traffic prior. Proceedings of the TEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17355 2022.
[41] C. Schroeder de Witt, T. Gupta, D. Makoviichuk, V. Makoviichuuk, P. H. Torr, M. Sun, and S. Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv e-prints, pages arXiv-2011. 2020.
[42] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High digh-dimensional continuous control using generalized advantage estimation, 2018
[43] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 3:x3Yv/777.06344 2017.
[44] M. Schwall, T. Daniel, T. Victor, F. Favaro, and H. Hohnhold. Waymo public road safety performance data. arXiv preprint arXiv:201 100038, 2020.
[45] S. Shi, I.. Jiang, D. Dai, and B. Schicle. Motion transformer with global intention localization and local movement refinement. arXiv preprint arXiv:2209. 01.22299..350.... 13508, 2022.
[46] J. Song, H. Ren, D. Sadigh, and S. Ermon. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
[47] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2446-2454, 2020.
12

---

[48] S. Suo, S. Regalado, S. Casas, and R. Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10395-10404, 2021.
[49] D. Team. Deepdrive: a simulator that allows anyone with pe to push the state-of- ofte----rtttttttt in selvidriving.. https: h //tithubtbtt//tt com hetp//deediiee//eee www.vv/ddddeeeee.eeeeee
[50] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 1807.0668,, 2018.
[51] C. E. Tuncali, G. Fainckos, H. Ito, and J. Kapinski. Simulation based adversarial test generation for autonomous vehicles with machine learning components. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1555-1562. IEEE, 2018.
[52] L.. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[53] E. Vinitsky, A. Kreidich, L. Le Flem, N. Kheterpal, K. Jang, C. Wu, F. Wu, R. Liaw, E. Liang, and A. M. Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference on Robot Learning, pages 399-409, 2018.
[54] E. Vinitsky, N. Lichtlé, X. Yang, B. Amos, and J. Foerster. Nocturne scalable driving benchmark for bringing multi multi-agent agent learning one step closer to the real world. arXiv preprint arXiv:2206. XiV:22006009889, 2022.
[55] L. Wang, Y. Hu. L. Sun, W. Zhan, M. Tomizuka, and C. Liu. Transferable and adaptable driving behavior prediction. arXiv preprint arXiv:2202.05144 2022.
[56] L. Wang. J. Liu, Shao, W. Wang, R. Chen, Y. Liu, and Waslander. Efficient reinforcement learning for autonomous driving with parameterized skills and priors. arXiv preprint arXiv: 04412, 2023.
[57] T.-H. Wang, A. Amini, W. Schwarting. 1. Gilitschenski, S. Karaman, and D. Rus. Learning interactive driving policies via data-driven simulation. In 2022 International Conference on Robotics and Automation (ICRA). TEFF, 2022.
[58] Waymo. Waymo safety report. Waymo Safery Repor, 2021.
[59] B. Wilson, W. Qi. T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, et al. Argaverse 2: Next generation datasets for self-driving perception and forecasting, arXiv preprint ar 23Xx:2331.00099 2023.
[60] 7. Wu, T. Liu, I.. Luo, Z. Zhong, Chen, H. Xiao, C. Hou, H. Lou, Y. Chen, R. Yang, et al. Mars: An instance-aware, modular and realistic simulator for autonomous driving. arXiv preprint arXiv:2307. 15058, 2023.
[61] Z. Xie, J. Zhang, W. Li, F. Zhang, and L. Zhang. S-nerf: Neural radiance fields for street views. arXiv preprint arXiv:2303. 00749, 2023.
[62] C. Xu, W. Ding, W. Lyu. Z. Liu, S. Wang. Y. He, H. Hu, D. Zhao, and B. Li. Safebench: A benchmarking platform for safety evaluation of autonomous vehicles. arXiv preprint wxXiv:2200.0968) 2022.
[63] R. Xu, Y. Guo, X. Han, X. Xia, H. Xiang, and J. Ma. Opencda: an open cooperative driving automation framework integrated with co-simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1162. IEEE, 2021.
[64] Z. Yang. Y. Chen, J. Wang. S. Manivasagam, W.-C. Ma, A. Yang, and R. Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1389-1399, 2023.
[65] L. Yu, J. Song, and S. Ermon. Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning, pages 7194-7201. PMLR, 2019.
[66] H. Zhang, S. Feng, C. Liu, Y. Ding, Zhu, Z. Zhou, W. Zhang, Yu, H. Jin, and Z. Li. Cityflow: multi-agent reinforcement learning environment for large scale city traffic scenario. In The World Wide Web Conference, WWW 19, page 3620-3624, New York, NY, USA, 2019. Association for Computing Machinery.
[67] L. Zhang. Z. Peng. Q. Li, and B. Zhou. Cat: Closed-loop adversarial training for safe end-to-end driving. In 7th Annual Conference on Robot Learning, 2023.
[68] Z. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, Che, B. Ray, and M. Pavone. Guided conditional diffusion for controllable traffic simulation, arXiv preprint arXiv:2210.1 17366, 2022.
13

---

Appendix
In this appendix, we provide more details about the four experiments and some scenario examples from the three databases used in the experiments. Section A is on the scenario generation model; Section B is on the single-agent cross-dataset generalization experiment: Section C is on the multi- agent reinforcement learning and imitation learning; Section E shows the interface of ROS bridge and the qualitative results of running Openpilot test. Section shows more rendered scenario examples from each database.
Codebase, documentation, videos, and scenario gallery are available at https://nettadriverss gi github. io/ /wsssrienenneet
A Scenario Generation Model
We use all the databases (nuPlan, Waymo, PG) in scenarioNet for cross-dataset traffic scenario generation experiments. The goal is show that scenarioNet contains diverse traffic scenario data for training deep neural networks. We conduct our experiments based on TrafficGen [19], neural generative model previously developed for reactive traffic scenario generation.
A.1 Task Definition
A traffic scenario is denoted as 7 (m, $1:7), which lasts T time steps and contains the High- Definition (HD) road map m and the state series of traffic vehicles $1.7 = [s,,,, ST|. Each element { .f.s' is set of states of N traffic vehicles at time step t. Given existing scenario T = (m, SI.T). the goal of TrafficGen to learn to generate new traffic scenarios 7' (m, s' I.))) that have similar distribution with r and different states s' and longer time steps T", After training, TrafficGen takes m as input and generates 7' as a totally different scenario.
A.2 Model Architecture
Traffic Scenario Encoder. TrafficGen uses vectorization to encode map and vehicle information,
representing lanes as sets of vectors, each vector representing a small region. A vector- vector-based coordinate system is established for each small region, with each vector comprising start point pf. endpoint pf, and information about vehicles this region. Thus, traffic snapshot 7 at time step f is represented as T = v V Cross attention mechanism is applied to the unordered set v to fuse information from different regions into single context vector.
Decoder for Scenario Generation. TrafficGen places vehicles by generating a set of weights for all regions, which is then turned into a categorical distribution. The local position of a tentative vehicle modeled by a mixture of K bivariate normal distributions, as are heading, speed, and size of the vehicle. Autoregressive sampling is used to create traffic snapshot. A motion forecasting model is used as the trajectory generator.
A.3 Experiment Setting
We train scenario generation model TrafficGen with mixed data. Specifically, we randomly sample 30% data from nuPlan, Waymo, and PG. We filter out the scenarios with less than cars and crop a rectangular area with side length of 120m centered on the ego vehicle. Each of the 20s scenarios is split into 10 traffic snapshots with 2s intervals. The training is executed on servers with x Nvidia 2080TI and 256 G memory in Distributed Data-Parallel (ddp) mode. We set the feature size to be 1024, and use 3-layer MLPs with hidden dimensions of [1024, 512, 256] for attribute modeling. The training takes 16 hours for 100 training epochs and the learning rate decays by 20% at every 30 epochs. The detailed hyperparameters are shown in Table
We plot some generated traffic scenarios in Fig. and the dynamics in Fig. 8. shows that the models trained on the processed scenario data from ScenarioNet can generate realistic and diverse traffic behaviors and interactions.
14

---

Figure 7: Traffic scenarios generated from the model trained on different databases: Waymo (L),
nuPlan (L), and PG (L),
Original
1=60
t=120
Simulation
Figure 8: Dynamics of the generated traffic scenarios. The first column is the original case. The middle columns show the generated scenarios different timesteps. The last column shows the corresponding scenarios imported in the simulation. The green and red dashed lines indicate the traffic light status of this lane at the intersection.
15

---

Figure 9: t-SNE visualizations of 3000 scenarios. PG scenarios (L) are located mostly on the left side, while the scenarios from nuPlan (L) and the Waymo scenarios (L) are scattered the right side.
A.4 t-SNE Visualizations
The trained model's encoder is used to extract feature embeddings a given scenario sample. The embeddings then visualized with t-SNE method to show the similarities and differences. The detailed hyperparameters of t-SNE are shown in Table 5.
The t-SNE result is plotted in Fig. 9. The clustering results show that there is a large domain gap between real-world scenarios (Waymo and nuPlan) and synthetic scenarios (PG). Besides, domain gap exists even in real-world datasets. One prominent feature of the Waymo dataset is the complex crossroad (shown in the lower right corner), which the nuPlan dataset is lacking. The t-SNE visualization reveals the differences between different traffic datasets.
16

---

Table TrafficGen
Table 5: t-SNE
Hyper-parameter
Value
Hyper-parameter
Value
Batch Size
256
Components Number
2
Feature Size
1024
Training epochs
100
Init I Learning Rate
pea
Learning Rate
3e-4
auto
Perplexity
30
Activation Function
"relu"
Early Exaggeration
12
MCG Layers
B Single-agent Cross-dataset Generalization Experiment
We use the PG database and nuPlan database for cross-dataset generalization experiments. The goal is to investigate how the sim-to-real gap affects the generalizability of the learning- based vehicle controller. To this end, we train agents on synthetic PG scenarios [30] and nuPlan [6] scenarios respectively, and test them on the same held-out real-world test set.
B.1
Task Setup
To be specific, the task is to follow the trajectory of the data collection car and drive as fast as possible while avoiding collisions.
Observation. The observation of the RL agents is as follows:
1. A 120-dimensional vector denoting the like point clouds with 50m maximum detecting distance centering at the target vehicle. Each entry is in [0. 1] with Gaussian noise and represents the relative distance of the nearest obstacle in the specified direction.
vector containing the data that summarizes the target vehicle's 's state such as the steering, heading,
velocity, and relative distance the trajectory follow.
3. The navigation information that guides the target vehicle toward the destination. Concretely, consists of 10 points sampled the future trajectory and the distance between two consecutive points is 2m. The points will be projected to the vehicle coordinates.
12-dimensional vector denoting the Lidar-like point clouds with 50m maximum detecting the boundary of the drivable area, like the solid lines or sidewalks. (Optional)
As the vehicle for collecting nuPlan data in Boston sometimes drives out of the drivable area for bypassing the cones or barriers, crossing the drivable area boundary usually happens. Therefore, we didn't 't use the boundary detector in experiments, and crossing the boundaries like solid lines won't 't terminate the episode nor penalize the agent.
The driving policy is fully end-to-end model and directly controls the low-level throttle and steering angle. The action a is continuous two-dimensional vector with entries in [-1, By multiplying coefficients and clipping the extreme value, the action will be converted into the engine force and steering angle for changing the vehicle states.
Action.
Reward and Cost Scheme. The reward function is composed of four parts as follows:
R Ci Riiip + C2 Psmooth C2Pocoliisin Rierm
(I)
The displacement reward Raiep de d4-1, wherein the de and d,-1 denotes the longitudinal
movement of the target vehicle in Frenet coordinates of the target trajectory between two consecutive time steps, providing dense reward encourage the agent move forward. The smooth penalty Pemooth = min (0, 1/v,- [a[0]]) incentives the agent to drive smoothly and avoid a large steering value change between two timesteps, especially, when the velocity is high. v, and a[0) denote the current velocity and the steering value respectively. In addition, if a collision with a vehicle, human, or object happens timestep t. the agent will receive collision penalty. The penalty is set to Protlisioo = 2 when colliding with a human or vehicle and P Peoltision. = 0.5 for colliding with an object like cones and barriers. We also define a sparse terminal reward Reerm. which is non-zero only at the last time step. At that step, we set Rdiss Ropeed 0 and assign Raerm according to the terminal state. Rterm is set to +10 if the vehicle reaches the destination, -5 for being 2.5m away from the reference trajectory. We set e₁ 2, C2 and c3 1.
17

---

Termination Conditions and Evaluation Metrics. The episode will be terminated only when: 1) the agent drives 2.5m away from the reference trajectory 2) the agent arrives at the destination and 3) the agent can not finish the episode in recorded recorded_episadee_ spisade_length + 50 steps. For each trained agent, we evaluate it in the held-out test environments and define the ratio of episodes where the agent arrives at the destination as the success rate. The definition is the same for out of road and Timeout. For evaluating the driving behavior, the speed in each scenario is collected and then averaged across all scenarios. Also, a metric similar to the success rate is measured and called route completion, which is the ratio of moving distance to the length of the whole reference trajectory. Since each agent trained across 5 random seeds, this evaluation process will be executed for 5 agent which has the same training setting but different random seeds. We report the average and std on the metrics mentioned above.
B.2 Curriculum Training System
In the single-agent experiments, 40,000 scenarios used for training agents in both PG scenarios and nuPlan scenarios. Loading each scenario from scratch costs a significant amount of time, and thus we would like to buffer the scenarios in RAM for repeated use. However, the memory consumption is nonnegligible, especially when we train 5 policies concurrently and launch 20 workers to collect rollout for each policy. Assuming each scenario consumes about 10MB of memory, buffering 40,000 training scenarios in each worker process consumes 40,000 x 100x I memory, which requests a powerful and expensive cluster train agents in large-scale scenarios. We adopt curriculum training scheme for overcoming this issue, which reduces the memory by 99.9%.
of
Curriculum Training. We first sort the training scenarios according to the difficulty score calculated by: track track_length x cumulative collection A greater moving distance corresponds to a higher velocity, which in turn indicates a higher difficulty score. This value then will be multiplied by a weight cumulative, curvuturr which quantifies the level of bending in the trajectory. After this, scenarios can be divided into 100 levels with 400 scenarios in each level. Only when the Success Rate reaches 75%, the worker will move to the next level and release the memory used to store scenarios of the previous level. To further reduce memory usage, we split the scenarios in each level into 20 subsets, and each worker only loads scenarios from the corresponding subset. Finally, only 0.05% scenarios (20 scenarios) are actually loaded in each worker, which saves memory by a large margin.
cormulatvecuurvauure curvature. track_ _lenghh is the moving distance of the data
This curriculum training scheme not only makes it possible to finish training single server but boosts training efficiency and performance. We conduct an ablation study without the curriculum training. which costs a long time to train. This training scheme releases every scenario after using and reloading it when needed. The inferior performance highlights the necessity of having curriculum training.
B.3 Results
As shown in Fig. 10, we present experiment results for four training settings: nuPlan with curriculum training, nuPlan without curriculum training, PG with curriculum training, and Combine PG and nuPlan w/ curriculun training. The four used databases are PG-train, nuPlan-train, PG-nuPlan-train, and nuPlan-test, comprising 40,000, 40,000, 40,000, and 5,000 scenarios respectively. Considering nuPlan with curriculum training as the baseline, our analysis focuses on two factors: the data source and the inclusion of curriculum training. The evaluation of real-world scenarios underscores the significance of in-distribution real-world training data and the curriculum training system.
Actually, the poor performance of nuPlan without curriculum training is also exposed at the training stage. In spite of increasing the data coverage, the agents under this setting always have poor training success rate and training route completion throughout the whole training process. However, agents trained in nuPlan with curriculum training setting are more stable and have increasing training success rate as they can exploit the scenarios that properly match their current driving ability.
For investigating the influence of data sources, the conclusion can be drawn only from the test results which show that agents trained under PG with carriculum training can not generalize to real-world settings well. And the failure mode is that agents trained in synthetic scenarios can not learn to drive at high speed. And they don't move until other vehicles move far from them. This behavior reflected on the test speed as well. However, it doesn' 't mean that the synthetic data is useless. When
18

---

1.0
Pa
1.0
0.5
succes Rate
Route Completeon
0.6
Cenbrr cuntudar
0.8
0.6
0.4
nulPian ourtccuuun rulfinn curriatire
0.4
ruPPan our culam ruffan wis cumiaiui FG w/ suricuuumm
0.2
curroulum
0.2
Success
Out Rood
Timbouu
Speed
Trairing Timeshopa
PPO Training
100
1.0
nuPlon ou ountcclum
Cumictum Level
Data Coverag
Scoenero Oiftosky
6000
run carisutre
curbaatum
4030
40
0.4
suPler
2000
20
cumbouuum
0.
0.0
0.5
1.0
0.5
1.0
1.5
20
0.0
0.5
1.0
200
PPO Toining Timesteps
PPO Training Timestaps
PPO Training Timestpss
600
2000
The upper left figure shows the evaluation results for agents trained in different settings. For the remaining training details, we plot the results of three settings together and the other dataset individually. This is because training agents on Combine w/ carriculum takes more steps and rollouts to converge and reach the maximum level, compared training agents on another 3 datasets (2e7 steps 8e7 steps). The curriculam level calculated as the average worker level across seeds, with 20 workers per seed. Data coverage refers to the proportion of scenarios in the training set that the agent encounters throughout the entire training process. A value of 1.0 indicates that the agent has visited all scenarios. Scenario difficulty is determined by calculating the average difficulty score across all scenarios collected during PPO optimization epoch.
Figure 10:
agents are trained on the combined datasets, despite consuming more data, they can actually manifest the best performance. This is because the PG scenarios contain a number of curved roads and thus agents trained in these scenarios can have better ability follow curvature trajectory.
Another interesting phenomenon is the drop of training success rate when the scenario difficulty and curriculum level increase, which is also reported in [9] well. And this value finally coverage the test success rate. Therefore, we infer that given enough training scenarios, the test performance of learning-based agents can be roughly reflected by the training-time performance.
The hyper-parameters of the RL training are listed in Table 11.
Multi-agent Policy Learning
C.1
Experiment Setting
We can load the real-world dataset into MetaDrive simulator and create multi-agent interactive policy environment. We first instantiate agents in the environment where their initial states are loaded from the real-world dataset. The initial states include position, heading, velocity and the size. Then, we assign EnvInputPolicy to all agents and allow them to be controlled by external RL policies.
The ground-truth (GT) trajectories are not accessible to the learning agents but serve as the supervision via reward function (in RL) or observations (in IL).
Concretely, we create the reward function similar to Eq. I with slightly different weights:
R = Raiep + P Pooltision + Rterm-
(2)
The displacement reward Raisp = de d,-1, wherein the de and d,- denotes the longitudinal movement of the target vehicle in Frenet coordinates of the target trajectory between two consecutive time steps, provides a dense reward to encourage the agent to move forward. In addition, if a collision with a vehicle, human, or object happens at timestep t, the agent will receive collision penalty.
19

---

The penalty is set to Pooliision = when colliding with a human or vehicle. We also define sparse terminal reward Rterms which non-zero only at the last time step. At that step, we set Rdisp = Repeed = 0 and assign Rierm according to the terminal state. Reerm is set to +10 if the vehicle reaches the destination, -1 for being 10m away from the reference trajectory. We transform the GT trajectory into the simulator to create the target trajectory and use the displacement reward as the primary supervision from the dataset.
On the other hand, in multi- imitation learning we use the GT trajectory to form a dataset of observations and follow the setting of learning from observations or say action-free imitation learning. Specifically, for each frame in the scenario, we load the states of all actors at this frame into the simulator and utilize the sensor simulation functionality of MetaDrive to simulate the observations of each actor. The observation follows Sec. B. We form the dataset of observation sequences of all actors in the dataset.
The ground-truth trajectory can be used to measure the learned behaviors. The the route completion rate is the ratio between the length of projected agent trajectory and the length of GT trajectory. The average distance between agent trajectory and the GT trajectory is computed as follows:
T Σ T
p08agent,t posGT,
where 7' is the minimum length of agent trajectory and the length of GT trajectory. Therefore, for the agents that terminate quickly after spawning, the average distance will be quite small. The final distance is computed as distance between the last position of GT trajectory and the last position of agent trajectory. The cost the number of crashes of an agent in one episode. There are two terminal conditions. If the route completion rate exceeds 95%, the agent is marked successful. If the agent moves out of 10m aways from the reference trajectory, the agent is marked out of road and failed. We don't 't terminate agent's episode if it crash with other objects.
C.2 Baseline Details
MA-GAIL. We use GAIL [25] in multi-agent setting [46] but the discriminator distinguishes state- next state pair, instead of state-action pair [50]. We use PPO as the underlying RL algorithm in GAIL. The arper-parameter are listed in Table
MA-AIRL.
We also train multi-agent Adversarial Inverse RL [65] (AIRL) with an additional inverse dynamics model for estimating the expert actions. The inverse dynamics model is trained concurrently with the AIRL policies and learns the action given state-next state pairs from the environment interactions. AIRL will learn reward function and use the reward function build discriminator as GAIL.. We also use PPO as the underlying RL algorithm. The typer-parametent are listed in Table
MARL baselines. We train independent PPO [43, 41] and TD3 [23] agents as well as the Coordinated Policy Optimization (CoPO) agents [39] as MARL baselines. The hyper parameters are given in the next section.
20

---

D Hyper-parameters
Table 7: Action-free Multi-agent AIRI.
Table 6: Action-free Multi-agent GAIL
pappeerarrmeter
Value
Hyper-parameter
Value
Discriminator Loss LR
3e-4
Discriminator L2 Norm
le-5
Discriminator LR
5e-5
Discriminator SGD Num Iters
5
Discriminator L2 Norm
le-5
Discriminator SGD Minibatch Size
1024
Discriminator SGD Num Iters
1
Inverse Dynamics SGD Num Iters
100
Discriminator SGD Minibatch Size
1024
Inverse Dynamics SGD Minibatch Size
512
PPO SGD Minibatch Size
512
Inverse Dynamics LR
le-4
PPO Batch Size
2000
PPO SGD Minibatch Size
512
PPO LR
le-4
PPO Batch Size
2000
PPO SGD Num Iters
10
PPO LR
le-4
PPO Clip Parameter
0.2
PPO SGD Num Iters
10
PPO Lambda
0.95
PPO Clip Parameter
0.2
PPO Gamma
0.99
PPO Lambda
0.95
PPO Gamma
0.99
Table 9: CoPO
Table 8: Multi-agent PPO
Hyper-parameter
Value
Hyper-parameter
Value
LCF LR
le-4
LCF Num Iters
PPO SGD Minibatch Size
512
Neighborhood Distance
40 m
PPO Batch Size
2000
PPO SGD Minibatch Size
512
PPO LR
le-4
PPO Batch Size
2000
PPO SGD Num Iters
10
PPO LR
1e-4
PPO Clip Parameter
0.2
PPO SGD Num Iters
10
PPO Lambda
0.95
PPO Clip Parameter
0.2
PPO Gamma
0.99
PPO Lambda
0.95
PPO Gamma
0.99
Table 11: Single-Agent PPO
Hyper-parameter
Value
KL Coefficient
0.2
Table 10: TD3
X for GAE [42]
0.95
Discounted Factor
0.99
Hyper-parameter
Value
Number of SGD epochs
20
Critic LR
le-4
Train Batch Size
50,000
Actor LR
le-4
SGD mini batch size
200
Tau for Target Update
5e-3
Learning Rate
1e-4
Train Batch Size
100
Clip Parameter e
0.2
Activation Function
"tanh"
MLP Hidden Units
[512, 256, 128]
MLP Layers
3
21

---

E AD stack testing
E.1
ROS bridge
As shown in Fig. 11, We provide a ROS bridge along with the ScenarioNet, allowing users from the ROS community develop and test their systems with massive real-world data. As shown in Fig. 12, information like camera, lidar, and mid-level representations can be retrieved from the simulation.
NY
Oud Kgg
Meted Maam
Olub Malio defteess
Figure 11: The interface ROS bridge allowing connecting ScenarioNet and ROS community.
Image
Dapth
Top-down Rendering
Light
Lidar
Figure 12: Multimodal sensory data provided by ScenarioNet.
ScenarioNet provides not only mid-level scenario representations but multiple sensor outputs like top-down view, RGB camera, depth camera, and cloud points. The deluxe RGB camera is supported by the deferred rendering pipeline. This figure shows the ocene-0061 from nuPlan-mini-spli and its digital twin counterparts. It is worth investigating how to reconstruct meshes from the recorded lidar cloud points and images that can simulate the sensor output from views different from the recorded ones. Besides, it is promising topic to study the closed-loop sensor fusing and learning-based control together, which can be supported by ScenarioNet.
22

---

E.2 Qualitative Results of Openpilot test
Openpilot [10] is an end-to-end solution for driver assistance. Therefore, platform providing 3D rendering is necessary, if one would like to study or test the system. ScenarioNet is the only one that provides both real-world scenarios and 3D graphies support. As its navigation module is still a beta version, we only test Openpilot in scenarios without diverged roads. We build small database containing mainly lane-keeping scenarios for conceptually demonstrating that ScenarioNer can connect with commercial AD stack. As shown in Fig. 13, the Openpilot system is robust to common scenarios like turning right, lane-keeping, stopping traffic lights, and passing traffic lights. The demo video is available https: /youtuyuuuuuuuu be/ wwtee///pee (eeejjPPBOCCTg
Turning Right
17
17
17
17
17
t=00s
t=2s
tm4s
t=6s
t=8s
16
+
17
17
+
17
17
+
t=10s
t=12s
t-14s
t=16s
t=18s
Lane Keeping
24
+
24
24
24
24
t=0s
t=2s
[+4g
[+66
[-8a
24
+
24
25
+
24
24
+
I-10s
[=12s
[=14s
f=16s
(=18s
Stopping at Traffic Light
20
19
11
t=8s
0
+
(=10s
[=12a
[=16a
(-18a
Passing Traffic Light
20
t=0s
t=2s
t=6s
t=8s
21
21
in
25
Figure 13: Openpilot manages to overcome four representative scenarios.
23

---

Scenario Databases
In the database statistics Table, the Intersection Ratio is the ratio scenarios having traffic lights for real-world data, and the ratio of scenarios containing intersections and roundabouts for synthetic data.
F1 Waymo Database
Figure 14: Rendered traffic scenarios and their top-down views from the Waymo database.
We construct the Waymo database from the 20 seconds version [47] motion data with Google Cloud path opeen cetasset_motioo_____ 1 wwwpeeesssdsceearioioooo creeaaio/trainima 20s. As Waymo data is collected with the altitude calibrated, we can filter the overpass scenarios by excluding the scenarios with significant changes height. addition, we exclude the scenarios where the ego car waits at the red light for a long time by selecting scenarios where the ego car moving distances are greater than meters. We provide the rendered scenario examples from this dataset in Fig. 14. Top-down views show that Waymo scenarios contain diverse road structures and large number of vehicles.

---

F2
nuPlan Database
Figure 15: Rendered traffic scenarios and their top-down views from the nuPlan database.
According to https: h/wp::///ww... http://wwwwwccce..s/e/e/mm/ nwscenes.eeee...m..... org/nuplan, nuPlan has more than 1500 hours of driving data collected in 4 different cities: Las Vegas, Singapore, Pittsburgh, and Boston, here we only use the data collected in Boston as want keep all databases in experiments to have similar sizes. Data collected in other cities can also be converted to our scenario format.
We use the V.I.O version data, which contains approximately 50,000 scenarios after excluding scenarios where the ego car moving distances are less than 10 meters. It is noticeable that nuPlan updated the data version v1.1 recently (one week before the NeurlPS 2023 dataset track deadline), which may induce some differences if building a database from this new version. As shown in Fig. 15, the scenario examples reveal that nuPlan Boston split has cluttered scenes and contains many traffic cones and barriers besides vehicles. We additionally find that in some nuPlan scenarios, the ego car trajectory out of the drivable for sidestepping the barriers or cones.
25

---

F.3 PG Database
Figure 16: Rendered traffic scenarios and their top-down views from the PG database.
Unlike the previous two datasets collected the real world, PG scenarios are synthesized according set of rules. For map generation, two blocks are sampled from set of candidate roadblocks and connected to form a map. Those blocks include intersections, roundabouts, straight roads, curved roads, Ramp, and so on. Once the map is defined, traffic generation rule will be applied to scatter vehicles and road objects like traffic cones on the map. The detailed scenario generation config such as the block distribution for sampling can be found at https://github: httpr//eeeeeeeaaareeereooo/oooo. /ttp://///......0000000-00000000
metadrived wttpe///oooome//aaleeeeee waloeennt/aaaiaiaeeeeeeeeeiieiin whw.oothtttttittaeoo ocks saccks_prbb__eeee dist. dist..yhL4. All vehicles choose a destina- tion automatically and be actuated by IDM policy which can keep proper distance from the front vehicle and perform lane change when the front object is static. The scenario data collected by an IDM policy well. Some scenario examples shown in Fig. 16.
26

---

G Discussion: how to improve visual fidelity?
The visual fidelity of the closed-loop simulation can be further improved with the collected raw sensor data. We already have some plans to improve it and will include this discussion in the paper for sharing our thoughts regarding realistic sensor simulation.
Currently, the sensor simulation including the camera and lidar is achieved in Computer Graphics (CG) way, where people try restoring the mesh and texture for objects from real-world data and shading these models based on lights and materials to make them visually realistic. To improve the rendering results of ScenarioNet,
do plan reconstruct the geometry data for traffic participants and objects from the driving videos. Besides, we also consider connecting ScenarioNer with Unreal Engine Nvidia Omniverse the future as they could provide better shading results.
On the other hand, NeRF an alternative to improve the quality of sensor simulation. Through volume rendering, it can directly synthesize new camera views and point clouds from the driving videos when traffic participants and the ego car move with different trajectories and poses in the closed-loog loop training. This way is purely like objects and buildings. Recent results [61, 64, 60] already demonstrate its potential terms of camera and lidar simulation. However, how make the NeRF scene editable is still open problem; hence, we plan to investigate this in the future.
and can exempt the need for restoring 3D assets
27


Reviews:
[{"review_id": "li5ZspPHGW", "reviewer": "Reviewer_7uZv", "summary": "", "questions": "", "limitations": "The authors have discussed the limitations of their work in Section 5. They have also discussed potential negative societal impacts of their work in the same section.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper's main strength lies in its approach to addressing the data accessibility issue in autonomous driving and the platform's ability to ingest diverse real world data sources. By using real-world traffic scenarios, the authors have been able to benchmark the decision-making of the AD system and improve the generalizability of the data-driven planning and control components. The paper also highlights the importance of aggregating as many scenarios as possible, as solely relying on driving data from one region or one database is insufficient to cover all potential traffic situations. The authors have also provided a ROS bridge for connecting open-sourced AD stacks and tested Openpilot, an end-to-end AD system. \n\nThere is definite strength to this submission and the underlying concept as it opens up the possibility of creating open-ended simulations out of static datasets, and to enhance the diversity of scenarios depicted in the datasets. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "pF3cKPkVC1", "reviewer": "Reviewer_DPk1", "summary": "", "questions": "", "limitations": "The authors have built a framework to load data from various sources. Unfortunately, this data (to my understanding) seems to only be usable in the metadrive simulator. \n\nIt would be great to also allow an export to open standards such as OpenScenario and OpenDrive. Due to the open source nature of this work, this might also be appendable by third parties.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main contribution lies in the automated loading, converting and handeling of massive real-world scenarios from different sources.\nThis capability allows to utilize a vastly greater number of more realistic scenarios in ADAS/AD development and validation. It could also be applicable for other domains, e.g. in smart city planning, cooperative driving etc.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "S13wLIp2F7", "reviewer": "Reviewer_Wnbm", "summary": "", "questions": "", "limitations": "The limitation and societal impact are discussed. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The strength is the combined dataset to a uniform description and show case the benefit of combined dataset on performance evaluation. Future researchers can follow the guideline to provide dataset that's more universal and be used together. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "a14lpNRIBV", "reviewer": "Reviewer_KCPD", "summary": "", "questions": "", "limitations": "The value of the procedurally generated data has not been fully demonstrated. Section 4.3 shows how it unable to generalize to nuPlan's test dataset: PG training learns slower movement, has a higher timeout rate, and has a lower success rate. Ideally, it should be demonstrated that joint nuPlan-PG training has a synergistic effect and achieves better scores on at least some metrics than nuPlan alone.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors explain their framework in granular detail in the main text and technical appendix.\nThey have also provided qualitative and quantitative evaluations for some forecasting tasks.\nThe framework offers a closed loop evaluation with both nonreactive and reactive neighboring agents.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vbGu8qQ4S5", "reviewer": "Reviewer_Ei6A", "summary": "", "questions": "", "limitations": "There was not a clear discussion about the limitations, just a small paragraph about the potential lack of variety in the current 3D assets and a disclaimer about safety regulation. It seems short given the software engineering nature of this project, that no other possible bottlenecks in processing datasets were identified (i.e. parallelism, multi-GPU support, etc.) and also potential (but understandable) limitations related to the scope of this project (simulation, interactive visualization, etc.).\n", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "### Main strengths\n\n#### Framework\nThe implementation of ScenarioNet which is based on a unified scene description that serves as a bridge to utilize multiple datasets under a common framework. The framework is structured in a modular way, allowing users to start with just the data construction module and progress to running a 3D visualization of single or multi-agent training policies. This flexibility empowers researchers to adapt the platform to their specific needs.\n\n### Additional contributions in the paper:\n\n#### Benchmarks for Autonomous Driving Evaluation.\nThe ScenarioNet platform serves as a benchmark for evaluating the safety of autonomous driving systems in simulation before their real-world deployment. By using data-driven scenarios and diverse traffic behaviors from real-world datasets, researchers can assess the performance and robustness of autonomous driving stacks more effectively in a simulated environment.\n\n#### Advancements in Machine Learning and Autonomous Driving.\nThe authors demonstrate how ScenarioNet facilitates several research opportunities in machine learning and autonomous driving. It supports large-scale scenario generation, imitation learning, and reinforcement learning tasks in both single-agent and multi-agent settings.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Significance of Contributions: As clearly articulated by the reviews, the strength of the work is the development of a large-scale interactive scenario generation and benchmarking system for autonomous driving. The system provides enhanced data accessibility by allowing the collection traffic scenarios from real-world driving datasets (Waymo, nuScenes etc.), replaying them in simulation and allowing benchmarking of decision making in autonomous driving settings. A range of experiments include: 
training a large scenario generation model, conducting cross-dataset single-agent training and testing experiments, and benchmarking Openpilot, an end-to-end AD stack in the reconstructed scenarios.

Strengths:  Enhanced data accessibility and sampling from diverse real-world traffic scenarios
Benchmarking for generalizing data driven planning and control in autonomous driving
ROS bridge for connecting open-sourced AD stacks and tested Openpilot, an end-to-end AD system.
Possibility of creating open-ended simulations out of static datasets, and to enhance the diversity of scenarios depicted in the datasets.

Limitations: The authors have discussed the limitations of their work and they have also discussed potential negative societal
impacts of their work in the same section.

The rebuttal and discussions that follow are convincing and the reviewers are overall positive about acceptance of the paper.

Recommendation:  I recommend the paper for a poster. It may be appropriate also for a spotlight if the relative ranking of this paper is high compared to papers accepted.
Decision: Accept (Poster)