ADT-CSE
Creators: name: Song, Qunying
name: Avner, Bensoussan
name: Mohammad, Reza Mousavi
Publisher: name: Zenodo
Year: 2024

description: ADT-CSE is short for Autonomous Driving Testing – Critical Scenario Evaluation, and is designed to evaluate the realism of a synthetic critical scenario set (i.e., DeepScenario [2][3]) by comparing them to real-world critical scenarios (from DMV California [1]) for testing autonomous driving systems.





DMV (Department of Motor Vehicles) California [1] collects and publishes collision reports submitted by autonomous vehicle manufacturers permitted to test on public roads. These reports, available as standard PDFs, contain information on real-world disengagements and collisions. Initially, we retrieved 536 reports since 2018, after filtering for readability and quality of information.




DeepScenario [2][3] is an open driving scenario set generated using Apollo ADS and SVL simulator. DeepScenario employed two weather (sunny, rainy), lighting (day, night), and four roads in San Francisco, and optimized for critical scenarios with respect to three objective functions (distance to object, jerk, and time to collision), by using three strategies (random, greedy search, and reinforcement learning). Originally, we retrieved all generated scenarios, including 1050 collision scenarios.



ADT-CSE contains two main sub-directories, namely data (the two scenario sets used), and tools (Python scripts) developed for evaluating and analysing the two data sets, in the following structure. ADT-CSE requires Python 3.7 (3.7.4 is used and suggested) or higher to run scripts in tools directory. Also, check tools/requirements.txt for all dependencies required and their suggested versions.

 



├───data
├──────original                                 # original data (33530 scenarios for DeepScenario & 536 reports for DMV California)
├─────────DeepScenario
├────────────....
├─────────DMV_California
├────────────....
├──────selected                                 # selected collision scenarios (1049 for DeepScenario & 282 for DMV California)
├─────────DeepScenario
├────────────....
├─────────DMV_California
├────────────....
├──────extracted                                # attributes extracted from selected collision scenarios and stored in csv files
├─────────DeepScenario.csv
├─────────DMV_California.csv 
├──────vectorised                               # vectorised version of extracted attributes in csv files
├─────────DeepScenario.csv
├─────────DMV_California.csv
├───tools
├──────analysis
├─────────multiple_attribute_comparison.py      # multiple attribute (e.g., weather and collision type) distribution analysis
├─────────single_attribute_comparison.py        # single attribute (e.g., weather) distribution analysis
├─────────distance_comparison.py                # distance-based comparison
├──────extraction                               
├─────────DeepScenario.py                       # attributes extraction from DeepScenario scenario description files
├─────────DMV_California.py                     # attributes extraction from DMV California collision reports
├──────util
├─────────DeepScenario_2DReconstruction.py      # 2D reconstruction of DeepScenario scenario description files    
├─────────DMV_California_CRTemplate.py          # collision report template for DMV California data
├─────────pca_helper.py                         # analyse and plot variance captured by target dimensions
├─────────vectorisation.py                      # vectorisation of extracted scenario attribute data
├──────requirements.txt                         # libraries required to run scripts in this directory


 


ADT-CSE selects 1049 collision scenarios for DeepScenario and 282 collision reports for DMV California from data/original/ directory, and has placed them in data/selected/ directory. It uses corresponding scripts (i.e., DeepScenario.py, DMV_California.py) in tools/extraction/ to extract attributes from the selected data and stored them as csv files. The csv files are reviewed and manually compensated by the authors for, e.g., missing fields in DMV California collision reports, and final results are stored as DeepScenario.csv and DMV_California.csv in data/extracted/ directory. Then, the extracted data is vectorised using tools/util/vectorisation.py and stored in data/vectorised/ directory. Finally, the vectorised data is analysed using the scripts in data/analysis/ for single attribute analysis, multiple attribute analysis, and distance-based comparison.



References


[1] California Department of Motor Vehicles. Autonomous vehicle collision reports. Available online at: https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/ (last accessed: September 1 2023).

[2] Chengjie Lu, Tao Yue, Shaukat Ali. DeepScenario: An Open Driving Scenario Dataset for Autonomous Driving System Testing. Zenodo, Mar 2023. DOI: 10.5281/zenodo.7714194

[3] Chengjie Lu, Tao Yue, Shaukat Ali. DeepScenario: An Open Driving Scenario Dataset for Autonomous Driving System Testing. 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pp.52-56, 2023.