DiaVio: Safety Violation Diagnosis for Autonomous Driving Simulation Testing

Overview

Simulation testing has been widely adopted by leading companies to ensure the safety of autonomous driving systems (ADSs). A number of scenario-based testing approaches have been developed to generate diverse driving scenarios for simulation testing, and demonstrated to be capable of finding safety violations. However, there is no automated way to diagnose whether these violations are caused by the ADS under test and which category these violations belong to. As a result, great effort is required to manually diagnose violations.

To bridge this gap, we propose DiaVio to automatically diagnose safety violations in simulation testing by leveraging large language models (LLMs). It is built on top of a new domain specific language (DSL) of crash to align real-world accident reports described in natural language and violation scenarios in simulation testing. DiaVio fine-tunes a base LLM with real-world accident reports to learn diagnosis capability, and uses the fine-tuned LLM to diagnose violation scenarios in simulation testing. Our evaluation has demonstrated the effectiveness and efficiency of DiaVio in violation diagnosis.

Methodology

DiaVio fine-tunes a base LLM with real-world accident reports, enabling it to diagnose safety violations in simulation testing.

Integration with Testing Approaches

DiaVio is integrated with two state-of-the-art open-source scenario-based testing approaches:

AV-Fuzzer: Setup with Apollo 8.0 ADS and the SORA-SVL simulator.
DriveFuzz: Setup with Behavior Agent ADS and the Carla simulator.

"loras" Folder

The "loras" folder contains fine-tuned models based on the Llama2 series chat models. These models are trained using real traffic accident data from the DiaVio method.

How to Use

Researchers and practitioners in autonomous driving can leverage DiaVio for:

Automated diagnosis of safety violations in simulation testing.
Integration with scenario-based testing approaches.
Fine-tuning language models for safety scenario analysis.

Citation

If you use DiaVio in your work, please cite our related publication.