# Selecting Non-Line of Sight Critical Scenarios for Connected Autonomous Vehicle Testing
## 2 Related Work
### Automotive Software Testing  
There are several methods for assessing the safety capabilities of AVs. These include: formal verification [25; 26; 27; 28], a function-based approach [29], shadow mode [30; 31; 32], real-world testing [33; 34; 35], a simulation-based approach [36; 37; 38; 39; 40] and a scenario-based approach [41; 42; 43; 44; 45]. Throughout the literature, the most common approaches used are real-world testing, the simulation-based approach and the scenario-based approach [46]. Real-world testing is the most realistic of the validation methods, but it necessitates more effort and comes at a higher cost due to the high level of realism required. Furthermore, the authors [PERSON] and [PERSON] [47] stated that driving 11 billion miles is required to prove statistically with 95% confidence that an autonomous vehicle is 20% better than a human driver (based on the failure rate). They concluded that in the real world, the traditional \"miles driven\" approach is not economically viable. Another issue with this statistical approach is that it fails to account for how eventful the miles driven are. For example, an AV can complete the miles requirement by driving on a relatively simple road. To address this, companies in California are now required to publish \"disengagement reports\" as per California State law. Disengagements are situations in which human intervention is required during an automated operation due to an unsafe decision made by an AV due to the criticality of the situation or a technological failure [48]. AVs can be tested through simulations with varying levels of fidelity instead of being tested in real-world settings. Dedicated software with a simple or complex mathematical representation of the subsystems can be used depending on the purpose. In April 2020, Waymo announced that it had driven 20 million miles on public roads and 15 billion miles in its simulator [49]. Tesla's Dojo supercomputer replays recorded data in order to retrain its full self-driving software stack [38]. Scalability and cost-efficiency are two advantages of simulation testing. Simulation models, on the other hand, must be tested in the real world. Nonetheless, AV testing can be accelerated by combining simulation and real-world testing. Scenario-based testing focuses on specific scenarios, such as safety-critical scenario cases, to reduce testing time and effort by generating proof of correct functionality using a scenario coverage matrix rather than relying on the statistical distribution used in real-world testing. In the PEGASUS project, scenario-based testing was used to test ADAS systems. The ability to use only the data that are required is a significant benefit of scenario-based testing. Several proposed methods, including clustering, rule-based and machine learning-based methods, are used to identify representative scenarios [41; 42; 43]. Other methods focus on categorising scenarios based on their rarity. Scenario-based testing, in which individual traffic scenarios are tested using virtual simulation, is a promising method [50; 51]. It has a number of advantages. It can be significantly faster than other approaches because it does not require real-time execution and testing can be used in parallel. Another significant advantage of this method is the assurance of safety during testing. Critical accident scenarios can be simulated in a virtual environment without endangering real-world traffic participants. However, because the simulated world deviates from the real world at times, model-world mismatches can occur. The primary goal of reproducing failure scenarios after they have occurred is to reduce as many failures as possible ahead of time. Although there has been some research looking at test case generation using scenario-based methods, none of these focused on incorporating scenarios that included blind spots, i.e., the presence of obstacles that are not in direct sight of the onboard sensors.