# Measures and Methods for the Evaluation of ATO Algorithms
## Literature Review
### Testing in Other Modes of Transportation
#### 2.3.3 Scenario-Based Testing  
Performance and safety assessment is often done by evaluating specific predefined test cases also referred to as scenario-based testing. These scenarios describe relevant situations within traffic during the operation of an AV. Only critical or challenging situations are selected. However, it is not possible to identify all potentially occurring traffic scenarios due to an indefinite combination of external influences as mentioned in [20]. Scenario-based testing shows to be a promising virtual test strategy.  
In [21], authors from the industry and scientific research proposed a first taxonomy for testing an Advanced Driver Assistance System (ADAS). To successfully carry out virtual validation of systems and software components, references based on collected data need to be defined. In [21], two ways to retrieve ground truth information are listed:  
* Reference by measurement in real-world traffic,
* References based on simulation.  
The authors conclude that both are necessary to validate technical and software-based components of ADAS. The _references_ are used to verify the outcome of test _scenarios_, which can be defined based on test criteria. Therefore, _metrics_ are necessary to quantify the results of the test scenarios. The combination of these three elements yields a taxonomy for testing ADAS. It is outlined that test scenarios can be defined as either knowledge-driven or data-driven. Secondly, the scope of the scenario is different, either being used to evaluate the functionality or safety for specific use-cases with case-by-case evaluation or traffic-based, providing an overview of the system's overall impact and revealing unwanted side-effects (see also [22]).  
In the field of avionics, scenario testing is used during the verification and evaluation of subsystems. In [23], a vision-augmented landing technology for general aircraft is presented. Simulations were used for functional validation and performance testing of essential algorithms of the system within a self-developed simulation environment. The performance was checked for \"integrity, accuracy, availability and continuity\" to reach common aviation standards ([24]). In [25], an automatic flight path controller for general aircraft was developed and described. Besides the description of the model, the paper also discusses the various testing stages that were carried out. First, model-in-the-loop testing provided the necessary model and functional validation. Hardware-in-the-loop and aircraft-in-the-loop tests were used for functional and implementation validation of the system. Performance tests were performed again by flight tests, also evaluating turbulent situations. Unmanned aerial vehicles make also use of automatic flight functionalities that allow an autonomous operation. In [26], a combination of different simulation sources isproposed to provide a testing environment where seamless transitions are possible. Using live data, virtual and constructed entities enable multiple inputs that can be handled in the same way by the system under test without the need to change the simulation environment.