# Measures and Methods for the Evaluation of ATO Algorithms
## 5 Conclusions  
With the approach presented in this paper, two set goals are achieved, namely identifying a way to efficiently test ATO algorithms in a virtual environment and enabling the comparison of multiple algorithms based on logical measures. Its application may allow train operators and infrastructure managers to select the best suiting ATO systems according to their requirements. The approach consists of universal KPIs taken from the literature and supplemented with sub-level indicators to measure the performance of ATO algorithms, a framework for the structured definition of test scenarios taking advantage of the layer model for the railway sector as well as different scenario types, and the identification of performance relevant factors. The KPIs introduced for this approach can provide valuable information about the performance of an ATO algorithm in specific situations and scenarios. It proved difficult to strike a balance between the need for further interpretation of the values and providing  
Figure 5: Selection of plots comparing indicator values for concrete scenarios of logical scenario one. (**a**) Arrival delay at stations. (**b**) Stopping accuracy at stations. (**c**) Energy consumption per track part. (**d**) Capacity score. (**e**) Maximum acceleration per track part. (**f**) Maximum jerk per track part. Figure 6: Additional plots generated by the evaluation tool for the original base scenario simulation of logical scenario one. quantified metrics that are comparable when scenario definitions change. Calculating the KPIs for each departure and arrival pair results in a more or less extensive list of values, depending on the length of the scenario. If the KPIs were determined only for the entire run, a simpler but less detailed overview of the performance for a given scenario would be possible. Furthermore, it is necessary to provide a range or overview of reference values for some indicators to be able to classify the measured quantitative values. This proved to be particularly difficult for the comfort indicator since the classification of these values is not commonly realized. Within the approach, different scenario types are introduced. These are helpful to infer from general test cases to exactly defined, complete, and thus also simulatable scenarios in a structured manner. However, the proposed approach only abstractly defines those scenario types. This decision was made due to the specific demands of each simulation software concerning the scenario definition. If more standardization for the design and exchange of scenarios is achieved, an improved and standardized concept for delineating the different types of scenarios will also be beneficial and easier to establish. With the layer model, a foundation is laid, based on the definition of entities and their properties. Another critical aspect of the approach is the identification of factors potentially having an impact on the ATO performance. Their extraction from the literature leads to an unequal grade of detail. What is meant by this is that one factor can have a variety of impacts on the performance of ATO systems due to the different effects it induces. Influences of several causative constituents of railway operation were partially combined, or the causing entity itself was included as a factor. This was done to achieve the conciseness of the overview but resulted in redundancies. The list can certainly be extended and umbrella terms might be divided into more specific impacts on ATO. During the identification of the factors having a potential impact on the ATO algorithm performance it was ascertained, that the layer model is applicable for defining the simulation scenario, the entities within those scenarios, and also including the definition of potential influencing factors of these entities. In contrast to that, it does not provide a possibility to allocate factors directly related to the train under test, the so-called _ego train_.