# Measures and Methods for the Evaluation of ATO Algorithms
## 4 Demonstration of Approach
### Simulation Results  
For demonstration purposes and to evaluate the use of the introduced approach, nine simulation runs as part of two logical scenarios have been performed. Their results have been analyzed accordingly. As intended, the KPIs were calculated based on the logging data of the simulation environment. In addition to the effects that could be expected from the introduction of \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layer** & **Entity** & **Property** & **Parameter Range** \\ \hline
6 & Timetable & Reduction of available running time & [0..6] min \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Description of the entities and their parameters relevant for testing the influence of different timetables on the ATO performance. Only showing relevant definitions differing from the base scenario. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layer** & **Entity** & **Property** & **Parameter Range** \\ \hline \multirow{3}{*}{6} & \multirow{3}{*}{Timetable} & Reduction of available running time & [0..6] min \\  & & Moving TW & [False, True] \\ \cline{3-4}  & & Varying TW length & [False, True] \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Description of the entities and their parameters relevant for testing the influence of different timetables and the variation of TWs of CTPs on the ATO performance. Only showing relevant definitions differing from the base scenario. restrictive boundary conditions in the timetable, important insights were gained about the simulation environment and the ATO algorithm itself. Within Figure 5 the KPI values for different concrete scenarios of logical scenario one are shown exemplarily. Logical and expected results, e.g., punctuality, can be identified. Other KPIs need to be evaluated in a greater context. For example, the values of the accuracy KPI vary greatly from one concrete scenario to the other. This variation can be attributed to inaccuracies in the simulation environment, which resulted in a non-optimal stopping process. Regarding maximum acceleration and jerk levels, measured values for the available scenarios are within the defined limits. Energy consumption falls within the expected ranges as well. In the case of the capacity evaluation, a measurement area from the beginning of the track up to a speed restriction area before the station of Amsterdam Zuid was selected. This measurement allows determining how the algorithm tries to reduce the travel time to a minimum by following the allowed maximum speed as closely as possible. Within the base scenario, enough running time is available. Thus, it is not necessary to drive at full speed leading to a greater area enclosed by the permitted and actual speed curve. When the algorithm tries to achieve a time-optimal drive, the area enclosed by the two curves should be minimal, since it tries to maintain the maximum permissible speed as closely as possible. How the capacity KPI is evaluated in this approach might not provide all information important for capacity determination. Because the scope of the simulator is limited to a microscopic assessment, the capacity evaluation is adapted to this characteristic. If analysis on a network scale was possible, more general statements potentially of greater relevance and significance may be drawn. In Figure 6 visualization of the speed profile including relevant restrictions like the maximum allowed track speed and ETCS emergency braking curve is displayed. Additionally, track data like station locations and height are given. With this, a more convenient and easier evaluation of the KPIs is possible by providing context to the plain numbers. Figure 5: _Cont._  
The scenarios are illustrating how the algorithm is reacting to a set of challenging timetables. This is observable by the change in the KPI values. Additionally, the speed plots provide viable information to further interpret the KPIs for example for a more in-depth analysis of weaknesses of the algorithm under test or to understand better the reason for certain specific behaviours.