# Measures and Methods for the Evaluation of ATO Algorithms
## 3 Approach for ATO Testing
### Key Performance Indicators  
Key performance indicators (KPI) are intended to provide a better understanding of whether an algorithm can fulfil the performance requirements under certain conditions in specific operating situations. They represent a universal and flexible metric, also allowing the comparison of multiple algorithms. To gain a concise representation we suppose a multilevel structure, i.e., a top-level KPI is characterized by several indicators that can be measured or calculated and thus represent a quantitative value that allows further interpretation and comparison. These given KPIs are punctuality, energy and safety and are commonly used in the railway sector when evaluating train operation. The same applies to the sub-level indicators of these KPIs. Within the literature of other projects in the context of testing ATO systems like [8], similar KPIs are used. Furthermore, accuracy, comfort and capacity were mentioned as relevant measures and are included in the presented approach. Table 1 is showing the complete list of the selected KPIs. The calculation of the KPIs is performed for each track section, defined by a departure and arrival station (or point). This shall provide an additional grade of detail for the subsequent analysis of the measures. A more condensed evaluation is possible by determining each KPI for the complete journey of the train under test. The KPI \"Punctuality\" seeks a detailed evaluation at multiple locations during a train journey. The comprehensive analysis of punctuality is of particular importance due to its significant operational impact. Besides the obvious evaluation points _departure_ and _arrival_, passing points are part of the scenario definition. These are characterized by the fact that the train does not stop at such locations. For each point, a passing time is defined which allows an evaluation of the punctuality at this specific point of the journey. Emerging delays can be identified and possibly traced back to their origin. Moreover, it is possible to assess whether and how well the ATO algorithm can reduce given delays. Passing points can also be _critical timing points_ (CTP). CTPs are vital points within a timetable equipped with a time window that is characterized by the earliest and latest passing time. These time windows define a frame in which a train must pass the CTP in order to not negatively influence the timetable. Both delays and early arrivals are to be avoided. As a consequence, they can have a great influence on punctuality as well as on capacity as explained later. The accuracy or precision of an ATO system is mainly defined by the accuracy of the stopping process and the resulting deviation of the specified and actual stopping position. \begin{table}
\begin{tabular}{c l l} \hline \hline
**KPI** & **Indicator** & **Unit** \\ \hline \multirow{3}{*}{Punctuality} & Departure deviation & [s] \\  & Passing point deviation & [s] \\  & Arrival deviation & [s] \\ \hline \multirow{3}{*}{Accuracy} & Stopping position deviation at station & [m] \\  & Stopping position deviation at red signals & [m] \\ \hline \multirow{3}{*}{Energy} & Energy consumption & [kWh] \\ \hline \multirow{3}{*}{Safety} & Yellow signal approaches & - \\  & Yellow signal passages & - \\  & Red signal approaches & - \\  & Red signal stops & - \\  & Minimal differences to ETCS braking curves & [km/h] \\  & (vInd, vPerm, vWarn, vSBI, vEBI) & \\ \hline \multirow{3}{*}{Comfort} & Max. Acceleration & [m/s\({}^{2}\)] \\  & Max. Jerk & [m/s\({}^{3}\)] \\ \hline \multirow{3}{*}{Capacity} & Meeting time windows & Pass / Fail \\  & Area enclosed by vPerm and actual speed curve & - \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of KPIs for the evaluation of the performance of ATO algorithms. A stop is not only necessary at stations but also in front of red signals. Thus, two indicators are introduced. If using PSD in higher GoA, the acceptable range of deviation of the stopping position is significantly lower.