# Measures and Methods for the Evaluation of ATO Algorithms
## 3 Approach for ATO Testing
### Key Performance Indicators
In [4], \"the need to the ATO function of achieving a centimetre-level accuracy for specific operations\" is outlined. For stopping at stations with PSD a precision of less than 10 cm is seen as a prerequisite. As ATO systems shall contribute to a more efficient drive, characterized by reduced energy demand, measuring the total energy consumption is essential. The total consumption can be separated into different components such as traction and subsystem energy consumption as well as regenerative braking energy. The most important component of the overall energy use is the consumption of traction energy which is influenced by multiple parameters. Those influences can be traction resistance, air resistance, efficiency rates of system components and others. In this paper, the energy consumption is calculated based on the data available within the simulation log files of the simulator. Therefore, the energy consumption is calculated per time step and summed up for each track section. In general, the following equations apply:  
\[F_{\text{trac}}=m\cdot(a-a_{\text{r}}), \tag{1}\] \[E=\sum_{i=1}^{n}\frac{F_{\text{trac},i}\cdot v_{i}+F_{\text{trac },i+1}\cdot v_{i+1}}{2}\cdot\Delta t. \tag{2}\]  
where \(F_{\text{trac}}\) is traction force in N, \(a_{\text{r}}\) is the coasting resistance [m/s\({}^{2}\)] determined by the simulator based on an empirical Equation (_Davis equation_), \(a\) is the actual resulting acceleration [m/s\({}^{2}\)] and \(m\) is the train mass in kg. In Equation (2), \(E\) is the total energy consumption in [kWh], \(v\) is the actual speed in [m/s] and \(\Delta t=t_{i+1}-t_{i}\) is the duration of the time step and \(n\) is the number of time steps. Although ATO does not include the execution of safety-critical tasks, the performance of the connected algorithms can be evaluated by measuring safety-related indicators. These measures are well-accepted values and are commonly in use when examining rail operations. * **Yellow Signal Approaches:** number of times driving up to a yellow signal,
* **Yellow Signal Passages:** number of times passing yellow signals,
* **Red Signal Approaches:** number of times driving up to a red signal,
* **Red Signal Stops:** number of times stopping at red signals,
* **Minimal Differences to ETCS braking curves:** minimal differences between the actual speed and the speed defined by the indication, permitted, warning, service brake indication and emergency brake indication speed curves of the ETCS on-board unit. These values can be compared with a reference run on the evaluated route under standardized (optimal) conditions, allowing a statement to be made about the quality of the calculated speed profile. Furthermore, they can also serve as indicators for capacity analyses, whereby further interpretation is necessary. The indicators describe the number of approaches toward yellow and red signals as well as counting the passages of yellow signals and stops at red signals. They provide information on how unhindered a train can pass a track section. In addition, approaching a yellow or particularly a red signal is always a situation that involves a higher risk than passing a signal that indicates to proceed. By calculating the minimal differences between the actual speed and the ETCS braking curves, information is obtained about how close the algorithm comes to forced braking or to what extent speed restrictions are observed. With a continuous evaluation of these values (for each time step), more in-depth analyses would be possible. In addition to the aforementioned KPIs, customer satisfaction is also influenced by other factors related to comfort. One possibility to determine (parts of) this subjective feeling is to calculate the physical values of acceleration and jerk. Thereby it is possible to get a statement about how comfortable or uncomfortable a drive is. Based on relevant literature [37; 38; 39], ranges for the acceleration for the defined comfort levels are introduced (Table 2). The absolute value of jerk should not exceed \(3\frac{\text{m}}{\text{s}^{3}}\).