## 6 Metrics for TSN Schedule Evaluation
### Evaluation Metrics  
Different aspects can be used in order to evaluate a TSN schedule that determines the network configuration. While we argue that determinism is the most important feature that is added by the TSN standards, more attention should be given to it when formulating the problem. For the evaluation of a schedule, determinism is also a very important criterion. However, most of the surveyed approaches do not really evaluate the quality of the resulting schedule. As shown in Table 5, the top two most occurring evaluation metrics are runtime and schedulability, while jitter is at number six in the ranking. In this section, we suggest a set of evaluation metrics for TSN scheduling. As shown in Figure 11, we consider four different metric classes. The first and most important one concerns the evaluation of the schedule against the traffic requirements. This is because a schedule is considered feasible and can be applied only if it satisfies the QoS requirements of the streams. The second level of evaluation metrics concerns the quality of the schedule in relation to its determinism. The more deterministic the schedule is, the better the guarantees that the network can give to the users. The next level of evaluation treats the objective specific to the work. As different works target the optimization of different objectives, it is important to evaluate how optimal the solution is according to the chosen objective function. The three previous levels of evaluation consider the performance of the algorithm according to its output. The fourth level evaluates the algorithm independently of its output. In the following, we present a noncomprehensive list of metrics that could be used in each level of evaluation. Figure 10: Examples of the slot slope problem. \begin{table}
\begin{tabular}{l c} \hline \hline
**Metric** & **Occurrence in Literature** \\ \hline Runtime & 70 \\ \hline Schedulability & 45 \\ \hline Latency & 40 \\ \hline Objective specific & 33 \\ \hline Resource utilization (memory, CPU, link/bandwidth/GB, network) & 19 \\ \hline Jitter & 10 \\ \hline Scalability & 7 \\ \hline Schedule length & 5 \\ \hline Optimality & 3 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Frequency of occurrence of metrics in the surveyed literature. The requirements of the use case could be expressed as constraints for the problem formulation. The following metrics could be used to evaluate them:  
* Worst-case latency [11, 15, 24, 29, 33, 35, 36, 37, 40, 46, 47, 48, 49, 53, 61, 62, 70, 72, 73, 75, 76, 80, 81, 83, 86, 87, 88, 91, 97, 99, 100, 104, 105, 106, 111, 116, 118, 119, 123];
* Worst-case jitter [29, 41, 53, 62, 80, 81, 84, 86, 87, 104];
* Maximum schedule length [37, 38, 90, 97]. Afterward, the evaluation of the schedule should be against the optimization objectives. Metrics evaluating the primary objective, i.e., determinism, include:  
* The strictness of temporal and spatial isolation;
* The probability of slot slope;
* The probability of frames/streams interleaving;
* Average jitter [41, 62]. Metrics at the third level could be very different as they evaluate the secondary objectives that are specific to each work.