## 6 Metrics for TSN Schedule Evaluation
### Evaluation Metrics
In the following, we list some of the ones used in the literature as examples:  
* Load balance (expressed in [33, 59] as slot occupancy, and in [34, 40, 75, 87, 96] as link/network utilization);
* Bandwidth efficiency( expressed in [90] as bandwidth loss and in [80, 91] as bandwidth utilization);
* Load on the bottleneck (expressed in [88] as the number of available flows);
* Frequency of preemption [95];
* Cost of topology [30, 126];
* Extensibility of the schedule [12] (measures the deviation in the duration of unused spaces in the schedule);
* Porosity of the schedule [52] (measures the spacing between HTS windows to allow LTS and BE transmission);
* Security level [104];
* Quality of control ( expressed in [85, 107] as update duration and expressed in [86] as execution jitter of control application);
* Reconfigurable depth( expressed in [67] as the number of successful global reconfigurations because of core failure before the reconfiguration operation is unfeasible);
* Frame loss/drop [83, 107];
* Path length [53];
* Queue usage/count [32, 44, 46, 57, 74, 118]. Figure 11: Classification of evaluation metrics. The evaluation of the schedule quality is a very important step before applying the configuration of the TSN elements to the network. This operation could be performed in the Central Network Controller (CNC) (e.g., in the TSN microservice according to the architecture presented in [89]). However, those metrics are not criteria for comparison between different scheduling algorithms since the schedule evaluation is use case-dependent. Consequently, different algorithms could perform better with some specific use cases but not with others. The evaluation of the scheduling algorithm/problem formulation could be based on metrics including:  
* Optimality gap of the solution in relation to the objective function [74, 112];
* Scalability [15, 29, 37, 41, 50, 116, 124] (expressed in [116] as the number of generated constraints and in [37] as runtime with regards to the number of flows);
* Runtime for a given problem complexity determined by the size of the topology and the number of streams to schedule [11, 14, 15, 29, 30, 31, 32, 33, 35, 37, 38, 40, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 57, 58, 59, 60, 62, 65, 65, 66, 67, 68, 70, 74, 76, 77, 78, 82, 84, 86, 87, 91, 95, 96, 97, 98, 101, 104, 105, 106, 107, 109, 110, 112, 113, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126];
* Scope of applicability of the algorithm based on its assumptions (e.g., algorithms with very restrictive assumptions are not applicable in the general case but for specific use cases);
* Resource utilization (e.g., CPU, memory, etc.) [61, 109, 119];
* The schedulability (ratio of scheduled streams per traffic type) under variable traffic loads [14, 29, 31, 32, 35, 40, 42, 45, 46, 47, 49, 58, 60, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 77, 85, 87, 93, 95, 96, 100, 102, 105, 107, 112, 113, 117, 118, 120, 121, 122, 123, 124, 125, 126];
* Settling time [85];
* Probability of failure per hour [102];
* Algorithm convergence [82]. The evaluation model that we suggest in this section could be used as a reference to assess the depth of the evaluation. Although most of the works use metrics from the first and second levels, we suggest giving more importance to the determinism of communication in the evaluation and consider it the second most important after the satisfaction of the hard constraints.