# Location-Awareness for Failure Management in Cellular Networks: An Integrated Approach
## 4 Evaluation
The fact of having the contextualized information leads to a better classification performance, as it can be seen by the reduction in the number of misclassified sample (those not in the main diagonal), which is translated to a better performance on the system. Going beyond this, and in order to fully evaluate the capabilities of the approach, the diagnosis is going to be evaluated applying three different classifiers implemented by CAFAIM: kNN, DISC, and ECOC. The automatic FE stage and the classification process are repeated 100 times for each of the possible configurations to obtain statistical significance. The samples are randomly divided in each iteration, while using the same data for all configurations. When considering the need of evaluation multiple iterations, confusion matrices are not the most feasible approach to represent the performance of the classification. Instead, the F1 score is used. This popular metric allows evaluating multi-class classifiers in terms of true and false positives (TP and FP) as well as false negatives (FN) with values up to 1. In the same way, this metric can also be expressed based on the precision and recall, with its expression being as below:  
\[\text{F1 score}=2\times\frac{\text{precision}\times\text{recall}}{\text{ precision}+\text{recall}} \tag{5}\]  
Both the precision and recall metric of the models are two primordial factors to take into account in this scope. On the one hand, precision will give the accuracy with which  
Figure 6: Confusion matrices. the model truly diagnoses the problem on the network. This is expressed as the proportion of predicted positives (TP and FP) that are truly positive (TP):  
\[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{6}\]  
On the other hand, recall will assess how the actual network status is correctly predicted by the model by representing the proportion of true labels (TP and FN) that have been predicted as true (TP). This can be expressed as the equation below. \[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{7}\]  
In this way, the arithmetic mean of all F1 score values obtained class-wise (denominated as the macro-averaged F1 score) provides a key figure of merit to compare different indicators + FE + classifier configurations [33; 34]. Therefore, Figure 7 shows the results of the macro F1 score, where each boxplot is generated from the 100 iterations performed for each configuration. Meanwhile, the execution times boxplots that are represented in Figure 8 are based on 10,000 executions (100 for each of the 100 iterations indicated before) of the inference block for each configuration in a \"high end\" personal computer (6 th Gen Intel Core i7-6700 (Quad Core 3.40 GHz, 4.0 GHz), RAM 16 GB 2400 MHz DDR4). The fact of using a joint set of Fusion NoFE improves the weighted F1 score of the models in comparison with Classic indicators, since it can be also seen that this improvement is greater with DISC classifier, as depicted in Figure 7. Nonetheless, in Figure 8, the increase of computational cost is seen, which this enhancement can carry out. Thus, it shows how the presence of a massive amounts of indicators can contribute to improving the diagnosis, if it will also require more time. Moreover, for Fusion Manual it is assessed how, contrary to the reasoning of the expert, the manual selection leads to highly degraded F1 score. Figure 7: Macro-averaged F1 score of the classifiers for the different configurations of classic and contextualized scenarios and feature engineering (FE) techniques. Nonetheless, the advantage of using FE techniques mostly lies in the models' execution time. The fact of applying them allows obtaining a huge reduction in terms of time for the model to predict, which is even lower than that obtained with the other configurations. In this way, taking both F1 score and time execution into account, NCA + PPCA achieves the best trade-off, which allows for positioning it as the best technique.