## 4 Evaluation
### Failure Detection Evaluation--Results and Analysis  
We conducted experiments with different datasets to evaluate, in terms of Accuracy, Recall, Precision, FAR, and F-Score, several candidate algorithms for realizing FDM; this mechanism tags the detected outliers and generates the tagged outliers dataset that is the FRM's input. This evaluation aims at selecting an algorithm for carrying out FDM. In this sense, we define the best performance condition for the outliers detection algorithm: high Accuracy, high Precision, and low FAR. In particular, we vary the parameters from each algorithm described in Section 3.3 as follows: for DBSCAN, the eps (e) parameter from 0.1 to 0.8 (with min_samples set in 3); for IF, the contamination (c) parameter from 0 to 0.5; and, for SVM, the nu parameter from 0.01 to 0.1. We evaluated the algorithms using the datasets described in Table 2 aggregating to each of them 1%, 5%, and 10% of outliers. Table 5 compares in the Per_Hour dataset the candidate algorithms for performing FDM. DBSCAN and IF obtained the same high performance in any outliers' percentage, meaning that these algorithms can operate correctly with a small dataset and few outliers. SVM obtained the worst performance; in this dataset with 10% of outliers, this algorithm got 33% and 80% of FAR and Recall, respectively, which indicates that SVM did not find all the outliers and present many false positives. These results are because SVM ignores the spatial correlation of neighboring nodes, which makes the results of local outliers inaccurate [37]. Table 6 compares in the Per_Day dataset the candidate algorithms for carrying out FDM. All evaluated algorithms' performance decreased in this dataset (12 times bigger than the Per_Hour dataset). The three evaluated algorithms obtained their best performance in this dataset with 1% of outliers, confirming that they function well when operating with a small dataset and few outliers. Overall, DBSCAN obtained the best results for the evaluated metrics. However, note that this algorithm got 7% and 3.5% of FAR when operating with 5% and 10% of outliers in the dataset, respectively, \begin{table}
\begin{tabular}{l l l} \hline \hline  & **Predicted Outlier** & **Predicted Normal** \\ \hline
**Actual outlier** & True Negative (TN) & False Positive (FP) \\
**Actual normal** & False Negative (FN) & True Positive (TP) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Confusion matrix. meaning in the datasets mentioned, the outliers are closer to standard data (DBSCAN has problems to identify these types of outliers). SVM obtained excellent results in almost all evaluated metrics but slightly lower than DBSCAN did. In addition, FAR in SVM increased up to 6.9 when the dataset contains 10% of outliers. The IF algorithm got excellent results regarding Precision, F-Score, and FAR. However, its Accuracy and Recall are around 97% and 90% when this dataset contains 5% and 10% of outliers, respectively. This algorithm had problems with false positives. Table 7 compares in the Per_Month dataset the candidate algorithms for performing FDM. All evaluated algorithms obtained excellent results regarding Accuracy, Recall, Precision, and F-Score. Accuracy and Precision in DBSCAN decreased slighter to 99.7% and 99.4% when the percentage of outliers moved from 5% and 10%, respectively. In this dataset, DBSCAN also had problems with false positives (FAR = 5.6 with 10% of outliers), indicating limitations to identify outliers closer to normal data. Regarding SVM, it is to highlight that its false positives increase when the percentage of outliers is equal to 10% (FAR = 7.09), which suggests inefficiency in classifying many outliers. FAR obtained by IF is perfect because the contamination parameter used was very close to the real percentage of outliers existing in the dataset; this is difficult to know in practice for an online operation. Based on the results above, we consider DBSCAN as the best option for carrying out FDM.