# Time Series Anomaly Detection for KPIs Based on Correlation Analysis and HMM
## 4 Experiment
### Performance Evaluation of Anomaly Detection  
After the construction of the KPI correlation matrix, alarm clustering is employed to filter out KPIs with low similarity or no exception in the known period. We then train the HMM, and finally infer the most likely implicit state corresponding to a KPI. In order to verify the effectiveness of the anomaly screening model proposed in this paper, the detection effects of Isolation Forest, One-Class SVM, and this model are tested on the 2018 AIOps [36] Challenge dataset and the 2021 AIOps Challenge dataset, named Dataset 1 and Dataset 2, respectively. We use precision, recall, and F1-score to evaluate the performance of anomaly screening. TP is the number of abnormal points correctly detected, and FP is the number of normal points wrongly identified as abnormal points. FN is the number of abnormal points wrongly identified as normal points. Table 10 shows the F1-scores of the three methods. Compared with the other two methods, our evaluation results are the best, indicating that our method achieves better results. However, the experimental results show that there are still false positives in this method. Moreover, because Dataset 2 contains a large number of indicator types and quantities, its F1-score will be lower than that of Dataset 1. Without the addition of a correlation analysis method, a wide variety of indicators in the system will affect the results. This shows the importance of correlation analysis when confronted with a large quantity of diverse data. Figure 8: Heat map of KPI correlation matrix: (**a**) correlation matrix for the first set of KPIs; (**b**) correlation matrix for the second set of KPIs. In order to verify the generalization ability of 1D-CNN-TCN, the model is tested by 5-fold crossover validation, and the evaluation index is accuracy. In the experiment, we use the 2018 AIOps challenge data, and divide the KPI data into five equal scores. One of these is taken as the test set each time, without repetition, and the other four are used as the training set to train the model. Finally, the specific accuracy values are obtained; please see Table 11.  
It can be seen from Table 11, after 5-fold crossover validation, that the accuracy of the model in this paper is in the range of 0.97-0.99, and the average value is 0.9824. It shows that the model has good generalization ability. However, we can see from the comparison between F1-score and accuracy that there are still some false positives in the anomaly screening model in this paper. Therefore, correlation analysis is needed to reduce the number of false positives of abnormal KPIs. In Table 12, if the final correlation between \(X\) and \(Y\) is higher than the threshold, then \(X\sim Y\); otherwise, \(X\sim Y\). If \(X\sim Y\), we continue to judge the temporal order. If \(X\) affects \(Y\), that is, \(X\) shows an anomaly before \(Y\), then \(X\to Y\); otherwise, \(Y\to X\). The F1-score (see Formulas (14)-(16)) is used to evaluate the performance of anomaly detection in this paper. Table 12 describes FP (false positive) and FN (false negative) in related issues of anomaly recognition. The experimental results of anomaly detection in this paper are shown in Table 13.  
According to the evaluation results of F1-score shown in Table 13, the optimal F1-scores of anomaly detection and temporal order (based on the two groups of correlation matrices) are 0.94 and 0.90, respectively, indicating that this method has a good effect on anomaly detection. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Isolation Forest** & **One-Class SVM** & **1D-CNN-TCN** \\ \hline Dataset 1 & 0.8432 & 0.8064 & 0.8602 \\ Dataset 2 & 0.8356 & 0.7807 & 0.8534 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: F1-score evaluation results of different methods.