# LogEDL: Log Anomaly Detection via Evidential Deep Learning
## 5 Experiment and Analysis
### Results and Analysis  
#### 5.2.1 Comparison Methods  
Our method LogEDL is compared with traditional machine learning and deep learning approaches. The comparative experiments are shown in Table 3; traditional anomaly detection methods, such as PCA, Isolation Forest, and OCSVM, demonstrate suboptimal performance across the three datasets. Although these methods sometimes achieve high _precision_ or _recall_, for instance, PCA's _recall_ reaches 100% on all datasets, their _F1 score_ remain exceedingly low. This indicates an inability to balance _precision_ and _recall_ effectively, likely due to their failure to capture the temporal sequence information inherent in log data, resulting in inadequate anomaly detection capabilities. LogCluster, specifically designed for log anomaly detection, shows improvements over traditional methods, particularly on the HDFS and BGL datasets. However, its performance on the Thunderbird dataset remains unsatisfactory, with _F1 score_ failing to reach high levels. DeepLog and LogAnomaly, two deep learning-based methods, significantly outperform the traditional methods and LogCluster, showcasing the advantages of deep learning in capturing log sequence patterns. Both methods achieve high _F1 score_ across the three datasets, especially excelling on the HDFS and Thunderbird datasets. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**HDFS**} & \multicolumn{2}{c}{**BGL**} & \multicolumn{2}{c}{**Thunderbird**} \\ \cline{2-9}  & **Precision** & **Recall** & **F1** & **Precision** & **Recall** & **F1** & **Precision** & **Recall** & **F1** \\ \hline PCA & 5.89 & 100.00 & 11.12 & 9.07 & 98.23 & 16.61 & 37.35 & 100.00 & 54.39 \\ iForest & 53.60 & 69.41 & 60.49 & 99.70 & 18.11 & 30.65 & 34.45 & 1.68 & 3.20 \\ OCSVM & 2.54 & 100.00 & 4.95 & 1.06 & 12.24 & 1.96 & 18.89 & 39.11 & 25.48 \\ LogCluster & 99.26 & 37.08 & 53.99 & 95.46 & 64.01 & 76.63 & 98.28 & 42.78 & 59.61 \\ DeepLog & 88.44 & 69.49 & 77.34 & 89.74 & 82.78 & 86.12 & 87.34 & 99.61 & 93.08 \\ LogAnomaly & 94.15 & 40.47 & 56.19 & 73.12 & 76.09 & 74.08 & 86.72 & 99.63 & 92.73 \\ LogBERT & 87.02 & 78.10 & 82.32 & 89.40 & 92.32 & 90.83 & 96.75 & 96.52 & 96.64 \\ LogBERT (EVT) & 29.24 & 69.01 & 41.07 & 28.11 & 73.05 & 40.60 & 30.78 & 74.24 & 43.51 \\
**LogEDL (ours)** & 90.06 & 92.80 & **91.41** & **99.84** & 97.26 & **98.53** & 97.80 & 98.08 & **97.91** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance comparison of LogEDL and baseline methods. LogBERT, a method based on pre-trained models and self-supervised learning tasks, demonstrates superior performance across all three datasets, with \(F1\)_score_ markedly higher than those of traditional methods and other deep learning approaches. LogBERT successfully models normal log sequences and accurately identifies anomalous sequences through self-supervised pre-training tasks. Extreme value theory (EVT) [36], also known as the Fisher-Tippett Theorem, is used as a simple evidence theory for anomaly detection based on the LogBERT model. However, since it relies on empirical thresholds, the results are not ideal. Our proposed method, LogEDL, surpasses all other methods across the three datasets, exhibiting exceptional performance. On the HDFS dataset, LogEDL achieves an \(F1\)_score_ of 91.41, significantly higher than LogBERT's 82.32. On the BGL dataset, LogEDL's \(F1\)_score_ of 98.53 substantially exceeds LogBERT's 90.83. On the Thunderbird dataset, LogEDL's \(F1\)_score_ of 97.91 outperforms LogBERT. This demonstrates that LogEDL maintains high _precision_ and _recall_ while achieving the best \(F1\)_score_. The comparative analysis indicates that LogEDL outperforms other methods in log anomaly detection tasks. Traditional methods (PCA, Isolation Forest, OCSVM) exhibit significant shortcomings in balancing _precision_ and _recall_, resulting in low \(F1\)_score_. The specialized method LogCluster shows some improvements but still lags behind deep learning methods.