# MDFULog: Multi-Feature Deep Fusion of Unstable Log Anomaly Detection Model
## 4 Experiment and Analysis
### Result Analysis
#### 4.3.2 Unstable Dataset  
We trained the evaluated models on the original HDFS and then tested the trained model on the synthetic test set (the injection ratio is the proportion of unstable log injection, respectively 5%, 10%, 15%, and 20%). The average results of multiple experiments are shown in Table 2.  
With the increase in the proportion of unstable logs, the F1-scores of PCA, DeepLog, LogRobust, ROEAD, and DeepSyslog decreased significantly, but the F1-scores of MDFULog did not change significantly. Even when the proportion of unstable logs was as high as 20%, the MDFULog model could obtain a high F1-measure (more than 0.95). MDFULog classifies log messages of the log sequence by calculating the semantic similarity, and the noise injection log rules will not significantly impact the semantics of log messages. Increasing the ratio of infused unstable logs will not significantly impact the F1-measure of MDFULog's anomaly detection. The traditional method cannot handle the instability log data created during the system update process and the noise generated during the processing. However, MDFULog can maintain higher performance, which confirms its robustness. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{2}{c}{**HDFS**} & \multicolumn{2}{c}{**OpenStack**} \\ \hline  & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline PCA & 0.86 & 0.68 & 0.759 & 0.51 & 0.68 & 0.583 \\ DeepLog & 0.89 & 0.93 & 0.910 & 0.78 & 0.81 & 0.795 \\ LogRobust & 0.93 & 0.96 & 0.945 & 0.89 & 0.81 & 0.848 \\ ROEAD & 0.96 & **0.98** & 0.970 & 0.94 & 0.94 & 0.940 \\ DeepSyslog & 0.96 & 0.95 & 0.955 & 0.87 & **0.97** & 0.917 \\ MDFULog & **0.98** & **0.98** & **0.96** & 0.96 & **0.960** \\ \hline \hline \end{tabular} Note: The best results are highlighted in bold.  
\end{table} 
Table 1: Results of comparing MDFULog with the baseline methods on stable datasets. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Injection Ratio** & **Model** & **P** & **R** & **F1** \\ \hline  & PCA & 0.85 & 0.66 & 0.743 \\  & DeepLog & 0.86 & 0.79 & 0.824 \\
5\% & LogRobust & 0.93 & 0.91 & 0.920 \\  & ROEAD & 0.96 & 0.95 & 0.955 \\  & DeepSyslog & 0.92 & 0.9 & 0.910 \\  & MDFULog & **0.98** & **0.97** & **0.975** \\ \hline  & PCA & 0.78 & 0.64 & 0.703 \\  & DeepLog & 0.75 & 0.58 & 0.654 \\
10\% & LogRobust & 0.89 & 0.9 & 0.895 \\  & ROEAD & 0.95 & 0.92 & 0.935 \\  & DeepSyslog & 0.89 & 0.81 & 0.848 \\  & MDFULog & **0.98** & **0.98** & **0.980** \\ \hline  & PCA & 0.73 & 0.42 & 0.533 \\  & DeepLog & 0.75 & 0.51 & 0.607 \\
15\% & LogRobust & 0.89 & 0.88 & 0.885 \\  & ROEAD & 0.90 & 0.86 & 0.880 \\  & DeepSyslog & 0.88 & 0.72 & 0.792 \\  & MDFULog & **0.96** & **0.96** & **0.960** \\ \hline  & PCA & 0.72 & 0.37 & 0.489 \\  & DeepLog & 0.70 & 0.42 & 0.525 \\
20\% & LogRobust & 0.90 & 0.89 & 0.895 \\  & ROEAD & 0.85 & 0.83 & 0.840 \\  & DeepSyslog & 0.87 & 0.69 & 0.770 \\ MDFULog & **0.95** & **0.97** & **0.960** \\ \hline \hline \end{tabular} Note: The best results are highlighted in bold.  
\end{table} 
Table 2: Results for comparing MDFULog versus the baseline methods on unstable datasets. The Precision of DeepLog on unstable datasets declined from 0.86 to 0.7. The factor affecting this is that DeepLog treats certain regeneration normal logs as anomalies. The F1-measure of PCA, DeepLog, LogRobust, ROEAD, and DeepSyslog decreased by 25%, 29%, 2%, 11%, and 14%, respectively. After injecting noise into a stable dataset, models (such as PCA) trained on the initial dataset dealt with the changing events as novel templates, bringing error alerts. Although LogRobust had a low F1-measure drop ratio, the overall F1-measure was low because of its coarse parsing of semantic information. On the contrary, MDFULog's Precision, Recall, and F1-measure did not drop significantly. The reason is that the MDFULog model uses the Bert model embedded with contrastive learning to learn the semantic vector of the log, which can better handle the updated log.