# LogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation
## 4 Experiment and Analysis
### Results and Analysis
It can also be seen that the performance of deep learning methods was better than that of machine learning methods, indicating that the machine learning model only used the log template count vector as the input feature, without considering the log content itself. Machine learning can detect the abnormal information in the log to a certain extent, but it cannot achieve good accuracy and coverage of anomaly detection. For example, PCA is based on the log template index for anomaly detection, only retains the main features of the original data, and loses a lot of key information, making it difficult to learn features from the sparse count matrix. LogCluster is based on clustering for log anomaly detection, but it cannot play a good role in the face of complex log structure, nor can it fully learn the features in the log, and the detection effect is not good. DeepLog regards the log sequence as a digital sequence and replaces the log template with a number. It not only uses the log parameter features but also integrates the log sequence features. However, it does not extract the semantic information in the log template, and it can easily treat the log sequence that has not appeared in the training data as an exception, resulting in lower accuracy and more false alarms. Compared with the machine learning methods PCA and LogCluster, it has a more obvious improvement. The LogAnomaly method uses the semantic and grammatical information of the log template, and then introduces Template2 Vec for the synonyms in the log. It uses the word vector weighted average to obtain the vector representation, which makes some improvements on the basis of DeepLog. However, it does not consider the problem of polysemous words, only considers the representation of a single word vector, and does not consider the context information; additionally, the learned feature information is not comprehensive enough. LogBERT uses BERT to capture the pattern of normal log sequence, and uses two self-supervised task training models, mask log template prediction and hypersphere minimization. LogBERT uses the hypersphere objective function as in LogBD, but the performance is not as good as LogBD, because LogBD uses domain adaptation to obtain more data with the same characteristics for training. LogTransfer is a supervised transfer learning method that uses normal and abnormal labeled data of the source system and the target system to train a cross-system anomaly detection model. LogTransfer achieves good performance when sufficient tag data are available. In this experiment, we tested how many labeled abnormal samples are needed to train LogTransfer for it to have similar performance to LogBD. When using 100 abnormal sequences from the source system and 10 abnormal sequences from the target system to train LogTransfer, LogTransfer could achieve the best performance on the source system. The detection results in the two scenarios are shown in Tables 5 and 6. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Source**} & \multicolumn{2}{c}{**Target**} \\ \cline{2-5}  & \(F_{\mathbf{1}}\) & **AUC** & \(F_{\mathbf{1}}\) & **AUC** \\ \hline LogTransfer & **0.971** & 0.972 & 0.792 & 0.828 \\ LogBD (ours) & 0.933 & **0.978** & **0.841** & **0.854** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The case of BGL-TB. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Source**} & \multicolumn{2}{c}{**Target**} \\ \cline{2-5}  & \(F_{\mathbf{1}}\) & **AUC** & \(F_{\mathbf{1}}\) & **AUC** \\ \hline LogTransfer & **0.981** & **0.975** & 0.788 & 0.833 \\ LogBD (ours) & 0.880 & 0.880 & **0.938** & **0.973** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The case of TB-BGL. For the TB\(\rightarrow\)BGL scenario, training LogTransfer with 10 abnormal sequences of the target system was not enough to be better than LogBD. It was better than LogBD on the source system, but worse on the target system.