# LogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation
## 4 Experiment and Analysis
### Results and Analysis  
Since PCA, LogCluster, DeepLog, LogAnomaly, and LogBERT are unsupervised models and are not designed for cross-system detection, these models were evaluated in two cases; that is, the training dataset used or did not use the samples of the target system, represented by W/O. Firstly, the Thunderbird dataset was used as the source system, and the BGL dataset was used as the target system. This situation is abbreviated as TB-BGL. The \(F_{1}\) value of 0.880 and the AUC value of 0.880 were obtained on the source system Thunderbird, and the \(F_{1}\) value of 0.938 and the AUC value of 0.973 were obtained on the target system BGL. Then, the BGL dataset was used as the source system and the Thunderbird dataset was used as the target system. This case is abbreviated as BGL-TB. The \(F_{1}\) value of 0.933 and the AUC value of 0.978 were obtained on the source system BGL, and the \(F_{1}\) value of 0.841 and the AUC value of 0.854 were obtained on the target system Thunderbird. Tables 3 and 4 show the experimental results of the six methods except LogTransfer in the case of TB-BGL and BGL-TB. W/O represents the training set using or not using the target system. Firstly, PCA, LogCluster, DeepLog, LogAnomaly, and LogBERT do not use domain adaptation. For Thunderbird as the source system and BGL as the target system (TB\(\rightarrow\)BGL) or BGL as the source system and Thunderbird as the target system (BGL\(\rightarrow\)TB), these five methods could produce good \(F_{1}\) values and AUC values on the source system even if they did not use the sample training of the target system. When the training dataset used samples from the target system, they could obtain better \(F_{1}\) and AUC values on the target system, but worse values on the source system, indicating that these five methods do not have cross-system adaptive ability. Using a mixed log sequence of the source system and the target system would only make the training data distribution diverse, whereby the detection model would confuse the distribution of the samples, resulting in poor model detection. Compared with these five methods, LogBD achieved better results in any scenario. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Source**} & \multicolumn{2}{c}{**Target**} \\ \cline{2-5}  & \(\mathbf{F_{1}}\) & **AUC** & \(\mathbf{F_{1}}\) & **AUC** \\ \hline PCA W/O & 0.322/0.642 & 0.587/0.816 & 0.731/0.558 & 0.776/0.504 \\ LogClusterW/O & 0.530/0.713 & 0.746/0.829 & 0.677/0.559 & 0.716/0.504 \\ DeepLog W/O & 0.834/0.878 & 0.824/0.867 & 0.720/0.656 & 0.779/0.600 \\ LogAnomaly & 0.859/0.905 & 0.901/0.936 & 0.784/0.643 & 0.809/0.691 \\ LogBERT W/O & 0.882/0.926 & 0.915/0.954 & 0.809/0.709 & 0.828/0.733 \\ LogBD (ours) & **0.933** & **0.978** & **0.841** & **0.854** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The experimental results of six methods in the case of BGL-TB. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Source**} & \multicolumn{2}{c}{**Target**} \\ \cline{2-5}  & \(\mathbf{F_{1}}\) & **AUC** & \(\mathbf{F_{1}}\) & **AUC** \\ \hline PCA W/O & 0.689/0.760 & 0.698/0.779 & 0.577/0.229 & 0.773/0.658 \\ LogClusterW/O & 0.708/0.724 & 0.688/0.716 & 0.597/0.223 & 0.686/0.500 \\ DeepLog W/O & 0.757/0.790 & 0.701/0.777 & 0.627/0.223 & 0.843/0.500 \\ LogAnomaly W/O & 0.793/0.813 & 0.754/0.791 & 0.697/0.287 & 0.859/0.647 \\ LogBERT W/O & 0.821/0.847 & 0.805/0.857 & 0.732/0.349 & 0.851/0.682 \\ LogBD (ours) & **0.880** & **0.880** & **0.938** & **0.973** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The experimental results of six methods in the case of TB-BGL. On the source system, the detection performance of LogBD was still better than the five models, which indicates that LogBD captured the pain points of log anomaly detection, i.e., the accuracy of log template analysis, the use of log semantic information, and the method of anomaly detection.