# LogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation
## 4 Experiment and Analysis
### Experiment Setting  
#### 4.1.1 Dataset  
The experimental dataset comes from a large log dataset loghub [26] released by the intelligent operation and maintenance team of the Chinese University of Hong Kong. The total size of the maintained log data set is more than 77 GB. The data were specially collected for log analysis and research. Some of the logs were production data released in previous studies, while others were real data collected from laboratory systems. The logs in the dataset came from various types of systems, including HDFS, BGL, Thunderbird, and OpenSSH. This paper used BGL and Thunderbird datasets for experiments:  
* The BlueGene/L (BGL) dataset, which comes from the BlueGene/L supercomputer system of Lawrence Livermore National Labs, contains 4,747,963 logs, including 4,399,494 normal logs and 348,469 abnormal logs.
* The Thunderbird (TB) dataset, from the Thunderbird supercomputer system of Sandia National Labs, contains 3,737,209 logs, including 3,015,004 normal logs and 722,205 abnormal logs.  
The statistics of the two datasets are shown in Table 1.  
Simple outlier detection is not needed for group logs because a single abnormal event is considered to be an independent anomaly. However, deep learning models are used to reveal abnormal models of multiple log events, such as changes in event sequences or time log correlations. Therefore, it is necessary to organize log events into groups and then analyze them individually or interrelated. There are two common grouping methods. One is the sliding time window, where a log will appear in multiple windows. The other is a fixed time window, where a log can only appear in one group. In order to ensure the similarity of the experimental environment, a sliding window of 20 was used to cut the log file into a short sequence with a step size of 4.  
After sliding window segmentation, the BGL dataset was divided into 1,175,002 log sequences. When BGL was used as the source system, there were 60,000 normal log sequences for training, 1,014,762 normal log sequences for testing, and 100,240 abnormal log sequences for testing. When BGL was used as the target system, 1000 normal log sequences were used for training, 1,073,762 normal log sequences were used for testing, and 100,240 abnormal log sequences were used for testing. The Thunderbird dataset was divided into 1,126,852 log sequences. When Thunderbird was used as the target system, there were 1000 normal log sequences for training, 803,783 normal log sequences for testing, and 322,069 abnormal log sequences for testing. When Thunderbird was used as the source system, there were 60,000 normal logs for training, 744,783 normal log sequences for testing, and 322,069 abnormal log sequences for testing, as shown in Table 2.  
In the training phase, 60,000 normal sequences from the source system and 1000 normal sequences from the target system were mixed as the model training set.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Total** & **Normal** & **Abnormal** \\ \hline BGL & 4,747,963 & 4,399,494 & 348,469 \\ TB & 3,737,209 & 3,015,004 & 722,205 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset information.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Dataset** & **Normal Log Sequence** & **Abnormal Log Sequence** \\ \hline BGL & 1,074,762 & 100,240 \\ TB & 804,783 & 322,069 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The number of log sequences after sliding window division.