# LogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation
## 3 Methodology
### Anomaly Detection Based on Domain Adaptation
#### 3.4.1 Feature Extractor  
LogBD trains the anomaly detection model using only normal log data. Let D be a dataset consisting of two parts: normal log messages \(\text{D}^{\text{S}}=\left\{\text{L}_{\text{i}}\right\}_{\text{i=1}}^{\text{Max} _{\text{s}}}\) of the source system and a small number of normal log messages \(\text{D}^{\text{T}}=\left\{\text{L}_{\text{i}}\right\}_{\text{i=1}}^{\text{ Max}_{\text{T}}}\) of the target system. After generating the log template vector representation using the pretraining model BERT, each log template is converted to a semantic vector representation. A log sequence is represented by \(X=\left\{\text{x}_{n}\right\}_{1}^{n}\), where \(\text{x}_{n}\) denotes the vector representation of the n-th log message. Inspired by the Deep SVDD method, LogBD's feature extractor maps log sequences to hypersphere space [35], which can be processed by neural networks. Deep neural networks are better at extracting the depth information contained in log data than traditional machine learning methods. Since log data carry temporal information, many scholars treat them as time-ordered sequences and use RNNs to learn log the temporal relationships between log entries. Although RNNs establish connections in temporal order with some memorability, they are prone to gradient disappearance or gradient explosion. [PERSON] proposed an LSTM network based on RNN. Compared with RNN, LSTM introduces a new temporal chain on the basis of RNN. Long-range dependencies can be better captured using the LSTM model because it possesses a gating mechanism to increase or decrease the information of cell states, and ultimately decide which information to use. However, LSTM has some problems; it cannot operate in parallel, only one step can be processed in one operation, and latter steps must wait for the previous steps to be processed before performing the operation. Keeping the intermediate results is very memory-consuming. Therefore, LogBD uses a TCN [8] with better performance and less memory consumption as the feature extractor \(\text{G}_{\text{f}}\). As shown in Figure 6, a vector representation of log sequences is input to the TCN for encoding and processed to obtain the log sequence mapping v. The i-th sequence  
Figure 5: The framework of DANN. from the source system is mapped as \(\rm v_{i}^{S}\), and the i-th sequence from the target system is mapped as \(\rm v_{i}^{T}\). For the feature extractor \(\rm G_{f}\), to have all log sequences mapped into a hypersphere space, we first set the center of the hypersphere to which all log sequences of the source and target systems are mapped as the mean value of all log sequences in the training dataset. The formula is as follows:  
\[c=Mean\left(\sum_{i=1}^{Max_{c}}\
u_{i}^{e}\right),\epsilon\in\{S,T\}, \tag{1}\]  
where S denotes the source domain or source system, and T denotes the target domain or target system. Since each log sequence is represented by the feature extractor as \(\rm v=G_{f}(x;\rm\phi_{f})\), the distance of the log sequence mapping to the center of the hypersphere sphere is \(\rm v_{i}^{e}-c\). In order to make the normal log sequence mapping close to the center of the hypersphere c, the distances of the log sequence mapping to the center of the sphere all need to be close to the minimum, setting the objective function of the feature extractor as follows:  
\[\mathcal{L}_{ex}=\sum_{\epsilon\in\{S,T\}}\sum_{i=1}^{Max_{c}}\|\
u_{i}^{e}-c \|^{2}+\frac{\lambda}{2}\sum_{l=1}^{L}\left\|W^{l}\right\|_{F}^{2}. \tag{2}\]  
By minimizing this formula, all normal log sequences can be kept close to the center. The \(ex\) in the formula represents the feature extractor, the first term in the formula makes all log sequences mapped as close to the center of the sphere as possible, and the second term is regularized.