# LogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation
## 3 Methodology
### Feature Extraction Based on BERT  
The log templates are textual data, while the data processed by machine learning and deep learning algorithms are usually fixed-length numerical inputs. The log templates must, therefore, be converted into vectors for computer processing. The word embedding method is used to obtain the semantic vectors.  
During the development of natural language processing, word embeddings have undergone an evolution from static to dynamic [27]. Static word embeddings have evolved in two broad stages. In the first stage, mainly sparse and high-dimensional vectors are used to represent words, such as one-hot encoding. Each word of a sentence corresponds to a high-dimensional vector, and all bits in the vector are \"0\" except for a certain bit which is \"1\". However, this cannot extract semantic similarity, and the vector dimension of this method is very large as there are many zero values, which wastes storage space. In the second stage, to solve these problems, neural networks are trained using large amounts of text data to generate low-dimensional vectors, with Word2 Vec [28] and Glove [29] among the classical approaches. Glove optimizes Word2 Vec by considering not only the adjacent local information but also using the co-occurrence matrix to count the global information for better results. Static word embeddings produce vector representations and words in a one-to-one correspondence, which cannot solve the problem of multiple meaning words in log utterances, resulting in incomplete extracted semantic information. Dynamic word embedding methods have emerged, including Embeddings from Language Models (ELMo) [30], Generative Pretraining (GPT) [31], and Bidirectional Encoder Representations from Transformers (BERT) [7]. ELMo extracts the forward and reverse sequence information based on the input text information using a bidirectional LSTM [32] model. The extracted information is used after stitching. The word vector is generated dynamically on the basis of the whole contextual information. GPT uses a one-way transformer as a feature extractor for extracting the above information without considering the reverse contextual information. In contrast, BERT changes the LSTM to a transformer based on ELMo to extract bidirectional context information at the same time, changing from word-level embedding to sentence-level embedding and preserving more semantic information. Firstly, the tokenizer of natural language toolkit (NLTK) package is used to remove the punctuation from the log template, and the * in the template is removed altogether, leaving the token of the  
Figure 3: Example of log parsing.  
log template to recompose the sentence input to BERT. The embedding layer outputs the vector representation of the sentence by token embedding, position embedding, and segment embedding. The vector representation is then input to the encoding layer, where the encoder module of the two transformers [25] in the encoding layer processes the vector representation. Each word is computed with the attention of the words before and after it to obtain a more accurate contextual representation. After the encoder module is used to achieve the semantic vector encoding of the log sequence, the final pooling layer performs an average pooling operation on the output of all tokens to obtain a fixed size vector as the representation of the whole input sequence. The feature representation process is shown in Figure 4.