## 3 Materials and Methods
### Vectorization of Text Data  
To utilize the potential of machine learning algorithms, a text needs to be properly encoded. The simplest and most common method is one-hot encoding. However, the orthogonality of the vectors obtained by one-hot encoding contributes to the loss of information about the semantic relationships between words, which are crucial in the anomaly detection process.  
In our approach, we used the fastText algorithm, which provides two models for computing word representations: _skipgram_ and continuous bag-of-words (_cbow_). skipgram learns to predict a target word based on a nearby word. In contrast, cbow predicts the target word according to its context. The context is represented as a bag-of-words contained in a fixed-length window around the target word. cbow tends to learn several times faster than skipgram, but skipgram performs better on small datasets. In this paper, we made use of the cbow model as the BGL dataset is rather large and the cbow model is able to compute embeddings precise enough for anomaly detection in a much shorter time than skipgram. Furthermore, initial experiments showed that using skipgram instead of cbow did not improve the results.  
As shown in Figure 2, the cbow model is a neural network consisting of three layers: input, hidden, and output. The input data of the neural network are one-hot encoded words that surround the target word context window (\(x_{1k}\) to \(x_{ck}\)). By words, we are referring to log templates. \(\mathbf{W}_{V\times N}\) is a weight matrix of size \(V\times N\). The results of multiplying the weight matrix \(\mathbf{W}_{V\times N}\) with each vector from the input layer are averaged to obtain the activation value of the hidden layer (\(N\)-dimensional vector \(h\)). The matrix \(\mathbf{W}_{N\times V}\) is a weight matrix of size \(N\times V\). The result of multiplying the weight matrix \(\mathbf{W}^{*}_{N\times V}\) with the activation vector from the hidden layer is the output layer vector of length \(V\). During model training, the output layer vector is passed as an argument to a loss function to determine the prediction error. The cbow model also makes use of negative sampling to make the training process more efficient [38].  
Model training is the process of optimizing the loss function by changing the values of the weights in weight matrices. Once the training process is complete, the model can represent words in vector form. The model stores the activation values for all pairs containing a given word from the dictionary and its context window, making it possible to calculate the target word vector representation by averaging all the activation vectors of that word.  
The approach based on the fastText vectorization without character n-grams, which we used in this study, is different from other log vectorization methods, such as word2 vec log embedding [15], log2 Vec [19], or NeurallLog [31]. All these approaches take words that compose a single log entry as the smallest units, which are then vectorized and transformed once (as in NeurallLog and log2 Vec) or twice (as in word2 vec log embedding) to a log event vector. Such an approach is resource-intensive because the number of words in the dataset is much larger than the number of log events. In this paper, we treat the entire log entry as the smallest unit by vectorizing the template indices using the fastText algorithm. The sequence vector is then obtained by a single transformation of all vectorized log events. The fastText algorithm itself, without n-grams, is almost identical to the word2 vec word embedding method [16] (not to confuse it with the word2 vec log embedding method), which could be used interchangeably in this approach. In the remainder of our article, we show that the vectors extracted this way are sufficient for efficient anomaly detection, once the sequence length has been adapted to a given binary classifier.