## 3 Proposed Models
### Anomaly Detection  
LTAnomaly uses LSTM and Transformer to model the feature matrix of the input and detect anomaly logs, taking advantage of LSTM modeling of the time dimension and the self-attention mechanism in Transformer to capture global contextual information. In doing so, we add LSTM to the Transformer subset (Figure 6). For Transformer with LSTM, the input eigenmatrix is processed by LSTM and transmitted to the Transformer block.  
The final feature matrix \(E\), including the log sequence feature matrix and component sequence feature matrix, is processed by Word2 Vec and TF-IDF. The feature matrix \(E\) in the proposal is input to the anomaly detection model based on LSTM-Transformer, which is processed by the Embedding layer and transmitted to the LSTM layer and the PositionalEncoding layer. The output matrix of the LSTM layer and the PositionalEncoding layer are combined and sent to the Transformer block. The calculation process can be summarized as follows:  
\[\mathbf{emb}=Encoder(E), \tag{8}\]  
\[\mathbf{out\_1,h}=LSTM(\mathbf{emb}), \tag{9}\]  
\[\mathbf{src}=Pos\_encoder(\mathbf{emb}\times\sqrt{ninp}), \tag{10}\]  
\[\mathbf{x}=Fusion(\mathbf{src},\mathbf{out\_1}). \tag{11}\]  
Figure 6: LT model: Transformer with LSTM.  
where \(\mathbf{emb}\) denotes the output of Embedding, \(\mathbf{out\_1}\) and \(\mathbf{src}\) are the outputs of LSTM and PositionalEncoding, respectively, and \(\mathbf{ninp}\) is the embedded dimension in the Embedding layer. \(Fusion(\cdot)\) is a combined operation for the Fusion layer, and \(\mathbf{x}\) is the output of the Fusion layer, which is input to the next module.  
We next discuss the decoder parts of the Transformer model, involving the MultiheadAttention, normalization, and full connection layers, as shown in Figure 7. The output \(\mathbf{x}\) of the Fusion layer is regarded as the input of the module, and the calculation process of the module is as follows:  
\[Q=\mathbf{x}W^{Q}, \tag{12}\]  
\[K=\mathbf{x}W^{K}, \tag{13}\]  
\[V=\mathbf{x}W^{V}, \tag{14}\]  
\[H_{i}=Softmax\left(\frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}}\right)V_{i} \tag{15}\]  
\[\mathbf{O}=Concat(H_{1},H_{2},\dots,H_{n-1},H_{n})W^{O}. \tag{16}\]  
where \(W^{Q}\), \(W^{K}\), and \(W^{V}\) are learnable parameter matrices, and the input data are transferred to query, key, and value. The dimension of the key vector is \(d_{k}\). The \(Concat(\cdot)\) function concatenates the output of each header with the help of the parameter matrix \(W^{O}\).  
The \(\mathbf{O}\) processed by multiple attention is added and connected with its input \(\mathbf{x}\) to provide \(\mathbf{Ot}\) through the full connection layer,  
\[\mathbf{Ot}=FNN(\mathbf{O}+\mathbf{x}), \tag{17}\]  
where \(FNN(\cdot)\) is a feedforward network. An \(\mathbf{Ot}\) with the same size and shape as \(\mathbf{x}\) can be input to the next module.  
Softmax calculates the prediction label y_pred for the log sequence, in which 0 indicates that the input data are normal and 1 indicates abnormality. Cross-entropy for  
Figure 7: Transformer block.  
our loss function can be utilized in the training phase, making full use of the adaptive moment estimation optimization algorithm (Adam) when tuning model parameters. LSTM and Transformer are combined in a powerful and robust model through intersectional information representation.