## 2 Background and Related Work  
### Large Language Models  
In recent years, the development of Large Language Models (LLMs) [19] has garnered widespread attention in the field of artificial intelligence, as shown in Figure 1.  
This attention stems from their remarkable achievements in Natural Language Processing (NLP) tasks, ranging from basic text classification [20] to complex question-answering systems [21] and natural language understanding [22]. These achievements are built on the foundation of the Transformer architecture [23], proposed by [PERSON] et al. in 2017, which rapidly became the dominant technology for NLP tasks [24]. The core of the Transformer architecture is the self-attention mechanism, allowing models to focus on different parts of the sequence data. The mathematical expression of the self-attention mechanism is as follows:  
\[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V. \tag{1}\]  
Here, \(Q\), \(K\), and \(V\) represent the matrices for Query, Key, and Value, respectively, with \(d_{k}\) denoting the dimension of the key vectors. This formula calculates the output by computing the similarity between the queries and keys, then using these softmax scores to weight the values. By stacking multiple attention layers and incorporating feed-forward neural networks, residual connections, and layer normalization in each layer, Transformer models build deep architectures. The advantage of this architecture lies in its strong parallel processing capabilities, ability to capture long-distance dependencies, and flexible adjustment of model parameters through layer and width expansion. BERT (Bidirectional Encoder Representations from Transformers) is a Transformer-based model introduced by Google in 2018 [25], as shown in Figure 2.  
Figure 1: The depiction illustrates the development trajectory of Large Language Models (LLMs) from 2019 to 2023, meticulously documenting the introduction of significant models and technologies during this period. This includes the launch of models, from T5 and GPT-3 to the latest GPT-4, LLaMA, demonstrating the continual progress and evolution of large language models in the field of Natural Language Processing (NLP). Moreover, the illustration includes models optimized for specific domains and tasks, such as Codex, AlphaCode, reflecting the expansion and deepening of large language model technology in multiple directions. Figure 2: The depiction reveals the fundamental architecture of the BERT model, which encompasses the Masked Language Model (MLM) and Next Sentence Prediction (NSP) pretraining tasks. It can be seen that Sentence A and Sentence B are connected through the special tokens [CLS] and [SEP], along with the process of certain vocabulary being masked, with the aim of training the model to capture deep linguistic features and understand the relationships between sentences. Its innovation lies in the use of a bidirectional Transformer encoder. BERT's training process includes two stages: pretraining and fine-tuning. During the pretraining stage, BERT learns deep representations of language through the Masked Language Model (MLM) and Next Sentence Prediction (NSP) tasks. The MLM task involves predicting randomly masked tokens, while the NSP task predicts whether two sentences are consecutive. Unlike BERT, GPT adopts a pretraining plus fine-tuning framework [26]; however, as an autoregressive model, it uses all previous words as context when generating each word. The key to GPT is its extensive pretraining on a large corpus, followed by fine-tuning on specific tasks, leading to significant generalization capabilities. Large language models such as BERT and GPT learn rich linguistic features and knowledge information by pretraining on large-scale text data [27]. These models can understand semantic information in text data and capture complex contextual relationships, offering potential for in-depth analysis of network attack behaviors.