## 2 Definition of Large Language Models
### Architecture of Large Language Models  
Large language models are typically built upon the Transformer architecture, a framework widely recognized as a milestone in the evolution of deep learning for natural language processing. The Transformer architecture follows a dual-structured encoder-decoder design, in which the encoder first transforms a sequence of input symbols into continuous representations, or feature vectors [27]. These continuous vectors are then passed to the decoder to initiate the process of generating the output text [28]. Figure 2 illustrates the overall Transformer architecture. In this structure, the encoder utilizes two key operations the self-attention mechanism and Feed-Forward Neural Layers to extract rich feature representations from the input. These operations are reinforced through residual connections and layer normalization to maintain a stable flow of information across multiple layers [27]. Subsequently, the decoder uses these extracted vectors to generate the output text in a step-by-step, auto-regressive  
Figure 2: Overview of the Transformer architecture based on self-attention. manner, meaning that each generated token serves as input for the next prediction step [27]. To preserve causality and prevent the model from accessing future tokens during generation, masking is applied within the decoder. This architecture not only achieves high accuracy but also supports non-sequential processing, enhancing the scalability and efficiency of Transformers when dealing with large and diverse datasets [27; 28]. In contrast to earlier architectures such as RNNs and LSTM networks, Transformer models are fully based on the self-attention mechanism and do not rely on sequential computation, a key factor that has significantly improved their efficiency and scalability [2]. To gain a deeper understanding of how the two core components of the Transformer the encoder and the decoder function, it is helpful to begin with a detailed examination of the encoder mechanism. In the Transformer architecture, the encoder acts as a meticulous and context-aware reader. It has full access to the input text (e.g., a source sentence in machine translation) and examines all words simultaneously to determine how different parts of the input relate to each other. The encoder first applies the Self-Attention mechanism, which determines how much attention each word should pay to every other word in the sequence. Following this, a Position-Wise Feed-Forward Neural Network is applied to each position in the sequence to extract higher-level features and capture more complex patterns and meanings [2]. As the information flows through these two stages, residual connections and layer normalizations are employed to stabilize the flow and prevent the excessive alteration of intermediate representations from occurring. By stacking this structure over multiple layers, the encoder eventually generates a context-aware semantic representation for each word, embedding the meaning of the entire input sequence into each individual token. These representations serve as the foundational information that the decoder later references to generate coherent and contextually appropriate outputs [29]. Each layer of the encoder consists of two sublayers [30]:  
1. A multi-head self-attention mechanism, which captures intra-sequence dependencies. 2. A fully connected feed-forward neural network, applied in a position-wise manner to each token. Both sublayers are wrapped with residual connections and followed by layer normalization. Additionally, all sublayers and embedding layers produce outputs with a fixed dimensionality, typically set to 512. On the other side of the architecture, the decoder shares a structure similar to that of the encoder, with one key addition: an extra sublayer dedicated to cross-attention, which allows the decoder to attend to the encoder's output representations [31].