## 2 Methods
### Transformer  
The Transformer consists of an encoder and a decoder. The encoder maps the input sequence \(x=(x_{1},\ldots,x_{n})\) to a continuous sequence \(z=(z_{1},\ldots,z_{n})\). The decoder generates a sequence \(y=(y_{1},\ldots,y_{m})\) step by step based on the given values of \(z\). The stacked encoder and decoder component structure is illustrated in Figure 4, where the left side represents the encoder component and the right side represents the decoder component. Each encoder has the same structure, consisting of two sub-layers with different weights. The input passes through a self-attention layer and then is fed to a feed-forward neural network layer. In addition to these two sub-layers, the decoder also includes an encoder-decoder attention layer, similar to Seq2 Seq, which helps the decoder focus on relevant parts of the input data. Between these layers, connections are made through Add and Normalize, involving mainly residual connections and normalization operations. Add and Normalize breaks the symmetry of the neural network, improving the issue of network degradation. It accelerates model convergence by optimizing the normalization process and addresses the problem of gradient vanishing in deep learning.  
Figure 3: GAN training mechanism.  
By treating data as a sequence, the Transformer initially processes the input data through embedding to obtain vector representations. Subsequently, position encoding is applied to the corresponding data in the sequence to prevent data misalignment. Through a self-attention mechanism, the data are mapped to triplets Q (Query), K (Key), and V (Value) for representation, and vectors are output. The formula for the self-attention mechanism is as follows:  
\[\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V. \tag{6}\]  
In the above equation, \(Q\) represents the query matrix, and \(K\) represents the key matrix, both of which use dot product matching. The result is the attention matrix, which effectively reflects the semantic relevance between \(Q\) and \(K\). Additionally, \(d_{k}\) in the equation denotes the channel dimension, and the weights obtained by applying the Softmax function to the calculated values are further used to obtain the values of self-attention.  
In each layer of the Transformer encoder and decoder structure, in addition to the multi-head self-attention mechanism, there is also a fully connected feed-forward network. The fully connected feed-forward network consists of two linear transformations followed by a ReLU activation function. The computational formula is as follows:  
\[\text{FFN}(x)=\text{max}(0,xW_{1}+b_{1})W_{2}+b_{2}. \tag{7}\]  
Although the linear transformations at different positions between layers in the fully connected feed-forward network are the same, the network parameters used within them are different. The purpose of positional encoding is to obtain accurate information about the positions of input sequences, thus compensating for the lack of positional information in Transformer models compared to recursive networks and convolutional neural networks. By adding positional encoding to the bottom input of the encoder and decoder, the bottom input and positional encoding have the same dimensions for addition, achieving an accurate description of the relative and absolute positions of input sequence data. There are various  
Figure 4: Stacked encoder and decoder structure diagram.  
methods for performing positional encoding, and the Transformer model uses cosine and sine functions with different frequencies.  
\[PE_{(pos,2i)}=sin\big{(}pos/10000^{2i/d_{model}}\big{)}, \tag{8}\]  
\[PE_{(pos,2i+1)}=cos\big{(}pos\ /\ 10000^{2i/d_{model}}\big{)}, \tag{9}\]  
where \(i\) represents the dimension and \(pos\) represents the position, indicating that each dimension of the positional encoding corresponds to a sine function.