## 3 Materials and Methods
### Transformer  
**Transformer Overall Structure:** The Transformer still adopts the encoder-decoder structure, but unlike Seq2 Seq, it eliminates the use of convolutional neural networks (CNNs) and RNNs internally and instead employs the self-attention mechanism. In RNNs, the computation of each unit is not parallel, as each unit needs to wait for the computation of other input units requiring their information. [PERSON] et al. (2017) solved this problem using CNNs, but it requires stacking many layers to capture long-range information [22]. The Transformer can be parallelized directly, and it also follows the encoder-decoder architecture completely.  
**Encoder:** The encoder of the Transformer model consists of two sublayers: the multi-head self-attention layer and the feed-forward neural network layer. The multi-head self-attention layer is composed of multiple attention layers concatenated together, with the attention calculation method in each attention layer using the scaled dot product formula. The formula is in Equations (2) and (3):  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V, \tag{2}\]  
\[Multi-Head=[Attention_{1};Attention_{2};\ldots;Attention_{h}], \tag{3}\]  
In Equation (2), \(Q\), \(K\), and \(V\) represent the query matrix, key matrix, and value matrix of the attention layer, respectively, which are obtained by passing the input vector through  
Figure 3: Seq2 Seq+attention model diagram. The encoder part utilizes a two-layer bidirectional GRU, while the decoder consists of a two-layer unidirectional GRU with the Luong global attention mechanism computed in a general fashion.  
h different linear layers, representing the feature information of the text in h different dimensions. dk represents the dimensionality of vectors.  
The formula for the feed-forward neural network layer is as in Equation (4):  
\[FFN=ReLu(W_{1}x+b_{1})W_{2}+b_{2}, \tag{4}\]  
In Equation (4), \(W_{1}\), \(W_{2}\), \(b_{1}\), and \(b_{2}\) are the weights and biases of the two linear layers. The process involves expanding the input vector \(x\) to the dimensionality of \(W_{1}\) and then reducing it back to its original dimensionality using \(W_{2}\).  
After each sublayer, there is a residual connection and normalization to alleviate gradient vanishing caused by the product of many factors during backpropagation. The formula for normalization is as in Equation (5):  
\[SubLayer_{output}=LayerNorm(x+(SubLayer(x))), \tag{5}\]  
In Equation (5), \(x\) represents the vector before processing by the sublayer, \(SubLayer(x)\) represents the vector after processing by the sublayer, and \(LayerNorm\) is the normalization function.  
**Decoder:** The decoder in this model has two sublayers from the encoder and an additional multi-head attention layer. This enables each decoder position to access all encoder input sequence information, enhancing relevant feature information. To prevent the model from capturing subsequent word information, the input sentence must be masked in the first self-attention sublayer of the decoder. This ensures that the model's prediction only relies on previous information, maintaining consistency between training and prediction.