# Research on Named Entity Recognition Methods in Chinese Forest Disease Texts
## 3 Materials and Methods
### The Proposed Approach
#### 3.2.2 Transformer Encoder Layer with Position Information  
The transformer model is completely based on an attention mechanism. Instead of using the original cycle and convolution structure, it adopts the self-attention mechanism, which can outperform RNNs and CNNs in tasks such as machine translation. The transformer model processes the characters in the sequence in parallel, and uses the self-attention mechanism to directly model the relationship between the characters in the sentence. It has better computational performance. The transformer model includes two main components: an encoder and a decoder. Because the decoder is often used for the generation task, the model proposed in this paper only uses the transformer encoder to model the context distance features. The specific structure is shown in Figure 2.  
First, the input text sequence is embedded with multiple features to obtain the multi-feature embedded input sequence \(x\in R^{K_{x}}\), where \(K_{x}\) represents the size of the input batch in the sequence thesaurus. As the original transformer model does not have the ability to capture sequential sequences, the positional encoding feature is added to represent the absolute position information of each character. Then, we calculate the sine transform, according to Formula (1), and the cosine transform, according to Formula (2), in order to encode the position and obtain the relative position information.  
\[PE_{(pos,2i)}=\sin(pos/10,000^{2i/d_{\text{model}}}) \tag{1}\]\[PE_{(pos,2i+1)}=\cos(pos/10,000^{2i/d_{\text{model}}}) \tag{2}\]  
where \(pos\) refers to the position of the current character in the sentence and \(i\) refers to the dimension of the vector. In even positions, sinusoidal coding is used; in odd positions, cosine coding is used. Then, the position coding and multi-feature embedded input sequence are concatenated. The dimension is the same as that of the multi-feature input sequence, which is obtained as the input of the transformer encoder. Then, the multi-head attention mechanism layer is input and decomposed according to Equation (3). The number of heads is 8.  
\[Q_{i}=XW_{i}^{Q},\ K_{i}=XW_{i}^{K},\ V_{i}=XW_{i}^{V},\ i=1,\ldots,8 \tag{3}\]  
where \(X\in\mathbb{R}^{n\times d_{k}}\), \(W_{i}^{Q},W_{i}^{K},W_{i}^{V}\in\mathbb{R}^{d_{k}\times d_{\text{model}}}\), and \(d_{k}\) is the dimension of the multi-feature embedding space. Then, we calculate the self-attention according to Equation (4) and learn a weight for each input word. The multi-head attention mechanism is equivalent to the integration of the different self-attention mechanisms of the head. The weighted characteristic matrices are spliced and calculated according to Equation (5), in order to obtain a large characteristic matrix.  
\[Z_{i}=\text{Attention}(Q_{i},K_{i},V_{i})_{i}=\text{softmax}(\frac{Q_{i}K_{ i}^{T}}{\sqrt{d_{k}}})V_{i} \tag{4}\]  
\[Z=\text{MultiHead}(Q,K,V)=\text{Concat}(Z_{1},\ldots,Z_{8})W^{O} \tag{5}\]  
\[\text{FN}(Z)=\text{max}(0,ZW_{1}+b_{1})W_{2}+b_{2} \tag{6}\]  
where \(Z_{i}\in\mathbb{R}^{n\times d_{\text{model}}}\). Then, we calculate the residual connection to solve the problem of gradient disappearance when the number of back-propagation layers is large, such that the gradient can reach the input layer quickly. Then, we normalize the data (i.e., normalizing the output to the standard normal distribution), accelerate the convergence, and increase the training speed. Finally, full connection of the two layers is carried out, according to Equation (6), and the dimension transformation is completed, which enhances the expression ability of the model. In the other layers, the input is the output of the previous layer.  
Figure 2: Structure of the transformer encoder.