# Combined Self-Attention Mechanism for Chinese Named Entity Recognition in Military
## 3 BiLSTM-Self-Attention-CRF Model
### Self-Attention Layer  
The greatest limitation of the traditional encoder-decoder structure is that the only connection between them is a fixed-length of the intermediate semantic vector. That is to say, the encoder compresses the whole sequence of information into a fixed-length vector. This approach has two drawbacks. One is that the semantic vector cannot fully represent the information of the entire sequence, which results in the information in the input sequence not being fully utilized. The other is, as the length of the input sequence grows, the information entered first may be overwritten by the information entered later. Recently, the self-attention mechanism has been widely used in image recognition, natural language understanding, and other fields to break the limitations of the traditional encoder-decoder structure, so as to extract as much feature information as possible from the input sequences [24]. To enhance the ability of the model to process information, we combine BiLSTM with a self-attention mechanism to capture lexical features and semantic information deeply. In the task of MNER, the semantic information of each Chinese character in a sentence does not have the same effect on this task. This mechanism can automatically focus on the specific Chinese characters that play a decisive role in entity recognition and capture the important semantic information in the input sequences, whilst reducing the focus on useless information. The self-attention mechanism is a selective mechanism for allocating limited information processing capacity, which can be used to find the relationship between the internals of the sequence. It can selectively focus on some important information and give high weights to important information, while assigning smaller weights to other information received at the same time. The attention mechanism can be described essentially as a mapping relationship between a query and a series of key-value pairs. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key [25]. Considering the small scale of military text corpus and the existence of many irregular texts, it is necessary to extract more text semantic features from multi-angle and multi-level perspectives. The demand cannot be met by an attention mechanism alone. Therefore, this paper uses the multi-attention mechanism proposed by [PERSON] et al. [25] in 2017 to improve the performance of MNER. Figure 2 is the structure of the multi-attention mechanism. Figure 2: The structure of multi-attention mechanism. For the complete hidden state sequence of the BiLSTM output, \(\mathbf{H}=(\mathbf{h_{1}},\mathbf{h_{2}},\ldots,\mathbf{h_{n}})\), the vector matrix \(\mathbf{H}\) is mapped to the three weight matrices obtained by training,\(w_{i}^{Q},\ w_{i}^{K},\ w_{i}^{V}\), for h times. The \(Q\), \(K\), and \(V\) vectors obtained from each mapping are fed into the attention function to generate the output \(\mathbf{z_{i}}\). Finally, the output matrix \(Z\) is obtained by splicing the h outputs together, so that more comprehensive feature information can be obtained. The attention function here uses the scaled dot-product attention function, which is essentially an attention mechanism that uses point multiplication to calculate similarity [26]. The specific calculation formula is as follows:  
\[\text{attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}V \tag{7}\]  
The first step is to compute the dot products of the query with all keys to get the similarity scores between the character vectors. The second step is to divide by \(\sqrt{d_{k}}\), which leads to having more stable gradients. The third step is applying a softmax function to calculate the weight coefficient. Finally, according to the weighted coefficients, a weighted summation operation is performed on the value to get an attention output. The multi-attention mechanism is composed of several scaled dot-product attentions.