# Named Entity Recognition Networks Based on Syntactically Constrained Attention
## 3 Methodology
### Syntactically Constrained Attention Networks  
First, the syntactic adjacency matrix \(A=(a_{ij})\in R^{n\times n}\) of the text is constructed using the syntactic dependency tree. The elements \(a_{ij}\) in the syntactic adjacency matrix \(A\) indicate the word \(x_{i}\) and word \(x_{j}\), and whether there is a dependency relationship between them. Then, by using a multi-head attention score matrix, this yields the attention matrix \(A_{score}^{i}\in R^{n\times n}\ (1\leq l\leq h)\), where \(h\) denotes the number of heads, as shown in Formula (1) below:  
\[A_{score}^{i}=\frac{QK^{T}}{\sqrt{d_{k}}} \tag{1}\]  
where \(Q\) and \(K\) are the abstract matrices obtained by projecting the context vector \(H\in R^{n\times d}\) from the BERT output. For each attention score matrix \(A_{score}^{i}\), the goal is to make each word  
Figure 3: Illustration of the SCAN-Net model. focus, as much as possible, on only the words in the sentence that have a dependency on itself, subject to the constraints of syntactic information. Syntactically constrained attention \(A_{\text{SC}}^{i}\) is defined as follows: words with dependencies are selected separately for each head, based on the attention score matrix. Then, for each syntactically constrained attention \(A_{\text{SC}}^{i}\), a softmax operation is applied. To obtain a syntactic constraint attention that is relatively better for the model, in order to understand the context \(A_{\text{SC}}^{i}\), we designed two strategies to integrate the syntactic information with the attention matrix, namely, the soft-pruning syntactic attention enhancement strategy and the hard-pruning syntactic concentration attention strategy. **Soft-pruning syntactic enhancement attention strategy** (hereinafter referred to as soft-pruning strategy): the attention weights of the strong words in each attention score matrix \(A_{\text{score}}^{i}\) are increased, according to the syntactic dependency information provided by the adjacency matrix A, and the attention weights of the weak words are kept constant. After obtaining \(A_{\text{SC}}^{i}\), the softmax operation is then performed as follows. The specific operations are shown in Formulas (2) and (3). \[A_{\text{SC}}^{i}=A\oplus A_{\text{score}}^{i} \tag{2}\]  
\[A_{\text{SC}_{\text{score}}}^{i}=softmax\Big{(}A_{\text{SC}}^{i}\Big{)} \tag{3}\]  
where \(A_{\text{SC}}^{i}\) denotes the syntactic constraint attention of the ith head, which is obtained after a softmax operation is conducted on it, in order to sharpen the constraint attention of the ith head \(A_{\text{SC}_{\text{score}}}^{i}\). **Hard-pruning syntax concentration attention strategy** (hereinafter referred to as hard-pruning strategy): the attention of weak words in each attention score matrix \(A_{\text{score}}^{i}\) is directly removed, based on the syntactic dependency information provided by the adjacency matrix A. The specific operations are shown in Formulas (4) and (5). \[A_{\text{SC}}^{i}=A\circ A_{\text{score}}^{i} \tag{4}\]  
\[A_{\text{SC}_{\text{score}}}^{i}=softmax\Big{(}A_{\text{SC}}^{i}\Big{)} \tag{5}\]  
While both of these strategies can assist the model in comprehending the context, the principles behind them are different. The soft-pruning strategy aims to strengthen the model by increasing the attention weights of strong words, enabling the model to pay more attention to them while still considering weak words. On the other hand, the hard-pruning strategy restricts the model by eliminating the attention of weak words, forcing it to focus solely on the strong words that have not been eliminated. After obtaining \(A_{\text{SC}_{\text{score}}}^{i}\), the abstract matrix \(V\) is obtained by projecting the text vector \(H\in R^{n\times d}\) output from BERT again, by using the method used to calculate \(Q\), \(K\) above. Afterwards, \(A_{\text{SC}_{\text{score}}}^{i}\) is introduced into the abstract matrix \(V\) to obtain the syntactically constrained contextual representation \(H_{\text{SC}}\in R^{n\times d}\).