# Syntax-Informed Self-Attention Network for Span-Based Joint Entity and Relation Extraction
## 3 Methodology
### Syntax-Informed Multi-Head Self-Attention  
Multi-head self-attention is the core component of Transformer [14], which not only allows the model to jointly attend to information from different feature subspaces at different positions, but also can be parallelized. Each of the heads learns a distinct attention function to attend to all of tokens in the sequence, and the feature of these heads will be concatenated to create the final representation. Many experimental results have shown that it has a powerful ability to learn context features and achieves impressive performance in neural machine translation benchmark [14; 49]. However, there are many studies find that the multi-head self-attention's capability has not been fully exploited [15; 16; 17; 18], and we expect to incorporate syntactic features into multi-head self-attention architecture to enhance its modeling capabilities. Therefore, we propose to incorporate syntax knowledge into multi-head self-attention by employing part of the heads to focus on syntactic parents of each token from pruned dependency trees, and use the rest of the heads to focus on semantic information in order to exploit the ability of the heads even further. In this layer, we will apply this mechanism on the global context. For semantic feature, we let \(h_{1}\) attention heads to attend to it. For each head, we map the input matrix \(X\) of N tokens representations into query \(Q_{h}\), key \(K_{h}\), and value \(V_{h}\) with \(h\) different learnable matrices:  
\[\left\{ \begin{array}{l}Q_{h}=Linear_{h}^{Q}(X)=X*W_{h}^{Q}\\ K_{h}=Linear_{h}^{R}(X)=X*W_{h}^{K}\\ V_{h}=Linear_{h}^{V}(X)=X*W_{h}^{V}\end{array} \right. \tag{3}\]  
After projecting, our \(Q_{h}\), \(K_{h}\), and \(V_{h}\) are of dimensions \(N\times d_{Q}\), \(N\times d_{K}\) and \(N\times d_{V}\), \(0\leq h\leq h_{1}\), and \(d_{Q},d_{K},d_{V}\) are dimensions of one head. Then, we use scaled dot-product to calculate attention weights, which is formulated as follows:  
\[A_{h}^{Sem}=softmax(\frac{Q_{h}K_{h}^{T}}{\sqrt{d_{h}}}) \tag{4}\]  
where \(\sqrt{d_{h}}\) is a scaled weight, and the value is a square root of its embedding dimension. Then, semantic attention weights will be multiplied by \(V_{h}\) for each token, and we will get the semantic-attended token representations:  
\[M_{h}^{Sem}=A_{h}^{Sem}\times V_{h} \tag{5}\]  
Then, we concatenate the results of these heads, and get semantic representation \(M^{Sem}\):  
\[M^{Sem}=[M_{0}^{Sem},\cdots,M_{h_{1}}^{Sem}] \tag{6}\]  
For syntactic features, we let \(h_{2}\) attention heads to attend to it and map the input matrix into query, key, and value as (4). We use external tools to generate a dependency tree of the input sentence, and prune the tree with different methods according to the candidate entity pairs generated from NER task. Similar to [6], we test three methods for pruning:  
* **ALL**: Keep the entire syntax tree without any pruning operations;
* **SDP (Shortest Dependency Path)**: Keep the shortest dependency path between entities in the syntax tree;
* **LCA (Lowest Common Ancestors)**: Keep the subtree under the lowest common ancestor of the entities pair. Then, we convert the prune tree into an adjacency matrix, and denote it as \(A_{prune}\). The attention weights are calculated as  
\[A_{h}^{Sym}=A_{prune} \tag{7}\]  
where \(A_{prune}\) is a \(N\times N\) one-hot matrix, and \(0\leq h\leq h_{2}\). Similarly, syntactic attention weights will be multiplied by \(V_{h}\) for each token, and we will get the syntax-attended token representations \(M^{Sym}\):  
\[M_{h}^{Sym}=A_{h}^{Sym}\times V_{h} \tag{8}\]  
\[M^{Sym}=[M_{0}^{Sym},\cdots,M_{h_{2}}^{Sym}] \tag{9}\]  
After that, we concatenate syntactic and semantic representation together, and denote it as \(M\):  
\[M=M^{Sym}\oplus M^{Sem} \tag{10}\]  
Finally, we employ a residual connection to our model. A layer of linear transformation is performed on the feature representation \(M\) and then added to the input matrix \(X\) (see Section 3.2).