## 3 The Proposed Model
### Syntactic Integration Unit  
As is shown in Figure 3, the Syntactic Integration Unit is designed to integrate syntax into BERT and is the most crucial component of this approach. In a multi-layer GCN, the  
Figure 2: Model architecture. Better representation for the entity nominal “South Korea” is obtained based on the pre-trained language representation from the BERT unit by integrating the syntactic constraints using a soft dependency-based attention. node representation \(h_{i}^{(l)}\) is produced by applying a graph convolution operation in layers from \(1\) to \(l-1\), described as follows:  
\[h_{i}^{(l)}=\rho(\sum_{j=1}^{n}A_{ij}W^{(l)}h_{j}^{(l-1)}+b^{(l)}) \tag{3}\]  
where \(W^{(l)}\) represents the weight matrix, \(b^{(l)}\) stands for the bias vector, and \(\rho\) is an activation. \(h^{(l-1)}\) and \(h^{(l)}\) are the hidden state in prior and current layer, respectively. Syntactic Integration Unit contains attention guided layer, densely connected layer, and linear combination layer. **Attention Guided Layer** Most existing methods adopt hard dependency relations (i.e., 1, 0 denote relation exists or not) to impose syntactic constraints. However, these methods require the pre-defined pruning strategy based on expert experience and simply set the dependency relations considered \"irrelevant\" as zero-weight (not attended). These rules may bias representations, especially toward a larger dependency graph. Reversely, the attention guided layer helps to launch the \"soft pruning\" strategy. This layer generally generates the attention guided adjacency matrix \(A^{(l)}\) whose weights range from 0 to 1 by multi-head attention [34]. The shape of \(A^{(l)}\) is the same as the original adjacency matrix \(A\) for convenience. Precisely, \(A^{(l)}\) is calculated as follows:  
\[A^{(l)}=softmax(\frac{QW_{i}^{Q}\times(KW_{i}^{K})^{T}}{\sqrt{d}})V \tag{4}\]  
where \(Q\), \(K\), \(V\) are, respectively, query, key, value in multi-head attention, \(Q\), \(K\) are both equal to the input representation \(R^{(m-1)}\) (i.e., output of the prior module), \(d\) is the dimension of \(R^{(m-1)}\), \(W_{i}^{Q}\) and \(W_{i}^{K}\) are both learnable parameters \(\in\mathbb{R}^{d\times d}\), \(A^{(l)}\) is the \(t\)-th attention guided adjacency matrix corresponding to the \(t\)-th head. In this way, the attention guided layer outputs a large fully connected graph to reallocate the importance of each dependency relation rather than pruning the graph into a smaller structure as tradition. Figure 3: The structure in Syntactic Integration Unit. **Densely Connected Layer** This layer helps to learn more local and non-local information and train a deeper model using densely connected operations. Each densely connected layer has \(L\) sub-layers. \(L\) is a hyper-parameter for each module. These sub-layers are placed in regular sequence, and each sub-layer takes all preceding sub-layers' output as input. The structure of Densely Connected Layer is shown in Figure 4. \(g_{j}^{(l)}\) is calculated in this layer, which is defined as the concatenation of the initial representation and the representations produced in each preceding sub-layer:  
\[g_{j}^{(l)}=[x_{j},h_{j}^{(1)},...h_{j}^{(l-1)}], \tag{5}\]  
where \(x_{j}\) is initial input representation, \(h_{j}^{(1)},...h_{j}^{(l-1)}\) are the outputs of all preceding sub-layers. In addition, the dimension of representations in these sub-layers is shrunk to improve the parameter efficiency, i.e., \(d_{hidden}=d/L\), where \(L\) is the number of sub-layers, \(d\) is the input dimension. For example, the number of sub-layer is 2 and input dimension is 1024, \(d_{hidden}=d/L=512\). Then a new representation whose dimension is \(1024(512\times 2)\) is formed by concatenating all these sub-layer outputs. \(N\) densely connected layers compute \(N\) adjacency matrixes produced by attention guided layer.