# Extraction of Joint Entity and Relationships with Soft Pruning and GlobalPointer
## 3 Methodology
### Graph Model
#### 3.2.2 KL Divergence  
In probability theory or information theory, KL divergence (Kullback-Leibler divergence), also known as relative entropy, is a way to describe the difference between two probability distributions \(p\) and Q. It can measure data objects whose geometric distances  
Figure 3: The graph module is shown with an example of four nodes and an adjacency matrix. The node embeddings and adjacency matrix are generated with KL divergence as inputs. Then, employing multi-head attention to construct \(N\) attention-guided adjacency matrices, the resulting matrices are fed into \(N\) separate densely connected layers, generating new representations. Finally, a linear combination layer is applied to integrate the outputs of the \(N\) different densely connected layers. (such as cosine distance, Euclidean distance) are difficult to measure [41]. When the two distributions are closer, the KL divergence value is smaller. KL divergence is defined as follows:  
\[D_{KL}(p||q)=\sum_{i=1}^{n}p(x_{i})\log\left(\frac{p(x_{i})}{q(x_{i})}\right) \tag{2}\]  
where \(p(x)\) is the target distribution, \(q(x)\) is the matching distribution, \(x_{i}\) is the discrete random variable, and \(N\) is the length of the distribution. To evaluate the differences in the distributions of the two nodes, KL divergence is utilized to determine the connection strength between nodes. The propagation of messages between token representations with considerable semantic variances is encouraged to obtain the potential links between tokens. As an example, the semantic gap between \"we\" and \"our\" is extremely tiny, so the strength of association between two words is small and the weight assigned is negligible; the semantic gap between \"[PERSON]\" and \"Apple\" is large, so the connection strength of the two words is vast and the weight assigned is significant. The edge weight between the \(i\)-th and \(j\)-th nodes is calculated as follows:  
\[e_{ij}=D_{KL}(N_{i}(\mu_{i},\sigma_{i}^{2})\Big{|}\Big{|}N_{j}(\mu_{j},\sigma_{ j}^{2})) \tag{3}\]  
where \(D_{KL}\) denotes KL divergence calculation. \(N_{i}(\mu_{i},\sigma_{i}^{2})\) and \(N_{j}(\mu_{j},\sigma_{j}^{2})\) are the distributions of the \(i\)-th node and \(j\)-th node. We obtain the graph G = (V, A) with a directed graph due to the asymmetry of the KL divergence. V denotes the set of all nodes, and A denotes the adjacency matrix. GCNs are neural networks that operate directly on graph structures. Let us take a look at how the nodes are updated first for multi-layer GCNs. Given a graph with \(n\) nodes, we can represent the graph with an \(n\times n\) adjacency matrix A. In traditional GCN operations, A\({}_{ij}\) = 1 and A\({}_{ji}\) = 1 if there is an edge going from node \(i\) to node \(j\), otherwise A\({}_{ij}\) = 0 and A\({}_{ji}\) = 0. The purpose of using GCN is to aggregate adjacent nodes and learn the information of K-order nodes. The convolution calculation of node \(i\) at the \(l\)-th layer takes \(h^{l-1}\) as input and outputs \(h^{(l)}\):  
\[h_{i}^{(l)}=\rho\left(\sum_{j}^{n}\text{A}_{ij}\text{W}^{(l)}h_{j}^{(l-1)}+ \text{b}^{(l)}\right) \tag{4}\]  
where A denotes the adjacency matrix and W represents the weight matrix. \(\rho\) is the activation function, and \(h_{i}^{(0)}\) is the initial input representation \(v_{i}\). We know that in most previous research, the input graph was pruned. However, rule-based pruning strategies might eliminate some crucial information from the entire tree. As a result, we use the attention guidance layer to turn the graph into \(N\) fully connected graphs [42]. The self-attention mechanism is used to create the adjacency matrix, which can be written as follows:  
\[\text{A}^{(t)}=softmax\left(\frac{Q\text{W}_{i}^{Q}\times\left(\text{K}\text{ W}_{i}^{K}\right)^{T}}{\sqrt{d}}\right)V \tag{5}\]  
where \(Q\) and \(K\) are BERT output and \(d\) denotes the dimension of the BERT output. With the above operation, we do not need to rely on external NLP toolkits, and avoid pruning operations.