## 3 The Proposed CTHE Method
### BERT-Based Att-Bi-GRU Specific Relation Prediction Module  
The main goal of this module is to accurately predict the entity pair relation with a specific pattern. If the entity pair meets this pattern, the trained BERT-based Att-Bi-GRU model is used to predict the relation. For example, we input the entity pair ([PERSON], PER, Harvard University, ORG, [PERSON] graduated from Harvard University), and output the possible prediction result as Graduated_University ([PERSON], Harvard University). The framework of the BERT-based Att-Bi-GRU is shown in Figure 3. The model contains Input, Word segmentation, BERT embedding, Bi-GRU, Attention, Concat, Normalized, and Output layers. As a supervised method, it requires annotation corpora to complete the training of the model. The input sample includes a head entity, a tail entity, an original sentence, and the relation. For the word segmentation layer, we used the pyhanlp (pyhanlp-0.1.66-cp35) module. It is important to note that to improve the accuracy of segmentation, all of the entities from named entity recognition results are added into the user-defined dictionary in the segmentation stage.  
The BERT embedding layer is based on the fine-tuned pre-trained Chinese-BERT-wwm-ext vector embedding model to obtains the vector transform results [x\({}_{1}\), x\({}_{2}\),..., x\({}_{\text{k}}\)] of all words. The remaining layers are implemented using the functions provided by Tensorflow (Tensorflow-1.12.0-cp35).  
Figure 3: The framework of the BERT-based Att-Bi-GRU model.  
The calculation formulas of the Bi-GRU layer and attention layer are as follows. For each word \(t\), the GRU cell unit computes \(h_{t}\) with input \(x_{t}\) and previous state \(ht_{-1}\), as:  
\[r_{t}=\sigma(W_{r}x_{t}+U_{t}h_{t-1}) \tag{1}\]  
\[u_{t}=\sigma(W_{u}x_{t}+U_{u}h_{t-1}) \tag{2}\]  
\[\widetilde{h_{t}}=\tan h(W_{c}x_{t}+U(r_{t}\odot h_{t-1})) \tag{3}\]  
\[h_{t}=(1-u_{t})\odot h_{t-1}+u_{t}\odot\widetilde{h_{t}} \tag{4}\]  
where \(h_{t}\) is hidden state, \(r_{t}\) is reset gate, and \(u_{t}\) is update gate. \(W_{r}\), \(W_{u}\), \(W_{c}\), and \(U_{r}\), \(U_{u}\), \(U\) represent the parameters of GRU. \(\sigma\) is sigmoid function, and \(\odot\) refers to the production with element-wise.  
For the word \(t\), we use the hidden state \(\xrightarrow[h_{t}]{}\) and \(\xleftarrow[h_{t}]{}\) to represent the encode results from the forward and backward GRU. Then, we use the concatenation \(h_{t}=[\xrightarrow[h_{t}]{}\xleftarrow[h_{t}]{}\) as the output of the Bi-GRU layer of word \(t\).  
For the task of relation classification of the entity pair, to find the hidden features between the head entity and tail entity, we introduced a position detection attention mechanism to calculate the weight of each word. The feature vector \(h_{T}\) of the word \(t\) is defined as a weighted sum, which is computed as follows:  
\[h_{T}=\sum_{i=1}^{k}\alpha_{i}h_{i} \tag{5}\]  
where \(h_{i}\) is the \(i\)-th cell unit output of the Bi-GRU layer. In addition, the \(\alpha_{i}\) was calculated as followed:  
\[\alpha_{i}=Aw_{i}+B \tag{6}\]  
\[w_{i}=\left\{\begin{array}{ll}0,\ t\
otin Set[\textit{head entity},\ \textit{tail entity}]\\ 1,\ t\in Set[\textit{head entity},\ \textit{tail entity}]\end{array}\right. \tag{7}\]  
where, Set[head entity, tail entity] is the word set between the head entity and tail entity in the original sentence, \(k\) is the total number of words in the original sentence, \(t\) is the \(t\)-th word in the original sentence, and \(A\), \(B\) are the parameters of the network. In our method, we pay more attention to the words between the head entity and tail entity.