# GREG: A Global Level Relation Extraction with Knowledge Graph Embedding
## 4 Experiment and Results
### Model Performance Comparison  
Our final experiment is comparing our proposed model's performance to other existing models. For this, we compared the proposed model's performance from three different perspectives. For this experiment, we will measure the module's capabilities on local relation extraction, global relation extraction, and overall performance. The following models shall be used for comparison: an End-to-End Convolutional Neural Network(CNN) based model by [PERSON] et al. [22] and the End-to-End LSTM based model by [PERSON] et al. [23] shall act as our baseline comparisons. This will be followed by the context-aware LSTM based relation extraction module by [PERSON] et al. [16], which forms the base for the first part of our module. Additionally, to evaluate our proposed approach itself, we have also added two variations of our model that use the End-to-End CNN based model [22] and LSTM based model [23] to act as the first part of our model. Each of the three tasks is tested through different datasets. The Wikipedia-Wikidata sentence-level relation annotations [16] shall be used for the local relation extraction task, and the Korean document dataset of history [6] shall be used for the global relation extraction task. For our measurement of overall performance, we used both the Korean document dataset of history [6] and DocRED dataset [4] and evaluated their performances separately. Among the 10,470 lines of documented Korean text of history [6] used for this task, we separated 1201 lines to be used as a test set, and 550 lines to be used as a validation set. Thus, each of these 1751 lines of text was not used as a training set. This is approximately about 7.5% and 3% of the whole dataset. The total sum of the two consists of 50 documents and corresponding knowledge graphs out of the entire 503 documents. On the other hand, the DocRED dataset contains 132,375 entities and 56,354 relational facts annotated on 5053 Wikipedia documents. Additionally, as at least 40.7% of the relational facts in DocRED are multisentence level relations, it is a perfect dataset to evaluate the overall performance of this model. Both datasets are human-annotated document-level datasets. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Initial**} & \multicolumn{6}{c}{**Alternated Training**} \\ \cline{3-8}  & \multicolumn{1}{c}{**Training**} & \multicolumn{1}{c}{**1 st**} & \multicolumn{1}{c}{**10 th**} & \multicolumn{1}{c}{**50 th**} & \multicolumn{1}{c}{**100 th**} & \multicolumn{1}{c}{**200 th**} & \multicolumn{1}{c}{**500 th**} \\  & \multicolumn{1}{c}{**Only**} & \multicolumn{1}{c}{**Epoch**} & \multicolumn{1}{c}{**Epoch**} & \multicolumn{1}{c}{**Epoch**} & \multicolumn{1}{c}{**Epoch**} & \multicolumn{1}{c}{**Epoch**} & \multicolumn{1}{c}{**Epoch**} \\ \hline
**Korean dataset** & 95.03\% & 67.60\% & 70.78\% & 84.78\% & 88.99\% & 93.64\% & 93.61\% \\ \hline
**DocRED dataset** & 47.83\% & 29.53\% & 33.33\% & 35.55\% & 42.02\% & 51.07\% & 51.05\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance in the LSTM based Relation Extraction module during the process of alternated training. As seen in Table 2, the LSTM based models tend to exceed the CNN based models. The main difference seems to come from the fact that the CNN based models tend to perform worse than the other LSTM based models in general. First, we compare the models' performances on local relation extraction. Initially, one can see that the CNN baseline model [22] performs the worst with the LSTM baseline model [23] performing almost the same. The LSTM Context model [16] is the highest performer with the GREG(Context) following behind. The performance difference between the LSTM Context model [16] and GREG(Context) on the local extraction task once again indicates the facts we found in our previous experiment. In addition, we can suspect this is a universal effect as the performance defection from the synchronization process seems to apply to the other GREG variants.