# Korean Historical Documents Analysis with Improved Dynamic Word Embedding
## 4 Experiment
### Results of Named Entity Recognition and Neural Machine Translation  
To evaluate the performance of the proposed method, various word embeddings were performed before the NER task; Table 2 summarises these results. In the table, W2V represent the Word2 Vec [28] method; DW2V is dynamic word embedding from [PERSON] et al. (2018) [32]; and DBE is dynamic Bernoulli embedding from [PERSON] et al. (2017) [33]. The * mark indicates that the method adopted a bilinear function to contain information about the kings.  
When word embedding was pre-trained using the skip-gram method in Word2 Vec (generally used without DWE), the F1-score was 0.61, thereby suggesting that it achieved the worst performance compared to the others that used DWE. The proposed method that combined DWE with factorised embedding parameterization showed a higher F1-score than other existing word embedding methods. This implies that word embedding has limitations in reflecting temporal information; further, it demonstrated that the addition of temporal information through factorised embedding parameterization and the bilinear function enhances the performance in various tasks.  
Further, NMT was compared with GRU [47] and a transformer-based model to test the performance of the proposed method. Learning in GRU also used the seq2 seq [48] with attention mechanism [49] method supported by the encoder and the decoder, and the hyperparameters were set to the same values as those in the proposed model. Three metrics (BLEU4 [50], METEOR [51], and ROUGE-L [52]) were used in the test; the results are summarised in Table 3.  
Table 3 indicates that the proposed method achieved higher performance compared to the other methods for historical documents. Results shows our model performs better than other NMT task-specifically learned model. It demonstrated that our model enhance NMT performance. Table 4 summarises the results of the translation test for the test dataset; the first sentences were translated accurately, although most of them contain place-names. This is attributed to the fact that the application of the NER improved the translation of texts related to names of places, and therefore, vectors pre-trained through NER are effective for learning the representation of historical documents.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Precision** & **Recall** & **F1-Score** \\ \hline W2V [28] & 0.581 & 0.637 & 0.607 \\ DW2V [32] & 0.627 & 0.640 & 0.633 \\ DW2V * [32] & 0.582 & 0.665 & 0.620 \\ DBE [33] & 0.629 & 0.639 & 0.633 \\ Ours & 0.650 & 0.646 & 0.648 \\ Ours * & 0.679 & 0.690 & 0.684 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results of proposed NER method. Test results was evaluated on parameter set with best validation F1-score. W2V represent the Word2 Vec [28] method; DW2V is dynamic word embedding from [PERSON] et al. (2018) [32]; and DBE is dynamic Bernoulli embedding from [PERSON] et al. (2017) [33]. The * mark indicates that the method adopted a bilinear function to contain information about the kings.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **BLEU1** & **BLEU2** & **BLEU3** & **BLEU4** & **METEOR** & **ROUGE-L** \\ \hline RNN Seq2 seq (GRU) [47] & 0.450228 & 0.350884 & 0.281607 & 0.250351 & 0.298210 & 0.428376 \\ Transformer (from scratch) [35] & 0.547702 & 0.438370 & 0.359203 & 0.314636 & 0.372762 & 0.585471 \\ Ours & 0.542257 & 0.447176 & 0.379800 & 0.337939 & 0.394580 & 0.656273 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results of the performance of the translation task. “Transformer (from scratch)” and “Ours” represent the model trained only using the machine translation task and the model trained using NER and bilinear function to encoder and trained decoder only using the NMT task, respectively.