SlowGuess/NAACL_2021_OCR huggingface dataset:
where  $\| \cdot \| _F$  represents the Frobenius norm, and  $\psi$  and  $\alpha$  represent the  $L_{1}$  regularization function and the regularization weight, respectively. We set the number of topics  $K$  as  $20^{8}$  and the regularization weight  $\alpha$  as 0.1.
This section describes the results of the performances of our model for restoration and translation, followed by qualitative examples of each task as well as topic modeling results.
5.1 Document Restoration
We evaluate the performance of our model on the document restoration task on the test dataset. We also compare performance between the model trained with and without multi-task learning. Table 3 shows the results of top-K (HITS@K). The top-10 accuracy of our proposed model is almost  $89\%$ , which indicates the high performance of our model and demonstrates that our model provides analysts with appropriate options. However, the baseline model, trained without multi-task learning, performs slightly better than the one with multi-task learning. This shows that the baseline model is more specialized in the document restoration task. However, although our model performance is slightly lower than the baseline model, the benefits of the multi-task learning approach are significantly manifested in the NMT task as shown in Table 5. As our model shows the acceptable performances on both the restoration and the translation tasks, we conclude that our model learns the purpose of our research well via multi-task learning. We will further discuss the main benefits of multi-task learning in Section 5.2.
We further investigate the qualitative results of the document restoration task. Table 2 shows four randomly sampled, example pairs. As shown in the first three rows of this table, the model also has the ability to predict bi-gram and tri-gram character-level tokens because the model is trained using
the results of NMF are slightly better than those of LDA.
<sup>8</sup>We set the number of topics as 20 after we conducted experiments by varying the topic numbers, such as 10, 15, 20, 30, and 50.
7Topic modeling includes several methods such as latent Dirichlet allocation (LDA) (Blei et al., 2003)-based and nonnegative matrix factorization-based models (Lee and Seung, 2001). We additionally tested topic modeling with LDA, but
Original
Predicted	上在慶德宮. 停常參. 經筵.
上在慶熙宮. 停常參. 經筵.
Original
Predicted	右承旨李世用疏曰云云. 省疏具悉. 疏辭, 下該曹稟處.
右承旨李世用疏曰云云. 省疏具悉. 疏辭, 下該曹稟處.
Original
Predicted	玉堂子. 答曰, 省具悉. 辭當採用焉. 内下記草
玉堂子. 答曰, 省具悉. 辭當採用焉. 内下記草
Original
Predicted	又曰, 假注書金基龍, 身病猝重, 势難察任, 今姑改差, 何如? 傅曰, 允.
又曰, 假注書李基淳, 身病猝重, 势難察任, 今姑改差, 何如? 傅曰, 允.
Table 2: Our model prediction results. Blue- and red-colored letters represent masked and predicted ones, respectively.
HITS@1	HITS@5	HITS@10
Baseline	77.83%	88.29%	90.89%
Full model	75.20%	86.21%	89.09%
Table 3: Top-K accuracies for the restoration task.
n-gram-based MLM. Furthermore, although each character is not exactly the same as the original one, the last example in the table shows that our model restores the proper format of the name part. However, predicting the exact name is a difficult task for human experts, even when considering the context of the sentence, as prior knowledge is necessary to predict the exact name. Therefore, we quantitatively measured the model performance on the proper nouns, e.g. person and location names, using 200 samples of them. The average top-10 accuracy is only  $8.3\%$ , significantly lower than the overall accuracy, which is larger than  $89\%$ . We conjecture that the degradation is mainly due to the difficulty in maintaining the information of the proper nouns, which would require external knowledge. We leave it as our future work.
5.2 Machine Translation Quality
To investigate the performance of the machine translation task, we translate the Hanja sentences in the test dataset and then evaluate the model performance. As shown in Table 5, the results for the translation task are evaluated by BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE-L (Lin, 2004). In this result, "Full" represents our proposed model trained by multitask learning of the translation and the restoration tasks. Therefore, the model is trained to take both the translated and untranslated sentences. On the other hand, "Base" represents the model trained only by the translation task, and thus, the model is trained to accept only the translated sentences. Our model outperforms the baseline model with a significant margin.
Furthermore, we generate sentences using the beam search method with the length normalization. In this study, we compare the greedy search and the beam search with a beam size of 3. As shown in Table 5, results obtained with a beam size of 3 are slightly better than the greedy search method. Finally, the BLEU score of our model is obtained as 0.5410, which indicates that our model performs reasonably well, compared to other recent models trained in other languages.
We additionally compared our model to the model trained via the pretraining-then-finetuning approach. As shown in Table 6, the BLEU score of this approach is 0.3755, which is  $5.9\%$  higher than that of the model trained from scratch but  $28.7\%$  lower than our multi-task learning approach. The results can be explained for two reasons. First, as the size of unpaired data is much larger than that of paired data, the multi-task learning fully utilizes the paired and unpaired data for the translation task, compared to the pretraining-then-finetuning approach. Second, The pretraining-then-finetuning approach has a catastrophic forgetting problem (Chen et al., 2020). In other words, the finetuning step can fail to maintain the knowledge acquired at the pretraining step. However, as both reconstruction and translation tasks are crucial for historical documents, such a forgetting issue is critical to our tasks.
We also tested the quality of the Hanja-Korean translation task using a Chinese-Korean machine translator. As few publicly available machine translation models for Chinese-Korean exist, we used Google Translate instead. The translator failed to translate given Hanja sentences in most cases, mainly because Hanja and Chinese have different properties in terms of grammar and word meanings.
To investigate the translation performance qualitatively, we sampled translated samples. Table 4
<sup>9</sup>https://translate.google.co.kr
Original
Predicted
Predicted (Eng.)	上在昌慶宮.停常參·經筵.
상이 charteng公关에 툘렸다.상화과 전쟁을 썼지收到了.
King was in the Changkyeong palace. He stopped the discussion of political affairs with other officers.
Original
Predicted
Predicted (Eng.)	答大司憲南龍翼疏曰,省疏具悉.內局提調之任,當勉副,卿其勿辭,救護母病,從速上來察職.
내사원 남용AIR의상ESCO에 갑가기를,“상ESCO을 bg고 갑일都不会.내국제조의 직人在부지려해마지목해공有的들을 사라수니,공은 사고 하지에 mock해울라와 직일을 죽PUR Rahman.”涨停了.
Replying to the Prosecutor General Namyongik's memorial, the king said, “I looked at the memorial and thoroughly understood what it meant. As the position of the director at the office of the royal physicians cannot help but agree to your message, you should not resign your position, care for your mother's illness, and come back to be responsible for your duties quickly.”
Original
Predicted
Predicted (Eng.)	夜一更,月暈.五更,西方坤方,有氣如火光.
dawn 1GeV에 갑무리가战胜了.5GeV에 출방,相关内容在上海的番道上的位置出现了.
The moon has a ring around it at 7-9 PM. At 3-5 AM, there was the light of the fire in the west and south-west.
Table 4: Examples of original Hanja sentences, ground-truth sentences, and predicted sentences. For readability, we appended English sentences corresponding to the predicted sentences in each row.
BLEU	METEOR	ROUGE-L
Base (1)	0.3547	0.3488	0.6082
Base (3)	0.3536	0.3482	0.6127
Full (1)	0.5269	0.4594	0.7463
Full (3)	0.5410	0.4719	0.7606
Table 5: Results of the performance of the translation task. "Base" and "Full" represent the model trained only using the machine translation task and the model trained using multi-task learning with machine translation and restoration tasks, respectively.
Multi-task	Scratch	Pipelining
BLEU	0.5410	0.3536	0.3755
Table 6: BLEU scores of the models trained with multi-task learning, scratch, and the pretraining-then-finetuning approach (pipelining), respectively.
shows the sentences translated from the untranslated documents by our model. For readability, we append English sentences corresponding to the predicted sentences in each row. Each result indicates that our model generates the modern sentences corresponding to contexts of the source Hanja sentences. Interestingly, the third example in the table is related to the astronomical observation of the aurora. Later, we found prior studies confirming that the red energy mentioned in our document was an aurora (Zhang, 1985; Stephenson and Willis, 2008). This highlights the importance of the ma
chine translation task of the historical records, as it is essential to survey by researchers in various fields such as astrophysics and geology. Therefore, we further analyze the documents with the topic modeling approach.
5.3 Results of Topic Modeling
As described in Section 4.3, we calculate the term-topic weight matrix  $\mathbf{W}$  and date-topic weight matrix  $\mathbf{H}$ . We select three interesting topics from the total of  $K$  topics and visualize the term-topic weights in  $\mathbf{W}$  using the word cloud and the datatopic matrix  $\mathbf{H}$  in a smoothed time-series graph for each topic. Fig. 4 shows the results.
The first topic is related to troops and military exercise. As shown in the red dashed box in the time-series graph, the weights dramatically decrease in 1882, while the weights continuously increase after the biggest war in 1592. In fact, a coup attempt of the old-fashioned soldiers occurred in 1882, causing the national intervention of neighboring countries and the decline of self-reliant defense. The fifteenth topic is related to war and national defense. Although this topic is related to the preceding military topic, it is more related to the international relationship compared to the first one. In the early years of the dynasty, northern enemies and pirates frequently invaded Joseon, which reveals as the large topical weights in the beginning.
Figure 4: Three topics extracted from topic modeling. We translated topic keywords into English for readability.
The weights increase in the late sixteenth century, and the weight maintains at a high level until 1637 when three great wars broke out in Joseon.
The eighteenth topic is related to astronomical observations such as a halo and a meteor shower. In the mid-sixteenth century, people observed the Leonids, as shown in the first red box of the graph. We later found that experts in astronomy also discovered this in the past, using AJD (Yang et al., 2005). Moreover, from the mid-seventeenth century to the early eighteenth century, the number of sunspots was low. Solar observers name this event as the Maunder minimum (Eddy, 1976; Shindell et al., 2001). This event caused abnormal climate phenomena, such as the third example in Table 4, as shown in the second red box of the graph. This topic demonstrates the importance of the use of historical records since it is difficult to easily spot the phenomena that occurred centuries ago.
Note that previous studies mainly attempted to exploit only AJD or translated parts of DRS. However, we utilize both AJD and the majority of DRS records by applying advanced NMT techniques. When performing topic modeling by using only those manually translated sentences, it failed to include topics such as the health of the royal families and actions against treason sinners, which were revealed by our approach. It is because the voluminous documents that have not been manually translated contain their own topics. Thereby, we extract several valuable topics even with no special knowledge in the Hanja domain. Translating the historical records into modern languages expands our knowledge base, and analysis of the records using machine translation and text mining tech
niques may help the analysts effectively explore the historical records.
In this paper, we proposed a novel approach to translate and restore the historical records of the Joseon dynasty by formulating the multi-task learning task based on the self-attention mechanism. Our approach significantly increases the translation quality by learning the rich contents in large documents. We anticipate these tasks are the first steps towards translating the ancient Korean historical records into modern languages such as English. Furthermore, the model effectively predicts the original words from the damaged parts of the documents, which is an essential step for restoring the damaged documents. Results from text mining approaches show that our approaches have the potential in supporting analysts in effectively exploring the large volume of historical documents. We also expect researchers from diverse domains can explore documents and discover historical findings such as astronomical phenomena and undiscovered international affairs, with no special domain knowledge. As future work, we will also leverage the transfer learning approach to translate historical documents into other languages, such as English or French. We also plan to apply knowledge graph-based machine learning approaches, e.g. knowledge graph embedding and graph neural networks, to discover historical events and relations.
This work was supported by Institute for Information & communications Technology Planning
& Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques, No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), and No.2021-0-01341, Artificial Intelligence Graduate School Program (Chung-Ang University)) and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No.NRF-2019R1A2C4070420).
Yannis Assael, Thea Sommerscheid, and Jonathan Prag. 2019. Restoring ancient text using deep learning: a case study on Greek epigraphy. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6368-6375, Hong Kong, China. Association for Computational Linguistics.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5360-5369, Hong Kong, China. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. the International Conference on Learning Representations (ICLR).
JinYeong Bak and Alice Oh. 2015. Five centuries of monarchy in korea: mining the text of the annals of the joseon dynasty. In Proceedings of the SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities.
JinYeong Bak and Alice Oh. 2018. Conversational decision-making model for predicting the king's decision in the annals of the joseon dynasty. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the Proc. the Annual Meeting of the Association for Computational Linguistics (ACL) workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research.
Gulcin Caner and Ismail Haritaoglu. 2010. Shape-dna: effective character restoration and enhancement for arabic text documents. In International Conference on Pattern Recognition.
Kai Chen, Mathias Seuret, Jean Hennebert, and Rolf Ingold. 2017. Convolutional neural networks for page segmentation of historical document images. In International Conference on Document Analysis and Recognition.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870-7881, Online. Association for Computational Linguistics.
Tarin Clanuwat, Alex Lamb, and Asanobu Kitamoto. 2019. Kuronet: Pre-modern japanese kuzushiji character recognition with deep learning. In 2019 International Conference on Document Analysis and Recognition (ICDAR).
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019. Electra: Pre-training text encoders as discriminators rather than generators. In Proc. the International Conference on Learning Representations (ICLR).
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Proc. the Advances in Neural Information Processing Systems (NIPS).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAProc. the Annual Meeting of the Association for Computational Linguistics (ACL).
Miguel Domingo and Francisco Casacuberta Nolla. 2018. Spelling normalization of historical documents by using a machine translation approach. In Proceedings of the Conference of the European Association for Machine Translation.
John A Eddy. 1976. The maunder minimum. Science.
Hisashi Hayakawa, Kiyomi Iwahashi, Yusuke Ebihara, Harufumi Tamazawa, Kazunari Shibata, Delores J Knipp, Akito D Kawamura, Kentaro Hattori, Kumiko Mase, Ichiro Nakanishi, et al. 2017. Long-lasting extreme magnetic storm activities in 1770 found in historical documents. The Astrophysical Journal.
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proc. the Advances in Neural Information Processing Systems (NIPS).
Junhyeok Jeon, Sung-Jun Noh, and Dong-Hee Lee. 2018. Relationship between lightning and solar activity for recorded between ce 1392–1877 in korea. Journal of Atmospheric and Solar-Terrestrial Physics.
Ho Chul Ki, Eun-Kyoung Shin, Eun Jin Woo, Eunju Lee, Jong Ha Hong, and Dong Hoon Shin. 2018. Horse-riding accidents and injuries in historical records of joseon dynasty, korea. International journal of paleopathology.
Hannah Kim, Jaegul Choo, Jingu Kim, Chandan K Reddy, and Haesun Park. 2015. Simultaneous discovery of common and discriminative topics via joint nonnegative matrix factorization. In Proc. the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Neethu S Kumar, Dinesh S Kumar, S Swathikiran, and Alex Pappachen James. 2014. Ancient indian document analysis using cognitive memory network. In International Conference on Advances in Computing, Communications and Informatics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In Proc. the International Conference on Learning Representations (ICLR).
Daniel D Lee and H Sebastian Seung. 2001. Algorithms for non-negative matrix factorization. In Proc. the Advances in Neural Information Processing Systems (NIPS).
Ki-Won Lee, Hong-Jin Yang, and Myeong-Gu Park. 2009. Orbital elements of comet c/1490 y1 and the quadrantid shower. Monthly Notices of the Royal Astronomical Society.
Yanyang Li, Tong Xiao, Yinqiao Li, Qiang Wang, Changming Xu, and Jingbo Zhu. 2018. A simple and effective approach to coverage-aware neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 292-297, Melbourne, Australia. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Dayiheng Liu, Kexin Yang, Qian Qu, and Jiancheng Lv. 2019a. Ancient-modern chinese translation with a new large training dataset. ACM Transactions on Asian and Low-Resource Language Information Processing.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019b. On the variance of the adaptive learning rate and beyond. In Proc. the International Conference on Learning Representations (ICLR).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692.
David Mimno. 2012. Computational historiography: Data mining in a century of classics journals. Journal on Computing and Cultural Heritage.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. the Annual Meeting of the Association for Computational Linguistics (ACL).
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog.
Drew T Shindell, Gavin A Schmidt, Michael E Mann, David Rind, and Anne Waple. 2001. Solar forcing of regional climate change during the maunder minimum. Science.
F Richard Stephenson and David M Willis. 2008. 'vapours like fire light' are korean aurorae. *Astronomy & Geophysics*.
Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre. 2018. An evaluation of neural machine translation models on historical spelling normalization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1320-1331, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. the Advances in Neural Information Processing Systems (NIPS).
Hong-Jin Yang, Changbom Park, and Myeong-Gu Park. 2005. Analysis of historical meteor and meteor shower records: Korea, china, and japan. Icarus.
Tze-I Yang, Andrew Torget, and Rada Mihalcea. 2011. Topic modeling on historical newspapers. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 96-104, Portland, OR, USA. Association for Computational Linguistics.