SlowGuess/NAACL_2022_OCR huggingface dataset:
Office of Governor-General during the Japanese colonial period with the view of Empire of Japan who ruled Korea<sup>6</sup>.
The mean absolute error of the predicted order of each king achieved the difference around one, except for the first King Taejo and the second King Jeongjong, whose errors are almost doubled. We hypothesize that this is mainly because there are too small number of examples in those classes. A similar tendency where the more samples are, the less mean absolute error be has been observed in other
classes. Also, the writing style of AJD had settled down from the third King Taejong, and those noisy records might confuse models to predict the exact dates.
6.2 Do Time Changes Affect Written Texts?
It stands to reason that AJD written in five centuries reveal the features of the language changes. In this section, we investigate the hypothesis above with respect to the named entities and  $n$ -grams.
(a) Taejo (1st King)
(b) Seongjong  $(9^{\mathrm{th}}$  King)
Figure 2: Chronological attribution results with AnchiB-ERT + AJD/DRS. Figure (a) shows the confusion matrix. In Figure (b), each bar indicates the mean absolute error and the line indicates the number of samples for each king.
Figure 3: Number of overlapped trigrams per king era. It shows the changes of trigrams over kings whose x-axis shows the kings and the y-axis shows the number of overlapped trigrams.
(c) Hyojong  $(17^{\mathrm{th}}$  King)
(d) Cheoljong  $(25^{\mathrm{th}}$  King)
Words Change over Time We analyze how frequently words change over time. For each king, we plot how many trigrams overlap by each king era in the order. Figure 3 shows overlapped trigrams in the 1st, 9th, 17th, and 25th king and the detailed results with all kings are described in Appendix. It is consistently observed that the closer the king era is, the more trigrams are overlapped. These changes result from not named entity but lexical choices, considering that person and location enti
http://esilllok-history.go.kr/
(a) Person
Figure 4: Relative frequency change of top-10 named entities per king. Each line indicates the change of the relative frequency of one entity over time, and the color of the line indicates the king era in which the entity is contained in the top-10 entities. The x-axis represents the time (the kings in the order) and the y-axis represents the relative frequency in each king era.
(b) Location
ties account for  $6.38\%$  and  $2.05\%$  in the characters of AJD, respectively. It verifies that words used in Joseon dynasty had changed over time gradually, and it enables the language models to capture those features.
Named Entity Changes over Time We investigate how the named entities had been used over time. In particular, we show frequency rates of top-10 frequently-used named entities by each king era and how they change over time in Figure 4. It implies a strong correlation between person entity and the passage of time, while there is no explicit correlation to location entity. Most person entities include officials of the time or the previous kings, relevant to the time. In contrast, most location entities include neighboring countries or place names in the Joseon, which are less dependent on the time. The examples of frequently appeared named entities are described in Appendix.
ML based NLP techniques have been recently applied to anthology to discover historical documents such as authorship attribution (Ouamour and Sayoud, 2012; Sayoud and Ouamour, 2017; Reisi and Mahboob Farimani, 2020; Hossain et al., 2020), NER (Won et al., 2018; Palladino et al., 2020), and manuscript age detection (Adam et al., 2018). Assael et al. (2022) proposed Ithaca to restore ancient Greek inscriptions and perform geographical attribution and chronological attribution of them.
Along with these works, several works provide language models suited for historical texts in ancient languages and evaluate those models on existing NLU tasks, which aims to support understanding those documents considering that the target languages are mostly extinct. Bamman and Burns (2020) propose Latin BERT for part-of-speech tagging in ancient Latin script. Tian et al. (2021) suggest AnchiBERT and evaluate their model on some NLP tasks including poem topic classification.
However, there has been no research attempting to propose language models in Hanja, which is a dead language in Korea but absolutely necessary to explore Korean history. Most of the studies with Hanja only shed lights on translating historical Hanja documents and use AJD as their corpus (Park et al., 2020; Jin et al., 2020; Kang et al., 2021).
We present HUE (Hanja Understanding Evaluation) dataset and BERT-based pretrained language models for classical Hanja documents. HUE dataset includes diverse tasks that can support analyzing historical documents written in Hanja which is an extinct language in Korea: Chronological Attribution (CA), Topic Classification (TC), Named Entity Recognition (NER), and Summary Retrieval (SR). Our models pretrained on Hanja corpora outperform other language models and we observe their performance on zero-shot settings with DRRI which is the dataset never been introduced in NLP community. The experimental results in king prediction imply that our models capture the historical events or facts disclosed in the texts. We also explore several methods to support Hanja language models such as masking named entities and giving document age as input sources, based on the analyses on textual features in AJD.
Help of adequate resources in Hanja documents might could fill some caveats in our work which lacks additional experiments and analyses on the records of different genre such as poetry, novel, and humanities resulting from the low resources that we can exploit. However, we expect that our dataset and accompanying language models might facilitate future works on historical documents written in Hanja by providing fundamental resources to leverage unknown Hanja corpora.
We would like to thank Yoonman Heo (Institute for the Translation of Korean Classics) providing expertise on hanja and Korean Classical Chinese. This research was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921). This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00421, Artificial Intelligence Graduate School Program (Sungkyunkwan University)). Kyunghyun Cho was supported by the NYU Center for Data Science National Science Foundation (Award 1922658) and Samsung Advanced Institute of Technology (Next Generation Deep Learning: from pattern recognition to AI).
Kalthoum Adam, Asim Baig, Somaya Al-Maadeed, Ahmed Bouridane, and Sherine El-Menshawy. 2018. Kertas: dataset for automatic dating of ancient arabic manuscripts. International journal on Document Analysis and Recognition (IJDAR).
Yannis Assael, Thea Sommerschield, Brendan Shillingford, Maharay Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag, and Nando de Freitas. 2022. Restoring and attributing ancient texts using deep neural networks. Nature, 603(7900):280-283.
JinYeong Bak and Alice Oh. 2015. Five centuries of monarchy in Korea: Mining the text of the annals of the Joseon dynasty. In Proceedings of the 9th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH).
David Bamman and Patrick J. Burns. 2020. Latin bert: A contextual language model for classical philology.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL.
Shantanu Godbole and Sunita Sarawagi. 2004. Discriminative methods for multi-labeled classification. In Advances in Knowledge Discovery and Data Mining.
Anika Samiha Hossain, Nazia Akter, and Md. Saiful Islam. 2020. A stylometric approach for author attribution system using neural network and machine learning classifiers. In Proceedings of the International Conference on Computing Advancements.
KyoHoon Jin, JeongA Wi, KyeongPil Kang, and YoungBin Kim. 2020. Korean historical documents analysis with improved dynamic word embedding. Applied Sciences.
Kyeongpil Kang, Kyohoon Jin, Soyoung Yang, Soojin Jang, Jaegul Choo, and Youngbin Kim. 2021. Restoring and mining the records of the Joseon dynasty via neural language modeling and machine translation. In Proceedings of the NAACL: Human Language Technologies.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. CoRR.
Siham Ouamour and Halim Sayoud. 2012. authorship attribution of ancient texts written by ten arabic travelers using a smo-svm classifier. In International Conference on Communications and Information Technology (ICCIIT).
Chiara Palladino, Farimah Karimi, and Brigitte Mathiak. 2020. Ner on ancient greek with minimal annotation. In https://dh2020.adho.org/.
Chanjun Park, Chanhee Lee, Yeongwook Yang, and Heuseok Lim. 2020. Ancient korean neural machine translation. IEEE Access.
Ehsan Reisi and Hassan Mahboob Farimani. 2020. Authorship attribution in historical and literary texts by a deep learning classifier. journal of Applied Intelligent Systems and Information Sciences.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. The Probabilistic Relevance Framework (PRF).
Halim Sayoud and Siham Ouamour. 2017. Score fusion based authorship attribution of ancient arabic texts. In Florida Artificial Intelligence Research Society Conference.
Huishuang Tian, Kexin Yang, Dayiheng Liu, and Jiancheng Lv. 2021. Archibert: A pre-trained model for ancient Chinese language understanding and generation. In 2021 International Joint Conference on Neural Networks (IJCNN).
Margarida Vale de Gato. 2015. The collaborative anthology in the literary translation course. The Interpreter and Translator Trainer.
Miguel Won, Patricia Murrieta-Flores, and Bruno Martins. 2018. Ensemble named entity recognition (ner): Evaluating ner tools in the identification of place names in historical corpora. Frontiers in Digital Humanities.
William J Youden. 1950. Index for rating diagnostic tests. Cancer, 3(1):32-35.
A Model
Table 8 shows hyperparameter settings of our models. We used Intel(R) Xeon(R) Silver 4114 (40 CPUs) and GeForce RTX 2080 Ti 10GB (4 GPUs) for all experiments including training, fine-tuning, and inference.
Hyperparameter	Value
Batch Size	32
Early Stopping Patience	3
Hidden Size	768
Learning Rate	2e-5
Learning Rate Scheduler	Linear
Max Sequence Length	512
Number of Hidden Layers	12
Optimizer	AdamW
Vocab Size	11270
B HUE Dataset
B.1 Dataset Size
Table 8: Model configuration
Train	Dev	Test
CA	330,469	41,309	41,309
TC	330,424	41,303	41,304
NER	385,915	13,417	13,418
SR	169,840	21,570	21,296
Table 9: Data split in HUE dataset
B.2 Source Corpora
B.2.1 Data Collection Process
We crawl AJD<sup>7</sup>, DRS<sup>8</sup>, and DRRI<sup>9</sup> from the comprehensive database for Korean classics which are publicly available published by IKTC. All source corpora are fully tagged with the written ages and named entities, while their entity maps differ to each other. AJD also provides topics which is tagged by the experts in the translation process.
B.2.2 Dataset Preprocessing
Table 10 shows good and bad example in DRRI to use as summary retrieval dataset. Bad examples mostly written from the  $21^{\mathrm{st}}$  King Yeongjo to early in the  $22^{\mathrm{nd}}$  King Jungjo describe daily lives of the crown prince who is King Jeongjo. The bad example in Table 10 is depicting his study. These examples tend to present extremely short moks which cannot be treated as summary and content, while the official records on administrative has much longer moks.
C.1 Trigram Changes Over Time
Figure 5 shows the changes of trigrams over all kings. It clearly delivers the changes of trigrams as time goes by. We can observe the same trend in either unigram or bigram.
C.2 Top-5 Named Entities by Kings
Table 11 shows top-5 person and location named entities in three King reigns. All person entities except King Sejong, the most frequent entity in Munjong, are officials in the reign periods, which is different by time changes. In contary, all location entities are the name of place, palace or site, showing some entities overlap among kings.
D Experimental Results
For CA, we measure the Quadratic Weighted Kappa score (QWK score) as metrics that treat each king label hierarchically.
Since TC is a multi-label classification task whose example might have multiple labels as the answer, we measure Hamming score along with accuracy. In this case, accuracy is the exact match score, and the hamming score is the accuracy of subset matched,  $|T \cap P| / |T \cup P|$ , where  $T$  is set of true labels and  $P$  is set of predicted labels (Godbole and Sarawagi, 2004). For the evaluation results in Table 13, we find and set the best threshold to each label by Youden's index. All pretrained models outperform BERT without pretraining, and two LMs re-trained on hanja documents show the best performances.
7http://sillok-history.go.kr/main/main. do
8http://sjw-history.go.kr/main.do
http://kyudb.snu.ac.kr/series/main.  
do?item_cd=ILS
Good Examples	Bad Examples
Date	King Jeongjo 17 (1793) Feb 06	King Yeongjo 50 (1774) May 15
Gang	遞承旨徐榮輔以沈晉賢代之前望也	行召對於尊賢閣。兼弼善洪景顔。說書李駿。翊贊李應重。
Mok	榮輔不仕進政院請牌招以許遞前望單子入之待下批牌招察任	講續綱目
Gang (En)	Royal secretary Yeong-bo Seo was replaced and Jin-hyeon Shim was appointed.	Crown Prince held a royal lecture at the Office of Crown Prince. Kyung-ahn Hong, a lecturer for the Crown Prince, Sangjoon Lee, the second tutor of the Office of Lectures for the Crown Prince, and Eungjoon Lee, a Guard of Crown Prince attended.
They delivered the lecture on《Sokgangmok (Comprehensive Mirror for Aid in Government)》.
Mok (En)	When Yeong-bo Seo did not resign, the king summoned his servants through his royal secretary and ordered him to do so, saying, “wait for appointing the royal secretary among candidates and let him check his job”.
Table 10: Good and bad examples in DRRI
Figure 5: Trigram changes over time
King
(Reigning period)	Munjong (5thKing)
(1450-1452)	Seonjo (14thKing)
(1567-1608)	Sunjo (23rdKing)
(1790-1834)
Person	世宗	2.42%	柳成龍	1.08%	南公輓	1.21%
金宗瑞	2.26%	李德馨	0.81%	金載瓊	1.09%
李季甸	1.97%	尹斗壽	0.72%	李時秀	0.85%
皇甫仁	1.82%	李恒福	0.65%	沈象奎	0.83%
鄭苯	1.78%	李元翼	0.57%	李相璜	0.77%
King Sejong	(1392-1397)	Seongryong Ryu	(1542-1607)	Gongcheol Nam	(1760-1840)
Jongseo Kim	(1383-1453)	Deokhyeong Lee	(1561-1613)	Jaechan Kim	(1746-1827)
Person (En)	Kyejeon Lee	(1404-1459)	Dushu Yun	(1533-1601)	Sisu Lee	(1745-1821)
Boin Hwang	(1387-1453)	Hangbok Lee	(1556-1618)	Sangkyu Shim	(1766-1838)
Bun Jeong	(1394-1454)	Weonik Lee	(1547-1634)	Sanghwang Lee	(1763-1841)
Location	平安	4.40%	平壤	2.10%	春塘臺	2.02%
咸吉	3.24%	朝鮮	1.93%	漢城府	1.19%
黃海	2.28%	京畿	1.17%	仁政殿	1.71%
輝德殿	2.21%	全羅	1.76%	平安	1.51%
京畿	2.10%	慶	1.64%	景慕宮	1.49%
Pyong-an	(Province)	Pyongyang	(Province)	Chundangdae	(Site)
Hamgyong	(Province)	Joseon	(Country)	Hanseong Magistracy	(Province)
Location (En)	Hwanghae	(Province)	Gyeonggi	(Province)	Injeongjeon	(Hall)
Hwideokjeon	(Hall)	Jeolla	(Province)	Pyong-an	(Province)
Gyeonggi	(Province)	Gyeongsang	(Province)	Gyeongmogung	(Palace)
Table 11: Top-5 named entities in  ${5}^{\text{th }},{14}^{\text{th }}$  ,and  ${23}^{\text{rd }}$  kings
Acc	F1	Pre	Rec	QWK
BERT not pretrained	56.59	54.26	55.08	56.59	87.09
AnchiBERT (Tian et al., 2021)	76.31	75.74	76.45	76.31	93.99
mBERT (Devlin et al., 2019)	76.02	75.29	75.80	76.02	93.76
AnchiBERT + AJD/DRS	79.50	79.33	80.06	79.50	95.46
mBERT + AJD/DRS	77.99	77.78	78.92	77.99	95.04
Table 12: Evaluation results for our PLMs on chronological attribution
Major (4 classes)	Minor (106 classes)
Acc	F1	Pre	Rec	Ham	Acc	F1	Pre	Rec	Ham
BERT not pretrained	68.52	68.91	74.82	70.65	79.04	26.48	61.29	54.86	84.59	42.45
AnchiBERT (Tian et al., 2021)	68.99	85.81	85.24	88.44	83.00	31.47	69.30	61.63	90.26	51.63
mBERT (Devlin et al., 2019)	56.78	79.59	77.60	87.78	73.64	32.85	70.01	62.17	90.84	54.28
AnchiBERT + AJD/DRS	70.48	88.33	86.61	92.71	84.80	31.77	72.07	64.87	91.24	52.25
mBERT + AJD/DRS	69.15	87.13	85.48	91.36	83.81	33.96	71.28	63.50	91.56	55.91
Table 13: Evaluation results for our PLMs on topic classification
Overall	Person	Location
Acc	F1	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec
BERT not pretrained	98.67	89.40	90.58	88.25	92.13	93.98	90.36	87.10	87.76	86.46
AnchiBERT	98.72	90.30	90.98	89.62	93.13	94.47	91.83	87.91	88.08	87.75
mBERT	98.52	88.57	89.52	87.64	91.63	93.60	89.74	86.02	86.18	85.85
AnchiBERT + AJD/DRS	98.76	90.42	91.31	89.55	93.28	94.77	91.84	88.01	88.43	87.59
mBERT + AJD/DRS	98.69	90.16	90.36	89.95	92.83	93.51	92.17	87.90	87.74	88.06
Table 14: Evaluation results for our PLMs on named entity recognition
MRR	Top-1	Top-10
BM25	29.87	25.58	33.98
BERT not pretrained	52.85	99.20	99.64
AnchiBERT (Tian et al., 2021)	67.92	99.20	99.85
mBERT (Devlin et al., 2019)	67.06	99.32	99.50
AnchiBERT + AJD/DRS	74.29	99.64	99.91
mBERT + AJD/DRS	73.88	99.44	99.59
Table 15: Evaluation results for our PLMs on summary retrieval