HistRED: A Historical Document-Level Relation Extraction Dataset
Year: 2023

Abstract:
Despite the extensive applications of relation extraction (RE) tasks in various domains, little has been explored in the historical context, which contains promising data across hundreds and thousands of years. To promote the historical RE research, we present HistRED constructed from Yeonhaengnok. Yeonhaengnok is a collection of records originally written in Hanja, the classical Chinese writing, which has later been translated into Korean. HistRED provides bilingual annotations such that RE can be performed on Korean and Hanja texts. In addition, HistRED supports various self-contained subtexts with different lengths, from a sentence level to a document level, supporting diverse context settings for researchers to evaluate the robustness of their RE models. To demonstrate the usefulness of our dataset, we propose a bilingual RE model that leverages both Korean and Hanja contexts to predict relations between entities. Our model outperforms monolingual baselines on HistRED, showing that employing multiple language contexts supplements the RE predictions. The dataset is publicly available at: https://huggingface.co/ datasets/Soyoung/HistRED under CC BYNC-ND 4.0 license.

heading: 1 Introduction
text: Relation extraction (RE) is the task of extracting relational facts from natural language texts. To solve RE problems, diverse datasets and machine learning (ML) methods have been developed. Earlier work limits the scope of the problem to sentencelevel RE, in which the task is to predict a relationship between two entities in a single sentence (Doddington et al., 2004; Walker et al., 2006; Hendrickx et al., 2010; Alt et al., 2020; Stoica et al., 2021). However, such a setting is impractical in real-world applications where relations between entities can exist across sentences in large unstructured texts. Therefore, document-level RE datasets for general and biomedical domains have been introduced (Li
et al., 2016; Yao et al., 2019; Wu et al., 2019; Zaporojets et al., 2021; Luo et al., 2022), serving as benchmarks for document-level RE mod-
3207
els (Huguet Cabot and Navigli, 2021; Tan et al., 2022; Xiao et al., 2022; Xie et al., 2022; Xu et al., 2021).
Despite the vast amount of accumulated historical data and the ML methods available for extracting information from it, research on information extraction targeting historical data has been rarely conducted. We believe this is due to the high complexity of analyzing historical records which are written in early languages and cover hundreds and thousands of years. For instance, early languages pose a challenge for accurate translation and knowledge extraction due to their differences in expressions, styles, and formats compared to contemporary languages. Also, since historical records are translated a long time after their creation, reading bilingual texts is necessary to fully understand the text. Such discrepancy requires domain experts who are able to understand both languages in order to accurately annotate the data. There has been a demand from historical academics to utilize ML algorithms to extract information from the huge amount of records; however, because of the aforementioned challenges, the historical domain has been overlooked by most ML communities.
In response, we introduce HistRED, a documentlevel RE dataset annotated on historical documents for promoting future historical RE studies. HistRED contains 5,816 documents extracted from 39 books in the Yeonhaengnok corpus (see Section 2 for details). As described in Table 11, our dataset is the first dataset that extracts relational information from the historical domain and dif-
1The statistics of our dataset is calculated when SL is 2.
fers from other RE datasets in that it supports both sentence-level and document-level contexts, as well as two languages: Korean and Hanja. Furthermore, researchers can select different sequence levels (SL), which we define as a unit of context lengths, when evaluating their RE models. Such independent subtexts are constructed by considering evidence sentences, which the annotators have tagged. The intuition is that evidence sentences, which provide context for deriving a certain relation between two entities, should not be separated from the original text when splitting a document; thus, we introduce an algorithm that properly splits a full document into several self-contained subtexts. Finally, we propose a novel architecture that can fully utilize bilingual contexts using pretrained language models (PLMs). Experimental results demonstrate that our bilingual RE model outperforms other monolingual ones.
Our contributions are summarized as follows:
• We introduce HistRED, a historical RE dataset built from scratch on Yeonhaengnok, a historical record written between the 16th and 19th centuries.
• We define new entity and relation types fit for our historical data and proceed with the dataset construction in collaboration with domain experts.
• We introduce a sequence level (SL) as a unit of varying sequence lengths, which properly splits a full document into several independent contexts, serving as a testbed for evaluating RE models on different context lengths.

heading: 2 Dataset Construction
text: To the best of our knowledge, HistRED is the first RE dataset in the historical domain; thus, there is no consensus regarding the dataset construction process on the historical corpus. In the process of designing our dataset, we collaborated with experts in the linguistics and literature of Hanja to arrive at a consensus. This section describes how we collaborated with the domain experts to construct HistRED without losing annotation quality.

heading: 2.1 Background
text: Joseon, the last dynastic kingdom of Korea, lasted just over five centuries, from 1392 to 1897, and many aspects of Korean traditions and customs trace their roots back to this era. Numerous historical documents exist from the Joseon dynasty, including Annals of Joseon Dynasty (AJD) and Diaries of the Royal Secretariats (DRS). Note that the majority of Joseon’s records were written in Hanja, the archaic Chinese writing that differs from modern Chinese, because the Korean language had not been standardized until much later. We considered a number of available historical texts and selected Yeonhaengnok, taking into account the amount of text and the annotation difficulty. Yeonhaengnok is essentially a travel diary from the Joseon period. In the past, traveling to other places, particularly to foreign countries, was rare. Therefore, intellectuals who traveled to Chung (also referred to as the Qing dynasty) meticulously documented their journeys, and Yeonhaengnok is a compilation of these accounts. Diverse individuals from different generations recorded their business trips following similar routes from Joseon to Chung, focusing on people, products, and events they encountered. The Institute for the Translation of Korean Classics (ITKC) has open-sourced the original and their translated texts for many historical documents, promoting active historical research2.

heading: 2.2 Dataset Schema
text: We engaged in rounds of deliberate discussions with three experts who have studied the linguistics and literature of Hanja for more than two decades and defined our dataset schema.
Documents Written between the 16th and 19th centuries, the books in Yeonhaengnok have different formats and contexts depending on the author
2The entire documents were collected from an open-source database at https://db.itkc.or.kr/
or the purpose of the book. After consulting with the experts, a total of 39 books that contain rich textual information were selected for our dataset, excluding ones that only list the names of people or products. The collection consists of a grand total of 2,019 complete documents, with each document encompassing the text for a single day. This arrangement is made possible because each book separates its contents according to date, akin to a modern-day diary.
Entity and Relation Types Since Yeonhaengnok is a unique record from the Joseon dynasty, entity and relation types used in typical RE tasks are not fit for our dataset. After conferring with the experts, we newly define the entity and relation types appropriate for our historical data. The details are described in Appendix A.2.

heading: 2.3 Annotate and Collect
text: Annotators 15 annotators were recruited, who can comprehend the Hanja texts with the Korean translations and have studied the linguistics and literature of Hanja for at least four years.
Data Annotation The annotation process was divided into two steps: Each annotator first annotates the text from scratch, and then a different annotator cross-checks the annotations. Prior to each step, we provided the annotators with guidelines and promptly addressed any inquiries they had throughout the annotation process. The annotators were instructed to tag four types of information: entities, relation types, coreferences, and evidence sentences. Entities are annotated in both Korean and Hanja texts, whereas the relations between entities are tagged in the Korean text only, reducing redundant workload for the annotators. Coreferences, which are words or expressions that refer to the same entity, are also tagged such that they are all used to represent a single entity during model training. Evidence sentences, which provide context why the entities have a particular relation, are labeled as well, following Yao et al. (2019). For 2,019 parallel texts, the average number of sentences is 24, and the average number of characters in a sentence is 45 in Korean, and 65 and 7 in Hanja, respectively.
Preprocessing The initial annotated data is preprocessed to facilitate model training due to several issues it presents. First, some texts contain quotes from other books and poems, which may be unnec-
essary information for performing the RE task, and thus we exclude them from our dataset. Second, the annotators have found no relation information in some texts either because they were too short or the author of the text had not written any meaningful information. We filter out such texts accordingly. Lastly, the average number of sentences is quite high, with a high variance of 1,503 characters in Korean and 12,812 characters in Hanja. This is because the writing rule of Yeonhaengnok is not stringent. Therefore, we divide these texts with respect to different sequence levels, as described in Section 2.4. Consequently, the original 2,019 texts yield a total of 5,852 data instances3. The mean and the variance of the number of sentences are reduced from 24(1503) to 2(4.15) in Korean and from 65(12812) to 5(57.62) in Hanja.
Statistics of HistRED The collected dataset is split into the training, validation, and test sets, and their statistics are demonstrated in Table 2. Since the sequence length of each document varies, we first sort all data by Korean character lengths, followed by random sampling in a 2:1:1 ratio for the training, validation, and test sets, respectively.

heading: 2.4 Sequence Level
text: A length of a document is a major obstacle to training a PLM such as BERT, which can take sequences of length only up to a specified length, e.g., 512 tokens. Naively, we can split long documents into multiple chunks; however, a problem may arise when the context for identifying a certain relation exists in a different chunk of text. To resolve this issue, we introduce a sequence level (SL), a unit of sequence length for extracting self-contained subtexts without losing context information for each relation in the text. This is achieved since we have instructed the annotators beforehand to mark evidence sentence(s), which are contextual sentences that help identify the corresponding relation. As a result, we can utilize these sentences as indicators when varying the lengths of a document.
Formally, let T ka represent a subtext for relation A when SL is k. Assume two relations exist in separate sentences of a document, i.e., D = [s1, · · · , sn], which consists of n sentences. When SL is 0 and i + 1 < j, the two subtexts can be defined as T 0a = [si, si+1], T 0 b = [sj ], where relation A exists in si and its context in si+1, while relation B exists and has its context
3When SL is 0. The detailed statistics are in Table 2.
in sj . If SL is set as k, each subtext is expanded to T ka = [si−k, · · · , si+k], T kb = [sj−k, · · · , sj+k], where 1 ≤ i − k, 1 ≤ j − k, i + k ≤ n, and j + k ≤ n. Note that the expansion is based on the sentence where the relation exists, i.e., si and sj . If i− k < 1 or j − k < 1, we set the initial index of T k as 1, and if n < i+ k or n < j + k, we set the last index of T k as n.
In addition, we must verify whether duplication occurs between the subtexts. If si+k of T ka becomes the same sentence as sj−k of T kb , we combine two subtexts to a new subtext T ka+b to remove the duplication between them. As shown in Table 2, the size of the dataset decreases as SL increases due to the removal of duplication. Based on this process, we produce five versions of our dataset, where {0, 1, 2, 4, 8} ∈ k. Because our dataset contains the bilingual corpus, the new documents are first generated in Korean text, followed by constructing the corresponding Hanja subtexts.

heading: 3 Data Analysis
text: In this section, we analyze various aspects of HistRED to provide a deeper understanding and highlight several characteristics of our historical data. Table 1 shows the properties and statistical aspects of HistRED with three most related datasets: I.PHI (Assael et al., 2022), DocRED (Yao et al., 2019), and KLUE-RE (Park et al., 2021). The tokenizer of mBERT (Devlin et al., 2019) is utilized to obtain the number of tokens in diverse languages. HistRED is the first dataset comprised of historical texts targeting the document-level RE task. There have been several studies on the historical corpus (Assael et al., 2019, 2022); however, most RE datasets are based on a general or biomedical domain (Yao et al., 2019; Luo et al., 2022), making it hard to derive historical knowledge.
Named Entity Types HistRED contains 10 entity types, including Location (35.91%), Person (34.55%), Number (13.61%), DateTime (4.82%),
and Product (4.40%)4. On average, approximately 11 entities appear in a single document, with the median being 10. The aforementioned types are the five most frequent entity types. This can be explained that Yeonhaengnok is a business-travel journal from Joseon to Chung; thus, the authors described whom they had met and when and where they had traveled. The full description is in Appendix Table 7.
Relation Types Our dataset encloses 20 relation types, including “per:position_held” (32.05%), “nearby” (27.28%), “alternate_name” (7.59%), “per:country_of_citizenship” (5.35%), and “product:provided_by” (3.82%)5. The frequent occurrence of “per:position_held” can be explained by the distinctive writing style during the Joseon dynasty. For instance, people wrote the name of another person along with their title (e.g., “Scientist Alan Turing” rather than “Alan Turing.”) People referred to each other by their titles or alternative names, such as pseudonyms because using a person’s given name implied a lack of respect and courtesy. The second most common relation is “nearby,” which indicates that the place or organization is located nearby6. This demonstrates that the authors were interested in geographic information when traveling. The full description is in Appendix Table 8.
Varying Sequence Length As described in Section 2.4, the input text length can be altered via the sequence level (SL). Table 3 shows a distribution of the number of tokens within a document when SL changes. When SL is 1, our sequence length becomes longer than the sentence-level RE dataset, including KLUE-RE. Additionally, when SL ≥ 4, our dataset exceeds the length of other document-level RE datasets, including DocRED.
Annotation Procedure Statistics Since our dataset construction consists of annotation and cross-checking steps, we summarize the statistics of this procedure. As shown in Table 4, each annotator tagged an average of 51.3 Korean entities, 50.6 Hanja entities, and 4.9 relations on each raw text. At the cross-checking step, a different annotator added an average of 6.5 Korean entities, 6.2
4The percentage is calculated when SL is 1. 5The percentage is calculated when SL is 1, same as entity. 6Since there were no mechanical mobilities and the diplomatic group moved with about 200 people, the authors could not move fast and usually walked inside a city.
Hanja entities, and 0.5 relations, while deleting 2.2 Korean entities, 2.0 Hanja entities, and 0.3 relations. As a result, the final annotations consist of 55.6 Korean entities, 54.8 Hanja entities, and 5.1 relations for each raw text on average.

heading: 4 Bilingual Relation Extraction Model
text: Unlike translation between modern languages, such as translation from English to Korean, historical records have been translated hundreds of years after their creation. As a result, the gap between ancient and present makes the translation task from Hanja into Korean difficult. Also, the translated texts can vary across translators; thus, the domain experts read both Hanja and Korean texts to fully understand the original text. Based on this observation, we hypothesize that understanding the bilingual text would help a model extract valuable information and design our bilingual RE model.
As shown in Figure 2, our model is a joint model of two separate encoders for Hanja and Korean,
along with a cross-attention block from the Transformer architecture (Vaswani et al., 2017). For a document D of length n in Hanja and m in Korean, we have Dhan = [xt]nt=1 and Dkor = [yt] m t=1, where x and y are input tokens of each document. We use the PLM encoder to obtain contextualized embeddings: Hkor, Hhan. Based on these hidden representations, we adopt the multi-head crossattention block, which consists of a cross-attention layer and residual connection layer (Vaswani et al., 2017). For instance, when the encoder process the Hanja text, we set the query as the Hanja token and the key and value to the Korean tokens. Crossattended representation H ′ is defined as
H ′han = softmax(Qhan,Kkor)Vkor, (1)
where we denote query Qhan = WQHhan, key Kkor = WKHkor, and value Vkor = WV Hkor, which are all linear projections of hidden representation H . WQ ∈ Rd×d, WK ∈ Rd×d, and WV ∈ Rd×d are learnable weight matrices. After the cross attention, H ′han is further processed in a residualconnection layer, Zhan = Linear(Hhan +H ′han). We get Zkor in the same manner. Our model pools entity embeddings from Zhan and Zkor. Each bilinear classifier predicts relation types, returning separate logits: logithan and logitkor. At last, our model generates final logits as follows:
logitout = α · logithan + (1− α) · logitkor, (2)
where logit ∈ Rk×c denotes the output logits of k entity pairs for all c relations, and α is a hyperparameter.

heading: 5 Experiments

heading: 5.1 Settings
text: Models Since our dataset consists of two languages, we build separate models for each language. We implement all models based on Huggingface Transformers (Wolf et al., 2020). For Korean, the baselines are mBERT (Devlin et al., 2019), KoBERT (a Korean BERT)7, and KLUE (Park et al., 2021). For Hanja, the baselines are mBERT and AnchiBERT (Tian et al., 2021). For our bilingual model, we consider combinations of these PLMs, i.e., KLUE, KoBERT, and mBERT for the Korean encoder and mBERT and AnchiBERT for the Hanja encoder. In our experiments, the combination of KLUE and AnchiBERT shows consistent
7https://github.com/SKTBrain/KoBERT
scores when varying SL. Therefore, our model consists of KLUE and AnchiBERT for Korean and Hanja encoders.
Evaluation Metric Following previous work in RE (Yao et al., 2019), precision, recall, and microF1 scores are used for evaluating models.
Hyper-parameters Hyper-parameters are set similarly to the BERT-base model in Devlin et al. (2019). The size of the embedding and hidden vector dimensions are set to 768, and the dimension of the position-wise feed-forward layers to 3,072. All encoders consist of 12 layers and 12 attention heads for each multi-head attention layer. Also, the cross-attention block consists of 8 multi-head attention, and α is set as 0.5 when we get the final logits (Lout). However, when SL is 2, 4, and 8, α is set to 0.6. The batch size for all experiments is set to 8. The learning rate is set to 5e-5 using the Adam optimizer (Kingma and Ba, 2015). All models are trained for 200 epochs and computed on a single NVIDIA TESLA V100 GPU. Computational details are in Appendix B.1.

heading: 5.2 Results
text: As shown in Table 5, our model outperforms other monolingual baselines and consistently demonstrates the best performance even as SL grows. Even though KLUE as a monolingual model per-
forms worse than mBERT when SL is 1, our model, which combines KLUE and AnchiBERT, outperforms mBERT. This indicates that exploiting bilingual contexts improves performance. We believe that the cross-attention module and the joint architecture not only incorporate the knowledge from the Korean model, but also create synergy between the Korean and Hanja language models by compensating for each other’s deficiencies. We test this hypothesis with analysis in Section 6. Consequently, the experimental results imply that utilizing a bilingual model would be efficient in analyzing other historical records if the record is written in an early language and translated into a modern one.
As our dataset also supports using only one language, we also make note of the monolingual performance. In the Korean dataset, KLUE outperforms mBERT and KoBERT when SL is 0 and 2, while mBERT performs better than KLUE when SL is 1. We also find that KoBERT shows worse performance than mBERT, even though KoBERT was trained specifically on the Korean corpus. This demonstrates that our historical domain is dissimilar from the modern Korean one. In Hanja, AnchiBERT performs best regardless of input text length. Additional experimental results are reported in Appendix Table 6.

heading: 6 Analysis
text: In this section, we introduce a real-world usage scenario and analyze our model on HistRED, describing how our historical dataset can be utilized in detail.
6.1 Usage Scenario of HistRED
Let us assume that a domain expert aims to collect information about the kings of Chung. In our
dataset, he or she can extract the facts via the entity of “Hwang Jae (황제)” in Korean, which is a particular word to indicate the emperors of Chung, and chronologically order the events around the title. Note that this is possible because our dataset contains (i) the text in both Korean and Hanja and (ii) the year when the text was written. In total, 34 relational facts are derived from eight distinct years between 1712 and 1849, including that (a) the king in 1713 had the seventh child via the “person:child” class, and (b) the king in 1848 presented the various products with specific names, including “五絲緞” and “小荷包,” to Joseon via the “product:given_by” class. Since most of the historical records only mentioned a crown prince of Chung, describing the seventh child of the king of Chung is a rare event, which can be a motive for other creative writings. In addition, the exact name of the products the king gives reveals that those products were produced in Chung in 1848 and would be a cue to guess the lifestyle of Chung.
The expert can derive the facts from our dataset only by reading the 34 relational facts. However, if he or she has to extract them from the raw corpus, they must read at least 20 raw documents containing 1,525 sentences in Korean and 4,995 in Hanja. This scenario illustrates how HistRED can accelerate the analysis process in the historical domain.

heading: 6.2 Advantage of the Bilingual RE Model
text: To analyze the stability of our joint model, we compare three models on random samples from the test set. We use KLUE and AnchiBERT models independently for a monolingual setting, whereas we combine them for our joint model. The SL is set to 4. As shown in Figure 3, we sample two examples: case A and B, each of which displays the
most representative sentences that contain the relations for the sake of readability. In both examples, our model successfully predicts accurate relation classes. In the case of A, the ground truth (GT) label is “per:worn_by” for first and second relation triplets. Despite the successful prediction of our model with relatively high confidence scores, the Korean model matches only one of the two, while the Hanja model fails to predict both. In the case of B, the GT label is “nearby” for the third and fourth ones. Since the third and fourth relations exist across sentences, predicting them is crucial for a document-level RE task. Our model successfully predicts both relation types even with a low confidence score, while the other monolingual models fail. This case study confirms our hypothesis on our joint model; the jointly trained model can improve the performance by compensating for each monolingual model’s weaknesses, and our model successfully harmonizes the separate PLMs.

heading: 7 Related Work

heading: 7.1 Relation Extraction
text: RE datasets (Yao et al., 2019; Alt et al., 2020; Stoica et al., 2021; Park et al., 2021; Luo et al., 2022) have been extensively studied to predict relation types when given the named entities in text. RE dataset begins at the sentence level, where the input sequence is a single sentence. This includes human-annotated datasets (Doddington et al., 2004; Walker et al., 2006; Hendrickx et al., 2010) and utilization of distant supervision (Riedel et al., 2010) or external knowledge (Cai et al., 2016; Han et al., 2018). Especially, TACRED (Alt et al.,
2020; Stoica et al., 2021) is one of the most representative datasets for the sentence-level RE task. However, inter-sentence relations in multiple sentences are difficult for models trained on a sentencelevel dataset, where the model is trained to extract intra-sentence relations. To resolve such issues, document-level RE datasets (Li et al., 2016; Yao et al., 2019; Wu et al., 2019; Zaporojets et al., 2021; Luo et al., 2022) have been proposed. Especially, DocRED (Yao et al., 2019) contains large-scale, distantly supervised data, and human-annotated data. KLUE-RE (Park et al., 2021) is an RE dataset constructed in the Korean language. However, KLUE-RE is a sentence-level RE dataset, making it challenging to apply document-level extraction to the historical Korean text. To the best of our knowledge, our dataset is the first document-level RE dataset in both Korean and Hanja.

heading: 7.2 Study on Historical Records
text: Several studies have been conducted on the application of deep learning models in historical corpora, particularly in Ancient Greece and Ancient Korea. The restoration and attribution of ancient Greece (Assael et al., 2019, 2022) have been studied in close collaboration with experts of epigraphy, also known as the study of inscriptions. In Korea, thanks to the enormous amount of historical records from the Joseon dynasty, a variety of research projects have been conducted focusing on AJD and DRS (Yang et al., 2005; Bak and Oh, 2015; Hayakawa et al., 2017; Ki et al., 2018; Bak and Oh, 2018; Yoo et al., 2019; Kang et al., 2021; Yoo et al., 2022). In addition, using the Korean
text of AJD, researchers have discovered historical events such as magnetic storm activities (Hayakawa et al., 2017), conversation patterns of the kings of Joseon (Bak and Oh, 2018), and social relations (Ki et al., 2018). Kang et al. (2021) also suggests a translation model that restores omitted characters when both languages are used. Yoo et al. (2022) introduce BERT-based pretrained models for AJD and DRS. As interests in historical records grow, numerous research proposals have emerged. However, most studies only utilize the translated text to analyze its knowledge. In this paper, we aim to go beyond the studies that rely solely on the text.

heading: 8 Conclusion
text: In this paper, we present HistRED, a documentlevel relation extraction dataset of our historical corpus. Our study specializes in extracting the knowledge in Yeonhaengnok by working closely with domain experts. The novelty of HistRED can be summarized by two characteristics: it contains a bilingual corpus, especially on historical records, and SL is used to alter the length of input sequences. We also propose a bilingual RE model that can fully exploit the bilingual text of HistRED and demonstrate that our model is an appropriate approach for HistRED. We anticipate not only will our dataset contribute to the application of ML to historical corpora but also to research in relation extraction.

heading: Limitations
text: We acknowledge that our dataset is not huge compared to other sentence-level relation extraction datasets. However, HistRED is the first bilingual RE dataset at the document level on the historical corpus. In addition, we constructed 5,816 data instances, and our bilingual model trained on HistRED achieved an F1 score of 63.48 percent when SL is 2. This reveals that our dataset is sufficient for finetuning the pretrained language models. Also, because Yeonhaengnok is a collection of travel records, the domain is not as expansive as other Joseon dynasty records. Additional research on massive corpora covering a broader domain is required in future studies.

heading: Ethical Consideration
text: We conducted two separate meetings before the first and second steps of data construction. At first, we introduced the reason we built this dataset and the goal of our study and clarified what the relation
extraction task is and how the dataset will be used. All annotators agreed that their annotated dataset would be used to build an RE dataset and train neural networks. We explained each type of the named entity and the relation with multiple examples and shared user guidance. In the second meeting, we guided the annotators in evaluating and modifying the interim findings in an appropriate manner.
We adjusted the workload of each annotator to be similar by assigning different text lengths during the first and second steps. We compensated each annotator an average of $1,700, which is greater than the minimum wage in Korea. Among 15 annotators, 14 were Korean, one was Chinese, 11 were female, and four were male. 30% of annotators are in a doctorate and 65% are in a master’s degree. Regarding copyrights, since our corpus is a historical record, all copyrights belong to ITKC. ITKC officially admit the usage of their corpus under CC BY-NC-ND 4.0 license.

heading: Acknowledgement
text: This research was supported by the KAIST AI Institute (“Kim Jae-Chul AI Development Fund” AI Dataset Challenge Project) (Project No. N11210253), the National Supercomputing Center with supercomputing resources including technical support (KSC-2022-CRE-0312), and the Challengeable Future Defense Technology Research and Development Program through the Agency For Defense Development (ADD) funded by the Defense Acquisition Program Administration (DAPA) in 2022 (No. N04220080). We also thank Junchul Lim, Wonseok Yang, Hobin Song of Korea University, and the Institute for the Translation of Korean Classics (ITKC) for their discussions and support.

heading: A Dataset Construction
text: The procedure consists of the following five steps: 1) collecting corpus from the open-source data of ITKC; 2) defining the schema of the named entities and relations; 3) identifying the entities in given documents; 4) annotating corresponding relations; and 5) modifying the interim results. This section illustrates the overall procedure.
Note that the construction process is divided into two phases because the raw text of Yeonhaengnok is significantly long, where the average length of Korean text is 1,106 characters, and the historyspecialized annotators are rare. Before beginning the first phase, the annotators received instructions on the purpose of this study, the types of entities and relations, and how to operate the user interface (UI) for data tagging. After instructions, annotators identified the named entities and the relations between them. In the second phase, the annotators cross-checked the intermediate results and modified incorrect annotations. During both phases, we provided the annotators with user guidance and maintained real-time communication.

heading: A.1 Corpus Collection
text: As mentioned in 2.2, we selected 39 books from Yeonhaengnok and divided them into 2,019 texts, each containing a single day’s content. We did not divide the text into shorter texts before providing it to the annotators because a relation may exist across multiple sentences or have its evidence sentence distant from where the relation appears. We provided the entire text to the annotators to reduce the possibility of losing relational data. Due to the highly variable length of the text, an additional process step was required to extract relational information in a manageable length. To select the sentences containing all the information that can indicate the relational fact, we guided the annotators to detect the evidence sentence(s) when they annotated the relation types.

heading: A.2 Defining Schema

heading: A.2.1 Types of Named Entities
text: As shown in Table 7, we defined 10 entity types. Here, we added the date and time as entity type; thus, we can estimate the exact time because most of the corpus includes the time when the text was written. For example, if a text contains tomorrow’s plan by mentioning “tomorrow” and the written
date is June 6, we can recognize the date of tomorrow as June 7.
In historical studies, it is essential to understand the lifestyle of ancient times. Lifestyle includes clothing, food, and utilized products. For instance, humans began consuming grains such as wheat and rice after the agricultural revolution. Since lifestyle has changed according to time and location, detecting food, clothes, and products on our corpus becomes a non-trivial task.
We also excluded two text types in the preprocessing: poems and quotations. When writing the Yeonhaengnok, the writers commonly composed poems or quoted related or ancient books, including the Analects of Confucius and Mencius. We decided to detect the books’ name because it helps us imply the political status of the writer. However, the poems usually describe the sentiments or thoughts of the writer, and the quotations are written in a more ancient time than Joseon. Since we concentrated on finding objective relational facts about the Joseon dynasty, we determined to exclude the poems and quotations. A special “exclude” entity type was provided to the annotators, and the annotators tagged such subtexts if the text was a poem or a quotation.

heading: A.2.2 Types of Relations
text: Since our corpus is a collection of travel reports, the authors wrote the people they had met and the places they had visited. As shown in Table 8, we defined 20 relation classes, including 14 personal and 4 location relations. In the Joseon dynasty, it was a convention to refer to one another by their alternative name or title; thus, identifying the alternative name of a specified person is essential for tracking the individual’s life. Also, since the name of a particular location can vary depending on time and place, we added “alternate name” as a relation class to account for these instances. Additionally, in Yeonhaengnok, the number indicates the distance traveled from one location to another. We hypothesized that the locations are close to each other if the text contains the distance between the locations where the author moved because there was no mechanical mobility and they usually walked the cities. In addition, they described the characteristics of a location, such as its regional product or cuisine and its functional role. Therefore, “loc:famous_for” and “loc:function_as” were added to the set of relation types.

heading: A.3 Entity Detection
text: The annotators annotated entities using a predefined set of entity types. We provided the original Hanja and the translated Korean texts, as shown in Fig. 4. As most annotators’ native language is Korean, we recommended detecting the entities in the Korean text first and the parallel entities in the Hanja text after. After detecting entities in both texts, the annotators drew a line connecting the same entity between the two languages (as in apple and pomme in English and French texts). The annotators also drew a line connecting entities that express a certain relation. To avoid confusion, the two lines are colored in blue and orange, respectively, as shown in Figure 4.

heading: A.4 Relation Annotation
text: After identifying the relations in the previous step, the annotators added relations by using the “add relation” button and selected a relation class for the relation triplet. They also tagged the indices of evidence sentences on the Korean and Hanja texts.

heading: A.5 Cross-Checking and Modification
text: After the first phase, we analyzed the intermediate result and updated the user manual, focusing on instructions for editing initial annotations. Before the cross-checking stage, we conducted a second tutorial for the annotators using the updated manual. We assigned annotators to texts such that they had not seen them during the first phase. If they found an error(s) during cross-checking, they revised the
annotations by adding or removing the entity(s) or relation(s).

heading: B Experiments

heading: B.1 Computational Details
text: Our experiments include monolingual and bilingual settings. For each model, we describe the number of total parameters and computational budget (hours) for training on 200 epochs on our dataset when SL is 0. For the Korean model, mBERT consists of 178M parameters and consumes about 4.2 hours, KoBERT is 93M and 3.3 hours, and KLUE is 111M and 4.0 hours, respectively. For the Hanja model, mBERT consists of 178M parameters and requires 4.6 hours, and AnchiBERT is 95M and 3.3 hours. Our joint model consists of 206M parameters and consumes 6.6 hours because our model adopts two separate PLMs.

heading: B.2 Performance Comparison on Large SL
text: As shown in Table 6, our joint model outperforms other baseline models when SL is 2, 4, and 8, where the average length of documents is 153, 250, and 427 tokens on the Korean text. Our model scores better when α is 0.6 rather than 0.5 when SL is 2, 4, and 8. This can be explained by the fact that ours is affected by the low performance of the Hanja encoder, i.e., AnchiBERT. The Hanja encoder significantly drops its scores as SL increases.

heading: C Dataset Examples
text: We include additional full data samples: Table 9, Table 10, and Table 11.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation (9)
3 A2. Did you discuss any potential risks of your work? Limitation section (9)
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract, 1. Introduction
3 A4. Have you used AI writing assistants when working on this paper? language check: tools like Grammarly, QuillBot, spell checkers, dictionaries, and synonym tools
B 3 Did you use or create scientific artifacts? 5; Huggingface and Pytorch tool.
3 B1. Did you cite the creators of artifacts you used? 5
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 5
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 5, Limitation (9)
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 1, 2, 3
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? B in appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? A
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 2, Ethical Consideration (10)
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Ethical Consideration (10)
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Since our corpus is historical records in Joseon dynasty, the copyrights of all text belongs to the Institute for the Translation of Korean Classics (ITKC). Our work is approved by ITKC to utilize the corpus, therefore the ethics is hard to be applied to our dataset.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Ethical Consideration (10)

heading: Future Work
text: N11210253), the National Supercomputing Center with supercomputing resources including technical support (KSC-2022-CRE-0312), and the Challengeable Future Defense Technology Research and Development Program through the Agency For Defense Development (ADD) funded by the Defense Acquisition Program Administration (DAPA) in 2022 (No. N04220080)