SlowGuess/NAACL_2021_OCR huggingface dataset:
Kyeongpil Kang
Scatter Lab
kyeongpil@scatterlab.co.kr
Kyohoon Jin
Chung-Ang University
fhzh123@cau.ac.kr
Soyoung Yang
sy_yang@kaist.ac.kr
Soojin Jang
sujin0110@cau.ac.kr
Jaegul Choo
jchoo@kaist.ac.kr
Youngbin Kim
ybkim85@cau.ac.kr
Understanding voluminous historical records provides clues on the past in various aspects, such as social and political issues and even natural science facts. However, it is generally difficult to fully utilize the historical records, since most of the documents are not written in a modern language and part of the contents are damaged over time. As a result, restoring the damaged or unrecognizable parts as well as translating the records into modern languages are crucial tasks. In response, we present a multi-task learning approach to restore and translate historical documents based on a self-attention mechanism, specifically utilizing two Korean historical records, ones of the most voluminous historical records in the world. Experimental results show that our approach significantly improves the accuracy of the translation task than baselines without multi-task learning. In addition, we present an in-depth exploratory analysis on our translated results via topic modeling, uncovering several significant historical events.
Historical records are invaluable sources of information on the lifestyle and scientific records of our ancestors. Humankind has learned how to handle social and political problems by learning from the past. The historical records also serve as the evidence of intellectual accomplishment of humanity over time. Given such importance, there has been a great deal of nationwide efforts to preserve these historical records. For instance, UNESCO protects world heritage sites, and experts from all around the world have been converting and restoring historical
records in a digital form for long-term preservation. A representative example is the Google Books Library Project<sup>1</sup>. However, despite the importance of the historical records, it has been challenging to properly utilize the records for the following reasons. First, the nontrivial amounts of the documents are partially damaged and unrecognizable due to unfortunate historical events or environments, such as wars and disasters, as well as the weak durability of paper documents. These factors result in difficulties to translate and understand the records. Second, as most of the records are written in ancient and outdated languages, non-experts are difficult to read and understand them. Thus, for their in-depth analysis, it is crucial to recover the damaged parts and properly translate them into modern languages.
To address these issues existing in historical records, we formulate them as the task of language modeling, especially for the recovery and neural machine translation, by leveraging the advanced neural networks. Moreover, we apply topic modeling to the translated historical records to efficiently discover the important historical events over the last hundreds of years. In particular, we utilize two representative Korean historical records: the Annals of the Joseon Dynasty and the Diaries of the Royal Secretariat (hereafter we refer to them as AJD and DRS respectively). These records, which contain 50 million and 243 million characters respectively, are recognized as the largest historical records in the world. Considering their high value, UNESCO recognized them as the Memory of the
<sup>1</sup>https://support.google.com/websearch/answer/9690276
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4031-4042 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Overview of the proposed approach of recovering, translating, and mining historical documents.
World. $^{2,3}$  These two historical corpora contain the contents of five hundred years from the fourteenth century to the early twentieth century. In detail, AJD consists of administrative affairs with national events, and DRS contains events that occurred around the kings of the Joseon Dynasty. These corpora are valuable as they contain diverse information including international relations and natural disasters. In addition, the contents of the records are objective since the writing rules are strict that political intervention, even from the kings, is not allowed by their independent institution.
Although DRS contains a much larger amount of information than AJD, only  $10 - 20\%$  of DRS has been translated into the modern Korean language by a few dozens of experts for the last twenty years. The complete translation of DRS is currently expected to additionally take more than 30-40 years if only human experts continue to translate them. Applying the neural machine translation models into the historical records contains several issues. First, the pre-trained models for Chinese are not suitable to DRS and AJD, mainly because of the differences between Hanja and the Chinese language. In the past, Korean historiographers borrowed the Chinese character to write the sentences spoken by Koreans. As a result, diverse characters had been moderated or created, and considerable grammatical differences exist between the Chinese language and Hanja. Furthermore, several parts of those records are damaged and require restoration as shown in Fig. 2. Therefore, these damaged parts should be restored in order to translate them correctly. In order to address these issues, we propose
a model suitable for the historical documents using the self-attention mechanism.
Overall, we propose a novel multi-task approach to restore the damaged parts and translate the records into a modern language. Afterward, we extract the meaningful historical topics from the world's largest historical records as shown in Fig. 1. This study makes the following contributions:
- We design a model based on the self-attention mechanism with multi-task learning to restore and translate the historical records. Results demonstrate that our methods are effective in restoring the damaged characters and translating the records into a modern language.
- We translate all the untranslated sentences in DRS. We believe that this dataset would be invaluable for researchers in various fields.<sup>4</sup>
- We present a case study that extracts meaningful historical events by applying topic modeling, highlighting the importance of analysis of historical documents.
This work broadly incorporates three different tasks: document restoration, machine translation, and document analysis. Therefore, this section describes studies related to the restoration of damaged documents, neural machine translation, and the analysis of historical records.
Recently, neural machine translation (NMT) has achieved outstanding achievements. Based on the encoder-decoder architecture, the attention mechanism (Bahdanau et al., 2015) significantly improves the performance of NMT, by calculating the target context vector in the current time step via dynamically combining the encoding vectors of source
<sup>4</sup>The codes, trained model, and datasets are accessible via https://github.com/Kyeongpil/deep-joseon-record-analysis.
$^{2}$ http://www.unesco.org/new/en/communication-and-information/memory-of-the-world/register/full-list-of-register-ed-heritage/registered-heritage-page-8/the-annals-of-the-c-hoson-dynasty/
<sup>3</sup>http://www.unesco.org/new/en/communication-and-in-formation/memory-of-the-world/register/full-list-of-register-ed-heritage/registered-heritage-page-8/seungjeongwon-ilgithi-thi-diaries-of-the-royal-secretariat/
(a)
(b)
(c) (d)
Figure 2: Examples of damaged documents. Those characters that should be put in rectangles are damaged or unrecognizable.
(a) 川口□李時盃，行己悖戾
(b) 尹知敬,  $\square \square$  李敏求·趙誠立
(c) □□悶迫不得已先赴母所
(d) 抄同趙□陪臣來時
words. The self-attention-based networks (Vaswani et al., 2017) consider the correlations among all word pairs in the source and target sentences. Based on the success of self-attention networks, Transformer architecture for language modeling has been proposed, showing the forefront performances (Devlin et al., 2019; Radford et al., 2019). Especially, the pre-training approaches further improve the performances, since they train the model robustly with several tasks using a large document corpus. In addition, lightweight models, such as ALBERT (Lan et al., 2019), are proposed to reduce the model size while preserving the model performance. However, as most of the recent approaches focus on pre-training with documents written in a modern language, the model for historical datasets does not exist. Therefore, we adopt a lightweight model in the same manner as ALBERT to efficiently reconstruct and translate millions of documents.
Regarding the translation task for the historical documents, several studies attempt to translate the ancient Chinese documents into modern Chinese language (Zhang et al., 2019b; Liu et al., 2019a). However, as they mainly attempt to translate archaic characters into the modern language using paired corpus, they do not fully utilize the unpaired corpus. Therefore, we improve the performance of machine translation for historical corpora with multi-task learning with the translation and restoration tasks, which fully utilize the paired and unpaired corpora.
2.2 Restoration of Historical Documents
Unfortunately, lots of characters in the historical records are damaged or misspelled. As shown in Fig. 2, the damaged parts are prevalent in DRS, which significantly degrade the quality of subse
quent translation tasks. To address this problem, several studies focus on normalizing the misspelled words (Tang et al., 2018; Domingo and Nolla, 2018), and others further apply language modeling to restore the parts of the documents via deep neural networks (DNNs) (Caner and Haritaoglu, 2010; Assael et al., 2019).
Recently, the Cloze-style approach of machine reading comprehension (masked language modeling; MLM) predicts the original tokens for those positions where the words in the original sentence are randomly chosen and masked or replaced (Hermann et al., 2015). Several studies significantly improved the model performance by pre-training the model via the Cloze-style approach. By utilizing the MLM approach with the self-attention mechanism and the large-scale training dataset, numerous models improve the performances of various downstream tasks including NMT task (Baevski et al., 2019; Devlin et al., 2019; Zhang et al., 2019a; Conneau and Lample, 2019; Liu et al., 2019c; Clark et al., 2019). However, to our knowledge, few studies apply such an MLM approach to restore the damaged parts.
Motivated by these studies, we design our model using masked language modeling based on the self-attention architecture to recover the damaged documents considering their contexts.
2.3 Analysis on Historical Records
Various studies apply the machine learning approaches to analyze the historical records (Zhao et al., 2014; Kumar et al., 2014; Mimno, 2012; Kim et al., 2015; Bak and Oh, 2015, 2018). In addition, researchers adopt neural networks such as convolutional neural networks and autoencoders, for page segmentation and optical character recognition to convert the historical records in a digital form (Chen et al., 2017; Clanuwat et al., 2019). Given such digital-form records, analysts attempt to utilize the topic modeling to discover the historically meaningful events (Yang et al., 2011).
Especially, using the translated AJD, researchers discover historical events such as magnetic storm activities (Yoo et al., 2015; Hayakawa et al., 2017), meteors (Lee et al., 2009), and solar activities (Jeon et al., 2018). In political science, researchers analyze the decision patterns of a royal family in the Joseon Dynasty (Bak and Oh, 2015, 2018). Besides, the dietary patterns and dynamic social relations among key figures during the Joseon Dynasty have
Figure 3: Overview of the proposed model for the restoration and translation tasks.
been investigated (Ki et al., 2018). However, existing studies mainly rely on the documents translated by human experts. Therefore, we first translate the documents in AJD and DRS. Afterward, we apply topic modeling approaches to mine meaningful historical events over large-scale data.
This section describes a multi-task learning approach based on the Transformer networks to effectively restore and translate the historical records. The overview of our model is shown in Fig. 3.
AJD and DRS datasets consist of Hanja sentences  $\mathcal{H} = \{h_1,\dots ,h_N\}$  and Korean sentences  $\mathcal{K} = \{k_{1},\ldots ,k_{N}\}$ , where each Korean sentence is translated from its corresponding Hanja sentence. Here, the Hanja represents the Chinese characters borrowed to write the Korean language in the past. Especially, DRS contains additional Hanja sentences  $\widetilde{\mathcal{H}} = \{h_{N + 1},\dots ,h_M\}$  that are not translated yet. Hence, we have in total  $M$  Hanja sentences in the Hanja corpus such that  $\hat{\mathcal{H}} = \mathcal{H}\cup \widetilde{\mathcal{H}}$  and  $N$  Korean sentences in the Korean corpus  $\mathcal{K}$ .
Considering the properties of AJD and DRS, we design a multi-task learning approach with document restoration and machine translation, based on the Transformer networks. As shown in Fig. 3, our model consists of embedding and output layers for Hanja and Korean, and three Transformer modules: the shared encoder, the restoration encoder, and the translation decoder. The restoration encoder is an encoder for the restoration task. The translation decoder is used for translating Hanja sentences into modern Korean sentences, and the shared encoder is used for both the restoration and translation tasks. By sharing the encoder module
for both tasks, the shared encoder is trained with a large-scale corpus, i.e., the Hanja-Korean paired dataset and the additional unpaired Hanja dataset. The parameter sharing technique assists the model to learn rich information from the Hanja corpus. We apply the cross-layer parameter-sharing technique in the same manner as used in ALBERT (Lan et al., 2019), which shares the attention parameters for each Transformer encoder and decoder modules to reduce the model size and the inference time.
3.1 Restoration of Damaged Documents
The restoration task for damaged documents is similar to the MLM approach, which masks randomly chosen tokens in the input sentence and then predicts their original tokens in the corresponding position. We apply the MLM technique to restore the damaged documents, especially in the case of the Hanja sentences  $\hat{\mathcal{H}}$ .
For word indices  $(w_{1}^{h_{i}},\dots ,w_{L_{i}}^{h_{i}})$  in the Hanja sentence  $h_i$  where  $L_{i}$  is the length of the  $i$ -th sequence, several words are randomly selected and replaced by a [MASK] token. We extract word embedding vectors  $(e_1^{h_i},\ldots ,e_{L_i}^{h_i})\in \mathbb{R}^{d_{emb}}$  from the Hanja embedding layer combined with positional embedding vectors, where  $d_{emb}$  represents the dimension size of the embedding space. Here, we apply the factorized embedding parameterization technique to reduce model parameters (Lan et al., 2019). These embedding vectors are projected onto the  $d_{model}$ -dimensional embedding space through a linear layer. Subsequently, the embedding vectors are transformed into the Hanja context vectors  $(\hat{s}_1^{h_i},\dots ,\hat{s}_{L_i}^{h_i})$  via the shared encoder and the restoration encoder as
$$
s _ {1} ^ {h _ {i}}, \dots , s _ {L _ {i}} ^ {h _ {i}} = f _ {S} \left(e _ {1} ^ {h _ {i}}, \dots , e _ {L _ {i}} ^ {h _ {i}}\right), \tag {1}
$$
$$
\hat {s} _ {1} ^ {h _ {i}}, \dots , \hat {s} _ {L _ {i}} ^ {h _ {i}} = f _ {R} \left(s _ {1} ^ {h _ {i}}, \dots , s _ {L _ {i}} ^ {h _ {i}}\right), \tag {2}
$$
where  $f_{S}$  and  $f_{R}$  functions represent the shared encoder and the restoration encoder, respectively. The Hanja context vectors are non-linearly transformed into the output vector  $z_{k}^{h_{i}} \in \mathbb{R}^{d_{emb}}$  via the output layer. We also apply the factorized embedding parameterization technique to the output layers for parameter reduction. We calculate the probability  $P(\hat{w}_{k,m}^{h_i}|w_1^{h_i},\dots ,w_{L_i}^{h_i})$  for the index  $m$  of the original token  $\hat{w}_k^{h_i}$ , using the softmax function as
$$
P \left(\hat {w} _ {k, m} ^ {h _ {i}} \mid w _ {1} ^ {h _ {i}}, \dots , w _ {L _ {i}} ^ {h _ {i}}\right) = \frac {\exp \left(\mathbf {W} _ {m} ^ {h} \top z _ {k} ^ {h _ {i}}\right)}{\sum_ {j} ^ {| V _ {h} |} \exp \left(\mathbf {W} _ {j} ^ {h} \top z _ {k} ^ {h _ {i}}\right)}, \tag {3}
$$
where  $|V_h|$  is the size of the Hanja vocabulary.
3.2 Neural Machine Translation for Historical Records
In order to facilitate the training of our translation module, we exploit the Hanja-Korean paired dataset  $\{(h_i,k_i)|h_i\in \mathcal{H},k_i\in \mathcal{K}\}$ . As shown in Fig. 3, we first extract the Hanja context vectors  $(s_1^{h_i},\ldots ,s_{L_i}^{h_i})$  from the word tokens in the Hanja sentence  $h_i$ , using the shared encoder in the same manner as in Eq. 1. Utilizing the Hanja context vectors and previously predicted Korean words  $(w_{1}^{k_{i}},\dots,w_{t - 1}^{k_{i}})$ , we subsequently calculate the  $d_{model}$ -dimensional Korean context vector  $s_t^{k_i}$  for the current time step  $t$  as
$$
s _ {t} ^ {k _ {i}} = f _ {D} (s _ {1} ^ {h _ {i}}, \ldots , s _ {L _ {i}} ^ {h _ {i}}, w _ {1} ^ {k _ {i}}, \ldots , w _ {t - 1} ^ {k _ {i}}), \qquad (4)
$$
where  $f_{D}$  represents the translation decoder layers. After calculating the Korean context vector  $s_t^{k_i}$ , we non-linearly transform the context vector to the output vector  $z_t^{k_i} \in \mathbb{R}^{d_{emb}}$ , through the output layer, along with the above-mentioned factorized embedding parameterization for parameter reduction. Finally, we yield the probability that the word  $V_m$  is generated from the  $t$ -th step as
$$
P \left(w _ {t, m} ^ {k _ {i}} \mid h _ {i}, w _ {1: t - 1} ^ {k _ {i}}\right) = \frac {\exp \left(\mathbf {W} _ {m} ^ {k} {} ^ {\top} z _ {t} ^ {k _ {i}}\right)}{\sum_ {j} ^ {| V _ {k} |} \exp \left(\mathbf {W} _ {j} ^ {k} {} ^ {\top} z _ {t} ^ {k _ {i}}\right)}, \tag {5}
$$
where  $|V_k|$  is the size of the vocabulary for the Korean corpus, and  $\mathbf{W}^k \in \mathbb{R}^{|V_k| \times d_{emb}}$  is the output layer for the Korean corpus.
As previously mentioned, we employ the parameter sharing approach for the encoder module, (i.e., the shared encoder), thus enhancing the robustness of our model, especially with the Hanja dataset.
In order to train our model, we use the cross-entropy loss to maximize the probability of the original token indices for the masked tokens and the target sentence for the translation task as
$$
\mathcal {L} _ {r s t} = - \frac {1}{M} \sum_ {h _ {i} \in \hat {\mathcal {H}}} \mathbb {E} _ {k \sim \xi \left(h _ {i}\right)} \left[ \log P \left(w _ {k} ^ {h _ {i}} \mid h _ {i}\right) \right], \tag {6}
$$
$$
\mathcal {L} _ {t r s} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \frac {1}{| k _ {i} |} \sum_ {t = 1} ^ {| k _ {i} |} P \left(w _ {t} ^ {k _ {i}} \mid h _ {i}, w _ {1: t - 1} ^ {k _ {i}}\right) \right], \tag {7}
$$
where  $\xi (\cdot)$  is an operator that randomly selects the tokens from each sentence for MLM. In this study, we apply not only unigram masking but also the n-gram masking techniques (i.e., bigrams and trigrams), as previously applied (Zhang et al., 2019a). Finally, the total loss is defined as
$$
\mathcal {L} = \mathcal {L} _ {r s t} + \mathcal {L} _ {t r s}. \tag {8}
$$
Paired Hanja	Unpaired Hanja	Korean
(#(Train data)	239,226	1,377,320	239,226
(#(Test data)	20,000	20,000	20,000
1st Quartile	26	27	22
Mean	143.81	165.66	123.68
3rd Qquatile	106	113	80
Median	52	55	40
Vocab size	8,742	8,742	24,000
Table 1: Dataset summary. The third to the sixth rows indicate the statistics for the length of each document.
Our model is optimized by using the rectified Adam (Liu et al., 2019b) with the layer-wise adaptive rate scheduling technique (You et al., 2017). We also apply the gradient accumulation technique and update our model for each loss asynchronously, to increase the batch size and efficiently manage the GPU memory.
After training the model, the damaged tokens are replaced by the [MASK] token during the restoration stage, and the model obtains the top-K characters with the highest probabilities, among which users can choose and confirm a correct character in the position of the damaged parts. In addition, we translate all the Hanja records that are not yet translated for further in-depth analysis. When translating the Hanja sentence, we additionally apply beam search with length normalization. The translation task for all the untranslated records using 20 V100 GPUs had a duration of approximately five days.
This section first describes our datasets and experimental settings.
4.1 Datasets and Preprocessing
To train our model, we collect most of the documents of AJD and DRS, including those manually translated to date, provided by the National Institute of the Korean History. The records contain approximately 250K documents for AJD and 1.4M documents for DRS.
After collecting documents, we tokenize each Hanja sentence into the character-level tokens, similar to previous studies (Zhang et al., 2014; Li et al., 2018), and also tokenize each Korean sentence based on the unigram language model (Kudo, 2018) provided by Google's SentencePiece library.6 Here, we included those words appearing more than ten times in the Hanja vocabulary, the size of which
<sup>5</sup>http://www-history.go.kr/
$^{6}$ https://github.com/google/sentencepiece
is about 8.7K words. For the Korean corpus, we limit the size of the Korean vocabulary to 24K. The out-of-vocabulary words are replaced with UNK (unknown) tokens. To improve the stability and efficiency during the training stage, we filter out those Hanja sentences with less than four tokens or more than 350 tokens and those Korean sentences with less than four tokens or more than 300 tokens. Note that the portion of sentences filtered out from each dataset is less than  $10\%$ .
To evaluate the performance of our model, we randomly select 20K sentences as a test dataset for each of the paired and the unpaired sets. The sizes of the training set for the Hanja-Korean paired corpus and the unpaired Hanja corpus are 240K and 1.38M, respectively. The statistics of the dataset are summarized in Table 1.
We set hyper-parameters similarly to the BERT (Devlin et al., 2019) base model. We set the size of the embedding dimension  $d_{emb}$ , the hidden vector dimension  $d_{model}$ , and the dimension of the position-wise feed-forward layers as 256, 768, and 3,072, respectively. The shared encoder, the translation decoder, and the restoration encoder consist of 12, 12, and 6 layers, respectively. We use 12 attention heads for each multi-head attention layer. Overall, the total number of parameters is around 168.8M.
4.3 Mining Historical Records via Topic Modeling
After obtaining machine-translated outputs of the remaining records, we apply topic modeling to the full set of documents for exploratory analysis of historical events. To be specific, the full set of documents include all of the manually translated records as well as machine-translated records by our model. By using each translated record  $k_{i}$  and its written date information  $d_{i}$ , we first parse the document into morphemes and then use the only noun and adjective tokens. Afterward, we build the term-date matrix  $\mathbf{V} \in \mathbb{R}^{V \times D}$  where  $V$  is the vocabulary size and  $D$  is the number of dates in the total set of historical documents.
In this study, we utilize non-negative matrix factorization (NMF) (Lee and Seung, 2001) as a topic modeling method. We first assume that there ex
ist  $K$  topics in the corpus. The term-date matrix  $\mathbf{V}$  is decomposed into the term-topic weight matrix  $\mathbf{W} \in \mathbb{R}^{V \times K}$  and the date-topic weight matrix  $\mathbf{H} \in \mathbb{R}^{D \times K}$  as
$$
\mathbf {W}, \mathbf {H} = \underset {\mathbf {W}, \mathbf {H} \geq 0} {\arg \min } \| \mathbf {V} - \mathbf {W} \mathbf {H} ^ {\top} \| _ {F} ^ {2} + \alpha \cdot \psi (\mathbf {W}, \mathbf {H}), \tag {9}
$$