SlowGuess/ACL_2020_OCR huggingface dataset:
Pranav A 徒 and Isabelle Augenstein
$^{1}$ Independent Researcher, Hong Kong
$^{\text{師}}$ Department of Computer Science, University of Copenhagen, Denmark
cs.pranav.a{at}gmail.com，augenstein{at}di.ku.dk
Simplified Chinese to Traditional Chinese character conversion is a common preprocessing step in Chinese NLP. Despite this, current approaches have insufficient performance because they do not take into account that a simplified Chinese character can correspond to multiple traditional characters. Here, we propose a model that can disambiguate between mappings and convert between the two scripts. The model is based on subword segmentation, two language models, as well as a method for mapping between subword sequences. We further construct benchmark datasets for topic classification and script conversion. Our proposed method outperforms previous Chinese Character conversion approaches by 6 points in accuracy. These results are further confirmed in a downstream application, where 2kenize is used to convert pretraining dataset for topic classification. An error analysis reveals that our method's particular strengths are in dealing with code mixing and named entities. The code and dataset is available at https://github.com/pranav-ust/2kenize
Chinese character (or script) conversion is a common preprocessing step for Chinese NLP practitioners (Zhang, 2014; Shi et al., 2011). Traditional Chinese (TC) and Simplified Chinese (SC) are the two standardized character sets (or scripts) for written Chinese. TC is predominantly used in Taiwan, Hong Kong, and Macau, whereas SC is mainly adopted in mainland China and SC characters are simplified versions of TC characters in terms of strokes and parts. Therefore, Chinese NLP practitioners apply script converters<sup>1</sup> to translate the
SC Sentence	维护发展中国家共同利益	Comments
Segmentation	维护发展中国家共同利益	护发: haircare
Conversion	維護髮展中國家共同利益	× Conversion
Segmentation	维护发展中国家共同利益	发展: develop
Conversion	維護發展中國家共同利益	✓ Conversion
Table 1: Example sentence with two different segmentations, and resulting different conversions. The sentence translates to Safeguarding the common interests of developing countries. This is a recurring example in this paper. Also refer §F.5.
dataset into their desired language. This is especially useful for TC NLP practitioners because TC is less widely used and under-resourced as compared to SC.
Converting from TC to SC is generally straightforward because there are one-to-one correspondences between most of the characters, so conversion can be performed using mapping tables (Denisowski, 2019; Chu et al., 2012). However, conversion from SC to TC is an arduous task as some SC characters can be mapped to more than one TC character depending on the context of the sentence. A detailed analysis by Halpern and Kerman (1999) shows that SC to TC conversion is a challenging and crucial problem, as  $12\%$  of SC characters have one-to-many mappings to TC characters. Our experiments show that current script converters achieve sentence accuracy results of  $55 - 85\%$  ( $\S 3$ ).
Another issue is that varying tokenization would lead to different results as Chinese is an unsegmented language, see Table 1 for an example. Off-the-shelf script converters would translate 维护发展中国家共同利益 into 維護髮展中國家共同利益,² whereas the correct conversion is 維
2Throughout this paper, we color code ambiguous SC characters with brown, ambiguous TC characters with violet, vernacular Cantonese characters with teal. By scripts, we refer as to character sets, and we interchangeably use them in this paper.
<sup>1</sup>Most of these tools like OpenCC, Mafan, Hanziconv are generally widely used within Chinese NLP community, which can be attested popularity of their Github repos.
7257
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7257-7272 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
護發展中國家共同利益. Here, the SC character 发 (hair; issue) has two TC mappings, 髖 (hair, issue) and 發 (hair; issue), depending on the context and tokenization; which shows that this task is non-trivial.
Despite this being an important task, there is a lack of benchmarks, $^{3}$  which implies that this problem is understudied in NLP. In this study, we propose 2kenize, a subword segmentation model which jointly considers Simplified Chinese and forecasting Traditional Chinese constructions. We achieve this by constructing a joint Simplified Chinese and Traditional Chinese language model based Viterbi tokenizer. Performing mapping disambiguation based on this tokenization method improves sentence accuracy by 6 points as compared to off-the-shelf converters and supervised models. Our qualitative error analysis reveals that our method's particular strengths are in dealing with code-mixing and named entities. Additionally, we address the issue of a lack of benchmark datasets by constructing datasets for script conversion and TC topic classification.
2 2kenize: Joint Segmentation and Conversion
We employ subword tokenization, as it addresses the issue of rare and unknown words (Mikolov et al., 2012) and has been shown advantageous for the language modelling of morphologically-rich languages (Czapla et al., 2018; Mielke and Eisner, 2019). This achieves improvements in accuracy for neural machine translation (NMT) tasks and has now become a prevailing practice (Denkowski and Neubig, 2017). The most widely-utilized method is Byte Pair Encoding (BPE, Sennrich et al. (2016)), a compression algorithm that combines frequent sequences of characters, which results in rare strings being segmented into subwords. Unigram (Kudo, 2018) and BPE-Drop (Provilkov et al., 2019) use subword ambiguity as noise, as well as stochastically-corrupted BPE segmentation to make it less deterministic. For NMT tasks generally, subword segmentation is seen as a monolingual task and applied independently on source and target corpora. We hypothesize that translation tasks, and specifically conversion tasks, as investigated here, would have a bet-
ter performance if segmentation were performed jointly. Hence, in this section, we describe our proposed method 2kenize, which jointly segments by taking the source and its approximate target sentences into account. This motivates the main idea of this paper: We propose 2kenize which jointly considers the source sentence and its corresponding target conversions by doing lookaheads with mappings.
2.1 Outline of the proposed approach
Given the possible SC character sequence  $\mathbf{s} = s_1s_2\ldots s_n$  and TC character sequence  $\mathbf{t} = t_1t_2\ldots t_n$ , we want to find the most likely  $\mathbf{t}$ , which is given by the Bayes decision rule as follows:
$$
\mathbf {t} = \underset {\mathbf {t} ^ {\prime} \in T ^ {*}} {\arg \max } p (\mathbf {s}, \mathbf {t} ^ {\prime}) \tag {1}
$$
where  $T^{*}$  denotes the set of all strings over symbols  $(t_{i})$  in  $T$  (Kleene star). We divide this problem into two parts: finding the mapping sequence (2) and finding the TC sequence from mappings (7).
We define a mapping, which is given by  $m_{i} = (\mathfrak{s}_{i},\mathfrak{t}_{i}) = (s_{j:k},\mathfrak{t}_{j:k})$ . Here,  $\mathfrak{t}_{j:k} = \{t_{j:k}^{1}\ldots t_{j:k}^{n}\}$  is a set of TC characters that correspond to the SC character in the mapping. Thus, a mapping sequence can be defined as a concatenation of mappings, which is  $\mathbf{m} = m_1m_2\dots m_l$ . Let  $\mathcal{M}$  be the superset of all possible mapping sequences and  $\mathcal{M}(\mathbf{s})$  be the all mapping sequences resulting from  $\mathbf{s}$ . Then, the best possible mapping sequence is given by
$$
\mathbf {m} = \underset {\mathbf {m} ^ {\prime} \in \mathcal {M} (\mathbf {s})} {\arg \max } p \left(\mathbf {m} ^ {\prime}\right) \tag {2}
$$
Moreover,  $p(\mathbf{m})$  can be expanded as such:
$$
\begin{array}{l} p (\mathbf {m}) = p \left(m _ {1} m _ {2} \dots m _ {l}\right) (3) \\ = p \left( \begin{array}{l l l l} \mathfrak {s} _ {1} & \mathfrak {s} _ {2} & \dots & \mathfrak {s} _ {l} \\ \mathfrak {t} _ {1} & \mathfrak {t} _ {2} & \dots & \mathfrak {t} _ {l} \end{array} \right) (4) \\ \approx p \left(\mathfrak {s} _ {1} \mathfrak {s} _ {2} \dots \mathfrak {s} _ {l}\right) + p \left(\mathrm {t} _ {1} \mathrm {t} _ {2} \dots \mathrm {t} _ {l}\right) (5) \\ = p _ {L M} \left(\mathfrak {s} _ {1: l}\right) + \sum_ {t \in \prod_ {i} t _ {i}} p _ {L M} \left(t _ {1: l}\right) (6) \\ \end{array}
$$
After expanding the mapping sequences (4), we take an approximation by estimating this as the sum of likelihoods of two sequences formed due to co-segmentations (5). The set of possible TC sequences is given by the Cartesian product of  $t_i$ . These likelihoods can then be estimated using language model (LM) probabilities as shown in (6).
$$
\mathbf {t} = \underset {\mathbf {t} ^ {\prime} \in \mathbf {m} _ {\mathbf {t}}} {\arg \max } p \left(\mathbf {t} ^ {\prime}\right) \tag {7}
$$
3The ChineseNLP website states that script conversion benchmarks and experiments currently do not exist: https://chinesenlp.xyz/#/docs/simplified_traditional_Chinese_conversion
7258
Figure 1: Language model architecture with subword and subsequence sampling. (Alt text: §F.1).
Once the mapping sequence  $\mathbf{m}$  has been found, all possible TC sequences are found from the set  $\mathbf{m}_{\mathbf{t}}$  which is the Cartesian product for all  $t_i$  in  $\mathbf{m}$ . From (7), we calculate approximate final sequence using beam search.
2.2 Model Architecture
Viterbi, a dynamic programming (DP) algorithm, considers phrases (or subsequences) and performs segmentation in a 'bottom-up' fashion (Nagata, 1994; Sproat et al., 1996). RNN-based language models are theoretically considered to be '  $\infty$  ’-gram (Khandelwal et al., 2018), which constitutes a challenge. Consider this sentence,维护发展中国家共同利益. A potential challenge could be to adequately estimate the probability of共同利益. As this sequence occurs infrequently in the beginning of sentences in the corpus, an RNN would under-estimate the probability of this subsequence. Moreover, an RNN would likely lose some useful context and perform worse without it (Kim et al., 2019). So for Viterbi to perform well with an RNN, we train the language model on subsequences. We approach this by training our model in such a way that it samples subsequences ran
Figure 2: From the given SC sentence, we create possible TC sequences using mappings. We input these to Viterbi, which recursively calls LSTM. Using Eq. (6) as the scoring function, Viterbi outputs the mapping sequence. We perform beam search to find the best TC sequence from the mapping sequence. (Alt text: §F.2).
domly in each epoch. As shown in Fig 1, we randomly split the sentence and use subsequences in separate epochs.
Using Kudo (2018) regularization method, we sample from the  $n$ -best segmentations in each epoch. This is done so that the model can understand different segmentations of a subsequence using a similar motivation as above. Recent works have shown that varying subword segmentations lead to a better downstream model performance (Provilkov et al., 2019; Kudo, 2018; Hiraoka et al., 2019); therefore, we use it as a data augmentation strategy. Once we get the  $n$ -best segmentations with scores, we normalize them, and then use the normalized scores as sampling probabilities (see Fig 1). As opposed to other subword tokenizers where the vocabulary size is fixed, we do not limit the vocabulary in our model. Hence, there are numerous possibilities of segment combinations which raises a need of caching most frequent tokens. Inspired by the work related to cache-based LMs (Kawakami et al., 2017) and ghost batches (Hoffer et al., 2017), we only consider the top- $k$  tokens in the main network memory and keep track of gradients of less recently used token embeddings (commonly known as LRU, Least Recently Used policy). This could be thought of as
7259
HK Literature	HK News	TW Literature	TW News
Sources	Liu (1962)	Singpao (2017-2018)	Jiubadao (2011)	AS subset Emerson (2005)
Lau Yee (1972)	Mingpao (2017-2018)	Ko (2010)	Liberty Times (2017-2018)
Foon (1988)	CityU subset Emerson (2005)	Yao (1964)	United Daily News (2017-2018)
Average Length	194.8	214.6	188.2	223.6
IAA	0.982	0.979	0.981	0.971
Mapping Examples	干-［幹,乾,干,幹］
須-［須,髒］	苏-［蘇,噉,甦］
暗-［暗,闊］	复-［復,複,覆］
叹-［嘆,歎］	胡-［胡,衛,鬍］
迹-［蹟,跡］
Table 2: An overview of the dataset used for intrinsic evaluation. We report sources, average character lengths and sentence level inter-annotator agreements (IAA, reported in  $\kappa$ ) and some examples of ambiguous SC-TC mappings.
virtual embeddings as delayed gradient accumulation allows to accommodate larger number of tokens. This virtual size embedding architecture is related to the continuous cache implementation and stochastic tokenization architectures (Grave et al., 2016; Hiraoka et al., 2019).
2.3 Segmentation and Disambiguation
This optimal sequencing problem can be formulated as an overlapping subsequence approach, which can be solved using LM based Viterbi (Nagata, 1994; Sproat et al., 1996). Fig. 2 explains this process of joint subword modelling. Here, we take Eq. (6) as the objective function for finding the mapping sequence, however, we use subword perplexities (Cotterell et al., 2018; Mielke et al., 2019; Mielke, 2019) in our implementation. For the TC LSTM, we add the probabilities of the beams of the possible sequences.
As discussed in §2.1 and Eq. (7), beam search is needed to select the best subword sequence for TC. Once the sentences are tokenized, the mapping table is used to convert each SC token to the corresponding TC token. We extract the final TC sentence by resolving ambiguities through beam search using the TC LSTM (Fig. 2).
3 Intrinsic Evaluation
3.1 Dataset for Intrinsic Evaluation
We construct a gold standard corpus for both Chinese scripts consisting of 4 domains: HK Literature and Newswire, and Taiwanese Literature and Newswire (Table 2) with each domain containing 3000 sentences. SC-TC mapping tables are constructed from existing resources (Denisowski, 2019; Chu et al., 2012). We heuristically convert selected TC sentences to SC using OpenCC. We asked the annotators to manually correct any incorrect conversions. $^{4}$
3.2 Language Model Training
We choose the SIGHAN-2005 Bakeoff dataset to train the segmentation-based language model (Emerson, 2005). For SC, we select the PKU and MSR partitions, and for TC, we use the Academia Sinica and CityU partitions. We apply maximal matching (or heuristic dictionary-based word segmenter) to pre-process these datasets by segmenting words into subwords (Wong and Chan, 1996). Here, 'dictionary' refers to the word-list in the mapping table. We then train a 2-layer LSTM language model LSTM with tied weights, and embedding and hidden sizes of 512 (Sundermeyer et al., 2012) on this segmented dataset with subsequence sampling and stochastic tokenization as discussed in  $\S 2.2$ .
3.3 Baselines and Ablations
We implement the following baselines for the experimentation:
Off-the-shelf Converters: Hanziconv $^{6}$  and Mafan $^{7}$  are dictionary-based script character converters. Evaluating this could be useful to understand the lower accuracy bound. OpenCC $^{8}$  uses a hybrid of characters and words (specifically trie based tokenizer) for script conversion (Pranav A et al., 2019).
Language Model Disambiguation: A strong baseline to this problem would be to build a language model to disambiguate between the characters, which is quite similar to STCP (Xu et al., 2017). We use a 2-layer LSTM language model trained on Traditional Chinese corpus.
Neural Sequence Models: We heuristically convert Traditional Chinese Wikipedia to Simplified Chinese using OpenCC and use it for training the seq2seq model (Sutskever et al., 2014). We
4 A detailed data statement is given in the appendix.
${}^{5}$  If in case you are looking for 'Related Work' section.
<sup>6</sup>https://github.com/berniey/hanziconv
7https://github.com/hermanschaaf/mafan
<sup>8</sup>https://github.com/BYVoid/OpenCC
7260
Conversion System	HK Lit	HK News	TW Lit	TW News	Overall
DED	SA	DED	SA	DED	SA	DED	SA	DED	SA
Dictionary based conversion, Hanziconv	34.1	54.7	37.7	59.1	31.3	60.0	39.3	58.9	34.2	55.6
Dictionary based conversion, Mafan	14.7	71.2	17.7	72.5	14.5	73.8	13.3	72.7	14.4	72.6
Trie dictionary based conversion, OpenCC	5.5	87.3	5.1	83.4	4.1	84.7	3.8	88.5	4.3	85.3
Language Model Disambiguation, STCP	6.3	85.6	5.4	79.9	4.7	84.1	5.2	83.9	5.3	84.0
Convolutional Sequence Models	6.7	85.8	5.3	79.3	4.8	84.5	5.2	83.9	5.4	84.4
2kenize with word tokenization	11.2	84.3	12.1	81.3	11.3	82.1	10.0	81.1	11.5	82.7
2kenize with maximal matching	5.2	88.7	3.3	93.1	4.0	88.6	4.8	87.7	4.5	88.9
2kenize with Unigram subwords	3.4	91.9	3.8	90.9	4.3	88.1	3.9	87.8	3.7	89.3
2kenize with joint LSTM modelling	2.8	94.9	3.1	93.7	3.8	91.3	2.9	91.9	3.0	92.4
Table 3: Results of the intrinsic evaluation experiments which are reported as a mean across 10 different seeds. We use disambiguation error density (DED, the lower, the better) and sentence accuracy (SA, the higher the better) metrics for evaluation. **Bold:** best, **Underlined:** second-best.
construct a 20-layer neural convolutional sequence model (Gehring et al., 2017) (both in encoder and decoder) using fairseq (Ott et al., 2019).
We perform ablation tests by inserting following segmentation models.
Word tokenization: We use Jieba, which is a commonly used hidden markov model based word tokenizer for Chinese NLP.
Dictionary substrings: We apply maximal string matching, which is a dictionary based greedy tokenizer (Pranav A et al., 2019; Wong and Chan, 1996).
Unigram from Sentencepiece: Subword segmentation is performed by sampling unigram language model perplexity values (Kudo, 2018).
Joint subwords: As discussed in §2.3, we use joint SC-TC subwords.
3.4 Results for Intrinsic Evaluation
We evaluate our models using the metrics of disambiguation error density (DED) and sentence accuracy (SA). DED is the average of total edit distances per 1000 ambiguous Simplified characters, which is  $\frac{\sum\text{edit distances}}{\sum\text{ambiguous Simplified characters}} \times 1000$ . SA is the number of sentences correctly converted in percentages. Contrary to previous papers, we do not report character based accuracy values, as generally most characters have straightforward mappings — a reason why we opt for a less forgiving metric like SA where every character in a sentence has to be correctly converted.
Results are shown in Table 3, broken down by domain, and overall. Our model attains an average DED of 3.0 and a SA of  $92.4\%$  overall, whereas the best existing converter, OpenCC, only achieves a DED of 4.3 and a SA of  $85.3\%$ . We
find that seq2seq and LM based disambiguation perform almost on par with OpenCC, due to the large number of false positive errors by these models. Jieba achieves an average DED of 11.2 as it does not handle OOV words well. For maximal matching of segmented words and Unigram subwords, it achieves an overall DED of 4.5 and 3.7, respectively — showing that joint segmentation yields better results. We observe that accuracy values are slightly worse on news text, due to the relatively high number of new entities in those datasets. We find that seq2seq and LM based disambiguation gives rise to many false positives. Heuristically converting TC to SC results in certain conversion errors in the training dataset; and additionally, seq2seq approaches tend to reword the target sentence, which shows that they are unsuitable for this task.
3.5 Qualitative Error Analysis
We manually inspect incorrect conversions in the intrinsic evaluation and find four interesting recurring linguistic patterns which confused the converters. We instructed the annotators to classify the items in the dataset (overall 12000 sentences in intrinsic evaluation dataset) if the sentences contain any of these patterns. In Table 4, we provide an overview of statistical information of these patterns and the performance by the converters.
Code mixing: Vernacular Cantonese characters (zh-yue) are a subset of TC characters but do not follow the norms of the standard written Chinese (Snow, 2004). We find that some of the sentences in our dataset are code-mixed with zh-yue (e.g. speech transcription) or English (e.g. named entities). Consider the snippet, "...古惑架 BENZ 190E 撞埋支...", which is code-mixed with
<https://github.com/fxsjy/jieba>
7261
Case	Method	SA	Example
Code mixing with	肯尼迪咾多囉做,掂唔掂呀?
With so much to do in Kennedy, can you handle it?
Cantonese	OpenCC	20.5	肯尼迪咾多囉做,掂唔掂呀?
(34 cases, 0.3%)	STCP	8.8	肯尼迪咾多囉做,掂唔掂呀?
2kenize	91.1	甘迺迪咾多囉做,掂唔掂呀?
Code mixing with English	OpenCC	95.6	自从我捲住大古惑架 BENZ 190E 撞埋支电灯柱嚱度之後, After I drove Slick's Benz 190E into the telephone pole,
(1532 cases, 12.8%)	STCP	86.5	自從我捲住大古惑架 BENZ 190E 撞埋支電燈柱嚱度之後, 自從我捲住大古惑架 BENZ 190E 撞埋支電燈柱嚱度之后,
2kenize	98.7	自從我捲住大古惑架 BENZ 190E 撞埋支電燈柱嚱度之後,
Disguised	维护发展中国家共同利益
Named	Safeguard the common interests of developing countries
Entities	OpenCC	85.7	維護髮展中國家共同利益
(378 cases, 3.15%)	STCP	82.1	維護髮展中國家共同利益
2kenize	93.2	維護發展中國家共同利益
Repeated	乔治亞来到了乔治亞洲旅游
Named	Georgia came to Georgia for travelling.
Entities	OpenCC	84.4	佐治亞來到了佐治亞洲旅遊
(428 cases, 3.57%)	STCP	17.9	佐治亞來到了喬治亞洲旅遊
2kenize	87.8	喬治亞來到了喬治亞洲旅遊
Table 4: Casewise breakdown of common errors. The first sentence is SC, second is the English translation and rest are TC outputs from the converters.
both zh-yue and English. The characters "BENZ 190E", 架 and 埋支 are not a part of standard written Chinese. We find that OOV words are 2kenized into single-character tokens which results in: "古 惑 | 架 | B|E|N|Z| 1|9|0|E| 撞 | 埋 | 支" Thus, 2kenize distributes the entropy over multiple tokens rather than a single token (generally UNK is used in such cases). This allows the language model to have more space for multiple guesses, which shows a massive advantage over word models or just UNK-ing it, a reason why subword tokenizers outperform closed-vocabulary models (Merit, 2019).
Disguised Named Entities: Take the recurring sentence: "维护发展中国家共同利益". Observe that the sentence contains a frequent word 中国 (China). However, the actual meaning and English translation do not include "China" at all. This is an interesting linguistic trait of Chinese, where words often appear in the sentence, but are not being interpreted. This could easily trip up a tokenizer, as the probability of 中国 being a token independently is high. Having 中国 as a separate token in the sentence could lead into an incorrect conversion (Table 1). We find in 2kenizer's trellis<sup>10</sup> that "维护 | 发展 | 中" has a higher probability than other possible segmentations. Substructure lookups and beam search in our setup considerably reduces the probability of getting wrong tokenization. The sen
tence is 2kenized into “维护 | 发展 | 中 | 国家 | 共同 | 利益”, which results in the correct conversion –維護發展中國家共同利益.
Repetitions: We find that in  $3.57\%$  of sentences, named entities are repeated. Interestingly, STCP, which uses a language model for disambiguation, often only converts one out of the repeated tokens correctly, which we can see in the table. As also shown, STCP prefers 佐治亞 over 喬治亞 in the first occurrence, but then prefers 喬治亞11 in the second occurrence as it gets more context. 2kenize converts both of the entities correctly, very likely due to substructure lookups.
Failure Cases: Dictionary-based converters (OpenCC, HanziConv and Mafan) only use the first conversion candidate<sup>12</sup> if multiple candidates are available. STCP often converts named entities wrongly, especially the ones which have long-range dependencies and repetitions. Although we find that 2kenize converts some of the unseen named entities perfectly, some of the errors caused were due to infrequent characters. Few cases are mainly related to variant characters<sup>13</sup> which are often used interchangeably.
11 Annotators and sources preferred 嚼治亞 over 佐治亞.
12We are not sure how did they define the "order", but we have observed that they select more frequent characters as their most highly ranked ones.
13For example,了解and 瞬解could be both used for "understand", and 裹面 and 裹面 could be used for "inside".
This is a probability lookup table in Viterbi to keep track of the segment information in a subsequence.
7262