Venue: ACL 2020

# 2kenize: Tying Subword Sequences for Chinese Script Conversion

Pranav A\({}^{\circledR}\) and Isabelle Augenstein\({}^{\circledR}\)

\({}^{\circledR}\)Independent Researcher, Hong Kong

\({}^{\circledR}\)Department of Computer Science, University of Copenhagen, Denmark cs.pranav.a{at}gmail.com, augenstein{at}di.ku.dk

###### Abstract

Simplified Chinese to Traditional Chinese character conversion is a common preprocessing step in Chinese NLP. Despite this, current approaches have insufficient performance because they do not take into account that a simplified Chinese character can correspond to multiple traditional characters. Here, we propose a model that can disambiguate between mappings and convert between the two scripts. The model is based on subword segmentation, two language models, as well as a method for mapping between subword sequences. We further construct benchmark datasets for topic classification and script conversion. Our proposed method outperforms previous Chinese Character conversion approaches by 6 points in accuracy. These results are further confirmed in a downstream application, where 2kenize is used to convert pretraining dataset for topic classification. An error analysis reveals that our method's particular strengths are in dealing with code mixing and named entities. The code and dataset is available at [https://github.com/pranav-ust/2kenize](https://github.com/pranav-ust/2kenize)

+
Footnote †: _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7257–7272

+
Footnote †: _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7257–7272

## 1 Introduction

Chinese character (or script) conversion is a common preprocessing step for Chinese NLP practitioners [30, 4]. Traditional Chinese (TC) and Simplified Chinese (SC) are the two standardized character sets (or scripts) for written Chinese. TC is predominantly used in Taiwan, Hong Kong, and Macau, whereas SC is mainly adopted in mainland China and SC characters are simplified versions of TC characters in terms of strokes and parts. Therefore, Chinese NLP practitioners apply script converters1 to translate the dataset into their desired language. This is especially useful for TC NLP practitioners because TC is less widely used and under-resourced as compared to SC.

Footnote 1: Most of these tools like OpenCC, Mafan, Hanziconv are generally widely used within Chinese NLP community, which can be attested popularity of their Github repos.

Converting from TC to SC is generally straightforward because there are one-to-one correspondences between most of the characters, so conversion can be performed using mapping tables [9, 2]. However, conversion from SC to TC is an arduous task as some SC characters can be mapped to more than one TC character depending on the context of the sentence. A detailed analysis by Halpern and Kerman [19] shows that SC to TC conversion is a challenging and crucial problem, as 12% of SC characters have one-to-many mappings to TC characters. Our experiments show that current script converters achieve sentence accuracy results of 55-85% (SS3).

Another issue is that varying tokenization would lead to different results as Chinese is an unsegmented language, see Table 1 for an example. Off-the-shelf script converts would translate \({}^{\circledR}\) into \({}^{\

* (_hair, issue_) has two TC mappings,
* (_hair, issue_) and
* (_hair, issue_), depending on the context and tokenization; which shows that this task is non-trivial.

Footnote 3: The ChineseNLP website states that script conversation benchmarks and experiments currently do not exist: [https://chinesenlp.xyz/#/docs/simplified_traditional_Chinese_conversion](https://chinesenlp.xyz/#/docs/simplified_traditional_Chinese_conversion)

Despite this being an important task, there is a lack of benchmarks,4 which implies that this problem is understudied in NLP. In this study, we propose _2kenize_, a subword segmentation model which jointly considers Simplified Chinese and forecasting Traditional Chinese constructions. We achieve this by constructing a joint Simplified Chinese and Traditional Chinese language model based Viterbi tokenizer. Performing mapping disambiguation based on this tokenization method improves sentence accuracy by 6 points as compared to off-the-shelf converters and supervised models. Our qualitative error analysis reveals that our method's particular strengths are in dealing with code-mixing and named entities. Additionally, we address the issue of a lack of benchmark datasets by constructing datasets for script conversion and TC topic classification.

Footnote 4: The ChineseNLP website states that script conversation benchmarks and experiments currently do not exist: [https://chinesenlp.xyz/#/docs/simplified_traditional_Chinese_conversion](https://chinesenlp.xyz/#/docs/simplified_traditional_Chinese_conversion)

## 2 **2kenze: Joint Segmentation and Conversion**

We employ subword tokenization, as it addresses the issue of rare and unknown words Mikolov et al. (2012) and has been shown advantageous for the language modelling of morphologically-rich languages Czapla et al. (2018); Mielke and Eisner (2019). This achieves improvements in accuracy for neural machine translation (NMT) tasks and has now become a prevailing practice Denkowski and Neubig (2017). The most widely-utilized method is Byte Pair Encoding BPE, Sennrich et al. (2016)), a compression algorithm that combines frequent sequences of characters, which results in rare strings being segmented into subwords. Unigram Kudo (2018) and BPE-Drop Provilkov et al. (2019) use subword ambiguity as noise, as well as stochastically-corrupted BPE segmentation to make it less deterministic. For NMT tasks generally, subword segmentation is seen as a monolingual task and applied independently on source and target corpora. We hypothesize that translation tasks, and specifically conversion tasks, as investigated here, would have a better performance if segmentation were performed jointly. Hence, in this section, we describe our proposed method _2kenize_, which jointly segments by taking the source and its approximate target sentences into account. This motivates the main idea of this paper: We propose _2kenize_ which jointly considers the source sentence and its corresponding target conversions by doing lookaeheads with mappings.

### Outline of the proposed approach

Given the possible SC character sequence \({\bf s}=s_{1}s_{2}\ldots s_{n}\) and TC character sequence \({\bf t}=t_{1}t_{2}\ldots t_{n}\), we want to find the most likely \({\bf t}\), which is given by the Bayes decision rule as follows:

\[{\bf t}=\operatorname*{arg\,max}_{{\bf f^{\prime}}\in T^{*}}p({\bf s},{\bf f^{ \prime}}) \tag{1}\]

where \(T^{*}\) denotes the set of all strings over symbols (\(t_{i}\)) in \(T\) (Kleene star). We divide this problem into two parts: finding the mapping sequence (2) and finding the TC sequence from mappings (7).

We define a mapping, which is given by \(m_{i}=(\mathfrak{s}_{i},\mathfrak{t}_{i})=(s_{j:k},\mathfrak{t}_{j:k})\). Here, \(\mathfrak{t}_{j:k}=\{t_{j:k}^{1}\dots t_{j:k}^{n}\}\) is a set of TC characters that correspond to the SC character in the mapping. Thus, a mapping sequence can be defined as a concatenation of mappings, which is \({\bf m}=m_{1}m_{2}\ldots m_{l}\). Let \(\mathcal{M}\) be the superset of all possible mapping sequences and \(\mathcal{M}({\bf s})\) be the all mapping sequences resulting from \({\bf s}\). Then, the best possible mapping sequence is given by

\[{\bf m}=\operatorname*{arg\,max}_{{\bf m^{\prime}}\in\mathcal{M}({\bf s})}p({ \bf m^{\prime}}) \tag{2}\]

Moreover, \(p({\bf m})\) can be expanded as such:

\[p({\bf m}) =p(m_{1}m_{2}\ldots m_{l}) \tag{3}\] \[=p\begin{pmatrix}\mathfrak{s}_{1}&\mathfrak{s}_{2}&\cdots& \mathfrak{s}_{l}\\ \mathfrak{t}_{1}&\mathfrak{t}_{2}&\cdots&\mathfrak{t}_{l}\end{pmatrix}\] (4) \[\approx p(\mathfrak{s}_{1}\mathfrak{s}_{2}\ldots\mathfrak{s}_{ l})+p(\mathfrak{t}_{1}\mathfrak{t}_{2}\ldots\mathfrak{t}_{l})\] (5) \[=p_{LM}(\mathfrak{s}_{1:l})+\sum_{l\in\prod_{i}\mathfrak{t}_{i}}p _{LM}(\mathfrak{t}_{1:l}) \tag{6}\]

After expanding the mapping sequences (4), we take an approximation by estimating this as the sum of likelihoods of two sequences formed due to co-segmentations (5). The set of possible TC sequences is given by the Cartesian product of \(\mathfrak{t}_{i}\). These likelihoods can then be estimated using language model (LM) probabilities as shown in (6).

\[{\bf t}=\operatorname*{arg\,max}_{{\bf f^{\prime}}\in{\bf m_{t}}}p({\bf f^{ \prime}})\] (7) 

Once the mapping sequence \({\bf m}\) has been found, all possible TC sequences are found from the set \({\bf m_{t}}\), which is the Cartesian product for all \({\rm t}_{i}\) in \({\bf m}\). From (7), we calculate approximate final sequence using beam search.

### Model Architecture

Viterbi, a dynamic programming (DP) algorithm, considers phrases (or subsequences) and performs segmentation in a 'bottom-up' fashion Nagata (1994); Sproat et al. (1996). RNN-based language models are theoretically considered to be '\(\infty\)'-gram Khandelwal et al. (2018), which consitutes a challenge. Consider this sentence, . A potential challenge could be to adequately estimate the probability of . As this sequence occurs infrequently in the beginning of sentences in the corpus, an RNN would under-estimate the probability of this subsequence. Moreover, an RNN would likely lose some useful context and perform worse without it Kim et al. (2019). So for Viterbi to perform well with an RNN, we train the language model on subsequences. We approach this by training our model in such a way that it samples subsequences randomly in each epoch. As shown in Fig 1, we randomly split the sentence and use subsequences in separate epochs.

Using Kudo (2018) regularization method, we sample from the \(n\)-best segmentations in each epoch. This is done so that the model can understand different segmentations of a subsequence using a similar motivation as above. Recent works have shown that varying subword segmentations lead to a better downstream model performance Provikov et al. (2019); Kudo (2018); Hiraoka et al. (2019); therefore, we use it as a data augmentation strategy. Once we get the \(n\)-best segmentations with scores, we normalize them, and then use the normalized scores as sampling probabilities (see Fig 1). As opposed to other subword tokenizers where the vocabulary size is fixed, we do not limit the vocabulary in our model. Hence, there are numerous possibilities of segment combinations which raises a need of caching most frequent tokens. Inspired by the work related to cache-based LMs Kawakami et al. (2017) and ghost batches Hoffer et al. (2017), we only consider the top-\(k\) tokens in the main network memory and keep track of gradients of less recently used token embeddings (commonly known as LRU, Least Recently Used policy). This could be thought of as

Figure 1: Language model architecture with subword and subsequence sampling. (Alt text: §F.1).

Figure 2: From the given SC sentence, we create possible TC sequences using mappings. We input these to Viterbi, which recursively calls LSTM. Using Eq. (6) as the scoring function, Viterbi outputs the mapping sequence. We perform beam search to find the best TC sequence from the mapping sequence. (Alt text: §F.2).

 

virtual embeddings as delayed gradient accumulation allows to accommodate larger number of tokens. This virtual size embedding architecture is related to the continuous cache implementation and stochastic tokenization architectures Grave et al. (2016); Hiraoka et al. (2019).

### Segmentation and Disambiguation

This optimal sequencing problem can be formulated as an overlapping subsequence approach, which can be solved using LM based Viterbi Nagata (1994); Sproat et al. (1996). Fig. 2 explains this process of joint subword modelling. Here, we take Eq. (6) as the objective function for finding the mapping sequence, however, we use subword perplexities Cotterell et al. (2018); Mielke et al. (2019); Mielke (2019) in our implementation. For the TC LSTM, we add the probabilities of the beams of the possible sequences.

As discussed in SS2.1 and Eq. (7), beam search is needed to select the best subword sequence for TC. Once the sentences are tokenized, the mapping table is used to convert each SC token to the corresponding TC token. We extract the final TC sentence by resolving ambiguities through beam search using the TC LSTM (Fig. 2).

## 3 Intrinsic Evaluation

### Dataset for Intrinsic Evaluation

We construct a gold standard corpus for both Chinese scripts consisting of 4 domains: HK Literature and Newswire, and Taiwanese Literature and Newswire (Table 2) with each domain containing 3000 sentences. SC-TC mapping tables are constructed from existing resources Denisowski (2019); Chu et al. (2012). We heuristically convert selected TC sentences to SC using OpenCC. We asked the annotators to manually correct any incorrect conversions.4

Footnote 4: A detailed data statement is given in the appendix.

### Language Model Training

We choose the SIGHAN-2005 Bakeoff dataset to train the segmentation-based language model Emerson (2005). For SC, we select the PKU and MSR partitions, and for TC, we use the Academia Sinica and CityU partitions. We apply maximal matching (or heuristic dictionary-based word segmenter) to pre-process these datasets by segmenting words into subwords Wong and Chan (1996). Here, 'dictionary' refers to the word-list in the mapping table. We then train a 2-layer LSTM language model LSTM with tied weights, and embedding and hidden sizes of 512 Sundermeyer et al. (2012) on this segmented dataset with subsequence sampling and stochastic tokenization as discussed in SS2.2.

### Baselines and Ablations5
Footnote 5: If in case you are looking for ‘Related Work’ section.

We implement the following baselines for the experimentation:

**Off-the-shelf Converters:** Hanziconv6 and Mafan7 are dictionary-based script character converters. Evaluating this could be useful to understand the lower accuracy bound. OpenCC8 uses a hybrid of characters and words (specifically trie based tokenizer) for script conversion Pranav A et al. (2019).

Footnote 6: [https://github.com/hernnaschaaf/mafan](https://github.com/hernnaschaaf/mafan)

Footnote 8: [https://github.com/BYVoid/OpenCC](https://github.com/BYVoid/OpenCC)

**Language Model Dismabiguation:** A strong baseline to this problem would be to build a language model to disambiguate between the characters, which is quite similar to STCP Xu et al. (2017). We use a 2-layer LSTM language model trained on Traditional Chinese corpus.

**Neural Sequence Models:** We heuristically convert Traditional Chinese Wikipedia to Simplified Chinese using OpenCC and use it for training the seq2seq model Sutskever et al. (2014). We

\begin{table}
\begin{tabular}{l r r r r}  & **HK Literature** & **HK News** & **TW Literature** \\ \hline _Sources_ & Liu (1962) & Singpao (2017-2018) & Jiubado (2011) & AS subset Emerson (2005) \\  & Lau Yee (1972) & Mingpao (2017-2018) & Ko (2010) & Liberty Times (2017-2018) \\ _Average Length_ & Foon (1988) & CityUSDENSENCen (2005) & Yao (1964) & United Daily News (2017-2018) \\ _Average Length_ & 194.8 & 214.6 & 188.2 & 223.6 \\ _IAA_ & 0.982 & 0.979 & 0.981 & 0.971 \\ _Mapping Examples_ & [] - [] 

construct a 20-layer neural convolutional sequence model Gehring et al. (2017) (both in encoder and decoder) using fairseq Ott et al. (2019).

We perform ablation tests by inserting following segmentation models.

**Word tokenization**: We use Jieba, which is a commonly used hidden markov model based word tokenizer for Chinese NLP. 9

Footnote 9: [https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)

**Dictionary substrings**: We apply maximal string matching, which is a dictionary based greedy tokenizer Pranav A et al. (2019); Wong and Chan (1996).

**Unigram** from Sentencepiece: Subword segmentation is performed by sampling unigram language model perplexity values Kudo (2018).

**Joint subwords**: As discussed in SS2.3, we use joint SC-TC subwords.

### Results for Intrinsic Evaluation

We evaluate our models using the metrics of disambiguation error density (DED) and sentence accuracy (SA). DED is the average of total edit distances per 1000 ambiguous Simplified characters, which is \(\frac{\sum\text{edit distances}}{\sum\text{ambiguous Simplified characters}}\times 1000\). SA is the number of sentences correctly converted in percentages. Contrary to previous papers, we do not report character based accuracy values, as generally most characters have straightforward mappings -- a reason why we opt for a less forgiving metric like SA where every character in a sentence has to be correctly converted.

Results are shown in Table 3, broken down by domain, and overall. Our model attains an average DED of 3.0 and a SA of 92.4% overall, whereas the best existing converter, OpenCC, only achieves a DED of 4.3 and a SA of 85.3%. We find that seq2seq and LM based disambiguation perform almost on par with OpenCC, due to the large number of false positive errors by these models. Jieba achieves an average DED of 11.2 as it does not handle OOV words well. For maximal matching of segmented words and Unigram subwords, it achieves an overall DED of 4.5 and 3.7, respectively -- showing that joint segmentation yields better results. We observe that accuracy values are slightly worse on news text, due to the relatively high number of new entities in those datasets. We find that seq2seq and LM based disambiguation gives rise to many false positives. Heuristically converting TC to SC results in certain conversion errors in the training dataset; and additionally, seq2seq approaches tend to reword the target sentence, which shows that they are unsuitable for this task.

### Qualitative Error Analysis

We manually inspect incorrect conversions in the intrinsic evaluation and find four interesting recurring linguistic patterns which confused the converters. We instructed the annotators to classify the items in the dataset (overall 12000 sentences in intrinsic evaluation dataset) if the sentences contain any of these patterns. In Table 4, we provide an overview of statistical information of these patterns and the performance by the converters.

**Code mixing**: Vernacular Cantonese characters (_zh-que_) are a subset of TC characters but do not follow the norms of the standard written Chinese Snow (2004). We find that some of the sentences in our dataset are code-mixed with _zh-yue_ (e.g. speech transcription) or English (e.g. named entities). Consider the snippet, "... "BNZ 190E [1]", which is code-mixed with

\begin{table}
\begin{tabular}{l c c c c c c c c c c}  & \multicolumn{2}{c}{**HK Lit**} & \multicolumn{2}{c}{**HK News**} & \multicolumn{2}{c}{**TW Lit**} & \multicolumn{2}{c}{**TW News**} & \multicolumn{2}{c}{**Overall**} \\ \hline
**Conversion System** & \multirow{2}{*}{DD} & \multirow{2}{*}{SA} & \multirow{2}{*}{DED} & \multirow{2}{*}{SA} & \multirow{2}{*}{DED} & \multirow{2}{*}{SA} & \multirow{2}{*}{DED} & \multirow{2}{*}{SA} & \multirow{2}{*}{DED} & \multirow{2}{*}{SA} & \multirow{2}{*}{DED} & \multirow{2}{*}{SA} \\ \cline{1-1} \cline{6-11} \cl 

both zh-yue and English. The characters "BENZ 190E", and are not a part of standard written Chinese. We find that OOV words are _2kenized_ into single-character tokens which results in: "Thus, _2kenize_ distributes the entropy over multiple tokens rather than a single token (generally UNK is used in such cases). This allows the language model to have more space for multiple guesses, which shows a massive advantage over word models or just UNKing it, a reason why subword tokenizers outperform closed-vocabulary models Merity [35].

* in the first occurrence, but then prefers in the second occurrence as it gets more context. _2kenize_ converts both of the entities correctly, very likely due to substructure lookups.

Footnote 10: This is a probability lookup table in Viterbi to keep track of the segment information in a subsequence.

**Failure Cases**: Dictionary-based converters (OpenCC, HanziConv and Mafan) only use the first conversion candidate12 if multiple candidates are available. STCP often converts named entities wrongly, especially the ones which have long-range dependencies and repetitions. Although we find that _2kenize_ converts some of the unseen named entities perfectly, some of the errors caused were due to infrequent characters. Few cases are mainly related to variant characters13 which are often used interchangeably.

\begin{table}
\begin{tabular}{l l r c} \hline
**Case** & **Method** & **SA** & **Example** \\ \hline Code mixing & & & _SC_ \\ with & & _With so much to do in Kennedy, can you handle it?_ & _HK Lit_ \\ Cantonese & OpenCC & 20.5 & \(X\) \\ (34 cases, & STCP & 8.8 & \(X\) \\
0.3\%) & 2kenize & 91.1 & \(Y\) \\ \hline  & & \(X\) \\  & & _After I drove Slick’s Benz 190E into the telephone pole,_ & _KK Lit_ \\  & & OpenCC & 95.6 & BENZ 190E += 



## 4 Extrinsic Evaluation

An accurate script converter should produce a less erroneous dataset, which should in turn improves the accuracy of the downstream tasks. In this section, we demonstrate the effect of script conversion on topic classification tasks to examine this assumption. We also study the impact of tokenization and pooling on the accuracy of topic classification. We apply the converter to the language modelling corpus (Wikitext), then train a classifier for informal and formal topic classification on that translated data. This allows us to measure the performance of the converter compared to other ones for a specific downstream task.

### Dataset for Extrinsic Evaluation

This section describes the dataset that we used for extrinsic evaluation experiments. It involves a pre-training dataset which consists Chinese Wikipedia and topic classification datasets.

#### 4.1.1 Pretraining Dataset

We use Chinese Wikipedia articles for pretraining the language model. Script conversion is an issue in Chinese Wikipedia, and currently, they use a server-side mechanism to automatically convert the scripts (dictionary-based) based on the location of the user. However, Wikipedia provides an option to view the article without conversion, which we use in the corpus.15 We use _zh-CN_, _zh-HK_ and _zh-yue_ wikis to retrieve articles originally written SC, TC and vernacular Cantonese + TC respectively with the help of wikiextractor.16 We pretrain the formal text classification models on articles from _zh-HK_ and converted _zh-CN_; and classification models for informal text on articles from _zh-HK_, _zh-yue_, and converted _zh-CN_.

Footnote 15: Note that it would not be straight-forward to compare against Wikipedia’s conversions directly, because they also perform some degree of manual post-processing [23].

#### 4.1.2 Classification Datasets:

We choose two classification tasks: formal news and informal topic classification (Table 5). For formal news, we scrape recent articles (2017-2019) from Singtao,17 for informal topics, we scrape posts (2017-2018) from LIHKG.18

Footnote 17: [https://github.com/attardi/wikietractor](https://github.com/attardi/wikietractor)

### Performance of various classifiers

For classification baselines, we use character-based SVM (Support Vector Machines, Joachims (1998)), CNN (Convolutional Nets, Zhang et al. (2015)) and Chinese BERT [23]. We also employ a state-of-the-art text classifier, MultiFiT [11], a lightweight RNN-based language model based classifier, which has shown to achieve a performance competitive with BERT [23] and ULMFiT [23]. The base architecture of MultiFiT is a 4-layer QRNN [22] with classifier head. We choose rectified Adam [15] with Lookahead [19] as the optimizer. We employ the cosine cyclic learning scheduler [20], where the limits of learning rate cycles are found by increasing the learning rate logarithmically and computing the evaluation loss for each learning rate [20]. To compute the batch size, we apply gradient noise scale to each batch size candidate and pick the one which gives the highest gradient noise scale [24]. We apply label smoothing [21] and use mixed precision training on RTX 2080. We implement our experiments using Pytorch [19] and FastAI [23].

MultiFiT uses _concat pooling_ after the last layer of QRNN, which means that the last time step is concatenated with an average and maximum

\begin{table}
\begin{tabular}{l r} \hline \multicolumn{2}{c}{**Formal Text Classification Dataset Overview**} \\ \hline _Source_ & Singtao \\ _Pretraining Corpus Size_ & 17500 \\ _Training Size_ & 3000 \\ _Validation Size_ & 450 \\ _Testing Size_ & Financial, Educational, Local \\  & International, Sports \\ _Language_ & zh-hart-hk \\ \hline \multicolumn{2}{c}{**Informal Text Classification Dataset Overview**} \\ \hline _Source_ & LIHKG \\ _Pretraining Corpus Size_ & 21000 \\ _Training Size_ & 4000 \\ _Validation Size_ & 450 \\ _Testing Size_ & 450 \\ _Categories_ & Sports, Opinions, IT \\  & Financial, Leisure, Memes \\ _Languages_ & zh-hant-hk, zh-yue, en-HK \\ \hline \end{tabular}
\end{table}
Table 5: Characteristics of classification dataset (Traditional Chinese) for extrinsic evaluation experiments.



pooled over previous time steps. Studies show that in LM based classifiers, different layers capture different types of knowledge-the last layer would be domain-specific and initial layers would be more generalized (Yosinski et al., 2014; Peters et al., 2019). We speculate that _concat pooling_ only on the last layer limits the information available to the classifier head and we hypothesise that the classifer would perform better if domain-specific as well as generalized knowledge were available to the head. For this reason, we augment the original MultiFIT architecture with _layer pooling_, which is concat pooling from all the layers, and pass that to the dense layer in the classifier, as shown in Fig 3.

We fine-tune the BERT language model and pretrain the MultiFiT language model on Chinese Wikipedia subsets (SS4.1.1). All classifiers are then trained on the given training set (character based models) and evaluated on the test set in terms of accuracy as number of items in each class are roughly equal. This experiment (and subsequent experiments in this section) is repeated across ten different seeds (Reimers and Gurevych, 2018) and data splits (Gorman and Bedrick, 2019) and the results are shown in Table 6. _Layer pooling_ shows an absolute improvement of 0.4% improvement over _concat pooling_ on formal and informal topic classification, thus confirming our hypothesis.

### Effect of Conversion on Classification

For each converter (OpenCC, STCP, _2kenize_), we translate _zh-CN_ wiki dataset and augment it with the TC wiki dataset. Then, we pretrain on this dataset, finetune on the domain data and train MultiFiT with layer pooling on these three datasets. We demonstate test set accuracies in Table 7. The dataset translated by _2kenize_ outperforms other converters, giving an absolute improvement of 0.9 % on formal and 4.5% over second-best converters on informal topic classification. These results emphasise that better script conversion improves the quality of the pretraining dataset, which boosts the performance of the downstream tasks like topic classification.

### Effect of Tokenization on Classification

Studies show that tokenization affects classification accuracy; open-vocabulary methods generally perform best (Eisenschlos et al., 2019; Hiraoka et al., 2019). For this experiment, we perform further ablations on our previous best classifier setup (MultiFiT with layer pooling on _2kenize_) to understand the effect of various subword tokenizers. Pretraining generally takes a long time (1-2 GPU days), hence we pretrain the classifier once for each tokenized corpus and do not perform sub

\begin{table}
\begin{tabular}{l r r}  & **Formal** & **Informal** \\ \hline Char-SVM & 73.2 & 63.7 \\ Char-CNN & 78.5 & 64.9 \\ Chinese BERT (base) & 84.5 & 66.3 \\ MultiFiT with no pooling & 87.5 & 68.5 \\ MultiFiT with concat pooling & 88.6 & 69.9 \\ MultiFiT with layer pooling & **89.0** & **70.3** \\ \end{tabular}
\end{table}
Table 6: Performance of various architectures on topic classification in terms of accuracy. The results are reported as a mean result across 10 different seeds and data splits. **Bold**: best, underlined: second best.

Figure 3: Proposed architecture for topic classification where we tweak MultiFiT to concatenate _concat-pools_ from all layers. (Alt text: §F.3).

\begin{table}
\begin{tabular}{l c c}
**Corpus Tokenization** & **Formal** & **Informal** \\ \hline Char & 93.2 & 77.9 \\ Jieba & 92.4 & 78.3 \\ BPE & 92.7 & 81.0 \\ BPE-Drop & 93.7 & 82.7 \\ Unigram & **94.8** & 82.2 \\ lresize & **94.8** & **83.2** \\ \hline \end{tabular}
\end{table}
Table 7: Ablation test of MultiFiT on different script converters. The results are reported as a mean accuracy result across 10 different seeds and data splits. **Bold**: best, underlined: second best.

 

word sampling for this experiment. For closed vocabulary methods, we use character and word segmentations (here with Jieba). Likewise, for open-vocabulary methods, we employ BPE, BPE-Drop and Unigram subword tokenizers.

Subword tokenizers mostly rely on frequency and do not take likelihood (something similar to \(n\)-gram language model) of tokenized sentence into consideration. Hence, we choose LM-based Viterbi segmentation (henceforth referred as _1kenize_), and here the LM would be the TC LSTM described in SS2.2. We report results in Table 8. We find that for formal classification, _1kenize_ and Unigram perform best. _1kenize_ outperforms other subword tokenizers for the noisier informal dataset, giving an absolute improvement of 0.5% over the second best method, which is BPE-Drop.

We plot a log frequency of tokens vs log order rank, which is shown in Figure 4. This distribution is based on the LIHKG dataset, which is noisier than other domains. We observe that character and word distributions are steeper than language model based subword tokenizers. This indicates that subword tokeniers produce a less skewed token distribution. Subword tokenizers like BPE and Unigram are deterministic and rely on frequency for segmentation. Since _1kenize_ is contextual, being LM-based, we find that it produces the least skewed distribution (lowest Zipf's law coefficient Zipf, 1949)), which also reduces variance, a reason why this simple segmentation method outperforms others for informal text classification.

## 5 Takeaways and Open Questions

The **contributions** of our work are:

* _2kenize_, a subword segmentation model, which jointly segments source sentence and its corresponding approximate target conversions.
* An **unsupervised script converter** based on _2kenize_ which shows a significant improvement over existing script converters and supervised models.
* _1kenize_, a variant of _2kenize_ which performs tokenization on only Traditional Chinese sentences which improves accuracy on topic classification tasks.
* **Character conversion evaluation datasets**: spanning Hong Kong and Taiwanese literature and news genres.
* **Traditional Chinese topic Classification datasets**: formal (scraped from Singtao) and informal (scraped from LIHKG) styles spanning genres like news, social media discussions, and memes.

The **key findings** of our work are:

* Our script converter shows a strong performance when dealing with code mixing and named entities. Supervised models are prone to anaphora and unseen entities related errors.
* A simple LM-based Viterbi segmentation model outperforms other subword tokenizers on topic classification tasks and reduces skewness of token distribution on a noisy dataset.

We leave some open questions to explore:

* How can we exploit subword variations to reduce skewness in the NLU tasks?
* Would subword-segmentation-transfer be helpful for other NMT-NLU task pairs like we did for _2kenize_ (script conversion) to _1kenize_ (classification)?

We anticipate that this study would be useful to TC NLP practitioners, as we address several research gaps, namely script conversion and a lack of benchmark datasets.

## Acknowledgements

The first author would like to thank Daya AI Limited, S.F. Hui, I-Tsun Cheng, Ishaan Batra, Conrad Ho, Roy Fork, Abhishek Gupta, Ajay Singh, Eugene Ho, Patrick Tu, Alex Chu, and Leland So for making valuable additions to this work. The second author would like to acknowledge funding from the Swedish Research Council for the project under grant agreement 2019-04129, which partly funded this work.

Figure 4: Log-log plots for different tokenizers. This is plotted frequency vs rank for the first 10000 tokens. Negative slopes calcuated from least squares are in the legend (lower means less skewed). (Alt text: §F.4).

 



## References

* Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. _Transactions of the Association for Computational Linguistics_, 6:587-604.
* Bradbury et al. (2016) James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016. Quasi-recurrent neural networks.
* Chu et al. (2012) Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2012. Chinese characters mapping table of Japanese, traditional Chinese and simplified Chinese. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012)_, pages 2149-2152, Istanbul, Turkey. European Languages Resources Association (ELRA).
* Contributors (2019) Wikipedia Contributors. 2019. Chinese script conversion and word processing in wikipedia. Page Version ID: 56925003.
* Cotterell et al. (2018) Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 536-541, New Orleans, Louisiana. Association for Computational Linguistics.
* Czapla et al. (2018) Piotr Czapla, Jeremy Howard, and Marcin Kardas. 2018. Universal language model fine-tuning with subword tokenization for polish. _ArXiv_, abs/1810.10222.
* Denisowski (2019) Paul Denisowski. 2019. Ce-cedict. [https://cc-cedict.org/](https://cc-cedict.org/).
* Denkowski and Neubig (2017) Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. In _Proceedings of the First Workshop on Neural Machine Translation_, pages 18-27, Vancouver. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Eisenschlos et al. (2019) Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingual language model fine-tuning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5706-5711, Hong Kong, China. Association for Computational Linguistics.
* Emerson (2005) Thomas Emerson. 2005. The second international Chinese word segmentation bakeoff. In _Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing_.
* Foon (1988) A Foon. 1988. Diary of the little man. Book. Pp.5-6.
* Volume 70_, ICML '17, page 1243-1252. JMLR.org.
* Gorman and Bedrick (2019) Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2786-2791, Florence, Italy. Association for Computational Linguistics.
* Grave et al. (2016) Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. _ArXiv_, abs/1612.04426.
* Halpern and Kerman (1999) Jack Halpern and Jouni Kerman. 1999. Pitfalls and complexities of chinese to chinese conversion. In _International Unicode Conference (14th) in Boston_.
* Hiraoka et al. (2019) Tatsuya Hiraoka, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Stochastic tokenization with a language model for neural text classification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1620-1629, Florence, Italy. Association for Computational Linguistics.
* Hoffer et al. (2017) Elad Hoffer, Itay Hubara, and Daniel Soudry. 2017. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In _NIPS_.
* Howard and Gugger (2020) Jeremy Howard and Sylvain Gugger. 2020. Fastai: A layered api for deep learning. _Information_, 11(2):108.
* Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
* Juibadoao (2011) Jiubadoao. 2011. _You Are the Apple of My Eye_. Chun Tian Chu Ban.
* Joachims (1998) Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. In _ECML_.
* Kawakami et al. (2017) Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in opensvocabulary neural language modeling. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1492-1502, Vancouver, Canada. Association for Computational Linguistics

Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 284-294, Melbourne, Australia. Association for Computational Linguistics.
* Kim et al. (2019) Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Koncro, Chris Dyer, and Gabor Melis. 2019. Unsupervised recurrent neural network grammars. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1105-1117, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ko (2010) Giddens Ko. 2010. _Cafe, Waiting, Love_. Spring Press.
* Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
* Yee (1972) Cheung Lau Yee. 1972. _Intersection_. Benefits Publishing Co., Ltd.
* Liu et al. (2019) Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. _ArXiv_, abs/1908.03265.
* Liu (1962) Yichang Liu. 1962. _Drunkard_. Benefits Publishing Co., Ltd.
* McCandlish et al. (2018) Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. An empirical model of large-batch training. _ArXiv_, abs/1812.06162.
* Merity (2019) Stephen Merity. 2019. Single headed attention rnn: Stop thinking with your head. _ArXiv_, abs/1911.11423.
* Mielke (2019) Sabrina J. Mielke. 2019. Can you compare perplexity across different segmentations?
* Mielke et al. (2019) Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4975-4989, Florence, Italy. Association for Computational Linguistics.
* Mielke and Eisner (2019) Sabrina J. Mielke and Jason Eisner. 2019. Spell once, summon anywhere: A two-level open-vocabulary language model. _AAAI_.
* Mikolov et al. (2012) Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cernocky. 2012. Subword language modeling with neural networks. _preprint (http://www. fit. vuthr. cz/imikolov/rnnlm/char. pdf)_, 8.
* Nagata (1994) Masaaki Nagata. 1994. A stochastic Japanese morphological analyzer using a forward-DP backward-A* n-best search algorithm. In _COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics_.
* Nianwen et al. (2016) Xue Nianwen, Zhang Xiuhong, Jiang Zixin, Palmer Martha, Xia Fei, Chiou Fu-Dong, and Meiyu Chang. 2016. Chinese treebank 9.0. LDC2016T13. Web Download. Philadelphia: Linguistic Data Consortium.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Dipl.-Ing. Kopf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjiie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _NeurIPS 2019_.
* Peters et al. (2019) Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_, pages 7-14, Florence, Italy. Association for Computational Linguistics.
* Guan et al. (2019) Pranav A, S.F. Hui, I-Tsun Cheng, Ishaan Batra, and Chiu Yik Hei. 2019. Learn languages first and then convert: Towards effective simplified to traditional chinese conversion. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Student Research Workshop, non-archival)_, Minneapolis, Minnesota. Association for Computational Linguistics.
* Provilkov et al. (2019) Ivan Provilkov, Dimitri Emelianenko, and Elena Voita. 2019. Bpe-dropout: Simple and effective subword regularization. _ArXiv_, abs/1910.13267.
* Reimers and Gurevych (2018) Nils Reimers and Iryna Gurevych. 2018. Why comparing single performance scores does not allow to draw conclusions about machine learning approaches.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.

 

Xiaodong Shi, Yidong Chen, and Xiuping Huang. 2011. Key problems in conversion from simplified to traditional chinesecharacters. In _International Conference on Asian Language Processing_.
* Smith (2015) Leslie N. Smith. 2015. Cyclical learning rates for training neural networks. _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_, pages 464-472.
* learning rate, batch size, momentum, and weight decay. _ArXiv_, abs/1803.09820.
* Snow (2004) Don Snow. 2004. _Cantonese as written language: The growth of a written Chinese vernacular_, volume 1. Hong Kong University Press.
* Sproat et al. (1996) Richard W. Sproat, Chilin Shih, William Gale, and Nancy Chang. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. _Computational Linguistics_, 22(3):377-404.
* Sundermeyer et al. (2012) Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In _INTERSPEECH_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_, pages 3104-3112. Curran Associates, Inc.
* Szegedy et al. (2015) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2818-2826.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv_, abs/1910.03771.
* Wong and Chan (1996) Pak-kwong Wong and Chorkin Chan. 1996. Chinese word segmentation based on maximum matching and word binding force. In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_.
* Xu et al. (2017) Jiarui Xu, Xuezhe Ma, Chen-Tse Tsai, and Eduard Hovy. 2017. Stcp: Simplified-traditional chinese conversion and proofreading. _Proceedings of the IJCNLP 2017, System Demonstrations_, pages 61-64.
* Yao (1964) Chiung Yao. 1964. Fire and rain. Book. ISBN 0-330-36076-0.
* Yosinski et al. (2014) Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In _NIPS_.
* Zhang et al. (2019) Michael Ruogu Zhang, James Lucas, Geoffrey E. Hinton, and Jimmy Ba. 2019. Lookahead optimizer: k steps forward, 1 step back. _ArXiv_, abs/1907.08610.
* Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _NIPS_.
* Zhang (2014) Xiaoheng Zhang. 2014. A comparative study on simplified-traditional chinese translation. In _Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data_, pages 212-222. Springer.
* Zipf (1949) George Kingsley Zipf. 1949. Human behavior and the principle of least effort.

 

Summary in Traditional Chinese:

1. 2kenize : [30]

[MISSING_PAGE_POST]

 



#### b.1.3 Speech Situation

The publication dates and sources are listed in the Table 2. HK and TW literature consists of popular books for which many movie and drama adaptations are made.18 Specifically, for HK literature, the text contains code-mixed characters with Vernacular Cantonese, which is quite unusual in formal publishing practices, and these books are often cited as an example for popularizing Cantonese in the 60s [45]. We also found code-mixing with English and numerous transliterated named entities which we have used for qualitative error analysis in the Table 4.

Footnote 18: We highly recommend these movies and novels as well.

#### b.1.4 Text Characteristics

Although Hong Kong and Taiwan both use Traditional Chinese, they are stylistically different as the dominant spoken language in HK is Cantonese and in TW is Taiwanese Mandarin. Thus, it is quite essential to test the performance of our algorithms on these two styles. We collected two genres for each style: informal literature and formal news. We found more variation within informal HK-TW literature as compared to the formal news. We intentionally chose long sentences (average length of 200 characters), especially which contain more ambiguous characters to make the dataset more challenging for testing.

## Appendix C Data Statement for Extrinsic Evaluation

This subsection describes the characteristics of the topic classification in Traditional Chinese. For the short overview, please see Table 5.

### Curation Rationale

We choose two different styles for curating this dataset: formal and informal. The formal text consists of news dataset scraped from Singtao, one of the popular newswire in Hong Kong. The classes in this dataset consist of Financial, Educational, Local, International, and Sports subsections. There are 17500 unlabelled and 3900 labelled items in this section. Authors would like to credit I-Tsun Cheng for giving us helpful suggestions in curating this dataset.

The informal text consists of social media posts dataset scraped from LIHKG, a Twitter equivalent in Hong Kong. The classes in this dataset consist of Sports, Opinions, Memes, IT, Financial and Leisure. There are 21000 unlabelled and 4900 labelled items in this section. Authors would like to credit Leland So for giving us helpful suggestions in curating this dataset.

### Language Variety

The texts in the formal subsection are typically written in Hong Kong style Traditional Chinese (zh-hant-hk). The posts scraped from LIHKG are predominantly in Traditional Chinese (zh-hanthk), and they are often code-mixed with Vernacular Cantonese (zh-yue) and English (en-HK).

### Speaker Demographic

Speakers were not directly approached for inclusion in this dataset and thus could not be asked for demographic information. Our best guess for demographic of LIHKG forum users are typically university students (19-23 years), and the majority of them speak Cantonese as a native language.

### Text Characteristics

The news articles are scraped from 2017-2019 and LIH KG posts are scraped from 2017-2018. Some of the posts in LIHKG are in the transliterated Cantonese form and some of them are not written in Standard Written Chinese. The news posts are generally quite long and often contains more than 5 sentences (average length of nearly 300 characters). On the other hand, the LIHKG posts are shorter and forums titles are generally one sentence each (average length of nearly 50 characters). Please note that due to the current situations in Hong Kong, we do not include political posts and news from mid-2019.

\begin{table}
\begin{tabular}{l c c}  & **RA** & \(\alpha\) \\ Character Level & 0.98 & 0.97 \\ Word Level & 0.95 & 0.94 \\ Sentence Level & 0.93 & 0.92 \\ \hline \end{tabular}
\end{table}
Table 10: Inter-annotator agreements as per different levels

\begin{table}
\begin{tabular}{c l|c c}  & & **RA** & \(\alpha\) \\ \hline
**HK** & & 0.98 & 0.98 \\  & Lit & 0.982 & 0.98 \\  & News & 0.979 & 0.97 \\ \hline
**TW** & & 0.98 & 0.98 \\  & Lit & 0.981 & 0.98 \\  & News & 0.971 & 0.97 \\ \hline \end{tabular}
\end{table}
Table 9: Inter-annotator agreements

Description of Intrinsic Evaluation Experiments

### Heuristic Grid Search of Learning Rate and Batch Size Hyperparameters

We employ the cosine cyclic learning scheduler (Smith, 2015), where the limits of learning rate cycles are found by increasing the learning rate logarithmically and computing the evaluation loss for each learning rate (Smith, 2018). To compute the batch size, we apply gradient noise scale to each batch size candidate and pick the one which gives the highest gradient noise scale (McCandlish et al., 2018).

### Training of SC and TC Language Model

The datasets are described in SS3.2. The model architecture is 2-layer LSTM language model with tied weights. Embedding size is 512 and hidden size is 512. We perform a concat pooling in the last layer where we concatenate the last output of the word, mean pool and max pool of all representations. We adopt comparable subword perplexity as suggested by Cotterell et al. (2018); Mielke et al. (2019); Mielke (2019), where we use a common denominator, referring to the number of segments per word in order to compare. On average, we achieve a perplexity of 168.6 on the Chinese Treebank test set (Nianwen et al., 2016). Also refer to Chinese LM Benchmark: [https://chinesenlp.xyz/#/docs/language_modeling](https://chinesenlp.xyz/#/docs/language_modeling). The training took 2 days on RTX 2080 with FP16 training, with a batch size of 256 and number of epochs of 250.

### Training of Convolutional seq2seq

Training dataset is a heuristically converted Traditional Chinese Wikipedia with OpenCC. We use 20 layers in encoder and decoder with the embedding size of 512 implemented in Fairseq (Ott et al., 2019). Dropout is 0.1 and we use adaptive softmax to speed up the training. The training took 1 day on RTX 2080 with FP16 training, with a batch size of 128 and number of epochs of 250.

## Appendix E Description of Extrinsic Evaluation Experiments

### Character CNN training

The datasets are described in SS4.1.2. The model architecture is 7-layer CNN with tied weights and residual blocks. Embedding size is 512 and hidden size is 512. We perform a concat pooling in the last layer where we concatenate the last output of the word, mean pool and max pool of all representations. The training took 16 hours on RTX 2080 with FP16 training, with a batch size of 256 and number of epochs of 350.

### Chinese BERT training

The datasets are described in SS4.1.2. We use Chinese BERT base (12-layer, 768-hidden, 12-heads, 110M parameters) using Transformers library (Wolf et al., 2019). We use sequence length of 384 and batch size of 12. Finetuning language model took 2 hours (learning rate of 3e-5) and finentuning classifier took 1 hour each on both datasets, including grid search on learning rates: 3e-4, 1e-4, 5e-5, 3e-5, where 3e-5 gives the best results (on RTX 2080 with FP16 training).

### MultiFiT training

We found MultiFiT is highly reproducible as compared to other models as it gives the least variance across the seeds and data splits. Hyperparameters are chosen by heuristic grid search on learning rate and batch size. The datasets are described in SS4.1.2. Pretraining language model takes 1 GPU day for each experiment of MultiFiT. Finetuning language model takes 3 hours where we used a patience of 2 epochs. Finetuning classifiers takes 3 hours where we used a patience of 2 epochs. All experiments of MultiFiT are implemented using FastAI Howard and Gugger (2020).

## Appendix F Alternative texts for figures and Chinese explanations

### Alternative text for Figure 1

The recurring Chinese sentence is split and we take one subsequence of it. The other subsequence is used in next iteration. We perform Unigram viterbi segmentation on this and get the probabilities. The probabilities are normalized and we sample a segmentation using this probability. This segmentation goes into the model which goes through cached embeddings, followed by stacked LSTM layers, followed by concat pooling (which consists of last output, mean pooling and max pooling) which then goes through a linear layer. We cache the top-k embeddings in the main memory and for the least frequent embeddings we track the gradients and do not keep them in the main network (we used gradient accumulation) 



### Alternative text for Figure 2

From the given SC sentence, we create possible TC sequences using mappings. We input these to Viterbi, which recursively calls LSTM. Using Eq. (6) as the scoring function, Viterbi outputs the mapping sequence. We perform beam search to find the best TC sequence from the mapping sequence where we used the same TC LSTM again.

### Alternative text for Figure 3

The architecture contains 4 stacked QRNN layers. Each layer has QRNN cells. After every layer we perform a concat pool (taking the last output, max pool and mean pool). We aggregate these pools in the final layer which goes into a linear layer. We highly recommend this for making the training more stable.

### Alternative text for Figure 4

We have plotted log-log token distribution. On x-axis we have order rank and on y-axis we have frequencies. Character based tokenization gives a slope of 1.703, BPE-Drop gives 1.31, BPE gives 1.27, word tokenization (Jieba) gives 1.41, uniform sampling gives 1.28 and 1kenize gives the least skewed distribution with a slope of 1.1. Note that these are negative slope and lower the slope is, more efficiently vocabulary is tokenized.

### Recurring Chinese sentence

Here, we explain the recurring sentence in this paper. In Table 1 we had SC sentence [], which means _Safeguarding the common interests of developing countries_. This is pronounced as _Weihu fazhan zhong guojia gongtong liyi_ in Mandarin. Its correct TC translation is [], which is pronounced as _wait 4wu6 faata3 zin2 zung1 gwok3 gaa1 gung6 tung4 lei9 ik1_ in Cantonese (note that the numerals are the tones).