SlowGuess/ACL_2020_OCR huggingface dataset:
Michael Ruogu Zhang, James Lucas, Geoffrey E. Hinton, and Jimmy Ba. 2019. Lookahead optimizer: k steps forward, 1 step back. ArXiv, abs/1907.08610.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS.
Xiaoheng Zhang. 2014. A comparative study on simplified-traditional chinese translation. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pages 212-222. Springer.
George Kingsley Zipf. 1949. Human behavior and the principle of least effort.
A Summary in Traditional Chinese: 簡體中文到繁體中文的文本轉換器
研究中文NLP時，將文本進行繁簡轉換是常見的數據預處理步驟。在簡繁轉換過程中，經常出現多個繁字轉換成同一簡體字，反之亦然。藉此透過測試現行的繁簡轉換算法，發現只有  $55 - 85\%$  準確度。進一步的調查發現，現代的神經網絡，譬如神經語言模型的字符歧義消除(neural language model disambiguation)和神經序列模型(neural sequence models)，均只達到  $84 - 85\%$  的句子準確性，都是由第一類錯誤(Type I error)所致。我們推斷上述問題，是由於模型未能有效釐清子詞(subword)的邊界所導致。
在此，我們提出了2kenize，一個子詞分割模型(subword segmentation model)，同時利用先行式繁體中文以及簡體中文進行建構。我們將聯合簡體中文及繁體中文共同訓練Viterbi分詞器。即使利用較具挑戰性的數據集測試，本模型亦達到  $91 - 95\%$  消歧準確度。透過定性誤差分析(qualitative error analysis)，展示了本模型更擅長處理code-mixing以及命名個體(named entities)除此以外，我們亦在主題分類領域中进行了外部評估，本模型更在主題分類的字符及詞語模型(character and word-based models)的領域中表現出眾，更在子詞正則化(subword regularization)中，獲得比BPE更好的名次。然後針對繁體中文句子對2kenize進行調整，誕生了1kenize。1kenize分別在正式數據集與其他子詞分詞器(subword tokenizers)名列前茅，在非正式數據集上更表現超群。由此，我們推斷子詞分詞器會嚴重地受token的分佈及偏度而影響
是次研究的貢獻：
1. 2kenize：簡體中文到繁體中文的文本轉換器
2. 字符轉換評估數據集：跨越香港和台灣文獻及新聞等多個類型的數據集
3. 主題分類數據集：繁體中文的正式和非正式文本數據涵蓋新聞，社交媒體討論，改圖，改歌，memes等二次創作文本。
B Data Statement for Intrinsic Evaluation
B.1 Corpus
In this subsection, we discuss the annotation procedure and the characteristics of the corpus used for the intrinsic evaluation. We have used Bender
and Friedman (2018) data statement design for the description.
B.1.1 Curation Rationale
The script conversion task is understudied in NLP and we could not find good quality parallel corpora to evaluate our approaches. The idea is to curate a diverse collection of TC works and convert them to SC, due to its one-to-one correspondence. However, we find out that some of the conversions were wrong because
1. sometimes dictionaries resulted in incorrect conversion,
2. stylistic differences between HK and TW characters and phrasing,
3. code-mixing of Cantonese and Traditional Chinese,
4. code-mixing with non-Chinese characters,
5. some characters in TC-SC conversion have one-to-many mappings as well.
Hence, we need quality control with human annotators to validate our conversions.
B.1.2 Annotation Process
Demographic: We opted for 4 trained annotators, 2 for annotating HK-style TC and 2 for annotating TW-style TC and thus going for double annotation for the corpus. They ranged in age from 18-20 years, included 2 men and 2 women, gave their ethnicity as Hong Kongers (2) and Taiwanese (2), and their native spoken languages were Cantonese (2) and Taiwanese Mandarin (2).
Workload: Annotators approximately validated 100 sentences per hour, comprising of total workload of 60 hours. They were given a month to annotate and were paid 5000 Hong Kong Dollars on completion.
Procedure: The annotators were shown TC and converted SC sentences (we used OpenCC to convert) and were asked to validate and correct any conversion mistakes. In case of disagreement, we used majority voting between automatically converted and annotators' corrections.
We provide raw agreement and Krippendorf's  $\alpha$  in Table 1 for pooled data and various sub-groups of the dataset. We also report inter-annotator agreements on character and phrasal levels in Table 2. These agreement values are difficult to interpret, but generally  $\alpha \geq 0.8$  is considered to be substantial.
7269
RA	α
HK	0.98	0.98
Lit	0.982	0.98
News	0.979	0.97
TW	0.98	0.98
Lit	0.981	0.98
News	0.971	0.97
Table 9: Inter-annotator agreements
RA	α
Character Level	0.98	0.97
Word Level	0.95	0.94
Sentence Level	0.93	0.92
Table 10: Inter-annotator agreements as per different levels
B.1.3 Speech Situation
The publication dates and sources are listed in the Table 2. HK and TW literature consists of popular books for which many movie and drama adaptations are made.[18] Specifically, for HK literature, the text contains code-mixed characters with Vernacular Cantonese, which is quite unusual in formal publishing practices, and these books are often cited as an example for popularizing Cantonese in the 60s (Snow, 2004). We also found code-mixing with English and numerous transliterated named entities which we have used for qualitative error analysis in the Table 4.
B.1.4 Text Characteristics
Although Hong Kong and Taiwan both use Traditional Chinese, they are stylistically different as the dominant spoken language in HK is Cantonese and in TW is Taiwanese Mandarin. Thus, it is quite essential to test the performance of our algorithms on these two styles. We collected two genres for each style: informal literature and formal news. We found more variation within informal HK-TW literature as compared to the formal news. We intentionally chose long sentences (average length of 200 characters), especially which contain more ambiguous characters to make the dataset more challenging for testing.
C Data Statement for Extrinsic Evaluation
This subsection describes the characteristics of the topic classification in Traditional Chinese. For the
short overview, please see Table 5.
C.1 Curation Rationale
We choose two different styles for curating this dataset: formal and informal. The formal text consists of news dataset scraped from Singtao, one of the popular newswire in Hong Kong. The classes in this dataset consist of Financial, Educational, Local, International, and Sports subsections. There are 17500 unlabelled and 3900 labelled items in this section. Authors would like to credit I-Tsun Cheng for giving us helpful suggestions in curating this dataset.
The informal text consists of social media posts dataset scraped from LIHKG, a Twitter equivalent in Hong Kong. The classes in this dataset consist of Sports, Opinions, Memes, IT, Financial and Leisure. There are 21000 unlabelled and 4900 labelled items in this section. Authors would like to credit Leland So for giving us helpful suggestions in curating this dataset.
C.2 Language Variety
The texts in the formal subsection are typically written in Hong Kong style Traditional Chinese (zh-hant-hk). The posts scraped from LIHKG are predominantly in Traditional Chinese (zh-hant-hk), and they are often code-mixed with Vernacular Cantonese (zh-yue) and English (en-HK).
C.3 Speaker Demographic
Speakers were not directly approached for inclusion in this dataset and thus could not be asked for demographic information. Our best guess for demographic of LIHKG forum users are typically university students (19-23 years), and the majority of them speak Cantonese as a native language.
C.4 Text Characteristics
The news articles are scraped from 2017-2019 and LIHKG posts are scraped from 2017-2018. Some of the posts in LIHKG are in the transliterated Cantonese form and some of them are not written in Standard Written Chinese. The news posts are generally quite long and often contains more than 5 sentences (average length of nearly 300 characters). On the other hand, the LIHKG posts are shorter and forums titles are generally one sentence each (average length of nearly 50 characters). Please note that due to the current situations in Hong Kong, we do not include political posts and news from mid-2019.
<sup>18</sup>We highly recommend these movies and novels as well.
7270
D Description of Intrinsic Evaluation Experiments
D.1 Heuristic Grid Search of Learning Rate and Batch Size Hyperparameters
We employ the cosine cyclic learning scheduler (Smith, 2015), where the limits of learning rate cycles are found by increasing the learning rate logarithmically and computing the evaluation loss for each learning rate (Smith, 2018). To compute the batch size, we apply gradient noise scale to each batch size candidate and pick the one which gives the highest gradient noise scale (McCandlish et al., 2018).
D.2 Training of SC and TC Language Model
The datasets are described in §3.2. The model architecture is 2-layer LSTM language model with tied weights. Embedding size is 512 and hidden size is 512. We perform a concat pooling in the last layer where we concatenate the last output of the word, mean pool and max pool of all representations. We adopt comparable subword perplexity as suggested by Cotterell et al. (2018); Mielke et al. (2019); Mielke (2019), where we use a common denominator, referring to the number of segments per word in order to compare. On average, we achieve a perplexity of 168.6 on the Chinese Treebank test set (Nianwen et al., 2016). Also refer to Chinese LM Benchmark: https://chinesenlp.xyz/#/docs/language_modeling. The training took 2 days on RTX 2080 with FP16 training, with a batch size of 256 and number of epochs of 250.
D.3 Training of Convolutional seq2seq
Training dataset is a heuristically converted Traditional Chinese Wikipedia with OpenCC. We use 20 layers in encoder and decoder with the embedding size of 512 implemented in Fairseq (Ott et al., 2019). Dropout is 0.1 and we use adaptive softmax to speed up the training. The training took 1 day on RTX 2080 with FP16 training, with a batch size of 128 and number of epochs of 250.
E Description of Extrinsic Evaluation Experiments
E.1 Character CNN training
The datasets are described in  $\S 4.1.2$ . The model architecture is 7-layer CNN with tied weights and residual blocks. Embedding size is 512 and hidden size is 512. We perform a concat pooling in
the last layer where we concatenate the last output of the word, mean pool and max pool of all representations. The training took 16 hours on RTX 2080 with FP16 training, with a batch size of 256 and number of epochs of 350.
E.2 Chinese BERT training
The datasets are described in §4.1.2. We use Chinese BERT base (12-layer, 768-hidden, 12-heads, 110M parameters) using Transformers library (Wolf et al., 2019). We use sequence length of 384 and batch size of 12. Finetuning language model took 2 hours (learning rate of 3e-5) and finetuning classifier took 1 hour each on both datasets, including grid search on learning rates: 3e-4, 1e-4, 5e-5, 3e-5, where 3e-5 gives the best results (on RTX 2080 with FP16 training).
E.3 MultiFiT training
We found MultiFiT is highly reproducible as compared to other models as it gives the least variance across the seeds and data splits. Hyperparameters are chosen by heuristic grid search on learning rate and batch size. The datasets are described in §4.1.2. Pretraining language model takes 1 GPU day for each experiment of MultiFiT. Finetuning language model takes 3 hours where we used a patience of 2 epochs. Finetuning classifiers takes 3 hours where we used a patience of 2 epochs. All experiments of MultiFiT are implemented using FastAI (Howard and Gugger, 2020).
F Alternative texts for figures and Chinese explanations
F.1 Alternative text for Figure 1
The recurring Chinese sentence is split and we take one subsequence of it. The other subsequence is used in next iteration. We perform Unigram viterbi segmentation on this and get the probabilities. The probabilities are normalized and we sample a segmentation using this probability. This segmentation goes into the model which goes through cached embeddings, followed by stacked LSTM layers, followed by concat pooling (which consists of last output, mean pooling and max pooling) which then goes through a linear layer. We cache the top-k embeddings in the main memory and for the least frequent embeddings we track the gradients and do not keep them in the main network (we used gradient accumulation).
7271
F.2 Alternative text for Figure 2
From the given SC sentence, we create possible TC sequences using mappings. We input these to Viterbi, which recursively calls LSTM. Using Eq. (6) as the scoring function, Viterbi outputs the mapping sequence. We perform beam search to find the best TC sequence from the mapping sequence where we used the same TC LSTM again.
F.3 Alternative text for Figure 3
The architecture contains 4 stacked QRNN layers. Each layer has QRNN cells. After every layer we perform a concat pool (taking the last output, max pool and mean pool). We aggregate these pools in the final layer which goes into a linear layer. We highly recommend this for making the training more stable.
F.4 Alternative text for Figure 4
We have plotted log-log token distribution. On x-axis we have order rank and on y-axis we have frequencies. Character based tokenization gives a slope of 1.703, BPE-Drop gives 1.31, BPE gives 1.27, word tokenization (Jieba) gives 1.41, unigram sampling gives 1.28 and 1kenize gives the least skewed distribution with a slope of 1.1. Note that these are negative slope and lower the slope is, more efficiently vocabulary is tokenized.
F.5 Recurring Chinese sentence
Here, we explain the recurring sentence in this paper. In Table 1 we had SC sentence维护发展中国家共同利益, which means Safeguarding the common interests of developing countries. This is pronounced as Weihu fāzhàn zhong guójiā gòngtóng liyi in Mandarin. Its correct TC translation is 維護發展中國家共同利益, which is pronounced as wai4 wu6 faat3 zin2zung1 gwok3 gaa1 gung6 tung4 lei6 jik1 in Cantonese (note that the numerals are the tones).