SlowGuess/ACL_2020_OCR huggingface dataset:
Formal Text Classification Dataset Overview
Source	Singtao
Pretraining Corpus Size	17500
Training Size	3000
Validation Size	450
Testing Size	450
Categories	Financial, Educational, Local International, Sports
Language	zh-hant-hk
Informal Text Classification Dataset Overview
Source	LIHKG
Pretraining Corpus Size	21000
Training Size	4000
Validation Size	450
Testing Size	450
Categories	Sports, Opinions, IT
Financial, Leisure, Memes
Languages	zh-hant-hk, zh-yue, en-HK
Table 5: Characteristics of classification dataset (Traditional Chinese) for extrinsic evaluation experiments.
4 Extrinsic Evaluation
An accurate script converter should produce a less erroneous dataset, which should in turn improves the accuracy of the downstream tasks. In this section, we demonstrate the effect of script conversion on topic classification tasks to examine this assumption. We also study the impact of tokenization and pooling on the accuracy of topic classification. We apply the converter to the language modelling corpus (Wikitext), then train a classifier for informal and formal topic classification on that translated data. This allows us to measure the performance of the converter compared to other ones for a specific downstream task.
4.1 Dataset for Extrinsic Evaluation
This section describes the dataset that we used for extrinsic evaluation experiments. It involves a pretraining dataset which consists Chinese Wikipedia and topic classification datasets.
4.1.1 Pretraining Dataset
We use Chinese Wikipedia articles for pretraining the language model. Script conversion is an issue in Chinese Wikipedia, and currently, they use a server-side mechanism to automatically convert the scripts (dictionary-based) based on the location of the user. However, Wikipedia provides an option to view the article without conversion, which
we use in the corpus. $^{14}$  We use  $zh-CN$ ,  $zh-HK$  and  $zh-yue$  wikis to retrieve articles originally written SC, TC and vernacular Cantonese + TC respectively with the help of wikiextractor. $^{15}$  We pretrain the formal text classification models on articles from  $zh-HK$  and converted  $zh-CN$ ; and classification models for informal text on articles from  $zh-HK$ ,  $zh-yue$ , and converted  $zh-CN$ .
4.1.2 Classification Datasets:
We choose two classification tasks: formal news and informal topic classification (Table 5). For formal news, we scrape recent articles (2017-2019) from Singtao, $^{16}$  for informal topics, we scrape posts (2017-2018) from LIHKG. $^{17}$
4.2 Performance of various classifiers
For classification baselines, we use character-based SVM (Support Vector Machines, Joachims (1998)), CNN (Convolutional Nets, Zhang et al. (2015)) and Chinese BERT (Devlin et al., 2019). We also employ a state-of-the-art text classifier, MultiFiT (Eisenschlos et al., 2019), a lightweight RNN-based language model based classifier, which has shown to achieve a performance competitive with BERT (Devlin et al., 2019) and ULMFiT (Howard and Ruder, 2018). The base architecture of MultiFiT is a 4-layer QRNN (Bradbury et al., 2016) with classifier head. We choose rectified Adam (Liu et al., 2019) with Lookahead (Zhang et al., 2019) as the optimizer. We employ the cosine cyclic learning scheduler (Smith, 2015), where the limits of learning rate cycles are found by increasing the learning rate logarithmically and computing the evaluation loss for each learning rate (Smith, 2018). To compute the batch size, we apply gradient noise scale to each batch size candidate and pick the one which gives the highest gradient noise scale (McCandlish et al., 2018). We apply label smoothing (Szegedy et al., 2015) and use mixed precision training on RTX 2080. We implement our experiments using Pytorch (Paszke et al., 2019) and FastAI (Howard and Gugger, 2020).
MultiFiT uses concat pooling after the last layer of QRNN, which means that the last time step is concatenated with an average and maximum
<sup>14</sup> Note that it would not be straight-forward to compare against Wikipedia's conversions directly, because they also perform some degree of manual post-processing (Contributors, 2019).
<sup>15</sup>https://github.com/attardi/wikiextractor
16https://std.stheadline.com
17https://lihkg.com
7263
Figure 3: Proposed architecture for topic classification where we tweak MultiFiT to concatenate concat-pools from all layers. (Alt text: §F.3).
Formal	Informal
Char-SVM	73.2	63.7
Char-CNN	78.5	64.9
Chinese BERT (base)	84.5	66.3
MultiFiT with no pooling	87.5	68.5
MultiFiT with concat pooling	88.6	69.9
MultiFiT with layer pooling	89.0	70.3
pooled over previous time steps. Studies show that in LM based classifiers, different layers capture different types of knowledge—the last layer would be domain-specific and initial layers would be more generalized (Yosinski et al., 2014; Peters et al., 2019). We speculate that concat pooling only on the last layer limits the information available to the classifier head and we hypothesise that the classifier would perform better if domain-specific as well as generalized knowledge were available to the head. For this reason, we augment the original MultiFIT architecture with layer pooling, which is concat pooling from all the layers, and pass that to the dense layer in the classifier, as shown in Fig 3.
We fine-tune the BERT language model and pretrain the MultiFiT language model on Chinese Wikipedia subsets (§4.1.1). All classifiers are then trained on the given training set (character based models) and evaluated on the test set in terms of accuracy as number of items in each class are roughly equal. This experiment (and subsequent experiments in this section) is repeated across ten different seeds (Reimers and Gurevych, 2018) and data splits (Gorman and Bedrick, 2019) and the results are shown in Table 6. Layer pooling shows an absolute improvement of  $0.4\%$  improvement over concat pooling on formal and informal topic clas
Table 6: Performance of various architectures on topic classification in terms of accuracy. The results are reported as a mean result across 10 different seeds and data splits. **Bold:** best, **underlined:** second best.
Pretraining data of MultiFiT	Formal	Informal
No Conversions	89.0	70.3
Including conversions with OpenCC	91.7	75.6
Including conversions with STCP	92.3	73.4
Including conversions with 2kenize	93.2	77.9
Table 7: Ablation test of MultiFiT on different script converters. The results are reported as a mean accuracy result across 10 different seeds and data splits. **Bold:** best, **underlined:** second best.
Corpus Tokenization	Formal	Informal
Char	93.2	77.9
Jieba	92.4	78.3
BPE	92.7	81.0
BPE-Drop	93.7	82.7
Unigram	94.8	82.2
1kenize	94.8	83.2
Table 8: Ablation test of MultiFiT on tokenizers. The results are reported as a mean accuracy result across 10 different seeds and data splits. **Bold:** best, **underlined:** second best.
sification, thus confirming our hypothesis.
4.3 Effect of Conversion on Classification
For each converter (OpenCC, STCP, 2kenize), we translate  $zh-CN$  wiki dataset and augment it with the TC wiki dataset. Then, we pretrain on this dataset, finetune on the domain data and train MultiFiT with layer pooling on these three datasets. We demonstrate test set accuracies in Table 7. The dataset translated by 2kenize outperforms other converters, giving an absolute improvement of  $0.9\%$  on formal and  $4.5\%$  over second-best converters on informal topic classification. These results emphasise that better script conversion improves the quality of the pretraining dataset, which boosts the performance of the downstream tasks like topic classification.
4.4 Effect of Tokenization on Classification
Studies show that tokenization affects classification accuracy; open-vocabulary methods generally perform best (Eisenschlos et al., 2019; Hiraoka et al., 2019). For this experiment, we perform further ablations on our previous best classifier setup (MultiFiT with layer pooling on 2kenize) to understand the effect of various subword tokenizers. Pretraining generally takes a long time (1-2 GPU days), hence we pretrain the classifier once for each tokenized corpus and do not perform sub
7264
Figure 4: Log-log plots for different tokenizers. This is plotted frequency vs rank for the first 10000 tokens. Negative slopes calculated from least squares are in the legend (lower means less skewed). (Alt text: §F.4).
word sampling for this experiment. For closed vocabulary methods, we use character and word segmentations (here with Jieba). Likewise, for open-vocabulary methods, we employ BPE, BPE-Drop and Unigram subword tokenizers.
Subword tokenizers mostly rely on frequency and do not take likelihood (something similar to  $n$ -gram language model) of tokenized sentence into consideration. Hence, we choose LM-based Viterbi segmentation (henceforth referred as lkenize), and here the LM would be the TC LSTM described in §2.2. We report results in Table 8. We find that for formal classification, lkenize and Uni-gram perform best. lkenize outperforms other subword tokenizers for the noisier informal dataset, giving an absolute improvement of  $0.5\%$  over the second best method, which is BPE-Drop.
We plot a log frequency of tokens vs log order rank, which is shown in Figure 4. This distribution is based on the LIHKG dataset, which is noisier than other domains. We observe that character and word distributions are steeper than language model based subword tokenizers. This indicates that subword tokenizers produce a less skewed token distribution. Subword tokenizers like BPE and Unigram are deterministic and rely on frequency for segmentation. Since Ikenize is contextual, being LM-based, we find that it produces the least skewed distribution (lowest Zipf's law coefficient (Zipf, 1949)), which also reduces variance, a reason why this simple segmentation method outperforms others for informal text classification.
5 Takeaways and Open Questions
The contributions of our work are:
- 2kenize, a subword segmentation model, which jointly segments source sentence and its corresponding approximate target conversions.
- An unsupervised script converter based on 2kenize which shows a significant improvement over existing script converters and supervised models.
- 1kenize, a variant of 2kenize which performs tokenization on only Traditional Chinese sentences which improves accuracy on topic classification tasks.
- Character conversion evaluation datasets: spanning Hong Kong and Taiwanese literature and news genres.
- Traditional Chinese topic Classification datasets: formal (scraped from Singtao) and informal (scraped from LIHKG) styles spanning genres like news, social media discussions, and memes.
The key findings of our work are:
- Our script converter shows a strong performance when dealing with code mixing and named entities. Supervised models are prone to anaphora and unseen entities related errors.
- A simple LM-based Viterbi segmentation model outperforms other subword tokenizers on topic classification tasks and reduces skewness of token distribution on a noisy dataset.
We leave some open questions to explore:
- How can we exploit subword variations to reduce skewness in the NLU tasks?
- Would subword-segmentation-transfer be helpful for other NMT-NLU task pairs like we did for 2kenize (script conversion) to 1kenize (classification)?
We anticipate that this study would be useful to TC NLP practitioners, as we address several research gaps, namely script conversion and a lack of benchmark datasets.
The first author would like to thank Dayta AI Limited, S.F. Hui, I-Tsun Cheng, Ishaan Batra, Conrad Ho, Roy Fork, Abhishek Gupta, Ajay Singh, Eugene Ho, Patrick Tu, Alex Chu, and Leland So for making valuable additions to this work. The second author would like to acknowledge funding from the Swedish Research Council for the project under grant agreement 2019-04129, which partly funded this work.
7265
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016. Quasi-recurrent neural networks.
Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2012. Chinese characters mapping table of Japanese, traditional Chinese and simplified Chinese. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2149-2152, Istanbul, Turkey. European Languages Resources Association (ELRA).
Wikipedia Contributors. 2019. Chinese script conversion and word processing in wikipedia. Page Version ID: 56925003.
Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 536-541, New Orleans, Louisiana. Association for Computational Linguistics.
Piotr Czapla, Jeremy Howard, and Marcin Kardas. 2018. Universal language model fine-tuning with subword tokenization for polish. *ArXiv*, abs/1810.10222.
Paul Denisowski. 2019. Cc-ceduct. https://cc-ceduct.org/.
Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 18-27, Vancouver. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingual language model fine-tuning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP), pages 5706-5711, Hong Kong, China. Association for Computational Linguistics.
Thomas Emerson. 2005. The second international Chinese word segmentation bakeoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
A Foon. 1988. Diary of the little man. Book. Pp.5-6.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML '17, page 1243-1252. JMLR.org.
Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786-2791, Florence, Italy. Association for Computational Linguistics.
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. *ArXiv*, abs/1612.04426.
Jack Halpern and Jouni Kerman. 1999. Pitfalls and complexities of chinese to chinese conversion. In International Unicode Conference (14th) in Boston.
Tatsuya Hiraoka, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Stochastic tokenization with a language model for neural text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1620-1629, Florence, Italy. Association for Computational Linguistics.
Elad Hoffer, Itay Hubara, and Daniel Soudry. 2017. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In NIPS.
Jeremy Howard and Sylvain Gugger. 2020. Fastai: A layered api for deep learning. Information, 11(2):108.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Jiubadao. 2011. You Are the Apple of My Eye. Chun Tian Chu Ban.
Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. In ECML.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1492-1502, Vancouver, Canada. Association for Computational Linguistics.
7266
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284-294, Melbourne, Australia. Association for Computational Linguistics.
Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019. Unsupervised recurrent neural network grammars. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1105-1117, Minneapolis, Minnesota. Association for Computational Linguistics.
Giddens Ko. 2010. Cafe, Waiting, Love. Spring Press.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Cheung Lau Yee. 1972. Intersection. Benefits Publishing Co., Ltd.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. ArXiv, abs/1908.03265.
Yichang Liu. 1962. *Drunkard*. Benefits Publishing Co., Ltd.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. An empirical model of large-batch training. *ArXiv*, abs/1812.06162.
Stephen Merity. 2019. Single headed attention rnn: Stop thinking with your head. ArXiv, abs/1911.11423.
Sabrina J. Mielke. 2019. Can you compare perplexity across different segmentations?
Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975-4989, Florence, Italy. Association for Computational Linguistics.
Sabrina J. Mielke and Jason Eisner. 2019. Spell once, summon anywhere: A two-level open-vocabulary language model. AAAI.
Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-son Le, Stefan Kombrink, and Jan Cernocky. 2012. Subword language modeling with neural networks. preprint (http://www.fit.vutbr.cz/imikolov/rnml/char.pdf), 8.
Masaaki Nagata. 1994. A stochastic Japanese morphological analyzer using a forward-DP backward-A* n-best search algorithm. In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics.
Xue Nianwen, Zhang Xiuhong, Jiang Zixin, Palmer Martha, Xia Fei, Chiou Fu-Dong, and Meiyu Chang. 2016. Chinese treebank 9.0. LDC2016T13. Web Download. Philadelphia: Linguistic Data Consortium.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Dipl.-Ing. Kopf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS 2019.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7-14, Florence, Italy. Association for Computational Linguistics.
Pranav A, S.F. Hui, I-Tsun Cheng, Ishaan Batra, and Chiu Yik Hei. 2019. Learn languages first and then convert: Towards effective simplified to traditional chinese conversion. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Student Research Workshop, non-archival), Minneapolis, Minnesota. Association for Computational Linguistics.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019. Bpe-dropout: Simple and effective subword regularization. *ArXiv*, abs/1910.13267.
Nils Reimers and Iryna Gurevych. 2018. Why comparing single performance scores does not allow to draw conclusions about machine learning approaches.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
7267
Xiaodong Shi, Yidong Chen, and Xiuping Huang. 2011. Key problems in conversion from simplified to traditional chinesesecharacters. In International Conference on Asian Language Processing.
Leslie N. Smith. 2015. Cyclical learning rates for training neural networks. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464-472.
Leslie N. Smith. 2018. A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. *ArXiv*, abs/1803.09820.
Don Snow. 2004. Cantonese as written language: The growth of a written Chinese vernacular, volume 1. Hong Kong University Press.
Richard W. Sproat, Chilin Shih, William Gale, and Nancy Chang. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics, 22(3):377-404.
Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In INTERSPEECH.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Pak-kwong Wong and Chorkin Chan. 1996. Chinese word segmentation based on maximum matching and word binding force. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
Jiarui Xu, Xuezhe Ma, Chen-Tse Tsai, and Eduard Hovy. 2017. Stcp: Simplified-traditional chinese conversion and proofreading. Proceedings of the IJCNLP 2017, System Demonstrations, pages 61-64.
Chiung Yao. 1964. Fire and rain. Book. ISBN 0-330-36076-0.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In NIPS.