SlowGuess/NAACL_2021_OCR huggingface dataset:
Models	PKU	MSRA	CITYU	AS	CKIP	NCC	SXU	CTB6	CNC	Avg.
Chen et al. (2017)	94.32	96.04	95.55	94.64	94.26	92.83	96.04	-	-	-
Ma et al. (2018)	96.10	97.40	97.20	96.20	-	-	-	96.70	-	-
He et al. (2019)	95.78	97.35	95.60	95.47	95.73	94.34	96.49	-	-	-
Gong et al. (2019)	96.15	97.78	96.22	95.22	94.99	94.12	97.25	-	-	-
Yang et al. (2019)	95.80	97.80	-	-	-	-	-	96.10	-	-
Meng et al. (2019)	96.70	98.30	97.90	96.70	-	-	-	-	-	-
Yang (2019)	96.50	98.40	-	-	-	-	-	-	-	-
Duan and Zhao (2020)	95.50	97.70	96.40	95.70	-	-	-	-	-	-
Huang et al. (2020)	97.30	98.50	97.80	97.00	-	-	97.50	97.80	97.30	-
Qiu et al. (2020)	96.41	98.05	96.91	96.44	96.51	96.04	97.61	-	-	-
Tian et al. (2020)	96.53	98.40	97.93	96.62	-	-	-	97.25	-	-
BERT-Base (ours)	96.72	98.25	98.19	96.93	96.49	96.13	97.61	97.85	97.45	97.29
METASEG (w/o fine-tune)	96.76	98.02	98.12	97.04	96.81	97.21	97.51	97.87	97.25	97.40
METASEG	96.92	98.50	98.20	97.01	96.72	97.24	97.88	97.89	97.55	97.55
Table 3: F1 scores on test sets of pre-training criteria. The first block displays results from previous works. The second block displays three models implemented by us.
Models	WTB	UD	ZX	Avg.
Ma et al. (2018)	-	96.90	-	-
Huang et al. (2020)	93.20	97.80	97.10	96.03
BERT-Base (ours)	93.00	98.32	97.06	96.13
METASEG (w/o fine-tune)	89.53	83.84	88.48	87.28
METASEG	93.97	98.57	97.22	96.59
Table 4: F1 scores on test sets of downstream criteria.
Models	WTB	UD	ZX	Avg.
METASEG	93.97	98.57	97.22	96.59
-ML	93.71	98.49	97.22	96.47
-ML-MP	93.00	98.32	97.06	96.13
Table 5: F1 scores for ablation studies on downstream criteria. -ML indicates METASEG without meta learning. -ML-MP indicates METASEG without meta learning and multi-criteria pre-training.
4.3.1 Low-Resource Settings
To better explore the downstream generalization ability of METASEG, we perform experiments on the downstream WTB criterion in low-resource settings. Specifically, we randomly sample a given rate of instances from the training set and fine-tune the pre-trained METASEG model on down-sampling training sets. These settings imitate the realistic low-resource circumstance where human-annotated data is insufficient.
The performance at different sampling rates is evaluated on the same WTB test set and reported in Table 6. Results show that METASEG outperforms BERT-Base at every sampling rate. The margin is larger when the sampling rate is lower, and reaches
$6.20\%$  at  $1\%$  sampling rate. This demonstrates that METASEG could generalize better on the downstream criterion in low-resource settings.
When the sampling rate drops from  $100\%$  to  $1\%$ , F1 score of BERT-Base decreases by  $7.60\%$  while that of METASEG only decreases by  $2.37\%$ . The performance of METASEG at  $1\%$  sampling rate still reaches  $91.60\%$  with only 8 instances, comparable with performance of BERT-Base at  $20\%$  sampling rate. This indicates that METASEG can make better use of prior segmentation knowledge and learn from less amount of data. It shows that METASEG would reduce the need of human annotation significantly.
4.3.2 Out-of-Vocabulary Words
Out-of-Vocabulary (OOV) words denote the words which exist in inference phase but don't exist in training phase. OOV words are a critical cause of errors on CWS tasks. We evaluate recalls for OOV words on test sets of all twelve criteria in Table 7.
Results show that METASEG outperforms BERT-Base on ten of twelve criteria and improves recalls for OOV words by  $0.99\%$  on average. This indicates that METASEG could benefit from our proposed pre-training methodology and recognize more OOV words in inference phase.
4.3.3 Non-Pretraining Setup
To investigate the contribution of multi-criteria pretraining towards performance of METASEG, we perform experiments on a non-pretraining baseline Transformer. Transformer has the same architecture and is directly trained from scratch on the same nine datasets (Section 4.2.1), but doesn't have any pre-training phase as METASEG. Comparison of
Sampling Rates	1%	5%	10%	20%	50%	80%	100%
#Instances	8	40	81	162	406	650	813
BERT-Base (ours)	85.40	87.83	90.46	91.15	92.80	93.14	93.00
METASEG	91.60	92.29	92.54	92.63	93.45	94.11	93.97
Table 6: F1 scores on WTB test set in low-resource settings.
Models	PKU	MSRA	CITYU	AS	CKIP	NCC	SXU	CTB6	CNC	WTB	UD	ZX	Avg.
BERT-Base	80.15	81.03	90.62	79.60	84.48	79.64	84.75	89.10	61.18	83.57	93.36	87.69	82.93
METASEG	80.90	83.03	90.66	80.89	84.42	84.14	85.98	89.21	61.90	85.00	93.59	87.33	83.92
Table 7: Recalls for OOV words on test sets of all twelve criteria.
F1 scores between Transformer and METASEG is displayed in Table 8.
Results show that METASEG outperforms the non-pretraining Transformer on each criterion and achieves a  $2.40\%$  gain on average, even with the same datasets and architecture. It demonstrates that multi-criteria pre-training is vital for the effectiveness of METASEG and the performance gain is not merely from the large dataset size.
Moreover, METASEG has the generalization ability to transfer prior knowledge to downstream unseen criteria, which could not be achieved by the non-pretraining counterpart Transformer.
4.3.4 Visualization
To visualize the discrepancy between pre-trained models and downstream criteria, we plot similarities of three downstream criteria with METASEG and BERT.
Figure 3: Cosine similarities between three downstream criteria and two pre-trained models. The dashed line indicates the positions where one criterion has equal similarities with two pre-trained models.
Specifically, we extract the criterion token embeddings of three downstream criteria WTB, UD and ZX. We also extract the undefined criterion
token embeddings of METASEG and BERT as representations of these two pre-trained models. We compute cosine similarities between three criteria embeddings and two pre-trained model embeddings, and illustrate them in Figure 3.
We can observe that similarities of all three downstream criteria lie above the dashed line, indicating that all three downstream criteria are more similar to METASEG than BERT. The closer one criterion is to the upper left corner, the more similar it is to METASEG. Therefore, we can conclude that WTB is the most similar criterion to METASEG among all these criteria, which qualitatively corresponds to the phenomenon that WTB criterion has the largest performance gain in Table 4. The above visualization results show that our proposed approach could solidly alleviate the discrepancy between pre-trained models and downstream CWS tasks. Thus METASEG is more similar to downstream criteria.
In this paper, we propose a CWS-specific pretrained model METASEG, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. Experiments show that METASEG could make good use of common prior segmentation knowledge from different existing criteria, and alleviate the discrepancy between pre-trained models and downstream CWS tasks. METASEG also gives better generalization ability in low-resource settings, and achieves new state-of-the-art performance on twelve CWS datasets.
Since the discrepancy between pre-training tasks and downstream tasks also exists in other NLP tasks and other languages, in the future we will explore whether the approach of pre-training with
Models	PKU	MSRA	CITYU	AS	CKIP	NCC	SXU	CTB6	CNC	Avg.
Transformer	95.33	94.79	95.36	95.22	95.17	93.90	95.66	96.45	94.51	95.15
METASEG	96.92	98.50	98.20	97.01	96.72	97.24	97.88	97.89	97.55	97.55
Table 8: Comparison of F1 scores on test sets of nine criteria between non-pretraining baseline Transformer and METASEG.
meta-learning in this paper could be applied to other tasks and languages apart from Chinese word segmentation.
Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuanjing Huang. 2015. Long Short-Term Memory Neural Networks for Chinese Word Segmentation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1197-1206, Lisbon, Portugal. Association for Computational Linguistics.
Xinchi Chen, Zhan Shi, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial Multi-Criteria Learning for Chinese Word Segmentation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1193-1203, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2019. ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations. ArXiv, abs/1911.00720.
Sufeng Duan and Hai Zhao. 2020. Attention Is All You Need for Chinese Word Segmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Thomas Emerson. 2005. The Second International Chinese Word Segmentation Bakeoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135, International Convention Centre, Sydney, Australia. PMLR.
Jingjing Gong, Xinchi Chen, Tao Gui, and Xipeng Qiu. 2019. Switch-LSTMs for Multi-Criteria Chinese Word Segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence.
Han He, Lei Wu, Hua Yan, Zhimin Gao, Yi Feng, and George Townsend. 2019. Effective Neural Solution for Multi-criteria Word Segmentation. In Smart Intelligent Computing and Applications, pages 133-142, Singapore. Springer Singapore.
Weipeng Huang, Xingyi Cheng, Kunlong Chen, Taifeng Wang, and Wei Chu. 2020. Towards Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning. In Proceedings of the 38th International Conference on Computational Linguistics.
Guangjin Jin and Xiao Chen. 2008. The Fourth International Chinese Language Processing Bakeoff: Chinese Word Segmentation, Named Entity Recognition and Chinese POS Tagging. In Proceedings of the Sixth SIGHAN Workshop on Chinese Language Processing.
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2020. SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. FLAT: Chinese NER Using Flat-Lattice Transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6836-6842. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
Shangwen Lv, Yuechen Wang, Daya Guo, Duyu Tang, Nan Duan, Fuqing Zhu, Ming Gong, Linjun Shou, Ryan Ma, Daxin Jiang, Guihong Cao, Ming Zhou, and Songlin Hu. 2020. Pre-training Text Representations as Meta Learning. ArXiv, abs/2004.05568.
Ji Ma, Kuzman Ganchev, and David Weiss. 2018. State-of-the-art Chinese Word Segmentation with Bi-LSTMs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4902-4908, Brussels, Belgium. Association for Computational Linguistics.
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for Chinese Character Representations. In Advances in Neural Information Processing Systems 32, pages 2746-2757. Curran Associates, Inc.
Xipeng Qiu, Hengzhi Pei, Hang Yan, and Xuanjing Huang. 2020. A Concise Model for Multi-Criteria Chinese Word Segmentation with Transformer Encoder. EMNLP Findings.
Yuanhe Tian, Yan Song, Fei Xia, Tong Zhang, and Yonggang Wang. 2020. Improving Chinese Word Segmentation with Wordhood Memory Networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8274-8285, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
William Yang Wang, Lingpeng Kong, Kathryn Mazaitis, and William W. Cohen. 2014. Dependency Parsing for Weibo: An Efficient Probabilistic Logic Programming Approach. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1152-1158, Doha, Qatar. Association for Computational Linguistics.
Qingrong Xia, Zhenghua Li, and Min Zhang. 2019. A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5382-5392, Hong Kong, China. Association for Computational Linguistics.
Mengge Xue, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020. Coarse-to-Fine Pre-training for Named Entity Recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The Penn Chinese TreeBank: Phrase Structure Annotation of A Large Corpus. Natural Language Engineering, page 207-238.
Nianwen Xue. 2003. Chinese Word Segmentation as Character Tagging. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 8, Number 1, February 2003: Special Issue on Word Formation and Chinese Language Processing, pages 29-48.
Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. A Graph-based Model for Joint Chinese Word Segmentation and Dependency Parsing. Transactions of the Association for Computational Linguistics, 8:78-92.
Haiqin Yang. 2019. BERT Meets Chinese Word Segmentation. ArXiv, abs/1909.09292.
Jie Yang, Yue Zhang, and Shuailong Liang. 2019. Subword Encoding in Lattice LSTM for Chinese Word Segmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2720-2725, Minneapolis, Minnesota. Association for Computational Linguistics.
Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacová, Václava Kettnerová, Zdenka Urešová, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Héctor Martínez Alonso, Cagrí Öltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadóva, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonça, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1-19, Vancouver, Canada. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. In Proceedings of the 37th International Conference on Machine Learning.
Meishan Zhang, Yue Zhang, Wanxiang Che, and Ting Liu. 2014. Type-Supervised Domain Adaptation for Joint Segmentation and POS-Tagging. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 588-597, Gothenburg, Sweden. Association for Computational Linguistics.
Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. 2013. Deep Learning for Chinese Word Segmentation and POS Tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 647-657, Seattle, Washington, USA. Association for Computational Linguistics.