SlowGuess/ACL_2020_OCR huggingface dataset:
Settings The task of machine reading comprehension (MRC) (Seo et al., 2016; Wang et al., 2016; Wang and Jiang, 2016; Wang et al., 2016; Shen et al., 2017; Chen et al., 2017) predicts the answer span in the passage given a question and the passage. We followed the standard protocols in Seo et al. (2016), in which the start and end indexes of answer are predicted. We report Extract Match (EM) as well as F1 score on validation set. We use three datasets on this task: SQuAD v1.1, SQuAD v2.0 (Rajpurkar et al., 2016, 2018) and Quoref (Dasigi et al., 2019).
- QANet: Yu et al. (2018b) builds a model based on convolutions and self-attention. Convolutions are used to model local interactions and self-attention are used to model global interactions.
- BERT: Devlin et al. (2018) scores each candidate span and the maximum scoring span is used as a prediction.
- XLNet: Yang et al. (2019) proposes a generalized autoregressive pretraining method that
Model	SQuAD v1.1	SQuAD v2.0	QuoRef
EM	F1	EM	F1	EM	F1
QANet (Yu et al., 2018b)	73.6	82.7	-	-	34.41	38.26
BERT (Devlin et al., 2018)	84.1	90.9	78.7	81.9	58.44	64.95
BERT+FL	84.67(+0.57)	91.25(+0.35)	78.92(+0.22)	82.20(+0.30)	60.78(+2.34)	66.19(+1.24)
BERT+DL	84.83(+0.73)	91.86(+0.96)	78.99(+0.29)	82.88(+0.98)	62.03(+3.59)	66.88(+1.93)
BERT+DSC	85.34(+1.24)	91.97(+1.07)	79.02(+0.32)	82.95(+1.05)	62.44(+4.00)	67.52(+2.57)
XLNet (Yang et al., 2019)	88.95	94.52	86.12	88.79	64.52	71.49
XLNet+FL	88.90(-0.05)	94.55(+0.03)	87.04(+0.92)	89.32(+0.53)	65.19(+0.67)	72.34(+0.85)
XLNet+DL	89.13(+0.18)	95.36(+0.84)	87.22(+1.10)	89.44(+0.65)	65.77(+1.25)	72.85(+1.36)
XLNet+DSC	89.79(+0.84)	95.77(+1.25)	87.65(+1.53)	89.51(+0.72)	65.98(+1.46)	72.90(+1.41)
Table 6: Experimental results for MRC task.
Model	MRPC F1	QQP F1
BERT (Devlin et al., 2018)	88.0	91.3
BERT+FL	88.43 (+0.43)	91.86 (+0.56)
BERT+DL	88.71 (+0.71)	91.92 (+0.62)
BERT+DSC	88.92 (+0.92)	92.11 (+0.81)
XLNet (Yang et al., 2019)	89.2	91.8
XLNet+FL	89.25 (+0.05)	92.31 (+0.51)
XLNet+DL	89.33 (+0.13)	92.39 (+0.59)
XLNet+DSC	89.78 (+0.58)	92.60 (+0.79)
Table 7: Experimental results for PI task.
enables learning bidirectional contexts.
Results Table 6 shows the experimental results for MRC task. With either BERT or XLNet, our proposed DSC loss obtains significant performance boost on both EM and F1. For SQuADv1.1, our proposed method outperforms XLNet by +1.25 in terms of F1 score and +0.84 in terms of EM. For SQuAD v2.0, the proposed method achieves 87.65 on EM and 89.51 on F1. On QuoRef, the proposed method surpasses XLNet by +1.46 on EM and +1.41 on F1.
4.4 Paraphrase Identification
Settings Paraphrase identification (PI) is the task of identifying whether two sentences have the same meaning or not. We conduct experiments on the two widely-used datasets: MRPC (Dolan and Brockett, 2005) and QQP. F1 score is reported for comparison. We use BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019) as baselines.
Results Table 7 shows the results. We find that replacing the training objective with DSC introduces performance boost for both settings, +0.58 for MRPC and +0.73 for QQP.
5.1 Datasets imbalanced to different extents
It is interesting to see how differently the proposed objectives affect datasets imbalanced to different extents. We use the paraphrase identification dataset QQP (37% positive and 63% negative) for studies. To construct datasets with different imbalance degrees, we used the original QQP dataset to construct synthetic training sets with different positive-negative ratios. Models are trained on these different synthetic sets and then test on the same original test set.
- Original training set (original) The original dataset with 363,871 examples, with  $37\%$  being positive and  $63\%$  being negative
- Positive augmentation (+ positive) We created a balanced dataset by adding positive examples. We first randomly chose positive training examples in the original training set as templates. Then we used Spacy to retrieve entity mentions and replace them with new ones by linking mentions to their corresponding entities in DBpedia. The augmented set contains 458,477 examples, with  $50\%$  being positive and  $50\%$  being negative.
- Negative augmentation (+ negative) We created a more imbalanced dataset. The size of the newly constructed training set and
<https://github.com/explosion/spaCy>
original	+ positive	+ negative	- negative	+ positive & negative
BERT	91.3	92.27	90.08	89.73	93.14
BERT+FL	91.86(+0.56)	92.64(+0.37)	90.61(+0.53)	90.79(+1.06)	93.45(+0.31)
BERT+DL	91.92(+0.62)	92.87(+0.60)	90.22(+0.14)	90.49(+0.76)	93.52(+0.38)
BERT+DSC	92.11(+0.81)	92.92(+0.65)	90.78(+0.70)	90.80(+1.07)	93.63(+0.49)
the data augmented technique are exactly the same as  $+$  negative, except that we chose negative training examples as templates. The augmented training set contains 458,477 examples, with  $21\%$  being positive and  $79\%$  being negative.
- Negative downsampling (- negative)
We down-sampled negative examples in the original training set to get a balanced training set. The down-sampled set contains 269,165 examples, with  $50\%$  being positive and  $50\%$  being negative.
- Positive and negative augmentation (+ positive & +negative)
We augmented the original training data with additional positive and negative examples with the data distribution staying the same. The augmented dataset contains 458,477 examples, with  $50\%$  being positive and  $50\%$  being negative.
Results are shown in Table 8. We first look at the first line, with all results obtained using the MLE objective. We can see that  $+$  positive outperforms original, and  $+$  negative underperforms original. This is in line with our expectation since  $+$  positive creates a balanced dataset while  $+$  negative creates a more imbalanced dataset. Despite the fact that  $-$  negative creates a balanced dataset, the number of training data decreases, resulting in inferior performances.
DSC achieves the highest F1 score across all datasets. Specially, for  $^+$  positive, DSC achieves minor improvements  $(+0.05\mathrm{F}1)$  over DL. In contrast, it significantly outperforms DL for  $^+$  negative dataset. This is in line with our expectation since DSC helps more on more imbalanced datasets. The performance of FL and DL are not consistent across different datasets, while DSC consistently performs the best on all datasets.
5.2 Dice loss for accuracy-oriented tasks?
We argue that the cross-entropy objective is actually accuracy-oriented, whereas the proposed losses perform as a soft version of F1 score. To
Table 8: The effect of different data augmentation ways for QQP in terms of F1-score.
SST-2	SST-5
Model	Acc	Acc
BERT+CE	94.90	55.57
BERT+DL	94.37	54.63
BERT+DSC	94.84	55.19
Table 9: The effect of DL and DSC on sentiment classification tasks. BERT+CE refers to fine-tuning BERT and setting cross-entropy as the training objective.
explore the effect of the dice loss on accuracy-oriented tasks such as text classification, we conduct experiments on the Stanford Sentiment Treebank (SST) datasets including SST-2 and SST-5. We fine-tuned  $\mathrm{BERT}_{\mathrm{Large}}$  with different training objectives. Experimental results for SST are shown in Table 9. For SST-5, BERT with CE achieves 55.57 in terms of accuracy, while DL and DSC perform slightly worse (54.63 and 55.19, respectively). Similar phenomenon is observed for SST-2. These results verify that the proposed dice loss is not accuracy-oriented, and should not be used for accuracy-oriented tasks.
5.3 Hyper-parameters in Tversky Index
As mentioned in Section 3.3, Tversky index (TI) offers the flexibility in controlling the tradeoff between false-negatives and false-positives. In this subsection, we explore the effect of hyperparameters (i.e.,  $\alpha$  and  $\beta$ ) in TI to test how they manipulate the tradeoff. We conduct experiments on the Chinese OntoNotes4.0 NER dataset and English QuoRef MRC dataset. Experimental results are shown in Table 10. The highest F1 on Chinese OntoNotes4.0 is 84.67 when  $\alpha$  is set to 0.6 while for QuoRef, the highest F1 is 68.44 when  $\alpha$  is set to 0.4. In addition, we can observe that the performance varies a lot as  $\alpha$  changes in distinct datasets, which shows that the hyperparameters  $\alpha, \beta$  actually play an important role in TI.
In this paper, we propose the dice-based loss to narrow down the gap between training objective and evaluation metrics (F1 score). Experimental results show that the proposed loss function help
α	Chinese Onto4.0	English QuoRef
α = 0.1	80.13	63.23
α = 0.2	81.17	63.45
α = 0.3	84.22	65.88
α = 0.4	84.52	68.44
α = 0.5	84.47	67.52
α = 0.6	84.67	66.35
α = 0.7	81.81	65.09
α = 0.8	80.97	64.13
α = 0.9	80.21	64.84
Table 10: The effect of hyperparameters in Tversky Index. We set  $\beta = 1 - \alpha$  and thus we only list  $\alpha$  here.
to achieve significant performance boost without changing model architectures.
We thank all anonymous reviewers, as well as Qinghong Han, Wei Wu and Jiawei Wu for their comments and suggestions. The work is supported by the National Natural Science Foundation of China (NSFC No. 61625107 and 61751209).
Bernd Bohnet, Ryan T. McDonald, Goncalo Simões, Daniel Andor, Emily Pittler, and Joshua Maynez. 2018. Morphosyntactic tagging with a meta-bilSTM model over context sensitive token encodings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2642-2652.
Haw-Shiuan Chang, Erik G. Learned-Miller, and Andrew McCallum. 2017. Active bias: Training more accurate neural networks by emphasizing high variance samples. In NIPS.
N. V. Chawla, K. W. Bowyer, Lawrence O. Hall, and W. P. Kegelmeyer. 2002. Smote: Synthetic minority over-sampling technique. J. Artif. Intell. Res., 16:321-357.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051.
Kean Chen, Jianguo Li, Weiyao Lin, John See, Ji Wang, Lingyu Duan, Zhibo Chen, Changwei He, and Junni Zou. 2019. Towards accurate one-stage object detection with ap-loss. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5119-5127.
Shijuan Chen, Haibo He, and Edwardo A. Garcia. 2010. Ramoboost: Ranked minority oversampling in boosting. IEEE Transactions on Neural Networks, 21:1624-1642.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018. Semi-supervised sequence
modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1914-1925.
Pradeep Dasigi, Nelson F Liu, Ana Marasovic, Noah A Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. arXiv preprint arXiv:1908.05803.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Lee R Dice. 1945. Measures of the amount of ecologic association between species. *Ecology*, 26(3):297-302.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Yang Fan, Fei Tian, Tao Qin, Xiuping Li, and Tie-Yan Liu. 2018. Learning to teach. ArXiv, abs/1805.03643.
Ross B. Girshick. 2015. Fast r-cnn. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1440-1448.
Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2013. Rich feature hierarchies for accurate object detection and semantic segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 580-587.
Frédéric Godin. 2019. Improving and Interpreting Neural Networks for Word-Level Prediction Tasks in Natural Language Processing. Ph.D. thesis, Ghent University, Belgium.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2017. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML.
H. Kahn and A. W. Marshall. 1953. Methods of reducing sample size in monte carlo computations. Operations Research, 1(5):263-278.
Anil Kanduri, Mohammad Hashem Haghbayan, Amir M. Rahmani, Muhammad Shafique, Axel Jantsch, and Pasi Liljeberg. 2018. adboost: Thermal aware performance boosting through dark silicon patterning. IEEE Trans. Computers, 67(8):1062-1077.
Angelos Katharopoulos and François Fleuret. 2018. Not all samples are created equal: Deep learning with importance sampling. In ICML.
Tomáš Kočisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading
comprehension challenge. Transactions of the Association of Computational Linguistics, 6:317-328.
Oldrich Kodym, Michal Spanel, and Adam Herout. 2018. Segmentation of head and neck organs at risk using CNN with batch dice loss. In *Pattern Recognition - 40th German Conference*, GCPR 2018, Stuttgart, Germany, October 9-12, 2018, Proceedings, pages 105-114.
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada., pages 1189-1197.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108-117, Sydney, Australia. Association for Computational Linguistics.
H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua. 2015. A convolutional neural network cascade for face detection. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5325-5334.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2019. A unified MRC framework for named entity recognition. CoRR, abs/1910.11476.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354.
Tomasz Malisiewicz, Abhinav Gupta, and Alexei A. Efros. 2011. Ensemble of exemplar-svms for object detection and beyond. In IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, pages 89-96.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Yuxian Meng, Muyu Li, Wei Wu, and Jiwei Li. 2019. *Dsreg: Using distant supervision as a regularizer.* arXiv preprint arXiv:1905.11658.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. 2016. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In
2016 Fourth International Conference on 3D Vision (3DV), pages 565-571. IEEE.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. *Lingvisticae Investigationes*, 30(1):3-26.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. 2019. Libra R-CNN: towards balanced learning for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 821-830.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Sameer Pradhan, Mitchell P. Marcus, Martha Palmer, Lance A. Ramshaw, Ralph M. Weischedel, and Nianwen Xue, editors. 2011. Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task. ACL.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:1137-1149.
Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1524-1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceed
ings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 142-147.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
Reuben R. Shamir, Yuval Duchin, Jinyoung Kim, Guillermo Sapiro, and Noam Harel. 2019. Continuous dice coefficient: a method for evaluating probabilistic segmentations. CoRR, abs/1906.11031.
Yan Shao, Christian Hardmeier, Jörg Tiedemann, and Joakim Nivre. 2017. Character-based joint segmentation and pos tagging for chinese using bidirectional rnncrf. arXiv preprint arXiv:1704.01314.
Chen Shen, Holger R. Roth, Hirohisa Oda, Masahiro Oda, Yuichiro Hayashi, Kazunari Misawa, and Kensaku Mori. 2018. On the influence of dice loss function in multi-class organ segmentation of abdominal CT using 3d fully convolutional networks. CoRR, abs/1801.05912.
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1047-1055. ACM.
Th A Sorensen. 1948. A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons. Biol. Skar., 5:1-34.
Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M. Jorge Cardoso. 2017. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support - Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, September 14, 2017, Proceedings, pages 240-248.
Amos Tversky. 1977. Features of similarity. Psychological review, 84(4):327.
Sergi Valverde, Mariano Cabezas, Eloy Roura, Sandra González-Villà, Deborah Pareto, Joan C Vilanova, Lluis Ramíó-Torrentà, Ålex Rovira, Arnau Oliver, and Xavier Llado. 2017. Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach. NeuroImage, 155:159-168.
Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905.
Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context match
ing for machine comprehension. arXiv preprint arXiv:1612.04211.
Wei Wu, Yuxian Meng, Qinghong Han, Muyu Li, Xiaoya Li, Jie Mei, Ping Nie, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. arXiv preprint arXiv:1901.10125.
Naiwen Xue, Fei Xia, Fudong Choiu, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Natural Language Engineering, 11(2):207-238.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018a. Janet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018b. Qanet: Combining local convolution with global self-attention for reading comprehension. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Yue Zhang and Jie Yang. 2018. Chinese ner using lattice lstm. arXiv preprint arXiv:1805.02023.
A.1 Part-of-Speech Tagging
Datasets We conduct experiments on three widely used benchmark, i.e., Chinese Treebank  $5.0^{2} / 6.0^{3}$  and  $\mathrm{UD1.4^4}$ .
- CTB5 is a Chinese dataset for tagging and parsing, which contains 507,222 words, 824,983 characters and 18,782 sentences extracted from newswire sources, including 698 articles from Xinhua (1994-1998), 55 articles from Information Services Department of HK-SAR (1997) and 132 articles from Sinorama Magazine (1996-1998 & 2000-2001).
- CTB6 is an extension of CTB5, containing 781,351 words, 1,285,149 characters and 28,295 sentences.
- UD is the abbreviation of Universal Dependencies, which is a framework for consistent
$^{2}$ https://catalog.ldc.upenn.edu/LDC2005T01  
 $^{3}$ https://catalog.ldc.upenn.edu/LDC2007T36  
 $^{4}$ https://universaldependencies.org/
annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages. In this work, we use UD1.4 for Chinese POS tagging.
A.2 Named Entity Recognition
Datasets For the NER task, we consider both Chinese datasets, i.e., OntoNotes4.0 $^5$  and MSRA $^6$ , and English datasets, i.e., CoNLL2003 $^7$  and OntoNotes5.0 $^8$ .
- CoNLL2003 is an English dataset with 4 entity types: Location, Organization, Person and Miscellaneous. We followed data processing protocols in (Ma and Hovy, 2016).
- English OntoNotes5.0 consists of texts from a wide variety of sources and contains 18 entity types. We use the standard train/dev/test split of CoNLL2012 shared task.
- Chinese MSRA performs as a Chinese benchmark dataset containing 3 entity types. Data in MSRA is collected from news domain. Since the development set is not provided in the original MSRA dataset, we randomly split the training set into training and development splits by 9:1. We use the official test set for evaluation.
- Chinese OntoNotes4.0 is a Chinese dataset and consists of texts from news domain, which has 18 entity types. In this paper, we take the same data split as Wu et al. (2019) did.
A.3 Machine Reading Comprehension
Datasets For MRC task, we use three datasets: SQuADv1.1/v2.0 and Queref<sup>10</sup> datasets.