SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 1: Results on the development set of the GLUE benchmark for base-size pre-trained models. Models using 16GB corpora are all pre-trained with a batch size of 256 sequences for 1M steps. STS-B and CoLA are reported by Pearson correlation coefficient (PCC) and Matthews correlation coefficient (MCC), other tasks are reported by accuracy (Acc). Note that results of ERNIE-Gram are the median of over ten runs with different random seeds.
Models	SQuAD1.1	SQuAD2.0
EM	F1	EM	F1
Models pre-trained on base-scale text corpora (16GB)
BERT (Devlin et al., 2019)	80.8	88.5	73.7	76.3
RoBERTa (Liu et al., 2019)	-	90.6	-	79.7
XLNet (Yang et al., 2019)	-	-	78.2	81.0
MPNet (Song et al., 2020)	85.0	91.4	80.5	83.3
UNILMv2 (Bao et al., 2020)	85.6	92.0	80.9	83.6
ERNIE-Gram	86.2	92.3	82.1	84.8
Models pre-trained on large-scale text corpora (160GB)
RoBERTa (Liu et al., 2019)	84.6	91.5	80.5	83.7
XLNet (Yang et al., 2019)	-	-	80.2	-
ELECTRA (Clark et al., 2020)	86.8	-	80.5	-
MPNet (Song et al., 2020)	86.8	92.5	82.8	85.6
UNILMv2 (Bao et al., 2020)	87.1	93.1	83.3	86.1
ERNIE-Gram	87.2	93.2	84.1	87.1
Table 2 presents the results on SQuAD for base-size pre-trained models, ERNIE-Gram achieves better performance than current strong baselines on both base-scale and large-scale pre-training text corpora.
4.5 Results on RACE and Text Classification Tasks
The ReAding Comprehension from Examinations (RACE; Lai et al., 2017) dataset collects 88K long passages from English exams at middle and high schools, the task is to select the correct choice from four given options according to the questions and
Table 2: Performance comparison between base-size pre-trained models on the SQuAD development sets. Exact-Match (EM) and F1 score are adopted for evaluations. Results of ERNIE-Gram are the median of over ten runs with different random seeds.
Models	RACE	IMDb	AG
Total	High	Middle	Err.	Err.
Pre-trained on base-scale text corpora (16GB)
BERTa	65.0	62.3	71.7	5.4	5.9
XLNetb	66.8	-	-	4.9	-
MPNetc	70.4	67.7	76.8	4.8	-
F-TFMdELECTRA	-	-	-	5.2	5.4
ERNIE-Gram	72.7	68.1	75.1	4.6	5.0
Pre-trained on large-scale text corpora (160GB)
MPNetc	72.0	70.3	76.3	4.4	-
ERNIE-Gram	77.7	75.6	78.8	3.9	4.9
Table 3: Comparison on the test sets of RACE, IMDb and AG. The listed models are all in base-size. In the results of RACE, "High" and "Middle" represent the training and evaluation sets for high schools and middle schools respectively, "Total" is the full training and evaluation set.  ${}^{a}$  (Devlin et al.,2019);  ${}^{b}$  (Yang et al., 2019);  ${}^{c}$  (Song et al.,2020);  ${}^{d}$  (Dai et al.,2020).
passages. We also evaluate ERNIE-Gram on two large scaled text classification tasks that involve long text and reasoning, including sentiment analysis datasets IMDb (Maas et al., 2011) and topic classification dataset AG's News (Zhang et al., 2015). The results are reported in Table 3. It can be seen that ERNIE-Gram consistently outperforms previous models, showing the advantage of ERNIE-Gram on tasks involving long text and reasoning.
4.6 Results on Chinese NLU Tasks
We execute extensive experiments on six Chinese language understanding tasks, including natural language inference (XNLI; Conneau et al., 2018),
Models	XNLI Acc Dev Test	LCQMC Acc Dev Test	DRCD EM / F1 Dev	TMRC2018 EM / F1 Dev	TMRC2018 Em / F1 Dev	TMRC2018 Em / F1 Dev	TMRC2018 Em / F1 Dev	TMRC2018 Em / F1 Dev	TMRC2018 Em / F1 Dev	TMRC2018 Em / F1 Dev
RoBERTa-wn-ext* LARGE	82.1	81.2	90.4	87.0	89.6 / 94.8	89.6 / 94.5	68.5 / 88.4	- / -	-	-
NEZHALARGE (Wei et al., 2019)	82.2	81.2	90.9	87.9	- / -	- / -	- / -	- / -	-	-
MacBERTLARGE (Cui et al., 2020)	82.4	81.3	90.6	87.6	91.2 / 95.6	91.7 / 95.6	70.7 / 88.9	- / -	-	-
BERT-wn-ext* BASE	79.4	78.7	89.6	87.1	85.0 / 91.2	83.6 / 90.4	67.1 / 85.7	- / -	-	-
RoBERTa-wn-ext* BASE	80.0	78.8	89.0	86.4	86.6 / 92.5	85.6 / 92.0	67.4 / 87.2	- / -	-	-
ZENBASE (Diao et al., 2020)	80.5	79.2	90.2	88.0	- / -	- / -	- / -	- / -	-	-
NEZHABASE (Wei et al., 2019)	81.4	79.3	90.0	87.4	- / -	- / -	- / -	- / -	-	-
MacBERTBASE (Cui et al., 2020)	80.3	79.3	89.5	87.0	89.4 / 94.3	89.5 / 93.8	68.5 / 87.9	- / -	-	-
ERNIE1.0BASE (Sun et al., 2019b)	79.9	78.4	89.7	87.4	84.6 / 90.9	84.0 / 90.5	65.1 / 85.1	57.9 / 72.1	95.0	93.8
ERNIE2.0BASE (Sun et al., 2020)	81.2	79.7	90.9	87.9	88.5 / 93.8	88.0 / 93.4	69.1 / 88.6	61.3 / 74.9	95.2	93.8
ERNIE-GramBASE	81.8	81.5	90.6	88.5	90.2 / 95.0	89.9 / 94.6	74.3 / 90.5	64.2 / 76.8	96.5	95.3
Table 4: Results on six Chinese NLU tasks for base-size pre-trained models. Results of models with asterisks “*” are from Cui et al., 2019. M-NER is in short for MSRA-NER dataset. “BASE” and “LARGE” donate different sizes of pre-training models. Large size models have  $L = 24$ ,  $H = 1024$ ,  $A = 16$  and total Parameters=340M.
machine reading comprehension (CMRC2018; Cui et al., 2018, DRCD; Shao et al., 2018 and DuR-eker; He et al., 2018), named entity recognition (MSRA-NER; Gao et al., 2005) and semantic similarity (LCQMC; Liu et al., 2018).
Results on six Chinese tasks are presented in Table 4. It is observed that ERNIE-Gram significantly outperforms previous models across tasks by a large margin and achieves new state-of-the-art results on these Chinese NLU tasks in base-size model group. Besides, ERNIE-Gram<sub>BASE</sub> are also better than various large-size models on XNLI, LCQMC and CMRC2018 datasets.
We further conduct ablation experiments to analyze the major components of ERNIE-Gram.
Effect of Explicitly N-gram MLM. We compare two models pre-trained with contiguously MLM and explicitly  $n$ -gram MLM objectives in the same settings (the size of  $n$ -gram lexicon is 300K). The evaluation results for pre-training and fine-tuning are shown in Figure 4. Compared with contiguously MLM, explicitly  $n$ -gram MLM objective facilitates the learning of  $n$ -gram semantic information with lower  $n$ -gram level perplexity in pre-training and better performance on downstream tasks. This verifies the effectiveness of explicitly  $n$ -gram MLM objective for injecting  $n$ -gram semantic information into pre-training.
Size of N-gram Lexicon. To study the impact of  $n$ -gram lexicon size on model performance, we extract  $n$ -gram lexicons with size from 100K to 400K for pre-training, as shown in Figure 5. As the
Figure 4: (a) N-gram level perplexity which is calculated by  $\left(\prod_{i=1}^{k}\mathsf{PPL}(\boldsymbol{w}_i)\right)^{\frac{1}{k}}$  for contiguously MLM, where  $\boldsymbol{w}_i$  is the  $i$ -th masked  $n$ -gram. (b) Performance distribution box plot on MNLI, QNLI, SST-2 and SQuAD1.1.
lexicon size enlarges, performance of contiguously MLM becomes worse, presumably because more  $n$ -grams are matched and connected as longer consecutive spans for prediction, which is more difficult for representation learning. Explicitly  $n$ -gram MLM with lexicon size being 300K achieves the best results, while the performance significantly declines when the size of lexicon increasing to 400K because more low-frequent  $n$ -grams are learning unnecessarily. See Appendix C for detailed results of different lexicon choices on GLUE and SQuAD.
Effect of Comprehensive N-gram Prediction and Enhanced N-gram Relation Modeling. As shown in Table 5, we compare several ERNIE-Gram variants with previous strong baselines under the BERT<sub>BASE</sub> setting. After removing comprehensive  $n$ -gram prediction (#2), ERNIE-Gram degenerates to a variant with explicitly  $n$ -gram MLM and  $n$ -gram relation modeling and its performance drops slightly by 0.3-0.6. When removing enhanced  $n$ -gram relation modeling (#3), ERNIE-Gram degenerates to a variant with comprehension
#	Models	MNLI	SST-2	SQuAD1.1	SQuAD2.0
m	mm	Acc	EM	F1	EM	F1
XLNeta	85.6	85.1	93.4	-	-	78.2	81.0
RoBERTa b	84.7	-	92.7	-	90.6	-	79.7
MPNetc	85.6	-	93.6	84.0	90.3	79.5	82.2
UNILMv2d	85.6	85.5	93.0	85.0	91.5	78.9	81.8
#1	ERNIE-Gram	86.5	86.4	93.2	85.2	91.7	80.8	84.0
#2	- CNP	86.2	86.2	92.7	85.0	91.5	80.4	83.4
#3	- ENRM	85.7	85.8	93.5	84.7	91.3	79.7	82.7
#4	- CNP - ENRM	85.6	85.7	92.9	84.5	91.2	79.5	82.4
Table 5: Comparisons between comprehensive  $n$ -gram prediction (CNP) and enhanced  $n$ -gram relation modeling (ENRM) methods. All the listed models are pre-trained following the same settings of BERT<sub>BASE</sub> (Devlin et al., 2019) and without relative position bias. Results of ERNIE-Gram variants are the median of over ten runs with different random seeds. Results in the upper block are from <sup>a</sup> (Yang et al., 2019), <sup>b</sup> (Liu et al., 2019), <sup>c</sup> (Song et al., 2020) and <sup>d</sup> (Bao et al., 2020).
(b)
Figure 5: Quantitative study on the size of extracted  $n$ -gram lexicon. (a) Comparisons on GLUE and SQuAD. Note that SQuAD is presented by the average scores of SQuAD1.1 and SQuAD2.0. (b) Performance distribution box plot on MNLI and SQuAD1.1 datasets.
Figure 6: (a) Recall rate of whole named entities on different evaluation subsets, which have incremental average length of named entities. (b-d) Mean attention scores of 12 attention heads in the last self-attention layer. Texts in green and orange boxes are named entities standing for organizations and locations.
sive  $n$ -gram MLM and the performance drops by 0.4-1.3. If removing both comprehensive  $n$ -gram prediction and relation modeling (#4), ERNIE-Gram degenerates to a variant with explicitly  $n$ -gram MLM and the performance drops by 0.7-1.6. These results demonstrate the advantage of comprehensive  $n$ -gram prediction and  $n$ -gram relation modeling methods for efficiently  $n$ -gram semantic injecting into pre-training. The detailed results of ablation study are supplemented in Appendix C.
4.8 Case Studies
To further understand the effectiveness of our approach for learning  $n$ -grams information, we fine-tune ERNIE-Gram, contiguously MLM and lower
cased BERT on CoNLL-2003 named entity recognition task (Sang and De Meulder, 2003) for comparison. We divide the evaluation set into five subsets based on the average length of the named entities in each sentence. As shown in Figure 6(a), it is more difficult to recognize whole named entities as the length of them increases, while the performance of ERNIE-Gram declines slower than contiguously MLM and BERT, which implies that ERNIE-Gram models tighter intra-dependencies of  $n$ -grams.
As shown in Figure 6(b-d), we visualize the attention patterns in the last self-attention layer of fine-tuned models. For contiguously MLM, there are clear diagonal lines in named entities that tokens prefer to attend to themselves in named entities. While for ERNIE-Gram, there are bright blocks over named entities that tokens attend to most of tokens in the same entity adequately to construct tight representation, verifying the effectiveness of ERNIE-Gram for  $n$ -gram semantic modeling.
In this paper, we present ERNIE-Gram, an explicitly  $n$ -gram masking and predicting method to eliminate the limitations of previous contiguously masking strategies and incorporate coarse-grained linguistic information into pre-training sufficiently. ERNIE-Gram conducts comprehensive  $n$ -gram prediction and relation modeling to further enhance the learning of semantic  $n$ -grams for pre-training. Experimental results on various NLU tasks demonstrate that ERNIE-Gram outperforms XLNet and RoBERTa by a large margin, and achieves state-of-the-art results on various benchmarks. Future work includes constructing more comprehensive  $n$ -gram
lexicon  $(n > 3)$  and pre-training ERNIE-Gram with large-size model for more downstream tasks.
We would like to thank Zhen Li for his constructive suggestions, and hope everything goes well with his work. We are also indebted to the NAACL-HLT reviewers for their detailed and insightful comments on our work.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, et al. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In Proceedings of the International Conference on Machine Learning, pages 7006-7016.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pretraining text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pretrained models for Chinese natural language processing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 657-668, Online. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101.
Yiming Cui, Ting Liu, Li Xiao, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. 2018. A span-extraction dataset for chinese machine reading comprehension. CoRR, abs/1810.07366.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment
challenge. In Proceedings of the First International Conference on Machine Learning Challenges, pages 177-190.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V Le. 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. In Advances in Neural Information Processing Systems.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2020. ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4729–4740, Online. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, volume 32, pages 13063-13075. Curran Associates, Inc.
Jianfeng Gao, Mu Li, Andi Wu, and Chang-Ning Huang. 2005. Chinese word segmentation and named entity recognition: A pragmatic approach. Computational Linguistics, 31(4):531-574.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 37-46, Melbourne, Australia. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Guolin Ke, Di He, and Tie-Yan Liu. 2020. Rethinking the positional encoding in language pre-training. arXiv preprint arXiv:2006.15595.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. 2019. Is word segmentation necessary for deep learning of Chinese representations? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3242-3252, Florence, Italy. Association for Computational Linguistics.
Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018. LCQMC:a large-scale Chinese question matching corpus. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1952-1962, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Erik F Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2018. DRCD: a chinese machine reading comprehension dataset. arXiv preprint arXiv:1806.00920.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. In Advances in Neural Information Processing Systems.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019a. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics, pages 194-206. Springer.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019b. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8968-8975.
Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. ArXiv, abs/1806.02847.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. NEZHA: Neural contextualized representation for chinese language understanding. arXiv preprint arXiv:1909.00204.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. *Erniegen: An enhanced multi-flow pre-training and finetuning framework for natural language generation.* In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, *IJCAI* 20, pages 3997-4003. International Joint Conferences on Artificial Intelligence Organization. Main track.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32, pages 5753-5763. Curran Associates, Inc.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28, pages 649-657. Curran Associates, Inc.
Xinsong Zhang and Hang Li. 2020. Ambert: A pretrained language model with multi-grained tokenization. arXiv preprint arXiv:2008.11869.
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 19-27.
A Hyperparameters for Pre-Training
As shown in Table 6, we list the detailed hyperparameters used for pre-training ERNIE-Gram on base and large scaled English text corpora and Chinese text corpora. We follow the same hyperparameters of BERT<sub>BASE</sub> (Devlin et al., 2019) to pre-train ERNIE-Gram on the base-scale English text corpora (16GB). We pre-train ERNIE-Gram on the large-scale text corpora (160GB) with the settings in RoBERTa (Liu et al., 2019) except the batch size being 5112 sequences.