SlowGuess/NAACL_2021_OCR huggingface dataset:
Dongling Xiao, Yu-Kun Li, Han Zhang, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang
Baidu Inc., China
{xiaodongling,liyukun01,zhanghan17,sunyu02,
tianhao,wu_hua, wanghaifeng}@baidu.com
Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT's Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of  $n$  tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly  $n$ -gram masking method to enhance the integration of coarse-grained information into pre-training. In ERNIE-Gram,  $n$ -grams are masked and predicted directly using explicit  $n$ -gram identities rather than contiguous sequences of  $n$  tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible  $n$ -gram identities as optional  $n$ -gram masks and predict them in both coarse-grained and fine-grained manners to enable comprehensive  $n$ -gram prediction and relation modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and finetune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art methods. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.
Pre-trained on large-scaled text corpora and finetuned on downstream tasks, self-supervised representation models (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2020; Clark et al., 2020) have achieved remarkable improvements in natural language understanding (NLU). As one of the most prominent
pre-trained models, BERT (Devlin et al., 2019) employs masked language modeling (MLM) to learn representations by masking individual tokens and predicting them based on their bidirectional context. However, BERT's MLM focuses on the representations of fine-grained text units (e.g. words or subwords in English and characters in Chinese), rarely considering the coarse-grained linguistic information (e.g. named entities or phrases in English and words in Chinese) thus incurring inadequate representation learning.
Many efforts have been devoted to integrate coarse-grained semantic information by independently masking and predicting contiguous sequences of  $n$  tokens, namely  $n$ -grams, such as named entities, phrases (Sun et al., 2019b), whole words (Cui et al., 2019) and text spans (Joshi et al., 2020). We argue that such contiguously masking strategies are less effective and reliable since the prediction of tokens in masked  $n$ -grams are independent of each other, which neglects the intradependencies of  $n$ -grams. Specifically, given a masked  $n$ -gram  $\boldsymbol{w} = \{x_1, \dots, x_n\}$ ,  $x \in \mathcal{V}_F$ , we maximize  $p(\boldsymbol{w}) = \prod_{i=1}^{n} p(x_i | c)$  for  $n$ -gram learning, where models learn to recover  $\boldsymbol{w}$  in a huge and sparse prediction space  $\mathcal{F} \in \mathbb{R}^{|V_F|^n}$ . Note that  $\mathcal{V}_F$  is the fine-grained vocabulary<sup>1</sup> and  $c$  is the context.
We propose ERNIE-Gram, an explicitly  $n$ -gram masked language modeling method in which  $n$ -grams are masked with single [MASK] symbols, and predicted directly using explicit  $n$ -gram identities rather than sequences of tokens, as depicted in Figure 1(b). The models learn to predict  $n$ -gram  $\pmb{w}$  in a small and dense prediction space  $\mathcal{N} \in \mathbb{R}^{|\mathcal{V}_N|}$ , where  $\mathcal{V}_N$  indicates a prior  $n$ -gram lexicon² and normally  $|\mathcal{V}_N| \ll |\mathcal{V}_F|^n$ . To
$^{1}\mathcal{V}_{F}$  contains 30K BPE codes in BERT (Devlin et al., 2019) and 50K subword units in RoBERTa (Liu et al., 2019).
$^{2}\mathcal{V}_{N}$  contains  $300\mathrm{K}n$  -grams, where  $n\in [2,4)$  in this paper,  $n$  -grams are extracted in word-level before tokenization.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1702-1715 June 6-11, 2021. ©2021 Association for Computational Linguistics
(a) Contiguously MLM
(b) Explicitly N-gram MLM
(c) Comprehensive N-gram MLM
Figure 1: Illustrations of different MLM objectives, where  $x_{i}$  and  $y_{i}$  represent the identities of fine-grained tokens and explicit  $n$ -grams respectively. Note that the weights of fine-grained classifier  $(W_{F} \in \mathbb{R}^{h \times |\mathcal{V}_{F}|})$  and N-gram classifier  $(W_{N} \in \mathbb{R}^{h \times |\langle \mathcal{V}_{F}, \mathcal{V}_{N} \rangle|})$  are not used in fine-tuning stage, where  $h$  is the hidden size and  $L$  is the layers.
learn the semantic of  $n$ -grams more adequately, we adopt a comprehensive  $n$ -gram prediction mechanism, simultaneously predicting masked  $n$ -grams in coarse-grained (explicit  $n$ -gram identities) and fine-grained (contained token identities) manners with well-designed attention mask metrics, as shown in Figure 1(c).
In addition, to model the semantic relationships between  $n$ -grams directly, we introduce an enhanced  $n$ -gram relation modeling mechanism, masking  $n$ -grams with plausible  $n$ -grams identities sampled from a generator model, and then recovering them to the original  $n$ -grams with the pair relation between plausible and original  $n$ -grams. Inspired by ELECTRA (Clark et al., 2020), we incorporate the replaced token detection objective to distinguish original  $n$ -grams from plausible ones, which enhances the interactions between explicit  $n$ -grams and fine-grained contextual tokens.
In this paper, we pre-train ERNIE-Gram on both base-scale and large-scale text corpora (16GB and 160GB respectively) under comparable pre-training setting. Then we fine-tune ERNIE-Gram on 13 English NLU tasks and 6 Chinese NLU tasks. Experimental results show that ERNIE-Gram consistently outperforms previous well-performed pre-training models on various benchmarks by a large margin.
2.1 Self-Supervised Pre-Training for NLU
Self-supervised pre-training has been used to learn contextualized sentence representations though various training objectives. GPT (Radford et al., 2018) employs unidirectional language modeling (LM) to exploit large-scale corpora. BERT (Devlin et al., 2019) proposes masked language modeling (MLM) to learn bidirectional representations efficiently, which is a representative objective for pre-training and has numerous extensions such as RoBERTa (Liu et al., 2019), UNILM (Dong et al., 2019) and ALBERT (Lan et al., 2020). XL-
Net (Yang et al., 2019) adopts permutation language modeling (PLM) to model the dependencies among predicted tokens. ELECTRA introduces replaced token detection (RTD) objective to learn all tokens for more compute-efficient pre-training.
2.2 Coarse-grained Linguistic Information Incorporating for Pre-Training
Coarse-grained linguistic information is indispensable for adequate representation learning. There are lots of studies that implicitly integrate coarse-grained information by extending BERT's MLM to contiguously masking and predicting contiguous sequences of tokens. For example, ERNIE (Sun et al., 2019b) masks named entities and phrases to enhance contextual representations, BERT-wwm (Cui et al., 2019) masks whole Chinese words to achieve better Chinese representations, SpanBERT (Joshi et al., 2020) masks contiguous spans to improve the performance on span selection tasks.
A few studies attempt to inject the coarse-grained  $n$ -gram representations into fine-grained contextualized representations explicitly, such as ZEN (Diao et al., 2020) and AMBERT (Zhang and Li, 2020), in which additional transformer encoders and computations for explicit  $n$ -gram representations are incorporated into both pre-training and fine-tuning. Li et al., 2019 demonstrate that explicit  $n$ -gram representations are not sufficiently reliable for NLP tasks because of  $n$ -gram data sparsity and the ubiquity of out-of-vocabulary  $n$ -grams. Differently, we only incorporate  $n$ -gram information by leveraging auxiliary  $n$ -gram classifier and embedding weights in pre-training, which will be completely removed during fine-tuning, so our method maintains the same parameters and computations as BERT.
In this section, we present the detailed implementation of ERNIE-Gram, including  $n$ -gram lexicon
$\mathcal{V}_N$  extraction in Section 3.5, explicitly  $n$ -gram MLM pre-training objective in Section 3.2, comprehensive  $n$ -gram prediction and relation modeling mechanisms in Section 3.3 and 3.4.
To inject  $n$ -gram information into pre-training, many works (Sun et al., 2019b; Cui et al., 2019; Joshi et al., 2020) extend BERT's masked language modeling (MLM) from masking individual tokens to contiguous sequences of  $n$  tokens.
Contiguously MLM. Given input sequence  $\pmb{x} = \{x_{1},\dots,x_{|\pmb{x}|}\}$ ,  $x \in \mathcal{V}_F$  and  $n$ -gram starting boundaries  $\pmb{b} = \{b_1,\dots,b_{|\pmb{b}|}\}$ , let  $z = \{z_{1},\dots,z_{|\pmb{b}| - 1}\}$  to be the sequence of  $n$ -grams, where  $z_{i} = \pmb{x}_{[b_{i}:b_{i + 1})}$ , MLM samples  $15\%$  of starting boundaries from  $\pmb{b}$  to mask  $n$ -grams, donating  $\mathcal{M}$  as the indexes of sampled starting boundaries,  $\pmb{z}_{\mathcal{M}}$  as the contiguously masked tokens,  $\pmb{z}_{\backslash \mathcal{M}}$  as the sequence after masking. As shown in Figure 1(a),  $\pmb{b} = \{1,2,4,5,6,7\}$ ,  $\pmb{z} = \{x_{1},\pmb{x}_{[2:4)},x_{4},x_{5},x_{6}\}$ ,  $\mathcal{M} = \{2,4\}$ ,  $\pmb{z}_{\mathcal{M}} = \{\pmb{x}_{[2:4)},x_{5}\}$ , and  $\pmb{z}_{\backslash \mathcal{M}} = \{x_{1},[\mathbb{M}], [\mathbb{M}],x_{4},[\mathbb{M}],x_{6}\}$ . Contiguously MLM is performed by minimizing the negative likelihood:
$$
- \log p _ {\theta} \left(\boldsymbol {z} _ {\mathcal {M}} \mid \boldsymbol {z} _ {\backslash \mathcal {M}}\right) = - \sum_ {z \in \boldsymbol {z} _ {\mathcal {M}}} \sum_ {x \in z} \log p _ {\theta} (x \mid \boldsymbol {z} _ {\backslash \mathcal {M}}). \tag {1}
$$
3.2 Explicitly N-gram Masked Language Modeling
Different from contiguously MLM, we employ explicit  $n$ -gram identities as pre-training targets to reduce the prediction space for  $n$ -grams. To be specific, let  $\pmb{y} = \{y_{1},\dots,y_{|\pmb{b}| - 1}\}$ ,  $y\in \langle \mathcal{V}_F,\mathcal{V}_N\rangle$  to be the sequence of explicit  $n$ -gram identities,  $\pmb{y}_{\mathcal{M}}$  to be the target  $n$ -gram identities, and  $\bar{z}_{\backslash \mathcal{M}}$  to be the sequence after explicitly masking  $n$ -grams. As shown in Figure 1(b),  $\pmb{y}_{\mathcal{M}} = \{y_2,y_4\}$ , and  $\bar{z}_{\backslash \mathcal{M}} = \{x_1,[\mathrm{M}],x_4,[\mathrm{M}],x_6\}$ . For masked  $n$ -gram  $\pmb{x}_{[2:4)}$ , the prediction space is significantly reduced from  $\mathbb{R}^{|\mathcal{V}_F|^2}$  to  $\mathbb{R}^{|\langle \mathcal{V}_F,\mathcal{V}_N\rangle |}$ . Explicitly  $n$ -gram MLM is performed by minimizing the negative likelihood:
$$
- \log p _ {\theta} (\boldsymbol {y} _ {\mathcal {M}} | \bar {z} _ {\backslash \mathcal {M}}) = - \sum_ {y \in \boldsymbol {y} _ {\mathcal {M}}} \log p _ {\theta} (y | \bar {z} _ {\backslash \mathcal {M}}). \tag {2}
$$
3.3 Comprehensive N-gram Prediction
We propose to simultaneously predict  $n$ -grams in fine-grained and coarse-grained manners corresponding to single mask symbol [M], which helps to extract comprehensive  $n$ -gram semantics,
Figure 2: (a) Detailed structure of Comprehensive N-gram MLM. (b) Self-attention mask  $M$  without leaking length information of masked  $n$ -grams.
as shown in Figure 1(c). Comprehensive  $n$ -gram MLM is performed by minimizing the joint negative likelihood:
$$
\begin{array}{l} - \log p _ {\theta} (\boldsymbol {y} _ {\mathcal {M}}, \boldsymbol {z} _ {\mathcal {M}} | \bar {\boldsymbol {z}} _ {\backslash \mathcal {M}}) = \\ - \sum_ {y \in \mathcal {Y} _ {\mathcal {M}}} \log p _ {\theta} (y | \bar {z} _ {\backslash \mathcal {M}}) - \sum_ {z \in z _ {\mathcal {M}}} \sum_ {x \in z} \log p _ {\theta} (x | \bar {z} _ {\backslash \mathcal {M}}). \tag {3} \\ \end{array}
$$
where the predictions of explicit  $n$ -gram  $\mathbf{y}_{\mathcal{M}}$  and fine-grained tokens  $\mathbf{x}_{\mathcal{M}}$  are conditioned on the same context sequence  $\bar{\mathbf{z}}_{\backslash \mathcal{M}}$ .
In detail, to predict all tokens contained in a  $n$ -gram from single [M] other than a consecutive sequence of [M], we adopt distinctive mask symbols  $[\mathrm{Mi}]$ ,  $\mathrm{i} = 1,\dots,n$  to aggregate contextualized representations for predicting the i-th token in  $n$ -gram. As shown in Figure 2(a), along with the same position as  $y_{2}$ , symbols [M1] and [M2] are used as queries  $(Q)$  to aggregate representations from  $\bar{z}_{\backslash M}(K)$  for the predictions of  $x_{2}$  and  $x_{3}$ , where  $Q$  and  $K$  donate the query and key in self-attention operation (Vaswani et al., 2017). As shown in Figure 2(b), the self-attention mask metric  $M$  controls what context a token can attend to by modifying the attention weight  $W_{A} = \text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}} + M)$ ,  $M$  is assigned as:
$$
M _ {i j} = \left\{ \begin{array}{l l} 0, & \text {a l l o w t o a t t e n d} \\ - \infty , & \text {p r e v e n t f r o m a t t e n d i n g} \end{array} \right. \tag {4}
$$
We argue that the length information of  $n$ -grams is detrimental to the representations learning, because it will arbitrarily prune a number of semanti-
Figure 3: (a) Detailed architecture of  $n$ -gram relation modeling, where  $L'$  donates the layers of the generator model. (b) An example of plausible  $n$ -gram sampling, where dotted boxes represent the sampling module, texts in green are the original  $n$ -grams, and the italic texts in blue donate the sampled  $n$ -grams.
cally related  $n$ -grams with different lengths during predicting. From this viewpoint, for the predictions of  $n$ -gram  $\{x_2,x_3\}$ , 1) we prevent context  $\bar{z}_{\backslash M}$  from attending to  $\{[\mathsf{M}1],[\mathsf{M}2]\}$  and 2) prevent  $\{[\mathsf{M}1],[\mathsf{M}2]\}$  from attending to each other, so that the length information of  $n$ -grams will not be leaked in pre-training, as displayed in Figure 2(b).
3.4 Enhanced N-gram Relation Modeling
To explicitly learn the semantic relationships between  $n$ -grams, we jointly pre-train a small generator model  $\theta'$  with explicitly  $n$ -gram MLM objective to sample plausible  $n$ -gram identities. Then we employ the generated identities to preform masking and train the standard model  $\theta$  to predict the original  $n$ -grams from fake ones in coarse-grained and fine-grained manners, as shown in Figure 3(a), which is efficient to model the pair relationships between similar  $n$ -grams. The generator model  $\theta'$  will not be used during fine-tuning, where the hidden size  $H_{\theta'}$  of  $\theta'$  has  $H_{\theta'} = H_{\theta} / 3$  empirically.
As shown in Figure 3(b),  $n$ -grams of different length can be sampled to mask original  $n$ -grams according to the prediction distributions of  $\theta'$ , which is more flexible and sufficient for constructing  $n$ -gram pairs than previous synonym masking methods (Cui et al., 2020) that require synonyms and original words to be of the same length. Note that our method needs a large embedding layer  $E \in \mathbb{R}^{|\langle \mathcal{V}_F, \mathcal{V}_N \rangle| \times h}$  to obtain  $n$ -gram vectors in pretraining. To keep the number of parameters consis
tent with that of vanilla BERT, we remove the auxiliary embedding weights of  $n$ -grams during finetuning  $(E \to E' \in \mathbb{R}^{|\mathcal{V}_F| \times h})$ . Specifically, let  $\boldsymbol{y}_{\mathcal{M}}'$  to be the generated  $n$ -gram identities,  $\bar{z}_{\mathcal{M}}'$  to be the sequence masked by  $\boldsymbol{y}_{\mathcal{M}}'$ , where  $\boldsymbol{y}_{\mathcal{M}}' = \{y_2', y_4'\}$ , and  $\bar{z}_{\mathcal{M}}' = \{x_1, y_2', x_4, y_4', x_6\}$  in Figure 3(a). The pre-training objective is to jointly minimize the negative likelihood of  $\theta'$  and  $\theta$ :
$$
- \log p _ {\theta^ {\prime}} \left(\boldsymbol {y} _ {\mathcal {M}} \mid \bar {\boldsymbol {z}} _ {\backslash \mathcal {M}}\right) - \log p _ {\theta} \left(\boldsymbol {y} _ {\mathcal {M}}, \boldsymbol {z} _ {\mathcal {M}} \mid \bar {\boldsymbol {z}} _ {\backslash \mathcal {M}} ^ {\prime}\right). \tag {5}
$$
Moreover, we incorporate the replaced token detection objective (RTD) to further distinguish fake  $n$ -grams from the mix-grained context  $\bar{z}_{\backslash \mathcal{M}}'$  for interactions among explicit  $n$ -grams and fine-grained contextual tokens, as shown in the right part of Figure 3(a). Formally, we donate  $\hat{z}_{\backslash \mathcal{M}}$  to be the sequence after replacing masked  $n$ -grams with target  $n$ -gram identities  $y_{\mathcal{M}}$ , the RTD objective is performed by minimizing the negative likelihood:
$$
\begin{array}{l} - \log p _ {\theta} \left(\mathbb {1} \left(\bar {z} _ {\backslash \mathcal {M}} ^ {\prime} = \hat {z} _ {\backslash \mathcal {M}}\right) | \bar {z} _ {\backslash \mathcal {M}} ^ {\prime}\right) \\ = - \sum_ {t = 1} ^ {| \hat {z} _ {\backslash \mathcal {M}} |} \log p _ {\theta} \left(\mathbb {1} \left(\bar {z} _ {\backslash \mathcal {M}, t} ^ {\prime} = \hat {z} _ {\backslash \mathcal {M}, t}\right) | \bar {z} _ {\backslash \mathcal {M}} ^ {\prime}, t\right). \tag {6} \\ \end{array}
$$
As the example depicted in Figure 3(a), the target context sequence  $\hat{z}_{\backslash \mathcal{M}} = \{x_1, y_2, x_4, y_4, x_6\}$ .
3.5 N-gram Extraction
N-gram Lexicon Extraction. We employ T-test to extract semantically-complete  $n$ -grams statistically from unlabeled text corpora  $\mathcal{X}$  (Xiao et al., 2020), as described in Algorithm 1. We first calcu
Algorithm 1 N-gram Extraction with T-test
Input: Large-scale text corpora  $\mathcal{X}$  for pre-training
Output: Semantic  $n$ -gram lexicon  $\mathcal{V}_N$
$\triangleright$  given initial hypothesis  $H_0$ : a randomly constructed  $n$ -gram  $\boldsymbol{w} = \{x_1, \dots, x_n\}$  with probability  $p'(w) = \prod_{i=1}^{n} p(x_i)$  cannot be a statistically semantic  $n$ -gram for  $l$  in range  $(2, n)$  do
late the  $t$ -statistic scores of all  $n$ -grams appearing
in  $\mathcal{X}$  since the higher the  $t$ -statistic score, the more likely it is a semantically-complete  $n$ -gram. Then, we select the  $l$ -grams with the top  $k_{l}$ $t$ -statistic scores to construct the final  $n$ -gram lexicon  $\nu_{N}$ .
N-gram Boundary Extraction. To incorporate  $n$ -gram information into MLM objective,  $n$ -gram boundaries are referred to mask whole  $n$ -grams for pre-training. Given an input sequence  $x = \{x_{1},\dots,x_{|\pmb{x}|}\}$ , we employ maximum matching algorithm to traverse valid  $n$ -gram paths  $\mathcal{B} = \{b_1,\dots,b_{|\mathcal{B}|}\}$  according to  $\nu_{N}$ , then select the shortest paths as the final  $n$ -gram boundaries  $\pmb{b}$ , where  $|b| \leq |b_i|, \forall i = 1,\dots,|\mathcal{B}|$ .
In this section, we first present the pre-training configuration of ERNIE-Gram on Chinese and English text corpora. Then we compare ERNIE-Gram with previous works on various downstream tasks. We also conduct several ablation experiments to access the major components of ERNIE-Gram.
4.1 Pre-training Text Corpora
English Pre-training Data. We use two common text corpora for English pre-training:
- Base-scale corpora: 16GB uncompressed text from WIKIPEDIA and BOOKSCORPUS (Zhu et al., 2015), which is the original data for BERT.
- Large-scale corpora: 160GB uncompressed text from WIKIPEDIA, BOOKSCORPUS, OPENWEBTEXT<sup>3</sup>, CC-NEWS (Liu et al., 2019) and STORIES (Trinh and Le, 2018), which is the original data used in RoBERTa.
Chinese Pre-training Data. We adopt the same Chinese text corpora used in ERNIE2.0 (Sun et al., 2020) to pre-train ERNIE-Gram.
4.2 Pre-training Setup
Before pre-training, we first extract 200K bi-grams and 100K tri-grams with Algorithm 1 to construct the semantic  $n$ -gram lexicon  $\mathcal{V}_N$  for English and Chinese corpora. and we adopt the sub-word dictionary (30K BPE codes) used in BERT and the character dictionary used in ERNIE2.0 as our fine-grained vocabulary  $\mathcal{V}_F$  in English and Chinese.
Following the previous practice, we pre-train ERNIE-Gram in base size  $(L = 12, H = 768, A = 12$ , Total Parameters=110M), and set the
length of the sequence in each batch up to 512 tokens. We add the relative position bias (Raffel et al., 2020) to attention weights and use Adam (Kingma and Ba, 2015) for optimizing. For pre-training on base-scale English corpora, the batch size is set to 256 sequences, the peak learning rate is  $1e - 4$  for 1M training steps, which are the same settings as  $\mathrm{BERT}_{\mathrm{BASE}}$ . As for large-scale English corpora, the batch size is 5112 sequences, the peak learning rate is  $4e - 4$  for 500K training steps. For pre-training on Chinese corpora, the batch size is 256 sequences, the peak learning rate is  $1e - 4$  for 3M training steps. All the pre-training hyper-parameters are supplemented in the Appendix A.
In fine-tuning, we remove the auxiliary embedding weights of explicit  $n$ -grams identities for fair comparison with previous pre-trained models.
4.3 Results on GLUE Benchmark
The General Language Understanding Evaluation (GLUE; Wang et al., 2018) is a multi-task benchmark consisting of various NLU tasks, which contains 1) pairwise classification tasks like language inference (MNLI; Williams et al., 2018, RTE; Dagan et al., 2006), question answering (QNLI; Rajpurkar et al., 2016) and paraphrase detection (QQP, MRPC; Dolan and Brockett, 2005), 2) single-sentence classification tasks like linguistic acceptability (CoLA; Warstadt et al., 2019), sentiment analysis (SST-2; Socher et al., 2013) and 3) text similarity task (STS-B; Cer et al., 2017).
The fine-tuning results on GLUE of ERNIE-Gram and various strong baselines are presented in Table 1. For fair comparison, the listed models are all in base size and fine-tuned without any data augmentation. Pre-trained with base-scale text corpora, ERNIE-Gram outperforms recent models such as TUPE and F-TFM by 1.7 and 1.3 points on average. As for large-scale text corpora, ERNIE-Gram achieves average score increase of 1.7 and 0.6 over RoBERTa and ELECTRA, demonstrating the effectiveness of ERNIE-Gram.
4.4 Results on Question Answering (SQuAD)
The Stanford Question Answering (SQuAD) tasks are designed to extract the answer span within the given passage conditioned on the question. We conduct experiments on SQuAD1.1 (Rajpurkar et al., 2016) and SQuAD2.0 (Rajpurkar et al., 2018) by adding a classification layer on the sequence outputs of ERNIE-Gram and predicting whether each token is the start or end position of the answer span.
$^{3}$ http://web.archive.org save/http://Skylion007.github.io/OpenWebTextCorpus  
 $^{4}$ We donate the number of layers as  $L$ , the hidden size as  $H$  and the number of self-attention heads as  $A$ .
Models	#Param	MNLI Acc	QNLI Acc	QQP Acc	SST-2 Acc	CoLA MCC	MRPC Acc	RTE Acc	STS-B PCC	GLUE Avg
Results of single models pre-trained on base-scale text corpora (16GB)
BERT (Devlin et al., 2019)	110M	84.5	91.7	91.3	93.2	58.9	87.3	68.6	89.5	83.1
TUPE (Ke et al., 2020)	110M	86.2	92.1	91.3	93.3	63.6	89.9	73.6	89.2	85.0
F-TFMELECTRA (Dai et al., 2020)	110M	86.4	92.1	91.7	93.1	64.3	89.2	75.4	90.8	85.4
ERNIE-Gram	110M	87.1	92.8	91.8	93.2	68.5	90.3	79.4	90.4	86.7
Results of single models pre-trained on large-scale text corpora (160GB or more)
XLNet (Yang et al., 2019)	110M	86.8	91.7	91.4	94.7	60.2	88.2	74.0	89.5	84.5
RoBERTa (Liu et al., 2019)	135M	87.6	92.8	91.9	94.8	63.6	90.2	78.7	91.2	86.4
ELECTRA (Clark et al., 2020)	110M	88.8	93.2	91.5	95.2	67.7	89.5	82.7	91.2	87.5
UNILMv2 (Bao et al., 2020)	110M	88.5	93.5	91.7	95.1	65.2	91.8	81.3	91.0	87.3
MPNet (Song et al., 2020)	110M	88.5	93.3	91.9	95.4	65.0	91.5	85.2	90.9	87.7
ERNIE-Gram	110M	89.1	93.2	92.2	95.6	68.6	90.7	83.8	91.3	88.1