Venue: NAACL 2021

ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding

Dongling Xiao, Yu-Kun Li, Han Zhang, Yu Sun, Hao Tian,

Hua Wu and Haifeng Wang

Baidu Inc., China {xiaodongling,liyukun01,zhanghan17,sunyu02,tianhao,wu_hua,wanghaifeng}@baidu.com

###### Abstract

Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT's Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of \(n\) tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly \(n\)-gram masking method to enhance the integration of coarse-grained information into pre-training. In ERNIE-Gram, \(n\)-grams are masked and predicted directly using explicit \(n\)-gram identities rather than contiguous sequences of \(n\) tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible \(n\)-gram identities as optional n-gram masks and predict them in both coarse-grained and fine-grained manners to enable comprehensive \(n\)-gram prediction and relation modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art methods. The source codes and pre-trained models have been released at [https://github.com/PaddlePaddle/ERNIE](https://github.com/PaddlePaddle/ERNIE).

## 1 Introduction

Pre-trained on large-scaled text corpora and fine-tuned on downstream tasks, self-supervised representation models Radford et al. (2018); Devlin et al. (2019); Liu et al. (2019); Yang et al. (2019); Lan et al. (2020); Clark et al. (2020) have achieved remarkable improvements in natural language understanding (NLU). As one of the most prominent pre-trained models, BERT Devlin et al. (2019) employs masked language modeling (MLM) to learn representations by masking individual tokens and predicting them based on their bidirectional context. However, BERT's MLM focuses on the representations of fine-grained text units (e.g. words or subwords in English and characters in Chinese), rarely considering the coarse-grained linguistic information (e.g. named entities or phrases in English and words in Chinese) thus incurring inadequate representation learning.

Many efforts have been devoted to integrate coarse-grained semantic information by independently masking and predicting contiguous sequences of \(n\) tokens, namely \(n\)-grams, such as named entities, phrases Sun et al. (2019), whole words Cui et al. (2019) and text spans Joshi et al. (2020). We argue that such contiguously masking strategies are less effective and reliable since the prediction of tokens in masked n-grams are independent of each other, which neglects the Intra-dependencies of n-grams. Specifically, given a masked \(n\)-gram \(\boldsymbol{w}\!=\!\left\{x_{1},...,x_{n}\right\},x\!\in\!\mathcal{V}_{F}\), we maximize \(p(\boldsymbol{w})\!=\!\prod_{i=1}^{n}p(x_{i}|\boldsymbol{c})\) for \(n\)-gram learning, where models learn to recover \(\boldsymbol{w}\) in a huge and sparse prediction space \(\mathcal{F}\!\in\!\mathbb{R}^{[\mathcal{V}_{F}]^{n}}\). Note that \(\mathcal{V}_{F}\) is the fine-grained vocabulary1 and \(\boldsymbol{c}\) is the context.

Footnote 1: V\({}_{F}\) contains \(30\)K BPE codes in BERT Devlin et al. (2019) and \(50\)K subword units in RoBERTa Liu et al. (2019).

We propose ERNIE-Gram, an **explicitly \(\boldsymbol{n}\)-gram masked**anguage modeling method in which \(n\)-grams are masked with single [MASK] symbols, and predicted directly using explicit \(n\)-gram identities rather than sequences of tokens, as depicted in Figure 1(b). The models learn to predict \(n\)-gram \(\boldsymbol{w}\) in a small and dense prediction space \(\mathcal{N}\!\in\!\mathbb{R}^{|\mathcal{V}_{N}|}\), where \(\mathcal{V}_{N}\) indicates a prior \(n\)-gram lexicon2 and normally \(|\mathcal{V}_{N}|\ll|\mathcal{V}_{F}|^{n}\). To 

learn the semantic of \(n\)-grams more adequately, we adopt a **comprehensive \(n\)-gram prediction** mechanism, simultaneously predicting masked \(n\)-grams in coarse-grained (explicit \(n\)-gram identities) and fine-grained (contained token identities) manners with well-designed attention mask metrics, as shown in Figure 1(c).

In addition, to model the semantic relationships between \(n\)-grams directly, we introduce an **enhanced \(n\)-gram relation modeling** mechanism, masking \(n\)-grams with plausible \(n\)-grams identities sampled from a generator model, and then recovering them to the original \(n\)-grams with the pair relation between plausible and original \(n\)-grams. Inspired by ELECTRA Clark et al. (2020), we incorporate the replaced token detection objective to distinguish original \(n\)-grams from plausible ones, which enhances the interactions between explicit \(n\)-grams and fine-grained contextual tokens.

In this paper, we pre-train ERNIE-Gram on both base-scale and large-scale text corpora (16GB and 160GB respectively) under comparable pre-training setting. Then we fine-tune ERNIE-Gram on 13 English NLU tasks and 6 Chinese NLU tasks. Experimental results show that ERNIE-Gram consistently outperforms previous well-performed pre-training models on various benchmarks by a large margin.

## 2 Related Work

### Self-Supervised Pre-Training for NLU

Self-supervised pre-training has been used to learn contextualized sentence representations though various training objectives. GPT Radford et al. (2018) employs unidirectional language modeling (LM) to exploit large-scale corpora. BERT Devlin et al. (2019) proposes masked language modeling (MLM) to learn bidirectional representations efficiently, which is a representative objective for pre-training and has numerous extensions such as RoBERTa Liu et al. (2019), UniLM Dong et al. (2019) and ALBERT Lan et al. (2020). XLNet Yang et al. (2019) adopts permutation language modeling (PLM) to model the dependencies among predicted tokens. ELECTRA introduces replaced token detection (RTD) objective to learn all tokens for more compute-efficient pre-training.

### Coarse-grained Linguistic Information Incorporating for Pre-Training

Coarse-grained linguistic information is indispensable for adequate representation learning. There are lots of studies that implicitly integrate coarse-grained information by extending BERT's MLM to contiguously masking and predicting contiguous sequences of tokens. For example, ERNIE Sun et al. (2019b) masks named entities and phrases to enhance contextual representations, BERT-wwm Cui et al. (2019) masks whole Chinese words to achieve better Chinese representations, SpanBERT Joshi et al. (2020) masks contiguous spans to improve the performance on span selection tasks.

A few studies attempt to inject the coarse-grained \(n\)-gram representations into fine-grained contextualized representations explicitly, such as ZenDiao et al. (2020) and AMBERT Zhang and Li (2020), in which additional transformer encoders and computations for explicit \(n\)-gram representations are incorporated into both pre-training and fine-tuning. Li et al. (2019) demonstrate that explicit \(n\)-gram representations are not sufficiently reliable for NLP tasks because of \(n\)-gram data sparsity and the ubiquity of out-of-vocabulary \(n\)-grams. Differently, we only incorporate \(n\)-gram information by leveraging auxiliary \(n\)-gram classifier and embedding weights in pre-training, which will be completely removed during fine-tuning, so our method maintains the same parameters and computations as BERT.

## 3 Proposed Method

In this section, we present the detailed implementation of ERNIE-Gram, including \(n\)-gram lexicon

Figure 1: Illustrations of different MLM objectives, where \(x_{i}\) and \(y_{i}\) represent the identities of fine-grained tokens and explicit \(n\)-grams respectively. Note that the weights of fine-grained classifier (\(W_{F}\in\mathbb{R}^{h\times|V_{F}|}\)) and N-gram classifier (\(W_{N}\in\mathbb{R}^{h\times|(\left|\mathcal{P}_{r},\mathcal{V}_{N}\right|)}\)) are not used in fine-tuning stage, where \(h\) is the hidden size and \(L\) is the layers.

 

\({\mathcal{V}_{N}}\) extraction in Section 3.5, explicitly \(n\)-gram MLM pre-training objective in Section 3.2, comprehensive \(n\)-gram prediction and relation modeling mechanisms in Section 3.3 and 3.4.

### Background

To inject \(n\)-gram information into pre-training, many works Sun et al. (2019b); Cui et al. (2019); Joshi et al. (2020) extend BERT's masked language modeling (MLM) from masking individual tokens to contiguous sequences of \(n\) tokens.

Contiguously MLM.Given input sequence \(\boldsymbol{x}=\left\{x_{1},...,x_{|\boldsymbol{z}|}\right\},x\in{\mathcal{V}_ {F}}\) and \(n\)-gram starting boundaries \(\boldsymbol{b}=\left\{b_{1},...,b_{|\boldsymbol{b}|}\right\}\), let \(\boldsymbol{z}=\left\{z_{1},...,z_{|\boldsymbol{b}|-1}\right\}\) to be the sequence of \(n\)-grams, where \(z_{i}\!=\!\boldsymbol{x}_{[b_{i}:b_{i+1}]}\), MLM samples \(15\%\) of starting boundaries from \(\boldsymbol{b}\) to mask \(n\)-grams, donating \({\mathcal{M}}\) as the indexes of sampled starting boundaries, \(\boldsymbol{z}_{\mathcal{M}}\) as the contiguously masked tokens, \(\boldsymbol{z}_{\backslash\mathcal{M}}\) as the sequence after masking. As shown in Figure 1(a), \(\boldsymbol{b}=\left\{1,2,4,5,6,7\right\}\), \(\boldsymbol{z}=\left\{x_{1},\boldsymbol{x}_{[2:4)},x_{4},x_{5},x_{6}\right\}\), \({\mathcal{M}}=\left\{2,4\right\}\), \(\boldsymbol{z}_{\mathcal{M}}=\left\{\boldsymbol{x}_{[2:4)},x_{5}\right\}\), and \(\boldsymbol{z}_{\setminus\mathcal{M}}=\left\{x_{1},\texttt{ [M]}\,,\allowbreak\texttt{[M]}\,,x_{4},\texttt{ [M]}\,,x_{6}\right\}\). Contiguously MLM is performed by minimizing the negative likelihood:

\[-\log p_{\theta}(\boldsymbol{z}_{\mathcal{M}}|\boldsymbol{z}_{\backslash \mathcal{M}})=\!-\!\sum_{z\in\boldsymbol{z}_{\mathcal{M}}}\sum_{x\in z}\log p _{\theta}(x|\boldsymbol{z}_{\backslash\mathcal{M}}). \tag{1}\]

### Explicitly N-gram Masked Language Modeling

Different from contiguously MLM, we employ explicit \(n\)-gram identities as pre-training targets to reduce the prediction space for \(n\)-grams. To be specific, let \(\boldsymbol{y}=\left\{y_{1},...,y_{|\boldsymbol{b}|-1}\right\},y\in\langle{ \mathcal{V}_{F}},{\mathcal{V}_{N}}\rangle\) to be the sequence of explicit \(n\)-gram identities, \(\boldsymbol{y}_{\mathcal{M}}\) to be the target \(n\)-gram identities, and \(\bar{\boldsymbol{z}}_{\backslash\mathcal{M}}\) to be the sequence after explicitly masking \(n\)-grams. As shown in Figure 1(b), \(\boldsymbol{y}_{\mathcal{M}}=\left\{y_{2},y_{4}\right\}\), and \(\bar{\boldsymbol{z}}_{\backslash\mathcal{M}}=\left\{x_{1},\texttt{ [M]}\,,\allowbreak x_{4},\texttt{ [M]}\,,\allowbreak x_{6}\right\}\). For masked \(n\)-gram \(\boldsymbol{x}_{[2:4)}\), the prediction space is significantly reduced from \(\mathbb{R}^{|{\mathcal{V}_{F}}|^{2}}\) to \(\mathbb{R}^{|\langle{\mathcal{V}_{F}},{\mathcal{V}_{N}}\rangle|}\). Explicitly \(n\)-gram MLM is performed by minimizing the negative likelihood:

\[-\!\log p_{\theta}(\boldsymbol{y}_{\mathcal{M}}|\bar{\boldsymbol{z}}_{ \backslash\mathcal{M}})=\!-\!\sum_{y\in\boldsymbol{y}_{\mathcal{M}}}\log p_{ \theta}(y|\bar{\boldsymbol{z}}_{\backslash\mathcal{M}}). \tag{2}\]

### Comprehensive N-gram Prediction

We propose to simultaneously predict \(n\)-grams in fine-grained and coarse-grained manners corresponding to single mask symbol [M], which helps to extract comprehensive \(n\)-gram semantics, as shown in Figure 1(c). Comprehensive \(n\)-gram MLM is performed by minimizing the joint negative likelihood:

\[-\log p_{\theta}(\boldsymbol{y}_{\mathcal{M}},\boldsymbol{z}_{ \mathcal{M}}|\bar{\boldsymbol{z}}_{\backslash\mathcal{M}})=\\ -\!\sum_{y\in\boldsymbol{y}_{\mathcal{M}}}\log p_{\theta}(y|\bar{ \boldsymbol{z}}_{\backslash\mathcal{M}})-\!\sum_{z\in\boldsymbol{z}_{ \mathcal{M}}}\sum_{x\in z}\!\log p_{\theta}(x|\bar{\boldsymbol{z}}_{\backslash \mathcal{M}}). \tag{3}\]

where the predictions of explicit \(n\)-gram \(\boldsymbol{y}_{\mathcal{M}}\) and fine-grained tokens \(\boldsymbol{x}_{\mathcal{M}}\) are conditioned on the same context sequence \(\bar{\boldsymbol{z}}_{\backslash\mathcal{M}}\).

In detail, to predict all tokens contained in a \(n\)-gram from single [M] other than a consecutive sequence of [M], we adopt distinctive mask symbols [Mi], i = \(1,...,n\) to aggregate contextualized representations for predicting the i-th token in \(n\)-gram. As shown in Figure 2(a), along with the same position as \(y_{2}\), symbols [M1] and [M2] are used as queries (\(Q\)) to aggregate representations from \(\bar{\boldsymbol{z}}_{\backslash\mathcal{M}}\) (\(K\)) for the predictions of \(x_{2}\) and \(x_{3}\), where \(Q\) and \(K\) donate the query and key in self-attention operation Vaswani et al. (2017). As shown in Figure 2(b), the self-attention mask metric \(M\) controls what context a token can attend to by modifying the attention weight \(W_{A}\!=\!\mathtt{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}}+M)\), \(M\) is assigned as:

\[M_{ij}=\begin{cases}0,&\mathrm{allow~{}to~{}attend}\\ -\infty,&\mathrm{prevent~{}from~{}atending}\end{cases} \tag{4}\]

We argue that the length information of \(n\)-grams is detrimental to the representations learning, because it will arbitrarily prune a number of semati

Figure 2: (a) Detailed structure of Comprehensive N-gram MLM. (b) Self-attention mask \(M\) without leaking length information of masked \(n\)-grams.

 

cally related \(n\)-grams with different lengths during predicting. From this viewpoint, for the predictions of \(n\)-gram {\(x_{2},x_{3}\)}, 1) we prevent context \(\bar{\mathbf{z}}_{\backslash\mathcal{M}}\) from attending to {[MI],[M2]} and 2) prevent {[M1],[M2]} from attending to each other, so that the length information of \(n\)-grams will not be leaked in pre-training, as displayed in Figure 2(b).

### Enhanced N-gram Relation Modeling

To explicitly learn the semantic relationships between \(n\)-grams, we jointly pre-train a small generator model \(\theta^{\prime}\) with explicitly \(n\)-gram MLM objective to sample plausible \(n\)-gram identities. Then we employ the generated identities to preform masking and train the standard model \(\theta\) to predict the original \(n\)-grams from fake ones in coarse-grained and fine-grained manners, as shown in Figure 3(a), which is efficient to model the pair relationships between similar \(n\)-grams. The generator model \(\theta^{\prime}\) will not be used during fine-tuning, where the hidden size \(H_{\theta^{\prime}}\) of \(\theta^{\prime}\) has \(H_{\theta^{\prime}}=H_{\theta}/3\) empirically.

As shown in Figure 3(b), \(n\)-grams of different length can be sampled to mask original \(n\)-grams according to the prediction distributions of \(\theta^{\prime}\), which is more flexible and sufficient for constructing \(n\)-gram pairs than previous synonym masking methods Cui et al. (2020) that require synonyms and original words to be of the same length. Note that our method needs a large embedding layer \(E\in\mathbb{R}^{|\langle V_{F},\mathcal{V}_{N}\rangle|\times h}\) to obtain \(n\)-gram vectors in pre-training. To keep the number of parameters consistent with that of vanilla BERT, we remove the auxiliary embedding weights of \(n\)-grams during finetuning (\(E\to E^{\prime}\in\mathbb{R}^{|\mathcal{V}_{F}|\times h}\)). Specifically, let \(\mathbf{y}^{\prime}_{\mathcal{M}}\) to be the generated \(n\)-gram identities, \(\bar{\mathbf{z}}^{\prime}_{\mathcal{M}}\) to be the sequence masked by \(\mathbf{y}^{\prime}_{\mathcal{M}}\), where \(\mathbf{y}^{\prime}_{\mathcal{M}}=\{y^{\prime}_{2},y^{\prime}_{4}\}\), and \(\bar{\mathbf{z}}^{\prime}_{\backslash\mathcal{M}}=\{x_{1},y^{\prime}_{2},x_{4}, y^{\prime}_{4},x_{6}\}\) in Figure 3(a). The pre-training objective is to jointly minimize the negative likelihood of \(\theta^{\prime}\) and \(\theta\):

\[-\log p_{\theta^{\prime}}(\mathbf{y}_{\mathcal{M}}|\bar{\mathbf{z}}_{\backslash \mathcal{M}})-\log p_{\theta}(\mathbf{y}_{\mathcal{M}},\mathbf{z}_{\mathcal{M}}|\bar{ \mathbf{z}}^{\prime}_{\backslash\mathcal{M}}). \tag{5}\]

Moreover, we incorporate the replaced token detection objective (RTD) to further distinguish fake \(n\)-grams from the mix-grained context \(\bar{\mathbf{z}}^{\prime}_{\backslash\mathcal{M}}\) for interactions among explicit \(n\)-grams and fine-grained contextual tokens, as shown in the right part of Figure 3(a). Formally, we donate \(\hat{\mathbf{z}}_{\backslash\mathcal{M}}\) to be the sequence after replacing masked \(n\)-grams with target \(n\)-gram identities \(\mathbf{y}_{\mathcal{M}}\), the RTD objective is performed by minimizing the negative likelihood:

\[-\log p_{\theta}\big{(}1(\bar{\mathbf{z}}^{\prime}_{\backslash\mathcal{ M}}=\hat{\mathbf{z}}_{\backslash\mathcal{M}})|\bar{\mathbf{z}}^{\prime}_{\backslash \mathcal{M}}\big{)} \tag{6}\] \[=-\sum_{t=1}^{\infty}\;\log p_{\theta}\big{(}1(\bar{\mathbf{z}}^{ \prime}_{\backslash\mathcal{M},t}=\hat{\mathbf{z}}_{\backslash\mathcal{M},t})|\bar {\mathbf{z}}^{\prime}_{\backslash\mathcal{M}},t\big{)}.\]

As the example depicted in Figure 3(a), the target context sequence \(\hat{\mathbf{z}}_{\backslash\mathcal{M}}=\{x_{1},y_{2},x_{4},y_{4},x_{6}\}\).

### N-gram Extraction

N-gram Lexicon Extraction.We employ T-test to extract semantically-complete \(n\)-grams statistically from unlabeled text corpora \(\mathcal{X}\) Xiao et al. (2020), as described in Algorithm 1.

```
Input: Large-scale text corpora \(\mathcal{X}\) for pre-training Output: Semantic \(n\)-gram lexicon \(\mathcal{V}_{N}\) \(\triangleright\) given initial hypothesis \(H_{0}\): a randomly constructed \(n\)-gram \(\mathbf{w}=\{x_{1},...,x_{n}\}\) with probability \(p^{\prime}(\mathbf{w})=\prod_{i=1}^{n}p(x_{i})\) cannot be a statistically semantic \(n\)-gram forl in range(2, \(n\))do\(\mathcal{V}_{N_{l}}\leftarrow\langle\rangle\)\(\triangleright\) initialize the lexicon for l-gram forl-gram \(\mathbf{w}\) in \(\mathcal{X}\)do\(s\leftarrow\frac{(p(\mathbf{w})-p^{\prime}(\mathbf{w}))}{\sqrt{\sigma^{2}/N_{l}}}\); \(t\)-statistic score \(\triangleright\) where statistical probability \(p(\mathbf{w})=\frac{\text{count}(\mathbf{w})}{N_{l}}\), deviation \(\sigma^{2}=p(\mathbf{w})(1-p(\mathbf{w}))\), \(N_{l}\) donates the count of \(l\)-grams in \(\mathcal{X}\) \(\mathcal{V}_{N_{l}}\)_append(\{\mathbf{w},s\}) \(\mathcal{V}_{N_{u}}\)\(\leftarrow\)topk\((\mathcal{V}_{N_{l}},k_{l})\quad\triangleright\)\(k_{l}\) is the number of \(l\)-gram \(\mathcal{V}_{N}\leftarrow\langle\mathcal{V}_{N_{2}},...,\mathcal{V}_{N_{n}}\rangle\)\(\triangleright\) merge all lexicons return\(\mathcal{V}_{N}\)
```

**Algorithm 1** N-gram Extraction with T-test

In the following, we first calculate the \(t\)-statistic scores of all \(n\)-grams appearing

Figure 3: (a) Detailed architecture of \(n\)-gram relation modeling, where \(L^{\prime}\) donates the layers of the generator model. (b) An example of plausible \(n\)-gram sampling, where dotted boxes represent the sampling module, texts in green are the original \(n\)-grams, and the italictexts in blue donate the sampled \(n\)-grams.

 

in \({\cal X}\) since the higher the \(t\)-statistic score, the more likely it is a semantically-complete \(n\)-gram. Then, we select the \(l\)-grams with the top \(k_{l}\)\(t\)-statistic scores to construct the final \(n\)-gram lexicon \({\cal V}_{N}\).

N-gram Boundary Extraction.To incorporate \(n\)-gram information into MLM objective, \(n\)-gram boundaries are referred to mask whole \(n\)-grams for pre-training. Given an input sequence \(\mathbf{x}=\{x_{1},...,x_{|\mathbf{x}|}\}\), we employ maximum matching algorithm to traverse valid \(n\)-gram paths \(\mathcal{B}=\{\mathbf{b}_{1},...,\mathbf{b}_{|\mathcal{B}|}\}\) according to \({\cal V}_{N}\), then select the shortest paths as the final \(n\)-gram boundaries \(\mathbf{b}\), where \(|\mathbf{b}|\leq|\mathbf{b}_{i}|,\forall i=1,...,|\mathcal{B}|\).

## 4 Experiments

In this section, we first present the pre-training configuration of ERNIE-Gram on Chinese and English text corpora. Then we compare ERNIE-Gram with previous works on various downstream tasks. We also conduct several ablation experiments to access the major components of ERNIE-Gram.

### Pre-training Text Corpora

English Pre-training Data.We use two common text corpora for English pre-training:

* [leftmargin=0pt]
* **Base-scale corpora:** 16GB uncompressed text from Wikipedia and BooksCorpus Zhu et al. (2015), which is the original data for BERT.
* **Large-scale corpora:** 160GB uncompressed text from Wikipedia, BooksCorpus, OpenWebText3, CC-News Liu et al. (2019) and Stories Trinh and Le (2018), which is the original data used in RoBERTa.

Footnote 3: [http://web.archive.org/save/http://Skylion007.github.io/OpenWebTextCorpus](http://web.archive.org/save/http://Skylion007.github.io/OpenWebTextCorpus)

Chinese Pre-training Data.We adopt the same Chinese text corpora used in ERNIE2.0 Sun et al. (2020) to pre-train ERNIE-Gram.

### Pre-training Setup

Before pre-training, we first extract \(200\)K bi-grams and \(100\)K tri-grams with Algorithm \(1\) to construct the semantic \(n\)-gram lexicon \({\cal V}_{N}\) for English and Chinese corpora. and we adopt the sub-word dictionary (\(30\)K BPE codes) used in BERT and the character dictionary used in ERNIE2.0 as our fine-grained vocabulary \({\cal V}_{F}\) in English and Chinese.

Following the previous practice, we pre-train ERNIE-Gram in base size (\(L=12,H=768,A=12\), Total Parameters=\(110\)M)4, and set the length of the sequence in each batch up to \(512\) tokens. We add the relative position bias Raffel et al. (2020) to attention weights and use Adam Kingma and Ba (2015) for optimizing. For pre-training on base-scale English corpora, the batch size is set to \(256\) sequences, the peak learning rate is \(1e\text{-}4\) for \(1\)M training steps, which are the same settings as BERTBASE. As for large-scale English corpora, the batch size is \(5112\) sequences, the peak learning rate is \(4e\text{-}4\) for \(500\)K training steps. For pre-training on Chinese corpora, the batch size is \(256\) sequences, the peak learning rate is \(1e\text{-}4\) for 3M training steps. All the pre-training hyper-parameters are supplemented in the Appendix A.

Footnote 4: We donate the number of layers as \(L\), the hidden size as \(H\) and the number of self-attention heads as \(A\).

In fine-tuning, we remove the auxiliary embedding weights of explicit \(n\)-grams identities for fair comparison with previous pre-trained models.

### Results on GLUE Benchmark

The General Language Understanding Evaluation (GLUE; Wang et al., 2018) is a multi-task benchmark consisting of various NLU tasks, which contains 1) pairwise classification tasks like language inference (MNLI; Williams et al. (2018), RTE; Dagan et al. (2006), question answering QNLI; Rajpurkar et al. (2016) and paraphrase detection QQP, MRPC; Dolan and Brockett (2005), 2) single-sentence classification tasks like linguistic acceptability CoLA; Warstadt et al. (2019), sentiment analysis (SST-2; Socher et al., 2013) and 3) text similarity task STS-B; Cer et al. (2017).

The fine-tuning results on GLUE of ERNIE-Gram and various strong baselines are presented in Table 1. For fair comparison, the listed models are all in base size and fine-tuned without any data augmentation. Pre-trained with base-scale text corpora, ERNIE-Gram outperforms recent models such as TUPE and F-TFM by \(1.7\) and \(1.3\) points on average. As for large-scale text corpora, ERNIE-Gram achieves average score increase of \(1.7\) and \(0.6\) over RoBERTa and ELECTRA, demonstrating the effectiveness of ERNIE-Gram.

### Results on Question Answering (SquAD)

The Stanford Question Answering (SQuAD) tasks are designed to extract the answer span within the given passage conditioned on the question. We conduct experiments on SQuAD1.1 Rajpurkar et al. (2016) and SQuAD2.0 Rajpurkar et al. (2018) by adding a classification layer on the sequence outputs of ERNIE-Gram and predicting whether each token is the start or end posit 

Table 2 presents the results on SQuAD for base-size pre-trained models, ERNIE-Gram achieves better performance than current strong baselines on both base-scale and large-scale pre-training text corpora.

### Results on RACE and Text Classification Tasks

The ReAding Comprehension from Examinations (RACE; Lai et al., 2017) dataset collects \(88\)K long passages from English exams at middle and high schools, the task is to select the correct choice from four given options according to the questions and passages. We also evaluate ERNIE-Gram on two large scaled text classification tasks that involve long text and reasoning, including sentiment analysis datasets IMDb (Maas et al., 2011) and topic classification dataset AG's News (Zhang et al., 2015). The results are reported in Table 3. It can be seen that ERNIE-Gram consistently outperforms previous models, showing the advantage of ERNIE-Gram on tasks involving long text and reasoning.

### Results on Chinese NLU Tasks

We execute extensive experiments on six Chinese language understanding tasks, including natural language inference (XNLI; Conneau et al., 2018),

\begin{table}
\begin{tabular}{l|c c c c c c c c c|c} \hline \hline \multirow{2}{*}{**Models**} & \multirow{2}{*}{**\#Param**} & **MNLI** & **QNLI** & **QQP** & **SST-2** & **CoLA** & **MRPC** & **RTE** & **STS-B** & **GLUE** \\  & & Acc & Acc & Acc & Acc & MCC & Acc & Acc & PCC & **Avg** \\ \hline \multicolumn{9}{l}{_Results of single models pre-trained on **base-scale** text corpora (16GB)_} \\ \hline BERT Devlin et al. (2019) & 110M & 84.5 & 91.7 & 91.3 & 93.2 & 58.9 & 87.3 & 68.6 & 89.5 & 83.1 \\ TUPe Ke et al. (2020) & 110M & 86.2 & 92.1 & 91.3 & **93.3** & 63.6 & 89.9 & 73.6 & 89.2 & 85.0 \\ F-TFMELECTRA Dai et al. (2020) & 110M & 86.4 & 92.1 & 91.7 & 93.1 & 64.3 & 89.2 & 75.4 & **90.8** & 85.4 \\ \hline ERNIE-Gram & 110M & **87.1** & **92.8** & **91.8** & 93.2 & **68.5** & **90.3** & **79.4** & 90.4 & **86.7** \\ \hline \multicolumn{9}{l}{_Results of single models pre-trained on **large-scale** text corpora (160GB or more)_} \\ \hline XLNet Yang et al. (2019) & 110M & 86.8 & 91.7 & 91.4 & 94.7 & 60.2 & 88.2 & 74.0 & 89.5 & 84.5 \\ RoBERTa Liu et al. (2019) & 135M & 87.6 & 92.8 & 91.9 & 94.8 & 63.6 & 90.2 & 78.7 & 91.2 & 86.4 \\ ELECTRA Clark et al. (2020) & 110M & 88.8 & 93.2 & 91.5 & 95.2 & 67.7 & 89.5 & 82.7 & 91.2 & 87.5 \\ UniLMv2 Bao et al. (2020) & 110M & 88.5 & **93.5** & 91.7 & 95.1 & 65.2 & **91.8** & 81.3 & 91.0 & 87.3 \\ MPNet Song et al. (2020) & 110M & 88.5 & 93.3 & 91.9 & 95.4 & 65.0 & 91.5 & **85.2** & 90.9 & 87.7 \\ \hline ERNIE-Gram & 110M & **89.1** & 93.2 & **92.2** & **95.6** & **68.6** & 90.7 & 83.8 & **91.3** & **88.1** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results on the development set of the GLUE benchmark for base-size pre-trained models. Models using 16GB corpora are all pre-trained with a batch size of 256 sequences for 1M steps. STS-B and CoLA are reported by Pearson correlation coefficient (PCC) and Matthews correlation coefficient (MCC), other tasks are reported by accuracy (Acc). Note that results of ERNIE-Gram are the median of over ten runs with different random seeds.

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**SQuAD1.1**} & **SQuAD2.0** \\  & EM & F1 & EM & F1 \\ \hline \multicolumn{5}{l}{_Models pre-trained on **base-scale** text corpora (16GB)_} \\ \hline BERT Devlin et al. (2019) & 80.8 & 88.5 & 73.7 & 76.3 \\ RoBERTa Liu et al. (2019) & - & 90.6 & - & 79.7 \\ XLNet Yang et al. (2019) & – & - & 78.2 & 81.0 \\ MPNet Gong et al. (2020) & 85.0 & 91.4 & 90.5 & 83.3 \\ UniLMv2 Bao et al. (2020) & 85.6 & 92.0 & 80.9 & 83.6 \\ ERNIE-Gram & **86.2** & **92.3** & **82.1** & **84.8** \\ \hline \multicolumn{5}{l}{_Models pre-trained on **large-scale** text corpora (160GB)_} \\ \hline RobBERta Liu et al. (2019) & 84.6 & 91.5 & 80.5 & 83.7 \\ XLNet Yang et al. (2019) & – & - & 80.2 & - \\ ELECTRA Clark et al. (2020) & 86.8 & - & 80.5 & - \\ MPNet Song et al. (2020) & 86.8 & 92.5 & 82.8 & 85.6 \\ UniLMv2 Bao et al. (2020) & 87.1 & 93.1 & 83.3 & 86.1 \\ ERNIE-Gram & **87.2** & **93.2** & **84.1** & **87.1** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison between base-size pre-trained models on the SQuAD development sets. Exact-Match (EM) and F1 score are adopted for evaluations. Results of ERNIE-Gram are the median of over ten runs with different random seeds.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**RACE**} & **IMdb** & **AG** \\  & Total & High & Middle & Err. & Err. \\ \hline \multicolumn{5}{l}{_Pre-trained on **base-scale** text corpora (16GB)_} \\ \hline BERT \({}^{a}\) & 65.0 & 62.3 & 71.7 & 5.4 & 5.9 \\ XLNet \({}^{b}\) & 66.8 & - & - & 4.9 & - \\ MPNet \({}^{c}\) & 70.4 & 67.7 & 76.8 & 4.8 & - \\ F-TFMELECTRA & - & - & - & 5.2 & 5.4 \\ ENIIE-Gram & **72.7** & **68.1** & 75.1 & **4.6** & **5.0** \\ \hline \multicolumn{5}{l}{_Pre-trained on **large-scale** text corpora (160GB)_} \\ \hline MPNet \({}^{c}\) & 72.0 & 70.3 & 76.3 & 4.4 & - \\ ERNIE-Gram & **77.7** & **75.6** & **78.8** & **3.9** & **4.9** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison on the test sets of RACE, IMDb and AG. The listed models are all in base-size. In the results of RACE, “High” and “Middle” represent the training and evaluation sets for high schools and mid-dle schools respectively, “Total” is the full training and evaluation set. \({}^{a}\)Devlin et al. (2019); \({}^{b}\)Yang et al. (2019); \({}^{c}\)Song et al. (2020); \({}^{d}\)Dai et al. (2020).

 

machine reading comprehension (CMRC2018; Cui et al., 2018, DRCD; Shao et al., 2018 and DuReader; He et al., 2018), named entity recognition (MSRA-NER; Gao et al., 2005) and semantic similarity (LCQMC; Liu et al., 2018).

Results on six Chinese tasks are presented in Table 4. It is observed that ERNIE-Gram significantly outperforms previous models across tasks by a large margin and achieves new state-of-the-art results on these Chinese NLU tasks in base-size model group. Besides, ERNIE-\({\rm Gram}_{\rm BASE}\) are also better than various large-size models on XNLI, LCQMC and CMRC2018 datasets.

### Ablation Studies

We further conduct ablation experiments to analyze the major components of ERNIE-Gram.

**Effect of Explicitly N-gram MLM.** We compare two models pre-trained with contiguously MLM and explicitly \(n\)-gram MLM objectives in the same settings (the size of \(n\)-gram lexicon is \(300\)K). The evaluation results for pre-training and fine-tuning are shown in Figure 4. Compared with contiguously MLM, explicitly \(n\)-gram MLM objective facilitates the learning of \(n\)-gram semantic information with lower \(n\)-gram level perplexity in pre-training and better performance on downstream tasks. This verifies the effectiveness of explicitly \(n\)-gram MLM objective for injecting \(n\)-gram semantic information into pre-training.

**Size of N-gram Lexicon.** To study the impact of \(n\)-gram lexicon size on model performance, we extract \(n\)-gram lexicons with size from \(100\)K to \(400\)K for pre-training, as shown in Figure 5. As the lexicon size enlarges, performance of contiguously MLM becomes worse, presumably because more \(n\)-grams are matched and connected as longer consecutive spans for prediction, which is more difficult for representation learning. Explicitly \(n\)-gram MLM with lexicon size being \(300\)K achieves the best results, while the performance significantly declines when the size of lexicon increasing to \(400\)K because more low-frequent \(n\)-grams are learning unnecessarily. See Appendix C for detailed results of different lexicon choices on GLUE and SQuAD.

**Effect of Comprehensive N-gram Prediction and Enhanced N-gram Relation Modeling.** As shown in Table 5, we compare several ERNIE-Gram variants with previous strong baselines under the BERT\({}_{\rm BASE}\) setting. After removing comprehensive \(n\)-gram prediction (#2), ERNIE-Gram degenerates to a variant with explicitly \(n\)-gram MLM and \(n\)-gram relation modeling and its performance drops slightly by \(0.3\)-\(0.6\). When removing enhanced \(n\)-gram relation modeling (#3), ERNIE-Gram degenerates t

Figure 4: (a) N-gram level perplexity which is calculated by \((\prod_{i=1}^{k}\texttt{PPL}(\mathbf{w}_{i}))^{\frac{1}{k}}\) for contiguously MLM, where \(\mathbf{w}_{i}\) is the \(i\)-th masked \(n\)-gram. (b) Performance distribution box plot on MNLI, QNLI, SST-2 and SQuAD1.1.

 

sive \(n\)-gram MLM and the performance drops by \(0.4\text{-}1.3\). If removing both comprehensive \(n\)-gram prediction and relation modeling (#4), ERNIEGram degenerates to a variant with explicitly \(n\)-gram MLM and the performance drops by \(0.7\text{-}1.6\). These results demonstrate the advantage of comprehensive \(n\)-gram prediction and \(n\)-gram relation modeling methods for efficiently \(n\)-gram semantic injecting into pre-training. The detailed results of ablation study are supplemented in Appendix C.

### Case Studies

To further understand the effectiveness of our approach for learning \(n\)-grams information, we fine-tune ERNIE-Gram, contiguously MLM and lower-cased BERT on CoNLL-2003 named entity recognition task Sang and De Meulder (2003) for comparison. We divide the evaluation set into five subsets based on the average length of the named entities in each sentence. As shown in Figure 5(a), it is more difficult to recognize whole named entities as the length of them increases, while the performance of ERNIE-Gram declines slower than contiguously MLM and BERT, which implies that ERNIE-Gram models tighter intra-dependencies of \(n\)-grams.

As shown in Figure 5(b-d), we visualize the attention patterns in the last self-attention layer of fine-tuned models. For contiguously MLM, there are clear diagonal lines in named entities that tokens prefer to attend to themself in named entities. While for ERNIE-Gram, there are bright blocks over named entities that tokens attend to most of tokens in the same entity adequately to construct tight representation, verifying the effectiveness of ERNIE-Gram for \(n\)-gram semantic modeling.

## 5 Conclusion

In this paper, we present ERNIE-Gram, an explicitly \(n\)-gram masking and predicting method to eliminate the limitations of previous contiguously masking strategies and incorporate coarse-grained linguistic information into pre-training sufficiently. ERNIE-Gram conducts comprehensive \(n\)-gram prediction and relation modeling to further enhance the learning of semantic \(n\)-grams for pre-training. Experimental results on various NLU tasks demonstrate that ERNIE-Gram outperforms XLNet and RoBERTa by a large margin, and achieves state-of-the-art results on various benchmarks. Future work includes constructing more comprehensive \(n\)-gram 

lexicon (\(n>3\)) and pre-training ERNIE-Gram with large-size model for more downstream tasks.

## Acknowledgments

We would like to thank Zhen Li for his constructive suggestions, and hope everything goes well with his work. We are also indebted to the NAACL-HLT reviewers for their detailed and insightful comments on our work.

## References

* Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, et al. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In _Proceedings of the International Conference on Machine Learning_, pages 7006-7016.
* Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
* Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In _International Conference on Learning Representations_.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Cui et al. (2020) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pretrained models for Chinese natural language processing. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 657-668, Online. Association for Computational Linguistics.
* Cui et al. (2019) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. _arXiv preprint arXiv:1906.08101_.
* Cui et al. (2018) Yiming Cui, Ting Liu, Li Xiao, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. 2018. A span-extraction dataset for chinese machine reading comprehension. _CoRR_, abs/1810.07366.
* Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In _Proceedings of the First International Conference on Machine Learning Challenges_, pages 177-190.
* Dai et al. (2020) Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V Le. 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. In _Advances in Neural Information Processing Systems_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Diao et al. (2020) Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2020. ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4729-4740, Online. Association for Computational Linguistics.
* Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
* Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wien Hon. 2019. Unified language model pre-training for natural language understanding and generation. In _Advances in Neural Information Processing Systems_, volume 32, pages 13063-13075. Curran Associates, Inc.
* Gao et al. (2005) Jianfeng Gao, Mu Li, Andi Wu, and Chang-Ning Huang. 2005. Chinese word segmentation and named entity recognition: A pragmatic approach. _Computational Linguistics_, 31(4):531-574.
* He et al. (2018) Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. DuReader: A Chinese machine reading comprehension dataset from real-world applications. In _Proceedings of the Workshop on Machine Reading for Question Answering_, pages 37-46, Melbourne, Australia. Association for Computational Linguistics.
* Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. _Transactions of the Association for Computational Linguistics_, 8:64-77.
* Ke et al. (2020) Goolin Ke, Di He, and Tie-Yan Liu. 2020. Rethinking the positional encoding in language pre-training. _arXiv preprint arXiv:2006.15595_ 

* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations_, San Diego, CA.
* Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAdging comprehension dataset from examinations. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In _International Conference on Learning Representations_.
* Li et al. (2019) Xiaoya Li, Yuxiang Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. 2019. Is word segmentation necessary for deep learning of Chinese representations? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3242-3252, Florence, Italy. Association for Computational Linguistics.
* Liu et al. (2018) Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018. LCQMC:a large-scale Chinese question matching corpus. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1952-1962, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_, pages 142-147.
* Shao et al. (2018) Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2018. DRCD: a chinese machine reading comprehension dataset. _arXiv preprint arXiv:1806.00920_.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Ti-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. In _Advances in Neural Information Processing Systems_.
* Sun et al. (2019a) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019a. How to fine-tune bert for text classification? In _China National Conference on Chinese Computational Linguistics_, pages 194-206. Springer.
* Sun et al. (2019b) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019b. Ernie: Enhanced representation through knowledge integration. _arXiv preprint arXiv:1904.09223_.
* Sun et al. (2020) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):8968-8975.
* Trinh and Le (2018) Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. _ArXiv_, abs/1806.02847.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008. Curran Associates, Inc.

 

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
* Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. _Transactions of the Association for Computational Linguistics_, 7:625-641.
* Wei et al. (2019) Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. NEZHA: Neural contextualized representation for chinese language understanding. _arXiv preprint arXiv:1909.00204_.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Xiao et al. (2020) Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernigen: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_, pages 3997-4003. International Joint Conferences on Artificial Intelligence Organization. Main track.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In _Advances in Neural Information Processing Systems_, volume 32, pages 5753-5763. Curran Associates, Inc.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Advances in Neural Information Processing Systems_, volume 28, pages 649-657. Curran Associates, Inc.
* Zhang and Li (2020) Xinsong Zhang and Hang Li. 2020. Ambert: A pretrained language model with multi-grained tokenization. _arXiv preprint arXiv:2008.11869_.
* Zhu et al. (2015) Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _2015 IEEE International Conference on Computer Vision (ICCV)_, pages 19-27.

 

Hyperparameters for Pre-Training

As shown in Table 6, we list the detailed hyperparameters used for pre-training ERNIE-Gram on base and large scaled English text corpora and Chinese text corpora. We follow the same hyperparameters of BERT\({}_{\mathrm{BASE}}\)Devlin et al. (2019) to pre-train ERNIE-Gram on the base-scale English text corpora (16GB). We pre-train ERNIE-Gram on the large-scale text corpora (160GB) with the settings in RoBERTaLiu et al. (2019) except the batch size being 5112 sequences.

## Appendix B Hyperparameters for Fine-Tuning

The hyperparameters for each tasks are searched on the development sets according to the average score of ten runs with different random seeds.

### GLUE benchmark

The fine-tuning hyper-parameters for GLUE benchmark Wang et al. (2018) are presented in Table 7.

### SQuAD benchmark and RACE dataset

The fine-tuning hyper-parameters for SQuAD Rajpurkar et al. (2016);Rajpurkar et al. (2018) and RACE Lai et al. (2017) are presented in Table 8.

### Cluster Classification tasks

Table 9 lists the fine-tuning hyper-parameters for IMDb Maas et al. (2011) and AG'news Zhang et al. (2015) datasets. To process texts with a length larger than \(512\), we follow Sun et al. (2019) to select the first \(512\) tokens to perform fine-tuning.

### Chinese NLU tasks

The fine-tuning hyperparameters for Chinese NLU tasks including XNLI Conneau et al. (2018), LCQMC Liu et al. (2018), DRCD Shao et al. (2018), DuReader He et al. (2018), CMRC2018 and MSRA-NER Gao et al. (2005) are presented in Table 10.

## Appendix C Detailed Results for Ablation Studies

We present the detailed results on GLUE benchmark for ablation studies in this section. The results on different MLM objectives and sizes of \(n\)-gram lexicon are presented in Table 11. The detailed

\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Hyperparameters** & **Base-scale** & **Large-scale** & **Chinese** \\ \hline Layers & & 12 & \\ Hidden size & & 768 & \\ Attention heads & & 12 & \\ Training steps & 1M & 500K & 3M \\ Batch size & 256 & 5112 & 256 \\ Learning rate & 1e-4 & 4e-4 & 1e-4 \\ Warmup reports & 10,000 & 24,000 & 4,000 \\ Adam \(\beta\) & (0.9, 0.99) & (0.9, 0.98) & (0.9, 0.99) \\ Adam \(\epsilon\) & & 1e-6 & \\ Learning rate schedule & & Linear & \\ Weight decay & & 0.01 & \\ Dropout & & 0.1 & \\ GPUs (Nvidia V100) & 16 & 64 & 32 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyperparameters used for pre-training on different text corpora.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Hyperparameters** & **GLUE** \\ \hline Batch size & \{16, 32\} \\ Learning rate & \{5e-5, 1e-4, 1.5e-4\} \\ Epochs & 3 for MNLI and \{10, 15\} for others \\ LR schedule & Linear \\ Layerwise LR decay & 0.8 \\ Warmup proportion & 0.1 \\ Weight decay & 0.01 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Hyperparameters used for fine-tuning on the GLUE benchmark.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Hyperparameters** & **SQuAD** & **RACE** \\ \hline Batch size & 48 & 32 \\ Learning rate & \{1e-4, 1.5e-4, 2e-4\} & \{8e-5, 1e-4\} \\ Epochs & \{2, 4\} & \{4, 5\} \\ LR schedule & Linear \\ Layerwise LR decay & Linear \\ Warmup proportion & 0.1 \\ Weight decay & 0.01 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Hyperparameters used for fine-tuning on the SQuAD benchmark and RACE dataset.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Hyperparameters** & **IMDb** & **AG’news** \\ \hline Batch size & 32 \\ Learning rate & \{5e-5, 1e-4, 1.5e-4\} \\ Epochs & 3 \\ LR schedule & Linear \\ Layerwise LR decay & 0.8 \\ Warmup proportion & 0.1 \\ Weight decay & 0.01 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Hyperparameters used for fine-tuning on IMDb and AG‘news.

 

results on ERNIE-Gram variants to verify the effectiveness of comprehensive \(n\)-gram prediction and enhanced \(n\)-gram relation modeling mechanisms are presented in Table12. Results of ablation study on relative position bias Raffel et al. (2020) are presented in Table13.

## Appendix D More cases on CoNLL2003 Dataset

We visualize the attention patterns of three supplementary cases from CoNLL2003 named entity recognition dataset Sang and De Meulder (2003) to compare the performance of ERNIE-Gram, contiguously MLM and BERT (lowercased), as shown

Figure 7: (a-c) Mean attention scores in the last self-attention layer. Texts in green, orange, red and blue boxes are named entities standing for organizations, locations, person and miscellaneous respectively.

 

in Figure 7. For contiguously MLM, there are clear diagonal lines in named entities that tokens prefer to attend to themselves. While for ERNIE-Gram, there are bright blocks over named entities that tokens attend to most of tokens in the same entity adequately to construct tight representation.

\begin{table}
\begin{tabular}{l|c c c c c c c} \hline \hline  & **MNLI** & **SST-2** & **SQuAD1.1** & **SQuAD2.0** \\  & m & mm & Acc & EM & F1 & EM & F1 \\ \hline MPNet Song et al. (2020) & 86.2 & - & **94.0** & 85.0 & 91.4 & 80.5 & 83.3 \\ \(-\)relative position bias & 85.6 & - & 93.6 & 84.0 & 90.3 & 79.5 & 82.2 \\ UniLMv2 Bao et al. (2020) & 86.1 & 86.1 & 93.2 & 85.6 & 92.0 & 80.9 & 83.6 \\ \(-\)relative position bias & 85.6 & 85.5 & 93.0 & 85.0 & 91.5 & 78.9 & 81.8 \\ \hline ERNIE-Gram & **87.1** & **87.1** & 93.2 & **86.2** & **92.3** & **82.1** & **84.8** \\ \(-\)relative position bias & 86.5 & 86.4 & 93.2 & 85.2 & 91.7 & 80.8 & 84.0 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Ablation study on relative position bias Raffel et al. (2020) for ERNIE-Gram and previous strong pre-trained models like MPNet and UniLMv2.