SlowGuess/NAACL_2021_OCR huggingface dataset:
Hyperparameters	Base-scale	Large-scale	Chinese
Layers	12
Hidden size	768
Attention heads	12
Training steps	1M	500K	3M
Batch size	256	5112	256
Learning rate	1e-4	4e-4	1e-4
Warmup steps	10,000	24,000	4,000
Adam β	(0.9, 0.99)	(0.9, 0.98)	(0.9, 0.99)
Adam ε	1e-6
Learning rate schedule	Linear
Weight decay	0.01
Dropout	0.1
GPUs (Nvidia V100)	16	64	32
Table 6: Hyperparameters used for pre-training on different text corpora.
B Hyperparameters for Fine-Tuning
The hyperparameters for each tasks are searched on the development sets according to the average score of ten runs with different random seeds.
B.1 GLUE benchmark
The fine-tuning hyper-parameters for GLUE benchmark (Wang et al., 2018) are presented in Table 7.
Hyperparameters	GLUE
Batch size	{16, 32}
Learning rate	{5e-5, 1e-4, 1.5e-4}
Epochs	3 for MNLI and {10, 15} for others
LR schedule	Linear
Layerwise LR decay	0.8
Warmup proportion	0.1
Weight decay	0.01
Table 7: Hyperparameters used for fine-tuning on the GLUE benchmark.
B.2 SQuAD benchmark and RACE dataset
The fine-tuning hyper-parameters for SQuAD (Rajpurkar et al., 2016; Rajpurkar et al., 2018) and RACE (Lai et al., 2017) are presented in Table 8.
Hyperparameters	SQuAD	RACE
Batch size	48	32
Learning rate	{1e-4, 1.5e-4, 2e-4}	{8e-5, 1e-4}
Epochs	{2, 4}	{4, 5}
LR schedule	Linear	Linear
Layerwise LR decay	0.8	0.8
Warmup proportion	0.1	0.1
Weight decay	0.0	0.01
Table 8: Hyperparameters used for fine-tuning on the SQuAD benchmark and RACE dataset.
B.3 Text Classification tasks
Table 9 lists the fine-tuning hyper-parameters for IMDb (Maas et al., 2011) and AG'news (Zhang et al., 2015) datasets. To process texts with a length larger than 512, we follow Sun et al., 2019a to select the first 512 tokens to perform fine-tuning.
Hyperparameters	IMDb	AG’news
Batch size	32
Learning rate	{5e-5, 1e-4, 1.5e-4}
Epochs	3
LR schedule	Linear
Layerwise LR decay	0.8
Warmup proportion	0.1
Weight decay	0.01
Table 9: Hyperparameters used for fine-tuning on IMDb and AG'news.
B.4 Chinese NLU tasks
The fine-tuning hyperparameters for Chinese NLU tasks including XNLI (Conneau et al., 2018), LCQMC (Liu et al., 2018), DRCD (Shao et al., 2018), DuReader (He et al., 2018), CMRC2018 and MSRA-NER (Gao et al., 2005) are presented in Table 10.
Tasks	Batch size	Learning rate	Epoch	Dropout
XNLI	256	1.5e-4	3	0.1
LCQMC	32	4e-5	2	0.1
CMRC2018	64	1.5e-4	5	0.2
DuReader	64	1.5e-4	5	0.1
DRCD	64	1.5e-4	3	0.1
MSRA-NER	16	1.5e-4	10	0.1
Table 10: Hyperparameters used for fine-tuning on Chinese NLU tasks. Note that all tasks use the layerwise lr decay with decay rate 0.8.
C Detailed Results for Ablation Studies
We present the detailed results on GLUE benchmark for ablation studies in this section. The results on different MLM objectives and sizes of  $n$ -gram lexicon are presented in Table 11. The detailed
Models	Size of Lexicon	MNLI Acc	QNLI Acc	QQP Acc	SST-2 Acc	CoLA MCC	MRPC Acc	RTE Acc	STS-B PCC	GLUE Avg	SQuAD1.1 EM	F1	SQuAD2.0 EM	F1
BERTReimplementation	0K	84.9	91.8	91.3	92.9	58.8	88.1	69.7	88.6	83.4	83.4	90.2	76.4	79.2
Contiguously MLM	100K	85.4	92.3	91.3	92.9	60.4	88.7	72.6	89.6	84.1	84.2	90.8	78.4	81.5
200K	85.3	92.0	91.5	92.7	59.3	89.0	71.5	89.5	83.9	84.2	90.9	78.3	81.3
300K	85.1	92.1	91.3	92.8	59.3	88.6	73.3	89.5	84.0	83.9	90.7	78.5	81.4
400K	85.0	92.0	91.3	93.1	58.3	89.2	71.8	89.1	83.7	83.9	90.7	78.0	81.1
Explicitly N-gram MLM	100K	85.3	92.2	91.4	92.9	62.3	88.6	72.5	88.0	84.2	84.2	90.9	78.6	81.4
200K	85.4	92.3	91.3	92.8	62.1	88.4	74.5	88.6	84.4	84.5	91.3	78.9	81.9
300K	85.7	92.3	91.3	92.9	62.6	88.7	75.8	89.4	84.8	84.7	91.2	79.5	82.4
400K	85.3	92.2	91.4	92.9	61.3	88.5	73.2	89.3	84.3	84.6	91.3	79.0	81.7
Table 11: Results on the development set of the GLUE and SQuAD benchmarks with different MLM objectives and diverse sizes of  $n$ -gram lexicon.
#	Models	MNLI	QNLI	QQP	SST-2	CoLA	MRPC	RTE	STS-B	GLUE
m mm	Acc	Acc	Acc	MCC	Acc	Acc	PCC	Avg
#1	ERNIE-GramBASE	87.1	87.1	92.8	91.8	93.2	68.5	90.3	79.4	90.4
#2	#1 - relative position bias	86.5	86.4	92.5	91.6	93.2	68.1	90.3	79.4	90.6
#3	#2 - comprehensive n-gram prediction (CNP)	86.2	86.2	92.4	91.7	92.7	65.5	90.0	78.7	90.5
#4	#2 - enhanced n-gram relation modeling (ENRM)	85.7	85.8	92.6	91.2	93.5	64.8	88.9	76.9	90.0
#5	#4 - comprehensive n-gram prediction (CNP)	85.6	85.7	92.3	91.3	92.9	62.6	88.7	75.8	89.4
Table 12: Comparisons between several ERNIE-Gram variants on GLUE benchmark. All the listed models are pre-trained following the same settings of  $\mathrm{BERT}_{\mathrm{BASE}}$  (Devlin et al., 2019).
Figure 7: (a-c) Mean attention scores in the last self-attention layer. Texts in green, orange, red and blue boxes are named entities standing for organizations, locations, person and miscellaneous respectively.
results on ERNIE-Gram variants to verify the effectiveness of comprehensive  $n$ -gram prediction and enhanced  $n$ -gram relation modeling mechanisms are presented in Table 12. Results of ablation study on relative position bias (Raffel et al., 2020) are presented in Table 13.
D More cases on CoNLL2003 Dataset
We visualize the attention patterns of three supplementary cases from CoNLL2003 named entity recognition dataset (Sang and De Meulder, 2003) to compare the performance of ERNIE-Gram, contiguously MLM and BERT (lowercased), as shown
Models	MNLI	SST-2 Acc	SQuAD1.1	SQuAD2.0
m	mm	EM	F1	EM	F1
MPNet (Song et al., 2020)	86.2	-	94.0	85.0	91.4	80.5	83.3
–relative position bias	85.6	-	93.6	84.0	90.3	79.5	82.2
UNILMv2 (Bao et al., 2020)	86.1	86.1	93.2	85.6	92.0	80.9	83.6
–relative position bias	85.6	85.5	93.0	85.0	91.5	78.9	81.8
ERNIE-Gram	87.1	87.1	93.2	86.2	92.3	82.1	84.8
–relative position bias	86.5	86.4	93.2	85.2	91.7	80.8	84.0
Table 13: Ablation study on relative position bias (Raffel et al., 2020) for ERNIE-Gram and previous strong pretrained models like MPNet and UNILMv2.
in Figure 7. For contiguously MLM, there are clear diagonal lines in named entities that tokens prefer to attend to themselves. While for ERNIE-Gram, there are bright blocks over named entities that tokens attend to most of tokens in the same entity adequately to construct tight representation.