SlowGuess/NAACL_2021_OCR huggingface dataset:
Similar to other pre-trained language models (Strubell et al., 2019), the study of Lattice-BERT inevitably involves lots of computing time/power. The incorporation of multi-granularity tokens generally introduces more computational costs than character-level PLMs (Diao et al., 2020; Zhang and Li, 2020). In this section, we elaborate our efforts in reducing the energy costs as well as discuss the energy comparison with the previous state-of-the-art multi-granularity Chinese PLM, Zhang and Li (2020, AMBERT).
Efforts in Reducing the Energy Costs. In this work, our efforts in reducing the energy costs can be summarized in two folds, i.e., (1) adopting more efficient experiment procedures, and (2) reporting performances of lite-size models to encourage the followers to compare in lightweight architectures.
From the perspective of experiment procedures, we adopt the mixed-precision arithmetic methods to speed up pre-training. We also utilize the two-phase pre-training procedures (Devlin et al., 2019, BERT), where the model processes smaller sequence lengths in  $90\%$  steps (See Appendix C.1). In downstream tasks, we search the learning rates with relatively larger strides with heuristic strategies to avoid pointless attempts (See Appendix C.2). We adopt ablation studies in lite-size. The lightweight architecture with six-layer models and 128-character inputs could save much power comparing with training the full-length base-size models repeatedly.
From the perspective of reporting strategies, we report the performance of base-size models together with lite-size models. As far as we know, all previous Chinese PLMs only report base- or large-size settings (Wei et al., 2019; Diao et al., 2020; Sun et al., 2020; Cui et al., 2020; Zhang and Li, 2020). Thus, the followers have to implement at least a 12-layer pre-training model to make a fair comparison. This reporting of lite-size performances facilitate the followers to fast validate in lite-settings by directly comparing with our LBERT-lite and BERT-our-lite. For the text classification and sequence labeling tasks, we use the model trained with 128-sequence length. Thus the followers could make a fair comparison with our methods without pre-training a full-length model (See Appendix C.2). Specifically, the 128-length LBERT-lite model (33M parameters) only costs
3.8 days pre-training with  $8 \times$  NVIDIA V100 16G cards,  $1/4$  of the original base-size Google-BERT model. In conclusion, by reporting the performances of lite-size models, we encourage followers to compare with our models with less computational costs.
Energy Comparison. We make the energy comparison between LBERT and the previous state-of-the-art multi-granularity Chinese PLM, Zhang and Li (2020, AMBERT), from the following perspectives: (1) How many computational costs are introduced to process the additional coarse-grained word-level inputs in pre-training. (2) How many additional parameters are introduced by the word-level vocabulary comparing with vanilla fine-grained Chinese PLMs.
Specifically, from the view of additional computational costs, comparing with the character-level BERT with other settings fixed, AMBERT (Zhang and Li, 2020) introduces  $100\%$  more computational costs by adopting two encoders to deal with word- and character-level inputs separately. Our proposed LBERT-base adopts a uniform architecture to consume words and characters from word-lattices simultaneously. Therefore, LBERT only introduces  $48\%$  more computational costs comparing with the corresponding character-level baselines (See Appendix C.1). On the other side, from the view of parameter scales, AMBERT has 176M parameters, where additional 68M parameters are introduced by the embedding of word-level tokens. With the embedding decomposition trick, LBERT only introduces 10M parameters by the word-level units in vocabulary. As a result, LBERT-base has 100M parameters, comparable to the character-level BERTour (90M) and RoBERTa (102M) models, and much smaller than AMBERT (176M). In conclusion, with fewer parameters and smaller computational costs, LBERT is more efficient than AMBERT.
In the Tasks section, we list eleven downstream tasks. This note presents several basic statistics, including the sentence length in characters, numbers of tokens in lattices, and dataset scales, which is shown in Table 5. For the cloze task, ChID, we focus on a small context of each blank in fine-tuning,
<sup>8</sup>We estimate the pre-training cost of Google-BERT according to Table-3 in Strubell et al. (2019).
thus, we do not report the statistics of the whole passages.
We show the number of training instances and the training/development/test split in Table 5 as well. Specifically, we count the number of questions/blanks in MRC tasks, the number of words/entities for CWS and NER tasks, and the numbers of sentences/sentence pairs in the CLUE classification tasks. Following the previous effort (Diao et al., 2020), we use the test set for the validation of the MSR-CWS dataset.
C Implement Details
Our code is available at https://github.com/alibaba/pretrained-language-models/LatticeBERT. Here, we specify some issues in pre-training and fine-tuning.
C.1 Pre-training Details.
Lattice-BERT models. We train Lattice-BERT with Adam optimizer (Kingma and Ba, 2015). The hyper-parameters of the Lattice-BERT models are shown in Table 4. The ratio of masked tokens in language modeling is  $15\%$ . The embedding sizes are different from hidden sizes because we factorize embedding matrices following Lan et al. (2020, ALBERT) to reduce the additional parameters introduced by word-level inputs. For the base-size models, the numbers of parameters in the character-level BERT-our and LBERT are 90M and 100M, respectively, less than the corresponding RoBERTa-base (102M) and AMBERT-base (176M). The numbers of parameters for lite-size models are 23M and 33M for BERT-our and LBERT models, respectively.
To speed up the pre-training, we adopt the two-phase procedures of Devlin et al. (2019, BERT), which first pre-trains the model with a sequence length of 128 characters per instance for  $90\%$  steps, and then trains the rest  $10\%$  steps with a sequence length of 512 characters per instance. We base our code on the optimized version of BERT released by NVIDIA by leveraging the mixed-precision arithmetic and the multi-GPU techniques. For the base-size LBERT models, the two pre-training phases take about 9.2 and 6.7 days, separately, with NVIDIA  $8 \times \mathrm{V}100$  16G cards. For lite-size models, the time consumption for the two phases takes 4.0 and 3.1 days. We discuss the energy resources in the Ethical Considerations section.
To make fair comparison, we expand the maxi
Hyper-param	LBERT-base	LBERT-lite
Number of Layers	12	6
Hidden Size	768	512
Embedding Size	128	128
FFN Inner Hidden Size	3072	2048
Attention Heads	12	8
Attention Head Size	64	64
Dropout	0.1	0.1
Attention Dropout	0.1	0.1
Activation Func.	GELU	GELU
Warmup Steps	5K	5K
Peak Learning Rate	6e-4	6e-4
Batch Size	8192	8192
Max Steps	100K	100K
Learning Rate Decay	Linear	Linear
Adam ε	1e-6	1e-6
Adam β1	0.9	0.9
Adam β2	0.999	0.999
Table 4: Hyper-parameters for pre-training.
mum size of input tokens in pre-training of LBERT to process the additional word-level lattice tokens, following previous multi-granularity PLMs in Chinese (Diao et al., 2020; Zhang and Li, 2020).9 Particularly, we ensure each training instance of character-level baselines (i.e., BERT-our) and the Lattice-BERT models contains the same number of character-level tokens, which is 128 and 512 for the first and second pre-training phase, respectively. As a result, from the view of the pre-training data creation, fixing the corpus size, the numbers of instances per epoch are the same between character-level BERT-our and LBERT in pre-training given the same corpus. Thus, in the same steps, we can train BERT-our and LBERT for the same number of epochs. In another word, BERT-our and LBERT process the same size of corpus with the same training steps.
In practice, based on the statistics on the pretraining corpus, we expand the input size by  $35\%$ . For example, the instances in the first pre-training phase of LBERT have 173 lattice tokens, where 128 Chinese characters are expected to be contained. For the second pre-training phase of LBERT, the input token size is 692.
However, via an empirical estimation, this extension makes the pre-training procedure of the Lattice-BERT models cost  $48\%$  more computational resources than the corresponding BERT-our settings.[10] This increment of time complexity is
Under the context of Lattice-BERT, the input token length or the max sequence length means the maximum number of lattice tokens per instance that the Lattice-BERT consumes.
10Theoretically, the time complexity of the attention layers to the input length is  $O\left(n^{2}\right)$ , and that of the fully connected
Task	Sentence Length	Dataset Scale	Hyper-parameters
tp995-C	tp995-T	Avg-C	Avg-T	Train	Dev.	Test	max-len.	#ep.	bs
CLUE Text Classification tasks
CMNLI	136	179	56.1	74.1	391.8K	12.2K	13.9K	256	4	16
TNEWS	104	133	41.2	53.0	53.4K	10.0K	10.0K	256	5	16
IFLY.	1028	1316	291.1	384.9	12.1K	2.6K	2.6K	512	5	8
AFQMC	78	106	29.7	39.6	34.3K	4.3K	3.9K	256	5	16
WSC.	147	187	71.7	91.8	1.2K	0.3K	0.3K	256	10	16
CLS	812	949	298.3	384.4	20.0K	3.0K	3.0K	512	5	8
CLUE MRC tasks
CMRC	991	1301	526.7	662.9	10.1K	1.0K	3.2K	512	5	8
ChID	-	-	-	-	577.2K	23.0K	23.0K	64	2	24
C3	1088	1479	224.9	329.1	11.9K	4.3K	3.9K	512	8	12
Sequence Labeling tasks
MSR-CWS	171	238	48.6	66.6	2.37M	-	106.9K	512	5	8
MSRA-NER	174	247	49.0	67.1	34.0K	3.8K	7.7K	512	10	8
Table 5: Statistics of the datasets and hyper-parameters for downstream tasks. tp995-C and tp995-T are the top  $99.5\%$  length of the input sequence in characters and the top  $99.5\%$  number of lattice tokens, respectively. Avg-C and Avg-T are the average numbers of the characters and the lattice tokens per instance. max-len., #ep., and bs represent the max sequence length of the BERT model (max lattice tokens that Lattice-BERT models consume), the number of epochs, and the batch size, respectively.
much lower than other multi-granularity PLMs like AMBERT (Zhang and Li, 2020), which introduces  $100\%$  additional computational resources comparing with the corresponding character-level setting. We further compare the performances of BERTour and LBERT in the downstream tasks under the same pre-training computational costs in Appendix D, where LBERT still outperforms BERTour by a large margin.
BERT-our Baselines. The pre-training settings of BERT-our are almost the same as our proposed LBERT. These settings provide a fair comparison to support the argument that the improvements are attributed to better utilizing the multi-granularity information in word lattices. The differences between BERT-our and LBERT in model architectures are: (1) BERT-our use the same vocabulary and tokenization functions as the Chinese version of Google BERT (Devlin et al., 2019), and taking character level inputs, while LBERT takes word lattices as inputs. (2) BERT-our has shorter input sequences to ensure the Lattice-BERT and BERT-our training for the same epochs in the same steps. (3) LBERT incorporates the positional relations between lattice tokens in attention layers, which is
layers is  $O(n)$ . Thus, the overall increment of time cost is between  $35\%$  and  $82\%$  ( $1.35 \times 1.35 \approx 1.82$ ). To empirically estimate the complexity, we use the ratio of pre-training time between taking 128 and 173 input lengths separately, under the setting of lite-size BERT-our.
not suitable for BERT-our with sequential inputs. The other components in lattice position attention, including the absolute position attention and the distance information, are adopted in BERT-our as well. (4) LBERT adopts whole segment prediction to avoid potential leakage, while BERT-our adopts the whole word masking (Cui et al., 2019, wwm) trick to utilize the word level information.
Vocabulary. The vocabulary used to construct lattices is a superset of the vocabulary in BERT-our and the Chinese version of Google-BERT (Devlin et al., 2019). Particular, the vocabulary of LBERT consists of 21K tokens (including characters and word pieces) from the vanilla vocabulary of BERT-our and 81K additional high-frequency words from the pre-training corpus. To obtain the high-frequency words, we randomly sample  $10\%$  of the pre-training corpus, running an in-house built tokenizer, and counting the word frequency after tokenization. All the English tokens in this vocabulary are lower-cased, which means LBERT can be seen as an uncased model. However, LBERT is a Chinese PLM, thus, only a few English tokens exist in pre-training and fine-tuning.
C.2 Fine-tuning Details.
Hyper-Parameters and Settings We tune the learning rates in (8e-6, 1e-5, 1.5e-5, 2e-5, 3e-5, 5e-5, 8e-5, 1e-4, 1.5e-4) on the development sets
dataset	best lr.	avg.	std.	best lr.	avg.	std.	best lr.	avg.	std.	best lr.	avg.	std.
BERT-our-base	BERT-our-lite	LBERT-base	LBERT-lite
CMNLI	1.5e-5	80.6	0.4	8.0e-5	77.6	0.2	1.0e-5	81.6	0.2	3.0e-5	79.2	0.2
TNEWS	1.5e-5	66.7	0.2	5.0e-5	65.4	0.2	3.0e-5	67.8	0.3	3.0e-5	67.2	0.2
IFLY.	3.0e-5	60.8	0.9	1.0e-4	59.3	0.2	2.0e-5	61.7	0.1	8.0e-5	60.0	0.3
AFQMC	2.0e-5	74.3	0.5	5.0e-5	72.3	0.6	1.0e-5	74.6	0.3	3.0e-5	73.1	0.4
WSC.	5.0e-5	81.5	1.3	1.5e-4	66.3	1.9	2.0e-5	85.5	1.1	8.0e-5	75.3	1.3
CSL	1.5e-5	81.4	0.3	1.0e-4	78.5	0.7	1.0e-5	83.3	0.2	1.0e-4	81.4	0.4
CMRC/F1	5.0e-5	86.9	0.2	1.5e-4	83.7	0.4	3.0e-5	87.4	0.5	8.0e-5	84.9	0.2
CMRC/EM	-	67.6	0.4	-	62.6	0.4	-	68.3	0.6	-	64.9	0.6
ChID	1.5e-5	84.6	-	5.0e-5	79.0	-	1.0e-5	86.4	-	2.0e-5	81.7	-
C3	5.0e-5	69.1	0.3	5.0e-5	61.9	0.6	5.0e-5	72.7	0.4	8.0e-5	63.0	0.5
MSRA-NER/EF1	2.0e-5	94.6	0.8	5.0e-5	92.8	0.2	5.0e-5	95.6	0.2	5.0e-5	94.1	0.1
MSRA-NER/LF1	-	96.5	0.5	-	95.5	0.1	-	97.1	0.1	-	96.2	0.1
MSR-CWS	3.0e-5	98.4	0.0	8.0e-5	98.1	0.0	8.0e-5	98.6	0.0	8.0e-5	98.4	0.0
Average	-	79.1	-	-	75.1	-	-	80.6	-	-	77.2	-
Table 6: The full experimental results, with best learning rates (best lr.), corresponding average (avg.) and standard deviation (std.) for five runs. For the NER task, EF1 and LF1 are entity-level F1 scores and label-level F1 scores, respectively. We average the multiple metrics within each task (e.g., CMRC) before averaging over tasks.
with the other hyper-parameters, including max sequence length of the BERT model (max-len), the number of epochs (ep.), and the batch size (batch), fixed. The hyper-parameters in fine-tuning are shown in Table 5.
Specifically, for the sequence labeling tasks, including CWS and NER, we choose the best learning rates based on the label-level F1 scores on the development sets. In practice, we adopt heuristic strategies to avoid pointless attempts. For example, if a model performs monotonic increasing in a down-stream task with the learning rates of 2e-5, 3e-5, and 5e-5, we will not try the learning rates lower than 2e-5. The best learning rates, together with the average and the standard deviation of the scores for five runs, are shown in Table 6. We report the average and deviation scores on development sets for CLUE tasks and those on the test sets for sequence labeling tasks (NER & CWS). In the ChID task, we run the models once with the best learning rates for efficiency, because its training set is too large.
In text classification and sequence labeling tasks, the max sequence length are the same between Lattice-BERT and the character-level BERT-our, thus, their time complexities are the same in finetuning. However, in MRC tasks, where the documents are relatively longer, it is essential to read the long passages as complete as possible. Thus, to make fair comparisons, we expand the size of input lattices in LBERT settings to ensure the numbers of Chinese characters per instances are the same
between LBERT and BERT-our settings.
Selection of Pre-training Models. We adopt the PLMs after the first pre-training phase, i.e., pretraining with the 128-character input size, for the text classification, sequence labeling, and ChID tasks. The details of the two-phase pre-training procedure refer to Appendix C.1. Pilot experiments demonstrate that, for these tasks, PLMs with shorter pre-training lengths perform comparably with the full-length versions. We think this is because the sentence lengths are relatively small in these tasks except for IFLY, the long text classification task, where the first few sentences are more crucial for the predictions. This strategy makes our explorations much more efficient. We adopt the full-length pre-training versions for the CMRC and  $\mathbf{C}^3$  tasks.
Detailed Implementation. For the implementation of fine-tuning tasks, we adopt the simplest methods following the CLUE official examples. $^{11}$  For example, we adopt logistic regressions over the [CLS] tokens for classifications and softmax regressions over the [CLS] tokens of all options for multiple choices in  $\mathbf{C}^3$  and ChID. To deal with the long documents in MRC tasks, we truncate document to at most 512 characters in  $\mathbf{C}^3$ . For CMRC, we split the document into segments of at most 512 characters with the stride of 128 characters.
11https://github.com/CLUEbenchmark/ CLUE/tree/master/baselines/models_ pytorch
CLUE-Classification	Seq. Lab.	avg.
NLI	TC	SPM	CoRE	KwRE	avg.	CWS	NER
Para.	Cmplx.	Epoch.	CMNLI	TNEWS	IFLY.	AFQMC	WSC.	CSL	MSR	MSRA
BERT-our	23M	T	N	77.6	65.4	59.3	72.3	66.3	78.5	69.9	98.1	95.5	76.7
+seql	23M	1.48T	1.35N	77.6	65.3	59.3	72.4	68.4	80.1	70.5	98.1	95.6	77.1
+seql-EmbDe	31M	1.48T	1.35N	78.4	65.8	59.7	71.6	71.4	79.9	71.1	98.3	95.8	77.6
LBERT	33M	1.48T	N	79.2	67.2	60.0	73.1	75.3	81.4	72.7	98.4	96.2	78.9
Table 7: The performances on development sets in lite-size settings. Para., Cmplx., and Epoch. are the parameter sizes, time complexities, and the training epochs, respectively. We mark the time complexity and the training epochs of the BERT-our as T and N.
Then the selected spans from each segment are aggregated to find the final answer to the question. Furthermore, for ChID, the contexts no far than 32 characters to the masked idiom are incorporated.
We conduct data augmentation for TNEWS and CSL tasks. In TNEWS, we use both keywords and titles for classifications. Moreover, for the CSL task, we concatenate all the keywords. These augmentations are also conducted in the previous works, which is either explicitly mentioned in the paper (Zhang and Li, 2020, AMBERT), or can be inferred from the performances (Wei et al., 2019, NEZHA).
D Further Experiments
Mentioned in §3.4 and Appendix C.1, we expand the maximum size of input tokens in pre-training of LBERT to process the additional word-level lattice tokens, following previous multi-granularity PLMs in Chinese (Diao et al., 2020; Zhang and Li, 2020). We expand the input token length for the pre-training of LBERT by  $35\%$ , which makes the LBERT and BERT-our have the same training epochs when the training steps are equal, but introducing  $48\%$  more computational resources (discussed in Appendix C). Meanwhile, the additional word-level tokens in the vocabulary of LBERT introduce  $11\%$  and  $43\%$  more parameters in the embedding matrix for the base-size and lite-size settings, respectively. To illustrate the gains of LBERT attribute to the incorporation of lattices instead of additional computations or parameters, we investigate the BERT-our with longer input sequences in pre-training and the BERT-our model without the embedding decomposition trick, which has more parameters in embedding.
To reduce computational costs, we base the experiments on the first pre-training phase and the lite-size setting (see Appendix C.1 for details). Since we do not conduct full-length pre-training, we use CLUE classification and sequence labeling tasks. We report the average scores on development sets over five runs. The compared system is listed below:
+seql is the character-level BERT-our taking the same sequence length (i.e., 173) as LBERT. Comparing with LBERT, this setting results in the same time consumption, together with  $35\%$  more character-level tokens per instance comparing to LBERT models, which results in  $35\%$  more corpus/epochs to process with the same training steps. -EmbDe is the character-level BERT-our without embedding decomposition tricks, i.e., the size of embedding matrix is the same as the hidden size, 512. In the lite-size setting, wihout embedding decomposition, the character-level BERT-our have 8M more parameters, and the total parameter scale is 31M, comparable to that of LBERT (33M).
Results. As we can see in Table 7, LBERT remarkably outperforms the BERT-our settings, even after the extension of time complexity and embedding parameters. Specifically, on CLUE classification tasks, LBERT outperforms BERT-our+seql and BERT-our+seql-EmbDe by  $2.2\%$  and  $1.6\%$ , respectively. On sequence labeling tasks, the improvements of +seql and -EmbDe are also marginal. These results demonstrate that the performance gains of LBERT attribute to the usage of multi-granularity representations in word lattices instead of the additional pre-training time complexities and embedding parameters.
12The TNEWS performance in NEZHA is  $67.4\%$ . However, without data augmentation, even the large models like ALBERT-xxlarge and RoBERTa-wwm-large could not obtain an accuracy more than  $60\%$  (Xu et al., 2020).