SlowGuess/NAACL_2021_OCR huggingface dataset:
Yuxuan Lai $^{1,2,*}$ , Yijia Liu $^{3}$ , Yansong Feng $^{1,2,\dagger}$ , Songfang Huang $^{3}$  and Dongyan Zhao $^{1,2}$
$^{1}$ Wangxuan Institute of Computer Technology, Peking University, China  $^{2}$ The MOE Key Laboratory of Computational Linguistics, Peking University, China
{erutan, fengyansong, zhaody}@pku.edu.cn
{yanshan.lyj, songfang.hsf}@alibaba-inc.com
Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese — Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sentence in a multi-granularity manner. Specifically, we construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We design a lattice position attention mechanism to exploit the lattice structures in self-attention layers. We further propose a masked segment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks. Experiments on 11 Chinese natural language understanding tasks show that our model can bring an average increase of  $1.5\%$  under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. Further analysis shows that Lattice-BERT can harness the lattice structures, and the improvement comes from the exploration of redundant information and multi-granularity representations. $^{1}$
Pre-trained Language Models (PLMs) have achieved promising results in many Chinese Natural Language Understanding (NLU) tasks (Cui et al., 2019; Liu et al., 2020; Sun et al., 2020). These models take a sequence of fine-grained units — Chinese characters — as the input, following the English PLMs' practice (Devlin et al., 2019, BERT).
Figure 1: An illustration of the word lattice for sentence  
研究生活很充实/Research Life is very fulfilling.
However, the meanings of many Chinese words cannot be fully understood through direct compositions of their characters' meanings. For example,老板/boss does not mean '老/elder 板/board.' The importance of word-level inputs in Chinese has been addressed in different tasks, including relation classification (Li et al., 2019), short text matching (Lai et al., 2019; Chen et al., 2020; Lyu et al., 2021), trigger detection (Lin et al., 2018), and named entity recognition (Zhang and Yang, 2018; Gui et al., 2019; Li et al., 2020a). The coarse-grained inputs benefit these tasks by introducing word-level semantics with multi-granularity representations, which is potentially complementary in character-level Chinese PLMs.
In this work, we discuss how to pre-train a Chinese PLM over a word lattice structure to exploit multi-granularity inputs. We argue that by incorporating the coarse-grained units into PLM, models could learn to utilize the multi-granularity information for downstream tasks. Specifically, we organize characters and words in sentences as word lattices (see Figure 1), which enable the models to explore the words from all possible word segmentation results.
However, it is not straightforward to learn a BERT-like PLM over the word lattices. The major challenges are two-folded. Firstly, BERT's original input is a sequence of characters ordered by their positions, making it difficult to consume the word lattices and preserve the positional relation
2For clarity, we use中文/English translation to represent an example in Chinese with its translation followed by.
*Work done during an internship at Alibaba DAMO Academy.
1Our code will be available at https://github. com/alibaba/pretrained-language-models/ LatticeBERT.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1716-1731 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: An illustration of our pre-training framework, Lattice-BERT.
ship between multi-granularity units. Secondly, the conventional masked language modeling (MLM) task may make the word-lattice based PLMs learn unexpected tricks. The reason is that such a word lattice naturally introduces redundancy, that is, one character can be contained in multiple text units. In MLM, models may refer to the other text units overlapping with the randomly masked one instead of the real context, which brings information leakages.
To address these challenges, we propose a Lattice-based Bidirectional Encoder Representation from Transformers (Lattice-BERT). Specifically, we design a lattice position attention (LPA) to help the transformers directly exploit positional relationship and distances between text units in lattices. Moreover, we propose a masked segment prediction (MSP) task to avoid the potential leakage between overlapping text units in language modeling. With LPA and MSP, the Lattice-BERT could harness the multi-granularity structures in lattices, thus, directly utilize the lattice structures to aggregate the coarse-grained word information to benefit various downstream tasks.
We evaluate our model on 11 Chinese NLU tasks in various paradigms, including the CLUE benchmarks (Xu et al., 2020) as well as two sequence labeling tasks. Compared with the baseline that only takes characters as inputs, Lattice-BERT brings an average increase of  $1.5\%$  and  $2.0\%$  under the settings of 12 and 6 layers, respectively. The 12-layer Lattice-BERT model beats all other base-size models on CLUE benchmarks. Moreover, we show that Lattice-BERT can harness the multi-granularity in
puts and utilize word-level semantics to outperform vanilla fine-grained PLMs.
Our contributions can be summarized as 1) We propose Lattice-BERT to leverage multi-granularity representations from word lattices in Chinese PLMs. 2) We design lattice position attention and masked segment prediction to facilitate Chinese PLMs to exploit the lattice structures. 3) Lattice-BERT brings remarkable improvements on 11 Chinese tasks and achieves new state of the arts among base-size models at the CLUE benchmarks.
2 Lattice-BERT
This section, we detail the implementation of Lattice-BERT, and its overall framework is presented in Figure 2.
2.1 Preliminary: BERT
BERT (Devlin et al., 2019, Bidirectional Encoder Representations from Transformers) is a pre-trained language model comprising a stack of multi-head self-attention layers and fully connected layers. For each head in the  $l_{th}$  multi-head self-attention layer, the output matrix  $\mathbf{H}^{\mathrm{out},l} = \left\{\mathbf{h}_1^{\mathrm{out},l},\mathbf{h}_2^{\mathrm{out},l},\dots ,\mathbf{h}_n^{\mathrm{out},l}\right\} \in \mathbb{R}^{n\times d_k}$  satisfies:
$$
\mathbf {h} _ {i} ^ {\text {o u t}, l} = \sum_ {j = 1} ^ {n} \left(\frac {\exp \alpha_ {i j} ^ {l}}{\sum_ {j ^ {\prime}} \exp \alpha_ {i j ^ {\prime}} ^ {l}} \mathbf {h} _ {j} ^ {\text {i n}, l} \mathbf {W} ^ {v, l}\right)
$$
$$
\alpha_ {i j} ^ {l} = \frac {1}{\sqrt {2 d _ {k}}} \left(\mathbf {h} _ {i} ^ {\text {i n}, l} \mathbf {W} ^ {q, l}\right) \left(\mathbf {h} _ {j} ^ {\text {i n}, l} \mathbf {W} ^ {k, l}\right) ^ {T} \tag {1}
$$
where  $\mathbf{H}^{\mathrm{in},l} = \left\{\mathbf{h}_1^{\mathrm{in},l},\mathbf{h}_2^{\mathrm{in},l},\dots ,\mathbf{h}_n^{\mathrm{in},l}\right\} \in \mathbb{R}^{n\times d_h}$  is the input matrix, and  $\mathbf{W}^{q,l},\mathbf{W}^{k,l},\mathbf{W}^{v,l}\in$
3https://www.cluebenchmarks.com/rank.html, until Oct. 31st, 2020.
$\mathbb{R}^{d_h\times d_k}$  are learnable parameters.  $n$  and  $d_{h}$  are sequence length and hidden size, and the attention size  $d_{k} = d_{h} / n_{h}$ , where  $n_h$  is the number of attention heads.
To capture the sequential features in languages, previous PLMs adopt position embedding in either input representations (Devlin et al., 2019; Lan et al., 2020) or attention weights (Yang et al., 2019; Wei et al., 2019; Ke et al., 2020). For the input-level position embedding, the inputs of the first layer are  $\widetilde{\mathbf{h}}_i^{\mathrm{in},0} = \mathbf{h}_i^{\mathrm{in},0} + \mathbf{P}_i$ , where  $\mathbf{P}_i$  is the embedding of the  $i_{th}$  position. The other works incorporate position information in attention weights, i.e.,  $\widetilde{\alpha}_{ij}^{l} = \alpha_{ij}^{l} + f(i,j)$ , where  $f$  is a function of the position pair  $(i,j)$ .
The BERT model is pre-trained on an unlabeled corpus with reconstruction losses, i.e., Masked Language Modeling (MLM) and Next Sentence Prediction (NSP), and then fine-tuned on downstream tasks to solve specific NLU tasks. Readers could refer to Devlin et al. (2019) for details.
2.2 Multi-granularity Inputs: Word Lattices
We adopt a word lattice to consume all possible segmentation results of a sentence in one PLM. Each segmentation can be a mixture of characters and words. As shown in Figure 1, a word lattice is a directed acyclic graph, where the nodes are positions in the original sentences, and each directed edge represents a character or a plausible word. Word lattices incorporate all words and characters so that models could explicitly exploit the inputs of both granularities, despite some of the words are redundant. In the rest of this work, we use lattice tokens to refer to text units, including the characters and words, contained in lattice graphs.
As shown in Figure 2, we list the lattice tokens in a line and consume these tokens to transformers straightforwardly. However, the challenges of learning PLMs as BERT over the lattice-like inputs include: 1) encoding the lattice tokens while preserving lattice structures; 2) avoiding potential leakage brought by redundant information.
2.3 Interaction: Lattice Position Attention
Since the original BERT is designed for sequence modeling, it is not straightforward for BERT to consume a lattice graph. The word lattices encode not only the character sequences but also nested and overlapping words from different segmentations. To accurately incorporate positional information from lattice graphs into the interactions between
Figure 3: An illustration of the positional relations. Each rectangle represents a lattice token, corresponding to a character span in the original sentence.  $T.1 \sim T.7$  are the target tokens with the seven different position relation to the Source token.
tokens, we extend the attention-level position embedding and propose lattice position attention.
The lattice position attention aggregates the attention score of token representations,  $\alpha_{ij}$  in Eq. 1, with three position related attention terms, encoding the absolute positions, the distance, and the positional relationship, which can be formulated as:
$$
\widetilde {\alpha} _ {i j} = \alpha_ {i j} + \operatorname {a t t} _ {i j} + b _ {i j} + r _ {i j} \tag {2}
$$
The  $\text{att}_{ij}$  in Eq. 2 is the attention weight between the absolute positions:
$$
\mathrm {a t t} _ {i j} = \frac {1}{\sqrt {2 d _ {k}}} \left(\left[ \mathbf {P} _ {s _ {i}} ^ {S}; \mathbf {P} _ {e _ {i}} ^ {E} \right] \mathbf {W} ^ {q}\right) \left(\left[ \mathbf {P} _ {s _ {j}} ^ {S}; \mathbf {P} _ {e _ {j}} ^ {E} \right] \mathbf {W} ^ {k}\right) ^ {T}
$$
$[\cdot ;]$  means the concatenation of vectors.  $\mathbf{W}^q,\mathbf{W}^k\in \mathbb{R}^{2d_e\times d_k}$  are learnable parameters,  $d_{e}$  and  $d_{k}$  are embedding size and attention size.  $s_i,e_i$  are positions of start and end characters of the  $i_{th}$  token. Taking the word research/research in Figure 1 as an example, it starts at the first character and ends at the second one, thus, its  $s_i$  and  $e_i$  are 1 and 2, respectively.  $\mathbf{P}^S$  and  $\mathbf{P}^E$  are learnable position embedding matrices.  $\mathbf{P}_t^S,\mathbf{P}_t^E\in \mathbb{R}^{d_e}$  is the  $t_{th}$  embedding vector of  $\mathbf{P}^S$  or  $\mathbf{P}^E$ . The attij exploit the prior of attention weight between the start and end positions of the token pairs.
The  $b_{ij}$  in Eq. 2 is the attention term for the distance between the  $i_{th}$  and  $j_{th}$  tokens, which consists of four scaling terms considering the combinations of the start and end positions:
$$
b _ {i j} = b _ {s _ {j} - s _ {i}} ^ {s s} + b _ {s _ {j} - e _ {i}} ^ {s e} + b _ {e _ {j} - s _ {i}} ^ {e s} + b _ {e _ {j} - e _ {i}} ^ {e e}
$$
$b_{t}^{ss}$  reflects the attention weight brought by the relative distance  $t$  between the start positions of two tokens. The other terms, i.e.,  $b_{t}^{se}, b_{t}^{es}$ , and  $b_{t}^{ee}$ , have similar meanings. In practice, the distance  $t$  is clipped into  $[-128, 128]$ .
$r_{ij}$  in Eq. 2 is a scaling term represents the positional relation between the  $i_{th}$  and  $j_{th}$  tokens. We
consider seven relations, including (1) self, (2) left and detached, (3) left and overlapped, (4) containing, (5) contained by, (6) right and overlapped, (7) right and detached. Figure 3 shows an illustration of these 7 relations. Formally, for the  $i_{th}$  and  $j_{th}$  tokens, they are overlapped means  $s_i \leq s_j < e_i \leq e_j$  or  $s_j \leq s_i < e_j \leq e_i$ , and if  $e_i < s_j$  or  $e_j < s_i$ , they are detached. If  $s_i \leq s_j \leq e_j \leq e_i$  and  $i \neq j$ , the  $i_{th}$  token contains the  $j_{th}$  token and the  $j_{th}$  token is contained by the  $i_{th}$  token. Intuitively, only two detached tokens can be concurrent in one Chinese word segmentation result. Moreover, the containing relation reflects a sort of lexical hierarchy in the lattices. We think  $r_{ij}$  can explicitly model the positional relations between tokens in lattice graphs.
We argue that the attention scores for distances and token relations capture different aspects of the multi-granularity structures in lattice graphs, thus, meeting the needs of various downstream tasks, such as distance for coreference resolution and positional relation for named entity recognition. With the information of absolute positions, distances, and positional relations, PLMs could accurately exploit the lattice structures in attention layers.
The lattice position attention weights are shared over all layers.  $b_{ij}, r_{ij}, \mathbf{W}^q,$  and  $\mathbf{W}^k$  are diverse in different attention heads to capture diverse attention patterns. We follow Ke et al. (2020) to reset the positional attention scores related to [CLS] tokens, which is the special token as prefix of the input sequences to capture the overall semantics.
2.4 Pre-training Tasks: Masked Segment Prediction
Vanilla BERT is trained to predict the randomly masked tokens in the sentences, i.e., the masked language modeling (MLM). For the case of consuming multi-granularity inputs, the input tokens are redundant which means a character can occur in its character forms and multiple words it belongs to. Directly adopting the randomly masking strategy may simplify the prediction problem in our case because the masked token can be easily guessed via peeking the unmasked tokens overlapping with the masked one. Taking the word research/research in Figure 2 as an example, supposing the masked input is [M]/究/研究, the model will consult research rather than the context to predict the masked token,研.
We investigate this problem and find that the
tokens within a minimal segment of the lattice provide strong clue for the prediction of other tokens. A segment is a connected subgraph of a lattice where no token exists outside the subgraph that overlaps with any token inside the subgraph. To identify these minimal segments, we enumerate the character-level tokens in sentence order, checking if all the word-level tokens which contain this character end at this character. If so, all the tokens containing previous and current characters are considered as a segment, and the next segment starts from the next character, see the example in Figure 2. After the segment detection, we propose a masked segment prediction (MSP) task as a replacement of the MLM in the original BERT. In MSP, we mask out all the tokens in a segment and predict all these tokens (see Figure 2) to avoid the potential leakage.
In addition to MSP, we also pre-train our models with the sentence order prediction (SOP) task in Lan et al. (2020), where the model predicts whether two consecutive sentences are swapped in inputs.
2.5 Downstream tasks with Lattice-BERT
We explore four kinds of downstream tasks, i.e., sentence/sentence-pair classification, multiple choices, sequence labeling, and span selection machine reading comprehension (MRC). For the sentence/sentence-pair classification, both vanilla and Lattice-BERT classify input instances base on logistic regressions over the representation of [CLS] tokens in the last layer. The circumstances are similar in multiple choice tasks, where softmax regressions are conducted over the representations of [CLS] tokens to choose the best options. However, for the span selection MRC, and the sequence labeling tasks like named entity recognition (NER), models need to perform token-wise classification. Vanilla BERT predicts labels for the input characters, but lattice-BERT has additional words. In Lattice-BERT, we extract the character chains (word pieces for numbers and English words) from lattices for training and prediction for a fair comparison with vanilla BERT. Pilot studies show that this strategy performs comparably with the more complex strategies, which supervise the labels over words and obtain a character's label via ensembles of all tokens containing that character.
2.6 Implementation
Lattice Construction. We construct the word lattices based on a vocabulary consisting of 102K high-frequency open domain words. All the sub
strings of the input sequence that appear in the vocabulary are considered lattices tokens of the input. With Aho-Corasick automaton (Aho and Corasick, 1975), this construction procedure can complete in linear time to the size of the corpus and the vocabulary. To deal with English words and numbers where the substrings are meaningless, we use the character sequences for those out-of-vocabulary non-Chinese inputs and remain the in-vocabulary words and word pieces.
We construct word lattices using all possible words according to a vocabulary instead of more sophisticated lattice construction strategies. Previous research efforts (Lai et al., 2019; Chen et al., 2020; Li et al., 2020b) on lattice construction suggests that using all possible words usually yields better performance. We think an overly-designed lattice construction method may bias our model on certain types of text, and would probably harm the generalization. So, in our case, we let the model learn by itself to filter the noise introduced by using all possible words during pre-training on a large-scale corpus.
Pre-training Details. To compare with previous pre-training works, we implement the base-size models, which contains 12 layers, 768-dimensional of hidden size, and 12 attention heads. To demonstrate how lattice gains in shallower architectures and provide lightweight baselines, we also conduct the lite-size models with 6 layers, 8 attention heads, and the hidden size of 512.
To avoid the large vocabulary introducing too many parameters in embedding matrix, we adopt the embedding decomposition trick following Lan et al. (2020, ALBERT). Consequently, the parameters of Lattice-BERT is 100M in base-size, only  $11\%$  more than its character-level counterpart (90M), and smaller than the RoBERTa-base (Liu et al., 2019) (102M) and AMBERT (Zhang and Li, 2020) (176M). The modeling of positional relation and distances in lattice position attention introduces only 12K parameters.
A collection of Chinese text, including Chinese Wikipedia, Zhihu, and web news, is used in our BERT models' pre-training stage. The total number of characters in our unlabeled data is 18.3G. We follow Liu et al. (2019) and train the PLMs with a large batch size of 8K instances for 100K
steps. The hyper-parameters and details are given in Appendix C.
We present the details of the Lattice-BERT finetuning results on 11 Chinese NLU tasks. Answering the following questions: (1) Whether the Lattice-BERT performs better than monogranularity PLMs and other multi-granularity PLMs? (2) How the proposed lattice position attention and masked segment prediction contribute to the downstream tasks? (3) How Lattice-BERT outperforms the original character-level PLMs?
We test our models on 11 Chinese NLU tasks, including the text classification and Machine Reading Comprehension (MRC) tasks in the Chinese Language Understanding Evaluation benchmark (Xu et al., 2020, CLUE), and two additional tasks to probe the effectiveness in sequence labeling.
CLUE text classification: natural language inference CMNLI, long text classification IFLY-TEK (IFLY.), short text classification TNEWS, semantic similarity AFQMC, coreference resolution (CoRE) CLUEWSC 2020 (WSC.), and key word recognition (KwRE) CSL.
CLUE MRC: Span selection based MRC CMRC 2018 (CMRC), multiple choice questions  $\mathbf{C}^3$ , and idiom cloze ChID.
Sequence Labeling: Chinese word segmentation (CWS) MSR dataset from SIGHAN2005 (Emerson, 2005), and named entity recognition (NER) MSRA-NER (Levow, 2006).
We probe our proposed Lattice-BERT model thoroughly with these various downstream tasks. The statistics and hyper-parameters of each task are elaborated in Appendix B. We tune learning rates on validation sets and report test results with the best developing performances for CLUE tasks. For MSR and MSRA-NER, we run the settings with the best learning rates five times and report the average scores to ensure the reliability of results.
RoBERTa (Cui et al., 2020) is the Chinese version RoBERTa model (Liu et al., 2019), which adopts
Formally, the time complexity is  $O((N + M) \times L)$ .  $N$  is the corpus size,  $M$  is the vocabulary size, and  $L$  is the average length of the words in characters, which is a small constant.
<sup>5</sup>We report accuracy / exact-match scores for CLUE tasks, and label-F1 / F1 for NER / CWS tasks. CLUE results see https://www.cluebenchmarks.com/rank.html.
Task	CLUE-Classification	CLUE-MRC	Seq. Labeling	avg.
NLI	TC	SPM	CoRE	KwRE	avg.	MRC	avg.	CWS	NER
CMNLI	TNEWS	IFLY.	AFQMC	WSC.	CSL	CMRC	ChID	C3	MSR	MSRA
base-size settings
RoBERTa	80.5	67.6	60.3	74.0	76.9	84.7	74.0	75.2	83.6	66.5	75.1	98.2	96.8	78.5
NEZHA	81.1	67.4	59.5	74.5	-	83.7	-	72.2	84.4	71.8	76.1	-	-	-
BERT-word	80.0	68.2	60.0	73.5	75.5	85.2	73.7	41.3	80.9	67.0	63.1	-	-	-
AMBERT	81.9	68.6	59.7	73.9	78.3	85.7	74.7	73.3	86.6	69.6	76.5	-	-	-
BERT-Our	80.3	67.7	62.2	74.0	79.3	81.6	74.2	72.7	84.1	68.6	75.1	98.4	96.5	78.7
LBERT	81.1	68.4	62.9	74.8	82.4	84.0	75.6	74.0	86.6	72.7	77.8	98.6	97.1	80.2
lite-size settings
BERT-Our	77.9	66.7	60.7	72.1	62.4	78.7	69.7	68.3	78.7	61.6	69.5	98.1	95.5	74.6
LBERT	79.1	68.2	61.9	72.4	70.0	81.9	72.3	69.9	81.3	63.6	71.6	98.4	96.2	76.6
Table 1: The results on testing sets of 11 Chinese tasks. The bold numbers are the best scores in each column.
the whole word masking trick and external pretraining corpus, known as RoBERTa-wwm-ext.6
NEZHA (Wei et al., 2019) is one of the best Chinese PLMs with a bag of tricks, which also explores attention-level position embedding.
AMBERT (Zhang and Li, 2020) is the state-of-the-art multi-granularity Chinese PLM, with two separated encoders for words and characters.
BERT-word is a Chinese PLM baseline, taking words as single-granularity inputs. We obtain the results from Zhang and Li (2020) directly.
BERT-our is our implemented BERT model, with the same pre-training corpus, model structures, hyper-parameters, and training procedure with Lattice-BERT, but taking characters as inputs. We also adopt the whole word masking trick.
LBERT is our proposed Lattice-BERT model, with word lattices as inputs, equipping with lattice position attentions and masked segment prediction.
In Table 1, we can see in text classification, MRC, and sequence labeling tasks, with both base and lite sizes, LBERT works better than our character-level baselines consistently. LBERT-base outperforms all previous base-size PLMs in average scores and obtain the best performances in 7 of the 11 tasks.