SlowGuess/ACL_2020_OCR huggingface dataset:
Yanzeng Li, Bowen Yu, Mengge Xue, Tingwen Liu*
Institute of Information Engineering, Chinese Academy of Sciences
School of Cyber Security, University of Chinese Academy of Sciences
{liyanzeng, yubowen, xuemengge, liutingwen}@iie.ac.cn
Most Chinese pre-trained models take character as the basic unit and learn representation according to character's external contexts, ignoring the semantics expressed in the word, which is the smallest meaningful utterance in Chinese. Hence, we propose a novel word-aligned attention to exploit explicit word information, which is complementary to various character-based Chinese pre-trained language models. Specifically, we devise a pooling mechanism to align the character-level attention to the word level and propose to alleviate the potential issue of segmentation error propagation by multi-source information fusion. As a result, word and character information are explicitly integrated at the fine-tuning procedure. Experimental results on five Chinese NLP benchmark tasks demonstrate that our method achieves significant improvements against BERT, ERNIE and BERT-wwm.
Pre-trained language Models (PLM) such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), ERNIE (Sun et al., 2019), BERT-wwm (Cui et al., 2019) and XLNet (Yang et al., 2019) have been proven to capture rich language information from text and then benefit many NLP applications by simple fine-tuning, including sentiment classification (Pang et al., 2002), natural language inference (Bowman et al., 2015), named entity recognition (Sang and De Meulder, 2003) and so on.
Generally, most popular PLMs prefer to use attention mechanism (Vaswani et al., 2017) to represent the natural language, such as word-to-word self-attention for English. Unlike English, in Chinese, words are not separated by explicit delimiters. Since without word boundaries information, it is
intuitive to model characters in Chinese tasks directly. However, in most cases, the semantic of a single Chinese character is ambiguous. For example, the character "拍" in word "球拍 (bat)" and "拍卖 (auction)" has entirely different meanings. Moreover, several recent works have demonstrated that considering the word segmentation information can lead to better language understanding, and accordingly benefits various Chinese tasks (Wang et al., 2017; Li et al., 2018; Zhang and Yang, 2018; Gui et al., 2019; Mengge et al., 2019).
All these factors motivate us to expand the character-level attention mechanism in Chinese PLMs to represent the semantics of words  ${}^{1}$  . To this end, there are two main challenges. (1) How to seamlessly integrate the segmentation information into character-based attention module of PLM is an important problem. (2) Gold-standard segmentation is rarely available in the downstream tasks, and how to effectively reduce the cascading noise caused by Chinese word segmentation (CWS) tools (Li et al., 2019) is another challenge.
In this paper, we propose a new architecture, named Multi-source Word Aligned Attention (MWA), to solve the above issues. (1) Psycholinguistic experiments (Bai et al., 2008; Meng et al., 2014) have shown that readers are likely to pay approximate attention to each character in one Chinese word. Drawing inspiration from such findings, we introduce a novel word-aligned attention, which could aggregate attention weight of characters in one word into a unified value with the mixed pooling strategy (Yu et al., 2014). (2) For reducing segmentation error, we further extend our word-aligned attention with multi-source segmentation produced by various segmenters and deploy
<sup>1</sup>Considering the enormous cost of re-training a language model, we hope to incorporate word segmentation information to the fine-tuning process to enhance performance, and leave how to improve the pre-training procedure for a future work.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3442-3448 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Architecture of Word-aligned Attention
a fusion function to pull together their disparate outputs. As shown in Table 1, different CWS tools may have different annotation granularity. Through comprehensive consideration of multi-granularity segmentation results, we can implicitly reduce the error caused by automatic annotation.
Extensive experiments are conducted on various Chinese NLP tasks including sentiment classification, named entity recognition, sentence pair matching, natural language inference and machine reading comprehension. The results and analysis show that the proposed method boosts BERT, ERNIE and BERT-wwm significantly on all the datasets  ${}^{2}$  .
2.1 Character-level Pre-trained Encoder
The primary goal of this work is to inject the word segmentation knowledge into character-level Chinese PLMs and enhance original models. Given the strong performance of deep Transformers trained on language modeling, we adopt BERT and its updated variants (ERNIE, BERT-wwm) as the basic encoder in this work, and the outputs from the last layer of encoder are treated as the character-level enriched contextual representations  $\mathbf{H}$ .
2.2 Word-aligned Attention
Although character-level Chinese PLM has remarkable ability to capture language knowledge from text, it neglects the semantic information expressed in the word level. Therefore we apply a word-aligned layer on top of the encoder to integrate the
Chinese	北京西山森林公园
Segmenter	thulac	北京	西山	森林	公园
ictclas	北京	西	山	森林	公园
hanlp	北京	西山	森林公园
Table 1: Results of different popular CWS tools over "北京西山森林公园(Beijing west mount forest park)".
word boundary information into the representation of characters with an attention aggregation module.
For an input sequence with  $n$  characters  $S = [c_1, c_2, \dots, c_n]$ , where  $c_j$  denotes the  $j$ -th character, CWS tool  $\pi$  is used to partition  $S$  into non-overlapping word blocks:
$$
\pi (S) = \left[ w _ {1}, w _ {2}, \dots , w _ {m} \right], (m \leq n) \tag {1}
$$
where  $w_{i} = \{c_{s},c_{s + 1},\dots ,c_{s + l - 1}\}$  is the  $i$  -th segmented word with a length of  $l$  and  $s$  is the index of  $w_{i}$  's first character in  $S$  .We apply self-attention operation with the representations of all input characters to get the character-level attention score matrix  $\mathbf{A}_c\in \mathbb{R}^{n\times n}$  . It can be formulated as:
$$
\mathbf {A} _ {c} = \mathcal {F} (\mathbf {H}) = \operatorname {s o f t m a x} \left(\frac {\left(\mathbf {K W} _ {k}\right) \left(\mathbf {Q W} _ {q}\right) ^ {T}}{\sqrt {d}}\right) \tag {2}
$$
where  $\mathbf{Q}$  and  $\mathbf{K}$  are both equal to the collective representation  $\mathbf{H}$  at the last layer of the Chinese PLM,  $\mathbf{W}_k\in \mathbb{R}^{d\times d}$  and  $\mathbf{W}_q\in \mathbb{R}^{d\times d}$  are trainable parameters for projection. While  $\mathbf{A}_c$  models the relationship between two arbitrarily characters without regard to the word boundary, we argue that incorporating word as atom in the attention can better represent the semantics, as the literal meaning of each individual character can be quite different from the implied meaning of the whole word, and the simple weighted sum in the character level may lose word and word sequence information.
To address this issue, we propose to align  $\mathbf{A}_c$  in the word level and integrate the inner-word attention. For ease of exposition, we rewrite  $\mathbf{A}_c$  as  $[\mathbf{a}_c^1,\mathbf{a}_c^2,\dots,\mathbf{a}_c^n]$ , where  $\mathbf{a}_c^i\in \mathbb{R}^n$  denotes the  $i$ -th row vector of  $\mathbf{A}_c$ , that is,  $\mathbf{a}_c^i$  represents the attention score vector of the  $i$ -th character. Then we deploy  $\pi$  to segment  $\mathbf{A}_c$  according to  $\pi (S)$ . For example, if  $\pi (S) = [\{c_1,c_2\},\{c_3\},\ldots ,\{c_{n - 1},c_n\} ]$ , then
$$
\pi \left(\mathbf {A} _ {c}\right) = \left[ \left\{\mathbf {a} _ {c} ^ {1}, \mathbf {a} _ {c} ^ {2} \right\}, \left\{\mathbf {a} _ {c} ^ {3} \right\}, \dots , \left\{\mathbf {a} _ {c} ^ {n - 1}, \mathbf {a} _ {c} ^ {n} \right\} \right] \tag {3}
$$
In this way, an attention vector sequence is divided into several subsequences and each subsequence represents the attention of one word.
2The source code of this paper can be obtained from https://github.com/lsvih/MWA.
Then, motivated by the psycholinguistic finding that readers are likely to pay similar attention to each character in one Chinese word, we devise an appropriate aggregation module to fuse the innerword attention. Concretely, we first transform  $\{\mathbf{a}_c^s,\dots ,\mathbf{a}_c^{s + l - 1}\}$  into one attention vector  $\mathbf{a}_w^i$  for  $w_{i}$  with the mixed pooling strategy (Yu et al., 2014)3. Then we execute the piecewise upsampling operation over each  $\mathbf{a}_w^i$  to keep input and output dimensions unchanged for the sake of plug and play. The detailed process can be summarized as:
$$
\begin{array}{l} \mathbf {a} _ {w} ^ {i} = \lambda \operatorname {M a x p o o l i n g} \left(\left\{\mathbf {a} _ {c} ^ {s}, \dots , \mathbf {a} _ {c} ^ {s + l - 1} \right\}\right) \tag {4} \\ + (1 - \lambda) \text {M e a n p o o l i n g} \left(\left\{\mathbf {a} _ {c} ^ {s}, \dots , \mathbf {a} _ {c} ^ {s + l - 1} \right\}\right) \\ \end{array}
$$
$$
\hat {\mathbf {A}} _ {c} [ s: s + l - 1 ] = \mathbf {e} _ {l} \otimes \mathbf {a} _ {w} ^ {i} \tag {5}
$$
where  $\lambda \in R^1$  is a weighting trainable variable to balance the mean and max pooling,  $\mathbf{e}_l = [1,\dots,1]^T$  represents a  $l$ -dimensional all-ones vector,  $l$  is the length of  $w_i$ ,  $\mathbf{e}_l \otimes \mathbf{a}_w^i = [\mathbf{a}_w^i,\dots,\mathbf{a}_w^i]$  denotes the Kronecker product operation between  $\mathbf{e}_l$  and  $\mathbf{a}_w^i$ ,  $\hat{\mathbf{A}}_c \in \mathbb{R}^{n \times n}$  is the aligned attention matrix. Eqs. 4 and 5 can help incorporate word segmentation information into character-level attention calculation process, and determine the attention vector of one character from the perspective of the whole word, which is beneficial for eliminating the attention bias caused by character ambiguity. Finally, we can obtain the enhanced character representation produced by word-aligned attention as follows:
$$
\hat {\mathbf {H}} = \hat {\mathbf {A}} _ {c} \mathbf {V} \mathbf {W} _ {v} \tag {6}
$$
where  $\mathbf{V} = \mathbf{H}$ ,  $\mathbf{W}_v \in \mathbb{R}^{d \times d}$  is a trainable projection matrix. Besides, we also use multi-head attention (Vaswani et al., 2017) to capture information from different representation subspaces jointly, thus we have  $K$  different aligned attention matrices  $\hat{\mathbf{A}}_c^k (1 \leq k \leq K)$  and corresponding representation  $\hat{\mathbf{H}}^k$ . With multi-head attention architecture, the output can be expressed as follows:
$$
\overline {{\mathbf {H}}} = \operatorname {C o n c a t} \left(\hat {\mathbf {H}} ^ {1}, \hat {\mathbf {H}} ^ {2}, \dots , \hat {\mathbf {H}} ^ {K}\right) \mathbf {W} _ {o} \tag {7}
$$
2.3 Multi-source Word-aligned Attention
As mentioned in Section 1, our proposed word-aligned attention relies on the segmentation results
of CWS tool  $\pi$ . Unfortunately, a segmenter is usually unreliable due to the risk of ambiguous and non-formal input, especially on out-of-domain data, which may lead to error propagation and an unsatisfactory model performance. In practice, the ambiguous distinction between morphemes and compound words leads to the cognitive divergence of words concepts, thus different  $\pi$  may provide diverse  $\pi(S)$  with various granularities. To reduce the impact of segmentation error and effectively mine the common knowledge of different segmenters, it's natural to enhance the word-aligned attention layer with multi-source segmentation inputs. Formally, assume that there are  $M$  popular CWS tools employed, we can obtain  $M$  different representations  $\overline{\mathbf{H}}^1, \dots, \overline{\mathbf{H}}^M$  by Eq. 7. Then we propose to fuse these semantically different representations as follows:
$$
\tilde {\mathbf {H}} = \sum_ {m = 1} ^ {M} \tanh  \left(\overline {{\mathbf {H}}} ^ {m} \mathbf {W} _ {g}\right) \tag {8}
$$
where  $\mathbf{W}_g$  is a parameter matrix and  $\tilde{\mathbf{H}}$  denotes the final output of the MWA attention layer.
3.1 Experiments Setup
To test the applicability of the proposed MWA attention, we choose three publicly available Chinese pre-trained models as the basic encoder: BERT, ERNIE, and BERT-wwm. In order to make a fair comparison, we keep the same hyper-parameters (such maximum length, warm-up steps, initial learning rate, etc.) as suggested in BERT-wwm (Cui et al., 2019) for both baselines and our method on each dataset. We run the same experiment for five times and report the average score to ensure the reliability of results. Besides, three popular CWS tools: thulac (Sun et al., 2016), ictclas (Zhang et al., 2003) and hanlp (He, 2014) are employed to segment sequence.
The experiments are carried out on five Chinese NLP tasks and six public benchmark datasets:
Sentiment Classification (SC): We adopt ChnSentiCorp<sup>4</sup> and weibo-100k sentiment dataset<sup>5</sup> in this task. ChnSentiCorp dataset has about 10k sentences, which express positive or negative emotion. weibo-100k dataset contains 1.2M microblog
4https://github.com/pengming617/bert classification
$^{5}$ https://github.com/SophonPlus/ChineseNlpCorpus/
3Other pooling methods such as max pooling or mean pooling also works. Here we choose mixed pooling because it has the advantages of distilling the global and the most prominent features in one word at the same time.
Dataset	Task	Max length	Batch size	Epoch	lr*	Dataset Size
Train	Dev	Test
ChnSentiCorp	SC	256	16	3	3 × 10-5	9.2K	1.2K	1.2K
weibo-100k	128	64	2	2 × 10-5	100K	~10K	10K
ontonotes	NER	256	16	5	3 × 10-5	15.7K	4.3K	4.3K
LCQMC	SPM	128	64	3	3 × 10-5	~239K	8.8K	12.5K
XNLI	NLI	128	64	2	3 × 10-5	~392K	2.5K	2.5K
DRCD	MRC	512	16	2	3 × 10-5	27K	3.5K	3.5K
Table 2: Summary of datasets and the corresponding hyper-parameters setting. Reported learning rates* are the initial values of BertAdam.
texts and each microblog is tagged as positive or negative emotion.
Named Entity Recognition (NER): this task is to test model's capacity of sequence tagging. We use a common public dataset Ontonotes 4.0 (Weischedel et al., 2011) in this task.
Sentence Pair Matching (SPM): We use the most widely used dataset LCQMC (Liu et al., 2018) in this task, which aims to identify whether two questions are in a same intention.
Natural Language Inference (NLI): this task is to exploit the contexts of text and concern inference relationships between sentences. XNLI (Connieu et al., 2018) is a cross-language language understanding dataset; we only use the Chinese language part of XNLI to evaluate the language understanding ability. And we processed this dataset in the same way as ERNIE (Sun et al., 2019) did.
Machine Reading Comprehension (MRC): MRC is a representative document-level modeling task which requires to answer the questions based on the given passages. DRCD (Shao et al., 2018) is a public span-extraction Chinese MRC dataset, whose answers are spans in the document.
We implement our model with PyTorch (Paszke et al., 2019), and all baselines are converted weights into PyTorch version. All experiments employ modified Adam (Devlin et al., 2019) as optimizer with 0.01 weight decay and 0.1 warm-up ratio. All pre-trained models are configured to 12 layers and 768 hidden dimension. The detail settings are shown in Table 2.
Table 3 shows the performances on five classical Chinese NLP tasks with six public datasets. Generally, our method consistently outperforms all baselines on all five tasks, which demonstrates the effectiveness and universality of the proposed approach. Moreover, the Wilcoxon's test shows that a significant difference  $(p < 0.05)$  exits between our model and baseline models.
In detail, on the two datasets of SC task, we observe an average of  $0.53\%$  and  $0.83\%$  absolute improvement in F1 score, respectively. SPM and NLI tasks can also gain benefits from our enhanced representation. For the NER task, our method obtains  $0.92\%$  improvement averagely over all baselines. Besides, introducing word segmentation information into the encoding of character sequences improves the MRC performance on average by 1.22 points and 1.65 points in F1 and Exact Match (EM) score respectively. We attribute such significant gain in NER and MRC to the particularity of these two tasks. Intuitively, Chinese NER is correlated with word segmentation, and named entity boundaries are also word boundaries. Thus the potential boundary information presented by the additional segmentation input can provide better guidance to label each character, which is consistent with the conclusion in (Zhang and Yang, 2018). Similarly, the span-extraction MRC task is to extract answer spans from document (Shao et al., 2018), which also faces the same word boundary problem as NER, and the long sequence in MRC exacerbates the problem. Therefore, our method gets a relatively greater improvement on the DRCD dataset.
To demonstrate the effectiveness of our multisource fusion method, we carry out experiments on the DRCD dev set with different segmentation inputs. Besides, we also design two strong baselines by introducing a Transformer layer  $(1T)$  and a random tokenizer model  $\mathrm{(WA_{random})}$  to exclude the benefits from additional parameters. As shown in Table 4, adding additional parameters by introducing an extra transformer layer can benefit the PLMs. Compared with  $1T$  and  $\mathrm{WA}_{\text{random}}$ , our proposed word-aligned attention gives quite stable improvements no matter what CWS tool we use, which again confirms the effectiveness and rationality of incorporating word segmentation information into character-level PLMs. Another observation is that
Task	SC	NER	SPM	NLI	MRC
Dataset	ChnSenti2,3	weibo-100k2	Ontonotes4	LCQMC2,3,4	XNL11,2,3,4	DRCD2,3	[EM|F1]
Prev. SOTA†	93.1(2019a)	-	74.89(2019b)	85.68(2019c)	67.5(2017d)	75.12(2019e)	87.26(2019e)
BERT	94.72	97.31	79.18	86.50	78.19	85.57	91.16
+MWA	95.34(+0.62)	98.14(+0.83)	79.86(+0.68)	86.92(+0.42)	78.42(+0.23)	86.86(+1.29)	92.22(+1.06)
BERT-wwm	94.38	97.36	79.28	86.11	77.92	84.11	90.46
+MWA	95.01(+0.63)	98.13(+0.77)	80.32(+1.04)	86.28(+0.17)	78.68(+0.76)	87.00(+2.89)	92.21(+1.75)
ERNIE	95.17	97.30	77.74	87.27	78.04	87.85	92.85
+MWA	95.52(+0.35)	98.18(+0.88)	78.78(+1.04)	88.73(+1.46)	78.71(+0.67)	88.61(+0.76)	93.72(+0.87)
Table 3: Evaluation results regarding each model on different datasets. Bold marks highest number among all models. Numbers in brackets indicate the absolute increase over baseline models. Superscript number $^{1,2,3,4}$  respectively represents that the corresponding dataset is also used by BERT (Devlin et al., 2019), BERT-wwm (Wu et al., 2016; Cui et al., 2019), ERNIE (Sun et al., 2019) and Glyce (Meng et al., 2019a), respectively. The results of all baselines are produced by our implementation or retrieved from original papers, and we report the higher one among them. The improvements over baselines are statistically significant  $(p < 0.05)$ .† denotes the results of previous state-of-the-art models on these datasets without using BERT.
Model	BERT	BERT-wwm	ERNIE
Original	92.06	91.68	92.61
+1T	92.37	92.22	93.42
+WArandom	91.83	90.33	92.12
+WAthulac	92.84	92.73	93.89
+WAtictclas	93.05	92.90	93.75
+WAhanlp	92.91	93.21	93.91
+MWA	93.59	93.72	94.21
employing multiple segmenters and fusing them together could introduce richer segmentation information and further improve the performance.
3.4 Parameter Scale Analysis
For fair comparison and demonstrating the improvement of our model is not only rely on more trainable parameters, we also conduct experiments on the DRCD dev set to explore whether the performance keeps going-up with more parameters by introducing additional transformer blocks on top of the representations of PLMs.
Table 4: F1 results of ablation experiments on the DRCD dev set.
Model	F1	Param. Number
BERT-wwm	91.68	110M
BERT-wwm+1T	92.23	110M+7.1M
BERT-wwm+2T	91.99	110M+14.2M
BERT-wwm+3T	91.68	110M+21.3M
BERT-wwm+MWA	93.72	110M+7.6M
Robust-BERT-wwm-ext-large	94.40	340M
Table 5: Comparison on the DRCD dev set. The  $nT$  denotes the number of additional transformer layers.
In Table 5,  $+1T$  denotes that we introduce another one Transformer layer on top of BERT-wwm and  $+2T$  means additional 2 layers,  $M$  denotes million. As the experimental results showed, when the number of additional layers exceeds 1, the performance starts to decline, which demonstrates
that using an extensive model on top of the PLM representations may not bring additional benefits. We can conclude that MWA doesn't introduce too many parameters, and MWA achieves better performance than  $+1\mathrm{T}$  under the similar parameter numbers. Besides, we also make comparison with the current best Chinese PLM: Robust-BERT-wwm-ext-large (Cui et al., 2019), a 24-layers Chinese PLM with 13.5 times more pre-training data and 3.1 times more parameters than BERT-wwm, experimental results show that our model can achieve comparable performance, which again confirms the effectiveness of incorporating word segmentation information into character-level PLMs.
In this paper, we develop a novel Multi-source Word Aligned Attention model (referred as MWA), which integrates word segmentation information into character-level self-attention mechanism to enhance the fine-tuning performance of Chinese PLMs. We conduct extensive experiments on five NLP tasks with six public datasets. The proposed approach yields substantial improvements compared to BERT, BERT-wwm and ERNIE, demonstrating its effectiveness and universality. Furthermore, the word-aligned attention can also be applied to English PLMs to bridge the semantic gap between the whole word and the segmented Word-Piece tokens, which we leave for future work.
We would like to thank reviewers for their insightful comments. This work is supported by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDC02040400.