Pre-trained language Models (PLM) such as ELMo BIBREF0, BERT BIBREF1, ERNIE BIBREF2 and XLNet BIBREF3 have been proven to capture rich language information from text and then benefit many NLP applications by simple fine-tuning, including sentiment classification, natural language inference, named entity recognition and so on.Generally, most of PLMs focus on using attention mechanism BIBREF4 to represent the natural language, such as word-level attention for English and character-level attention for Chinese. Unlike English, in Chinese, words are not separated by explicit delimiters, which means that character is the smallest linguistic unit. However, in most cases, the semantic of single Chinese character is ambiguous. UTF8gbsn For example, in Table 1, using the attention over word 西山 is more intuitive than over the two individual characters 西 and 山. Moreover, previous work has shown that considering the word segmentation information can lead to better language understanding and accordingly benefits various Chines NLP tasks BIBREF5, BIBREF6, BIBREF7.All these factors motivate us to expand the character-level attention mechanism in Chinese PLM to represent attention over words . To this end, there are two main challenges. (1) How to seamlessly integrate word segmentation information into character-level attention module of PLM is an important problem. (2) Gold-standard segmentation is rarely available in the downstream tasks, and how to effectively reduce the cascading noise caused by automatic segmentation tools BIBREF8 is another challenge.In this paper, we propose a new architecture, named Multi-source Word Alignd Attention (MWA), to solve the above issues. (1) Psycholinguistic experiments BIBREF9, BIBREF10 have shown that readers are likely to pay approximate attention to each character in one Chinese word. Drawing inspiration from such finding, we introduce a novel word-aligned attention, which could aggregate attention weight of characters in one word into a unified value with the mixed pooling strategy BIBREF11. (2) For reducing segmentation error, we further extend our word-aligned attention with multi-source segmentation produced by various segmenters, and deploy a fusion function to pull together their disparate output. In this way, we can implicitly reduce the error caused by automatic annotation.Extensive experiments are conducted on various Chinese NLP datasets including named entity recognition, sentiment classification, sentence pair matching, natural language inference, etc. The results show that the proposed model brings another gain over BERT BIBREF1, ERNIE BIBREF2 and BERT-wwm BIBREF12, BIBREF13 in all the tasks.The primary goal of this work is to inject the word segmentation knowledge into character-level Chinese PLM and enhance original models. Given the strong performance of recent deep transformers trained on language modeling, we adopt BERT and its updated variants (ERNIE, BERT-wwm) as the basic encoder for our work, and the outputs $\mathbf {H}$ from the last layer of encoder are treated as the enriched contextual representations.Although the character-level PLM can well capture language knowledge from text, it neglects the semantic information expressed in the word level. Therefore we apply a word-aligned layer on top of the encoder to integrate the word boundary information into representation of character with the attention aggregation mechanism.For an input sequence with with $n$ characters $S=[c_1, c_2, ... , c_n]$, where $c_j$ denotes the $j$-th character, Chinese words segmentation tool $\pi $ is used to partition $S$ into non-overlapping word blocks:where $w_i = \lbrace c_s, c_{s+1}, ..., c_{s+l-1}\rbrace $ is the $i$-th segmented word of length $l$ and $s$ is the index of $w_i$'s first character in $S$. We apply the self-attention with the representations of all input characters to get the character-level attention score matrix $\textbf {A}_c \in \mathbb {R}^{n \times n}$. It can be formulated as:where $\textbf {Q}$ and $\textbf {K}$ are both equal to the collective representation $\textbf {H}$ at the last layer of the Chinese PLM, $\textbf {W}_k \in \mathbb {R}^{d\times d}$ and $\textbf {W}_q \in \mathbb {R}^{d\times d}$ are trainable parameters for projection. While $\textbf {A}_c$ models the relationship between two arbitrarily characters without regard to the word boundary, we argue that incorporating word as atoms in the attention can better represent the semantics, as the literal meaning of each individual characters can be quite different from the implied meaning of the whole word, and the simple weighted sum in character-level cannot capture the semantic interaction between words.To this end, we propose to align $\textbf {A}_c$ in the word level and integrate the inner-word attention. For the sake of simplicity, we rewrite $\textbf {A}_c$ as $[\textbf {a}_c^1, \textbf {a}_c^2, ... ,\textbf {a}_c^n]$, where $\textbf {a}_c^i \in \mathbb {R}^n $ denotes the $i$-th row vector of $\textbf {A}_c$ and the attention score vector of the $i$-th character. Then we deploy $\pi $ to segment $\textbf {A}_c$ according to $\pi (S)$. For example, if $\pi (S) = [\lbrace c_1, c_2\rbrace , \lbrace c_3\rbrace , ...,\lbrace c_{n-1}, c_{n}\rbrace ]$, thenIn this way, an attention vector sequence is segmented into several subsequences and each subsequence represents the attention of one word. Then, motivated by the psycholinguistic finding that readers are likely to pay approximate attention to each character in one Chinese word, we devise an appropriate aggregation module to fuse the inner-word character attention. Concretely, we first transform $\lbrace \textbf {a}_c^s,..., \textbf {a}_c^{s+l-1}\rbrace $ into one attention vector $\textbf {a}_w^i$ for $w_i$ with the mixed pooling strategy BIBREF11. Then we execute the piecewise up- mpling operation over each $\textbf {a}_w^i$ to keep input and output dimensions unchanged for the sake of plug and play. The detailed process can be summarized as follows:where $\lambda \in R^1 $ is a weighting trainable variable to balance the mean and max pooling, $\textbf {e}_l=[1,...,1]^T$ represents a $l$-dimensional all-ones vector, $l$ is the length of word $w_i$, $\textbf {e}_l \otimes \textbf {a}_w^i=[\textbf {a}_w^i,...,\textbf {a}_w^i]$ denotes the kronecker product operation between $\textbf {e}_l$ and $\textbf {a}_w^i$, $\hat{\textbf {A}}_c \in \mathbb {R}^{n \times n}$ is the aligned attention matrix. The Eq. (DISPLAY_FORM9-) can help incorporate word segmentation information into character-level attention calculation process, and determine the attention vector of one character from the perspective of the whole word, which is beneficial for eliminating the attention bias caused by character ambiguity. Finally, we get the enhanced character representation produced by word-aligned attention:where $\textbf {V} = \textbf {H}$, $\textbf {W}_v \in \mathbb {R}^{d\times d}$ is a trainable projection matrix. Besides, we also use multi-head attention BIBREF4 to capture information from different representation subspaces jointly, thus we have $K$ different aligned attention matrices $\hat{\textbf {A}}_c^k (1\le k\le K)$ and corresponding output $\hat{\textbf {H}}^k$. With multi-head attention architecture, the output can be expressed as follows:As mentioned in Section SECREF1, our proposed word-aligned attention relies on the segmentation results of CWS tool $\pi $. Unfortunately, a segmenter is usually unreliable due to the risk of ambiguous and non-formal input, especially on out-of-domain data, which may lead to error propagation and an unsatisfactory model performance. In practice, The ambiguous distinction between morphemes and compound words leads to the cognitive divergence of words concepts, thus different $\pi $ may provide diverse $\pi (S)$ with various granularities. To reduce the impact of segmentation error and effectively mine the common knowledge of different segmenters, it’s natural to enhance the word-aligned attention layer with multi-source segmentation input. Formally, assume that there are $M$ popular CWS tools employed, we can obtain $M$ different representations $\overline{\textbf {H}}^1, ..., \overline{\textbf {H}}^M $ by Eq. DISPLAY_FORM11. Then we propose to fuse these semantically different representations as follows:where $\textbf {W}_g$ is the parameter matrix and $\tilde{\textbf {H}}$ is the final output of the MWA attention layer.To test the applicability of the proposed MWA attention, we choose three publicly available Chinese pre-trained models as the basic encoder: BERT, ERNIE, and BERT-wwm. In order to make a fair comparison, we keep the same hyper-parameters (such maximum length, warm-up steps, initial learning rate, etc) as suggested in BERT-wwm BIBREF13 for both baselines and our method on each dataset. We run the same experiment for five times and report the average score to ensure the reliability of results. For detailed hyper-parameter settings, please see Appendix. Besides, three popular CWS tools thulac BIBREF14, ictclas BIBREF15 and hanlp BIBREF16 are employed to segment the Chinese sentences into words.We carried out experiments on four Chinese NLP tasks, including Emotion Classification (EC), Named Entity Recognition (NER), Sentence Pair Matching (SPM) and Natural Language Inference (NLI). The detail of those tasks and the corresponding datasets are introduced in Appendix.Table TABREF14 shows the experiment measuring improvements from the MWA attention on test sets of four datasets. Generally, our method consistently outperforms all baselines on all of four tasks, which clearly indicates the advantage of introducing word segmentation information into the encoding of character sequences. Moreover, the Wilcoxon’s test shows that significant difference ($p< 0.01$) exits between our model with baseline models.In detail, On the EC task, we observe 1.46% absolute improvement in F1 score over ERINE. SPM and NLI tasks can also gain benefits from our enhanced representation, achieving an absolute F1 increase of 0.68% and 0.55% over original models averagely. For the NER task, our method improves the performance of BERT by 1.54%, and obtains 1.23% improvement averagely over all baselines. We attribute such significant gain in NER to the particularity of this task. Intuitively, Chinese NER is correlated with word segmentation, and named entity boundaries are also word boundaries. Thus the potential boundary information presented by the additional segmentation input can provide a better guidance to label each character, which is consistent with the conclusion in BIBREF6, BIBREF7.To demonstrate the effectiveness of our multi-source fusion method in reducing the segmentation error introduced by CWS tools, We further carry out experiments on the EC task with different segmentation inputs. Table TABREF16 presents the comprehensive results on the three segmentation inputs produced by three CWS tools aforementioned. Experimental results show that our model gives quite stable improvement no matter the segmentation input quality. This again suggests the effectiveness of incorporating word segmentation information into character-level PLMs. And by employing multiple segmenters and fusing them together could introduce richer segmentation information and reduce the impact of general existent segmentation error.In this paper, we propose an effective architecture Word-aligned Attention to incorporate word segmentation information into character-based pre-trained language models, which is adopted to a variety of downstream NLP tasks as an extend layer in fine-tuned process. And we also employ more segmenters into via proposed Multi-source Word-aligned Attention for reducing segmentation error. The experimental results show the effectiveness of our method. Comparing to BERT, ERNIE and BERT-wwm, our model obtains substantial improvements on various NLP benchmarks. Although we mainly focused on Chinese PLM in this paper, our model would take advantage the capabilities of Word-aligned Attention for word-piece in English NLP task. We are also considering applying this model into pre-training language model for various Language Model task in different grain to capture multi-level language features.

Q: What pre-trained models did they compare to?

Q: How does the fusion method work?

Q: What dataset did they use?
A: weibo-100k, Ontonotes, LCQMC and XNLI

Q: What benchmarks did they experiment on?