Abstract: Most Chinese pre-trained encoders take a character as a basic unit and learn representations according to character's external contexts, ignoring the semantics expressed in the word, which is the smallest meaningful unit in Chinese. Hence, we propose a novel word aligned attention to incorporate word segmentation information, which is complementary to various Chinese pre-trained language models. Specifically, we devise a mixed-pooling strategy to align the character level attention to the word level, and propose an effective fusion method to solve the potential issue of segmentation error propagation. As a result, word and character information are explicitly integrated at the fine-tuning procedure. Experimental results on various Chinese NLP benchmarks demonstrate that our model could bring another significant gain over several pre-trained models.

Introduction: Pre-trained language Models (PLM) such as ELMo BIBREF0, BERT BIBREF1, ERNIE BIBREF2 and XLNet BIBREF3 have been proven to capture rich language information from text and then benefit many NLP applications by simple fine-tuning, including sentiment classification, natural language inference, named entity recognition and so on. Generally, most of PLMs focus on using attention mechanism BIBREF4 to represent the natural language, such as word-level attention for English and character-level attention for Chinese. Unlike English, in Chinese, words are not separated by explicit delimiters, which means that character is the smallest linguistic unit. However, in most cases, the semantic of single Chinese character is ambiguous. UTF8gbsn For example, in Table 1, using the attention over word 西山 is more intuitive than over the two individual characters 西 and 山. Moreover, previous work has shown that considering the word segmentation information can lead to better language understanding and accordingly benefits various Chines NLP tasks BIBREF5, BIBREF6, BIBREF7. All these factors motivate us to expand the character-level attention mechanism in Chinese PLM to represent attention over words . To this end, there are two main challenges. (1) How to seamlessly integrate word segmentation information into character-level attention module of PLM is an important problem. (2) Gold-standard segmentation is rarely available in the downstream tasks, and how to effectively reduce the cascading noise caused by automatic segmentation tools BIBREF8 is another challenge. In this paper, we propose a new architecture, named Multi-source Word Alignd Attention (MWA), to solve the above issues. (1) Psycholinguistic experiments BIBREF9, BIBREF10 have shown that readers are likely to pay approximate attention to each character in one Chinese word. Drawing inspiration from such finding, we introduce a novel word-aligned attention, which could aggregate attention weight of characters in one word into a unified value with the mixed pooling strategy BIBREF11. (2) For reducing segmentation error, we further extend our word-aligned attention with multi-source segmentation produced by various segmenters, and deploy a fusion function to pull together their disparate output. In this way, we can implicitly reduce the error caused by automatic annotation. Extensive experiments are conducted on various Chinese NLP datasets including named entity recognition, sentiment classification, sentence pair matching, natural language inference, etc. The results show that the proposed model brings another gain over BERT BIBREF1, ERNIE BIBREF2 and BERT-wwm BIBREF12, BIBREF13 in all the tasks.

Question: What is the size of the dataset?
Answer: 3029