# An ERNIE-Based Joint Model for Chinese Named Entity Recognition
## 1 Introduction  
Named Entity Recognition (NER), as the fundamental task of Natural Language Processing (NLP), aims to recognize entities with specific meanings from unstructured text, such as the names of people, locations, and organizations [1]. It is the initial step in extracting valuable knowledge from unstructured text and building a Knowledge Graph (KG). The performance of NER may affect downstream knowledge extraction tasks, such as the Relation Extraction (RE) [2]. In the early years, researchers used rule-based or dictionary-based methods for NER tasks [3, 4]. However, these methods lack generalization because they are proposed for particular types of entities. Machine learning and deep learning methods emerging in recent years are also used in NER tasks [5, 6]. Nevertheless, the performance of these methods often suffers from small-scale human-labelled trainingdata, resulting in poor generalization capability, especially for rare words. Therefore, it is of interest to know whether the prior semantic knowledge can be learned from large amounts of unlabelled corpora to improve the performance of NER. Recently, BERT (Bidirectional Encoder Representations from Transformers) [7] achieved state-of-the-art (SOTA) results in various NLP tasks. It can obtain prior semantic knowledge from large-scale unlabelled corpora through pre-training tasks and improve the performance of downstream tasks by transferring this knowledge to them [7]. However, Chinese NER is still a more challenging task for BERT because there are no physical separations between Chinese words. Therefore, BERT can only obtain character-level representations during pre-training. For example, the meaning of the two sentences inputted into BERT in Figure 1 is the same. The token of the English sentence is a word, while the token of the Chinese sentence is a character. During the procedure of \"Masked Language Model (MLM)\", which is a pre-training task, BERT will mask some tokens at random and predict it in order to learn the prior semantic knowledge about the tokens. Therefore, BERT can only learn the character-level representations as the tokens are all Chinese characters. However, the Chinese NER cannot be handled well when only using character-level representations, because in general, the meaning of a Chinese word is quite different from that of the characters, which make up the word. ERNIE (Enhanced Representation through kNowledge IntEgration) [8], which is an improved pre-training model of BERT, is more suitable for Chinese NER because it is designed to learn language representations enhanced by knowledge masking strategy. Unlike the character-level masking strategy of BERT, which can only learn the character-level representations of Chinese, the knowledge masking strategy of ERNIE consists of entity-level and phrase-level masking strategies and can learn the prior semantic knowledge of Chinese entities and phrases implicitly during pre-training. The model has better generalization and adaptability due to the knowledge masking strategy [8]. However, the potential of ERNIE has not been fully explored. In order to transfer the prior semantic knowledge to a downstream task (e.g., NER), ERNIE must be fine-tuned over a task-specific dataset. As shown in Figure 2, for a NER task, the representation \(h_{d}\) of a token \(t_{d}\) (for \(d=1,\dots,D\)) can be used to classify this token with respect to the target categories, and \(h=\{h_{1},\dots,h_{D}\}\) can be regarded as the token-level features. For a text classification task, the representation \(h_{C}\) of the [CLS] is a fixed dimensional pooled representation of the sequence, and \(h_{C}\) can be regarded as the sentence-level feature. Therefore, it would be of special interest to know whether the performance of NER can be improved by utilizing both the token-level and sentence-level features. In this paper, we aim to improve the performance of Chinese NER by utilizing both the token-level and sentence-level features.