Title: CLASSIFICATION ATTENTION FOR CHINESE NER

Abstract: The character-based model, such as BERT, has achieved remarkable success in Chinese named entity recognition (NER). However, such model would likely miss the overall information of the entity words. In this paper, we propose to combine priori entity information with BERT. Instead of relying on additional lexicons or pre-trained word embeddings, our model has generated entity classification embeddings directly on the pre-trained BERT, having the merit of increasing model practicability and avoiding OOV problem. Experiments show that our model has achieved state-of-the-art results on 3 Chinese NER datasets.

1. INTRODUCTION:

The study of NER is of great importance as a pre-processing step for a variety of downstream applications in Information Extraction. Through supervised learning, the NER task can be cast to a sequence labeling task which involves identifying both entity boundaries and entity types Li et al. (2018) . By doing so, a named entity is considered correctly recognized only if both of its boundary and type match the ground truth. Sang & De Meulder (2003) Pradhan et al. (2012) Due to the differences in language structure, Chinese sentences are constructed in units of character with no uppercase identifier to special words such as names or locations, making it inherently unfriendly to the boundary identifying problem. To apply the word-based model in Chinese, traditionally, a segmentation step is required ahead of time in a pipeline architecture, where the errors of word boundaries could pass to the consequent NER process.

In recent years, character based Chinese NER models including BILSTM-CRF Lample et al. (2016) and BERT Devlin et al. (2018) have become dominant and achieved state-of-the-art results. However, using only character-based model would lead to potential loss of overall information. A common mitigation is to add external lexicon as a reference. Whether directly integrated into characterbased models by means of word embeddings, or directly repairing the boundaries of entities recognized by character-based models, the performance of lexicon-assisted model is strongly related to the quality of the lexicon. However, building a complete pre-defined lexicon is costly and can hardly handle OOV issues.

In this paper, an entity classification-assisted model is proposed to alleviate this problem. Above all, we design a mechanism to generate embeddings for each entity class using only the characterbased BERT pre-trained model. It mainly consists of two procedures. The first step is to form word embeddings of entities appearing in the trainning sentences through character embeddings and the second step is to aggregate entity embeddings by category and generate classification embeddings. Benefited from the inherent contextual information given in BERT representation, entities appearing under identical context can have identical representations. To further eliminate noise and extract common attributes between entities in the same class, a weighted projection removal method is applied additionally.

After that, we designed a novel Attention mechanism to integrate entity classification embeddings into the character-based BERT model. The input sources of our attention architecture consists of a set of preprocessed entity classes and an input sentence. In order to get a series of weighted representation of the input sentence, the traditional Scaled Dot-Product Attention is revised and increased in dimension, providing greater weight to characters identical to each entity class.

Results show that our model outperforms other Chinese NER models over a variety of datasets across different domains without external resources. Our code will be released at XXXX

2. RELATED MODELS:

Although character sequence labeling has been the dominant approach for Chinese NER, many works seek to add further information into the model.

As mentioned above, word-based NER models suffer from the segmentation-NER pipeline, one of the solutions is to import a multitask structure. It is a common way to share mutual information between related tasks and boost the general performance. Peng & Dredze (2016) jointly trained CWS (Chinese word segmentation) task with NER task and yield significant improvements in NER for Chinese social media. To further explore language regularities, more works Yang et al. (2016 ) Ruder12 et al. (2017) consider adding additional related tasks into the mission, employing parameter sharing or selective sharing. Transfer learning is another popular way of improvement. Sun et al. (2019a ) Feng et al. (2018) utilized the abundant resources in English to boost the NER performances in low resource languages including Chinese. Besides, Zhang & Yang (2018) investigate a latticestructured LSTM model for Chinese NER, which encodes a sequence of input characters as well as all potential words that match a lexicon, having the merit of leveraging explicit word information over character sequence labeling without suffering from segmentation error. Liu et al. (2019) further adjust the structure of the lattice model, assigning word information to a single character and ensuring that there is no shortcut paths between characters to prevent the model from degenerating into a partial word-based model. Different from the previous, our method does not require external resources including labeled data for other tasks or predefined lexicons, making the model more practical.

Recently, many works exploit language processing with deep language models to improve the performance of downstream NLP tasks, such as ELMo Peters et al. (2018) , GPT Radford et al. (2018) , BERT Devlin et al. (2018 ), ERNIESun et al. (2019b) Zhang et al. (2019) , XLNET Yang et al. (2019) . These methods first pretrain neural networks on large-scale unlabeled text corpora, and then finetune the models or representations on downstream tasks. The dynamic embeddings learned from these models outperform the traditional static embeddings by containing more contextual information as well as relieving the polysemy problem. Tagging Models based on these structures bring impressive results Sun et al. (2019a) Kaneko & Komachi (2019) Lee et al. (2019) . Our method proposed a way to extract the representation of the entity classification embeddings through the rich information contained in the pre-trained language model, which eliminates the need for external word embeddings.

What's more, the attention mechanism has shown impressive performance on many NLP tasks. Kaneko & Komachi (2019) investigate the effect of utilizing information not only from the final layer but also from intermediate layers of a pre-trained language representation model to detect grammatical errors. Rei et al. (2016) proposed a novel architecture for combining the characterbased representation with the word embedding by using an attention mechanism, allowing the model to choose which information to use from each information source dynamically. Zhu et al. (2019) investigate a practical Convolutional Attention Network for Chinese NER which not depending on any external resources. The Scaled Dot-Product Attention proposed by Vaswani et al. (2017) is one of the mainstream models currently Kaneko & Komachi (2019) , Devlin et al. (2018) , Sun et al. (2019b) , Zhang et al. (2019) , Yang et al. (2019) , Lee et al. (2019) . In this paper, we revise the Scaled Dot-Product Attention to Classification Attention which would give a weighted representation of the input sentences through a series of entity classes.

3. MODEL:



3.1. OVERALL ARCHITECTURE:

As shown in Figure 1 , the overall architecture of the proposed model is mainly composed of three parts: Embedding extraction for entity class and Classification Attention. The embedding extraction step is performed firstly in a pipeline structure and the results will be aggregated to the attention model. Details of the 2 parts will be elaborated as follow. 

3.2. EMBEDDING EXTRACTION FOR ENTITY CLASS:

Word Lexicon has played an important role in dealing model NER problems. However, a highquality lexicon construction requires a lot of manpower and might be incompetent to handle OOV issues. Since the ideal word embedding assumes that words with identical classes would have clustered representations, it is suggested that the embedding of known entities may help to find new entities of the same class.

Pre-trained with a large amount of Chinese corpus, the last hidden layer of BERT integrates context information into each character of the input sentences. It not only excludes synonym interference compared with traditional static embeddings, but also provides potential information for the embedding vector of entities in the context. Inspired from Arora et al. (2016) , in this paper, we illustrate a simple way to extract word embeddings of entities in the hidden outputs of BERT. Figure 2 .a gives the flow chart of the whole procedure.

Let X = {x 1 , x 2 . . . x L } denotes the input sentence in character level, where L is the total length and Y = {y 1 , y 2 . . . y L } denotes the label sequences accordingly. Let H = {h 1 , h 2 . . . h L } be the last hidden layer of BERT. Under the BMES label scheme Wang & Xu (2017) , both the class and location of each entity can be inferred by the label sequence Y . To fix the size, the embedding of an entity is the averaged embedding of its composed character.

Emb(W ord) = Emb (char m , char m+1 , . . . char n ) = 1 n -m n i=m Emb (char i ) = 1 n -m n i=m h i

Different from the original method in Arora et al. (2016) , the smooth inverse frequency is abandoned due to the dynamic semantic information encoded in H. To further get the most representative embedding of a given entity, the weighted projection of the word embeddings on their first singular vector is removed. Let M atrix be the collected embeddings of a certain class, whose columns are {Emb 1 , Emb 2 , . . . Emb S }, and let u be its first singular vector and α be the weight parameter α [0, 1]. For all Emb i , do:

Emb i = Emb i -αuu T Emb i

It is assumed that when the gap between the training data and the testing data is large, α may takes a larger value to show a more common representation of the entity class. Finally, the embedding of a certain entity class can be gathered through the output of averagePooling layer. The whole processure is shown in Figure2(a). Firstly, consider a situation with only one entity class. Matrix Q and Matrix K are adopted to extract the characteristics of the input sentence and the entity. Here, the length of the input entity class can be seen as an input sentence with only one character. By applying the traditional Scaled Dot-Product Attention, the similarity between each character of the input sentence and the entity class can be calculated by:

Similarity(Q, K) = softmax QK T √ d k

The output product gives indication of how likely a character in the input sentence is to be the input entity. Since the final result we would like to get is the weighted embedding of the input sentence. So, the V matrix in our architecture is calculated from the input sentence other than the entity. The final result can be got through

Attention(Q, K, V ) = similarity(Q, K) V Figure 2(b)

gives an illustration of the attention architecture for multiple entity classes. For the sake of simplicity, the softmax step and the dropout step are omitted, and the normalized scaling dot product between the entity classification embedding and the sentence embedding after the Key-layer and then Query-layer respectively is directly displayed. The intensity of the weight is proportional to the depth of color giving indication of how likely the input character is to the particular entity.

After applying the Hadamard product, the weighted representation of the original sentence after Value-layer is finally obtained.

During the calculation, Multi-head mechanism is adopted for each entity class and the entire calculation is done once by a matrix one-dimensional higher than BERT. After the attention stage, the merged output is then passed along with the original character embeddings of the input sentence through the same follow-up layers as BERT, namely Self-output layer, Intermediate layer, and Output layer. Figure 1 gives the detailed structure of the 3 layers where two residential nets are applied to ensure the validity of the deep network.

4. EXPERIMENT:

An extensive set of experiments on varies Chinese NER datasets are conducted to demonstrate the effectiveness of our proposed model. In the following sections, we will list the details of datasets, Especially, since our model is conducted on the basis of BERT, the maximum number of input characters is limited. To keep a balance between the computing resources and maintaining the original data, we set the maximum number of inputting characters as 256, where data with exceeded numbers of input characters are cut into shorter sentences. The statistics of the datasets are shown in Table 1 , where the original total number of sentences for each data set as well as the total number of sentences after cutting are both given. It should be noticed that in order to obtain a fair verification, the test results of the split sentences are merged together to produce the final result.

4.1.2. IMPLEMENTATION DETAILS:

We use the 12-layer pre-trained BERT as the base model to generate the classification embeddings for each entity class and to conduct classification attention. We set the word embedding size of the entity class to 768, the same with the BERT's hidden size. The classification attention layer is set to 1. The maximum length of the input sentences is set to 256 and the training batches are set to 16. Other shared Hyper-parameters defined in the configuration json file along with the released BERT-Base model remain unchanged. We use BERT-Adam to optimize all the trainable parameters.

Learning rate is set to 5 × 10 -5 initially and decays during training at a rate of 0.01.

4.1.3. EXPERIMENTAL RESULTS:

In this section, we will give the experimental results of our proposed model and previous state-ofthe-art methods on Weibo dataset, Chinese Resume dataset, OntoNotes 4 dataset and MSRA dataset, respectively. To exclude the performance improvements from BERT's own mechanism, we design 2 Baselines based on BERT along with the Classification Attention method, namely BERT + CRF model, BERT with 13 hidden layers (i.e. one additional layer to exclude the improvements of selfattention itself) + CRF model. We treat NER as a sequential labeling problem and adopt BMOES tagging style in this paper since it has been shown that models using BMOES are remarkably better than BIO Yang et al. (2018) 4.1.4 WEIBO DATASET

Table 3 shows the comparisons on Weibo Dataset. We compare our proposed model with the latest models on Weibo dataset. It could be observed that our proposed model achieves state-of-the-art performance. In the first block of Table 3 , we report the performances of our baselines. Our baseline BERT-base + CRF achieves an F1-score of 71.05 and BERT with 13 hidden layers + CRF achieves an F1-score of 69.14. In the second block of 2019) are built on the basis of LSTM of which the feature extraction ability is relatively pooler than Transformer.The performance of our proposed model is presented in the last line in Table 3 . Since the F1-scores of the 2 baseline models are around 70, it is assumed that the gap between the training data and the test data is relatively large. So, the entity embeddings obtained from the training data of a certain class may also get relatively large distribution differences with the test class. Hence, the weight for direction removal should be aligned with a larger weight. By doing so, most of the divergences in the distribution of the embeddings in the training set can be excluded, while the common component can be retained. It can be recognized from the result that both the Precision score and Recall score of our model are all higher than the 2 base-line model which demonstrate the effectiveness of our proposed model. 

4.1.5. ONTONOTES DATASET:

Table 4 shows the comparisons on OntoNotes 4 dataset. We compare our proposed model with the latest models on OntoNotes dataset. In the first block of Table 4 , we report the performance of the baseline models. Our baseline BERT-base + CRF achieves an F1-score of 79.64 and BERT with 13 hidden layers + CRF achieves an F1-score of 78.78.

In the second block of Table 4 , the performances of the 5 latest models are given. Zhang & Yang (2018) An entity-sentence relation heatmap is shown in Figure 3 , where the strength of the relationship is proportional to the depth of the color. The y coordinate shows the categories of different entities and the x coordinate shows the input Chinese character as well as their corresponding labels.

According to the ground truth, most entities are accurately given greater weight after attention though the example sentence contains several irregular symbols. One exception is that only "Cloud" was marked as class "PER" in the ground truth but the whole term "Dear cloud" may also be seen as a whole username that belongs to class "PER" .

It should also be noticed that though our method presents indication of all potential entities, the weight of each entity class is not ideally proportional to the ground truth label. West Lake Yintai store is assigned with a greater likelihood in "GPE" than "LOC". At the same time, the organization McDonald's is assigned with equal weight in "GPE"," PER" and "ORG".

4.2. RESULTS ANALYSIS:

Our proposed model outperforms previous work on MSRA, Weibo and Chinese Resume dataset and gains competitive results on OntoNotes 4 datasets without using any external resources. The experiments results demonstrate the effectiveness of our proposed model. Compared to Baseline BERT + CRF, the performance improvement after adding Classification Attention mechanism verifies that entity classification embeddigns obtained by our model can give extra information to locate entities. Moreover, compared to Baseline BERT with 13 layers + CRF, the better performance of our model excludes the benefits from deeper self-attention layer.

5. CONCLUSION:

In this paper, we seek to find an entity classification-assisted model for Chinese NER by investigating a mechanism to generate classification embeddings for each entity class using only the character-based BERT pre-trained model and designing a Classification Attention for Chinese NER. It observes a series of weighted representation of the input sentences so that characters similar to a particular entity class could have greater feature weights. In future work, we plan to conduct deeper research on classification embeddings generation and model structure improvements.

:



Figures and Tables:

Figure 1: The overall architecture of the proposed model.The detailed structure for Classification Embedding Extraction is will be shown in following Section

Figure 2: Figure2(a) :Embedding extraction for entity classsification. Figure 2(b) Illustration of the attention architecture.

,Liu et al. (2019),Zhu et al. (2019) andZhao et al. (2018) are developed based on LSTM whereas Sun et al. (2019a) utilized the character embeddings of BERT. For external resources, Zhang & Yang (2018), Liu et al. (2019) and Zhao et al. (2018) utilized predefined lexicon embeddings and Sun et al. (2019a) utilized pre-trained English NER model. It can be illustrated from the result in

Figure 3: Entity-sentence relation heatmap of the case in Weibo Dataset

Data splitting for each dataset

Table content: <table><tr><td>Dataset</td><td>Type</td><td colspan="2">Train Dev</td><td>Test</td></tr><tr><td>OntoNotes4</td><td colspan="4">Original Max length 256 15729 4301 4347 15724 4301 4346</td></tr><tr><td/><td>Original</td><td>46364</td><td>-</td><td>4365</td></tr><tr><td>MSRA</td><td colspan="4">Split Dev Data 42000 4364 4365</td></tr><tr><td/><td colspan="4">Max length 256 42080 4370 4375</td></tr><tr><td>Weibo</td><td colspan="2">Original Max length 256 1350 1350</td><td>270 270</td><td>270 270</td></tr><tr><td>Resume</td><td colspan="2">Original Max length 256 3821 3821</td><td>463 463</td><td>477 477</td></tr><tr><td colspan="5">parameter settings as well as experimental results and conclude result analysis. Standard precision</td></tr><tr><td colspan="4">(P), recall (R) and micro F1-score (F1) are used as evaluation metrics.</td></tr><tr><td>4.1 EXPERIMENTAL SETTINGS</td><td/><td/><td/></tr><tr><td>4.1.1 DATA</td><td/><td/><td/></tr></table>

Table 3, the performances of 3 latest models for Chinese NER are given where Zhang & Yang (2018) and Liu et al. (2019) utilized external resources including predefined lexicon and pre-trained word embeddings. The lattice structure proposed by Zhang & Yang (2018) achieves F1-score of 58.79. By excluding the no shortcut paths between characters, Liu et al. (2019) improves the F1-score to 59.84. Zhu et al. (2019) does not rely on any external resources and achieves the F1-score of 59.31. Compared with the baseline model, Zhang & Yang (2018), Liu et al. (2019) and Zhu et al. (

Table content: <table/>

Results of Weibo Dataset

Table content: <table><tr><td>Models</td><td>P</td><td>R</td><td>F1</td></tr><tr><td>BERT + CRF</td><td colspan="3">70.38 71.74 71.05</td></tr><tr><td>13-layers BERT + CRF</td><td colspan="3">70.71 67.63 69.14</td></tr><tr><td>Zhang &amp; Yang (2018)</td><td>-</td><td>-</td><td>58.79</td></tr><tr><td>Liu et al. (2019)</td><td>-</td><td>-</td><td>59.84</td></tr><tr><td>Zhu, Wang and Karlsson (2019)</td><td colspan="3">55.38 62.98 59.31</td></tr><tr><td>Our model</td><td colspan="3">71.16 73.91 72.51</td></tr></table>

Results of OntoNotes Dataset

Table content: <table><tr><td>Models</td><td>P</td><td>R</td><td>F1</td></tr><tr><td>BERT + CRF</td><td colspan="3">79.08 80.21 79.64</td></tr><tr><td>13-layers BERT + CRF</td><td colspan="3">74.06 84.15 78.78</td></tr><tr><td>Zhang &amp; Yang (2018)</td><td colspan="3">76.35 71.56 73.88</td></tr><tr><td>Liu et al. (2019)</td><td colspan="3">76.09 72.85 74.43</td></tr><tr><td>(Zhu, Wang and Karlsson 2019)</td><td colspan="3">75.05 72.29 73.64</td></tr><tr><td>Sun et al. (2019a)</td><td>-</td><td>-</td><td>80.21</td></tr><tr><td colspan="4">Zhao et al. (2018) 76.69 71.91 74.22</td></tr><tr><td>Our model</td><td colspan="3">79.23 80.92 80.07</td></tr></table>

Zhu et al. (2019) with no external resources achieves F1-score of 73.64,Zhang & Yang  (2018)  with lattice structure achieves F1-score of 73.88.Zhao et al. (2018) andLiu et al. (2019) achieve 74.22 and 74.43 respectivelyThe performance of our proposed model is shown in the last line in Table4. Since the F1-scores of the 2 baseline models are around 79, the weight for direction removal should also be aligned with a lager weight. It can be recognized from the result that our model got the higest Precision score and competitive F1 score without requiring external resources.4.1.6 RESUME DATASETTable5shows the comparisons on Resume dataset. We compare our proposed model with the latest models on Resume dataset. In the first block of Table5, we report the performances of the baseline models. Our baseline BERT-base + CRF achieves an F1-score of 95.76 and BERT with 13 hidden layers + CRF achieves an F1-score of 95.97. In the second block of Table5, the performances of 5 latest model are given.Liu et al. (2019),Zhu et al. (2019) andZhang & Yang (2018)  are all built on the basis of LSTM. For external resources,Liu et al. (2019) andZhang & Yang (2018)  utilized predefined lexicon and pre-trained word embeddings. It can be told from the result in Table5,Zhu et al. (2019) with no external resources achieves F1-score of 94.94,Zhang & Yang (2018)  with lattice structure achieves F1-score of 95.21. By excluding the no shortcut paths between characters,Liu et al. (2019) improves the F1-score to 95.21. The performance of our proposed model is given in the last line in Table5. Since the F1-scores of the 2 baseline models are around 95, it is assumed that the gap between the training data and the test data is relatively small. So, the entity embeddings obtained from the training data of a certain class may also get relatively small distribution differences with the test class. Hence, the weight for direction removal should be aligned with a smaller weight. It can be told from the result that our model got both competitive Precision score and Recall score than the Baseline model and forms the highest F1-score finally.

Table content: <table/>

Results of Resume Dataset

Table content: <table><tr><td>Models</td><td>P</td><td>R</td><td>F1</td></tr><tr><td>BERT + CRF</td><td colspan="3">95.94 95.58 95.76</td></tr><tr><td>13-layers BERT + CRF</td><td colspan="3">95.45 96.50 95.97</td></tr><tr><td colspan="4">Liu et al. (2019) 95.27 95.15 95.21</td></tr><tr><td>(Zhu, Wang and Karlsson 2019)</td><td colspan="3">95.05 94.82 94.94</td></tr><tr><td>Zhang &amp; Yang (2018)</td><td colspan="3">94.81 94.11 94.46</td></tr><tr><td>Our model</td><td colspan="3">95.73 96.38 96.06</td></tr></table>

Results of MSRA Dataset

Table content: <table><tr><td>Models</td><td>P</td><td>R</td><td>F1</td></tr><tr><td>BERT + CRF</td><td colspan="3">94.98 94.64 94.81</td></tr><tr><td>13-layers BERT + CRF</td><td colspan="3">94.91 94.24 94.57</td></tr><tr><td>Zhang &amp; Yang (2018)</td><td colspan="3">93.57 92.79 93.18</td></tr><tr><td>Liu et al. (2019)</td><td colspan="3">94.58 92.91 93.74</td></tr><tr><td>(Zhu, Wang and Karlsson 2019)</td><td colspan="3">93.53 92.42 92.97</td></tr><tr><td colspan="4">Zhao et al. (2018) 93.66 93.05 93.35</td></tr><tr><td>Our model</td><td colspan="3">95.20 94.49 94.86</td></tr></table>

shows the comparisons on MSRA dataset. We compare our proposed model with the latest models on MSRA dataset. In the first block of Table6, we report the performances of the baseline models. Our baseline BERT-base + CRF achieves an F1-score of 94.81 and BERT with 13 hidden layers + CRF achieves an F1-score of 94.57.In the second block of Table6, the performances of 4 latest model are given.Zhang & Yang (2018),Liu et al. (2019),Zhu et al. (2019) andZhao et al. (2018) are all built on the basis of LSTM. For external resources,Zhang & Yang (2018),Liu et al. (2019) andZhao et al. (2018) utilized predefined lexicon and pre-trained word embeddings. It can be told from the result in Table6,Zhu et al. (2019) with no external resources achieves F1-score of 92.97,Zhang & Yang (2018)  with lattice structure achieves F1-score of 93.18.Zhao et al. (2018) andLiu et al. (2019) achieve 93.35 and 93.74 respectively.The performance of our proposed model is given at the last line in Table6. Since the F1-scores of the 2 baseline models are around 90, the weight for direction removal should be aligned with a smaller weight.It can be told from the result that our model got competitive Recall score and the highest Recall score and forms the highest F1-score finally.

Table content: <table/>

Peer review (iclr2020):
This paper tries to improve the performance of Chinese NER by developing a novel attention mechanism that leverages BERT pre-trained model which considers bi-directional context. Experiments on a number of tasks show that the proposed approach is effective. Comments: [1] A bunch of experiments are conducted [2] Chinese NER is a hard problem, but it would be great to see the proposed approach generalizable to other tasks. So, the contribution of this paper is limited [3] The proposed algorithm is simple and effective, but the novelty is a bit low