Classification Attention for Chinese NER
Authors: Yuchen Ge
FanYang
PeiYang
Venue:  (2020)
Keywords: Chinese NER
NER
tagging
deeplearning
nlp

Abstract: The character-based model, such as BERT, has achieved remarkable success in Chinese named entity recognition (NER). However, such model would likely miss the overall information of the entity words. In this paper, we propose to combine priori entity information with BERT. Instead of relying on additional lexicons or pre-trained word embeddings, our model has generated entity classification embeddings directly on the pre-trained BERT, having the merit of increasing model practicability and avoiding OOV problem. Experiments show that our model has achieved state-of-the-art results on 3 Chinese NER datasets.

Under review as a conference paper at ICLR 2020
CLASSIFICATION ATTENTION FOR CHINESE NER
Anonymous authors
Paper under double-blind review
ABSTRACT
The character- based model, such as BERT. has achieved remarkable success in Chinese named entity recognition (NER). However, such model would likely miss the overall information of the entity words. In this paper, we propose to combine priori entity information with BERT. Instead of relying on additional lexicons pre-trained word embeddings, our model has generated entity classification em- beddings directly on the pre-trained BERT, having the merit of increasing model practicability and avoiding OOV problem. Experiments show that our model has achieved state-of- the-art results on 3 Chinese NER datasets.
INTRODUCTION
The study of NER is great importance as a pre-processing step for a variety of downstream applications in Information Extraction. Through supervised learning, the NER task can be cast to sequence labeling task which involves identifying both entity boundaries and entity types Li et al. (2018). By doing so, a named entity is considered correctly recognized only both of its boundary and type match the ground truth. Sang & De Meulder (2003) Pradhan et al. (2012)
Due to the differences in language structure, Chinese sentences are constructed in units of charac- ter with no uppercase identifier to special words such as names or locations, making it inherently unfriendly to the boundary identifying problem. To apply the word-based model in Chinese, tradi- tionally, segmentation step is required ahead of time in pipeline architecture, where the errors of word boundaries could pass to the consequent NER process.
In recent years, character based Chinese NER models including BILSTM-CRF Lample et al. (2016) and BERT Devlin et al. (2018) have become dominant and achieved state-of-the-an results. How- ever, using only character-basee based model would lead to potential loss of overall information. A com- mon mitigation is to add external lexicon as a reference. Whether directly integrated into character- based models by means of word embeddings, directly repairing the boundaries of entities recog- nized by character- bhased models, the performance of lexicon-assisted model strongly related to the quality of the lexicon. However, building a complete pre-defined lexicon costly and can hardly handle OOV issues.
In this paper, entity classificttin--asiiteed model is proposed to alleviate this problem. Above all, we design a mechanism to generate embeddings for each entity class using only the character- based BERT pre-trained model. It mainly consists of two procedures. The first step form word embeddings of entities appearing in the trainning sentences through character embeddings and the second step to aggregate entity embeddings by category and generate classification embeddings. Benefited from the inherent contextual information given in BERT representation, entities appearing under identical context can have identical representations. To further eliminate noise and extract common attributes between entities in the same class, weighted projection removal method is applied additionally.
After that, designed novel Attention mechanism to integrate entity classification embeddings into the character-based BERT model. The input sources of our attention architecture consists of set of preprocessed entity classes and an input sentence. In order to get series of weighted representation of the input sentence, the traditional Scaled Dot-Product Attention is revised and increased in dimension, providing greater weight characters identical to each entity class.
Results show that our model outperforms other Chinese NER models over a variety of datasets across different domains without external resources. Our code will be released XXXX

---

Under review as a conference paper at ICLR 2020
2
RELATED MODELS
Although character sequence labeling has been the dominant approach for Chinese NER, many works seek to add further information into the model.
As mentioned above, word-based NER models suffer from the segmentation-NER pipeline, one of the solutions is to import multitask structure. It is a common way to share mutual information between related tasks and boost the general performance. [Pnng & Dredze (2016) jointly trained CWS (Chinese word segmentation) task with NER task and yield sigmificant improvements in NER for Chinese social media. To further explore language regularities, more works Yang et al. (2016)
Ruder12 al. (2017) consider adding additional related tasks into the mission, employing parameter sharing selective sharing. Transfer learning is another popular way of improvement. [Sun et al. (2019a) Feng et al. (2018) utilized the abundant resources in English to boost the NER performances low resource languages including Chinese. Besides, Zhang & Yang (2018) investigate lattice- structured LSTM model for Chinese NER, which encodes a sequence of input characters as well as all potential words that match lexicon, having the merit of leveraging explicit word information over character sequence labeling without suffering from segmentation error. Liu et al. (2019) fur- ther adjust the structure of the lattice model, assigning word information ensuring that there is no shortcut paths between characters to prevent the model from degenerating into a partial word-based model. Different from the previous, our method does not require exter- nal resources including labeled data for other tasks predefined lexicons, making the model more practical.
single character and
Recently, many works exploit language processing with deep language models to improve the per- formance of downstream NLP tasks, such as ELMo Peters et al. (018), Radford et al. (2018). BeRTiDevlin et al. (2018), ERNIF et al. (2019b) Zhang et al. 2019), XLNET et al. 2019). These methods first pretrain neural networks on large-scale unlabeled text corpora, and then finetune downstream tasks. The dynamic embeddings learned from these
the models or representations models outperform the traditional static embeddings by containing more contextual information as well relieving the polysemy problem. Tagging Models based on these structures bring impres- sive results Sun et al. (2019) Kaneko & Komachil (2019) Lee et al. (2019). Our method proposed way extract the representation of the entity classification embeddings through the rich infor- mation contained in the pre-trained language model, which eliminates the need for external word embeddings.
What's more, the attention mechanism has shown impressive performance on many NLP tasks. Kaneko & Komachi (2019) investigate the effect of utilizing information not only from the final layer but also from intermediate layers of a pre-trained language representation model to detect grammatical errors. Rei al.) (2016) proposed novel architecture for combining the character- based representation with the word embedding by using attention mechanism, allowing the model to choose which information to use from each information source dynamically. Zhhu al. (2019) investigate a practical Convolutional Attention Network for Chinese NER which not depending on any external resources. The Scaled Dot-Product Attention proposed by Vaswani et al. (017)) is one of the mainstream models currently Kancko & Komachi (2019), Devlin et al. (2018). Sun al. (2019b). ZZang et al. (2019), Yang et al. (2019), Lee et (2019). In this paper, we revise the Scaled Dot-Product Attention to Classification Attention which would give weighted representation of the input sentences through series of entity classes.
3 MODEL
3.1 OVERALL ARCHITECTURE
As shown in Figure 1, the overall architecture of the proposed model is mainly composed of three parts: Embedding extraction for entity class and Classification Attention. The embedding extraction step is performed firstly in a pipeline structure and the results will be aggregated to the attention model. Details of the 2 parts will be elaborated as follow.
2

---

Under review as a conference paper at ICLR 2020
Eatraciia
prasen
Kontee
Valaple
Sofaer
Diqae
protiee
Desped
Rorse
nikgpe
Nigh
Kostee
Malgle
Sofaes
Degue
Tidarar
"kclaapp not canotlag pernoko and ae
Use lam
Burt Jasaba
Englsh Amscraties
18:
w...nnghh
Figure 1: The overall architecture of the proposed model.The detailed structure for Classification Embedding Extraction is will be shown in following Section
3.2 EMBEDDING EXTRACTION FOR ENTITY CLASS
Word Lexicon has played an important role in dealing model NER problems. However, a high- quality lexicon construction requires lot of manpower and might be incompetent to handle OOV issues. Since the ideal word embedding assumes that words with identical classes would have clus- tered representations, it is suggested that the embedding of known entities may help find new entities of the same class.
Pre-trained with large amount of Chinese corpus, the last hidden layer of BERT integrates context information into each character of the input sentences. It not only excludes synonym interference compared with traditional static embeddings, but also provides potential information for the embed- ding vector of entities in the context. Inspired from Arora et al. (2016), in this paper, we illustrate simple way to extract word embeddings of entities in the hidden outputs of BERT. Figure 2.a gives the flow chart of the whole procedure.
Let X {21, T2 ... ..) denotes the input sentence in character level, where the total length and Y = {11, ... Vr. denotes the label sequences accordingly. Let H {h1, .2...hh. be the h2)
last hidden layer BERT. Under the BMES label scheme Wang Xu (2017). both the class and location of each entity can be inferred by the label sequence Y. To fix the size, the embedding of an entity the averaged embedding of its composed character.
Emb(W ( ord) = Emb (charm. m. char char,)
Emb (char;)
1
h
Different from the original method in Arora et al. (016)), the smooth inverse frequency is abandoned due to the dynamic semantic information encoded in H. To further get the most representative embedding of a given entity, the weighted projection of the word embeddings on their first singular
vector is removed. Let Matrix be the collected embeddings of certain class, whose columns are { (Emb, Embg... ... Embg]. and let u be its first singular vector and a be the weight parameter ac(0, 1]. For all Emb; do: Emb, Emb; awru" Emb;
is assumed that when the gap between the training data and the testing data large, a may takes larger value to show more common representation of the entity class. Finally, the embedding of a certain entity class can be gathered through the output of averagePooling layer. The whole processure is shown in Figure2(aa.
3

---

Under review as a conference paper at ICLR 2020
IDC
Ny Npw
LDC
ORG
Ny
Bnn uenoeeeeee
(a)
(b)
Figure 2: (agure2(a) EEmbedding extraction for entity classsification. Figure 2(b) Illustration of the attention architecture.
3.3 CLASSIFICATION ATTENTION
As shown in Figure 1, the Classification Attention architecture is designed the basis of Multi- Head Attention proposed by Transformer Vaswani et al. (2017). The biggest difference for this architecture lies in the input source which contains set of entity classification embeddings and character embeddings of the input sentence. It aims to get a series of weighted representation of the input sentences so that characters similar to a particular entity class could have greater feature weights.
Firstly, consider situation with only one entity class. Matrix Q and Matrix K adopted to extract the characteristics of the input sentence and the entity. Here, the length of the input entity class can be seen as an imput sentence with only one character. By applying the traditional Scaled Dot-Product Attention, the similarity between each character of the input sentence and the entity class can be calculated by:
QK VTTE
Similarity(Q,kK) softmax
The output product gives indication of how likely character in the input sentence is to be the input entity. Since the final result we would like to get is the weighted embedding of the input sentence. So, the V matrix in architecture is calculated from the input sentence other than the entity. The final result can be got through
Attention(Q2 (Q, V) similarity (Q,tit(QQQ4 K)OV V O
Figure 2(b) gives an illustration of the attention architecture for multiple entity classes. For the sake of simplicity, the softmax step and the dropout step are omitted, and the normalized scaling dot product between the entity classification embedding and the sentence embedding after the Key-layer and then Query-laye respectively is directly displayed. The intensity of the weight is proportional to the depth color giving indication of how likely the input character to the particular entity. After applying the Hadamard product, the weighted representation of the original sentence after Value-layer is finally obtained.
During the calculation, Multi-head mechanism is adopted for each entity class and the entire cal- culation is done once by matrix one-dimensional higher than BERT. After the attention stage, the merged output then passed along with the original character embeddings of the input sentence through the same follow-up layers as BERT, namely Self-output layer, Intermediate layer, and Out- put layer. Figure gives the detailed structure of the layers where two residential nets are applied to ensure the validity of the deep network.
4 EXPERIMENT
An extensive set of experiments on varies Chinese NER datasets are conducted to demonstrate the effectiveness of our proposed model. In the following sections, will list the details of datasets,
4

---

Under review as a conference paper at ICLR 2020
Table 1: Data splitting for each dataset
Dataset
Type
Train
Dev
Test
Original
15724
4301
4346
OntoNotes4
Max length 256
15729
4301
4347
Original
46364
4365
MSRA
Split Dev Data
42000
4364
4365
Max length 256
42080
4370
4375
Original
1350
270
270
Weibo
Max length 256
1350
270
270
Original
3821
463
477
Resume
Max length 256
3821
463
477
parameter settings well experimental results and conclude result analysis. Standard precision (P), recall (R) and micro Fl-score (FI) are used as evaluation metrics.
4.1
EXPERIMENTAL SETTINGS
4.1.1 DATA DESCRIPTION
The four datasets in our experiments namely OntoNotes 4 Weischedel al. (2011). MSRA Levow (006), Weibo Peng & Dredze (2013) He & Sun (2116) and Resume Zhang & Yang (2018) are widely used evaluate the performances of Chinese NER. The OntoNotes4 dataset carries pre- cise annotation from the news field which contains 4 classes: PER (Person), ORG (Organization)) LOC (Location) and GPE (Geo-Political Entity). The training, development and test splits of the OntoNotes4 datasets adopted in experiments are the same with Che et al. (2013). MSRA NER dataset of SIGHAN Bakeoff 2006 is another precisely annotated sources from the news field with larger amount of data. The training set and test set provided directly with 3 classes: PER, ORG, LOC. To make a development set, several data from the training set are randomly picked up in experiments. Details for data splitting of MSRA shown in Table 1. For generated data, small dataset extracted from Sina Weibo is considered. The Weibo dataset is annotated with four en- tity types: PER (Person), ORG (Organization), LOC (Location) and GPE (Geo-Political Entity) and is already divided into training, development and test sets. The resume dataset is released by Zhang Yang (2018). With the success of the Lattice model, the Resume dataset has also become popular experimental dataset in many papers. It is annotated with eight types of named entities: CONT (Country), EDU (Educational Institution), LOC, PER, ORG, PRO (Profession), RACE (Ethnicity Background) and TITLE (Job Title).
Especially, since our model conducted on the basis of BERT, the maximum number of input characters is limited. To keep balance between the computing resources and maintaining the original data, we set the maximum number of inputting characters as 256, where data with exceeded numbers of input characters Table where the original total number of sentences for each data set as well as the total number of sentences after cutting are both given. It should be noticed that in order obtain fair verification, the test results of the split sentences are merged together to produce the final result.
cut into shorter sentences. The statisties of the datasets are shown in
4.1.2 IMPLEMENTATION DETAILS
We use the 12-layer pre-trained BERT the base model to generate the classification embeddings for each entity class and to conduct classification attention. We set the word embedding size of the entity class to 768, the same with the BERT's hidden size. The classification attention layer is set to 1. The maximum length of the input sentences is set to 256 and the training batches are set to 16. Other shared BERT-BE BERT-Baase model remain unchanged. We use BERT-Ade Adam to optimize all the trainable parameters. Learning rate is set to 5 10-5 initially and decays during training rate of 0.01.
defined in the configuration json file along with the released
5

---

Under review as a conference paper at ICLR 2020
4.1.3 EXPERIMENTAL RESULTS
In this section, we will give the experimental results of our proposed model and previous state-of- the-art methods on Weibo dataset, Chinese Resume dataset, OntoNotes 4 dataset and MSRA dataset, respectively. To exclude the performance improvements from BERT's own mechanism, we design 2 Baselines based on BERT along with the Classification Attention method, namely BERT CRF model, BERT with 13 hidden layers (i.e. one additional layer to exclude the improvements self- attention itself) + CRF model. We treat NER sequential labeling problem and adopt BMOES lagging style in this paper since it has been shown that models using BMOES are remarkably better than BIO Yang et al. 2018
4.1.4 WEIBO DATASET
Table 3 shows the comparisons on Weibo Dataset. We compare our proposed model with the latest models on Weibo dataset. It could be observed that our proposed model achieves state-of-the-an performance. In the first block of Table 3, we report the performances of our baselines. Our baseline BERT-base + CRF achieves an Fi-score of 71.05 and BERT with 13 hidden layers + CRF achieves an F1-score of 69.14. In the second block of Table 3, the performances of 3 latest models for Chinese NER are given where Zhang & Yang (0118) and et al. (2019) utilized external resources including predefined lexicon and pre-trained word embeddings. The lattice structure proposed by Zhang & Yangl (2018) achieves Fi-score of 58.79. By excluding the no shortcut paths between characters, Liu al. (219)) improves the Fl-score to 59.84. Zhu al. (2019) does not rely on any external resources and achieves the Fi-score of 59.31. Compared with the baseline model, Zhang Yang (018)) Liu et al. (2019) and Zhu et al. (2019) are built on the basis of LSTM of which the feature extraction ability is relatively pooler than Transformer. The performance of our proposed model is presented in the last line in Table 3. Since the Fl-scores of the 2 baseline models are around 70, it is assumed that the gap between the training data and the test data is relatively large. So, the entity embeddings obtained from the training data of a certain class may also get relatively large distribution differences with the test class. Hence, the weight for direction removal should be aligned with a larger weight. By doing so, most of the divergences in the distribution of the be retained. It
embeddings in the training set can be excluded, while the common component can be recognized from the result that both the Precision score and Recall score of our model are all higher than the base-line model which demonstrate the effectiveness of our proposed model.
Table 3: Results of OntoNotes Dataset
Table 2: Results of Weibo Dataset
Models
R
Models
R
F1
BERT + CRF
79.08
80.21
79.64
BERT + CRF
70.38
71.74
71.05
13-layers BERT
74.06
84.15
78.78
13-layers BERT + CRF
CRR
70.71
67.63
69.14
Zhang & Yang (2018)
76.35
71.56
73.88
Zhang & Yang (2018)
58.79
Liu et al. 2019
76.09
72.85
74.43
Liu et al. 2019)
59.84
(Zhu, Wang
75.05
72.29
73.64
Zhu, Wang and
Karlsson 2019)
55.38
62.98
59.31
Karlsson (2019)
Sun et al. 2019a)
80.21
Our model
71.16
73.91
72.51
Zhao et al. 2018
76.69
71.91
74.22
Our model
79.23
80.92
80.07
4.1.5 ONTONOTES DATASET
Table 4 shows the comparisons on OntoNotes 4 dataset. We compare our proposed model with the latest models on OntoNotes dataset. the first block of Table 4, we report the performance of the baseline models. Our baseline BERT-base CRF achieves Fl-score of 79.64 and BERT with 13 hidden layers + CRF achieves an Fl-score of 78.78.
In the second block of Table 4, the performances of the 5 latest models are given. Zhang & Yang 2018), Liu et al. (201)) Zhu et al. (2019) and Zhao et al. (2018) are developed based on LSTM whereas Sun et al. 2019a) utilized the character embeddings of BERT. For external resources, Zhang & Yang 2018). Liu et al. 2019) and Zhao et al. (2018) utilized predefined lexicon embeddings and Sun al. (2019a) utilized pre-trained English NER model. It can be illustrated from the result in

---

Under review as a conference paper at ICLR 2020
Table 4. Zhu al. (2019) with no external resources achieves Fl-score of 73.64, Zhang & Yang 2018 with lattice structure achieves Fl-score of 73.88. Zhao et al. (2018) and Liu et al. (2019) achieve 74.22 and 74.43 respectivelyThe The performance of our proposed model is shown in the last line in Table 4. Since the Fi-scores of the 2 baseline models are around 79, the weight for direction removal should also be aligned with lager weight. It can be recognized from the result that model got the higest Precision score and competitive Fl score without requiring external resources.
4.1.6 RESUME DATASET
Table 5 shows the comparisons on Resume dataset. We compare our proposed model with the latest models on Resume dataset. In the first block of Table 5, we report the performances of the baseline models. Our baseline BERT-base CRF achieves an Fl-score of 95. and BERT with 13 hidden layers + CRF achieves an F1-score of 95.99 97. In the second block of Table 5, the performances of latest model are given. [iu et al. (2019), Zhu et al. (2019) and Zhang & Yang (2018) are all built on the basis of LSTM. For external resources, Liu et al. (2019) and Zhang & Yang (2018) utilized predefined lexicon and pre-trained word embeddings, can be told from the result in Table 5. Zhu et al.] (2019) with no external resources achieves Fi-score of 94.94, Zhhang & Yang (2018) with lattice structure achieves F1-score of 95.21. By excluding the no shortcut paths between characters,
Liu et al. (2019) improves the F1 F1-score to 95.21. The performance of our proposed model is given the last line Table 5. Since the F1-scores of the baseline models are around 95, it assumed that the gap between the training data and the test data is relatively small. So, the entity embeddings obtained from the training data of a certain class may also get relatively small distribution differences with the test class. Hence, the weight for direction removal should be aligned with smaller weight. can be told from the result that our model got both competitive Precision score and Recall score than the Baseline model and forms the highest F1 -cooe finally.
Table 5: Results of MSRA Dataset
Table 4: Results of Resume Dataset
Models
R
Models
P
R
F1
BERT CRF
94.98
94.64
94.81
BERT + CRF
95.94
95.58
95.76
13-layers BERT
13-layers BERT + CCR
94.91
94.24
94.57
95.45
96.50
95.97
CRF
Zhang & Yang (2018)
Liu et al. 2019)
95.27
95.15
95.21
93.57
92.79
93.18
(Zhu, Wang and
95.05
94.82
94.94
Liu et al. 2019
94.58
92.91
93.74
Karlsson 2019)
(Zhu, Wang and
Zhang & Yang (2018)
93.53
92.42
92.97
94.81
94.11
94.46
Karlsson 2019)
Zhao et al. (0118)
93.66
93.05
93.35
Our model
95.73
96.38
96.06
Our model
95.20
94.49
94.86
4.1.7 MSRA DATASET
Table 6 shows the comparisons on MSRA dataset. We compare our proposed model with the latest models on MSRA dataset. In the first block of Table 6, we report the performances of the baseline models. Our baseline BERT-base CRF achieves an Fl-score of 94.81 and BERT with 13 hidden layers + CRF achieves an F1-score of 94.57.
In the second block of Table the performances of 4 latest model are given. Zhang & Yang 2018). Liu et al.] (2019). Zhu et al. (2019) and Zhao et al. (2018) are all built on the basis of LSTM. For external resources, /Zhang & Yang (2018), Liu et al. (2019) and Zhao et al. (2018) utilized predefined lexicon and pre-trained word embeddings. It can be told from the result in Table 6. Zhu ct al. (2019) with no external resources achieves F1-score of 92.97, Zhang & Yang 2018 with lattice structure achieves Fl-score of 98.18. Zhao et al. (2018) and Liu et al. (2019) achieve 93.35 and 93.74 respectively.' The performance of our proposed model is given at the last line in Table 6. Since the Fi-scores of the 2 baseline models are around 90, the weight for direction removal should be aligned with a smaller It can be told from the result that our model got competitive Recall score and the highest Recall score and forms the highest F1-score finally.
7

---

Under review as a conference paper at ICLR 2020
oRG
a
040
Chrice
8:1
CRG
McDonalds
0.032
Dae Cluud
PER
Cloud
-0.004
Figure 3: Entity-sentent relation heatmap of the case in Weibo Dataset
4.1.8 CASE STUDY
To explore the process of the Classification Attention, case study of Weibo Dataset will be shown this section. We pick a real test sentence from Weibo. The four entity classification embeddings extracted from the training data which do not contain entities in the test sentence. The model is trained following the hyperparameters described in the Implementation Details section. The English translation of the input sentence is:
took photo at McDonald's West Lake Yintai store: Gather with Dear Cloud at McDonald's.
entity-sentence relation heatmap shown in Figure 3, where strength of the relationship proportional to the depth of the color. The coordinate shows the categories of different entities and the x coordinate shows the input Chinese character as well as their corresponding labels.
According the ground truth, most entities accurately given greater weight after attention though the example sentence contains several irregular symbols. One exception is that only "Cloud" was marked class "PER" in the ground truth but the whole term "Dear cloud" may also be as whole username that belongs to class "PER"
It should also be noticed that though method presents indication of all potential entities, the weight of each entity class not ideally proportional to the ground truth label. West Lake Yintai store assigned with greater likelihood "GPE" than "LOC". At the same time, the organization McDonald's is assigned with equal weight in "GPE"," PER" and "ORG".
4.2 RESULTS ANALYSIS
Our proposed model outperforms previous work on MSRA, Weibo and Chinese Resume dataset and gains competitive results on OntoNotes 4 datasets without using any external resources. The exper- iments results demonstrate the effectiveness of our proposed model. Compared to Baseline BERT CRE, performance improvement after adding Classification Attention mechanism verifies that entity classification embeddigns obtained by our model can give extra information to locate entities. Moreover, compared to Baseline BERT with 13 layers + CRF, the better performance of our model excludes the benefits from deeper self-attention layer.
CONCLUSION
In this paper, we seek to find an entity classification-assitee model for Chinese NER by inves- tigating mechanism to generate classification embeddings for each entity class using only the character-based BERT pre-trained model and designing a Classification Attention for Chinese NER. It observes series of weighted representation of the input sentences that characters similar to particular entity class could have greater feature weights. In future work, we plan to conduct deeper research classification embeddings generation and model structure improvements.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. simple but tough-to-beat baseline for sentence embeddings. 2016.
Wanxiang Che, Mengqiu Wang. Christopher D Manning, and Ting Liu. Named entity recognition with bilingual constraints. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 52,62, 2013.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480). 2018.
Xiaocheng Feng, Xiachong Feng, Bing Qin, Zhangyin Feng, and Ting Liu, Improving low resource named entity recognition using cross- loos-llnguua knowledge transfer. In IJCAI, pp- 4071-4077, 2018.
Hangfeng He and Xu Sun. F-score driven max margin neural network for named entity recognition in chinese social media. arXiv preprint arXiv:1671.04234 2016
Masahiro Kaneko and Mamoru Komachi. Multi-head multi-layer attention deep language repre- sentations for grammatical error detection. arXiv preprint arXiv:1904.07334, 2019.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. arXiv preprint /603.00 2016.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jac- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv: 900.06646 2019.
Gina-Anne Levow. The third international chinese language processing bakeoff:: Word segmenta- tion and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pp. 108-117, 2006.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. arXiv preprint (.XKv:88/2.0944// 2018.
Wei Liu, Tongge Xu, Qinghua Xu, Jiayu Song, and Yueran Zu. An encoding strategy based word- character Istm for chinese ner. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Pupers), pp. 2379-2389, 2019.
Nanyun Peng and Mark Dredze. Named entity recognition for chinese social media with jointly trained embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 548-554, 2015.
Nanyun Peng and Mark Dredze. Multi-task domain adaptation for sequence tagging. arXiv preprint arXiv:1608.0 02689, 2016.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint wrXiv:8802.05365, 2018.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. Conll- 2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Confer- ence on EMNLP and CoNLL-Shared Task, pp. 1-40. Association for Computational Linguistics, 2012.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technical report, Technical report, OpenAl, 2018.
Marek Rei, Gamal KO Crichton, and Sampo Pyysalo. Attending to characters in neural sequence labeling models. arXiv preprint 1611:6361 2016.
Sebastian R2 Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Sluice networks: Learning what share between loosely related tasks. stat, 1050:23, 2017.

---

Under review as a conference paper at ICLR 2020
Erik F Sang and Fien De Meulder. Introduction to the con1l-2003 shared task: Language- independent named entity recognition. arXiv preprint cs/0306050, 2003.
Linghao Sun, Huixiong Yi, and Huanhuan Chen. Back attention knowledge transfer for low-resource named entity recognition. arXiv preprint arXiv:1906.01 183, 2019a.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint ar 1904. 09223, 2019b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Chunqi Wang and Bo Xu. Convolutional neural network with word embeddings for chinese word segmentation. arXiv preprint arXiv:1711.04411 2017.
Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Mar- cus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. Ontonotes release 4.0. LDC201 /T03, Philadelphia, Penn.: Linguistic Data Consortium, 2011.
Jie Yang. Shuailong Liang, and Yue Zhang. Design challenges and misconceptions in neural se- quence labeling. arXiv preprint
2018.
Zhilin Yang, Ruslan Salakhutdinov, and William Cohen. Multi-task cross-lingual sequence tagging 2016.
from scratch. arXiv preprint
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint orXiv:1906.08237 2019.
Yue Zhang and Jie Yang. Chinese ner using lattice Istm. arXiv preprint arXiv:/805.02023. 2018
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Emii: Enhanced language representation with informative entities. arXiv preprint 1rXiv:190..07129, 2019.
Dong Yang Zhao, JiuMing Huang, and Yan Jia. Chinese name entity recognition using highway- Istm-erf. In Proceedings of the 2018 International Conference on Algorithms, Computing and Artificial Intelligence, pp. 64. ACM, 2018
Yuying Zhu, Guoxin Wang, and Börje F Karlsson. Can-ner: Convolutional attention network forchi- nese named entity recognition. arXiv preprint 19Xiv:1904.02141, 2019.
10


Reviews:
[{"review_id": "rylcHyzA9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Comments by sections : \n\nSummary : \n\nThe use of entity is not clear. Are you referring to named-entities ?\n\n1 INTRODUCTION\n\n\"Due to the differences in language structure\" : this paragraph is not clear. It should say explicitly that in Chinese word can be composed of one or a couple of characters and that word boundaries can not detected graphically. \n\n \"A common mitigation is to add external lexicon as a reference\"   references are needed on how to includes words in embedding (except just training word embeddings)\n \n \" an entity classification-assisted model\" : not very clear : the classification is assisted ?\n \n \" The first step is to form word embeddings of entities appearing in the trainning sentences through character embeddings and the second step is to aggregate entity embeddings by category and generate classification embeddings\" : is it a multi-task training ?\n \n \"After that, we designed a novel Attention mechanism to integrate entity \" : is it the same model or two different propositions of the paper ? \"After that\" is not very clear as a transition.\n \n Section 2 :\n \n \"more works Yang et al. (2016) Ruder12 et al. (2017) \" : strange formulation\n \n \"What’s more, the attention mechanism...\" : odd expression.\n \n \"In this paper, we revise the Scaled Dot-Product Attention to Classification Attention which would give a weighted representation of the input sentences through a series of entity classes.\" : maybe it should be moved to the introduction as a novelty proposed by the paper.\n \n \n 3.2  EMBEDDING EXTRACTION FOR ENTITY CLASS\n \n \" the smooth inverse frequency is abandoned\" : why ? please explain this choice.\n \"the weighted projection of the word embeddings on their first singular vector is removed.\" : explain. If it is a common practice, give a citation otherwise justify this choice.\n \n 3.3 CLASSIFICATION ATTENTION \n \n Here again, the proposed attention system is described but not justified : why would a class specific attention system be better ? What are the expected advantages ?\n \n 4.1.3 EXPERIMENTAL RESULTS\n Experiments are conducted on 4 dataset and the proposed model is compared to a \"standard\" BERT-based model and several results form the litterature. The proposed model outperforms sometimes the other models, often by a small margin as it is usually the case in NER experiments. \n But more insight on the strengths of the models should be given by conducting an ablation study. \n  \n  \n In conclusion ,this paper present an incremental improvement over BERT-based NER for Chinese. The proposed approach is not sufficiently justified and experiments, even if showing improvements over state-of-the-art models or published results, does not sufficiently explore the benefits of the proposed model (with ablation study for example).  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ryejz8JhFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n      This paper discussed an approach to do named entity resolution (NER, the paper focuses only on Chinese NER but I think it could generalize to other languages as well). The idea is based on smart integration and extension of multiple existing building blocks: 1) BERT pre-trained model 2) a previous work to get document embedding by doing weighted average of word embedding (https://openreview.net/pdf?id=SyK00v5xx) and 3) Scaled dot-product attention mechanism applied directly to multi-label classification. The \"Introduction\", \"Related work\", and \"Experiment Settings\" sections are well written and covers many details and decent references. Especially, the \"experiments\" section is described in a great amount of details, which should be very helpful for reproducibility. \n      \nContributions:\n      * The author found an interesting application of the original algorithm (https://openreview.net/pdf?id=SyK00v5xx) to represent the entity class embedding based on averaging \"BERT\" embeddings of all the component words. This could be implemented as a pre-processing step against any training dataset to derive \"pre-learned\" entity class embedding.\n      * Instead of the common approach of connecting the BERT sequence outputs directly to CRF layer, the author added an intermediate layer to calculate the classification attention between a sentence (sequence of token embedding) and any entity class (based on the above pre-learned entity embedding). This result plus the original sentence embedding are concatenated.  The concatenation is further fed into a few additional layers to produce the final inputs into CRF layer.\n\nWeakness:\n     * The paper lacks novelty. As pointed above, I did not see that the contribution from the paper is sufficiently original. It is a good application of various existing methods though.\n\nI also have a few suggestions/questions below:\n\n* The ERNIE paper (https://arxiv.org/abs/1907.12412v1) is mentioned in the related work. Since ERNIE can potentially learn a good vocab for Chinese, did you ever compare your approach vs ERNIE+CRF? \n* There is one paper that I know which is pretty relevant to what you are doing here, which is probably worth a reference. https://arxiv.org/abs/1805.04174.  In that paper, the idea is to co-learn a class embedding and perform text classification. Their class attention is performed through dot-production attention though.\n* The Table index seems wrong in your paper. (I think Table 2 is not mentioned in your paper, but all tables (3-6) is offset by 1). \n* There are some minor typos or places that need some clarifications.\n   - in the abstract: \"character-based\" model. This is a little confusing. Because BERT is a word-piece based model. word-piece could across multiple characters for English. IIUC, You probably want to say \"Chinese-character\" instead of character.\n   - in \"Introduction\", \"providing greater weight to characters identical to each entity class\", you might want to revise this sentence to clarify its meaning further.\n   - In section 3.2, you might want to give some explanation to some notations (the first time you refer to it). For example, what is $L$, what is $m$ and $n$.  What is $S$?  Also why the denominator of Emb(Word) is not $n-m+1$? \n\n   - The last paragraph in section 3.3 needs more clarification as well. How do you merge the three tensors after attention stage? (a concatenation ?) . The last sentence mentioned \"residential\", I guess instead you want to say \"residual\".  You might also want to clarify where the \"3 layers\" of residual appear in your network.\n\n  - In your experiment, (if I did not miss), did you freeze the BERT parameters and entity embeddings when finetuning your NER model?\n\n  - in Table 2 and Table 3, why the 13-layer BERT + CRF performs significantly worse on Recall (Table 2) and significantly better on Recall (Table 3)? \n\n\n  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeUmO-R5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to improve the performance of Chinese NER by developing a novel attention mechanism that leverages BERT pre-trained model which considers bi-directional context. Experiments on a number of tasks show that the proposed approach is effective.\n\nComments:\n[1] A bunch of experiments are conducted\n[2] Chinese NER is a hard problem, but it would be great to see the proposed approach generalizable to other tasks. So, the contribution of this paper is limited\n[3] The proposed algorithm is simple and effective, but the novelty is a bit low\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is interested in Chinese Name Entity Recognition, building on a BERT pre-trained model. All reviewers agree that the contribution has limited novelty. Motivation leading to the chosen architecture is also missing. In addition, the writing of the paper should be improved.

Decision: Reject