Venue: NAACL 2019

# An Encoding Strategy Based Word-Character LSTM for Chinese NER

Wei Liu\({}^{1,2}\), Tongge Xu\({}^{3,1}\), Qinghua Xu\({}^{1,2}\), Jiayu Song\({}^{2}\), Yueran Zu\({}^{2}\)

\({}^{1}\)Hefei Innovation Research Institute, Beihang University

\({}^{2}\)School of Computer Science and Engineering, Beihang University

\({}^{3}\)School of Cyber Science and Technology, Beihang University

{liuwei1206, xutg, kuqh_buaa, heavensyc, yueranzu}@buaa.edu.cn

###### Abstract

A recently proposed lattice model has demonstrated that words in character sequence can provide rich word boundary information for character-based Chinese NER model. In this model, word information is integrated into a shortcut path between the start and the end characters of the word. However, the existence of shortcut path may cause the model to degenerate into a partial word-based model, which will suffer from word segmentation errors. Furthermore, the lattice model can not be trained in batches due to its DAG structure. In this paper, we propose a novel word-character LSTM(WC-LSTM) model to add word information into the start or the end character of the word, alleviating the influence of word segmentation errors while obtaining the word boundary information. Four different strategies are explored in our model to encode word information into a fixed-sized representation for efficient batch training. Experiments on benchmark datasets show that our proposed model outperforms other state-of-the-arts models.

## 1 Introduction

Name Entity Recognition(NER) is a basic task of many NLP systems including Information Retrieval Virga and Khudanpur (2003), Relationship Extraction Miwa and Bansal (2016), Question Answering Molla et al. (2006). The main task of NER is to identify named entities such as person, location, organization, etc. in given text. Various methods have been proposed to tackle this problem, including Hidden Markov Models(HMMs) Saito and Nagata (2003), Maximum Entropy Models(ME) Chieu and Ng (2003), Support Vector Machines(SVM) Ekbal and Bandyopadhyay (2010) and Conditional Random Fields(CRF) Feng et al. (2006). With the development of deep learning, neural networks Huang et al. (2015); Lample et al. (2016); Habibi et al. (2017) have been introduced to NER task. To avoid the segmentation errors, most of neural Chinese NER models are character-based.

Although character-based method has achieved good performance, it does not exploit word information in character sequence. Entity boundaries usually coincide with some word boundaries, which suggests that words in character sequence can provide rich boundary information for character-based model. To integrate words information into character-based model, Zhang and Yang (2018) propose a lattice-structured LSTM

Figure 1: An example of the lattice model degenerates into a partial word-based model. Due to the shortcut path ”\(\mathbb{IL}\)(River)” \(\rightarrow\) “\(\mathbb{IL}\)(River Water)” \(\rightarrow\) “\(\mathbb{IK}\)(Water)”, the model incorrectly predicts that “\(\mathbb{IL}\)(River)” and “\(\mathbb{JK}\)(Water)” belong to the same entity. Red labels(without underline) denote predicted labels, and blue labels(with underline) denote gold labels.



model to encode a sequence of input characters as well as all potential words that match a lexicon. Their model is an extension of character-based LSTM-CRF model and uses extra "shortcut paths" to link the memory cell between the start and the end characters of a word for utilizing word information. And the gated recurrent unit is used to control the contribution of shortcut paths and path between adjacent characters. However, as the study of Yang et al. (2018) shown, the gate mechanism fails to choose the right path sometimes. As shown in Figure 1, wrong choices may cause lattice model to degenerate into a partial word-based model, which suffers from word segmentation errors. In addition, due to the variable length of words, the length of the whole path is not fixed. Besides, each character is bounded with a variable-sized candidate word sets, which means the amount of incoming and outcoming paths is not fixed either. In this case, lattice LSTM model is deprived of the power of batch training, and hence it is highly inefficient.

To address the above problems, we propose a novel word-character LSTM(WC-LSTM) to integrate word information into character-based model. To prevent our model from degenerating into a partial word-based model, we assign word information to a single character and ensure that there are no shortcut paths between characters. Specifically, word information is assigned to its end character and start character in forward WC-LSTM and backward WC-LSTM respectively. We introduce four strategies to extract fixed-sized useful information from different words, which ensures that our proposed model can perform batch training without losing word information.

We demonstrate the effectiveness of our architecture on four widely used datasets. Experimental results show that our proposed model outperforms other state-of-the-art models on the four datasets.

Our contributions of this paper can be concluded as follows:

* We propose a novel word-character LSTM(WC-LSTM) to incorporate word information into character-based model.
* We explore four different strategies to encode word information into a fixed-sized vector, which enables our proposed model to be trained in batches and adapted to various application scenarios.
* Our proposed model outperforms other models and achieves new state-of-the-art over four Chinese NER datasets. We release the source code for further research1. Footnote 1: [https://github.com/liuwei1206/CCW-NER](https://github.com/liuwei1206/CCW-NER)

## 2 Related Work

Neural Networks have been shown to achieve impressive results on Name Entity Recognition task Gregoric et al. (2018); Lin and Lu (2018). Based on the level of granularity, most of the models can be divided into three categories: word-based models, character-based models, and hybrid models.

**Word-Based Models**. Collobert and Weston (2008) propose one of the first word-based models for NER, with feature constructed from orthographic features, dictionaries and lexicons Yadav and Bethard (2018). Collobert et al. (2011) replace the hand-crafted features with word embeddings. Huang et al. (2015) propose a BiLSTM-CRF model for NER and achieves good performance. Ma and Hovy (2016) and Chiu and Nichols (2016) use CNN to capture spelling characteristics and Lample et al. (2016) use LSTM instead. When applied to Chinese NER, the above models all suffer from segmentation errors, since Chinese word segmentation is compulsory for those models.

**Character-Based Models**. Peng and Dredze (2015) propose to add segmentation features for better recognition of entity boundary. Dong et al. (2016) integrate radical-level features into character-based model. To eliminate the ambiguity of character, Sun and He (2017) take the position of character into account. Although the above models have achieved good results, they all ignore word information in character sequence.

**Hybrid Models**. Some efforts have been made to integrate word boundary information into character-based models. Motivated by the success of multi-task learning for Natural Language Processing Liu et al. (2016, 2017); Zhang et al. (2018), Peng and Dredze (2016) first proposed to jointly train Chinese NER with Chinese word segmentation(CWS) task. Cao et al. (2018) apply adversarial transfer learning framework to integrate the task-shared word boundary information into Chinese NER task. Another way to obtain word boundary information is proposed by Zhang and Yang (2018), using a lattice LSTM to integrate word information into character-based model, which is similar to what is proposed in 

this paper. The main differences are as follows. Firstly, they exploit word information by a DAG-structured LSTM, while we use a chain-structured LSTM. Secondly, instead of integrating to the hidden state of LSTM, our model add word information into the input vector. Finally, our model can be trained in batches and is more efficient.

## 3 Method

The architecture of our proposed model is shown in Figure 2. Same as the widely used neural Chinese NER model, we use LSTM-CRF as our main network structure. The differences between our model and a standard LSTM-CRF model are mainly on the embedding layer and LSTM and can be summarized as follows. First, we represent a Chinese sentence as a sequence of character-words pairs to integrate word information into each character. Second, to enable our model to train in batches and to meet different application requirements, we introduce four encoding strategies to extract fixed-sized but different information from words. Finally, a chain-structured word-character LSTM is used to extract features from both character and word for better predicting.

Next, we will explain the main ideas for each component, including word-character embedding layer, word encoding strategy, and word-character LSTM.

Formally, we denote a Chinese sentence as \({\bf s}=\{c_{1},c_{2},...,c_{n}\}\), where \(c_{i}\) denotes the \(i_{th}\) character. We use \(c_{b,e}\) to denote a character subsequence in \({\bf s}\), which begins with \(b_{th}\) character and ends with \(e_{th}\) character. Take the sentence in Figure 2 for example, \(c_{1,2}\) is "\(\Box\)(Rise)". We use \(\overrightarrow{ws_{i}}\) to denote words assigned to \(i_{th}\) character in forward WC-LSTM, which are a set of character subsequences \(c_{b,i}\), where \(b<i\) and \(c_{b,i}\) matches a word in lexicon **D**. The lexicon **B** is the same as the one used in Zhang and Yang (2018), which is built by using automatically segmented large raw text. Similarly, we use \(\overleftarrow{ws_{i}}\) to denote the words for \(i_{th}\) character in backward WC-LSTM, which are a set of character subsequences \(c_{i,e}\), where \(e>i\) and \(c_{i,e}\) matches a word in lexicon **D**. Finally, the sentence \({\bf s}\) is represented as \(\overrightarrow{{\bf rs}}=\{(c_{1},\overrightarrow{ws_{1}^{*}}),(c_{2}, \overrightarrow{ws_{2}^{*}}),...,(c_{n},\overrightarrow{ws_{n}^{*}})\}\) in our model, and its reverse representation is \(\overleftarrow{{\bf fs}}=\{(c_{n},\overleftarrow{ws_{n}}),(c_{n-1}, \overleftarrow{ws_{n-1}}),...,(c_{1},\overleftarrow{ws_{1}})\}\).

### Word-Character Embedding Layer

In our model, Each position \(i\) in \(\overrightarrow{{\bf rs}}\) consists of two parts: \(i_{th}\) character \(c_{i}\) and the assigned words \(\overrightarrow{ws_{i}}\). The origin number of words in \(\overrightarrow{ws_{i}}\) is \(s_{i}^{t}\), and words are sorted by their length. We ensure each \(\overrightarrow{ws_{i}}\) has the same number \(s_{i}^{p}\)2 in the whole batch by padding. We embed each character \(c_{i}\) in dis

Figure 2: The architecture of our unidirectional model. The blue part can be seen as a standard character-based model but with a word-character LSTM(WC-LSTM), and the red part indicates the process of encoding word information into a fixed-size representation. Word information is integrated into the end character of the word. Where ”\(<\)PAD\(>\)” denotes padding value; ”Stgy” denotes a certain encoding strategy and \(\oplus\) denotes concatenation operation.

 

tributional space as \({\bf x}_{i}^{c}\):

\[{\bf x}_{i}^{c}={\bf e}^{c}(c_{i}) \tag{1}\]

where \({\bf e}^{c}\) denotes a pre-trained character embedding lookup table. Similarly, for each \(\overrightarrow{ws_{i}}=\{\overrightarrow{w_{i1}},...,\overrightarrow{ws_{is_{ i}^{p}}}\}\), the \(l_{th}\) word \(\overrightarrow{w_{il}}\) in \(\overrightarrow{ws_{i}}\) is represented using

\[{\bf x}_{il}^{\overrightarrow{u}}={\bf e}^{w}(\overrightarrow{w_{il}}) \tag{2}\]

where \({\bf e}^{w}\) denotes a pre-trained word embedding lookup table. As a result, the distributional representation of words \(\overrightarrow{ws_{i}}\) is \(\{{\bf x}_{i1}^{\overrightarrow{u}},...,{\bf x}_{is_{i}^{p}}^{\overrightarrow{ u}}\}\).

### Words Encoding Strategy

Although the number of assigned words \(s_{i}^{p}\) for each character \(c_{i}\) is same in one batch, the number varies from batch to batch. As a result, the size of input to the model is not fixed, which is not conducive to batch training. To acquire fixed-sized input, we introduce four different encoding strategies in this section. And we use \({\bf x}_{i}^{\overrightarrow{uw}}\) to denote the final representation of word information for position \(i\) in following sections.

**Shortest Word First:** For each word set \(\overrightarrow{ws_{i}}=\{\overrightarrow{w_{i1}},...,\overrightarrow{ws_{is_{ i}^{p}}}\}\), we simply select word whose length is the shortest, i.e. \(\overrightarrow{w_{i1}}\). Then

\[{\bf x}_{i}^{\overrightarrow{uw}}={\bf x}_{i1}^{\overrightarrow{u}} \tag{3}\]

**Longest Word First:** Contrary to the shortest word first, we select word whose length is the longest, i.e. \(\overrightarrow{ws_{is_{i}^{p}}}\). Note that \(s_{i}^{t}\) may be \(0\), in this case, we set it to \(1\). Then

\[{\bf x}_{i}^{\overrightarrow{uw}}={\bf x}_{is_{i}^{t}}^{\overrightarrow{w}} \tag{4}\]

**Average:** While the first two strategies can only use the information of partial words, we introduce an average strategy to utilize all word information. As its name indicates, the average strategy computes the centroid of the embeddings of all elements except paddings in word set , i.e. \(\{\overrightarrow{w_{i1}},...,\overrightarrow{ws_{is_{i}^{t}}}\}\). If \(s_{i}^{t}=0\), we simply average all the padding value in the word set. Then

\[{\bf x}_{i}^{\overrightarrow{uw}}=\begin{cases}\frac{1}{s_{i}^{t}}\sum_{l=1} ^{s_{i}^{t}}{\bf x}_{il}^{\overrightarrow{uw}},&\text{if $s_{i}^{t}>0$}\\ \frac{1}{s_{i}^{t}}\sum_{l=1}^{s_{i}^{p}}{\bf x}_{il}^{\overrightarrow{uw}},& \text{if $s_{i}^{t}=0$}\end{cases} \tag{5}\]

**Self-Attention:** Inspired by self-attention mechanism applied to sentence embedding Lin et al. (2017), we exploit self-attention to better capture useful information from assigned words. For simplicity, we denote all the \({\bf x}_{il}^{\overrightarrow{u}}\) as \(W_{i}\), which has the size \(s_{i}^{p}\)-by-\(d^{w}\), where \(d^{w}\) denotes the dimensionality of word embedding \({\bf e}^{w}\).

\[W_{i}=({\bf x}_{i1}^{\overrightarrow{u}},...,{\bf x}_{is_{i}^{p}}^{\overrightarrow {u}}) \tag{6}\]

We use self-attention mechanism to obtain a linear combination of \(s_{i}^{p}\) word embeddings in \(W_{i}\). The attention mechanism takes \(W_{i}\) as input, and generates a weight vector \({\bf a}_{i}\).

\[{\bf a}_{i}=softmax({\bf w}_{2}tanh(W_{1}W_{i}^{T})) \tag{7}\]

\(W_{1}\) is a weight matrix with the size of \(d_{a}\)-by-\(d^{w}\) and \({\bf w}_{2}\) is a \(d_{a}\) dimensional vector, where \(d_{a}\) is a hyperparameter. Both of them are trainable parameters.

If \(s_{i}^{t}>0\), we use the mask to exclude the padding values; otherwise we reserve them. Finally, we use \({\bf a}_{i}\) to get the weighted sum of all words.

\[{\bf x}_{i}^{\overrightarrow{uw}}=\begin{cases}\sum_{l=1}^{s_{i}^{t}}a_{il}{ \bf x}_{il}^{\overrightarrow{u}},&\text{if $s_{i}^{t}>0$}\\ \sum_{l=1}^{s_{j}^{p}}a_{il}{\bf x}_{il}^{\overrightarrow{uw}},&\text{if $s_{i}^{ t}=0$}\end{cases} \tag{8}\]

where \(a_{il}\) denotes the \(l_{th}\) value in \({\bf a}_{i}\).

### Word-Character LSTM(WC-LSTM)

Inspired by the way character bigram is integrated into sequence labeling model Chen et al. (2015); Yang et al. (2017), we concatenate each \({\bf x}_{i}^{c}\) with \({\bf x}_{i}^{\overrightarrow{uw}}\) to utilize word information. And this is quite different from the way used in Zhang and Yang (2018), since they use extra shortcut paths to integrate word information into the hidden layer of LSTM. By concatenating, there is no shortcut path in our model and information can only flow between adjacent characters, which ensures that our model will not degenerate into a partial word-based model. Then the WC-LSTM functions are:

\[\begin{bmatrix}\tilde{\mathbf{c}_{i}}\\ {\bf o}_{i}\\ {\bf i}_{i}\\ {\bf f}_{i}\end{bmatrix}=\begin{bmatrix}tanh\\ \sigma\\ \sigma\\ \sigma\end{bmatrix}\left({\bf W}_{p}\begin{bmatrix}{\bf x}_{i}\\ {\bf h}_{i-1}\end{bmatrix}+{\bf b}_{p}\right) \tag{9}\]

\[\begin{split}{\bf x}_{i}&={\bf x}_{i}^{c}\oplus{\bf x}_{i}^{ \overrightarrow{uw}}\\ {\bf c}_{i}&=\tilde{\mathbf{c}}_{i}\odot{\bf i}_{i}+{\bf c}_{i-1}\odot{\bf f }_{i}\\ {\bf h}_{i}&={\bf o}_{i}\odot tanh({\bf c}_{i})\end{split} \tag{10}\]

where \({\bf o}_{i}\), \({\bf i}_{i}\) and \({\bf f}_{i}\) denote output gate, input gate and forget gate respectively. \({\bf W}_{p}\) and \({\bf b}_{p}\) are parameters of affine transformation; \(\sigma\) denotes the 

logistic sigmoid function; \(\oplus\) denotes concatenation operation and \(\odot\) denotes elementwise multiplication.

The bidirectional WC-LSTM is applied in our model to leverage both information from the past and the future. To get the future information, we use a second WC-LSTM that reads the reverse representation of \(\overrightarrow{\mathbf{rs}}\), i.e., \(\overleftarrow{\mathbf{fs}}=\{(c_{n},\overleftarrow{ws}_{n}),(c_{n-1}, \overleftarrow{ws}_{n-1}),...,(c_{1},\overleftarrow{ws}_{1})\}\). And the following operations to get each backward WC-LSTM hidden vector \(\overleftarrow{\mathbf{h}_{i}}\) is the same as the one in the forward WC-LSTM. Finally, the update of each bidirectional WC-LSTM unit can be written as follows:

\[\begin{split}&\overrightarrow{\mathbf{x}_{i}}=\mathbf{x}^{c}_{i} \oplus\mathbf{x}^{\overrightarrow{w}_{i}}\\ &\overleftarrow{\mathbf{s}_{i}}=\mathbf{x}^{c}_{i}\oplus \mathbf{x}^{\overleftarrow{ws}}_{i}\\ &\overrightarrow{\mathbf{h}_{i}}=\overline{\mathrm{WC}-\mathrm{ LSTM}}(\overleftarrow{\mathbf{h}_{i-1}},\overrightarrow{\mathbf{x}_{i}})\\ &\overleftarrow{\mathbf{h}_{i}}=\overleftarrow{\mathrm{WC}- \mathrm{LSTM}}(\overleftarrow{\mathbf{h}_{i+1}},\overleftarrow{\mathbf{x}_{i}}) \\ &\mathbf{h}_{i}=\overrightarrow{\mathbf{h}_{i}}\oplus \overleftarrow{\mathbf{h}_{i}}\end{split} \tag{11}\]

where \(\overrightarrow{\mathbf{h}_{i}}\) and \(\overleftarrow{\mathbf{h}_{i}}\) are hidden states at position \(i\) of forward and backward WC-LSTM respectively, and \(\oplus\) denotes concatenation operation.

### Decoding and Training

Considering the dependencies between successive labels, we use a CRF layer to make sequence tagging. We define matrix \(\mathbf{O}\) to be scores calculated based on the output \(\mathbf{H}=\{\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{n}\}\):

\[\mathbf{O}=\mathbf{W}_{o}\mathbf{H}+\mathbf{b}_{o} \tag{12}\]

For a label sequence \(\mathbf{y}=\{y_{1},y_{2},...,y_{n}\}\), we define its probability to be:

\[p(\mathbf{y}|\mathbf{s})=\frac{exp\left(\sum_{i}\big{(}\mathbf{O}_{i,y_{i}}+ \mathbf{T}_{y_{i-1},y_{i}}\big{)}\right)}{\sum_{\tilde{\mathbf{y}}}exp\left( \sum_{i}\big{(}\mathbf{O}_{i,\tilde{y}_{i}}+\mathbf{T}_{\tilde{y}_{i-1},\tilde {y}_{i}}\big{)}\right)} \tag{13}\]

Where \(\mathbf{W}_{\mathbf{o}}\) and \(\mathbf{b}_{\mathbf{o}}\) are paramters to calculate \(\mathbf{O}\); \(\mathbf{T}\) is a transition score matrix and \(\tilde{\mathbf{y}}\) denotes all possible tag sequences.

While decoding, we use the Viterbi algorithm to find the label sequences that obtained the highest score:

\[\mathbf{y}^{*}=arg\max_{\mathbf{y}\in\tilde{\mathbf{Y}}}\sum_{i}\big{(} \mathbf{O}_{i,y_{i}}+\mathbf{T}_{y_{i-1},y_{i}}\big{)} \tag{14}\]

Given N manually labeled data \(\{(\mathbf{s}_{j},\mathbf{y}_{j})\}|_{j=1}^{N}\), we minimize the sentence-level negative log-likelihood loss to train the model:

\[L=-\sum_{j}log(p(\mathbf{y}_{j}|\mathbf{s}_{j})) \tag{15}\]

## 4 Experiments

### Experimental Settings

**Dataset**. We evaluate our model on four datasets, including OntoNotes4 (Weischedel et al., 2011), MSRA (Levow, 2006), Weibo NER (Peng and Dredze, 2015) and a Chinese resume dataset (Zhang and Yang, 2018). Both OntoNotes4 and MSRA datasets are news in simplified Chinese. Weibo NER dataset is social media data, which is drawn from the Sina Weibo. Chinese resume dataset consists of resumes of senior executives, which is annotated by (Zhang and Yang, 2018). For OntoNotes, we use the same training, development and test splits as (Che et al., 2013). For other datasets which have already been split, and we don't change them. We summarize the datasets in Table 1.

**Implementation Details**. We utilize the character and word embeddings used in (Zhang and Yang, 2018), both of which are pre-trained on Chinese Giga-Word using word2vec model. Following (Zhang and Yang, 2018), we use the word embedding dictionary as Lexicon **D** in our model. For characters and words that do not appear in the pretrained embeddings, we initialize them with a uniform distribution3. When training the model, character embeddings and word embeddings are updated along with other parameters.

Footnote 3: The range is \(\left[-\sqrt{\frac{3}{dim}},+\sqrt{\frac{3}{dim}}\right]\), where \(dim\) demotes the size of embedding.

For hyper-parameter configurations, we mostly refer to the settings in (Zhang and Yang, 2018). We set both character embedding size and word embedding size to 50. The dimensionality of each unidirectional multi-input LSTM hidden states is 100 for Weibo NER and Chinese Resume, and 200 for OntoNote 4 and MSRA. For self-attention strategy, we set the \(d_{a}\) to 50. To avoid overfitting, we apply dropout to both embeddings and LSTM with a rate of 0.5. We use SGD to optimize all the trainable parameters. Learning rate is set to 0.015 initially and decays during training at a rate

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
* & 4365 \\ \hline Weibo NER & 1350 & 270 & 270 \\ \hline Chinese resume & 3821 & 463 & 477 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the datasets
 

of 0.05.

For evaluation, we use the Precision(P), Recall(R) and F1 score as metrics in our experiments.

### Experimental Results

**OntoNotes**. Table 2 shows the experimental results on OntoNote 4 dataset. The "Input" column shows the representation of input sentence, where "Gold seg" means a sequence of words with gold-standard segmentation, and "No seg" means a sequence of character without any segmentation.

The first block in Table 2 are the results of word-based models Wang et al. (2013); Che et al. (2013); Yang et al. (2016). By using gold-standard segmentation and external labeled data, all of them achieve good performance. But the only resource used in our model are pretrained character and word embeddings.

The first two rows in the second block show the performance of the lattice model and character-based model. The character baseline denotes the original character-based BiLSTM-CRF model. Zhang and Yang (2018) propose a lattice LSTM to exploit word information in character sequence, giving the F1 score of 73.88%. Compared with the character baseline, lattice model gains 8.92% improvement in F1 score, which shows the importance of word information in character sequence.

In the last four rows, we list the results of our proposed model. The results show that all of our models outperform other character-based models, and the one with self-attention strategy achieves the best result. Without gold-standard segmentation and external labeled data, our model gives competitive results to the word-based models on this dataset. Compared with the character baseline, our model with self-attention obtains 9.48% improvement in F1 score, which proves the effectiveness of our way to integrating word information. Compared with lattice model, all of our models achieve better results, which shows that our approach to integrating word information is more reasonable than lattice model.

**MSRA**. Table 3 shows the results on MSRA dataset. Zhang et al. (2006) and Zhou et al. (2013) use the statistical model with rich hand-crafted features. Dong et al. (2016) exploit radical features in Chinese character. Cao et al. (2018) joint train Chinese NER task with Chinese word segmentation, in which adversarial learning and self-attention mechanism are applied for better performance. We can observe that our proposed models outperformance the above models and the one with average strategy achieves new state-of-the-art performance.

**Weibo**. Table 4 shows the results4 on Weibo dataset. The "NE", "NM" and "Overall" columns denote F1-score for named entities, nominal entities(excluding named entities) and both respectively. We can see that WC-LSTM model with longest word first strategy achieves new state-of-the-art performance. Multi-task learning Peng and Dredze (2015, 2016); Cao et al. (2018) and semi-supervised learning Sun and He (2017); He and Sun (2017) are the most common methods

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**Models** & **NE** & **NM** & **Overall** \\ \hline Peng and Dredze (2015) & 51.96 & 61.05 & 56.05 \\ Peng and Dredze (2016) & **55.28** & 62.97 & 58.99 \\ Sun and He (2017) & 54.50 & 62.17 & 58.23 \\ He and Sun (2017) & 50.60 & 59.32 & 54.82 \\ Cao et al. (2018) & 54.34 & 57.35 & 58.70 \\ Lattice Zhang and Yang (2018) & 53.04 & 62.25 & 58.79 \\ \hline Character baseline & 47.98 & 57.94 & 52.88 \\ \hline WC-LSTM + shortest & 52.99 & 65.75 & **59.20** \\ WC-LSTM + longest & 52.55 & **67.41** & **59.84** \\ WC-LSTM + average & **53.19** & 64.17 & 58.67 \\ WC-LSTM + self-attention & 49.86 & 65.31 & 57.51 \\ \hline \end{tabular}
\end{table}
Table 4: Results on Weibo NER

\begin{table}
\begin{tabular}{|l|l|c|c|} \hline
**Input** & **Models** & **P** & **R** & **F1** \\ \hline \multirow{3}{*}{Gold seg} & Wang et al. (2013) & 76.43 & 72.32 & 74.32 \\  & Cho et al. (2013) & 77.71 & 75.21 & **75.02** \\  & Yang et al. (2016) & 65.9 & 71.84 & 68.57 \\ \hline \multirow{6}{*}{No seg} & Lattice Zhang and Yang (2018) & 76.35 & 71.56 & 73.88 \\  & Character baseline & 70.08 & 60.53 & 64.95 \\ \cline{2-4}  & WC-LSTM + shortest & **76.39** & 72.39 & **74.34** \\ \cline{1-1}  & WC-LSTM + longest & 76.52 & 72.76 & **74.16** \\ \cline{1-1}  & WC-LSTM + average & 76.04 & 72.03 & **73.98** \\ \cline{1-1}  & WC-LSTM + self-attention & 76.09 & **72.85** & **74.43** \\ \hline \end{tabular}
\end{table}
Table 2: Results on OntoNotes 

for Weibo NER task due to the small amount of training data. All of the above models require additional cross-domain or semi-supervised data. Compared with those models, our model does not need additional labeled data; we only exploit pre-trained character and word embeddings.

**Resume**. Table 5 shows the results on Chinese Resume dataset. Consistent with the previous results, our models outperform lattice model Zhang and Yang (2018). The above experimental results strongly verify that our method to utilize word information is more effective than the lattice model.

Our proposed model has achieved state-of-the-art results on **various domains** such as news, social media, and Chinese resume.

### Efficiency

To further explore the efficiency of our model, we conduct some comparative experiments on training time and convergence speed. The lattice model proposed in Zhang and Yang (2018) is our principal comparison object, since it also utilizes the word information in character sequence. Our model is an extension of the character-based model, so we also report the results on character-based model as character baseline. We only conduct our experiments on OnteNotes dataset due to space limitation. And we choose the model with the self-attention strategy for the comparative experiments, as it outperforms other strategies on OntoNotes dataset.

The training time of each epoch for all models is shown in Table 6. The lattice model needs the most training time for each epoch, since it can only be trained with batch_size=1 due to its complex DAG structure. Compared with it, our model with batch_size=1 only need half of the training time. Which shows that our model is more efficient. With batch_size=8, our model is nearly 6 times faster than the lattice model, which further demonstrates the efficiency of our model. Compared with the character baseline, our model only adds a small amount of training time but greatly improves the performance. All the experiments are conducted on a single GPU with NVIDIA Tesla K40m.

Figure 3 shows the learning curve of the models in Table 6. As we can see from the figure, whether with batch_size=1 or 8, our model can converge within the same epochs as lattice model does. But compared with the lattice model, our model with batch_size=8 only takes about 1/7 of their training time per epoch. Besides, we can observe from Figure 3, both our model and lattice model significantly outperform the character baseline, which shows the importance of the word information again.

### Detailed Analysis Case Study

Word information is very useful for Chinese NER task, since it can provide rich word boundary information. To verify that our model can better utilize the boundary information, we analyze an example from OntoNotes dataset. As shown in Table 7, the character-based model cannot detect the existence of the entity "\(\nexists[X]{}\)(Northeast Asia)" without word information. The lattice model incorrectly recognizes "\(\nexists[X]{}\)

\begin{table}
\begin{tabular}{l c} \hline  & **Time(s)/epoch** \\ \hline Character baseline (\(batch\_size\)=1) & 880 \\ Character baseline (\(batch\_size\)=8) & 253 \\ Lattice & 2245 \\ WC-LSTM (\(batch\_size\)=1) & 980 \\ WC-LSTM (\(batch\_size\)=8) & 350 \\ \hline \end{tabular}
\end{table}
Table 6: Time per epoch of models

Figure 3: Convergence curve of models. Our model can converge within the same epochs as lattice model does. ”1” and ”8” denotes batch_size. Lattice model can only be trained with batch_size=1 due to its DAG structure.

 

* [leftmargin=0.5cm]
**State:** (Northeast Asian Continental Bridge)" as an entity, which is caused by the wrong selection of paths. Different from the lattice model, our models are not disturbed by the wrong boundary information and make the correct predictions.
**Strategies Analysis**. In this part, we analyze the difference between strategies. The application scenarios of shortest word first and longest word first can be explained by Nested Name Entity Recognition Ju et al. (2018); Sohrab and Miwa (2018). Short word first is good at identifying inner nested entities due to the short word information, while longest word first tends to identify flat entities with the help of long word information. Taking ""XIL\(\Xi\)H"Yangtze River Delta)" as an example, shortest word first recognizes ""XL(Yangtze)" and ""XHL(Delta)" as entities, but longest word first tend to think that they are part of the entity ""XIL\(\Xi\)H"Yangtze River Delta)". Both results are reasonable, but the right result depends on specific needs.

The average and self-attention strategies are the combination of all words information and can use more information. Intuitively, they should outperform the shortest word first and the longest word first. But results on Weibo NER(Table 4) and Resume(Table 5) show the opposite effect. We conjecture that this is caused by the small amount of training data since more word information but small dataset will lead to overfitting. The average strategy is a special case of the self-attention strategy where all weights are the same, so we would like to see the latter outperforms the former when training data is sufficient. Surprisingly, the average strategy achieves higher F1 score than the self-attention strategy in MSRA dataset(Table 3). We carefully analyze the experimental results and find that there are a large number of informal texts in the MSRA test set. Specifically, the MSRA test set contains some very long sentences, in which there are a series of Chinese person name without delimiter. As shown in Table 8, when applied to such informal text, the self-attention strategy fails to determine the entity boundary sometimes while the average strategy correctly recognizes the entities. And we conjecture that, with more trainable parameters, the self-attention strategy can better fit the formal text in the training set but cannot adapt well to the informal data in the test set, so it performs worse than the average strategy.

Finally, the application scenarios of different strategies can be summarized as followings. If the training data is sufficient, we recommend using self-attention for formal texts and average strategy for informal texts. If there is only a very small amount of annotated data, we recommend using the shortest words first for inner nested entities and longest word first strategy for flat entities.

**Lexicon and Embeddings**. To further analyze the contribution from word lexicon and pretrained word embeddings, we conduct some comparative experiments by using the same word lexicon with and without pretrained embeddings. We choose the strategy that achieving the best performance for each dataset. We estimate the contribution of the lexicon by replacing pretrained word embeddings with randomly initialized embeddings5. As shown in Table 9, both lexicon

\begin{table}
\begin{tabular}{|l|c||c c c c c c|} \hline
**Source** & \#!f? & R?+R? & X?+R? & Y?+R? \\ (ruvenated) & New & Nontext Asian & Continental Bridge & & & & \\  & & & & & & & \\ \hline \multirow{2}{*}{**Latent words**} & Nott 

and pretrained word embeddings are useful to our model. However, different from the result in lattice modelYang et al. (2018), pretrained word embeddings contribute more than lexicon to our model. Taking the result on Ontonote for example, the contribution of pretrained embeddings can be estimated as \((9.48\%-2.86\%)=6.62\%\), which is higher than the contribution of lexicon \(2.86\%\). The results show that our model relies more on pre-trained embeddings instead of the lexicon, which explains the excellent performance of our model in different domains.

## 5 Conclusion and Future Work

In this paper, we propose a novel method to utilize word information in character sequence for Chinese NER. Four encoding strategies are introduced to extract fixed-sized but different information for batch training. By using WC-LSTM to extract features from the character vector and word vector, our model can effectively exploit word boundary information and mitigate the influence of word segmentation errors. Experiments on datasets in different domains show that our model is more efficient and faster than the lattice model and also outperforms other state-of-the-art models.

In the future, we plan to further improve and perfect the proposed method, such as exploring some strategies to handle OOV words. Also, the proposed methods can be further extended to other Chinese NLP tasks, such as CWS, Text Classification, and Sentiment Analysis.

## Acknowledgments

The research work is supported by the No. BHKX-17-07 project of Hefei Innovation Research Institute, Beihang University. We would like to thank Jie Yang for his open-source toolkit "NCRF++" Yang and Zhang (2018). Moreover, we sincerely thank all anonymous reviewers for their valuable comments.

## References

* November 4, 2018_, pages 182-192.
* Che et al. (2013) Wanxiang Che, Mengqiu Wang, Christopher D. Manning, and Ting Liu. 2013. Named entity recognition with bilingual constraints. In _Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9-14, 2013, Westin Beachtree Plaza Hotel, Atlanta, Georgia, USA_, pages 52-62.
* Chen et al. (2015) Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuanjing Huang. 2015. Long short-term memory neural networks for chinese word segmentation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 1197-1206.
* June 1, 2003_, pages 160-163.
* Chiu and Nichols (2016) Jason P. C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. _TACL_, 4:357-370.
* Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In _Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008)_, pages 160-167.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12.
* Dong et al. (2016) Chuanhai Dong, Jiajun Zhang, Chengqing Zong, Masanori Hattori, and Hui Di. 2016. Character-based LSTM-CRF with radical-level features for chinese named entity recognition. In _In Natural Language Understanding and Intelligent Applications_, pages 239-250.
* Ekbal and Bandyopadhyay (2010) Asif Ekbal and Sivaji Bandyopadhyay. 2010. Named entity recognition using support vector machine: A language independent approach. _International Journal of Electrical and Electronics Engineering_, 4(2):155-170.
* Feng et al. (2006) Yuanyong Feng, Le Sun, and Yuanhua Lv. 2006. Chinese word segmentation and named entity recognition based on conditional random fields models. In

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline  & **OntoNotes** & **Resume** & **MSRA** & **Welho** \\ \hline Character baseline & 64.95 & 93.35 & 88.37 & 52.88 \\ WC+STM + init & 67.81(+2.86) & 96.51(+1.16) & 90.68(+2.31) & 54.94(+2.06) \\ WC+STM + protein & 74.43(+9.48) & 95.21(+1.86) & 93.74(+5.37) & 59.84(+6.90) \\ \hline \end{tabular}
\end{table}
Table 9: Comparison F1 scores between our proposed model with and without pretrained word embeddings. Where ”init” and ”pretrain” denote without and with pretrained embeddings respectively. “+” denotes the boost value to baseline.

 

Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing_, pages 181-184.
* Gregoric et al. (2018) Andrej Zukov Gregoric, Yoram Bachrach, and Sam Coope. 2018. Named entity recognition with parallel recurrent neural networks. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_, pages 69-74.
* Habibi et al. (2017) Maryam Habibi, Leon Weber, Mariana L. Neves, David Luis Wiegandt, and Ulf Leser. 2017. Deep learning with word embeddings improves biomedical named entity recognition. _Bioinformatics_, 33(14):i37-ii38.
* He and Sun (2017) Hangfeng He and Xu Sun. 2017. A unified model for cross-domain and semi-supervised named entity recognition in chinese social media. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA._, pages 3216-3222.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _Computing Research Repository_, arXiv:1508.01991.
* Ju et al. (2018) Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 1446-1459.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 260-27.
* Levow (2006) Gina-Anne Levow. 2006. The third international chinese language processing bakeoff: Word segmentation and named entity recognition. In _Proceedings of the Fifth Workshop on Chinese Language Processing, SIGHAN@COLING/ACL 2006, Sydney, Australia, July 22-23, 2006_, pages 108-117.
* November 4, 2018_, pages 2012-2022.
* Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In _International Conference on Learning Representations 2017_.
* Liu et al. (2016) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Deep multi-task learning with shared memory for text classification. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 118-127.
* Liu et al. (2017) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_, pages 1-10.
* Ma and Hovy (2016)Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_.
* Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 1105-1116.
* Molla et al. (2006) Diego Molla, Menno van Zaanen, and Daniel Smith. 2006. Named entity recognition for question answering. In _Proceedings of the 2006 Australasian Language Technology Workshop (ALTW2006)_, pages 51-58.
* Peng and Dredze (2015)Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 548-554.
* Peng and Dredze (2016) Nanyun Peng and Mark Dredze. 2016. Improving named entity recognition for chinese social media with word segmentation representation learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016_.
* Peng and Dredze (2017) Nanyun Peng and Mark Dredze. 2017. Supplementary results for named entity recognition on chinese social media with an updated dataset. Technical report.
* Saito and Nagata (2003) Kuniko Saito and Masaaki Nagata. 2003. Multilanguage named-entity recognition system based on hmm. In _Proceedings of the ACL 2003 Workshop on Multilingual and Mixed-language Named Entity Recognition_.
* November 4, 2018_, pages 2843-2849.
* Sun and He (2017) Xu Sun and Hangfeng He. 2017. F-score driven max margin neural network for named entity recognition in chinese social media. In _Proceedings of the 15th 

Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017_, pages 713-718.
* Virga and Khudanpur (2003) Paola Virga and Sanjeev Khudanpur. 2003. Transliteration of proper names in cross-lingual information retrieval. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_.
* Wang et al. (2013) Mengqiu Wang, Wanxiang Che, and Christopher D. Manning. 2013. Effective bilingual constraints for semi-supervised learning of named entity recognizers. In _Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, July 14-18, 2013, Bellevue, Washington, USA_.
* Weischedel et al. (2011) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ann Taylor Nianwen Xue, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2011. Ontonotes release 4.0 ldc2011t03. dvd. In _Linguistic Data Consortium_.
* Yadav and Bethard (2018) Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2145-2158. Association for Computational Linguistics.
* 17th International Conference, CICLing 2016, Konya, Turkey, April 3-9, 2016, Revised Selected Papers, Part I_, pages 140-154.
* Yang and Zhang (2018) Jie Yang and Yue Zhang. 2018. Ncrf++: An open-source neural sequence labeling toolkit. In _Proceedings of ACL 2018, System Demonstrations_, pages 74-79. Association for Computational Linguistics.
* August 4, Volume 1: Long Papers_, pages 839-849.
* Yang et al. (2018) Jie Yang, Yue Zhang, and Shuailong Liang. 2018. Subword encoding in lattice LSTM for chinese word segmentation. _Computing Research Repository_, abs/1810.12594.
* Zhang et al. (2018) Qi Zhang, Xiaoyu Liu, and Jinlan Fu. 2018. Neural networks incorporating dictionaries for chinese word segmentation. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18)_, pages 5682-5689. Springer Berlin Heidelberg.
* Zhang et al. (2006) Suxiang Zhang, Ying Qin, Juan Wen, and Xiaojie Wang. 2006. Word segmentation and named entity recognition for SIGHAN bakcoffs. In _Proceedings of the Fifth Workshop on Chinese Language Processing, SIGHAN@COLING/ACL 2006, Sydney, Australia, July 22-23, 2006_, pages 158-161.
* Zhang and Yang (2018) Yue Zhang and Jie Yang. 2018. Chinese mer using lattice lstm. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018_, pages 1554-1564. Springer Berlin Heidelberg.
* Zhou et al. (2013) Junsheng Zhou, Weiguang Qu, and Fen Zhang. 2013. Chinese named entity recognition via joint identification and categorization. _Chinese journal of electronics_, 22(2):225-230.