F-Score Driven Max Margin Neural Network for Named Entity Recognition in Chinese Social Media

Abstract: We focus on named entity recognition (NER) for Chinese social media. With massive unlabeled text and quite limited labelled corpus, we propose a semi-supervised learning model based on B-LSTM neural network. To take advantage of traditional methods in NER such as CRF, we combine transition probability with deep learning in our model. To bridge the gap between label accuracy and F-score of NER, we construct a model which can be directly trained on F-score. When considering the instability of F-score driven method and meaningful information provided by label accuracy, we propose an integrated method to train on both F-score and label accuracy. Our integrated model yields 7.44\% improvement over previous state-of-the-art result.

With the development of Internet, social media plays an important role in information exchange. The natural language processing tasks on social media are more challenging which draw attention of many researchers BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . As the foundation of many downstream applications BIBREF4 , BIBREF5 , BIBREF6 such as information extraction, named entity recognition (NER) deserves more research in prevailing and challenging social media text. NER is a task to identify names in texts and to assign names with particular types BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 . It is the informality of social media that discourages accuracy of NER systems. While efforts in English have narrowed the gap between social media and formal domains BIBREF3 , the task in Chinese remains challenging. It is caused by Chinese logographic characters which lack many clues to indicate whether a word is a name, such as capitalization. The scant labelled Chinese social media corpus makes the task more challenging BIBREF11 , BIBREF12 , BIBREF13 .
To address the problem, one approach is to use the lexical embeddings learnt from massive unlabeled text. To take better advantage of unlabeled text, Peng and Dredze peng-dredze:2015:EMNLP evaluates three types of embeddings for Chinese text, and shows the effectiveness of positional character embeddings with experiments. Considering the value of word segmentation in Chinese NER, another approach is to construct an integrated model to jointly train learned representations for both predicting word segmentations and NER BIBREF14 .
However, the two above approaches are implemented within CRF model. We construct a semi-supervised model based on B-LSTM neural network to learn from the limited labelled corpus by using lexical information provided by massive unlabeled text. To shrink the gap between label accuracy and F-Score, we propose a method to directly train on F-Score rather than label accuracy in our model. In addition, we propose an integrated method to train on both F-Score and label accuracy. Specifically, we make contributions as follows:
We construct a semi-supervised model which is based on B-LSTM neural network and combine transition probability to form structured output. We propose a method to train directly on F-Score in our model. In addition, we propose an integrated method to train on both F-Score and label accuracy.
B-LSTM neural network can learn from past input features and LSTM layer makes it more efficient BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 . However, B-LSTM cannot learn sentence level label information. Huang et al. huang2015bidirectional combine CRF to use sentence level label information. We combine transition probability into our model to gain sentence level label information. To combine transition probability into B-LSTM neural network, we construct a Max Margin Neural Network (MMNN) BIBREF19 based on B-LSTM. The prediction of label in position INLINEFORM0 is given as: DISPLAYFORM0 
where INLINEFORM0 are the transformation parameters, INLINEFORM1 the hidden vector and INLINEFORM2 the bias parameter. For a input sentence INLINEFORM3 with a label sequence INLINEFORM4 , a sentence-level score is then given as: DISPLAYFORM0 
where INLINEFORM0 indicates the probability of label INLINEFORM1 at position INLINEFORM2 by the network with parameters INLINEFORM3 , INLINEFORM4 indicates the matrix of transition probability. In our model, INLINEFORM5 is computed as: DISPLAYFORM0 
We define a structured margin loss INLINEFORM0 as Pei et al. pei-ge-chang:2014:P14-1: DISPLAYFORM0 
where INLINEFORM0 is the length of setence INLINEFORM1 , INLINEFORM2 is a discount parameter, INLINEFORM3 a given correct label sequence and INLINEFORM4 a predicted label sequence. For a given training instance INLINEFORM5 , our predicted label sequence is the label sequence with highest score: INLINEFORM6 
The label sequence with the highest score can be obtained by carrying out viterbi algorithm. The regularized objective function is as follows: DISPLAYFORM0 INLINEFORM0 
By minimizing the object, we can increase the score of correct label sequence INLINEFORM0 and decrease the score of incorrect label sequence INLINEFORM1 .
Max Margin training method use structured margin loss INLINEFORM0 to describe the difference between the corrected label sequence INLINEFORM1 and predicted label sequence INLINEFORM2 . In fact, the structured margin loss INLINEFORM3 reflect the loss in label accuracy. Considering the gap between label accuracy and F-Score in NER, we introduce a new training method to train directly on F-Score. To introduce F-Score driven training method, we need to take a look at the subgradient of equation ( EQREF9 ): INLINEFORM4 
In the subgradient, we can know that structured margin loss INLINEFORM0 contributes nothing to the subgradient of the regularized objective function INLINEFORM1 . The margin loss INLINEFORM2 serves as a trigger function to conduct the training process of B-LSTM based MMNN. We can introduce a new trigger function to guide the training process of neural network.
F-Score Trigger Function The main criterion of NER task is F-score. However, high label accuracy does not mean high F-score. For instance, if every named entity's last character is labeledas O, the label accuracy can be quite high, but the precision, recall and F-score are 0. We use the F-Score between corrected label sequence and predicted label sequence as trigger function, which can conduct the training process to optimize the F-Score of training examples. Our new structured margin loss can be described as: DISPLAYFORM0 
where INLINEFORM0 is the F-Score between corrected label sequence and predicted label sequence.
F-Score and Label Accuracy Trigger Function The F-Score can be quite unstable in some situation. For instance, if there is no named entity in a sentence, F-Score will be always 0 regardless of the predicted label sequence. To take advantage of meaningful information provided by label accuracy, we introduce an integrated trigger function as follows: DISPLAYFORM0 
where INLINEFORM0 is a factor to adjust the weight of label accuracy and F-Score.
Because F-Score depends on the whole label sequence, we use beam search to find INLINEFORM0 label sequences with top sentece-level score INLINEFORM1 and then use trigger function to rerank the INLINEFORM2 label sequences and select the best.
Word segmentation takes an important part in Chinese text processing. Both Peng and Dredze peng-dredze:2015:EMNLP and Peng and Dredze peng-dredze:2016:P16-2 show the value of word segmentation to Chinese NER in social media. We present two methods to use word segmentation information in neural network model.
Character and Position Embeddings To incorporate word segmentation information, we attach every character with its positional tag. This method is to distinguish the same character at different position in the word. We need to word segment the text and learn positional character embeddings from the segmented text.
Character Embeddings and Word Segmentation Features We can treat word segmentation as discrete features in neural network model. The discrete features can be easily incorporated into neural network model BIBREF20 . We use word embeddings from a LSTM pretrained on MSRA 2006 corpus to initialize the word segmentation features.
We use a modified labelled corpus as Peng and Dredze peng-dredze:2016:P16-2 for NER in Chinese social media. Details of the data are listed in Table TABREF19 . We also use the same unlabelled text as Peng and Dredze peng-dredze:2016:P16-2 from Sina Weibo service in China and the text is word segmented by a Chinese word segmentation system Jieba as Peng and Dredze peng-dredze:2016:P16-2 so that our results are more comparable to theirs.
We pre-trained embeddings using word2vec BIBREF22 with the skip-gram training model, without negative sampling and other default parameter settings. Like Mao et al. mao2008chinese, we use bigram features as follow: INLINEFORM0 
We use window approach BIBREF20 to extract higher level Features from word feature vectors. We treat bigram features as discrete features BIBREF20 for our neural network. Our models are trained using stochastic gradient descent with an L2 regularizer.
As for parameters in our models, window size for word embedding is 5, word embedding dimension, feature embedding dimension and hidden vector dimension are all 100, discount INLINEFORM0 in margin loss is INLINEFORM1 , and the hyper parameter for the INLINEFORM2 is INLINEFORM3 . As for learning rate, initial learning rate is INLINEFORM4 with a decay rate INLINEFORM5 . For integrated model, INLINEFORM6 is INLINEFORM7 . We train 20 epochs and choose the best prediction for test.
We evaluate two methods to incorporate word segmentation information. The results of two methods are shown as Table TABREF22 . We can see that positional character embeddings perform better in neural network. This is probably because positional character embeddings method can learn word segmentation information from unlabeled text while word segmentation can only use training corpus.
We adopt positional character embeddings in our next four models. Our first model is a B-LSTM neural network (baseline). To take advantage of traditional model BIBREF23 , BIBREF24 such as CRF, we combine transition probability in our B-LSTM based MMNN. We design a F-Score driven training method in our third model F-Score Driven Model I . We propose an integrated training method in our fourth model F-Score Driven Model II .The results of models are depicted as Figure UID11 . From the figure, we can know our models perfrom better with little loss in time.
Table TABREF23 shows results for NER on test sets. In the Table TABREF23 , we also show micro F1-score (Overall) and out-of-vocabulary entities (OOV) recall. Peng and Dredze peng-dredze:2016:P16-2 is the state-of-the-art NER system in Chinese Social media. By comparing the results of B-LSTM model and B-LSTM + MTNN model, we can know transition probability is significant for NER. Compared with B-LSTM + MMNN model, F-Score Driven Model I improves the result of named entity with a loss in nominal mention. The integrated training model (F-Score Driven Model II) benefits from both label accuracy and F-Score, which achieves a new state-of-the-art NER system in Chinese social media. Our integrated model has better performance on named entity and nominal mention.
To better understand the impact of the factor INLINEFORM0 , we show the results of our integrated model with different values of INLINEFORM1 in Figure UID13 . From Figure UID13 , we can know that INLINEFORM2 is an important factor for us to balance F-score and accuracy. Our integrated model may help alleviate the influence of noise in NER in Chinese social media.
The results of our experiments also suggest directions for future work. We can observe all models in Table TABREF23 achieve a much lower recall than precision BIBREF25 . So we need to design some methods to solve the problem.
Thanks to Shuming Ma for the help on improving the writing. This work was supported in part by National Natural Science Foundation of China (No. 61673028), and National High Technology Research and Development Program of China (863 Program, No. 2015AA015404). Xu Sun is the corresponding author of this paper. The first author focuses on the design of the method and the experimental results. The corresponding author focuses on the design of the method.

Figure and table captions:
Table 1: Details of Weibo NER corpus.
Table 2: Two methods to incorporate word segmentation information.
Table 3: NER results for named and nominal mentions on test data.

Questions about this paper:
What is F-score obtained?

Answers:
Best F1 score obtained is 54.82% overall