Active Learning for Chinese Word Segmentation in Medical Text

Abstract: Electronic health records (EHRs) stored in hospital information systems completely reflect the patients' diagnosis and treatment processes, which are essential to clinical data mining. Chinese word segmentation (CWS) is a fundamental and important task for Chinese natural language processing. Currently, most state-of-the-art CWS methods greatly depend on large-scale manually-annotated data, which is a very time-consuming and expensive work, specially for the annotation in medical field. In this paper, we present an active learning method for CWS in medical text. To effectively utilize complete segmentation history, a new scoring model in sampling strategy is proposed, which combines information entropy with neural network. Besides, to capture interactions between adjacent characters, K-means clustering features are additionally added in word segmenter. We experimentally evaluate our proposed CWS method in medical text, experimental results based on EHRs collected from the Shuguang Hospital Affiliated to Shanghai University of Traditional Chinese Medicine show that our proposed method outperforms other reference methods, which can effectively save the cost of manual annotation.

Electronic health records (EHRs) systematically collect patients' clinical information, such as health profiles, histories of present illness, past medical histories, examination results and treatment plans BIBREF0 . By analyzing EHRs, many useful information, closely related to patients, can be discovered BIBREF1 . Since Chinese EHRs are recorded without explicit word delimiters (e.g., “UTF8gkai糖尿病酮症酸中毒” (diabetic ketoacidosis)), Chinese word segmentation (CWS) is a prerequisite for processing EHRs. Currently, state-of-the-art CWS methods usually require large amounts of manually-labeled data to reach their full potential. However, there are many challenges inherent in labeling EHRs. First, EHRs have many medical terminologies, such as “UTF8gkai高血压性心脏病” (hypertensive heart disease) and “UTF8gkai罗氏芬” (Rocephin), so only annotators with medical backgrounds can be qualified to label EHRs. Second, EHRs may involve personal privacies of patients. Therefore, they cannot be openly published on a large scale for labeling. The above two problems lead to the high annotation cost and insufficient training corpus in the research of CWS in medical text.
CWS was usually formulated as a sequence labeling task BIBREF2 , which can be solved by supervised learning approaches, such as hidden markov model (HMM) BIBREF3 and conditional random field (CRF) BIBREF4 . However, these methods rely heavily on handcrafted features. To relieve the efforts of feature engineering, neural network-based methods are beginning to thrive BIBREF5 , BIBREF6 , BIBREF7 . However, due to insufficient annotated training data, conventional models for CWS trained on open corpus often suffer from significant performance degradation when transferred to a domain-specific text. Moreover, the task in medical domain is rarely dabbled, and only one related work on transfer learning is found in recent literatures BIBREF8 . However, researches related to transfer learning mostly remain in general domains, causing a major problem that a considerable amount of manually annotated data is required, when introducing the models into specific domains.
One of the solutions for this obstacle is to use active learning, where only a small scale of samples are selected and labeled in an active manner. Active learning methods are favored by the researchers in many natural language processing (NLP) tasks, such as text classification BIBREF9 and named entity recognition (NER) BIBREF10 . However, only a handful of works are conducted on CWS BIBREF2 , and few focuses on medical domain tasks.
Given the aforementioned challenges and current researches, we propose a word segmentation method based on active learning. To model the segmentation history, we incorporate a sampling strategy consisting of word score, link score and sequence score, which effectively evaluates the segmentation decisions. Specifically, we combine information branch and gated neural network to determine if the segment is a legal word, i.e., word score. Meanwhile, we use the hidden layer output of the long short-term memory (LSTM) BIBREF11 to find out how the word is linked to its surroundings, i.e., link score. The final decision on the selection of labeling samples is made by calculating the average of word and link scores on the whole segmented sentence, i.e., sequence score. Besides, to capture coherence over characters, we additionally add K-means clustering features to the input of CRF-based word segmenter.
To sum up, the main contributions of our work are summarized as follows:
The rest of this paper is organized as follows. Section SECREF2 briefly reviews the related work on CWS and active learning. Section SECREF3 presents an active learning method for CWS. We experimentally evaluate our proposed method in Section SECREF4 . Finally, Section SECREF5 concludes the paper and envisions on future work.
In past decades, researches on CWS have a long history and various methods have been proposed BIBREF13 , BIBREF14 , BIBREF15 , which is an important task for Chinese NLP BIBREF7 . These methods are mainly focus on two categories: supervised learning and deep learning BIBREF2 .
Supervised Learning Methods. Initially, supervised learning methods were widely-used in CWS. Xue BIBREF13 employed a maximum entropy tagger to automatically assign Chinese characters. Zhao et al. BIBREF16 used a conditional random field for tag decoding and considered both feature template selection and tag set selection. However, these methods greatly rely on manual feature engineering BIBREF17 , while handcrafted features are difficult to design, and the size of these features is usually very large BIBREF6 .
Deep Learning Methods. Recently, neural networks have been applied in CWS tasks. To name a few, Zheng et al. BIBREF14 used deep layers of neural networks to learn feature representations of characters. Chen et al. BIBREF6 adopted LSTM to capture the previous important information. Chen et al. BIBREF18 proposed a gated recursive neural network (GRNN), which contains reset and update gates to incorporate the complicated combinations of characters. Jiang and Tang BIBREF19 proposed a sequence-to-sequence transformer model to avoid overfitting and capture character information at the distant site of a sentence. Yang et al. BIBREF20 investigated subword information for CWS and integrated subword embeddings into a Lattice LSTM (LaLSTM) network. However, general word segmentation models do not work well in specific field due to lack of annotated training data.
Currently, a handful of domain-specific CWS approaches have been studied, but they focused on decentralized domains. In the metallurgical field, Shao et al. BIBREF15 proposed a domain-specific CWS method based on Bi-LSTM model. In the medical field, Xing et al. BIBREF8 proposed an adaptive multi-task transfer learning framework to fully leverage domain-invariant knowledge from high resource domain to medical domain. Meanwhile, transfer learning still greatly focuses on the corpus in general domain. When it comes to the specific domain, large amounts of manually-annotated data is necessary. Active learning can solve this problem to a certain extent. However, due to the challenges faced by performing active learning on CWS, only a few studies have been conducted. On judgements, Yan et al. BIBREF21 adopted the local annotation strategy, which selects substrings around the informative characters in active learning. However, their method still stays at the statistical level. Unlike the above method, we propose an active learning approach for CWS in medical text, which combines information entropy with neural network to effectively reduce annotation cost.
Active learning BIBREF22 mainly aims to ease the data collection process by automatically deciding which instances should be labeled by annotators to train a model as quickly and effectively as possible BIBREF23 . The sampling strategy plays a key role in active learning. In the past decade, the rapid development of active learning has resulted in various sampling strategies, such as uncertainty sampling BIBREF24 , query-by-committee BIBREF25 and information gain BIBREF26 . Currently, the most mainstream sampling strategy is uncertainty sampling. It focuses its selection on samples closest to the decision boundary of the classifier and then chooses these samples for annotators to relabel BIBREF27 .
The formal definition of uncertainty sampling is to select a sample INLINEFORM0 that maximizes the entropy INLINEFORM1 over the probability of predicted classes: DISPLAYFORM0 
where INLINEFORM0 is a multi-dimensional feature vector, INLINEFORM1 is its binary label, and INLINEFORM2 is the predicted probability, through which a classifier trained on training sets can map features to labels. However, in some complicated tasks, such as CWS and NER, only considering the uncertainty of classifier is obviously not enough.
Active learning methods can generally be described into two parts: a learning engine and a selection engine BIBREF28 . The learning engine is essentially a classifier, which is mainly used for training of classification problems. The selection engine is based on the sampling strategy, which chooses samples that need to be relabeled by annotators from unlabeled data. Then, relabeled samples are added to training set for classifier to re-train, thus continuously improving the accuracy of the classifier. In this paper, a CRF-based segmenter and a scoring model are employed as learning engine and selection engine, respectively.
Fig. FIGREF7 and Algorithm SECREF3 demonstrate the procedure of CWS based on active learning. First, we train a CRF-based segmenter by train set. Then, the segmenter is employed to annotate the unlabeled set roughly. Subsequently, information entropy based scoring model picks INLINEFORM0 -lowest ranking samples for annotators to relabel. Meanwhile, the train sets and unlabeled sets are updated. Finally, we re-train the segmenter. The above steps iterate until the desired accuracy is achieved or the number of iterations has reached a predefined threshold. [!ht] Active Learning for Chinese Word Segmentation labeled data INLINEFORM1 , unlabeled data INLINEFORM2 , the number of iterations INLINEFORM3 , the number of samples selected per iteration INLINEFORM4 , partitioning function INLINEFORM5 , size INLINEFORM6 a word segmentation model INLINEFORM7 with the smallest test set loss INLINEFORM8 Initialize: INLINEFORM9 
 train a word segmenter INLINEFORM0 
 estimate the test set loss INLINEFORM0 
 label INLINEFORM0 by INLINEFORM1 
 INLINEFORM0 to INLINEFORM1 INLINEFORM2 compute INLINEFORM3 by branch information entropy based scoring model
 select INLINEFORM0 -lowest ranking samples INLINEFORM1 
relabel INLINEFORM0 by annotators
form a new labeled dataset INLINEFORM0 
form a new unlabeled dataset INLINEFORM0 
train a word segmenter INLINEFORM0 
estimate the new test loss INLINEFORM0 
compute the loss reduction INLINEFORM0 
 INLINEFORM0 INLINEFORM1 
 INLINEFORM0 
 INLINEFORM0 INLINEFORM1 with the smallest test set loss INLINEFORM2 INLINEFORM3 
CWS can be formalized as a sequence labeling problem with character position tags, which are (`B', `M', `E', `S'). So, we convert the labeled data into the `BMES' format, in which each character in the sequence is assigned into a label as follows one by one: B=beginning of a word, M=middle of a word, E=end of a word and S=single word.
In this paper, we use CRF as a training model for CWS task. Given the observed sequence, CRF has a single exponential model for the joint probability of the entire sequence of labels, while maximum entropy markov model (MEMM) BIBREF29 uses per-state exponential models for the conditional probabilities of next states BIBREF4 . Therefore, it can solve the label bias problem effectively. Compared with neural networks, it has less dependency on the corpus size.
First, we pre-process EHRs at the character-level, separating each character of raw EHRs. For instance, given a sentence INLINEFORM0 , where INLINEFORM1 represents the INLINEFORM2 -th character, the separated form is INLINEFORM3 . Then, we employ Word2Vec BIBREF30 to train pre-processed EHRs to get character embeddings. To capture interactions between adjacent characters, K-means clustering algorithm BIBREF31 is utilized to feature the coherence over characters. In general, K-means divides INLINEFORM4 EHR characters into INLINEFORM5 groups of clusters and the similarity of EHR characters in the same cluster is higher. With each iteration, K-means can classify EHR characters into the nearest cluster based on distance to the mean vector. Then, recalculating and adjusting the mean vectors of these clusters until the mean vector converges. K-means features explicitly show the difference between two adjacent characters and even multiple characters. Finally, we additionally add K-means clustering features to the input of CRF-based segmenter. The segmenter makes positional tagging decisions over individual characters. For example, a Chinese segmented sentence UTF8gkai“病人/长期/于/我院/肾病科/住院/治疗/。/" (The patient was hospitalized for a long time in the nephrology department of our hospital.) is labeled as `BEBESBEBMEBEBES'.
To select the most appropriate sentences in a large number of unlabeled corpora, we propose a scoring model based on information entropy and neural network as the sampling strategy of active learning, which is inspired by Cai and Zhao BIBREF32 . The score of a segmented sentence is computed as follows. First, mapping the segmented sentence to a sequence of candidate word embeddings. Then, the scoring model takes the word embedding sequence as input, scoring over each individual candidate word from two perspectives: (1) the possibility that the candidate word itself can be regarded as a legal word; (2) the rationality of the link that the candidate word directly follows previous segmentation history. Fig. FIGREF10 illustrates the entire scoring model. A gated neural network is employed over character embeddings to generate distributed representations of candidate words, which are sent to a LSTM model.
We use gated neural network and information entropy to capture the likelihood of the segment being a legal word. The architecture of word score model is depicted in Fig. FIGREF12 .
Gated Combination Neural Network (GCNN)
To effectively learn word representations through character embeddings, we use GCNN BIBREF32 . The architecture of GCNN is demonstrated in Fig. FIGREF13 , which includes update gate and reset gate. The gated mechanism not only captures the characteristics of the characters themselves, but also utilizes the interaction between the characters. There are two types of gates in this network structure: reset gates and update gates. These two gated vectors determine the final output of the gated recurrent neural network, where the update gate helps the model determine what to be passed, and the reset gate primarily helps the model decide what to be cleared. In particular, the word embedding of a word with INLINEFORM0 characters can be computed as: DISPLAYFORM0 
where INLINEFORM0 and INLINEFORM1 are update gates for new combination vector INLINEFORM2 and the i-th character INLINEFORM3 respectively, the combination vector INLINEFORM4 is formalized as: DISPLAYFORM0 
where INLINEFORM0 and INLINEFORM1 are reset gates for characters.
Left and Right Branch Information Entropy In general, each string in a sentence may be a word. However, compared with a string which is not a word, the string of a word is significantly more independent. The branch information entropy is usually used to judge whether each character in a string is tightly linked through the statistical characteristics of the string, which reflects the likelihood of a string being a word. The left and right branch information entropy can be formalized as follows: DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 denotes the INLINEFORM1 -th candidate word, INLINEFORM2 denotes the character set, INLINEFORM3 denotes the probability that character INLINEFORM4 is on the left of word INLINEFORM5 and INLINEFORM6 denotes the probability that character INLINEFORM7 is on the right of word INLINEFORM8 . INLINEFORM9 and INLINEFORM10 respectively represent the left and right branch information entropy of the candidate word INLINEFORM11 . If the left and right branch information entropy of a candidate word is relatively high, the probability that the candidate word can be combined with the surrounded characters to form a word is low, thus the candidate word is likely to be a legal word.
To judge whether the candidate words in a segmented sentence are legal words, we compute the left and right entropy of each candidate word, then take average as the measurement standard: DISPLAYFORM0 
We represent a segmented sentence with INLINEFORM0 candidate words as [ INLINEFORM1 , INLINEFORM2 ,..., INLINEFORM3 ], so the INLINEFORM4 ( INLINEFORM5 ) of the INLINEFORM6 -th candidate word is computed by its average entropy: DISPLAYFORM0 
In this paper, we use LSTM to capture the coherence between words in a segmented sentence. This neural network is mainly an optimization for traditional RNN. RNN is widely used to deal with time-series prediction problems. The result of its current hidden layer is determined by the input of the current layer and the output of the previous hidden layer BIBREF33 . Therefore, RNN can remember historical results. However, traditional RNN has problems of vanishing gradient and exploding gradient when training long sequences BIBREF34 . By adding a gated mechanism to RNN, LSTM effectively solves these problems, which motivates us to get the link score with LSTM. Formally, the LSTM unit performs the following operations at time step INLINEFORM0 : DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 are the inputs of LSTM, all INLINEFORM3 and INLINEFORM4 are a set of parameter matrices to be trained, and INLINEFORM5 is a set of bias parameter matrices to be trained. INLINEFORM6 and INLINEFORM7 operation respectively represent matrix element-wise multiplication and sigmoid function. In the LSTM unit, there are two hidden layers ( INLINEFORM8 , INLINEFORM9 ), where INLINEFORM10 is the internal memory cell for dealing with vanishing gradient, while INLINEFORM11 is the main output of the LSTM unit for complex operations in subsequent layers.
We denotes INLINEFORM0 as the word embedding of time step INLINEFORM1 , a prediction INLINEFORM2 of next word embedding INLINEFORM3 can be computed by hidden layer INLINEFORM4 : DISPLAYFORM0 
Therefore, link score of next word embedding INLINEFORM0 can be computed as: DISPLAYFORM0 
Due to the structure of LSTM, vector INLINEFORM0 contains important information of entire segmentation decisions. In this way, the link score gets the result of the sequence-level word segmentation, not just word-level.
Intuitively, we can compute the score of a segmented sequence by summing up word scores and link scores. However, we find that a sequence with more candidate words tends to have higher sequence scores. Therefore, to alleviate the impact of the number of candidate words on sequence scores, we calculate final scores as follows: DISPLAYFORM0 
where INLINEFORM0 denotes the INLINEFORM1 -th segmented sequence with INLINEFORM2 candidate words, and INLINEFORM3 represents the INLINEFORM4 -th candidate words in the segmented sequence.
When training the model, we seek to minimize the sequence score of the corrected segmented sentence and the predicted segmented sentence. DISPLAYFORM0 
where INLINEFORM0 is the loss function.
We collect 204 EHRs with cardiovascular diseases from the Shuguang Hospital Affiliated to Shanghai University of Traditional Chinese Medicine and each contains 27 types of records. We choose 4 different types with a total of 3868 records from them, which are first course reports, medical records, chief ward round records and discharge records. The detailed information of EHRs are listed in Table TABREF32 .
We split our datasets as follows. First, we randomly select 3200 records from 3868 records as unlabeled set. Then, we manually annotate remaining 668 records as labeled set, which contains 1170 sentences. Finally, we divide labeled set into train set and test set with the ratio of 7:3 randomly. Statistics of datasets are listed in Table TABREF33 .
To determine suitable parameters, we divide training set into two sets, the first 80% sentences as training set and the rest 20% sentences as validation set.
Character embedding dimensions and K-means clusters are two main parameters in the CRF-based word segmenter.
In this paper, we choose character-based CRF without any features as baseline. First, we use Word2Vec to train character embeddings with dimensions of [`50', `100', `150', `200', `300', `400'] respectively, thus we obtain 6 different dimensional character embeddings. Second, these six types of character embeddings are used as the input to K-means algorithm with the number of clusters [`50', `100', `200', `300', `400', `500', `600'] respectively to capture the corresponding features of character embeddings. Then, we add K-means clustering features to baseline for training. As can be seen from Fig. FIGREF36 , when the character embedding dimension INLINEFORM0 = 150 and the number of clusters INLINEFORM1 = 400, CRF-based word segmenter performs best, so these two parameters are used in subsequent experiments.
Hyper-parameters of neural network have a great impact on the performance. The hyper-parameters we choose are listed in Table TABREF38 .
The dimension of character embeddings is set as same as the parameter used in CRF-based word segmenter and the number of hidden units is also set to be the same as it. Maximum word length is ralated to the number of parameters in GCNN unit. Since there are many long medical terminologies in EHRs, we set the maximum word length as 6. In addition, dropout is an effective way to prevent neural networks from overfitting BIBREF35 . To avoid overfitting, we drop the input layer of the scoring model with the rate of 20%.
Our work experimentally compares two mainstream CWS tools (LTP and Jieba) on training and testing sets. These two tools are widely used and recognized due to their high INLINEFORM0 -score of word segmentation in general fields. However, in specific fields, there are many terminologies and uncommon words, which lead to the unsatisfactory performance of segmentation results. To solve the problem of word segmentation in specific fields, these two tools provide a custom dictionary for users. In the experiments, we also conduct a comparative experiment on whether external domain dictionary has an effect on the experimental results. We manually construct the dictionary when labeling EHRs.
From the results in Table TABREF41 , we find that Jieba benefits a lot from the external dictionary. However, the Recall of LTP decreases when joining the domain dictionary. Generally speaking, since these two tools are trained by general domain corpus, the results are not ideal enough to cater to the needs of subsequent NLP of EHRs when applied to specific fields.
To investigate the effectiveness of K-means features in CRF-based segmenter, we also compare K-means with 3 different clustering features, including MeanShift BIBREF36 , SpectralClustering BIBREF37 and DBSCAN BIBREF38 on training and testing sets. From the results in Table TABREF43 , by adding additional clustering features in CRF-based segmenter, there is a significant improvement of INLINEFORM0 -score, which indicates that clustering features can effectively capture the semantic coherence between characters. Among these clustering features, K-means performs best, so we utlize K-means results as additional features for CRF-based segmenter.
In this experiment, since uncertainty sampling is the most popular strategy in real applications for its simpleness and effectiveness BIBREF27 , we compare our proposed strategy with uncertainty sampling in active learning. We conduct our experiments as follows. First, we employ CRF-based segmenter to annotate the unlabeled set. Then, sampling strategy in active learning selects a part of samples for annotators to relabel. Finally, the relabeled samples are added to train set for segmenter to re-train. Our proposed scoring strategy selects samples according to the sequence scores of the segmented sentences, while uncertainty sampling suggests relabeling samples that are closest to the segmenter’s decision boundary.
Generally, two main parameters in active learning are the numbers of iterations and samples selected per iteration. To fairly investigate the influence of two parameters, we compare our proposed strategy with uncertainty sampling on the same parameter. We find that though the number of iterations is large enough, it has a limited impact on the performance of segmenter. Therefore, we choose 30 as the number of iterations, which is a good trade-off between speed and performance. As for the number of samples selected per iteration, there are 6078 sentences in unlabeled set, considering the high cost of relabeling, we set four sizes of samples selected per iteration, which are 2%, 5%, 8% and 11%.
The experimental results of two sampling strategies with 30 iterations on four different proportions of relabeled data are shown in Fig. FIGREF45 , where x-axis represents the number of iterations and y-axis denotes the INLINEFORM0 -score of the segmenter. Scoring strategy shows consistent improvements over uncertainty sampling in the early iterations, indicating that scoring strategy is more capable of selecting representative samples.
Furthermore, we also investigate the relations between the best INLINEFORM0 -score and corresponding number of iteration on two sampling strategies, which is depicted in Fig. FIGREF46 .
It is observed that in our proposed scoring model, with the proportion of relabeled data increasing, the iteration number of reaching the optimal word segmentation result is decreasing, but the INLINEFORM0 -score of CRF-based word segmenter is also gradually decreasing. When the proportion is 2%, the segmenter reaches the highest INLINEFORM1 -score: 90.62%. Obviously, our proposed strategy outperforms uncertainty sampling by a large margin. Our proposed method needs only 2% relabeled samples to obtain INLINEFORM2 -score of 90.62%, while uncertainty sampling requires 8% samples to reach its best INLINEFORM3 -score of 88.98%, which indicates that with our proposed method, we only need to manually relabel a small number of samples to achieve a desired segmentation result.
To relieve the efforts of EHRs annotation, we propose an effective word segmentation method based on active learning, in which the sampling strategy is a scoring model combining information entropy with neural network. Compared with the mainstream uncertainty sampling, our strategy selects samples from statistical perspective and deep learning level. In addition, to capture coherence between characters, we add K-means clustering features to CRF-based word segmenter. Based on EHRs collected from the Shuguang Hospital Affiliated to Shanghai University of Traditional Chinese Medicine, we evaluate our method on CWS task. Compared with uncertainty sampling, our method requires 6% less relabeled samples to achieve better performance, which proves that our method can save the cost of manual annotation to a certain extent.
In future, we plan to employ other widely-used deep neural networks, such as convolutional neural network and attention mechanism, in the research of EHRs segmentation. Then, we believe that our method can be applied to other tasks as well, so we will fully investigate the application of our method in other tasks, such as NER and relation extraction.
The authors would like to appreciate any suggestions or comments from the anonymous reviewers. This work was supported by the National Natural Science Foundation of China (No. 61772201) and the National Key R&D Program of China for “Precision medical research" (No. 2018YFC0910550).

Figure and table captions:
Fig. 1. The diagram of active learning for the Chinese word segmentation.
Fig. 2. The architecture of the information entropy based scoring model, where ‘/’ represents candidate word separator, xi represents the one-hot encoding of the i-th character, cj represents the j-th character embedding learned by Word2Vec, wm represents the distributed representation of the mth candidate word and pn represents the prediction of the (n+1)-th candidate word.
Fig. 3. The architecture of word score, where ‘/’ represents candidate word separator, ci represents the i-th character embedding, wj represents the j-th candidate word embedding and ScoreWord(wk) represents the word score of the k-th candidate word.
Fig. 4. The architecture of GCNN.
TABLE I DETAILED INFORMATION OF EHRS
TABLE III HYPER-PARAMETER SETTING.
TABLE IV EXPERIMENTAL RESULTS WITH DIFFERENT WORD SEGMENTATION TOOLS.
Fig. 5. The relation between F1-score and K-means class with different character embedding dimensions.
TABLE II STATISTICS OF DATASETS
TABLE V COMPARISON WITH DIFFERENT CLUSTERING FEATURES.
Fig. 7. The relations between the best F1-score and corresponding iteration on two sampling strategies with different relabeled sample sizes.
Fig. 6. The results of two sampling strategies with different relabeled sample sizes.

Questions about this paper:
How does the active learning model work?

Answers:
Active learning methods has a learning engine (mainly used for training of classification problems) and the selection engine (which chooses samples that need to be relabeled by annotators from unlabeled data). Then, relabeled samples are added to training set for classifier to re-train, thus continuously improving the accuracy of the classifier. In this paper, CRF-based segmenter and a scoring model are employed as learning engine and selection engine, respectively.