Venue: EMNLP 2017

# Multi-Grained Chinese Word Segmentation

Chen Gong, Zhenghua Li; Min Zhang, Xinzhou Jiang

Soochow University, Suzhou, China cgong@stu.suda.edu.cn, {zhli13,minzhang}@suda.edu.cn, xzjiang.hw@gmail.com

Correspondence author

###### Abstract

Traditionally, word segmentation (WS) adopts the single-granularity formalism, where a sentence corresponds to a single word sequence. However, Sproat et al. (1996) show that the inter-native-speaker consistency ratio over Chinese word boundaries is only 76%, indicating single-grained WS (SWS) imposes unnecessary challenges on both manual annotation and statistical modeling. Moreover, WS results of different granularities can be complementary and beneficial for high-level applications.

This work proposes and addresses multi-grained WS (MWS). First, we build a large-scale pseudo MWS dataset for model training and tuning by leveraging the annotation heterogeneity of three SWS datasets. Then we manually annotate 1,500 test sentences with true MWS annotations. Finally, we propose three benchmark approaches by casting MWS as constituent parsing and sequence labeling. Experiments and analysis lead to many interesting findings.

## 1 Introduction

As the first processing step of Chinese language processing, word segmentation (WS) has been extensively studied and made great progress during the past decades, thanks to the annotation of large-scale benchmark datasets, among which the most widely-used are Microsoft Research Corpus (MSR) Huang et al. (2006), Peking University People Daily Corpus (PPD) Yu et al. (2003), and Penn Chinese Treebank (CTB) Xue et al. (2005). Table 1 gives an example sentence segmented in different guidelines. Meanwhile, WS approaches gradually evolve from maximum matching based on lexicon dictionaries Liu and Liang (1986), to path searching from segmentation graphs based on language modeling scores and other statistics Zhang and Liu (2002), to character-based sequence labeling Xue (2003), to shift-reduce incremental parsing Zhang and Clark (2007). Recently, neural network models have also achieved success by effectively learning representation of characters and contexts Zheng et al. (2013); Pei et al. (2014); Ma and Hinrichs (2015); Chen et al. (2015); Zhang et al. (2016); Cai and Zhao (2016); Liu et al. (2016).

To date, all the labeled datasets adopt the single-granularity formalization, and previous research mainly focuses on single-grained WS (SWS), where one sentence is segmented into a single word sequence. Although different WS guidelines share the same high-level criterion of word boundaries - a character string combined closely and used steadily forms a word, people greatly diverge due to individual differences on knowledge and living environments, etc. An anonymous reviewer kindly points

out that Vladimir Skalicka of the Prague School claimed that unlike the "isolating" languages such as French and English, Chinese belongs to the "polysynthetic" type, in which compound words are normally produced from indigenous morphemes (Jernudd and Shapiro, 1989). The vague distinction between morphemes and compounds also contribute to the cognition divergence on the concept of words. Sproat et al. (1996) show that the consensus ratio over word boundaries is only 76% among Chinese native speakers without trained on a common guideline.To fill this gap, WS guidelines need to further group words into many types and provide illustration examples for each type. Nevertheless, it is very challenging even for well-trained annotators to fully grasp the guidelines and to be consistent on uncovered cases. For example, Xiu (2013) (in Tables 1-3) shows that about 3% characters are inconsistently segmented in the PPD training data used in SIGHAN Bakeoff 2005 Emerson (2005). We have also observed many inconsistency cases in all MSR/PPD/CTB during this work. In a word, SWS imposes great challenge on data annotation, and as a side effect, enforces statistical models to learn subtleness of annotation guidelines rather than the true WS ambiguities.

From another perspective, WS results of different granularities may be complementary in supporting applications such as information retrieval (IR) Liu et al. (2008) and machine translation (MT) Su et al. (2017). On the one hand, coarse-grained words enable statistical models to perform more exact matching and analyzing. On the other hand, fine-grained words are helpful in both reducing data sparseness and supporting deeper understanding of language.1

Footnote 1: Words in CTB are generally more fine-grained than those in PPD and MSR, probably due to the requirement of annotating syntactic structures.

To solve the above two issues for SWS, this paper proposes and addresses multi-grained WS (MWS). Given an input sentence, the goal is to produce a hierarchy structure of all words of different granularities, as illustrated in Figure 1. To tackle the lack of labeled data, we build a large-scale pseudo MWS dataset for model training and tuning by automatically converting annotations of three heterogeneous SWS datasets (i.e. MSR/PPD/CTB) based on the recently proposed coupled sequence labeling approach of Li et al. (2015). In order to fully investigate the problem, we manually annotate 1,500 test sentences with true MWS annotations. Finally, we propose three benchmark approaches by casting MWS as constituent parsing and sequence labeling problems. Experiments and data analysis lead to many interesting findings.

We will release the newly annotated data and the codes of the benchmark approaches at [http://hlt.suda.edu.cn/~zhli](http://hlt.suda.edu.cn/~zhli). However, due to the license issue, we may not directly release all the pseudo MWS datasets. Instead, we will launch a web service for obtaining MWS annotations given a sentence with one of MSR/PPD/CTB annotations.

## 2 Pseudo MWS Data Conversion

This section introduces the process of gathering pseudo MWS data by making use of the annotation heterogeneity of the three existing datasets, i.e., MSR/PPD/CTB.

### Annotation Heterogeneity

MSR is a manually labeled corpus with word boundaries and named entity tags, and is annotated by Microsoft Research Asia for supporting Chinese text processing Huang et al. (2006). The key characteristic of MSR is treating named entities as single words. For example, \(``\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangledown{}\!\! \bigtriangleup{}\!\!\bigtriangleup{}^{\#}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\! \bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\! \bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigcirc{}\!\!\bigvee{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{} \!\!\bigotimesup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\! \!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\!\bigtriangleup{}\!\! 

nese information processing, and is annotated by Institute of Computational Linguistics at Peking University (Yu et al., 2003). Based on the Penn Chinese Treebank Project, CTB is built to create a Mandarin Chinese corpus with syntactic bracketing Xue et al. (2005). We find that CTB is more fine-grained in word boundaries than MSR and PPD, since syntactic annotation tends to require deeper understanding of a sentence. For example, Table 5 reports the averaged number of characters per word in each corpus, and confirms our observations.

For better understanding of annotation heterogeneity, we summarize high-frequency differences among the three datasets observed and gathered during this study in Appendix A. However, it is difficult to obtain a complete list of annotation correspondences among the three datasets, since there are too many low-frequency and irregular cases. Moreover, we also observe a lot of inconsistency annotations of the same word or words with similar structures in all three datasets, as shown in Appendix B.

### Coupled WS for Conversion: MSR/PPD as Example

This section introduces how to automatically produce high-quality PPD-side WS labels for a sentence with MSR-side gold-standard WS labels, by leveraging the two non-overlapping SWS data of MSR and PPD with the coupled sequence labeling approach of Li et al. (2015) and Li et al. (2016). Figure 2 shows the workflow.

Given a sentence \({\bf x}=[c_{1},...,c_{i},...,c_{n}]\), the coupled model aims to produce a sequence of bundled tags \({\bf t}=[t^{a}_{1}t^{b}_{1},...,t^{a}_{i}{}^{b}_{i},...,t^{a}_{n}{}^{b}_{n}]\), where \(t^{a}_{i}\) and \(t^{b}_{i}\) are two labels corresponding to two heterogeneous guidelines respectively. Table 2 gives an example of coupled WS on MSR/PPD. We employ the standard four-tag label set to mark word boundaries of one granularity, among which B, I, E respectively represent that the concerned character situates at the _beginning, inside, end_ position of a word, and S represents a single-character word. The bottom row shows the gold-standard bundled tag sequence.

One key advantage of the coupled model is to directly learn from two _non-overlapping_ heterogeneous training datasets, where each dataset only contains single-side gold-standard labels. To deal with this partial (or incomplete) labeling issue, they project each single-side label to a set of bundled labels by considering all labels at the missing side, as shown in the second row in Table 2. Such _ambiguous labelings_ are used for model supervision.

Under a traditional CRF, the coupled model defines the score of a bundled tag sequence as

\[Score({\bf x},{\bf t};\theta)=\theta\cdot{\bf f}({\bf x},{\bf t })\\ =\sum_{i=1}^{n+1}\theta\cdot\left[\begin{array}{l}{\bf f}_{joint}({ \bf x},i,t^{a}_{i-1}t^{b}_{i-1},t^{a}_{i}t^{b}_{i})\\ {\bf f}_{sep\_a}({\bf x},i,t^{a}_{i-1},t^{a}_{i})\\ {\bf f}_{sep\_b}({\bf x},i,t^{b}_{i-1},t^{b}_{i})\end{array}\right]\]

where \({\bf f}_{joint}(.)\) are the _joint features_ whereas \({\bf f}_{sep\_a/sep\_b}(.)\) are the separate features. Li et al. (2015) demonstrate that the joint features capture the implicit mappings between heterogeneous annotations, while the back-off separate features work as a remedy for the sparseness of the joint features.

\begin{table}
\begin{tabular}{p{48.4pt}||p{27.9pt}|p{27.

 

In their case study of POS tagging, Li et al. (2015) show the coupled model improves tagging accuracy by \(95.0-94.1=0.9\%\) on CTB5-test over the baseline non-coupled model trained on a single training data.

More importantly, they show that the coupled model can be naturally used for the task of annotation conversion, where second-side labels are automatically annotated, given one-side gold-standard labels. The given one-side tags are used to obtain ambiguous labelings, as shown in Table 2, and the coupled model finds the best bundled tag sequence in the _constrained search space_, instead of in the whole bundled tag space, hence greatly reducing the difficulty. Li et al. (2015) report that the coupled model can improve conversion accuracy on POS tagging by \(93.9-90.6=3.3\%\) over the non-coupled model.2

Footnote 2: The accuracy seems quite low. The reason is only the 20% most ambiguous words of each sentence are manually labeled and evaluated in their experiments.

### Producing Pseudo MWS Data

Figure 3 shows the workflow of producing pseudo MWS data with three separately trained coupled models. Please note that one coupled model is able to perform conversion between one pair of annotation standards, and thus three coupled models are required for three kinds of annotation standards. Another alternative is that we could directly train one coupled model on MSR/PPD/CTB by extending the approach of Li et al. (2015) from two guidelines into three, which would lead to a much larger bundled tag space. For simplicity, we directly employ their released codes in this work, and leave that for future exploration.

After conversion, we obtain 9 pseudo MWS datasets (i.e., MSR/PPD/CTB-train/dev/test) and represent each sentence in a hierarchy structure as shown in Figure 1. Please kindly note that the guideline-specific information are thrown away, since we do not care which word belongs to which guideline.

In the resulting pseudo MWS data, we find about 0.08% of words overlap with other words, meaning a string "ABC" is segmented into "A/BC" and "AB/C" in two different annotations. We have manually checked these words, and find almost all those cases are caused by conversion errors. This confirms that our treatment of MWS as a hierarchy structure is reasonable.

## 3 Manual Annotation

In order to fully investigate the MWS problem, we have manually created a true MWS data of 1,500 sentences for final evaluation. From each test dataset in Table 5, we randomly sample 500 sentences with converted pseudo MWS annotations for manual correction. First, two coauthors of this work spent about two hours each day on manual correction of the pseudo MWS annotations for two weeks. During this period, we have summarized a list of high-frequency corresponding patterns among the three guidelines (see Appendix A), and have also written a simple program to automatically detect inconsistent annotations of given words in different training datasets, so that annotators can use the outputs of the program to decide ambiguous cases, which we find is extremely helpful for annotation.

Then, we employ 10 postgraduate students as our annotators who are at different familiarity in WS annotation. Before formal annotation, the annotators are trained for two hours on the basic concepts of MWS, high-frequency correspondences among the three guidelines, and the use of the outputs of the program. We also encourage the annotators to access the three training datasets directly for studying concrete cases under real contexts. Moreover, annotators are asked to recheck their annotations before final submission to improve quality.

To measure the inter-annotator consistency, 150 sentences (10%) are sampled for double annotation, and are grouped into four batches for four pairs of annotators. After annotation, two annotators on the same batch compare

Figure 3: Producing pseudo MWS data.

 

their results and produce a consensus submission through discussion.

The annotation process lasts for four days, and each annotator spends about 8 hours in total on completing 160 sentences on average. Table 3 compares data statistics on the \(1,500\) sentences before and after manual annotation. The second column reports the number of words, and the last three columns report the distribution of words according to their granularity levels. To illustrate how to gain the distribution, we take Figure 1 as an example, which contains 1 single-grained words, 9 two-grained words, and 7 three-grained words.4

Footnote 4: Formally, we call a word \(s\) three-grained if there are two other words \(s1\) and \(s2\) satisfying any one conditions: 1) \(s2\in s1\in s\) (like “\(\!\!\pm\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! 

bidirectional LSTM layers to encode the input token sequence, as shown in Figure 4; 2) two separate multilayer perceptrons (MLPs) to make structural/labeling decisions based on \(4/3\) simple LSTM span features. A span feature represents a sentence span \((i,j)\) by concatenating the element-wise differences of BiLSTM outputs:

\[{\bf r}_{(i,j)}=[{\bf f}^{\rm d}_{j}-{\bf f}^{\rm l}_{i-1};{\bf b}^{\rm l}_{i}-{ \bf b}^{\rm l}_{j+1};{\bf f}^{\rm g}_{j}-{\bf f}^{\rm G}_{i-1};{\bf b}^{\rm B}_{ i}-{\bf b}^{\rm Z}_{j+1}]\]

To adapt the original parsing model to our MWS task, we concatenate bichar embeddings \({\bf e}_{c_{i-1}c_{i}}\)with single char embedding \({\bf e}_{c_{i}}\) as inputs to the first-layer BiLSTM, inspired by Pei et al. (2014), who show that bichar embeddings are very helpful for SWS.

### MWS as Sequence Labeling

It is also straightforward to model MWS as a sequence labeling task by replacing SWS labels with MWS labels for each character. Table 4 encodes the MWS structure in Figure 1 with a sequence of MWS labels. The idea is to concatenate multiple SWS tags simultaneously for one character to denote the positions of the character under words of different granularities. Please note that each MWS label contains at most three SWS labels since we only consider three SWS datasets in this work. Here, we organize the SWS labels in the order of fine-to-coarse granularities.

For simplicity and fair comparison, we adopt a similar network architecture as the parsing model described in Section 4.1. To decide the MWS label of a character \(c_{i}\) in the input sentence, we feed the outputs of the two-layer BiLSTM outputs \([{\bf f}^{\rm l}_{i};{\bf b}^{\rm l}_{i},{\bf f}^{\rm z}_{i};{\bf b}^{\rm Z}_{i }]\) into a single-hidden-layer MLP.

### MWS as SWS Aggregation

Instead of directly training a MWS model on the three pseudo MWS training datasets, we can also train three separate SWS models on the three SWS training datasets. Given an input sentence, we apply the three SWS models and then merge their outputs as MWS results.

The network architecture is the same with the sequence labeling model in Section 4.2, except the MLP outputs correspond to SWS labels instead of MWS labels.

## 5 Experiments

**Data:** for MSR, we adopt the training/test datasets of the SIGHAN Bakeoff 2005 (Emerson, 2005), and cut off 10% random training sentences as the dev data following Zhang et al. (2016); for PPD and CTB, we follow Li et al. (2015) and directly adopt their datasets and data split. Table 5 shows the data statistics.7

Footnote 7: A DBC-to-SBC (double/single-byte characters) case preprocessing is performed on all datasets to avoid encoding inconsistency.

**Evaluation Metrics:** the goal of MWS is to precisely produce all words of different granularities given the input sentence. Therefore, to reach a balance of both precision (\(P=\frac{\#Word_{gold\cap tvxy}}{\#Word_{sys}}\)) and recall (\(R=\frac{\#Word_{gold\cap tvy}}{\#Word_{gold}}\)), we use the F1 score (\(=\frac{2PR}{P+R}\)) as in SWS.

**Hyper-parameter:** we implement all our approaches based on the codes released by Cross and Huang (2016), by making extensions such as adding bichar embeddings and

Figure 4: Two-layer BiLSTM architecture.

\begin{table}
\begin{tabular}{l||r|r|r|r|r|r} Chars & Train & Dev & Test & \#Char per Word \\ \hline MSR & 78,232 & 8,692 & 3,985 & 1.71 \\ PPD & 46,815 & 2,000 & 5,000 & 1.67 \\ CTB & 16,091 & 803 & 1910 & 1.63 \\ \end{tabular}
\end{table}
Table 5: Data statistics (in sentence number). The last column reports the averaged character number of each word.

 

supporting sequence labeling.8 For simplicity, char and bichar embeddings are randomly initialized following Cross and Huang (2016). The dimensions of char and bichar embeddings are both 50 and other hyper-parameters are the same with Cross and Huang (2016). In our preliminary experiments, we observe that under their neural network framework, the MWS performance is quite stable when rerunning under random initialization or reasonably altering other hyper-parameters. Due to time limitation, we leave the use of pre-trained embeddings and more hyper-parameter tuning for future exploration.

Footnote 8: [https://github.com/jhcross/span-parser](https://github.com/jhcross/span-parser). We are very grateful for their helping us solve some code issues at the early stage of this work.

**Training/test settings:** when training the parsing and sequence labeling based MWS models (not SWS aggregation) on MSR/PPD/CTB-train, we adopt the simple corpus weighting strategy used in Li et al. (2015) to balance the contributions of each training dataset. Before each iteration, we randomly sample \(10{,}000\) sentences from each training dataset, and merge and shuffle them for one-iteration training. We use merged MSR/PPD/CTB-dev as the MWS dev data for model selection.9

Footnote 9: For MSR-dev, only the first \(3,000\) sentences are used during training due to efficiency concern.

For the SWS aggregation model, three SWS models are separately trained on the three training/dev datasets. For evaluation, three SWS outputs produced independently are merged as one MWS result given a sentence.

In all experiments, training stops when F-score on the dev data does not improve in 20 consecutive iterations, and we choose the model that performs best on the dev data for final evaluation.

**Main results:** Table 6 reports the performance of different approaches on both the pseudo MWS dev data and the manually annotated MWS test data. The "#Word" column reports the total number of words returned by the corresponding model; the following three columns show the percentages of words of different granularities; the last "Overlapping" column gives the percent of words that overlap with other words, which only happens in the "SWS aggregation" approach, since no constraint can be applied to the three separate SWS models during testing. From the results, we can draw the following findings.

First, the results suggest that using pseudo training and dev datasets to build a MWS model is feasible, based on two evidences: 1) our simple benchmark model can reach a high F-score of \(96.07\%\) on the manually annotated test data, which is \(1.77\%\) higher than directly aggregating outputs of three SWS models; 2) the P/R/F scores on the pseudo dev data and on the manually labeled test data are quite consistent in general, indicating that it is reliable to use the pseudo dev data for model selection and tuning.

Second, the parsing approach and the sequence labeling approach (with or without bichar embeddings) achieve very similar performance (within \(0.15\%\) vibration), More importantly, the parsing approach produces more words and more multi-grained words than the sequence labeling approach, indicating that it is potentially more proper to model MWS as a parsing problem in order to better capture and represent multi-granularity structures. Another possible disadvantage of the sequence labeling approach is that the trained model cannot produce more granularity levels (e.g., four-grained) beyond

\begin{table}
\begin{tabular}{c|c c c|c c c|c c c|c c}  & \multicolumn{4}{c|}{Dev (Pseudo)} & \multicolumn{4}{c}{Test (Manual)} \\ \hline  & P & R & F & P & R & F & \#Words & Single & Two & Three & Overlapping \\ \hline Parsing & 96.55 & 96.40 & 96.48 & 97.00 & 95.16 & **96.07** & 44,408 & 74.9\% & 23.5\% & 1.6\% & – \\ w/o Bichar Emb & 95.58 & 95.04 & 95.51 & 96.37 & 94.11 & 95.22 & 44,434 & 74.2\% & 24.1\% & 1.7\% & – \\ \hline Sequence Labeling & **96.86** & 96.26 & **96.59** & **97.01** & 94.96 & 95.97 & 44,323 & 75.8\% & 22.7\% & 1.5\% & – \\ w/o Bichar Emb & 95.88 & 94.94 & 95.41 & 96.56 & 94.18 & 95.35 & 44,162 & 75.7\% & 22.8\% & 1.5\% & – \\ \hline SWS Aggregation & 90.43 & **97.44** & 93.80 & 92.11 & **96.59** & 94.30 & 47,478 & 64.6\% & 31.4\% & 4.0\% & 1.0\% \\ \hline \end{tabular}
\end{table}
Table 6: Performance of different MWS approaches.

 

those in the training data. Nevertheless, compared against the manual annotations in Table 3, both the parsing and sequence labeling approaches retrieve much less multi-grained words, which is caused by the under-representation issue of the pseudo training data, as discussed in Section 3.

Third, the SWS aggregation approach achieves the best recall at the price of very low precision on both dev/test data. We believe the reason is that training three SWS models separately on one of the three training datasets has two disadvantages: 1) connections among different guidelines are totally ignored, leading to many overlapping words (\(1.0\%\)); 2) smaller training data also degrades the performance of each SWS model.

Finally, using bichar embeddings turns out very helpful for MWS, and leads to \(0.97\sim 1.18\%\) F-score improvement on dev data and \(0.62\sim 0.85\%\) on test data, which is consistent with the SWS results in Pei et al. (2014).

## 6 Related Work

As far as we know, this is the first work that formally proposes and addresses the problem of Chinese MWS under the data-driven machine learning framework. It is true that the industrial community, driven by practical demand, has long been interested in retrieving words of different granularities from the engineering perspective, based on lexicon dictionaries and heuristic rules Zhu and Li (2008); Hou et al. (2010). We also discover two publicly released toolkits, i.e., IKAnalyzer10 and PoolWord11, which consider all substrings in a sentence and return those above a threshold probability as candidate words. In contrast, this paper defines MWS as a strict hierarchy structure, and propose a supervised learning framework for the problem.

Footnote 10: [https://github.com/medcl/elasticsearch-analysis-ik](https://github.com/medcl/elasticsearch-analysis-ik)

Footnote 11: [http://pullword.com/](http://pullword.com/)

To alleviate the high OOV-ratio issue of character-based sequence labeling, Zhang et al. (2006) and Zhao and Kit (2007) propose subword-based sequence labeling for word segmentation by extracting high-frequency subword and treating them as the basic labeling units. Li (2011) and Li and Zhou (2012) propose to jointly parse the internal structures of words and syntactic structure of a sentence. Their definition of internal structures mainly considers prefix or suffix information. They manually annotate the internal structures of words that have high-frequency prefixes or suffixes and left other words with flat structures in CTB. Zhang et al. (2013) further annotate internal structures of all words in CTB and then perform character-level parsing with WS labels. Cheng et al. (2015) propose to cope with the multiple WS standard problem based on internal word structures. After close study of the above works, we find that the MWS annotations automatically built in this work actually capture a lot of subwords and word internal structures in previous works. Most importantly, the main focus of previous works is to improve SWS or parsing performance, whereas this work aims to build a hierarchy structure of multi-grained words. We leave the integration of MWS and parsing for future work.

It has been a long debate whether there exists an optimal WS granularity for MT, which is further complicated by the inevitable mistakes contained in 1-best WS outputs. Dyer et al. (2008) propose an MT model based on source-language word lattices, obtained by merging the outputs of different segmenters. Xiao et al. (2010) propose joint SWS and MT based on word lattices. Recently, Su et al. (2017) propose a word lattice-based neural MT model. They train many segmenters on MSR/PPD/CTB, and merge the outputs to produce word lattices for source-language sentences, which is similar to our SWS aggregation approach. All above works show the usefulness of word lattices instead of a single SWS output. In help IR, Liu et al. (2008) propose a ranking based WS approach for producing words of different granularities. We believe this work can further help both IR and MT by supplying with more accurate MWS results.

## 7 Conclusion

This work proposes and addresses the problem of MWS, so that all words of different granularities can be captured in a hierarchy structure given a sentence. We can draw the 

following interesting findings.

(1) Our annotation conversion approach can gather high-quality pseudo MWS training/dev datasets, and hence it is feasible to use them for model training and tuning.

(2) Manual MWS data annotation tells us that about \(28.4\%\) words are multi-grained, and among them \(94.4\%\) are two-grained words.

(3) The parsing and sequence labeling approaches achieve very similar performance, and outperform the SWS aggregation approach by a large margin.

We believe there are many exploration directions for this new task, among which we are particularly interested in three in the near future: 1) improving our benchmark approaches by considering task-specific features and neural network architectures, 2) verifying the usefulness of MWS to high-level applications such as MT, 3) integrating MWS with syntactic parsing in some way by exploiting existing treebanks.

## Acknowledgments

The authors would like to thank the anonymous reviewers for the helpful comments. We are greatly grateful to all students in LA group for their hard work as annotators. Especially, we thank Jiawei Sun for her extensive participant and support through this work. We also thank Wenliang Chen and Guodong Zhou for the helpful discussions. This work was supported by National Natural Science Foundation of China (Grant No. 61525205, 61373095, 61502325).

## References

* Cai and Zhao (2016) Deng Cai and Hai Zhao. 2016. Neural word segmentation learning for chinese. In _Proceedings of ACL_, pages 409-420.
* Chen et al. (2015) Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuanjing Huang. 2015. Long short-term memory neural networks for chinese word segmentation. In _Proceedings of EMNLP_, pages 1197-1206.
* Cheng et al. (2015) Fei Cheng, Kevin Duh, and Yuji Matsumoto. 2015. Synthetic word parsing improves chinese word segmentation. In _ACL_, pages 262-267.
* Cross and Huang (2016) James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In _Proceedings of EMNLP_, pages 1-11.
* Dyer et al. (2008) Christopher Dyer, Smaranda Muresan, and Philip Resnik. 2008. Generalizing word lattice translation. In _Proceedings of ACL_, pages 1012-1020.
* Emerson (2005) Thomas Emerson. 2005. The second international chinese word segmentation bakeoff. In _Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing_, pages 123-133.
* Hou et al. (2010) Lei Hou, Min Chu, Jingming Tang, Jian Sun, Xiaoling Liao, Rengang Peng, Yang Yang, and Bingjing Xu. 2010. Method and device for providing multi-granularity word segmentation result. Chinese Patent (CN102479191A).
* Huang et al. (2006) Chang-Ning Huang, Yumei Li, and Xiaodan Zhu. 2006. Tokenization Guidelines of Chinese Text (V5.0, in Chinese). Microsoft Research Asia.
* Jernudd and Shapiro (1989) Bjorn H. Jernudd and Michael J. Shapiro, editors. 1989. _The Politics of Language Purism (page 214)_. Mouton de Gruyter.
* Li et al. (2015) Zhenghua Li, Jiayuan Chao, Min Zhang, and Wenliang Chen. 2015. Coupled sequence labeling on heterogeneous annotations: POS tagging as a case study. In _Proceedings of ACL_, pages 1783-1792.
* Li et al. (2016) Zhenghua Li, Jiayuan Chao, Min Zhang, and Jiwen Yang. 2016. Fast coupled sequence labeling on heterogeneous annotations via context-aware pruning. In _Proceedings of EMNLP_, pages 753-762.
* Li (2011) Zhongguo Li. 2011. Parsing the internal structure of words: A new paradigm for chinese word segmentation. In _Proceedings of ACL_, pages 1405-1414.
* Li and Zhou (2012) Zhongguo Li and Guodong Zhou. 2012. Unified dependency parsing of chinese morphological and syntactic structures. In _Proceedings of EMNLP_, pages 1445-1454.
* Liu et al. (2016) Yijia Liu, Wanxiang Che, Jiang Guo, Bing Qin, and Ting Liu. 2016. Exploring segment representations for neural segmentation models. In _Proceedings of AAAI_, pages 2880-2886.
* Liu et al. (2008) Yixuan Liu, Bin Wang, Fan Ding, and Sheng Xu. 2008. Information retrieval oriented word segmentation based on character association strength ranking. In _Proceedings of EMNLP_, pages 1061-1069.
* Liu and Liang (1986) Yuan Liu and Nanyuan Liang. 1986. Fundation of Chinese processing: statistics of modern Chinese word frequencies. _Journal of Chinese Information Processing (in Chinese)_, 0(1):17-25.
* Ma and Hinrichs (2015) Jianqiang Ma and Erhard Hinrichs. 2015. Accurate linear-time chinese word segmentation via embedding matching. In _Proceedings of ACL-IJCNLP_, pages 1733-1743 

Wenzhe Pei, Tao Ge, and Baobao Chang. 2014. Max-margin tensor neural network for chinese word segmentation. In _Proceedings of ACL_, pages 293-303.
* Sproat et al. (1996) Richard Sproat, William Gales, Chilin Shih, and Nancy Chang. 1996. A Stochastic Finite-State Word-Segmentation Algorithm for Chinese. _Computational Linguistics_, 22(3).
* Su et al. (2017) Jinsong Su, Zhixing Tan, Deyi Xiong, Rongrong Ji, Xiaodong Shi, and Yang Liu. 2017. Lattice-based recurrent neurla network encoders for neural machine translation. In _Proceedings of AAAI_.
* Xiao et al. (2010) Xinyan Xiao, Yang Liu, YoungSook Hwang, Qun Liu, and Shouxun Lin. 2010. Joint tokenization and translation. In _Proceedings of COLING_, pages 1200-1208.
* Xiu (2013) Chi Xiu. 2013. _The research and implementation of method for domain Chinese word segmentation_. Ph.D. thesis, Beijing University of Technology.
* Xue (2003) Nianwen Xue. 2003. Chinese word segmentation as character tagging. _Computational Linguistics and Chinese Language Processing_, 8(1):29-48.
* Xue et al. (2005) Nianwen Xue, Fei Xia, Fu-Dong Chiou, and Martha Palmer. 2005. The Penn Chinese Treebank: Phrase structure annotation of a large corpus. _Natural Language Engineering_, 11(2):207-238.
* Yu et al. (2003) Shiwen Yu, Huiming Duan, Xuefeng Zhu, Bin Swen, and Baobao Chang. 2003. Specification for corpus processing at Peking University: Word segmentation, POS tagging and phonetic notation (In Chinese). _Journal of Chinese Language and Computing_, 13(2):121-158.
* Zhang and Liu (2002) Hua-Ping Zhang and Qun Liu. 2002. Model of Chinese words rough segmentation based on n-shortest-paths method. _Journal of Chinese Information Processing (in Chinese)_, 16(5):1-7.
* Zhang et al. (2013) Meishan Zhang, Yue Zhang, Wanxiang Che, and Ting Liu. 2013. Chinese parsing exploiting characters. In _Proceedings of ACL_, pages 125-134.
* Zhang et al. (2016) Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Transition-based neural word segmentation. In _Proceedings of ACL_, pages 421-431.
* Zhang et al. (2006) Ruiqiang Zhang, Genichiro Kikui, and Eiichiro Sumita. 2006. Subword-based tagging for confidence-dependent chinese word segmentation. In _Proceedings of COLING/ACL_, pages 961-968.
* Zhang and Clark (2007) Yue Zhang and Stephen Clark. 2007. Chinese segmentation with a word-based perceptron algorithm. In _Proceedings of ACL_, pages 840-847.
* Zhao and Kit (2007) Hai Zhao and Chunyu Kit. 2007. Effective subsequence-based tagging for Chinese word segmentation. _Journal of Chinese Information Processing (in Chinese)_, 21(5):8-13.
* Zheng et al. (2013) Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. 2013. Deep learning for Chinese word segmentation and POS tagging. In _Proceedings of EMNLP_, pages 647-657.
* Zhu and Li (2008) Jian Zhu and Shan Li. 2008. Method and device for large- and small-grained segmentation of Chinese text. Chinese Patent (CN101246472A).

## Appendix A: Collected Annotation Inconsistencies

### In MSR

* [(1)] "--Il (one day)" is annotated as ""Il (one day)" or ''---(one)//Il (day)":
* [(2)] "If you mock\({}^{\prime}\) is annotated as ""Il (too much)" or ""Il (foo only/g smo)".
* [(3)] "Il (this one)" and ""Il (this item)" have the same structure of ""Il (this')" + #, however they are annotated as ""Il (this one)" and ""Il (this (if)/Il (item)" respectively.
* [(4)] "Elf (the same story)" and ""Il (foo !)" have the same structure of ""Il (to; however they are annotated as ""Il (same)//Il (tow)" and ""Il (tow to?)" respectively.
* [(5)] "Elf (nuclear weapon)" and ""Il (nuclear technology)" have the same structure of ""Il (nuclear)" + #, however they are annotated as ""Il (nuclear weapons)" and ""Il (nuclear)/Il (technology)" respectively.
* [(6)] "Elf (the next step)" and ""Il (the next game)" have the same structure of ""Il (the next)" + #, however they are annotated as ""Il (the next step)" and ""Il (rest)// (none)//Il (game)" respectively.
* [(7)] "Elf (deputy direction")" and ""Il (give-president)" have the same structure of ""Il (give)" + #, however they are annotated as ""Il (deputy)/Elf (director)" and ""Il (vice-president)" respectively.
* [(8)] "Elf (work)" and ""Il (create)" have the same structure of ""Il (ze (or/or)", however they are annotated as ""Il (worker)" and ""Il (create)/Il (cor)" respectively.
* [(7)] "Elf (cross country)" and ""Il (cross border)" have the same structure of ""Il + # (cross + #), however they are annotated as ""Il (cross contrary)" and ""Il (cross)/Il (border)"

**In PPD**

* [(1)] "Elf (ministerial level)" is annotated as ""Elf (miniterial level)" or ""Il (ministerial/Il (level)".
* [(one day)]" is annotated as ""Il (one day)" or "" (ones)//Il (day)".
* [(3)] "Elf (two mach)" is annotated as ""Il (too much)" or ""Il (toso)// (smo)".
* [(4)] "Elf (biggest)" is annotated as ""Il (biggest)" or ""Il (most)/% (bits)".
* [(5)] "Elf (and also)" is annotated as ""Il (and also)" or ""Il (also)/fit (have)".
* [(6)] "Elf (reward greatly)" is annotated as ""Il (reward)" or "" (reward)/% (greally)".
* [(7)] "Elf (by means of")" and ""Il (wowe o)" have the same structure of ++++ (co), however they are annotated as ""Il (type manos)" and ""Il (we)//T (top)" respectively.

 



\begin{table}
\begin{tabular}{|l|l|l|l|l|} \hline category & Chinese example & CTB & PPD & MSR \\ \hline
[HT

- "+H-\(\#\)B (the central government"), however they are annotated at "+H-\(\#\)B ("Credit Center Committee") and "+R R (Vistnamese Communist Party/0-\(\#\)B (central)" respectively.
* "F-\(\#\)B (the next step)" and "F-\(\#\)B (the next game)" have the same structure of "F-- (the next)" + H, however they are annotated as "F--\(\#\)B (the next step)" and "F (next)//~one (low)" respectively.
* "H-\(\#\)B (or beyond the year)" and "H-\(\#\)B (cross border)" have the same structure of "F" + H (cross + H), however they are annotated as "H (go beyond)/\(\#\)B (year)" and "H-\(\#\)B (cross border)" respectively.

**In CTB**

* "H-\(\#\)B (heavyweight)" is annotated as "H-\(\#\)B (heavyweight)" or "H-\(\#\)B (heavy)/R (weight)".
* "H-\(\#\)B (one day)" is annotated as "--H (one day)" or "--(one)/I (day)".
* "H-\(\#\)B (re-comployment)" is annotated as "H-\(\#\)B (re-employment)" or "H-\(\#\)B (once again)/\(\#\)B (employment)".
* "H-\(\#\)B (with cow)" is annotated as "H-\(\#\)B (wild cow)" or "H-\(\#\)B (two)".
* "H-\(\#\)B (biggest)" is annotated as "H-\(\#\)B (bigest)" or "H-\(\#\)B (most)/\(\%\)B (bit)".
* "H-\(\#\)B (and also)" is annotated as "E-\(\#\)B (and also)" or "E-\(\#\)B (also)/\(\#\)B (have)".
* "F-\(\#\)B (the next step)" is annotated as "F--\(\#\)B (the next step)" or "F-\(\#\)B (next)//~one (only/\(\#\)B (step)".
* "H-\(\#\)B ( vice-president)" is annotated as "H-\(\#\)B (vice-president)" or "H-\(\#\)B (vee1)/\(\#\)B (president)".
* "H-\(\#\)B (change into)" is annotated as "G-\(\#\)B (change into)" or "B (change in)".
* "H-\(\#\)B (beneficial to)" and "H-\(\#\)B (we too)" have the same structure of \(\#\) + F-\(\#\)B (to)"; however they are annotated as "H-\(\#\)B (beneficial to)" and "H-\(\#\)B (ow)/\(f\)B (too)" respectively.
* "H-\(\#\)B (for beyond the year)" and "H-\(\#\)B (cross border)" have the same structure of "W-\(\#\)B (cross + H), however they are annotated as "H-\(\#\)B (go beyond)/\(\#\)B (year)" and "H-\(\#\)B (cross border)" respectively.