Venue: NAACL 2019

# Subword Encoding in Lattice LSTM for Chinese Word Segmentation

Jie Yang\({}^{\clubsuit}\), Yue Zhang\({}^{\spadesuit}\), Shuailong Liang\({}^{\clubsuit}\)

\({}^{\spadesuit}\)Brigham and Women's Hospital. Boston, USA

\({}^{\clubsuit}\)Harvard Medical School, Harvard University. Boston, USA

\({}^{\spadesuit}\)School of Engineering, Westlake University. Hangzhou, China

\({}^{\clubsuit}\)Singapore University of Technology and Design. Singapore

jieynlp@gmail.com

yue.zhang@wias.org.cn

shuailong_liang@mymail.sutd.edu.sg

###### Abstract

We investigate subword information for Chinese word segmentation, by integrating sub word embeddings trained using byte-pair encoding into a Lattice LSTM (LaLSTM) network over a character sequence. Experiments on standard benchmark show that subword information brings significant gains over strong character-based segmentation models. To our knowledge, this is the first research on the effectiveness of subwords on neural word segmentation.

## 1 Introduction

Chinese word segmentation (CWS) is a traditional NLP task [25], the features for which have been a central research topic. Statistical methods consider characters [30], subwords [46], and words [47] as input features. Among these, both characters [8] and words [49, 50] have also shown useful in recent neural models. However, how to utilize the subword features in neural networks has not been investigated yet.

In this paper, we fill this gap by proposing a subword-based neural word segmentor, by integrating two strands of works: the byte pair encoding (BPE) algorithm [14] and the lattice LSTM structure [46]. The BPE algorithm constructs a subword list from raw data and lattice LSTM introduces subwords into character LSTM representation. In particular, our baseline is a BiLSTM-CRF segmentor [8] and we replace LSTM with lattice LSTM using subwords to encode character composition information. Our code1 is based on NCRF++ [46].

Footnote 1: Our code is released at [https://github.com/jiesudt/SubwordEncoding-CWS](https://github.com/jiesudt/SubwordEncoding-CWS).

Compared with character-based neural segmentors, our model can utilize abundant character combination (subword) information, which is effective to disambiguate characters. For example, in Figure 1, the subword "\({}^{\clubsuit}\)" means that the character "\({}^{\clubsuit}\)" means "Academy(noun)" rather than "study(verb)". Compared with the word-based neural models [46, 50], ambiguous subwords in a context can provide additional information for disambiguation. For instance, the subword "\({}^{\clubsuit}\)" \({}^{\clubsuit}\)" (Academy of Sciences)" and "\({}^{\clubsuit}\)" (Academy)" can be useful in determining the correct segmentation, which is "\({}^{\clubsuit}\)" (Academy of Sciences/)".

To our knowledge, we are the first to use subwords in a neural network segmentor. We investigate the contributions of subword lexicons and their pretrained embeddings through controlled experiments. Results on four benchmarks show that the proposed model can give comparable results with state-of-the-art models.

## 2 Related Work

State-of-the-art statistical segmentors use either sequence labeling methods e.g. CRF [15] with character features [23, 22] or the transition-based models with word features [46, 48]. Neural segmentors [8],

Figure 1: Segmentation with ambiguous words.



2015a; Cai and Zhao, 2016) generally take the same framework except using neural networks as automatic feature extractor.

Lattice LSTM was proposed by Zhang and Yang (2018) for Chinese named entity recognition (NER). It integrates the character sequence features and all lexicon word embeddings that match a character subsequence in the input into a sequence labeling model. Zhu et al. (2016) proposed a DAG-structured LSTM structure which is similar to the lattice LSTM model but binarizing the paths in the merging process. Chen et al. (2017) also built a DAG-LSTM structure for word segmentation but without memory cells. Our model consistently gives better performance.

BPE is a data compression algorithm (Gage, 1994) which has been used in neural machine translation (NMT) by capturing the most frequent subwords instead of words (Sennrich et al., 2016). Here we use it for collecting subwords in Chinese, similar to the use in Chinese NMT.

## 3 Models

We take the state-of-the-art LSTM-CRF framework as our baseline. For an input sentence with \(m\) characters \(s=c_{1},c_{2},\ldots,c_{m}\), where \(c_{i}\) denotes the \(i\)th character, the segmentor is to assign each character \(c_{i}\) with a label \(l_{i}\). Figure 2 shows the segmentor framework on input character sequence "[Fellow of the Chinese Academy of Sciences]", where the black part represents the baseline LSTM-CRF model and the red part shows the lattice structure.

### Baseline Model

As shown in Figure 2, for each input character \(c_{i}\), the corresponding character unigram embeddings and character bigram embeddings are represented as \(\textbf{e}_{c_{i}}\) and \(\textbf{e}_{c_{i}c_{i+1}}\), respectively. The character representation is calculated as following:

\[\textbf{x}_{i}=\textbf{e}_{c_{i}}\oplus\textbf{e}_{c_{i}c_{i+1}}, \tag{1}\]

where \(\oplus\) represents _concatenate_ operation.

Unlike Zhang et al. (2016) which uses a window to strengthen the local features, or Zhou et al. (2017) which adds a non-linear layer before the LSTM layer, we feed \(\{\textbf{x}_{1},\textbf{x}_{2},\ldots,\textbf{x}_{m}\}\) into a bidirectional LSTM:

\[\begin{array}{ll}\overrightarrow{\textbf{h}}_{1},&\overrightarrow{\textbf{ h}}_{2},\ldots,\overrightarrow{\textbf{h}}_{m}=\overline{LSTM}(\textbf{ x}_{1},\textbf{x}_{2},\ldots,\textbf{x}_{m})\\ \overleftarrow{\textbf{h}}_{1},&\overleftarrow{\textbf{h}}_{2},\ldots, \overrightarrow{\textbf{h}}_{m}=\overline{LSTM}(\textbf{x}_{1},\textbf{x}_{2 },\ldots,\textbf{x}_{m}),\end{array} \tag{2}\]

where \(\overline{LSTM}\) and \(\widehat{LSTM}\) represent the forward and backward LSTM, respectively. The detailed equations are listed in _Appendix_. The hidden vector of character \(c_{i}\) is

\[\textbf{h}_{i}=\overrightarrow{\textbf{h}}_{i}\oplus\overleftarrow{\textbf{ h}}_{i} \tag{3}\]

### Lattice LSTM

The lattice LSTM adds "shortcut paths" (red part in Figure 2) to LSTM. The input of the lattice LSTM model is character sequence and all subsequences which are matched words in a lexicon \(\mathbb{D}\), collected from BPE. Following Zhang and Yang (2018), we use \(w_{b,e}\) to represent the subsequence that has a start character index \(b\) and a end character index \(e\), and the embeddings of the subsequence is represented as \(\textbf{e}_{wb,e}\).

During the forward lattice LSTM calculation, the "cell" in Figure 2 of a subsequence \(w_{b,e}\) takes the hidden vector of the start character \(\textbf{h}_{b}\) and the subsequence embeddings \(\textbf{e}_{wb,e}\) as input, an extra

Figure 2: Models. Only forward LSTM is illustrated here.

 

LSTM cell is applied to calculate the memory vector of the sequence \({\bf c}_{w_{b,e}}\):

\[{\bf c}_{w_{b,e}}=LSTMCell({\textbf{h}}_{b},{\bf e}_{w_{b,e}}), \tag{4}\]

where the \(LSTMCell\) is a simplified LSTM unit which calculate the memory only. The output memory vector \({\bf c}_{w_{b,e}}\) links to the end character \(c_{e}\) to calculate its hidden vector \(\overrightarrow{\textbf{h}}_{e}\). For character with multiple memory cell inputs2, we assign a gate for each subsequence input to control its contribution. The detailed equations are listed in _Appendix_. The final output \(\overrightarrow{\textbf{h}}_{i}\) includes both the character sequence history information and all the matched subsequence information.

Footnote 2: e.g. The first “\(\Box\)” in Figure 2 takes two subsequence memory vectors of both “+->%” and “+->%” as input.

### Decoding and Training

We use a standard CRF layer for inference (details in _Appendix_). Viterbi (1967) is used to find the highest scored label sequence over the input. During training, we choose sentence-level log-likelihood as the loss function.

\[Loss=\sum_{i=1}^{N}log(P(y_{i}|s_{i})), \tag{5}\]

where \(y_{i}\) is the gold labels of sentence \(s_{i}\).

## 4 Experiments

### Experimental Settings

**Data**. We evaluate our model on four standard Chinese word segmentation datasets: CTB6, PKU, MSR, and Weibo. PKU and MSR are taken from the SIGHAN 2005 bake-off Emerson (2005) and Weibo dataset is the NLPCC 2016 shared task Qiu et al. (2016), standard split are used. We take CTB6 as the main dataset and split the train/dev/test following Zhang et al. (2016). The statistics of the datasets are listed in _Appendix_.

**Hyperparameters**. We keep the hyperparameters the same among all datasets. Standard gradient descent (SGD) with a learning rate decay is used as the optimizer. The embedding sizes of character unigram/bigram and subword are all 50. Dropout Srivastava et al. (2014) is used on both the character input and the subword input to prevent overfitting. Details are listed in Table 1.

**Embeddings**. We take the same character unigram and bigram embeddings as Zhang et al. (2016), who pretrain embeddings using word2vec Mikolov et al. (2013) on Chinese Gigaword3. The vocabulary of subword is constructed with 200000 merge operations and the subword embeddings are also trained using word2vec Heinzerling and Strube (2018). Trie Fredkin (1960) is used to accelerate lattice building. All the embeddings are fine-tuned during training.

Footnote 3: [https://catalog.ldc.upenn.edu/LDC2011T13](https://catalog.ldc.upenn.edu/LDC2011T13).

### Development Experiments

We perform experiments on the CTB6 development dataset to investigate the contribution of character bigram information and the subword information. Figure 3 shows the iteration curve of F-scores against different numbers of training iterations with different character representations. "_Bigram" represents the model using both character unigram and bigram information (embedding concatenation). Character bigram information can improve the baseline significantly. When the "LaLSTM+Subword" structure is added, the model performance is further improved. This shows that subword information has a great ability to disambiguate the characters.

Figure 3: F1-value against training iterations.

 

Zhang and Yang (2018) observed that character bigram information has a negative effect in lattice LSTM on Chinese NER task, while we find a different result on Chinese word segmentation where character bigram information gives significant improvements in the lattice LSTM. This is likely because character bigrams are informative but ambiguous. They can provide more useful character disambiguation evidence in segmentation than in NER where lattice LSTM works well in disambiguating characters.

### Results

Table 2 shows the main results and the recent state-of-the-art neural CWS models. Zhang et al. (2016) integrated both discrete features and neural features in a transition-based framework. Xu and Sun (2016) proposed the dependency-based gated recursive neural network to utilize long distance dependencies. Yang et al. (2017) utilized pretrained character representations from multitasks. We examine their non-pretrained model performance for fair comparison. Ma et al. (2018) built a bidirectional LSTM model with carefully hyperparameter selection. These methods are orthogonal to and can be integrated into our lattice structure.

As shown in Table 2, the subword lattice LSTM gives significant improvements on all evaluated datasets. In the PKU dataset, our model is slightly behind Xu and Sun (2016) which preprocesses the dataset by replacing all the Chinese idioms, leading the comparison not entirely fair. Our model gives the best performance on MSR and Weibo datasets, which demonstrates that subword encoding can help the lattice LSTM model gives comparable performance to the state-of-the-art word segmentation models.

### Analysis

**Lexicon and Embeddings**. To distinguish the contribution of subword lexicon and their pre-trained embeddings, we conduct a set of experiments by using the same subword lexicon with randomly initialized embeddings4 on CTB6 data. As shown in Table 3, the contribution of the error reduction by the lexicon is \(4.5\%\). While \(6.9\%\) error reduction comes from both lexicon and pre-trained embeddings. We can estimate that the contribution of pretraining is \((6.9\%-4.5\%)=2.4\%\). This roughly shows that both lexicon and pretraining are useful to lattice LSTM, and the former contributes more than the latter.

Footnote 4: Within \([-\sqrt{\frac{3}{dim}},\sqrt{\frac{3}{dim}}]\), \(dim\) is the embedding size.

**OOV Analysis**. Table 3 also shows the recall of in-vocabulary (**R\({}_{IV}\)**) and out-of-vocabulary (**R\({}_{COV}\)**) words, respectively. As shown in the table, the **R\({}_{OOV}\)** can be largely improved with the lattice structure (\(2.43\%\) absolute improvement).

**Sentence Length**. We compare the baseline model with our proposed model on the sentence length distribution in Figure 4. The performance of the baseline has a valley in around 30-character length and decreases when the sentence length over 90. This phenomenon has also been observed in transition-based neural segmentor Yang et al. (2017). While "LaLSTM+Subword" gives a more stable performance along sentence length.

Figure 4: F1-value against the sentence length.

 



**Subword Coverage in lexicon.** Table 45 shows the subword coverage rate in four datasets. Subword level coverage is consistently higher than the entity level coverage in Zhang and Yang (2018). We can see that higher subword coverage (PKU/MSR, \(>90\%\)) gives better error reduction rate. Weibo dataset gets the minimum improvement due to the low subword coverage.

Footnote 5: #Word is the word number in the corresponding dataset, #Match is the matched words number between the dataset and subword lexicon, #Ratio represents the subword coverage rate. #ER is the error reduction compared with baseline model.

**Case Study**. Figure 5 shows an example of CTB6 test dataset. In this example, there are two matched subwords "\(\mathbf{\hat{x}}\) & #

Yijia Liu, Wanxiang Che, Jiang Guo, Bing Qin, and Ting Liu. 2016. Exploring segment representations for neural segmentation models. In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_.
* Ma et al. (2018) Ji Ma, Kuzman Ganchev, and David Weiss. 2018. State-of-the-art chinese word segmentation with bi-lstms. In _Proceedings of EMNLP_, pages 4902-4908, Brussels, Belgium. Association for Computational Linguistics.
* Ma and Hinrichs (2015) Jianqiang Ma and Erhard Hinrichs. 2015. Accurate linear-time chinese word segmentation via embedding matching. In _Proceedings of ACL-IJCNLP_, pages 1733-1743, Beijing, China.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Pei et al. (2014) Wenzhe Pei, Tao Ge, and Baobao Chang. 2014. Maxmargin tensor neural network for chinese word segmentation. In _Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics_, pages 293-303. Association for Computational Linguistics.
* Peng et al. (2004) Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. In _Proceedings of the International Conference on Computational Linguistics_, page 562.
* Qiu et al. (2016) Xipeng Qiu, Peng Qian, and Zhan Shi. 2016. Overview of the nlpcc-iccpol 2016 shared task: Chinese word segmentation for micro-blog texts. In _International Conference on Computer Processing of Oriental Languages_, pages 901-906. Springer.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of ACL_, pages 1715-1725, Berlin, Germany.
* Sproat et al. (1996) Richard Sproat, William Gale, Chilin Shih, and Nancy Chang. 1996. A stochastic finite-state word-segmentation algorithm for chinese. _Computational linguistics_, 22(3):377-404.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_, 15(1):1929-1958.
* Sun (2010) Weiwei Sun. 2010. Word-based and character-based word segmentation models: Comparison and combination. In _Proceedings of the International Conference on Computational Linguistics_, pages 1211-1219.
* Viterbi (1967) Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. _IEEE transactions on Information Theory_, 13(2):260-269.
* Xu and Sun (2016) Jingjing Xu and Xu Sun. 2016. Dependency-based gated recursive neural network for chinese word seggmentation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, page 567. Association for Computational Linguistics.
* Xue et al. (2003) Nianwen Xue et al. 2003. Chinese word segmentation as character tagging. _Computational Linguistics and Chinese Language Processing_, 8(1):29-48.
* Yang and Zhang (2018) Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Yang et al. (2017) Jie Yang, Yue Zhang, and Fei Dong. 2017. Neural word segmentation with rich pretraining. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_, pages 839-849, Vancouver, Canada.
* Zhang et al. (2016) Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Transition-based neural word segmentation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_.
* Zhang et al. (2006) Ruiqiang Zhang, Genichiro Kikui, and Eiichiro Sumita. 2006. Subword-based tagging by conditional random fields for chinese word segmentation. In _Proceedings of HLT-NAACL_, pages 193-196, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Zhang and Clark (2007) Yue Zhang and Stephen Clark. 2007. Chinese segmentation with a word-based perceptron algorithm. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics_, volume 45, page 840.
* Zhang and Yang (2018) Yue Zhang and Jie Yang. 2018. Chinese net using lattice lstm. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_, Melbourne, Australia.
* Zhao et al. (2006) Hai Zhao, Chang-Ning Huang, Mu Li, and Bao-Liang Lu. 2006. Effective tag set selection in chinese word segmentation via conditional random field modeling. In _Proceedings of PACLIC_, volume 20, pages 87-94. Citeseer.
* Zheng et al. (2013) Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. 2013. Deep learning for chinese word segmentation and pos tagging. In _Proceedings of EMNLP_, pages 647-657.
* Zhou et al. (2017) Hao Zhou, Zhenting Yu, Yue Zhang, Shujian Huang, XIN-YU DAI, and Jiajun Chen. 2017. Word-context character embeddings for chinese word segmentation. In _Proceedings of EMNLP_, pages 760-766, Copenhagen, Denmark.
* Zhu et al. (2016) Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2016. Dag-structured long short-term memory for semantic compositionality. In _Proceedings of NAACL-HLT_, pages 917-926