Abstract: Neural word segmentation research has benefited from large-scale raw texts by leveraging them for pretraining character and word embeddings. On the other hand, statistical segmentation research has exploited richer sources of external information, such as punctuation, automatic segmentation and POS. We investigate the effectiveness of a range of external training sources for neural word segmentation by building a modular segmentation model, pretraining the most important submodule using rich external sources. Results show that such pretraining significantly improves the model, leading to accuracies competitive to the best methods on six benchmarks.

Introduction: There has been a recent shift of research attention in the word segmentation literature from statistical methods to deep learning BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . Neural network models have been exploited due to their strength in non-sparse representation learning and non-linear power in feature combination, which have led to advances in many NLP tasks. So far, neural word segmentors have given comparable accuracies to the best statictical models. With respect to non-sparse representation, character embeddings have been exploited as a foundation of neural word segmentors. They serve to reduce sparsity of character ngrams, allowing, for example, “猫(cat) 躺(lie) 在(in) 墙角(corner)” to be connected with “狗(dog) 蹲(sit) 在(in) 墙角(corner)” BIBREF0 , which is infeasible by using sparse one-hot character features. In addition to character embeddings, distributed representations of character bigrams BIBREF6 , BIBREF1 and words BIBREF2 , BIBREF5 have also been shown to improve segmentation accuracies. With respect to non-linear modeling power, various network structures have been exploited to represent contexts for segmentation disambiguation, including multi-layer perceptrons on five-character windows BIBREF0 , BIBREF6 , BIBREF1 , BIBREF7 , as well as LSTMs on characters BIBREF3 , BIBREF8 and words BIBREF2 , BIBREF4 , BIBREF5 . For structured learning and inference, CRF has been used for character sequence labelling models BIBREF1 , BIBREF3 and structural beam search has been used for word-based segmentors BIBREF4 , BIBREF5 . Previous research has shown that segmentation accuracies can be improved by pretraining character and word embeddings over large Chinese texts, which is consistent with findings on other NLP tasks, such as parsing BIBREF9 . Pretraining can be regarded as one way of leveraging external resources to improve accuracies, which is practically highly useful and has become a standard practice in neural NLP. On the other hand, statistical segmentation research has exploited raw texts for semi-supervised learning, by collecting clues from raw texts more thoroughly such as mutual information and punctuation BIBREF10 , BIBREF11 , and making use of self-predictions BIBREF12 , BIBREF13 . It has also utilised heterogenous annotations such as POS BIBREF14 , BIBREF15 and segmentation under different standards BIBREF16 . To our knowledge, such rich external information has not been systematically investigated for neural segmentation. We fill this gap by investigating rich external pretraining for neural segmentation. Following BIBREF4 and BIBREF5 , we adopt a globally optimised beam-search framework for neural structured prediction BIBREF9 , BIBREF17 , BIBREF18 , which allows word information to be modelled explicitly. Different from previous work, we make our model conceptually simple and modular, so that the most important sub module, namely a five-character window context, can be pretrained using external data. We adopt a multi-task learning strategy BIBREF19 , casting each external source of information as a auxiliary classification task, sharing a five-character window network. After pretraining, the character window network is used to initialize the corresponding module in our segmentor. Results on 6 different benchmarks show that our method outperforms the best statistical and neural segmentation models consistently, giving the best reported results on 5 datasets in different domains and genres. Our implementation is based on LibN3L BIBREF20 . Code and models can be downloaded from http://gitHub.com/jiesutd/RichWordSegmentor

Question: What external sources are used?
Answer: Raw data from Gigaword, Automatically segmented text from Gigaword, Heterogenous training data from People's Daily, POS data from People's Daily