Venue: EMNLP 2017

# Natural Language Processing with Small Feed-Forward Networks

Jan A. Botha Emily Pitler Ji Ma Anton Bakalov Alex Salcianu David Weiss Ryan McDonald Slav Petrov Google Inc. Mountain View, CA {jabot,epitler,maji,abakalov,salcinau,djweiss,ryanmcd,slav}@google.com

###### Abstract

We show that small and shallow feed-forward neural networks can achieve near state-of-the-art results on a range of unstructured and structured language processing tasks while being considerably cheaper in memory and computational requirements than deep recurrent models. Motivated by resource-constrained environments like mobile phones, we showcase simple techniques for obtaining such small neural network models, and investigate different tradeoffs when deciding how to allocate a small memory budget.

## 1 Introduction

Deep and recurrent neural networks with large network capacity have become increasingly accurate for challenging language processing tasks. For example, machine translation models have been able to attain impressive accuracies, with models that use hundreds of millions Bahdanau et al. (2014); Wu et al. (2016) or billions Shazeer et al. (2017) of parameters. These models, however, may not be feasible in all computational settings. In particular, models running on mobile devices are often constrained in terms of memory and computation.

Long Short-Term Memory (LSTM) models Hochreiter and Schmidhuber (1997) have achieved good results with small memory footprints by using character-based input representations: e.g., the part-of-speech tagging models of Gillick et al. (2016) have only roughly 900,000 parameters. Latency, however, can still be an issue with LSTMs, due to the large number of matrix multiplications they require (eight per LSTM cell): Kim and Rush (2016) report speeds of only 8.8 words/second when running a two-layer LSTM translation system on an Android phone.

Feed-forward neural networks have the potential to be much faster. In this paper, we show that small feed-forward networks can achieve results at or near the state-of-the-art on a variety of natural language processing tasks, with an order of magnitude speedup over an LSTM-based approach.

We begin by introducing the network model structure and the character-based representations we use throughout all tasks (SS2). The four tasks that we address are: language identification (Lang-ID), part-of-speech (POS) tagging, word segmentation, and preordering for translation. In order to use feed-forward networks for structured prediction tasks, we use transition systems Titov and Henderson (2007, 2010) with feature embeddings as proposed by Chen and Manning (2014), and introduce two novel transition systems for the last two tasks. We focus on _budgeted_ models and ablate four techniques (one on each task) for improving accuracy for a given memory budget:

1. Quantization: Using more dimensions and less precision (Lang-ID: SS3.1).
2. Word clusters: Reducing the network size to allow for word clusters and derived features (POS tagging: SS3.2).
3. Selected features: Adding explicit feature conjunctions (segmentation: SS3.3).
4. Pipelines: Introducing another task in a pipeline and allocating parameters to the auxiliary task instead (preordering: SS3.4).

We achieve results at or near state-of-the-art with small (\(<3\) MB) models on all four tasks.

## 2 Small Feed-Forward Network Models

The network architectures are designed to limit the memory and runtime of the model. Figure 1 illustrates the model architecture

1. Discrete features are organized into groups (e.g., \({\bf E}_{bigrams}\)), with one embedding matrix \({\bf E}_{g}\in\mathbb{R}^{V_{g}\times D_{g}}\) per group.
2. Embeddings of features extracted for each group are reshaped into a single vector and concatenated to define the output of the embedding layer as \({\bf h}_{0}=[{\bf X}_{g}{\bf E}_{g}~{}|~{}\forall g]\).
3. A single hidden layer, \({\bf h}_{1}\), with \(M\) rectified linear units Nair and Hinton (2010) is fully connected to \({\bf h}_{0}\).
4. A softmax function models the probability of an output class y: \(P(y)\propto\exp(\beta_{y}^{T}{\bf h}_{1}+b_{y})\), where \(\beta_{y}\in\mathbb{R}^{M}\) and \(b_{y}\) are the weight vector and bias, respectively.

Memory needs are dominated by the embedding matrix sizes (\(\sum_{g}V_{g}D_{g}\), where \(V_{g}\) and \(D_{g}\) are the vocabulary sizes and dimensions respectively for each feature group \(g\)), while runtime is strongly influenced by the hidden layer dimensions.

Hashed Character _n_-gramsPrevious applications of this network structure used (pretrained) word embeddings to represent words Chen and Manning (2014); Weiss et al. (2015). However, for word embeddings to be effective, they usually need to cover large vocabularies (100,000+) and dimensions (50+). Inspired by the success of character-based representations Ling et al. (2015), we use features defined over character _n_-grams instead of relying on word embeddings, and learn their embeddings from scratch.

We use a distinct feature group \(g\) for each _n_-gram length \(N\), and control the size \(V_{g}\) directly by applying _random feature mixing_Ganchev and Dredze (2008). That is, we define the feature value \(v\) for an _n_-gram string \(x\) as \(v=\mathcal{H}(x)\mod V_{g}\), where \(\mathcal{H}\) is a well-behaved hash function. Typical values for \(V_{g}\) are in the 100-5000 range, which is far smaller than the exponential number of unique raw _n_-grams. A consequence of these small feature vocabularies is that we can also use small feature embeddings, typically \(D_{g}\)=16.

QuantizationA commonly used strategy for compressing neural networks is quantization, using less precision to store parameters Han et al. (2015). We compress the embedding weights (the vast majority of the parameters for these shallow models) by storing scale factors for each embedding (details in the supplementary material). In SS3.1, we contrast devoting model size to higher precision and lower dimensionality versus lower precision and more network dimensions.

TrainingOur objective function combines the cross-entropy loss for model predictions relative to the ground truth with L2 regularization of the biases and hidden layer weights. For optimization, we use mini-batched averaged stochastic gradient descent with momentum Bottou (2010); Hinton (2012) and exponentially decaying learning rates. The mini-batch size is fixed to 32 and we perform a grid search for the other hyperparameters, tuning against the task-specific evaluation metric on held-out data, with early stopping. Full feature templates and optimal hyperparameter settings are given in the supplementary material.

## 3 Experiments

We experiment with small feed-forward networks for four diverse NLP tasks: language identification, part-of-speech tagging, word segmentation, and preordering for statistical machine translation.

Evaluation MetricsIn addition to standard task-specific quality metrics, our evaluations also consider model size and computational cost. We skirt implementation details by calculating size as the number of kilobytes (1KB=1024 bytes) needed to represent all model parameters and resources. We approximate the computational cost as the number of floating-point operations (FLOPs) performed for one forward pass through the network given an embedding vector \({\bf h}_{0}\). This cost is dominated by the matrix multiplications to compute (unscaled) activation unit values, hence our metric excludes the non-linearities and softmax normal

Figure 1: An example network structure for a model using bigrams of the previous, current and next word, and trigrams of the current word. Does not illustrate hashing.

 

ization, but still accounts for the final layer logits. To ground this metric, we also provide indicative absolute speeds for each task, as measured on a modern workstation CPU (3.50GHz Intel Xeon E5-1650 v3).

### Language Identification

Recent shared tasks on code-switching Molina et al. (2016) and dialects Malmasi et al. (2016) have generated renewed interest in language identification. We restrict our focus to single language identification across diverse languages, and compare to the work of Baldwin and Lui (2010) on predicting the language of Wikipedia text in 66 languages. For this task, we obtain the input \({\bf h}_{0}\) by separately _averaging_ the embeddings for each \(n\)-gram length (\(N=[1,4]\)), as summation did not produce good results.

Table 1 shows that we outperform the low-memory nearest-prototype model of Baldwin and Lui (2010). Their nearest neighbor model is the most accurate but its memory scales linearly with the size of the training data.

Moreover, we can apply quantization to the embedding matrix without hurting prediction accuracy: it is better to use less precision for each dimension, but to use more dimensions. Our subsequent models all use quantization. There is no noticeable variation in processing speed when performing dequantization on-the-fly at inference time. Our 16-dim Lang-ID model runs at 4450 documents/second (5.6 MB of text per second) on the preprocessed Wikipedia dataset.

**Relationship to Compact Language Detector** These techniques back the open-source Compact Language Detector v3 (CLD3)1 that runs in Google Chrome browsers.2 Our experimental Lang-ID model uses the same overall architecture as CLD3, but uses a simpler feature set, less involved preprocessing, and covers fewer languages.

Footnote 1: github.com/google/cld3

Footnote 2: As of the date of this writing in 2017.

### POS Tagging

We apply our model as an unstructured classifier to predict a POS tag for each token independently, and compare its performance to that of the byte-to-span (BTS) model Gillick et al. (2016). BTS is a 4-layer LSTM network that maps a sequence of bytes to a sequence of labeled spans, such as tokens and their POS tags. Both approaches limit model size by using small input vocabularies: byte values in the case of BTS, and hashed character \(n\)-grams and (optionally) cluster ids in our case.

#### Bloom Mapped Word Clusters

It is well known that word clusters can be powerful features in linear models for a variety of tasks Koo et al. (2008); Turian et al. (2010). Here, we show that they can also be useful in neural network models. However, naively introducing word cluster features drastically increases the amount of memory required, as a word-to-cluster mapping file with hundreds of thousands of entries can be several megabytes on its own.3 By representing word clusters with a Bloom map Talbot and Talbot (2008), a key-value based generalization of Bloom filters, we can reduce the space required by a factor of \(\sim\)15 and use 300KB to (approximately) represent the clusters for 250,000 word types.

Footnote 3: For example, the commonly used English clusters from the BLLIP corpus is over 7 MB – people.csail.mit.edu/maestro/papers/bllip-clusters.gz

In order to compare against the monolingual setting of Gillick et al. (2016), we train models for the same set of 13 languages from the Universal Dependency treebanks v1.1 Nivre et al. (2016) corpus, using the standard predefined splits.

As shown in Table 2, our best models are 0.3% more accuate on average across all languages than the BTS monolingual models, while using 6x fewer parameters and 36x fewer FLOPs. The cluster features play an important role, providing a 15% relative reduction in error over our vanilla model, but also increase the overall size. Halv

\begin{table}
\begin{tabular}{l|c c}
* \\ Baldwin and Lui (2010): NP & 87.0 &
- \\ Small FF, 6 dim & 87.3 & 334 KB \\ Small FF, 16 dim & 88.0 & 800 KB \\ Small FF, 16 dim, _quantized_ & 88.0 & 302 KB \\ \end{tabular}
\end{table}
Table 1: Language Identification. Quantization allows trading numerical precision for larger embeddings. The two models from Baldwin and Lui (2010) are the nearest neighbor (NN) and nearest prototype (NP) approaches.

\begin{table}
\begin{tabular}{l|c c c c}
* & 6.63m \\ Small FF & 94.76 & 241k & 0.6 & 0.27m \\ +Clusters & 95.56 & 261k & 1.0 & 0.31m \\ \(\frac{1}{2}\) Dim. & 95.39 & 143k & 0.7 & 0.18m \\ \end{tabular}
\end{table}
Table 2: POS tagging. Embedded word clusters improves accuracy and allows the use of smaller embedding dimensions.
 

ing all feature embedding dimensions (except for the cluster features) still gives a 12% reduction in error and trims the overall size back to 1.1x the vanilla model, staying well under 1MB in total. This halved model configuration has a throughput of 46k tokens/second, on average.

Two potential advantages of BTS are that it does not require tokenized input and has a more accurate multilingual version, achieving 95.85% accuracy. From a _memory_ perspective, one multilingual BTS model will take less space than separate FF models. However, from a _runtime_ perspective, a pipeline of our models doing language identification, word segmentation, and then POS tagging would still be faster than a single instance of the deep LSTM BTS model, by about 12x in our FLOPs estimate.4

Footnote 4: Our calculation of BTS FLOPs is very conservative and favorable to BTS, as detailed in the supplementary material.

### Segmentation

Word segmentation is critical for processing Asian languages where words are not explicitly separated by spaces. Recently, neural networks have significantly improved segmentation accuracy Zhang et al. (2016); Cai and Zhao (2016); Liu et al. (2016); Yang et al. (2017); Kong et al. (2015). We use a structured model based on the transition system in Table 3, and similar to the one proposed by Zhang and Clark (2007). We conduct the segmentation experiments on the Chinese Treebank 6.0 with the recommended data splits. No external resources or pretrained embeddings are used. Hashing was detrimental to quality in our preliminary experiments, hence we do not use it for this task. To learn an embedding for unknown characters, we cast characters occurring only once in the training set to a special symbol.

Selected FeaturesBecause we are not using hashing here, we need to be careful about the size of the input vocabulary. The neural network with its non-linearity is in theory able to learn bigrams by conjoining unigrams, but it has been shown that explicitly using character bigram features leads to better accuracy Zhang et al. (2016); Pei et al. (2014). Zhang et al. (2016) suggests that embedding manually specified feature connections further improves accuracy Zhang et al. (2016)-combo' in Table 4). However, such embeddings could easily lead to a model size explosion and thus are not considered in this work.

The results in Table 4 show that spending our memory budget on small bigram embeddings is more effective than on larger character embeddings, in terms of both accuracy and model size. Our model featuring bigrams runs at 110KB of text per second, or 39k tokens/second.

### Preordering

Preordering source-side words into the target-side word order is a useful preprocessing task for statistical machine translation Xia and McCord (2004); Collins et al. (2005); Nakagawa (2015); de Gispert et al. (2015). We propose a novel transition system for this task (Table 5), so that we can repeatedly apply a small network to produce these permutations. Inspired by a non-projective parsing transition system Nivre (2009), the system uses a swap action to permute spans. The system is sound for permutations: any derivation will end with all of the input words in a permuted order, and complete: all permutations are reachable (use shift and swap operations to perform a bubble sort, then append\(n-1\) times to form a single span). For training and evaluation, we use the English-Japanese manual word alignments from Nakagawa (2015).

\begin{table}
\begin{tabular}{l|c c}
**Transition** & **Precondition** \\ \hline Append & \(([\sigma]i|j],[\beta])\rightarrow([\sigma|[ij]],[\beta])\) \\ Shift & \(([\sigma],[i|\beta])\rightarrow([\sigma|i],[\beta])\) \\ Swap & \(([\sigma|i|j],[\beta])\rightarrow[\sigma|j],[i|\beta])\); \(i<j\) \\ \end{tabular}
\end{table}
Table 4: Segmentation results. Explicit bigrams are useful.

\begin{table}
\begin{tabular}{l l}
**Transition** & **Precondition** \\ \hline  & \(([\sigma|\)\(i|\beta)\)\(\rightarrow\)\(([\sigma|i],[\beta])\) \\ Merge & \(([\sigma],[i|\beta))\)\(\rightarrow\)\(([\sigma],[\beta])\) \\ \end{tabular}
\end{table}
Table 3: Segmentation Transition system. Initially all characters are on the buffer \(\beta\) and the stack \(\sigma\) is empty: \(([],[c_{1}c_{2}...c_{n}])\). In the final state the buffer is empty and the stack contains the first character for each word.

\begin{table}
\begin{tabular}{l l r}
**Model** & **Accuracy** & **Size** \\ \hline   Zhang et al. (2016) & 95.01 & – \\   Zhang et al. (2016)-combo & 95.95 & – \\ Small FF, 64 dim & 94.24 & 846KB \\ Small FF, 256 dim & 94.16 & 3.2MB \\ Small FF, 64 dim, _bigrams_ & 95.18 & 2.0MB \\ \end{tabular}
\end{table}
Table 5: Preordering Transition system. Initially all words are part of singleton spans on the buffer: \(([],[[w_{1}][w_{2}]...[w_{n}]])\). In the final state the buffer is empty and the stack contains a single span.

 

PipelinesFor preordering, we experiment with either spending all of our memory budget on reordering, or spending some of the memory budget on features over predicted POS tags, which also requires an additional neural network to predict these tags. Full feature templates are in the supplementary material. As the POS tagger network uses features based on a three word window around the token, another possibility is to add all of the features that would have affected the POS tag of a token to the reorderer directly.

Table 6 shows results with or without using the predicted POS tags in the preorderer, as well as including the features used by the tagger in the reorderer directly and only training the downstream task. The preorderer that includes a separate network for POS tagging and then extracts features over the predicted tags is more accurate and smaller than the model that includes all the features that contribute to a POS tag in the reorderer directly. This pipeline processes 7k tokens/second when taking pretokenized text as input, with the POS tagger accounting for 23% of the computation time.

## 4 Conclusions

This paper shows that small feed-forward networks are sufficient to achieve useful accuracies on a variety of tasks. In resource-constrained environments, speed and memory are important metrics to optimize as well as accuracies. While large and deep recurrent models are likely to be the most accurate whenever they can be afforded, feed-foward networks can provide better value in terms of runtime and memory, and should be considered a strong baseline.

## Acknowledgments

We thank Kuzman Ganchev, Fernando Pereira, and the anonymous reviewers for their useful comments.

## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_. Baldwin and Lui (2010) Timothy Baldwin and Marco Lui. 2010. Language identification: The long and the short of the matter. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 229-237. Association for Computational Linguistics. Bottou (2010) Leon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In _Proceedings of COMPSTAT'2010_, pages 177-186. Springer. Cai and Zhao (2016) Deng Cai and Hai Zhao. 2016. Neural word segmentation learning for Chinese. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 409-420, Berlin, Germany. Association for Computational Linguistics. Chen and Manning (2014) Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 740-750. Collins et al. (2005) M. Collins, P. Koehn, and I. Kucerova. 2005. Clause restructuring for statistical machine translation. In _Proceedings of ACL_, pages 531-540. Ganchev and Dredze (2008) Kuzman Ganchev and Mark Dredze. 2008. Small statistical models by random feature mixing. In _Proceedings of the ACL-08- HLT Workshop on Mobile Language Processing_, pages 18-19. Association for Computational Linguistics. Gillick et al. (2016) Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarang Subramanya. 2016. Multilingual language processing from bytes. In _Proceedings of NAACL-HLT_, pages 1296-1306, San Diego, USA. Association for Computational Linguistics. de Gispert et al. (2015) Adria de Gispert, Gonzalo Iglesias, and Bill Byrne. 2015. Fast and accurate preordering for SMT using neural networks. In _Proceedings of NAACL_, pages 1012-1017. Han et al. (2015) Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. _arXiv preprint arXiv:1510.00149_. Hinton (2012) Geoffrey E. Hinton. 2012. A practical guide to training restricted Boltzmann machines. In _Neural Networks: Tricks of the Trade (2nd ed.)_, Lecture Notes in Computer Science, pages 599-619. Springer. Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.

\begin{table}
\begin{tabular}{l|c c}
* \\ Small FF & 75.2 & 0.5MB \\ Small FF + POS tags & 81.3 & 1.3MB \\ Small FF + Tagger input fits. & 76.6 & 3.7MB \\ \end{tabular}
\end{table}
Table 6: Preordering results for English — Japanese. _FRS_ (in \([0,100]\)) is the fuzzy reordering score (Talbot et al., 2011).
 

Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
* Kong et al. (2015) Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2015. Segmental recurrent neural networks. _CoRR_, abs/1511.06018.
* Koo et al. (2008) Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In _Proceedings of ACL-08: HLT_, pages 595-603.
* Ling et al. (2015) Wang Ling, Tiago Luis, Luis Marujo, Ramon Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1520-1530. Association for Computational Linguistics.
* Liu et al. (2016) Yijia Liu, Wanxiang Che, Jiang Guo, Bing Qin, and Ting Liu. 2016. Exploring segment representations for neural segmentation models. In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016_, pages 2880-2886.
* Malmasi et al. (2016) Shervin Malmasi, Marcos Zampieri, Nikola Ljubesic, Preslav Nakov, Ahmed Ali, and Jorg Tiedemann. 2016. Discriminating between similar languages and Arabic dialect identification: A report on the third DSL shared task. In _Proceedings of the Third Workshop on NLP For Similar Languages, Varieties and Dialects (VarDial3)_, pages 1-14, Osaka, Japan. The COLING 2016 Organizing Committee.
* Molina et al. (2016) Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the second shared task on language identification in code-switched data. In _Proceedings of the Second Workshop on Computational Approaches to Code Switching_, pages 40-49, Austin, Texas. Association for Computational Linguistics.
* Nair and Hinton (2010) Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In _Proceedings of the 27th International Conference on Machine Learning_, pages 807-814.
* Nakagawa (2015) Tetsuji Nakagawa. 2015. Efficient top-down BTG parsing for machine translation preordering. In _Proceedings of ACL_, pages 208-218.
* Nivre (2009) Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 351-359.
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter Yoav Goldberg, Jan Hajiac, Christopher D. Manning, Ryan McDonald, Slav Petrov, Samapo Pysalo, Natalia Sliverra, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A Multilingual Treebank Collection. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_, Paris, France. European Language Resources Association (ELRA).
* Pei et al. (2014) Wenzhe Pei, Tao Ge, and Baobao Chang. 2014. Max-margin tensor neural network for Chinese word segmentation. In _Proceedings of the 52nd Annual Meettics of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 293-303, Baltimore, Maryland. Association for Computational Linguistics.
* Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. _arXiv preprint arXiv:1701.06538_.
* Talbot et al. (2011) David Talbot, Hideto Kazawa, Hiroshi Ichikawa, Jason Katz-Brown, Masakazu Seno, and Franz J. Och. 2011. A lightweight evaluation framework for machine translation reordering. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_, WMT '11, pages 12-21, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Talbot and Talbot (2008) David Talbot and John Talbot. 2008. Bloom maps. In _Proceedings of the Meeting on Analytic Algorithmics and Combinatorics_, pages 203-212. Society for Industrial and Applied Mathematics.
* Titov and Henderson (2007) Ivan Titov and James Henderson. 2007. Fast and robust multilingual dependency parsing with a generative latent variable model. In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 947-951.
* Titov and Henderson (2010) Ivan Titov and James Henderson. 2010. A latent variable model for generative dependency parsing. In Harry Burt, Paola Merlo, and Joakim Nivre, editors, _Trends in Parsing Technology: Dependency Parsing, Domain Adaptation, and Deep Parsing_, pages 35-55. Springer Netherlands, Dordrecht.
* Turian et al. (2010) Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word Representations: A Simple and General Method for Semi-supervised Learning. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 384-394, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Weiss et al. (2015) David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics_, pages 323-333 

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macheuff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _CoRR_, abs/1609.08144.
* Xia and McCord (2004) Fei Xia and Michael McCord. 2004. Improving a statistical MT system with automatically learned rewrite patterns. In _Proceedings of COLING_, page 508.
* Yang et al. (2017) Jie Yang, Yue Zhang, and Fei Dong. 2017. Neural word segmentation with rich pretraining. _CoRR_, abs/1704.08960.
* Zhang et al. (2016) Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Transition-based neural word segmentation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 421-431, Berlin, Germany. Association for Computational Linguistics.
* Zhang and Clark (2007) Yue Zhang and Stephen Clark. 2007. Chinese segmentation with a word-based perceptron algorithm. In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_, pages 840-847, Prague, Czech Republic. Association for Computational Linguistics.