Introduction
Neural Machine Translation (NMT) is an end-to-end learning approach to machine translation which has recently shown promising results on multiple language pairs BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . Unlike conventional Statistical Machine Translation (SMT) systems BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 which consist of multiple separately tuned components, NMT aims at building upon a single and large neural network to directly map input text to associated output text. Typical NMT models consists of two recurrent neural networks (RNNs), an encoder to read and encode the input text into a distributed representation and a decoder to generate translated text conditioned on the input representation BIBREF13 , BIBREF14 .
Driven by the breakthrough achieved in computer vision BIBREF15 , BIBREF16 , research in NMT has recently turned towards studying Deep Neural Networks (DNNs). Wu et al. wu2016google and Zhou et al. zhou2016deep found that deep architectures in both the encoder and decoder are essential for capturing subtle irregularities in the source and target languages. However, training a deep neural network is not as simple as stacking layers. Optimization often becomes increasingly difficult with more layers. One reasonable explanation is the notorious problem of vanishing/exploding gradients which was first studied in the context of vanilla RNNs BIBREF17 . Most prevalent approaches to solve this problem rely on short-cut connections between adjacent layers such as residual or fast-forward connections BIBREF15 , BIBREF16 , BIBREF18 . Different from previous work, we choose to reduce the gradient path inside the recurrent units and propose a novel Linear Associative Unit (LAU) which creates a fusion of both linear and non-linear transformations of the input. Through this design, information can flow across several steps both in time and in space with little attenuation. The mechanism makes it easy to train deep stack RNNs which can efficiently capture the complex inherent structures of sentences for NMT. Based on LAUs, we also propose a NMT model , called DeepLAU, with deep architecture in both the encoder and decoder.
Although DeepLAU is fairly simple, it gives remarkable empirical results. On the NIST Chinese-English task, DeepLAU with proper settings yields the best reported result and also a 4.9 BLEU improvement over a strong NMT baseline with most known techniques (e.g, dropout) incorporated. On WMT English-German and English-French tasks, it also achieves performance superior or comparable to the state-of-the-art.
Neural machine translation
A typical neural machine translation system is a single and large neural network which directly models the conditional probability INLINEFORM0 of translating a source sentence INLINEFORM1 to a target sentence INLINEFORM2 .
Attention-based NMT, with RNNsearch as its most popular representative, generalizes the conventional notion of encoder-decoder in using an array of vectors to represent the source sentence and dynamically addressing the relevant segments of them during decoding. The process can be explicitly split into an encoding part, a decoding part and an attention mechanism. The model first encodes the source sentence INLINEFORM0 into a sequence of vectors INLINEFORM1 . In general, INLINEFORM2 is the annotation of INLINEFORM3 from a bi-directional RNN which contains information about the whole sentence with a strong focus on the parts of INLINEFORM4 . Then, the RNNsearch model decodes and generates the target translation INLINEFORM5 based on the context INLINEFORM6 <t INLINEFORM7 p(yi|y<i, INLINEFORM8 is dynamically obtained according to the contribution of the source annotation made to the word prediction. This is called automatic alignment BIBREF14 or attention mechanism BIBREF0 , but it is essentially reading with content-based addressing defined in BIBREF19 . With this addressing strategy the decoder can attend to the source representation that is most relevant to the stage of decoding.
Deep neural models have recently achieved a great success in a wide range of problems. In computer vision, models with more than 100 convolutional layers have outperformed shallow ones by a big margin on a series of image tasks BIBREF15 , BIBREF16 . Following similar ideas of building deep CNNs, some promising improvements have also been achieved on building deep NMT systems. Zhou et al. zhou2016deep proposed a new type of linear connections between adjacent layers to simplify the training of deeply stacked RNNs. Similarly, Wu et al. wu2016google introduced residual connections to their deep neural machine translation system and achieve great improvements. However the optimization of deep RNNs is still an open problem due to the massive recurrent computation which makes the gradient propagation path extremely tortuous.
Model Description
In this section, we discuss Linear Associative Unit (LAU) to ease the training of deep stack of RNNs. Based on this idea, we further propose DeepLAU, a neural machine translation model with a deep encoder and decoder.
Recurrent Layers
A recurrent neural network BIBREF20 is a class of neural network that has recurrent connections and a state (or its more sophisticated memory-like extension). The past information is built up through the recurrent connections. This makes RNN applicable for sequential prediction tasks of arbitrary length. Given a sequence of vectors INLINEFORM0 as input, a standard RNN computes the sequence hidden states INLINEFORM1 by iterating the following equation from INLINEFORM2 to INLINEFORM3 : DISPLAYFORM0
INLINEFORM0 is usually a nonlinear function such as composition of a logistic sigmoid with an affine transformation.
Gated Recurrent Unit
It is difficult to train RNNs to capture long-term dependencies because the gradients tend to either vanish (most of the time) or explode. The effect of long-term dependencies is dropped exponentially with respect to the gradient propagation length. The problem was explored in depth by BIBREF21 , BIBREF17 . A successful approach is to design a more sophisticated activation function than a usual activation function consisting of gating functions to control the information flow and reduce the propagation path. There is a long thread of work aiming to solve this problem, with the long short-term memory units (LSTM) being the most salient examples and gated recurrent unit (GRU) being the most recent one BIBREF21 , BIBREF22 . RNNs employing either of these recurrent units have been shown to perform well in tasks that require capturing long-term dependencies.
GRU can be viewed as a slightly more dramatic variation on LSTM with fewer parameters. The activation function is armed with two specifically designed gates called update and reset gates to control the flow of information inside each hidden unit. Each hidden state at time-step INLINEFORM0 is computed as follows DISPLAYFORM0
For Chinese-English, our training data consists of INLINEFORM0 M sentence pairs extracted from LDC corpora, with INLINEFORM1 M Chinese words and INLINEFORM2 M English words respectively. We choose NIST 2002 (MT02) dataset as our development set, and the NIST 2003 (MT03), 2004 (MT04) 2005 (MT05) and 2006 (MT06) datasets as our test sets.
For English-German, to compare with the results reported by previous work BIBREF0 , BIBREF18 , BIBREF6 , we used the same subset of the WMT 2014 training corpus that contains 4.5M sentence pairs with 91M English words and 87M German words. The concatenation of news-test 2012 and news-test 2013 is used as the validation set and news-test 2014 as the test set.
To evaluate at scale, we also report the results of English-French. To compare with the results reported by previous work on end-to-end NMT BIBREF13 , BIBREF14 , BIBREF6 , BIBREF23 , BIBREF18 , we used the same subset of the WMT 2014 training corpus that contains 12M sentence pairs with 304M English words and 348M French words. The concatenation of news-test 2012 and news-test 2013 serves as the validation set and news-test 2014 as the test set.
Training details
Our training procedure and hyper parameter choices are similar to those used by BIBREF14 . In more details, we limit the source and target vocabularies to the most frequent INLINEFORM0 words in both Chinese-English and English-French. For English-German, we set the source and target vocabularies size to INLINEFORM1 and INLINEFORM2 , respectively.
For all experiments, the dimensions of word embeddings and recurrent hidden states are both set to 512. The dimension of INLINEFORM0 is also of size 512. Note that our network is more narrow than most previous work where hidden states of dimmention 1024 is used. we initialize parameters by sampling each element from the Gaussian distribution with mean 0 and variance INLINEFORM1 .
Parameter optimization is performed using stochastic gradient descent. Adadelta BIBREF24 is used to automatically adapt the learning rate of each parameter ( INLINEFORM0 and INLINEFORM1 ). To avoid gradient explosion, the gradients of the cost function which had INLINEFORM2 norm larger than a predefined threshold INLINEFORM3 were normalized to the threshold BIBREF25 . We set INLINEFORM4 to INLINEFORM5 at the beginning and halve the threshold until the BLEU score does not change much on the development set. Each SGD is a mini-batch of 128 examples. We train our NMT model with the sentences of length up to 80 words in the training data, while for the Moses system we use the full training data. Translations are generated by a beam search and log-likelihood scores are normalized by sentence length. We use a beam width of 10 in all the experiments. Dropout was also applied on the output layer to avoid over-fitting. The dropout rate is set to INLINEFORM6 . Except when otherwise mentioned, NMT systems are have 4 layers encoders and 4 layers decoders.
Results on Chinese-English Translation
Table TABREF7 shows BLEU scores on Chinese-English datasets. Clearly DeepLAU leads to a remarkable improvement over their competitors. Compared to DeepGRU, DeepLAU is INLINEFORM0 BLEU score higher on average four test sets, showing the modeling power gained from the liner associative connections. We suggest it is because LAUs apply adaptive gate function conditioned on the input which make it able to automatically decide how much linear information should be transferred to the next step.
To show the power of DeepLAU, we also make a comparison with previous work. Our best single model outperforms both a phrased-based MT system (Moses) as well as an open source attention-based NMT system (Groundhog) by INLINEFORM0 and INLINEFORM1 BLEU points respectively on average. The result is also better than some other state-of-the-art variants of attention-based NMT mode with big margins. After PosUnk and ensemble, DeepLAU seizes another notable gain of INLINEFORM2 BLEU and outperform Moses by INLINEFORM3 BLEU.
Results on English-German Translation
The results on English-German translation are presented in Table TABREF10 . We compare our NMT systems with various other systems including the winning system in WMT’14 BIBREF26 , a phrase-based system whose language models were trained on a huge monolingual text, the Common Crawl corpus. For end-to-end NMT systems, to the best of our knowledge, Wu et al. wu2016google is currently the SOTA system and about 4 BLEU points on top of previously best reported results even though Zhou et al. zhou2016deep used a much deeper neural network.
Following Wu et al. wu2016google, the BLEU score represents the averaged score of 8 models we trained. Our approach achieves comparable results with SOTA system. As can be seen from the Table TABREF10 , DeepLAU performs better than the word based model and even not much worse than the best wordpiece models achieved by Wu et al. wu2016google. Note that DeepLAU are simple and easy to implement, as opposed to previous models reported in Wu et al. wu2016google, which dependends on some external techniques to achieve their best performance, such as their introduction of length normalization, coverage penalty, fine-tuning and the RL-refined model.
Results on English-French Translation
To evaluate at scale, we also show the results on an English-French task with INLINEFORM0 sentence pairs and INLINEFORM1 vocabulary in Table TABREF13 . Luong et al. luong2014addressing achieves BLEU score of INLINEFORM2 with a six layers deep Encoder-Decoder model. The two attention models, RNNSearch and RNNsearch-LV achieve BLEU scores of INLINEFORM3 and INLINEFORM4 respectively. The previous best single NMT Deep-Att model with an 18 layers encoder and 7 layers decoder achieves BLEU score of INLINEFORM5 . For DeepLAU, we obtain the BLEU score of INLINEFORM6 with a 4 layers encoder and 4 layers decoder, which is on par with the SOTA system in terms of BLEU. Note that Zhou et al. zhou2016deep utilize a much larger depth as well as external alignment model and extensive regularization to achieve their best results.
Analysis
Then we will study the main factors that influence our results on NIST Chinese-English translation task. We also compare our approach with two SOTA topologies which were used in building deep NMT systems.
Residual Networks (ResNet) are among the pioneering works BIBREF27 , BIBREF28 that utilize extra identity connections to enhance information flow such that very deep neural networks can be effectively optimized. Share the similar idea, Wu et al. wu2016google introduced to leverage residual connections to train deep RNNs.
Fast Forward (F-F) connections were proposed to reduce the propagation path length which is the pioneer work to simplify the training of deep NMT model BIBREF18 . The work can be viewed as a parametric ResNet with short cut connections between adjacent layers. The procedure takes a linear sum between the input and the newly computed state.
Table TABREF18 shows the effect of the novel LAU. By comparing row 3 to row 7, we see that when INLINEFORM0 and INLINEFORM1 are set to 2, the average BLEU scores achieved by DeepGRU and DeepLAU are INLINEFORM2 and INLINEFORM3 , respectively. LAU can bring an improvement of INLINEFORM4 in terms of BLEU. After increasing the model depth to 4 (row 4 and row 6), the improvement is enlarged to INLINEFORM5 . When DeepGRU is trained with larger depth (say, 4), the training becomes more difficult and the performance falls behind its shallow partner. While for DeepLAU, as can be see in row 9, with increasing the depth even to INLINEFORM6 and INLINEFORM7 we can still obtain growth by INLINEFORM8 BLEU score. Compared to previous short-cut connection methods (row 5 and row 6), The LAU still achieve meaningful improvements over F-F connections and Residual connections by INLINEFORM9 and INLINEFORM10 BLEU points respectively.
DeepLAU introduces more parameters than DeepGRU. In order to figure out the effect of DeepLAU comparing models with the same parameter size, we increase the hidden size of DeepGRU model. Row 3 shows that, after using a twice larger GRU layer, the BLEU score is INLINEFORM0 , which is still worse than the corresponding DeepLAU model with fewer parameters.
Next we will study the model size. In Table TABREF18 , starting from INLINEFORM0 and INLINEFORM1 and gradually increasing the model depth, we can achieve substantial improvements in terms of BLEU. With INLINEFORM2 and INLINEFORM3 , our DeepLAU model yields the best BLEU score. We tried to increase the model depth with the same hidden size but failed to see further improvements.
We then tried to increase the hidden size. By comparing row 2 and row 3, we find the improvements is relative small with a wider hidden size. It is also worth mentioning that a deep and thin network with fewer parameters can still achieve comparable results with its shallow partner. This suggests that depth plays a more important role in increasing the complexity of neural networks than width and our deliberately designed LAU benefit from the optimizing of such a deep model.
A more detailed comparison between DeepLAU (4 layers encoder and 4 layers decoder), DeepLAU(2 layer encoder and 2 layer decoder) and DeepGRU (4 layers encoder and 4 layers decoder), suggest that with deep architectures are essential to the superior performance of our system. In particular, we test the BLEU scores on sentences longer than INLINEFORM0 on the merged test set. Clearly, in all curves, performance degrades with increased sentence length. However, DeepLAU models yield consistently higher BLEU scores than the DeepGRU model on longer sentences. These observations are consistent with our intuition that very deep RNN model is especially good at modeling the nested latent structures on relatively complicated sentences and LAU plays an important role on optimizing such a complex deep model.
Conclusion
We propose a Linear Associative Unit (LAU) which makes a fusion of both linear and non-linear transformation inside the recurrent unit. On this way, gradients decay much slower compared to the standard deep networks which enable us to build a deep neural network for machine translation. Our empirical study shows that it can significantly improve the performance of NMT.
acknowledge
We sincerely thank the anonymous reviewers for their thorough reviewing and valuable suggestions. Wang's work is partially supported by National Science Foundation for Deep Semantics Based Uighur to Chinese Machine Translation (ID 61662077). Qun Liu's work is partially supported by Science Foundation Ireland in the ADAPT Centre for Digital Content Technology (www.adaptcentre.ie) at Dublin City University funded under the SFI Research Centres Programme (Grant 13/RC/2106) co-funded under the European Regional Development Fund.

Question: Do they use the same architecture as LSTM-s and GRUs with just replacing with the LAU unit?
Answer: Yes