Venue: EMNLP 2017

# Massive Exploration of Neural Machine Translation Architectures

Denny Britz, Anna Goldie1, Minh-Thang Luong, Quoc V. Le

{dennybritz,agoldie,thangluong,qvl}@google.com Google Brain

Both authors contributed equally to this work.Work done as a member of the Google Brain Residency program (g.co/brainresidency).

Footnote 2: footnotemark:

###### Abstract

Neural Machine Translation (NMT) has shown remarkable progress over the past few years, with production systems now being deployed to end-users. As the field is moving rapidly, it has become unclear which elements of NMT architectures have a significant impact on translation quality. In this work, we present a large-scale analysis of the sensitivity of NMT architectures to common hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on a WMT English to German translation task. Our experiments provide practical insights into the relative importance of factors such as embedding size, network depth, RNN cell type, residual connections, attention mechanism, and decoding heuristics. As part of this contribution, we also release an open-source NMT framework in TensorFlow to make it easy for others to reproduce our results and perform their own experiments.

## 1 Introduction

Neural Machine Translation (NMT) Kalchbrenner and Blunsom (2013); Sutskever et al. (2014); Cho et al. (2014) is an end-to-end approach to machine translation. NMT has shown impressive results Jean et al. (2015); Luong et al. (2015); Sennrich et al. (2016); Wu et al. (2016) surpassing those of phrase-based systems while addressing shortcomings, such as the need for hand-engineered features. The most popular approaches to NMT are based on sequence-to-sequence models, an encoder-decoder architecture consisting of two recurrent neural networks (RNNs) and an attention mechanism that aligns target with source tokens Bahdanau et al. (2015); Luong et al. (2015).

One drawback of current NMT architectures is the huge amount of compute required to train them. Training on real-world datasets of several million examples typically requires dozens of GPUs and convergence time is on the order of days to weeks Wu et al. (2016). While sweeping across large hyperparameter spaces is common in Computer Vision Huang et al. (2016), such exploration would be prohibitively expensive for NMT models, limiting researchers to well-established architecture and hyperparameter choices. Furthermore, there have been no large-scale studies of how these hyperparameters affect the performance of NMT systems. As a result, it remains unclear why these models perform as well as they do or how we might improve them.

In this work, we present an extensive analysis of architectural hyperparameters for NMT systems. Using a total of more than 250,000 GPU hours, we explore common variations of NMT architectures and provide insights into which architectural choices matter most. We report BLEU scores, perplexities, model sizes, and convergence time for all experiments, including variance numbers calculated across several runs of each experiment. In addition, we release the software framework that we wrote to facilitate this exploration.

In summary, the main contributions of this work are as follows:

* We provide immediately applicable insights into the optimization of NMT models, as well as promising directions for future research. For example, we found that deep encoders are

more difficult to optimize than decoders, that dense residual connections yield better performance than regular residual connections, and that a well-tuned beam search is surprisingly critical to obtaining state-of-the-art results. By presenting practical advice for choosing baseline architectures, we help researchers avoid wasting time on unpromising model variations.
* We also establish the extent to which metrics such as BLEU are influenced by random initialization and slight hyperparameter variation, allowing researchers to better distinguish statistically significant results from noise.
* Finally, we release an open-source TensorFlow package, specifically designed to implement reproducible state-of-the-art sequence-to-sequence models. All experiments were run using this framework and we include all configuration files and processing scripts needed to reproduce the experiments in this paper. We hope to accelerate future research by releasing this framework to the public.

## 2 Background and Preliminaries

### Neural Machine Translation

Our models are based on an encoder-decoder architecture with attention mechanism Bahdanau et al. (2015); Luong et al. (2015), as shown in figure 1. An encoder function \(f_{enc}\) takes as input a sequence of source tokens \(\mathbf{x}=(x_{1},...,x_{m})\) and produces a sequence of states \(\mathbf{h}=(h_{1},...,h_{m})\). In our base model, \(f_{enc}\) is a bi-directional RNN and the state \(h_{i}\) corresponds to the concatenation of the states produced by the backward and forward RNNs, \(h_{i}=[\overline{h_{i}};\overline{h_{i}}]\). The decoder \(f_{dec}\) is an RNN that predicts the probability of a target sequence \(\mathbf{y}=(y_{1},...,y_{k})\) based on \(\mathbf{h}\). The probability of each target token \(y_{i}\in{1,...V}\) is predicted based on the recurrent state in the decoder RNN \(s_{i}\), the previous words, \(y_{<i}\), and a context vector \(c_{i}\). The context vector \(c_{i}\) is also called the attention vector and is calculated as a weighted average of the source states.

\[c_{i} =\sum_{j}a_{ij}h_{j} \tag{1}\] \[a_{ij} =\frac{\hat{a}_{ij}}{\sum_{j}\hat{a}_{ij}}\] (2) \[\hat{a}_{ij} =att(s_{i},h_{j}) \tag{3}\]

Here, \(att(s_{i},h_{j})\) is an attention function that calculates an unnormalized alignment score between the encoder state \(h_{j}\) and the decoder state \(s_{i}\). In our base model, we use a function of the form \(att(s_{i},h_{j})=\langle W_{h}h_{j},W_{s}s_{i}\rangle\), where the matrices \(W\) are used to transform the source and target states into a representation of the same size.

The decoder outputs a distribution over a vocabulary of fixed-size \(V\):

\[P(y_{i}|y_{1},...,y_{i-1},\mathbf{x})\] \[=\text{softmax}(W[s_{i};c_{i}]+b)\]

The whole model is trained end-to-end by minimizing the negative log likelihood of the target words using stochastic gradient descent.

## 3 Experimental Protocols

### Datasets and Preprocessing

We run all experiments on the WMT'15 English\(\rightarrow\)German task consisting of 4.5M sentence pairs, obtained by combining the Europarl v7, News Commentary v10, and Common Crawl corpora. We use newstest2013 as our validation set and newstest2014 and newstest2015 as our test sets. We focus on WMT English\(\rightarrow\)German because it is a morphologically rich language therefore has been a standard benchmark in previous important work in Neural Machine Translation Jean et al. (2015); Luong et al. (2015); Sennrich et al. (2016); Zhou et al. (2016); Wu et al. (2016)

To test for generality, we also ran a small number of experiments on English\(\rightarrow\)French translation, and we found that the performance was highly correlated with that of English\(\rightarrow\)German but that it took much longer to train models on the larger English\(\rightarrow\)French dataset. Given that translation from the morphologically richer German is also considered a more challenging task, we felt justified in using the English\(\rightarrow\)German translation task for this hyperparameter sweep 

We tokenize and clean all datasets with the scripts in Moses1 and learn shared subword units using Byte Pair Encoding (BPE) Sennrich et al. (2016) using 32,000 merge operations for a final vocabulary size of approximately 37k. We discovered that data preprocessing can have a large impact on final numbers, and since we wish to enable reproducibility, we release our data preprocessing scripts together with the NMT framework to the public. For more details on data preprocessing parameters, we refer the reader to the code release.

Footnote 1: [https://github.com/moses-smt/mosesdecoder/](https://github.com/moses-smt/mosesdecoder/)

### Training Setup and Software

All of the following experiments are carried out using our own implementation based on TensorFlow Abadi et al. (2016). We built this framework to enable reproducible state-of-the-art implementations of Neural Machine Translation architectures. As part of our contribution, we are releasing the framework and all configuration files needed to reproduce our results. Training is performed on Nvidia Tesla K40m and Tesla K80 GPUs, distributed over 8 parallel workers and 6 parameter servers per experiment. We use a batch size of 128 and decode using beam search with a beam width of 10 and the length normalization penalty of 0.6 described in Wu et al. (2016). BLEU scores are calculated on tokenized data using the _multi-bleu.perl_ script in Moses.2 Each experiment is run for a maximum of 2.5M steps and replicated 4 times with different initializations. We save model checkpoints every 30 minutes and choose the best checkpoint based on the validation set BLEU score. We report _mean and standard deviation_ as well as _highest scores_ (as per cross validation) for each experiment.

Footnote 2: [https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl)

### Baseline Model

Based on a review of recent literature, we chose a baseline model that we knew would perform reasonably well. Our goal was to keep the baseline model simple and standard, not to advance the start of the art. The model (described in 2.1) consists of a 2-layer bidirectional encoder (1 layer in each direction), and a 2 layer decoder with a multiplicative Luong et al. (2015) attention mechanism. We use 512-unit GRU Cho et al. (2014) cells for both the encoder and decoder and apply Dropout of 0.2 at the input of each cell. We train using the Adam optimizer and a fixed learning rate of \(0.0001\) without decay. The embedding dimensionality is set to 512. A more detailed description of all model hyperparameters can be found in the supplementary material.

In each of the following experiments, the hy

Figure 1: Encoder-Decoder architecture with attention module. Section numbers reference experiments corresponding to the components.

 

perparameters of the baseline model are held constant, except for the one hyperparameter being studied. We hope that this allows us to isolate the effect of various hyperparameter changes. We recognize that this procedure does not account for interactions between hyperparameters, and we perform additional experiments when we believe such interactions are likely to occur (e.g., skip connections and number of layers).

## 4 Experiments and Results

For the sake of brevity, we only report mean BLEU, standard deviation, highest BLEU in parentheses, and model size in the following tables. Log perplexity, tokens/sec and convergence times can be found in the supplementary material tables. All reported p-values were calculated with a two-sample t-test that assumed equal variances.

### Embedding Dimensionality

With a large vocabulary, the embedding layer may account for a significant fraction of the model parameters. Historically, researchers have used 620-dimensional Bahdanau et al. (2015) or 1024-dimensional Luong et al. (2015a) embeddings. We expected larger embeddings to result in better BLEU scores, or at least lower perplexities, but this wasn't always the case. While table 1 shows that 2048-dimensional embeddings yielded the overall best result, they only outperformed the smallest 128-dimensional embeddings by a narrow yet statistically significant margin (p = 0.01), but took nearly twice as long to converge. Gradient updates to both small and large embeddings did not differ significantly from each other and the norm of gradient updates to the embedding matrix stayed approximately constant throughout training, regardless of size. We did not observe overfitting with large embeddings and training log perplexity was almost equal across experiments, suggesting that the model does not make efficient use of the extra parameters and that there may be a need for better optimization techniques. Alternatively, it could be the case that models with large embeddings simply need far more than 2.5M steps to converge to the best solution.

### RNN Cell Variant

Both LSTM Hochreiter and Schmidhuber (1997) and GRU Cho et al. (2014) cells are commonly used in NMT architectures. While there exist studies Greff et al. (2016) that explore cell variants on small sequence tasks of a few thousand examples, we are not aware of any such studies in large-scale NMT settings.

The vanishing gradient problem is a motivation for gated cells, such as the GRU and LSTM. Using vanilla RNN cells, deep networks cannot efficiently propagate information and gradients through multiple layers and time steps. We initialize the decoder state to zero instead of passing the encoder state, and we experiment with using a vanilla RNN cell in the decoder only Vanilla-Dec below). For the LSTM and GRU variants, we vary cell types in both the encoder and decoder. We use LSTM cells without peephole connections and initialize the forget bias of both LSTM and GRU cells to 1.

In our experiments, _LSTM cells consistently outperformed GRU cells_, a result which was statistically significant (p \(<\) 0.00001). Since the computational bottleneck in our architecture is the softmax operation, we did not observe large differences in training speed between LSTM and GRU cells. Somewhat to our surprise, we found that the vanilla decoder is unable to learn nearly as well as the gated variant. This suggests that the decoder indeed passes information in its own state through multiple time steps instead of relying solely on the attention mechanism and current input (which includes the previous attention context). It could also be the case that the gating mechanism is necessary to mask out irrelevant parts of the input.

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Dim** & **newstest2013** & **Params** \\ \hline
128 & \(21.50\pm 0.16\) (21.66) & 36.13M \\
256 & \(21.73\pm 0.09\) (21.85) & 46.20M \\
512 & \(21.78\pm 0.05\) (21.83) & 66.32M \\
1024 & \(21.36\pm 0.27\) (21.67) & 106.58M \\
2048 & \(\mathbf{21.86}\pm 0.17\) (22.08) & 187.09M \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 1: BLEU scores on newstest2013, varying the embedding dimensionality.

\begin{table}
\begin{tabular}{|l|l|l|} \hline
**Cell** & **newstest2013** & **Params** \\ \hline LSTM & \(\mathbf{22.22}\pm 0.08\) (22.33) & 68.95M \\ GRU & \(21.78\pm 0.05\) (21.83) & 66.32M \\ Vanilla-Dec & \(15.38\pm 0.28\) (15.73) & 63.18M \\ \hline \end{tabular}
\end{table}
Table 2: BLEU scores on newstest2013, varying the type of encoder and decoder cell.

 



### Encoder and Decoder Depth

We generally expect deeper networks to converge to better solutions than shallower ones [14]. While some work [20, 37, 56] has achieved state-of-the-art results using deep networks, others [8, 9, 10] have produced similar results with far shallower ones. Therefore, it is unclear how important depth is, and whether shallow networks are capable of producing results competitive with those of deep networks. Here, we explore the effect of both encoder and decoder depth up to 8 layers. For the bidirectional encoder, we separately stack the RNNs in both directions. For example, the Enc-8 model corresponds to one forward and one backward 4-layer RNN. For deeper networks, we also experiment with two variants of residual connections [14, 15] to encourage gradient flow. In the standard variant, shown in equation (4), we insert residual connections between consecutive layers. If \(h_{t}^{(l)}(x_{t}^{(l)},h_{t-1}^{(l)})\) is the RNN output of layer \(l\) at time step \(t\), then:

\[x_{t}^{(l+1)}=h_{t}^{(l)}(x_{t}^{(l)},h_{t-1}^{(l)})+x_{t}^{(l)} \tag{4}\]

where \(x_{t}^{(0)}\) are the embedded input tokens. We also explore a dense ("ResD" below) variant of residual connections similar to those used by [20] in Image Recognition. In this variant, we add skip connections from each layer to all other layers:

\[x_{t}^{(l+1)}=h_{t}^{(l)}(x_{t}^{(l)},h_{t-1}^{(l)})+\sum_{j=0}^{l}x_{t}^{(j)} \tag{5}\]

Our implementation differs from [20] in that we use addition instead of concatenation in order to keep the state size constant.

Table 3 shows results of varying encoder and decoder depth with and without residual connection. We found no benefit to increasing encoder depth beyond two layers, as we observed no statistically significant improvement from going to four layers and even deeper models generally diverged during training. The best deep residual models achieved good results, but only one of four runs converged, as suggested by the large standard deviation.

On the decoder side, deeper models outperformed shallower ones by a small but statistically significant margin (p < 0.00001), but without residual connections, we were unable to train decoders with 8 or more layers. Across the deep decoder experiments, dense residual connections consistently outperformed regular residual connections (p < 0.00001) and converged much faster in terms of step count, as shown in figure 2. We expected deep models to perform better [21, 56, 56] across the board, and we believe that our experiments demonstrate the need for more robust techniques for optimizing deep sequential models. For example, we may need a better-tuned SGD optimizer or some form of batch normalization, in order to robustly train deep networks with residual connections.

### Unidirectional vs. Bidirectional Encoder

In the literature, we see bidirectional encoders [1], unidirectional encoders [20], and a mix of both [56] being used. Bidirectional encoders are able to create representations that take into account both past and future inputs, while unidirectional encoders can only take past inputs into account. The benefit of unidirectional encoders is that their computation can be easily parallelized on GPUs, allowing them to run faster than their bidirectional counterparts. We are not aware of any studies that explore the necessity of bidirectional

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Depth** & **newstest2013** & **Params** \\ \hline Enc-2 & \(21.78\pm 0.05\) (21.83) & 66.32M \\ Enc-4 & \(\mathbf{21.85}\pm 0.32\) (22.23) & 69.47M \\ Enc-8 & \(21.32\pm 0.14\) (21.51) & 75.77M \\ Enc-8-Res & \(19.23\pm 1.96\) (21.97) & 75.77M \\ Enc-8-ResD & \(17.30\pm 2.64\) (21.03) & 75.77M \\ \hline Dec-1 & \(21.76\pm 0.12\) (21.93) & 64.75M \\ Dec-2 & \(21.78\pm 0.05\) (21.83) & 66.32M \\ Dec-4 & \(\mathbf{22.37}\pm 0.10\) (22.51) & 69.47M \\ Dec-4-Res & \(17.48\pm 0.25\) (17.82) & 68.69M \\ Dec-4-ResD & \(21.10\pm 0.24\) (21.43) & 68.69M \\ Dec-8 & \(01.42\pm 0.23\) (1.66) & 75.77M \\ Dec-8-Res & \(16.99\pm 0.42\) (17.47) & 75.77M \\ Dec-8-ResD & \(20.97\pm 0.34\) (21.42) & 75.77M \\ \hline \end{tabular}
\end{table}
Table 3: BLEU scores on newstest2013, varying the encoder and decoder depth and type of residual connections.



ity. In this set of experiments, we explore unidirectional encoders of varying depth with and without reversed source inputs, as this is a commonly used trick that allows the encoder to create richer representations for earlier words. Given that errors on the decoder side can easily cascade, the correctness of early words has disproportionate impact.

Table 4 shows that bidirectional encoders generally outperform unidirectional encoders, but not by a statistically significant margin. The encoders with reversed source consistently outperform their non-reversed counterparts (p = 0.009 for one layer models, p = 0.0003 for two layers, p = 0.2751 for four layers), but do not beat shallower bidirectional encoders.

### Attention Mechanism

The two most commonly used attention mechanisms are the additive Bahdanau et al. (2015) variant, equation (6) below, and the computationally less expensive multiplicative variant Luong et al. (2015a), equation (7) below. Given an attention key \(h_{j}\) (an encoder state) and attention query \(s_{i}\) (a decoder state), the attention score for each pair is calculated as follows:

\[\text{score}(h_{j},s_{i}) =\langle v,tanh(W_{1}h_{j}+W_{2}s_{i})\rangle \tag{6}\] \[\text{score}(h_{j},s_{i}) =\langle W_{1}h_{j},W_{2}s_{i}\rangle \tag{7}\]

We call the dimensionality of \(W_{1}h_{j}\) and \(W_{2}s_{i}\) the "attention dimensionality" and vary it from 128 to 1024 by changing the layer size. We also experiment with using no attention mechanism by initializing the decoder state with the last encoder state (None-State), or concatenating the last encoder state to each decoder input (None-Input). The results are shown in table 5.

We found that the parameterized additive attention mechanism slightly but consistently outperformed the multiplicative one (p = 0.013 for 128 units, p = 0.5 for 256 units, p = 0.0012 for 512 units, and p < 0.00001 for 1024/8 units), with the attention dimensionality having little effect.

While we did expect the attention-based models to significantly outperform those without an attention mechanism, we were surprised by just how poorly the "Non-Input" models fared, given that they had access to encoder information at each time step. Furthermore, we found that the attention-based models exhibited significantly larger gradient updates to decoder states throughout training. This suggests that the attention mechanism acts more like a "weighted skip connection" that optimizes gradient flow than like a "memory" that allows the encoder to access source states, as is commonly stated in the literature. We believe that further research in this direction is necessary to shed light on the role of the attention mecha

\begin{table}
\begin{tabular}{|l|c|l|} \hline
**Cell** & **newstest2013** & **Params** \\ \hline Bidi-2 & \(\mathbf{21.78}\pm 0.05\) (21.83) & 66.32M \\ Uni-1 & \(20.54\pm 0.16\) (20.73) & 63.44M \\ Uni-1R & \(21.16\pm 0.35\) (21.64) & 63.44M \\ Uni-2 & \(20.98\pm 0.10\) (21.07) & 65.01M \\ Uni-2R & \(21.76\pm 0.21\) (21.93) & 65.01M \\ Uni-4 & \(21.47\pm 0.22\) (21.70) & 68.16M \\ Uni-4R & \(21.32\pm 0.42\) (21.89) & 68.16M \\ \hline \end{tabular}
\end{table}
Table 4: BLEU scores on newstest2013, varying the type of encoder. The ”R” suffix indicates a reversed source sequence.

Figure 2: Training plots for deep decoder with and without residual connections, showing log perplexity on the eval set.

 

nism and whether it may be purely a vehicle for easier optimization.

### Beam Search Strategies

Beam Search is a commonly used technique to find target sequences that maximize some scoring function \(s({\bf y},{\bf x})\) through tree search. In the simplest case, the score to be maximized is the log probability of the target sequence given the source. Recently, extensions such as coverage penalties Tu et al. (2016) and length normalizations Wu et al. (2016) have been shown to improve decoding results. It has also been observed Tu et al. (2017) that very large beam sizes, even with length penalty, perform worse than smaller ones. Thus, choosing the correct beam width can be crucial to achieving the best results.

Table 6 shows the effect of varying beam widths and adding length normalization penalties. A beam width of 1 corresponds to greedy search. We found that a well-tuned beam search is crucial to achieving good results, and that it leads to consistent gains of more than one BLEU point. Similar to Tu et al. (2017) we found that very large beams yield worse results and that there is a "sweet spot" of optimal beam width. We believe that further research into the robustness of hyperparameters in beam search is crucial to progress in NMT. We also experimented with a coverage penalty, but found no additional gain over a sufficiently large length penalty.

### Final System Comparison

Finally, we compare our best performing model across all experiments, as chosen on the newstest2013 validation set, to historical results found in the literature in Table 8. Interestingly, the best performing model turned out to be nearly equivalent to the base model (described in Section 3.3), differing only in that it used 512-dimensional additive attention. While not the focus on this work, we were able to achieve further improvements by combining all of our insights into a single model described in Table 7.

Although we do not offer architectural innovations, we do show that through careful hyperparameter tuning and good initialization, it is possible to achieve state-of-the-art performance on standard WMT benchmarks. Our model is outperformed only by Wu et al. (2016), a model which is significantly more complex and lacks a public

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Beam** & **newstest2013** & **Params** \\ \hline B1 & \(20.66\pm 0.31\) (21.08) & 66.32M \\ B3 & \(21.55\pm 0.26\) (21.94) & 66.32M \\ B5 & \(21.60\pm 0.28\) (22.03) & 66.32M \\ B10 & \(21.57\pm 0.26\) (21.91) & 66.32M \\ B25 & \(21.47\pm 0.30\) (21.77) & 66.32M \\ B100 & \(21.10\pm 0.31\) (21.39) & 66.32M \\ B10-LP-0.5 & \(21.71\pm 0.25\) (22.04) & 66.32M \\ B10-LP-1.0 & \(\mathbf{21.80}\pm 0.25\) (22.16) & 66.32M \\ \hline \end{tabular}
\end{table}
Table 6: BLEU scores on newstest2013, varying the beam width and adding length penalties (LP).

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline
* \\ Luong & 20.9 &
- \\ BPE-Char & 21.5 & 23.9 \\ BPE &
- & 20.5 \\ RNNSearch-LV & 19.4 &
- \\ RNNSearch &
- & 16.5 \\ \hline Deep-Att* & 20.6 &
- \\ GNMT* & 24.61 &
- \\ Deep-Conv* &
- & 24.3 \\ \hline \end{tabular}
\end{table}
Table 7: Hyperparameter settings for our final combined model, consisting of all of the individually optimized values.

\begin{table}
\begin{tabular}{|l|l|} \hline
* & **Value*
* \\ \hline embedding dim & 512 \\ rnn cell variant & LSTMCell \\ encoder depth & 4 \\ decoder depth & 4 \\ attention dim & 512 \\ attention type & Bahdanau \\ encoder & bidirectional \\ beam size & 10 \\ length penalty & 1.0 \\ \hline \end{tabular}
\end{table}
Table 8: Comparison to RNNsearch Jean et al. (2015), RNNSearch-LV Jean et al. (2015), BPE Sennrich et al. (2016), BPE-Char Chung et al. (2016), Deep-Att Zhou et al. (2016), Luong Luo et al. (2015a), Deep-Conv Gehring et al. (2016), GNMT Wu et al. (2016), and OpenNMT Klein et al. (2017). Systems with an
* do not have a public implementation.
 

implementation.

To test whether our findings generalize to other languages, we also trained a model with the same hyperparameter configurations on the AS-PEC Japanese to English translation task and achieved a BLEU score of 38.87, which is state-of-the-art.

## 5 Open-Source Release

We demonstrated empirically how small changes to hyperparameter values and different initialization can affect results, and how factors such as a well-tuned beam search are critical to high quality translation results. To move towards reproducible research, we believe it is important that researchers start building upon common frameworks and data processing pipelines. With this goal in mind, we built a modular software framework that allows researchers to explore novel architectures with minimal code changes, and define experimental parameters in a reproducible manner. While our initial experiments are in Machine Translation, our framework can easily be adapted to problems in Summarization (e.g., Nallapati et al. (2016)), Conversational Modeling (e.g., Vinyals and Le (2015); Shang et al. (2015); Sordoni et al. (2015); Li et al. (2015)) or Image-To-Text (e.g., Vinyals et al. (2015); Karpathy and Fei-Fei (2015); Xu et al. (2015)).

Although there exist open-source libraries such as OpenNMT Klein et al. (2017) that share similar goals, they have not yet achieved state-of-the-art results (see table 8) and lack some important features, such as support for distributed training. We hope that by open sourcing our experimental toolkit, we can help to accelerate research in neural machine translation and sequence-to-sequence modeling.

## 6 Conclusion

We conducted a large-scale empirical analysis of architecture variations for Neural Machine Translation, teasing apart the key factors to achieving state-of-the-art results. We demonstrated a number of surprising insights, including the fact that beam search tuning is just as crucial as most architectural variations, and that with current optimization techniques deep models do not always outperform shallow ones. Here, we summarize our practical findings:

* Large embeddings with 2048 dimensions achieved the best results, but only by a small margin. Even small embeddings with 128 dimensions seem to have sufficient capacity to capture most of the necessary semantic information.
* LSTM Cells consistently outperformed GRU Cells.
* Bidirectional encoders with 2 to 4 layers performed best. Deeper encoders were significantly more likely to diverge, but show potential if they can be optimized well.
* Deep 4-layer decoders slightly outperformed shallower decoders. Residual connections were necessary to train decoders with 8 layers and dense residual connections offer additional robustness.
* Parameterized additive attention yielded the overall best results.
* A well-tuned beam search with length penalty is crucial. Beam widths of 5 to 10 along with a length penalty of 1.0 seemed to work well.

We highlighted several important research questions, including the efficient use of embedding parameters (4.1), the role of attention mechanisms as weighted skip connections (4.5) as opposed to memory units, the need for better optimization methods for deep recurrent networks (4.3), and the need for a better beam search (4.6) robust to hyperparameter variations.

Finally, given the recent surge in new applications for sequence-to-sequence models, we believe our new findings and state-of-the-art open-source package can significantly accelerate the pace of research in this domain.

## References

* Abadi et al. [2016] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In _OSDI_.

 

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _ICLR_.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _EMNLP_.
* Chung et al. (2016) Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In _ACL_.
* Gehring et al. (2016) Jonas Gehring, Michael Auli, David Grangier, and Yann N. Dauphin. 2016. A convolutional encoder model for neural machine translation. _CoRR_ abs/1611.02344.
* Greff et al. (2016) Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnik, Bas R. Steunebrink, and Jurgen Schmiduber. 2016. LSTM: A search space odyssey. _IEEE Transactions on Neural Networks and Learning Systems_ PP(99):1-11.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _CVPR_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_ 9(8):1735-1780.
* Huang et al. (2016a) Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. 2016a. Densely connected convolutional networks. _CoRR_ abs/1608.06993.
* Huang et al. (2016b) Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy. 2016b. Speed/accuracy trade-offs for modern convolutional object detectors. _CoRR_ abs/1611.10012.
* Jean et al. (2015) Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In _ACL_.
* Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In _EMNLP_.
* Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In _CVPR_.
* Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. _CoRR_ abs/1701.02810.
* Li et al. (2015) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. _arXiv preprint arXiv:1510.03055_ .
* Luong and Manning (2016) Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In _ACL_.
* Luong et al. (2015a) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In _EMNLP_.
* Luong et al. (2015b) Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In _ACL_.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. _arXiv preprint arXiv:1602.06023_ .
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. In _ACL_.
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In _ACL_.
* Shang et al. (2015) Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. _arXiv preprint arXiv:1503.02364_ .
* Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. _arXiv preprint arXiv:1506.06714_ .
* Srivastava et al. (2015) Rupesh Kumar Srivastava, Klaus Greff, and Jurgen Schmidhuber. 2015. Highway networks. _arXiv preprint arXiv:1505.00387_ .
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _NIPS_.
* Tu et al. (2017)Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In _AAAI_.
* Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In _ACL_.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. _arXiv preprint arXiv:1506.05869_ .
* Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Kriktun, Yuan Cao, Qin Gao, Klaus 

Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _CoRR_ abs/1609.08144.
* Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In _ICML_.
* Zhou et al. (2016) Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep recurrent models with fast-forward connections for neural machine translation. _TACL_ .