Title: A Convolutional Encoder Model for Neural Machine Translation
Authors: Jonas Gehring
Michael Auli
David Grangier
Yann N. Dauphin
Venue: ICLR 2017

Abstract: The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence.
In this paper we present a faster and simpler architecture based on a succession of convolutional layers. 
This allows to encode the entire source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies.
On WMT'16 English-Romanian translation we achieve competitive accuracy to the state-of-the-art and we outperform several recently published results on the WMT'15 English-German task. 
Our models obtain almost the same accuracy as a very deep LSTM setup on WMT'14 English-French translation.
Our convolutional encoder speeds up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM baseline. 

TL;DR: Investigate encoder models for translation and demonstrate that convolutions can outperform LSTMs as encoders.

Review discussion:
Hi, I have a few questions about your submission.
First, can you please explain in more detail why your convnet model is faster for decoding? Is it because less FLOPs are required? Or is it because you were able to use less hidden units? Are both implementations efficient? It's not obvious to me why 15+5 layers of convolution are two times faster than 2 layers of LSTM when a single CPU thread is used.
Second, a very interesting finding of the paper is that two convolutional network are needed for good performance, CNN-a and CNN-c. Can you provide a qualitative explanation for this phenomenon?
Finally, I think that the wording "conceptually simpler encoder" that you used in the conclusion is a very subjective judgement. Especially given that you have to use 2 different convnets.
**Why is the convnet encoder faster at decoding?**
Our architecture works very well with smaller embedding sizes which allows us to match or outperform the accuracy of the LSTM baseline at lower capacity. In the particular experiment you are referring to, we use a smaller embedding size and encoder output dimensionality which makes computation faster despite having more layers.
Also, the 2-layer bi-directional LSTM encoder baseline contains 4 LSTMs, because it is bi-directional. The accuracy of the baseline (Table 3b) has been tuned and the results are very competitive to other published work (cf. Table 2 WMT'15 English-German).
The smaller embedding size is not the only reason for the speed-up. In Table 3 (a), we compare a Conv 6/3 encoder and a BiLSTM with equal embedding sizes. The convolutional encoder is still 1.34x faster (at 0.7 higher BLEU) although it requires roughly 1.6x as many FLOPs. We believe that this is likely due to better cache locality for convolutional layers on CPUs: a LSTM with fused gates requires two big matrix multiplications with different weights as well as additions, multiplications and non-linearities  for each word in the input sentence, while each convolutional layer can be computed in a tight loop with the same weight matrix. Both architectures might benefit from a further optimized multi-core implementation, but in comparison to other NMT implementations our setup is quite fast already.
Our bi-directional LSTM implementation is based on torch rnnlib which uses fused LSTM gates ( https://github.com/facebookresearch/torch-rnnlib/blob/master/rnnlib/cell.lua) and which we consider an efficient implementation.
**Why are there two convolutional stacks in the encoder? **
Using only a single stack of convolutions was indeed much worse (22.9 BLEU vs 28.5 BLEU). The encoder output is used for two important tasks: first, it is matched to the decoder state to identify which parts of the input are relevant to generate the next token.
Second, it actually needs to encode the source sentence to provide useful inputs to the decoder. Tackling both these tasks with a simple deep convolutional network is much harder than for a bi-directional LSTM which has a gating mechanism.  Our goal was to use very simple conv-tanh blocks but there may be a more complex setting that enables a single encoder stack.
**Why do we claim our encoder model to be conceptually simpler? **
We acknowledge that this statement is subjective. We removed it and highlighted the differences between RNNs and convnets to support this statement, namely: (i) bi-directional LSTMs process past and future words in separate stacks which then need to be combined, whereas convnets process the surrounding context at every step, (ii) convnets perform the same number of non-linear operations for each word, whereas RNNs  apply a very large number of non-linearities to words far away, (iii) parallelization is easier with convnets.
The paper reports a very clear and easy to understand result that a convolutional network can be used instead of the recurrent encoder for neural machine translation.
Apart from the known architectural elements, such as convolution, pooling, residual connections, position embeddings, the paper features one unexpected architectural twist: two stacks of convolutions, one for computing alignment and another for computing the representations.
The empirical evidence that this was necessary is provided, however the question of *why* it is necessary remains open.
The experimental evaluation is very extensive and leaves no doubt that the proposed approach works well. The convnet-based model was faster at evaluation, but it is not very clear what is the main speed-up factor. It’s however hard to argue against the fact that the speed advantage of convnets is likely to increase if a more parallel implementation is considered.
My main concern is whether or not the paper is appropriate for ICLR, because the contribution is quite incremental and rather application-specific. ACL, EMNLP and other NLP conferences would be a better venue, I think.
Paper is application specific: we chose to focus on machine translation which is a challenging task with strong competition systems, dedicated product teams interested in improvements, and MT has been used in the past to introduce sequence to sequence models (e.g. Sutskever et al, Bahdanau et al). It is very likely that tasks relying on architectures similar to attention-based translation will benefit by our architecture as well, e.g. summarization, constituency parsing, or dialog modeling. In future work, we plan to evaluate on other sequence to sequence problems as well.
Two stacks: our experimentation with the two stacks suggests that the attention weights (CNN-a) need to integrate information from a wide context which can be done with a deep stack;
at the same time, the vectors which are averaged (CNN-c) seem to benefit from a shallower, more local representation closer to the input words. Two stacks is an easy way to achieve these contradicting requirements. We will add this discussion to the text to make the point clearer. Inspired by LSTMs, we are currently exploring non-linearities with gating that could potentially derive a single representation satisfying both requirements.
Faster evaluation: we will clarify this in the next version of the paper. Please refer to our previous answer to this question in the thread below.
Thank you for your comments. As far as I understand there has been no next version of the paper, and I do not think I can increase the score without seeing it in this case. It is still very unclear for me why 1-layer CNN-a + 1-layer CNN-c can perform so much better than just a 1-layer CNN, this would require a  thorough investigation, as this seems to be the main finding of the paper.
The same applies for the reasons behind the faster decoding.
We just updated the paper according to our reply to the reviewers on January 9th.
We respectfully disagree that requiring two stacks is the main finding of the paper. Our goal is to show that convolutional encoders with finite contexts can compare favorably to RNNs. This claim is supported with extensive experiments.
We also edited the discussion around the two-stack architecture.
In particular, Figure 1a already presents extensive experimentation around the depth of the two stacks. At this point we do not see how to extend our analysis further.
My last 5 cents: the fact that deep CNNs can handle long-term dependencies has been already discussed in the works on Pixel RNNs and Pixel CNNs (https://arxiv.org/abs/1601.06759, https://arxiv.org/pdf/1606.05328v2.pdf). In my view this limits the contribution of this paper to showing that the same holds for the machine translation encoder.
This paper is the first (I believe) to establish a simple yet important result that Convnets for NMT encoders can be competitive to RNNs. The authors present a convincing set of results over many translation tasks and compare with very competitive baselines. I also appreciate the detailed report on training and generation speed. I find it's very interesting when position embeddings turn out to be hugely important (beside residual connections); unfortunately, there is little analysis to shed more lights on this aspect and perhaps compare other ways of capturing positions (a wild guess might be to use embeddings that represent some form of relative positions). The only concern I have (similar to the other reviewer) is that this paper perhaps fits better in an NLP conference.
One minor comment: it's slight strange that this well-executed paper doesn't have a single figure on the proposed architecture :) It will also be even better to draw a figure for the biLSTM architecture as well (it does take some effort to understand the last paragraph in Section 2, especially the part on having a linear layer to compute z).
We will add figures illustrating our model architecture better; our baseline biLSTM architecture closely follows [Zhou et al, TACL 2016] https://arxiv.org/pdf/1606.04199v3.pdf who have detailed figures illustrating the model.
Position embeddings are important. Without them our encoder has no notion of position. We investigated position embeddings numbered from the left to right and right to left but found that one direction was enough.
Regarding conference fit: we show that RNNs are not necessary to perform encoding in a complex sequence to sequence task and that CNNs are comparable or better.
Of course, this is of interest to the NLP community but also to the ML community. In term of representation learning, the two stack CNN highlight that attention weights and the input representation might benefit from different depth, which can impact other tasks, and other models. Our work also highlights  that the success of RNNs in MT cannot be attributed to their unique ability to model long term dependencies. We believe that those findings are of great interest to the ML community.
The system described works comparably to bi-directional LSTM baseline for NMT, and CNN's are naturally parallelizable.
Key ideas include the use of two stacked CNN's (one for each of encoding and decoding) for translation, with res connections and position embeddings. The use of CNN's for translation has been attempted previously (as described by the authors), but presumably it is the authors' combination of various architectural choices (attention, position embeddings, etc) that make the present system competitive with RNN's, whereas earlier attempts were not. They describe system's sensitivity to some of these choices (e.g. experiments to choose appropriate number of layers in each of the CNN's).
The experimental results are well reported in detail.
One or two figures would definitely be required to help clarify the architecture.
This paper is less about new ways of learning representations than about the combination of choices made (over the set of existing techniques) in order to get the good results that they do on the reported NMT tasks. In this respect, while I am fairly confident that the paper represents good work in machine learning, I am not quite as confident about its fit for this particular conference.
We will add figures illustrating the architecture. Please see our response regarding conference fit in the replies to the other reviewers below.
I appreciate the additional comments about conference fit, although I do agree with AnonReviewer1 that, e.g. PixelCNN also demonstrates CNN's ability to model long-term structure (i.e. that RNN's are not unique in this regard).
PixelCNN was applied to vision and CNNs have been known to work on vision tasks for quite some time. The same is not true for language. Our paper is the first work that successfully demonstrates that CNNs can be very competitive to LSTMs on language and in particular translation. It demonstrates that CNNs can model long-range dependencies for language.

Meta-review:
role: AC
data: comment: Authors, It would be great to have a rebuttal for this paper as reviewers will be discussing over the next week. Thanks.
role: PC
data: comment: This work demonstrates architectural choices to make conv nets work for NMT. In general the reviewers liked the work and were convinced by the results but found the main contributions to be "incremental". 
 
 Pros:
 - Clarity: The work was clearly presented, and besides for minor comments (diagrams) the reviewers understood the work
 - Quality: The experimental results were thorough, "very extensive and leaves no doubt that the proposed approach works well".
 
 Mixed:
 - Novelty: There is appreciation that the work is novel. However as the work is somewhat "application-specific" the reviewers felt the technical contribution was not an overwhelming contribution.
 - Impact: While some speed ups were shown, not all reviewers were convinced that the benefit was sufficient, or "main speed-up factor(s)" were. 
 
 This work is clearly worthwhile, but the reviews place it slightly below the top papers in this area.

Decision: Reject