Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors proposed a new neural-network based machine translation method that generates the target sentence by generating multiple partial segments in the target sentence from different positions in the source information. The model is based on the SWAN architecture which is previously proposed, and an additional "local reordering" layer to reshuffle source information to adjust those positions to the target sentence.

Using the SWAN architecture looks more reasonable than the conventional attention mechanism when the ground-truth word alignment is monotone. Also, the concept of local reordering mechanism looks well to improve the basic SWAN model to reconfigure it to the situation of machine translation tasks.

The "window size" of the local reordering layer looks like the "distortion limit" used in traditional phrase-based statistical machine translation methods, and this hyperparameter may impose a similar issue with that of the distortion limit into the proposed model; small window sizes may drop information about long dependency. For example, verbs in German sentences sometimes move to the tail of the sentence and they introduce a dependency between some distant words in the sentence. Since reordering windows restrict the context of each position to a limited number of neighbors, it may not capture distant information enough. I expected that some observations about this point will be unveiled in the paper, but unfortunately, the paper described only a few BLEU scores with different window sizes which have not enough information about it. It is useful for all followers of this paper to provide some observations about this point.
In addition, it could be very meaningful to provide some experimental results on linguistically distant language pairs, such as Japanese and English, or simply reversing word orders in either source or target sentences (this might work to simulate the case of distant reordering).

Authors argued some differences between conventional attention mechanism and the local reordering mechanism, but it is somewhat unclear that which ones are the definite difference between those approaches.

A super interesting and mysterious point of the proposed method is that it achieves better BLEU than conventional methods despite no any global language models (Table 1 row 8), and the language model options (Table 1 row 9 and footnote 4) may reduce the model accuracy as well as it works not so effectively. This phenomenon definitely goes against the intuitions about developing most of the conventional machine translation models. Specifically, it is unclear how the model correctly treats word connections between segments without any global language model. Authors should pay attention to explain more detailed analysis about this point in the paper.

Eq. (1) is incorrect. According to Fig. 2, the conditional probability in the product operator should be revised to p(a_t | x_{1:t}, a_{1:t-1}), and the independence approximation to remove a_{1:t-1} from the conditions should also be noted in the paper.
Nevertheless, the condition x_{1:t} could not be reduced because the source position is always conditioned by all previous positions through an RNN.


review_title: Reasonable modeling but some unclear point

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces a neural translation model that automatically discovers phrases.  This idea is very interesting and tries to marry phrase-based statistical machine translation with neural methods in a principled way. However, the clarity of the paper could be improved.

The local reordering layer has the ability to swap inputs, however, how do you ensure that it actually does swap inputs rather than ignoring some inputs and duplicating others?

Are all segments translated independently, or do you carry over the hidden state of the decoder RNN between segments? In Figure 1 both a BRNN and SWAN layer are shown, is there another RNN in the SWAN layer, or does the BRNN emit the final outputs after the segments have been determined?
review_title: Official review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces a new architecture for end to end neural machine translation. Inspired by the phrase based approach, the translation process is decomposed as follows : source words are embedded and then reordered; a bilstm then encodes the reordered source; a sleep wake network finally generates the target sequence as a phrase sequence built from left to right. 

This kind of approach is more related to ngram based machine translation than conventional phrase based one.  

The idea is nice. The proposed approach does not rely on attention based model. This opens nice perpectives for better and faster inference. 

My first concern is about the architecture description. For instance, the swan part is not really stand alone. For reader who does not already know this net, I'm not sure this is really clear. Moreover, there is no link between notations used for the swan part and the ones used in the reordering part. 

Then, one question arises. Why don't you consider the reordering of the whole source sentence. Maybe you could motivate your choice at this point. This is the main contribution of the paper, since swan already exists.

Finally, the experimental part shows nice improvements but: 1/ you must provide baseline results with a well tuned phrase based mt system; 2/ the datasets are small ones, as well as the vocabularies, you should try with larger datasets and bpe for sake of comparison. 
review_title: This paper introduces a new architecture for end to end neural machine translation with promising results on small datasets.

Meta-review: this submission introduces soft local reordering to the recently proposed SWAN layer [Wang et al., 2017] to make it suitable for machine translation. although only in small-scale experiments, the results are convincing.
Decision: Accept (Poster)