Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors propose to pretrain the encoder/decoder of seq2seq models on a large amount of unlabeled data using a LM objective. They obtain improvements using this technique on machine translation and abstractive summarization.

While the effectiveness of pretraining seq2seq models has been known among researchers and explored in a few papers (e.g. Zoph et al. 2016,  Dai and Le 2015), I believe this is the first paper to pretrain using a LM for both the encoder/decoder. The technique is simple, but the gains are large (e.g. +2.7 BLEU on NMT). In addition, the authors perform extensive ablation studies to analyze where the performance is coming from. Hence, I think this paper should be accepted.

review_title: good paper with strong experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors propose the use of layer-wise language model-like pretraining for encoder-decoder models. This allows to leverage separate source and target corpora (in unsupervised manner) without necessity of large amounts of parallel training corpora. The idea is in principle fairly simple, and rely on initial optimising both encoder and decoder with LSTM tasked to perform language modelling. 

The ideas are not new, and the paper is more like a successful compilation of several approaches that have been around for some time. The experimental validation, though, offers some interesting insights into importance of initialization, and the effectiveness of different initialisations approaches in enc-dec setting.

The regulariser you propose to use on page 3, looks like typical multi-task objective function, especially it is used in an alternating manner would be interesting to see whether similar performance might have been obtained starting with this objective, from random initialisation.

You should probably give credit for encoder-decoder like-RNN models published in 1990s.

Minors:
Pg. 2, Sec 2.1 2nd paragraph: can be different sizes -> can be of different sizes
review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: strengths:

A method is proposed in this paper to initialize the encoder and decoder of the seq2seq model using the trained weights of language models with no parallel data. After such pretraining, all weights are jointly fine-tuned with parallel labeled data with an additional language modeling loss.

It is shown that pretraining accelerates training and improves generalization of seq2seq models.

The main value of the proposed method is to leverage separate source and target corpora, contrasting the common methods of using large amounts of parallel training corpora.


weaknesses:

The objective function shown in the middle of pg 3 is highly empirical, not directly linked to how non-parallel data helps to improve the final prediction results. The paper should compare with and discuss the objective function based on expectation of cross entropy which is directly linked to improving prediction results as proposed in arXiv:1606.04646, Chen et al.:  Unsupervised Learning of Predictors from Unpaired Input-Output Samples, 2016.

The pre-training procedure proposed in this paper is also closely connected with the DNN pretraining method presented in Dahl et al. 2011, 2012. Comparisons should be made in the paper, highlighting why the proposed one is conceptually superior if the authors believe so. 

review_title: the paper addresses a very important issue of exploiting non-parallel training data, but it should add detailed discussions on comparing with two pieces of prior arts detailed in the review below

final_score: rating: null
review_content: Can you compare unsupervised Pretraining for Sequence to Sequence proposed in this paper with its counterpart for DNN (Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, IEEE Transactions on Audio, Speech, and Language Processing, 2012)? what is the theoretical insight you may have out of this comparison?
review_title: Unsupervised Pretraining 

Meta-review: This paper effectively demonstrates that the use of pretraining can improve the performance of seq2seq models for MT and summarization tasks. However, despite these empirical gains, the reviewers were not convinced enough by the novelty of the work itself and did not feel like there were technical contributions to make this a fit for ICLR.
 
 Pros:
 - All reviewers agree that the empirical gains in this paper are convincing and lead to BLEU improvements on a large scale translation and translation like tasks. Reviewer 4 also praises the detailed analysis that demonstrates that these gains come from the pretraining process itself. 
 - From an impact perspective, the reviewers found the approach clear and implementable. 
 
 Cons:
 - Novelty criticisms are that the method is a "compilation" of past approaches (although at a larger scale) and therefore primarily experimental, and that the objectives given are "highly empirical" and not yet motivated by theory. The authors did respond, but the reviewer did not change their score.
 - There are suggestions that this type of work would perhaps be more widely impactful in an NLP venue, where a BLEU improvement of this regard is a strong supporting piece on its own.
Decision: Reject