Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 

This paper investigates the effectiveness of a massively large parallel corpus in NMT training, which consists of more than 40 billion En-Zh parallel sentences.
To the best of my knowledge, the 40 billion parallel corpus for the NMT training is the largest reported in the paper published so far.
 
For preventing long training time, this paper proposes a practical data split and utilization method, which the authors call “dynamic-data-split.”
The key idea of their method is to dynamically assign training instances to different model components and update different components according to the assigned instances.
 
This paper reports the BLEU score of WMT17 Chinese-English dataset for 32.3, which significantly outperformed the best score, and improved the performance of existing state-of-the-art results.
They also provide several deeper analyses of the proposed method by changing the model training strategy (pretrain only, pretrain+finetune), data split strategy, data size, and tokenization (word, BPE, character).
 



The main concern of this paper is the reproducibility of the experiments.
 
Their main focus is to investigate the effectiveness of 40B massive parallel data.
However, the origin and how the authors correct the data is fully unknown; in the paper, they only say, “The data comes from diverse sources such as web pages (∼2 billion), digitized books (∼1 billion) and private purchase from translation agencies (∼46 billion).”
What is the “private purchase from translation agencies.”
No one knows how they were collected except the authors.
It is impossible to reproduce the results of the experiments conducted in this paper in future validation.


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
This work conducts a large scale study on pretraining for neural machine translation. Overall, this work makes good contributions to the community, but the experiments need improvements. 

Pros: 
	1. The data scale is huge, with 40 billion sentence pairs.
	2. The results are promising, with 3.2 BLEU improvement over STOA results.

Cons:
	1. It is pity that the trained model is only evaluated on one test set and experiments are conducted on one language pair. Thus, it is not clear to me whether the improvement is general across datasets and language pairs. I understand that it is costly to conduct such a large scale study on another language pair. At least, it is easy to test the models on other datasets for the same language pair. For example, I'd like to see the results on WMT 2019 Chinese->English translation dataset.
	2. I'm curious how large-scale pretraining compare with large-scale back translation. The following paper shows that large-scale back translation can also significantly improve the final translation accuracy. Note that back translation only needs monolingual data, while the pretraining in this work needs bilingual sentence pairs. 
Edunov, Sergey, Myle Ott, Michael Auli, and David Grangier. "Understanding back-translation at scale." arXiv preprint arXiv:1808.09381 (2018).

Besides, will the model be shared to the public?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes an approach to train NMT models on extremely large parallel corpora. Because of the dataset size, training several epochs on the full dataset with a single model is too expensive. As a result, the dataset is split into several chunks, on which different models are trained independently. The different models are combined to form an ensemble model. Different strategies are proposed to split the dataset effectively. The resulting model achieves a performance of 32.3, outperforming the previous SOTA by 3.2 BLEU.

In Section 4.3, I'm curious about why only 6 layers are used in the encoder and decoder? Large scale pretraining like in BERT usually benefits from very large datasets, but also very large models. 6 layers seems very small given the size of the training set. Table 2 reports results with "Small" and "Large" models. What does this correspond to? Only Section 4.3 discusses the size of the model, but it does not mention different architectural choices.

In Section 5.1 the paper mentions "the single-model achieves a BLEU score of 29.7, already outperforming the current best system", but in Tables 1 and 2 it seems that the best score with single model is 28.7, not 29.7.

I feel that the results are a bit disappointing given the scale of the experiments. Table 2 suggests that a single model, even trained with 40B sentence pairs, does not outperform a single model trained with 20M sentence pairs as in "He et al., 2019", while being significantly more expensive to train. Also, the comparisons in Table 1 are done between single and ensemble models, which is not a fair comparison. The model with a BLEU score of 32.0 uses an ensemble of 10 models. What would be the performance of an ensemble of 10 models trained with the regular 20M parallel sentence pairs?

Also, did you try the approach of "Hassan et al (2018)" suggested in Section 4.1, where only in-domain sentences are selected? It is true that  "every time we encounter a new domain, we have to retrain the model", but I think this is still a more viable approach than pretraining on the full 40B sentences. Why not trying to train on the top-100M sentence pairs that are the most in-domain?

In the back-translation (BT) experiments, did you select 100M monolingual sentences randomly? If that is the case, this is expected to see a drop in performance, BT is usually a very effective, but not so much when the monolingual data is noisy or out of domain. Although it is critical to work with cleaned data in NLP (especially in the context of generation), dataset cleaning is not really addressed in the paper.

Overall, the experimental setup is impressive, but the improvements in terms of BLEU are relatively small, and the technical contributions seem quite thin to me for a ML conference. Moreover, the dataset used in the paper is not available to the research community, which prevents reproducibility. Also, as mentioned above, I think several important experiments are missing.
review_title: Official Blind Review #1

Meta-review: The authors address the problem of training an NMT model on a really massive parallel data set of 40 billion Chinese-English sentence pairs, an order of magnitude bigger than other cz-en experiments. To address noise and training time problems they propose pretraining + a couple of different ways of creating a fine-tuning data set. Two of the reviewers assert that the technical contribution is thin, and the results are SOTA but not really as good as you might hope with this amount of data. This combined with the fact that the data set is not released, makes me think that this paper is not a good fit with ICLR and would more appropriate for an application focussed conference. The authors engaged strongly with the reviewers, adding more backtranslation results. The reviewers took their responses into account but did not change their scores. 
Decision: Reject