Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper describes a method for training self-supervised neural machine translation systems from a document-aligned comparable corpus (Wikipedia in en, fr, de and es).

The proposed training method consists of two concurrent processes: a pseudo-parallel sentence pair extraction process, where average word embeddings and encoder states are used to construct sentence embeddings which are compared to extract candidate sentence pairs, and a conventional model optimization process that uses online batches of the extracted sentence pairs as training data.

Experimental results on automatically evaluated translation quality on standard test sets are reported, in addition to parallel sentence extraction quality evaluated on the Europarl corpus and additional analyses on the self-induced curriculum resulting from the training process.

The proposed methodology is solid. The main issue with the paper is the lack of proper baseline comparison. The authors compare only with supervised and unsupervised systems trained on different corpora, and not with other approaches based on pseudo-parallel data extraction from Wikipedia.

EDIT:

I have increased my score based on the author's response.
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper studies how to extract/select suitable training data from comparable —rather than parallel— corpora. The idea sounds reasonable.

My major concern is about the evaluation: it didn't compare with any existing work. Actually there quite a few papers  on mining parallel sentences from comparable corpora such as Wikipedia, as shown below. Seems the authors are not aware of those works and didn't review and compare with them. Without such comparisons, it is difficult to judge the effectiveness of the proposed method and the quality of this work. 
[1] Finding similar sentences across multiple languages in Wikipedia, Proceedings of the Workshop on NEW TEXT Wikis and blogs and other dynamic text sources. 2006.
[2] Method for building sentence-aligned corpus from wikipedia, 2008 AAAI Workshop on Wikipedia and Artificial Intelligence (WikiAI08). 2008.
[3] Extracting parallel sentences from comparable corpora using document-level alignment, Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2010.
[4] "Improving machine translation performance by exploiting non-parallel corpora." Computational Linguistics2006.
[5] https://www.aclweb.org/anthology/W04-3208.pdf
[6] https://openreview.net/pdf?id=ryza73R9tQ

Minor issues:
	1. "For each language pair, a shared byte-pair encoding (BPE) (Sennrich et al., 2016) of 100k merge operations is applied." Most papers on neural machine translation don't use such a large BPE size, which is likely to lead to better performance. It would be better to use the same setting as previous work for fair comparisons.

	2. "In the case of SS-NMT, both tasks —data extraction and learning NMT— enable and enhance each other, such that this mutual supervision leads to a self-induced curriculum, which is the subject to our analysis." Similar idea, mutual boosting between data selection and model training, has been explored in the following paper, although not for machine translation. What's the difference between these two papers?
Learning to Teach, ICLR 2018.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: ** Paper summary **
Self-supervised machine translation (SS-NMT) is a problem where extracting data and training an NMT model are simultaneously conducted. (Ruiter et al., 2019) proposed several rules to select data and train models. This paper analyzes the following aspects of self-supervised machine translation (SS-NMT):
1.	Data extraction quality: precision and recall increase w.r.t. training iterations.
2.	Closeness to translation tasks: For the extracted sentences, as training goes on, the complexity decreases to the average level of potential bilingual corpus; the similarly of extracted sentences becomes closer. The authors also find that a joint process of extracting data and training models outperforms training a model with the extracted data.
3.	Complexity and similarity: The extracted sentences become harder w.r.t training epochs (measured by Gunning Fog Index). The presence of homographs becomes weaker and weaker.

** Details **
1.	The analysis is solid, but the findings are in general not quite surprising to readers. Besides, how should we leverage the findings in the paper?
2.	For the results in Table 3, what if we use all the data discovered by the initial, middle and end epochs (which might be duplicated) instead of ``unique’’ data?
3.     Can you show more statistics about data extraction time, training time?
4.     The relation with [ref1] should be discussed.

[ref1] Machine Translation with Weakly Paired Documents, https://openreview.net/pdf?id=ryza73R9tQ
review_title: Official Blind Review #2

Meta-review: This paper presents a method for curriculum learning based on extracting parallel sentences from comparable corpora (wikipedia), and continuously retraining the model based on these examples. Two reviewers pointed out that the initial version of the paper lacked references and baselines from methods of mining parallel sentences from comparable corpora such as Wikipedia. The authors have responded at length and included some of the requested baseline results. This changed one reviewer's score but has not tipped the balance strongly enough for considering this for publication. 
Decision: Reject