Self-Induced Curriculum Learning in Neural Machine Translation
Authors: Dana Ruiter
Cristina España-Bonet
Josef van Genabith
Venue:  (2020)
Keywords: curriculum learning
neural machine translation
self-supervised learning

Abstract: Self-supervised neural machine translation (SS-NMT) learns how to extract/select suitable training data from comparable (rather than parallel) corpora and how to translate, in a way that the two tasks support each other in a virtuous circle.  SS-NMT has been shown to be competitive with state-of-the-art unsupervised NMT. In this study we provide an in-depth analysis of the sampling choices the SS-NMT model takes during training. We show that, without it having been told to do so, the model selects samples of increasing (i) complexity and (ii) task-relevance in combination with (iii) a denoising curriculum. We observe that the dynamics of the mutual-supervision of both system internal representation types is vital for the extraction and hence translation performance. We show that in terms of the human Gunning-Fog Readability index (GF), SS-NMT starts by extracting and learning from Wikipedia data suitable for high school (GF=10--11) and quickly moves towards content suitable for first year undergraduate students (GF=13).

Under review as a conference paper at ICLR 2020
SELF-INDUCED CURRICULUM LEARNING IN NEURAL MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised neural machine translation (SS-NMT) learns how to extract/select suitable training data from comparable rather than parallel- corpora and how to translate, in way that the two tasks support each other in virtuous circle. SS- NMT has been shown to be competitive with state-of-the-art unsupervised NMT. In this study we provide an in-depth analysis of the sampling choices the SS-NMT model takes during training. We show that, without it having been told to do so, the model selects samples of increasing (i) complexity and (ii) task-relevance in combination with (iii) denoising curriculum. We observe that the dynamics of the mutual-supervision of both system internal representation types is vital for the extraction and hence translation performance. We show that in terms of the human Gunning-Fog Readability index (GF), SS-NMT starts by extracting and learning from Wikipedia data suitable for high school (GF=10-11) and quickly moves towards content suitable for first year undergraduate students (GF=13).
INTRODUCTION
Human Iearners, when faced with a new task, generally focus on simple examples before applying their gained knowledge more complex instances. This approach to learning based on sampling from a curriculum of increasing complexity has also been shown to be beneficial for machines and has been named curriculum learning (CL) (Bengio et al., (2009) by the machine learning community. Previous research on curriculum learning has focused on selecting the best distribution of data i.e order, difficulty and closeness to the final task- to train a system. In this approach, data prepared for the system to ease the learning task. In this work, we follow complementary approach: we design a system that selects by itself the data to be trained on, and we analyse the selected distribution of data -order, difficulty and closeness to the final task without imposing it beforehand. Our method resembles self-paced learning (SPL) (Kumar et al. 2010), in that it uses the emerging model hypothesis to select samples online that fit into its space opposed to most curriculum learning approaches that rely on judgements by the target hypothesis, i.c. an external teacher (Hacohen & Weinshall 2019) to design the curriculum.
The task explored in our work is machine translation (MT). In particular, we focus self-supervised machine translation (SS-NMT) (Ruiter et al.) (019). which exploits the internal representations of an emergent neural machine translation (NMT) system to select useful data for training, where each selection decision is dependent on the current state of the model. Self-supervised learning (Raina) et 2007 Bengio et al. 2013) involves primary task (PT), for which labelled data is not available, and an auxiliary task (AT) that enables the PT to learned by exploiting supervisory signals within the data. In the case of SS-NMT, both tasks -data extraction and learning NMT- enable and enhance each other, such that this mutual supervision leads to a self-induced curriculum, which is the subject to our analysis.
Curriculum learning has been widely studied. In Section 2 we describe the related work on CL, especially focusing on MT. Section introddces the main aspects of self-supervised neural machine translation. Here, we analyse the quality of both the primary and the auxiliary tasks. This is followed by a detailed study of the self-induced curriculum Section 4 where we analyse the characteristics of the distribution of training data obtained in the auxiliary task of the system. Finally, we draw our conclusion in Section 5

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
In recent years, machine translation (MT) has experienced major improvements in translation qual- ity by the introduction of neural architectures including RNNs (Cho et al., 2014 Bahdanau et al. 2014) and transformers (Vaswani et al.] 2017). However, these rely on the availability of large amounts of parallel data. To overcome the need for labelled data, unsupervised neural machine trans- lation (ample et al. [0 Artetxe et al. 2018b Yang et al., 2018) focuses on the exploitation of vast amounts of monolingual data by combining denoising autoencoders with back-translation and multilingual encoders. Further combining these with phrase tables from statistical machine transla- tion (SMT) leads to impressive results (Lample et al. 20188) Artetxe et al.) 201 Sa; Ren et al. 2019] these still rely on hundreds of millions of monolingual sen-
Artetxe et al. 2019). Nevertheless, tences - which are not easy to come by for most languages alternative methods focusing on the exploitation of smaller amounts of comparable corpora have recently been introduced in the form of self-supervised NMT (Ruiter et al., 2019). Here, the internal representations of an emergent NMT system are used to identify useful sentences in aligned documents. As the selection is dependent on the current state of the model, it resembles type of self- self-paced learning Kumar al. (2010). Also Wu et al. 2019) exploit comparable corpora and include similar sentences in NMT training. A comparison between both approaches is given in Section 3.1
The data selection in SS-NMT is directly related to curriculum learning, which, in its essence, is the idea of presenting training samples in meaningful order to benefit the learning, e.g. in the form of faster convergence or better performance (Bengio et al., 2009). Inspired by human learners, Elman (193)) argue that a neural network's optimization can be accelerated by providing samples in order of increasing complexity. While sample difficulty an intuitive measure on which base a learning schedule on, a variety of curricula focus on other metrics such as task-relevance or noise.
Up to now, curriculum learning in NMT has had strong focus the relevance of training samples to a given translation task, e.g. domain adaptation, where the task is the optimization for a specific domain. (van der Wees et al. (217)) train on increasingly relevant samples by the gradual exclusion of irrelevant ones. In line with the basic hypotheses of CL, they observed an increase BLEU over a static NMT baseline and significant speed-up in training as the data size is incrementally re- duced. Reversely, /Zhang et al. (2099) adapt an NMT model domain-distant (difficult) samples. This contradictory behavior of henefiting from both increasingly difficult Noomain-distant and casy (domain-relevam) samples has been analyzed by Weinshall et al.) 2018). showing that the initial phases of training benefit from easy samples w.r.t the target hy- pothesis. while also being boosted (Freund & Schapire| 1996) by samples that are difficult w.r.t. the current hypothesis (Hacohen Weinshall 2019). Such boosting methods. where the model focuses longer on difficult samples, have been applied to NMT by ZZang et al. (2017). In a more complex setup, Wang et al. (2019b) adapt an NMT model to several domains by introducing samples with increasing relevance to all selected domains. In Waan et al. (2019a) both domain-relevance and NMT denoising are combined into single curriculum.
domain by introducing increasingly
The denoising curriculum for NMT proposed by Wang et al. 2018 is quite related to our approach in that they also apply an online data selection approach to build the curriculum based on the current hypothesis of the model. However, the noise scores for the dataset at each training step depend on fine-tuning the model on a small selection of clean data, which comes with high computational cost. To alleviate this cost, Kumar et al. (2019) use reinforcement learning the pre-scored noisy corpus to jointly learn the denoising curriculum with NMT. In Section 3.2 we show that our model exploits its self-supervised nature perform denoising by selecting parallll pairs with increasing accuracy without the need of additional noise metrics.
Difficulty- curricula for NMT that take into account sentence length and vocabulary frequency have been shown to improve translation quality when samples are presented in increasing complexity Koemi & Bojar (2017). Platanios et al. (0119) link the introduction of difficult samples with the NMT models' competence. Other difficulty
have been explored extensively in [Zhagg et al. (2018), showing that they, too, can speed-up uup training without a loss in translation performance.
As a by-product, SS-NMT extracts cross-lingual similar sentence pairs, so the extractions can be compared to parallel data mining systems where strictly parallel sentences are expected. Early approaches exploited structural information of web-pages (Resnik & Smith 2003) or Wikipedia (Adafre & de Rijke| (006)). Others relied on statistical measures such as cross lingu-lingual information retrieval (Utiyama & Isahara (2003), maximum entropy classifiers (Munteanu & Marcu, (205). con-
ditional random fields (Smith al. 2010). SMT or combination of methods (Fung & Cheung
2

---

Under review as a conference paper at ICLR 2020
L1 (en)
L2 (fr/de/ee))
# Sents (M)
# Tokens (M)
Ø
# Sents (M)
# T'Teees (M)
Ø
WP
117/42
2693/1205
28
38/25
644/710
16
WPende
117/37
2693/987
29
51/30
1081/742
24
WPenes 117/35
2693/937
32
27/20
691/572
17
enfr
1+6
25+80
28
1+3
27+87
16
EPende
1+9
25+180
29
1+7
26+192
24
EPenes
1+7
24+84
32
144
26+91
17
Table 1: Number of sentences and tokens for all corpora used. For WP, report the sizes for both the monolingual/comparabe editions. Sizes for ceeudo-comparable EP corpora are reported for both truc+false splits. Average sentences per article (2) is given for all comparable corpora.
2004 Yasuda & Sumita 2008 Abdul-Rauf & Schwenk 20099 CaarónnCedeenn et al. 2015). Nowa-
days, sentence representations obtained from NMT systems or devoted architectures are achieving new state of the art on parallel sentence extraction and filtering, see for instance (España- Bonnctt-Bontte et al, 2017 Grégoire & Langlais 2018) [Schwenk] (2018) Bouamor & Sajjad [2018] Artetxe & Schwenk, 2019a Hangya & Fraser 2019 2019]Chaudhan Chaudhary et al. 2019). Due to multilingual aspect, we consider
the state-of-the-an method used in Schhenk et al. (2019) as a comparison point (see Section 3.2)
3 SELF-SUPERVISED NEURAL MACHINE TRANSLATION, SS-NMT
SS-NMT is a joint data selection and training framework for machine translation, which was origi- nally introduced in Ruiter et al.] (2019). It enables the learning of NMT on comparable rather than parallel data; where comparable data collection of multilingual topic-aligned documents. The basic architecture is a standard NMT system that uses the semantic information encoded in the inter- nal representations of the network to determine at training time if input sentence pair is parallel enough or not, and therefore whether should be used for training or not. The selection online, so, the more the semantic representations improve during training, the more truly parallel sentence pairs selected. Because of this, the nature of the selected pairs evolves during training. and this evolution is what we analyze as induced curriculum learning in Section 4 SS-NMT can be applied to different NMT architectures. |Riter al. (2019) showed good performance of SS-NMT both for RNN based and transformer neural systems, differing in the internal semantic representa- tions used to select the data. In this work, we focus on transformer architectures as they nowadays reach a higher translation quality (Barrault et al., 2019).
made
Let us assume a bidirectional NMT system {L1, L2) + late simultancously from language L1 into another language L2 and vice-versa with single encoder and a single decoder. This is important in the self-supervised architecture because allows represent the two languages in the same semantic space. The input data to train the system is monolingual corpus of sentences L1 and a monolingual corpus sentences in L2 and the system learns to select the adequate pairs. In order to speed-up the training, we use a comparable corpus such as Wikipedia, where we can safely assume that there are comparable (similar) and parallel sentence pairs in related documents DLI, D12-
where the engine learns to trans-
Given a document pair Dr.,, D1.,, the SS-NMT system encodes each sentence of each document into two fixed-length vectors C.,, and Ch such that
T
T
C
Wis
Ch
ht:
where w, the word embedding at time step and 1 the encoder output. For each the sentence representations referred to as all combinations of sentences 8..1 X812 || E Dr. and 8 Dra are encoded and then scored using the margin- measure by Artetxe & Schwenk (2019b):
sim 81.2)
margin(SLI, 8L2) aVTENN Pt)/2 avT (SL2, Qx)/2'
(2)
3

---

Under review as a conference paper at ICLR 2020
SS-NMT
SOTA
LI-L2
L2-LI
LI-L2
L2-LI
BLEU
TER
METEOR
BLEU
TER
METEOR
BLEU
BLEU
en en-fr fr en-de
29.48
51.94
57.35
27.69
53.39
64.22
45.6/25.1/36.
-124.2/33.5
14.40
69.28
39.95
18.84
62.15
55.13
35.0/17.2/22.5 -/--
-/21.0/2..5
en-es
28.57
52.60
55.63
28.43
54.09
63.86
+
Table 2: BLEU, TER and METHOR scores of the self-supervised NMT systems trained on the three en-(fr, de, es} comparable WPs and tested on NT13/NTI4. We compare the performance with current State-of-the-art (SOTA) systems in supervised NMT (Edunov et al.) / unsupervvsed NMT
[ampple al. 2018b) unsupervised NMT+SMT (Artetxe 2019).
where aVTENN (X, Y)) corresponds to the average similarity between a sentence and (NN(X), its k nearest neighbors Y2 in the other language:
si
aVIKNN (x,
E
(3)
k
YERNN(X)
What follows is a selection process, that identifies the top scoring $12 for each 811 and vice- versa. If a pair {SL1: }L2) } is top scoring for both language directions and for both sentence representations, it is accepted without the addition of any hyperparameter or threshold. This is the high precision, medium recall approach in Ruiter et al. (2019) and we use k = as in Artetxe & Schwenk (2019b). Whenever enough pairs have been collected to create a batch, the system trains on it, updating its weights afterwards and proceeding fill the next batch.
3.1 TRANSLATION QUALITY
Experimental Setup We use Wikipedia (WP) as a comparable corpus to train the self-supervised system. We download the English, French, German and Spanish WP dumps] pre-process them and extract the comparable articles per language pair using Wikitailor (Barrón- All articles are normalized, tokenized and truecased using standard Moses (Koehn et al. 2007)
et al. 2015).
scripts. For each language pair, a shared byte-pair encoding (BPE) (Sennrich et al.j 2016) of 100%
merge operations is applied. The number of sentences, tokens and average article length is reported
in Table T For validation we use newstes12012 (NT12) and for testing n (NT13) for
en-es and newstes/2014 (NT14) for en-(Jr, d We use an OpenNMT-based (Klein et al. 2017) implementation of SS-NMT. Our models follow transformer base defined in Vaswani al. (017).
All systems are trained on a single GPU GTX TITAN using a batch size of 50 sentences with maximum sentence length being 50 tokens.
Monolingual embeddings trained using do2ve on the complete WP editions are projected into a common multilingual space via vecmap (Artetxe et al., (017) to attain bilingual embeddings between enn(ffrddddr (d.eees.s). These embeddings initialise the word embeddings (C,,) our system.
Finally, as a control experiment and purely in order to analyse the quality of the data selection auxiliary task, we use the Europarl (EP) corpus (Koehn 2005). The corpus is pre-processed in the same way as WP, and we create synthetic comparable corpus from it as explained in Section 3.2 For these experiments, we use the same corpora for validation and testing as mentioned above.
Automatic Evaluation Translation quality is evaluated by means of three automatic metries: BLEU (Papineni et al.] 2002), TER (Snover et al. 2006) and METEOR (Lavie & Agarwal (007). SS-NMT translation performance training the en- es) comparable Wikipeddaas is reported in Table 2 together with a comparison to the current state-of-the-an (SOTA) in supervised and un- supervised NMT(+SMT). SS-NMT'S is par with the current SOTA in unsupervised NMT, slightly outperforming it by <-3-4 BLEU points in en-fr while having a lower performance on en-de (~3 BLEU). Notice that unsupervised systems such as Lample et al. (2018b) use more than 400M mono- lingual sentences for training while we use an order of magnitude less by exploiting comparable corpora. However, once unsupervised NMT is combined with statistical methods, these outper- form SS-NMT by large margins, i.e. ~6 for en-fr and ~5- BLEU for en-de. In a recent study
Dumps were downloaded January 2019 from https: / htwpp/urppsuuaeo/e////eee wikimedi oorg/
4

---

Under review as a conference paper at ICLR 2020
Steps (1k)
Steps (1k)
16
30
44
58
1
16
30
44
58
72
1.0
Precision
Recall
0.5
0.0
3
Epoch
Epoch
Dual (ac)
Dual (ep)
Cw (ac)
- Cw (ep)
Ch (ac)
Ch (ep)
Figure 1: Accumulated (ac) and epoch-wise (ep) precision and recall on the en-fr EP-based control
experiments. A non-linearity on training steps per epoch can be observed in the ar-axis.
by Wu al. (2099), online data extraction from comparable documents has been combined with unsupervised methods for NMT, which allows for the additional exploitation of large amounts of monolingual data. They achieve similar BLEU scores SS-NMT for en2es (28.1) and es2en (27.6) on NT13; notably SS-NMT trains on comparable data only.
3.2 DATA EXTRACTION QUALITY
Experimental Setup To get an idea of the extraction performance of an SS-NMT system, we perform control experiments on synthetic comparable corpora, as there is no underlying ground truth Wikipedia. For these purposes, we use the en-(fr,de,es) versions of EP. After setting aside 1M parallel pairs as true samples to evaluate the extraction performance on, all remaining pairs scrambled create negative samples (false). In order keep the synthetic comparable corpora close to the statistics of the original comparable WPs, we control the true:false sentence pair ratio and mimic what we observe in our extractions from WP. To estimate it, we start with the assumption that all accepted sentences are true (parallel) examples, and that the false examples (non- (non-parallel) can be obtained as the total number of sentences minus the accepted ones. With this, we expect base true:false ratios of 1:4 for en-(fr,es) and 1:8 for en-de. The negative samples are oversampled in order to meet this ratio given that there are 1M true samples. Further, we calculate the average article length of the comparable WPs and split the mixed corpus of true and false samples into articles. The detailed statistics of the synthetic pseudo-comparable EPs are reported in Table We extract and train the SS-NMT system these synthetic corpora after having initialized each model with the same bilingual embeddings as before.
Automatic Evaluation The extracted pairs from EP at each epoch are compared to the 1M ground truth pairs to calculate the epoch-wise extraction precision (P) and recall (R). Further, we also take the concatenation of all extracted sentences up certain epoch in training in order report the accumulated and R. As we are interested in the final extraction decision based on the votes of both representations C.w and C₁, (dual), but also in the decisions of each single representation (C.,. C,), report the performance for all three representation combinations on EP cnfr in Figure T Similar curves can be observed for EPendde and EP EPennss which are considered in the discussion below.
At the beginning of training, the extraction precision of each representation itself is rather low with Pe[0.45,0.66) for C. and PE[O.14, 14,0.40] for Ch- The fact that C... is initialized using pre- trained embeddings, while C₁ is not, leads to this large difference in initial precision between the two. However, both representations are combined via their intersections, the final decision of the model is high precision already at the beginning of training with values around ~ 0.78 - 0.87. As training progresses and the internal representations are adapted the task, the precision of C2. is greatly improved, leading to an overall high precision extraction which converges at ~ 0.96 - 0.99. This development of extracting parallel pairs with increasing precision be viewed type of denoising curriculum as described by Wang et al. (2018).
The recall of the model, being bounded by the performance of the weakest representation, is very low at the beginning of training (Re [0.03.0.04) due to the lack of task knowledge in C₁, However, as training progresses and C₁ improves, the accumulated extraction recall of the model rises to high values of 0.95 0.98. Interestingly, the epoch-wise recall is much lower than the accumulated,
5

---

Under review as a conference paper at ICLR 2020
en-fr
en-de
en-es
#Pairs
en22fr
fr
#Pairs
em2dde
de2en
#Pairs
en2es
e82em
NMT inst
2.14M
21.80
21.07
0.32M
3.44
4.88
2.51M
26.98
28.85
NMT mid
3.14M
28.91
26.61
1.13M
11.18
15.70
3.96M
28.33
27.92
NMT
3.17M
28.80
26.48
1.18M
11.89
15.91
3,99M
28.34
28.09
NMT aul
5.38M
26.79
25.24
2.21M
11.63
15.52
5.41M
27.85
27.62
SS-NMT
5.38M
29.48
27.69
2.21M
14.40
18.84
5.41M
28.57
28.43
Wikimatrix
2.76M
33.50
30.12
1.57M
13.61
12.17
3.38M
29.60
28.30
Table 3: BLEU scores of a supervised NMT system trained on Wikimatrix as well as the unique pairs collected by SS-NMT in the first (NMT mit)) intermediate (NMTTia)) final (NMT end) and all (NMT epochs training. Evaluation performed on NT14 (en-(fr,de)) and NT13 (en-es) respectively.
which provides evidence for the hypothesis that SS-NMT models visit different relevant samples at different points in training, such that it has visited most of the samples at some point during training. but not at every epoch.
It should be stressed that the successful extraction of increasingly precise pairs in combination with high recall is the result of the dynamics of both internal representations C and C₁ As C₁ less informative at the beginning of training, C guides the final decision to ensure high precision; and as C.r, is high in recall throughout the training, C₁ ensures gentle growth in final recall by setting a good lower bound. The intersection of both ensures that errors committed by one have a chance of being caught by the other; a mutual supervision between representations.
Comparison We compare on the en - {fr, de, es} corpora provided by Wikimatrix (Schwenk al. 2019), which we pre-process as described in 3.1 As these consist of collection of mined sentence pairs together with their similarity scores, a manual threshold 0 needs to be set to extract pairs. We the extraction script using 0 = 1.04, which has been suggested to be a good choice for most language pairs We then use the resulting corpus to train a supervised NMT system using the same specifications as in 3.1
The results can be viewed in table outperforms SS-NMT by 3-4 BLEU points, while the opposite is the case for en-de, where SS- NMT achieves 1-5 BLEU points more. For en-es, both approaches perform similarly. The variable performance of the two approaches may be due to the varying appropriateness of the extraction threshold 0 in Wikimatrix. This yields the problem that for each language and corpus, new optimal threshold needs to be found; a problem that SS-NMT overcomes by its use of two representation types that compliment each other during extraction without the need of manually set threshold.
3 For en-fr, the supervised system trained on Wikimatrix
4
SELF-INDUCED CURRICULA
4.1 ORDER & CLOSENESS TO THE TRANSLATION TASK
As first indicator of the existence of a preferred choice in the order of the extracted sentence pairs, we compare the performance of SS-NMT with different supervised NMT models trained on the data extracted by SS-NMT at different points in training. We consider the first (NMT intermediate (NMT and final (NMT epochs of training as well as the concatenation of all unique pairs (. We then train four supervised NMT models on this data. The difference in the translation quality one can obtain using only the data selected at different epochs illustrates the closeness of these data to the final translation task: we expect data extracted towards the end of the SS-NMT training to include more sentences which are parallel as demanded by a translation task and therefore to achieve higher translation quality.
For each language pair and system, Table 3 shows the number of sentence pairs used for training and the achieved BLEU score. The SS-NMT training outperforms all the supervised versions across all tested languages. Notably, the performance is 1-3 BLEU points above the supervised system trained

---

Under review as a conference paper at ICLR 2020
1.12
12.0
Average Similarity
1.10
Perplexity
11.5
1.08
1.06
11.0
1.04
1.02
10
20
5
10
15
Steps (1k)
Steps (10k)
en (enfr)
en (ende)
en (enes)
enfr
ende
enes
Figure 2: Perplexities on the English data extracted by SS-NMT (left) and average similarity scores of the accepted pairs (right).
Gunning Fog Index
13
Homographs (%)
0,40
0.35
0.30
11
0.25
10
20 Steps (1k)
10
20
30
40
Steps (1k)
..... en (enfr)
. en (ende)
vake en (enes)
enfr
ende
enes
Gunning Fog Index (left) and percentage of homographs (right) of extracted English data seen during the first 40 k steps training.
Figure 3:
on all extracted data, despite the fact that the SS-NMT system uses a much smaller amount of data in its first epochs. This suggests that the SS-NMT system is able to exclude previously accepted false positives at later epochs, while training directly on the entirety of its extracted data leads recurring visitation of the same erroneous samples. Further, the quality and quantity of the extracted data grows training continues for all languages analysed. This can be observed by the fact that the concatenation of the data extracted across epochs (NMT, is always outperformed by the last and thus largest epoch (NMT despite being much larger in size.
second indicator of the closeness of the curriculum to the final task is of course the similarity between the selected sentence pairs during training. In our case, we estimate similarity between pairs by their margin-han scores (Eq. P during training. At the beginning of training, the average similarity between extracted pairs is comparatively low, but it quickly rises within the first 100 training steps to values close to mar ginnii1~1.077 and margin~1.12 depending on the language pair. This evolution is depicted in Figure (right). The increase in the mean similarity of the accepted pairs provides evidence for the base hypothesis that internal representations of translations grow closer in the cross-lingual space, and the system able to exploit this fact by extracting increasingly pairs
similar and accurate seen in Section 3.2
4.2 ORDER & COMPLEXITY
Establishing the complexity of sentence is a complex task by itself. Complexity can estimated by the loss of an instance with respect to the gold or target. In our self-supervised approach, there is no target for the sentence extraction task, we try to infer complexity by other means.
First, we study the behaviour of the average perplexity throughout training. Perplexities of the extracted data estimated using language model trained with KenLM (Heafield 2011) the

---

Under review as a conference paper at ICLR 2020
Gunning Fog Index
Gunning Fog Index
20
10
10
de
fr
2
5 Epoch
10
Language
Figure 4: Kernel density estimated Gunning Fog distributions and box plots over the monolingual Wikipedias (left) and extracted en (en-de sentences different points in training (right). For comparison, the distribution over monolingual WP is included in the latter case.
monolingual WPs for the four languages in our study. We observe the same behaviour in the four cases which can be illustrated by the English curves plotted in Figure (left). Perplexity drops heav- ily within the first 10 l steps for all languages and models. This indicates that the data extracted in the first epoch includes more outliers, and the distribution of extracted sentences moves closer to the average observed in the monolingual WPs training advances. The larger number of outliers the beginning training can be attributed to the larger number of homographs and short sentences (see Figure 3 (right)) at the beginning of training, leading skewed distribution of selected sentences.
The presence of homographs vital for the supervupe system in initialization phase. At the beginning of training, only word embeddings, and therefore C... are initialized with pre-trained data, while C₁ randomly initialized. Thus, words that have the same index in the vocabulary homographs- play an important role in identifying similar sentences using Ch, making up around 1/3 of all tokens observed in the first epoch. As training progresses, and both Cw and C₁ are adapted to the training data, the prevalence of homographs drops and the extraction is now less dependent on shared vocabulary.
Finally, we analyze the complexity of the sentences that a SS-NMT system selects at different points of training by measuring their readability. For this, we apply a modified version of the Gunning Fog Index (GF) (uunning 1952), which is measure of the years of schooling needed to understand a written text given the complexity of its sentences and vocabulary. It defined as:
"10 GF = 0.4 [(
100 (-
(4)
where w and s are the number of words and sentences in text. e is the number of complex words, which are defined as words containing more than 2 syllables. The original formula excluded sev- eral linguistic phenomena from the complex word definition such compound words, inflectional suffixes or familiar jargon; we do not apply all the language-dependen linguistic analysis.
Since our training data is build up with Wikipedia articles, the diversity in the complexity of the sentences is limited to the range of complexities observed in Wikipedia. Figure 4 (left) shows the Gunning Fog distributions over the monolingual WPs. We plot the probability density function for the Gunning Fox Index for the four WP editions estimated via a kernel density estimation. We observe that each distribution is made up of two overlapping distributions; one with low sentence complexity containing article titles and headers, and one with higher average complexity and larger standard deviation containing content sentences.
To study the behaviour during training, compare the Gunning Fog distributions of the English data extracted at the beginning, middle and end of SS-NMTend, with that of the original mono WP e In the extracted data, we observe that the overlapping distributions are less pronounced and that there no trail of highly complex sentences. This due (i) the pre-processing of the input data, which removes sentences containing less than 6 tokens, thus removing most WP titles and short sentences, and (ii) the length accepted in our batches, which is constrained 50 tokens per sentence, removing highly complex strings. Apart from this, the distributions in the middle and the end of training come close the underlying one, but observe large shift towards very
8

---

Under review as a conference paper at ICLR 2020
simple sentences in the first epoch. This shows that the system extracts mostly simple content the beginning training, but soon moves towards complex sentences that were previously not yet identifiable as parallel.
more detailed evolution is depicted in Figure (left). We accumulate extracted sentences within the period of l steps each and report their GF scores. Here we observe how the complexity of the sentences extracted rises strongly within the first 20 steps of training. For English, most models start with text that is deemed to be suitable for high school students (grade 10-11) and quickly turns to more complex sentences suited for undergraduate students in their first year (~13 years of schooling). The mean of the full set of sentences in the English Wikipedia is of ~12 which corresponds to high school senior. For all other languages, similar trend of growing sentence complexity can be observed.
5 SUMMARY AND CONCLUSIONS
This paper explores self-superviied NMT systems where learning the MT model done simulta- neously with the selection of parallel sentences. This association makes the system define its own curriculum. We observe that the dynamics of mutual-supervision of both system's 's internal repre- sentations, Cn and Ch. is imperative to the high recall and precision extraction of SS-NMT. Their combination for data selection over time resembles a denoising curriculum architecture in that the percentage of unprecise pairs -i... non-translations- decreases from 18% to 2%, with an espe- cially fast descend at the beginning of training.
Even if the quality of the extraction increases with time, the lower-similarity sentences used at the beginning of the training still relevant for the translation engine. We analyze the translation quality of a supervised NMT system trained on the extracted data at each epoch of the SS-NMT and observe a continuous increase in BLEU which cannot only be accounted for by the varying numbers of extracted pairs per epoch. Analogously, we also analyze the similarity scores of extracted sentences and see that they also increase over time. As extracted pairs are increasingly similar and precise argued before- the extraction itself resembles secondary curriculum of growing task-relevance, where the task at hand is NMT learning with parallel sentences.
A tertiary curriculum of increased sample complexity could be observed via an analysis of the ex- tracted data's 's Gunning Fog indexes. Here, the system starts with sentences suitable for high school students (GF=10-11) and quickly moves towards content suitable for first year undergraduate stu- dents (GF=13); an overachiever indeed as the norm over the complete WP is around GF=12.
Lastly, by estimating the perplexity with an external language model trained on WP, we observe a steep decrease at the beginning of training with fast convergence. This indicates that the extracted data quickly starts to resemble the underlying distribution of all WP data, with a larger amount of outliers at the beginning. These outliers can be accounted for by the importance of homographs at the beginning of training. This raises the question of how SS-NMT will perform on really distant languages, which is something that will need to be examined in our future work.
REFERENCES
Sadaf Abdul-Rauf and Holger Schwenk. On the use of comparable corpora to improve SMT perfor- mance. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pp. 16 16-23, Athens, Greece, March 2009. Association for Computational Linguistics.
Sisay Fissaha Adafre and Maarten de Rijke. Finding similar sentences across multiple languages in Wikipedia. In Proceedings of the Workshop NEW TEXT Wikis and blogs and other dynamic text sources, 2006. URL/https: wttp:/wwwwwwww....//.//w/w.w.tta aclweb org/ hogghoothhooyyyoooooooooooooooot 2006//06253220 2810
Mikel Artetxe and Holger Schwenk. Margin-based parallel corpus mining with multilingual sen- tence embeddings. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pp. 3197-3203, Florence, Italy, July 2019a. Association for Computational Linguisties.
Mikel Artetxe and Holger Schwenk. Margin- based parallel corpus mining with multilingual sen- tence embeddings. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp- 3197-3203, 2019b.

---

Under review as a conference paper at ICLR 2020
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 451-462, 2017.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3632 1632-3642, Brussels, Belgium, November 201 8a. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, Encko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In Proceedings of the Sixth International Conference on Learming Representations, ICLR, April 2018b.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. An effective approach to unsupervised machine translation. August 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, September 2014.
Loc Barrault, Ondej Bojar, Marta R. Costa-juss, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Miler, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on ma- chine translation (wmt19). In Proceedings of the Fourth Conference on Machine Translation (Volume Shared Task Papers, Day 1). pp. 1-6,, Florence, Italy, August 2019. Association for Computational Linguisties.
Alberto Barrón-Cedeño, Cristina España-Bonet, Josu Boldoba, and Lluís Mårquez. A Factory of Comparable Corpora from Wikipedia. In Proceedings of the Eighth Workshop on Building and Using Comparable Corpora, pp. 3-13, 2015.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41 -88 New York, NY, USA, 2009. ACM.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new August 2013.
perspectives. IEEE Trans. Pattern Anal. Mach. Intell.,
Houda Bouamor and Hassan Sajjad. H2@bucc18: Parallel sentence extraction from comparable cor- pora using multilingual sentence embeddings. In Reinhard Rapp. Pierre Zweigenbaum, and Serge Sharoff (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France, may 2018. European Language Resources Association (ELRA).
Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. Low- resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pp. 261-266, Flo- rence, Italy, August 2019, Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations Using RNN Encoder- Decoder for Statistical Machine Translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), pp. 1724-1734, Doha, Qatar, October 2014. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 489-500, Brussels, Belgium. Association for Computational Linguistics.
Jeffrey L. Elman. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71 99, 1993.
Cristina España-Bonet, Adám Csaba Varga, Alberto Barrón-Cedefio, and Josef van Genabith. An empirical analysis of NMTT-derived interlingual embeddings and their use in parallel sentence identification. IEEE Journal of Selected Topics in Signal Processing, 11(8):1340-13555 2017.
10

---

Under review as a conference paper at ICLR 2020
Yoav Freund and Robert E. Schapire. Experiments with new boosting algorithm. In Proceedings of the Thirteenth International Conference on Internaiional Conference on Machine Learning. pp. 148-156, San Francisco, CA, USA, 1996. Morgan Kaufmann Publishers Inc.
Pascale Fung and Percy Cheung. Mining very-non very-non-parallel corpora: Parallel sentence and lexicon extraction via bootstrapping and em. In Proceedings of the 2004 Conference Empirical Meth- ods Natural Language Processing, pp. 57-63, Barcelona, Spain, July 2004. Association for Computational Linguistics.
Francis Grégoire and Philippe Langlais. Extracting parallel sentences with bidirectional recurrent neural networks to improve machine translation. In Proceedings of the 27th International Confer- ence on Computational Linguisties, pp. 1442-1453, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.
Robert Gunning. The Technique of Clear Writing. McGraw-Hill, 1952.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep net- works. arXiv preprint arXiv: 1904.036266 2019.
Viktor Hangya and Alexander Fraser. Unsupervised parallel sentence extraction with parallel seg- ment detection helps machine translation. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pp. 1224-1234, Florence, Italy, July 2019. Association for Computational Linguistics.
Kenneth Heafield. Kenlm: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 187-197, Stroudsburg, PA, USA, 2011, Asso- ciation for Computational Linguisties.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. Opennmt: Open- source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstra- tions, pp. 67-72. Association for Computational Linguistics, 2017.
Tom Koemi and Ondrej Bojar. Curriculum learning and minibatch bucketing in neural machine translation. Recent Advances Natural Language Processing (RANLP, 2017.
Philipp Koehn. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit, pp. 79-86, Phuket, Thailand, 2005. AAMT, AAMT.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison- Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bo- jar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp. 177-180. As- sociation for Computational Linguistics, 2007.
Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. Reinforcement learning based curriculum optimization for neural machine translation. In Proceedings of the Anmual Conference of the North American Chapter of the Association for Computational Linguisties (NAACL), 2019.
M. P. Kumar, Benjamin Packer, and Daphne Koller. Self- Self- leff-paced learning for latent variable models. In J. D. Lafferty, C. K. I. Williams, J. Shawe- Shawe-Taylor, S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 119-11997. Curran Associates, Inc., 2010.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In Proceedings of the Sixth International Conference on Learning Representations, ICLR, April 2018a.
Guillaume Lample, Myle Ott, Alexis Comneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pp. 5039-5049. Association for Computational Linguistics, 2018b
11

---

Under review as a conference paper at ICLR 2020
Alon Lavie and Abhaya Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Muchine Translation, pp. 228-222-22 231, Stroudsburg, PA, USA, 2007. Association for Computational Linguisties.
Dragos Stefan Munteanuu and Daniel Marcu. Improving machine translation performance by ex- ploiting non non-parallel corpora. Comput. Linguist., 37(4):777-504, December 2005.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311-318, Stroudsburg, PA, USA, 2002. Association for Compu- tational Linguistics.
Emmanouil Antonios Platanios, Otilia Streteu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. Competence-base based curriculum learning for neural machine translation. arXiv preprint arXiv: 1903.0984 2019.
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y. Ng. Self-taught learning: Transfer learning from unlabeled data. In Proceedings of the 24th International Conference on Machine Learning, pp. 759-766, New York, NY, USA, 2007. ACM.
Shuo Ren, Zhirui Zhang, Shujie Liu, Ming Zhou, and Shuai Ma. Unsupervised neural machine translation with smt as posterior regularization, 2019.
LinguitstsniL,uLggsissiisii Philip Resnik and Noah A. Smith. The web as a parallel corpus. Computational Linguistics, 29(3): 349-380, 2003. doi: 10.1162/089120003322711
URL htttps://wwwwt... hwtp://wwwwwww..bbb.o/...o org/
Dana Ruiter, Cristina España-Bonet, and Josef van Genabith. Self-supervised neural machine trans- lation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- fics, pp. 1828- 1834, Florence, Italy, July 2019. Association for Computational Linguistics.
Holger Schwenk. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pp. 228-234, Melbourne, Australia, July 2018. Association for Computational Linguistics.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. Wiki- matrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv preprint 197757190..0571, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics, (ACI. 2016), Volume 1: Long Papers, pp. 1715-1725, Berlin, Germany, 2016.
Jason R. Smith, Chris Quirk, and Kristina Toutanova. Extracting parallel sentences from comparable corpora using document level alignment. In Human Language Technologies: The 2010 Annual Conference of the North Americun Chapter of the Association for Computational Linguistics, HLT 10. pp. 403-411, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and Ralph Weischedel. A study of translation error rate with targeted human annotation. In In Proceedings of the Association for Machine Transaltion in the Americas (AMTA 2006, 2006.
Masao Utiyama and Hitoshi Isahara. Reliable measures for aligning Japanese-Engihe news articles and sentences. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pp. 72-79, Sapporo, Japan, July 2003. Association for Computational Linguistics.
Marlies van der Wees, Arianna Bisazza, and Christof Monz. Dynamic data selection for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.. 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser. and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017.
12

---

Under review as a conference paper at ICLR 2020
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference on Machine Translation, pp. 133-143. Association for Computational Linguisties, 2018.
Wei Wang, Isaac Caswell, and Ciprian Chelba. Dynamically composing domain-dats selection with clean-data selection by "co-curricular learning" for neural machine translation. arXiv preprint arXiv:1906.01 000, 2019a.
Wei Wang, Ye Tian, Jiquan Ngiam, Yinfei Yang, Isaac Caswell, and Zarana Parekh. Learning multitask curriculum for neural machine translation. arXiv preprint arXiv:1908 10940, 2019b.
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International Conference on Machine Learning (ICML), volume 36, 2018.
Lijun Wu, Jinhua Zhu, Di He, Fei Gao, Tao QIN, Jianhuang Lai, and Tie- Liu. Machine transla- tion with weakly paired documents. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL) pp. 426644366499 4375, Hong Kong, China, November 2019. Association for Computational Linguisties.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Unsupervised neural machine translation with weight sharing. In Proceedings of the 56th Anmual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 46-55. Association for Computational Linguis- tics, 2018
Keiji Yasuda and Eiichiro Sumita. Method for building sentence-aligned corpus from wikipedia. In AAAI Workshop Wikipedia and Artificial Intelligence (WikiA108), 2008.
Dakun Zhang, Jungi Kim, Josep Crego. and Jean Senellart. Boosting neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Vol- ume 2: Short Papers, pp. 27 271-276, Taipei, Taiwan, 2017. Asian Federation of Natural Language Processing.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna Mar- tindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint arXiv:1811.0073: 2018
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. Curriculum learning for domain adaptation in neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume (Long Papers), June 2019.
13


Reviews:
[{"review_id": "SkxqBn8OYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Paper summary **\nSelf-supervised machine translation (SS-NMT) is a problem where extracting data and training an NMT model are simultaneously conducted. (Ruiter et al., 2019) proposed several rules to select data and train models. This paper analyzes the following aspects of self-supervised machine translation (SS-NMT):\n1.\tData extraction quality: precision and recall increase w.r.t. training iterations.\n2.\tCloseness to translation tasks: For the extracted sentences, as training goes on, the complexity decreases to the average level of potential bilingual corpus; the similarly of extracted sentences becomes closer. The authors also find that a joint process of extracting data and training models outperforms training a model with the extracted data.\n3.\tComplexity and similarity: The extracted sentences become harder w.r.t training epochs (measured by Gunning Fog Index). The presence of homographs becomes weaker and weaker.\n\n** Details **\n1.\tThe analysis is solid, but the findings are in general not quite surprising to readers. Besides, how should we leverage the findings in the paper?\n2.\tFor the results in Table 3, what if we use all the data discovered by the initial, middle and end epochs (which might be duplicated) instead of ``unique’’ data?\n3.     Can you show more statistics about data extraction time, training time?\n4.     The relation with [ref1] should be discussed.\n\n[ref1] Machine Translation with Weakly Paired Documents, https://openreview.net/pdf?id=ryza73R9tQ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylrgUNXiS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method for training self-supervised neural machine translation systems from a document-aligned comparable corpus (Wikipedia in en, fr, de and es).\n\nThe proposed training method consists of two concurrent processes: a pseudo-parallel sentence pair extraction process, where average word embeddings and encoder states are used to construct sentence embeddings which are compared to extract candidate sentence pairs, and a conventional model optimization process that uses online batches of the extracted sentence pairs as training data.\n\nExperimental results on automatically evaluated translation quality on standard test sets are reported, in addition to parallel sentence extraction quality evaluated on the Europarl corpus and additional analyses on the self-induced curriculum resulting from the training process.\n\nThe proposed methodology is solid. The main issue with the paper is the lack of proper baseline comparison. The authors compare only with supervised and unsupervised systems trained on different corpora, and not with other approaches based on pseudo-parallel data extraction from Wikipedia.\n\nEDIT:\n\nI have increased my score based on the author's response.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bygx092pFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to extract/select suitable training data from comparable —rather than parallel— corpora. The idea sounds reasonable.\n\nMy major concern is about the evaluation: it didn't compare with any existing work. Actually there quite a few papers  on mining parallel sentences from comparable corpora such as Wikipedia, as shown below. Seems the authors are not aware of those works and didn't review and compare with them. Without such comparisons, it is difficult to judge the effectiveness of the proposed method and the quality of this work. \n[1] Finding similar sentences across multiple languages in Wikipedia, Proceedings of the Workshop on NEW TEXT Wikis and blogs and other dynamic text sources. 2006.\n[2] Method for building sentence-aligned corpus from wikipedia, 2008 AAAI Workshop on Wikipedia and Artificial Intelligence (WikiAI08). 2008.\n[3] Extracting parallel sentences from comparable corpora using document-level alignment, Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2010.\n[4] \"Improving machine translation performance by exploiting non-parallel corpora.\" Computational Linguistics2006.\n[5] https://www.aclweb.org/anthology/W04-3208.pdf\n[6] https://openreview.net/pdf?id=ryza73R9tQ\n\nMinor issues:\n\t1. \"For each language pair, a shared byte-pair encoding (BPE) (Sennrich et al., 2016) of 100k merge operations is applied.\" Most papers on neural machine translation don't use such a large BPE size, which is likely to lead to better performance. It would be better to use the same setting as previous work for fair comparisons.\n\n\t2. \"In the case of SS-NMT, both tasks —data extraction and learning NMT— enable and enhance each other, such that this mutual supervision leads to a self-induced curriculum, which is the subject to our analysis.\" Similar idea, mutual boosting between data selection and model training, has been explored in the following paper, although not for machine translation. What's the difference between these two papers?\nLearning to Teach, ICLR 2018.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for curriculum learning based on extracting parallel sentences from comparable corpora (wikipedia), and continuously retraining the model based on these examples. Two reviewers pointed out that the initial version of the paper lacked references and baselines from methods of mining parallel sentences from comparable corpora such as Wikipedia. The authors have responded at length and included some of the requested baseline results. This changed one reviewer's score but has not tipped the balance strongly enough for considering this for publication. 
Decision: Reject