Cross-model Back-translated Distillation for Unsupervised Machine Translation
Authors: Phi Xuan Nguyen
Shafiq Joty
Kui Wu
AiTi Aw
Venue:  (2021)
Keywords: unsupervised machine translation
NMT
machine translation

Abstract: Recent unsupervised machine translation (UMT) systems usually employ three main principles: initialization, language modeling and iterative back-translation, though they may apply them differently. Crucially, iterative back-translation and denoising auto-encoding for language modeling provide data diversity to train the UMT systems. However, these diversification processes may have reached their limit. We introduce a novel component to the standard UMT framework called Cross-model Back-translated Distillation (CBD), that is aimed to induce another level of data diversification that existing principles lack. CBD is applicable to all previous UMT approaches. In our experiments, it boosts the performance of the standard UMT methods by 1.5-2.0 BLEU. In particular, in WMT'14 English-French, WMT'16 German-English and English-Romanian, CBD outperforms cross-lingual masked language model (XLM) by 2.3, 2.2 and 1.6 BLEU, respectively. It also yields 1.5-3.3 BLEU improvements in IWSLT English-French and English-German tasks. Through extensive experimental analyses, we show that CBD is effective because it embraces data diversity while other similar variants do not.

Under review as a conference paper at ICLR 2021
CROSS-MODEL BACK-TRANSLATED DISTILLATION FOR UNSUPERVISED MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent unsupervised machine translation (UMT) systems usually employ three main principles: initialization, language modeling and iterative back-translation, though they may apply them differently. Crucially, iterative back-translation and denoising auto-encoding for language modeling provide data diversity train the UMT systems. However, the gains from these diversification processes has seemed plateau. We introduce novel component to the standard UMT framework called Cross-model Back translatdd Distillation (CBD), that is aimed induce another level of data diversification that existing principles lack. CBD is applicable to all previous UMT approaches. In our experiments, it boosts the performance of the standard UMT methods by 1.5-2.0 BLEU. particular, WMT 14 English-French WMT 16 German-Englin and English- English-Romaniann CBD outperforms cross- linsu-l masked language model (XLM) by 2.3, 2.2 and 1.6 BLEU,, respectively. It also yields 1.5-3.3 BLEU improvements in IWSLT English- French and English-Geerna show that CBD is effective because it embraces data diversity while other similar variants do not
tasks. Through extensive experimental analyses, we
INTRODUCTION
Machine translation (MT) is core task in natural language processing that involves both language understanding and generation. Recent neural approaches (Vaswani et al. 2017 Wu et al. 2019) have advanced the state of the art with near human-leve performance (Hassan al. 2018). However, they continue to rely heavily on large parallel data. As result, the search for unsupervised alternatives using only monolingual data has been active. While Ravi & Knight (201T) and Klementiev al.) (2012) proposed various unsupervised techniques for statistical MT (LMp), et al. (2018a c) established a general framework for modern unsupervised MT (UMT) that works for both SMT and neural MT (NMT) models. The framework has three main principles: model initialization, language modeling and iterative tockktranslation. Model initialization bootstraps the model with a knowledge prior like word-level transfer (Lample et al., 2018b). Language modeling, which takes the form of denoising eiooonnn (DAE) in NMT (Lample et al.) 2018c). trains the model to generate plausible sentences language. Meanwhile, iterative back trak-trannlttion (IBT) facilitates cross-lingual translation training by generating noisy source sentences for original target sentences. The recent approaches differ how they apply each of these three principles. For instance, Lample al.) (20T8a) use an unsupervised word- translatiiion model (Lample et al. 2018b) for model initialization, while Conneau & Lample (219) use a pretrained cross-lingual masked language model (XLM).
In this paper, we focus on a different aspect of the UMT framework, namely, its data diversification process. If we look from this view, the DAE and IBT steps of the UMT framework also perform some form of data diversification to train the model. Specifically, the noise model in the DAE process generates new, but noised, versions of the input data, which are used to train the model with a reconstruction objective. Likewise, the IBT step involves the same UMT model to create synthetic parallel pairs (with the source being synthetic), which are then used to train the model. Since the NMT model is updated with DAE and IBT simultancously, the model generates fresh translations in each back-trane teek-ranslation step. Overall, thanks to DAE and IBT, the model gets better at translating
Anonymized code: https:/tinyurl.com/// com/y2ru8res By diversification, mean sentence level variations (not expanding to other topics genres).

---

Under review as a conference paper at ICLR 2021
by iteratively training the newly created and diversified data whose quality also improves over time. This argument also applies to statistical UMT. except for the lack of the DAE (Lample et al.) 2018c). However, conjecture that these diversification methods may have reached their limit as the performance does not improve further the longer we train the UMT models.
In this work, we introduce fourth principle the standard UMT framework: Cross- Back- translated Distillation or CBD $ with the aim to induce another level of diversification that the existing UMT principles lack. CBD initially trains two UMT agents (models) using existing approaches. Then, one of the two agents translates the monolingual data from one language s to another in the first level. In the second level, the generated data are back-translated from to s by the other agent. In the final step, the synthetic parallel data created by the first and second levels are used to distill supervised MT model. CBD applicable to any existing UMT method and is more efficient than ensembling approaches (Freitag et al. 2017) 5.3)
In the experiments ($4) we have evaluated CBD on the WMT' 14 English-French. WMT 16 English-German and WM English-Romanian unsupervised translation tasks. CBD shows consistent improvements of 1.0-2.0 BLEU compared to the baselines these tasks. It also boosts the performance on IWSLT" 14 English-German and IWSLT" 13 English-French tasks significantly. In our analysis, we explain with experiments why other similar variants ($5.1) and other alternatives from the literature ($.44) do not work well and cross moosl back- transsaatnnn is crucial for our method. We further demonstrate that CBD) enhances the haselines by achieving greater diversity as measured by back-translation BLEU ($5.2)
BACKGROUND
Ravi & Knight 201T) were among the first to propose a UMT system by framing the problem as a
decipherment task that considers -on-English text cipher for English. Nonetheless, the method is
limited and may not be applicable to the current well-established NMT systems (Luong et al.] [015 Vaswani al. 2017 Wu al., 2019). Lample et al. (2018)) set the foundation for modern UMT.
They propose to maintain two encoder- languages, and train them via denoising auto-encoding, iterative back-translation and adversarial training. In their follow-up work. Lample et al. (2018c) formulate a common UMT framework for both PBSMT and NMT with three basic principles that can be customized. The three principles are:
networks simultaneously for both source and target
Initialization: A non condrandomize cross- multi-lingual initialization that represents a knowledge prior to bootstrap the UMT model. For instance, Lample et al. (2018a) and Artetxe et al. (2019) use
an unsupervised word-translation model MUSE (Lample et al. (018b) initialization to promote word-to-word cross- -ingu-l transfer. Lample et al. (2018c) use a shared jointly trained sub-word (Sennrich et al., (2016b) dictionary. On the other hand, Conneau Lample (2019) use a pretrained
cross-lingual masked language model (XLM) to initialize the unsupervised NMT model.
Language modeling: Training language model on monolingual data helps the UMT model to generate fluent texts. The neural UMT approaches (Lample et al.] [2018a]e Conneau & Lample 2019) use denoising auto-encoder training to achieve language modeling effects in the neural model. Meanwhile, the PBSMT variant proposed by |ample et al. (2018c) uses the KenL. smoothed n- n-gram language models (Heafield 2011).
Iterative brck-translation: Back-translation (Sennrich et al., 2016a) brings about the bridge between source and target languages by using a backward model that translates data from target to source. The (source and target) monolingual data is translated back and forth iteratively to progress the UMT model in both directions.
During training, the initialization step conducted once, while the denoising and maak-translation steps are often executed in an alternating manner It is worth noting that depending on different implementations, the parameters for backward and forward components may be separate (Lample) et al., 2018a) or shared (Lample et al. 2018c) Conneau & Lample| 2019). A parameter-shared cross-lingual NMT model has the capability to translate from either source or target, while a UMT system with ceaumettr-ssppraate models has to maintain two models. Either way, we deem a standard UMT system to be bidirectional, i.e. is capable of translating from either source or target language.
'The Kenl. M language model in PBSMT Lample al. 2018c was kept fixed during the training process.
2

---

Under review as a conference paper at ICLR 2021
Our proposed cross-model back-translated distillation (CBD) works outside this well-established framework. It employs two UMT agents to create extra diversified data apart from what existing methods already offer, rendering it useful add-on to the general UMT framework. Furthermore, different implementations of UMT' as discussed above can be plugged into the CBD system to achieve performance boost, even for future methods that may potentially not employ the three principles.
3 CROSS- BACK-TRANSLATE DISTILLATION
Let X and X be the sets of monolingual data for languages s and t, respectively. We first train two UMT agents independently with two different parameter sets 0₃ and 02 using existing methods (Lample al., 201 2018a e Conneau & Lample, 2019) Since UMT agent with parameter set is deemed bidirectional in our setup, we denote Yr (- 0) to be the translation sample from language s to & of input sentence Ta using parameters
addition, for each language direction s + (and vice versa for - s). we introduce a supervised MT model with parameters that will be trained through our CBD. With these notations, we denote the following training loss L with respect input variables O.. 0g {01, )2), the supervised parameters 0 and the learning rate r,. Figure also illustrates the translation flows of function L.
L(Oa, 0g. 0) -E,,~ P,, (2) 0 B.). ~x, log P(ys [r,, 0) + log P(ys 2.. 0) Ex - P(... a. (11. log P(rz Us. 0) + log P Wy 0)]
(1)
In the CBD strategy (Algorithm] I Eq. at each step, each agent E {01. 02} samples translations from the monolingual data of both languages and f acquire the first level of synthetic parallel data (Ia.We) and ). In the second level, the other agent {01, 02] \ 0 is used to sample the translation 2x of the translation y/ of I, (and similarly for 2, from the translation ys of 2:).. This process is basically back-translation, but with the backward model coming from a different regime than that of the forward model. The fact that the first level agent must be different from the second level agent is crucial to achieve the desirable level of diversity in data generation. After this, we update the final model i using all the synthetic pairs (x, y) and (y, 2) with maximum likelihood estimation (MLE).
Algorithm 1 Cross-model Back-translated Distillation: Given monolingual data Xg and X, of languages s and return s -> UMT model with parameters A.
X, T2
Ou 8-48
Vt
2 1
8,
0
1,
X2 - It.
y
Ag
21
(Weiee)
θ
1: procedure CBD(s, L)
(uasze)
Train the 1st UMT agent with parameters
3: Train the 2nd UMT agent with parameters 02
Figure 1: Translation flows of loss function L in Eq. [ Variable set (0.. 0z) is replaced with (0,, 02) and (02, 01) iteratively in Algorithm I Parallel pairs (æ,, ye (2.. ye) ve)-(yss are) and (ys- 21) are used to train the final model 0.
4: Randomly initialize the parameters for MT model. 0 while until convergence do
5:
6:
i i
V (01, 0z 0) n Vi 0,
7:
0
0
8: return
0
In this way, the model Ô first gets trained on the translated products - y) of the UMT teachers, making it as capable as the teachers. Secondly, the model 9 is also trained on the second-level data (y - 2), which is slightly different from the first-level data. Thus, this mechanism provides extra data diversification to the system Ô in addition to what the UMT teachers already offer, resulting in our final model outperforming the UMT baselines ($4). However, one may argue that since 0 and 02 are trained in similar fashion, 2 will be the same v. resulting in duplicate pair. In experiments, on the contrary, the back-translated dataset contains only around 14% duplicates across different language pairs, shown in our analysis on data diversity in $5.2 In Appendix I8.1 provide more generalized version of CBD with n (2 2) UMT agents, where we also analyze its effectiveness on the IWSLT translation tasks.
For neural approaches, changing the random seeds would do the trick, while PBSMT methods would need randomize the initial embeddings and/or subsample the training data.
3

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
We present our UMT experiments on the WMT tasks (84.1) followed by IWSLT (4..2)
4.1 WMT EXPERIMENTS
Setup. We use the default data setup and codebase Conneau & Lample (011 code repository Specifically, we use the News Crawl 2007-2008 datasets for English (En), French (Fr) and German (De), and News Crawl 2015 dataset for Romanian (Ro), and limit the total number of sentences per language to 5M. This results in a total 10M, 1OM and 7M sentences of combined monolingual data for En-Fr, En-De and En-Ro, respectively. Note that in their paper, Lample al. (2018c) and Conneau & 1.ample| (2019) conducted their UMT experiments on very large datasets of 274M, 509M and 195M sentences (combination of source and target monolingual data) for En-Fr, En-De and En-Ro respectively, which are derived from News Crawl 2007-2017 datasets. Unfortunately. we were unable reproduce these (very) large scale experiments because we encountered Cud-o--CPU-mmmonn error when attempting to load the large datasets, even though our system has 128GB RAM. Instead, we rerun the NMT and PBSMT models (Lample al. 2018c) and the pretrained XLM (Conneau & Lample 2019) on the default setup to acquire the corresponding NMT, PBSMT and XLM baselines.
For the NMT models, we follow [ample et al. (2018c) train the UMT agents with a parameter- shared Transformer (Vaswani et al.] 2017) that has 6 layers and 512 model dimensions and a batch size of 32 sentences. We use joint Byte-Pair Encoding (BPE) (Sennrich et al.) (01 and train fastText (Bojanowski et al. (017) on the BPE tokens to initialize the token embeddings. For the PBSMT Koehn et al.] 2003) models, following Lample et al. (2018c), we use MUSE (Lample et al. 2018b to generate the initial phrase table and run 4 iterations of back-translatice
We subsample 500K sentences from the 5M monolingual sentences at each iteration to train the PBSMT models different seeds for MUSE training and randomly
To ensure randomness in the PBSMT agents, subsample different sets of data during PBSMT training. For XLM (Conneau & Lample 2019), we also follow their setup. For each agent, we first pretrain big XL. model (6 layers, 1024 model
dimensions) on the default 10M data with 4K tokens per batch. Then we initialize the encoder-decoder system of the UMT agent with the XLM and train the model for 30 epochs The NMT and XLM agents are trained with a 4-GPU setup. We use the big Transformer (Ott et al., 2018) as the final supervised model. We choose the best model based on validation loss and use beam size of
Table 1: BLEU scores on WMT' 14 English-French (En-Fr), WMT' 16 English-German German (En-De) and WMT' 16 English-Rone Romanian (En-Ro) unsupervised translation tasks.
Method Data
En-Fr
Fr-En
En-De
De-En
En-Ro
Ro-En
Results reported from previous papers on large scale datasets
Data Used
274M
274M
509M
509M
195M
195M
NMT (Lample et al. 2018c)
25.1
24.2
17.2
21.0
21.1
19.4
PBSMT (Lample et al. 2018c
27.8
27.2
17.7
22.6
21.3
23.0
XLM
(Conneau & Lample 2019)
33.4
33.3
26.4
34.3
33.3
31.8
Results from our runs on smaller datasets
Data Used
10M
10M
10M
10M
7M
7M
NMT (Lample al.
2018c)
24.7
24.5
14.5
18.2
16.7
16.3
CBD
26.5
25.8
16.6
20.4
18.3
17.7
PBSMT (Lample et al. CBD
2018c
17.1
16.4
10.9
13.6
10.5
11.7
21.6
20.6
15.0
17.7
11.3
14.5
XLM (Conneau & Lample 2019)
33.0
31.5
23.9
29.3
30.6
27.9
CBD
35.3
33.0
26.1
31.5
32.2
29.1
For PBSMT, Lar.M.,,Lammpl... Lample al. (2018c) subsampled 5M out of 193M sentences of monolingual data. For NMT, they train for epochs, which almost 60x longer than with data size taken into account.
4

---

Under review as a conference paper at ICLR 2021
Results. Table shows the experimental results of different UMT approaches with and without CBD. First of all, with the datasets that are 30-50 times smaller than the ones used in previous papers, the baselines perform around 2 to BLEU worse than the reported BLEU. As it can be seen, the CBD-enhanced model with the pretrained XL.M achieves 35.3 and 33.0 BLEU on the WMT' 14 En-Fr and Fr-En tasks respectively, which are 2.3 and 1.5 BLEU improvements over the baseline. It also surpasses Conneau & Lample (2019) by 1.9 BLEU, despite the fact that the SOTA was trained with 274M sentences (compared our setup of 10M sentences). CBD also boosts the scores for XLM in En-De, De-En, En-Ro, Ro-En by around 2.0 BLEU. For the NMT systems, CBD also outperforms the baselines by to 2 BLEU. More interestingly, PBSMT models are known be deterministic, but CBD is still able to improve data diversity and provide performance boost by up to 4.0 BLEU points.
4.2 IWSLT EXPERIMENTS
We also demonstrate the effectiveness of CBD on relatively small datasets for IWSLT En-Fr and En-De translation tasks. The IWSLT' 13 En-Fr dataset contains 200K sentences for each language. We use the (WSLTIS.TED.ss22112 set for validation and the D.SLL15.TED.st2000 The IWSLT" 14 En-De dataset contains 160K sentences for each language. We split it into 95% for training and 5% for validation, and we use IWSLT14,TED dev20010 dev2012, tst2010,tst1011 lst2012] for testing. For these experiments, we use the neural UMT method (Lample et al.. 2018c) with Transformer of 5 layers and 512 model dimensions, and trained using only GPU.
set for testing.
From the results in Table we can see that CBD improves the performance in all the four tasks by 2-3 BLEU compared to the NMT baseline of Lample et al. (2018c).
Table 2: BLEU scores on the unsupervised IWSLT' 13 Boglish-Frenchh (En-Fr) and IWSLT' 14 English-German (En-De) tasks.
Method/ / Data
En-Fr
Fr-En
En-De
De-En
NMT (ample et al. 2018ch + CBD
29.6
30.7
15.8
19.1
31.8
31.9
18.5
21.6
5 UNDERSTANDING CBD
5.1 CROSS-MODEL DACKSSRANNNN IS KEY
As mentioned, erucial to our strategy's 's success is the cross-model back-translation, where the agent operating at the first level must be different from the one in the second level. To verify this, we compare CBD with similar variants that do not employ the cross-model element in the WMT tasks. We refer to these variants commonly as fack-translatio distillation (BD). The first variant BD ( (Ist level, I agent) has only UMT agent that translates the monolingual data only once and uses these synthetic parallel pairs to distill the model Å. The second variant BD (/st level, 2 agents) employs UMT agents, similar to CBD, to produce 2 sets of synthetic parallel data from the monolingual data and uses both of them for distillation. Finally, the third variant BD (2nd level, 2 agents) uses 2 UMT agents to sample translations from the monolingual data in forward and backward directions using the same respective agents. In other words, BD (2nd level, 2 agents) follows similar procedures in Algorithm [ except that it optimizes the following loss with 02 being alternated between 0n and Oz:
I(Oa, -E Pi-le, D. v. 0n Ja log P( 2.., 0) + log P(uel=s. 0)]
(2)
-E Plls Po log P 1 yo. 0) + log P P( 0)]
From the comparison in Table 3 we see that none of the BD variants noticeably improves the performance across the language pairs, while CBD provides consistent gains of 1.0-2.0 BLEU In particular, the BD (Ist level, agent) variant fails to improve as the distilled model is trained on the same synthetic data that the UMT agent is already trained on. The variant BD (1st level, agents) is in fact similar in sprit to (Nguyen et al.) 2020). which improves supervised and semi-supervised MT. However, it fails to do so in the unsupervised setup, due to the lack of supervised agents. The variant BD (2nd level, 2 agents) also fails because the 2nd level synthetic data is already optimized
5

---

Under review as a conference paper at ICLR 2021
during iterative trckstranslttion training of the UMT agents, leaving the distilled model with extra information to exploit. On the other hand, cross-model fack-translation enables CBD to translate the second-level data by agent other than the first-level agent. In this strategy, the second agent produces targets that the first agent not aware of, while the second agent receives as input the sources that are foreign to it. This process creates corrupted but information, which the supervised MT model can leverage to improve the overall MT performance through more data diversity.
Table 3: BLEU comparison of CBD vs. no cross-model variants in the WMT' 14 English-Frev French (En-Fr), WMT" 16 Anglish-German (En-De) and English-Romanian (En-Ro) tasks.
Method
En-Fr
Fr-En
En-De
De-En
En-Ro
Ro-En
NMT Baseline (Lample et al. 2018c)
24.7
24.5
14.5
18.2
16.7
16.3
BD (1st level, agent)
24.5
24.4
13.8
17.6
16.0
15.9
BD (1st level, 2 agents)
24.6
24.6
14.0
17.9
16.5
16.1
BD (2nd level, 2 agents)
24.7
24.7
14.4
18.0
16.7
16.4
CBD (2nd level, agents)
26.5
25.8
16.6
20.4
18.3
17.7
5.2 CBD PRODUCES DIVERSE DATA
Having argued that cross-mis moosl boccktranslltion creates extra information for the supervised MT model leverage on, hypothesize that such extra information be measurable by the diversity of the generated data. To measure this, we compute the reconstruction BLEU and compare the scores for BD and CBD (both 2nd level, 2 agents) in the WMT En-Fr, En-De and En-Ro tasks. The scores are obtained by using the first agent to translate the available monolingual data in language s to and then the second agent to translate those translations back to language s. After that, BLEU score is measured by comparing the reconstructed text with the original text. In BD, the first and second agents are identical, while they are distinct for CBD. From the results in Table we observe that the reconstruction BLEU scores of CBD are more than 10 points lower than those of BD, indicating that newly generated data by CBD are more diverse and different from the original data.
Table 4: Reconstruction BLEU scores of BD and CBD in different languages for the WMT unsupervised translation tasks. Lower BLEU means more diverse.
Method
En-Fr-En
Fr-En-Fr
En-De-En
De-En-De
En-Ro-En
Ro-En-Ro
BD 76.5
72.7
75.1
63.3
73.6
71.2
CBD
63.1
59.5
60.7
50.6
61.0
56.8
we further report the ratio of duplicate source-targ pairs to the amount of synthetic parallel data created by CBD. We sample 30M synthetic parallel data using the CBD strategy and examine the amount of duplicate pairs for the WMT En-Fr, En-De and En-Ro tasks. We can notice that across the language pairs, only around 14% of the parallel data are duplicates. Given that only about 5M (3.5M for En-Ro) sentences are real sentences and the remaining 25M sentences are synthetic, this amount of duplicates is surprisingly low. This fact also explains why CBD is able to exploit extra information better than any standard UMT to improve the performance.
In Table 5
Table 5: Comparison between the amount of real data, generated data by CBD and the duplicates per language pair for the WMT" En-Fr, WMT' 16 En-De and En-Ro unsupervised MT tasks.
Method
En-Fr
En-De
En-Ro
Real data
5M
5M
3.5M
Generated data
30M
30M
29M
Duplicate pairs
4.4M (14.5%)
3.9M(13%)
3.9M (13.4%)

---

Under review as a conference paper at ICLR 2021
5.3 COMPARISON WITH ENSEMBLES OF MODELS AND ENSEMBLE KNOWLEDGE DISTILLATION
Since CBD utilizes outputs from two UMT agents for supervised distillation, it is interesting to see how it performs compared to ensemble of UMT models and ensemble knowledge distillation FFeiiag et al., 2017). To perform ensembling, we average the probabilities of the two UMT agents at each decoding step. For ensemble distillation, we generate synthetic parallel data from an ensemble of UMT agents, which is then used to train the supervised model.
Table 6: BLEU comparison of CBD vs. an ensemble of UMT agents and ensemble knowledge distillation (Freitag et al. 2017) on WMT 14 En-Fr, WMT" 16 En-De and En-Ro translation tasks.
Method
En-Fr
Fr-En
En-De
De-En
En-Ro
Ro-En
NMT Baseline (Lample et al. 2018c)
24.7
24.5
14.5
18.2
16.7
16.3
Ensemble of 2 NMT agents
25.2
24.8
15.3
19.1
17.7
17.1
Ensemble distillation
17.3
20.0
3.5
3.7
1.2
1.1
CBD with NMT
26.5
25.8
16.6
20.4
18.3
17.7
From the results on the WMT translation tasks in Table 6 we observe that ensembles of models improve the performance only by 0.5-1.0 BLEU, while CBD provides larger gains (1.0-2.0 BLEU) across all the tasks. These results demonstrate that CBD is capable of leveraging the potentials of multiple UMT agents better than how an ensemble of agents does. This is in contrast to data diversification (Nguyen et al.) 2020), which is shown to mimic and perform similarly to model ensembling. More importantly, during inference, computations (twice in this case) to store and execute multiple models. In contrast, CBD can throw away the UMT teacher agents after training and needs only one single model for inference. Meanwhile, ensemble knowledge distillation (Freitag et al.) 2017). which performs well with supervised agents, performs poorly in unsupervised MT tasks. The reason could be that the UMT agents may not be suitable for the method originally intended for supervised learning. Further inspection in Appendix $8.2 suggests that many samples the ensemble translations contain incomprehensible repetitions.
ensemble of models requires more memory and
5.4
COMPARISON WITH OTHER POTENTIAL. ALTERNATIVES
In this section, we compare CBD) with other alternatives in the text generation literature that also attempt to increase diversity. While many of these methods are generic, we adopt them in the UMT framework and compare their performance with CBD technique in the WMT En-Fr, Fr-en, En-De, and De-en tasks, taking the XLM (Conneau & Lample 2019) as the base model.
One major group of alternatives is sampling based methods, where the model samples translations following multinomial distributions during iterative back trcck-translatin. Specifically, we compare the
CBD with (i) sampling with temperature 0.3 (Edunov et al.) [0118] Fan et al. 2018). (ii) top-k sampling (Radford et al. 2019), and (iii) nucleus or top-p sampling (Holtzman et al. 2020). Plus, we compare CBD with target noising, where we add random noises to the translations iterative back-translation. Finally, multi-agent dual learning (Wang et al.) 2019) is also considered as another alternative, where multiple unsupervised agents are used train the end supervised model.
the UMT model during
The results are reported in Table 7 We can see that while the sampling based methods indeed increase the diversity significantly, they do not improve the performance as much as CBD does. The reason could be that the extra data generated by (stochastic) sampling are noisy and their quality is not as good as deterministic predictions from the two UMT agents via cross- moosl mack-translation. On the other hand, target noising does not provide a consistent improvement while multi-agent dual learning achieves less impressive gains compared to CBD.
6 RELATED WORK
The first step towards utilizing the vast monolingual data to boost MT quality is through semi-
supervised training. Back- back-translationn (Sennrich et al. 2016a] Edunov et al. 2018) is an effective approach to exploit target-side monolingual data. Dual learning (He et al. [016 Wang et al. 2019). meanwhile, trains backward and forward models concurrently and intertwines them together. Recently, Zheng et al. (2020) proposed variational method to couple the translation and language models
7

---

Under review as a conference paper at ICLR 2021
Table 7: Comparison with other alternatives on the WMT En-Fr, Fr-En, En-De and De-En, with XLM as the base model.
WMT
En-Fr
Fr-En
En-De
De-En
XLM Baseline (Conneau & Lample 2019)
33.0
31.5
23.9
29.3
Sampling with temperature 0.3
33.5
32.2
24.3
30.2
Top-k sampling
33.18
32.26
24.0
29.9
Nucleus (top-p) sampling
Diverge
Target noising
32.8
30.7
24.0
29.6
Multi-agent dual learning
33.5
31.7
24.6
29.9
CBD
35.3
33.0
26.1
31.5
through shared latent space. There have also been attempts in solving low-resource translation problems with limited parallel data (Gu et al. 20188] Irvine & Callison- Burch, 2014 Guzmán et al. 2019). In the realm of SMT, cross-lingual dictronaries have been used to reduce parallel data reliance (Irvine & Callison-Bonn Burch 2016 Klementiev et al. 2012).
In recent years, unsupervised word-translation via cross-lingual word embedding has seen huge success (Lample et al. 2018b; Artetxe et al. 2017 20188). This opened the door for UMT methods that employ the three principles described in $2 Lample et al. (20T8a) and Artetxe et al. (2018c)
among the first of this kind, who use denoising autodecoder for language modeling and iterative transaaannll.... Lample et al. (2018a) use MUSE (Lample et al.) 2018b) as the initialization to bootstrap the model, while Artetxe et al. (20T8c) use cross-lingual word embeddings (Artetxe et al. 2017). [Lample et al. (2018c) later suggested the use of BPE (Sennrich et al.] 2016b) and fest (Bojanowski et al. (2017) to initialize the model and the parameters sharing. Pretrained language models (Devlin et al. 2019) are then used to initialize the entire network (Conneau & Lample
2019). Song et al. (2019) proposed to pretrain an encoder-decoder model while Artttxe et al. (2018b)
suggested a combination of PBSMT and NMT with subword information [ Plus, pretraining BART (Lewis et al.)
2020) on multi-lingual corpora improves the initialization process (Liu et al.) 2020).
Our proposed CBD works outside the three-principle UMT framework and is considered as an add-on to any underlying UMT system. There exist some relevant approaches to CBD. First, similar to Nguyen et (002)). which generates a diverse set of data from multiple supervised MT agents. Despite being effective in supervised and semi-supervised settings, a direct implementation of in UMT underperforms due to lack of supervised signals (S5.Ip. In order to successfully exploit unsupervised agents, CBD requires cross-model back- trasstrranlttion which is the key to its effectiveness. Second, CBD can be viewed
augmentation technique (Fadaee al., 2017] Wang et al.) 2018). Although the denoising autoencoding built in a typical UMT system also performs augmentation, the noising process is rather naive, while CBD augments data by well-trained agents. Third, CBD is related to ensemble knowledge distillation (Kim & Rush 2016 Freitag et al.) 2017). These methods use multiple models to perform ensemble inference to generate one-way synthetic targets from the original source data, which are then used to distill the final model. However, these distillation schemes only apply to supervised settings and the agents are used to jointly produce high quality translations rather than diverse candidates. Similar to our method, multi-agent dual learning (Wang) et al. 2019) also uses multiple models both forward and backward directions, but to minimize the reconstruction losses instead of to generate diverse synthetic data.
CONCLUSION
We have proposed cross-model back-translated distillation (CBD) a method that works outside the three existing principles for unsupervised MT and is applicable to any UMT methods. CBD improves the performance of various underlying UMT approaches in the WMT' 14 English-Frence WMT' 16 English-German and baselines in the IWSLT' 14 German- -nnglish and IWSLT' 13 English-Frende tasks by up to 3.0 BLEU. Our analysis shows that CBD embraces data diversity and extracts more model-specifie intrinsic information than what an ensemble of models would do.
translation tasks by 1.0-2.0 BLEU. It also outperforms the
Since Artetxe al. 2018b) did not provide the code, we were unable to apply CBD to their work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 451-462, 2017.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. ACL, 2018a. URL hRttps:/:/www aclweb org http//anthooooyyee----11
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Unsupervised statistical machine translation. In
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
pp. 3632-3642, Brussels, Belgium, Novembee 2018b. Association for Computational Linguisties. doi: 10. 10.1865/
URL https: www. ac www.bb..w.............. org/ anthology/
D18 018-1399 1399
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In International Conference on Learning Representations, 2018c. URL https
7/openreview. net Stte://ooommmmd--yyoyggrg
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. An effective approach to unsupervised machine translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 194-203, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v 1//699/////19./// 1019. (tthttts:: www. aclweb. wttp://atoooooyy/ss----eeo-
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. ISSN 2307-387X.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 7059- 7069. Curran Associates, Inc., 2019. URL http:://paprr.niiie ccllpperc/paper/ 6928-cross- iinginnllaagggaaaa -moeeeagge-mmeeeeee www.raiiiig.gpp.........
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1423. URI. https
www. ac 1 web. org/ ana / -19-1423
Sergey Edunov, Myle Out, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489-500, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DD 1/D11/////D11-/// 1045, URL httlt:tt:://// www aclweb. org/ aataaaoooo//1--11100000
Sergey Edunov, Myle Ott, Mare' Aurelio Ranzato, and Michael Auli On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2836-28466 2846, Online, July 2020. Association URL httpt:tt///// /www
for Computational Linguistics. doi: 10. 10.18653/11/2020.adlll
ac web. www.orrg//nthoooooo.o loghhol0gy22200 a.cl-aaaaaa moal-mmin.225. 253
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Pupers), pp. 567-573, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.17-53/ URL https:/ www www.w.wwwbbb......wwwwwwwwww...w org wtt.:////aaaooggg/11----0000
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P 1082. URI.https. wwww.wa..m ac web. www.orgg/atthhoooogo/////eeeeeee 1082

---

Under review as a conference paper at ICLR 2021
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural machine translation. arXiv preprint
2017.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. Universal neural machine translation for extremely low resource languages. arXiv preprini arXiv: 1002.08322. 2018.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Mare' Aurelio Ranzato. The FLORES evaluation datasets for low-resource machine translation: Depali-Enghin and Sinhala-Enghin In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--ICCLL)) pp. (099-6111, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v1/D194 www. aww.eb.ww....................... org/ www.hoogtooaooooooooooooo 1632
URL https
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dow Dowemu,, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou Achieving human parity on automatic chinese to english news translation. CoRR, abs/1803.05567, 2018. URL http:
//arxiv. org.iv.orrgaaae abs/ 803. 05567
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in neural information processing systems, pp. 820-828 2016.
Kenneth Heafield. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pp. 187-197. Association for Computational Linguisties, 2011.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2020. URL https? //openreview net
Ann Irvine and Chris Callison-Burch. Hallucinating phrase translations for low resource mt. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pp. 160-170, 2014.
Ann Irvine and Chris Callison- End-to-end statistical machine translation with zero or small parallel texts. Natural Language Engineering, 22(4):517-548, 2016.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-11
URL WtRLhttps:// www. a locccleeee www. org/ hwt.://wwwwb....r..toooooo////// 6139
Alexandre Klementiev, Ann Irvine, Chris Callison- and David Yarowsky. Toward statistical machine translation without parallel corpora. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguisties, pp. 130-140. Association for Computational Linguistics, 2012.
Philipp Koehn, Franz Josef Och, and Daniel Marcu. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology Volume NAACL '03, pp. 48-54, Stroudsburg, PA, USA, 2003. Association for Computational Linguistics. doi: 10.3115/1073445. 1073462. 3ttpa///AAAA//////o///////.//../
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations, 2018a. URL
wwwp://wwwiiee.eee//onunnnt..eee net /
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. International Conference on Learning Representations, 8b. URL htt.itttts.://peerreveeee 2ttpe/rrrrreeeeeeeeeoornm2222 96sain
10

---

Under review as a conference paper at ICLR 2021
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase- based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 0039-5049, Brussels, Belgium, October- November 2018c. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18-15544 URL
https: wwwp://wwww.. ac web wwt.://w.b...o........o..o....ee http//attooolo..o. ogy/
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising eqquence-t--sqquene pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871 Online, July 2020. Association for Computational Linguistics. doi: 10. (0.18853/v112022..m main.703. URL
https: //w::///wwwwwwo www.wb.wwbb.................w... org htt.:/e.....ggaaoooooogo////////
main. 703,
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.0822 2020.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLPP), pp. 1412-1421. ACI., 2015.
Xuan-Phi Nguyen, Shafiq Joty, Wu Kui, and Ai Ti Aw. Data diversification: A simple strategy for neural machine translation. In Advances in Neural Information Processing Systems 34. Curran Associates, Inc., 2020.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT), 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Sujith Ravi and Kevin Knight. Deciphering foreign language. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp- 12-21, 2011.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86-96, Berlin, Germany, August 2016a. Association for Computational Linguistics. doi: 10.1663//1/P11-1000. URL https 7/ 7/www. www.wb.ww.b........... wwtp:eww....g/aaooogg...ooo/o/// 20060000//111110099
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725. Association for Computational Linguisties,
2016b. doi: 10.18653/v1/PI--11 162. URL http: /ttt://www..o. aww.wb.cclbb........ wwwpor/ggttooooo..oeo hog.ggoytoooogy// 016-1163
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence sequence pre-training for language generation. arXiv preprint arXiv: /905.02450, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 856-861, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/w1/D18-. 1100. URL https www. aclweb. org/ awtatooohoooyy.oo wwapoyyyyll D18-111111 1100
Yiren Wang, Yingee Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Multi- agent dual learning. In International Conference on Learning Representations, 2019. URL
https::/oppnrest net htte://oooommmiitty///23555 httpss///ggeerrrreeeeee.e//orooo

---

Under review as a conference paper at ICLR 2021
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In Internationaa Conference on Learning Representations, 2019.
Zaixiang Zheng, Hao Zhou, Shujian Huang, Lei Li, Xin- Yu Dai, and Jiajun Chen. Mirror-generative neural machine translation. In International Conference on Learning Representations, 2020. URL
https://openrevier net/ forum HK*ITNNT
12

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Generalized Cross-model Back-translated Distillation (GCBD): Given monolingual data X, and Xx languages & and and hyper-parameter n, return 8 - UMT model with parameters 0.
1: procedure GCBD(n, for 1...,, nt do
3: Train UMT agent with parameters 0g
4: Randomly initialize the parameters for the MT model, 0
5: while until convergence do
6: for i 1, ...... do
7:
for E ..... where # do 0
8:
&V 0z, 0)
9:
return i
8 APPENDIX
In the following supplementary material, we provide the generalized version of our method cross- model back-translated distillation, or GCBD, and measure its effectiveness in the IWSLT English- German, German-Eng English, English-French and French-English unsupervised tasks ($8.T). In addition, we investigate why ensemble knowledge distillation (Freitag et al. 2017). which boosts the performance supervised setup, fails to do so an unsupervised setup where we replace the supervised agents used in the method with the UMT agents ($8.2). Lastly, We also show that method is uneffected by the translationese effect (Edunov et al. 2020 (8.3).
8.1 GENERALIZED VERSION
In this section, we describe a generalized version of our CBD, which involves multiple UMT agents instead of just two. Then, we test this method in the IWSLT experiments demonstrate its effectiveness and characteristics. Specifically, in addition to the input monolingual data X, and X, of languages and and the supervised model Ó, we introduce another hyper-parameter to indicate the number of unsupervised agents used to perform cross-modes pack-translation. The generalized
cross-model back translatdd distillation (GCBD) strategy is presented in Algorithm 2 In this method, instead of training only two agents, the method trains a set of n UMT agents = =(1 0n)- During training, iteratively select two orderly distinct agents 0 and 0, from ② and use them to perform cross-modes
and train the model
To evaluate GCBD in comparison with CBD, we conduct experiments with the IWSLT' 13 English- French (En-Fr) and IWSLT' 14 English-German (En-De) tasks. The setup for these experiments (2. 4) to
identical to the experiment in $4.2 except that vary the pappe--tr determine the optimal number of agents. The results are reported in Table As can seen, increasing the number of agents the standard CBD. Moreover, using 4 UMT agents does not improve the performance over using just 3 UMT, despite that this setup still outperforms the standard CBD. The results indicate that increasing the system complexity further is not always optimal and diminishing return is observed as we add more agents the system.
3 adds additional 0.4 1.0 BLEU improvement compared to
Table 8: BLEU scores on the unsupervised IWSLT' 13 English-Free Freschh (En-Fr) and IWSLT' 14 English-German (En-De) tasks with varying number of agents of GCBD.
Method / Data
En-Fr
Fr-En
En-De
De-En
NMT (Lample et al. 2018c) (baseline)
29.6
30.7
15.8
19.1
+ GCBD (n = 2) or CBD
31.8
31.9
18.5
21.6
+ GCBD (n 3)
32.8
32.2
19.1
22.2
+ GCBD (n = 4)
32.3
32.1
19.1
22.1
13

---

Under review as a conference paper at ICLR 2021
8.2 ANALYSIS OF DEGENERATION ENSEMBLE KNOWLEDGE DISTILLATION
Ensemble knowledge distillation (Friitag et al. 2017) has been used to enhance supervised machine translation. It uses multiple strong (supervised) teachers generate synthetic parallel data from both sides of the parallel corpora by averaging the decoding probabilities of the teachers at each step. The synthetic data are then used to train the student model. Having seen its effectiveness in the supervised setup, we apply this same tactic to unsupervised MT tasks by replacing the supervised teachers with unsupervised MT agents. However, the method surprisingly causes drastic performance drop in the WMT 14 En-Fr, WMT" 16 En-De and En-Ro unsupervised MT tasks.
By manual inspection, we found that many instances of the synthetic data are incomprehensible and contain repetitions, which is a degeneration behavior. We then quantitatively measure the percentage of sentences in the synthetic data containing tri-gram repetitions by counting the number of sentences where a word/sub- word is repeated at least three consecutive times. As reported in Table from 30% to 86% of the synthetic data generated by the ensemble knowledge distillation method are incomprchensible and contain repetitions. Relative to the performance of CBD, the performance drop in ensemble distillation is also more dramatic for language pairs with higher percentage of degeneration (En-Ro and En-De). This explains why the downstream student model fails to learn from these corrupted data. The results indicate that UMT agents are unable to jointly translate through ensembling strategy the monolingual data that they were trained on. This phenomenon may require further research to be fully understood.
On the other hand, with less than 0.1% tri-gram repetitions. CBD generates little to no repetitions, which partly explains why it is able to improve the performance.
Table 9: Percentage of tri-gram repetitions in the synthetic data generated by ensemble knowledge distillation (Freitag al. 2017). compared to those created by CBD; and the respective test BLEU scores in WMT 14 En-Fr, WMT' 16 En-De and En-Ro unsupervised tasks.
Method
En-Fr
Fr-En
En-De
De-En
En-Ro
Ro-En
% tri-gram repetition Ensemble distillation
30.3%
34%
73%
76%
43%
86%
CBD (2nd level, 2)
10
10
10
10-1g
10 10-2%
10-2s
BLEU on test set Ensemble distillation
17.3
20.0
3.5
3.7
1.2
1.1
CBD (2nd level, n 2)
26.5
25.8
16.6
20.4
18.3
17.7
8.3 THE TRANSLATIONESE EFFECT
It can be that cross back-translation method is indeed modified version of back- translation (Sennrich et al. (2) Therefore, it is necessary to test if this method suffer from the translationese effect (Edunov al. 2020). As pointed out in their work, back-translation only shows performance gains with translationese source sentences but does not improve when the sentences are natural text Nguyen al. (2200) shows that the translationese effect only exhibits in semi- supervised setup, where there are both parallel and monolingual data. However, while they show that their supervised mack-rranslation technique not impacted by the translationese effect, they left out the question whether unsupervised counterparts are affected.
Therefore, we test our unsupervised CBD method against the translationese effect by conducting the same experiment. More precisely, we compare the BLEU scores of our method versus the XLM baseline (Conneau & Lample, 2019) in the WMT" English-German test sets in the three setups devised by Edunov et al. (2020):
Natural source > translationese target (X + Y*).
Translationese source + natural target (X* Y)
Translationese of translationese of source to translationese of target (X** Y*).
Translationese is human translation of a natural text by professional translator. Translationese tends to be simpler, more grammatically correct, but lacks contextual sentiments and fidelity.
14

---

Under review as a conference paper at ICLR 2021
Table 10
shows that method outperforms the baseline significantly in the natural source translationese target scenario (X - Y"), while it may not improve the translationese source scenario (X" + Y) considerably. The results demonstrate that
method behaves differently than what the translationese effect indicates. More importantly, the translations of the natural source sentences are improved, which indicates the practical usefulness of our method. Furthermore, in combination with the findings in Nguyen et al. (2020), the experiment shows that the translationese effect may only exhibit in the semi-supervised setup, but not supervised or unsupervised setups.
Table 10: BLEU scores of our method and the baseline (Conneau & Lample 2019 on the translationese effect (Edunov et al. 2020), in the WMT" 14 English- setup.
WMT' 14 En-De
X
Y
X
-Y
X"*
Y
UMT Baseline (XLM)
18.63
18.01
25.59
Our method
20.79
18.24
27.85
15


Reviews:
[{"review_id": "yrhIBVUpkjj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method to enhance unsupervised machine translation through data augmentation. The idea is pretty straight-forward, if not altogether intuitive, you begin by training two bidirectional (i.e.: they can translate source to target and target to source) unsupervised MT systems A and B. The tested scenarios always have A and B be identical architectures trained with different initializations. They then produce synthetic source-target pairs by first having A (source->target) translate the provided source sentence x to y, and then having B (target->source) translate y back to z. They then train supervised MT on both x,y and z,y. The same procedure can be repeated with source and target reversed. The authors show substantial (1-2) BLEU improvements with 3 different UMT systems in 5 low-data scenarios (En-Fr, Fr-En, En-De, En-Ro, Ro-En), all subsampled to 5M monolingual sentences for each language. In En-Fr and Fr-En and En-De, they are able to match reported XLM results from Conneau and Lample 2019, despite using much less data.\n\nThis simple idea is explored extremely thoroughly. The paper reads more like a journal paper that has undergone several stages of review than a conference paper. The authors make connections to and compare against a number of relevant ensembling strategies (to account for two systems being used) and back-translation-diversification strategies (to account for multiple sources being produced for the same target), and consistently show that only their specific recipe leads to the same levels of improvement. The authors really leave no stone unturned.\n\nThe biggest knock against this paper is the relatively small data scenario. Having two UMT systems allows them to provide two source sentences (one original and one synthetic) for each target sentence (always synthetic), but how important is this when we have 25x more original source sentences? I can imagine arguments for why the high-data UMT scenario is unrealistic (many monolingual sentences implies the likely presence of parallel data), but those arguments aren’t presented in the paper. It would be greatly strengthened by a full-data experiment for even just one or two of the language pairs.\n\nBeyond that, I have few concerns. The paper is clear, easy to follow, and as I said, very thorough. But I’ll do my best to make some constructive criticisms:\n\nI think the Related Work section feels a little superfluous after all of the comparisons made to related work in the Background and in the Experiments. I think I would like to have seen more discussion of the highlighly related work in sections 5.3 and 5.4. In particular, a more detailed discussion of this method’s relation to multi-agent dual learning would be worth giving up parts of Related work that are already mentioned in Background (like pre-neural statistical unsupervised MT).\n\nIt would be useful to specify how BLEU is calculated, to help readers understand just how useful the cross-paper BLEU comparisons in Table 1 are.", "extra_scores": {}, "extra_text": {}}, {"review_id": "p7W-X0jkVas", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new component to the unsupervised machine translation framework called cross-model back-translated distillation. The proposed approach is applicable to the other unsupervised methods. Experimental results in several translation tasks show that the proposed approach improves the translation accuracy of the standard unsupervised machine translation models, outperforming the cross-lingual masked language model. \n\n- The analyses are interesting to understand the proposed approach. Table 4 reports the diverse of synthetic data, but what about the quality as parallel data? Can you use parallel data instead and conduct the same analyses so that you can use BLEU score as an evaluation metric?\n- Is it possible to apply the proposed approach to supervised NMT training, by creating BT data from monolingual data? \n- Table 1 reports that the results from your experiments show that the equivalent/better performance agains the existing models with much fewer data. What about scaling up the monolingual data size 5x/10x more? Will the performance be improved better and better?\n- \"the translated products (x-y) of the UMT teachers.\" at p.6.  What does this \"x-y\" mean?\n\nTypo:\np.3 5.2.In Appendix -> 5.2. In Appendix", "extra_scores": {}, "extra_text": {}}, {"review_id": "8slankuRNKO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes an additional stage of training for unsupervised NMT models utilizing synthetic data generated from multiple independently trained models. The generated synthetic data uses two stages of back-translation, with different models, in order to \"diversify\" the set of training data used for fine-tuning the models. This is similar to the approach in [1], but uses an additional stage of back-translation with a different model. The authors add this additional stage of training to unsupervised NMT models using different pipelines (PB unsupervised MT, Neural Unsupervised MT, XLM) and show that their approach improves all of these approaches by 1.5-2 Bleu on WMT En-Fr, De-En and En-Ro.\n\nStrengths:\n1. The paper is well written, the approach is simple and seems to improve quality by significant amounts in a variety of experimental settings.\n2. The authors do a great job of comparing against several relevant approaches (sampling during back-translation, ensembling, multi-agent dual learning). The paper compares against most of the relevant approaches I could think of while reading the paper.\n\nWeaknesses / Questions for authors:\n1. As with any NMT model trained with synthetic data, it would be better to report results on source and target original splits of the test data to provide a clearer evaluation [2,3]. Also clarify the Bleu scripts, tokenization and other post-processing used for evaluation.\n2. The datasets used for experimentation are much smaller than the ones used for the baseline unsupervised-NMT approaches. It would be great to report results in the original training conditions (this is not a major limitation however, since the proposed approach seems to improve over baselines trained with more data).\n3. Did the authors try any experiments with unsupervised models utilizing parallel data in unrelated languages, similar to [4,5] or in real low resource settings [6]? These are more practical conditions for unsupervised MT in true low resource languages.\n\nRecommendation: Overall, this is a good paper and I would recommend acceptance. While I would have also liked to see experiments in more realistic low-resource settings, the current paper does a good enough job of evaluating the approach in standard unsupervised NMT settings on related high resource languages.\n\nReferences:\n[1] Data Diversification: A Simple Strategy For Neural Machine Translation, Nguyen et al.\n[2] APE at Scale and its Implications on MT Evaluation Biases, Freitag et al.\n[3] On The Evaluation of Machine Translation Systems Trained With Back-Translation, Edunov et al.\n[4] Multilingual Denoising Pre-training for Neural Machine Translation, Liu et al.\n[5] Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation, Siddhant et al.\n[6] When Does Unsupervised Machine Translation Work?, Marchisio et al.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3pLTvufPPm1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, two unsupervised agents are utilized at cross-model by using the dual nature of the unsupervised machine translation model, in which forward translation of agent_1 is combined with the backward translation of agent_2, more synthetic translation pairs are obtained to train a new supervised machine translation model. The result is improved on multiple unsupervised machine translation, and this paper claims that more diversity is brought to the synthetic data, so a better translation model can be trained. This paper uses a reconstruction BLEU or BT BLEU [1] metric to compare the effect of the inside-model with that of cross-model, and finds that cross model translation has a lower back-translation effect, which shows that the diversity is enhanced. Furthermore, CBD is compared with the ensemble method and achieves better performance. The proposed method is quite simple yet effective, but it is also a kind of data enhancement.\n\nIn addition to these contributions, the paper also has some shortcomings\n\n1. The evidence in this paper can not support the claim that the current performance bottleneck of UMT is due to the lack of diversity: the performance upper limit of UMT is still due to the lack of clear supervision signal, which limits the further performance growth. Because the training of CBD is divided into two stages, the diversity of the second stage only brings more training data to enhance the supervised machine translation model, rather than unsupervised machine translation effect.\n\n2. Source of promotion: the second stage of CBD method adopts (x_s, y_t), (z_s, y_t), (y_s, x_t), (y_s, z_t) synthetic translation pairs, it is not clear how much performance growth comes from increased data and how much growth comes from the new model implementation (ott et al., 2018). It is not appropriate to attribute all contributions to the diversity brought by CBD. I suggest that the author should use (y_s, x_t) data to train based on the (ott et al., 2018) model, and report the effect comparison (In my experiments, the second stage model implemented with fairseq trained only on (y_s, x_t) surpass both agents trained with XLM due to more efficient implementation in fairseq).\n\n3. Unfair comparison with the enable distillation: authors need to compare CBD with the model trained on the synthetic data (y_s, x_t) of the ensemble of agent_1 and agent_ 2. In the training data (x_s, y_t), (z_s, y_t), (y_s, x_t), (y_s, z_t) for the second stage of CDB,  x_t, golden language sequences as translation target is stronger than synthetic language sequences (silver) as target. Therefore, It is necessary to report the real result of ensembled distillation. The current results are very unreliable. In addition, it is necessary to compare the training time of the CBD method and ensemble distillation training (including the decoding process after the first stage of training) to show the efficiency of CBD.\n\n4. The non-golden language sequence as a translation target is called pseudo-NMT (PNMT). The author adopts a variety of model structures, which is slightly redundant. They can directly add the synthetic data decoded by cross model to continually train the original XLM model with a supervised translation objective (which is naturally supported in XLM from my experience), and report the effect comparison between them.\n\n5. The essence of the CDB approach is a process of self-supervised training, so it is necessary to compare self-training/tri-training introduced in [2].\n\n\nIn general, the CBD method in this paper is a simple and effective data enhancement method to improve the performance of the model. However, due to the lack of many important details of the implementation, despite the promotion, the source of promotion is unknown. In addition, the unreasonable comparison of the baseline models deepens my concern about the real promotion of this CBD method.\n\n[1] Li, Zuchao, et al. \"Reference Language based Unsupervised Neural Machine Translation.\" arXiv preprint arXiv:2004.02127 (2020).\n\n[2] Sun, Haipeng, et al. \"Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios.\" arXiv preprint arXiv:2004.04507 (2020).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an additional training objective for unsupervised neural machine translation (UNMT). They first train two UNMT models and use these models to generate pseudo parallel corpora.  These parallel corpora are used to optimize the UNMT training objective. The experiments are conducted on several language pairs and they also compared with several alternative works. 

All the reviewers admit that the proposed method is straightforward and effective. The authors claim that the new training objective is used to enhance the "data diversification". This point has been questioned by the reviewers. Some reviewers are convinced by the response and some still have different opinions.  From my point of view, the proposed method can also be considered as a kind of combination of  (pseudo) supervised NMT and unsupervised NMT. 

The presentation and description of its key contributions seem unclear. However, we encourage the authors to modify their paper and we believe this proposed method can inspire the MT community for further research. At the moment, the paper is seen as not yet ready for publication at this time.
Decision: Reject