Demystifying Learning of Unsupervised Neural Machine Translation
Authors: Guanlin Li
lemao liu
Taro Watanabe
Conghui Zhu
Tiejun Zhao
Venue:  (2021)
Keywords: Unsupervised Neural Machine Translation
Marginal Likelihood Maximization
Mutual Information

Abstract: Unsupervised Neural Machine Translation or UNMT has received great attention
in recent years. Though tremendous empirical improvements have been achieved,
there still lacks theory-oriented investigation and thus some fundamental
questions like \textit{why} certain training protocol can work or not under
\textit{what} circumstances have not yet been well understood. This paper
attempts to provide theoretical insights for the above questions. Specifically,
following the methodology of comparative study, we leverage two perspectives,
i) \textit{marginal likelihood maximization} and ii) \textit{mutual information}
from information theory, to understand the different learning effects from the
standard training protocol and its variants. Our detailed analyses reveal
several critical conditions for the successful training of UNMT.

Under review as a conference paper at ICLR 2021
DEMYSTIFYING LEARNING OF UNSUPERVISED NEU- RAL MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised Neural Machine Translation or UNMT has received great attention in recent years. Though tremendous empirical improvements have been achieved, there still lacks theory-oriented investigation and thus some fundamental questions like why certain training protocol can work or not under what circumstances have not yet been well understood. This paper attempts to provide theoretical insights for the above questions. Specifically, following the methodology of comparative study, we leverage two perspectives, marginal likelihood maximization and ii) mutual information from information theory, to understand the different learning effects from the standard training protocol and its variants. Our detailed analyses reveal several critical conditions for the successful training of UNMT.
INTRODUCTION
Unsupervised Neural Machine Translation or UNMT have grown from its infancy (Artetxe et al.
2018 Lample et al. 2018a 8a) close-to- susperviee performance recently on some translation scenar-
ios (I aample & Conneau 2019 Song et al. (2019). Early UNMT works (Artetxe et al. [017 Lample
ct al.) [2018|] Yang et al. 2018) adopt complex training strategies including model initialization, syn- thetic parallel data for warming the model, ad adversarial loss for making encoder universal, differ- ent weight sharing mechanisms etc. Then Lample et al. (2018b) simplifies all these and establishes a two-components framework, involving an initialization strategy followed by iterative training on two tasks, i.e. denoising auto-encoding with the DAE loss and online back-translation with the BT loss. Works afterwards mainly focus on developing better initialization strategies (Lample & Conneau 2019: Ren al. 2019 (ong et al., [219 Liu et al. 2020). Although obtaining impressive performance, it is unclear why this standard training protocol possible to be successful. Kim et al. (020) and Marchisio et al. (2020) consider the standard training as a black-box and empirically analyze its success failure regarding different data settings (i.e. text domains and language pairs). Unfortunately, due to the lack of theoretical guidelines, some fundamental questions are still re- mained unknown: what standard training tries to minimize under the general unsupervised training paradigm GGhahramani] (2004) and when a certain training protocol can work for training UNMT? In this paper, we attempt open the back-box training of UNMT and understand its theoretical essence from two angles: i) a marginal likelihood maximization view; and ii) an ceermaaiin-thhoreee view by ablating standard training protocol with other variants. Our contributions are as follows.
A. By making an analogy of standard training protocol with marginal likelihood or Evidence Lower BOund (ELBO) optimization, we visualize the learning curves of the two terms in ELBO objective, and found that optimizing ELBO is not sufficient for training a successful UNMT model, indicating that specific regularization design i.e. the DAE loss, quite matters.
B. By leveraging information theory, we present formal definition what does it mean to success- fully train an UNMT model, and then readily derive sufficient condition and necessary condition for successfully training UNMT in principle. In addition, conditions through empirical experiments, and find that both conditions indeed explain why standard training protocol works while others suffer from degeneration to learning sub-optimal tasks.
validate both sufficient and necessary
C. Based on explanations for those failed protocols, we continue experiments to settle the role played by DAE and BT. Firstly, BT is the main task while DAE is critical auxiliary. Then clarify that DAE has more important role than just learning word order, accepted as common knowledge in almost all previous works, but also preserving the mutual information between encoder input and

---

Under review as a conference paper at ICLR 2021
encoder output, which is necessary for successful training. Furthermore, DAE also functions behavior regularizer for decoding with online BT, and prevents BT from yielding degenerated data.
2 UNDERSTANDING UNMT FROM TWo PERSPECTIVES
In this section, we first introduce background about the standard training protocol proposed in Lam- al. (2018b), which is adopted by almost all later works. Then we introduce the basic concept of two perspectives on which we rely for analyzing the learning of different training protocol variants. Due to the space limit, please refer to appendix A.] for timely literature review of recent advance.
2.1 STANDARD TRAINING PROTOCOL
The standard training protocol involves standard initialization strategy and standard iterative train- ing procedure, and they both are built upon a specific design of encoder-decoder parameterization.
Parameterization and initialization UNMT model adopts shared embedding matrix for shared vocabulary with joint BPE (Sennrich et al. 2016), and the two languages share the same encoder and decoder with only a language embedding for distinguishing the input from different languages. As result, unconstrained decoding might generate tokens from the same language as the input. Standard initialization means using fasT (Bojanowski
al. 2017) to initialize the embedding matrix, denoted as JointEmb. XLM (Lample & Conncau 2019) uses trained encoder to initialize both encoder and decoder of the UNMT model. We also consider random initialization for completeness.
Iterative training strategy The iterative training strategy involves optimization of two critical losses by turss, i.e. the DAE loss and the BT loss as defined in Eq. [ and Eq. where s and denote the two languages. DAE loss is constructed through sampling monolingual sentence (or y), construct its noisy version C(x) (C)) and minimize the reconstruction error or RecErr:
doee
log Po (xx)) + |- Pr-xcylC((
(1)
BT loss is constructed through sampling a monolingual sentence x (or y), construct its corresponding translation via the current model M(x) (M(y)) through back-translation and minimize the RecErr:
che Eg>-(=>]-- log (1(()) + Ee- [- log Pa- (
The online BT process involved in the iterative training strategy can be seen as Co-Training (Blum & Mitchell 1998), where two models (with shared weights) constructed on two views (source/target sentence) generate pseudo labels as the other view (pseudo translation) for training the correspond- ing dual model. We summarize the whole standard training protocol in Algorithm I in appendix A.2
Constrained decoding Besides the basics, we further introduce the concept of constrained decoding where the model should be constrained to decode tokens only in the target language regardless of the shared embedding parameterization. This could give us simple definition of cross-lingual RecEr beyond naive RecErr in Eq, 2 Details of the algorithm and the definition are shown in appendix A.3
2.2 A MARGINAL MAXIMIZATION VIEW
The standard training of UNMT model takes advantage of sole monolingual corpora D., Dr, which is similar to the generative modeling setting where only unlabeled data is available (Ghahramani 2004). Here we take an analogy of the standard UNMT training as implicitly maximizing marginal of the monolingual data. Due the duality of translation (He al. 2016), the target sentence not only plays the role of label, but also the input in reverse translation direction. So in essence the standard UNMT training can be seen as maximizing the marginal log likelihood of D. and DD simultaneously. However, since marginals involve infinite summation over a certain view (larget/source), lower bound is often optimized via Monte Carlo approximation (Kingma & Welling 2014).
In the following derivation of ELBO Kingma& & Welling 2019), (g(w)zr) is the posterior distribution of y when taking y as the latent variable. Here we only derive the bound for x E D.- A detailed analogy of the standard UNMT objective and the ELBO objective is presented in Table I As you can see, both objectives have the same reconstruction error terms but different regularization terms: for ELBO, the model is optimized stay close with the language model via the KI. loss.
2

---

Under review as a conference paper at ICLR 2021
Table I: An anology of the standard UNMT objective and the negative ELBO objective. Note that both objectives have the reconstruction term; since in VAE training KL divergence is often referred to as regularization (Prokhorov al. 2019) so we call DAE loss the regularization term as well.
reconstruction error
regularization term
standard UNMT
M(x)(() Ptg
lo C(x))
negative ELBO
E (u(l))) po
DKz(9o(y (o(y))
log po()) 2 Bas(ula)) log po (=lw)] DEL (vole)))))) (ELBO)
(3)
Worth noting that, we are not the first to make a connection between marginal maximization and the standard UNMT training. Hn et al. (2020), they have already proposed an ELBO formulation for unsupervised sequence transduction task. However, they tend to focus on replacing the standard UNMT-iraining-styll objective function with the ELBO objective, and propose several critical tricks such as Gumbel softmax and self-reconstruction for making ELBO really work. Instead, we leverage ELBO mainly
anology to the standard UNMT training objective, and through comparative study with other protocol variants, we can further understand why standard objective and its variants work or not, even though they all tend have similar ELBO values. Details are in appendix A.4
2.3 AN INFORMATION-
VIEW
If we denote Y" = M(X) as the random variable (F.V) generated by the model M over X. There- fore, if (Y", X) gradually contains more and more bilingual knowledge, the model will be able to generate better translations, eventually leading to the success of UNMT training. Suppose c con- stant predefined by users which controls the satisfactory level for translation performance, we give a definition to formalize the success of UNMT training from an information-theoree viewpoint.
Definition 2.1. If I( (X) > c after training, we say that UNMT training fails. (Caveat, c is concetual quantity, we never instantiate its value.)
we say that UNMT training is successful; otherwise,
Suppose p( x. s/) is the true distribution of (z, v/). and Pt-s<( [x] y/) is an estimator of p( g/). We obtain the following two conditions for success of training UNMT based on the above definition 2.1
Proposition (Sufficient condition) Eptz.y) log pt- 2 e HX)), then UNMT training will be successful.
Proof. Based on Definition 2.1 and the definition of mutual information (MI) and Jensen's 's inequal- ity, we can derive the following inequality (Pimentel et al.) 2020):
T(X, Y") (()
( Y')= H(X) - H (X)) H(X) + Ep(r,s) log (1 (/') 2
(4)
Since the sufficient condition relies on true distribution p(x, 3/) which is unknown in practice, we sample (Ir, 3/) from the empirical distribution of X and Ps-re (3(((r))) (xr) as approximation. Then the ideal sufficient condition is reduced to a practical one: If Σx Ep. Ep.-1((0'ler) (2/la))) log (xly(()ly ve) 2 c HX),, then UNMT training will be successful. Since MI is symmetric, we can also have similar formula regarding - direction with Ey(a' s)) log ( x²) 2 H(Y). They together connect success of training to the BT loss in Eq. 2 lower BT loss more likely to make UNMT training successful.
Furthermore, if we denote the encoder output as EV. Z enc(X) (X), we can obtain the following necessury condition:
Proposition 2. (Necessary condition). If UNMT training is successful, then [[ Z) 2 c.
Proof. Following the Data Processing Inequality Coovrr & Thomas 1991). we have the following inequality that holds all the time:
T(X, 2) 2 T(X,Y') 2c
(5)
3

---

Under review as a conference paper at ICLR 2021
with the Markov chain (or data processing order) X m/ Z dec Y
In subsequent experiments, we follow Pimentel et al. 20200 and estimate I(X, 2) = H(X)
[[( by calculating [ through a statistical 1-gram language model and [ through probing (Conneau al., 2018) respectively. For estimating II(X, Y"), we use the loken-by-loken point-wise mutual information (PMI) over some pseudo bitext as surrogate to the sentence-by- sentence MI. The detailed estimation methods are presented in appendix AA.5
EXPERIMENT
In this section, we conduct a series of carefully-designed comparative studies of the standard UNMT training pro- tocol and its variants well as using the ELBO objective. Different training protocols that we investigate and their name abbreviations are listed in Table 2 The logic flow of this section could be summarized as Tollows. i) Firstly, we present the basic experimental settings, the overall per- formance of carefully-desiineed ablations, and some high- lighted observations from those results; ii) as promised in Sec. 2.2 we visualize the learning curves of different terms in negative ELBO with a conclusion that marginal maximization is only necessary but not sufficient condition for successfully learning translation; iii) we explain both quantitatively and qualitatively why dae-only and bt-only that implicitly optimizes ELBO cannot work from an information theoretti perspective, and highlight the importance of task task-specific regularization such as the DAE loss; iv) we clarify the main and auxiliary relationship between BT and DAE loss, and further investigate the critical regularization effects of DAE loss.
Table 2: Variants of training protocol.
Protocol
Loss used
standard
DAE (Eq.))
+ BT (Fq. 2
dae-only
DAE (Eq.
bt-only
BT (Eq.
elbo
ELBO
elbo-dae
ELBO (Eq.
DAE (Eq.
3.1 EXPERIMENTAL SETTINGS AND OVERALL PERFORMANCE
Dataset and Reproducibility We adopt the publically accessible WMT14 En-Fr and En-De datasets for our experiments. We strictly follow the data pre-processing pipeline and instructions for training in the official XLM code repoitory| The monolingual data for each language is set to about 5M sentences from the newscrawl monolingual collection [ Though we do not focus improving over state of the art, adding more monolingual data can indeed largely improve the final performance. For XLM initialization, we download the pretrained models from the XLM repo; and for JointEmb initialization, we use fastTEXT (Bojanowski et al. 2017) for learning the word embeddings on the concatenated monolingual corpora for each language pair (about 10M).
Outline of Overall Performance Here in Table 3 we first report the overall performances with the standard training protocol and two of its variants dae-only and bt-only under the three initial- ization strategies (random, jointEmb, XLM); performances of optimizing elbo are also shown with XLM initialization. There are several observations to be highlighted here. i) Only the standard and the elbo-dae training protocols lead to decent performances, and the later requires using DAE loss as well and it is necessary to set the coefficient of the KI. regularization term lower than 0.05. ii) Simply optimizing elbo leads to failure of training. iii) although dae-only seems to lead to very low performance through largely copying the input, if we continue training with the BT loss alone, we can surprisingly obtain similar (or sometimes even better) performance compared with standard training protocol (the +BT loss row), though the initial performance is very low (about 2 BLEU points), which indicating that it is not necessary that we have initial model with decent performance to make Co- Trairining successful. iv) For bt-only, if we continue with standard training, the final performance still struggles to reach that of standard; actually, for weak initialization methods (ran- dom, JointEmb), the model could even hardly learn copy, and stay failure all the time. This may imply that bt-only is learning poisoned inner representation. Actually, according the information inequality (5) bt-only will make I(X, Z) to be very low, that is the output of the encoder can hardly identify the input sentence (Brunner et al. 2020), therefore data quality of Co- fo-Training stays low all the time. We will design an experiment to prove this in Sec. 3.4.2
hhttps///gttub.... hhtpss/githbb.mmtfccchooees cwm/ftacchookresennccII...L
Please refer to the "get-data-mmt.shh sh' script in the XI.M M repo for more details; we the default setting.
4

---

Under review as a conference paper at ICLR 2021
Table Performances Paaiinnni al. and bt-only under three initializations; together with elbo-related protocols with XLM initialization.
2002) of the standard training and its two variants dae-only
random init.
JointEmb init.
XLM init.
Protocol
en-fr
fr-en
en-de
de-en
en-fr
fr-en
en on-de
de-en
en-ft
fr-en
en-de
de-en
standard
12.41
12.36
6.75
8.88
27.37
25.13
17.68
22.34
33.12
29.96
27.29
33.03
dae-only
1.39
1.35
2.68
2.75
2.20
2.19
2.74
2.79
2.32
2.31
3.40
9.61
+BT loss
11.19
10.71
7.79
9.66
26.75
24.96
18.93
23.31
33.14
29.88
26.67
32.87
bt-only
0.25
0.
0.
0.69
0.29
0.
0.15
0.73
0.18
0.19
0.18
0.55
standard
0.13
0.59
0.16
11.85
15.54
19:30
18.16
20.52
25.65
elbo
0.
elbo-dae
32.67
29.81
En-Fr
(c-
XLM Jointmm random
XLM
XLM
ezt
1990
Rec extimate
JointEmb random
Kl extimate
JointEmb
800
490
390-
220
120
146
Number updates x 50
Number of updates x 50
Number of updacess x
Fr.En
2]
2)
En-Fr
XLM
730
XLM
500
XLM
jointEmb random
JointEmb
JointEmb
estimate
Bec En estiminng
random
XL extimate
220
490
45
120
49
65
120
20
146
Namber of updates x 50
Number of updates x 50
Number of updaiess x
(a)
(b)
Figure 1: Negative ELBO learning curve visualization on WMT14 En-Fr validation set under stan- dard training; each column represents (a) ELBO, (b) reconstruction error and (c) KL estimation.
3.2 VISUALIZING ELBO LEARNING CURVES
In this subsection, set up to visualize the learning curves of ELBO, together with the two terms in ELBO, i.e. reconstruction error and KL divergence. The actual ELBO values are negative, but learning commonly means to minimize certain loss, here visualize the negative ELBO. The lower the value is, the better the ELBO is being optimized. We first demonstrate the learning curves of standard training, describe some observed phenomena, and then turn to visualize the curves of other failure training protocols. Note that, since we use two samples (k=2) in the Monte Carlo approximation, all the terms are twice the value should be.
Figure [ (a) demonstrates the ELBO learning curves of the standard UNMT training under three initialization strategies. The overall ELBO on the two monolingual datasets is the sum of En-Frr and En directions. As you can see, across all initialization strategies, even though there is clear mismatch between the standard UNMT objective and the ELBO objective over the regularization term, we can conclude that standard training implicitly minimizes negative ELBBO.
Figure (b) is the visualization of the reconstruction error term within ELBO. It is evid-et that for standard training, the reconstruction loss represents the lross-lingual translation ability of the model, so the original paper (Lample et al. 20) the reconstruction BLEU, which correlates well with the reconstruction error, is used for model selection without given bitext development set. Figure I (c) shows the visualization of the KL divergence term. is interesting that for all initialization strategies, the KL value first goes down and then goes up quite bit till convergence. This learning phenomenon can be summarized as: the standard training protocol tends to make the model first fit to behave like language model of the two languages and then fit to translation model
5

---

Under review as a conference paper at ICLR 2021
(K-
Roc (c
(K
standar) ean wif
6624
352
ELBO astimate
estimate
342-
203
122
KL estimate
200
300
409-
183
123
53
Number updates 330
Number updates
Number of updates
(a)
(b)
(c)
Figure 2: Negative ELBO learning curve visualization on WMT14 En-Fr validation set for all failure variants comparing with the standard training protocol with XLM initialization.
in later stage. And the 'going-up' reflects the existence of large distance between the distribution of a language model p(y) and the translation model p(y (z), given same target y.
Next, we continue to visualize the ELBO curves of some failure variants of the training protocol. In Figure we draw all the ELBO learning curves for those training protocols presented in Table that fail to train well-performed model under XLM initialization. As you can see in Figure (a). other than standard, before (30x50=) 1500 updates, all variants seem to reach similar ELBO values. As the training goes on, dae-only and bt-only tend to have exactly the same ELBO value with the standard training protocol. However, the reconstruction error of bt-only is much higher than standard, while the KI. divergence of dae-only is much higher than that of standard as well. Both low reconstruction error and a low KI. distance are necessary for successful training. For elbo-only, there is a quick posterior collapse phenomenon at initial training (before 1 updates) (in Figure 2 (c) KL becomes very low), however, then the KL slowly goes up, which might be resulted from the unstableness of ELBO optimization with REINFORCE (He et al., 2020). This indicates that only requiring ELBO to be optimized a whole is only the necessary but not the sufficient condition of successful learning the target translation task.
3.3 WHY MINIMIZED ELBO CAN still LEAD TO TRAINING FAILURE?
An intuitive explanation is that unsupervised learning through marginal likelihood maximization is under-determined. There are many plausible tasks like language modeling, paraphrasing, sim- ple sequence copying, translation, that satisfy the inductive bias of the parameterization, and freely learning with objectives like ELBO can make the model learn any of the plausible tasks if optimiza- tion finally converges. And learning any of them can induce minimized ELBO. So what tasks on earth have dae-only and bt-only finally learned respectively? Table in appendix demonstrates the decoding behavior of the final models given certain source inputs. We can assume that dac-only degenerates to the sequence copy task while bt-only degenerates to the language modeling task.
3.3.1 ANALYSIS ON FAILURE OF DAE-ONLY
Learned copy Figure (b) informs us that dae-only has even lower reconstruction error than stan- dard, which means that even if dac-only is not trained with the language embedding feeding of another language in the target-side, it can still minimize the reconstruction loss when feed with the target language embedding. However, Table 7 demonstrates that dae-only has learned almost per- feet sequence copy. Here we use the definition of cross- mooss-lingual RecErr to clarify this phenomenon, since unconstrained decoding might generate tokens that come from source language and this might prevent us from distinguishing RecErr of mono-lingual reconstruction (copying, paraphrasing) and cross-lingual reconstruction (translation).
In previous subsection, we draw all the learning curves in Figure based on naive RecErr without considering the above situation. Here in Figure 3 according to modified Definition A.2 we can draw the cross-ling lingslln RecErr for standard, dae-only and bt-only accordingly. As shown, the cross-lingual RecErr curve is much higher than the RecErr for dae-only, and is the highest among all, indicating that, essentially, the target translation task is learned only by learning a low cross- lingual reconstruction loss. That is why later work like Liu et al. (2020) directly uses constrained

---

Under review as a conference paper at ICLR 2021
Table 4: Token-to-toker mutual information esti- mated over the generated pseudo-bitext.
Table 5: Controlling the update frequency ra- tio of DAE and BT losses under XLM init.
greedy
sample
7
cn-fr
fr-en
en-fr
Ir-en
en-Ir
Ir-en
0.02
24.37
22.82
standard
0.55
0.53
0.46
0.52
0.05
27.21
24.55
bt-only
0.06
0.08
0.11
0.12
0.1
33.46
29.78
dae-only
0.57
0.50
0.56
0.58
33.12
29.96
0.60
0.49
0.43
0.37
32.02
29.18
random
0.27
0.27
0.27
0.27
2.30
2.29
Fi-En Cross -ingual Rec Err
Mi estimate along training
etnneari (anee) daccedy
Mutual Information (PP41)
2:50
Rec Er estimnaee
307
lc bienly jeras
2.45
102
2.40
3:36
Hardarr, samslingg
802
0:30
-J6 das ionse) sammpp
din-wrr simplin
0:35
booetl. sumpling
3:20
102-
3:15
330
10
Number of up dates x
Epoch
Figure 3: The learning curve of cross-lingual Re- сErr based on Definition A.2 the RecErr of dae- only is shown for comparison as well.
Figure 4: The mutual information estimation of the pseudo-bitext generated by sampling-basee decoding along the training process.
decoding for BT to accelerate training. We also calculate the correlation between RecErr/cross- lingual RecErr and BLEU in F the appendix), the later is much better correlated (-0.37 verses -0.87) with final performance. Moreover, in Table 4 we calculate the mutual information in pscudo-bitext generated by dae-only under constrained decoding, suprisingly
find that the
MI is very high. However, since dae-only does not expose the model on such data, it never learns the cross-lingual alignment in the data, which indicates the important role of BT that can actually learn such cross-lingual MI. These findings support our practical sufficient condition: a lower cross- lingual RecErr is more likely to make UNMT training successful.
3.3.2 ANALYSIS ON FAILURE OF BT-ONLY
Degeneration to LM As you can see in Figure the sentence-level KL distance between bt-only and the language model is very small, much lower than standard and others. This indicates that the learned behavior of bt-only may resemble the behavior of language model. That is, the UNMT model learned with bt-only might largely ignore the potential predictive information contained at the source side, and only relies decoder's LM prior.
MI of Pseudo-Bitexs intuitively visualized by showing the mutual information contained in the pseudo-bitext generated with online iterative BT. Table 4 shows the mutual information of the final checkpoints obtained by standard, bt-only and dae-only. As you source and target of the generated bitext. And even if we instead use sampling for generation, the 0.27). We also conduct experiment
The reason why such degeneration happens during the training process can be
bt-only has the lowest mutual information between
mutual information is still lower than random bitext (0.12 when at BT phase the model uses sampling instead of greedy, this will alleviate the degeneration a little bit, but the learning still fails. We further draw the mutual information of pseudo-bitext along training in Figure 4 and bt-only stays low all the time, while standard have growing values.
3.4
THE CRITICAL ROLE OF BT AND DAE LOSSES
3.4.1
BT LOSS IS THE MAIN TASK WHILE DAE LOSS THE AUXILIARY
In the standard UNMT training protocol, we can think of consecutive learning of DAE and BT losses multitasking (Caruana 1998). However, since the (cross-lingual) BT-loss is directly related to

---

Under review as a conference paper at ICLR 2021
Table 6: The estimated mutual information between encoder input X and output 2 enc(X).
protocol
standard
bt-only
elbo-only
XI.M init.
en
7.603 0.079
7.603
2.863
7.603
4.299
7.603 00.14
fr
7.449 0.096 7.449 2.571
7.449 3.980
7.449 -0.123
[2r, 1)) which is used guarantee the success of translation in principle as mentioned in Sec. 2.3 we may think BT task as the main task and DAE task as the auxiliary task with respect to the target translation task. This section gives an affirmative answer to this intuition. Since DAE and BT losses are updated one after another, we can control the update frequency ratio between them. This can control the degree of the model learning towards certain loss and setting. Table 5 shows the results. The larger is, the more frequent DAE loss is being updated. As you can see, it seems the more DAE loss is updated. the training tends to become the dae-only setting, that is, the model starts to learn copy instead of translation. On the contrary, it does not hurt so much even BT is updated 10 times more than DAE (T = 0.1), though increasing BT updates lot (larger than 20 times, T 0.05) will definitely degrade final performance, which reflects the important role of DAE's regularization effect on BT, that is to constrain online inference at BT phase so to prevent the model from learning unexpected noise.
corresponds to the standard
3.4.2 DAE NOT JUST HELPS WITH LANGUAGE MODELING, BUT PRESERVES MI
In previous works DAE loss have been recognized learning word order aka. the language model- ing ability of the UNMT model Lample et al. 20): Artetxe et al.) 2018): Lample et al. (201 8b); Yang et al. (2018) Kim et al. (2020). Here we would like to clarify that the most critical functional- ity of DAE is not just learning language model, but at least preserving the MI between encoder input and output which matches the necessary condition we introduced in Sec.2.3 2.3 As a result, it can prevent the model from degeneration during online BT. To this end, we first experiment with a differ- ent version of DAE loss that ignores word order, that is, when constructing DAE loss from 2 we first permute the order of x. denoted by Perm [x] and then optimize - log P (Perm [x:] C(Perm[ (z) instead. For XLM initialization, the final performance only drops from 33. 12.12 to 31.02. This indi- cates that DAE are not only learning word order, but something more critical, i.e. preserving the MI between input X and encoder output Z.
We verify this by estimating the MI between X and Z for encoders trained with standard, bt-only, elbo-only, and a baseline encoder initialized from XLM. Then we only train a random initialized de- coder over the encoder. In Table 6 each entry consists of two terms with the first term estimation of H() and the second term an estimation of H (IXIZ) Z). As you can see, without the regularization effect of DAE, bt-only and elbo-only has very large entropy of X|Z. even much larger than the XLM initialized encoder. This can explain the previous highlighted phenomenon in Table 3 that is, after bt-only, if we continue train with standard, only XLM initialization can recover certain performance while the other two stay failure. The reason is that without DAE, the encoder represen- tation is being contaminated without contain any useful information of X. Actually, DAE not only preserve the MI between encoder input and output, in Table Fad Figure we have drawn the cross- lingual MI, i.c. I(X, Y"), contained in pseudo- bitext generated by dae-only trained model, it seems that dac-only with XLM initialization have already learned initial word-to-word word translation ability. This can be further leveraged by online BT to learn towards real centence-by-senteee translation.
CONCLUSION
This paper conducts thorough comparative studies of the standard UNMT training protocol and its variants from two theoretical views, i) marginal likelihood maximization and ii) mutual information. We find that standard training implicitly optimizes ELBO so as other failed variants, indicating the importance of DAE as a regularization for helping the model learn the correct target task. Low BT reconstruction loss) is a self-evident sufficient condition for successful training of UNMT, and the high mutual information between X and Z ene(X) is necessary condition for preventing the model from degeneration. In addition, DAE loss plays the role of preserving T(X,, Z) as well as T(X, Y"); meanwhile, online BT is the main task that enables the model to actually learn from emerging cross-lingual signals unveiled by DAE in the pseudo-bitext.
losses (cross-linguus
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 451 451-462, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1000 URL https: wwwp://wwwwwww..... www aclweb.
org/ htt//aathlooyyyy111//
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. ArXiv, abs/1710. 1041, 2018.
A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In COLT" 98, 1998.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. ISSN 2307-387X.
Gino Brunner, Y. Liu, Damián Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Watten- hofer. On identifiability in transformers. arXiv: Computation and Language, 2020.
R. Caruana. Multitask learning. In Encyclopedia of Machine Learning and Data Mining, 1998.
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint urXiv:1710.04087, 2017.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and M. Baroni. What you can cram into single vector: Probing sentence embeddings for linguistic properties. ArXiv, abs/I Ne0///05.01000 2018.
T. Cover and J. Thomas. Elements of information theory. In Elements of Information Theory, 1991.
Xiangyu Duan, Baijun Ji, Hao Jia, Min Tan, Min Zhang, Boxing Chen, Weihua Luo, and Yue Zhang. Bilingual dictionary based neural machine translation without using parallel sentences. ArXiv, 12422007.02677. 2020.
Yansong Gao and Pratik Chaudhari. A free-energy principle for representation learning. In Ka- malika Chaudhuri and Ruslan Salakhutdinov (eds.), A Free-Energy Principle for Representa- tion Learning, volume 98 of Proceedings of Machine Learning Research. PMLR, 2020. URL http:://proceedings mlr. press v97 http:/999/aaaaaaaaaa htmll
Zoubin Ghahramani. Unsupervised Learning, pp. 72-112. Springer Berlin Heidelberg, Berlin, 5. httpp:
Heidelberg, 2004. ISBN 978-3-540-286550-9 doi: 10.
//doi. 100 1007 1ww-1110777---50--8555--0
Di He, Yingce Xia, Tao Qin, I.. Wang, N. Yu, T. Liu, and W. Ma. Dual learning for machine translation. In NIPS, 2016.
Junxian He, X. Wang, Graham Neubig, and Taylor Berg- Kirkpatrick. A probabilistic formulation of unsupervised text style transfer. ArXiv, abs/2002.03912, 2020.
Yunsu Kim, M. Graça, and H. Ney. When and why is unsupervised neural machine translation useless? ArXiv, abs/2004. 10581, 2020.
Diederik P. Kingma and M. Welling. Auto-encoding variational bayes. CoRR, 2444132.111 2014.
Diederik P. Kingma and M. Welling. An introduction variational autoencoders. Found. Trends 2019.
Mach. Learn., 12:3073392, 2019.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. ArXiv, abs/1901.07291 2019.
Guillaume Lample, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine transla- tion using monolingual corpora only, ArXiv, abs/1711 11177100044, 2018a.

---

Under review as a conference paper at ICLR 2021
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In EMNLP, 2018b.
M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, A. Mohamed, Omer Levy, V. Stoy- anov, and Luke Zettlemoyer. Bart: Denoising sequence-to--sequuenee pre-training for natural lan- guage generation, translation, and comprehension. ArXiv, abs/1910.13400 2020a.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettle- moyer. Pre-training via paraphrasing. arXiv preprint arXiv:2006. /5020, 2020b.
Ke Li and Jitendra Malik. Implicit maximum 1oXivv/809.09000 2018.
likelihood
estimation.
arXiv preprint
Yinhan Liu, Jiatao Gu, Naman Goyal, Xiongmin Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. ArXiv, abs/2001.08210 2020.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. When does unsupervised machine translation work? ArXiv, abs/2004.05516, 2020.
Kishore Papineni, S. Roukos, T. Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-thhoreet, probing for linguistic structure. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4609-4622, Online, July 2020.
Association for Computational Linguistics. doi: 10. 101118665/1//22000. acl-main.444 URL https:
7/ //www.aclwed org/ anthologg/22220 http.//..wwocoraa...ooooooo///// acl ma main...n.....................m 420
Victor Prokhorov, Ehsan Shareghi, Yingzhen Li, and Nigel Collier. On the importance of the kullback-leibler divergence term in variational autoencoders for text generation. ArXiv, 0bs//1999.13663 2019.
Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, and Shuai Ma. Explicit cross-lingual pre-training for unsupervised machine translation. arXiv preprint arXiv:9909.0018/ 2019.
Rico Sennrich, B. Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. ArXiv, abs/1 001.508.000 508.07909, 2016.
Kaitao Song, Xu Tan, Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In ICML, 2019.
Chau B. Tran, Yuging Tang, Xiongmin Li, and Jiatao Gu. Cross-lingual retrieval for iterative self- supervised training. ArXiv, 000/2006.095266 2020.
Jiawei Wu, Xin Wang, and William Yang Wang. Extract and edit: An alternative to for unsupervised neural machine translation. NAACL-HLT, 2019.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Unsupervised neural machine translation with weight sharing. In ACL, 2018.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 RELATED LITERATURE
1. BETTER INITIALIZATION STRATEGIES FOR UNMT
In our introduction, we have mentioned that if the encoder-decoder model has already achieved certain decent initial performance, then by using BT loss solely can reach comparative even bet- ter performance than XLM initialization. MASS (Song et al. 2019) is the first work that achieves this, they pretrain sequence-to-sequence model for predicting span of sentence given the span- dropped sentence as input. They find in their experiments that dropping half of the sentence as a contiguous span achieves the best result. Recently, mBART (Liu et al. 2020) extends the idea of denoising pre-training on Lewis et al. (2020a) to the multilingual setting. They pre-train sequence- to-sequence model monolingual corpora of 25 languages, and only use BT loss to finetune the model for UNMT. In their paper, they claim that when relying only on BT, they use constrained decoding obtain sentence on the other language at initial epochs overcome the copy issue. Conceptually, mBART actually redefines the role of DAE loss as a pre-training objective, and this largely matches our findings in Table B third row (=BT los),, that is we use DAE loss alone to train the model under certain initialization and then continue to train it solely with BT, and this matches the standard training protocol. However, when using DAE as pre-training loss from random ini- tialization, the model could only achieve 10+ BLEU far less than 30+. Some reason for this gap might be: 1) the noisy function of the DAE loss in standard training is a little bit different than that in BART; 2) we haven't 't use such larger corpora for pre- rre-training. We think to find out the reason might have significant contribution to the community on questions like: a) what is the data- scale for pre-training actually work? and b) what kind of self-supervision is more effective than others? Other works like |en et al.] (2019) directly conduct masked language model pre-training with explicitly constructed cross-lingual prediction signals, which is obtained from cross-lingual word translation techniques (Conneau et al., 2017). Although they do not apply their method on sequence pre-training and their method could be applied directly to MASS.
A.1.2 PRACTICAL ISSUES OF UNMT
Recently, several works start to criticize the practicality of the standard UNMT training. Kim et al. (020) and Marchisio et al. (2020) both claim that domain mismatch of the two monolingual corpora and the dissimilarity of the language pair correlate well with performance degradation. In Kim et al. (2020). they investigate pratical scenario with three factors: i) linguistic distance; ii) availability of large-scale bitext; iii) availability of large-scale monolingual text. They instantiate the factors with chosen language pairs and find that the standard UNMT training protocol only works for pairs with close linguistic distance and abundent monolingual text. Similar to Kim et al. (2020), Marchisio et al.] (2020) also conduct an extensive empirical evaluation of for unsupervised machine translation using dissimilar language pairs, domains and authentic low-resource languages. However, instead of using pure NMT model, they also rely on statistical machine translation model for warming up the NMT model, which is not the standard training protocol that we have investigated. In fact, although using different training protocol, they find similar observations that in Kim et al. (2020).
A.1.3 IMPROVED TRAINING PROTOCOL
Tran et al. 2020) proposes a novel cross-lingual retrieval method for finding comparative sentence pairs from monolingual corpora of the two language. They use the multilingual praitedined encoder of mBART (Liu et al. 2020) to get universal semantic representations of sentences (by doing this, they just average the token-level vectors from mBART as single vector for nearest neighbor search) for retrieval of potentially aligned sentence pairs for iterative self-supervised training. This method resembles that of Wu et al. (2019) who use the UNMT model's s own encoder representation instead of a self-supervised pre-trained encoder, and can be seen as its multilingual pre-training extension. All of the above proposed specific training methods for UNMT together with recent paraphrase-bar based pre- pre-training objective (Lewis al. 2020b) can all be thought of as implicit maximum likelihood training (Li & Malik 2018), since the retrieval phase is certain instantiation of k-Nearest-Neighbob search. Duan al. (2020) also proposes new training method by constructing mixed code pseudo-
11

---

Under review as a conference paper at ICLR 2021
bitext. Their method proves the effectiveness of using unsupervisedly induced bilingual lexicon as 'anchor' for better preventing BT from learning self self-generated noised bitext.
A.2 THE STANDARD TRAINING PROTOCOL
Please refer Algorithm for detailed description of the standard training protocol.
Algorithm 1: The Standard UNMT Training Protocol
Input:
A large-scale pre- pre-training corpus Ppt:
Two monolingual fine-tuning corpora D. and De; An untrained encoder-decoder .Mo... with de and 0d and specifically 0% 0., 0' 04 as the embeddings.
Output:
The estimated UNMT model Ma,..
1: // initialization
2: Learn joint BPE code on D, UD;;;
3: Apply BPE to the pre-training corpus Dpci
4: if pretrain 'JointEmb' then
5: Apply fastText on Ppt to learn embeddings
6: Initialize 05 and 0% with the learned joint embeddings
7: else if pretrain = 'XIM" then
8: Train Meg. with self- supervied losse(s) on P
9: Initialize the 0e,d with the learned parameters:
10: else
11: Initialize the Og and Og randomly;
12: end if
13: fine-runing
slep=0; O:
15: Sample monolingual batch b. E D... b, E D.
16: Construct denoising language modelling loss
according to [q.[]
Update model parameters using ADAM by back propagaannn Eq.
18: Sample monolingual batch b, E DD. b. E De:
19: Use Mo... to translate each batch to the other
language side b. and b.;
20: Construct nocktrranslation loss according to
Eq. on the two paired bilingual
batches (bc.b.) b.), (b., bz):
Update model parameters using ADAM by propapaaagg Eq.
22: step += 1;
23: if step=MAX.S M-X-MAXXE STEP then
24: End training;
25: else
Go line 15
27: end if
28: return Mezai
A.3 CONSTRAINED DECODING AND CROSS-LINGUAL RECERR
Let us take En-Fi translation task as an example. Since English language and French language share a large amount of vocabularies, and the sharing will be enhanced due to subword tokenization, i.c. BPE. The percentage shared vocabulary of En and are above 70% in our data setting. Thus, here we leverage a simple but effective heuristic to divide the shared vocabulary into En-dominant and Fr-dominant subones. The idea is to use a token's 'ss frequency ratio over the English monolingual and French monolingual corpus an indicator of language it mostly likely belongs to. Say given token f, we can compute its frequency ratio as r = freqen (t)/ ff/freq (t). the ratio r is larger than say t belongs English since it mostly
certain threshold and vice versa. In experiment, we set r = 2 to get reasonable vocabulary division. And then during decoding, set the logits of tokens in the other language
in English than in French,
for satisfying our constraint.
12

---

Under review as a conference paper at ICLR 2021
Table 7: Some examples of the degeneration phenomenon for dae-only and bt-only variants.
dag-only Ir-en
sre
avocat Manning déposé une plainte formelle pour les traifements subis por Manning en janvier 2011 avocat Manning déposé une plainte formelle pour les mauvais traisements subis par Manning janvier 2011.
ped
Lgt
Manning 's lawyer filed a formal objection to Manning 's treatmenn in Junuary 2011
bt-only ft-C0
src
Le télescope Hubble observé la naissance de étoiles dans galaxie spirale M83.
ped
The is the only way to ensure that the Eurogean Union is able act.
tgr
View of the Furopean Southern Observatory ESO the Chilean Andes.
src The the only way ensure the European Union
Le télescope Hubble a observé la naissance de ves etoiles dans la galaxie spirale M83.
ped
play part in the fight against temorism. This star birth was captured by the Hubble telescope in the M83 spiral galaxy..
tgt
Definition A.1. (ReeErr) Given an UNMT model M. its reconstruction error on a mono-lingual text D is defined as: jol LEED Tिं log P[a[)[[a]). Here M denotes the model's output sequence through greedy decoding or sampling.
Definition A.2. (Cross-Lingual RecErr) Given an UNMT model M, the cross- lingsll reconstruc- tion error notes the result of constrained decoding for predicting a sequence of tokens in the other language.
lingull text D is defined as: [D] Σxen 1 æ log [xx]) Here [M de-
A.4 COMPUTE ELBO
Since in our following experiments, we are going to visualize the learning curve of ELBO along the training life cycle, we should be able empirically compute the two terms of ELBO which both involve expectation over 2o (y(z)) Instead of using greedy decoding for obtaining samples, we use sampling (k=2) to compute reconstruction error and KL. divergence, and both terms are computed via Monte Carlo method. For the KL term, po(y) is a language model trained on D.. Instead of using token-level ELBO like that used in He et al. (2016), we do not normalize the ELBO values by the number of tokens in y. and use the sentence- level ELBO value for visualization.
A.5 ESTIMATING H(X), HX AND I(X, Y")
For estimating I(X, Z) which is the MI between discrete and continuous r.v.s, we use the equality I(X, Z) = HX) H(X Z), and then estimate HXX) and [X[ respectively. For the entropy of X, we use a I-gram language model on the same training corpus for training the UNMT model, and use the average token-level entropy
surrogate; as for the entropy of X|Z, train extra reconstruction model over fixed UNMT encoder to make sure we are using the representation from certain UNMT model, then we use the reconstruction model (a decoder)'s token-level entropy as a surrogate. This is motivated by Gao & Chaudhari (2020) who leverage the reconstruction error as a measure for how much information has discarded in the hidden representation, and also similar to recent probing methodology (Conneau et al. 2018).
For estimating I(X, Y"), we use the token-by-token point-wise mutual information (PMI) over some MI. Note that since we use different es-
pseudo bitext as surrogate to the contence-by-senten timators (continuous v.s. discrete) for computing II( X,, Y') and T( [(,, Z), moreover, according to Pimentel et al. 2222)), the model-based estimation of HX)) HH [X|)Z) is lower bound, so it is hard to compare between I(X, Y') and I(X, Z). But values within one estimator are comparable.
Here we give detailed introduction of how we estimate the above statistics. For estimating the first two terms, we follow the formula introduced in Pimentel et al. 2020) to estimate the entropy:
N
go (x;
log go (X
(6)
Here if = Ø is null, we use to estimate H(X); or C = Z. we use it to estimate ()).
Estimate HI( We estimate the token-level entropy instead of sentence-level, that is, X denotes token Y.V.. The qo we use is an I-gram language model on the concatenated En, Fr corpora.
Estimate H(X| Z) The to we to estimate H[X ZZ) is Transformer decoder over fixed encoder that provides hidden representations Z. So we first train a decoder over the training corpus, and then use the decoder provide the log likelihood of every token X.
13

---

Under review as a conference paper at ICLR 2021
RecErr/BLEU Correlation
RecErr/BLEU Correlation
Fr-En, 1=-0.37
70-
Fr-En, r=-0.87
15
BLEU
*
RecErr
Cross-Lingual RecErr
(8)
(b)
Figure 5: Correlation of the mono-fcross-lingaal rec_err and the final performance.
Here, to make H()) and ZZ) comparable, the estimates of (X) and HX) are calculated the same training held-out, about 50k sentences.
Estimate [x, Y") Given large amount of pseudo-bitext, we use the point-wise mutual information token-level estimates of the actual mutual information in paired sentences, that is:
1
P(
PMI(X, Y')
Σ
N
L by
((x))
14


Reviews:
[{"review_id": "vbMoVwMHiW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to explain the learning of unsupervised neural machine translation(UNMT) theoretically from two perspectives: \n1)` marginal likelihood maximization, where the objective of UNMT is analogous to ELBO and it's two terms are visualized during the training of standard UNMT. \n2) information-theoretic view, where the authors derive the success of UNMT training as maximizing the mutual information between source and target translation (sufficient condition), as well as source input and its encoder output (necessary condition). Later in the experiments, the paper presents the MI during training to support their arguments.\n\nPros\n1. This paper attempts to explain UNMT theoretically, which is a significant step for understanding UNMT inspiring the future direction of UNMT. \n2. The experiments are properly aligned with the theoretical arguments made in the paper and give good empirical explanations.\n\n\nCons:\n1. It seems that we can't really evaluate if a model meets the real sufficient condition using the practical sufficient condition.\n2. Although this paper provides propositions that what is a success UNMT training, it could've been better if showed how we can create a better UNMT system based on these definitions. \n3. ELBO-only protocol fails to train, which makes the analogy b/w *standard* and ELBO in the perspective of marginal likelihood maximization not pretty much hold thus less convincing. \n\n----\n\nQuestions:\n1. fig 4: I don't really understand why dae-only always preserves a high(est) MI during training. The authors barely explained anything in terms of this.\n2. though c in the def 2.1 & the sufficient condition is a conceptual quantity, to what extend we can consider a UNMT model is successfully learned? Even an intuitive thought is welcome.\n\n\n------\nReason for score: This paper gives a theoretical view of understanding UNMT, which is a good contribution. Some of the analogy and propositions are not well supported by the empirical results. This paper could have given some future directions based on their theoretical understanding. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hByCy33pwqN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper performs an ablative study on the two components involved in training unsupervised MT systems: 1) back-translation loss, 2) denoising autoencoding loss. It links the reconstruction loss to ELBO (where the q distribution is a back-translation model). It shows that the original loss with both the components is important for unsupervised MT and ELBO needs to be augmented with denoising autoencoding loss to be effective at training unsupervised MT models.\n\n-- The graphs show comparisons on ELBO across different models. However, ELBO is not really comparable across models by definition.\n\n-- Many findings in the paper are unsurprising and add little to our current understanding of unsupervised MT systems. For example, DAE-only collapsing to copying and BT-only collapsing to language modeling is rather expected. This can be followed from the proposed conceptualization of mutual information in the paper. The MI is highest when Y' = X! Therefore, unregularized training (with just one of the loss components) is expected to result in degenerate behavior.\n\n-- The only slightly surprising results is that ELBO on its own collapses and fails to learn anything. Technically, it does have the two components in its loss functions that make unsupervised MT work. The reasons could range from optimization issues to poor Monte-Carlo sample based approximation of expectations in ELBO (reparametrization trick or score matching is not explored in this paper for better approximation).", "extra_scores": {}, "extra_text": {}}, {"review_id": "ckdHnUfy6eR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes a closer look at the inner workings of unsupervised MT training.\n\nThe authors provide two alternate views on the backtranslation+DAE objectives used in unsupervised-MT. This interpretation sheds more light on the relationship between the two: for example, it appears that the DAE loss is critical to preserve mutual information between input and encodings, thus preventing the translation models from degenerating into unconditional language models. The authors' conclusions are well supported by experiments. However, I find that the paper falls short by being purely descriptive. While the many observations made by the authors are interesting, the reader is left hanging trying to figure out what to do with them. What is the point of these observations? Why are they useful? This paper is missing a second, more \"prescriptive\" half answering these questions.\n\n**Pros**\n\n- Sheds light on the role of the two loss terms\n- Convincing experiments: the authors claims are well supported by their experiments\n\n**Cons**\n\n- Purely descriptive: what is the takeaway? How can we improve unsupervised MT based on these findings? Better model selection? No need for denoising if the autoencoder is trained less?\n- The paper is very hard to read. I attribute this to the abundance of notations and acronyms, as well as the unwieldy phrasing. In addition, the progression of the experimental section is hard to follow. This could be remedied by summarizing and clarifying the contributions better. Right now, it reads like a list of facts about UNMT (this is also related to my first point)\n- I am not 100% convinced that the mutual information between model prediction and input is a good measure of MT success. Wouldn't that make a model that just copies its input successful (if H(x)>c )? Could the authors provide more intuition on why that is a good metric?\n\n**Remarks**\n- Grammar errors and typos: there are a lot of grammar errors and typos. I understand that the authors may not be native speakers of English. Nevertheless, the amount of errors is high enough to be distracting. I highly encourage the authors to seek proof-reading. Some examples:\n   * \"why this standard training protocol is ~possible to be~ successful\"\n   * \"some fundamental questions are still remained unknown\" -> \"some fundamental questions remain unanswered\"\n   *\"~If~ We denote Y 0 = M(X) as the random variable (r.v.) generated by the model M over X.\"\n   * \"repoitory\" -> \"repository\"\n   * \"BT LOSS IS THE MAIN TASK WHILE DAE LOSS THE AUXILIARY\" -> \"BT IS THE MAIN TASK WHILE DAE IS AUXILIARY\"\n- Style: I understand that this is a personal preference to some extent (which is why I didn't count it as a con), but the writing style is not very \"academic\": Examples: addressing the reader in the second person (2.2: \"As you can see\"), awkward/informal phrasing (1: \"we present a formal definition on what does it mean to [...], \"3: \"As promised in Section [...]\")\n- There are vague statements that should either be made more precise or removed altogether. For example in Sec. 2.3: \"Therefore, if (Y 0;X) gradually contains more and more bilingual knowledge,\" what does \"containing more bilingual knowledge\" mean for a pair of sentences?\n- I didn't quite follow the point of the fine-tuning experiments in 3.1 (the +bt loss and +standard rows in table 3). Specifically, I'm not sure I understand what the authors mean by saying that this shows that bt-only \"poisons\" the model. What is meant by \"poisoning\"? I'm not sure how this fits in the overall narrative of the paper, and I found this observation confusing more than anything else.\n\n---\n\nPost rebuttal:\n\nI thank the authors for responding to my questions. While some minor points are cleared up, I am still not completely satisfied with the rather vague notion of \"bilingual knowledge\" (what even is the \"correct axiomatic translation correspondence\", is it the (idealistic) true data distribution one tries to model?). Similarly, the use of the MI as a measure of success is still unsatisfying to me since it relies on a \"trick\" (mismatch of vocabulary between source and target) which might not even be relevant in practice (since most models use sub-words anyway).\n\nOverall, I think this research direction is promising, but I keep my recommendation the same. The paper would greatly benefit from another round of revision to clear up these points and clarify the presentation if it is to be useful to the research community.", "extra_scores": {}, "extra_text": {}}, {"review_id": "L28mnwOgfHv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to explain why UNMT using back-translation (BT) and denoising autoencoding (DAE) has been successful.\n\nI generally found this paper difficult to follow. Both the language and logic of its arguments were often not very clear to me.\n\nSection 2 presents some theoretical arguments, or analogies. \n\nSection 2.2 draws an analogy between UNMT and ELBO; the reconstruction-error terms are similar but the regularization terms are different.\n\nSection 2.3 tries to connect UNMT to mutual information, but I didn’t understand why Definition 2.1 tries to define the “success” of UNMT in terms of MI (wouldn’t setting M to the identity function achieve maximal MI?).\n\nBoth of these analogies are very rough, and I feel that this section falls short of providing understanding of UNMT.\n\nSection 3 presents some experiments to support the arguments made in Section 2.\n\nSection 3.2 shows that UNMT minimizes the ELBO term that is similar to a term in the UNMT loss and doesn’t minimize the ELBO term that isn’t similar to a term in the UNMT loss. \n\nSection 3.3 argues that using only the BT loss will cause UNMT to degenerate to generating fluent target-language sentences, while using only the DAE loss will cause UNMT to degenerate to copying the source to the target.\n\nSection 3.4.1 argues that BT is more important than DAE because UNMT performs best when BT is weighted 10 times more heavily than DAE.\n\nSection 3.4.2 argues that DAE’s job is to ensure that the encoder does not lose information.\n\nOverall, it seems to me that these findings agree with the intuitive understanding of UNMT that was already present in the UNMT literature, with the exception of the finding in 3.4.1 that the BT term should be given higher weight than the DAE term. \n\nWhile I think that explaining why UNMT works is an excellent research goal and there are some interesting ideas here, I do not think that this paper is ready for publication yet. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to explain why popular UNMT training objective components (back-translation and denoising autoencoding) are effective. The paper provides experimental analysis and draws connections with ELBO and mutual information. Reviewers generally agree that the paper's goal is worthy: trying to form a better theoretical understanding of successful approaches to UNMT. 
However, most reviewers raised serious concerns about the current draft and suggested another round of revision and resubmission. Specifically, reviewers were concerned that some of the analogies used to explain UNMT are underdeveloped. Further, reviewers pointed to issues with clarity that made some of the arguments hard to follow. Finally, one reviewer argued that many of the results are expected and agree with common understanding of UNMT in the literature, thus undermining their value to some extent. I tend to agree with reviewers that this paper is not ready for publication in its current form. Thus I recommend rejection. 
Decision: Reject