Mirror-Generative Neural Machine Translation
Authors: Zaixiang Zheng
Hao Zhou
Shujian Huang
Lei Li
Xin-Yu Dai
Jiajun Chen
Venue:  (2020)
Keywords: neural machine translation
generative model
mirror

Abstract: Training neural machine translation models (NMT) requires a large amount of parallel corpus, which is scarce for many language pairs. However, raw non-parallel corpora are often easy to obtain. Existing approaches have not exploited the full potential of non-parallel bilingual data either in training or decoding. In this paper, we propose the mirror-generative NMT (MGNMT), a single unified architecture that simultaneously integrates the source to target translation model, the target to source translation model, and two language models. Both translation models and language models share the same latent semantic space, therefore both translation directions can learn from non-parallel data more effectively. Besides, the translation models and language models can collaborate together during decoding. Our experiments show that the proposed MGNMT consistently outperforms existing approaches in a variety of scenarios and language pairs, including resource-rich and low-resource situations. 

Published as a conference paper at ICLR 2020
MIRROR-GENERATIVE NEURAL MACHINE TRANSLA- TION
Zaixiang Zheng', Hao Zhou?, Shujian Huang', Lei Li?, Xin-Yu Dai', Jiajun Chen' National Key Laboratory for Novel Software Technology, Nanjing University
zhengzx0 smail. nju. edu. en, [huangsj, daixinyu, chenj j @nju. edu. on "ByteDance AI Lab nllp, lilei 1 ab fbb.eabbbybdytcccoooom com com
ABSTRACT
Training neural machine translation models (NMT) requires large amount of par- allel corpus, which scarce for many language pairs. However, raw non-parallel corpora are often easy to obtain. Existing approaches have not exploited the full potential of non-parallel bilingual data either in training or decoding. In this paper, NMT (MGNMT), a single unified architecture
we propose the cuiror-ggneralie that simultancously integrates the source to target translation model, the target to source translation model, and two language models. Both translation models and language models share the same latent semantic space, therefore both translation directions can learn from non-parallel data more effectively. Besides, the transla- tion models and language models can collaborate together during decoding. Our experiments show that the proposed MGNMT consistently outperforms existing approaches in variety of language pairs and scenarios, including resource-rich and low-resource situations.
INTRODUCTION
Neural machine translation (NMT) systems (Sutskevrr et al. 2014 Bahdanau et al. 2015 Gehring al., 2017; Vaswani et al.) 2017) have given quite promising translation results when abundant parallel bilingual data are available for training. But obtaining such large amounts of parallel data non-trivial in most machine translation scenarios. For example, there are many low-resource language pairs (e.g.. English-to-Tanne Tamil), which lack adequate parallel data for training. Moreover, it is often difficult to adapt NMT models to other domains there only limited test domain parallel data (e.g., medical domain), due to the large domain discrepancy between the test domain and the parallel data for training (usually news-wires). For these cases where the parallel bilingual data not adequate, making the most of non- parallel bilingual data (always quite cheap to get) is crucial to achieving satisfactory translation performance.
We argue that current NMT approaches of exploiting non non-parallel data are not necessarily the best, in both training and decoding phases. For training, transaatnnltiinn (Sennrich et al.) 2016b) is the most widely used approach for exploiting monolingual data. However, fack-translation individually updates the two directions of machine translation models, which is not the most effective. Specif- ically, given monolingual data x (of source language) and y (of target language back-translation utilizes y by applying rgt2sre translation model (TMy-xx) to get predicted translations â. Then the pseudo translation pairs (ă, y) are used to update the sre2tgt translation model (TM+-yy) x can be used in the same way to update TMy-xx Note that here TMy-xx and TMx are independent and up- dated individually. Namely. each updating of TMy will not directly benefit TM Some related work like joint transsaranlttioon Zhang et al. (2018) and dual learning He et al. (2016a) introduce iterative training to make TM and M, benefit from each other implicitly and iteratively. But translation models these approaches are still independent. Ideally, gains from non-parall paraalalll data can be enlarged if we have relevant TMy-xx and TMx-y. In that case, after every updating of TMy- we may directly get better TMx-.y and vice versa, which exploits non-parallel data more effectively.
Please refer Section 2 for the notation in details.

---

Published as a conference paper at ICLR 2020
MGNNT
target LM [aC (2R,))
src2tgt TM Iaky x.2:0,,11
IOAAME
source [p(]]116)]
tat2erc [etalss::0p)
Figure 1: The graphical model of MGNMT.
Figure 2: Illustration of the mirror property of MGNMT.
For decoding, some related works Gulcehre al. 2015) propose interpolate external language models LM, (trained separately on target monolingual data) to translation model TM which in- cludes knowledge from target monolingual data for better translation. This particularly useful for domain adaptation because we may obtain better translation output quite fitting test domain (c.g., social networks), through better LMy. However, directly interpolating an independent language model in decoding maybe not the best. First, the language model used here is external, still inde- pendently learned to the translation model, thus the two models may not cooperate well by a simple interpolation mechanism (even conflict). Additionally, the language model is only included in de- coding, which is not considered in training. This leads to the inconsistency of training and decoding, which may harm the performance.
In this paper, we propose the problems for effectively exploiting non-parallel data in NMT. MGNMT is proposed to jointly train translation models (i.e., TM--yy and -My and language models (ie.. LM, and LM,) in a unified framework, which is non-trivial. Inspired by generative NMT (Shah & Barber 2018). propose to introduce a latent semantic variable z shared between x and y. Our method exploits the symmetry, or mirror property, in decomposing the conditional joint probability p(1, y =), namely:
NMT (MGNMT) to address the aforementioned
logp(r; p(x, y/2) log p(x 22) + logp(yll 2) log ppyl 22) log p(x 2)
1
2 log ((gp(y) logp(yl=) + log pugp(lyy 2) + log [((]))))
sno2egr TM.-s
targer LMy
zgr2xne TMy-x
source LMM.
The graphical model of MGNMT illustrated Figure 1 MGNMT aligns the bidirectional trans- lation models as well as language models in two languages through a shared latent semantic space (Figure 2]. so that all of them are relevant and become conditional independent given 2. In such case, MGNMT enables following advantages:
(i) For training, thanks z as a bridge, TMy->x and TM, are not independent, thus ev- ery updating of one direction will directly benefit the other direction. This improves the efficiency of using non-parallel data. 3.II
(ii) For decoding, MGNMT could naturally take advantages of its internal target-side language model, which is jointly learned with the translation model. Both of them contribute to the better generation process together. (Section 3.2)
Note that MGNMT is orthogonal to dual learning (He et al. 2016a) and joint back transsatann (Zhang et al. 2018). Translation models in MGNMT are dependent, and the two translation models could directly promote each other. Differently, dual learning and joint back-translation works in an im- plicit way, and these two approaches can also be used to further improve MGNMT. The language models used in dual learning faces the same problem as Gulcehre et al. 2015). Given GNMT, the proposed MGNMT is also non-trivial. GNMT only has a source-side language model, thus it cannot enhance decoding like MGNMT. Also, in Shah & Barber (2018), they require GNMT to share all the parameters and vocabularies between translation models so as utilize monolingual data, which is not best suited for distant language pairs. We will give more comparison in the related work.
Experiments show that MGNMT achieves competitive performance on parallel bilingual data, while it does advance training on non-parallel data. MGNMT outperforms several strong baselines in different scenarios and language pairs, including resource-rich scenarios, as well as resource-poor circumstances low-resource language translation and cross-domain translation. Moreover, we show that translation quality indeed becomes better when the jointly learned translation model and language model of MGNMT work together. We also demonstrate that MGNMT is architecture-ff which can be applied to any neural sequence model such as Transformer and RNN. These pieces of evidence verify that MGNMT meets our expectation of fully utilizing non-paralled data.
2

---

Published as a conference paper at ICLR 2020
target (,))
src2hgt TM [pts (0::1
Inference Model Lotzl [4,00]
M(x,y)
tgt22rc (19)))
5
Sample foom JF
source LM [pptec:811
Figure 3: Illustration of the architecture of MGNMT.
2 BACKGROUND AND RELATED WORK
Notation Given a pair of sentences from source and target languages, e.g., (ar, 3). we denote x as sentence of the "source" language, and sentence of the "larget" language. Additionally, use the terms "source-side" and "target-side" of a translation direction to denote the input and the output sides of it, e.g., the source-side the "tgt2sre" translation is the targer language.
Neural machine translation Conventional neural machine translation (NMT) models often adopt an encoder-decoder framework (Bahdanau et al., (015) with discriminative learning. Here NMT models aim to approximate the conditional distribution log p(ypplle [x; 0sy) over target sentence y (yI..... VL,) given a source sentence models as discriminative NMT models. Training criterion for a discriminative NMT model is to maximize the conditional log- log-likelihood log p(y] Bsy) on abundant parallel bilingual data Day (w("", (n) In I...N} of i.i.d observations.
(.1...., LL,)- Here we refer to such regular NMT
As pointed out by ZZhang et al. (2066) and Su et al. (20T8), the shared semantics between æ and y are learned in an implicit way discriminative NMT, which is insufficient to model the semantic equivalence in translation. Recently, (Shah & Barber (2018) propose a generative NMT (GNMT) by modeling the joint distribution p(a;, y) instead of ply with latent variable 2:
log p(z, u/2; (xy. log p(22; 0.) + log plyla, 0xy
where GNMT models log p(z) 02) as a source variational language model. Eikema & Aziz 2019 addition,/Chan
also propose a similar approach. In addition, [Chan et al. (2019) propose generative insertion-basee modeling for sequence, which also models the joint distribution.
Exploiting non-parallel data for NMT Both discriminative and generative NMT could not di- rectly learn from non-parallel bilingual data. To remedy this, back-translation and its variants (Sen- nrich et al. 2016b Zhang et al. 2018 exploit non-parallel bilingual data by generating synthetic parallel data. Dual learning (He et al. 2016a| Xia et al.) (017) learns from non-parallel data in round-trip game via reinforcement learning, with the help of pretrained language models. Although these methods have shown their effectiveness, the independence between translation models, and between translation and language models (dual learning) may lead to inefficiency to utilize non- parallel data for both training and decoding as MGNMT does. In the meantime, iterative learning schemes like them could also complement MGNMT.
Some other related studies exploit non-parallel bilingual data by sharing all parameters and vocab- ularies between source and target languages, by which two translation directions can be updated by either monolingual data (Dong et 2015 Johnson et al.) 2017 Firat et al.) 2016 Artetxe et al.) 2018 Lample et al. 2018a b). and GNMT well in an auto-encoder fashion. However, they may still fail to apply to distant language pairs (Zhang Komachi] 2019) such as English-to-Chiness due to the potential issues of non-overlapping alphabets, which also verified in experiments.
Additionally, as aforementioned, integrating language model is another direction to exploit monolin- gual data (Gulcehre et al. 2015 Stahlberg al. 2018 Chu & Wang 2018)) for NMT. However, this kind methods often resorts to external trained language models, which is agnostic to translation task. Besides, although GNMT contains source-side side language model, it cannot help decoding. In contrast, MGNMT jointly learns translation and language modeling probabilisticaly and can natu- rally rely both together for better generation.

---

Published as a conference paper at ICLR 2020
Algorithm Training MGNMT from Non-Parallel Data
Input: (pretrained) MGNMT MM(0). source monolingual dataset D,, target monolingual dataset D,
1: while not converge do
2: Draw source and target sentences from non-parallel data: w()) ~ Dr, y(t) ~ Dr
3: Use M to translate (x() to construct a pscudo-paraliel sentence pair (a(*)) ppamn)
4:
Compute C(x!"); 02, Oyx, 6) with (z(*), .ypeeu) by Equation 5
5: Use M to translate y to construct a pseudo-parallel sentence pair (arponn yy (0
6:
Compute ((y 0y, Oxy, .) with (apsen y(9) by Equation (4)
7: Compute the deviation V by Equation 6
8: Update parameters 0 + 10
end while
3 MIRROR GROORGGENERATIVE NEURAL MACHINE TRANSLATION
We propose the mirror-gennn generative NMT (MGNMT), a novel deep generative model which simulta- neously models pair of sre2tgt and tgt2src (variational) translation models, well pair of source and target (variational) language models, in a highly integrated way with the mirror property. As result, MGNMT learn from non- non-parallel bilingual data, and naturally interpolate its learned language model with the translation model in the decoding process.
The overall architecture of MGNMT is illustrated graphically in Figure 3 MGNMT models the joint distribution over the bilingual sentences pair by exploiting the mirror property of the joint probability: log p(z, 8|2) 2/2 p(y]g, + log p(yl2) + log p(zly, + log p(x| where the latent variable 2 (we use standard Gaussian prior 2 ~ NN0,, I)) stands for the shared semantics between x and y, serving as a bridge between all the integrated translation and language models.
3.1 TRAINING
3.1.1 LEARNING FROM PARALLEL DATA
We first introduce how to train MGNMT on regular parallel bilingual data. Given parallel bilin- gual sentence pair (I, y), we use stochastic gradient variational Bayes (SGVB) (Kingma & Welling 2014 to perform approximate maximum likelihood estimation of log p(z. y). We parameterize the approximate posterior q((|2, @) N N(pp(z)) y). Ex(x,)) Then from Equation { we can have the Evidence Lower BOund (ELBO) C(x, yr 0: 6) of the likeliood of the joint probability as:
logp(z, v) 2 C(x, y: 0 の) = E ( 2 (log p(yla, 2; Bxy) + log p(yl=; f,)
+ log p(r ( y, 2; 0yx) + log p(x|2;0,))
(2)
where (0,, 0yx, Oxy) the set of the parameters of translation and language models. The first term is the (expected) logelikel of the sentence pair. The expectation is obtained by Monte Carlo sampling. The second term is the KL-divergence between z's approximate posterior and prior reparameterization trick (Kingma & Welling 2014), we now
distributions. By relying jointly train all the components using gradient- gradient-based algorithms.
3.1.2 LEARNING FROM NON-PARALLEL DATA
Since MGNMT has intrinsically a pair of mirror translation models, we design an iterative training approach to exploit non-parallel data, in which both directions of MGNMT could benefit from the monolingual data mutually and boost each other. The proposed training process on non-parallel bilingual data is illustrated in Algorithm
Formally, given non-parallel bilingual sentences, i.e., (+) from source monolingual dataset D, = {x(+) s = 1...S} and (D) from target monolingual dataset D, = 01 I..T), we aim to maximize the lower-bounds of the likelihood of their marginal distributions mutually:
log (()(())))) + log (() (() 0z. Byx, 3) ()) 0y, Bay: x)
(3)
4

---

Published as a conference paper at ICLR 2020
where C(x(*); 8x Byx: Φ) and ((y(t); By, Bky, Φ) are the lower-bounds of the source and target marginal log- likeliikkihoods, respectively.
Let us take ((y("); ; By, Oxy. Φ) for example. Inspired by Zhang al. (2018). we sample with (()ly(0) in source language as y translation (i.e., transsatanll and obtain a secudo-paralle sentence pair (a, ( Accordingly, we give the form of ((y(";; 0y, Oxy. Φ) in Equation (4). Like- wise, Equation (5) is for ((v ; Oy, Oxy, 6). (See Appendix for the their derivation).
Cly Oxy, Φ) = Ep(elo()) [ [ (11 plg lz; 02) + log ply 2, 2; 0xy)
DKLL9 x. ' [)]p(()]]]
(4)
[(x(*); 0x 02. ②) = Eppulaton [B (o wo))
(log p( (1 22 0,) + log p(x y, 8y))
DxL(g(=) [=){[
(5)
The parameters included in Equation
can be updated via gradient-based algorithm, where the deviations are computed as Equation in mirror and integrated behavior:
3
V 80, ((x + 0.. (0y + V V (o(((x!)))) + (.))
(6)
The overall training process of exploiting non-parallel data does to some extent share a similar idea with joint back-translation (Zhang et al. 2018). But they only utilize one side of non-parallel data to update one direction of translation models for each iteration. Thanks to from the shared approxi- mate posterior q(2 [z, yr o) as a bridge, both directions of MGNMT could benefit from either of the monolingual data. Besides, MGNMT'S "back-trame translated" pseudo translations have been improved by advanced decoding process (see Equation ( ) which leads better learning effect.
3.2 DECODING
Thanks to simultaneously modeling of translation models and language models, MGNMT is now able to generate translation by the collaboration of translation and language models together. This endows MGNMT'S translation in target-side language with more domain-related fluency and quality.
Due to the mirror nature of MGNMT, the decoding process also of symmetry: given a source sentence x (or target sentence y), we want to find translation by y argmax, p(y]z) argmax, p(x, y) (x argmax, p(zly) argmax, p(z, y)), which is approximated by a mir- ror variant of the idea of EM decoding algorithm in GNMT (Shah & Barber 2018). Our decoding process is illustrated in Algorithm
Let's take the srg2tgr translation as example. Given source sentence 1) we first sam- ples an initial from the standard Gaussian prior and then obtain an initial draft translation as ý argmax, p(y|., 2); 2) this translation is iteratively refined by re-sampling this time from the approximate posterior g(2) ((222,, (z, Ø: Φ), and re-decoding with beam search by maximizing the ELBO:
ū t argmax, C(x, U; 0,
argmax, Bygtl ply ar, + log Pogp(vle 2) + p(rl() + log logp(a.ly. =)]
argmnxx, Ey(cla,see) log p(y weee, x, x) + log p(y Mees =)] + log logp(1|>) + log p( (yl,, z)
Decoding Score
Reconstructive Reranking Score
The decoding scores at each step are now given by TMx-y and LMy, which is helpful to find sen- tence not only being the translation of x but also being more possible in the target language The reconstructive reranking scores are given by LMg and TMy-x- which are employed after translation candidates are generated. MGNMT can leverage this kind of scores to sort the translation candi- dates and determine the most faithful translation the source sentence. is essentially share the
Empirically, we find that using log p( Vei, x, 2) + log pwppp((////ea a) with a coefficient 8 RJ 0.3 leads robust results, which shares the similar observations with Gulechre al. (2015).
5

---

Published as a conference paper at ICLR 2020
Algorithm 2 MGNMT Decoding with EM Algorithm
Input: MGNMT M(Ø). input sentence æ, input language
Output: x's translation
procedure: DECODING(I, I)
I: i the "target" language then
2: Swap the parameters of M(0) regarding language: (0x, Oxx + 10.. 0xy
3: end if
4: y y= (UN(((>>)
return translation y
procedure: RUN(#)
1: Sample from standard Gaussian: NO 1)
2: Generate initial draft translation: ÿ argmaxx, log pvy 2)
while not converage do
4: Sample z (a 1==1
j.k from variational distribution: 2(() g(2|2, 0)
Generate translation candidates
search by maximizing "decoding scores" in Equation Determine the best intermediate translation via ranking (0) by maximizing & Eg [log p( 200)
{0}
beam
K Eg 2 E, log
(*)) log p(ulue []
6:
D "reconstructive reranking scores" in Equation
7: end while
return translation y
Table 1: Statistics of datasets for each translation tasks.
Dataset
WMr14 EN
NIST Exe+ZH
WMr16 ENe=Ro
I EXe+DE
Parallel
3.50m
1.34m
0.62m
0.20m (TEn)
Non-parallel
0,20m (News)
Dew/eet
newstesr2013/A4
MT06/MT03
16W5test2015/16
same idea as Ng al. (2019) and Chen et al. (2019), which propose a neural noisy channel rerank- ing to incorporate reconstructive score to rerank the translation candidates. Some studies like Tu al. (017). Cheng al. (2066) also exploit this bilingual semantic equivalence as reconstruction regularization for training.
EXPERIMENT
Dataset To evaluate our model in resource-poor scenarios, we conducted experiments on WMT16 English-to/from-RRan -Rmmainnn (WMT16 EN-+RO) translation task as low-resource translation and IWSLT16 English-to/from ww/rroo-Germaaan (1WSLT16 EN++DE) parallel data of TED talk as cross-domain translation. As for resource-rich scenarios, we conducted experiments German (WMT14 EN+DD), NIST seggiish-tofrom-Chhiie all the languages, the Chinese monolingual data were extracted from LDC corpus. Table [ lists the statistics. In par- ticular, for cross-domain translation, all models trained using parallel data from TED domain, and non-parallel data from NEWS domain if applicable. We then evaluate these models on both TED and NEWS testsets, respectively.
WMT14 English-to/fromm (NIST EN++ZI) translation tasks. For the non-parallel data from News Crawl, except for NIST EN+*ZH, where
Experimental settings We implemented models the top of Transformer (Vaswani al. 2017] and RNMT (Bahdanau et al. 2015) and GNMT (Shah & Barber 2018) as well on Pytorch In this section, we only compare experimental results on Transformer implementation
For all languages pairs, sentence were encoded using byte pair encoding (Sennrich et al. 201 6a BPE) with 32k merge operations, jointly learned from the concatenation of the parallel training dataset only (except for NIST ZH-EN whose BPEs were learned separately). We used the Adam op- timizer (Kingma & Ba 2014) with the same learning rate schedule strategy as Vaswani et al.) (2017) with 4k warmup steps. Each mini-batch consists of about 4,096 source and target tokens respec-
"The original GNMT is based on RNN, and we adapted GNMT to Transformer. See Appendix for results RNMT, which consistent Transformer.

---

Published as a conference paper at ICLR 2020
Table 2: Statistics of the training datasets for each translation tasks. These values of DKL [p))] are to some extent large, which means that MGNMT does rely on the latent variable.
Datasct
WMr14 EN
NIST EN++ZH
WMT16 Exe+Ro
IWSLT16 ENE+DE
KL-annealing steps
25k
13.5k
DEL [=())]lp(=))
6.78
Table BLEU scores on low-resource translation (WMT16 6 EN++RO) and cross-domaie translation (IWSLT EN++D).. Note that for cross-domain translation, all models are trained with TED domain as parallel data, and NEWS domain as monolingual data applicable, whereas these models are evaluated on the testsets of the both domains, respectively.
LOW-RESOURCE
CROSS-DOMAIN (para. TED & monoo. NEWS)
Model
W WMr16
Ro Ro-EN
TED
NEWS
Ex-Ro
EN-DE
DE-EN
EN-DE
DE-EN
Transfommer Vaswani et al. 2017
32.1
33.2
27.5
32.8
17.1
19.9
GNMT (haah Barber 2018
32.4
33.6
28.0
33.2
17.4
20.1
GNMT-M-SSL non-parallel
Shah & Barber
2018
34.1
35.3
28.4
33.7
24.9
Transformrr+BT non non-purullll Senurich
201
35.0
27.8
Transfomer++JB non-paralle!
Zhang
34,5
35.7
28.4
33.8
25.1
Transfonmer+Dual paraflel
[e
34.6
35.7
28.5
34.0
21.8
25.3
MGNMT
32.7
33.9
28.2
33.6
17.6
20.2
MGNMT non-parallee!
34.9
36.1
28.5
34.2
22.8
26.1
tively. We trained our models on single GTX 1080ti GPU. To avoid that the approximate posterior "collapses" to the prior that learns to ignore the latent representation while (x=)(()))lp())) trends closely to zero (Bowman al., 2016 Shah & Barber 2018), we applied KL-annealing and word dropout (Bowman et al. 2016) counter this effect. For all experiments, word dropout rates were set to a constant of 0.3. Honestly, annealing KL weight somewhat tricky. Table lists our best setting of KL-annealing for each task on the development sets. The translation evaluation metric is BLEU (Papineni et al. 2002). More details are included in Appendix.
4.1 RESULTS AND DISCUSSION
As shown in Table and Table MGNMT outperforms our competitive Transformer base- line (Vaswani et al. 2017). Transformer-based GNMT (Shah & Barber 2018) and related work in both resource-poor poorr scenarios and resource-rich scenarios.
MGNMT makes better use of non-parallel data. As shown in Table 3 MGNMT outperforms our competitive Transformer baseline (Vaswani et al.] (2077) Transformer based GNMT (Shah & Barber 2018) and related work in both resource-poor scenarios.
On low-resource language pairs. The proposed MGNMT obtains a bit improvement over Trans- former and GNMT on the scarce bilingual data. Large margins of improvement are obtained by exploiting non-parallel data.
2. On cross-domain translation. To evaluate the capability of our model in the cross-domain set- ting, we first trained our model on TED data from IWSLT benchmark to simulate general-doman training, and then exposed the model NEWS non parallel bilingual data from News Crawl cessing target domain knowledge. As shown in Table 3 being invisible to target domain training data leads poor performance target domain testset (NEWS) of both Transformer and MGNMT. In this case, non-parallel data of NEWS domain contributes significantly, leading to 5.7~6.4 BLEU gains. We also conduct case study on the cross-domain translation in Appendix.
3. On Resource-rich scenarios. We also conduct regular translation experiments on two resource- rich language pairs, i.e., EN and NIST EN As shown in Table 4 MGNMT can obtain comparable results compared to discriminative baseline RNMT and generative baseline GNMT on pure parallel setting. Our model can also achieve better performance by the aid of non-parallel bilingual data than the compared previous approaches, consistent with the experimental results in resource- poor scenarios.
4. Comparison to other semi-supervised work. We compare approach with well-established approaches which are also designed for leveraging non non-parallel data, including tank-trannlttion (Sennrich et al. 2016b Bransformer+BT), joint back-translation training (Zhang al., 2018 Trans-
7

---

Published as a conference paper at ICLR 2020
Table 4: BLEU scores on resource-rich language pairs.
WMr14
NIST
Model
Ex-DE
DE-EN
EN-ZH
ZH-EN
Transformer Vaswani et al. 2017)
27.2
30.8
39.02
45.72
GNMT (Shah & Barber 2018)
27.5
31.1
40.10
46,69
GNMT-M-SSLL non-parallel Shah Barber
2018
29.7
33.5
41.73
47.70
Transfomer+BTT non-paratlel SSnnniih m
2016b
29.6
33.2
41.98
48.35
Transfomer+JBTT non non-parallei
Zhang ct al
2018
30.0
33.6
42.43
48.75
Transformer+ Dual + non-parullel
(Hl el al.
2016h
29.6
33.2
42.13
48.60
MGNMT
27.7
31.4
40.42
46,98
MGNMT non-parallel
30.3
33.8
42.56
49.05
31
27
BLEU
23
TransornurJJBT En-De
19
PransformorJJT De-kin
MGami De-En
15
0.0m
0.2m
4m
1.0m
1.5m
2.0M
En mono.
DE mono.
En & De non-parallel
@Sentences the added nan-parallel data
Figure 5: BLEU increments vs. adding one side mono- lingual (w/o interactive training) or non-parallel bilin- gual data for MGNMT on IWSLT EN+ DE tasks.
Figure BLEU vs. scales of non-
parallel data on IWSLT EN tasks.
former+JBT), multi-lingual and semi- variant of GNMT (Shah & Barber 2018 GNMT- M-SSL), and dual learning (He et al.] 2016a Transformer+Dual), As shown in Table 3 while introducing non-parallel data to either low-resource language cross-domain translation, all listed semi-supervised approaches gain substantial improvements. Among them, our MGNMT achieves the best BLEU score. Meanwhile, in resource-rich language pairs, the results are consistent. We suggest that because the jointly trained language model and translation model could work coordi- nately for decoding, MGNMT surpasses joint back translation and dual learning. Interestingly, we can see that the GNMT-M- M-SLL performs poorly on NIST EN+ZZH, which means parameters-sharte not quite suitable for distant language pair. These results indicate promising strength of boost- ing low-resource translation and exploiting domain-related knowledge from non-parallel data for cross-domain scenarios.
MGNMT is better at incorporating language model in decoding In addition, we find from Ta- ble that simple interpolation of NMT and exter- nal LM (separately trained on target-side mono- lingual data) (Gulcehre al., [2015] Transformer- LM-FUSION) only produces mild effects. This can be attributed to the unrelated probabilistic model- ing, which means that more naturally integrated solution like MGNMT is necessary.
Table 5: Incorporating LM for decoding (IWSLT task).
Model
EN-DE
DE-EN
MGNMT: dec, w/o LM
21.2
24.6
MGNMT: dec. w/ LM
22.8
26.1
Transformer
17.1
19.9
Transfomert LUSSON
18.4
Comparison with noisy channel model reranking (Ng et al., 2019) We com- pare MGNMT with the noisy channel model reranking (Ng et al. 2019 NCMR).
Table 6: Comparison with NCMR (IWSLT task).
Model
EN-DE
DE-EN
MGNMT non-parallel
22.8
26.1
NCMR uses log 2))) + A logp(=ly) +
Transformer+B3 BT NCMR (w/o)
21.8 (20.9)
25.1 (24.3)
GNMT-M-SA
NCMR (w/o)
22.4 (22.0)
25.6 (24.9)
X₂ log poy) to rerank the translation candi-
dates obtained from beam search, where X1 = 1 and X2 = 0.3, which are similar to our decoding setting. As shown in T NCMR is indeed effective and easy-to-use. But MGNMT still works better. Specifically, the advantage of the unified probabilistic modeling in MGNMT not only im- proves the effectiveness and efficiency exploiting non-parallel data for training, but also enables the use of the highly-couphe language models and bidirectional translation models testing time.
8

---

Published as a conference paper at ICLR 2020
Effects of non-parallel data. We conduct experiments regarding the scales of non-parallel data on IWSLT EN+>DE to investigate the relationship between benefits and data scales. As shown in Fig- ure 4 as the amount of non-parallel data increases, all models become strong gradually. MGNMT consistently in all data scales. Nevertheless, the growth rate de-
outperforms Transformer+J] creases probably due noise of the non-parallel data. We also investigate if one side of non-parallel data could benefit both translation directions of MGNMT. As shown in Figure S we surprisingly find that only using one side monolingual data, for example, English, could also improve English- to Go-German translation little bit, which meets our expectation.
Effects of latent variable 2. Empirically; Figure 6 shows gains become little when KL. term gets close to O (z becomes uninfor- mative), while too large KL affects negatively; meanwhile, Table are relatively reason-
x
A
2 shows that the values of DKL [xx](g())]lp(a able; besides, decoding from a zero 2 leads to large drops. These suggest that MGNMT learns meaningful bilingual latent variable, and heavily relies on it to model the translation task. Moreover, MGNMT adds further improvements to decoding by involving lan- guage models that condition the meaningful semantic z. (Table 5). These pieces of evidence show the necessity of z.
X
Kedb Deds
5.5
11
16.5
22
KL-Divergence
Figure 6: ABLEU wrt DKL.
Speed comparison MGNMT introduces extra costs for training and decoding com- pared to Transformer baseline. When be- ing trained on parallel data, MGNMT only slightly increases the training cost. However, the training cost regarding non-parallel train- ing is larger than vanilla Transformer because of the on- on-fly sampling of pseudo-translatee pairs, which is also the cost of joint back- translation and dual learning. As shown in Table 7
Table 7: Training (hours until early stop) and decod-
ing cost comparison IWSLT task. All the exper- iments are conducted on a single 1080ti GPU.
Model
Training (hrs)
Decoding
Transformer
17
1.0x
Transforer++BT
25
1.0x
GNMT-M-SSIL
~00
2.1x
Transformer+JBi
<-34
1,0x
Transforer++Dull
552
1.0x
MGNMT
~2
2.7x
we can see that on-fly sampling im- plies time-consumption, MGNMT takes more training time than joint back-translation but less than dual learning. One possible way to improve the efficiency may be to sample and save these aneudo-translate pairs in advance to the next epoch of training.
MGNMT non-parallel
45
2.7x
As for inference time, Transformer+ (BT/////DD are roughly the same vanilla Transformer because essentially they do not modify decoding phase. Apart from this, we find that the decod- ing converges 2~3 iterations for MGNMT, which leads ~2.7 time cost as the Transformer baseline. To alleviate the sacrifice of speed will be one of our future directions.
Robustness of noisy source sentence We conduct ex- periments on noisy source sentence to investigate the ro- bustness of our models compared with GNMT. The exper- imental setting is similar to Shah Barber (2018), i.c., each word of the source sentence has 30% chance of being missing. We conduct experiments on WMT En
Table 8: Comparison on robustness of noisy source sentence.
Model
GNMT
MGNMT
En-De
27.5
27.7
De-Fn
31.1
31.4
De. As shown in Table g MGNMT is more robust than GNMT with noisy source input. This may be attributed to the unified probabilistic modeling of TMs and LMs in MGNMT, where the backward translation and language models are naturally and directly leveraged to better "denoise" the noisy source input. Nevertheless, the missing content in the noisy source input still very hard to recover, leading to a large drop to all methods. Dealing with noisy input is interesting and we will leave it for future study.
En-De (noisy)
19.4
20.3
De-En (noisy)
23.0
24.1
5 CONCLUSION
In this paper, we propose the mirror-generatie NMT model (MGNMT) to make better use of non- parallel data. MGNMT jointly learns bidirectional translation models well source and target

---

Published as a conference paper at ICLR 2020
language models in a latent space of the shared bilingual semantics. In such case, both translation directions MGNMT could simultaneously benefit from non con-parlllll data. Besides, MGNMT can naturally take advantage of its learned target-side language model for decoding, which leads to better generation quality. Experiments show that the proposed MGNMT consistently outperforms other approaches in all investigated scenarios, and verify its advantages in both training and decoding. We will investigate whether MGNMT can be used in completely unsupervised setting in future work.
6
ACKNOWLEDGEMENTS
We would like thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by National Science Foundation China (No. U1836221, 61672277), National Key R&D Program of China (No. 2019QY1800 1806).
REFERENCES
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. translation. In ICLR, 2018.
Unsupervised neural machine
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In ICLR, 2015.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from a continuous space. In CoNLL, 2016.
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. Kermit: Generative insertion-based modeling for sequences. arXiv preprint arXiv: 00060//044, 2019.
Peng-Jen Chen, Jiajun Shen, Matt Le, Vishrav Chaudhary, Ahmed El- Guillaume Wenzek, Myle Ott, and MareAurelio Ranzato. Facebook ais wat19 myanmar-english translation task sub- mission. WAT 2019, 2019.
Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. supervised learning for neural machine translation. In ACL, 2016.
Semi-
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mecoder-decoder for statistical machine translation. In EMNLP, 2014.
Chenhui Chu and Ying Wang. A survey of domain adaptation for neural machine translation. In COLING, 2018.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In ACL, 2015.
Bryan Eikema and Wilker Aziz. Auto-encoding variational neural machine translation. In RepLANLP,2011
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman- Varman-Vural, and Kyunghyun Cho. Zero-resource translation with multi-lingual neural machine translation. In EMNLP, 2016.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence sequence learning. In ICML, 2017.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural ma- chine translation. arXiv preprint arXiv:1503.03533 2015.
Di He, Yingee Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie- Yan Liu, and Wei- Ying Ma. Dual learning for machine translation. Advances in Neural Information Processing Systems, pp- 820-828, 2016a.
10

---

Published as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016b.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Googles multilingual neural machine translation system: Enabling zero-shot translation. Transactions of ACL, 5, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2018a.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and MarcAurelio Ranzato. Phrase-based & neural unsupervised machine translation. In EMNLP, 2018b.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Honza Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010.
Nathan Ng. Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fairs wmt19 news translation task submission. In WMT, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(1 (666732663-2,1,, 1997.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 201
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving Neural Machine Translation Models with Monolingual Data. In ACL, 2016b.
Harshil Shah and David Barber. Generative neural machine translation. Neurips, 2018.
Felix Stahlberg, James Cross, and Veselin Stoyanov. Simple fusion: Return of the language model. In WMT, 2018
Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. Variational recurrent neural machine translation. In AAAI, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. Neural machine translation with reconstruction. In AAAI, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. In NIPS, 2017.
Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie- Yan Liu. Dual supervised learning. In ICML, 2017.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. Variational neural machine translation. In EMNLP, 2016.
Longtu Zhang and Mamoru Komachi. Chinese-japanese unsupervised neural machine translation using sub-character level information. CoRR, 0ths/1933.00144 2019.
Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. Joint training for neural machine translation models with monolingual data. In AAAI, 2018

---

Published as a conference paper at ICLR 2020
A LEARNING FROM NON-PARALLEL DATA: DERIVATION
We first take the target marginal probability logp(y(()) for example to show its deviation. Inspired by Zhang et al. (2018), we introduce ( translation x in source language as intermediate hidden variable, and decompose log p(y(t)) as:
log = log 2 p(x, = log Σ [Q(x) p(a, ((x) y01)
Q(x)
Σ lo) p(x, Q(x)
(Jenson inequality)
(8)
= Q(x log p(z, y (0)) ((x) log Q(x)
In order to make the equal sign to be valid Equation 8). (xx) must be the true tg12sre translation
probability p" (x)ly(() which can be approximated by MGNMT through y (xlly/)) = p(zly/0) =
p(x,y!")) EE([p( ((),)))()))) via Monte Carlo sampling Analogously, the intermediate hidden variable the translation of yd given by MGNMT itself (described in Section 3.2), which pro- duces pair of pseudo parallel sentences (1r, y())) This is similar the nack-translation (Sennrich) iet al. (2016b), which requires an externally separate tgt2sre NMT model to provide the synthetic data other than the unified model itself as in MGNMT.
p"(y(")
Remember that we have derived the low-bound of logp(z, v) in Equation get the lower bound of log (()) as ((y Oy, Oxy, () by
As a result, we now
log 2 (0 By. Ox, = Ep [R xa) l2 (logp(al2) + p(rly
+ log pgg (1) lo =}}]
(9)
DKula(212 sf (h) の) [])]p - logp (xly
Since p(xly((,,) 2). p(æ[2) and p" ()) are irrelevant to parameters {0y. B,y ()) 0y. Oxy. の)
could be simplified optimization purpose, namely:
Cly By. Bay. の) = (p(r (B [E 1 (log
+ log |x,
DKLL(((Ela y の) [[=)]]
(10)
The lower-bound ((y("); 8y. Bay, Φ) of (() serves as a training objective to optimize {By, Bay, Likewise, the lower bound of the likelihood the target marginal probability log p(x(+)) could be derived as:
((x()); 0x Byx. Φ) = Ep(ulo(o) Bo(sjuet) ( moorr 2 flog p(z 0x) + p( 0y0H
DKLlg(2/ ... {))((())
(11)
B IMPLEMENTATION DETAILS
We follow the GNMT (Shah & Barber 2018) to implement our MGNMT. For machine translation, we have a source sentence x ( ) and a target sentence y = (yn....,, .V1,)). As afore- mentioned, MGNMT consists variational sre2tgt and a tgt2sre translation models (TMx-xy(Oxy).
Q(x) must satisfy the following condition pl where e is a constant and does not depend on z. Given Σ, Q(x) = 1, Q(z) can be calculated (1)
$ In order make the equal sign to be valid Equation g
as:
Q(x)= p(x, (0)
p" (1)
where p² ()) denotes the true tgt2sre translation probability, while the target marginal probability p²(v)= 1.. due to the assumption that the target sentences D, i.i.d.
12

---

Published as a conference paper at ICLR 2020
TM2--y(9x)),, well as source and target variational language model(LM, (8x).LM, (8y)). These four components are conditioned on a shared inference model ( y: Φ) as approximate posterior. The overall architecture is shown in Figure B
Now, we first introduce the implementations based on RNMT, which similar to (Shah & Barber 2018
Then we introduce the Transformer-basee variant.
B. RNMT-BASED MGNMT
Let's take the target language model (0g) example. LM,(0,) mod- els the computation of pyy|2; 0y), which implemented by GRU-based (Cho et al. 2014) RNNLM (Mikolov et al. 2010) with the latent variable additional input. The probabilities p(y(z), for I..., Lx are factorized by:
Language Model
Ln
p(u22) = II plu,luer 2) = sof))ax(E((j) "F (Wy)))
(12)
where Wy is learnable linear transformation matrix, E(y;) is the embedding of yj. and the hidden state hi is computed as:
h2 12
(13)
where [i-] is concatenation operation. Likewise, the source language model LMx (0,) models p(x|=;; 0x) in mirror way.
Translation Model Let's take the src2tg! translation model TMx (Oxy) an example. TM.-y(0sy)) models the computation of plyle xr, z; 0,y), which is implemented by the variational vari-
of the widely-used RNMT (Bahdanau al. 2015). RNMT uses encoder-decoder framework. The conditional probabilities p(y|z, 2), for
Lx are factorized by:
Ly p(ul,; 2) II pls,lyea, softmax(F((u) U,$})
UU,))
(14)
where U, is a learnable linear transformation matrix, and the decoder hidden state si is computed
= 12:
(15)
Ojji softmax(a(3),.m uf))
(16)
5 G ?
(17)
where of the i-th encoder hidden state, df is the attentive context vector, which is weighted average of the source hidden states by attentive weight ojz given by the attention model a. The encoder hidden state v² is modeled by a bidirectional GRU (Schuster & Paliwal 1997| Cho et al. 2014):
f BIGRU ((x+1))
(18)
Likewise, the rgt2sre translation model TM,-y (0yz)) models p(x) 2; 02) in a mirror way.
Inference Model The inference model ((=|x, y; Φ) serves approximate posterior, which is diagonal Gaussian:
q(2|=, (VHss Eg(,v))
(19)
13

---

Published as a conference paper at ICLR 2020
We first map the sentences z, and y to sentence representation vector using bidirectional GRU, followed by an average pooling, respectively:
px
(20)
Lx
BIGRU F(y(=1))
(21)
y
where and F is the fixed-length sentence vector which is the average of the hidden states of the bidirectional GRU of and y, respectively. We then parameterize the inference model by:
qz||z, y: (it
(22)
B.2 TRANSFORMER-BAN BBSED MGNMT
Theoretically, MGNMT is independent from neural architectures we choose. As for Transformer- based MGNMT, we substitute the translation models from RNMT to Transformer (Vaswani et al.) 2017). which is also extended to condition on latent semantic. The language models and inference model remain the same.
B.3 HYPERPARAMETERS
RNMT-based MGNMT adopts 512-dimensional GRUs, 512-dimensional word embeddings, and 100 100-dimensional latent variable 2. As for Transformer- based MGNMT, we use the same configu- rations as transformer--oe are shared in the MGNMT in our implementations. For KL-annealing (Bowman et al., 2016). we multiply the KL divergence term by a constant weight, which we linearly anneal from 0 to over the initial steps of training. The KL-annealing steps sensitive to languages and the amount of dataset. We include the KL-annealing steps best results for each language in the paper.
in Vaswani et al.) (2017). The embeddings of the same language
B.4 IMPLEMENTATION OF OTHER BASELINES
Back-translan translatoon (Sennrich dual learning (He et al. 2016 6a Dual) are effective training strategies which do not depend on specific architecture. Suppose that we have monolingual data D, and Dy, and bilingual parallel Day- Note that the forward and backward TMs here are all Transformer or RNMT.
2018, JBT), and 2016b BT), joint back-translation (Zhang al.
BT: To train TM we first petrain backward translation model TMy-xx And then we use TM to translate Dx into pseudo source corpus Dx by beam search (b = 2), and De and Dy form the pseudo parallel corpus, namely Deyy. We finally use the collection Dxyy and Day to train TM The BT training for TMy is similar by alternating the language.
JBT: JBT is extension of BT in alternative and iterative manner. 1) We first pretrain TMx-y and TMy-x on Dxy. respectively. 2) We use TMy-x/TMxx-y to generate pseudo parallel corpora Drty/Day' respectively. 3) We then re-train TMx on the collec- tion of Day and Dxyy/Day'' for epoch, respectively. So now we have pair of better TM,+y and TMy-xx- 4) We finally repeat 2) and 3) with the better TMs until training converges.
Dual: 1) We first pretrain TM and TMy-x on Dxy. respectively, and LM, and LMy D, and D, respectively. Note that in the following training process, the LMs are fixed. 2) To train TMs from monolingual corpora, the rest of the training process fol- lows He al. (2016a) to iteratively and alternatively optimize the language model ward and reconstruction reward. Our implementation is heavily inspired by https: 7/qithub.
-
14

---

Published as a conference paper at ICLR 2020
C EXPERIMENTS ON RNMT
C.I. IDENTICAL SET OF EXPERIMENTS AS TRANSFORMER
We show experiments RNMT in Table 9 and TO which shows the consistent trending Transformer- based experiments. These results suggest that MGNMT is architecture-free, which can theoretically and practically be adapted to arbitrary sequence-to-sequeeee architecture.
C.2 COMPARISON WITH GNMT ITS ORIGINAL SETTING.
The lack of official GNMT codes and their manually created datasets makes it impossible for to directly compare MGNMT with GNMT in their original setting. This why we initially resorted to standard benchmark datasets. Nevertheless, we try to conduct such comparisons (Table [[]. We followed Shah Barber (2018) to conduct English-French experiments. The parallel data are provided by Multi UN corpus. Similar to Shah & Barber (2018), we created a small, medium and large amount parallel data, corresponding to 40K, 400K and 4M sentence pairs, respectively. We created validation set of 5K and test set of 10K sentence pairs. For non non-parnee con-aaallll data, used the News Crawl articles from 2009 to 2012. Note that in [hah & Barber (2018), there is monolingual corpora consisting 20.9M monolingual sentences used for English, which is too large and time- consuming. Here we used 4.5M monolingual sentences for English and French, respectively, As shown in Table I1 MGNMT still outperforms GNMT.
Table 9: BLEU scores low-resource translation (WMT16 EN++Ro), and cross- translation (IWSLT EN+>DE).
LOW-RESOURCH
CROSS DOMAIN (para. TED mono. NEWS)
Model
WMr16 Eo
TED
NEWS
EN-Ro
Ro-EN
EN-DE
DE-EN
EN-DE
DE-EN
RNMT
29.3
29.9
23.1
28.8
13.7
16.6
GNMT
30.0
30.7
23.4
29.4
13.8
16.9
GNMT-M UNMT-M-SSLL SSL + non porallalll
31.6
32.5
23.6
29.6
17.5
22.0
RNMT-BT non- parallel
31.0
31.7
23.7
29.9
16.9
21.5
RNMT-JBT + non non-parallel
31.7
32.3
24.0
30.1
17.6
22.1
RNMT-DUAL non-parallel
31.9
32.5
23.4
29.6
17.3
21.9
RNMT-LM-FUSION + non-parallel
29.5
30.3
14.1
17.0
MGNMT
30.4
31.2
29.8
13.8
MGNMT non-parallel
32.5
32.9
24.2
30.4
18.7
23.3
D CASE STUDY
As shown in Table 12 we can see that without being trained on in-domain (NEWS) non-parallel bilingual data, the baseline RNMT shows obvious style mismatches phenomenon. Although all the enhanced methods alleviate this domain inconsistency problem to some extent, MGNMT produces the best wwedddommmlaateeeee translation.
Table 10: BLEU scores resource-rich language pairs. We report results of Newstest2014 testset for WM114, and MT03 testset for NIST.
WMT14
NIST
Model
EN-DE DE
DE-EN
EN-ZH
ZH EN
RNMT
Bahdanau et al. 2015
21.9
26.0
31.77
38.10
GNMT
Shah & Barber 2018)
22.3
26.5
32.19
38.45
GNMT- SSL. + non-parallel (Shah & Barber 2018)
24.8
28.4
32.06
38.56
RNMT-B BT non-parallel (Sennrich al. 2016b)
23.6
27.9
32.98
39.21
RNMT-JBB non-paralle! (Zhang et al. 2018)
25.2
28.8
33.60
39.72
MGNMT
22.7
27.9
32.61
38.88
MGNMT non-parallel
25.7
29.4
34.07
40.25
15

---

Published as a conference paper at ICLR 2020
Table 11: BLEU scores of RNMT-based experiments on English-French using similar settings as Shah & Barber 2018 Numbers in parentheses quoted from GNMT paper. Note that because we used 4.5M English monolingual sentences instead of the original MM (too cimecconsm,ngg,, the reproduced results of "GNMT-M-SsL" are bit lower.
Model
40k
400K
4M
A
Pi-FN
RNMT
11,96
12.30
27.81
28.13
37.20
38.00
25.88
0
12.33(12.47)
13.65(13.84)
69(29998))
29,51(29441
37.64(37.977)
38.49(38.4)
26.72
0.54
GNMM-MM-SI. non-paralied
18.60 (20
18.92 (20.99)
35 90(37.37)
36.72/39 66)
38.75(39.41)
50 70:30440600 69)
31.37
5.49
MONNT
12,52
14.02
29.38
30.10
35.21
38.89
27.19
1.31
paralle!
Table 12: An example from IWSLT DE-EN cross-domain translation. In this case, all the models were first trained on parallel bilingual data from TED talks (IWSLT2016), and exposed to non- parallel bilingual data of NEWS domain (News Crawl).
Source
"Die Benzinsteuer ist einfach nicht zukunftsfähig". Lee Munnich, ein Experte für Verkelnsgesetzgebung an der Universität von Minnesota.
Reference
"The gas lax is just not sustainable". said Lee Munnich, a transportation policy expert at the University of Minnesota.
RNMT
"The gasoline tax is simply not sustainable," so Lee Munnich, expert on the University of Minnesota.
RNMT-BT
"The gas tax is simply not sustainable," so Lee Munnich, an expert on traffic legislation the University of Minnesota.
RNMT-JBT
"The gas tax just not sustainable," say Lee Munnich, expert traffic legislation
the University Minnesota.
MGNMT
"The gas tax is just not sustainable," said Lee Munnich, an traffic legislation expert at the University of Minnesota.
16


Reviews:
[{"review_id": "rkeLk2RpFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach to neural MT in which the joint (source, target) distribution is modeled as an average over two different factorizations: target given source and source given target. This gives rise to four distributions - two language models and two translation models - which are parameterized separately but conditioned on a common variational latent variable.  The model is trained on parallel data using a standard VAE approach. It can additionally be trained on non-parallel data in an approach similar to iterated back-translation at sentence-level granularity, but with language and translation model probabilities for observed sentences coupled by the latent variable. Inference iterates between sampling a latent variable given the current best hypothesis, and using beam search plus rescoring to find a new best hypothesis given the current latent variable. The approach is evaluated in several different scenarios (low- and high-resource, domain adaptation - trained on parallel data only or parallel plus monolingual data) and found to generally outperform previous work on generative NMT and iterated back-translation.\n\nStrengths: clearly written, well motivated, very comprehensive experiments comparing to relevant baselines.\n\nWeaknesses: somewhat incremental relative to Shah and Barber (Neurips 2018), results are only marginally positive, framework is probably too cumbersome to justify widespread adoption based on the results.\n\nI think the paper should be accepted. Although it’s not highly original, it ties together three strands of work in a principled way: joint models, variational approaches, and back-translation / dual learning. The increment over Shah and Barber is bolstered by the addition of back-translation, which gives substantial improvements when using non-parallel data; and to a lesser extent by the argument about the advantage of separate models for distant language pairs. Using all possible LMs and TMs coupled with a latent variable feels like an area that was inevitably going to get explored, and this paper does a good job of it. Although the gains over the baselines are not overly compelling, they are quite systematic, indicating that the advantage is probably real, albeit slight. The authors are also to be commended on their use of not just the Shah and Barber baseline, but also the back-translation-based techniques, which are generally stronger competitors when monolingual data is incorporated.\n\nFurther comments/questions:\n\nWhy are there no results for Transformer+Dual in table 4? This omission looks odd, since Transformer+Dual was the strongest baseline in table 3.\n\nPlease add implementation details for the Transformer+{BT,JBT,Dual} baselines.\n\nIt was surprising not to see robustness experiments like Shah and Barber’s dropped source words, since robustness to source noise could be one of the advantages of having an explicit model of the source sentence.\n\nA few additional suggestions for related work: noisy channel approaches (eg, The Neural Noisy Channel, Yu et al, ICLR 2017); decipherment (eg, Beyond parallel data: Joint word alignment and decipherment improves machine translation, EMNLP 2014 - yes, from SMT days, but still); other joint modeling work (KERMIT: Generative Insertion-Based Modeling for Sequences, Chan et al, 2019).\n\nConsider dropping the “1+1 > 2” metaphor. It’s not clear to me exactly what it means, or what it adds to the paper.\n\n“deviation” is used in a couple places where you probably meant “derivation”?\n\nLine 6 in algorithm 2 should use both forward and backward scores for rescoring.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1x8BrKCFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a new translation model that combines translation models in two directions and language modelss in two languages by sharing a latent semantic representation. The basic idea to joint modeling of translations conditioning on the latent representations and the parameters are learned by generating pseudo translations in two directions. Decoding is also carefully designed by interchanging sampling in two directions in a greedy fashion. Empirical results show consistent gains when compared with heuristic methods to generate pseudo data, e.g., back translation. \n\nIt is an interesting work on proposing a unified framework to translation by conditioning on a shared latent space in four models. It is not only rivaling heuristic methods to generate pseudo data, but surpassing competitive Transformer baselines.\n\nOther comment:\n\n- It is a bit confusing that MGNMT was not experimented with Transformer, though the paper and appendix describe that it is easy to use the Transformer in the MGNMT setting. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ei7P7g9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose MGNMT (Mirror Generative NMT) which aims to integrate s2t, t2s, source and target language models in a single framework. They lay out the details of their framework and motivate the need for leveraging monolingual data in both source and target directions. They also talk about related work in this space. Finally, they perform experiments on low and high resource tasks. They also investigate certain specific phenomena like effect of non-parallel data, effect of target LM during decoding, and effect of adding one side monolingual data.\n\nPros:\n- Overall, the paper was clearly written and well motivated. The authors clearly lay out their new framework and establish it for the reader.\n- The set of experiments are very detailed and the authors make sure to compare against all semi-supervised works like BT, JBT and Dual learning.\n- The set of analyses at the end was also interesting and tried to dig deeper in certain phenomena.\n- All training details and hyperparameters have been laid it in the paper.\n\nCons:\n- For all the additional complexity, this newly proposed method only slightly outperforms other semi-supervised methods like BT, JBT & Dual learning as seen in Tables 3 and 4.\n- The authors could have been more upfront about training and inference costs of their proposed framework and compared it to the other setups. For example, decoding costs 2.7x more than a vanilla transformer. A comparison of decoding and training costs of all methods would have provided the right balance between complexity and quality. This additional complexity might outweigh the gains obtained in some cases.\n\nRating Justification:\nDespite the con of added complexity, I like the formulation of the new joint framework and I think this will serve as a good starting point for others to push in this direction further. Hence, I want to see this paper accepted.\n\nMinor comments:\nlast para of section 1: first line is too big. Please break into multiple lines.\n\"Exploiting non-parallel data for NMT\" - second para, please cite Dong et. al and Johnson et. al who also share al parameters and vocab in a single model.\nPage 5, section 3.2, second para - line 1 please rephrase.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for considering translations in both directions within the framework of generative neural machine translation, significantly improving accuracy.

All three reviewers appreciated the paper, although they noted that the gains were somewhat small for the increased complexity of the model. Nonetheless, the baselines presented are already quite competitive, so improvements on these datasets are likely to never be extremely large.

Overall, I found this to be a quite nice paper, and strongly recommend acceptance, perhaps as an oral presentation.
Decision: Accept (Talk)