Title: MIRROR-GENERATIVE NEURAL MACHINE TRANSLA-TION

Abstract: Training neural machine translation models (NMT) requires a large amount of parallel corpus, which is scarce for many language pairs. However, raw non-parallel corpora are often easy to obtain. Existing approaches have not exploited the full potential of non-parallel bilingual data either in training or decoding. In this paper, we propose the mirror-generative NMT (MGNMT), a single unified architecture that simultaneously integrates the source to target translation model, the target to source translation model, and two language models. Both translation models and language models share the same latent semantic space, therefore both translation directions can learn from non-parallel data more effectively. Besides, the translation models and language models can collaborate together during decoding. Our experiments show that the proposed MGNMT consistently outperforms existing approaches in a variety of language pairs and scenarios, including resource-rich and low-resource situations.

1. INTRODUCTION:

Neural machine translation (NMT) systems (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) have given quite promising translation results when abundant parallel bilingual data are available for training. But obtaining such large amounts of parallel data is non-trivial in most machine translation scenarios. For example, there are many low-resource language pairs (e.g., English-to-Tamil), which lack adequate parallel data for training. Moreover, it is often difficult to adapt NMT models to other domains if there is only limited test domain parallel data (e.g., medical domain), due to the large domain discrepancy between the test domain and the parallel data for training (usually news-wires). For these cases where the parallel bilingual data are not adequate, making the most use of non-parallel bilingual data (always quite cheap to get) is crucial to achieving satisfactory translation performance.

We argue that current NMT approaches of exploiting non-parallel data are not necessarily the best, in both training and decoding phases. For training, back-translation (Sennrich et al., 2016b) is the most widely used approach for exploiting monolingual data. However, back-translation individually updates the two directions of machine translation models, which is not the most effective. Specifically, given monolingual data x (of source language) and y (of target language) 1 , back-translation utilizes y by applying tgt2src translation model (TM y→x ) to get predicted translations x. Then the pseudo translation pairs x, y are used to update the src2tgt translation model (TM x→y ). x can be used in the same way to update TM y→x . Note that here TM y→x and TM x→y are independent and updated individually. Namely, each updating of TM y→x will not directly benefit TM x→y . Some related work like joint back-translation Zhang et al. (2018) and dual learning He et al. (2016a) introduce iterative training to make TM y→x and TM x→y benefit from each other implicitly and iteratively. But translation models in these approaches are still independent. Ideally, gains from non-parallel data can be enlarged if we have relevant TM y→x and TM x→y . In that case, after every updating of TM y→x , we may directly get better TM x→y and vice versa, which exploits non-parallel data more effectively.

1 Please refer to Section 2 for the notation in details.

1 Published as a conference paper at ICLR 2020 For decoding, some related works (Gulcehre et al., 2015) propose to interpolate external language models LM y (trained separately on target monolingual data) to translation model TM x→y , which includes knowledge from target monolingual data for better translation. This is particularly useful for domain adaptation because we may obtain better translation output quite fitting test domain (e.g., social networks), through a better LM y . However, directly interpolating an independent language model in decoding maybe not the best. First, the language model used here is external, still independently learned to the translation model, thus the two models may not cooperate well by a simple interpolation mechanism (even conflict). Additionally, the language model is only included in decoding, which is not considered in training. This leads to the inconsistency of training and decoding, which may harm the performance.

In this paper, we propose the mirror-generative NMT (MGNMT) to address the aforementioned problems for effectively exploiting non-parallel data in NMT. MGNMT is proposed to jointly train translation models (i.e., TM x→y and TM y→x ) and language models (i.e., LM x and LM y ) in a unified framework, which is non-trivial. Inspired by generative NMT (Shah & Barber, 2018) , we propose to introduce a latent semantic variable z shared between x and y. Our method exploits the symmetry, or mirror property, in decomposing the conditional joint probability p(x, y|z), namely:

EQUATION

The graphical model of MGNMT is illustrated in Figure 1 . MGNMT aligns the bidirectional translation models as well as language models in two languages through a shared latent semantic space (Figure 2 ), so that all of them are relevant and become conditional independent given z. In such case, MGNMT enables following advantages:

(i) For training, thanks to z as a bridge, TM y→x and TM x→y are not independent, thus every updating of one direction will directly benefit the other direction. This improves the efficiency of using non-parallel data. (Section 3.1) (ii) For decoding, MGNMT could naturally take advantages of its internal target-side language model, which is jointly learned with the translation model. Both of them contribute to the better generation process together. (Section 3.2)

Note that MGNMT is orthogonal to dual learning (He et al., 2016a) and joint back-translation (Zhang et al., 2018) . Translation models in MGNMT are dependent, and the two translation models could directly promote each other. Differently, dual learning and joint back-translation works in an implicit way, and these two approaches can also be used to further improve MGNMT. The language models used in dual learning faces the same problem as Gulcehre et al. (2015) . Given GNMT, the proposed MGNMT is also non-trivial. GNMT only has a source-side language model, thus it cannot enhance decoding like MGNMT. Also, in Shah & Barber (2018) , they require GNMT to share all the parameters and vocabularies between translation models so as to utilize monolingual data, which is not best suited for distant language pairs. We will give more comparison in the related work.

Experiments show that MGNMT achieves competitive performance on parallel bilingual data, while it does advance training on non-parallel data. MGNMT outperforms several strong baselines in different scenarios and language pairs, including resource-rich scenarios, as well as resource-poor circumstances on low-resource language translation and cross-domain translation. Moreover, we show that translation quality indeed becomes better when the jointly learned translation model and language model of MGNMT work together. We also demonstrate that MGNMT is architecture-free which can be applied to any neural sequence model such as Transformer and RNN. These pieces of evidence verify that MGNMT meets our expectation of fully utilizing non-parallel data. 

2. BACKGROUND AND RELATED WORK:

Notation Given a pair of sentences from source and target languages, e.g., x, y , we denote x as a sentence of the "source" language, and y as a sentence of the "target" language. Additionally, we use the terms "source-side" and "target-side" of a translation direction to denote the input and the output sides of it, e.g., the source-side of the "tgt2src" translation is the target language.

Neural machine translation Conventional neural machine translation (NMT) models often adopt an encoder-decoder framework (Bahdanau et al., 2015) where GNMT models log p(x|z; θ x ) as a source variational language model. Eikema & Aziz (2019) also propose a similar approach. In addition, Chan et al. (2019) propose a generative insertion-based modeling for sequence, which also models the joint distribution.

Exploiting non-parallel data for NMT Both discriminative and generative NMT could not directly learn from non-parallel bilingual data. To remedy this, back-translation and its variants (Sennrich et al., 2016b; Zhang et al., 2018) exploit non-parallel bilingual data by generating synthetic parallel data. Dual learning (He et al., 2016a; Xia et al., 2017) learns from non-parallel data in a round-trip game via reinforcement learning, with the help of pretrained language models. Although these methods have shown their effectiveness, the independence between translation models, and between translation and language models (dual learning) may lead to inefficiency to utilize nonparallel data for both training and decoding as MGNMT does. In the meantime, iterative learning schemes like them could also complement MGNMT.

Some other related studies exploit non-parallel bilingual data by sharing all parameters and vocabularies between source and target languages, by which two translation directions can be updated by either monolingual data (Dong et al., 2015; Johnson et al., 2017; Firat et al., 2016; Artetxe et al., 2018; Lample et al., 2018a; b) , and GNMT as well in an auto-encoder fashion. However, they may still fail to apply to distant language pairs (Zhang & Komachi, 2019) such as English-to-Chinese due to the potential issues of non-overlapping alphabets, which is also verified in our experiments.

Additionally, as aforementioned, integrating language model is another direction to exploit monolingual data (Gulcehre et al., 2015; Stahlberg et al., 2018; Chu & Wang, 2018) for NMT. However, this kind of methods often resorts to external trained language models, which is agnostic to translation task. Besides, although GNMT contains a source-side language model, it cannot help decoding. In contrast, MGNMT jointly learns translation and language modeling probabilistically and can naturally rely on both together for a better generation.

Algorithm 1 Training MGNMT from Non-Parallel Data:

Input: (pretrained) MGNMT M(θ) , source monolingual dataset Dx, target monolingual dataset Dy 1: while not converge do 2: Draw source and target sentences from non-parallel data: x (s) ∼ Dx, y (t) ∼ Dy 3:

Use M to translate x (s) to construct a pseudo-parallel sentence pair x (s) , y (s) pseu

4::

Compute L(x (s) ; θx, θyx, φ) with x (s) , y

pseu by Equation ( 5) 5:

Use M to translate y (t) to construct a pseudo-parallel sentence pair x

(t)

pseu, y (t) 6:

Compute L(y (t) ; θy, θxy, φ) with x (t) pseu, y (t) by Equation (4) 7:

Compute the deviation ∇θ by Equation ( 6) 8:

Update parameters θ → θ + η∇θ 9: end while

3. MIRROR-GENERATIVE NEURAL MACHINE TRANSLATION:

We propose the mirror-generative NMT (MGNMT), a novel deep generative model which simultaneously models a pair of src2tgt and tgt2src (variational) translation models, as well as a pair of source and target (variational) language models, in a highly integrated way with the mirror property. As a result, MGNMT can learn from non-parallel bilingual data, and naturally interpolate its learned language model with the translation model in the decoding process.

The overall architecture of MGNMT is illustrated graphically in Figure 3 . MGNMT models the joint distribution over the bilingual sentences pair by exploiting the mirror property of the joint probability:

log p(x, y|z) = 1 2 [log p(y|x, z) + log p(y|z) + log p(x|y, z) + log p(x|z)],

where the latent variable z (we use a standard Gaussian prior z ∼ N (0, I)) stands for the shared semantics between x and y, serving as a bridge between all the integrated translation and language models.

3.1. TRAINING:



3.1.1. LEARNING FROM PARALLEL DATA:

We first introduce how to train MGNMT on a regular parallel bilingual data. Given a parallel bilingual sentence pair x, y , we use stochastic gradient variational Bayes (SGVB) (Kingma & Welling, 2014) to perform approximate maximum likelihood estimation of log p(x, y). We parameterize the approximate posterior q(z|x, y; φ) = N (µ φ (x, y), Σ φ (x, y)). Then from Equation (1), we can have the Evidence Lower BOund (ELBO) L(x, y; θ; φ) of the log-likelihood of the joint probability as:

log p(x, y) ≥ L(x, y; θ, φ) = E q(z|x,y;φ) [ 1 2 {log p(y|x, z; θ xy ) + log p(y|z; θ y ) + log p(x|y, z; θ yx ) + log p(x|z; θ x )}] (2) -D KL [q(z|x, y; φ)||p(z)]

where θ = {θ x , θ yx , θ y , θ xy } is the set of the parameters of translation and language models. The first term is the (expected) log-likelihood of the sentence pair. The expectation is obtained by Monte Carlo sampling. The second term is the KL-divergence between z's approximate posterior and prior distributions. By relying on a reparameterization trick (Kingma & Welling, 2014), we can now jointly train all the components using gradient-based algorithms.

3.1.2. LEARNING FROM NON-PARALLEL DATA:

Since MGNMT has intrinsically a pair of mirror translation models, we design an iterative training approach to exploit non-parallel data, in which both directions of MGNMT could benefit from the monolingual data mutually and boost each other. The proposed training process on non-parallel bilingual data is illustrated in Algorithm 1.

Formally, given non-parallel bilingual sentences, i.e., x (s) from source monolingual dataset D x = {x (s) |s = 1...S} and y (t) from target monolingual dataset D y = {y (t) |t = 1...T }, we aim to maximize the lower-bounds of the likelihood of their marginal distributions mutually:

EQUATION

where L(x (s) ; θ x , θ yx , φ) and L(y (t) ; θ y , θ xy , φ) are the lower-bounds of the source and target marginal log-likelihoods, respectively.

Let us take L(y (t) ; θ y , θ xy , φ) for example. Inspired by Zhang et al. (2018) , we sample x with p(x|y (t) ) in source language as y (t) 's translation (i.e., back-translation) and obtain a pseudo-parallel sentence pair x, y (t) . Accordingly, we give the form of L(y (t) ; θ y , θ xy , φ) in Equation ( 4). Likewise, Equation ( 5) is for L(y (t) ; θ y , θ xy , φ). (See Appendix for the their derivation).

EQUATION

The parameters included in Equation ( 3) can be updated via gradient-based algorithm, where the deviations are computed as Equation ( 6) in a mirror and integrated behavior:

EQUATION

The overall training process of exploiting non-parallel data does to some extent share a similar idea with joint back-translation (Zhang et al., 2018) . But they only utilize one side of non-parallel data to update one direction of translation models for each iteration. Thanks to z from the shared approximate posterior q(z|x, y; φ) as a bridge, both directions of MGNMT could benefit from either of the monolingual data. Besides, MGNMT's "back-translated" pseudo translations have been improved by advanced decoding process (see Equation ( 7)), which leads to a better learning effect.

3.2. DECODING:

Thanks to simultaneously modeling of translation models and language models, MGNMT is now able to generate translation by the collaboration of translation and language models together. This endows MGNMT's translation in target-side language with more domain-related fluency and quality.

Due to the mirror nature of MGNMT, the decoding process is also of symmetry: given a source sentence x (or target sentence y), we want to find a translation by y = argmax y p(y|x) = argmax y p(x, y) (x = argmax x p(x|y) = argmax x p(x, y)), which is approximated by a mirror variant of the idea of EM decoding algorithm in GNMT (Shah & Barber, 2018) . Our decoding process is illustrated in Algorithm 2.

Let's take the srg2tgt translation as example. Given a source sentence x, 1) we first samples an initial z from the standard Gaussian prior and then obtain an initial draft translation as ỹ = argmax y p(y|x, z); 2) this translation is iteratively refined by re-sampling z this time from the approximate posterior q(z|x, ỹ; φ), and re-decoding with beam search by maximizing the ELBO: The decoding scores at each step are now given by TM x→y and LM y , which is helpful to find a sentence y not only being the translation of x but also being more possible in the target language 2 . The reconstructive reranking scores are given by LM x and TM y→x , which are employed after translation candidates are generated. MGNMT can leverage this kind of scores to sort the translation candidates and determine the most faithful translation to the source sentence. It is to essentially share the Algorithm 2 MGNMT Decoding with EM Algorithm

ỹ ←argmax y L(x, ỹ; θ, φ) = argmax y E q(z|x

Input: MGNMT M(θ), input sentence x, input language l Output: x's translation y procedure: DECODING(x, l) 1: if l is the "target" language then 2: Swap the parameters of M(θ) regarding language: {θx, θyx} ↔ {θy, θxy} 3: end if 4: y = RUN(x) 5: return translation y procedure: RUN(x) 1: Sample z from standard Gaussian: z ∼ N (0, I) 2: Generate initial draft translation: ỹ = argmax y log p(y|x, z) 3: while not converage do 4:

Sample z = {z (k) } K s=1 from variational distribution: z (k) ∼ q(z|x, ỹ) 5:

Generate translation candidates {ŷ} via beam search by maximizing

1 K z (k) [ i log p(yi|y<i, x, z (k) ) + log p(yi|y<i, z (k) )]

"decoding scores" in Equation ( 7) 6:

Determine the best intermediate translation ỹ via ranking {ŷ} by maximizing

1 K z (k) [log p(x|z (k) )+ log p(x|y, z (k) )]

"reconstructive reranking scores" in Equation ( 7) 7: end while 8: return translation y = ỹ 

4. EXPERIMENT:

Dataset To evaluate our model in resource-poor scenarios, we conducted experiments on WMT16 English-to/from-Romanian (WMT16 EN↔RO) translation task as low-resource translation and IWSLT16 English-to/from-German (IWSLT16 EN↔DE) parallel data of TED talk as cross-domain translation. As for resource-rich scenarios, we conducted experiments on WMT14 English-to/from-German (WMT14 EN↔DE), NIST English-to/from-Chinese (NIST EN↔ZH) translation tasks. For all the languages, we use the non-parallel data from News Crawl, except for NIST EN↔ZH, where the Chinese monolingual data were extracted from LDC corpus. Table 1 lists the statistics. In particular, for cross-domain translation, all models are trained using parallel data from TED domain, and non-parallel data from NEWS domain if applicable. We then evaluate these models on both TED and NEWS testsets, respectively.

Experimental settings We implemented our models on the top of Transformer (Vaswani et al., 2017) and RNMT (Bahdanau et al., 2015) and GNMT (Shah & Barber, 2018) as well on Pytorch3 . In this section, we only compare experimental results on Transformer implementation. 4For all languages pairs, sentence were encoded using byte pair encoding (Sennrich et al., 2016a, BPE) with 32k merge operations, jointly learned from the concatenation of the parallel training dataset only (except for NIST ZH-EN whose BPEs were learned separately). We used the Adam optimizer (Kingma & Ba, 2014) with the same learning rate schedule strategy as Vaswani et al. (2017) with 4k warmup steps. Each mini-batch consists of about 4,096 source and target tokens respec- tively. We trained our models on a single GTX 1080ti GPU. To avoid that the approximate posterior "collapses" to the prior that learns to ignore the latent representation while D KL (q(z)||p(z)) trends closely to zero (Bowman et al., 2016; Shah & Barber, 2018) , we applied KL-annealing and word dropout (Bowman et al., 2016) to counter this effect. For all experiments, word dropout rates were set to a constant of 0.3. Honestly, annealing KL weight is somewhat tricky. Table 2 lists our best setting of KL-annealing for each task on the development sets. The translation evaluation metric is BLEU (Papineni et al., 2002) . More details are included in Appendix.

4.1. RESULTS AND DISCUSSION:

As shown in Table 3 and Table 4 , MGNMT outperforms our competitive Transformer baseline (Vaswani et al., 2017) , Transformer-based GNMT (Shah & Barber, 2018) and related work in both resource-poor scenarios and resource-rich scenarios.

MGNMT makes better use of non-parallel data. As shown in Table 3 , MGNMT outperforms our competitive Transformer baseline (Vaswani et al., 2017) , Transformer-based GNMT (Shah & Barber, 2018) and related work in both resource-poor scenarios.

1. On low-resource language pairs. The proposed MGNMT obtains a bit improvement over Transformer and GNMT on the scarce bilingual data. Large margins of improvement are obtained by exploiting non-parallel data. 2. On cross-domain translation. To evaluate the capability of our model in the cross-domain setting, we first trained our model on TED data from IWSLT benchmark to simulate general-domain training, and then exposed the model to NEWS non-parallel bilingual data from News Crawl to accessing target domain knowledge. As shown in Table 3 , being invisible to target domain training data leads to poor performance in target domain testset (NEWS) of both Transformer and MGNMT.

In this case, non-parallel data of NEWS domain contributes significantly, leading to 5.7∼6.4 BLEU gains. We also conduct a case study on the cross-domain translation in Appendix.

3. On Resource-rich scenarios. We also conduct regular translation experiments on two resourcerich language pairs, i.e., EN↔DE and NIST EN↔ZH. As shown in Table 4 , MGNMT can obtain comparable results compared to discriminative baseline RNMT and generative baseline GNMT on pure parallel setting. Our model can also achieve better performance by the aid of non-parallel bilingual data than the compared previous approaches, consistent with the experimental results in resource-poor scenarios. 4. Comparison to other semi-supervised work. We compare our approach with well-established approaches which are also designed for leveraging non-parallel data, including back-translation (Sennrich et al., 2016b, Transformer+BT) , joint back-translation training (Zhang et al., 2018, Trans- former+JBT), multi-lingual and semi-supervised variant of GNMT (Shah & Barber, 2018 , GNMT-M-SSL), and dual learning (He et al., 2016a, Transformer+Dual) . As shown in Table 3 , while introducing non-parallel data to either low-resource language or cross-domain translation, all listed semi-supervised approaches gain substantial improvements. Among them, our MGNMT achieves the best BLEU score. Meanwhile, in resource-rich language pairs, the results are consistent. We suggest that because the jointly trained language model and translation model could work coordinately for decoding, MGNMT surpasses joint back-translation and dual learning. Interestingly, we can see that the GNMT-M-SLL performs poorly on NIST EN↔ZH, which means parameters-sharing is not quite suitable for distant language pair. These results indicate its promising strength of boosting low-resource translation and exploiting domain-related knowledge from non-parallel data for cross-domain scenarios. MGNMT is better at incorporating language model in decoding In addition, we find from Table 5 that simple interpolation of NMT and external LM (separately trained on target-side monolingual data) (Gulcehre et al., 2015, Transformer-LM-FUSION) only produces mild effects. This can be attributed to the unrelated probabilistic modeling, which means that a more naturally integrated solution like MGNMT is necessary. Comparison with noisy channel model reranking (Ng et al., 2019) We compare MGNMT with the noisy channel model reranking (Ng et al., 2019, NCMR) . NCMR uses log p ( y|x) + λ 1 log p(x|y) + λ 2 log p(y) to rerank the translation candidates obtained from beam search, where λ 1 = 1 and λ 2 = 0.3, which are similar to our decoding setting. As shown in Table 6 , NCMR is indeed effective and easy-to-use. But MGNMT still works better. Specifically, the advantage of the unified probabilistic modeling in MGNMT not only improves the effectiveness and efficiency of exploiting non-parallel data for training, but also enables the use of the highly-coupled language models and bidirectional translation models at testing time.

Effects of non-parallel data. We conduct experiments regarding the scales of non-parallel data on IWSLT EN↔DE to investigate the relationship between benefits and data scales. As shown in Figure 4 , as the amount of non-parallel data increases, all models become strong gradually. MGNMT outperforms Transformer+JBT consistently in all data scales. Nevertheless, the growth rate decreases probably due to noise of the non-parallel data. We also investigate if one side of non-parallel data could benefit both translation directions of MGNMT. As shown in Figure 5 , we surprisingly find that only using one side monolingual data, for example, English, could also improve Englishto-German translation a little bit, which meets our expectation. Effects of latent variable z. Empirically, Figure 6 shows gains become little when KL term gets close to 0 (z becomes uninformative), while too large KL affects negatively; meanwhile, Table 2 shows that the values of D KL [q(z)||p(z)] are relatively reasonable; besides, decoding from a zero z leads to large drops. These suggest that MGNMT learns a meaningful bilingual latent variable, and heavily relies on it to model the translation task. Moreover, MGNMT adds further improvements to decoding by involving language models that condition on the meaningful semantic z. (Table 5 ). These pieces of evidence show the necessity of z.

Table 7 : Training (hours until early stop) and decoding cost comparison on IWSLT task. All the experiments are conducted on a single 1080ti GPU. As for inference time, Transformer+{BT/JBT/Dual} are roughly the same as vanilla Transformer because essentially they do not modify decoding phase. Apart from this, we find that the decoding converges at 2∼3 iterations for MGNMT, which leads to ∼2.7× time cost as the Transformer baseline. To alleviate the sacrifice of speed will be one of our future directions. Robustness of noisy source sentence We conduct experiments on noisy source sentence to investigate the robustness of our models compared with GNMT. The experimental setting is similar to Shah & Barber (2018) , i.e., each word of the source sentence has a 30% chance of being missing. We conduct experiments on WMT En-De. As shown in Table 8 , MGNMT is more robust than GNMT with noisy source input. This may be attributed to the unified probabilistic modeling of TMs and LMs in MGNMT, where the backward translation and language models are naturally and directly leveraged to better "denoise" the noisy source input. Nevertheless, the missing content in the noisy source input is still very hard to recover, leading to a large drop to all methods. Dealing with noisy input is interesting and we will leave it for future study.

EQUATION

5. CONCLUSION:

In this paper, we propose the mirror-generative NMT model (MGNMT) to make better use of nonparallel data. MGNMT jointly learns bidirectional translation models as well as source and target language models in a latent space of the shared bilingual semantics. In such a case, both translation directions of MGNMT could simultaneously benefit from non-parallel data. Besides, MGNMT can naturally take advantage of its learned target-side language model for decoding, which leads to better generation quality. Experiments show that the proposed MGNMT consistently outperforms other approaches in all investigated scenarios, and verify its advantages in both training and decoding. We will investigate whether MGNMT can be used in completely unsupervised setting in future work.

A LEARNING FROM NON-PARALLEL DATA: DERIVATION

We first take the target marginal probability log p(y (t) ) for example to show its deviation. Inspired by Zhang et al. (2018) , we introduce y (t) 's translation x in source language as intermediate hidden variable, and decompose log p(y (t) ) as:

log p(y (t) ) = log x p(x, y (t) ) = log x Q(x) p(x, y (t) ) Q(x) ≥ x Q(x) log p(x, y (t) ) Q(x) (Jenson inequality) (8) = x Q(x) log p(x, y (t) ) -Q(x) log Q(x)

In order to make the equal sign to be valid in Equation ( 8), Q(x) must be the true tgt2src translation probability p * (x|y (t) ) , which can be approximated by MGNMT through p * (x|y t) |z)] via Monte Carlo sampling 5 . Analogously, the intermediate hidden variable x is the translation of y (t) given by MGNMT itself (described in Section 3.2), which produces a pair of pseudo parallel sentences x, y (t) . This is similar to the back-translation (Sennrich et al., 2016b) , which requires an externally separate tgt2src NMT model to provide the synthetic data other than the unified model itself as in MGNMT.

EQUATION

Remember that we have derived the low-bound of log p(x, y) in Equation ( 2). As a result, we now get the lower bound of log p(y (t) ) as L(y (t) ; θ y , θ xy , φ) by

EQUATION

-D KL [q(z|x, y 

EQUATION

The lower-bound L(y (t) ; θ y , θ xy , φ) of log p(y (t) ) serves as a training objective to optimize {θ y , θ xy , φ}. Likewise, the lower bound of the likelihood on the target marginal probability log p(x (s) ) could be derived as:

EQUATION

B IMPLEMENTATION DETAILS:

We follow the GNMT (Shah & Barber, 2018) to implement our MGNMT. For machine translation, we have a source sentence x = x 1 , . . . , x Lx and a target sentence y = y 1 , . . . , y Ly . As aforementioned, MGNMT consists of a variational src2tgt and a tgt2src translation models (TM x→y (θ xy ),

5 In order to make the equal sign to be valid in Equation 8, Q(x) must satisfy the following condition

p(x,y (t) ) Q(x)

= c, where c is a constant and does not depend on x. Given x Q(x) = 1, Q(x) can be calculated as:

Q(x) = p(x, y (t) ) c = p(x, y (t) ) x p(x, y (t) ) = p * (x|y (t) )

where p * (x|y (t) ) denotes the true tgt2src translation probability, while the target marginal probability p * (y) = c = 1 T due to the assumption that the target sentences in Dy are i.i.d.

TM x→y (θ yx )), as well as a source and a target variational language model(LM x (θ x ),LM y (θ y )). These four components are conditioned on a shared inference model q(z|x, y; φ) as approximate posterior.

The overall architecture is shown in Figure 3 . Now, we first introduce the implementations based on RNMT, which is similar to (Shah & Barber, 2018) . Then we introduce the Transformer-based variant.

B.1 RNMT-BASED MGNMT:

Language Model Let's take the target language model LM y (θ y ) as an example. LM y (θ y ) models the computation of p(y|z; θ y ), which is implemented by a GRU-based (Cho et al., 2014) RNNLM (Mikolov et al., 2010) with the latent variable z as additional input. The probabilities p(y|z), for i = 1, ..., L x are factorized by:

EQUATION

where W y is a learnable linear transformation matrix, E(y j ) is the embedding of y j , and the hidden state h y j is computed as:

EQUATION

where [•; •] is a concatenation operation.

Likewise, the source language model LM x (θ x ) models p(x|z; θ x ) in a mirror way.

Translation Model Let's take the src2tgt translation model TM x→y (θ xy ) as an example. TM x→y (θ xy ) models the computation of p(y|x, z; θ xy ), which is implemented by the variational variant of the widely-used RNMT (Bahdanau et al., 2015) . RNMT uses an encoder-decoder framework. The conditional probabilities p(y|x, z), for i = 1, ..., L x are factorized by: p(y|x, z) = Ly j p(y j |y <t , x, z) = softmax(E(y j ) U y s y j )

where U y is a learnable linear transformation matrix, and the decoder hidden state s y j is computed as: sy j = GRU(s y t-1 , [z; E(y t-1 )]) (15)

EQUATION

s y j = GRU(s y j , c y j ) '

where v x i is the i-th encoder hidden state, c y j is the attentive context vector, which is a weighted average of the source hidden states by attentive weight α ji given by the attention model a. The encoder hidden state v x i is modeled by a bidirectional GRU (Schuster & Paliwal, 1997; Cho et al., 2014) :

EQUATION

Likewise, the tgt2src translation model TM x→y (θ yx )) models p(x|z; θ x ) in a mirror way.

Inference Model The inference model q(z|x, y; φ) serves as an approximate posterior, which is a diagonal Gaussian:

q(z|x, y; φ) = N (µ φ (x, y), Σ φ (x, y))

Table 11 : BLEU scores of RNMT-based experiments on English-French using similar settings as Shah & Barber (2018) . Numbers in parentheses are quoted from GNMT paper. Note that because we used 4.5M English monolingual sentences instead of the original 20.9M (too time-consuming), the reproduced results of "GNMT-M-SSL" are a bit lower. Table 12 : An example from IWSLT DE-EN cross-domain translation. In this case, all the models were first trained on parallel bilingual data from TED talks (IWSLT2016), and exposed to nonparallel bilingual data of NEWS domain (News Crawl).

Source "Die Benzinsteuer ist einfach nicht zukunftsfähig", so Lee Munnich, ein Experte für Verkehrsgesetzgebung an der Universität von Minnesota. Reference "The gas tax is just not sustainable", said Lee Munnich, a transportation policy expert at the University of Minnesota. RNMT "The gasoline tax is simply not sustainable," so Lee Munnich, an expert on the University of Minnesota. RNMT-BT "The gas tax is simply not sustainable," so Lee Munnich, an expert on traffic legislation at the University of Minnesota . RNMT-JBT "The gas tax is just not sustainable," say Lee Munnich, an expert on traffic legislation at the University of Minnesota . MGNMT "The gas tax is just not sustainable," said Lee Munnich, an traffic legislation expert at the University of Minnesota.

:

Empirically, we find that using log p(yi|y<i, x, z) + β log p(yi|y<i, z) with a coefficient β ≈ 0.3 leads to more robust results, which shares the similar observations withGulcehre et al. (2015).

The original GNMT is based on RNN, and we adapted GNMT to Transformer.

See Appendix for results on RNMT, which is consistent to Transformer.

Figures and Tables:

Figure 1: The graphical model of MGNMT.

Figure 3: Illustration of the architecture of MGNMT.

Figure 4: BLEU vs. scales of nonparallel data on IWSLT EN↔DE tasks.

Figure 5: BLEU increments vs. adding one side monolingual (w/o interactive training) or non-parallel bilingual data for MGNMT on IWSLT EN↔DE tasks.

Figure 6: ∆BLEU wrt D KL .

2.7× Speed comparison MGNMT introduces extra costs for training and decoding compared to Transformer baseline. When being trained on parallel data, MGNMT only slightly increases the training cost. However, the training cost regarding non-parallel training is larger than vanilla Transformer because of the on-fly sampling of pseudo-translation pairs, which is also the cost of joint backtranslation and dual learning. As shown in Table 7, we can see that on-fly sampling implies time-consumption, MGNMT takes more training time than joint back-translation but less than dual learning. One possible way to improve the efficiency may be to sample and save these pseudo-translation pairs in advance to the next epoch of training.

with discriminative learning. Here NMT models aim to approximate the conditional distribution log p(y|x; θ xy ) over a target sentence y = y 1 , . . . , y Ly given a source sentence x = x 1 , . . . , x Lx . Here we refer to such regular NMT models as discriminative NMT models. Training criterion for a discriminative NMT model is to maximize the conditional log-likelihood log p(y|x; θ xy ) on abundant parallel bilingual data D xy = {x (n) , y (n) |n = 1...N } of i.i.d observations.

Table content: <table><tr><td>As pointed out by Zhang et al. (2016) and Su et al. (2018), the shared semantics z between x and</td></tr><tr><td>y are learned in an implicit way in discriminative NMT, which is insufficient to model the semantic</td></tr><tr><td>equivalence in translation. Recently, Shah &amp; Barber (2018) propose a generative NMT (GNMT) by</td></tr><tr><td>modeling the joint distribution p(x, y) instead of p(y|x) with a latent variable z:</td></tr><tr><td>log p(x, y|z; θ = {θ x , θ xy }) = log p(x|z; θ x ) + log p(y|x, z; θ xy )</td></tr></table>

Statistics of datasets for each translation tasks.

Table content: <table><tr><td>Dataset</td><td>WMT14 EN↔DE</td><td>NIST EN↔ZH</td><td>WMT16 EN↔RO</td><td>IWSLT16 EN↔DE</td></tr><tr><td>Parallel</td><td>4.50m</td><td>1.34m</td><td>0.62m</td><td>0.20m (TED)</td></tr><tr><td>Non-parallel</td><td>5.00m</td><td>1.00m</td><td>1.00m</td><td>0.20m (NEWS)</td></tr><tr><td>Dev/Test</td><td>newstest2013/14</td><td>MT06/MT03</td><td>newstest2015/16</td><td>tst13/14&amp;newstest2014</td></tr></table>

Statistics of the training datasets for each translation tasks. These values of D KL [q(z)||p(z)] are to some extent large, which means that MGNMT does rely on the latent variable.

Table content: <table><tr><td>Dataset</td><td>WMT14 EN↔DE</td><td>NIST EN↔ZH</td><td>WMT16 EN↔RO</td><td>IWSLT16 EN↔DE</td></tr><tr><td>KL-annealing steps</td><td>35k</td><td>13.5k</td><td>8k</td><td>4k</td></tr><tr><td>DKL[q(z)||p(z)]</td><td>6.78</td><td>8.26</td><td>6.36</td><td>7.81</td></tr></table>

BLEU scores on low-resource translation (WMT16 EN↔RO), and cross-domain translation (IWSLT EN↔DE). Note that for cross-domain translation, all models are trained with TED domain as parallel data, and NEWS domain as monolingual data if applicable, whereas these models are evaluated on the testsets of the both domains, respectively.

Table content: <table><tr><td/><td colspan="2">LOW-RESOURCE</td><td colspan="4">CROSS-DOMAIN (para. TED &amp; mono. NEWS)</td></tr><tr><td>Model</td><td colspan="2">WMT16 EN↔RO</td><td colspan="2">TED</td><td/><td>NEWS</td></tr><tr><td/><td>EN-RO</td><td>RO-EN</td><td>EN-DE</td><td>DE-EN</td><td>EN-DE</td><td>DE-EN</td></tr><tr><td>Transformer (Vaswani et al., 2017)</td><td>32.1</td><td>33.2</td><td>27.5</td><td>32.8</td><td>17.1</td><td>19.9</td></tr><tr><td>GNMT (Shah &amp; Barber, 2018)</td><td>32.4</td><td>33.6</td><td>28.0</td><td>33.2</td><td>17.4</td><td>20.1</td></tr><tr><td>GNMT-M-SSL + non-parallel (Shah &amp; Barber, 2018)</td><td>34.1</td><td>35.3</td><td>28.4</td><td>33.7</td><td>22.0</td><td>24.9</td></tr><tr><td>Transformer+BT + non-parallel (Sennrich et al., 2016b)</td><td>33.9</td><td>35.0</td><td>27.8</td><td>33.3</td><td>20.9</td><td>24.3</td></tr><tr><td>Transformer+JBT + non-parallel (Zhang et al., 2018)</td><td>34.5</td><td>35.7</td><td>28.4</td><td>33.8</td><td>21.9</td><td>25.1</td></tr><tr><td>Transformer+Dual + non-parallel (He et al., 2016a)</td><td>34.6</td><td>35.7</td><td>28.5</td><td>34.0</td><td>21.8</td><td>25.3</td></tr><tr><td>MGNMT</td><td>32.7</td><td>33.9</td><td>28.2</td><td>33.6</td><td>17.6</td><td>20.2</td></tr><tr><td>MGNMT + non-parallel</td><td>34.9</td><td>36.1</td><td>28.5</td><td>34.2</td><td>22.8</td><td>26.1</td></tr></table>

BLEU scores on resource-rich language pairs.

Table content: <table><tr><td>Model</td><td colspan="2">WMT14 EN-DE DE-EN</td><td colspan="2">NIST EN-ZH ZH-EN</td></tr><tr><td>Transformer (Vaswani et al., 2017)</td><td>27.2</td><td>30.8</td><td>39.02</td><td>45.72</td></tr><tr><td>GNMT (Shah &amp; Barber, 2018)</td><td>27.5</td><td>31.1</td><td>40.10</td><td>46.69</td></tr><tr><td>GNMT-M-SSL + non-parallel (Shah &amp; Barber, 2018)</td><td>29.7</td><td>33.5</td><td>41.73</td><td>47.70</td></tr><tr><td>Transformer+BT + non-parallel (Sennrich et al., 2016b)</td><td>29.6</td><td>33.2</td><td>41.98</td><td>48.35</td></tr><tr><td>Transformer+JBT + non-parallel (Zhang et al., 2018)</td><td>30.0</td><td>33.6</td><td>42.43</td><td>48.75</td></tr><tr><td>Transformer+Dual + non-parallel (He et al., 2016b)</td><td>29.6</td><td>33.2</td><td>42.13</td><td>48.60</td></tr><tr><td>MGNMT</td><td>27.7</td><td>31.4</td><td>40.42</td><td>46.98</td></tr><tr><td>MGNMT + non-parallel</td><td>30.3</td><td>33.8</td><td>42.56</td><td>49.05</td></tr></table>

Incorporating LM for decoding (IWSLT task).

Table content: <table><tr><td>Model</td><td>EN-DE</td><td>DE-EN</td></tr><tr><td>MGNMT: dec. w/o LM</td><td>21.2</td><td>24.6</td></tr><tr><td>MGNMT: dec. w/ LM</td><td>22.8</td><td>26.1</td></tr><tr><td>Transformer</td><td>17.1</td><td>19.9</td></tr><tr><td>Transformer+LM-FUSION</td><td>18.4</td><td>21.1</td></tr></table>

Comparison with NCMR (IWSLT task).

Table content: <table><tr><td>Model</td><td>EN-DE</td><td>DE-EN</td></tr><tr><td>MGNMT + non-parallel</td><td>22.8</td><td>26.1</td></tr><tr><td colspan="3">Transformer+BT w/ NCMR (w/o) 21.8 (20.9) 25.1 (24.3)</td></tr><tr><td>GNMT-M-SSL w/ NCMR (w/o)</td><td colspan="2">22.4 (22.0) 25.6 (24.9)</td></tr></table>

Comparison on robustness of noisy source sentence.

Table content: <table><tr><td>Model</td><td>GNMT</td><td>MGNMT</td></tr><tr><td>En-De</td><td>27.5</td><td>27.7</td></tr><tr><td>De-En</td><td>31.1</td><td>31.4</td></tr><tr><td>En-De (noisy)</td><td>19.4</td><td>20.3</td></tr><tr><td>De-En (noisy)</td><td>23.0</td><td>24.1</td></tr></table>

Peer review (iclr2020):
This paper proposes an approach to neural MT in which the joint (source, target) distribution is modeled as an average over two different factorizations: target given source and source given target. This gives rise to four distributions - two language models and two translation models - which are parameterized separately but conditioned on a common variational latent variable. The model is trained on parallel data using a standard VAE approach. It can additionally be trained on non-parallel data in an approach similar to iterated back-translation at sentence-level granularity, but with language and translation model probabilities for observed sentences coupled by the latent variable. Inference iterates between sampling a latent variable given the current best hypothesis, and using beam search plus rescoring to find a new best hypothesis given the current latent variable. The approach is evaluated in several different scenarios (low- and high-resource, domain adaptation - trained on parallel data only or parallel plus monolingual data) and found to generally outperform previous work on generative NMT and iterated back-translation. Strengths: clearly written, well motivated, very comprehensive experiments comparing to relevant baselines. Weaknesses: somewhat incremental relative to Shah and Barber (Neurips 2018), results are only marginally positive, framework is probably too cumbersome to justify widespread adoption based on the results. I think the paper should be accepted. Although it’s not highly original, it ties together three strands of work in a principled way: joint models, variational approaches, and back-translation / dual learning. The increment over Shah and Barber is bolstered by the addition of back-translation, which gives substantial improvements when using non-parallel data; and to a lesser extent by the argument about the advantage of separate models for distant language pairs. Using all possible LMs and TMs coupled with a latent variable feels like an area that was inevitably going to get explored, and this paper does a good job of it. Although the gains over the baselines are not overly compelling, they are quite systematic, indicating that the advantage is probably real, albeit slight. The authors are also to be commended on their use of not just the Shah and Barber baseline, but also the back-translation-based techniques, which are generally stronger competitors when monolingual data is incorporated. Further comments/questions: Why are there no results for Transformer+Dual in table 4? This omission looks odd, since Transformer+Dual was the strongest baseline in table 3. Please add implementation details for the Transformer+{BT,JBT,Dual} baselines. It was surprising not to see robustness experiments like Shah and Barber’s dropped source words, since robustness to source noise could be one of the advantages of having an explicit model of the source sentence. A few additional suggestions for related work: noisy channel approaches (eg, The Neural Noisy Channel, Yu et al, ICLR 2017); decipherment (eg, Beyond parallel data: Joint word alignment and decipherment improves machine translation, EMNLP 2014 - yes, from SMT days, but still); other joint modeling work (KERMIT: Generative Insertion-Based Modeling for Sequences, Chan et al, 2019). Consider dropping the “1+1 > 2” metaphor. It’s not clear to me exactly what it means, or what it adds to the paper. “deviation” is used in a couple places where you probably meant “derivation”? Line 6 in algorithm 2 should use both forward and backward scores for rescoring.