Venue: NAACL 2021

# Almost Free Semantic Draft for Neural Machine Translation

Xi Ai College of Computer Science Chongqing University barid.x.ai@gmail.comBin Fang College of Computer Science Chongqing University fb@cqu.edu.cn

###### Abstract

Translation quality can be improved by global information from the required target sentence because the decoder can understand both past and future information. However, the model needs additional cost to produce and consider such global information. In this work, to inject global information but also save cost, we present an efficient method to sample and consider a semantic draft as global information from semantic space for decoding with almost free of cost. Unlike other successful adaptations, we do not have to perform an EM-like process that repeatedly samples a possible semantic from the semantic space. Empirical experiments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. We will open all our source code on GitHub.

## 1 Introduction

Successful NMT (Neural Machine Translation) Vaswani et al. (2017); Bahdanau et al. (2015); Johnson et al. (2017); Ng et al. (2019) can translate sentences through left to right or through right to left. However, there is one critical limitation in this diagram. That is, the decoder can only have access to directional information (left-to-right or right-to-left) when processing auto-regressive Graves (2013).

To alleviate this pain, there have been three successful lines. **1)**_Generative NMT_: Zheng et al. (2020); Shah and Barber (2018); Su et al. (2018); Zhang et al. (2016); Eikema and Aziz (2019) adapt VAE (variational auto-encoder) Altieri and Duvenaud (2015); Kingma and Ba (2015); Bowman et al. (2016) for NMT that is trained in generative model settings, modeling the semantics of the source and target sentences in latent space. **2)**_Deliberation_: since the problem is caused by the one-pass process of decoding in the auto-regression process Xia et al. (2017) present a framework to predict a guess target sentence in the first-pass and jointly considers the encoding and the guess target sentence in the second-pass. **3)**_Soft-prototype_: Wang et al. (2019) present a framework to generate a prototype on the encoder side and then the decoder can jointly use the encoding and the prototype. Although empirical results show the previous methods can successfully inject global information into the decoder, these methods either introduce computational complexity to the encoder-decoder architecture or employ an EM-like process in inferring, thus requiring even more than 100% additional time to produce and consider global information in inferring.

In this work, we present an efficient method to sample and consider a semantic draft as global information for decoding with almost free of cost, following the line of generative NMT. Concretely, we sample the semantic draft from semantic space that is a Gaussian inference model with learnable parameters. In the classic utilization of the semantic space, e.g., generative NMT, inferring needs to work with the EM-like process that could degrade the inference efficiency significantly. To mitigate the degradation but still use the semantic space, we train the encoder of NMT in multilingual settings and simultaneously train a cross-lingual generator to obtain an approximation of the target-sentence semantic, hence modeling the required semantic space from the approximation and the source-sentence semantic. In inferring, based on the source-sentence semantic and an approximation made by the cross-lingual generator, the semantic draft can be sampled from the semantic space in a one-shot style. Once the semantic draft has been sampled, we aggregate the semantic draft and the encoding so that the variational decoder can simply decompose the aggregation.

We train the model in generative settings with additional loss of KL-divergence that is used to optimize the semantic space, similar to generative NMT training Zheng et al. (2020); Shah and Barber (

2018; Su et al., 2018; Zhang et al., 2016; Eikema and Aziz, 2019) and VAE training Altieri and Duvenaud, 2015; Kingma and Ba, 2015; Bowman et al., 2016). Our work can build upon Transformer Vaswani et al. (2017), LSTM/GRU Hochreiter and Schmidhuber, 1997; Chung et al., 2014) and Convolutional sequence Gehring et al. (2017). In this work, we use Transformer as an example to present our idea, evaluating our method on common translation tasks and 5 more comprehensive experiments.

Our empirical study shows that, compared to previously successful methods, our method can achieve competitive performance and has a clear advantage in inference efficiency. Since we do not change the architecture of the NMT model, our model is compatible with common technics in NMT.

## 2 Background

Notation\(x\) and \(y\) denotes word embeddings in the source language \(L_{1}\) and the target language \(L_{2}\), respectively. \(X=(x_{0},x_{2},...,x_{n})\in R^{N\times d}\) and \(Y=(y_{0},y_{2},...,y_{m})\in R^{M\times d}\) are the sentences sampled from corpora in \(L_{1}\) and \(L_{2}\) respectively, where \(N\) and \(M\) are the sequence length and \(d\) is the word embedding dimension. \(X\) and \(Y\) are parallel sentences that are used in our supervised training. The translation task \(X\to Y\) is denoted as \(Y=Dec(Enc(X))\), where \(Dec\) and \(Enc\) jointly construct an encoder-decoder model. \(s\) and \(t\) represent the source-sentence semantic for \(X\) and the target-sentence semantic for \(Y\) in translation, respectively. \(z\) is a latent variable to represent a semantic draft, sampled from the semantic space.

NMT Vaswani et al. (2017); Bahdanau et al. (2015); Johnson et al. (2017); Ng et al. (2019) utilizes \(seq2seq\) learning Sutskever et al. (2014) and auto-regressive Graves (2013) to facilitate training and inferring. Concretely, the current translation \(y_{j}\) at time-step \(j\) is conditional on \(Enc(X)\) and \(y_{<j}\), where \(y_{<j}\) is the previous translation before \(j\). The intrinsic problem is caused because the translation \(y_{j}\) can only consider \(y<j\) without considering \(y>j\) . Intuitively, a semantic draft or global information including \(y<j\) and \(y>j\) can benefit the translation \(y_{j}\) because the translation can be consistent with neighboring information.

Some impressive methods have been proposed to produce and consider a draft providing global information for better translation quality. _1)_ Generative NMT (including variational NMT) Shah and Barber (2018); Zheng et al. (2020); Su et al. (2018); Zhang et al. (2016); Eikema and Aziz (2019) study latent and continuous space of semantic Bowman et al. (2016) for NMT, which can sample \(z\). These methods inject \(z\) into NMT to provide global information for better translation. Meanwhile, the encoder is encouraged to consider \(z\). In this manner, generative NMT models the joint probability \(P_{nmt}(X,Y,z)=p(z)p(X|z)p_{nmt}(Y|X,z)\) in training. For inferring, the model utilizes the EM-like process to maximize a lower bound on \(log(p(X,Y))\) by repeatedly guessing or predicting possible \(Y\) and resampling \(z\). However, compared to NMT without \(z\), generative NMT costs over 100% additional time in inferring typically. _2)_ Sharing the same idea of the reconsideration of the current translation, _Deliberation_ Xia et al. (2017) is proposed to deliberate the complete output of the first-pass decoding as the attention context of the second-pass decoding. With the _Deliberation_, the final translation is based on the understanding of a possible translation in the target language. Although _Deliberation_ is employed without the EM-like process, which is more efficient than generative NMT in inferring, the doubled pass increases the time of auto-regressive in decoding that costs 80% additional time in inferring. _3)_Wang et al. (2019) further consider the inference efficiency and the storage cost, proposing _Soft-prototype_ framework to use a prototype. The prototype is an approximation of the target sentence \(Y^{\prime}=(y^{\prime}_{0},...,y^{\prime}_{i})\), produced by a probability generator \(R\) that accepts any \(x\) to generate a probability \(p(y^{\prime})\) over the target vocabulary to search \(y^{\prime}\).

These successful methods, although using different settings and frameworks, share the same idea to inject a draft of the required target sentence and introduce global information to the decoder. Therefore, the decoder can understand the target globally. Concretely, such an idea can be formulated into a framework as:

\[Y=Dec(Enc(X),draft) \tag{1}\]

However, these successful methods either introduce computational complexity to NMT Wang et al. (2019); Xia et al. (2017) or employ the EM-like process, showing significant degradation in inference efficiency, e.g., GNMTShah and Barber (2018) needs 110% additional inferring time. Intuitively, a high-quality draft should include two main aspects: 



**1)** a good draft should include a global semantic for the target sentence; **2)** a draft should not degrade inference efficiency significantly.

## NMT with Semantic Draft

In this section, we present our framework and method. We then discuss how to train the model in generative settings and how to tackle optimization challenges in practice.

### Framework

Inspired from previously successful models, we employ the general framework \(Y=Dec(Enc(X),draft)\) for our model, presenting the high-level architecture in Figure 1. Concretely, \(draft\) is instantiated to \(z\) that the general framework is modified to \(Y=Dec(Enc(X),z)\). Since \(z\) is sampled from the semantic space, our decoder is a _variational_ decoder Altieri and Duvenaud (2015); Kingma and Ba (2015); Bowman et al. (2016).

#### 3.1.1 Generative Semantic Draft

To obtain \(z\), we leverage a similar generative process of GNMT Shah and Barber (2018), sampling \(z\) from the semantic space that is a Gaussian inference model trained by \(s\) and \(t\) or approximations of \(s\) and \(t\) at the very least. Typically, \(s\) and \(t\) are obtained by modeling the semantics of \(X\) and \(Y\) with the same parameters.

Semantic for Source Sentence\(s\in R^{d}\) is computed by averaging a set of vector representation. Specifically, we first process \(X\) to the NMT encoder before averaging, obtaining \(Enc(X)\). Then, we compute \(s=\frac{1}{N}\sum_{k=0}^{n}Enc(X)_{k}\).

Semantic for Target SentenceWe encourage the model to learn an approximation of \(t\) instead of the "ground-truth target semantic". We assume \(G(s)\approx t\), where \(G\) is a two-layer cross-lingual generator. In other words, we compute a _dummy_ target-sentence semantic \(G(s)\) based on \(s\). We will discuss this assumption in SS4_Multilingual Encoder and Cross-lingual Generator_ and how to train the cross-lingual generator \(G\) in SS3.2_Encoder and Generator Tweaking_.

Semantic SpaceTypically, a Gaussian inference model is used for the semantic space, representing a variational distribution \(q_{z}(z|s,t)\) for sampling Shah and Barber (2018); Zhang et al. (2016); Zheng et al. (2020). It serves as an approximate posterior. Instead of \(q_{z}(z|s,t)\), in our model, we use \(q_{z}(z|s,G(s))\) for our required _semantic space_ because \(G(s)\) is encouraged to learn an approximation of \(t\). Specifically, we concatenate \(s\) and \(G(s)\) to compute the mean and variance of the diagonal Gaussian as:

\[S=[s,G(s)]\\ q_{z}(z|s,G(s))=\mathcal{N}(W^{\mu}S,diag(exp(W^{\sigma}S))) \tag{2}\]

#### 3.1.2 Decoding with Draft

As aforementioned, \(z\) is sampled from the semantic space \(q_{z}(z|s,G(s))\). We then aggregate \(z\) and \(Enc(X)\), processing the aggregation to the decoder for decoding. In other words, we add generative context to the encoding for the encoder-decoder attention in the decoder. Therefore, the decoder is a variational decoder that is conditional on \(z\) and \(X\).

### Training

NMT TrainingTo train the parameters of both NMT and the semantic space in generative settings, we follow the successful training strategy in previous works Bowman et al. (2016); Zhang et al. (2016), using SGVB (stochastic gradient variational Bayes) Kingma and Welling (2014); Rezende et al. (2014) to perform approximate maximum likelihood estimation:

\[\begin{split}&\mathcal{L}(Y|X)=\mathbb{E}_{q_{z}(z|s,G(s))}[ \log p_{nmt}(Y|X,z)]-\\ &\lambda D_{KL}q_{z}(z|s,G(s))||p(z)\end{split} \tag{3}\]

where \(\lambda\) weighs the KL divergence term and \(p(z)=\mathcal{N}(0,I)\).

Encoder and Generator TweakingIntuitively, the semantic space should consider the shared semantics between \(s\) and \(t\). Ideally, \(s\) and \(t\) should be obtained from a shared model by processing \(X\) and \(Y\), which is discussed in generative NMT Shah and Barber (2018); Zheng et al. (2020); Eikema and Aziz (2019). In spired by this idea, we use the same NMT encoder to compute \(Enc(Y)\), obtaining the "ground-truth target semantic" \(t=\frac{1}{M}\sum_{k=0}^{m}Enc(Y)_{k}\in R^{d}\). As aforementioned, we do not directly use \(t\) for our semantic space, which is different from generative NMT. Instead, we only use \(t\) to enforce and regularize \(G(s)\) in training. Concretely, we train the cross-lingual generator \(G\) to restore \(t\) from \(s\) so that \(G(s)\approx t\)



## 4 Discussion

### Inferring with Almost Free Draft

Costly DraftIn traditionally generative NMT, based on a random target sentence, the inference mode or the process of translation generating makes an initial guess \(z_{init}\) from the semantic space or the variational distribution \(q_{z}(z|s,t_{random})\), where \(s\) is computed by \(X\) and \(t_{random}\) is obtained from a random \(Y_{random}\). Then, it can generate a possible translation \(Y^{\prime}\) and its semantic \(t^{\prime}\). To obtain a good translation, based on the last translation, the inference mode can re-sample a better semantic from the semantic space and regenerate a new translation to maximize a lower bound on \(log(p(X,Y))\) in the EM-like process. Readers can also refer to Algorithm 1 in GNMT Shah Barber (2018) for more details.

Almost Free DraftUnlike traditionally generative NMT, we do not need to make an initial guess and also do not employ the EM-like process to sample \(z\) for inferring, which improves the inference efficiency. In our model, \(G(s)\), which is the _dummy_ target semantic, plays a prominent role that aims to approximate \(t\) instead of making an initial guess. Therefore, we do not have to make an initial guess, and we can also eliminate the whole EM-like process because \(z\) is not randomly sampled, which results in a one-shot sampling. Since \(G\) is a simple generator, sampling \(z\) from \(q_{z}(z|s,G(s))\) does not hurt the inference efficiency significantly and is almost free of cost.

### Multilingual Encoder and Cross-lingual Generator

Approximation of \(t\)In _Encoder and Generator Tweaking_ operation, we jointly train the encoder and the cross-lingual generator \(G\) to make \(G(s)\) and \(t\) as similar as possible. Since we input parallel sentences to the encoder, the encoder is encouraged to search multilingual properties. Specifically, we notice that \(s\approx t\) potentially1, which is studied and reported in previous works of multilingual BERT empirically Devlin et al. (2019); Karthikeyan et al. (2020); Wu and Dredze (2019). Meanwhile, _Soft-prototype_Wang et al. (2019) and multilingual NMT Wu et al. (2016); Johnson et al. (2017) also explore this aspect in NMT scenario. We further introduce the cross-lingual generator \(G\) to tweak/finetune the property, observing the significant benefits of regularizing. Most importantly, with the cross-lingual generator \(G\), the model can greedily gain a _dummy_\(t\) by \(G(s)\) so that the semantic draft can be sampled in a one-shot generative style without the EM-like process.

Footnote 1: There is a difference between \(s\) or \(t\) and the output of multilingual BERT. Specifically, \(s\) and \(t\) are sentence representations, whereas multilingual BERT outputs a sequence of the word representation.

Potential of \(s\) and \(G(s)\)Besides, we are aware that only injecting \(s\) or \(G(s)\) without processing to the semantic space may also provide global information or the shared semantic for decoding because \(s\approx t\) and \(G(s)\approx t\) potentially. We will present an ablation study in one of our comprehensive experiments SS6.5 _Necessity of Semantic Space and Multilingual Encoder_ to show the significance of \(G\), the semantic space and their combination.

Semantic in Encoder and DecoderOn the other hand, compared to generative NMT, which employs an auxiliary network to help the semantic space by feeding parallel sentences, our method simply processes the parallel sentences to the NMT

Figure 1: High-level view of NMT with a semantic draft. _Note that the “dotted line” is only used in training. \(s\)_ and \(t\) represent the sentence semantics. The semantic draft \(z\) is sampled from the semantic space that is a parameterized space to model the Gaussian inference distribution \(q_{z}(z|s,G(s))\), where \(G\) is our cross-lingual generator. \(\sigma\) linearly increases over the course of training so that the model learns to predict without \(t\). _cos_ denotes the similarity between \(G(s)\) and \(t\) that we encourage \(G(s)\approx t\). The variational decoder decomposes the sum of a draft and the encoding.



encoder that is equivalent to the auxiliary network in generative NMT. In this way, there is no need to pass \(z\) to the encoder to model a joint probability \(P_{nmt}(X,Y,z)=p(z)p(X|z)p_{nmt}(Y|X,z)\). Specifically, as discussed in VAE Altieri and Duvenaud (2015); Kingma and Ba (2015); Bowman et al. (2016); Zhang et al. (2016), if \(z\) involves in the process of encoding, \(z\) can guide and regularize the encoder to consider the shared semantic. Therefore, generative NMT models the joint probability in training, encouraged to consider \(z\) in both the encoder and the decoder. However, in our model, we let the multilingual encoder consider the implicitly shared semantic itself, and we inject \(z\) into the decoder that is encouraged to consider the shared semantic.

### Comparison

In Figure 2, we compare our framework with previous successful models: GNMT Shah and Barber (2018), _Deliberation Xia et al. (2017)_ and _Soft-prototype_Wang et al. (2019). We observe some significant differences from the perspective of our design:

* **vs GNMT _1)_** The semantic space is built upon the multilingual encoder and the cross-lingual generator in our model; _2)_ the semantic/global information is only used in the decoder.
* **vs Deliberation** The global information comes from semantic space instead of the first-pass decoding.
* **vs Soft-prototype** The global information is sampled from the semantic space instead of target prototypes.
* Additionally, we notice an optimization solution for the EM-like process. Eikema and Aziz (2019) study an approximating method to maximize the lower bound on \(log(p(X,Y))\) by employing an auxiliary distribution with only using source \(s\), which boosts the inference efficiency with a single call (without the EM-like process) to an \(argmax\) solver. Compared to their work, our model has three major differences: _1)_ our model depends on both \(s\) and \(G(s)\); _2)_ an auxiliary distribution is not necessary in our model; _3)_ we focus on the process of draft generating.

### Optimization Challenges

Collapse of DKLBowman et al. (2016) report the collapse of \(D_{KL}\) term in the objective function Eq.3. Following the instructions of Bowman et al. (2016); Shah and Barber (2018), we apply two common strategies: _1)_\(\lambda\) linearly increases from 0 to 1 over the initial 50k steps during training; _2)_ we randomly drop a constant of 30% words when encoding \(X\).

Warm-up of GeneratorTraining is somewhat tricky when using the cross-lingual generator \(G\). We apply a weight \(\sigma\in[0,1]\) for \(G(s)\) and a weight \(1-\sigma\) for \(t\), as presented in Figure 1. \(\sigma\) linearly increases from 0 to 1 over 50k steps after \(\lambda=1\). By this strategy, the semantic space is encouraged to rely on \(t\) in warm-up. Significantly, it avoids that \(cos(G(s),t)\) is close to 0 at the beginning of training. After warm-up, i.e., \(G(s)\approx t\), we use \(G(s)\) for the rest of training.

## 5 Experiment Settings

### Dataset

To be comparable, we train our model on language pairs \(\{French,German\}\leftrightarrow English\) and a relative low-resource language pair \(Romanian\leftrightarrow English\) which are commonly used in previous work Shah and Barber (2018); Vaswani et al. (2017); Bahdanau et al. (2015); Zheng et al. (2020). Concretely, we download parallel corpora \(\{French,German,\}\leftrightarrow English\) from WMT 20142Bojar et al. (2014). For \(Romanian\leftrightarrow\\)

Figure 2: Comparison between our model and previous models. The “dotted line” indicates the flow of global information. \(Z\) denotes Gaussian semantic space. _Net_ denotes an auxiliary network. \(G\) is a generator. _1st_ denotes first-pass decoding. _soft_ denotes soft-prototype.

 

\(English\), we retrieve parallel corpora from WMT 20163 Bojar et al. (2016). The preprocess is simple in our case that we only remove sentences with over 50-word length in our training datasets. Following standard evaluation, the model is evaluated on _newstest2014_ for \(\{French,German\}\leftrightarrow English\) and _newstest2016_ for \(Romanian\leftrightarrow English\). Case-sensitive BLEU score is computed by _multi-BLEU.perl4_ to report the performance. We also employ beam search with beam size 4 and length penalty 0.6.

Footnote 3: [http://www.statmt.org/wmt16/translation-task.html](http://www.statmt.org/wmt16/translation-task.html)

Footnote 4: [https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-BLEU.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-BLEU.perl)

### Model Settings

We implement presented model on Tensorflow 2.0 Abadi et al. (2016). To be comparable with other models and baselines, the NMT settings are identical to big-Transformer Vaswani et al. (2017). Specifically, we set model dimension, word embedding, head, encoder layer, decoder layer and FFN filter to 1024, 1024, 16, 6 and 4096. Adam optimizer Kingma and Ba (2015) is employed with parameters \(\beta_{1}=0.9\text{,}\beta_{2}=0.98\) and \(\epsilon=10^{-9}\). We use a dynamic learning rate over the course of NMT training Smith (2017); Vaswani et al. (2017)5. The dropout rate is set to \(rate=0.1\), and label smoothing is used with \(gamma=0.1\)Mezzini (2018). Parallel corpora for one translation task (e.g., \(Romanian\leftrightarrow English\)) are concatenated to train BPE Sennrich et al. (2016) with a balance strategy Lample and Conneau (2019) that forms a shared vocabulary with \(40,000\) subtokens. For data feeding efficiency, each mini-batch of similar-length sentences are padded to the same length and may have a different number of elements in each mini-batch, such that \(batch\_size\times padded\_length<=3000\).

Footnote 5: \(lr=peak\_lr\times min(1,step/warm\_up)\times(max(step,warm\_up))^{-0.5}\), where \(warm\_up=3000\) and \(peak\_lr=0.05\).

### Reimplementation and Reconfiguration

To be fair, we reimplement some models on our machine with the same mini-batch size. We compare the reimplemented results to the reported results on the same test set to ensure the difference less than 5% (or 1.5) in BLEU. Then, we can confirm the reimplementation and reconfiguration.

## 6 Performance

### Translation Task

We study the methods of how to produce and consider global information for NMT. Since we have discussed three successful directions, we compare our method with the baselines of Transformer Vaswani et al. (2017), generative NMT including GNMT Shah and Barber (2018) and MirrorGNNT Zheng et al. (2020), _Deliberation_Xia et al. (2017) and _Soft-prototype_Wang et al. (2019). Meanwhile, we have introduced some additional parameters to the model, which is the same as the comparable models. Therefore, we evaluate not only the performance but also the inference efficiency. The comparison of the inference efficiency is based on the inference speed of the vanilla big-Transformer. Besides, we reconfigure MirrorGNMT and GNMT to big-Transformer settings, and we additionally reimplement _Soft-prototype_ on \(English\to Romanian\). Table 1 presents the performance of our model and the baselines on the training dataset. We summarize the results that:

* **Competitive Translation Quality** Our method outperforms the baselines of big-Transformer and GNMT on all the language pairs. Compared to state-of-the-art models, our model gains competitive performance on all the language pairs.
* **Clear Advantage in Inference Efficiency** Besides competitive performance on all the language pairs, our model has a clear advantage in the comparison of inference efficiency. Specifically, GNMT, Mirror-GNMT and _Deliberation_ introduce computational complexity to the decoder that needs more than 1 iteration6 to consider a translation (+ 80% additional time at least), and _Soft-prototype_ increases the computational complexity on the encoder side (+ 34% additional time). However, our method only introduces a generator to the model so that the computational complexity in the encoder and the decoder is the same as in vanilla big-Transformer, which results in an efficient inferring and an almost free draft (only + 5% additional time). Footnote 6: During our test, generative NMT including GNMT and Mirror-GNMT always need 2-3 iterations for the EM-like process, and _Deliberation_ needs a constant of 2 iterations.
* **Improvement from 

port a result obtained by employing the EM-like process for our model in the last row. Although there is noticeable room for improvement, it degrades the inference efficiency significantly so that we do not suggest such a combination. We will discuss this result and integration in SS6.2 _Drafting with EM-like process_.

### Drafting with EM-like process

In the most discussion of this work, we sample \(z\) from \(q_{z}(z|s,G(s))\) in a one-shot generative style for the sake of inference efficiency. The previous evaluation shows that such an idea is feasible. Meanwhile, our model shares some properties with generative NMT, which makes us interested in the integration with the EM-like process for the sake of the best translation quality only.

In this scenario, we have two steps to translate \(X\):

1. We sample a semantic draft \(z\) from \(q_{z}(z|s,G(s))\) and gain a possible translation \(Y^{\prime}\).
2. We then sample a new semantic draft \(z^{\prime}\) from \(q_{z}(z|s,t^{\prime})\) to predict a possible and new translation \(Y^{\prime\prime}\), where \(t^{\prime}=\frac{1}{M^{\prime}}\sum_{k=0}^{M^{\prime}}Enc(Y^{\prime})_{k}\) and \(M^{\prime}\) is the length of \(Y^{\prime}\).

The second step can be repeated to maximize a lower bound on \(log(p_{nmt}(Y|X))\). We observe some improvements from employing the EM-like process, reporting the result in the last row of Table 1 that we achieve the best performance on all the language pairs. However, most significantly, the translation converges at \(2\sim 3\) iterations that increase the inference time by 137% (from \(1.05\times\) to \(2.42\times\)). Concretely, the model needs to re-encode the last translation to obtain a new draft and re-decode the new draft to generate a new translation, e.g., re-encode \(Y^{\prime}\) to obtain \(Enc(Y^{\prime})\) and its \(t^{\prime}\), re-sample the draft \(z^{\prime}\) from \(q_{z}(z|s,t^{\prime})\) and re-decode the aggregation of \(Enc(X)\) and \(z^{\prime}\). Thus, we suggest the one-shot generative style in practice.

Additionally, we realize that in this case the improvement may come from not only the re-sampled draft but also the adaptation of two ideas: _1)_"double encoding" in _Soft-prototype_Wang et al. (2019) because we encode the previously complete translation/prototype for the next translation; _2)_"double decoding" in _Deliberation_Xia et al. (2017) because we make more than one complete translation. We will justify the significance of the draft in SS6.3 _Test for Draft_ and SS6.4 _Draft Reliance Test_.

### Test for Draft

We are interested in whether the draft does indeed provide useful semantics/global information. In the last section, the improvement from the EM-like process can intuitively show the effect of the draft because a better-quality draft re-sampled from the last translation continuously improves the performance, but the improvement may only come from "double encoding" and "double decoding". Therefore, we conduct a test to demonstrate that the generative draft learns the desired semantics.

In this test, we share the same _missing word translation_ task with GNMT Shah and Barber (2018). Concretely, the model is forced to give a translation based on the draft heavily. We share the same settings that each word has a 30% chance of being missing independently. Note that we do not conduct this experiment for _Deliberation_Xia et al. (2017) and _Soft-prototype_Wang et al. (2019) because such discriminative models do not sample semantics from the semantic space. Table 2 shows the test result on training dataset \(German\leftrightarrow English\) and test dataset _newstest2014_. We observe that our model outperforms GNMT and achieves competitive performance to Mirror-GNMT Zheng et al. (2020). Specifically, compared to GNMT, our method trains a multilingual encoder and a cross-lingual generator to encourage shared semantics for the semantic space.

\begin{table}
\begin{tabular}{l l c c c|c c|c} \hline Model & \multicolumn{2}{c}{revised2014} & \multicolumn{2}{c}{revised2016} & Speed \\ \cline{2-7}  & \(Fr\to En\) & \(En\to Fr\) & \(De\gets En\) & \(En\to De\) & \(Ro\gets En\) & \(En\to Ro\) \\ \hline Transformer Vaswani et al. (2017), _DM baseline_ & 42.21 & 41.85 & 32.11 & 28.40 & 34.01 & 32.83 & \(1\times\) \\ GNMT Shah and Barber (2018), _GM baseline_ & 42.67 & 42.22 & 32.54 & 28.81 & 34.52 & 33.34 & \(2.1\times\) \\ Transformer + _Deliberation_Xia et al. (2017) & 42.58 & 42.58 & 29.11 & 29.11 & 1.8 \(\times\) \\ Mirror-GNMT Zheng et al. (2020) & & 42.99 & 33.11 & 29.22 & 34.91 & 33.87 & \(2.7\times\) \\ Transformer + _Soft-prototype_Wang et al. (2019) & & 42.99 & 39.46 & 34.12 & 1.34 \(\times\) \\ Our method & 42.94 & 42.73 & 33.03 & 29.20 & 34.89 & 33.82 & 1.05\(\times\) \\ \hline our method with EM-like process & 43.05 & 42.97 & 33.31 & 29.49 & 35.09 & 34.18 & 2.42\(\times\) \\ \hline \end{tabular}
\end{table}
Table 1: Performance of our method. Our method is competitive on translation quality and has a clear advantage in inference efficiency. _DM baseline_: discriminative model baseline. _GM baseline_: generative model baseline.

 

Compared to Mirror-GNMT, which gains the improvement from the simultaneously used LM (language model) and back-translation technic [14], our model is not integrated with LM to counter noisy input so that Mirror-GNMT gains slightly better performance. We leave the integration with denoising language modeling [23] for future experiments.

### Draft Reliance Test

We have demonstrated that the semantic draft is useful for the translation task. We further indicate how much the model relies on the semantic draft. Since the objective function Eq.3 is the same as in GNMT [7] and Mirror-GNMT [5], we report a comparison on the term of \(D_{KL}=D_{KL}q_{z}(z|X,Y)||p(z)\), presenting the result in Table 3. The test is conducted on training dataset \(German\leftrightarrow English\) and test dataset _newstest2014_ by averaging the value of \(D_{KL}=D_{KL}q_{z}(z|X,Y)||p(z)\). Our method relies on the semantic draft (or the latent variable from the semantic space) heavier than GNMT does. With the EM-like process, the reliance is higher than Mirror-GNMT.

### Necessity of Semantic Space and \(G\)

Although the semantic draft does indeed provide useful global information in SS6.3 _Test for Draft_ and SS6.4 _Draft Reliance Test_, we still question the necessity of the semantic space because \(G(s)\approx t\) and \(s\approx t\). In other words, we can simply process \(G(s)\) or \(s\) to the decoder, which can provide global information for decoding potentailly. To justify, we train the model on training dataset \(German\leftrightarrow English\) and test dataset _newstest2014_ with 4 different types of \(draft\) based on the framework \(Dec(Enc(X),draft)\):

* We use our full-packaged model \(draft=z\), where \(z\) comes from \(q_{z}(z|s,G(s))\).
* \(draft=G(s)\) is set for translation to test the significance of the semantic space.
* To test the significance of \(G\), we set \(draft=z^{\prime}\), where \(z^{\prime}\) comes from \(q_{z^{\prime}}(z^{\prime}|s)\).
* We test both the significance of \(G\) and the semantic space by setting \(draft=s\) for translation.

Besides the difference of \(draft\), all the other configurations are the same for this test. We report the result in Table 4, and our observations are that:

* According to "row 2 vs row 4", we can see the significance of the cross-lingual generator \(G\).
* "row 3 vs row 4" indicates the significance of the semantic space.
* When focusing on "row 2 vs row 3", \(G\) improves general translation performance (column 2&4), and the semantic space improves noisy translation (column 3&5)

We intuitively conclude that the semantic space and the cross-lingual generator \(G\) can further smooth and regularize the semantic for decoding, similar to that is found in GNMT [8] and [9]. Moreover, the cross-lingual generator \(G\) can only restore a coarse semantic so that the model cannot only rely on \(G(s)\) to maintain translation quality when testing in the _missing word translation_ task generally.

### Improvement from Non-parallel Data

We have mentioned the multilingual property of the encoder in our design, using the NMT encoder to process \(X\) and \(Y\) . As reported in multilingual BERT [13, 14, 15],

\begin{table}
\begin{tabular}{l l l l|l l} \hline \multicolumn{6}{c}{_newset2014_} \\ Model & \(De\to En\) & \(noisy~{}De\to En\) & \(En\to De\) & \(noisy~{}En\to De\) \\ \hline GNMT [8] & 32.54 & 23.28 & 28.81 & 19.93 \\ Mirror-GNMT [5] & 33.11 & 24.37 & 29.22 & 20.74 \\ \hline our method & 33.03 & 23.93 & 29.20 & 20.35 \\ our method with EM-like process & 33.31 & 24.21 & 29.49 & 20.52 \\ \hline \end{tabular}
\end{table}
Table 2: Performance of _missing word translation_. Our method is competitive in this scenario even without integrating language modeling to recover noisy input.

\begin{table}
\begin{tabular}{l c l l} \hline  & GNMT & Mirror-GNMT & our method & + EM-like process \\ \hline \(D_{KL}\) & 5.73 & 6.92 & 6.65 & 7.03 \\ \hline \end{tabular}
\end{table}
Table 3: Draft Reliance Test. The average value of \(D_{KL}=D_{KL}q_{z}(z|X,Y)||p(z)\) on test set \(German\leftrightarrow English\). Higher value indicates higher reliance on the semantic draft or the latent variable.

 

Wu and Dredze, 2019), sharing encoder for _non-parallel_ sentences in different languages can still build shared semantic space implicitly. This leads us to experiment with that we can jointly train the encoder with the objective of multilingual BERT. We then train on a relative low-resource language pair \(Romanian\leftrightarrow English\), and we use additional monolingual data _News Crawl articles 2015_ from WMT 2016 to jointly train the multilingual encoder with the objective of multilingual BERT. In Table 5, we report competitive results, and the performance is significantly improved by simultaneously using non-parallel data. Note that, when training on non-parallel data, we can pre-train the multilingual encoder with the BERT objective instead of joint training. We leave this idea for further experiments.

## 7 Conclusion and Future Work

Translation quality can be further improved by global information from the target sentence. Although there have been three feasible solutions, successful methods do not consider inference efficiency carefully, which leads to high cost in inferring. In this work, we present a method/framework to improve the performance of NMT. We sample a semantic draft from semantic space that the decoder can consider the semantic draft to obtain the required global information with high efficiency in inferring. Our empirical study shows that, compared to previously successful methods, our method can achieve competitive performance and has a clear advantage in inference efficiency. Since we do not change the architecture of the NMT model, our model can be further improved by employing pretraining (Lample and Conneau, 2019; Devlin et al., 2019; Radford et al., 2018), back-translation (Sennrich et al., 2016a) and other finetuning methods with non-parallel data. And, our model can also be used in unsupervised NMT (Artetxe et al., 2018; Lample et al., 2018). We leave all these experiments for future work.

## References

* Abadi et al. (2016) Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_, pages 265-283.
* Altieri and Duvenaud (2015) Nicholas Altieri and David Duvenaud. 2015. Variational Inference with Gradient Flows. _NIPS Workshop_, 37:3-6.
* Conference Track Proceedings_.
* Conference Track Proceedings_.
* Bojar et al. (2016) Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias

\begin{table}
\begin{tabular}{p{42.7pt} p{28.5pt}|p{28.5pt} p{28.5pt} p{28.5pt}} \hline  & \multicolumn{4}{c}{_newsid2016_} \\ \cline{2-4} Model & & \(Ro\to En\) & \(En\to Ro\) \\ \hline Transformer + XLM + non-parallel (Lample and Conneau, 2019) & 35.30 & 34.11 \\ Mirror-GSMT + non-parallel(Zheng et al., 2020) & 37.54 & 35.93 \\ Transformer + Soft-prototype + non-parallel (Wang et al., 2019) & 38.05 & 36.41 \\ \hline our method & 34.89 & 33.82 \\ our method + non-parallel & 37.42 & 35.77 \\ our method with EM-like process + non-parallel & 38.19 & 36.53 \\ \hline \end{tabular}
\end{table}
Table 5: Performance of training with additional non-parallel data. The performance of our method is competitive, significantly improved by non-parallel data.

 

Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In _Proceedings of the First Conference on Machine Translation_, pages 131-198, Berlin, Germany. Association for Computational Linguistics.
* Bojar et al. (2014) Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tambyna. 2014. Findings of the 2014 workshop on statistical machine translation. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Bowman et al. (2016) Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 10-21.
* Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Eikema and Aziz (2019) Bryan Eikema and Wilker Aziz. 2019. Auto-encoding variational neural machine translation. In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_, pages 124-141, Florence, Italy. Association for Computational Linguistics.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In _Proceedings of the 34th International Conference on Machine Learning_, volume 70 of _Proceedings of Machine Learning Research_, pages 1243-1252. PMLR.
* Graves (2013) Alex Graves. 2013. Generating Sequences With Recurrent Neural Networks. _arXiv preprint arXiv:1308.0850_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Computation_, 9(8):1735-1780.
* Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Kirkun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. _Transactions of the Association for Computational Linguistics_, 5:339-351.
* Conference Track Proceedings_.
* Conference Track Proceedings_.
* Conference Track Proceedings_.
* Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In _Advances in neural information processing systems_.
* Conference Track Proceedings_.
* Short papers proceedings_.
* Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In _Proceedings of the Fourth Conference on Machine Translation_, pages 314-319, Florence, Italy. Association for Computational Linguistics.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In _Proceedings of the 31st International Conference on Machine Learning_, volume 32 of _Proceedings of Machine Learning Research_, pages 1278-1286, Bejing, China. PMLR.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 86-96, Berlin, Germany. Association for Computational Linguistics 

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Shah and Barber (2018) Harshil Shah and David Barber. 2018. Generative neural machine translation. In _Advances in Neural Information Processing Systems_, volume 2018-Decem, pages 1346-1355.
* Smith (2017) Leslie N Smith. 2017. Cyclical learning rates for training neural networks. In _2017 IEEE winter conference on applications of computer vision (WACV)_, pages 464-472. IEEE.
* Su et al. (2018) Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. 2018. Variational recurrent neural machine translation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 32.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems_, volume 4, pages 3104-3112.
* Vaswani et al. (2017) Ashish Vaswani, Google Brain, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In _Advances in neural information processing systems_, pages 5998-6008.
* Vincent (2010) Pascal Vincent. 2010. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. _Journal of Machine Learning Research_, 11:3371-3408.
* Wang et al. (2019) Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, CengZiang Zhai, and Tie-Yan Liu. 2019. Neural Machine Translation with Soft Prototype. In _Advances in Neural Information Processing Systems_.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the qth International Joint Conference on Natural Language Processing_, pages 833-844, Hong Kong, China. Association for Computational Linguistics.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _ArXiv_, abs/1609.08144.
* Xia et al. (2017) Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation Networks: Sequence Generation Beyond One-Pass Decoding. In _Advances in Neural Information Processing Systems_, pages 1784-1794. Curran Associates, Inc.
* Zhang et al. (2016) Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 521-530, Austin, Texas. Association for Computational Linguistics.
* Conference Track Proceedings_