Meta Back-Translation
Authors: Hieu Pham
Xinyi Wang
Yiming Yang
Graham Neubig
Venue: ICLR 2021 Poster (2021)
Keywords: meta learning
machine translation
back translation

Abstract: Back-translation is an effective strategy to improve the performance of Neural Machine Translation~(NMT) by generating pseudo-parallel data. However, several recent works have found that better translation quality in the pseudo-parallel data does not necessarily lead to a better final translation model, while lower-quality but diverse data often yields stronger results instead.
In this paper we propose a new way to generate pseudo-parallel data for back-translation that directly optimizes the final model performance.  Specifically, we propose a meta-learning framework where the back-translation model learns to match the forward-translation model's gradients on the development data with those on the pseudo-parallel data. In our evaluations in both the standard datasets WMT En-De'14 and WMT En-Fr'14, as well as a multilingual translation setting, our method leads to significant improvements over strong baselines. 

Under review as a conference paper at ICLR 2021
META BACK-TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Back-translation (Sennrich et al., 2016) is an effective strategy to improve the performance of Neural Machine Translation (NMT) by generating pscudo-paralle data. However, several recent works have found that better translation quality of the pseudo-parallel data does not necessarily lead to better final translation model, while lower-quality but diverse data often yields stronger results instead. In this paper we propose a new way to generate pseudo-paralle data for back-translation that directly optimizes the final model performance. Specifically, we propose a meta-learning framework where the back-transkaa translationn model learns match the model's ' gradients on the development data with those on the saauddllalllll data. In our evaluations De' 14 and WMT En-Fr' 14, as well as a multilingual translation setting, our method leads to significant improvements over strong baselines.
both the standard datasets WMT En-
INTRODUCTION
While Neural Machine Translation (NMT) delivers state-of-the oftte-of-the-art performance across many trans- lation tasks, this performance is usually contingent on the existence of large amounts of training data (Sutskever et al., 2014; Vaswani et al., 2017). Since large parallel training datasets are often unavailable for many languages and domains, various methods have been developed to leverage abundant monolingual corpora (Gulcehre et al., 2015; Cheng et al., 2016; Sennrich et al., 2016; Xia et al., 2016; Hoang et al., 2018; Song et al., 2019; He et al., 2020), Among such methods, one particularly popular approach is
(BT; Sennrich et al. (2016)).
In BT, in order to train a cource-to-tee to-tagget translation model, i.c., the forward model, one first trains a target-to-souree translation model, i.e., the backward model. This backward model is then employed to translate monolingual data from the target language into the source language, resulting in pseudo- parallel corpus. This pseudo-parallel corpus is then combined with the real parallel corpus train the final forward translation model. While the resulting forward model from BT typically enjoys a significant boost in translation quality, we identify that BT inherently carries two weaknesses.
First, while the backward model provides natural way to utilize monolingual data in the target language, the backward model itself is still trained the parallel corpus. This means that the backward model's quality is as limited as that of forward model trained in the vanilla setting. Hoang al. (2018) proposed iterative BT to avoid this weakness, but this technique requires multiple rounds of retraining models in both directions which are slow and expensive.
Second, we do not understand how the the forward model's performance. For example, Edunov et al. (2018) has observed that pseudo- parallel data generated by sampling or by beam-searching with noise from the backward model train better forward models, even though these generating methods typically result in lower BLEU scores compared to standard beam search. While Edunov et al. (2018) associated their observation to the diversity of the generated eseudo-paralle, data, diversity alone is obviously insufficient some degree of quality is necessary as well.
data translated by the backward model affects
In summary, while BT is an important technique, training good backward model for BT is either hard or slow and expensive, and even if we have a good backward model, there is no single recipe how to use it to train good forward model.
In this paper, we propose a novel technique to alleviate both aforementioned weaknesses of BT. Unlike vanilla BT, which keeps the trained backward model fixed and merely uses it to generate pseudo-

---

Under review as a conference paper at ICLR 2021
update backsward model
Mondingual
Porado Purallel
Parallel
Wie geht es dir hamtef
Backward model
comples
How are you today?
gradient
How are you?
Entropp
Crees
Forwurd model (t-1)
geht
Forward model (t)
geht
An example training step of meta back-translation to train forward model translating English (En) into German (De). The step consists of two phases, illustrated from left to right in the figure. Phase backward model translates a De sentence taken from a monolingual corpus into a pseudo En sentence, and the forward model updates its parameters by prop-praaggaing from canonical training losses on the pair (pseudo En, mono De). Phase 2: the updated forward model computes cross-entropy loss pair of ground truth sentences (real En, real De). As annotated with the red path in the figure, this cross-entropy loss depends on the backward model, and hence can be baok-prop to update the backward model. Best viewed in colors.
Figure 1:
parallel data to train the forward model, we continue to update the backward model throughout the forward model's training. Specifically, update the backward model to improve the forward model's performance on a held-out set of ground truth parallel data. We provide an illustrative example of method in Fig. 1, where we highlight how the forward model's held-out set performance depends on the pseudo-parallel data sampled from the backward model. This dependency allows us to mathematically derive an end-to-end update rule continue training the backward model throughout the forward model's training. As our derivation technique is similar to meta-learning (Schmidhuber, 1992; Finn et al., 2017). we name our method Meta Back- Translation (MetaBT).
In theory, MetaBT effectively resolves both aforementioned weaknesses of vanilla BT. First, the backward model continues its training based on its own generated pseudo-paralls data, and hence is no longer limited to the available parallel data, Furthermore, MetaBT only trains one backward model and then trains one pair of forward model and backward model, eschewing the expense of multiple iterations in Iterative BT (Hoang et al., 2018). Second, since MetaBT updates its backward model in an end-to-end manner based on the forward model's performance on a held-out set, MetaBT no longer needs to explicitly understand the effect of its generated seeudo-parallel data on the forward model's 's quality.
Our empirical experiments verify the theoretical advantages of MetaBT with definitive improvements over strong BT baselines on various settings. In particular, on the classical benchmark of WMT En-De 2014, MetaBT leads to +1.66 BLEU score over sampling-base that MetaBT allows us to extend the initial parallel training set of the backward model by including parallel data from slightly different languages. Since MetaBT continues refine the backward model, the negative effect of language discrepancy is eventually rebated throughout the forward model's training, boosting up + +1.20 BLEU score for low-resource translation tasks.
BT. Additionally, we discover
A PROBABILISTIC PERSPECTIVE OF TRAASTRANNNATIO
To facilitate the discussion of MetaBT, we introduce probabilistic framework to interpret BT. Our framework helps to analyze the advantages and disadvantages of a few methods to generate escudo-paralle data such as sampling, beam- saanscchchhngg and beam-scarching with noise (Sennrich et al., 6016; Edunoy et al., 2018). Analyses these generating methods within our framework also motivates MetaBT and further allows mathematically derive MetaBT'ss update rules in $ 3.
Our Probabilistic Framework. We treat a language S as a probability distribution over all possible sequences of tokens. Formally, we denote by Ps(x) the distribution of random variable x, whose each instance x is sequence of tokens. To translate from source language S into target language T, we learn the conditional distribution Ps.r(y)x for sentences from the languages S and 7 with a parameterized probabilistic model Pyy| x: 0). Ideally, we learn 0 by minimizing the objective:
(0) = Er Ps. (x..) [[(zz yr 0)] where f(x,, yı 0) = -logP(y|
(I)
Since Ps. (x. y) Ps. (v) Ps (xly)
(xlv), we can refactor ((() from Eq. 1 as:
((0) Ey-Pr E.- Ps.r (x, y: 8)]
P
(2)
Motivating BT. In BT, since it is not feasible to draw exact samples y (y) and x ~ Ps.r (xly)) rely approximations. First, instead of sampling y Pr(y), we collect a corpus Dr of
2

---

Under review as a conference paper at ICLR 2021
monolingual data in the target language T and draw the samples y Uniform(Dr) Second, instead of sampling (xly( derive an approximate distribution P(xly) and sample P(x y).
Before we explain the derivation of P(x y), let us state that with these approximations, the objective (0) from Eq. becomes the BT following objective:
BT (0) = E yn--ooifomm (Dr)
P- B2 [E( y: 0:
(3)
Rather unsurprisingly, P(x y) in Eq. 3 above derived from pre-trained parameterized backward translation model P(x|y; v)) For example:
P(zly) - 1 argmax; p(cly: 4)) results in BT via beam- (Sennrich et al., 2016).
P(zly) _ P vb) results in BT via sampling (Edunov et al., 2018).
P(rly) - 1 Ix argmax; P(dly: v)) results in BT via noisy beam-search (Edunov al., 2018) where [xx yı +-) denotes the joint distribution of the backward model P(x yy 1)) and the noise.
Therefore, have shown that in probabilistic framework for BT, three common techniques generate preudo-parallel data from a pre-trained backward model correspond to different derivations from the backward model's distribution P(xly; Our framework naturally motivates two questions:
(1) given a translation task, how do we tell which derivation of P(xly) from P(xly; ; 1b) is better than another? and (2) can we derive better choices for P(xly) from a pre-trained backward model (p) according the answer of question (1)?
Metric for the Generating Methods. In the existing literature, the answer for our first question is relatively straightforward. Since most papers view the method of generating pseudo-parallel data as hyper-level design, i.e. similar to the choice of an architecture like Transformer or LSTM, and hence practitioners choose one method over another based on the performance of the resulting forward model on held-out validation sets.
Automatically Derive Good Generating Methods. We now turn to the second question that our probabilistic framework motivates. Thanks the generality of framework, every choice for Pyxly) results in an optimization objective. Using this objective, we can train forward model and measure its validation performance to evaluate our choice of P(xly). This process of choosing and evaluating Pxxly) can be posed as the following bi-level optimization problem:
Outer loop: P argmax ValidPerformmacc((()))
Inner loop:
0² argmin To P),
(4)
where Ter P) = En- (Dr) P(xl)) [((,, 0)]
The optimal solution of this bi-level optimization problem can potentially train forward model that generalizes well, as the forward model learns on a sscudo-paralle, dataset and yet achieves a good performance held-out validation set. Unfortunately, directly solving this optimization problem is not feasible. Not only is the inner loop quite expensive as it includes training a forward model from scratch according to P, the outer loop is also poorly defined as we do not have any restriction on the space that P can take. Next, in $ 3, we introduce a restriction on the space that can take, and show that restriction turns the task of choosing P into differentiable problem which can be solved with gradient descent.
3 META BACK- TACK-TRANSIATION
Continuing our discussion from $ 2, we design Meta Tack-Translation (MetaBT) which finds a strategy to generate pseudo-parally data from pre-trained backward model such that if forward model training on the generated parulllll data, it will achieve a strong performance on held-out validation set.
3

---

Under review as a conference paper at ICLR 2021
The Usage of "Validation" Data. Throughout this section, readers will that MetaBT makes extensive use of the "validation" set to provide feedback for refine the ceeudo-paralled data's generat- ing strategy. Thus, to avoid nullifying the meaning of held-out validation set, henceforth refer to the ground-truth parallel dataset where the forward model's performance is measured throughout its training the meta validation dataset and denote it by MetaDev- Other than this meta validation set, we also have separate validation set for hyper-parameter tuning and model selection.
Differentiable Bi-level Optimization Problem. We now discuss MetaBT, starting with formu- lating differentiable version of Problem 4. Suppose we have pre-trained paramterized backward translation model P(xly;:>) v)). Instead of designing the generating distribution P(xly) by applying actions such as sampling or beam-search to y: 1/), we let P(xly) - P(xly; v) and continue to update the backmodel's parameters & throughout the course of training the forward model. Clearly, under this association P(xly) _ P(xly; v)), the parameters controls the generating distribution of the secudo-paralled data to train the forward model. By setting the differentiable parameters & as the optimization variable for the outer loop, we turn the intractable Problem 4 into differentiable one:
Outer loop:
v = argmax Performance (f" (w), DMenaDev)
(5)
Inner loop: 0 (w) = argmin E En-UUnifomm Dr B P [6(x, y: v)
Bi-level optimization problems whose both outer and inner loops operate differentiable variables like Problem 5 have appeared repeatedly in the recent literature of meta-learning, spanning many areas such learning initialization (Finn al., 2017), learning hyper-paue spper-parameters (Baydin al., 2018), designing architectures (Liu et al., 2019), and reweighting examples (Wang et al., 2019b). We thus follow their successful techniques and design a two-phase alternative update rule for the forward model's parameters 0 in the inner loop and the backward model's parameters v in the outer loop:
Update the Forward Parameters 0. Given batch of monolingual target data ~ Uniform( Dr), we sample the pseudo-parallel data (3 ~ P(xly; v)), u) and update 0 as if (σz, y) was real data. For simplicity, assuming that 0 is updated using gradient descent on (i, y). using a learning rate n7. then we have:
Phase 1:
0, = 0, - Th of yr 0)
(6)
Phase 2: Update the Backward Parameters vi. Note that Eq. 6 means that Og depends on t. because T is sampled from a distribution parameterized by th. This dependency allows us to compute the meta validation loss of the forward model at 0,, which pookppro this loss to compute the gradient V (O, DMeaDev). Once we have this gradient, we can perform a gradient-based update on the backward parameter with learning rate my:
denote by (0,(w), Deeaaev), ), and
V/2 Viz-1 - To V V (o (0)) D Meeubee)
(7)
Computing V (1))) D MetaDerr) Our derivation of this gradient utilizes two techniques: (1) the chain rule to differentiate J (0.())), DeeaaDev) with respect to 1/s via 02; and (2) the log- graddint trick from reinforcement learning literature (Williams, 1992) to propagate gradients through the sampling of pseudo-source Z. We refer readers to $ A.I for the full derivation. Here, we present the final result:
V a (O, DMtetDee) ~ JO,, DMeeaDev) Vol(T, yi 0r-1)] WllogP(Z)us v)
(8)
In our implementation, we leverage the recent advances in high-order AutoGrad tools efficiently compute the gradient dot- product term via Jacobian-vector vector products. By alternating the update rules in Eq. 6 and Eq. 7, we have the complete MetaBT algorithm.
Remark: An Alternative Interpretation of MetaBT. The update rule of the backward model in Eq. strongly resembles the REINFORCE equation from the reinforcement learning literature. This similarity suggests that the backward model is trained as if it were an agent in reinforcement learning. From this perspective, the backward model is trained so that the would maximize the "reward":
data sampled from
R(G) VoJ DMMenDe)) T
(T, yj 0.-1)
Since this dot- dot-product measures the similarity in directions of the two gradients, it can be interpreted that MetaBT optimizes the backward model so that the forward model's gradient on pseudo-parallel
4

---

Under review as a conference paper at ICLR 2021
data sampled from the backward model is similar to the forward model's gradient computed on the meta validation set. This is a desirable goal because the reward guides the backward model's parameters to favor samples that are similar to those in the meta validation set.
A MULLTILINGUAL APPLICATION OF METABT
We find that the previous interpretation of MetaBT in Section 3 leads a rather unexpected application MetaBT.. Specifically, we consider the situation where the language pair of interest S-7 has very limited parallel training data. In such a situation, BT approaches all suffer from serious disadvantage: since the backward model needs to be trained on the parallel data T-S, when the amount of parallel data is small, the resulting backward model has very low quality. The pseudo-parallel corpus generated from the low- low-quality backward model can contaminate the training signals of the forward model (Currey et al., 2017).
To compensate for the lack of initial parallel data to train the backward model, propose to use parallel data from a related language -" for which we can collect substantially more data. Specifically, train the backward model the union of parallel data T-5' and T-S, instead of only T-5. Since this procedure results in a substantially larger set of parallel training data, the obtained backward model has higher quality. However, since the extra S' parallel data dominates the training set of the backward model, the pseudo source sentences sampled from the resulting backward model would have more features of the related language S', rather than our language of interest S.
In principle, MetaBT can fix this discrepancy by adapting the backward model using the forward model's ' gradient on the meta validation set that only contains parallel data for S-T' This would move the back-translated pseudo source sentences closer to our language of interest S.
5 EXPERIMENTS
We evaluate MetaBT in two settings: (1) a standard back-translation setting to verify that MetaBT can create more effective training data for the forward model, and (2) a multilingual NMT setting to confirm that MetaBT is also effective when the backward model is pre-trained on related language pair as discussed in $
5.1 DATASET AND PREPROCESSING
Standard For the standard setting, we consider two large datasets: WMT En-De 2014 and WMT En-Fr 2014', tokenized with SentencePiece (Kudo & Richardson, 2018) using a joint vocabulary size of 32K for each dataset. We filter all training datasets, keeping only sentence pairs where both source and target have more than 200 tokenized subwords, resulting in parallel training corpus of 4.5M sentence pairs for WMT En-De and 40.8M sentences for WMT En-Fr. For the target monolingual data, we collect 250M sentences in German and 61 million sentences in French, both from the WMT news datasets between 2007 and 2017. After de-duplic,,ion, we filter out the sentences that have more than 200 subwords, resulting in 220M German sentences and 60M French sentences.
The multilingual setting uses the multilingual TED talk dataset (Qi et al., 2018), which contains parallel data from 58 languages to English. We focus on translating 4 low fow-resource languages to English: Azerbaijani (az), Belarusian (be), Glacian (gl), Slovak (sk). Each low-resource language is paired with a corresponding related high-resource language: Turkish (tr), Russian (ru), Portuguese (pt). Czech (cs). We following the setting from prior work (Neubig & Hu, 2018: Wang et al., 2019a) and use SentencePiece with separate vocabulary of 8K for each language.
Multilingual
5.2 BASELINES
Our first baseline is No BT, where we train all systems using parallel data only. For the standard setting, we simply train the NMT model on the WMT parallel data. For the multilingual setting, we train the model on the concatenation of the parallel training data from both the low-resource language
Data link: http://www. statmt. wrrg wwt 14/
5

---

Under review as a conference paper at ICLR 2021
and the high-resouree language. The No BT baseline helps to verify the correctness of our model implementations. For the BT baselines, we consider two strong candidates:
we sample the pseudo source sentences from fixed backward model trained with MLE. This baseline the same with sampling-based BT (Edunov al., 2018). We choose sampling instead of beam-search and beam-search with noise as Edunov et al. (2018) found sampling to be stronger than beam-search and on par with noisy beam-search. Our data usage, as specified in $ 5.1, is also the same with Edunov et al. (2018) 8) on WMT. We call this baseline MLE to signify the fact that the backward model trained with MLE and then kept fixed throughout the course of the forward model's learning.
MLE:
DualNMT (Xia et al., 2016): this baseline further improves the quality of the backward model using reinforcement learning with reward that combines the language model score and the reconstruction score from the forward model.
Note that for the multilingual setting, we use top-10 sampling, which we find has better performance than sampling from the whole vocabulary in the preliminary experiments.
5.3 IMPLEMENTATION
We use the Transformer-Base architecture (Vaswani et al., 2017) for all forward and backward models in our experiments' NMT models. All hyper-parameters can be found in $ A.2. We choose Transformer- -oroomeer-BBeeeeee instead of Transformer-Larg because MetaBT requires storing in memory both the forward model and the backward model, as well as the two-step gradients for meta-leaming, which together exceeds our 16G of accelerator memory when we try running Transformer-Large We further discuss this in $ 7
For the standard setup, we pre-train the backward model on the WMT parallel corpora. In the meta-learning phases that we described in $ 3, we initialize the parameters 4 using this pre-trained checkpoint. From this checkpoint, at each training step, our forward model is updated using two sources of data: (1) a batch from the parallel training data, and (2) a batch of sentences from the monolingual data, and their source sentences are sampled by the backward model.
For the multilingual setup, we pre-train the backward model on the reverse direction of the parallel data from both the low-resource and the high-resource languages. From this checkpoint, at each meta-learning step, the forward model receives two sources of data: (1) a batch of the parallel data from the low-resource language. (2) batch of the target English data from the high-resource language, which are fed into the BT model to sample the pseudo-source data.
5.4 RESULTS
Multilingual
Standard
BT Model Objective
az-en
be-en
gl-en
sk-en
en-de
en-fr
No BT
11.50
17.00
28.44
28.19
26.49
38.56
MLE (Edunov al., 2018)
11.30
17.40
29.10
28.70
28.73
39.77
DuaINMT (Xia al., 2016)
11.69
14.81
25.30
27.07
25.71
Meta Back-Translation
11.92"
18.10"
30.30"
29.00
30.39*
40.28"
Table 1: BLEU scores of MetaBT and of our baselines in the standard bilingual setting and the multilingual setting. indicates statistically significant improvements with p < 0.001. Our tests follow (Clark et al., 201 1)..
We report the BLEU scores (Papineni et al., 2002) for all models and settings in Tab. 1. From the table, we observe that the most consistent baseline MLE, which significantly improves over the No BT baseline. Mcanwhile, DualNMT's performance is much weaker, losing to MLE all tasks except for az-en where its margin is only +0.19 BLEU compared to No BT. For WMT En-Fr, we even observe that DuaINMT often results in numerical instability before reaching 34 BLEU score and thus we do not report the result. By comparing the baselines' performance, we can see that continuing to train the backward model outperform MLE is a challenging mission.
Despite such challenge, MetaBT consistently outperforms all baselines in both settings. In particular, compared to the best baselines, MetaBT'ss gain up +1.20 BLEU in the low-resource multilingual

---

Under review as a conference paper at ICLR 2021
setting, and is +1.66 BLEU for WMT En-De 14. We remark that WMT En-De 14 is relatively classical benchmark for NMT and that a gain of +1.66 BLEU on this benchmark is very significant. While the gain of MetaBT over MLE statistical test shows that the gain is still significant. Therefore, our experimental results confirm the theoretical advantages of MetaBT, Next, in & 5.5, investigate the behaviors of MetaBTT to further understand how the method controls the generating process of pscudo-parallel data.
WMT En-Fr is somewhat smaller (+0.51 BLEU), our
5.5
ANALYSIS
27.5
Prob. of BT Samples
0.0089
A/V
MIFT
25.0
Tmin PPL
BLED
0.0080
22.0
0.0075
MLE
17.5
-MBT
1.0070
-MLEE
Figure 3: Training PPL and Validation BLEU for WMT En-De De throughout the forward model's training. MetaBT Icads to consistently higher validation BLEU by generating pseudo-parallel data that avoids overfitting for the forwarwd model, evident by higher training PPL.
Figure 2: Probability of pseudo-paralle from the forward model for WMT 14 En-Fr. MetaBT produces less diverse data fit the model better.
MetaBT Flexibly Avoids both Overfitting and Underfitting. We demonstrate two constrasting behaviors of MetaBT in Fig. 2 and Fig. 3. In Fig. 2, MetaBTT generates pseudo-parallel data for the forward model to learn in WMT En-Fr. Since WMT En-Fr is large (40.8 million parallel sentences), the Transformer- Base forward model underfits. By "observing" the forward model's underfitting, perhaps via a low meta validation performance, the backward model generates the pseudo-parallel data that the forward model assigns high probability, hence reducing the learning difficulty for the forward model. In contrast, Fig. 3 shows that for WMT En-De, the pseudo-parallel data generated by the backward model leads to a higher training loss for the forward model. Since WMT En-De has only 4.5 million parallel sentences which is about 10x smaller than WMT En-Fr, we suspect that MetaBTT generates harder pseudo-paralas data for the backward model to avoid overfitting. In both cases, we have no control over the behaviors of MetaBT, and hence we suspect that MetaBT can appropriately adjusts its behavior depending on the forward model's '' learning state.
-LE MBT 1k
be
50.00 80
0.27
0.85
0555
30.00 100
50000
0:57
0.50
6,7GG
0.78
0:50
0.000
Figure 4: Percentage of words in the pseudo source sentences that are the low-resource vocabulary throughout training. MetaBT learns to favor the sentences that are more similar to the data from the low-resource language.
MetaBT Samples Eseudo-Paralle Data Closer to the Meta Validation Set. After showing that MetaBT can affect the forward model's training opposite ways, we now show that MetaBT actually tries to generate pseudo- parallel data that are closed to the meta validation data. Note that this is the expected behavior of MetaBT, since the ultimate objective is for the forward model to perform well on this meta validation set. We focus the multilingual setting because this setting highlights the vast difference between the parallel data and the meta validation data. In particular, recall from $ 4 that in order translate low-resource language S into language T, we use extra data from language which is related to S but which has abundant parallel data S'- -T. Meanwhile, the meta validation set only consists of parallel sentences in S-T.
7

---

Under review as a conference paper at ICLR 2021
In Fig. 4, we group the sampled sentences throughout the forward model's training into 10 bins based on the training steps that they are generated, and plot the percentage of words in the pseudo source sentences that are from the vocabulary of S for each bin. As seen from the figure. MetaBT keeps increasing the vocabulary coverage throughout training, indicating that it favors the sentences that are more similar to the meta validation data, which are from the low-resource language S.
Qualitative Analysis: MetaBT Generates Fewer Pathological Outputs. In Fig. 5, we plot the histogram of length differences between the reference sentences and the translations of MetaBT and by our baseline MLE on WMT En-De. We observe a con- sistent trend of the MLE baseline generate more sentences with pathological length differences, i.c. more than 15-10 words different from the reference's 's lengths. One such example is il- lustrated in Tab. 2 in $ A.4. We suspect that this happens for MLE because while sampling-based back-translation increases diversity of the outputs and aids overall forward performance, it will still sometimes generate extremely bad pseudo-parallel parallel examples. Forward models that learn from these bad inputs will sometimes produces translations that are completely incorrect, for example being too short or too long, causing the trends in Fig. 5. MetaBT suffers less from this problem because the backward model continues training improve the forward model's dev set performance.
530-
232
Figure 5: Histogram of differences length between the reference and system outputs. MLE-trained BT tends to generate slightly more out- puts with lengths that greatly differ from the reference.
6 RELATED WORK
Our work is related to methods that leverage monolingual data either on the source side (He et al., 2020) or the target side (Sennrich et al., 2016; Edunov et al., 2018) to improve the final translation quality. Going beyond vanilla BT, IterativeBT (Hoang et al., 2018) trains multiple rounds of backward and forward models and observe further improvemntt. While MetaBTT cannot push the backward model's 's quality as well, MetaBT is also much cheaper than multiple training rounds of IterativeBT. DualNMT (Xia et al., 2016) jointly optimizes the backward model with the forward model, but relies on indirect indicators, leading to weak performances as we showed in $ 5.4.
As MetaBT essentially learns generate pseudo-parallel data for effective training, MetaBT is natural extensions of many methods that learn to re-weight or to select extra data for training. For example, Soto et al. (2020) and Dou et al. (2020) select saccstraddltted data from different systems using heuristic, while Wang & Neubig (2019); Lin et al. (2019); Wang et al. (2019b; 2020) select the multilingual data that most helpful for forward model. We find the relationship between MetaBT and these methods analogous to the relationship between sampling from a distribution and computing the distribution's 's density.
The meta-learning technique in our method has also been applied to other tasks, such as: learning initialization points (Finn et al., 2017; Gu et al., 2018), designing architectures (Liu et al., 2019), generating synthetic input images Such et al. (2019), and pseudo labeling (Pham et al., 2020).
7 LIMITATION, FUTURE WORK, AND CONCLUSION
We propose Meta Back- Translation (MetaBT), an algorithm that learns to adjust a fack-translation model to generate data that are most effective for the training of the forward model. Our experiments show that MetaBT outperforms strong existing methods on both a standard NMT setting and multilingual setting.
As discussed in $ 5.3 the large memory footprint is current weakness that makes it impossible to apply MetaBT to larger models. However, the resulting Transformer-Base model trained by MetaBT still outperform Transformer- smaller Transformer- -eaoo model are cheaper to deploy, MetaBT still has its values. In the future, expect this memory limitation will be lifted, e.g. when better technology, such automated model parallelism (Lepikhin et al., 2020) or more powerful accelerators, become available. When that happens, MetaBT's will better realize its potential.
models trained in the standard settings. Since the
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Atilim Gunes Baydin, Robert Cornish, David Rarbio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. ICLR, 2018.
Yong Cheng, Wei Xu, Zhongjun He, Wei Hc, Hua Wu, Maosong Sun, and Yang Liu. Semi-supervised learning for neural machine translation. In ACI., 2016.
Jonathan Clark, Chris Dyer, Alon Lavie, and Noah Smith. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In ACI, 2011. 6
Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. Copied monolingual data improves low-resource neural machine translation. In WMT, 2017.
Zi-Yi Dou, Antonios Anastasopoulos, and Graham Neubig. Dynamic data selection and weighting for iterative bock-translation. arXiv preprint arXiv:2004.03672 2020.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In EMNLP, 2018. 1, 2, 3, 6,
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017. 2, 4,
Jiatao Gu, Yong Wang. Yun Chen, Victor O. Li, and Kyunghyun Cho. Meta-learning for low- resource neural machine translation. In EMNLP, 2018. URL https: /ttps:///wwww.ee www.ww.wwwww.......... org/
anthology/DIs /D18 1398.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. arXiv preprint arXiv: 1503.0353, 2015. 1
Junxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020. 1, 8
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back-translation for neural machine translation.
ACI., 2018. 1,
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 12
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, 2018. 5
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. Arxiv 2006 2006.16668. 16668, 2020.
Yu-Hsiang Lin, Chian-Yo Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. Choosing transfer languages for cross-lingual learning. In ACL, 2019. 8
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. 2019.
4, 8
Graham Neubig and Junjie Hu. Rapid adaptation of neural machine translation to new languages. EMNLP, 2018. 5
Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. compare-mt: A tool for holistic comparison of language generation systems. In NAACL, 2019. 13
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. 6
Hieu Pham, Qizhe Xie, Zihang Dai, and Quoe V. Le. Meta pseudo labels. Arxiv 2003. 10580, 2020. 8

---

Under review as a conference paper at ICLR 2021
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. When and why are pre-trained word embeddings useful for neural machine translation? NAACL, 2018.
Jurgen Schmidhuber. Learning to control fast-weight memories: An alternative dynamic recurrent networks. In Neural Computation, 1992. 2
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL, 2016. 1, 2,
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint 1rX.::9905.225505 2019.
Xabier Soto, Dimitar Shterionov, Alberto Poncelas, and Andy Way. Selecting backtranslated data from multiple sources for improved neural machine translation. In ACL, 2020. 8
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In arxiv, 2019. 8
Ily Sutskever, Oriol Vinyals. and Quoc V. Le. Sequence sequence learning with neural networks. In Advances in Neural Information Processing Systems, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems, 2017. 1, 6, 12
Xinyi Wang and Graham Neubig. Target conditioned sampling: Optimizing data selection for multilingual neural machine translation. ACL, 2019.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham Neubig. Multilingual neural machine translation with soft decoupled encoding. In ICLR, 2019a. 5
Xinyi Wang, Hieu Pham, Paul Mitchel, Antonis Anastasopoulos, Jaime Carbonell, and Graham
Neubig. Optimizing data usage via differentiable rewards. In arxiv, 2019b. URL https:
www.xii..rg/abaa..... 1911. 10088. 4, 8
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural machine translation. In ACL, 2020. 8
Ronald J. Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning. Machine Learning. 1992.
Yingce Xia, Di He, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in Neural Information Processing Systems, 2016. 6,
10


Reviews:
[{"review_id": "x1IcJIMoO4d", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method to improve NMT training with backtranslation.\n\nRather than using a fixed t->s model to translate target monolingual data in order to augment the training set for the s->t model, the proposed approach first pretrains the t->s model as usual then jointly trains it with the forward s->t model using a meta-learning approach: the s->t model is trained on the syntetic backtranslated data and a \"meta-validation\" loss is computed on a paralled dataset, which is used to update the t->s model using REINFORCE.\nThe approach is similar to the DualNMT model by Xia et al, but rather than updating on monolingual data based on LM and reconstruction scores, it uses a reward based on the cross-entropy on parallel data.\nThe paper also proposes a way to adapt this method to a multi-lingual setting.\n\nExperiments are performed on WMT-14 En->De and En->Fr, and on 4 IWSLT-2018 language pairs. The authors report small but consistent improvements. Additional analyses are also reported.\n\nOverall the method seems valid, although it is described at a very high level and no code release is mentioned. In my experience successfully implementing RL-based method is strongly dependent on getting hyperparameters and implementation details right, so it could be hard to reproduce this work without the code or a more detailed description. Also it's not entirely clear what is being used as \"meta-validation\" data here, I suppose it's all the parallel training data, but the paper doesn't make it clear.\n\nMinor issues: the \"Tagged backtranslation\" paper by Caswell et al. 2019 contrasts the claim that improvements with sampling backtranslation are due to increased diversity. It should be referenced as relevant work. The Xia et al., 2016 Dual NMT paper is referenced multiple times in the text but not in the bibliography section", "extra_scores": {}, "extra_text": {}}, {"review_id": "wizsFpQYH8L", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an extension of the back-translation method which provides a means of leveraging monolingual data in NMT where the quality of the data generated by the back-translation model is controlled through a meta learning regime that trains the BT model jointly with the actual translation model.\nThe method is an interesting application of meta-learning to NMT and worth seeing the results. It is a well-written paper with a sound description of the method and evaluation. The motivation is the main weakness and a better discussion and comparison to related work would help clarify the applicability of the method.\n\nComments\n- The motivation for the method is supported by two claims, the first one related to the upper bound on quality of the BT model trained on the parallel data, and the second on the quality of the pseudo-parallel data generated by the BT model used then to train the actual NMT model together with the original parallel data. There is on the other hand not any empirical support in either claim that these create a weakness, thus, grounding the motivation for the paper. \nIn the former case, the method does not really modify in any matter the quality of the original parallel data, hence the discussion is irrelevant. In the latter case, especially, when using the pseudo-parallel data, there are many factors not discussed that affect the quality of the pseudo-parallel data generated by the BT model other the diversity, such as source domain drift and eg. generation of translationese. It is indeed limited the literature on this topic although doesn't validate that any claim should be supported either by references or within the study included in the paper. The second reference in this context is referred to as slow and expensive but isn't the meta learning also making the model much slower and more costly in a similar sense?\n- Can the authors analyze or discuss how well the backward model is improving and how does it distinguish from the previous methods?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "quN69SIwL6K", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an interesting approach for back-translation. The idea is to update both forward model and backward models during training. The forward model is updated in a standard way using synthetic samples generated from the backward model. The backward model is updated using gradient of the forward model on meta-validation dataset (i.e., parallel data). Evaluation shows that Meta BT performs better than offline BT ((Edunov et al., 2018)  in both multilingual setup and standard  en-fr, en-de translation direction. In general, I find the approach is nice.\nWhile the paper compared Meta-BT with offline BT (i.e., MLE as mentioned in the paper) and DualNMT, I think these two baselines are not sufficient to verify the claims made by the paper.\n\nThe paper claims that MetaBT allows to update the backward model unlike offline-BT and avoids expense of multiple iterations in iterative BT. Note that back-translation can be done on-the-fly (Artetxe et al., 2018, Conneau and Lample, 2019). Online back translation allows updating both forward and backward models using monolingual data during training. Thus, I think online BT should be a baseline for appropriate comparison. While MetaBT avoids multiple iterations of iterative BT, the evidence is not provided in the paper in terms of training time for MetaBT. The additional complexity of MetaBT lies in the update of the backward model per mini-batch and the computation of the Jacobian-vector products. As the author already mentioned that MetaBT has a large memory footprint, thus it’s slower to do one update in MetaBT. I wonder how MetaBT performs in comparison with iterative BT with 2 iterations with respect to BLEU scores and training time.\n\nCan the authors provide similar plots in Figure 2 for En-De and in Figure 3 for En-Fr? With respect to MetaBT avoids overfitting. I think it would be nice to have some analysis on the samples generated by the backward model. In comparison to offline BT, does the backward model in MetaBT generate more diverse output?\n\nWith respect to the presentation of the paper, I think Figure 1 is a bit confusing to read. I was hoping to get the main idea from Figure 1 but it didn’t help at all.\n\n\n\n\n**References**\n\nMikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho. Unsupervised Neural Machine Translation. ICLR 2018. \nGuillaume Lample, Alexis Conneau. Cross-lingual Language Model Pretraining. NeurIPS 2019\n\n\n**Post-response update**\nThanks authors for extra effort on semi-supervised experiments. I decided to increase the score to 6. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2aRXecuN6Np", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "summary:\n\nThis paper applies techniques from meta-learning to derive and end-to-end update rule for a workflow involving backtranslation, specificically maximizing translation performance of the forward model, while updating the backward model to produce backtranslations that are maximally useful to improve the forward model's quality (as measured on a meta validation set). The approach is evaluated on WMT EN-DE and EN-FR and compared against a simple sampling strategy for backtranslation, and dual learning. In addition, the paper considers a multilingual setup where the translation direction is low-resource, and the initial backtranslation model is trained on a mix of parallel data from the language pair of interest, as well as auxiliary data with a high-resource, related source language. \n\nstrengths:\n\n+ the idea of end-to-end optimization of the backward model to maximally benefit training of the forward model is novel and interesting.\n\n+ the mathematical derivation of the objective function is sound.\n\nweaknesses:\n\n- the empirical evaluation is not very convincing. Important information is missing, baselines are inexplicably weak, and some other simple baselines are missing. Specifically:\n\n -- what data sets do you use for meta validation? Is it one (or several) of the newstest sets? Since you're actually learning gradients on this data set, rather than just using it for early stopping or hyperparameter choice, I think you should consider using that data directly for training as one of the baselines. For example, Oravecz et al. (2019) report a 1.5-2 BLEU gain from fine-tuning on newstest2008-2017 for EN-DE in their submission to WMT.\n\n -- for the Edunov et al. 2018 baseline (sampling for back-translation), there is a gap of >6 BLEU between the best result they report (35 BLEU) and yours (28.73 BLEU). Some of this may be explainable by the fact that you use Transformer base rather than Transformer large, but even relatively speaking, Edunov et al. (2018) observe an improvement of ~4 BLEU with BT, while this paper only reports ~2. Authors discuss memory limitations as the reason why the did not train on Transformer Big, but they could conceivably use gradient accumulation to enable training even on limited hardware. Other reasons for the score discrepancy deserve discussion (for example, are BLEU scores reported tokenized or not?)\n \n -- please provide more detail how the high-resource data was used for the various baselines. You mention that you follow settings from previous work (Neubig & Hu, 2018; Wang et al., 2019a), but these actually use different techniques. I gather that you use some type (which?) of transfer learning for the forward models. Do you also apply these techniques to the backward models, and if so, is the initial model for meta back-translation initialized differently?\n  \n -- the dual learning baseline bears some conceptual similarity to the proposed objective in that both backward and forward model are continuously improved. I'm surprised to see that it leads to worse performance than the baseline trained on parallel data only, but there is too little information to build trust in this result. Which language model was used for the additional rewards? What beam size (or sample size) was used for the backtranslation? What monolingual data was used, only target-languge data or also source-language data? If the latter, which data was used?\n \n- I find the title problematic. I can see how it was coined as a combination of meta-learning and backtranslation, but it's misleading to a reader who doesn't know this intention. If we take \"meta\" to roughly express some self-referentiality (meta-learning is learning to learn; a meta-analysis is an analysis of analyses), is meta-backtranslation the backtranslation of backtranslations?\n\nrecommendation:\n\nOverall, I vote for rejection. I like the core idea and could see this work being published eventually, but feel that the empirical evaluation needs to be strengthened before I would recommend acceptance. There are open questions about the evaluation setup, and if authors can answer these questions, this will let me better judge the empirical rigour.\n\ntypos:\n\nmulltilingual -> multilingual\n\n\n**post-response update**:\n\nsome of my concerns about the evaluation were resolved in the author response (for example regarding the meta-validation datasets) , and I have slightly increased my rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a meta-learning-based technique to learn how to back-translate (generate a synthetic source-language translation of an observed target-language sentence) for the purpose of better optimising a source-to-target translation model. 

The approach is an interesting novel angle to jointly training the translation model and the back-translation component. Compared to techniques like UNMT and DualNMT, the approach offers reduced training time and a simpler formulation with fewer trainable components (and fewer hyperparameters). 

During the discussion phase the authors provided additional insight, clarifications, and results that improved our perception of the paper. I would personally appreciate if the authors would update their paper with the clarifications they made to points raised by R2, R3, and R4, especially on the details about meta-validation, the discussion about memory footprint, and the additional results on UNMT (and variants). 
Decision: Accept (Poster)