Meta Back-Translation

1 INTRODUCTION . While Neural Machine Translation ( NMT ) delivers state-of-the-art performance across many translation tasks , this performance is usually contingent on the existence of large amounts of training data ( Sutskever et al. , 2014 ; Vaswani et al. , 2017 ) . Since large parallel training datasets are often unavailable for many languages and domains , various methods have been developed to leverage abundant monolingual corpora ( Gulcehre et al. , 2015 ; Cheng et al. , 2016 ; Sennrich et al. , 2016 ; Xia et al. , 2016 ; Hoang et al. , 2018 ; Song et al. , 2019 ; He et al. , 2020 ) . Among such methods , one particularly popular approach is back-translation ( BT ; Sennrich et al . ( 2016 ) ) . In BT , in order to train a source-to-target translation model , i.e. , the forward model , one first trains a target-to-source translation model , i.e. , the backward model . This backward model is then employed to translate monolingual data from the target language into the source language , resulting in a pseudoparallel corpus . This pseudo-parallel corpus is then combined with the real parallel corpus to train the final forward translation model . While the resulting forward model from BT typically enjoys a significant boost in translation quality , we identify that BT inherently carries two weaknesses . First , while the backward model provides a natural way to utilize monolingual data in the target language , the backward model itself is still trained on the parallel corpus . This means that the backward model ’ s quality is as limited as that of a forward model trained in the vanilla setting . Hoang et al . ( 2018 ) proposed iterative BT to avoid this weakness , but this technique requires multiple rounds of retraining models in both directions which are slow and expensive . Second , we do not understand how the pseudo-parallel data translated by the backward model affects the forward model ’ s performance . For example , Edunov et al . ( 2018 ) has observed that pseudoparallel data generated by sampling or by beam-searching with noise from the backward model train better forward models , even though these generating methods typically result in lower BLEU scores compared to standard beam search . While Edunov et al . ( 2018 ) associated their observation to the diversity of the generated pseudo-parallel data , diversity alone is obviously insufficient – some degree of quality is necessary as well . In summary , while BT is an important technique , training a good backward model for BT is either hard or slow and expensive , and even if we have a good backward model , there is no single recipe how to use it to train a good forward model . In this paper , we propose a novel technique to alleviate both aforementioned weaknesses of BT . Unlike vanilla BT , which keeps the trained backward model fixed and merely uses it to generate pseudo- parallel data to train the forward model , we continue to update the backward model throughout the forward model ’ s training . Specifically , we update the backward model to improve the forward model ’ s performance on a held-out set of ground truth parallel data . We provide an illustrative example of our method in Fig . 1 , where we highlight how the forward model ’ s held-out set performance depends on the pseudo-parallel data sampled from the backward model . This dependency allows us to mathematically derive an end-to-end update rule to continue training the backward model throughout the forward model ’ s training . As our derivation technique is similar to meta-learning ( Schmidhuber , 1992 ; Finn et al. , 2017 ) , we name our method Meta Back-Translation ( MetaBT ) . In theory , MetaBT effectively resolves both aforementioned weaknesses of vanilla BT . First , the backward model continues its training based on its own generated pseudo-parallel data , and hence is no longer limited to the available parallel data . Furthermore , MetaBT only trains one backward model and then trains one pair of forward model and backward model , eschewing the expense of multiple iterations in Iterative BT ( Hoang et al. , 2018 ) . Second , since MetaBT updates its backward model in an end-to-end manner based on the forward model ’ s performance on a held-out set , MetaBT no longer needs to explicitly understand the effect of its generated pseudo-parallel data on the forward model ’ s quality . Our empirical experiments verify the theoretical advantages of MetaBT with definitive improvements over strong BT baselines on various settings . In particular , on the classical benchmark of WMT En-De 2014 , MetaBT leads to +1.66 BLEU score over sampling-based BT . Additionally , we discover that MetaBT allows us to extend the initial parallel training set of the backward model by including parallel data from slightly different languages . Since MetaBT continues to refine the backward model , the negative effect of language discrepancy is eventually rebated throughout the forward model ’ s training , boosting up to +1.20 BLEU score for low-resource translation tasks . 2 A PROBABILISTIC PERSPECTIVE OF BACK-TRANSLATION . To facilitate the discussion of MetaBT , we introduce a probabilistic framework to interpret BT . Our framework helps to analyze the advantages and disadvantages of a few methods to generate pseudo-parallel data such as sampling , beam-searching , and beam-searching with noise ( Sennrich et al. , 2016 ; Edunov et al. , 2018 ) . Analyses of these generating methods within our framework also motivates MetaBT and further allows us to mathematically derive MetaBT ’ s update rules in § 3 . Our Probabilistic Framework . We treat a language S as a probability distribution over all possible sequences of tokens . Formally , we denote by PS ( x ) the distribution of a random variable x , whose each instance x is a sequence of tokens . To translate from a source language S into a target language T , we learn the conditional distribution PS , T ( y|x ) for sentences from the languages S and T with a parameterized probabilistic model P ( y|x ; θ ) . Ideally , we learn θ by minimizing the objective : J ( θ ) = Ex , y∼PS , T ( x , y ) [ ` ( x , y ; θ ) ] where ` ( x , y ; θ ) = −logP ( y|x ; θ ) ( 1 ) Since PS , T ( x , y ) = PS , T ( y ) PS , T ( x|y ) = PT ( y ) PS , T ( x|y ) , we can refactor J ( θ ) from Eq . 1 as : J ( θ ) = Ey∼PT ( y ) Ex∼PS , T ( x|y ) [ ` ( x , y ; θ ) ] ( 2 ) Motivating BT . In BT , since it is not feasible to draw exact samples y ∼ PT ( y ) and x ∼ PS , T ( x|y ) , we rely on two approximations . First , instead of sampling y ∼ PT ( y ) , we collect a corpus DT of monolingual data in the target language T and draw the samples y ∼ Uniform ( DT ) . Second , instead of sampling x ∼ PS , T ( x|y ) , we derive an approximate distribution P̂ ( x|y ) and sample x ∼ P̂ ( x|y ) . Before we explain the derivation of P̂ ( x|y ) , let us state that with these approximations , the objective J ( θ ) from Eq . 2 becomes the BT following objective : ĴBT ( θ ) = Ey∼Uniform ( DT ) Ex∼P̂ ( x|y ) [ ` ( x , y ; θ ) ] ( 3 ) Rather unsurprisingly , P̂ ( x|y ) in Eq . 3 above is derived from a pre-trained parameterized backward translation model P ( x|y ; ψ ) . For example : • P̂ ( x|y ) 4= 1 [ x = argmaxẋ P ( ẋ|y ; ψ ) ] results in BT via beam-search ( Sennrich et al. , 2016 ) . • P̂ ( x|y ) 4= P ( x|y ; ψ ) results in BT via sampling ( Edunov et al. , 2018 ) . • P̂ ( x|y ) 4= 1 [ x = argmaxẋ P̃ ( ẋ|y ; ψ ) ] results in BT via noisy beam-search ( Edunov et al. , 2018 ) where P̃ ( x|y ; ψ ) denotes the joint distribution of the backward model P ( x|y ; ψ ) and the noise . Therefore , we have shown that in our probabilistic framework for BT , three common techniques to generate pseudo-parallel data from a pre-trained backward model correspond to different derivations from the backward model ’ s distribution P ( x|y ; ψ ) . Our framework naturally motivates two questions : ( 1 ) given a translation task , how do we tell which derivation of P̂ ( x|y ) from P ( x|y , ψ ) is better than another ? and ( 2 ) can we derive better choices for P̂ ( x|y ) from a pre-trained backward model P ( x|y ; ψ ) according to the answer of question ( 1 ) ? Metric for the Generating Methods . In the existing literature , the answer for our first question is relatively straightforward . Since most papers view the method of generating pseudo-parallel data as a hyper-level design , i.e . similar to the choice of an architecture like Transformer or LSTM , and hence practitioners choose one method over another based on the performance of the resulting forward model on held-out validation sets . Automatically Derive Good Generating Methods . We now turn to the second question that our probabilistic framework motivates . Thanks to the generality of our framework , every choice for P̂ ( x|y ) results in an optimization objective . Using this objective , we can train a forward model and measure its validation performance to evaluate our choice of P̂ ( x|y ) . This process of choosing and evaluating P̂ ( x|y ) can be posed as the following bi-level optimization problem : Outer loop : P̂ ∗ = argmax P̂ ValidPerformance ( θ∗ P̂ ) , Inner loop : θ∗ P̂ = argmin θ ĴBT ( θ ; P̂ ) , where ĴBT ( θ ; P̂ ) = Ey∼Uniform ( DT ) Ex∼P̂ ( x|y ) [ ` ( x , y ; θ ) ] ( 4 ) The optimal solution of this bi-level optimization problem can potentially train a forward model that generalizes well , as the forward model learns on a pseudo-parallel dataset and yet achieves a good performance on a held-out validation set . Unfortunately , directly solving this optimization problem is not feasible . Not only is the inner loop quite expensive as it includes training a forward model from scratch according to P̂ , the outer loop is also poorly defined as we do not have any restriction on the space that P̂ can take . Next , in § 3 , we introduce a restriction on the space that P̂ can take , and show that our restriction turns the task of choosing P̂ into a differentiable problem which can be solved with gradient descent . 3 META BACK-TRANSLATION . Continuing our discussion from § 2 , we design Meta Back-Translation ( MetaBT ) which finds a strategy to generate pseudo-parallel data from a pre-trained backward model such that if a forward model training on the generated pseudo-parallel data , it will achieve a strong performance on a held-out validation set . The Usage of “ Validation ” Data . Throughout this section , readers will see that MetaBT makes extensive use of the “ validation ” set to provide feedback for refine the pseudo-parallel data ’ s generating strategy . Thus , to avoid nullifying the meaning of a held-out validation set , we henceforth refer to the ground-truth parallel dataset where the forward model ’ s performance is measured throughout its training as the meta validation dataset and denote it by DMetaDev . Other than this meta validation set , we also have a separate validation set for hyper-parameter tuning and model selection . A Differentiable Bi-level Optimization Problem . We now discuss MetaBT , starting with formulating a differentiable version of Problem 4 . Suppose we have pre-trained a paramterized backward translation model P ( x|y ; ψ ) . Instead of designing the generating distribution P̂ ( x|y ) by applying actions such as sampling or beam-search to P ( x|y ; ψ ) , we let P̂ ( x|y ) 4= P ( x|y ; ψ ) and continue to update the backmodel ’ s parameters ψ throughout the course of training the forward model . Clearly , under this association P̂ ( x|y ) 4= P ( x|y ; ψ ) , the parameters ψ controls the generating distribution of the pseudo-parallel data to train the forward model . By setting the differentiable parameters ψ as the optimization variable for the outer loop , we turn the intractable Problem 4 into a differentiable one : Outer loop : ψ∗ = argmax ψ Performance ( θ∗ ( ψ ) , DMetaDev ) Inner loop : θ∗ ( ψ ) = argmin θ Ey∼Uniform ( DT ) Ex∼P̂ ( x|y ) [ ` ( x , y ; θ ) ] ( 5 ) Bi-level optimization problems whose both outer and inner loops operate on differentiable variables like Problem 5 have appeared repeatedly in the recent literature of meta-learning , spanning many areas such as learning initialization ( Finn et al. , 2017 ) , learning hyper-parameters ( Baydin et al. , 2018 ) , designing architectures ( Liu et al. , 2019 ) , and reweighting examples ( Wang et al. , 2019b ) . We thus follow their successful techniques and design a two-phase alternative update rule for the forward model ’ s parameters θ in the inner loop and the backward model ’ s parameters ψ in the outer loop : Phase 1 : Update the Forward Parameters θ . Given a batch of monolingual target data y ∼ Uniform ( DT ) , we sample the pseudo-parallel data ( x̂ ∼ P ( x|y ; ψ ) , y ) and update θ as if ( x̂ , y ) was real data . For simplicity , assuming that θ is updated using gradient descent on ( x̂ , y ) , using a learning rate ηθ , then we have : θt = θt−1 − ηθ∇θ ` ( x̂ , y ; θ ) ( 6 ) Phase 2 : Update the Backward Parameters ψ . Note that Eq . 6 means that θt depends on ψ , because x̂ is sampled from a distribution parameterized by ψ . This dependency allows us to compute the meta validation loss of the forward model at θt , which we denote by J ( θt ( ψ ) , DMetaDev ) , and back-propagate this loss to compute the gradient∇ψJ ( θt ( ψ ) , DMetaDev ) . Once we have this gradient , we can perform a gradient-based update on the backward parameter ψ with learning rate ηψ : ψt = ψt−1 − ηψ∇ψ∇θJ ( θt ( ψ ) , DMetaDev ) ( 7 ) Computing∇ψJ ( θt ( ψ ) , DMetaDev ) . Our derivation of this gradient utilizes two techniques : ( 1 ) the chain rule to differentiate J ( θt ( ψ ) , DMetaDev ) with respect to ψ via θt ; and ( 2 ) the log-gradient trick from reinforcement learning literature ( Williams , 1992 ) to propagate gradients through the sampling of pseudo-source x̂ . We refer readers to § A.1 for the full derivation . Here , we present the final result : ∇ψJ ( θt ( ψ ) , DMetaDev ) ≈ − [ ∇θJ ( θt , DMetaDev ) > · ∇θ ` ( x̂ , y ; θt−1 ) ] · ∇ψlogP ( x̂|y ; ψ ) ( 8 ) In our implementation , we leverage the recent advances in high-order AutoGrad tools to efficiently compute the gradient dot-product term via Jacobian-vector products . By alternating the update rules in Eq . 6 and Eq . 7 , we have the complete MetaBT algorithm . Remark : An Alternative Interpretation of MetaBT . The update rule of the backward model in Eq . 8 strongly resembles the REINFORCE equation from the reinforcement learning literature . This similarity suggests that the backward model is trained as if it were an agent in reinforcement learning . From this perspective , the backward model is trained so that the pseudo-parallel data sampled from it would maximize the “ reward ” : R ( x̂ ) = ∇θJ ( θt , DMetaDev ) > · ∇θ ` ( x̂ , y ; θt−1 ) ( 9 ) Since this dot-product measures the similarity in directions of the two gradients , it can be interpreted that MetaBT optimizes the backward model so that the forward model ’ s gradient on pseudo-parallel data sampled from the backward model is similar to the forward model ’ s gradient computed on the meta validation set . This is a desirable goal because the reward guides the backward model ’ s parameters to favor samples that are similar to those in the meta validation set .

Summary: This paper applies techniques from meta-learning to derive and end-to-end update rule for a workflow involving backtranslation, specificically maximizing translation performance of the forward model, while updating the backward model to produce backtranslations that are maximally useful to improve the forward model's quality (as measured on a meta validation set). The approach is evaluated on WMT EN-DE and EN-FR and compared against a simple sampling strategy for backtranslation, and dual learning. In addition, the paper considers a multilingual setup where the translation direction is low-resource, and the initial backtranslation model is trained on a mix of parallel data from the language pair of interest, as well as auxiliary data with a high-resource, related source language.