Self-supervised and Supervised Joint Training for Resource-rich Machine Translation

1 INTRODUCTION . Self-supervised pre-training of text representations ( Peters et al. , 2018 ; Radford et al. , 2018 ) has achieved tremendous success in natural language processing applications . Inspired by BERT ( Devlin et al. , 2019 ) , recent works attempt to leverage sequence-to-sequence model pre-training for Neural Machine Translation ( NMT ) . Generally , these methods comprise two stages : pre-training and finetuning . During the pre-training stage , a proxy task , e.g . the Cloze task ( Devlin et al. , 2019 ) , is used to learn the model parameters on abundant unlabeled monolingual data . In the second stage , the full or partial model is finetuned on a downstream translation task of labeled parallel sentences . When the amount of labeled data is limited , studies have demonstrated the benefit of pre-training for low-resource translation tasks ( Lewis et al. , 2019 ; Song et al. , 2019 ) . In many NMT applications , we are confronted with resource-rich languages which are characterized by millions of labeled parallel sentences . However , for these resource-rich tasks , pre-training representations rarely endows the NMT model with superior quality and , even worse , it sometimes can undermine the model ’ s performance if improperly utilized ( Zhu et al. , 2020 ) . This is partly due to catastrophic forgetting ( French , 1999 ) where prolonged finetuning on large corpora causes the learning to overwhelm the knowledge learned during pre-training . Several mitigation methods have been proposed for resource-rich machine translation ( Edunov et al. , 2019 ; Yang et al. , 2019 ; Zhu et al. , 2020 ) , such as freezing the pre-trained representations during the finetuning stage . In this paper , we study resource-rich machine translation through a different perspective of joint training where in contrast to the conventional two-stage approaches , we train NMT models in a single stage using the self-supervised objective ( on unlabeled monolingual sentences ) in addition to the supervised objective ( on labeled parallel sentences ) . The biggest challenge for this single-stage training paradigm is that self-supervised learning is less useful in joint training because it provides a much weaker learning signal that is easily dominated by the signal obtained through supervised learning . As a result , plausible approaches such as simply combining self-supervised and supervised learning objectives perform not much better than the supervised learning objective by itself . To this end , we introduce an approach to exploit complementary self-supervised learning signals to facilitate supervised learning in a joint training framework . Inspired by chromosomal crossovers ( Rieger et al. , 2012 ) , we propose a new task called crossover encoder-decoder ( or XEnDec ) which takes two training examples as inputs ( called parents ) , shuffles their source sentences , and produces a “ virtual ” sentence ( called offspring ) by a mixture decoder model . The key to our approach is to “ interbreed ” monolingual ( unlabeled ) and parallel ( labeled ) sentences through second filial generation with a crossover encoder-decoder , which we call F2-XEnDec , and train NMT models on the F2 offspring . As the F2 offspring exhibits combinations of traits that differ from those found in either parent , it turns out to be a meaningful objective to learn NMT models from both labeled and unlabeled sentences in a joint training framework . To the best of our knowledge , the proposed F2-XEnDec is among the first joint training approaches that substantially improve resource-rich machine translation . Closest to our work are two-stage approaches by Zhu et al . ( 2020 ) and Yang et al . ( 2019 ) who designed special finetuning objectives . Compared to their approaches , our focus lies on addressing a different challenge which is making self-supervised learning complementary to joint training of supervised NMT models on large labeled parallel corpora . Our experimental results substantiate the competitiveness of the proposed joint training approach . Furthermore , our results suggest that the approach improves the robustness of NMT models ( Belinkov & Bisk , 2018 ; Cheng et al. , 2019 ) . Contemporary NMT systems often lack robustness and therefore suffer from dramatic performance drops when they are exposed to input perturbations , even though these perturbations may not be strong enough to alter the meaning of the input sentence . Our improvement in robustness is interesting as none of the two-stage training approaches have ever reported this behavior . We empirically validate our approach on the WMT ’ 14 English-German and WMT ’ 14 EnglishFrench translation benchmarks which yields an improvement of 2.13 and 1.78 BLEU points over a vanilla Transformer model baseline . We also achieve a new state of the art of 46 BLEU on the WMT ’ 14 English-French translation task when further incorporating the back translation technique into our approach . In summary , our contributions are as follows : 1 . We propose a crossover encoder-decoder ( XEnDec ) that generates ” virtual ” examples over pairs of training examples . We discuss its relation to the standard self-supervised learning objective that can be recovered by XEnDec . 2 . We combine self-supervised and supervised losses in a joint training framework using our proposed F2-XEnDec and show that self-supervised learning is complementary to supervised learning for resource-rich NMT . 3 . Our approach achieves significant improvements on resource-rich translation tasks and exhibits higher robustness against input perturbations , particularly to code-switching noise . 2 BACKGROUND . 2.1 NEURAL MACHINE TRANSLATION . Under the encoder-decoder paradigm ( Bahdanau et al. , 2015 ; Gehring et al. , 2017 ; Vaswani et al. , 2017 ) , the conditional probability P ( y|x ; θ ) of a target-language sentence y = y1 , · · · , yJ given a source-language sentence x = x1 , · · · , xI is modeled as follows : The encoder maps the source sentence x onto a sequence of I word embeddings e ( x ) = e ( x1 ) , ... , e ( xI ) . Then the word embeddings are encoded into their corresponding continuous hidden representations . The decoder acts as a conditional language model that reads embeddings e ( y ) for a shifted copy of y along with the aggregated contextual representations c. For clarity , we denote the input and output in the decoder as z and y , i.e . z = 〈s〉 , y1 , · · · , yJ−1 , where 〈s〉 is a start symbol . Conditioned on the aggregated contextual representation cj and its partial target input z≤j , the decoder generates y as P ( y|x ; θ ) = ∏J j=1 P ( yj |z≤j , cj ; θ ) . The aggregated contextual representation c is often calculated by summarizing the sentence x with an attention mechanism ( Bahdanau et al. , 2015 ) . A byproduct of the attention computation is a noisy alignment matrix A ∈ RJ×I which roughly captures the translation correspondence between target and source words ( Garg et al. , 2019 ) . Generally , NMT optimizes the model parameters θ by minimizing the empirical risk over a parallel training set ( x , y ) ∈ S : LS ( θ ) = E ( x , y ) ∈S [ ` ( f ( x , y ; θ ) , h ( y ) ) ] , ( 1 ) where ` is the cross entropy loss between the model prediction f ( x , y ; θ ) and h ( y ) , and h ( y ) denotes the sequence of one-hot label vectors for y with label smoothing in the Transformer ( Vaswani et al. , 2017 ) . 2.2 PRE-TRAINING FOR NEURAL MACHINE TRANSLATION . Pre-training sequence-to-sequence models for language generation is receiving increasing attention in the machine translation community ( Song et al. , 2019 ; Lewis et al. , 2019 ) . These methods generally comprise two stages : pre-training and finetuning . The pre-training takes advantage of the abundant monolingual corpus U = { y } to learn representations through a self-supervised objective called denoising autoencoder ( Vincent et al. , 2008 ) . The denoising autoencoder aims at reconstructing the original sentence y from one of its corrupted counterparts . Let y be obtained by corrupting y with a noise function n ( · ) and masking words . Then the pseudo parallel data ( y , y ) is fed into the NMT model to compute the reconstruction loss . The self-supervised loss over the monolingual corpus U is defined as : LU ( θ ) = E y∈U [ ` ( f ( y , y ; θ ) , h ( y ) ) ] , ( 2 ) The optimal model parameters θ ? are learned via a self-supervised loss LU ( θ ) and used to initialize downstream models during the finetuning on the parallel training set S. 3 CROSS-BREEDING : F2-XEnDec For resource-rich translation tasks in which a large parallel corpus S and ( virtually ) unlimited monolingual corpora U are available , our goal is to improve translation performance by exploiting selfsupervised signals to complement the supervised learning . In F2-XEnDec , we jointly train NMT models with supervised and self-supervised learning objectives in a single stage . We design a new objective LF2 and construct virtual data ( x̃ , ỹ ) to bridge the parallel data ( in supervised learning ) and the pseudo parallel data ( in self-supervised learning ) . The training loss over the virtual data ( x̃ , ỹ ) is computed as : LF2 ( θ ) = E y∈U E ( xp , yp ) ∈S [ ` ( f ( x̃ , ỹ ; θ ) , h ( ỹ ) ) ] , ( 3 ) where generating ( x̃ , ỹ ) depends on the parallel data ( xp , yp ) and the pseudo parallel data ( y , y ) . We propose a method called crossover encoder-decoder ( XEnDec ) that operates on two sentence pairs . As illustrated in Fig . 1 , the first generation ( Fig . 1 ( 1 ) ) uses XEnDec to combine monolingual sentences , thereby incurring a self-supervised proxy loss LF1 which is equivalent to LU . The second generation ( Fig . 1 ( 2 ) ) applies XEnDec between the offspring of the first generation ( y , y ) and the parallel sentence ( xp , yp ) to introduce LF2 . The final NMT models are optimized jointly on the original translation loss and the above two auxiliary losses . L ( θ ) = LS ( θ ) + LF1 ( θ ) + LF2 ( θ ) , ( 4 ) LF2 in Equation 4 is used to deeply fuse self-supervised and supervised training at instance level , rather than mixing them across instances mechanically . In the remainder of this section , we first detail XEnDec . We then discuss the relation between our approach , pre-training , and adversarial training objectives . Finally , we summarize some important techniques used in our approach and present the overall algorithm . 3.1 CROSSOVER ENCODER-DECODER This section introduces the crossover encoder-decoder ( XEnDec ) – an essential subtask for the proposed method . Different from a conventional encoder-decoder , XEnDec takes two training examples as inputs ( called parents ) , shuffles the parents ’ source sentences and produces a virtual example ( called offspring ) through a mixture decoder model . Fig . 2 illustrates this process . Formally , let ( x , y ) denote a training example where x = x1 , · · · , xI represents a source sentence of I words and y = y1 , · · · , yJ is the corresponding target sentence . In supervised training , x and y are parallel sentences . As we shall see in Section 3.4 , XEnDec can be carried out with and without supervision , although we do not distinguish both cases for now . Given a pair of examples ( x , y ) and ( x′ , y′ ) called parents , the crossover encoder shuffles the two source sequences into a new source sentence x̃ calculated from : x̃i = mixi + ( 1−mi ) x′i , ( 5 ) where m = m1 , · · · , mI ∈ { 0 , 1 } I stands for a series of Bernoulli random variables with each taking the value 1 with probability p. If mi = 1 , then the i-th word in x will be substituted with the word in x′ at the same position . For convenience , the lengths of the two sequences are aligned by padding tokens to the end of the shorter sentence . The crossover decoder employs a mixture model to generate the virtual target sentence . The embedding of the decoder ’ s input z̃ is computed as : e ( z̃j ) = 1 Z [ e ( yj−1 ) I∑ i=1 A ( j−1 ) imi + e ( y ′ j−1 ) I∑ i=1 A′ ( j−1 ) i ( 1−mi ) ] , ( 6 ) where e ( · ) is the embedding function . Z = ∑I i=1A ( j−1 ) imi+A ′ ( j−1 ) i ( 1−mi ) is the normalization term where A and A′ are the alignment matrices for the source sequences x and x′ , respectively . Equation 6 averages embeddings of y and y′ through the latent weights computed by m , A and A′ . The alignment matrix measures the contribution of the source words for generating a specific target word ( Och & Ney , 2004 ; Bahdanau et al. , 2015 ) . For example , Aj : indicates the contribution scores of each source word for the j-th word in the target sentence . For simplicity , this paper uses the attention matrix learned in the NMT model as a noisy alignment matrix ( Garg et al. , 2019 ) . Likewise , we compute label vectors for the crossover decoder by : h ( ỹj ) = 1 Z [ h ( yj ) I∑ i=1 Ajimi + h ( y ′ j ) I∑ i=1 A′ji ( 1−mi ) ] , ( 7 ) The h ( · ) function projects a word onto its label vector , e.g . a one-hot vector . The loss of XEnDec over ( x , y , x′ , y′ ) is computed as the negative log-likelihood of generating the virtual sentence ỹ : ` ( f ( x̃ , ỹ ; θ ) , h ( ỹ ) ) = − logP ( ỹ|x̃ ; θ ) = ∑ j −Ey∈h ( ỹj ) logP ( y|z̃≤j , cj ; θ ) , ( 8 ) Notice that even though we do not directly observe the “ virtual sentences ” z̃ and ỹ , we are still able to compute the loss using their embeddings . In practice , the length of x̃ is |x| whereas lengths of both ỹ and z̃ are max ( |y| , |y′| ) .

Summary: This paper introduces a new approach to semi-supervised training of neural machine translation. During training, the traditional supervised loss is complemented with two auxiliary losses: a denoising autoencoder and what the authors call cross-breeding: shuffling together the source side of a sentence pair with an unrelated, noised monolingual sentence, and predicting a virtual target sentence whose embedings and labels are a linear interpolation of the paired target sentence and the monolingual sentence, with interpolation weights being based on an attention matrix of the input words selected during shuffling. Authors report an improvement of around 2 BLEU on WMT14 EN-DE and EN-FR over a purely supervised Transformer, and improvements of 1 BLEU (EN-DE) and 0.4 BLEU (EN-FR) over a system with back-translation. Additionally, the system shows robustness against a type of "code-switching" noise.