Self-supervised and Supervised Joint Training for Resource-rich Machine Translation

1 INTRODUCTION . Self-supervised pre-training of text representations ( Peters et al. , 2018 ; Radford et al. , 2018 ) has achieved tremendous success in natural language processing applications . Inspired by BERT ( Devlin et al. , 2019 ) , recent works attempt to leverage sequence-to-sequence model pre-training for Neural Machine Translation ( NMT ) . Generally , these methods comprise two stages : pre-training and finetuning . During the pre-training stage , a proxy task , e.g . the Cloze task ( Devlin et al. , 2019 ) , is used to learn the model parameters on abundant unlabeled monolingual data . In the second stage , the full or partial model is finetuned on a downstream translation task of labeled parallel sentences . When the amount of labeled data is limited , studies have demonstrated the benefit of pre-training for low-resource translation tasks ( Lewis et al. , 2019 ; Song et al. , 2019 ) . In many NMT applications , we are confronted with resource-rich languages which are characterized by millions of labeled parallel sentences . However , for these resource-rich tasks , pre-training representations rarely endows the NMT model with superior quality and , even worse , it sometimes can undermine the model ’ s performance if improperly utilized ( Zhu et al. , 2020 ) . This is partly due to catastrophic forgetting ( French , 1999 ) where prolonged finetuning on large corpora causes the learning to overwhelm the knowledge learned during pre-training . Several mitigation methods have been proposed for resource-rich machine translation ( Edunov et al. , 2019 ; Yang et al. , 2019 ; Zhu et al. , 2020 ) , such as freezing the pre-trained representations during the finetuning stage . In this paper , we study resource-rich machine translation through a different perspective of joint training where in contrast to the conventional two-stage approaches , we train NMT models in a single stage using the self-supervised objective ( on unlabeled monolingual sentences ) in addition to the supervised objective ( on labeled parallel sentences ) . The biggest challenge for this single-stage training paradigm is that self-supervised learning is less useful in joint training because it provides a much weaker learning signal that is easily dominated by the signal obtained through supervised learning . As a result , plausible approaches such as simply combining self-supervised and supervised learning objectives perform not much better than the supervised learning objective by itself . To this end , we introduce an approach to exploit complementary self-supervised learning signals to facilitate supervised learning in a joint training framework . Inspired by chromosomal crossovers ( Rieger et al. , 2012 ) , we propose a new task called crossover encoder-decoder ( or XEnDec ) which takes two training examples as inputs ( called parents ) , shuffles their source sentences , and produces a “ virtual ” sentence ( called offspring ) by a mixture decoder model . The key to our approach is to “ interbreed ” monolingual ( unlabeled ) and parallel ( labeled ) sentences through second filial generation with a crossover encoder-decoder , which we call F2-XEnDec , and train NMT models on the F2 offspring . As the F2 offspring exhibits combinations of traits that differ from those found in either parent , it turns out to be a meaningful objective to learn NMT models from both labeled and unlabeled sentences in a joint training framework . To the best of our knowledge , the proposed F2-XEnDec is among the first joint training approaches that substantially improve resource-rich machine translation . Closest to our work are two-stage approaches by Zhu et al . ( 2020 ) and Yang et al . ( 2019 ) who designed special finetuning objectives . Compared to their approaches , our focus lies on addressing a different challenge which is making self-supervised learning complementary to joint training of supervised NMT models on large labeled parallel corpora . Our experimental results substantiate the competitiveness of the proposed joint training approach . Furthermore , our results suggest that the approach improves the robustness of NMT models ( Belinkov & Bisk , 2018 ; Cheng et al. , 2019 ) . Contemporary NMT systems often lack robustness and therefore suffer from dramatic performance drops when they are exposed to input perturbations , even though these perturbations may not be strong enough to alter the meaning of the input sentence . Our improvement in robustness is interesting as none of the two-stage training approaches have ever reported this behavior . We empirically validate our approach on the WMT ’ 14 English-German and WMT ’ 14 EnglishFrench translation benchmarks which yields an improvement of 2.13 and 1.78 BLEU points over a vanilla Transformer model baseline . We also achieve a new state of the art of 46 BLEU on the WMT ’ 14 English-French translation task when further incorporating the back translation technique into our approach . In summary , our contributions are as follows : 1 . We propose a crossover encoder-decoder ( XEnDec ) that generates ” virtual ” examples over pairs of training examples . We discuss its relation to the standard self-supervised learning objective that can be recovered by XEnDec . 2 . We combine self-supervised and supervised losses in a joint training framework using our proposed F2-XEnDec and show that self-supervised learning is complementary to supervised learning for resource-rich NMT . 3 . Our approach achieves significant improvements on resource-rich translation tasks and exhibits higher robustness against input perturbations , particularly to code-switching noise . 2 BACKGROUND . 2.1 NEURAL MACHINE TRANSLATION . Under the encoder-decoder paradigm ( Bahdanau et al. , 2015 ; Gehring et al. , 2017 ; Vaswani et al. , 2017 ) , the conditional probability P ( y|x ; θ ) of a target-language sentence y = y1 , · · · , yJ given a source-language sentence x = x1 , · · · , xI is modeled as follows : The encoder maps the source sentence x onto a sequence of I word embeddings e ( x ) = e ( x1 ) , ... , e ( xI ) . Then the word embeddings are encoded into their corresponding continuous hidden representations . The decoder acts as a conditional language model that reads embeddings e ( y ) for a shifted copy of y along with the aggregated contextual representations c. For clarity , we denote the input and output in the decoder as z and y , i.e . z = 〈s〉 , y1 , · · · , yJ−1 , where 〈s〉 is a start symbol . Conditioned on the aggregated contextual representation cj and its partial target input z≤j , the decoder generates y as P ( y|x ; θ ) = ∏J j=1 P ( yj |z≤j , cj ; θ ) . The aggregated contextual representation c is often calculated by summarizing the sentence x with an attention mechanism ( Bahdanau et al. , 2015 ) . A byproduct of the attention computation is a noisy alignment matrix A ∈ RJ×I which roughly captures the translation correspondence between target and source words ( Garg et al. , 2019 ) . Generally , NMT optimizes the model parameters θ by minimizing the empirical risk over a parallel training set ( x , y ) ∈ S : LS ( θ ) = E ( x , y ) ∈S [ ` ( f ( x , y ; θ ) , h ( y ) ) ] , ( 1 ) where ` is the cross entropy loss between the model prediction f ( x , y ; θ ) and h ( y ) , and h ( y ) denotes the sequence of one-hot label vectors for y with label smoothing in the Transformer ( Vaswani et al. , 2017 ) . 2.2 PRE-TRAINING FOR NEURAL MACHINE TRANSLATION . Pre-training sequence-to-sequence models for language generation is receiving increasing attention in the machine translation community ( Song et al. , 2019 ; Lewis et al. , 2019 ) . These methods generally comprise two stages : pre-training and finetuning . The pre-training takes advantage of the abundant monolingual corpus U = { y } to learn representations through a self-supervised objective called denoising autoencoder ( Vincent et al. , 2008 ) . The denoising autoencoder aims at reconstructing the original sentence y from one of its corrupted counterparts . Let y be obtained by corrupting y with a noise function n ( · ) and masking words . Then the pseudo parallel data ( y , y ) is fed into the NMT model to compute the reconstruction loss . The self-supervised loss over the monolingual corpus U is defined as : LU ( θ ) = E y∈U [ ` ( f ( y , y ; θ ) , h ( y ) ) ] , ( 2 ) The optimal model parameters θ ? are learned via a self-supervised loss LU ( θ ) and used to initialize downstream models during the finetuning on the parallel training set S. 3 CROSS-BREEDING : F2-XEnDec For resource-rich translation tasks in which a large parallel corpus S and ( virtually ) unlimited monolingual corpora U are available , our goal is to improve translation performance by exploiting selfsupervised signals to complement the supervised learning . In F2-XEnDec , we jointly train NMT models with supervised and self-supervised learning objectives in a single stage . We design a new objective LF2 and construct virtual data ( x̃ , ỹ ) to bridge the parallel data ( in supervised learning ) and the pseudo parallel data ( in self-supervised learning ) . The training loss over the virtual data ( x̃ , ỹ ) is computed as : LF2 ( θ ) = E y∈U E ( xp , yp ) ∈S [ ` ( f ( x̃ , ỹ ; θ ) , h ( ỹ ) ) ] , ( 3 ) where generating ( x̃ , ỹ ) depends on the parallel data ( xp , yp ) and the pseudo parallel data ( y , y ) . We propose a method called crossover encoder-decoder ( XEnDec ) that operates on two sentence pairs . As illustrated in Fig . 1 , the first generation ( Fig . 1 ( 1 ) ) uses XEnDec to combine monolingual sentences , thereby incurring a self-supervised proxy loss LF1 which is equivalent to LU . The second generation ( Fig . 1 ( 2 ) ) applies XEnDec between the offspring of the first generation ( y , y ) and the parallel sentence ( xp , yp ) to introduce LF2 . The final NMT models are optimized jointly on the original translation loss and the above two auxiliary losses . L ( θ ) = LS ( θ ) + LF1 ( θ ) + LF2 ( θ ) , ( 4 ) LF2 in Equation 4 is used to deeply fuse self-supervised and supervised training at instance level , rather than mixing them across instances mechanically . In the remainder of this section , we first detail XEnDec . We then discuss the relation between our approach , pre-training , and adversarial training objectives . Finally , we summarize some important techniques used in our approach and present the overall algorithm . 3.1 CROSSOVER ENCODER-DECODER This section introduces the crossover encoder-decoder ( XEnDec ) – an essential subtask for the proposed method . Different from a conventional encoder-decoder , XEnDec takes two training examples as inputs ( called parents ) , shuffles the parents ’ source sentences and produces a virtual example ( called offspring ) through a mixture decoder model . Fig . 2 illustrates this process . Formally , let ( x , y ) denote a training example where x = x1 , · · · , xI represents a source sentence of I words and y = y1 , · · · , yJ is the corresponding target sentence . In supervised training , x and y are parallel sentences . As we shall see in Section 3.4 , XEnDec can be carried out with and without supervision , although we do not distinguish both cases for now . Given a pair of examples ( x , y ) and ( x′ , y′ ) called parents , the crossover encoder shuffles the two source sequences into a new source sentence x̃ calculated from : x̃i = mixi + ( 1−mi ) x′i , ( 5 ) where m = m1 , · · · , mI ∈ { 0 , 1 } I stands for a series of Bernoulli random variables with each taking the value 1 with probability p. If mi = 1 , then the i-th word in x will be substituted with the word in x′ at the same position . For convenience , the lengths of the two sequences are aligned by padding tokens to the end of the shorter sentence . The crossover decoder employs a mixture model to generate the virtual target sentence . The embedding of the decoder ’ s input z̃ is computed as : e ( z̃j ) = 1 Z [ e ( yj−1 ) I∑ i=1 A ( j−1 ) imi + e ( y ′ j−1 ) I∑ i=1 A′ ( j−1 ) i ( 1−mi ) ] , ( 6 ) where e ( · ) is the embedding function . Z = ∑I i=1A ( j−1 ) imi+A ′ ( j−1 ) i ( 1−mi ) is the normalization term where A and A′ are the alignment matrices for the source sequences x and x′ , respectively . Equation 6 averages embeddings of y and y′ through the latent weights computed by m , A and A′ . The alignment matrix measures the contribution of the source words for generating a specific target word ( Och & Ney , 2004 ; Bahdanau et al. , 2015 ) . For example , Aj : indicates the contribution scores of each source word for the j-th word in the target sentence . For simplicity , this paper uses the attention matrix learned in the NMT model as a noisy alignment matrix ( Garg et al. , 2019 ) . Likewise , we compute label vectors for the crossover decoder by : h ( ỹj ) = 1 Z [ h ( yj ) I∑ i=1 Ajimi + h ( y ′ j ) I∑ i=1 A′ji ( 1−mi ) ] , ( 7 ) The h ( · ) function projects a word onto its label vector , e.g . a one-hot vector . The loss of XEnDec over ( x , y , x′ , y′ ) is computed as the negative log-likelihood of generating the virtual sentence ỹ : ` ( f ( x̃ , ỹ ; θ ) , h ( ỹ ) ) = − logP ( ỹ|x̃ ; θ ) = ∑ j −Ey∈h ( ỹj ) logP ( y|z̃≤j , cj ; θ ) , ( 8 ) Notice that even though we do not directly observe the “ virtual sentences ” z̃ and ỹ , we are still able to compute the loss using their embeddings . In practice , the length of x̃ is |x| whereas lengths of both ỹ and z̃ are max ( |y| , |y′| ) .

Summary: This paper proposes a joint training strategy that combines supervised learning on parallel data and self-supervised learning on monolingual data for NMT. The monolingual sentences are corrupted with word order shuffling and masking. And a cross encoder-decoder is introduced to fuse the parallel source-side sentence and the corrupted monolingual sentence. The NMT models are optimized jointly with cross-entropy loss of parallel data, reconstruction loss for monolingual data, and cross-entropy of the virtual fused data.