Revisiting Self-Training for Neural Sequence Generation

1 INTRODUCTION . Deep neural networks often require large amounts of labeled data to achieve good performance . However , acquiring labels is a costly process , which motivates research on methods that can effectively utilize unlabeled data to improve performance . Towards this goal , semi-supervised learning ( Chapelle et al. , 2009 ) methods that take advantage of both labeled and unlabeled data are a natural starting point . In the context of sequence generation problems , semi-supervised approaches have been shown to work well in some cases . For example , back-translation ( Sennrich et al. , 2015 ) makes use of the monolingual data on the target side to improve machine translation systems , latent variable models ( Kingma et al. , 2014 ) are employed to incorporate unlabeled source data to facilitate sentence compression ( Miao & Blunsom , 2016 ) or code generation ( Yin et al. , 2018 ) . In this work , we revisit a much older and simpler semi-supervised method , self-training ( ST , Scudder ( 1965 ) ) , where a base model trained with labeled data acts as a “ teacher ” to label the unannotated data , which is then used to augment the original small training set . Then , a “ student ” model is trained with this new training set to yield the final model . Originally designed for classification problems , common wisdom suggests that this method may be effective only when a good fraction of the predictions on unlabeled samples are correct , otherwise mistakes are going to be reinforced ( Zhu & Goldberg , 2009 ) . In the field of natural language processing , some early work have successfully applied self-training to word sense disambiguation ( Yarowsky , 1995 ) and parsing ( McClosky et al. , 2006 ; Reichart & Rappoport , 2007 ; Huang & Harper , 2009 ) . However , self-training has not been studied extensively when the target output is natural language . This is partially because in language generation applications ( e.g . machine translation ) hypotheses are often very far from the ground-truth target , especially in low-resource settings . It is natural to ∗Equal Contribution . Most of the work is done during Junxian ’ s internship at FAIR . 1Code is available at https : //github.com/jxhe/self-training-text-generation . Algorithm 1 Classic Self-training 1 : Train a base model fθ on L = { xi , yi } li=1 2 : repeat 3 : Apply fθ to the unlabeled instances U 4 : Select a subset S ⊂ { ( x , fθ ( x ) ) |x ∈ U } 5 : Train a new model fθ on S ∪ L 6 : until convergence or maximum iterations are reached ask whether self-training can be useful at all in this case . While Ueffing ( 2006 ) and Zhang & Zong ( 2016 ) explored self-training in statistical and neural machine translation , only relatively limited gains were reported and , to the best of our knowledge , it is still unclear what makes self-training work . Moreover , Zhang & Zong ( 2016 ) did not update the decoder parameters when using pseudo parallel data noting that “ synthetic target parts may negatively influence the decoder model of NMT ” . In this paper , we aim to answer two questions : ( 1 ) How does self-training perform in sequence generation tasks like machine translation and text summarization ? Are “ bad ” pseudo targets indeed catastrophic for self-training ? ( 2 ) If self-training helps improving the baseline , what contributes to its success ? What are the important ingredients to make it work ? Towards this end , we first evaluate self-training on a small-scale machine translation task and empirically observe significant performance gains over the supervised baseline ( §3.2 ) , then we perform a comprehensive ablation analysis to understand the key factors that contribute to its success ( §3.3 ) . We find that the decoding method to generate pseudo targets accounts for part of the improvement , but more importantly , the perturbation of hidden states – dropout ( Hinton et al. , 2012 ) – turns out to be a crucial ingredient to prevent self-training from falling into the same local optimum as the base model , and this is responsible for most of the gains . To understand the role of such noise in self-training , we use a toy experiment to analyze how noise effectively propagates labels to nearby inputs , sometimes helping correct incorrect predictions ( §4.1 ) . Motivated by this analysis , we propose to inject additional noise by perturbing also the input . Comprehensive experiments on machine translation and text summarization tasks demonstrate the effectiveness of noisy self-training . 2 SELF-TRAINING . Formally , in conditional sequence generation tasks like machine translation , we have a parallel dataset L = { xi , yi } li=1 and a large unlabeled dataset U = { xj } l+u j=l+1 , where |U | > |L| in most cases . As shown in Algorithm 1 , classic self-training starts from a base model trained with parallel data L , and iteratively applies the current model to obtain predictions on unlabeled instances U , then it incorporates a subset of the pseudo parallel data S to update the current model . There are two key factors : ( 1 ) Selection of the subset S. S is usually selected based on some confidence scores ( e.g . log probability ) ( Yarowsky , 1995 ) but it is also possible for S to be the whole pseudo parallel data ( Zhu & Goldberg , 2009 ) . ( 2 ) Combination of real and pseudo parallel data . A new model is often trained on the two datasets jointly as in back-translation , but this introduces an additional hyper-parameter to weigh the importance of the parallel data relative to the pseudo data ( Edunov et al. , 2018 ) . Another way is to treat them separately – first we train the model on pseudo parallel data S , and then fine-tune it on real data L. In our preliminary experiments , we find that the separate training strategy with the whole pseudo parallel dataset ( i.e . S = { ( x , fθ ( x ) ) |x ∈ U } ) produces better or equal performance for neural sequence generation while being simpler . Therefore , in the remainder of this paper we use this simpler setting . We include quantitative comparison regarding joint training , separate training , and pseudo-parallel data filtering in Appendix B , where separate training is able to match ( or surpass ) the performance of joint training . In self-training , the unsupervised loss LU from unlabeled instances is defined as : LU = −Ex∼p ( x ) Ey∼pθ∗ ( y|x ) log pθ ( y|x ) , ( 1 ) where p ( x ) is the empirical data distribution approximated with samples from S , pθ ( y|x ) is the conditional distribution defined by the model . θ∗ is the parameter from the last iteration ( initially it Methods PT FT baseline – 15.6 ST ( scratch ) 16.8 17.9 ST ( baseline ) 16.5 17.5 Table 1 : Test tokenized BLEU on WMT100K . Self-training results are from the first iteration . “ Scratch ” denotes that the system is initialized randomly and trained from scratch , while “ baseline ” means it is initialized with the baseline model . is set as the parameter of the supervised baseline ) , and fixed within the current iteration . Eq . 1 reveals the connection between self-training and entropy regularization ( Grandvalet & Bengio , 2005 ) . In the context of classification , self-training can be understood from the view of entropy regularization ( Lee , 2013 ) , which favors a low-density separation between classes , a commonly assumed prior for semi-supervised learning ( Chapelle & Zien , 2005 ) . 3 A CASE STUDY ON MACHINE TRANSLATION . To examine the effectiveness of self-training on neural sequence generation , we start by analyzing a machine translation task . We then perform ablation analysis to understand the contributing factors of the performance gains . 3.1 SETUP . We work with the standard WMT 2014 English-German dataset consisting of about 3.9 million training sentence pairs after filtering long and imbalanced pairs . Sentences are encoded using 40K byte-pair codes ( Sennrich et al. , 2016 ) . As a preliminary experiment , we randomly sample 100K sentences from the training set to train the model and use the remaining English sentences as the unlabeled monolingual data . For convenience , we refer to this dataset as WMT100K . Such synthetic setting allows us to have high-quality unlabeled data to verify the performance of self-training . We train with the Base Transformer architecture ( Vaswani et al. , 2017 ) and dropout rate at 0.3 . Full training and optimization parameters can be found in Appendix A.1 . All experiments throughout this paper including the transformer implementation are based on the fairseq toolkit ( Ott et al. , 2019 ) , and all results are in terms of case-sensitive tokenized BLEU ( Papineni et al. , 2002 ) . We use beam search decoding ( beam size 5 ) to create the pseudo targets and to report BLEU on test set . 3.2 OBSERVATIONS . In Figure 1 , we use green bars to show the result of applying self-training for three iterations . We include both ( 1 ) pseudo-training ( PT ) : the first step of self-training where we train a new model ( from scratch ) using only the pseudo parallel data generated by the current model , and ( 2 ) finetuning ( FT ) : the fine-tuned system using real parallel data based on the pretrained model from the PT step . Note that in the fine-tuning step the system is re-initialized from scratch . Surprisingly , we find that the pseudo-training step at the first iteration is able to improve BLEU even if the model is only trained on its own predictions , and fine-tuning further boosts the performance . The test BLEU keeps improving over the first three iterations , until convergence to outperform the initial baseline by 3 BLEU points . This behaviour is unexpected because no new information seems to be injected during this iterative process – target sentences of the monolingual data are from the base model ’ s predictions , thus translation errors are likely to remain , if not magnified . This is different from back-translation where new knowledge may originate from an additional backward translation model and real monolingual targets may help the decoder generate more fluent sentences . One straightforward hypothesis is that the added pseudo-parallel data might implicitly change the training trajectory towards a ( somehow ) better local optimum , given that we train a new model from scratch at each iteration . To rule out this hypothesis , we perform an ablation experiment and initialize θ from the last iteration ( i.e . θ∗ ) . Formally , based on Eq . 1 we have : ∇θLU |θ=θ∗ = −Ex∼p ( x ) [ ∇θEy∼pθ∗ ( y|x ) log pθ ( y|x ) |θ=θ∗ ] = 0 , ( 2 ) because the conditional log likelihood is maximized when pθ ( y|x ) matches the underlying data distribution pθ∗ ( y|x ) . Therefore , the parameter θ should not ( at least not significantly ) change if we initialize it with θ∗ from the last iteration . Table 1 shows the comparison results of these two initialization schemes at the first iteration . Surprisingly , continuing training from the baseline model also yields an improvement of 1.9 BLEU points , comparable to initializing from random . While stochastic optimization introduces randomness in the training process , it is startling that continuing training gives such a non-trivial improvement . Next , we investigate the underlying reasons for this .

Summary: The paper introduces an interesting study that tries to explain why conditional text generation models with autoregressive decoders benefit from self-training on pseudo labels created from the same model. The paper introduces and verifies two hypotheses: 1) Decoding strategy: Since beam search is a biased estimator sampling using it doesn't reflect the learned distribution from the model and hence variations happen that benefit learning. (this partially help).