Generate, Annotate, and Learn: Generative Models Advance Self-Training and Knowledge Distillation

1 INTRODUCTION . Unlabeled data is abundant in the real world , but task-specific unlabeled data within the scope of a given machine learning problem can be challenging to find . For instance , one can not easily find in-domain unlabeled data conforming to the input distribution of a specific Natural Language Processing ( NLP ) task from the GLUE benchmark ( Wang et al. , 2019b ) . Some NLP tasks require an input comprising a pair of sentences with a particular relationship between them . Moreover , classification datasets typically represent a tailored distribution of text and only include a limited number of class labels . If task-specific unlabeled data were available , one could adopt self-training ( Yarowsky , 1995 ) to automatically annotate unlabeled data with pseudo labels to improve accuracy and robustness of classifiers ( Xie et al. , 2020 ; Carmon et al. , 2019b ) . In addition , one can use knowledge distillation ( Hinton et al. , 2015 ) on fresh task-specific unlabeled data to more effectively compress deep neural networks and ensembles ( Buciluǎ et al. , 2006 ; Chen et al. , 2020c ) . When task-specific unlabeled examples do not exist , one can try to retrieve them from a large and diverse open-domain dataset . For instance , Du et al . ( 2020 ) have used nearest neighbor retrieval to harvest in-domain unlabeled text from the internet , leading to a successful application of selftraining and knowledge distillation to certain NLP tasks . While retrieval can indeed help to find in-domain data for problems with simple inputs , it is not practical for problems with complex input schemes , e.g. , sentence pairs with certain relations and tabular data . Accordingly , self-training and retrieval-based methods have not been widely adopted for NLP tasks , e.g. , on the GLUE benchmark . This paper presents a deceptively simple and general framework called “ generate , annotate , and learn ( GAL ) ” to help advance semi-supervised learning and knowledge distillation on various applications that do not come with unlabeled data . We advocate for the use of language models to synthesize unlabeled tasks-specific data , in lieu of real unlabeled data . We build on recent advances in text generation ( Radford et al. , 2019 ; Gao et al. , 2021 ) , and use powerful generative models to synthesize unlabeled text and tables . Then , we use state-of-the-art classifiers to annotate generated unlabeled data with pseudo labels . Finally , we combine labeled data with pseudo labeled data to train more effective classifiers or for the purpose of knowledge distillation ( KD ) . We motivate GAL by making connections to empirical and vicinal risk minimization ( Vapnik , 1992 ; Chapelle et al. , 2001 ) , and demonstrate its utility by presenting empirical results on a wide range of applications . Our key contributions include : • We propose a simple way to advance SSL , KD , and few-shot learning on NLP by using language models to synthesize large amounts of task-specific unlabeled data . • We link GAL to empirical and vicinal risk minimization , helping explain why GAL works and why synthetic samples from class-conditional language models are not as effective . • We systematically dissect GAL and study the key components leading to its success . • GAL establishes a new SoTA for a single 6-layer transformer on the GLUE test set . • GAL improves prompt-based few-shot learning , providing an average improvement of 1.3 % on four 4-shot learning NLP tasks . • GAL advance self-training for tabular tasks , outperforming XGBoost on 2 out of 4 tasks . 2 RELATED WORK . There has been a surge of interest in improving accuracy and label efficiency of classifiers via : 1 . Self-Supervised pretraining on open-domain unlabeled data in a task-agnostic way ( Peters et al. , 2018 ; Devlin et al. , 2019 ; Chen et al. , 2020b ) , 2 . Self-Training using domain-specific unlabeled data in a task-specific way ( Rosenberg et al. , 2005 ; McClosky et al. , 2006 ; Xie et al. , 2020 ) . While self-supervised learning can be applied to a broad distribution of unlabeled data , self-training requires unlabeled data that at least can be annotated using the same set of class labels available for the downstream task ( Oliver et al. , 2018 ) . For instance , if one is interested in training a classifier to distinguish images of cats and dogs , self-training with images of aircraft is likely not helpful , but it is conceivable that self-supervised learning with images of aircraft can still help . A growing body of recent work suggests that perhaps self-supervised pretraining and self-training are compatible and can be combined to achieve the best semi-supervised learning performance ( Chen et al. , 2020c ; Du et al. , 2020 ) . We corroborate the existing evidence by showing gains from generative self-training . Semi-supervised learning ( SSL ) has a long and rich history in machine learning ( Cooper & Freeman , 1970 ; McLachlan & Ganesalingam , 1982 ; Riloff , 1996 ; Chapelle et al. , 2009 ; Van Engelen & Hoos , 2020 ) . One of the oldest family of SSL algorithms is self-training , a.k.a . self-learning or selflabeling ( Scudder , 1965 ; Fralick , 1967 ; Agrawala , 1970 ; Yarowsky , 1995 ) . Self-training encourages knowledge transfer between a teacher and a student model in such a way that the student can outperform the teacher . Specifically , one leverages the teacher ’ s knowledge to annotate unlabeled data with so-called pseudo labels , and the student learns from a mixture of pseudo- and human-labeled data . Self-training has recently seen renewed interest across vision and NLP applications ( Yalniz et al. , 2019 ; Xie et al. , 2020 ; Zoph et al. , 2020 ; Du et al. , 2020 ) . Recent work aims to combine self-training and consistency regularization to develop powerful SSL algorithms . The key idea is to ensure that the predictions of a classifier on unlabeled examples are robust to strong augmentations ( Berthelot et al. , 2019a ; Sohn et al. , 2020 ; Xie et al. , 2019 ) . We build on prior work and investigate the use of synthetic data within the broad family of self-training methods . Recent theoretical work analyzes self-training for linear models , often under the assumption that the data distribution is ( nearly ) Gaussian ( Carmon et al. , 2019a ; Raghunathan et al. , 2020 ; Chen et al. , 2020d ; Kumar et al. , 2020a ; Oymak & Gulcu , 2020 ) . Wei et al . ( 2021 ) prove that , under “ expansion ” and “ class separation ” assumptions , self-training can lead to more accurate neural network classifiers . We present a theoretical framing of GAL in terms of empirical and vicinal risk minimization ( Vapnik , 1992 ; Chapelle et al. , 2001 ) . An important family of related work uses generative models for SSL by learning features that are useful for both generation and discrimination ( e.g. , Chen et al. , 2020a ; Odena , 2016 ; Dai et al. , 2017 ) . For instance , Kingma et al . ( 2014 ) approach SSL by viewing missing class labels as a set of latent variables and use variational inference to impute missing labels as well as other factors of variation . By contrast , our work does not learn features using generative models and keeps the generative and discriminative processes separate . This offers more flexibility and allows GAL to use self-supervised pretraining methods that are not fully generative . Our work is closely related to recent work on the uses of generative models for data augmentation ( Norouzi et al. , 2020 ; Yang et al. , 2020 ) . Unlike Norouzi et al . ( 2020 ) , we do not use instance- based generative models . Yang et al . ( 2020 ) propose a complex scheme , including data relabeling , data filtering , and two-stage training , to utilize synthetic data . By contrast , we show that a simple mixture of the original data and synthetic data can provide sizable gains . Furthermore , we show a more broad use of generative models on KD and few-shot learning , in addition to tabular tasks . Knowledge Distillation ( KD ) ( Buciluǎ et al. , 2006 ; Hinton et al. , 2015 ) uses a procedure similar to self-training to distill knowledge of an expressive teacher model into a smaller student model . In contrast , self-distillation ( Furlanello et al. , 2018 ; Zhang et al. , 2019a ; Mobahi et al. , 2020 ) uses teacher and student models of equal size , hoping to iteratively refine class labels . Previous work uses unlabeled data ( Buciluǎ et al. , 2006 ) and adversarial training ( Wang et al. , 2018 ) to improve KD . We demonstrate that synthetic data generated by unconditional generative models can improve KD on NLP , outperforming strong KD baselines , which often add more complexity and additional hyper-parameters ( e.g. , Sun et al. , 2019a ; Jiao et al. , 2019 ; Xu et al. , 2020 ; Rashid et al. , 2021 ) . Advanced generative models are able to generate realistic images and text ( Karras et al. , 2017 ; Brock et al. , 2019 ; Karras et al. , 2019 ; Radford et al. , 2019 ; Brown et al. , 2020 ) . The quality of synthetic samples has improved to the extent that deep fake detection has become an important research topic itself ( Zellers et al. , 2019 ; Dolhansky et al. , 2019 ) . Recent work has aimed to utilize classconditional generative models to help improve supervised learning ( Antoniou et al. , 2017 ; Bowles et al. , 2018 ; Zhang et al. , 2019b ; Kumar et al. , 2020b ; Gao et al. , 2020 ) . However , Ravuri & Vinyals ( 2019 ) have shown that images generated by state-of-the-art class-conditional generative models fall short of improving ImageNet classification accuracy , despite strong sample quality scores ( Salimans et al. , 2016 ; Heusel et al. , 2017 ) . Similarly , Kumar et al . ( 2020b ) find that it is difficult for sentences generated by label-conditioned GPT-2 ( Radford et al. , 2019 ) to retain the semantics or pragmatics of a specified category , which leads to poor performance on downstream tasks . We discuss why class-conditional generative models are hardly effective for supervised learning , and instead , focus on unconditional generative models . 3 BACKGROUND ON SELF-TRAINING . Given a labeled dataset L = { ( xi , yi ) } Ni=1 and an unlabeled dataset U = { xj } Mj=1 , we summarize the general family of SSL algorithms known as self-training as : 1 . First , an initial model denoted f1 is trained using supervised learning on the labeled dataset L. 2 . Then , at iteration t , one adopts ft as the teacher model to annotate the unlabeled dataset U using pseudo labels . Optionally , one uses a selection method to pick a subset St ⊆ { ( xj , ft ( xj ) ) } Mj=1 of pseudo labeled examples . 3 . A student model ft+1 is trained to optimize a classification loss on the combination of L and St : ` t+1 = E ( x , y ) ∼ ( L∪St ) H ( y , ft+1 ( x ) ) , ( 1 ) where H ( q , p ) = q > log p is the softmax cross entropy loss , and y is assumed to be a one-hot vector ( original labels ) or a vector of class probabilities ( soft pseudo labels ) . 4 . Self-training iterations are repeated T times or until performance plateaus . Many different variants of the basic self-training algorithm discussed above exist in the literature . These variants differ in the type of pseudo labels used , the selection strategy to filter pseudo labeled examples , the speed at which ft is replaced with ft+1 , the choice of data augmentation strategy in the teacher and student models , and the weighting of the two datasets in the objective ( Berthelot et al. , 2019b ; a ; Xie et al. , 2020 ; Sohn et al. , 2020 ; Du et al. , 2020 ) . An important design choice is the type of pseudo labels used . One can simply use soft class probabilities predicted by a teacher ft ( Du et al. , 2020 ) , sharpened class probabilities ( Berthelot et al. , 2019b ) , or hard labels ( a one-hot vector that is zero except at argmaxft ( x ) ) ( Lee et al. , 2013 ) . Another important consideration is the selection strategy to retain a subset of pseudo-labeled examples . FixMatch ( Sohn et al. , 2020 ) uses a hyper-parameter τ to select examples on which the teacher model has a certain level of confidence , i.e. , St = { ( x , ft ( x ) ) | x ∈ U & max ( ft ( x ) ) ≥ τ } . ( 2 ) NoisyStudent ( Xie et al. , 2020 ) also uses a form of confidence filtering but ensures that the class labels in the selected subset are balanced . In principle , any method for out-of-distribution detection ( Hendrycks & Gimpel , 2016 ) can be adopted for filtering pseudo-labeled examples . We adopt the simplest variant of self-training and limit hyper-parameter tuning to a bare minimum .

Summary: This paper identifies challenges in self-training as a lack of in-domain data (input x). To overcome this challenge, the paper proposes to "generate" in-domain data using large self-supervised LMs. The rest of the process, which are annotation and learning, follows typical self-training and this paper takes the approach of learning with soft-target as (Du et al.) The paper examines the proposed approach (GAL) on GLUE benchmark on KD (table1), SSL on full data (tab2,3), and prompt-based few-shot (tab4), and also examines GAL in tabular tasks (tab 7). On KD, GAL usually shows the largest improvement on 1st iteration and shows minor fluctuations of performance during more iterations (Tab2, 7).