Year: 2021

Abstract:
Semi-Supervised Learning (SSL) has seen success in many application domains, but this success often relies on the availability of task-specific unlabeled data. Knowledge distillation (KD) has enabled compressing deep networks, achieving the best results when distilling knowledge on fresh task-specific unlabeled data. However, task-specific unlabeled data can be challenging to find, especially for NLP. We present a simple framework called “generate, annotate, and learn (GAL)” that uses unconditional language models to synthesize in-domain unlabeled data, helping advance SSL and KD on NLP and tabular tasks. To obtain strong taskspecific generative models, we either fine-tune a large language model (LLM) on inputs from specific tasks, or prompt a LLM with a few input examples to generate more unlabeled examples. Then, we use existing classifiers to annotate generated unlabeled examples with pseudo labels, which are used as additional training data or as additional prompts. GAL improves prompt-based few-shot learning on several NLP tasks. It also yields a new state-of-the-art for 6-layer transformers on the GLUE leaderboard. Finally, self-training with GAL offers large gains on four tabular tasks from the UCI repository.

heading: 1 INTRODUCTION
text: Unlabeled data is abundant in the real world, but task-specific unlabeled data within the scope of a given machine learning problem can be challenging to find. For instance, one cannot easily find in-domain unlabeled data conforming to the input distribution of a specific Natural Language Processing (NLP) task from the GLUE benchmark (Wang et al., 2019b). Some NLP tasks require an input comprising a pair of sentences with a particular relationship between them. Moreover, classification datasets typically represent a tailored distribution of text and only include a limited number of class labels. If task-specific unlabeled data were available, one could adopt self-training (Yarowsky, 1995) to automatically annotate unlabeled data with pseudo labels to improve accuracy and robustness of classifiers (Xie et al., 2020; Carmon et al., 2019b). In addition, one can use knowledge distillation (Hinton et al., 2015) on fresh task-specific unlabeled data to more effectively compress deep neural networks and ensembles (Buciluǎ et al., 2006; Chen et al., 2020c).
When task-specific unlabeled examples do not exist, one can try to retrieve them from a large and diverse open-domain dataset. For instance, Du et al. (2020) have used nearest neighbor retrieval to harvest in-domain unlabeled text from the internet, leading to a successful application of selftraining and knowledge distillation to certain NLP tasks. While retrieval can indeed help to find in-domain data for problems with simple inputs, it is not practical for problems with complex input schemes, e.g., sentence pairs with certain relations and tabular data. Accordingly, self-training and retrieval-based methods have not been widely adopted for NLP tasks, e.g., on the GLUE benchmark.
This paper presents a deceptively simple and general framework called “generate, annotate, and learn (GAL)” to help advance semi-supervised learning and knowledge distillation on various applications that do not come with unlabeled data. We advocate for the use of language models to synthesize unlabeled tasks-specific data, in lieu of real unlabeled data. We build on recent advances in text generation (Radford et al., 2019; Gao et al., 2021), and use powerful generative models to synthesize unlabeled text and tables. Then, we use state-of-the-art classifiers to annotate generated unlabeled data with pseudo labels. Finally, we combine labeled data with pseudo labeled data to train more effective classifiers or for the purpose of knowledge distillation (KD).
We motivate GAL by making connections to empirical and vicinal risk minimization (Vapnik, 1992; Chapelle et al., 2001), and demonstrate its utility by presenting empirical results on a wide range of applications. Our key contributions include: • We propose a simple way to advance SSL, KD, and few-shot learning on NLP by using language
models to synthesize large amounts of task-specific unlabeled data. • We link GAL to empirical and vicinal risk minimization, helping explain why GAL works and
why synthetic samples from class-conditional language models are not as effective. • We systematically dissect GAL and study the key components leading to its success. • GAL establishes a new SoTA for a single 6-layer transformer on the GLUE test set. • GAL improves prompt-based few-shot learning, providing an average improvement of 1.3% on
four 4-shot learning NLP tasks. • GAL advance self-training for tabular tasks, outperforming XGBoost on 2 out of 4 tasks.

heading: 2 RELATED WORK
text: There has been a surge of interest in improving accuracy and label efficiency of classifiers via: 1. Self-Supervised pretraining on open-domain unlabeled data in a task-agnostic way (Peters et al.,
2018; Devlin et al., 2019; Chen et al., 2020b), 2. Self-Training using domain-specific unlabeled data in a task-specific way (Rosenberg et al.,
2005; McClosky et al., 2006; Xie et al., 2020). While self-supervised learning can be applied to a broad distribution of unlabeled data, self-training requires unlabeled data that at least can be annotated using the same set of class labels available for the downstream task (Oliver et al., 2018). For instance, if one is interested in training a classifier to distinguish images of cats and dogs, self-training with images of aircraft is likely not helpful, but it is conceivable that self-supervised learning with images of aircraft can still help. A growing body of recent work suggests that perhaps self-supervised pretraining and self-training are compatible and can be combined to achieve the best semi-supervised learning performance (Chen et al., 2020c; Du et al., 2020). We corroborate the existing evidence by showing gains from generative self-training.
Semi-supervised learning (SSL) has a long and rich history in machine learning (Cooper & Freeman, 1970; McLachlan & Ganesalingam, 1982; Riloff, 1996; Chapelle et al., 2009; Van Engelen & Hoos, 2020). One of the oldest family of SSL algorithms is self-training, a.k.a. self-learning or selflabeling (Scudder, 1965; Fralick, 1967; Agrawala, 1970; Yarowsky, 1995). Self-training encourages knowledge transfer between a teacher and a student model in such a way that the student can outperform the teacher. Specifically, one leverages the teacher’s knowledge to annotate unlabeled data with so-called pseudo labels, and the student learns from a mixture of pseudo- and human-labeled data. Self-training has recently seen renewed interest across vision and NLP applications (Yalniz et al., 2019; Xie et al., 2020; Zoph et al., 2020; Du et al., 2020). Recent work aims to combine self-training and consistency regularization to develop powerful SSL algorithms. The key idea is to ensure that the predictions of a classifier on unlabeled examples are robust to strong augmentations (Berthelot et al., 2019a; Sohn et al., 2020; Xie et al., 2019). We build on prior work and investigate the use of synthetic data within the broad family of self-training methods.
Recent theoretical work analyzes self-training for linear models, often under the assumption that the data distribution is (nearly) Gaussian (Carmon et al., 2019a; Raghunathan et al., 2020; Chen et al., 2020d; Kumar et al., 2020a; Oymak & Gulcu, 2020). Wei et al. (2021) prove that, under “expansion” and “class separation” assumptions, self-training can lead to more accurate neural network classifiers. We present a theoretical framing of GAL in terms of empirical and vicinal risk minimization (Vapnik, 1992; Chapelle et al., 2001).
An important family of related work uses generative models for SSL by learning features that are useful for both generation and discrimination (e.g., Chen et al., 2020a; Odena, 2016; Dai et al., 2017). For instance, Kingma et al. (2014) approach SSL by viewing missing class labels as a set of latent variables and use variational inference to impute missing labels as well as other factors of variation. By contrast, our work does not learn features using generative models and keeps the generative and discriminative processes separate. This offers more flexibility and allows GAL to use self-supervised pretraining methods that are not fully generative.
Our work is closely related to recent work on the uses of generative models for data augmentation (Norouzi et al., 2020; Yang et al., 2020). Unlike Norouzi et al. (2020), we do not use instance-
based generative models. Yang et al. (2020) propose a complex scheme, including data relabeling, data filtering, and two-stage training, to utilize synthetic data. By contrast, we show that a simple mixture of the original data and synthetic data can provide sizable gains. Furthermore, we show a more broad use of generative models on KD and few-shot learning, in addition to tabular tasks.
Knowledge Distillation (KD) (Buciluǎ et al., 2006; Hinton et al., 2015) uses a procedure similar to self-training to distill knowledge of an expressive teacher model into a smaller student model. In contrast, self-distillation (Furlanello et al., 2018; Zhang et al., 2019a; Mobahi et al., 2020) uses teacher and student models of equal size, hoping to iteratively refine class labels. Previous work uses unlabeled data (Buciluǎ et al., 2006) and adversarial training (Wang et al., 2018) to improve KD. We demonstrate that synthetic data generated by unconditional generative models can improve KD on NLP, outperforming strong KD baselines, which often add more complexity and additional hyper-parameters (e.g., Sun et al., 2019a; Jiao et al., 2019; Xu et al., 2020; Rashid et al., 2021).
Advanced generative models are able to generate realistic images and text (Karras et al., 2017; Brock et al., 2019; Karras et al., 2019; Radford et al., 2019; Brown et al., 2020). The quality of synthetic samples has improved to the extent that deep fake detection has become an important research topic itself (Zellers et al., 2019; Dolhansky et al., 2019). Recent work has aimed to utilize classconditional generative models to help improve supervised learning (Antoniou et al., 2017; Bowles et al., 2018; Zhang et al., 2019b; Kumar et al., 2020b; Gao et al., 2020). However, Ravuri & Vinyals (2019) have shown that images generated by state-of-the-art class-conditional generative models fall short of improving ImageNet classification accuracy, despite strong sample quality scores (Salimans et al., 2016; Heusel et al., 2017). Similarly, Kumar et al. (2020b) find that it is difficult for sentences generated by label-conditioned GPT-2 (Radford et al., 2019) to retain the semantics or pragmatics of a specified category, which leads to poor performance on downstream tasks. We discuss why class-conditional generative models are hardly effective for supervised learning, and instead, focus on unconditional generative models.

heading: 3 BACKGROUND ON SELF-TRAINING
text: Given a labeled dataset L = {(xi, yi)}Ni=1 and an unlabeled dataset U = {xj}Mj=1, we summarize the general family of SSL algorithms known as self-training as: 1. First, an initial model denoted f1 is trained using supervised learning on the labeled dataset L. 2. Then, at iteration t, one adopts ft as the teacher model to annotate the unlabeled dataset U using
pseudo labels. Optionally, one uses a selection method to pick a subset St ⊆ {(xj , ft(xj))}Mj=1 of pseudo labeled examples.
3. A student model ft+1 is trained to optimize a classification loss on the combination of L and St:
`t+1 = E(x,y)∼(L∪St)H(y, ft+1(x)) , (1) where H(q, p) = q> log p is the softmax cross entropy loss, and y is assumed to be a one-hot vector (original labels) or a vector of class probabilities (soft pseudo labels).
4. Self-training iterations are repeated T times or until performance plateaus.
Many different variants of the basic self-training algorithm discussed above exist in the literature. These variants differ in the type of pseudo labels used, the selection strategy to filter pseudo labeled examples, the speed at which ft is replaced with ft+1, the choice of data augmentation strategy in the teacher and student models, and the weighting of the two datasets in the objective (Berthelot et al., 2019b;a; Xie et al., 2020; Sohn et al., 2020; Du et al., 2020).
An important design choice is the type of pseudo labels used. One can simply use soft class probabilities predicted by a teacher ft (Du et al., 2020), sharpened class probabilities (Berthelot et al., 2019b), or hard labels (a one-hot vector that is zero except at argmaxft(x)) (Lee et al., 2013). Another important consideration is the selection strategy to retain a subset of pseudo-labeled examples. FixMatch (Sohn et al., 2020) uses a hyper-parameter τ to select examples on which the teacher model has a certain level of confidence, i.e.,
St = {(x, ft(x)) | x ∈ U & max(ft(x)) ≥ τ} . (2) NoisyStudent (Xie et al., 2020) also uses a form of confidence filtering but ensures that the class labels in the selected subset are balanced. In principle, any method for out-of-distribution detection (Hendrycks & Gimpel, 2016) can be adopted for filtering pseudo-labeled examples. We adopt the simplest variant of self-training and limit hyper-parameter tuning to a bare minimum.

heading: 4 GENERATE, ANNOTATE, AND LEARN (GAL)
text: Given a labeled dataset L = {(xi, yi)}Ni=1, we first train an unconditional domain-specific generative model g(x) on Lx = {xi}Ni=1, and then use it to synthesize unlabeled data. Such synthetic unlabeled data is used within self-training and KD even in the absence of in-domain unlabeled data. We restrict our attention to basic KD and self-training methods, even though GAL can be combined with more sophisticated semi-supervised techniques too.
The objective function of GAL for self-training during iteration t, provided a teacher model ft, is expressed as:
`t+1 = λE(x,y)∼LH(y, ft+1(x)) + (1− λ)Ex̃∼g(x)H(ft(x̃), ft+1(x̃)) . (3) We use soft pseudo labels within self-training and KD. To improve computational efficiency of GAL, we do not generate unlabeled data on the fly, but generate as many unconditional samples as possible and store them in a synthetic unlabeled dataset U . We simply set λ = 0.5, unless stated otherwise.
Not surprisingly, the effectiveness of GAL depends on the fidelity and diversity of synthetic examples. If we had access to the oracle generative process, we were able to obtain the best KD and SSL results, as if we had access to real task-specific unlabeled data. Our preliminary experiments suggest that large language models are particularly effective within the GAL framework. Hence, as shown in Figure 1, to build the best domain-specific language model, we adopt a large large language model pretrained on lots of open-domain text, and fine-tune it on a given dataset’s inputs, i.e., Lx, ignoring class labels. Both our theory and ablations confirm that ignoring class labels is a good idea. Transferring the knowledge of large language models is particularly beneficial a small input dataset Lx of text is available (Hernandez et al., 2021). In what follows, we first discuss practical considerations around building domain-specific language models, and then link GAL to empirical and vicinal risk minimization, to motivate the approach.

heading: 4.1 DOMAIN-SPECIFIC GENERATIVE MODELS OF TEXT AND TABLES
text: Text. Many NLP tasks have a relatively small labeled dataset (Wang et al., 2019b;a). While selfsupervised pretraining, followed by supervised fine-tuning (Devlin et al., 2019; Liu et al., 2019b; Clark et al., 2020; Lewis et al., 2019) has become the prominent approach to NLP, previous work has also investigated different data augmentation methods to increase the size of the training datasets. In summary, existing approaches to data augmentation for NLP include lexicon replacement, sentence retrieval, and round-trip machine translation (Wang & Yang, 2015; Yu et al., 2018; Kobayashi, 2018; Wu et al., 2019; Lichtarge et al., 2019; Wei & Zou, 2019; Alberti et al., 2019; Du et al., 2020; Shen et al., 2020). By contrast, we propose the use of unconditional autoregressive LMs for data augmentation. This is simple, flexible, and powerful.
We take a pretrained GPT-2 language model (Radford et al., 2019) and fine-tune it separately on each dataset of interest after removing class labels. We find that training from scratch on these datasets
is hopeless, but the larger the pretrained GPT-2 variant, the better the validation perplexity scores are. For tasks modeling a relationship between multiple sentences, we concatenate a separator token “[SEP]” between consecutive sentences. Once a fine-tuned GPT-2 model is obtained, we generate task-specific synthetic data up to 40× larger than the original training sets. For some samples of generated text for GLUE see Table C.1 to C.6. We believe using bigger LMs and larger synthetic datasets will improve our results, but we are constrained by compute resources.
Tabular data. Features from tabular tasks are often in a well-structured format, i.e., each data point comprises a fixed number of attributes as in Table D.2. This property impedes the acquisition of task-specific unlabeled data, i.e., most augmentation techniques such as round-trip translation and retrieval are hardly applicable. To enable generative modeling on tabular data, we convert each data point (i.e., row from the table) into a sentence by concatenating all of its attributes. This reformatting enables the use of GPT-2 fine-tuning similar to text, and surprisingly GPT-2 samples are very effective.

heading: 4.2 AN EMPIRICAL RISK MINIMIZATION PERSPECTIVE
text: In supervised learning, one seeks to learn a mapping f that given an input x, predicts a reasonable output y. To define the supervised learning problem formally, one assumes that input-output pairs are drawn from a joint distribution P , i.e., (x, y) ∼ P (x, y), and a loss function H(y, f(x)) is used to assess the quality of a mapping f . This loss is used to define a notion of expected risk:
R(f) = EP (x,y)H(y, f(x)) . (4)
In almost all practical applications P (x, y) is unknown. Hence, a labeled dataset of examples L = {(xi, yi)}Ni=1 is used to approximate R(f) as
R̂(f) = 1
N ∑N i=1 H(yi, f(xi)) . (5)
This objective function is known as empirical risk, and learning f through minimizing R̂(f) is known as the empirical risk minimization principle (Vapnik, 1992). To compensate for the finite sample size in (5), one typically combines R̂(f) with a regularizer to improve generalization.
Beyond empirical risk minimization. Empirical risk minimization (5) is motivated as a way to approximateP (x, y) through a set of Dirac delta functions on labeled examples: Pδ(x, y) = ∑ i δ(x= xi, y=yi)/N . However, this approximation is far from perfect, hence one uses a heldout validation set for early stopping and hyper parameter tuning.
Vicinal risk minimization (Chapelle et al., 2001) approximates expected risk as EPν(x,y)H(y, f(x)), using a vicinity distribution, e.g., ν(x̃, ỹ | x, y) = N (x̃ − x, σ2)δ(ỹ = y) to approximate P (x, y) as
Pν(x, y) = 1
N ∑N i=1 ν(x̃ = x, ỹ = y | xi, yi) . (6)
The goal is to increase the support of each labeled data point and improve the quality and robustness of the risk function.
Recent work on mixup regularization (Zhang et al., 2018) proposes an effective way to construct another vicinity distribution by interpolating between two data points and their labels. Albeit its simplicity, these smoothing techniques tend to improve matters.
Generative models for risk minimization. One can factorize the joint distribution of input-output pairs as P (x, y) = P (x)P (y | x). Accordingly, if one is able to learn a reasonable unconditional generative model of x denoted g(x), then one can draw a pair (x, y) by first drawing x ∼ g(x) and then using the current instance of ft to draw y ∼ ft(x). Then, one can use ft and g to approximate expected risk as Rt(ft+1) = Ex∼g(x)Ey∼ft(x)H(y, ft+1(x)) . (7) The quality of this approximation highly depends on the quality of ft and g. If ft is far from an optimal classifier f∗ or g(x) is far from P (x), (7) yields a poor approximation.
The expected risk in (7) smoothens the risk landscape in complex ways beyond simple Gaussian smoothing and interpolation. This smoothing is applicable to any continuous, discrete, or structured domain as long as expressive generative models of P (x) are available. That said, for almost all
reasonable loss functions H (e.g., softmax cross entropy and squared error), (7) is minimized when ft+1 = ft, which is not ideal, especially when ft is far from f∗. On the other hand, empirical risk (5) anchors the problem in real labeled examples that are provided as ground truth.
GAL aims to combine the benefits of (5) and (7) via:
Rt(ft+1) = λ
N ∑N i=1 H(yi, ft+1(xi)) + (1− λ)Ex∼g(x)Ey∼ft(x)H(y, ft+1(x)) (8)
In this formulation, if ft represents the minimizer of empirical risk (5), then ft+1 = ft is the minimizer of (8) too. However, one does not seek the global minimizer of empirical risk, but rather the best performance on heldout data. If ft is obtained by stochastic gradient descent on any risk function, but early stopped according to empirical risk on a heldout set, then using such ft in (8) to define Rt(ft+1) promotes the selection of a mapping ft+1 that minimizes empirical risk while staying close to the best performing mapping so far (i.e., ft). This formulation motivates self-training and GAL as regularizers in the functional space and explains why they can conceivably work.
How about class-conditional generative models? One can also factorize the joint distribution P (x, y) as P (y)P (x | y) and accordingly utilize a class-conditional generative model g(x | y) to derive the following expected risk formulation:
R(f) = Ey∼P (y)Ex∼g(x|y)H(y, ft+1(x)) . (9) In this setting pseudo labeling is not needed as synthetic data is already labeled. One can show that the optimal classifier f∗g that minimizes (9) for cross entropy loss is given by,
f∗g (y | x) = g(x|y)P (y) /∑
y′ g(x|y′)P (y′) , (10)
that is turning the class-conditional generative model into a classifier by using the Bayes rule yields the optimal solution.
Provided that the accuracy of generative classifiers on natural image and text classification is far behind their discriminate counterparts (e.g., Ravuri & Vinyals, 2019), we think substituting (9) into (8) is not a good idea. Essentially, by substituting (9) into the classification objective, one is regularizing f to remain close to f∗g , which is not an effective strategy if f ∗ g is not competitive. This argument corroborates the evidence from our ablation studies and recent work showing that using class-conditional generative models to augment supervised learning does not provide big gains (Ravuri & Vinyals, 2019). That said, one can still use class-conditional generative models to synthesize high-fidelity samples. As long as these samples are treated as unlabeled examples and annotated using a classifier, e.g., ft, we believe this is a reasonable approach falling under GAL. Our argument above only applies to the scenario that class-conditional generative models are used to synthesize labeled examples.

heading: 5 EXPERIMENTS
text: We asses the effectiveness of GAL on KD, self-training and few-shot learning for NLP. We also present self-training results on tabular tasks. Appendix B shows the applicability of GAL to two image classification tasks as a proof of concept, but more advanced techniques such as Mixup (Zhang et al., 2018) are needed to bridge the gap with the state-of-the-art.

heading: 5.1 STATE-OF-THE-ART RESULTS ON KNOWLEDGE DISTILLATION ON GLUE
text: We use the GLUE benchmark (Wang et al., 2019b) for our KD experiments; see Appendix D for benchmark details and Appendix E for the details of synthetic text generation. Our synthetic unlabeled dataset U includes 40× as many examples as the original dataset for each task in the GLUE benchmark.
The goal of knowledge distillation (KD) (Buciluǎ et al., 2006; Hinton et al., 2015) is to distill the knowledge of a powerful teacher model into a compact student model with as little loss in performance as possible. This can help with model compression (Jiao et al., 2019; Sun et al., 2019a) and multi-task learning (Liu et al., 2019a; Clark et al., 2019). It is known that KD on fresh data, unseen during training, performs better (Buciluǎ et al., 2006; Chen et al., 2020c) than KD on original training data. Accordingly, we investigate the effectiveness of knowledge distillation using generated unlabeled data through GAL.
We use the HuggingFace implementation (Wolf et al., 2020) for KD experiments and adopt a standard experimental setup consistent with previous work (Sun et al., 2019a; Xu et al., 2020). Following Rashid et al. (2021), fine-tuned RoBERTa-large (24-layer transformer) represents the teacher and a DistilRoBERTa (6-layer transformer) (Sanh et al., 2019) is used as the student. We train the student model on U and L, where U is annotated by an ensemble of 10 models, achieving an average score of 87.9. We then mix U and L with a ratio of 1:4, which is equivalent to λ = 0.2. This ratio works best on the dev set.
Table 1 shows the results of individual 6-layer transformers on the GLUE test set. All of the baselines use an identical student architecture. GAL achieves the best entry on the GLUE leaderboard, marking a new state-of-the-art for KD on NLP. It outperforms strong KD baselines such as DistilRoBERTa (Sanh et al., 2019), BERT-PKD (Sun et al., 2019a), BERT-Theseus (Xu et al., 2020), tinyBERT (Jiao et al., 2019) and tinyRoBERTa (Rashid et al., 2021). It also outperforms our own DistilRoBERTa+KD baseline, which learns from soft labels produced by an identical RoBERTalarge ensemble on the original labeled dataset. While the use of soft labels outperform the vanilla fine-tuned DistilRoBERTa model, it significantly underperforms our KD+GAL baseline. We also compare with round-trip translation (RT), a strong data-augmentation baseline (e.g., Yu et al., 2018; Shleifer, 2019). We mirror the experimental setup of GAL and generate 40× unlabeled data using German as the bridge language (English →German→English). The translations are generated via the best model in WMT19 (Ng et al., 2019). Although DistilRoBERTa+RT is better than vanilla DistilRoBERTa and KD variants, it still significantly underperforms our approach.

heading: 5.2 NLP SELF-TRAINING EXPERIMENTS ON GLUE
text: We fine-tune pretrained RoBERTa models provided by fairseq (Ott et al., 2019) on each GLUE task. Fine-tuned RoBERTa serves as the first teacher model for self-training. Each student model is initialized with the original pretrained RoBERTa and fine-tuned with exactly the same hyperparameters as suggested by fairseq (Ott et al., 2019). We combine the labeled dataset L and the synthetic dataset U with a ratio of 1:1, by oversampling labeled data. This corresponds to λ = 0.5 in Eq. (8).
Table 2 shows that GAL provides an average improvement of +1.3% over RoBERTa-base. We see consistent improvements with more GAL iterations, but performance saturates after three iterations.
We further compare our approach with a self-distillation (Furlanello et al., 2018) baseline, in which the teacher and student models use the same architecture and transfer knowledge via the original labeled training set. Although self-distillation provides a slight improvement, the gains from GAL are more significant.
We delve deeper and combine GAL self-training with RoBERTa-large and report test results for both single model and ensemble model in Table 3. We observe consistent gains coming from GAL on RoBERTa-large. Our results underperform the latest and biggest LMs from the GLUE leaderboard, but we are optimistic that GAL can be effectively combined with enormous LMs to provide additional gains.

heading: 5.3 SELF-TRAINING ON TABULAR TASKS
text: We assess the effectiveness of GAL self-training on four tabular tasks, namely connect-4 (Burton & Kelly, 2006), Drug Review (Gräßer et al., 2018), Drybean (Koklu & Ozkan, 2020) and Spambase (Dua & Graff, 2017). The details of these tasks can be found in Appendix D. We follow the same protocol as GLUE tasks and generate 40× unlabeled data from a fine-tuned GPT-2-large. Table 4 shows that GAL achieves sizable gains on these tasks even though neither RoBERTa nor GPT-2 are optimized for tabular tasks. XGBoost (Chen & Guestrin, 2016), a strong supervised baseline for tabular tasks underperforms RoBERTa+GAL on connect-4 and Drug Review. It is worth noting that since inputs of Drug Review contain free form text, we convert them into numeric values through the max-pooled representation of the last hidden states of RoBERTa base. XGBoost outperforms RoBERTa on Drybean and Spambase, but it is important to note that these two datasets include floating point attributes, which we simply tokenize using BPE (Sennrich et al., 2016) into subwords. Nevertheless, GAL is capable of bridging the gap between transformers and XGBoost. We believe GAL can be successfully combined with XGBoost, but we leave this to future work, since the XGBoost library does not easily accommodate soft labels.

heading: 5.4 PROMPT-BASED FEW-SHOT EXPERIMENTS
text: GPT3 (Brown et al., 2020) has introduced an optimization-free paradigm for few-shot learning for NLP. Without updating the parameters, large LMs can correctly predict the labels of the inputs by conditioning on a prompt, which consists of an instruction, a few labeled instances and a new unlabeled input. We apply GAL to prompt-based few-shot learning. Specifically, we present k labeled examples as a prompt to GPT-J (Wang & Komatsuzaki, 2021), an open-sourced re-implementation of GPT-3-6B, and generate m synthetic examples, followed by the corresponding labels. Note that to mitigate noisy outputs, the generation of each synthetic example only conditions on the original k labeled examples. Finally, we concatenate the original k examples and m synthetic examples, and conduct a (k +m)-shot learning experiment with GPT-J.
Brown et al. (2020) studied a total of 51 few-shot learning tasks. Studying all of these tasks is prohibitively expensive. Thus, we filter tasks by following these two steps. First, since generating m synthetic examples for each test instance is computationally expensive, we exclude tasks that have more than 5k test examples. Second, we filter tasks on which GPT-3-6B achieves a score lower than 65% (please refer to Table H.1 in Brown et al. (2020) for more details). After applying the filtering steps, we use four datasets: SST-2 (Wang et al., 2019b), PIQA (Bisk et al., 2020), COPA and BoolQ (Wang et al., 2019a) as the testbed. We notice that in order to generate valid synthetic data, GPT-J requires to see at least 4 labeled examples. In addition, at most 16 examples of BoolQ can be fed into GPT-J without truncation. Thus, we set k andm to 4 and 12 respectively. As seen in Table 5, GAL leads to an average improvement of 1.2% over 4-shot learning, and reduces the gap between 4-shot and 16-shot learning. We noticed that the quality of some generated examples is low. We believe the performance of few-shot learning can be further improved with high-quality instances. One solution is to generate many synthetic examples, and select a high-quality subset. Since each test instance conditions on distinct labeled instances, one has to generate different synthetic instances for each test example from GPT-J, which causes expensive computation. Due to such computational constraints, we leave the investigation of data selection strategies to the future work.
5.5 ABLATING COMPONENTS OF GAL ON GLUE
We conduct an in-depth study of different components of GAL on GLUE datasets. Unless stated otherwise, we use a RoBERTabase model with a combination of the original training data and 40× synthetic data for each experiment. GPT-2 model size. Radford et al. (2019) present a few variants of the GPT-2 model including GPT-2, GPT-2-medium, GPT-2large, and GPT-2-XL. Larger GPT-2 models yield better perplexity scores and higher generation quality. We utilize these models except GPT-2-XL within the GAL framework to study the impact of the generative model’s quality on downstream task’s perfor-
mance. Table 6 shows that regardless of the GPT-2 model sizes, GAL consistently surpasses the vanilla RoBERTa base. Moreover, SST-2 and RTE datasets are not sensitive to the capacity of the GPT-2 model, but higher quality synthetic text improves the results on MRPC and CoLA datasets. We leave investigation of GPT-2-XL and even larger LMs such as GPT-3 (Brown et al., 2020) to future work.
Class-conditional synthetic data generation. Previous work (Kumar et al., 2020b; Ravuri & Vinyals, 2019) suggests that it is challenging to utilize synthetic data from class-conditional gener-
ative models to boost the accuracy of text and image classifiers. Our theory in Section 4.2 points to the potential drawback of class-conditional synthetic data. We empirically study this phenomenon, by fine-tuning GPT-2 in a class-conditional manner. Table 7 shows that not only class-conditional LMs underperform unconditional LMs in our GAL framework, but also they are much worse than the baseline.

heading: 6 CONCLUSION
text: We present Generate, Annotate, and Learn (GAL): a framework for self-training and knowledge distillation with generated unlabeled data. We motivate GAL from an expected risk minimization perspective and demonstrate both theoretically and empirically that the use of unconditional generative models for synthetic data generation is more effective than class-conditional generative models, previously used in the literature. GAL leverages advances in large pretrained language models to help supervised learning and can have implications for learning from limited labeled data. GAL works surprisingly well on NLP and tabular tasks, and helps improve knowledge distillation and prompt-based few-shot learning. We hope that GAL will stimulate new research on the evaluation and development of large language models.

heading: A PSEUDO-CODE OF ALGORITHMS
text: Algorithm 1 SelfTraining(L, U , f0, T ) Input: Labeled dataset L = {(xi, yi)}Ni=1
Unlabeled dataset U = {xj}Mj=1 Initial parameters of a classifier f0
Output: A better classifier fT+1 after T self-training steps 1: train a base model f1 by fine-tuning f0 on L 2: for t = 1 to T do: 3: apply ft to unlabeled instances of U 4: select a subset St ⊆ {(x, ft(x)) | x ∈ U} 5: train a new model ft+1 by either fine-tuning f0 on L ∪ St
or gradient descend on a minibatch from L ∪ St 6: return fT+1
Algorithm 2 GAL(L, g0, f0, k, T ) Input: Labeled dataset L = {(xi, yi)}Ni=1
Initial parameters of a generative model g0 Initial parameters of a classifier f0
Output: A better classifier fT+1 after T GAL steps 1: train a generative model g by fine-tuning g0 on Lx where Lx = {x | (x, y) ∈ L} 2: generate U = {x̃j}kNj=1 by drawing kN random samples i.i.d. from g(x), i.e., x̃j ∼ g(x) for j = 1 to kN .
3: return SelfTraining(L, U , f0, T )

heading: B GAL ON IMAGE CLASSIFICATION TASKS
text: As a proof of concept, in addition to NLP and tabular tasks, we assess the effectiveness of GAL on CIFAR-10 (Krizhevsky & Hinton, 2009) and Fashion MNIST (Xiao et al., 2017) as well. We adopt the NCSN model of Song & Ermon (2019) as the task-specific generative model. We use the CIFAR-10 model provided by the authors and train a model on Fashion MNIST using the same configuration as CIFAR-10. We select the model checkpoint resulting in the best FID score vs. training set (Heusel et al., 2017) based on 1000 samples. We then use the NCSN models to generate up to 10× synthetic unlabeled data, i.e., 500K for CIFAR-10 and 600K for Fashion MNIST. See Appendix C for representative samples.
We adopt FixMatch (Sohn et al., 2020) to conduct semi-supervised learning on vision tasks, since FixMatch has shown promising results on CIFAR-10. Specifically, we train a classifier on mini-batches of intertwined labeled and unlabeled data (synthetic). In each iteration, we obtain pseudo-labels for the unlabeled data, but filter unlabeled examples based on classifier’s confidence, i.e., examples are kept on which the largest class probability exceeds τ . Weak augmentation is used to define pseudo labels, but strong augmentations are used to obtain student model’s predictions. We randomly sample from the strong augmentations list defined in RandAugment (Cubuk et al., 2020). We only apply strong augmentations to the synthetic sam-
ples and not the original labeled data to ensure a fair comparison with the baseline.
We conduct experiments on three different convolutional neural network architectures: VGG19 (Simonyan & Zisserman, 2014), WideResnet28-10 (Zagoruyko & Komodakis, 2016), and ResNet110 (He et al., 2016). For the full list of hyperparameters and other implementation details, please refer to Appendix H. Each classifier is trained for 200 epochs and 3 synthetic datasets of size (1×, 5×, 10×) of the training dataset are used.
Table B.1 shows that GAL achieves an average error reduction of 0.78% over the baseline on CIFAR10 across the 3 architectures tested. Further, it appears that the larger the synthetic dataset size, the better the performance of GAL is. We note that the reported results are the average of 3 independent runs. Similarly on Fashion MNIST, we witness consistent gains across all architectures. Fashion MNIST results are included in Appendix I. Our image classification experiments confirm that even when the generative model of GAL is not pretrained on open domain data and solely trained on the dataset at hand, GAL can offer significant improvements.
Table B.2 presents GAL results on Fashion MNIST dataset. Similar to CIFAR-10, we observe a performance improvement across the three architectures.
C GENERATED UNLABELED EXAMPLES ANNOTATED WITH PSEUDO LABELS

heading: D DATASETS

heading: E GENERATING SYNTHETIC TEXT FOR GLUE
text: To generate domain-specific synthetic data, we fine-tune GPT-2-large on the training set of each downstream task, excluding labels. For tasks with multiple input sentences, we concatenate input sentences into a long sequences and separate sentences by special [SEP] tokens. We generate new domain-specific data by using top-k random sampling similar to Radford et al. (2019). We do not feed any prompt to the LM, but a special [BOS] token to initiate the generation chain. A generation episode is terminated when a special [EOS] token is produced. We generate diverse sentences by varying the random seed. After collecting enough synthetic data, we only retain unique sentences. For tasks with α input sentences, we discard generated samples that violate this constraint (approximately 10% of samples were rejected).
Quality of synthetic dataset. An effective generative model of text should learn the word preferences and genre associated with a given corpus, but still produce novel sentences. In order to study the characteristics of our synthetic datasets, Table E.1 reports the number of unique n-grams in the training and synthetic datasets, as well as the number of unique n-grams shared between them. The high degree of overlap on uni-grams suggests that the fine-tuned GPT-2-large is somewhat domainspecific. Meanwhile, the large number of unique n-grams in the synthetic dataset suggests that many novel word combinations are generated, which is helpful for GAL.
Table E.1: For each dataset we report the number of unique n-grams in (the original dataset, the synthetic dataset, shared between the two).
SST-2 QNLI RTE MRPC CoLA 1-gram (15k, 33k, 11k) (89k, 231k, 55k) (18k, 34k, 13k) (15k, 27k, 10k) (6k, 6k, 4k) 3-gram (107k, 2M, 38k) (2M, 10M, 513k) (120k, 750k, 30k) (105k, 562k, 27k) (39k, 198k, 14k) 5-gram (109k, 4M, 9k) (2M, 25M, 146k) (130k, 1M, 4k) (120k, 1M, 7k) (35k, 389k, 5k)
F IMPORTANCE OF PSEUDO-LABELS
We have argued and demonstrated that using class-conditional generative models to generate labeled synthetic examples is less effective than GAL in section 4 and section 5. To further verify this argument, we sample 100 instances from the synthetic RTE dataset generated by a class-conditional LM. Then we annotate these examples using a human annotator and the RoBERTa classifier. Finally, we compute the Accuracy, F1, Precision and Recall scores between human labels and RoBERTa labels, and between human labels and conditioning labels (i.e., labels that the class-conditional LM conditions on.). Table F.1 shows that class-conditional LM has difficulty generating sentences retaining the semantics or pragmatics of a specified category, which also corroborates our theoretical analysis in section 4. On the other hand, RoBERTa is able to produce higher quality labels that correlate better with human annotations.
Table F.1: Performance of RoBERTa annotation and conditioning labels on 100 random examples from the synthetic RTE dataset generated by a class-conditional LM.
Label type Accuracy F1 Precision Recall
RoBERTa 90.0 91.4 100.0 84.1 conditioning label 72.0 71.4 66.0 77.8

heading: G GPT-2 FOR CLASSIFICATION
text: We have conducted additional experiments, where we fine-tune GPT-2 as a classifier. We have considered two variants of the GPT-2 model. The first varant is the original GPT-2 model (GPT2original) pre-trained on open-domain text. The second variant is the GPT-2 model that was finetuned on the inputs of each task separately (GPT-2-finetuned). This model was used to generate task-specific (synthetic) unlabeled data. Finally, we also consider self-training with GAL on top of GPT2-original. Specifically, we use the GPT-2-finetuned model to synthesize 40x in-domain unlabeled data. Then we apply self-training to GPT-2-original, where the data is a combination of the original labeled data and pseudo-labeled synthetic data. Table G.1 suggests that the gains of GAL come from the pseudo-labeled synthetic data, i.e., both synthetic unlabeled data and teacher’s knowledge. Without the generation of synthetic unlabeled data, the domain-specific knowledge embedded in GPT-2-finetuned model cannot be utilized. As such, GPT-2-finetuned model is inferior to the GPT2-original model.
Table G.1: GLUE test results of different GPT-2 models.
Model MNLI CoLA SST-2 MRPC STS-B QQP QNLI RTE Avg GPT-2-original 85.9/85.6 54.8 94.5 86.9/82.2 86.3/85.2 72.5/89.3 91.2 69.8 80.9 GPT-2-finetuned 85.8/85.5 40.9 94.5 87.0/81.0 85.6/84.3 71.4/88.5 91.5 69.0 78.8 GPT-2-original+GAL 86.2/85.8 55.7 94.7 87.9/83.4 86.9/85.9 72.6/89.4 91.9 70.6 81.5

heading: H TRAINING DETAILS
text: We use the fairseq codebase (Ott et al., 2019) for implementing both NLP and tabular experiments. Training details are summarized in Table H.1 and Table H.2. We use the HuggingFace codebase (Wolf et al., 2020) for KD experiments. All NLP models are trained for 5 epochs with a learning rate of 2e-5 and a batch size of 32. All experiments are run on a single Nvidia V100 GPU.
For the CV tasks, we first use the official implementation of NCSN (Song & Ermon, 2019) to generate the synthetic images for CIFAR-10 and Fashion MNIST. We use the pretrained checkpoints provided by the authors for the generation of synthetic CIFAR-10 images and we train a new generative model for Fashion MNIST from scratch with the same hyperparameters of the CIFAR-10 network. After generating the synthetic images, we apply GAL using a FixMatch-like setup (Sohn et al., 2020), using the hyperparameters listed in Table H.4. We follow Cubuk et al.
Table H.1: Training details for NLP tasks.
MNLI CoLA SST-2 MRPC STS-B QQP QNLI RTE lr 1e-5 1e-5 1e-5 1e-5 2e-5 1e-5 1e-5 2e-5 #sent. 32 16 32 16 16 32 32 16 warmup steps 7432 320 1256 137 214 28318 1986 122 validate steps 12386 535 2093 203 360 11307 3310 203 #epoch 2 2 2 2 2 2 2 2
Table H.2: Training details for tabular tasks.
connect4 Drug Drybean Spambase lr 1e-5 1e-5 1e-5 1e-5 #sent. 32 16 16 16 warmup steps 1013 116 408 138 validate steps 1686 212 681 231 #epoch 2 2 2 4
Table H.3: Training details for KD on GLUE benchmark.
MNLI CoLA SST-2 MRPC STS-B QQP QNLI RTE lr 2e-5 2e-5 2e-5 2e-5 2e-5 2e-5 2e-5 2e-5 #sent. 32 32 32 32 16 32 32 16 validate steps 24542 534 4208 208 358 22740 6546 154 #epoch 1 1 1 1 1 1 1 1
(2020) for strong augmentations. Finally, the backbone of the classifiers is from this codebase: https://github.com/bearpaw/pytorch-classification.
Table H.4: Training details for CV experiments
Parameter Description Value τ Pseudo-labeling confidence threshold 0.95 batch size Number of labeled images per batch 64 µ Ratio between number of unlabeled and labeled images in each batch 7 images per epoch Number of labeled images per epoch 65536 #epoch Number of epochs of training 200 lr learning rate max value (10 epochs warmup then cosine decay) 0.03 weight decay Weight decay regualrization coefficient 5.00× 10−4 momentum Nesterov momentum for SGD optimizer 0.90

heading: I ADDITIONAL DETAILS
text: In Tables I.1, and I.2, we present some descriptive statistics of our CIFAR-10 synthetic image dataset to complement the samples shown in Figure C.1 and to help shed some light on the nature of the images generated by the NCSN network.

Review summaries:
The authors propose a new framework of data augmentation based on generative models. The idea is to generate some new samples and then to classify them in an unsupervised manner to improve a student model. The authors provide an extreme experiment where the sentences are composed of numerical data raw extracted from UCI datasets. The authors give a very clear formalization & discussion to bridge between data generation and semi-supervised learning. Then, the authors provide a large experimental section that show the interest of the GAL approach. In particular for small models but also for larger ones. The experimental section is very strong, investingating several interseting ablation. The last section (5.5) seems very promising for the future, in particular for data2text applications.

This paper identifies challenges in self-training as a lack of in-domain data (input x). To overcome this challenge, the paper proposes to "generate" in-domain data using large self-supervised LMs. The rest of the process, which are annotation and learning, follows typical self-training and this paper takes the approach of learning with soft-target as (Du et al.) The paper examines the proposed approach (GAL) on GLUE benchmark on KD (table1), SSL on full data (tab2,3), and prompt-based few-shot (tab4), and also examines GAL in tabular tasks (tab 7). On KD, GAL usually shows the largest improvement on 1st iteration and shows minor fluctuations of performance during more iterations (Tab2, 7).

The paper provides a novel framework for advancing self-supervised learning, Knowledge distillation and few-shot learning for NLP and Tabular data. The framework focuses on self-training and knowledge distillation by generating synthetic data based on an unconditional generative model. They conduct extensive experiments spanning NLP, Tabular data and Computer vision datasets, discussing the benefits and limitations of their proposed methods. They also conduct and extensive literature review for each of the components used in their framework.