Generate, Annotate, and Learn: Generative Models Advance Self-Training and Knowledge Distillation
Authors: Xuanli He
Islam Nassar
Jamie Ryan Kiros
Gholamreza Haffari
Mohammad Norouzi
Venue: ICLR 2022 Submitted (2022)
Keywords: deep generative models
semi-supervised learning
knowledge distillation
large language models

Abstract: Semi-Supervised Learning (SSL) has seen success in many application domains, but this success often relies on the availability of task-specific unlabeled data. Knowledge distillation (KD) has enabled compressing deep networks, achieving the best results when distilling knowledge on fresh task-specific unlabeled examples. However, task-specific unlabeled data can be challenging to find, especially for NLP problems. We present a simple framework called "generate, annotate, and learn (GAL)" that uses unconditional language models to synthesize in-domain unlabeled data, helping advance SSL and KD on NLP and tabular tasks. To obtain strong task-specific generative models, we either fine-tune a large language model (LLM) on inputs from specific tasks, or prompt a LLM with a few input examples to generate more unlabeled examples. Then, we use existing classifiers to annotate generated unlabeled examples with pseudo labels, which are used as additional training data or as additional prompts. GAL improves prompt-based few-shot learning on several NLP tasks. It also yields a new state-of-the-art for 6-layer transformers on the GLUE leaderboard. Finally, self-training with GAL offers large gains on four tabular tasks from the UCI repository.

Under review as a conference paper at ICLR 2022
GENERATE, ANNOTATE, AND LEARN: GENERATIVE MODELS ADVANCE SELF-TRAINING AND KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-biee blind review
ABSTRACT
Semi-Supervised Learning (SSL) has seen success in many application domains, but this success often relies on the availability of task-specific unlabeled data. Knowledge distillation (KD) has enabled compressing deep networks, achieving the best results when distilling knowledge on fresh task-specific unlabeled data. However, spec-iicc unlabeled data can be challenging to find, especially for NL.. We present a simple framework called "generate, annotate, and learn (GAL)" that uses unconditional language models to synthesize in-domain unlabeled data, helping advance SSL and KD on NLP and tabular tasks. To obtain strong task- specific generative models, we either fine-tune large language model (LLM) on inputs from specific tasks, or prompt a LLM with a few input examples to generate more unlabeled examples. Then, generated unlabeled examples with pseudo labels, which are used as additional training data as additional prompts. GAL improves prompt-based few-shot learning on several NI.P tasks. It also yields a new state-of-the-an for 6-layer transformers on the GLUE leaderboard. Finally, self-training with GAL offers large gains on four tabular tasks from the UCI repository.
existing classifiers to annotate
INTRODUCTION
Unlabeled data is abundant in the real world, but task-specific unlabeled data within the scope of a given machine learning problem can be challenging find. For instance, one cannot easily find in-domain unlabeled data conforming to the input distribution of specific Natural Language Pro- cessing (NIP) task from the GLUE benchmark (Wang et al. 2019b). Some NI.P tasks require an input comprising a pair of sentences with particular relationship between them. Moreover, classifi- cation datasets typically represent a tailored distribution of text and only include a limited number of class labels. If task-specific unlabeled data were available, one could adopt self-training (Yarowsky) 1995) to automatically annotate unlabeled data with pseudo labels to improve accuracy and robust- ness of classifiers (Xie et al.] 20200] [Carmon al.) 2019b). In addition, one use knowledge distillation (Hinton et al. 2015) on fresh task-specific unlabeled data to more effectively compress deep neural networks and ensembles (Buciluà al.
2006 Chen et al. 2020c).
When task-specific unlabeled examples do not exist, one can try to retrieve them from a large and diverse open-domain dataset. For instance, [uu al. (2020) have used nearest neighbor retrieval to harvest in-domain unlabeled text from the internet, leading to a successful application of self- training and knowledge distillation certain NLP tasks. While retrieval can indeed help to find in-domain data for problems with simple inputs, it is not practical for problems with complex input schemes, e.g., sentence pairs with certain relations and tabular data. Accordingly, self-training and retrieval-based methods have not been widely adopted for NLP tasks, e.g., on the GL.E benchmark.
This paper presents deceptively simple and general framework called "generate, annotate, and learn (GAL)" to advance cupesuupereieed learning and knowledge distillation on various applications that do not come with unlabeled data. We advocate for the use of language models to synthesize unlabeled tasks-specific data, in lieu of real unlabeled data. We build recent advances in text generation (Radford et al., 22099; Gao et al. (02I). ) and use powerful generative models to synthesize unlabeled text and tables. Then, data with pseudo labels. Finally, we combine labeled data with pseudo labeled data to train more effective classifiers or for the purpose of knowledge distillation (KD).
state-of-the otate-of-the-art thetthh-rt classifiers
annotate generated unlabeled

---

Under review as a conference paper at ICLR 2022
We motivate GAL by making connections to empirical and vicinal risk minimization (Vapnik 1992 Chapelle et al. ).01). and demonstrate its utility by presenting empirical results on a wide range of applications. Our key contributions include:
We propose simple way to advance SSL, KD, and few-shot learning on NLP by using language models to synthesize large amounts of task-specific unlabeled data.
We link GAL to empirical and vicinal risk minimization, helping explain why GAL works and why synthetic samples from dlass-conditioall language models are not as effective.
We systematically dissect GAL and study the key components leading to its success.
GAL establishes new SoTA for single 6-layer transformer on the GLUE test set.
GAL improves prompt-based few-shot learning, providing an average improvement of 1.3% on four 4-shot learning NLP tasks.
GAL advance self-training for tabular tasks, outperforming XGBoost on 2 out of 4 tasks,
2 RELATED WORK
There has been a surge of interest in improving accuracy and label efficiency of classifiers via: Self-Supervised pretraining on open-domain unlabeled data in a task-agnostic way (Peters et al.
2018 Devlin et al.) 2019 Chen et al., 2020b),
2. Self-Training using domain-specific unlabeled data in task-specific way (Rosenberg et al., 2005, McClosky et al., 2006; Xie et al. 2020).
While self-supervised learning can be applied to a broad distribution of unlabeled data, self-training requires unlabeled data that at least can be annotated using the same set of class labels available for the downstream task (Oliver et al., 2018). For instance, if one is interested in training a classifier to distinguish images of cats and dogs, self-training with images of aireraft is likely not helpful, but it is conceivable that self-supervised learning with images of aircraft can still help. A growing body of recent work suggests that perhaps self-supervised pretraining and self-training can be combined to achieve the best remi-supervisee learning performance (Chen et al. 2020c Du
compatible and
al. 2020). We corroborate the existing evidence by showing gains from generative self-traiming.
Semi-supervised learning (SSL) has a long and rich history in machine learning (Cooper & Free-
man 1970; McLachlan Ganesalingam 1982 Rilofi 1996 Chapelle et al., 2009 Van Engelen &
Hoos 2020)) One of the oldest family of SSL algorithms is self--raaining a.k.a. self- lelf-learning or self-
labeling (Scudder 1965 Fralick 1967, ( Agrawala 1970, Yarowsky 1995). Self-training encourages knowledge transfer between a teacher and student model in such way that the student can outper- form the teacher. Specifically, one leverages the teacher's knowledge to annotate unlabeled data with so-called pseudo labels, and the student learns from mixture of pscudo- and human-labelee data. Self-training has recently seen renewed interest across vision and NLP applications (Yalniz et al.) 2019 Xie et al. 20200 Zoph et al., 2020 [Du et 2020). Recent work aims to combine self-training and consistency regularization to develop powerful SSL algorithms. The key idea is to ensure that the predictions of a classifier on unlabeled examples are robust strong augmentations (Berthelot et al. 2019a; [Shnn et al. 2020 Xie et al. 2019). We build on prior work and investigate the use of synthetic data within the broad family of self-training methods.
Recent theoretical work analyzes self-training for linear models, often under the assumption that the data distribution (nearly) Gaussian (Carmon et al. (2019a] Raghunathan al., 2020 Chen al., 2020d Kumar et al. 2020a; Oymak& & Gulcu 2020). Wei et al. (2021) prove that, under "epaasiion" and "class separation" assumptions, self- self-training can lead to more accurate neural network classi- fiers. We present a theoretical framing of GAL in terms of empirical and vicinal risk minimization [Vapnik] 1992 Chapelle et al. (200))
An important family of related work uses generative models for SSL by learning features that are useful for both generation and diserimination (e.g., [Cen et al. 2020a| Odena 2016 Dai et al., 2017). For instance, Kingma et al. (2014) approach SSL by viewing missing class labels as a set of latent variables and use variational inference to impute missing labels as well as other factors of variation. By contrast, our work does not learn features using generative models and keeps the generative and discriminative processes separate. This offers more flexibility and allows GAL to use self- self-supe pretraining methods that are not fully generative.
Our work is closely related to recent work on the uses of generative models for data augmenta- tion (Norouzi et al. 2020 Yang al. 2020). Unlike Norouzi et al. (2020), we do not use instance-
2

---

Under review as a conference paper at ICLR 2022
based generative models. Yang et al. (2020) propose a complex scheme, including data relabeling, data filtering, and two-stage training, to utilize synthetic data. By contrast, we show that a simple mixture of the original data and synthetic data can provide sizable gains. Furthermore, show more broad use of generative models on KD and few-shot learning, in addition to tabular tasks.
Knowledge Distillation (KD) (Buciluá al. 2006 Hinton al. 2015) uses procedure similar to self-training to distill knowledge of an expressive teacher model into a smaller student model. In contrast, self-distillation (Furlanello et al. 2018 Zhang et al., 2019at Mobahi al. 2020) uses teacher and student models of equal size, hoping to iteratively refine class labels. Previous work uses unlabeled data (Buciluă et al., (006) and adversarial training (Wang et al., 2018) to improve KD. We demonstrate that synthetic data generated by unconditional generative models can improve KD on NIP,, outperforming strong KD baselines, which often add more complexity and additional hyper-parameters (e.g., Sun et al. 2019a Jiao et al., 2019. Xu et al.
2020 Rashid et al. (021).
Advanced generative models are able to generate realistic images and text (Krrras et al. |017)|Brock [Brock] eet al.) 2019 Karras et al.) [219 Radford et al.] [2019] [Bronn et al. (020). The quality of synthetic samples has improved to the extent that deep fake detection has become an important research topic itself (Zellers et al. 2019; Dolhansky et al. 2019). Recent work has aimed to utilize class-
conditional generative models to help improve supervised learning (Antoniou et al. 2017 Bowles 2019b; Kumar al. 2020b; Gao et al., (2020). However, Ravuri & Vinyals (2019) have shown that images generated by state-of-the-art class-conditional generative models fall short of improving ImageNet classification accuracy, despite strong sample quality scores (Salimans)
ct al. 2018 Zhang al.
it al. 2016 Heusel et al. (0177). Similarly, Kumar et al. (2020b) find that it is difficult for sentences generated by label-conditioned GPT-2 (Radford et al. 2019) retain the semantics or pragmatics of a specified category, which leads to poor performance on downstream tasks. We discuss why consitondiiional generative models are hardly effective for supervised learning, and instead, focus on unconditional generative models.
3 BACKGROUND ON SELF-TRAININ
Given a labeled dataset L = {xx; 1:)) and an unlabeled dataset U = {2 we summarize the general family of SSL algorithms known as self- self-training as:
First, an initial model denoted /1 is trained using supervised learning on the labeled dataset L. 2. Then, at iteration t, one adopts fe as the teacher model to annotate the unlabeled dataset U using pseudo labels. Optionally, one uses selection method to pick subset S. S {(zz, f((x;)))) M of pseudo labeled examples.
3. A student model f(+1 is trained to optimize a classification loss on the combination of L and S₁:
l++1 (( Hy f++1(x)),
(1)
where (g, = q log p is the softmax cross entropy loss, and is assumed to be a one-hot vector (original labels) or vector of class probabilities (soft pseudo labels).
4. Self-training iterations are repeated T' times or until performance plateaus.
Many different variants of the basic self-training algorithm discussed above exist in the literature. These variants differ in the type of pseudo labels used, the selection strategy to filter pseudo labeled examples, the speed at which fe is replaced with fi+1, the choice of data augmentation strategy in the teacher and student models, and the weighting of the two datasets in the objective (Berthelot
et al. 2019b a Xie et al., 2020: Sohn et al. 2020 Du et al.) 2020).
An important design choice the type of pseudo labels used, One can simply use soft class prob- abilitics predicted by teacher fe (Du et al. 2020), sharpened class probabilities (Berthelot et al.) 2019b), or hard labels (a one-hot vector that is zero except at argmaxf((x)) (Lee et al. 2013). An- other important consideration is the selection strategy to retain subset of psculo-labelldd examples. FixMatch (Sohn et al. 2020) uses a hyper-parameter т to select examples on which the teacher model has certain level of confidence, i.e.,
S = (2, f(()) x E U & max(fi()))
NoisySStudent (Xie et al.) 2020) also uses a form of confidence filtering but ensures that the class labels in the selected subset are balanced. In principle, any method for out-of-distribution detec- tion (Hendrycks & Gimpel, (016) can be adopted for filtering pseudo-labeled examples. We adopt the simplest variant of trainiraining and limit syper-parameter tuning to a bare minimum.
3

---

Under review as a conference paper at ICLR 2022
Selt- sut--supeeiied pretraining (Masked language modeling)
Supervised fine-tuning
Selt -r-iiirinnngggg distiliation
BERT and
Glassifier Ply [X]
Better smaller classifier
Open doraiin unlabeled
Synthetic in-domain unlabeled
Labeled
Large LMM (e.g- GPT-2)
In-domain LM P(x)
MLE
Fine tinentuning an in-domain data without labels
An illustration of GAL for NLP. We use open-domain data once for self-supervised pretraining (e.g.. BERT) and once for training a large LM (e.g., GPT-2). BERT is fine-tuned labeled data to yield classifier for the task of interest. GPT-2 fine-tuned on the same data without labels obtain unconditional specifc LM, which is used to generate lots of synthetic in-domain unlabeled data self- trainiainng and KD,
Figure 1:
4 GENERATE, ANNOTATE, AND LEARN (GAL)
Given a labeled dataset L {( wi) we first train unconditional domain-specific gener- ative model g(z) on L, {x, } and then use it to synthesize unlabeled data. Such synthetic unlabeled data is used within self self-training and KD even in the absence of in-domain unlabeled data. We restrict our attention to basic KI) and self-training methods, even though GAL can be combined with more sophisticated semi-supervised techniques too.
The objective function of GAL for self- elf-trriinng during iteration f, provided teacher model ft, is expressed as:
4+11 A (E(e.y)LL Hy, (y,, Je+1 (at) + (1 A) Ez)g(=) ( (c+
(3)
We use soft pseudo labels within self-training and KD. To improve computational efficiency of GAL, we do not generate unlabeled data on the fly, but generate as many unconditional samples as possible and store them in synthetic unlabeled dataset U. We simply set À 0.5, unless stated otherwise.
Not surprisingly, the effectiveness of GAL depends on the fidelity and diversity of synthetic exam- ples. If we had access the oracle generative process, were able obtain the best KD and SSL results, as if we had access to real task-specific unlabeled data. Our preliminary experiments suggest that large language models are particularly effective within the GAL framework. Hence, shown in Figure I to build the best domain-specific language model, we adopt a large large language model pretrained on lots of open-domain text, and fine-tune on a given dataset's inputs, i.e., Lg. ignoring class labels. Both our theory and ablations confirm that ignoring class labels is a good idea. Trans- ferring the knowledge of large language models particularly beneficial a small input dataset I.g of text is available (Hernandez et al. 2021]. I. In what follows, we first discuss practical considerations around building domain -peccffc language models, and then link GAL to empirical and vicinal risk minimization, motivate the approach.
4.1 DOMAIN-SPECIFES GENERATIVE MODELS OF TEXT AND TABLES
Text. Many NLP tasks have relatively small labeled dataset (Wang et al. 2019b a). While self- supervised pretraining, followed by supervised fine-tuning (Devlin et al. 2019 Liu et al., 2019b] Clark et al. 2020 Lewis et al. 2019) has become the prominent approach to NLP. previous work has also investigated different data augmentation methods to increase the size of the training datasets. In summary, existing approaches to data augmentation for NLP include lexicon replacement, sentence
retrieval, and round-trip machine translation (Wang& & Yangl 2015 Yu et al. 2018 Kobayashi 2018
Wu al. 2019 Lichtarge et al. 2019 Wei & Zou 2019 Alberti et al. 2019 Du al. 2020
Shen et al. 2020). By contrast, we propose the use of unconditional autoregressive LMs for data augmentation. This is simple, flexible, and powerful.
We take a pretrained GPT-2 language model (Radford et al. 2019) and fine-tune it separately on each dataset of interest after removing class labels. We find that training from scratch on these datasets
4

---

Under review as a conference paper at ICLR 2022
is hopeless, but the larger the pretrained GPT-2 variant, the better the validation perplexity scores are. For tasks modeling a relationship between multiple sentences, we concatenate a separator token "[SEP]" between consecutive sentences. Once fine-tuned GPT-2 model is obtained, we generate task-specific synthetic data up to 40x larger than the original training sets. For some samples of generated text for GLUE see Table C.1 to C.6] We believe using bigger LMs and larger synthetic datasets will improve our results, but we are constrained by compute resources.
Tabular data. Features from tabular tasks often in well-structured format, i.e., each data point comprises a fixed number of attributes as in Table D.2 This property impedes the acquisition of task-specific unlabeled data, i.e., most augmentation techniques such as round-trip translation and retrieval are hardly applicable. To enable generative modeling on tabular data, we convert each data point (i.e. row from the table) into a sentence by concatenating all of its attributes. This reformatting enables the use of GPT-2 fine-tuning similar to text, and surprisingly GPT-2 samples are very effective.
4.2 AN EMPIRICAL RISK MINIMIZAATION PERSPECTIVE
In supervised learning. one seeks to learn a mapping that given an input æ, predicts a reasonable output y. To define the supervised learning problem formally, one assumes that input-output pairs are drawn from a joint distribution P. i.e., (æt, y) ~ P(x, y), and a loss function H (y, f(x))) ) is used
to assess the quality of mapping f. This loss is used to define a notion of expected risk:
R(f) = En (v. f(x)).
(4)
In almost all practical applications P(æ, y) is unknown. Hence, labeled dataset of examples L {( W is used to approximate R(f) as
R(( 1 N Σ
H(ye, f(:)))
(5)
This objective function is known as empirical risk, and learning f through minimizing R(f) is known the empirical risk minimization principle (Vapnik) 1992). To compensate for the finite sample size in 5 one typically combines R(f) with regularizer to improve generalization.
Beyond empirical risk minimization. Empirical risk minimization ( is motivated way to ap- proximate P(x. 3) through a set of Dirac delta functions on labeled examples: Pa(x, y) = E, 8(2= T. y= /N. However, this approximation is far from perfect, hence uses heldout validation set for early stopping and hyper parameter tuning.
Vicinal risk minimization (Chapelle et al. 2001 approximates expected risk as (m.y) (v, fx))), using a vicinity distribution, e.g., v( (2 5 as
v,y)
= -i xr 02 )ā y) to approximate P(æ, y)
(z y) Σ v(ā x. ÿ
T v/)
The goal is increase the support of each labeled data point and improve the quality and robustness of the risk function.
Recent work mixup regularization (Zhang et al. 2018 proposes an effective way to construct another vicinity distribution by interpolating between two data points and their labels. Albeit its simplicity, these smoothing techniques tend to improve matters.
Generative models for risk minimization. One can factorize the joint distribution of input-output pairs as P(z, y) = P(z)) æ). Accordingly, if one is able to learn reasonable unconditional generative model of æ denoted g(x), then one can draw pair (æ, y) by first drawing æ g(x) and then using the current instance of fi to draw y ~ fe(x)) Then, one can use ft and g to approximate expected risk as
Re(f++1) Exwgge Be (e) H( (++1
The quality of this approximation highly depends the quality of fi and g. If fe is far from an optimal classifier f or g(x) is far from P(x)) (7) yields a poor approximation.
The expected risk in (7) smoothens the risk landscape in complex ways beyond simple Gaussian smoothing and interpolation. This smoothing is applicable to any continuous, discrete, or structured domain long expressive generative models of P(x)) are available. That said, for almost all
5

---

Under review as a conference paper at ICLR 2022
reasonable loss functions H (e.g., softmax cross entropy and squared error), is minimized when fi+1 = ft. which is not ideal, especially when fo is far from f*. On the other hand, empirical risk 5S anchors the problem in real labeled examples that are provided as ground truth.
GAL aims to combine the benefits of and
via:
À Σ ==1 HI,. Se+ + (1- (on Sc+1 (8) Re(fe+1) = N
In this formulation, if ft represents the minimizer of empirical risk (5) then fi41 = fr is the minimizer of (8) too. However, one does not seek the global minimizer of empirical risk, but rather the best performance
heldout data. If ft obtained by stochastic gradient descent on any risk function, but early stopped according to empirical risk on a heldout set, then using such fa in (8) define Re (f(+1) promotes the selection of a mapping fi+1 that minimizes empirical risk while staying close to the best performing mapping so far (i.e., ft). This formulation motivates self-training and GAL as regularizers in the functional space and explains why they can conceivably work.
How about class-conditional generative models? One can also factorize the joint distribution P(x, y) as P(x) y) and accordingly utilize a class-conditional generative model g(x y) to derive the following expected risk formulation:
R(f) P()) (y, f++1(x)).
In this setting pseudo labeling is not needed as synthetic data is already labeled. One can show that the optimal classifier f that minimizes
( for cross entropy loss is given by.
(y x=) g(aly)) E,, g(zly)))())))
(10)
that is turning the class-conditional generative model into a classifier by using the Bayes rule yields the optimal solution.
Provided that the accuracy of generative classifiers on natural image and text classification is far
behind their discriminate counterparts (e.g., Ravuri & Vinyals) 2019), we think substituting ( into 8 is not a good idea. Essentially, by substituting (9) into the classification objective, one regularizing I to remain close to J which is not an effective strategy fg not competi- tive. This argument corroborates the evidence from our ablation studies and recent work showing that using class-conis consiscondiiinaal generative models to augment supervised learning does not provide big gains (Ravuri & Vinyals 2019). That said, one can still use class-conditional generative models to synthesize high-fidelity samples. As long as these samples are treated as unlabeled examples and annotated using classifier, egg., Jt, we believe this is reasonable approach falling under GAL. Our argument above only applies to the scenario that class-conditional generative models are used synthesize labeled examples.
EXPERIMENTS
We asses the effectiveness of GAL on KD, self-training and few-shot learning for NLP. We also present self-training results on tabular tasks. Appendix B shows the applicability of GAL to two image classification tasks as proof of concept, but more advanced techniques such as Mixup (Zhang al. 2018) are needed to bridge the gap with the state-of-the-art.
5.1
STATE-OF-THE-ART RESULTS ON KNOWLEDGE DISTILLATION ON GLUE
We use the GLUE benchmark (Wang et al. 2019b) for our KD experiments; see Appendix D for benchmark details and Appendix E for the details of synthetic text generation. Our synthetic unla- beled dataset U includes 40x as many examples as the original dataset for each task in the GLUE benchmark.
The goal of knowledge distillation (KD) (Buciluă et al. 2006 Hinton et al. 2015) is to distill the knowledge of a powerful teacher model into a compact student model with as little loss in perfor- mance as possible. This can help with model compression (Jiao et al. 2019 Sun et al.) 2019a) and multi-task learning (Liu et al. 2019a Clark et al. 2019). It is known that KD on fresh data, unseen during training, performs better (Buciluă et al. 2006 Chen et al. 2020c) than KD original train- ing data. Accordingly, we investigate the effectiveness of knowledge distillation using generated unlabeled data through GAL..

---

Under review as a conference paper at ICLR 2022
of the art on KD for NLP. Baselines: BERT-Theseus (Xu et al. 2020). BERT-PKD (Snn al. 2019a), tinyBERT (Jiao let al. 2019) tiayRoBERR (Rashid et al. 2021). DostilRoBERTa (Sanh et al. 2019). J. and DistilRoBERTa KD (standard KD) and DistilRoBERTa RT (round-trip translation to generate unlabeled text). Accuracy scores on MN.l-matched/MNN BaatheeddMNLIImminnnee mimimmathhed are reported for MNLI, Matthew's correlation is reported for CoLA, F1/Accuracy scores are reported for QQP and MRPC, Pearson/Spearman correlations are reported for STS-B,
Table GLUE test results for single 6-layer transformer models. GAL establishes
and Accuracy is reported for QNLI and RTE.
Model
MNLI CoLA SST-2
MRPC
STS-B
QQP
QNLI RTE Avg
Previous work:
82,4/82.1
47.8
92.2
87.6/83.2
85.6/84.
71.6/89.3
89.6
66.2
78.6
BERT-PKI PKR
81.5/81.0
92.0
85.0/79.9
70.7/88.9
89.0
65.5
tinyBERT
84.6/83.2
51.1
87.3/82.6
85.0/83.7
71.6/8911
90.4
70.0
79.8
tinyRoBERTa
86.2/85.6
58.6
95.1
91.2/88.1 88.5/88.4
73.0/89.7
92.4
76.6
83.5
Our results:
DistilRoBERTa
83.8/83.4
55.9
93.2
87.4/83.1
87.5/87.5
71.7/89.1
90.6
73.3
81.2
DistilRoBERTa+ KD
84.7/84.5
54.9
94.1
88.0/84.4
87.4/86.6 72.1/89.2
91.6
73.8
81.6
DistilRoBERTa+ RT
86.1/86.1
53.0
94.6
91.0/87.8
89.2/88.8 73.1/89.9
92.4
76.9
82.7
DistilRoBERTa GAL
87.4/86.5
60.0
95.3
91.9/89.2 90.0/89.6 73.3/90.0
92.7
81.8
84.8
Table 2: RoBERTa base and GAL self-training results GLUE dev sets, averaged across 5 independent runs.
Model
MNLI CoLA SST-2
MRPC STS-B QQP QNLI RTE Avg
RoBERTa base
87.7
63.6
94.8
90.1
90.8
91.5
92.6
78.8
86.2
GAL (iter I)
87.9
65.1
95.3
91.7
91.4
91.8
93.1
81.4
87.2
GAL (iter 2)
88.0
65.2
95.3
92.2
91.5
91.7
93.2
82.4
87.4
4 GAL (iter 3)
87.9
65.5
95.3
92.2
91.7
91.7
93.2
82.0
87.4
RoBERTa base + self-distillation
88.1
63.7
95.2
90.3
90.4
91.5
93.1
79.7
86.5
We use the HuggingFace implementation (Wolf al. 2020) for KD experiments and adopt stan-
dard experimental setup consistent with previous work (Snn et al. 2019a Xu et al.) 2020). Following Rashid al. (202T), fine-tuned RoBERTa-large (24-layer transformer) represents the teacher and DiEtilRoBERTa (6-layer transformer) (Sanh et al. 2019) is used as the student. We train the student model on and L,, where U is annotated by an ensemble of models, achieving an average score of 87.9. We then mix U and L with ratio of 1:4, which is equivalent to À 0.2. This ratio works best on the dev set.
Table D shows the results of individual 6-layer transformers on the GLUE test set. All of the base- lines use an identical student architecture. GAL achieves the best entry on the GLUE leaderboard, marking
state-of of- the art for KD NLP. It outperforms strong KD baselines such Dis- tilRoBERTa (Sanh et al., 2019), BERT-PKD (Sun et al. 201 9a), BERT-Theseus (Xu et al.) 2020), tinyBERT (Jiao et al. 2019) and RonYRo (Rashid et al. 2021 It also outperforms
Distil baseline, which learns from soft labels produced by an identical RoBERTa- large ensemble on the original labeled dataset. While the use of soft labels outperform the vanilla fine-tuned BoERa model, it significantly underperforms our KD+GAL baseline. We also compare with round-trij trip translation (RT), strong lata--uumentatiinn baseline (e.g., Yu al. 2018 Shleifer 2019). We mirror the experimental setup of GAL and generate 40x unlabeled data using German as the bridge language (English -German--English) The translations are generated via the best model in WMT19 (Ng et al. 2019). Although DistilRo is better than vanilla DistilRoBERTa and KD variants, it still significantly underperforms our approach.
5.2 NLP SELF- TRAINING EXPERIMENTS ON GLUE
We fine-tune pretrained RoBERTa models provided by fairseq (Ott et al. 2019) on each GLUE task. Fine-tuned RoBERTa serves the first teacher model for self-training. Each student model initialized with the original pretrained RoBERTa and fine-tuned with exactly the same hyper- parameters suggested by fairseq (Ott et al. 2019). We combine the labeled dataset L and the synthetic dataset U with a ratio of 1:1, by oversampling labeled data. This corresponds to = 0.5 in Eq. (8).
Table 2 shows that GAL provides an average improvement of +1.3% over RoBERTa-base. We see consistent improvements with more GAL iterations, but performance saturates after three iterations.

---

Under review as a conference paper at ICLR 2022
We further compare our approach with self-distillation (Furlanello et al. 2018) baseline, in which the teacher and student models use the same architecture and transfer knowledge via the original labeled training set. Although self-distillation provides slight improvement, the gains from GAL are more significant.
We delve deeper and combine GAL self-training with RoBERTa-large and report test results for both single model and ensemble model in T We observe consistent gains coming from GAL on RoBERTa-large. Our results underperform the latest and biggest LMs from the GLUE leader- board, but we are optimistic that GAL can be effectively combined with enormous LMs to provide additional gains.
Table 3: RoBERTa-large with GAL self-training and SoTA methods evaluated GLUE sets. The benefit GAL on single models is larger ensembles. appears that self-training reduce the variance models.
Baselines including much larger models: RoBERTa-large (Liu et al. 2019b). ELECTRA (Clark et al.
2020
T5 (Raffel et al. 2020), ERNIE (Sun et al.
2019b). and DeBERTa (He et al. 2020
Model
MNLI
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
RTE
Avg
Individual Models (our implementation):
RoBERTa-large
90.1/89.7
63.8
96.1
91.2/88.3
90.9/90.7
72.5/89.6
94.5
85.9
86.5
RoBERTa-large GAL
90.2/89.8
66.2
96.4
92.0/89.2
90.7/90.5
73.6/89.9
95.0
86.3
87.1
Ensemble Models (our implementation:::
RoBERTa-large
91.2/90.5
66.8
96.9
92.8/90.3
91.9/91.6
74.5/90.4
95.5
87.7
87.9
RoBERTa-large GAL
91.0/90.7
67.9
97.1
93.1/90.8
91.6/91.4
74.5/90.4
95.8
88.2
88.2
State-of-the-art.
RoBERTa-large
90.8/90.2
67.8
96.7
92.3/89.8
92.2/91.9
74.3/90.3
95.4
88.2
88.0
ELECTRA
91.3/90.8
71.7
97.1
93.1/90.7
92.9/92.5
75.6/90.8
95.8
89.8
89.2
TS
92.2/91.9
71.6
97.5
92.8/90.4
93.1/92.8
75.1/90.6
96.9
92.8
89.8
ERNIE
91.9/91.4
74.4
97.8
93.9/91.8
93.0/92.6
75.2/90.9
97.3
92.0
90.2
DeBERTa
91.9/91.6
71.5
97.5
94.0/92.0
92.9/92.6
76.2/90.8
99.2
93.2
90.3
5.3 SELF- SELF-TRAININ ON TABULAR TASKS
We assess the effectiveness of GAL self-training four tabular tasks, namely connect-4 (Burton
& Kelly| 22006). Drug Review (Gräßer et al.) 2018). Drybean (Koklu & Ozkan 2020) and Spam- base (Dua & Graff 2017). The details of these tasks can be found in Appendix D We follow the
same protocol as GLUE tasks and generate 40x unlabeled data from fine-tuned GPT-2-large. Ta- shows that GAL achieves sizable gains on these tasks even though neither RoBERTa nor GPT-2 are optimized for tabular tasks. XGBoost (Chen & Guestrin, 2016). strong supervised baseline for tabular tasks underperforms ROBERTa+GAL on connect-4 and Drug Review. worth noting that since inputs of Drug Review contain free form text, we convert them into numeric values through the max-pooled representation the last hidden states of RoBERTa base. XGBoost outperforms RoBERTa on Drybean and Spambase, but it is important to note that these two datasets include floating point attributes, which we simply tokenize using BPE (Sennrich et al. (0166) into subwords.
Nevertheless, GAL is capable of bridging the gap between transformers and XGBoost. We believe GAL can be successfully combined with XGBoost, but we leave this to future work, since the XG- Boost library does not easily accommodate soft labels.
Table RoBERTa-base and GAL results four tabular datasets from the UCI repository. Accuracy reported for these datascts,
Model
connect-4
Drug Review
Drybean
Spambase
XGBoost
86.0
80.1
92.1
96.7
RoBERTa base
85.0
84.6
85.0
87.7
+ GAL (iter 1)
87.0
85.7
85.8
89.0
+ GAL (iter 2)
87.5
85,8
86.0
88.8
+ GAL (iter 3)
87.3
85.6
85.9
89.3
8

---

Under review as a conference paper at ICLR 2022
Table 5: sho- learning results for GPT-J (6B) Wang & Komatsuzaki racy is reported for these datasets.
2021
four NLP datasets. Accu-
Model
SST-2
PIQA
COPA
BoolQ
Avg
4-shot
89.8
76.0
79.0
64.3
77.3
8-shot
91.3
76.2
79.0
66.2
78.2
16-shot
92.7
77.0
81.0
66.8
79.4
4-shot synthetic 12-shot (GAL)
91.5
76.7
80.0
65,9
78.5
5.4
PROMPT-BASED FEW-SHOT EXPERIMENTS
GPT3 (Brown et 2020) has introduced an optimization- free paradigm for few-shot learning for NLP. Without updating the parameters, large LMs can correctly predict the labels of the inputs by conditioning on prompt, which consists of instruction, a few labeled instances and a new unla- beled input. We apply GAL to prompt-based few-shot learning. Specifically, examples as a prompt to GPT-J (Wang Komatsuzakij 202T), an open-sourced re-implementation of GPT-3-6B, and generate synthetic examples, followed by the corresponding labels. Note that to mitigate noisy outputs, the generation of each synthetic example only conditions on the original k labeled examples. Finally, concatenate the original is examples and synthetic examples, and conduct a (I: +m)-sott learning experiment with GPT-J.
present l: labeled
Brown al. 2020) studied total of 51 few-shot learning tasks. Studying all of these tasks is prohibitively expensive. Thus, we filter tasks by following these two steps. First, since generating m synthetic examples for each test instance is computationally expensive, we exclude tasks that have more than 5k test examples. Second, we filter tasks on which GPT-3-6B achieves a score lower than 65% (please refer Table H.1 in Brown et al.) (2020) for more details). After applying the filtering steps, we use four datasets: SST-2 (Wang et al. 2019b). PIQA (Bisk et al., 2020). COPA and BoolQ (Wang al. 2019a) as the testbed. We notice that order generate valid synthetic data, GPT-J requires to see at least 4 labeled examples. In addition, at most 16 examples of BoolQ can be fed into GPT-J without truncation. Thus, we set and m to and 12 respectively. As seen in Table GAL leads to average improvement of 1.2% over 4-shot learning, and reduces the gap between 4-shot and 16-shot learning. We noticed that the quality of some generated examples is low. We believe the performance of few-shot learning One solution is to generate many synthetic examples, and select a high-quality subset. Since each test instance conditions distinct labeled instances, one has generate different synthetic instances for each test example from GPT-J, which causes expensive computation. Due to such computational constraints, we leave the investigation of data selection strategies to the future work.
be further improved with high-quality instances.
5.5 ABLATING COMPONENTS OF GAL ON GLUE
We conduct an in-depth study of different components of GAL on GLUE datasets. Unless stated otherwise, we RoBERTa- base model with combination of the original training data and 40x synthetic data for each experiment.
Table 6: GAL with various GPT-2 model sizes on GLUE sets. NA indicates a vanilla RoBERTa base model.
GPT-2 model size. Radford et al. (2019) present few variants of the GPT-2 model including GPT-2, GPT-2-medium, GPT-2- large, and GPT-2-XL. Larger GPT-2 models yield better perplex- ity scores and higher generation quality. We utilize these models except -P. within the GAL framework to study the impact of the generative model's quality on downstream task's perfor- mance. Table [ shows that regardless of the GPT-2 model sizes, GAL consistently surpasses the
GPT-2
SST-2
RTE MRPC
CoL.A
NA
94.8
78.8
90.1
63.6
small
95.5
81.3
medium
95.3
81.3
91.3
63.7
large
95.3
81.4
91.7
65.1
vanilla RoBERTa base. Moreover, SST-2 and RTE datasets are not sensitive to the capacity of the GPT-2 model, but higher quality synthetic text improves the results MRPC and CoLA datasets.
We leave investigation of GPT-2-XL and even larger LMs such as GPT-3 (Brown et al. 2020 to future work.
Classtcon synthetic data generation. Previous work (Kumar et al. 2020b Ravuri &
Vinyals 2019) suggests that it is challenging utilize synthetic data from condicontiiiona gener-

---

Under review as a conference paper at ICLR 2022
ative models to boost the accuracy of text and image classifiers. Our theory in Section 4.2 points to the potential drawback of class-conding clas-conniiiinal synthetic data. We empirically study this phenomenon, by fine-tuning GPT-2 in class-conditional manner. Table shows that not only classcconditional LMs underperform unconditional LMs in our GAL framework, but also they are much worse than the baseline.
Table 7: Synthetic data from class-conditional LMs underperforms GAL and RoBERTa on GLUE dev sets.
Source of synthetic data
SST-2
RTE
MRPC
CoLA
No synthetic data (baseline)
94.8
78.8
90.1
63.6
Class-conditional LM
92.9
74.4
86.0
58.4
Unconditional LM (GAL.
95.3
81.4
91.7
65.1
CONCLUSION
We present Generate, Annotate, and Learn (GAL): framework for self-training and knowledge distillation with generated unlabeled data. We motivate GAL from expected risk minimization perspective and demonstrate both theoretically and empirically that the use of unconditional genera- tive models for synthetic data generation is more effective than class-conditional generative models, previously used in the literature. GAL leverages advances in large pretrained language models to help supervised learning and can have implications for learning from limited labeled data. GAL works surprisingly well on NLP and tabular tasks, and helps improve knowledge distillation and prompt-based few-shot learning. We hope that GAL will stimulate new research the evaluation and development of large language models.
REFERENCES
A. Agrawala. Learning with probabilistic teacher. IEEE Transactions Information Theory, 16 (4):373-379, 1970. doi: 10.1 1.1109//TT.19700 1054472.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic qa cor- pora generation with roundtrip consistency. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 6168-6173, 2019.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative ad adversarial networks. 171 2017.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv:1911.09788 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Infor- mation Processing Systems, pp. 5049-5059, 2019b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Ham- mers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueck- ert. Gan augmentation: Augmenting training data using generative adversarial networks. 11Xiv: 100.0033 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv:2005. 04.220055.4165. /4/65, 2020.
10

---

Under review as a conference paper at ICLR 2022
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 115555351,, 2006.
Ariel N Burton and Paul HJ Kelly. Performance prediction of paging workloads using lightweight tracing. Future Generation Computer Systems, 22(7):784-793, 2006.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. In H. Wal- lach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 11192-11 1203. Curran Associates, Inc., 2019a.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John Duchi. Unlabeled data improves adversarial robustness. arXiv:190S.13736 2019b.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in neural information processing systems, 2001.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Supervised Learning. MIT, 2009.
Mark Chen, Alec Radford. Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. ICMI., 2020a.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp- 1597-1607, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. NeurIPS, 2020c.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020d.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher Manning, and Quoc Le. Bam! bornaggai multi-task networks for natural language understanding. Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 5931-5937, 2019.
Kevin Clark, Minh-Thang Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2020.
David B Cooper and John H Freeman. On the asymptotic improvement in the out-come of super- vised learning provided by additional nonsupervised learning. IEEE Transactions on Computers, 1970.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan Salakhutdinov. Good semi- supervised learning that requires a bad gan. arXiv preprint Ar0iv:1705.00
2017.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
11

---

Under review as a conference paper at ICLR 2022
Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. The deep- fake detection challenge (dfdc) preview dataset. arXiv:1910.08854 2019.
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoy- anov, and Alexis Conneau, Self-training improves pre-training for natural language understand- ing. arXiv:2010.022 199404,9,, 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. bttp::////aachiiii ics. uci.ccc.e edu/ml
S Fralick. Learning recognize patterns without teacher. IEEE Transactions Information Theory, 1967.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. International Conference on Machine Learning, pp. 1607-1616, 2018.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot lan- guage model evaluation, September 2021. URL httph:t/////m 537162 623
52020//ooo2///.ooo zenodo.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arX/::201..15723, 2020.
Felix Gräßer, Surya Kallumadi, Hagen Malberg, and Sebastian Zaunseder. Aspect-based sentiment analysis of drug reviews applying cross-domain and cross-data learning. Proceedings of the 2018 International Conference on Digital Health, pp. 121-125, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Secoding-enhancee bert with disentangled attention. arXiv:2006.0366: 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and nut-of-distribution examples in neural networks. ICLR, 2016.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, pp.
2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv:1503.02531, 2015.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv:1909. 10351, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv: 1710. 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adver- sarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems, pp. 3581-3589, 2014.
12

---

Under review as a conference paper at ICLR 2022
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic rela- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 2 (Short Papers), pp-
452-457, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: N0.18653/11/N18-2000 wtt.https:://
ac web..web.... www.ww..b..ooa.................. wwtp://w.rgg.gatoooogoooo//////e ogy/N18-2722
Murat Koklu and lIker Ali Ozkan. Multiclass classification of dry beans using computer vision and machine learning techniques. Computers and Electromics in Agriculurre, 174:105507 2020. ISSN 0168-1699, doi: http:///oaaoog/////oooooooooooo2
URL httpps:/www
sciencedirects com/
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self- elf-training for gradual domain adaptation. Hal Daumé IIII and Aarti Singh (eds.), Proceedings of the 37th International Con- ference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5468-5479. PMLR, 13-18 Jul 2020a.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained trans- former models. arXiv:2003.0222 2020b.
Dong-Hyun Lee al. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICMI., 2013.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-seqqeecce pre-training for natural language generation, translation, and comprehension. arXiv: arXiv: 13461,, 2019.
Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong, Cor- pora generation for grammatical error correction. arXh::1904.05780, 2019.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neu- ral networks via knowledge distillation for natural language understanding. arXiv:/904.088 2019a
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 1rXiv: 1692, 2019b.
David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. Pro- ceedings of the Human Language Technology Conference of the NAACL, Main Conference, pp- 152-159, 2006.
Geoffrey McLachlan and S Ganesalingam. Updating a discriminant function on the basis of un- classified data. Communications in Statistex-SSmulattin and Computation, 1982.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. Self-distilation amplifies regularization in hilbert space. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's wmt19 news translation task submission. In Proceedings of the Fourth Conference Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 314-319, 2019.
Sajad Norouzi, David Fleet, and Mohammad Norouzi. Exemplar vaes for exemplar based genera- tion and data augmentation. arXiv:2004.04470 2020.
Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint 11X.v:1606.011888 2016.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evalu- ation of deep semi-supervised learning algorithms. NeurlPS, 2018.
13

---

Under review as a conference paper at ICLR 2022
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguisties (Demonstrations), pp. 48-53, 2019.
Samet Oymak and Talha Cihad Gulcu. Statistical and algorithmic insights for semi-supervised learning with self-training. CoRR, abs/2006. 1006, 2020. URL httpss///exxiv.or//aaabb
2006. 11006
Matthew Peters, Mark Neumann, Mohit Iyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Pupers), pp. 2227-2237, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang. John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7909-7919. PMLR, 13-18 Jul 2020.
Ahmad Rashid, Vasileios Lioutas, and Mehdi Rezagholizadeh, Mate-kd: Masked adversarial text, a companion to knowledge distillation. arXiv preprint (txXX::200500991,, 2021.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. Advances in Neural Information Processing Systems, pp. 12268-12279, 2019.
Ellen Riloff. Automatically generating extraction patterns from untagged text. Proceedings of the national conference on artificial intelligence, pp. 1044-1043 1996.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. Applications of Computer Vision and the IEEE Workshop on Motion and Video Computing, IEEE Workshop on, 1:29-36, 2005.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 2234-2242, 2016.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108 2019.
H Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 1965.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers). pp. 1715-1725,, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/w1/P16- 1162. URL htths:t:::/ wwwww//wwww.////////w/////ww//// aclweb.
orr//anto Sonttholoyy/P6611111
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A simple but tough- to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXiv:2009.1388 2020.
Sam Shleifer. Low resource text classification with ulmfit and backtranslation. arXiv preprint arXiv: 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv: 1556, 2014.
14

---

Under review as a conference paper at ICLR 2022
Kihyuk Sohn, David Berthelot, Chun-Liang Li. Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying supervvse learning with consistency and confidence. arXiv:2001.07683, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Proceedings of the 33rd Annual Conference on Neural Information Processing Systems, 2019.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. IJCNLP), pp. 4314-4323, 2019a.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv: 190400923 2019b.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learn- ing, 109()):373-444 2020.
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural informa- tion processing systems, 1992.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. SuperGLUE: A stickier benchmark for general-purposs language understanding systems. arXiv 1905.0057700 2019a.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GI.UE: A multi-task benchmark and analysis platform for natural language understanding. Inter- national Conference on Learning Representations, 2019b.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. /ttps://github.. com k fn o
wwanhhtrraaaooooo r jax May 2021.
William Yang Wang and Diyi Yang. That's so annoying!!!: A lexical and frame-semantic embed- ding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2557-2563, 2015.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with genera- tive adversarial networks. NeurlPS, 2018
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2021.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing LEMNLP-IJCNLPP pp. 6383-6389, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-ther the- natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 45, October 2020. doi: 10.18653/v1/ 2020.emnip-demos.6
Xing Wu, Shangwen L.v, Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional bert contextual augmentation. International Conference on Computational Science, pp. 95, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mminn a novel image dataset for benchmark- ing machine learning algorithms. 1rXiv:170..07740 2017.
15

---

Under review as a conference paper at ICLR 2022
Qizhe Xie. Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint 1xXiv:1904.12848, 2019.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2020.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseus: Compressing bert by progressive module replacing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7859-7869, 2020.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. / 2019.
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. G-daug: Generative data augmen- tation for commonsense reasoning. arXiv:2004.11546, 2020.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. 33rd annual meeting of the association for computational linguistics, pp. 189-196, 1995.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoe V Le. QANet: Combining local convolution with global self-attention for reading comprehension. ICLR, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.1 12. BMVA Press, September 2016. ISBN 1-901725-599 59-6. doi: 10.5244/C. 30.87.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in Neural Information Processing Systems, 12:9054-9065, 9065, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018.
Linfeng Zhang, Jieho Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3713-3722, 2019a.
Xiaofeng Zhang, Zhangyang Wang. Dong Liu, and Qing Ling. Dada: Deep adversarial data aug- mentation for extremely low data regime classification. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2807-2811, 2019b.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. Advances in Neural Information Processing Systems, 33, 2020.
16

---

Under review as a conference paper at ICLR 2022
A
PSEUDO-CODE OF ALGORITHMS
Algorithm SelfTraining U, fo.
Input:
Labeled dataset
vi)
Unlabeled dataset U [x; [=
Initial parameters of classifier
Output: A better classifier fr-1 after self-training steps
1: train base model fi by fine-tuning fo on do:
2: for t
3: apply ft to unlabeled instances of U
4: select subset S (ar, f(()) x U}
5: train new model fe41 by either fine-tuning fo on L S. or gradient descend on minibatch from L U.S. S:
6: return fr+1
Algorithm 2 GAL(L, 90. fo. k, T)
Input:
Labeled dataset L = {axs, 31) }}
Initial parameters of generative model go Initial parameters of classifier
Output: A better classifier fr-1 after GAL steps
1: train generative model by fine-tuning go on Lg where Lg = fæ (ee y) E L}
2: generate U f &N by drawing &N random samples ii.d from g(x), ie., 2y g(x)) for NNN.
3: return SelfTraining(4. U. fo.
B GAL ON IMAGE CLASSIFICATION TASKS
As a proof of concept, in addition to NLP and tabular tasks, we assess the effectiveness of GAL on CIFAR-10 (Krizhevsky Hinton 2009) and Fashion MNIST (Xiao al. 2017) well. We adopt the NCSN model of Song & Ermon 2019) as the task-specific generative model. We use the CIFAR-10 model provided by the authors and train a model on Fashion MNIST using the same configuration as CIFAR-10 10. We select the model checkpoint resulting in the best FID score vs. training set (Heusel al. 2017) based on 1000 samples. We then use the NCSN models generate up to 10x synthetic unlabeled data, i.e,, 500K for CIFAR-10 and 600K for Fashion MNIST. See Appendix C for representative samples.
We adopt FixMatch (Sohn et al. 2020) to conduct semi-supervised learning on vision tasks, since Fix- Match has shown promising results CIFAR-10. Specifically, we train classifier on mini-batcher of intertwined labeled and unlabeled data (synthetic). In each iteration, we obtain pseudo-labels for the unla- beled data, but filter unlabeled examples based on clas- sifier's 's confidence, i.e., examples are kept on which the largest class probability exceeds т. Weak augmenta- tion is used to define pseudo labels, but strong augmen- tations are used to obtain student model's predictions. We randomly sample from the strong augmentations list defined in RandAugment (Cubuk et al. (020). We only apply strong augmentations to the synthetic sam-
Table B.I: Classification error rates on CIFAR- 10 test set with varying amounts synthetic data for three different model architectures. Re- ported results are the average of 3 independent runs.
Model # params
VGG19
ResNetl 10
WRN28- 10
1.74M
20.1
36.48M
Baseline
6.62
5.85
3.87
GAL 1x
5.97
5.13
3.75
GAL 5x
5.80
5.11
3.25
GAL 10x
5.65
5.10
3.23
ples and not the original labeled data to ensure a fair comparison with the baseline.
We conduct experiments three different convolutional neural network architectures: VGG19 Si-
monyan Zisserman 2014). WideResnet28-10 (Zagoruyko & Komodakis 2016). and
ResNet1 (He et al.) 2016). For the full list of hyperparameters and other implementation de- tails, please refer to Appendix H Each classifier trained for 200 epochs and 3 synthetic datasets of size (1 x, 5x, 10x) of the training dataset used.
17

---

Under review as a conference paper at ICLR 2022
Table B.1
shows that GAL achieves average reduction of 0.78% over the baseline CIFAR- 10 across the 3 architectures tested. Further, appears that the larger the synthetic dataset size, the better the performance of GAL is. We note that the reported results are the average of 3 independent runs. Similarly on Fashion MNIST, we witness consistent gains across all architectures. Fashion MNIST results included in Appendix | Our image classification experiments confirm that even when the generative model of GAL is not pretrained on open domain data and solely trained on the dataset hand, GAL can offer significant improvements.
Table presents GAL results on Fashion MNIST dataset. Similar to CIFAR-IO, we observe performance improvement across the three architectures.
Table B.2: Classification rates Fashion MNIST test set with varying amounts of synthetic data for three different model architectures. Results reported are average over independent runs.
Model
VGG19
WRN28-2
ResNet110
# params
1.74M
1.98M
20.11M
Baseline
5.41
4.92
5.21
GAL 1x
5.06
4.63
4.74
GAL 5x
5.14
4.85
4.85
GAL 10x
4.90
4.74
4.75
18

---

Under review as a conference paper at ICLR 2022
C
GENERATED UNLABELED EXAMPLES ANNOTATED WITH PSEUDO LABELS
truck
ship
horse
frog
dog
deer
bird
car
airplane
Figure C.1: CIFAR-10 synthetic samples generated by NCSN (Song & Ermon 2019 and corresponding
pscudo-labels. Images are filtered based confidence threshold of 0.95 and categorized based on lscudo-labels. For each category, random samples are shown.
19

---

Under review as a conference paper at ICLR 2022
boot
bag
sneaker
shirt
sandal
coat
dress
pullover
trouser
t-shirt
Figure C.2: Fashion MNIST synthetic samples generated by NCSN (ong Ermon 2019 and pseudo-labele Images are filtered based on a confidence threshold of 7 = 0.95 and categorized based on pseudo-labels. For each category, 16 random samples are shown.
20

---

Under review as a conference paper at ICLR 2022
Table C.I: QNLI: Two labeled examples, along with nearest neighbors (based on RoBERTa representations) from our synthetic dataset. We include labels for original examples and pseudo-labels for synthetic examples in parenthesis.
When did the third Digimon series begin? [SEP] Unlike the two seasons before it and most the seasons that followed, Digimon Tamers takes a darker and more realistic approach to its story featuring Digimon who do not reincarnate after their deaths and more complex character development in the original Japanese. (not entailment)
KNN:
1: What is the name of the third season? [SEP] In addition to the first two seasons, the third season is the scason that introduced new characters such as Captain Malice, a supervillain who became the antagonist in scason two; and the villains known as the Heartbreakers, who introduced group of crime fighters. (not entailment)
When did the "Walking Dead" series end? [SEP] In 2013, AMC announced that it would develop a "superhero series", which would follow the storylines and characters from the "Walking Dead" series in order to bring the popular AMC original series to a new and younger audience. (not entailment)
3: What is the main objective of the first season of the X-Files? [SEP] The first season notable in that the characters were introduced and developed within the space of single season, as was the format of the show itself. (not entailment)
What did Arsenal consider the yellow and blue colors to be after losing a FA Cup final wearing red and white? [SEP] Arsenal then competed in three consecutive FA Cup finals between 1978 and 1980 wearing their "lucky" yellow and blue strip, which remained the club's 's's away strip until the release of green and navy away kit 1982-83, 883, (entailment)
KNN:
Who was the most important player for Arsenal Football Club in the 1950s? [SEP] Wenger continued to use Arsenal's famous red shirts and red kits throughout the 1950s and 1960s, and the red strip became the club's most recognised and recognizable symbol. (not entailment)
2: When were the first two teams to play for the trophy in the Premier I League? [SEP] The trophy was awarded to Manchester United in 1990-91 and was named after Sir Bobby Charlton, the club's manager until 1990, and later Sir Stanley Matthews, the club's most successful manager. (not entailment)
3: What were the last four players to wear the yellow in the final? [SEP] With Arsenal having won all four major trophies in the period, they became the only club to have won five in row. (not entailment)
Table C.2: QQP: Two labeled examples, along with 3 nearest neighbors (based on RoBERTa representations) from our synthetic dataset. We include labels for original examples and pseudo-labels for synthetic examples parenthesis.
How is the life of math student? Could you descrihe your own experiences? [SEP] Which level of prepration is enough for the exam jipı5? (not duplicated)
KNN:
1: What the best courses for mechanical engincering student? [SEP] What is the best course
to do after completing a B. Tech in mechanical engineering? (not duplicated)
2: How much marks are needed to get through the GATE with electronics? [SEP] What is the
average score of the Gate EE exam? What are the cut-offis? (not duplicated)
3: What is the best time table for students prepare for IAS? [SEP] How can one study for IAS in
a best time? (not duplicated)
How does an IQ test work and what is determined from IQ test? [SEP] How does IQ test works? (duplicated)
KNN:
1: What is the average IQ of the U.S. population? [SEP] How does an IQ test work? (not dupli- cated)
2: Is the Iq test an effective way to measure intelligence? [SEP] How do IQ tests work? (duplicated)
3: How is an IQ test on a scale from I to 100 scored? [SEP] How do you get your IQ tested? (not duplicated)
21

---

Under review as a conference paper at ICLR 2022
Table C.3: RTE: Two labeled examples, along with 3 nearest neighbors (based on RoBERTa representations) from our synthetic dataset. We include labels for original examples and pseudo-labels for synthetic examples parenthesis.
Like the United States, U.N. officials are also dismayed that Aristide killed a conference called by Prime Minister Robert Malval in Port-au-Prince in hopes of bringing all the feuding parties together. [SEP] Aristide had Prime Minister Robert Malval murdered in Port-au-Prince. (not entailment)
KNN:
The government has been criticized for failing prevent the protests that led to the ouster of President Nicolas Sarkozy earlier this month, which led his second clection defeat since assuming office two years ago. [SEP] Prime Minister Jcan-Mare Ayrault is a former president of France. (not entailment)
2: The French president, Jacques Chirac, has been urged by both the Vatican and the U.N. Security Council to step up efforts to prevent the return of former dictator Nicolas Sarkozy. [SEP) Nicolas Sarkozy left France. (not entailment)
3: The French newspaper
Monde says the French President Nicolas Sarkozy was ad- vised by U.S. President George W. Bush about a possible trip to Iraq on Thursday. [SEP) Nicolas Sarkozy is member of the United States. (not entailment)
Only week after it had comment on upping the storage capacity of its Hotmail e-il service, Microsoft early Thursday announced it was boosting the allowance to 250MB to follow similar moves by rivals such as Google, Yahoo, and Lycos. [SEP) Microsoft's Hotmail has raised its storage capacity to 250MB. (entailment)
KNN:
I: The company, known as Microsoft Office, said it plans to sell all of the copics of its popular Office suite at a loss in the wake of the launch of Microsoft Windows 7, saying it will also make $25 million in advertising costs, move likely to hurt its long-standing position among consumers and business leaders. [SEP] Microsoft Office popular office suite. (entailment)
The company's 's shares shot up more than 35% after the company said has sold of its remaining inventory of the new Kindle e-readers at $70 each. The shares rose to $65.20 on Wednesday, their highest since March 6. 2011. "The Kindle is our best selling product," said Jeff Bezos, founder and CEO of Amazon.com. [SEP] Amazon.com com is based Seattle. (not entailment)
response to concerns expressed by some investors, Microsoft last week said it would reduce the amount of shares that will be available to the public by 10 percent in the first quarter, with further reduction to 3 percent in the second quarter. The stock price has plunged from $24 to $17, and Microsoft is currently offering $17 to S19 a share to its most senior employces. Some investors had criticized Microsoft's response concerns about the price its stock and about the perception that the company in trouble, [SEP] Microsoft is struggling to sell stock. (not entailment)
22

---

Under review as a conference paper at ICLR 2022
Table C.4: MRPC: Two labeled examples, along with 3 nearest neighbors (based on RoBERTa representations) from our synthetic dataset. We include labels for original examples and pseudo-labels for synthetic examples in parenthesis.
BMI of 25 above is considered overweight ; or above considered obese [SEP] A BMI between 18.5 and 24.9 is considered normal, over 25 is considered overweight and 30 or greater is defined as obese... (paraphrase)
KNN:
1: The report said that the average woman in her twenties who takes oral contraceptives daily can expect a loss of around 40 cent of her bone density between the ages of 20 and 45 [SEP] The study said the average woman in her twenties who used the pill every day every day for up five years. can expect loss of about 40 per cent of her bone density between the ages 20 and (paraphrase)
2: The report found that 17 percent of U.S. adults between ages 18 and 64 have a body mass index at or above the normal 20. [SEP] For people of that age 17.1 pereent of adults have body mass index at above the normal 20. while 12.6 percent have body mass index of 30 above (not paraphrase)
The survey shows the proportion of women between and who were obese was 6.3 percent, up from 5.7 percent 2001 [SEP] The proportion women between 20 and 44 who were obese increased to 6.3 percent from 5.7 percent in 2001 (paraphrase)
Shares of Genentech, much larger company with several products the market than 2 percent [SEP] Shares of Xoma fell 16 percent in early trade, while shares of Genentech much larger company with several products on the market, were up percent (not paraphrase)
KNN:
Shares in Aventura fell as much as 5 percent. while shares Medi- climbed percent. . [SEP] Shares in Aventura were up 2.5 percent. while shares in Medi-Cal rose 2.5 percent. (paraphrase)
2: Shares of Amgen rose $ 2.29, or 2.2 percent, to 41.10 in after-hours trading [SEP] 1.62. 1.6 percent $ 41.06
Shares of Amgen division of Sanofi-Aventi,, after-hours trading (paraphrase)
3: Shares of General Electric Co GE.N rose more than percent on the New York Stock Exchange while shares of PepsiCo Inc. PEP.N rose 4.7 percent. [SPE] General Electric 's shares jumped almost percent on the New York Stock Exchange.. while PepsiCo climbed 4.7 percent. (paraphrase)
23

---

Under review as a conference paper at ICLR 2022
Table C.5: MNLI: Two laheled examples, along with 3 nearest neighbors (based RoBERTa representations) from our synthetic dataset. We include labels for original examples and pseudo-labels for synthetic examples in parenthesis.
One of our number will earry out your instructions minutely. [SEP) A member of my team will execute your orders with immense precision. (entailment)
KNN:
I: We are your disposal help you with your investigation and provide a full range of pro bono services. [SEP] We the only ones who can help you with your investigation. (neutral)
2: will speak with the chief officer of the contractor, who will be informed about the results of this effort. [SEP] The contractor is being informed about the results of the effort. (entailment)
3: We have office here to assist you. [SEP] An office is where will assist you, said the manager. (neutral)
Conceptually cream skimming has two basic dimensions product and geography, (SEP) Product and geography are what make cream skimming work. (ncutral)
KNN:
I: There are two main types of analysis and they are the case study and the case report.
[SEP] The case study is the most popular method used to analyze subject. (neutral) 2: A third approach capturing and using this type of experience is engage the program management and finance systems the organization. [SEP] There are two strategies to
capturing and using experience, (contradiction)
3: The first is to see the basic elements of a business model in action. [SEP] Basic elements of business models are the most important for the success of any company. (neutral)
don't 't mean be glib about your concers,, but if were you, I might be more concereed about the near-term rate implications of this $1. [SEP] I am concerned more about your issues than the near-term rate implications. (contradiction)
KNN:
I'm not here to tell you of my own experiences, but they are important to others who might have similar concerns. [SEP] If you were to have similar concerns, I'd like to encourage you to tell them to me. (neutral)
2: I don't 't mean sound judgmental, but as person, think that's issue you' 're probably pretty much your own if you think about it. [SEP] You'te 're probably right if you think about (neutral)
3: But I don't I mean to take your word for it. [SEP] know you are correct, but I want to make sure it's clear that I do not agrec. (contradiction)
Table C.6: SST-2: Two labeled examples, along with nearest neighbors (based on RoBERTa representations) from our synthetic dataset. We include labels for original examples and pscudo-labels for synthetic examples in parenthesis.
are more deeply thought through than in most fight-thiining's films (positive)
KNN:
1: is far more sophisticated, insightful and thought-prowoking than his previous films. (positive)
2: is more sophisticated than its more obvious and lsss-than-dazziigg counterparts (positive)
3: is about as well-thought the idea of a bad hair day. (negative)
contains no wit, only labored gags (negative)
KNN:
1: lacks insight, and lacks empathy (negative)
2: has little humor or intelligence (negative)
3: lacks all wit and humanity (negative)
24

---

Under review as a conference paper at ICLR 2022
D DATASETS
Table D.I: Summary of the three sets of tasks used for evaluation of GAL. STS-B is regression task, so #classes not applicable.
Dataset
task
domain
#train
#dev
#test
#classes
NLP GLUE Benchmark:
SST-2
sentiment analysis
movie reviews
67k
872
1.8k
QQP
paraphrase
social QA questions
364k
40k
391k
QNLI
QA/natural language inference
Wikipedia
105k
5k
5.4k
RIE
natural language inference
Wikipedia
2.5k
277
3k
MNLI
natural language inference
mise.
393k
20k
20k
MRPC
paraphrase
news
3.7k
408
1.7k
CoLA
acceptability
8.5k
1043
1k
STS-B
sentence similarity
misc.
5.8k
15k
1,4k
Tabular Data UCI:
connect-4
utility value sentiment analysis
gaming medical grain e-mail
54k
6.8k
6.8k
Drug Review
2.6k
Drybean
categurical classificution spam classification
10.9k
1.4k
1.4k
Spambase
3.7k
0.5k
0.5k
Computer Vision:
CIFAR-10
image classification image classificution
real images
50K 60K
N/A
Fashion MNIST
clothing grey scale
N/A
10K
Table D.2: examples of input and labels for the Dryhean tabular task.
Attributes
Label
1
37316, 718.058800 118.059....
0.6738775377027455 0.9981482213222
SIRA HOROZ
2 50634, 892.39199999999999 .........5880 0,480548833661 11584 0.99427334994473365
3
33631 669.076 0.8466656241 160356, 0.998119630548730
SEKER
E GENERATING SYNTHETIC TEXT FOR GLUE
To generate domain-specific synthetic data, we fine-tune GPT-2-large on the training set of each downstream task, excluding labels. For tasks with multiple input sentences, we concatenate input sentences into a long sequences and separate sentences by special [SEP] tokens. We generate new domain-specific data by using top-k random sampling similar to Radford et al. (2019). We do not feed any prompt to the LM, but a special [BOS] token to initiate the generation chain. A generation episode is terminated when special [EOS] token is produced. We generate diverse sentences by varying the random seed. After collecting enough synthetic data, we only retain unique sentences. For tasks with a input sentences, discard generated samples that violate this constraint (approximately 10% of samples were rejected).
Quality of synthetic dataset. An effective generative model of text should learn the word prefer- ences and genre associated with a given corpus, but still produce novel sentences. In order to study the characteristics of our synthetic datasets, Table E.] reports the number of unique n-grams in the training and synthetic datasets, as well as the number of unique n-grams shared between them. The high degree of overlap on uni-grat gramg suggests that the fine-tuned GPT-2-large is somewhat domain- specific. Mcanwhile, the large number of unique n- r-grams in the synthetic dataset suggests that many novel word combinations are generated, which helpful for GAL
Table E.1: For each dataset we report the number of unique n-grams in (the original dataset, the synthetic dataset, shared hetween the two).
SST-2
QNLI
RTE
MRPC
CoLA
1-gram
(15k, 33k, 11k)
(89k, 231k, 55k)
(18k, 34k, 13k)
(15k, 27k, 10k)
(6k, 6k, 4k) (39k. 198k, 14k) (35k, 389k, 5k)
3-gram
(107k, 2M, 38k)
(2M. 10M, 513k)
(120k, 750k. 30k)
(105k. 562k, 27k)
5-gram
(109k, 4M, 9k)
(2M, 25M, 146k)
(130k, 1M, 4k)
(120k, 1M, 7k)
25

---

Under review as a conference paper at ICLR 2022
F IMPORTANCE OF PSEUDO-LABELS
We have argued and demonstrated that using class-conditional generative models to generate labeled synthetic examples less effective than GAL in section and section To further verify this argument, we sample 100 instances from the synthetic RTE dataset generated by a class-conditions LM. Then we annotate these examples using human annotator and the RoBERTa classifier. Finally, compute the Accuracy, FI, Precision and Recall scores between human labels and RoBERTa labels, and between human labels and conditioning labels (i.e., labels that the class-conditional IMM conditions on.). Table F.T shows that class-conditional LM has difficulty generating sentences retaining the semantics or pragmatics of a specified category, which also corroborates our theoretical analysis in section 4 On the other hand, RoBERTa is able produce higher quality labels that correlate better with human annotations.
Table F1: Performance of RoBERTa annotation and conditioning labels on 100 random examples from the synthetic RTE dataset generated by class-conditinnal LM.
Label type
Accuracy
F1
Precision
Recall
RoBERTa
90.0
84.1 71.4
91.4
100.0
conditioning label
72.0
66.0
77.8
G GPT-2 FOR CLASSIFICATION
We have conducted additional experiments, where we tne-tune GPT-2 as a classifier. We have considered two variants of the GPT-2 model. The first varant is the original GPT-2 model (GPT2- original) pre-trained on open-domain text. The second variant is the GPT-2 model that was fine- tuned on the inputs of each task separately (GPT-2-finetuned). This model was used to generate task-specific (synthetic) unlabeled data. Finally, we also consider self-training with GAL on top of GPT2-original. Specifically, we use the GPT--2finetuned model to synthesize 40x in-domain to GPT-2-original, where the data is a combination of the original labeled data and pseudo-labeled synthetic data. Table G.I suggests that the gains of GAL come from the pseudo-labeled synthetic data, i.e., both synthetic unlabeled data and teacher's knowledge. Without the generation of synthetic unlabeled data, the domain-specífic knowledge embedded in GPT-2-finetuned model cannot be utilized. As such, to the GPT2-original model.
unlabeled data. Then we apply
model is inferior
Table G.1: GLUE test results of different GPT-2 models.
Model
MNLI
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
RTE
Avg
GPT-2-original
85.9/85.6 6
54.8
94.5
86.9/82.2
86.3/85.2
72.5/89.3
91.2
69.8
80.9
GPT-2-finetuned
85.8/85.5
40.9
94.5
87.0/8 1.0
65.6/84.3
71.4/88.5
91.5
69.0
78.8
GPT-2-orrginal-GAAL
86.2/85.8
55.7
94.7
87.9/83.4
86.9/85.9
72.6/89.4
91.9
70.6
81.5
H TRAINING DETAILS
We use the fairseq codebase (Ott et al. 2019) for implementing both NLP and tabular experiments. Training details are summarized in Table H.l and Table H.2 We use the HuggingFace code- base (Wolf et al.) 2020) for KD experiments. learning rate of 2c-5 and batch size of 32. All experiments
NL.P models are trained for epochs with a on single Nvidia V100 GPU.
For the CV tasks, we first use the official implementation of NCSN (Song & Ermon 2019) to generate the synthetic images for CIFAR-IO and Fashion MNIST. We use the pretrained check- points provided by the authors for the generation of synthetic CIFAR-10 images and we train generative model for Fashion MNIST from scratch with the same hyperparameters of the CIFAR-10 10 network. After generating the synthetic images, we apply GAL using a FixMatch like setup (Sohn et al. 2020), using the hyperparameters listed in Table H.4
We follow Cubuk et al.
26

---

Under review as a conference paper at ICLR 2022
Table H.l: Training details for NLP tasks.
MNLI
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
RTE
Ir
le-5
le-5
le-5
le-5
2c-5
le-5
le-5
2e-5
#sent.
32
32
32
16
warmup steps
7432
320
1256
137
214
28318
1986
122
validate steps
12386
535
2093
203
360
11307
3310
203
#epoch
2
2
2
Table H.2: Training details for tabular tasks.
connect4
Drug
Drybean
Spambase
Ir
le-5
le-5
le-5
le-5
#sent.
32
16
16
warmup steps
1013
116
408
138
validate steps
1686
212
681
231
#epoch
2
Table H.3: Training details for KD on GLUE benchmark.
MNLI
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
RTE
Ir
2e-5
2e-5
2e-5
2e-5
2e-5
2e-5
2e-5
2e-5
#sent.
32
32
32
16
32
32
validate steps
24542
534
4208
208
358
22740
6546
154
#epoch
Finally, the backbone of the classifiers is from this codebase: ht https::/ con con/bearpaw/pytoroae classification
2020 for strong augmentations.
Table H.4: Training details for CV experiments
Parameter
Description
Value
Pseudo-labele labeling confidence threshold
0.95
batch size
Number of labeled images per batch
64
Rutio between number of unlabeled and labeled images in each batch
7
images per epoch
Number of labeled images per epoch
65536
#epoch
Number of epochs of training
200
Ir
learning rate max value (10 epochs warmup then cosine decay)
0.03
weight decay
Weight decay regualrization coefficient
5.00 10-4
momentum
Nesterov momentum for SGD optimizer
0.90
ADDITIONAL DETAILS
In Tables and 1.2 we present some descriptive statisties of CIFAR-10 synthetic image dataset
to complement the samples shown in Figure C.I and to help shed some light on the nature of the images generated by the NCSN network.
27

---

Under review as a conference paper at ICLR 2022
Table Ll: Unfiltered CIFAR-10 synthetic data statisties sorted by Count. The Class pseudo-label for each synthetic image first obtained using teacher model trained on the original CIFAR-10 data. Count denotes the number of images per class in the entire synthetic dataset (500K images). Confidence statistics shows the mean and standard deviation of the teacher model confidence score aggregated over each class.
Confidence Mean
Class
Count
Std
truck
64519
0.932
0.141
ship
32800
0.912
0.156
horse
39604
0.916
0.158
frog
76194
0.887
0.168
dog
38784
0.826
0.188
deer
38829
0.865
0.183
cat
65969
0.826
0.185
bird 37255
0.806
0.193
car 36264
0.936
0.140
airplane 69782
0.897
0.161
Table 1.2: Filtered CIFAR-10 synthetic data statistics sorted by Count. The Class pseudo-label obtained
for each synthetic image using teacher model trained on the original CIFAR-10 data. The dataset is then filtered hased the teacher confidence score where only images with confidence 2 0.95 retained.
Count denotes the number of images per class in the filtered synthetic dataset. Confidence statistics shows the mean standard deviation the teacher model confidence score aggregated over each class.
Confidence Mean
Class
Count
Std
truck
48796
0.996
0,009
ship
22741
0.995
0.010
horse
28498
0.996
0.010
frog
45923
0.993
0.012
dog
15984
0.989
0.014
deer
21413
0.993
0.012
cat
26311
0.988
0.014
bird
13440
0.988
0.014
car
28329
0.997
0.008
airplane
43745
0.992
0.012
28


Reviews:
[{"review_id": "qMIp_OuxaD", "reviewer": "Reviewer_Xyfj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors study how to handle the challenge of lacking task-specific unlabeled data in a semi-supervised learning (SSL). The problem is important and well-motivated.\n\nStrengths:\n*   The paper introduces a clear flow and comprehensive studies in the experimental section. The authors show many interesting findings. It is also surprising to see GPT-j could generate quality data with given few-shot labeled data.\n\n\nWeakness:\n*  First, the authors explicitly mention the small amount of labeled data in Figure 1.  Most of the experiments are conducted under full supervision except Table 4.  It is not very true that only a small amount of labeled data is available considering MNLI includes 400k examples and RTE also has more than thousands of labeled examples. This claim matters since data augmentation is much more challenging and important for low-resource scenarios. \n\n* The proposed solution is to leverage data augmentation to generate task-specific unlabeled data. Considering that this idea is widely explored in the existing literature, I am more interested in the most distinguishing part: the authors use the unconditional autoregressive lanaguage model.  However, this claim is only investigated in Table 6. This still leaves many unanswered questions, like whether this observation is only validated for autoregressive models? especially considering that the previous papers[1] show class-conditional LM also brings sizable gains.\n\n* Even though authors review many related works, they did not include other data augmentation works as baselines. Since this area is widely explored, it is important to understand how much benefits can be brought by this method contrasting other data augmentation options. The model with data augmentation and without augmentation is only ablation studies. Considering a GPT-2 is incorporated additional for data generation, it is not surprising to observe improvements. Comparison with other state-of-the-art data augmentation is needed. Besides the large language model as a data generator[1], many works like EDA[2], UDA[3] also explore this idea. \n\nI have several additional questions:\n1.  In table 1, what data is used for knowledge distillation in DistiRoBERTa + KD? Since KD is usually based on unlabeled data, I am not sure how the unlabeled data is achieved? If KD is based on trained labeled data, this seems not a fair comparison. This can only show that using additional LM for data augmentation is useful and such a claim is provided in many existing papers. I will suggest adopting other simple ways to generate unlabeled data like UDA[3] and still KD loss on them\n\n2. I am not sure about the claim on class-conditional generative models. Since class-conditional generative models still generate noisy data, how about using pseudo labeling on class-conditional generated data? The negative effect seems to be brought by noisy labels instead of conditional generated data quality since codntional-class generation is also shown to be effective in a recent paper [4] with a more challenging setting.\n\n\n\n\n[1] Jason W Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks\n\n[2] Kumar, Varun  and Choudhary, Ashutosh  and Cho, Eunah Data Augmentation Using Pre-trained Transformer Models. worshop AACL 2020\n\n[3] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, Quoc V. Le Unsupervised Data Augmentation for Consistency Training. NeurIPS 2020\n\n[4] Zirui Wang, Adams Wei Yu, Orhan Firat, Yuan Cao Zirui Wang, Adams Wei Yu, Orhan Firat, Yuan Cao\n", "summary_of_the_paper": "The paper studies semi-supervised learning via generated synthetic data. More specifically, the authors handle the challenge of lacking task-specific unlabeled data and use unconditional language models to synthesize in-domain unlabeled data, showing the effectiveness of semi-supervised learning, knowledge distillation on NLP, and tabular tasks. ", "summary_of_the_review": "The authors study how to generate data for semi-supervised learning. The main claim is that the unconditional autoregressive language model is able to provide good quality data. However, this claim is still validated based on comparison with other state-of-the-art data augmentation methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gbtcKIbVwVj", "reviewer": "Reviewer_Ke74", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n\n- Clearly written, Appendix also has extra descriptions of task and side experiments on image task and fine-tuning GPT2.\n- Has good survey of related work in self-training.\n- Extensive experiments on KD, self-training, and few-shot + tabular task.\n\n## Weakness\n\nThe main challenge that this paper points out is in generating (or coming up with) the in-domain data.  Given this challenge,\n\n- I am not entirely sure whether this GPT-2 fine-tuning or GPT3 generation is very novel for this paper to be accepted as ICLR paper.\n- As *generation from fine-tuned large LM* is the main contribution of this paper, I suggest the authors to compare with other possible ways of collecting this in-domain data (such as kNN or generation without fine-tuning) to contrast and highlight their scientific contribution.\n\n\nThinking about other possible ways to generate in-domain data,\n- This paper points out that it is difficult to collect in-domain data such as pair-wise sentences and tabular data is hard to get.\n    - Q) For tasks that simply require single sentence as an input, what happens if you just use NYT, wikipedia corpus or if you do kNN given training set? \n    - Q) For the pair-wise input (x1, x2), what happens if you simply do nearest-neighbor with x1, x2 given popular unlabeled texts?\n    - Q) For the tabular dataset, I am not sure if pre-training of GPT2 really adds value, what happens if you just train separate LM for the tabular data.\n\n**Other questions/comments**\n\n- Have you considered using adapters instead of finetuning whole GPT2 which seems too expensive.\n- What would be the result like if you don't really finetune GPT2 and do prompt-based generation? (not for few-shot)\n- How about using g(x) notation in figure 1.\n- Do you require such a large pre-trained model such as GPT2 for something like tabular dataset? I am not sure how GPT2 pretraining helps the generation of feature-like datasets such as tabular data.\n- Maybe inappropriate to say outperforming XGBoost on 2 out of 4 tasks? (I think showing gains are fine over transformers, but besides connect-4 in Table 7, I am not sure the comparison to the XGBoost is that meaningful. The gaps between XGBoost and RoBERTa are remaining similar)\n- Table 2, \"We see consistent improvements with more GAL iterations,\", I am not sure if MNLI and RTE show that?\n- What are the two metrics (metric1/metric2 format) for MNLI, MRPC, STS-B, QQP? I recommend that the authors denote metrics for all their tables.\n- Any reason that table 6 has part of GLUE only?\n- How big is one iteration of GAL? Does it correspond to running through all 40x unlabeled data?\n\n**Grammar**\nIf we had access to the oracle generative process, we were able to obtain the best KD and SSL results —> we could / we would be able to (?)", "summary_of_the_paper": "This paper identifies challenges in self-training as a lack of in-domain data (input x). To overcome this challenge, the paper proposes to \"generate\" in-domain data using large self-supervised LMs. The rest of the process, which are annotation and learning, follows typical self-training and this paper takes the approach of learning with soft-target as (Du et al.)\nThe paper examines the proposed approach (GAL) on GLUE benchmark on KD (table1), SSL on full data (tab2,3), and prompt-based few-shot (tab4), and also examines GAL in tabular tasks (tab 7). On KD, GAL usually shows the largest improvement on 1st iteration and shows minor fluctuations of performance during more iterations (Tab2, 7).", "summary_of_the_review": "I want to first thank the authors for writing a clear paper and conducting extensive experiments. \n\nThe experimental results show performance gains, however, I am not convinced that this approach is novel as it has been shown in previous work that self-training can improve performance and as we know that GPT-2, 3 can generate quality examples.\n\nTo appreciate the result more, I recommend authors to actually focus on discussing challenges in finetuning generator given training data. If they could improve the quality of generation in a novel way, maybe the paper could become more appealing. At the current state, I feel like GAL is very straightforward self-training with the unlabeled data generation with large-LM generation.\n(Additional comments:\nIn case I missed some contributions of this paper, If we remove the good quality of GPT2,3  what are the benefits that GAL is bringing as a framework?  Could you elaborate on why generating in-domain data is a novel idea?)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qc71bvKwnoR", "reviewer": "Reviewer_uF4y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, I want to thank the authors for the effort in writing this paper. It is extremely well-written and I appreciate the section 2 and 3, where key concepts are detailed and formally presented, very easy to follow. I wish more papers dedicated this much space and effort for adding context for the reader. I also appreciate the fact that you share situations where the models fail, showing limitations of the approach.\n\nThe model seems solid and justification for decisions made are clear. Improvements are not large but, given they are being compared to very strong baselines, it is fair to say they are relatively solid.\n\nI do not have many weaknesses to highlight. Adding some of the things that could improve the publication below:\n\n1 - For figure 1, a NIT comment is that adding the unlabeled data to the same level as the models made it a bit confusing at first glance. Maybe moving the \"open domain unlabeled data\" cloud more to the left might make it more clear where the whole process actually starts (maybe something similar for the \"small amount of labeled data\" part.\n2- As you present the GLUE results on table 1, you split the scores with a \"/\". Since this is a paper that spans several areas of AI, it might be good to give more explanations on why this is the case and what are the metrics being considered in all your tables.\n3 - For table 1 and 2, some standard deviation for the multiple runs or significance testing might be useful to compare results that are similar to each other.\n4 - In section 5.3, a way to evaluate the impact of the quality for synthetic examples is to manually generate them by hand. I do think that is an important component for the paper that was not covered as well as the rest. Understanding the limitation of current generative models and understanding \"how far we can go if we improve them\" based on curated high quality data might make your point stronger.\n5 - In table 5, you might want to add a baseline comparison without GAL so it is easier to compare the improvements.\n6 - For tabular tasks, I think that section was more confusing than helpful. There was not a lot of analysis on why you decided to encode features the way you did and the impact it had on the two datasets where XGBoost outperforms your methods. It might be an interesting analysis to add or at least to discuss further.", "summary_of_the_paper": "The paper provides a novel framework for advancing self-supervised learning, Knowledge distillation and few-shot learning for NLP and Tabular data. The framework focuses on self-training and knowledge distillation by generating synthetic data based on an unconditional generative model. They conduct extensive experiments spanning NLP, Tabular data and Computer vision datasets, discussing the benefits and limitations of their proposed methods. They also conduct and extensive literature review for each of the components used in their framework.", "summary_of_the_review": "This work has extensive literature review and provides enough context for the reader to follow their thought process and the decisions made by the authors. Results are convincing and span multiple NLP tasks and research areas (tabular and CV data). There is in-depth discussions about limitations and benefits, and about the decisions made for the framework. My opinion is that this is a clear accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HAchQiR0f6X", "reviewer": "Reviewer_zaeu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "++ Clear & well written\n\n++ relevant combination of different language models\n\n++ strong & relevant experimental part\n\n++ great last experiment on UCI tabular dataset\n\n-- (probable) high computation cost\n\n-- anonymization issue\n\nThe fact that GAL appears in the GLUE learderbord associated with the name XXX is a major issue if XXX is indeed the author of this article. As the GLUE leaderboard is mentioned a lot of time, this anonymization leak is important.\n\n==\n\nFew detailed comments:\n\nSection 3 about distillation is very clear.\n\nOn Fig 1., authors could mention explicitely and/or graphically that GAL correspond to the bottom path (and not to the refining of the discriminative path).\n\nDiscussion of section 4.2 is clear and valuable.\n\nAppendix are numerous and very interesting. They contain all information required to reproduced those experiments.\n\nThe fact that GAL appears in the GLUE learderbord associated with the name XXX [to be discussed privately] is a major issue if XXX is indeed the author of this article. As the GLUE leaderboard is mentioned a lot of time, this anonymization leak is important.\n\nThe authors do not discuss the computation cost of combining GPT2 & RoBERTa-large modeling.\n\n", "summary_of_the_paper": "The authors propose a new framework of data augmentation based on generative models. The idea is to generate some new samples and then to classify them in an unsupervised manner to improve a student model. The authors provide an extreme experiment where the sentences are composed of numerical data raw extracted from UCI datasets. The authors give a very clear formalization & discussion to bridge between data generation and semi-supervised learning.\nThen, the authors provide a large experimental section that show the interest of the GAL approach. In particular for small models but also for larger ones. The experimental section is very strong, investingating several interseting ablation. The last section (5.5) seems very promising for the future, in particular for data2text applications.", "summary_of_the_review": "The proposed framework is simple, well described and very powerful. Experiments are numerous and relevant. I propose to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: To overcome the challenge of lacking task-specific unlabeled data in semi-supervised learning (SSL) or knowledge-distillation (KD) tasks, this paper presents a new framework called "generate, annotate, and learn (GAL)" that uses unconditional language models to synthesize in-domain unlabeled data to advance SSL and KD. Extensive experiments on both NLP and tabular tasks demonstrate positive results of the proposed method.  

Reviewers generally agree on several key strengths of the paper, e.g., the paper is well-written, literature review is comprehensive, experimental results are generally positive (the improvements over the standard baselines on GLUE benchmark looks solid despite not very significant). On the negative side, some reviewers did raise some major concerns about the novelty of the proposed framework and the lack of strong baselines for comparison. For example, the proposed GAL framework doesn’t seem particularly novel as neither of the proposed components is new, and the key value of the work seems on the contribution of evaluating the large LM's ability to generate good in-domain unlabeled data (as agreed by authors). Therefore, it is very important to compare with other existing data augmentation baselines in the empirical studies. While authors did try to add one round-trip-translation (RT) data augmentation baseline for comparison during the rebuttal, more stronger SOTA data augmentation baselines should be compared. 

Overall, this is a good paper which is worthy of publication in near future but it still needs some more work on more extensive comparison of more baselines and improvements on the writing of novelty and contribution claims.
Decision: Reject