Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
Authors: Yu Meng
Jiaxin Huang
Yu Zhang
Jiawei Han
Venue: NeurIPS 2022 Accept (2022)
Keywords: Zero-Shot Learning
Natural Language Understanding
Pretrained Language Models

Abstract: Pretrained language models (PLMs) have demonstrated remarkable performance in various natural language processing tasks: Unidirectional PLMs (e.g., GPT) are well known for their superior text generation capabilities; bidirectional PLMs (e.g., BERT) have been the prominent choice for natural language understanding (NLU) tasks. While both types of models have achieved promising few-shot learning performance, their potential for zero-shot learning has been underexplored. In this paper, we present a simple approach that uses both types of PLMs for fully zero-shot learning of NLU tasks without requiring any task-specific data: A unidirectional PLM generates class-conditioned texts guided by prompts, which are used as the training data for fine-tuning a bidirectional PLM. With quality training data selected based on the generation probability and regularization techniques (label smoothing and temporal ensembling) applied to the fine-tuning stage for better generalization and stability, our approach demonstrates strong performance across seven classification tasks of the GLUE benchmark (e.g., 72.3/73.8 on MNLI-m/mm and 92.8 on SST-2), significantly outperforming zero-shot prompting methods and achieving even comparable results to strong few-shot approaches using 32 training samples per class.

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
Yu Meng, Jiaxin Huang, Yu Zhang, Jiawei Han
Department of Computer Science, University of Illinois at Urbana-Champar {yumeng5, jiaxinh3, yuz9, hanj)eillinois edd
Abstract
Pretrained language models (PLMs) have demonstrated remarkable performance in various natural language processing tasks: Unidirectional PLMs (e.g., GPT) are well known for their superior text generation capabilities; bidirectional PLMs (e.g., BERT) have been the prominent choice for natural language understanding (NLU) tasks. While both types of models have achieved promising few-shot learning performance, their potential for zero-shot learning has been underexplored. In this paper, we present a simple approach that uses both types of PLMs for fully zero-shot learning of NLU tasks without requiring any task-specific data: A unidirectional PLM generates class-conditioned texts guided by prompts, which are used as the training data for fine-tume fine-tuning bidirectional PLM. With quality training data selected based on the generation probability and regularization techniques (label smoothing and temporal ensembling) applied to the fine-tuning stage for better generalization and stability, our approach demonstrates strong performance across seven classification tasks of the GLUE benchmark (e.g.. 72.3/73.8 8 on MNLI-m/mm and 92.8 on SST-2), significantly outperforming zero-shot prompting methods and achieving even comparable results strong few-shot approaches using 32 training samples per class
Introduction
have achieved human-level performance on natural language understanding (NLU) tasks (66 when fine-tuned on a large amount of task-specific training data. However, such a supervised fine-tuning paradigm is drastically different from how humans perform these tasks: We barely need see many task-specific training samples to perform well. Recently, many studies have revealed the intriguing few-shot learning potential of PLMs: By converting task descriptions to natural language prompts and injecting them into PLMs,
Pretrained language models (PLMs) [5
34 40 41
prompt-based approaches [5] 13 55 56, [59 leverage task-specific information for better training
data efficiency and have achieved remarkable few-shot results.
When prompt-based methods are applied to the zero-shot setting, however, the PLMs' predictions are much less accurate. For example, GPT-3's zero-shot performance much degraded relative to its few-shot performance [5), especially on challenging tasks like natural language inference (NLI). Without any task- spskispecific samples, it is indeed challenging for PLMs to effectively interpret the prompts that come different formats and are unseen the pretraining data. To familiarize PLMs with various prompts for zero-shot generalization to unseen tasks, a recent study proposes instruction tuning 770,, which fine-tunes PLMS large collection of different tasks described by instructions.
Despite its strong performance, its success is grounded in the large number of cross-tate -ask annotated datasets (e.g., train many non-NLI tasks and transfer to NLI tasks) and the gigantic model size (e.g., hundreds of billions of parameters), posing great challenges for training and using them.
I Code can be found at at
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work, we study zero-shot learning of PLMs on NLU tasks without any task task-specific or cross- task data. Motivated by the strong text generation power of recent PLMs 23 30 [2], we propose SuperGen, a Supervision Generation approach, wherein training data are created via a unidirectional PLM (i.e., the generator) which generates class-conditioned texts guided by label-descriptive prompts. bidirectional PLM (i.e., the classifier) is then fine-tuned on the generated texts to perform the corresponding task. Both PLMs can be of moderate size to fit in typical research hardware (e.g.. GPT-2-sized [5D generator and RoBERTa arge -ized 34 classifier). With supervision automatically created by the generator, SuperGen eliminates the need for task-specific annotations and provides the classifier PI.M with a larger amount of training data than in few-shot scenarios. We call such setting zero-shot because the entire process does not need any human annotated data, either from the target task or other tasks. The major difference from previous methods is that we synthesize training data for the target task, whereas existing veros-shot methods do not use any form of training data from the test domain (but may train on other domains) and directly perform inference on the target task.
Across seven classification tasks of the GLUE benchmark [66], SuperGen significantly outperforms the prompt-based zero-shot method and even achieves an overall better result in both average performance and stability than strong few-shot approaches that use 32 annotated samples per class. We identify several key factors to the strong performance of SuperGen through ablation studies: (I) selecting quality training data based on their generated probability, and (2) using label smoothing and temporal ensembling to regularize fine-tuning on generated data.
2 Related Work
2.1 Few-Shot and Zero-Shot Learning with PLMs
Instead of using a large amount of annotated training data for fine-tuning PLMs on downstream tasks, few-shot learning studies how to better leverage only a small amount of task-specific training data, a more realistic scenario in many applications. The most strict few-shot learning setting does not assume access to any unlabeled data or large validation sets for hyperparameter tuning [48], where
63 84) are prominently deployed to inject task descriptions into PLMs and make effective use of their language modeling capability for improved training data efficiency in low-data regimes. More broadly, semi-supervised learning additionally
prompt-based methods (5) 13 33 35 55 59
leverages unlabeled task-specific data, where data augmentation 7 73 bootstrapping (56) methods are commonly used.
regularization
43 and
Zero-shot learning, on the other hand, is a much more challenging setting with absolutely no access to any task-specific data. When prompt-based methods are directly used to obtain predictions from PLMs without any training, their zero-shot performance can be much worse [5] [3 -difficult NLU tasks can be barely formulated as prompts that resemble the format of pretraining data, posing great challenges for PLMs to accurately interpret and leverage the prompts without given any training samples. The current mainstream of zero-shot learning is based on transfer learning: By converting
a set of tasks with abundant annotations into instruction templates 42
54 70 74], entailment
pairs [79 80] or question- answer formats [50] 86) and fine-tuning PLMs on them, the PLMs acquire
the cross-task transfer ability [78] to execute unseen tasks when they are formulated in a similar format. Our work proposes a different approach from these studies: We use unidirectional PLM to generate training data for fine-tuning another PLM on the target task. This not only removes the need for a large amount of cross-tast task annotations, but also eliminates the task difference in training and inference. Moreover, different from previous studies [ 76 that rely on labeled data to fine-tune the generative PLM, we directly use prompts to guide data generation without fine-tuning.
2.2 Controlled Text Generation with PLMS
Controlled text generation [22) aims to steer the generated texts of language models towards desired contents, styles or domains. Through fine-tuming PLMs on -ppeuute--pecfiiccccc data, high-level control generating certain topics or sentiments (8)), fine-grained control (e.g., generating specific words
or phrases (6) or both [24] can be achieved. Adapting PLMs to generate texts of specific attributes
can also be realized at inference time without any further training of the PLMs [10 26
27
32
47 75).
Different text attributes also be represented during pretraining time as control codes 23 which
later can serve as explicit guidance for generating domain/atribute-speciff texts.
The idea of generating category-conditioned texts as training data has been explored for topic classification with ong-of-worde LSTM-based language models [38 39), which may not have

---

Label git negative
Seggle- Tasks geg. Sentiment Classification)
Smoothing
Sequence
Prompt my
Selected Quality Training Sample (a=".2) 3/)
This temible.
(Unidirectional PLM)
Generator
waste and money.
Regulariza
Classifler Co (Bidirectional PLM)
Fine-Tuning
Label yy entallment
Sequence-Pai Pair Tasks (e-g. Natural Language Inference)
(a", (e".a")))
Regultrize
Sampled Sequence at"
Prompt I6y
Generated Sequence
Selected Quality Training Sample
Temporal Temporal Ensembling
station was estimated be mid- 2020. In other words,
The opening date the
Generator (Unidirectional PLM)
open
station was
Figure 1: Overview of SuperGen for zero-shot learning of NLU tasks. A unidirectional PLM generates training data guided by label-descriptive prompts. Quality training samples are selected based on average log generation probability. A bidirectional PLM is fine-tuned on the selected training set with label smoothing and temporal ensembling as regularization to perform the classification task.
enough capacity to generate quality training data for challenging NLU tasks. With more powerful PLMs, the idea of using prompts guidance has emerged recently: Since natural language generation is largely based on contexts, using certain prompts to start a sequence can effectively steer the subsequent texts to be generated. The prompts can be either in natural language [57] or as learnable parameters [31] this work, we also guide text generation via prompts, but for the novel purpose of creating training data for NLU tasks. There have been studies with similar goals, such as generating similan/dissimilar sentences for training sentence embeddings (58) and using labeled samples as demonstrations to prompt large PLMS [8]] for creating novel training data. this work, we explore generating training data without using any labeled samples for wide range of different NLU tasks. The similar setting is also explored in concurrent study (77). Compared to annotated task-specific data, the generated texts may contain noise and have domain difference from the downstream task.
We introduce several important strategies for effective fine-tuning generated data.
3 Method
3.1 Preliminaries
Problem Formulation. We consider solving a classification problem[wherr we are only given the label space Y/ and mapping M y + W that converts each label E y into a dabe--descrppiive prompt (i.e., short phrase) Wy E W. We assume access a unidirectional PLM Go as the generator and bidirectional PLM C. which will be fine-tuned the classifier" We also assume the pretraining corpus D (e.g., Wikipedia) is available. Fig. shows an overview of our proposed SuperGen method.
Text Generation with Unidirectional PLMs. A unidirectional PLM Ge is pretrained to maximize the generation probability of each token in a sequence x [x1, T2, .. Fn]] conditioned on previous tokens:
exp(e, h.) IVI exp(e) h.))
max II pa (zv (x)), where pa(a;(
Here, (-)) is usually parameterized using token embeddings e and contextualized embeddings h given by a Transformer [65] encoder.
After pretraining, Go can be directly used generate new texts by recursively sampling tokens from its output probability distribution. Typically, temperature hyperparameter > is introduced during sampling [20] to adjust the sharpness of the probability distribution:
exp(e, h2/T)
po(os( EM exple] ha/T)'
We do not consider regression tasks in this work due to the difficulty of generating texts conditioned on a continuous label space. However, there exist approaches [14] (33 that solve regression tasks by training on classification tasks. We leave the integration of SuperGen with these methods future work for solving regression tasks.
We assume the classifier to be bidirectional PLMs since they generally work better than unidirectional PLMs in NLU tasks; we in principle use any PLM the classifier.
3

---

where r - 0 approximates greedily picking the most probable next token; T - oc induces a uniform distribution. Additionally, sampled tokens can be confined to the top-k most probable ones avoid low-quality tokens. In this work, we find such top-k sampling with temperature is sufficient to produce coherent and meaningful texts as training data for NLU tasks. Exploring more sophisticated sampling strategies [2] is left for future work.
3.2 Training Data Generation
When given a label-descriptive prompt such as "Write negative review:", humans able to produce texts pertaining to the cor- responding class. We aim to leverage the strong text generation power of unidirec- tional PLM Gg for the same purpose of cre- ating class-conditioned training data. We note that Gg directly used for generation without any parameter updates. The prompts used for different NLU tasks in GLUE are summarized in T
Table 1: Prompts used to generate clndiicodd texts for different GLUE tasks. SST-2 is a single- sequence classification task and the rest are sequence- pair classification tasks. Generation for CoLA does not use prompts but by varying sampling tempera- tures. x denotes a sequence randomly sampled from the pretraining corpus; x denotes the sequence to be generated by Gs; denotes skipping at least one sequence. See Appendix A for more details.
Task
Label
Prompt
Generating Single Sequences. For single- sequence NLU tasks such as sentiment clas- sification (e.g., SST-2), we simply use a prompt wy corresponding to label y as the beginning of the sequence and let Go gener- ate the remaining sequence:
positive negative
Rating: 5.0 x? Rating: 1.0 x
SST-2
entailment
xr. In other words, x
neutral
x=" Furthermore,
MNLI
There a rumor that xt". However, the truth is: x
contradiction
x + Go(yy)
entailment not entailment
"=? x
QNLI
x= ?......!"
where (yy denotes using Wy the in- put to Gg and recursively sampling tokens from the distribution in Eq. (I until a full sequence is generated; 29 denotes the gen- erated sequence (i.e., excluding the prompt), which will be paired with to form one train- ing sample (x², y).
entailment not entailment
x²" In other words, æ"
RTE
x²" Furthermore, "
equivalent
x². In other words, x
MRPC
not equivalent
x. Furthermore,
equivalent
? In other words, xn
QQP
not equivalent
" Furthermore, xx
For syntactic tasks like linguistic acceptabil-
ity classification (e.g., CoL.A) which requires generating both linguistically acceptable and unaccept- able sequences, we start the sequence with random stop words and use varying sampling temperatures for generating different sequences. A smaller temperature (e.g., - = 0. in Equation (I) sharpens the sampling probability distribution towards the most probable tokens, thus the resulting sequence is more likely to be linguistically acceptable. Using a larger temperature (e.g., r = 10 in Equation [T) flattens the sampling probability distribution to be more uniform, and the generated tokens will be nearly random, which create linguistically incorrect sequences.
Generating Sequence Pairs. Sequence-pair classification tasks require generating two sequences of specific relationships (e.g., entailment, contradiction). We sample" the first sequence a from the pretraining corpus D, concatenate the prompt wy with x", and generate the second sequence x":
x Go (I="; wyl) x D.
The sequence pair training sample will then be formed as (x*, x x..v).
Rewarding and Penalizing Repetitions for Sequence Pair Generation. A common issue text generation is degenerate repetition 21 23 71] where generated texts get stuck in repetition loops. To address this issue, one approach is to discourage repetition by reducing the logits of tokens that are already in the sequence before performing sampling [23]. In sequence pair generation, however, it is sometimes desirable to encourage the second sequence to repeat some words in the first sentence (e.g., for generating an entailment or a paraphrase). Therefore, we propose a simple modification of
'In principle, we can also generate the first sequence using Ge. but we find sampling from 2 improves the diversity of texts.

---

Eq. ① that rewards/penalizes repetition based on whether the token has appeared in x'/x9;
exp(e; hi/w) Σ exple] hw/w)
Ta
E
x
A
Ha
$ x
where
TB Ti E x
(2)
else
and a 0, > 0 are hyperparameters. By setting 1 and 3 > 1, we can promote tokens in xx that have not appeared in x have higher chance of being generated, and discourage the generation of repetitive tokens in a to mitigate degenerate repetition. The parameters used for different tasks are listed in Appendix B Table
3.3 Effective TineTTnnig on Generated Texts
With the generated training data, one can fine-tune a bidirectional PLM Cs the classifier to perform the NLU task. However, training Cg via standard supervised training on all generated texts is likely to yield suboptimal performance on downstream tasks because (1) the generated texts may contain noise Gg may not always produce texts pertaining the desired class, especially for challenging sequence pair tasks with subtle semantic relationships; and (2) the generated texts can be considered as originated from the domain of Gy pretraining data, with a potentially different distribution from the downstream task; straightforward application of supervised training will result in overfitting to the pretraining domain and diminishing generalization ability, a common challenge in transfer learning [64 (7]. To address these challenges, next introduce several simple and important strategies for more effective and stable fine-tuning on generated texts.
Selecting Quality Training Data. We aim select generated texts 2 that are most likely to pertain to the desired label y (i.e., with the highest p(x"ly) ly)). The true probability p(# ly) is unknown and we estimate it via the generation probability given by Go conditioned the prompt wy:
p( (y) ~ (o( = II (z: (wy: x221))
Since the above measure is biased towards shorter sequences, we instead use the geometric mean of the above conditional generation probability (or equivalently, the average log probability) of all tokens in ø the ranking score, following 82):
log Po (z; (uy x 21))
(3)
To construct a training set consisting of N samples per class, we will generate more samples (e.g., 10N), and select training data based on the score in Eq. 3 For all tasks except CoLA, the top-N ones of each class are selected; for CoLA, the top-N ones are used as linguistically acceptable training samples, and the bottom-N ones as linguistically unacceptable sequences.
Regularization for Better Generalization and Stability. Even with the above training data selection procedure, the resulting training set may still contain noise and there exists domain difference from the downstream tasks. We apply two regularization techniques, label smoothing [62] and temporal ensembling (28) for better fine-tuning stability and generalization.
Given a training sample (x3, y). label smoothing trains the classifier Co to minimize the standard cross-entropy loss between the label and the classifier's 's prediction P& (z)), except that the label is weighted average of the one-hot vector and uniform distribution over all labels:
D
min Σ q, log(p((>))g),
where = 1(j + and is the smoothing weight. By forcing the classifier to be less confident training data. label smoothing improves robustness to label noise (36) and prevents overfitting to the training set (44), thus improving generalization different domains.
The motivation for temporal ensembling is that neural networks usually first pick up easy and general patterns in the data before learning sophisticated and sppasic- features [83), and thus the
5

---

earlier states of the network offer better generalizability to different domains. We therefore record the predictions Ps- = pPs(x9) of Co on each training sample (a:9, y) different training steps, and use the accumulated moving-averags predictions & to regularize the latest model training. This also helps suppress the fluctuation in model predictions due to data noise, offering better noise-robustness 45]
We update ensembled predictions once every B batches:
(5) where & has a zero initialization; is the momentum parameter, is the number of updates & has received; the division (1 1-77 7²) is for bias correction (28). We also use the ensembled prediction # as
72 + (),7) t 2/(1-7")
a reliable signal to filter out noisy training samples: Only those samples on which & strongly agrees
with the label (i.e, &, > 8 where 8 > 0 is a threshold parameter) will be used for training.
We regularize model training by extending Eq. (4) to add KL divergence regularization term from the model prediction to the ensembled prediction weighed by A:
121
Ej log P (x:))) ) 3j
min
gi log(ps
A
(6)
We follow 28 to slowly ramp-up A during training.
3.4
Overall Algorithm
We summarize SuperGen for single- sequence NLU tasks in Algorithm [ Soly- ing sequence-pair problems follows the same algorithm except the pretraining cor- pus D is needed for sampling the first se- quence x..
Algorithm 1: SuperGen for Zero-Shoo Learning.
Input: ): Label space; P: Label-descriptive prompts; Gs:
Unidirectional PLM; Cg:: Bidircctional PLM. Parameter: N: Number of training samples per class to
generate; M(> N): Number of total training samples to generate; '": Number of training steps; B: Ensemble prediction
4 Experimental Setup
update interval; 8: Threshold parameter.
Output: Cg: Classifier that classifies input texts into y
Downstream Tasks and Metries. We
for y E ) do
use all the tasks included in GLUE [66] except STS-B which is a regression task. Please refer to Appendix C for more de- tails about GLUE tasks. We follow the
Ty + 0
// Class y train set init.
for E [1.2.....
do
x + Go(w,)
evaluation protocol of [13: We use FI score as the metric for QQP and MRPC, Matthews correlation for CoLA, and accu- racy for the rest of the tasks. The original development sets of these tasks are used for testing. For all reported results, we in- clude the average and standard deviation over 5 different random seeds.
To T, UT(x", .))
end
end T 0 // Selected train set. for y E y do
Sort T, in descending order by Fq.
T TU;
end
Models. Unless specified otherwise, we use CTRL (1.63B parameters) 23 as the generator Go and COCO-LM Large (367M parameters) 40 the classifier Cs. We also show the results using similar-sized
// Ensembled prediction init.
T + // Filtered train set. for i E 21, do
Fine-tune Co via Eq. 6 minibatch of T if 1%B = 0 then Update 2, z via Eq. 5 T ((x) y) B. > 8, (x y)
PLMs (GPT-2 [ST/RoBERTa 34) as the
generator/classifier in Section 5.6
E
ET
Fine-Tuning Settings and Hyperparam- eters. We note that SuperGen is compat- ible with any fine-tuning method; while us- ing more sophisticated methods may grant further performance improvement, we use the basic prompt-hased fine-tuning with manual templates approach for simplicity and clarity. For all tasks, we use the same templates and label words as in
end
end return C3 = Co
[13. Under the zero-shot learning setting, it is not possible to tune hyperparameters due to the lack

---

Table Results on seven GLUE classification tasks. We report average and standard deviation (as subscripts) performance over different random seeds. Results from LM-BFF [13].
MNLII(m/mm)
QQP (F1)
QNLI (Acc.)
SST-2 (Acc.)
CoLA (Matt)
RTE (Acc.)
MRPC
AVG
Method
(Acc.)
(F-1)
Zero-Shot Setting: No task-ipecific data (neither labeled unlabeled).
Prompting'
50 80...///5..7.
49.7o.o
50.8p.a
83. 83.6..o
2.0a.a
51.30.0
61.9u.
50.1
SuperGen
72.30.s/73.8p.5
66.11.1
73.31.9
92.81.a
32.75.s
65.31.2
82.2n.5
69.4
data selection
63. 63.71.5/64.2. //..
62.32.2
63.95.2
91.32.
30.5g.8
62.41.5
81.6n.2
65.1
label smooth
70.7o.a/72. 1o.7
65.10.9
71.42.5
91.0p.9
9.51.
64.81.1
83.0p.7
65.2
temporal ensemble
62. 6/63 o663./66.6.... 64.8
63.9u.s
72.42.0
92.5u.
23.57.0
63.51.v
78.82
65.3
Few-Shot Setting: Use 32 laheled samplesiclass (half for training and half for development).
Fine-tuning'
45.8s.4/47 .8.s
60.74.3
60.2
81.4s.s
33.914.3
54.49.9
76.62.s
59.1
Manual prompt'
68.32.s/70.51..
65.5s.3
64.5a.2
92.70.9
9.37.a
69.1a.6
74.55.3
63.6
demonstration
70.71.s/72.0..
69.81.s
69.21.9
92.6o.s
18.7s.s
68.72.a
77.82.0
66.9
Auto prompt
68.32.a/70.124
67.0s.o
68.37.4
92.31.o
14.014.1
73.92.2
76.22.3
65.8
demonstration
70.0s.o/7..0..
67.75.s
68.5%.4
93.0g.a
21.815.9
71.1..3
78.13.
67.3
Fully supervised!
89.8/89.5
81.7
93.3
95.0
62.6
80.9
91.4
84.9
Table 3: Results with different groups of prompts. CoLA does not use prompts for generation. The number of prompt groups is equal to the number of the task labels.
Prompt Group
MNLII-m/mm)
QQP
QNLI
SST-2
RTE
MRPC
#0 (Original)
72.30.
66.11.1
73.31.9
92.8p.a
65.31.2
82.2u.o
#1
70. 70.7..u/72.41.2
65.51.4
71.91.7
92.20.9
64.41.a
81.9p.4
#2
70.8u.u/72.lax.
65.61.1
72.22
92.4u.s
64.71.8
81.8p.s
#3
70.91.
Mixed
7/73.o/
66.91.5
73.01.7
92.8p.9
66.3
81.32.0
of validation sets. Therefore, we keep all fine-tuning hyperparameters (e.g., learning rate, batch size, training epochs, number of generated training samples, label smoothing and temporal ensembling hyperparameters) the same across all tasks. See Appendix B Table 10 for details.
Compared Methods and Ablations. We include the results of zero-shot prompting, standard few-shot fine-tuning and the four few-shot prompt-based fine-tuning methods proposed in [13]. We also conduct ablation studies by removing the following three techniques from SuperGen one at a time: (1) not using Eq. 33) for training data selection but randomly selecting the same amount of training data (- data selection); (2) not using label smoothing (- label smooth) but using one-hot labels; and (3) not using temporal ensembling (i.e., using Eq. (4) instead of Eq. (6) as the training objective) (- temporal ensemble). Lastly, include the fully supervised fine-tuning results trained on the entire training sets.
Evaluation
5.1 Main Results
We present the results of SuperGen, its ablations and compared methods in Table Overall, SuperGen significantly outperforms zero-shot prompting and achieves an overall better result than all few-shot methods. Notably, SuperGen results in much smaller variance over different random seeds than few-shot approaches on most tasks-wird access to more training data, fine-tuning of PLMs becomes much more stable. The ablation results demonstrate that all three strategies (i.e., quality training data selection, label smoothing and temporal ensembling) play important roles in improving and stabilizing the final performance, especially on challenging tasks like MNLI.
5.2 Using Different Prompts
One important factor of SuperGen is the choice of label-descriptive prompts as they directly influence the quality of generated training samples. To study the impact of different prompt choices on the
7

---

70
000
73-
30
50
70-
Fen, Sast Ords
40
FewoSost SigerGes
0.1 0.2
0.1
1.0
2.0
10
10.0
0.1
0.2
0.4
1.0
20
4.0
10.0
322
64
198
256
# Generaatd Thuining Date (xclel)
# Generated Troining Data Used (x1e3)
# Pew-Stool Samppee Per Labee
(a) MNLI-m.
(b) SST-2.
Figure 3: Classifier accuracy on MNLI-m fine-tuned on the few- shot samples only vs. on the few- shot and SuperGen generated set with varying few-shot set sizes.
Figure 2: Classifier accuracy fine-tuned on different amount of generated training data (after data selection). Dots and error bars are the average performance and the standard deviation over seeds, respectively.
final model performance, create different groups of prompts other than the original ones. We replace the prompt for one label used in Table T with synonymous one and keep other prompts unchanged when forming a different prompt group (Please refer to Appendix A Table for details). We also experiment with mixing the generated data by different prompt groups (mixed). The results are shown in Table 3 Overall, the model performance under different prompts is quite close, except on RTE whose test set is very small, potentially resulting in the higher variance. In this work, manually choose simple prompts that make intuitive sense, and we leave the automatic searching of optimal prompts as future work.
5.3 Results with Different Amount of Generated Data
With training data automatically created by the generator, we can have a virtually infinite amount of training samples. We show the results of using different amount of generated data (after quality data selection) for fine-tuning the classifier Cs in Fig. on MNLI-m and SST-2. When the number of training data is small (e.g., 100), the fine-tuning variance is high, resulting in the similar instability issue with few-shot settings. With more generated data used, both average performance and training stability improve, yielding comparable results (with smaller variance) to fine-tuning using few-shot task-specific data. However, when too many generated data (e.g., 10, 000) are used, the classifier's performance slightly drops, probably due increased label moise-recaa that the training data are selected based on the ranking score in Eq. (3). so using more data results in the inclusion of more lower-ranking texts in the training set and reduced data quality. One way to address this issue is to use fixed selection ratio and increase the total number of generated texts to obtain a larger number of high-quality training data. However, this comes at greater computation cost in the generation step. An important future direction is thus to develop better data selection strategies.
5.4 Using SuperGen in Few-Shot Settings
We present a simple extension of SuperGen to few-shot settings and show that the generated data of SuperGen may also improve the few-shot performance. When few-shot samples available, first fine-tune the classifier on the few-shot training set (standard prompt-based based fine-tuning without regularization), and then continue tie-ttunig the classifier on the generated data by SuperGen as described in Section 3.3 This allows the classifier to effectively leverage the knowledge from the few-shot training set to filter out noisy samples in the generated data, as temporal ensembling regularizes the classifier to remember the predictions learned previously and only keeps samples on which the model predictions agree with the label. We show the benefits of incorporating generated data for different few-shot sample sizes on MNLI in Fig. ( use half labeled samples for classifier training and half for development): When the few-shot training and validation sets are rather small (32 64 samples per label in total), fine-tuning the classifier on the SuperGen generated set further (after fine-tuning on the few-shot samples) brings notable performance improvements. However, such benefits diminish with more few-shot training samples: The generated data fail to improve the few-shot performance when there are 128 samples per label, and even worsen the classifier performance with 256 samples per label. This is probably because our synthetic data generation process is zero-shot and does not leverage any few-shot samples; the resulting generated samples may not be of high enough quality to boost the few-shot performance when there are relatively abundant annotated samples. Possible ways to few-shot samples for generation include using them as demonstrations [5], for creating augmentations (29] and for tuning the generators. We leave the explorations of generating higher quality data by leveraging shot samples for future work.
8

---

Table 4: Comparisons with using CTRL for zero- shot prompting and for knowledge distillation. I:
Table 5:
Results with different
genera-
tor/classifier PL.Ss
The entire training set used as unlabeled data.
PLMs (GolCa)
(NLI-(//mm)
SST-2
Method
MNLI-(m/mm)
SST-2
CTRL/COCO-LM
72.30.s/73.80.5
92.8o.6
SuperGen
72.3v.a/73.8u.s
92.8g.G
CTRL/RoBERTa
69.0o.s/70.66
93.01.5
CTRL Prompting
38. 0t.9
72.5u.o
GPT-2/COCO-LM
2
88 88.21.s
Knowledge Distill
10.8p.a/11.5m
73.6u.
GPT-2/RoBERTa
68.3o./699.o..7
88.6u.s
5.5
Using Generators for Knowledge Distillation
Apart from using unidirectional PLMs Ge for training data generation, one could also directly apply them to unlabeled data formulated as prompts to obtain zero-shot predictions (i.e., prompting (5 (3)
which can then be used as soft labels to train the classifier Co. In Table 4 we show (1) the zero-shot prediction accuracy of CTRL (the best out of three different prompts, details in Appendix D and (2) the classifier performance trained from CTRL's predictions on the entire unlabeled training set as
soft labels (i.e., knowledge distillation). Similar to the observations in previous studies 5 70 85],
the zero-shot predictions of unidirectional PLMs are quite inaccurate and directly using them as soft labels to train classifiers does not yield good results. We hypothesize that the advantages of using unidirectional PLMs for training data generation over using them for zero-shot predictions are twofold: (1) Better flexibility in prompt formats. When unidirectional PLMs used for zero-shot predictions, the prompts have to be designed so that the label word is the last token in the sequence to be predicted, as unidirectional PLMs cannot attend to subsequent tokens. Such constraints may result in the prompt being dissimilar to the pretraining data distribution and worsen the prediction quality of the PLMs. On the contrary, using unidirectional PLMs for generation is not subject to any prompt format constraints. (2) More direct uses of PLMs' language modeling ability. Using unidirectional PLMs for training data generation directly leverages the PLMs' output token probability. Applying PLMs for zero-shot prediction, however, requires an additional step to convert token predictions to label predictions (i.e., the verbalizer (56)). and such mapping process usually necessitates manual curation and can hardly be optimal 13 especially without abundant task-specific data.
5.6 Using Different PLMs
The final performance of SuperGen is relevant the choice of PLMs as the generatoo/classifier. Apart from the default PLM choice, we report the results of using GPT-2x (1.54B parameters) [51] as the generator and RoBERTaLung Lngee (356M parameters) 33 as the classifier in Table 5 with every- thing else unchanged. When using GPT-2, we change the prompt used for SST-2 to "The film is bad/terible/awfful. for the negative label and "The film is good/great/excellen
for the positive label, since the original prompts used for SST-2 in Tarle part of the control codes of CTRL and cannot be effectively leveraged by GPT-2. Overall, both CTRL and GPT-2 are able to generate quality training data for good fine-tuned classifier performance; CTRL consistently yields better results than GPT-2 regardless of the choice of the classifier PLM, probably because CTRL is pretrained with control codes which provide explicit guidance for generating texts of certain domains and attributes. We also observe that the generated text quality is strongly correlated to the generator's s model size- -uiin a smaller version of GPT-2 (e.g.. with 1 117M parameters) results significantly less coherent texts and can hardly serve as training data. An interesting future direction is to try larger generator PLMs (e.g., GPT-3) which may create training data of better quality.
5.7 Case Studies
We present concrete examples of generated texts guided by prompts of different labels in Table 6 The generated sequences are not only coherent, but also pertain to the corresponding labels. For easier tasks like SST-2, the generated texts almost always correctly reflect the desired sentiment polarity specified by the prompt. For more difficult tasks like MNLI, sometimes the generated texts are not of the correct label (Appendix E Table [3 shows some negative results). The existence of such label noise motivates our use of the regularization techniques in the fine-tuning stage. In the future, it will be interesting to develop methods to better control text generation towards the desired label.

---

Table Example generated texts for SST-2, MNLI and QQP. Sampled sequences from pretraining corpus (z*) are italicized; generated sequences (x*) ) are underlined; prompts (w") are in bold.
Task
Label
Generated Text
SST-2
positive
Rating: 5.0 The fil is a great example of the kind of movie that you can watch over and over.
negative
Rating: 1.0 The film was a total waste of time. I would not recommend this movie to anyone.
The consiraction of the station begon in 2016, when the opening dute was extimuted to mid-20201
cotailment
other words, the station was to opea in 2020.
MNLI
neutral
The construction of station began in
opening
estimated to mid-2020.
Furthermore, is expected that bus terminal will be built at this station.
There is rumor that The constraction of the station began in 2016, when the opening date war estimated to mid- mid-2020.
contradiction
However, the truth is: The constraction started in 2017, and the afficial opening date was set for March 31, 2018.
equivalent
What are the moxt wear resisiont steels? In other words, what are the most durable steels? What are the most weur resisiont steels? Furthermore, what is the best way to clean them?
QQP
not equivalent
Discussions and Conclusions
While PLMs have demonstrated remarkable text generation and un- derstanding capability, they come with potential risks harms 2 B such as generating misinformation [46) or amplifying harmful biases [49]. The focus of our work is on utilizing exist-
Ethical Considerations.
ing PLMs to generate training data for NLU tasks instead of developing new PLMs or generation methods. Therefore, our method can be used in company with any bias reduction and correction techniques [5 37 to mitigate the risks of PLMs.
Limitations. One inherent limitation with zero-shot learning is the lack of access to task-specific samples for hyperparameter tuning, whereas the performance of neural networks is usually heavily dependent on the choice of hyperparameters even when the training algorithm and training set are fixed [48) J. Also, without access to any labeled data, the generated training data quality may not be high enough to achieve good performance on challenging tasks, especially when the task distribution is significantly different from the pretraining data distribution (e.g., the "linguistically incorrect" label of CoLA requires generating sequences with grammar mistakes different distribution from the one used to train PLMs). A promising direction to address the above limitations is extending SuperGen to few-shot settings (e.g., the setting studied in Section 5.4) and leveraging small amount of labeled data for generating better quality data and for hyperparameter tuning.
Conclusions. We propose SuperGen, an automatic supervision generation approach for zero-shot learning of NLU tasks. By providing label-descriptive prompts guidance to unidirectional PLM, training data can be automatically created for fine-tuning a bidirectional PLM. Our framework differs from previous wransferleearing--are cross-task annotations and eliminates the task difference in training and inference. We show that several strategies are important for effective and stable fine-tuning on generated data, including quality training data selection, label smoothing and temporal ensembling. SuperGen achieves strong performance on seven classification tasks of the GI.EE benchmark, even yielding comparable or better results than sophisticated few-shot learning methods and offering better stability. There is large room for future work, including but not limited to: Extension to few-shot learning settings, exploring larger generator models [25] (68. better fine-tuning techniques to leverage generated data and better strategies for selecting quality training data.
zero-shot methods in that SuperGen does not rely on
Acknowledgments
Research was supported in part by US DARPA KAIROS Program No. FA8750-19-2-1000 and INCAS Program No. HR001121COII National Science Foundation IIS-19-56151, IIS- 11-417-,,,,, and IIS 17-04532, and the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897. and the Institute for Geospatial Understanding through Integrative Discovery Environment (I-UUDEE) by NSF under Award No. 2118329, Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Government. Yu Meng is supported by the Google PhD Fellowship. We thank anonymous reviewers for valuable and insightful feedback.
10

---

References
[1] Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! In AAAI, 2020.
[2] Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In ACI, 2020.
[3] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accounlability, and Transpurency, 2021.
[4] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In NeurIPS, 2020.
[6] Alvin Chan, Y. Ong. Bill Tuck Weng Pung, Aston Zhang, and Jie Fu. CoCon: A self-supervised approach for controlled text generation. In ICI.R, 2021.
[7] Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: kinguistically-nffoni interpolation of hidden space for semi-supervised text classification. In ACL, 2020.
[8] Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
[9] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
[10] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In ICIRR, 2020.
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[12] William B Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. International Workshop on Paraphrasing (/WP), 2005.
[13] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL, 2021.
[14] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In EMNLP, 2021.
[15) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxi- cityPrompts: Evaluating neural toxic degeneration language models. In Findings of EMNLP. 2020.
[16] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal rec- ognizing textual entailment challenge. ACL-PA workshop on textual entailment and paraphrasing, 2007.
[17] Aaron Gokaslan and Vanya Cohen. WebText corpus. http:://kkylccn077.giic. Lt/Oppe:NebTextCorrpus 2019.
[18] Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In PASCAL Challenges Workshop on Recognising Textual Entailment, 2006.
[19] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding- BERT with disentangled attention. In ICLR, 2021.

---

[20] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02533 2015.
[21] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020.
[22] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward controlled generation of text. In ICML, 2017.
[23] Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. ArXiv, hes/1909.058555 2019.
[24] Muhammad Khalifa, Hady ElSahar, and Marc Dymetman. A distributional approach to con- trolled text generation. In ICLR, 2021.
[25] Boseop Kim, HyoungSeok Kim, Sang-Wou Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. What changes can large- scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. In EMNLP, 2021.
[26] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Rajani. GeDi: Generative discriminator guided sequence generation. In EMNLP. 2021.
[27] Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. Controlled text generation as continuous optimization with multiple constraints. In NeurIPS, 2021.
[28] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. ICLR, 2017.
[29] Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural data 2220011133333 2021.
augmentation via example extrapolation. arXiv preprint
[30] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeeee pre-training for natural language generation, translation, and comprehension. In ACI, 2020.
[31] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
[32] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time time controlled text generation with experts and anti-experts. In ACL, 2021.
[33] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. ArXiv, abs/2103. 10385, 2021.
[34] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
[35] Robert I. Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.1 00t:22006.1353.. 2021.
[36] Michal Lukasik, Srinadh Bhojanapalli. Aditya Krishna Menon, and Surinder Kumar. Does label smoothing mitigate label noise? In ICML, 2020.
[37] Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. PowerTransformer: Unsupervised controllable revision for biased language correction. In EMNLP, 2020.
[38] Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. Weakly-supervised neural text classifi- cation. In CIKM, 2018
[39] Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. Weakly-supervice hicrarchical text classification. In AAAI, 2019.
12

---

[40] Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. COCO-LM: Correcting and contrasting text sequences for language model pretraining. In NeurlPS, 2021.
[41] Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Pretraining text encoders with adversarial mixture of training signal generators. In ICLR, 2022.
[42] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task gener- alization via natural language crowdsoureing instructions. arXiv preprint arXiv:2104.08773, 2021.
[43] Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. Adversarial training methods for semi-supervised text classification. In ICIRR, 2017.
[44] Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NeurlPS, 2019.
[45] Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. SELF: Learning to filter noisy labels with self- ensembling. In ICLR, 2020.
[46] Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstractive summarization with FRANK: benchmark for factuality metrics. In NAACL, 2021.
[47] Damian Pascual, Béni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer. A plug-and-plas method for controlled text generation. In Findings of EMNLP, 2021.
[48] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In NeurlPS, 2021.
[49] Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. Style transfer through back-translation. In ACL, 2018.
[50] Raul Puri and Bryan Catanzaro. Zero-shot text classification with generative language models. ArXiv, abs/1912. 10165, 2019.
[51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
[52] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2019.
[53] Nils Reimers and Iryna Gurevych. BERT-networks, In EMNLP, 2019.
Sentence-BERTT Sentence embeddings using siamese
[54] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.
[55] Teven Le Scao and Alexander M. Rush. How many data points is a prompt worth? In NAACI. 2021.
[56] Timo Schick and Hinrich Schütze. Exploiting clozeequestionn for few-shot text classification and natural language inference. In EACL, 2021.
[57] Timo Schick and Hinrich Schütze. Few-shot text generation with natural language instructions. In EMNLP, 2021.
[58] Timo Schick and Hinrich Schüizze Generating datasets with pretrained language models. In EMNLP, 2021.
[59] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In NAACL, 2021.
13

---

[60] Iyer Shankar. Dandekar Nikhil, and Csernai Koréel. First Quora dataset re-
lease: Question pairs, 2017. URL https://www. quora com/ haam/q/quoreeee quoradata/ F First-Quora-Datasee ootpraattttttaaaeeeeooottttooaaa
[61] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew
Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, 2013.
[62] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. CVPR, 2016.
[63] Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. In EMNLP, 2021.
[64] Lisa Torey and Jude Shavlik. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques. 2010.
[65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[66] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE:: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNI.P, 2018 8.
[67] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019.
[68] Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. arXiv preprint arXiv:2109.0919) 2021.
[69] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. In TACL, 2019,
[70] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
[71] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In ICLR, 2020.
[72] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-cor troad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLI, 2018.
[73] Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In NeurIPS, 2020.
[74] Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. ZeroPrompt: Scaling prompt-based pretraining to 1, 0000 tasks improves zero-shot generalization. ArXiv, abs/2201 .0690069 10, 2022.
[75] Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators, In NAACI, 2021.
[76] Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmenta- tion for commonsense reasoning. In Findings of EMNLP, 2020.
[77] Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Ling- peng Kong. ZeroGen: Efficient zero-shot learning via dataset generation. ArXiv, abs/2202.079922. 2022.
[78] Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. In EMNLP, 2021.
[79] Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In EMNLP, 2019.
14

---

[80] Wenpeng Yin, Nazneen Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong, Universal natural language processing with limited annotations: Try few-shot textual entailment start. In EMNLP, 2020.
[81] Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Woo Lee, and Woomyeong Park. GPT3Mix: Leveraging large-scale language models for text augmentation. In EMNLP Findings, 2021.
[82] Weizhe Yuan, Graham Neubig, and Pengfei Liu. BARTScore: Evaluating generated text text generation. In NeurIPS, 2021.
[83] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
[84] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In ICLR, 2022.
[85] Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
[86] Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In Findings of EMNLP, 2021.
[87] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 2020.
[88] Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, Abs//909.00899, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
15

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "u_jhedqFr3M", "reviewer": "Reviewer_V4Sv", "summary": "They study zero-shot learning of PLMs on NLU tasks without any task-specific or cross-task data.\n\nThey propose SuperGen approach: training data are created via unidirectional PLM. A bidirectional PLM is then finetuned on the generated texts to perform tasks.\n\nAcross seven classification tasks of the GLUE benchmark, SuperGen significantly outperforms the prompt-based zero-shot method. Their method for selecting quality training data based on their generated probability and using label smoothing and temporal ensembling demonstrates the strong performance of SuperGen.", "questions": "- In Table2, the performance of SuperGen without label smoothing for CoLA drops a lot. Is there a specific reason for that?\n- Generator selection: Why did you not use large size PLMs (e.g., GPT3) from the beginning for the high-quality generation, if the model size matters for high-quality text generation like you mentioned in page 8, line 280?\n- Are there other regularization methods other than label smoothing and temporal ensembling that you might have considered?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Supervision automatically created by the generator eliminates the need for task-specific annotations.\n- Rewarding and penalizing repetitions for sequence pair generation reduces the chance that the generated texts get stuck in repetition loops.\n- Reasonable method for selecting quality training data to take the average log probability.\n- Label smoothing forces the classifier to have less probability on training data. That improves robustness to label noise and prevent overfitting to the training set.\n\nWeaknesses\n- Since prompt would be a key factor for the performance, it would be nice to have prompt ablation studies for each task, not just three tasks(MNLI, RTE, MRPC).\n- Even though SuperGen significantly outperforms zero-shot prompting and achieves a better result than all the few-shot methods, the performance cannot match to their SOTA results.\n- The generated texts are not judged by humans.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RwoArVe8WHT", "reviewer": "Reviewer_kf6s", "summary": "This paper proposes generating training data for certain NLU tasks using a pre-trained generative model and using this data to fine-tune a pre-trained MLM for the target task. The generated data is filtered based on likelihood and the classifier is regularized using label smoothing for robustness to noise. They generate on the order of a few thousand examples using prompts and the label space for 7 GLUE tasks and compare against zero-shot and few-shot baselines.", "questions": "Have you considered generating synthetic data using the few-shot examples? Do you think that would make it possible to generate higher quality data with a greater value-add?\nE.g. https://arxiv.org/pdf/2102.01335.pdf ", "limitations": "I think the limitations section doesn't really discuss potential societal impact. Perhaps the synthetic data generated by the model might contain unwanted biases that could be harmful.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \nThe proposed approach to generate data using prompts is quite interesting and is generating data of good enough quality to achieve non-trivially good performance on the GLUE tasks. This has important implications for showing that using LLMs, zero- and few-shot settings don’t need to be as such by augmenting them with synthetic data, at least for the kinds of tasks explored in this paper. \n\nWeaknesses:  \nCalling this entire setup “zero-shot” is a bit of a stretch. The generator can be seen as zero-shot for generating task-specific examples, but the fine-tuned classifier not so much. For this reason, the truly zero-shot baseline seems weak and it is unsurprising that the classifier fine-tuned on synthetic data outperforms it. Perhaps an interesting comparison that was left out was fine-tuning on few-shot data vs. fine-tuning on a combination of few-shot and the synthetic data to quantify how much value is added by the synthetic data. \n\nThere’s nothing new and surprising about using synthetic data, it’s more the approach to generating the synthetic data that is the interesting contribution here. The story (and title) could use some revision to clarify this. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LQkDloDQOBd", "reviewer": "Reviewer_XfNA", "summary": "This paper proposes SuperGen, which targeting at automatic dataset generation for zero-shot learning of NLU tasks. The framework relies on manually designed prompts to \"extract\" task-specific knowledge from a pre-trained PLM, thus constructing a synthetic dataset. In the whole process, no human annotations are required. The experiments show that the proposed method can even outperform few-shot learning methods on some NLU tasks. ", "questions": "\"we include the average and standard deviation over 5 different random seeds.\"\nAre these 5 runs performed on 5 different few-shot train/dev sets or on the same pre-sampled set over 5 different runs? ", "limitations": "In spite of excellent results on easy tasks such as simple text classification, the overall performance is still poor on tough tasks like CoLA. This limitation has to be discussed. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe idea of modifying an auto-regressive PLM decision boundary to a discriminative one by generating synthetic data is interesting.\n\nThe empirical evaluation shows promising results of SuperGen beating the performance of PLMs used for the few-shot setting.\n\nWeakness:\n\nAlthough it's encouraging to see that the zero-shot SuperGen outperforms the few-shot setting, I'm not fully convinced by the results. As been widely discussed, the data selection process under a few-shot setting can have a huge impact on evaluation results. Not to mention the impact brought by prompts, decoding strategies, etc. I understand that some of these factors are difficult to control, so I suggest the following baselines for evaluation:\n\nPLMs fine-tuned on the original train set for each task. While I understand that these results will outperform the results of SuperGen, I think it is still relevant and important to know what is the gap in performance between fine-tuning the PLMs and the zero-shot evaluations as shown in the Table.\nA knowledge distillation baseline: using the PLM to generate the output label when provided with an unlabeled text input (from some corpus), and then using these pseudo labels to train a discriminative model. This would help shed light on what is the benefit of creating the synthetic data rather than directly transferring knowledge from the PLM to downstream models.\n\nIt would have been interesting to see if the generated synthetic data in a zero-shot manner could add to the labeled supervised data for the tasks via data augmentation, and help improve the performance of the fully supervised setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept