Meta-Weighted Language Model Tuning for Augmentation-Enhanced Few-Shot Learning
Authors: Yu Meng
Martin Michalski
Jiaxin Huang
Yu Zhang
Tarek Abdelzaher
Jiawei Han
Venue: Submitted to ICLR 2023 (2023)
Keywords: Few-Shot Learning
Natural Language Understanding

Abstract: Recent studies have revealed the intriguing few-shot learning ability of pretrained language models (PLMs): They can quickly adapt to a new task when fine-tuned on a small amount of labeled data formulated as prompts, without requiring abundant task-specific annotations. Despite their promising performance, most existing few-shot approaches that only learn from the small training set still underperform fully supervised training by nontrivial margins. In this work, we study few-shot learning with PLMs from a different perspective: We first tune an autoregressive PLM on the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples which augment the original training set. To encourage the generator to produce label-discriminative samples, we train it via weighted maximum likelihood where the weight of each token is automatically adjusted based on a discriminative meta-learning objective. A classification PLM can then be fine-tuned on both the few-shot and the synthetic samples with regularization for better generalization and stability. Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods.

Under review as a conference paper at ICLR 2023
META- WEETAWEEGGTED LANGUAGE MODEL TUNING FOR AUGMENTATION-ENHANGGI FEW-SHOT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent studies have revealed the intriguing few-shot learning ability of pretrained language models (PLMs): They can quickly adapt to new task when fine-tuned on small amount of labeled data formulated as prompts, without requiring abundant task-specific annotations. Despite their promising performance, most existing few- shot approaches that only learn from the small training set still underperform fully supervised training by nontrivial margins. In this work, we study few-shot learning with PLMs from different perspective: We first tune autoregressive PLM the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples which augment the original training set. To encourage the generator to produce label-discriminatiee samples, we train it via weighted maximum likelihood where the weight of each token is automatically adjusted based on a discriminative meta-learning objective. A classification PLM can then be fine-tuned on both the few-shot and the synthetic samples with regularization for better generalization and stability. Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods, improving no-augmentation methods by 5+ average points, and outperforming augmentation methods by 3+ average points]
INTRODUCTION
Recent research has demonstrated the appealing few-shot learning potential of pretrained language
models (PLMs) (Brown et al.) 2020 Clark al., 2020 Devlin al. 2019; He et al. [22TT] Liu et al.)
2019 Meng et al. 2021) on natural language understanding (NLU) tasks Wang et al. [201]] 2018):
Instead of relying on abundant task-specific annotations, PLMs can effectively leverage small set
of training samples to quickly learn new task. Such training data efficiency is usually achieved by
formulating downstream tasks as prompts (Brown et al. 2020; Gao et al. 2021 Scao & Rush 2021
Schick & Schütze,, 202 d) which allow the PLM adapt its language modeling ability acquired through pretraining to new downstream tasks.
The success of pramet-based methods has stimulated numerous explorations along the line of effective few-shot learning with PLMs: The training samples converted to natural language prompts can be used directly fine-tune PLMs (Gao al. 2021 Schick & Schütze 2021a) or in-context demonstrations to facilitate better inference (Brown et al.) 2020: Liu et al. (022b). More recent approaches aim automate the design of prompts by gradient-baset searching (Shin et al. 2020) or parameterizing prompts as continuous learnable embeddings (Lester et al.] 2021 et al. 2021b) Zhang al. 2022, Zhong et al. [02I). Other studies investigate and address specific issues in prompt- based few-shot learning (Liu et al. [202aa] Tam et al., 2021 Zhao et al.) (202)). While remarkable, the model performance still has nontrivial gap from fully supervised models trained on massive labeled data. Indeed, training deep models is inherently data demanding model generalization
usually benefits from more training samples (Baum& & Haussler 1988)
In this work, we study few-shot learning with PLMs from a different perspective: Instead of proposing new methods for fine-tuning on few-shot samples, we focus on the generation of quality training data based on few-shot samples and using these synthesized training samples to fine-tune the classification
models. Motivated by the strong text generation power of autoregressive PLMs (Brown et al. 2020
Code shared in the supplementary material.

---

Under review as a conference paper at ICLR 2023
Keskar al., 2019 Raffel et al. ) previous data augmentation methods enlarge the training set by synthesizing new samples based on the few-shot samples. They either fine-tune the generator on the training set with the standard maximum likelihood objective (Anaby- Tavor al. 2020 Kumar et al.] 2020) or use the training samples as demonstrations (Yoo et al., (021). However, these methods do not explicitly model the distinction across different labels and may struggle to generate accurate training samples pertaining to the desired labels for challenging NLU tasks.
In this paper, we study how use few-shot samples effectively tune PLMs to generate high quality label-disciminnatiee training samples. Our contributions are as follows: (1) We analyze the issues of using standard maximum likelihood for tuning the generator and propose a meta- meig-weighted maximum likelihood objective for generator tuning by automatically learning token weights that emphasize label diseriminativeness. (2) We propose simple and effective training procedure for fine-tuning classification PLMs on generated data by mitigating label noise. (3) Under the same few-shot learning setting, our method FewGen outperforms existing methods by 3+ average points on seven classification tasks of the GLUE benchmark (Wang al. 2018). Ablation studies demonstrate the effectiveness our proposed meta-weighted training objective and classifier fine-tuning method.
2 RELATED WORK
Few-Shot Learning with PLMs. Few-shot learning has gained much attention recently due to its minimal resource assumption-Withonn requiring massive annotated data but only leveraging a few training samples (e.g., 16 per label), few-shot methods can be widely adopted in many practical scenarios where obtaining large-scale annotations is unaffordable. Standard fine-tuning of PLMS for few-shot learning usually performs poorly because the limited training samples may not be sufficient for optimizing the parameters in the newly introduced classification head. To reuse the language modeling ability of PLMs without introducing randomly initialized parameters,
prompt-based approaches (Brown al. 2020: Gao al., 2021 Hu et al. 2022 Logan IV et al.) 2021 [in et al.) 2022 [chiick& & Schützz 2202a b d.d.Tamm et al. 2021) formulate training samples as natural language prompt templates so that various downsteam tasks can be solved as token prediction problem. They enjoy improved training data efficiency over standard fine-tuning in low- data regimes (Scao & Rush (02T) and achieve remarkable few-shot learning performance. Later developments in prompt- based methods replace the manual design of prompt templates with automatic search or learning (Cui et al., 2022 Hambardzumyan et al. 202 I Lester et al. 2021: Liu al.. 2021b Zhang et al., 2022 Zhong et al. (202]). There are also studies focusing on specific issues in prompt-base based methods such as densifying the supervision by revising the training objective (Liu et al., 2022al Tam et al.] 2021) and calibrating the biased predictions of PLMs before fine-tuning (Zhao ct al. 2021). Instead of focusing on fine-tuning methods for few-shot learning, we study how to effectively generate abundant quality training samples by learning from the few-shot samples and use them to improve the generalization of the classification model.
Data Augmentation. Data augmentation methods (Chen et al. 2020 Lee et al., 2021 Miyato al., 2017: Xie et al. 2020) aim to create similar samples to the existing ones so that the enlarged training set can benefit model generalization. Early approaches simply use manually designed rules (e.g., swapping or inserting tokens) for word-level alterations over the given samples to create new ones (Wei & Zou 2019). Later methods leverage the strong generation power of PLMs to synthesize novel samples from scratch. Given a training set, the PLMs can be either fine-tuned on the labeled
samples to learn label-conditioned generation probability (Kumar et al. 2020 Lee et al. 2021 Yang
et al., 2020) or take the labeled data as demonstrations (Wang et al. 2021 Yoo et al. 2021) to generate similar samples pertaining to the same label. In this work, we study how effectively tune
generators on few-shot training data for creating new dtaa-standard fine-tuning of PLMs on a small set of training data is prone overfitting, and the resulting model may struggle to generate accurate, diverse and novel training data. We address this challenge by leveraging prefix-tuning and proposing a new meta- weighted training objective to emphasize dibr-iiisriiinnatvee tokens for generator tuning.
Controlled Text Generation. Generating training samples for different labels can be viewed as form of controlled text generation (Hu et al.] (017). whose goal is to generate textual contents of desired semantics, styles or attributes. Such control can be realized through different stages of PLM training and deployment: During pretraining, control codes (Keskar al. 2019) can be used as
2

---

Under review as a conference paper at ICLR 2023
Generator Training
Classifier Training
Darain
Dirin
V-ggens
disse
Step 1: Supervised Training on Drain
gen
Igen
Classification PLM
D
Prefix
Autoregressive PLM (frozen)
Generate
gen
Step 2: Training with Regularization and Sample Filtering on D Zgen
Figure 1: Overview of FewGen. A generator PLM is first tuned on the few-shot samples with our proposed meta-weighted maximum likelihood objective and then used to synthesize new training samples. A classification PLM is finally trained on both the few-shot and the generated samples.
explicit guidance for training the model to generate Nomain/attributtesppccff texts; fine-tuning PLMs with attribute-specin data also grant high-level control (e.g., certain topics or sentiments (Ziegler
et al. 2019)). fine-gratd aneegrained control (e.g., specific words or phrases (Chan et al., (02II) or both (Khalifa
al. 202T ;: inference time, control over desired attributes
also be enforced without updating
the PLM parameters (Dathathri et al. [2020] Krause et al. [202]] Kumar et al.) 2021 Liu et al. 2021a
Pascual et al., [022]] Yang Klein [2021]. Recently, few studies explore fine-tuning autoregressive
PLMs (Anaby- Tavor et al. 2020; Yang et al. 2020) with the standard language modeling objective
on the training set or using label ssee-sseefficc prompts (Meng al., 2022 Schick Schütze 2021e
Wang et al. 2021 Ye et al. 2022) to steer text generation towards the desired label.
Meta-Learning for Sample Weighting. The idea of weighting training samples in the loss cal- culation originates from the class imbalance (Wang al. (017) and noisy label (Hendrycks et al.) 2018) learning scenarios- -By assigning higher weights to the samples from minority classes or lower weights to the noisy samples, the learning process is less impacted by the imbalance/label noise issues. Meta-learning (Andrychowicz et al. 2016 Finn et al. [017 Franceschi et al. 2018 Wu et al. 2018) is one way to automatically learn the weight for each sample. Specifically, meta objective, usually defined as the loss on clean unbiased validation set (Ren et al., 2018 Shhu et al.) 2019),, can be used to learn the sample weights which become hyperparameters that control the optimization of model parameters. Our work has different motivation and formulation of the meta objective for token-wiss weighted training: Not all tokens in a training sample are equally aabel-disrimminatie.. We thus design meta objective emphasize distinction across different labels (instead of using the validation loss as the meta objective) for learning the token weights.
3 METHOD
3.1 PRELIMINARIES
Overview. We consider the strict few-shot learning setting (Perez et al. 2021): The training set Dusin {(. y)} consists of K training samples per label where æ [1, Ø2. En, is text sequence with n tokens. The development set Davv is of the same size as Dtrain- There is no access additional task-specific unlabeled data. The number of training samples K is assumed to be very small (e.g., K = 16), making it challenging to train a classification model Cp that generalizes well to unseen data. To mitigate such training data scarcity issue, we propose first train an autoregressive PLM on Drmin. and then use it as generator Ge to synthesize a large amount of novel samples Dgen {6,, } that augment the original training set. Finally, a classification PLM C₂ is fine-tuned on both Dumin and Deen to perform the task. An overview of our FewGen method is shown in Fig.
Text Generation with Autoregressive PLMs. In standard fine-tuning for text generation, an autoregressive PLM Go is trained via the maximum likelihood generation loss of each token in
3

---

Under review as a conference paper at ICLR 2023
sequence æ conditioned previous tokens:
exple) L ,2)) y/= exp(e,,tt)
min log pe x<y),
where the token generation probability (e(.) is usually parameterized using token embeddings and hidden states of a Transformer (Vaswani et al. 2017) model. After training, Gg can used to generate novel texts by iteratively sampling tokens from its generation probability distribution.
Prefix-Tuning Unlike fine-tuning which updates all model parameters 0 of a PLM. prefix- tuning (Li & Liang 202T) freezes all pretrained Transformer parameters and only optimizes prefix vectors B that are prepended to each Transformer layer. We use prefix-tuning for training Gg on Dusin because (1) it offers better effectiveness than fine-tuning for small datasets (Li & Liang 2021) and (2) the generation models for different labels can share the same backbone Transformer parame- ters with only the prefix vectors being different, significantly reducing the memory requirement for multi-class classification tasks.
3.2 LABEL- SHEELL-DICIIMINATIV TEXT GENERATOR TUNING WITH META WEIGHTS
To model the conditional text generation probability (())))) different labels, straightforward way parameterize a generation model Gø,, for each label y via set of prefix vectors 02 = (6p, l/= so that p(z) y1) p0,, (z), and then tune Pp. on the training samples æ with label yı:
Motivation.
n
min Cyens O,
Lgen (p:))
log Po,y (xy (5)
However, such an approach only optimizes the generative like- lihood p(æ]ly)) without accounting for label discriminativennss p(yu [z) which is essential for generating unambiguous training samples to benefit the final classification task. Challenging NLU tasks can have largely similar distributions across different labels, with very nuanced differences reflected by a few key tokens. For example, a negative review text "a movie where the ending feels like a cop-our" may immediately become a positive one by just changing the last word "cop-out" to "revelation". Indeed, we find that such subtle distinctions over different labels may not be effectively captured by the generators if they trained with the standard generation objective in Eq. ( As shown in Fig. 2 Laise (defined in Eq. 2 can even increase during training- Is possible that the dominating patterns in the training samples are label-indise hhbel-indiscrmmnnte (e.g., movie review dataset may frequently mention "the movie"), making the generators of different labels eventually converge to similar distributions, especially when there are limited training samples per label.
Lake
Mapayn MM
w M
2010
Thsining Steps
Figure 2: (On MNLI) Training the generator only via Lgen does not automatically decrease Lisee
To promote the generation of dibeindiscrininative texts, we encourage each token xy to be more likely generated under the corresponding label y instead of other labels (i.e., maximize pe,, (zy (zz)) and minimize pe,,
( for 1' + via a discriminative loss L
n2
po,, (xz ((t))))))
Laise (0p)
c Ciso (0p), (0))
(2)
n
El po,, (xtg xr
Although one can directly combine Laise with Egen train Go, enforce distinction across different labels, doing so will result in two undesirable consequences: (1) A hyperparameter needs to be introduced to balance the weights of the two losses. whose optimal value is likely vary by task; and (2) the generation generation-irrelvaan irrelevant loss Lasee will unavoidably interfere the language modeling process, making the resulting model prone to generating less fluent and coherent texts.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Meta-Weighted Generator Tuning.
Input: rain: Few-shot training set.
Parameter: T:: Number of training steps.
Output: pg: Prefix parameters for all labels.
Initialize 0 (with task- task-descriptive prompts) and (0)
for t E 1,......
do
B + Sample a minibatch from Daiin
0 (w)()) <- Take one gradient step to descend Lu-gen (08) :w()) on B
w(++)) t- Take one gradient step to descend Lass (0))) (wo(1)) on B
0g ((++1) -
Take one gradient step to descend Lu-gon
(t) P wl(+1)) on B
end
return 0, O5T
Weighted Maximum Likelihood Generator Tuning. To preserve the generative learning of Ga while emphasizing abbe-diisriminaaiive tokens, we assume each token is associated with a weight in the maximum likelihood loss. Intuitively, when goal is to generate distinctive texts across different labels as training samples, not all tokens should contribute equally to generator training. For example, for sentiment classification tasks, one would expect "good/bad" to be more dibbldiiccrimiiative than "the movie", and the former should be paid more attention to during training. thus natural to generalize Lgen in Eq. ①1) to Lw-gen follows by assuming weight wy is given for each token.
min Lw-gons 0p
Cw-gon (0 w))
Σ j=1
W C. (On))
(0 log po,, (x))
(3)
Note that in Lu-gen. w is assumed to be the hyperparameter under which 6p. is optimized. When wy is the same for every token, Eq. (3) will be equivalent to Eq. {T). While it possible to manually design weighting rules for setting ur to promote label-discriminatie learning, they will likely necessitate task-specific knowledge and nontrivial tuning. To facilitate the automatic learning of these weights w, we propose to parameterize them as learnable hyperparameters using the idea of meta-learning.
Meta Weight Learning Setup. To automatically learn token weights using the idea of meta- learning, we formulate bi-level optimization problem, where the inner objective Lw-gen optimizes the generator parameters O,. and the outer objective Laise optimizes token weights that used as hyperparameters by the inner objective. We parameterize token weights w via a weighting network Iw so that wy wy wy(w). Details about the implementation of gw are in Appendix E Overall, the learning objectives are as follows:
p (w) argmin Cw Cu.gens
L (ep:()
Wi (w) C (0p)
gens
Op
i=1
(4)
w argmin Laises
Lasse (0) =
L (7)
Under the above formulation, the token weights (y( are automatically learned such that the resulting generator parameters 07(w)) capture discrimiiiei information (i.e., minimize Ldise)- Instead of solving the optimal w² and 0;, via nested optimization loops, we use an online optimization strategy (Shu et al. 2019) for training efficiency. It also guarantees convergence to the critical points of both Zw-gen and Laise under mild conditions. The initialization prompts can be found in Appendix C The overall training procedure is shown in Algorithm T
Analysis of Meta Weight Learning. We analyze the gradient update of meta weights to study its effect in generator tuning. The weighting network parameter optimized via Eq. 4 and its
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Classification model fine-tuning Danin and gen
Input: Driin Few-shot training set; Pgen Synthesized training set. Parameter: T: Number of training steps.
Output: Φ: Trained classification model parameters.
(0) <- Train on Duain with standard supervised learning
+- 0 // Ensemoled prediction initialization
for t E [0. T do
B f Sample a minibatch from Dgen
((++1) Take one gradient step to descend Lelass in Eq. B
$ f AAccmullate the current model prediction Update geen to exclude noisy samples based on z
end
return Φ (TT))
gradient is as follows (detailed derivation in Appendix A
DLasc 0 (w))
aaisc (p,)
Buy (w) Bw
ü. (0p)
dw
(=
00p
Ap=A
00,
of5
can be seen that the gradient descent direction of is determined by weighted sum of token weight gradient ascent direction (i.e., (w,(ww), ), where the weight dj characterizes the similarity between the gradient of the discriminative objective and the gradient of the generative objective on
the jth token. Therefore, the meta weights will be higher on those tokens where optimizing their generative objective is more beneficial for minimizing the discriminative objective.
3.3 CLASSIFIER FINE-TUNING
With the trained generator Ge we can synthesize novel training samples Dgen that augment 2 Priin for fine-tuning a PLM Cs for classification. The major challenge to effectively leverage D Dgen is that the label noise (i.e., some generated samples may not accurately pertain to the corresponding label) may deteriorate model performance if standard supervised learning is directly used. We propose simple noise-robust training procedure to improve the generalization and stability of training: First fine-tune C. on Dirain with standard supervised training, and then continue fine-tuning it on Dgen by applying temporal ensembling (Laine & Aila 2017) as regularization. Specifically, given a training sample (ø, E D Dgen. minimize the following classification loss:
Pø (T)) Z
min Lelasss Celass = log(ps())g))
2
21 log
(5)
1=1
where (œ) is the model prediction Ø; is a regularization weight for temporal ensembling: and z the accumulated moving-averan model predictions. We also use the ensembled prediction & to filter out noisy synthesized samples: We only include those samples for training where z strongly agrees with the label ý (i.e,, 2g > where term is the cross-entropy loss; the second regularization term corresponds temporal ensembling, which requires the current model prediction to be close to its past accumulated predictions. This not only neutralizes the fluctuation in model predictions for better training stability when label noise is present (Nguyen et al., (0200) but also helps prevent catastrophic forgetting (Kirkpatrick et al. 2017) of the information learned previously from the few-shot training D Please refer to Appendix C for details about the temporal ensembling implementation. The overall procedure of classifier fine-tuning is summarized in Algorithm
is a threshold parameter). In Eq. (5). the first classification
EXPERIMENTAL SETUP
Downstream Tasks and Metrics. We conduct evaluation on all tasks of the GLUE bench- mark (Wang et al. 2018) (more details in Appendix B) except STS-B which is regression task. We

---

Under review as a conference paper at ICLR 2023
Table 1: Results on seven classification tasks of the GLUE benchmark.
We report average and standard deviation (as subscripts) performance over 5 different maain Dike splits defined in Gao et al. (022)) Methods that use
(2021). t; Results from Gao et al. (2021). I, Results from Zhang et al.
additional models apart from the final classification model are marked.
Method
MNLL-mm/mm)
QQP
QNLI
SST-2
CoLAA (Mau))
RTE (Acc.)
MRPC (FI)
AVG
(Acc.)
(FI)
(Ace.)
(Arc.)
Methods without Angmentation: Few-shot somples are directly used for classifier tuning demonstrations for inference
Prompting
30.8/51.7
49.7
50.8
83.6
2.0
51.3
61.9
50.1
Fine-Tuning
45.8g.u447.86. 86.
60.74.5
60.2p.s
81.4
33.914.5
54.4s.9
76.62.5
59.1
Context!
52.0u, 4..6
36.1s.2
53.8u.a
84.81.
-1.52.4
60.41.4
45.76.
47.4
(Man.))
08.3v.//70.5..
65.5s.a
64.5a.2
92.7p.a
9.3ra
69.1s.6
74.5c.a
63.6
demonstration
77..r.a/7..0..
69.8
69.21 a
92.60.5
18.7a.
08.7s.a
77.8
66.9
LM-BFF (Auto)' (w. 2.9B T5)
08.32.s/70.12.
67.0s.o
68.31.4
92.31.
14.014.1
73.92
76.22.
65.8
+ demonstration" (w. 2.99 T5) P-Tuning"
70.0g./72.00...
67.76.8
68.55.4
93.0n.a
21.815.9
71.16.8
78.15.4
67.3
6/
65.6s.0
64.32.8
92.2n.4
74.5
DART
67.52el
67.8s.2
66.7u.
93.5u.s
78.34.5
Methods with Augmentation: Few-shot samples are used for creating synthesized samples and for classifier tuning
MixTest
65.12./666.22 22.a
60.6s.9
68.4s.1
89.12.s
12.86.2
66.54.1
61.6%
61.1
Back Translation (w. trained Marian)
66.9c./68.3kr
59.8
67.84.
91.11.9
7.55.7
62.4s.5
68.011.2
60.6
GPT3Mix (w. 175B GPT3)
61.5
70.41.
69.2p.
93.6..
48.9:
70.410.0
69.9
69.2
Generator Fine Tuning (w. 1.6B CTRL)
/70.8..
60.4g.7
70.9..1
91.21.a
18.810.0
66.14.4
60.815.4
62.6
FewGen (w. 1.6B CTRL)
75.7../77..1.
71.5
76.3..
93.1a.
40.07.s
71.22.4
81.12.
72.8
Fully Supervised Fine- Tuning'
89.8/89.5
80.9
91.4
84.9
follow the same data split and evaluation protocol as Gao et al. (202T): Both Dusin and Daey contain 16 samples per label and are sampled from the original training set with different random seeds. The original development sets are used for testing. For all reported results, we include the average and standard deviation over the different Drsii//Deev splits. F1 score used as the metric for QQP and MRPC, Matthews correlation for CoLAA, and accuracy for the remaining tasks.
Models, Training Settings and Hyperparameters. FewGen is a training data generation method and can be used with any fine-tuning method on any classification model. We use moderate- sized PLMs to ensure our results are reproducible on typical research hardware: CTRL (1.6B parameters) (Keskar et al.) 2019) as the generator Go and RoBERTa (356M parameters) (Liu al. 2019) as the classifier Cz. We use prefix-tuning for training Gg and prompt-based fine-tuning for training Cg. For simplicity, we use the most basic manual prompt version LM-BFF (Gao et al. 2021). The only exception is CoLA for which we use the standard fine-tuning since the input data might be out of the distribution of C (Gao et al. 2021). The hyperparameter tuning is performed on Davv. More details are in Appendix
Compared Methods. No-augmentation baselines include zero-shot prompting, standard fine- tuning, in-context learning, and the following strong few-shot learning methods: Four versions of LM-BFF (Gao et al. 22021), P- T-Tunng (Liu et al., 2021b) and DART (Zhang et al. 2022). We also compare FewGen with data augmentation methods for few-shot learning: Mix (Chen et al. 2020). using back translation systems generate paraphrases (UDA-style (Xie al. 2020) augmentation), GPT3Mix (Yoo et al. 202T) and standard fine-tuning of generator on the few-shot samples with prompts. All augmentation methods use LM-BFF (Man.) for fine-tuning the RoBERTa classifier. More details about data augmentation baselines can be found in Appendix
5 EVALUATION
5.1
MAIN RESULTS
We present the results of FewGen and baselines in Table I FewGen achieves overall better perfor- mance across the GLUE tasks, on average 5+ points higher than the previous best few-shot method without augmentation, and 3+ points better than GPT3Mi Yoo et al. (202)) which uses 100 times larger generator model (175B) than FewGen. The promising results confirm the effectiveness of our
TThe CoLA results reported in the original GPT3Mix paper use accuracy as the metric instead of Matthews correlation; reimplemented GPT3Mix achieves 79.4p. on CoLAA if measured by accuracy.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Ablation studies by removing (-) switching (w.) one component of FewGen.
Method
(NLI-(m/mm)
QQP
QNLI
SST-2
CoLA
RTE
MRPC
FewGen
7
71.51.
76.34.4
93.1p.
40.07.5
71.22.4
81.12.
W. Laeen
71.9../76.21.
70.71.9
75.04.8
92.5p.7
37.8a.2
69.5222
80.81.0
w. Cgen + Ldise
74.61
68.82.1
76.14.3
92.4p.8
41.2g.0
70.12.2
79.62.4
temporal ensemble
3/.42./
65.82.1
75.12.7
92.11.7
33.94.
66.62.4
80.43.2
w. fine-tune on Dirin U pen
68.91.//7066..9
64.31.5
71.14.1
91.81.3
34.03.2
59.6
80.4g.5
proposed FewGen method in generating quality training data and leveraging them in combination with the few-shot training set for tine-tuning the classification model.
Comparison with Back Translation. Using back translation to paraphrase the few-shot samples does not improve the results, even with prompt-based fine-tuning to train the classifier this probably because it does not produce samples that are sufficiently different from the few-shot training set. The success of UDA (Xie et al.) 2020) is grounded in the augmentations from abundant unlabeled data that improve the classifier generalization. However, under the strict few-shot learning setup, there is no access to additional task task-specific unlabeled data (Gao et al. (021), making it challenging for paraphrase-basee few-shot set. The new training samples produced by our FewGen method are not limited to the paraphrases of the few-shot samples, as the generator is trained via prefix-tuning to preserve the PLM's pretraining knowledge, based on which novel training samples can be synthesized.
methods to create sufficiently diverse training samples only based the small
Comparison with GPT3Mix. The gigantic size of GPT3 makes it challenging for tuning on few- shot samples. Therefore, GPT3Mix (Yoo et al.) 202T) uses few-shot samples as demonstrations for creating the augmentations. Such an approach suffers from two limitations: (1) Without any parameter update to the PLM, its learning ability is not fully leveraged to adapt to the few-shot training set. (2) The PLM can only use a small subset of the few-shot samples at time for creating each augmentation. as the number of demonstrations received by the model bounded by its maximum input sequence length. This makes the quality of the created augmentations more sensitive to the randomly drawn training samples. Our FewGen method, on the other hand, can use the entire few-shot set for tuning the PLM and achieves overall even better classification results with a much smaller PLM (< 1% the size of the GPT3 model) which can be deployed much more easily in practice.
5.2 ABLATION STUDIES
We further analyze the effectiveness of each important component in FewGen. Specifically, we compare FewGen with the following ablations: (1) Using the standard Egen in Eq. (1) instead of our proposed Lw-gen in Eq. (3) for generator tuning (w. Leen); (2) using the directly combined Egen and Ldise for generator tuning (w. Lgea + Laise); (3) without applying temporal ensembling in Eq. (5) (- temporal ensemble); (4) directly fine-tuning the classification model on the combination of D gen and Dirain (w. fine-tune Durain U DD geen Fl As shown in Table 2 (1) & (2) using the standard maximum likelihood loss or the combination of generation and discrimination losses to tune the generator both yield lower-quality training data and lead to degraded classification performance; (3) not applying temporal ensembling for fine-tuning the classifier is more prone to label noise in the generated samples; (4) fine-tuning the classifier on the combination of DD Dgen and Durain significantly underperforms our two-step fine-tuming method. To study the impact of the amount of generated training samples on the model performance, we plot the MNLI-m accuracy (mean and standard
deviation) with different sizes of D in Fig. Both the average model performance and stability improve with more generated samples.
5.3 ANALYSES OF LOSS FUNCTIONS FOR GENERATOR TUNING
As shown in Table 2 the choice of generator loss has a significant impact on the synthesized data quality and thus the final model performance. We conduct further analyses to compare the training processes of the generator under the following three loss functions and the resulting generated
For this ablation, upsample Deran by x 100 so that its size is comparable with Deen without upsampling. the result is much worse.
8

---

Under review as a conference paper at ICLR 2023
77.5
3.0
4ym
Lpp Lyp Cas
75.0
Eygur + Ciiee
Die Sen Tree
2.5-
Lugs
Langs
22.5
2.6-
70.0
2.4
20 8.0 15.0
100
200
300
400
2.2
200
300
400
# Geeerted Troining Dota (xiel)
Thimingg Stopp
Thiining Stops
Figure 3: MNLI-m accuracy with different amounts of generated training data.
Figure 4: With different loss functions used for generator tuning, (Left) Lase and (Right) standard language modeling loss on the development set. Best viewed in color.
samples: (1) geen which is the standard language modeling loss; (2) Laeen + Laise which directly adds the discriminative loss to generator training; and (3) Law-gen which is our meta-weighted objective. Fig. 4 shows the discriminative loss Laise and the standard language modeling loss on the held-out development set throughout training. Although using Eeen + Lasec helps reduce the discriminative loss, it comes at the cost of hindering language modeling- the generator loss on the development set is high. Using meta- weightee objective Lw-gen for tuning the generator not only encourages discrininativeness but also mitigates overfitting, yielding the lowest validation set loss. This is probably because the model receives contrastive information from other labels which facilitates more accurate modeling of the texts with the target label. We present more quantitative analyses of different generator training objectives in Appendix G We visualize the token weights w automatically learned
and used in L. gen in Appendix F
DISCUSSIONS AND CONCLUSIONS
Ethical Considerations. Despite the impressive text generation and representation power of PLMs. they can also come with the risk (Bender al. 202 Bender Koller 2020; Brown et al. 2020) of
generating disinformation (Pagnoni et al. 2021) exacerbating biases (Prabhumoye al.; 2018), Instead of improving upon PLM architectures or generation techniques, our work focuses on using existing PLMs create training data for NLU tasks. Therefore, our method can be combined with
any bias reduction and correction strategies (Gehman et al., 2020 Ma et al., 2020 in practice to reduce the adverse effects of PL.Ms.
Linitations. Compared to few-shot learning methods that directly train classification models the small training set, FewGen requires tuning a generator PLM and using it to synthesize novel training samples, resulting in higher computation costs and longer running time. Still, believe that our method may bring more good than harm-when the small training data size becomes the performance bottleneck for NLU tasks. simple yet costly solution is obtain more human annotations. Our method may replace or reduce the human efforts in such training data creation processes.
Conclusions. In this work, we propose FewGen, which leverages few-shot training samples to tune generator PLM for synthesizing novel training data. The generated data be then used in combination with few-shot samples to fine-tune a classification model for better generalization. To emphasize diberiiicriiinaaivee information during generator tuning, we propose weighted maximum likelihood objective where the token weights are automatically learned via a discriminative meta objective. Since the generated samples may contain label noise, we propose simple training procedure that first trains classifiers on the few-shot training set and then on the generated set by applying temporal ensembling for noise-robustness. Across seven classification tasks from the GLUE benchmark, FewGen significantly outperforms existing approaches under the same few-shot learning setting. The effectiveness each important component in FewGen is validated via ablation studies. Future work directions may include: Using larger PLMs the generator and the classifier, jointly training both models with each other's high-confident predictions, and developing systematic metrics for evaluating the quality of generated training samples.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, N. Teppee, and Naama Zwerdling. Do have enough data? deep learning to the rescue! In AAAI, 2020.
Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning learn by gradient descent by gradient descent. In NIPS, 2016.
Eric Baum and David Haussler. What size net gives valid generalization? In NIPS, 1988.
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understand- ing in the age of data. In ACL, 2020.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Manl Major, and Shmargaret Shmitchell, On the dangers of stochastic parrots: Can language models too big? In ACM Conference on Fairness, Accountability, and Transparency, 2021.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurlPS, 2020.
Alvin Chan, Y. Ong, Bill Tuck Weng Pung, Aston Zhang, and Jie Fu. CoCon: A self-supervised approach for controlled text generation. In ICLR, 2021.
Jiaao Chen, Zichao Yang, and Diyi Yang, MixText: Linguisticall-innormee interpolation of hidden space for semi-supervised text classification. In ACL, 2020.
Kevin Clark, Minh-Thang Luong, Quoe Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, and Zhiyuan Liu. Prototypical verbalizer for prompt-based few-shot tuning. In ACL, 2022.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In ICLR, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
William Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP), 2005.
Chelsea Finn, Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In ICML, 2018.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. In ACI., 2021.
10

---

Under review as a conference paper at ICLR 2023
Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A. Smith. RealToxi- cityPrompts: Evaluating neural toxic degeneration in language models. In EMNLP Findings, 2020.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASAAL workshop on textual entailment and paraphrasing. 2007.
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In PASCAL Challenges Workshop on Recognising Textual Entailment, 2006.
Karen Hambardzumyan, H. Khachatrian, and Jonathan May. WARP: Word-level adversarial repro- gramming. In ACL, 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. In ICLR, 2021.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NeurlPS, 2018.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juan-Zi Li, and Maosong Sun. Knowledge- able prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. In ACL, 2022.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward controlled generation text. ICML, 2017.
Marcin Junczys-Dowe Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu T. Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André T. Martins, and Alexandra Birch. Marian: Fast neural machine translation in C++. In ACL System Demo, 2018.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL:
A conditional transformer language model for controllable generation. ArXiv, 2019.
Muhammad Khalifa, Hady ElSahar, and Mare Dymetman. A distributional approach to controlled text generation. In /CLR, 2021.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barre Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq R. Joty, Richard Socher, and Nazneen Rajani. GeDi: Generative discriminator guided sequence generation. In EMNLP, 2021.
Sachin Kumar, Eric Malmi, Aliaksei Severyn. and Yulia Tsvetkov. Controlled text generation as continuous optimization with multiple constraints. In NeurIPS, 2021.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained transformer models. In Workshop Life-long Learning for Spoken Language Systems, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi semi-supervised learning. In ICLR, 2017.
Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural data augmentation via example extrapolation. arXiv preprint arXiv:2102.01333 2021.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. In EMNLP, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACI., 2021.
11

---

Under review as a conference paper at ICLR 2023
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta. Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-ting controlled text generation with experts and anti-experts. In ACL, 2021a.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NeurIPS, 2022a.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out, 2022b.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. ArXiv, abs/2103. 2021b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Robert Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2 13353, 2021.
Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. PowerTransformer: Unsupervised controllable revision for biased language correction. In EMNLP, 2020.
Yu Meng, Chenyan Xiong, Payal Bajaj. Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. COCO-LM: Correcting and contrasting text sequences for language model pretraining. In NeurIPS, 2021.
Yu Meng. Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. In NewrIPS, 2022.
Sewon Min, Michael Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In ACL, 2022.
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. Adversarial training methods for semi- supervised text classification. In ICLR, 2017.
Due Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. SELF: Learning to filter noisy labels with self-ensembling, In ICLR, 2020.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metries. In NAACL, 2021.
Damian Pascual, Béni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer. plug-and- play method for controlled text generation. In EMNLP Findings, 2021.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In NeurIPS, 2021.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan Black. Style transfer through back-translation. In ACL, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2019.
Mengye Ren, Wenyuan Zeng. Binh Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
Teven Le Scao and Alexander M. Rush. How many data points is a prompt worth? In NAACL, 2021.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for fhw-shot text classification and natural language inference. In EACI. 2021a.
12

---

Under review as a conference paper at ICLR 2023
Timo Schick and Hinrich Schütze. Few-shot text generation with natural language instructions. In EMNLP, 2021b. 2021b.
Timo Schick and Hinrich Schützee Generating datasets with pretrained language models. In EMNLP, 2021c.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In NAACL, 2021d.
Iyer Shankar, Dandekar Nikhil, and Csernai Korél.
First
Quora dataset release:
Question pairs, 2017. First Sestt-uorr-Dataseet Rec--Releaan -asesett-eelleee-ueeeeeeeeee ceniiiio-Pairrs
URL. https:/ htapps///wwww www. quora............o..... www.://o/////.ooo http///quorraaaaa
Taylor Shin, Yasaman Razeghi, Robert I. Logan IV, Eric Wallace, and Sameer Singh. Eliciting knowledge from language models using automatically generated prompts. In EMNLP,
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Samping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. In NeurlPS, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher DD Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Trevor Scott Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In JCML, 2019.
Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. In EMNLP, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: stickier benchmark for purpose language understanding systems. In NeurlPS, 2019.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. NIPS, 2017.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. ArXiv, abs/2109.09199 2021.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. In TACL, 2019.
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2018.
Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Learning to teach with dynamic loss functions. In NeurIPS, 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh- Minh-Thang Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In NeurIPS, 2020.
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In NAACI, 2021.
13

---

Under review as a conference paper at ICLR 2023
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. G-daug: Generative data augmenta- tion for commonsense reasoning. In EMNLP Findings. 2020.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng. Zhiyong Wu, Tao Yu, and Lingpeng Kong, Zerogen: Efficient zero-shot learning via dataset generation. ArXiv, abs/2202.07922, 2022.
Kang Min Yoo, Do-Hyoung Park, Jaewoo Kang, Sang- Woo Lee, and Woomyeong Park. GPT3Mix: Leveraging large-scale language models for text augmentation. In EMNLP Findings, 2021.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In ICLR, 2022.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing [mask]: Learning vs. learning to recall. In NAACI., 2021.
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019.
14

---

Under review as a conference paper at ICLR 2023
A
DERIVATION OF META WEIGHT GRADIENT UPDATE
We first write out the gradient update of 0²²) (w)()) and (1+1) according to Algorithm
as follows:
(w()) = =00 WL gen
(0p: (())
(w(
@Cien()))
(6)
00,
80,
0 0,-0,
j=1
0x=0
_ (w))
(e+1) w(t)
(7)
dw
(
where a and are step sizes.
The gradient in Equation is calculated as:
Loioc
- (w)
Sw
(+)
ddix
(pp)
00p (w) dw
00,
0==05 6
(!w(!)
Laix
(8))
a (0p)
Juy (w) Ew
Plugging in Eq. 6
00,
0y=0/²
00,
0p-0
w=w(t)
i ddisc
(0²)
a. (0p)
Owy (w)
00p
a0,
dr
6, 06.0
0--0$
Ad
Therefore,
LLiic ((5)) (w))
BLisc (0,) dj
Ow, (w)
O. (0p)
α
dj
Ow
dw
00,
00,
(4)
00=00
B GLUE TASKS
We provide the details of the seven classification tasks included in the GLUE benchmark.
MNLI: Multi-genre Natural Language Inference (Williams al. 2018 requires predicting whether a given premise sentence entails, contradicts or neutral with respect to a given hypothesis sentence.
QQP: Quora Question Pairs (Shankar et al. 2017) requires judging whether pair of questions asked are semantically equivalent.
QNLI: Question Natural Language Inference requires predicting whether given sentence contains the answer to a given question sentence.
SST-2: Stanford Sentiment Treebank (Socher et al. 2013 requires determining if movie review has positive or negative sentiment.
CoLA: Corpus of Linguistic Acceptability Warstadt et al. given sentence is linguistically acceptable or not.
2019 requires determining whether
RTE: Recognizing Textual Entailment (Bentivogli et al. 2009 Dagan et al. 2005 Giampiccolo
ct al. 2007; Haim et al.) 2006) requires predicting whether given premise sentence entails a given hypothesis sentence not.
15

---

Under review as a conference paper at ICLR 2023
MRPC: Microsoft Research Paraphrase Corpus (Dolan & Brockett 2005) requires predicting whether two sentences are semantically equivalent or not.
C IMPLEMENTATION DETAILS
Table 3: Prompts used for initializing the prefix vectors and control codes (required by CTRL. CTRI.KKeskal (Keskar et al.] (2019) used in generator training. The control codes are selected to approximiate the domain of the task. For single-sequence tasks, a denotes the training sample; for sequence-pain tasks, ø and 22 denote the first and second sequence in the training sample, respectively.
Task
Task Type
Control Code
Label
Iuitialization Prompt
SST-2
single-soquence
Reviews
positive negative
Rating: 1.0 negative movic roview; a
Rating: 5.0 positive movie review;
grammatical
Linguistically correct sentence: xp
CoLA
single-scqucoee
Links
not grammatical
Linguistically incomeet sentence: a1
entailment
Scateace implics Sentence 2. Senteace 1: 2) Sentence aPg
MNLI sequence-pair
Wikipedia
neutral
Sentence 2 supplements Sentence 1. Sentence I: Sentence 2:
contradiction
Sentence contradicts Sentence Sentencee mi Sentence 22
entailment
Purugraph is relevant Question. Question: Paragraph: in
QNLI sequence pair
Links
not entailment
Puragraph is irrelevant to Question. Question: an Paragraph: 25)
entailment not entailment
Sentence implies Sentence Sentence z: Sentence 21g
RTE
sequence-pair pair
Wikipedia
Sentence 2 supplements Senience 1. Sentence 1: Senience 2:
MRPC sequence puir
Wikipedia
equivalent not equivalent
Scatence differeut from Sentence 2. Sentence I: 2:; Sentence 2: 20
Sentence equivalent to Sentence 2. Sentence I: at Sentence 2: at2
QQP
sequenoe-paic
Links
equivalent equivalent
Question equivalent to Question 2. Question I: an Question 2: zr Question different from Question Question 2) Question 202
Details of Initialization Prompts
Used for Generator Tuning on Different Tasks. For generator
tuning, we find it beneficial to initialize the prefix vectors with task-descriptive prompts, similar to the observations in Li & Liang (2021). The prefix lengths (i.e., number of trained prefix token positions) are equal to the number of tokens in the prompts. We present details about the prompts used for initializing the prefix vectors for different tasks in Table 3 For sequence-pair tasks, an additional infix prompt is used between the two sequences, and we also tune the embeddings of the infix (i.e., prompt-tuning (Lester et al. 202Th) for generator training.
Details of Generator Tuning. The weighted generator tuning procedure (Algorithm involves three forward and backward passes, and thus its time complexity is approximately 3 times of standard generator training without meta learning. However, since the few-shot training sets have a small amount of training data, the extra time cost is usually affordable. In practice, our generator tuning with meta weight learning takes 10 minutes to train on each task (the standard generator training time without meta- meta-learning is 3.5 minutes). We use fixed set of hyperparamters for all tasks without task-specific hyperparamter tuning: We set batch size to be 2, the learning rate for optimizing O, to be 5e 3, the learning for optimizing w be le 2, and training epoch be 20.
Details of Generating Training Data. Following Meng al. (2022), for sequence-pai tasks (MNLI, QQP, QNLI, RTE and MRPC), we randomly sample the first sequence from the pretraining corpus (e.g., Wikipedia) and use greedy sampling for generating the second sequence. For single- sequence tasks (SST-2 and CoLA), we use top-k sampling with temperature to generate training data from scratch where h = 10. For all tasks, we generate 5, 000 samples per label.
For SST-2, we use one of the following tokens to start generation: "a", "one", "the", "this", "that", "i", "you", "it", "what". For CoLA, we use a random stop word to start generation.
Hyperparameters for Fine- Fine-Tuning Classifier PLMs. For fine-tuning on the few-shot training samples Dirinn search among the following hyperparameter ranges based on development set (Daw) model performance and pick the best performing model for futher fine-tuning on synthesized data: Learning rate in [le 5, 2e 5] and batch size in [2, 4. 8). The number of training steps is fixed to be 1000. We also find beneficial to apply label smoothing (smoothing weight set to 0.15) for fine-tuning on the fho-shot training set.
16

---

Under review as a conference paper at ICLR 2023
For fine-tuning on the synthesized training samples D gea. the following hyperparameters: 5e - 6 as the learning rate; 16 as the batch size; label smoothing weight = 0.15 temporal ensemble momentum y 0.9; temporal ensemble loss weight À 20; training steps T 6, 000.
Details of Temporal Ensembling for Fine-Tuning Classifier PLMs on Synthetic Data. We update ensembled predictions & as follows where Ps is the current model prediction, 7 is the momentum parameter, is the accumulated model prediction before bias correction, is the accumulated model prediction after bias correction, and & is the number of updates has received:
ト マ + (1 - ?))pp
2/(1-11
The accumulated model prediction has zero initialization; the division (1- - >²) is for bias correction (Laine & Aila 2017). After each update of 2, it will be compared to threshold value 6; each synthesized sample (z, y) will be included in training only if 2g > 6.
We update the ensembled predictions & on all samples in D gen every 200 steps, and set the threshold value for sample filtering 0.8.
Computation Environment.
The experiments are conducted on NVIDIA A100 GPUs.
D DATA AUGMENTATION BASELINE DETAILS
Details About Mixtext (Chen et al. 2020). We use the TMix version of MixText to perform data interpolation the few-shot labeled dataset (since there is access to unlabeled task-specific data under the strict few-shot learning setting Gao et al. (022I). We adapt the label mix-up operation fit prompt-based fine-tuning by interpolating the label words instead of categorical labels; we observe that this results in better few-shot performance than the original TMix, probably analogous to why prompt-base based men-tunnn outperforms standard fine-tuning for few-shot learning. We train the classifier with supervised loss combined with consistency loss over the interpolated samples as in the original paper. We follow the default hyperparameters in MixText.
Details About Back Translation. We use two trained Marian Junczys-Dowmunt Dowmunt et al. 2018) models to perform data augmentation via back translation. We translate our labeled examples from English to French, and then back to English. As in UDA (Xie et al., 2020), we employ random sampling with a tunable temperature to generate a diverse set of derivative examples. We generate 32 examples from each few-shot training example and let the synthesized samples share the same label with the original few-shot training sample. After combining with the original examples, fine-tune the classifier and observe performance.
Details About GPT3Mix (Yoo et al. 2021). We use the 175B GPT3 model for generating the augmentations. For creating each augmentation, we randomly sample = 4 (the optimal setting according GPT3Mix) examples from the few-shot training follow the suggested format proposed in the original paper (Yoo et al.) 2021) and are shown in Table We create 5, 000 augmented samples per label make the resulting training set size equal to that of FewGen. After obtaining the augmented examples and their pseudo labels (the probability predictions over all labels by GPT3), we use them along with the real few-shot samples for fine-tuning the classifier, following the setting in GPT3Mix (Yoo et al. 202T).
demonstrations. The prompts
Details About Standard Generator Fine-Tuning. We fine-tune the same 1.6B CTRL (Keskar ct al. 2019) model as used in FewGen with the standard maximum likelihood objective. Different from previous studies (Anaby-Tavon et al. 2020 Kumar al.. 2020 that prepend categorical labels to the training samples, we enhance the generator fine-tuning with label-descriptive prompts (shown in Table 3) used in FewGen. We create 5, 000 augmented samples per label to make the resulting training set size equal to that of FewGen
E DETAILS OF WEIGHTING NETWORK IMPLEMENTATION
17

---

Under review as a conference paper at ICLR 2023
Table 4: Prompts used for GPT3Mix augmentation. For sequence-pair tasks, I1 and 22 denote the first and second input sequence, respectively. For single-sequence tasks, a denotes the input sequence. y denotes the label name. Only one example is shown in the template for clarity; in practice, we concatenate k- 4 samples according to the optimal setting in GPT3Mix (Yoo et al. 2021).
Task
Template
Label name
Each item the following list contains movie review and the respective sentiment. The sentiment is one of ppositive" or "negative'.
positive: positive negative: negative
SST-2
Movic review; a (Seatiment: g) ...
Each item the following list contains text and the respective grammar. The grammar one of "correct' 'incorrect.
grammatical: correct grammatical: incorrect
CoLA
Text: æ (Grammar: y)...
Each item in the following list contains premise, a hypothesis and their logical relation.
entailment: entailment
MNLI
The logical relation one of 'entailment" "neutral" or "contradiction.
neutral: neutral
Premise: Hypothesis: an (Logical relation: g)
contradiction: contradiction
Each item in the following list contains question, an answer and their logical relation. The logical relation is one of 'entailment' or 'neutral".
entailment: entailment not entailment: neutral
QNLI
Question: an Answer: at (Loggical relation: 1) .-.
Each item following contains a premise, a hypothesis and their logical relation. The logical relation is one of 'entailment' 'ncutral'.
entailment: entailment catailment: neutral
RTE
Premise: x;; Hypothesis: at (Logical relation: y) y)...
Fach item in the following list contains two sentences and their semantic relation. The semantic relation
equivalent: equivalent not equivalent: different
MRPC
of 'equivalent' 'different".
Sentence I: an; Sentence 2: mr (Semantic relation: y)
Each item in the following list contains two questions and their semantic relation. The semantic relation is one of 'equivalent' or 'different'.
equivalent: equivalent not equivalent: different
QQP
Question Question (Semantic relation: 3/) ...
Since the token weights w used in Eq. 4 need to characterize the discriminativennss of each to- ken, we use the value of discriminative objective at each token disc as the input to the weighting network, and we use softmax to normalize the weights:
Table 5: Study of weighting network instantiation. The default architecture is feedforward network (FFN) with one hidden layer. We also explore adding self-attention layer on top of the generator PLM's output hidden states (Self-attention). We use the same two metrics with Table to evaluate the resulting generators.
exp g (Case))
wy(w)
=1 exp Iw (ofince)
Architecture
MNLI
SST-2
Acc. (1)
PPL (1)
Acc. (1)
PPL (L)
FFN Self-attention
72.3
11.9
93.2
43.5
Following Shu et al. (019 we instantiate Iw
70.3
12.9
92.3
44.2
to be a feedforward network (FFN) with only
one 100-dimension hidden layer by default. We
explore an alternative instantiation that adds one self-attention layer on top of the generator PLMM' output hidden states. The meta weights are finally obtained by projecting the outputs of the self- attention layer using another linear layer. We evaluate the resulting generator quality via the same two metrics as in Appendix G Table Sable5]show shows that using more complicated architectures (e.g., adding another self- afffattention layer) does not result in a better generator compared to using a simple FFN for meta weight learning. This is probably because the generator PLM's output representations are sufficiently contextualized and contain the information necessary for learning the token weights, thus a simple FFN as the weighting network will be enough. Using more complicated networks, on the other hand, will introduce more randomly initialized new parameters which may not be learned well using the limited amount of few-shot training data.
F VISUALIZATION OF TOKEN WEIGHT LEARNING
To gain intuitive understanding of what tokens are assigned more weight during generator tuning, we visualize the learned weights in Fig. 5 The tokens with higher weights (c.g., "weak" in the first example and "hates" in the second example) are learned to be important tokens that decide the relation of the second sentence to the first sentence (i.e., the label of the training sample). With such tokens emphasized during training, the generator is encouraged to capture dabel-discriminative information that facilitates the generation of unambiguous training samples.
18

---

Under review as a conference paper at ICLR 2023
Sentence 1: But prophecy is always strongest when based on colncidence-tthat is prime rule.
Label: Contradiction
Sentence 2: Prophecles based on coincidences are widely known to be weak and unrellable, 0:63 0,00,.35 0.33
weights
0.03
Sentence 1: But Rodgers did tell Lewis that he despises Amelio because Amelio supported Clinton, so it is Rodgers' mistake, not our author's, that we correcting.
Label: Entailment
Sentence 2:
Rodgers told Lewis he hates Amelio.
weights
0.14
0.08
0.07
0.08
0.47
0.17
Figure 5: Visualization of learned token weights on two samples from MNLI's few-shot training set. The generator is trained given the first sentence to generate the second. The tokens associated with higher weights are more label indicative.
Table 6: Evaluation of generator training objectives. We use two metrics: Generated data accuracy (Ace; higher is better) and generator's 's perplexity on the test set (PPL; lower is better). The results are averaged over 5 D Dusin /Ddv splits.
Objective
MNLI Aoc. (f)
QOP
QNLI
SST-2
CoLA
MRPC
PPL (1)
Acc. (f) PPL (1) Acc. (1)
PPL (1)
Acc. (1)
PPL (1) Acc. (1) PPL (1) Acc. (1) PPL Aoc. (1) PPL (L)
Eyen
69.4
13.1
87.5
10.9
57.0
23.4
91.5
85.6
82.9
87.6
5.0
Eya
Lax
70.2
13.5
87.3
11.2
57.2
21.8
92.0
19.5
58.2
87.0
82.8
9.6
86.3
5.3
Caign
72.3
11.9
60.1
23.2
93.2
60.7
83.8
83.4
90.5
Table 7: (For MNLI) Examples of generated second sequence (hypothesis) by generators tuned with three different objectives conditioned on a given first sequence (premise) "In 2009, hair samples from 1.137 Taiji residents were tested for mercury by the National Institute for Minamata Disease (NIMD)". The true label of the generated sequence is marked at the end of the sequence (if the generated sequence correctly pertains to the target label, it is marked in blue; otherwise, it is in red).
Objective
Label
Generated Second Sequence
entailment
The NIMD found that hair samples from 1,137 Taiji residents had mercury in their hair. (neutral) The NIMD found that there was no evidence of link between exposure to high levels of mercury and thyroid cancer. (neutrul)
Cyss
neutral
contradiction
There was Do evidence of mercury in hair samples from Tajji (neutral)
entailment
The number of hairs in sample equal to the number of people who lived in Taiji. (neutrul)
Cyes + Lase
neutral
The results showed that there was no significunt difference in levels of mercury. (neutral)
contradiction
Hair samples from 1,137 Taiji residents were not tested. (contradiction)
entailment
The NIMD tested hair samples from 1,137 residents of Taiji. (entailment) significant difference levels between people who lived near nickel mine and those living far away. (neutrul) The NIMD did not test any of the hair samples, (conradiction)
There
Daggs
neutral
contradiction
G QUANTITATIVE ANALYSES OF GENERATOR TRAINING OBJECTIVES
Apart from the final classification model performance which indirectly reflects the synthetic data quality, we additionally conduct more direct quantitative analyses of different generator training objectives. We use two metries: (1) The accuracy of generated texts, which is judged by fully- supervised RoBERTa models fine-tuned on the original training sets of each task. We choose to adopt such an automatic evaluation instead of human evaluation because it is efficient and reliable- fully-supervised RoBERTa arge models have comparable or better accuracy than human baselines according the GLUE benchmark (2) The generator's 's perplexity on the test sets, which reflects how well the generator models the task distribution. As shown in Table 6 using Lw.gen for generator training consistently outperforms using Egen or Cgen + Ldises both in generated text accuracy and in language modeling ability.
Comparing Lw-gen with Lgen- the meta weights automatically learned emphasize discriminative tokens in generator training and help the generator capture the subtle semantic differences across different labels, resulting in better language modeling quality and more distinctive generated data.
Comparing Lw-gen with Lgen + Ldise. the generator training objective is not directly impacted by the discriminative objective, thus avoiding the gradient interference issue in multi-task learning (Standley et al. 2019) the gradient for optimizing the generative probability p( will be interfered by
https: / //
19

---

Under review as a conference paper at ICLR 2023
that optimizing the discriminative probability p(u x:) if geen + Laise is used. Therefore, using Lw-gen results in better language modeling quality and more fluent and coherent generation results.
We also showease concrete generation results for the three labels of MNLI by models trained with the three different loss functions in Table 7 The model trained with geen produces fluent and coherent sentences, but the generated sentences do not accurately pertain the desired label (i.e., the "enttilment" and "contradiction" generation results are in fact neutral with respect to the given sentence), lacking label discriminativeness. When Ggen + Ldsse is used, the generated samples of different labels are more distinctive, but also become less natural and coherent due to the model's language modeling ability being hampered. The generator tuned with Lu-gen produces both coherent and aubel-discimminativ samples which can serve as quality training data.
Table 8: 16-shot training samples of SST-2.
Label
Example
Review Text
(ramsay) visually transforms the dreary expanse of dead-ead distaste characters inhabit into poem art, music metaphor
#1
#2
the film jults the laughs from the audience by cattle prod.
the film presents visceral and dangerously bonest revelations about men and machines behind curtains our planet
#3
#1
film will enthrall whole family-
serious movie-goers emharking upon this journey will the road to perdition leads to a satisfying destination.
#5
#6
sweet and memorable film.
shyamalan takes a potemtilly trite and overused concept (aliens come to earth) and infuses into rustic, realistic, and allogether creepy tale hidden invasion. .
#7
crisp psychological drama (and) a fascinating little thriller that would have been perfect for an old twillght zone cpisode.
#8
positive
my big far greek wedding not only the best date movie of the year, it 's also a - dare i say it twice
#9
delightfully charming and totally american. i might add 1 slice of comedic bliss
comedy-dranc of nearly opic proportions rooted in a sincere performance by the title character undergoing
#10
midlife crisis.
#11
digas and lathan are among the chief reasons brown sugar is such sweet and sexy film. you 're not merely watching history, you 're engulfed by it. the concept hoot.
#12
#13
the tilmmakers eyy for detail and the high standards of performance convey strong sense of the
#144
girls eavireament.
#15
haunting tale of marder and maybem..
neil burger here succeeded immeddiate mystery in the present.
muking the mystery of four decades back the springbourd for
#16
#1
nothing happens, and it happens to flat characters.
#2
as lively an account as seinfeld is deadpan. ;
so we got ten little indians meets Iriday the 13th by way of clean and sober filmed on the set carpenter 's
#3
the thing and loaded with actors you 're most likely to find un the nexl inevitable incurnation of the love boat
the plot is nothing but boilerplate cliches from start to finish, , and the scripi ussumes that not only would subtiety be lost on the target audicnce. , but that it 's also too stopid to realize that they 've already soon this exact same movie a hundred times
#4
ultimately. sarah is dedication to linding ber husband seems more psychotic than romantic. and nothing in the movie makes convincing case that one woman 's beoken heart outweighs all the loss we witness..
#5
the big finish is a bit like getting all excited about a choculate eclair and then biting into it and finding the filling missing.
#6
this picture is mostly a lump of run run-of the mill profanity sprinkled with a few remarks so geared toward engendering audience sympathy that you might think he was running for affice or trying to win over probation officer.
#7
negative
just because walk to remember is shrewd enough activate girlish tear ducts does n'l mean it good enough for girls.
#8
#9
often lingers just long the irrelevant as the engaging , which gradually what time there this movie. certain in particular, brought me uncomfortably close tn losing my lunch.
#10
#11
but it would be hetter wait for the video.
rude black comedy about the catalytic effect holy fool has upon those around him in the curthroat world of children 's television.
#12
#13 just collection of this and that whatever fills time- with unified whole
although god great addresses interesting matters identity and beritage. 's hard shake the foeling intended to different kind film.
#11
#15
the chocolate factory without charlie.
#16
that setting. , their struggle simply too Iudicrous and borderline insulting.
20

---

Under review as a conference paper at ICLR 2023
H CONCRETE GENERATION RESULTS
We present few-shot training samples (Drrin) of SST-2 in Table 8 and some concrete generation results (from pen) for all tasks Tables 10
13
14 and 15
As shown in Tables 8 and | the synthetic samples are accurate and quite different from the given training samples serve effective augmentations. For sequence-pair tasks, because randomly sample the first sequence from the pretraining corpus and let the generator create the second sequence given certain labels, the resulting generated samples will be certainly different from the given training samples.
Table 9: Examples of concrete generations for SST-2.
Example
Label
Generated Text
positive
i was bit skeptical when i first saw the trailer for this movie, but after watching it, was quite enjoyable.
bogative
i was expecting a fun and entertaining movie with some nice special effects, hut instead was just plain horing.
film that has the potential you foel something.
audience
state of mind where
positive
#2
film that has potcutial
of the moving and powerful expericnces ever made
negutive
instead marred by unfocused script, bad acting and lack emetional depth.
positive negutive
this good movic it's not just for people who like to see things blow up or have explosions or shoot guns.
#3
am a big fan of the first two movics but this one is just plain bad.
Table 10: Examples of concrete generations for CoLA.
Example
Label
Generated Text
grammatical
Between the of think you have good chance to make the linals. few weeks later them television. We thought there hundred wasn't that they came
#1
not grammatical
time.
grammatical
Doing
allowed go out and play my friends.
#2
not grammatical
Only those lives depend on only when you are its merey do harm.
grammatical
To be able to speak the language of the other person is prerequisite for communication.
#3
not grammatical
The only good way to avoid getting into trouble for being too friendly would to let me trip.
Table 11:
Examples of concrete generations for QQP. The first question (italicized) is randomly sampled from the pretraining corpus; the second question (underlined) is generated by Ge
Example
Label
Generated Text
Question 1:
How long does it take for project to be completed?
equivalent
Question 2: How long does take to complete a project?
#1
Question 1: How long does it take for project to be completed?
not equivalent
Question 2: What is the total cost of completing this project?
Question What mascots or characters would you like to see included as Super Mario Maker DLC in the future?
equivalent
Question 2:
What would you like
in Super Mario Maker DLC that you did not in the game?
#2
Question I:
What mascots characters would you like to see included Super Mario Maker DLC in the future?
not equivalent
Question 2: How do I get a copy of this game?
21

---

Under review as a conference paper at ICLR 2023
Table 12: Examples of conerete generations for MNLI. The first sentence (italicized) is randomly sampled from the pretraining corpus; the second sentence (underlined) is generated by Ga.-
Example
Label
Generated Text
Sentence Air provided for combustion by an electric blower.
entailment
Sentence 2: The blower provides air to combustion chamber.
Sentence I: Air is provided for the combustion by electric blower.
#1
neutral
Sentence 2: Electric blowers are available in most gas stations.
Sentence 1: Air is provided for the combustion by an electric blower.
contradiction
Sentence 2: The blower does not work.
Sentence I: Since its base is almost at sea level, it is only the 15th highest light the United States, the first 14 being built on higher ground.
entailment
Sentence 2: It the 15th highest light in United States. Sentence 1: Since base almost sea level, it only the 15th highest light in the United Stutes, the first 14 being built on higher ground.
#22
neutral
Sentence 2: The lighthouse was originally constructed to he beacon for ships passing by and as such has been used since before World War II. Sentence Since base almost at sea level, it only 15th highest light the United States, the first 14 being built on higher ground.
contradiction
Sentence 2: is located mountain top.
Table 13: Examples of concrete generations for QNLI The question (italicized) is randomly sampled from the pretraining corpus; the answer (underlined) is generated by Go.-
Example
Label
Generated Text
Question: What makes you want to step up the next level?
entailment
Answer: want to the best player can be.
#1
Question: What makes you want to step up the next level?
entailment
Answer: new program
called "Project and it expected total
of $450 million in federal funding would toward it.
Question: How do all those shops know what you would like to buy?
entailment
Answer: The
able track your preferences and provide you with list of products that are best for you.
#2
Question: How do all those shops know what you would like buy?
not entailment
Answer: The stores are not required by law or regulation in the United States and Canada
but they have been known for years
illegal sales on eBay.
Table 14: Examples concrete generations for RTE. The first sentence (italicized) is randomly sampled from the pretraining corpus; the second sentence (underlined) is generated by Ge
Example
Label
Generated Text
Sentence 1: In 1970, the Science Council of Canada recommended that the government of Canada immediately invest in industrial development the aviation industry, including construction of aircraft. navigation aids, and regulation of air trfffc..
entailment
Sentence 2: The government of Canada has invested in the uviation industry.
#1
Sentence 1: In 1970, the Science Council of Canada recommended that the government of Canada immediately invest in industrial development of the aviation industry, including construction of aireraft, navigation oids, and regulation of oir trffie.
not entailment
Sentence 2: The Aviation Industry estublished by Royal Decree on June 1, 1970.
Sentence 1: All of the Centre's stoff are fluently bilingual in both Fnglish and Chinese and are familiar with the traditions of Chinase culture.
entuilment
Sentence 2: The Centre is a bilingual institution.
Sentence 1: All of the Centre's stoff are fluently bilingual in beth Fnglish and Chinese and are familiar with the traditions of the Chinese culture.
not entailment
Sentence 2: The Centre is cultural centre for learning about China.
22

---

Under review as a conference paper at ICLR 2023
Table 15: Examples of concrete generations for MRPC. The first sentence (italicized) is randomly sampled from the pretraining corpus; the second sentence (underlined) is generated by Ge,-
Example
Label
Generated TTxt
Sentence It embodies aesthetic ideals of the Queen Anne style through design and detailing and shows the type of house that attracted presperous Napans at the end of the Victorian ero.
entailment
Sentence 2: It is fine example of the Queen Anne style of architecture through its desigm and detailing and at the end
type house that attracted prosperous Napans Victorian era.
#1
Sentence I: Ii embodies the aesthetic ideals of the Queen Anne style through its design and detailing and shows the type of house that attracted prosperous Napuns the end Victorian
not entailment
Sentence The building is a fine example in style, with an elegant facude reminiscent to those found on many grand mansions built by wealthy merchants during
America's 's Gilded Age.
Sentence 1: Crosbie ran unsuccessfully for the leadership of the Liberal Party of Newfoundland and Labrador in 1969, losing Smallwood, and was also candidate in the Progressive Conservative Party Canada's 1983 leadership election. placing third.
entailment
Sentence Crosbie was candidate in the Progressive Conservative Party of Canada's s 1983
#2
leadership election, placing third.
Sentence Crosbie ran unsaccessfally for the leadership of the Liberal Party of Newfoundland and Labrador 1969, losing Smallwood, and was candidate the Progressive Conservative Party of Canada's 1983 leadership election, placing third. Sentence 2: He lost his bid as leader after be failed twice at running against John Diefenbaker.
not entailment
23


Reviews:
[{"review_id": "UEdaDwRyQ0", "reviewer": "Reviewer_gbEj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new augmentation strategy to augment few-shot samples during fine-tuning. The core idea of the paper is to use a pre-trained language model (PLM) to generate texts which can be leveraged towards fine-tuning.  The authors propose two methods in the paper to achieve this: (i) a method to fine-tune the PLM to generate appropriate texts; (ii) a technique to fine-tune the classifier using the generated texts.", "summary_of_the_review": "Overall, the paper is well-written, has satisfactory novelty and strong results. However, I am leaning towards weak reject as I feel that the authors did not compare with enough baselines where augmentation is used. Given the strong results, I will be willing to increase my scores if the authors can : (i) provide some motivation on why these 3 augmentation techniques were chosen to compare to or add some more baselines to Table 1.  (ii) Some quantitative measurement of the mitigation of label noise on the meta-objective. ", "strength_and_weaknesses": "Strengths:\n\n\t- The paper shows strong improvements over other baseline few-shot fine-tuning techniques (with / without use of data-augmentation).  The comparisons with data-augmentation techniques is more fair, as extra data is used during downstream fine-tuning.\n\n\t- The main technical novelty is in formulating the fine-tuning process of the generator with the meta-objective. Conditioning the generator on a weighted version of the input is a good way to generate label-coherent examples and the authors make the framework to work which is a strength. \n\nWeakness:\n\n\t- Could the authors add the ablation where D_train and D_gen is together used for fine-tuning?  I could not find it in the paper and it would be beneficial to add this to truly understand the effectiveness of the proposed fine-tuning strategy (with temporal ensembles) for the classifier tuning stage. In Table. (2), I do find -temporal ensemble, however I am assuming it is a two step fine-tuning - first on D_train and then on D_gen without the regularizer. \n\n\t- The compared baselines are not comprehensive. The authors should add more baselines where data-augmentation is used for few-shot learning. Some of the baseline techniques from https://aclanthology.org/2021.findings-acl.84.pdf can be investigated to strengthen the contribution of the paper. While a comprehensive analysis of all the augmentations is out of scope for the paper, a few strongly performing augmentation techniques can be added as baselines to Table 1.\n\nI would like to see some quantitative analysis on what proportion of label noise is mitigated with the use of meta-objective in training the generator? Downstream performance is definitely one indirect way to measure it, but having a quantitative analysis would strengthen the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U6ARHFPv57F", "reviewer": "Reviewer_TXcp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is exploring the few-shot capabilities for pretrained language models, by proposing a two-step method called FewGen: first to generate novel training samples from few-shot samples to augment the original set; and second to perform classification tasks by fine-tuning a classification-based language model on these generated samples. Additionally, the generation of novel samples is guided by a meta-objective which uses a weighting mechanism to choose the most important tokens w.r.t a label and produce more discriminative text samples. The model achieves better results on the GLUE benchmark, which consists of natural language understanding tasks, compared to existing few-shot methods, both with and without data augmentation steps.", "summary_of_the_review": "This paper is interesting and proposes to smartly combine existing techniques from data augmentation and meta-learning. However, meta-learning comes a bit of a sudden and is not clearly motivated, nor the learning strategy is thoroughly explained. The authors should try to define the link between the proposed techniques, namely, data augmentation through text generation and meta-learning the weights, to have a more consistent storyline. Also, it would also strengthen the paper if the authors consider the points and concerns raised in the “weakness” part.", "strength_and_weaknesses": "Strengths\n\n+The idea of combining an autoregressive language model for data augmentation and another classification language model that performs the task, is simple and novel to some extent.\n\n+All used modules and components are reasonable and sound, including the pre-trained models in the framework and the considered datasets/tasks. There is also some additional analysis and ablations included.\n\n+The empirical study shows that the proposed method FewGen outperforms other SOTA methods, with and without data augmentation on the GLUE benchmark.\n\n+The paper has a good flow, is well-written and understandable for the most part.\n\nWeaknesses (and questions):\n\n-There is a lack of experiments without the data augmentation done by the autoregressive language model. To have a better understanding of how the data generation part helps, the authors should provide a baseline where the model is fine-tuned without augmentation. This will offer a fair comparison to the models trained w/o augmentation in Table 1, and also can highlight the contribution of the classification-only part of the method.\n\n-The paper states that “general idea of meta-learning is to formulate a meta objective to enable automatic learning of hyperparameters”, which is not entirely true, since that can be considered only as a subset of the meta-learning goals. In general, meta-learning is aiming to learn good representations and accrue meta-knowledge by observing batches of few-shot tasks. Therefore, it would be good if the authors can elaborate more how their method achieves this, i.e. to explain thoroughly the learning strategy, which is only briefly mentioned to be the online optimization strategy.\n\n-The paper uses a learnable “w” as a feedforward net with 100-dimension hidden layer, but it is not explained why this is the only considered option. For instance, other options such as self-attention layers, do not seem to be considered. Since this part is important for learning the token weights in the meta-weighted objective function, the structure of this part can be further explored, for instance by providing more ablation analysis.\n\n-The paper is lacking more qualitative analysis and discussion. It would be nice to provide more generated samples for given k-shots and for different tasks, and to discuss how they contribute to a better performance.\n\n-It is not entirely clear how the shots are defined in the downstream tasks. Are they predefined in the scope of the datasets? This part can be more elaborated.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NwL8C_2c5jM", "reviewer": "Reviewer_KP32", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, authors propose a text-generation method to augment training data for various discriminative few-shot NLP tasks (from the Glue benchmark) such that the model performs better on these tasks as opposed to training (e.g. prompt-tuning/fine-tuning) using the original train split alone. Towards this end, authors propose i) a discriminative loss to ensure generated texts produce enough diversity to differentiate between labels and ii) a meta-weighted generative loss to take into account the impact of different tokens towards a label. They also propose an ensembling classifier to add more robustness to deal with generated samples. Experimental results on various tasks from the GLUE benchmark shows that the proposed method outperforms other methods from the literature.", "summary_of_the_review": "Overall, I found the paper to have a solid technical contribution to solve the task at hand which is to improve few-shot performance of various discriminative NLP tasks by learning to generate diverse synthetic samples. Experimental results and ablation studies showcase the efficacy of the algorithm. At this point, I am inclined to accept this paper based on my opinion alone. ", "strength_and_weaknesses": "**Strengths**\n* The paper is enjoyable to read and the algorithm has been developed nicely throughout the narratives. The authors clearly describe the problem they are intending to solve followed by motivating and describing the three major components of their algorithm - the discriminative loss, meta-weighted generative loss and the ensembled classifier. Authors also provide the end-to-end algorithms which help the reader to understand and possibly implement the proposed method. \n* Experimental results on the GLUE benchmark show clear and non-trivial improvement over existing baselines including the ones that rely on generating additional samples and those which do not.\n* Although the final proposed method is somewhat complicated, ablation studies on the importance of each of the component show relative improvement caused by individual components and the importance of all these pieces. \nOverall, I find it to be a strong paper with a non-trivial contribution, clear narrative and strong empirical results.\n\n**Weaknesses/Comments**\n* Qualitative analysis - Given that it is a generative paper, I'd like to see examples of how the generated texts look like for different labels. How different are those from the ones present in the training sample? On the same line, I'd like to see how well the meta-weight generation can learn the importance of various tokens. For example, authors provide an example that the token \"movie\" might not be relevant to generate texts for positive/negative movie reviews. Does such properties automatically emerge from the meta-learning procedure?\n* Susceptibility to hyper-parameter tuning - The optimization procedure is performing three interleaved gradient updates at every iteration. I'd imagine that such an optimization procedure will require sophisticated hyper-parameter tuning to obtain stable convergence. I'd like authors to comment on that along with the overall computational cost for this procedure.\n* The whole generator-discriminator paradigm is very similar to many works from the literature which are based on GANs. I'd like the authors to clarify in the paper how their optimization process varies differently from the GAN.\n* Generalizability of the ensembled classifier - Can the ensembled classifier generalize to other methods which also produce synthetic samples to augment training set? If so, how much gain can we tentatively expect to obtain with those methods? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0bnXcBcpgoQ", "reviewer": "Reviewer_1UgY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to augment label-discriminative data in PLM few-shot learning setting. The main idea is to improve the PLM-generator by introducing a label-discriminative loss in a meta-learning framework. The improved PLM-generator is then used to generate augmentation data to finetune a classification PLM(Roberta-large). The paper compares the proposed methods with several few-shot baselines on seven classification tasks of GLUE.\n", "summary_of_the_review": "Overall, the paper is well-written with a clear flow. However, (1) the intuition that formulates label-discriminative text generation into a bi-level problem is not clear, as there is no strict hierarchy of the two objectives. No convincing intuitive explanation or theoretical justification is given to explain the necessity and superiority of such a meta-framework in this paper. (2) Some important baselines are missing, and I remain skeptical about the quality improvement of generated data and performance gain of the proposed method.\n", "strength_and_weaknesses": "Strength:\n1. The overall framework to investigate the few-shot learning ability of PLM  is somehow novel.\n2. It is interesting to design the $L_{disc}$ to encourage the PLM-generator to synthesize more label-discriminative augmentation examples.\n3. The paper is well-written with clear logic. Most part of the paper is easy to understand.\n\n\nWeakness:\n1. The intuition that formulates label-discriminative text generation into a bi-level problem is not clear. What is the interpretation behind it? It seems that the bi-level structure is not necessary in this case, since there is no strict hierarchy of the inner and outer objectives ($L_\\text{gen}$ and $L_\\text{disc}$): the order of inner and outer objectives can be simply swapped to achieve the same goal(i.e. using unweighted generation loss in the outer loop and weighted discriminative loss in the inner loop). The bi-level form here is more like to introduce “technical contribution”.\n2. The two reason for not directly combining $L_\\text{disc}$ and $L_\\text{gen}$ is not well justified ( \"the generation-irrelevant loss  $L_\\text{disc}$  will unavoidably interfere with the language modeling process” and “A hyperparameter needs to be introduced to balance the weights of the two losses ”.) I do not see why the bi-level formulation helps solve these issues. Specifically, suppose the token weights are learned to minimize the discriminative loss, how can it guarantee such weights will not affect the language modeling process? In fact, even without direct intervention, the $L_\\text{disc}$ will affect $w$ and therefore affect the optimization of $L_\\text{gen}$. Besides, it seems that introducing a hyperparameter to balance  $L_\\text{disc}$  and $L_\\text{gen}$ is acceptable, as the author introduces such a hyperparameter in Eqn. 5 to balance the two loss term for finetuning, which is actually in a similar situation.\n3. I remain skeptical about the performance gain of the proposed method. Firstly, the authors do not compare with the important baseline that directly uses few shots in the demonstration(as in-context examples) to generate augmentation data, and then use the labeled few shots and synthetic data to fine-tune a PLM classifier. I find that the cited paper (Meng et al., 2022) uses the same PLM generator and classifier, as well as a much restricted zero-shot setting, which can achieve comparable performance with FewGen in some task. It looks like the performance gain (if any) is quite marginal, given that few-shot should already introduce some performance boost compared with zero-shot. Besides, I am curious about to which extent more label-discriminative data can improve the moderate-size PLM classifier. Table2 should include all the glue tasks. Complete results of the important baseline(w. $L_\\text{gen}$) can help justify which margin of improvement can the proposed method achieve. \n4. The paper does not conduct quantitative and qualitative analysis to measure the quality of the generated text. Can the improved generator really generate more label-discriminative text? Some automatic metrics/human evaluation should be introduced to measure the quality of the generated data. \n5. Usually, the bi-level formulations used in previous work present justifications of why such formulation is adopted, such as meta weight net, MAML, etc. However, no theoretical justification or intuitive explanation is given to explain the necessity and superiority of such a formulation in this paper.\n6. The performance bottleneck of the proposed method is quite obvious, since it is not applicable to large PLMs, and moderate PLMs(e.g. 1.6B Crtl) usually do not have good generation ability. From the experimental details, it is quite concerning that for the experiments using Ctrl as PLM-generator, the batch size can be only set to 2 on A100 GPUs. This brings the usability of this method on larger PLM into question. Besides, it is well known that meta-learning methods are notoriously hard to tune and can be extremely sensitive to hyperparameter choices. It is hard to tune the framework on a large PLM with a small batch size.  In addition, the updating process of the generator looks time-consuming, as the gradient calculation for prefix tuning needs to backward pass all the gradients through the PLM’s parameters in each iteration.\n7. The authors only conduct experiments on 2-class or 3-class NLU classification tasks. It is questionable whether the proposed method is suitable for other tasks(e.g., multi-class classification and QA tasks). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject