PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation
Authors: Gaurav Sahu
Olga Vechtomova
Dzmitry Bahdanau
Issam H. Laradji
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
knowledge distillation
text classification
few-shot learning
TL;DR: Generating borderline examples and, crucially, relabelling them using a large language model like GPT3.5 effectively transfers the knowledge of massive GPT3.5 into much smaller models like DistilBERT and BERT for few-shot text classification.

Abstract: Data augmentation is a widely used technique to address the problem of text classification when there is a limited amount of training data. Recent work often tackles this problem using large language models (LLMs) like GPT3 that can generate new examples given already available ones. In this work, we propose a method to generate more helpful augmented data by utilizing the LLM's abilities to follow instructions and perform few-shot classifications.  Our specific PromptMix method consists of two steps: 1) generate challenging text augmentations near class boundaries; however, generating borderline examples increases the risk of false positives in the dataset, so we 2) relabel the text augmentations using a prompting-based LLM classifier to enhance the correctness of labels in the generated data. We evaluate the proposed method in challenging 2-shot and zero-shot settings on four text classification datasets: Banking77, TREC6, Subjectivity (SUBJ), and Twitter Complaints. Our experiments show that generating and, crucially, relabeling borderline examples facilitates the transfer of knowledge of a massive LLM like GPT3.5-turbo into smaller and cheaper classifiers like DistilBERT-base and BERT-base. Furthermore, 2-shot PromptMix outperforms multiple 5-shot data augmentation methods on the four datasets. Our code is available at https://github.com/ServiceNow/PromptMix-EMNLP-2023.

PromptMix: A Class Boundary Augmentation Method for Large Language
Model Distillation
Gaurav Sahu University of Waterloo gsahu@uwaterooo.cm ca
Olga Vechtomova University of Waterloo
Dzmitry Bahdanau
Issam H. Laradji ServiceNow Research
ServiceNow Research Mila, McGill University
Canada CIFAR AI Chair
Abstract
Data augmentation is widely used technique address the problem of text classification when there is limited amount of training data. Recent work often tackles this problem using large language models (LLMs) like GPT3 that can generate new examples given already avail- able ones. In this work, we propose method to generate more helpful augmented data by uti- lizing LLM's abilities to follow instructions and perform few-shot classifications. Our spe- cific PromptMix method consists of two steps: I) generate challenging text augmentations near class boundaries; however, generating border- line examples increases the risk false pos- itives in the dataset, so we 2) relabel the text augmentations using prompting-based L.IM classifier to enhance the correctness labels in the generated data. We evaluate the proposed method in challenging 2-shot and zero-shot set- tings on four text classification datasets: Bank- ing77, TREC6, Subjectivity (SUBJ), and Twit- ter Complaints. Our experiments show that gen- erating and, crucially, relabeling borderline ex- amples facilitates the transfer of knowledge of a massive L.LM like GPT3.5-turbo into smaller and cheaper classifiers like DistilBERT M..e and BERT base. Furthermore, 2-shot Prompt- Mix outperforms multiple 5-shot data aug- mentation methods the four datasets. Our code is available at https: /twpbbb..ttttt.///// com/ ServiceeNo//PrrmppMMMi---EEE
prosipie wib
LIA
PhangeMi
Figure PromptMix focuses on generating exam- ples near the class boundary of two classes, unlike other standard augmentation approaches like EDA (Wei and Zou, 2019) and prompting-based methods without Mixup (Sahu et al., 2022; Lin al., 2023), that only use the information of a single class for augmentation.
Rashtchian et al., 2010; Rajpurkar et al., 2016; Khot et al., 201 20.8). With the recent surge in the development of generative large language models (LLMs) (Brown et al., 2020; Chowdhery al., 2022: Zhang et al., 2022: Touvron et al., 2023). a large body of literature has emerged that em- ploys LLMs to generate additional data for vari- ous tasks (Kumar et al., 2020; Schick and Schütze, 2021; Wang et al., 2023).
this work. we focus on the task of few-shot text classification (Schick and Schütze, 2021; Alex et al., 2021; Bragg et al., 2021). Specifically, we explore zero-shot and 2-shot settings. Early works employing LLMs to generate additional data sam- ples for text classification first fine-tune genera- tive language model on an initial seed dataset and then use it to synthesize new training data (Wu et al., 2019; Kumar al., 2019, 2020; Anaby-Tavar et al., 2020); however, the fine-tuning step quickly becomes a bottleneck in the absence of sufficient
1 Introduction
Data scarcity is key challenge in numerous real- life scenarios, such deploying intent detection systems in task-oriented conversational agents and identifying hateful instances of speech on social media platforms. Crowdsoureing has been a tradi- tionally popular choice to obtain additional data, but it a financially expensive procedure incurring a high cost of human labor (Sheng et al., 2008;

---

1pt
Class limit
class age, limit
aceage dtapa for
tank Some
Step.li Gesende
Step 2: Relaist
close: ago Jint
25 sepport is abour uses asking low an ATM wher to fird one, uraryy oller chaifiations abous numsccion an ATM, Sonee ouarplo.
cf
LLM,
LLM,
Craap bur. accat aa XTM?
support
Generate ulkeaano nge,
sqppon
suppont
class suppoct
Smal Fasmples
Asquented Datasii
Figure 2: PromptMix framework. The dashed box shows the generation process for every class in the dataset.
Step I: we generate augmentations (yellow documents) by feeding the Mixup prompt proposed in Section 3. to an LLM. Step 2: relabel all the augmentations using LLM (as deseribed in Section 3.2) fix any incorrect labels from Step I. Note: LLL, and L.MM2 can be identical. Refer to Figures and 4 for detailed versions of our prompts.
seed examples. More recent works sidestep fine- tuning by designing natural language prompts for off-the-shelf LLMs (Yoo et al., 2021; Sahu et al., 2022; Lin et al., 2023). A key limitation of such works that their prompts only focus on using information from single class when generating augmentations. Additionally, they do not incen- tivize the LLM to diversify the generated exam- ples, and recent works show that instruction-tuned LLMs like InstructGPT (Ouyang et al., 2022) and Vicuna (Chiang et al., 2023) are prone to mode collapse (Zhu et al., 2023).
set of labeled examples for any text classifica- tion dataset; and b) we demonstrate that gener- ating borderline examples and relabeling them im- proves knowledge transfer from massive LLM like GPT3.5 into much smaller models like Dis- tiBBERT and BERT, even without abundant seed examples. We also show that 2-shot PromptMix outperforms multiple data augmentation baselines that use 5-shot or more seed data.
2 Related work
Our work intersects with the topics of data aug- mentation, few-shot classification, and knowledge distillation, which we explain in detail below.
To address the previous limitations, we propose a two-step prompting- technique, PromptMix. First, PromptMix instructs an LLM (in our case, GPT3.5-turbo') to generate new examples by mix- ing information from multiple classes. The degree of mixing is controlled by a parameter a, and us- ing a range of values diversifies the generated examples; however, promoting mixup increases the risk of false positive generations, PromptMix uses relabelling scheme in the second step to im- prove the faithfulness of the generated examples. particular, it uses an LLM as classifier to assign new labels to the generated examples. We find that training a classifier on these relabeled examples effectively transfers the knowledge of a massive LLM like GPT3.5 into much smaller models like BERT (Devlin et al., 2019) and DistilBERT (Sanh al., 2019). Figure demonstrates the complete PromptMix framework.
2.1 LLM-based Data Augmentation for Few-shot Text Classification
Kumar et al. (2019) evaluate different feature space data augmentation techniques, such as upsampling, perturbation, and linear interpolation, for vary- ing levels of data scarcity (ranging from 5% data availability to 20%); however, their performance gains are minor compared to a no-augmentation set- ting. Kumar al. (2020) consider 10-shot, 50-shot, and 100-shot text classification setups where they fine-tune pretrained language models like BERT, BART (Lewis et al., 2020), and GPT-2 (Radford et al., 2019) as data generators on k-shot data. Next, they condition the fine-tuned data generators to synthesize new training examples for individ- ual classes in the dataset. This method improves over classical data augmentation techniques, such as easy data augmentation (Wei and Zou, 2019) and back translation (Sennrich et al., 2016), but the experiments were performed on datasets with very
We summarize our contributions as follows: a) we propose PromptMix, a novel two-step prompting-based method to generate diverse

---

few classes (up seven). In reality, classification setups can have hundreds of classes, and the initial fine-tuning step would become bottleneck.
effectively distilled into much smaller student model, allowing a similar level of performance as the teacher at lower computational cost. Shrid- har et al. (2022) distill a GPT3 (6B) model into a GPT-2 model for a Chain-of-thought (CoT) reason- ing task. Liang et al. (2021) propose MixKD to encourage the student model to mimic the teacher's behavior on not only the available training exam- ples but also on interpolated examples. Finally, Sun et al. (2020) distill knowledge through intermediate layers of the teacher via contrastive objective. comparison, our approach allows knowledge dis- tillation of a massive teacher model like GPT3.5 (175B parameters) into significantly smaller stu- dent models like DistilBERT and BERT (67M and 110M parameters, respectively).
To alleviate the fine-tuning bottleneck, Yoo et al. (2021) use natural language prompts for data aug- mentation; however, they also experiment with less challenging classification setups (with fewer classes). Sahu ct al. (2022) propose a prompting- based approach using off-the-shelf GPT-3, to gener- ate large labeled corpus of augmented data. Their method improves across multiple classification se- tups with a large number of classes (up to 150) and varying levels of granularity; however, their method struggles on tasks where the classes carry very close semantic meanings. In particular, the generated samples have incorrect labels. In our method, we use a language model as a classifier to improve the label accuracy of the generated dataset.
Methodology
Some non-prompting brapptiin----edddd approaches include Wei et al. (2021), who propose curriculum data augmentation, where they first train a model on limited s-shot data and then incrementally intro- duce augmented data as training progresses. They use triplet loss (Schroff et al., 2015), which min- imizes the distance between data points with the same label and maximizes for differently labeled examples. Tunstall al. (2022) propose SetFit that first fine-tunes sentence transformers on a small number of text pairs in a contrastive Siamese man- ner (Dong and Shen, 2018) and then generate rich text embeddings for training a classification head. Finally, Kim et al. (2021) propose LINDA, which is first trained on one million sentence pairs randomly drawn from English Wikipedia. Later, they the Mixup algorithm (Zhang et al., 2018) interpolate between two English sentences of varying lengths. LINDA significantly improves the performance of a BERT base model across multiple few-shot clas- sification setups. In our work, we consider more extreme few-shot setups (2-shot, zero-shot) and per- form more controlled interpolation that promotes the generation of examples near class boundaries.
We hypothesize that training a robust text classifier requires the training data to have a good mix of borderline examples (Swayamdipta et al., 2020). This section describes method PromptMix. where first instruct LLM (GPT3.5-turbo, in our case) to generate difficult examples near class boundaries then relabel those generations to im- prove the faithfulness of the generated data (see Figure 2).
3.1 Step 1: Generating examples
First, we manually write short descriptions for ev- ery class in the dataset. We use descriptions our prompts to facilitate the usage of the approach in the extremely data-scarce zero-shot and two- shot settings. Next, we randomly select a group of t(= 4) classes C C classes, where C denotes the of all classes in the dataset. For each class in c, we combine the description and k examples in the prompt, & being the shot setting (see part in Figure 3). Lastly, for each class c,Vi e [1, t) in the subset, we instruct GPT3.5-turbe to generate example utterances that are mix of two classes: C and randomly selected class Cj E Ecl {ci). In particular, we instruct the LLM to generate utter- ances that belong a% to class C2 and (1 a)% to class ej (see part 2 in Figure 3). Figure 5 in Appedix A.l shows the distribution sampled from.
2.2 Knowledge Distillation
Knowledge distillation (Bucila et al., 2006; Hinton et al., 2015; West et al., 2022) refers training smaller student model mimicking the behavior of much larger teacher model. In particular, the objec- tive function aims to match the output distribution of the student model to that of the teacher model. By doing so, knowledge of the teacher model is
"e choose based on the results in Section A.2 "specifically, we use the gpt-3.5-turbe 5.5-trrbo-06. -6613 engine

---

Prometi
Input Proupt:
Considez the task ccansitying becveen the folloving
Consider the task of classitying betxson the collowing classes Islong with zone exemples);
celoses with cono exarples):
age,
aage_littt limit,
sbout
inquinies
1. age limit, which is about custonez inquiriea on age-zelaced restrictions fox opering bank account. Sone exemples of utcerences include: Can I get sccount for spn?
age-related restri onstrittions for opening bark account,
Sore exengles of utterances include: ny Can by Lasnages have an accounl?
get
Can my teenagen have an ausourt?
pipport, use an ATN, dlace to find one, or any other clarifinations about transaction exanples utterances incleda:
"s users neting
2. etn, support, which is about users anking how to an where
en AIM, Sone
one, or any other
slari fications about transaction at an ATM, Some Include:
Ia the closeat XIM to ze within 2 milea?
exanples of
Are there only coxtain MM nachines whaze I can
Is the clorest ATM to me within 2 miler?
Are there only certain machines where use this card?
Generate diverse each utcerance zelongs 756 to ago lini: and 255 to support.
unterences
Comsider the rowling test sentencee 1. Do need be over certain age ATNP
Example 1:
Classity the test sentence into of the previously sescribed classes.
Cospletions:
Can scncone undez 18 open on account with unlinited
1.
Can open an account for ny tzenaçe daughtor?
Example 1:
Do need to be over cextain age to use MI to
my fron the NTM?
roney
Completions:
support
Figure 3: PromptMix prompt. The demonstration highlights the two main parts of our prompt: Part I shows the description and examples, and Part 2 shows the mixup instruction. We use GPT3.5-ture -ttrbo to obtain the completions. generations in blue and bad generations in red. Note: for brevity, we only include two classes in the prompt.
Figure 4: Relabelling prompt. The generated sentence does not belong to the class age age_limit as the main objective of the query entails an ATM support query, which is fixed after relabeling. Note: relabel both good and bad examples. The color in the figure is just demonstration purposes.
this example, we highlight good
3.2 Step 2:
Improving Fidelity
not include all the classes in the prompt because a) some datasets can have hundreds of classes that would not fit in the context size of the LLM, and b) we found in our preliminary experiments that long contexts degraded GPT's classification ability. Figure 4 shows the prompt structure for relabeling a generated example.
Since borderline examples are inherently difficult put into category, the LLM may generate ex- amples for the minority class in the prompt (cz). In other words, the LLM can generate false positives. To address this issue, we employ GPT3.5-turbe as classifier and relabel the generated examples.
When constructing the classification prompt, we choose the top-5 closest classes according to the similarity between the SBERT sentence embed- ding (Reimers and Gurevych, 2019) of the gener- ated example and available examples in the few- shot training set. For the zero-shot setting, we use the class name's embedding instead of the available examples. We then follow a similar process as in Step to construct the prompt, but instead ask the LLM to classify the provided sentence into one of the classes in the prompt (see Figure 4). To ensure a valid prediction, we retrieve the closest class in the dataset based the cosine similarity of the SBERT embedding of the GPT-generated class and the ground- truth classes in the dataset 4. We do
After generating borderline examples and re- labeling them, we train a text classifier on the combined PromptMix-generaat original seed data. Specifically, we fine-tune DistilBERT base and BERT base classification mod- els. In the subsequent sections, we show that our method achieves text augmentation, pseudo- labeling, and knowledge distillation in one cohe- sive pipeline.
data and the
Does PromptMix Generate Borderline Examples?
Table shows that using mixup generates sentences that contain information from the majority and the minority class, compared sentences generated without mixup that only contain information about the majority class. This demonstrates that mixup
"we use the sentence- ttann wwteeooeeereaaallllaeeeeaaeaaeee saaess/allllmane 1el-mmnee-bbsseeee model from the transfoert lihrary

---

Input
age__liit
(75%)
support (25%))
B77
TREC6
SUBJ
TC
Can
under
open
with
Classes
77
2
unlimited ATM withirrwal limit")
# Train
9002*
5452
8000
100
Can
open
uecount
for
# Valid
1001
500
2000
w/ Mixup
my lcenage doughter
# Test
3080
500
2000
3349*
Do
need
to be over a certain age
ATM?
Table 2: Statistics of the text classification datasets we in experiments. indicates that we split the original data into training and validation/testing instead of using a split provided by the dataset authors.
Can
use my children's 's debit card to
withdraw from the ATM?
open account
What the minimum age requirement bank account?
Mixup
there an holders?
age restriction for account
Is there
minimum ago roquirement to open account?
bed for verifying the scalability and effectiveness of our approach. The following three datasets have coarse labels but cover a variety of domains, al- lowing us to test the adaptability of our method across different domains. TREC6 (Voorhees et al., 1999) is a question classification datasct with six broad classes of questions in English. The subjec- tivity dataset (SUBJ) (Pang and Lee, 2004) con- tains movie reviews with objectivity labels. Lastly, the twitter complaints dataset (TC) (Preoçiuc-Piettro et al., 2019) contains tweets annotated by whether they contain a complaint or not. We refer the reader to Table 2 for exact statistics of all the datasets.
Table I: Effect of Mixup. GPT..5-turbo generations for the prompt shown in Figure We' ve highlighted parts about age limit in yellow and parts about atm, _ in cyan We see clear evidence of mixup work as sentences generated using mixup contain information about both classes.
leads to the generation of borderline examples. Table also shows some false positives, where GPT generates sentences like, "Do need to be over certain age to use an ATM?" for the major- ity class age_limiie limit. The class lim covers all age-related queries about opening bank accouni, whereas the generated sentence is age-related query about using an ATM, making it better fit for the minority class atm_ suppus. By relabelling such false positives, we aim to rectify the mismatch between the generated examples and their assigned class labels. Figure shows that GPT correctly pre- dicts the new class of the sentence as atm_aammmaaaaa suppoot, verifying the importance of the relabeling step in approach.
5.2 Few-shot Setup
For our experiments, we consider 1) a 2-shot setup, where only k = 2 training examples are available for every class, and 2) zero-shot setup, where we do not have access any training examples.
Notations. We will use Dport to refer to the dataset parts, i.e., train, validation, and test. When augmenting the training data using any method, we generate N examples per class and refer to the resulting data as D A,train (obtained after Step I). We refer to the relabeled version of the resulting data (obtained after Step 2) as D A-
Overall, we find the observations in this section to be strong evidence that PromptMix can generate high-quality datasets even aggressive few-shot text classification setups. Next, we conduct an extensive suite of experiments to verify the effec- tiveness of PromptMix.
5.3 Training and Evaluation
Training. We fine-tune DistilBERT base and BERT buse models for text classification by adding a linear layer on top the [CLS] token (Wolf al., 2019). In all the setups, we fine-tune the classifier for 5 epochs. We use a learning rate of 6 10 and weight decay of 1 x 10-3 for B77 and a learning
5 Experimental Setup
5.1
Datasets
We use four text classification datasets with varying levels of granularity among the classes. Banking77 (B77) (Casanueva et al., 2020) is single-domain dataset with 77 banking-related classes, where the difference between multiple classes is nuanced. The fine- ane-grained nature combined with a high num- ber of target classes makes Banking77 a good test
rate of 4x 10 5 and weight decay of x 10 2 for all the other datascts. Finally, generate N = 50 amples per class for B77 and TREC6 and N 100 examples per class for SUBJ and TC. We used the validation sets of TREC6 and SUBJ to choose the specific N values as they provide a good cost-to- performance ratio. We chose TREC6 over B77 to

---

minimize our costs for using GPT3.5-turbo.
on 654 million Reddit sentences.
Several works in the past have explored 5-shot, 10-shot, and even 100-shot settings for data aug- mentation (Kumar et al., 2020; Zhang et al., 2021), But, to the best of our knowledge, we are the first to explore data augmentation in 2-shot and zero- shot settings. Through our experiments, we aim to measure the extent of data scarcity that an LLM like EPT3.5-turbo can handle, which is optimized to follow instructions.
We perform all hyperparameter tuning using DistilBERT base on B77 and TREC6. We limit our tuning to the two datasets to obtain two sets of hy- perparameters: one for large-scale datasets like B77 and another for small-scale datasets like TREC6, SUBJ, and TC. Additionally, we use the same set of hyperparameters for the BERT base classification model. We use the full validation set for tuning instead of a few-shot one to avoid issues with un- stable hyperparameters.
Evaluation. We measure the performance of the classifiers in terms of test classification accuracy and report the full set of results in Table 3. To note, A1 denotes the accuracy of the classifier on Dtrain U DA.train (augmented dataset after Step I in Section 3.1) and A2 denotes the ac- curacy of the classifier on Dtrain U DA R,train (augmented--rclabeled dataset after Step 2 in Sec- tion 3.2). We run each experiment for three random seeds and report the mean accuracy in Table 3.
We run experiments for the following scenar- ios: 1) Baseline (2-shot). All the classes are re- duced 2 examples per class, and we fine-tune a DistilBERT base /BER/ model on the reduced dataset. 2) NN+GPT3.5. We use the nearest- neighbor approach to populate the prompt as de- scribed in Section 3.2 and then prompt GPT3.5- turbo to classify the test set examples (see Fig- ure 4 for reference). 3) Sahu et al. (2022). A prompting-based approach for data augmentation that lists all the seed examples for a single class in the prompt and prompts the LLM generate more examples based on it. does not promote mixup in the generated examples. 4) PromptMix. Our proposed approach with multiple classes in the prompt, instructing the LLM to generate borderline examples. 5) PromptMix (zero-shot). We remove seed examples from PromptMix but still use the manually written class descriptions. 6) Easy Data Augmentation (EDA). An edit-based augmenta- tion technique proposed by Wei and Zou (2019) that applies rule-based changes to existing train- ing examples to generate additional examples. 7) GPT3Mix. A mixup-based augmentation method using soft labels for pseudolaheling proposed by Yoo et al. (2021). Notably, Yoo et al. (2021) mea- sure the degree of few-shot in terms of the percent- age of training examples used for augmentation (sub-sample percentage) and experiment with dif- ferent percentages. We report their rest-performing model for a sub-sample percentage of 1,, where GPT3Mix uses 55 training examples in TREC6 and 80 training examples in SUBJ (combined for all the classes). 8) CPFT. Zhang et al. (2021) use contrastive pre- pre-training before fine-tuning model for few-shot intent detection using RoBERTaaeee 9) USE. A large multilingual model pretrained on 16 languages (Yang et al., 2020). 10) CONVERT.
Results
Referring to Table 3, we first note that Al per- forms significantly better than baseline (2-shot) for DistilBERTTaee and BERT base across all four datasets. This confirms that data augmentation is helpful in data-scarce setups.
6.1 On the Effeet of Relabeling Step
Table 5 shows the percentage of generated exam- ples relabeled by 5PT3.--turbo for different aug- mentation methods. We note that relabeling per- centage is higher when we add mixup to the prompt (PromptMix v/s PromptMix w/o Mixup). High re- labeling percentages suggest that mixup generates more borderline examples than any other augmen- tation method. This verifies the importance of the relabeling step in our method, which is highly ef- fective in remedying the problem of false positive generations. This is further demonstrated by higher A2 values than Al across the board. Specifically, for PromptMix, we observe an improvement of 7.4% on B77, 8. 1% on TREC6, 12.4% on SUBJ, and 13.5% on TC, when we use DistilBERT) bases and 5.9% on B77, 10.4% on TREC6, 14.4% on SUBJ, and 11.6% on TC when use BERT basse In addition these significant improvements, we note that the degree of improvement increases as the classification task gets easier in terms of the number of target classes. Table shows some ex- amples of leaked generations that GPT rectifies
An intent detection model from Casanueva et al. (2020), which uses a dual encoder setup pretrained

---

Prompt Features
1877
TRRCS
SURI
TC
Method
Ex.
Dex:
caass
Misup
A1
A2
A2
DisiBERT
Baseline
16.0 (0.9)
31,7
64.4
38.8 (0.8)
EDA
47.8 (0.7)
40.9 (0.6)
(0.4)
42,9 (0.7)
GPT3Mix
(7.8(28)
(1.3(1.5)
Sahu (2022)
(1.9
(0.6)
(1.3)
(0.7)
(1.3)
83.7 (0.4)
$1.5 (0.6)
(5.400.5)
dese.
(1.1
72.4 (0.7)
63.8 (1.1)
64.9 (0.6)
(4.2.11)
36.3 (0.5)
57.2 (07)
67.9
PromptMix Mixup (verosshot)
72.2(113)
(0.8)
(1.0
(1.3)
PrompeMix (zcro shot)
(2.3)
(1.4
54.1 (1.7)
65.7 (0.7)
$0.
$5.4 (0.5)
56.0 (1.3)
71.5 (0.9)
PrompeMix Mixup
MrimptMix
72.3 (1.1)
79.7 (0.7)
60.6 (1.4)
68.7 (9.6
(1.)
89,9 (0.8)
61.8 (1.4)
75.3 (1.2)
Basclinc
21.6
33.0 (0.6)
(0.8
42.7 (0.6)
EDA
(0.9)
GPT3Mi
60.5 (6.1)
90.6(1.1)
LINDA (5-shot)
Sahu (2022)
(1.4
(1.6111)
(1.2)
57.8 (0.8
83.6(1.3)
(1.00
55.3
58.1 (0.3)
(2.)
(3.7
67.1 64.3
k7.(1.))
87.2
66.1
(1.2(1.1)
PromptMix Mixup (ccrosshot)
(4.0
764(11.)
61.2
(..)
$3,(.14)
85.9
64.3
(8.9
PromptMix (scro-shot)
(2.4
55.5 77.6(1.)
6..5
$0.
(0.7)
74.7 (2-4
Prompt Mix Mixap
(1.4(1.4)
78.5 (0.7)
70.1 (1.5
(0.6
(7.0(1))
(0.0
71.0 (09)
(2.3
Mix
(1.6 80.1 (0.9)
(2.5)
73.7
(2.3
(1.1)
66.8
78.4 (0.8)
NN4GPTRS5
79.9
74.4
Table 3: Test classification accuracy (out of 100%) on four datasets, averaged across three random seeds (with standard deviation in brackets). A1 is the accuracy of the classifier on the generated dataset, and A2 is the accuracy the generated-relabeled dataset. Note: use GPT3Mix results from Yoo et al. (2021), and LINDA results from Kim et al. (2021). "_" for a method shows that the particular prompt feature is inapplicable to that method.
Method
Accuracy
79.9) and SUBJ (91.7 v/s 90.4) and is competi- tive on TREC6 (73.7 v/s 74.4). The gap between PromptMix and NN+GPT3.5 is larger on TC com- pared to other datasets (78.4 v/s 88.6). This might be due to the nature of the dataset, which con- tains extensive use of social media language that GPT knows about, but two examples might be too few to cover the vast diversity of complaints in the wild. Overall, these results are promising as GPT3.5-tur with 175B parameters is 2600 larger than DistilBERT base and ~ 1600x larger than BERT which only have 67M and 10M parameters, respectively. In both cases, our fi- nal classifiers are > 99.9% smaller than GPT3.5- turbo. We do not see such strong performance for any other generation method, even after relabeling. This confirms that generating borderline examples in PromptMix makes for high-quality dataset gen- eration method that aids the transfer knowledge of large-scale models such as GPT3.5- -ptrrbo into much smaller classifiers data-scarce settings.
USE (5-shot)
76,3
CONVERT (5-shol)
75.3
USE+CONVERT (5-shot)
77.8
CPFT (5-shot)
80.9
PromptMix (2-shot)
80.1
Table 4: Comparing 2-shot PromptMix results with shot baselines on B77. Note: CPFT, USE, CONVERT, and USE+CONVERT as reported Zhang et al. (2021).
Method
B77
TREC6
SUBJ
TC
Sahu et al. (2022)
21.0
3.9
2.5
+ desc.
8.5
28.0
1.5
8.5
PromptMix (zero-shot)
32.8
36.2
28.8
16.2
w/o Mixup
22.5
18.9
2.6
7.8
PromptMix
33.9
33.8
42.0
23.4
w/o Mixup
22.2
20.3
6.1
14.0
Table 5: Percentage of generated examples relabeled by GPT3.5-turbo Step for different methods. Note: percentages are averaged across three runs.
6.3 PromptMix v/s Other Augmentation Methods
during the relabeling step.
6.2 Borderline Examples Aid Knowledge Distillation
In Table 3, compare the 2-shot performance of PromptMix against several strong baselines on the four datasets. First, note that PromptMix significantly better than the EDA baseline on all datasets. Next, we compare the results of Prompt- Mix (which uses GPT3.5 --trrbt for generation) with
In Table 3, we notice that PromptMix achieves almost similar performance as NN+GPT3.5 on three out of four datasets. Specifically, Prompt- Mix outperforms NN+GPT3.5 on B77 (80.1 v/s

---

GPT3.5-turbo generated sentences
Before Relabeling After Relabeling
Twitter Complaints
This also expresses an opinion about product. but includes minor complaint about the battery life.
no complaint complaint
appreciate the quick response
GoUber_Support but the
no_complaint complaint
driver's s behavior was unacoeptahle, #goodandhad
@SStarbucks this of beans you use? #Noo #NNo_Complll Complaint
latte amazing!
Can you tell me what kind
complaint + mo_complaint
I'm really impressed with &Nike's customer service. They
complaint + no. complaint
helped me solve my problem quickly and efficiently.
Banking77
Why my transfer declined after tried send money to my daughter's account?
uge limil - Bechinedd_ransllee
My transfers keep getting declined, but I'm old enough to have an account. Can you help me?
age soc_limitaae + declined transfer
My card got declined when tried top up my account. What's going on?
topping up. up. card declined card payment
there limit
much can top up with my card? It's topping_up_ be_iinn_u__b___rrdd top_up_limiii
not letting me add more money.
just tried topping up my account with my
card
topping up by barrddrd__.. + uop.up failee failed
and it didn't 't work.
Table Effect of Relabeling. GPT3.5-turbe sentences that were relabeled the Twitter Complaints and Banking77 datasets. Highlighted text denotes the difference with respect to the class of the generated sentence before relabeling.
GPT3Mix (which uses GPT3's Davinci model for generation) and LINDA as they are the closest methods to ours in terms of motivation. Prompt- Mix outperforms both LINDA and GPT3Mix by huge margin even though LINDA is 5-shot and uses pretraining, and GPT3Mix utilizes I 1% of training samples for augmentation, translating to 55 total seed examples compared our 12 seed examples on TREC6, and 80 seed examples compared to our seed examples on the SUBJ dataset.
adding class description to the prompt leads to decent performance gains on all the datasets (Sahu et al. (2022) w/ and w/o desc). Comparing Prompt- Mix (zero-shot) with and without Mixup against Sahu et al. (2022) + desc. shows that using only de- scriptions in the prompt (no seed examples) leads to a significant boost in model performance. We also observe that using multiple class descriptions in a prompt either better or equivalent to using a single class with its description and examples. This suggests that providing more context about other classes helps LLM generate better quality augmen- tations for given class. Therefore, it follows intu- itively that PromptMix, which combines the usage of multiple classes with deseriptions well as seed examples, leads to the best performance all the datasets.
Table 4 shows that 2-shot PromptMix outper- forms USE, CONVERT, and USE+CONVERT on the Banking77 dataset. It also achieves an accu- racy of 80.1% compared to 80.9% by CPFT, which is highly competitive. To emphasize, all the con- sidered baselines are 5-shot, with some models using additional data for pretraining. Moreover, it is possible to further improve the performance of PromptMix with more dataset- and model- specific hyperparameter-tuniee tuning; however, our core message is to show that generating borderline examples com- bined with relabeling a highly promising method for data augmentation in 2-shot and zero-shot text classification setups.
6.5 Open-source v/s Closed-source models
Table 3 and 4 showcase strong capabilities of GPT3.5-turbo for our task. However, GPT3.5- turbo is closed-source language model provided by OpenAT, and it can quickly get costly as we in- crease the number of classes in the dataset. There- fore. we conduct a small-scale experiment on the TREC6 dataset with open-source LLMs. In par- ticular, we prompt open-source LLMs instead of GPT3.5-turbo to synthesize new examples and to relabel them. Next, we finetune a BERT base classi-
6.4 Descriptions are Impactful
Our experiments that were used to evaluate the effectiveness using human written descriptions show promising results. First, we note that simply

---

Method
Accuracy
it highly promising data augmentation approach.
Baseline
33.0
Limitations
PromptMix (Llama---7b-chat-be
55.6
PromptMix (Llama---33b-chathh
66.6
Our results indicate a promising potential to use LLMs for generating data in highly-ageressive few- shot sctups; however, this work has few limi- tations, as detailed in this section. First, we rely completely on GPT's pseudolabels tackle false positive generations during the augmentation step. Secondly, since we use SBERT embeddings to en- sure that pseudolabel is a valid class in the dataset, we ignore potential out-of-scopccout-of-ooooo (OOS/OOD) generations. We did observe few in- stances where GPT pseudolabels were of the form, "This sentence does not belong to any of the pro- vided classes." This presents good avenue in- troduce human interventions that can judge GPT pseudolabels and can identify OOS/OOD exam- ples, which can be helpful in many real-life tasks. We also want to emphasize that while human in- terventions can greatly help, they also present the challenge minimizing human labor.
PromptMix (Llama---7bb-chatll
70.8
PromptMix (GPT3.5-turbo)
73.7
PromptMix (GPT-4)
86.2
NN+GPT3.5
74.4
NY-LLama--270b-chaah
76.2
NN+GPT4
88.2
Table 7: Comparison of different open-source LLMs v/s GPT on TREC6. Note: Baseline refers the shot accuracy of the BERThase classifier. We replace GPT3.5-trro with GPT4 and to get the NN+GPT4 and 1a-L-amm---70-chaa baselines, respectively.
fier on the augmented+rrlabblled dataset. GPT-Neo (1.3B), GPT-J (6B), and the more re- cent instruction-tuned Vicuna and Stable-vicuna (13B) (Chiang et al., 2023) models achieve very poor performance compared GPT3.5-turbo. We find that even for medium--izez LLMs like Vicuna and Stable-vicuna. inference time is major bottle- neck 5. However, our experiments with LLama-2 show promising results. We used different-sized Llama-2 models (7b, 13b, 70b) the TREC6 dataset and observe strong correlation between model size and test accuracy. In particular, we note that 2Lamm-2-70b-chat-hh might be a decent alter- native to the closed-source GPT3.5-turbo model. We also test GPT-4 (larger than the GPT3.5 model) and observe significant boost in classification ac- curacy. Table shows the results
Second, while our experiments with open-source models suggest that the larger open-source mod- els like LLama-2-70b might be a good alterna- tive closed-source models like GPT3.5, thereby greatly cutting API costs, they still demand signif- icant computational resources. Therefore, efforts towards distilling the knowledge of bigger LLMs like Llama-2-700 2- into smaller models would greatl aid in making these models more accessible and feasible for use.
Ethics Statement
We use GPT generate new examples, and even though GPT3.5-turbo is instruction-tunele some generations might depict undesirable biases with re- spect to the current objective. For the stated reason, we recommend using our proposed classification models with human monitoring to avoid any ethical issues due to false positive predictions of the down- stream classifiers. Practitioners may also consider explicitly debiasing language models for their spe- cific use cases (Barikeri al., 2021; Schick et al., 2021).
7 Conclusion
To conclude, we propose PromptMix, novel two-step prompting- based method for generating borderline augmented examples in extreme few- shot text classification setups. Our method com- bines the process of text augmentation, pseudola- beling, and knowledge distillation in a cohesive pipeline. We show that by generating borderline training examples and relabeling them using large teacher model like GPr3.5-turbo, we can transfer the knowledge of such massive LLMs into much smaller models like DistilBERT, base and BERT base- Furthermore, 2-shot PromptMix beats multiple shot or higher data augmentation baselines, making
References
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Car- lier, Michael Noetel, and Andreas Stuhlmüller. 2021.
'e tried on 32G V100 with 8-bit quantization "We use Anyscale endpoints to access 1.aam-22 models.

---

RAFT: A real-world few-shot text classification benchmark. In Thirty-fiith Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 2).
Xingping Dong and Jianbing Shen. 2018. Triplet loss in siamese network for object tracking. In Proceed- ings of the European conference on computer vision (ECCV), pages 459-474,
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor. George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 7383-7390.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. Redditbias: A real-world resource for bias evaluation and debiasing of conversational language models. arXiv preprint arXiv:2106.03::
Yekyung Kim, Seohyeong Jeong, and Kyunghyun Cho. 2021. Linda: Unsupervised learning to interpo- late in natural language processing. arXiv preprint arXiv: 13969..
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. Flex: Unifying evaluation for few-shot nlp. Advances in Neural Information Processing Systems, 14:15787-15800.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained trans- former models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems. pages 18-26.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Sys- tems, pages 1877-1901.
Varun Kumar, Hadrien Glaude, Cyprien de Lichy, and Wlliam Campbell. 2019. A closer look at feature space data augmentation for few-shot intent classifi- cation. EMNLP-IJCNLP 2019, page 1.
Cristian Bucila, Rich Caruana, Alexandru Niculescu- Mizil, et al. 2006. Model compression. In Kdd, volume 6, pages 1 150402-1150461
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlmmyyr. 2020. Bart: Denoising sequence-to-sen to-eequence pre- pre-training for nat- ural language generation. translation, and comprehen- sion. In P Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulié. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38 45, On- line. Association for Computational Linguistics.
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021. Mixkd: Towards efficient distillation of large- scale language models. In International Conference on Learning Representations.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yen-Ting Lin, Alexandros Papangelis, Scokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazi- far, Di Jin, Yang Liu, and Dilek Hakkani-Tur. 2023. Selective in-context data augmentation for intent de- tection using pointwise v viinformation. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1455-1468.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 022231204..2311..
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 41 4186.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cx/0409058.

---

Daniel Preotiuc-Piett Piettoo Mihaela Gaman, and Nikolaos Aletras. 2019. Automatically identifying complaints in social media. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 5008-5019, Florence, Italy. Associa- tion for Computational Linguisties.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Mod- els with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 86-96, Berlin, Germany. Association for Computa- tional Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Victor Sheng, Foster Provost, and Panagiotis G lpeiro- tis, 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614-622.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling multi-step reasoning ca- pabilities of large language models into smaller mod- els via semantic decompositions. arXiv preprint arXiv:2212.00199
Cyrus Rashtchian, Peter Young, Micah Hodosh, and Ju- lia Hockenmaier. 2010. Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACI. HLT 2010 workshop on creating speech and language data with Amazon's 's Mechanical Turk, pages 137-147.
Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. 2020. Contrastive distil- lation on intermediate representations for language model compression. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 498-508.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLA) pages 9992,3992, Hong Kong, China. Association for Com- putational Linguisties.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie. Yizhong Wang. Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online. Association for Computa- tional Linguistics.
Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bah- danau. 2022. Data augmentation for intent classi- fication with off-the-shelf large language models. In Proceedings of the 4th Workshop on NLP for Conver- sational AI, pages 47-57, Dublin, Ireland. Associa- tion for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rxiv:1910.01100
Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo. Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts, arXiv preprint 11Xi::2299.1105.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computarional Linguisticx: Human Language Technologies, pages 2339-2352.
Ellen M Voorhees, Dawn M Tice, et al. 1999. The trec-8 question answering track evaluation. In TREC, volume 1999, page 82.
Yufei Wang, Jiayi Zheng, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, and Daxin Jiang. 2023. KnowDA: All-in-one knowledge mixture model for data augmentation low-resource NLP. In The Eleventh International Conference on Learning Rep- resentations.
Timo Schick, Sahana Udupa, and Hinrich Schüitze. 2021. Self-diagnosis and self self-debi A proposal for re- ducing corpus-based bias in nlp. Transactions of the A Association for Computational Linguistics, 9: 1408 1424.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 815-823.
Jason Wei, Chengyu Huang. Soroush Vosoughi. Yu Cheng, and Shiqi Xu. 2021. Few-shot text clas- sification with triplet networks, data augmentation, and curriculum learning. In Proceedings of the 2021 Conference of the North American Chapter of the

---

Association for Computational Linguistics: Human Language Technologies. pages 5493-5500.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.001
Jason Wei and Kai Zou. 2019. EDA: Easy data augmen- tation techniques for boosting performance text classification tasks. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- (JCNP)), pages 6382-6338 6388, Hong Kong, China. As- sociation for Computational Linguistics.
Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I Jordan, and Jiantao Jiao. 2023. Fine-tuning language models with advantage-induee policy alignment. arXiv preprint mxXiv:3306.02231.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena
A
Appendix
Hwang. Liwei Jiang. Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 4602-4622 Seat- tle United States. Association for Computational Linguistics.
A.1 Distribution of for Mixup
We use round(10(z + 1))/20 V x ~ 3,, 2) in our experiments. We modify the standard B-distribution by restricting its range the half in- terval of (0.5, 1.0] with peak near 1.0. We choose >> 0.5 incentivize the LLM to generate exam- ples that are mixed up but still belong to G. We round off the a to the nearest 0.05 to avoid deci- mal values that would be arbitrary, given we are operating with natural language instructions.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of- the-art natural language processing, arXiv preprint 1r1.0:191..03771.
25000
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contex- tual augmentation. In Computational Science-ICCS 2019: 19th International Conference, Faro, Portugal, June 12-14, 2019, Proceedings, Part IV 19, pages 84 95. Springer.
20000
15000
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sen- tence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations. pages 87-94, Online. Association for Computational Linguistics.
5000
07
5.
1.0
Figure 5: a = rowndd 1)) 2 v (5,2)).
A.2 Choice of
Kang Min Yoo, Dongju Park, Jacwook Kang, Sang- Woo Lee, and Woomyoung Park. 2021. Gpt3mix: Lever- aging large-scale language models for text augmen- tation. In Findings of the Association for Computa- tional Linguisties: EMNLP 2021, pages 2225-2239,
We experiment with different values of f (classes to include in the prompt) and choose = based on the validation performance in Table 8.
t
Accuracy
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
2
73.3
4
76.8
6
69.1
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021. Few-shot intent detection via contrastive pre-training and fine-tuning. arXiv preprini arXiv:2109.06344
Table 8: Validation accuracy for different values of TREC6.


Reviews:
[{"review_id": "gd9el7Yzww", "reviewer": "Reviewer_Z33a", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful feedback. We are encouraged to know that the paper is easy to follow and that you found our insights helpful. We thank you for the relevant citation suggestions and a meaningful analysis we can add to that paper. Overall, they have helped us gain new insights and discussion points. We would also like to mention that we will release our codebase upon acceptance for reproducibility.\n\nWe now respond to your comments in detail:\n\n> *The reasoning behind why borderline examples are beneficial is not super clear from this draft. Some prior work has suggested that ambiguous examples are usually harder for models and having more of those usually helps improve model performance (see https://arxiv.org/abs/2009.10795).*\n\nThank you very much for bringing this work to our attention! The concept of Data Maps highly aligns with our case-in-point that we need ambiguous examples to build a good quality dataset and ultimately train a robust classifier. We will include this work in our paper.\n\n> *One analysis that could strengthen the paper: which models are being used for data augmentation.*\n\nWe use OpenAI's gpt-3.5-turbo for data augmentation (version 0613).\n\n> *How does the performance change with the size of the model considered for augmentation and so on?*\n\nGreat question! We tried the smaller models GPT-Neo (1.3B), GPT-J (6B), OpenAI's Curie (estimated 13B), and the more recent instruction-tuned Vicuna and Stable Vicuna (13B) models but they achieved low performance in comparison to GPT-3.5. \n\nHowever, new models have emerged recently in the LLM space, like Llama-2 and GPT-4. As seen in the Table below, we ran different-sized Llama-2 models (7b, 13b, and 70b) on **TREC6** and saw a strong correlation between model size and test accuracy. We also ran GPT-4 (which is larger than the GPT-3.5-turbo model we used) and achieved even better results. We have added these results to the paper.\n\n**Exp 1. Results on TREC 6**\n\n| Model |  Test Acc (A2) | \n| --- | --- | \n|**PromptMix Results**|\n| Llama-2-7b-chat-hf | 55.6$^\\dagger$ | \n| Llama-2-13b-chat-hf |  66.6$^\\dagger$ | \n| Llama-2-70b-chat-hf |  70.8$^\\dagger$ | \n| GPT-3.5-turbo (as in the paper)  |  73.7 | \n| GPT-4 |  **86.2*** | \n|**Upper Bound Results**|\n|NN+GPT-3.5-turbo (as in the paper) |  74.4 |\n|NN+Llama-2-70b-chat-hf | 76.2 |\n|NN+GPT4 | 88.2 |\n||\n\n$^\\dagger$ used Llama-2-70b-chat-hf for relabeling; *used GPT-4 for relabeling\n\n> *Some recent work on distillation from LLMs to improve downstream tasks should be discussed in related work or introduction. See https://arxiv.org/abs/2110.07178 and follow-up works.*\n\nThank you for pointing this out. We will cite this work and add a discussion of these models in our related work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed experimental setup is simple and easy to implement and leads to substantial improvements over baselines. \n2. The paper is well written, well organized, and provides extensive analysis and ablations showcasing the importance of each presented component. ", "reasons_to_reject": "Not necessarily reasons to reject but the paper could benefit from including some of these discussions.\n1. The reasoning behind why borderline examples are beneficial is not super clear from this draft. Some prior work has suggested that ambiguous examples are usually harder for models and having more of those usually helps improve model performance (see https://arxiv.org/abs/2009.10795). \n2. One analysis that could strengthen the paper: which models are being used for data augmentation. How does the performance change with the size of the model considered for augmentation and so on? ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a data augmentation strategy for text classification using LLMs. The authors propose to use LLMs like GPT3 to generate \"border-line\" examples by (1) prompting the model to generate examples that are x% in one class and (1-x)% in another (mixing), and then (2) prompting the model again to label the generated examples. By augmenting these examples to the original training datasets and fine-tuning BERT based models, the authors show significant improvements over other data augmentation based baselines on the 4 datasets considered and improved performance over GPT3.5 on 3/4 datasets considered. \n\nThe authors also provided ablations on each of the proposed components to showcase the importance of each of them.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Some recent work on distillation from LLMs to improve downstream tasks should be discussed in related work or introduction. See https://arxiv.org/abs/2110.07178 and follow-up works.\n"}, "extra_text": {}}, {"review_id": "n5pUcE4cAO", "reviewer": "Reviewer_XgUZ", "summary": "", "questions": "One question would be do you think PromptMix would perform worse in cases of task like Hate Speech detection ? \nBecause it's already shown in this paper (https://aclanthology.org/2023.findings-eacl.9/) that for tasks like Hate Speech there is no concept of mixup like 70% non-hate + 30% hate. Any sentence which has >0% hate is hateful irrespective of whether it's majority or minority class.\n\nSo it would be interesting to see this kind of analysis in the paper as well or maybe cite & mention the same in Limitations section.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your review. We are pleased to know that you found our methodology intuitive and that you liked our analysis. Thank you for the relevant work suggestions and for asking great questions. Getting answers to your questions allowed us to gain some interesting insights. We would also like to mention that we will release our codebase upon acceptance for reproducibility.\n\nWe now respond to your comments in detail:\n\n> *In Section 3.1, it's mentioned that authors choose random c=4 classes. What is the performance variance when this \"c\" set is changed for B77 and TREC6 which has >4 classes ?*\n\nGood question! We agree we didn't discuss our choice of $t=4$ in the paper, but we chose it based on the following validation accuracies obtain on the TREC6 for PromptMix (we have added these results to our paper):\n\n| t | Validation Acc (A2) |\n| --- | --- |\n| 2 |  73.3 |\n| 4 |  76.8 |\n| 6 |  69.1 |\n||\n\n> *An analysis on that would be nice i.e. which open-source LLMs would work with this approach ? or maybe find a LLM parameter count cutoff (say 7B) lower than that performance of PromptMix would start degrading ?*\n\nGreat question! We have tried the open-source models GPT-Neo (1.3B), GPT-J (6B), and the more recent instruction-tuned Vicuna and Stable Vicuna (13B) models but they achieved low performance in comparison to GPT-3.5.\n\nHowever, new open-source models have emerged recently in the LLM space, like Llama-2. As seen in the Table below, we ran different sized Llama-V2 (7b, 13b, and 70b) on TREC 6 and saw that Llama-V2 70b achieves close results to GPT-3.5, suggesting that current open-source models are also feasible for our proposed method. We have added these results to the paper.\n\n\n**Exp 1. Results on TREC 6**\n\n| Model |  Test Acc (A2) | \n| --- | --- | \n|**PromptMix Results**|\n| Llama-2-7b-chat-hf | 55.6$^\\dagger$ | \n| Llama-2-13b-chat-hf |  66.6$^\\dagger$ | \n| Llama-2-70b-chat-hf |  70.8$^\\dagger$ | \n| GPT-3.5-turbo (as in the paper)  |  73.7 | \n|**Upper Bound Results**|\n|NN+GPT-3.5-turbo (as in the paper)  |  74.4 |\n||\n\n$^\\dagger$ used Llama-70b-chat-hf for relabeling.\n\nFor the LLM parameter count cutoff, there's a significant performance drop when switching to the 7b Llama model, suggesting that 13B might be the minimum size for an LLM to be feasible for data augmentation in 2-shot text classification settings.\n\n> *Report the standard deviations for all the main results presented. Because in few-shot and LLM-based settings it's important to understand the stability of the results obtained.*\n\nThanks for raising this point. We have added standard deviations to our main results table in parentheses. Here's a sample of Table 3 with std values (A1 and A2 values are separated by a \";\").\n\n| Model | B77 | TREC6 | SUBJ | TC |\n| --- | --- | --- | --- | --- |\n| | | BERT-base | |\n| Baseline | 22.6 (1.2) | 33.0 (0.6) | 71.6 (0.8) | 42.7 (0.6) |\n| EDA | 49.2 (0.9) | 51.1 (0.6) | 84.5 (0.5) | 47.8 (0.4) |\n| GPT3Mix | $-$ | 60.5 (6.1) | 90.6 (1.1) | $-$ |\n| Linda (5-shot) | $-$ | 62.2 (3.1) | $-$ | $-$ |\n| Sahu et al., (2022) | 70.7 (1.4) ; 71.6 (1.1) | 51.3 (1.2) ; 57.8 (0.8) | 83.6 (1.3) ; 85.5 (1.0) | 55.3 (1.1); 58.1 (0.3) |\n| $\\quad$+desc | 72.8 (1.1) ; 73.0 (0.7) | 64.3 (1.3) ; 67.1 (0.2) | 87.0 (1.7) ; 87.2 (1.1) | 66.1 (1.4) ; 65.2 (1.1) |\n| PromptMix w/o Mixup (zero-shot) | 74.0 (1.9) ; 76.4 (1.1) | 64.2 (1.6) ; 68.5 (0.9) | 83.6 (1.4) ; 85.9 (0.8) | 64.3 (1.2) ; 68.9 (0.3) |\nPromptMix (zero-shot) | 71.3 (2.4) ; 77.6 (1.3) | 55.5 (1.7) ; 67.5 (0.4) | 80.7 (1.4) ; 89.5 (0.7) | 57.6 (2.4) ; 74.7 (1.5) |\nPromptMix w/o Mixup | 74.4 (1.4) ; 78.5 (0.7) | 70.1 (1.5) ; 71.6 (0.2) | 87.0 (1.3) ; 90.0 (0.1) | 71.0 (0.9) ; 72.3 (0.3) |\nPromptMix | 74.2 (1.6) ; 80.1 (0.9) | 63.3 (2.5) ; 73.7 (1.1) | 77.3 (2.2) ; 91.7 (1.1) | 66.8 (0.8) ; 78.4 (0.8) |\n||\n\n> *do you think PromptMix would perform worse in cases of task like Hate Speech detection ? Because it's already shown in this paper (https://aclanthology.org/2023.findings-eacl.9/) that for tasks like Hate Speech there is no concept of mixup like 70% non-hate + 30% hate. Any sentence which has >0% hate is hateful irrespective of whether it's majority or minority class.*\n\nNo, we think PromptMix would work for Hate Speech Detection as well. The example of 70% non-hate + 30% hate should be correctly classified as hate, but it's possible that an ML model puts more attention onto the non-hate part of the sentence and misclassifies it as non-hate. In our pipeline, we aim to minimize such mistakes by training the classifier on such hard/ambiguous examples where examples are not clear for the model on whether they are \"hate\" or \"non-hate\". We also thank you for pointing out the EasyMix paper--we agree it carries the same motivation as our approach, and we will cite it and discuss it in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper is very detailed and well-written.\n2) The idea of generating borderline examples using LLMs makes sense intuitively and also the authors show ablation as to why the re-labeling step is important. \n3) They compare with lot of other related baselines which makes for a complete study on the task at hand.\n4) Another interesting analysis, is adding class descriptions in zero-shot setting alone lead to improvements over other approaches.", "reasons_to_reject": "1) In Section 3.1, it's mentioned that authors choose random c=4 classes. What is the performance variance when this \"c\" set is changed for B77 and TREC6 which has >4 classes ? \n2) Another thing to note would be the performance of PromptMix inherently depends on the quality of questions / answers generated by the LLM. So a high-quality model is essential for the success of this task. An analysis on that would be nice i.e. which open-source LLMs would work with this approach ? or maybe find a LLM parameter count cutoff (say 7B) lower than that performance of PromptMix would start degrading ?\n3)  Report the standard deviations for all the main results presented. Because in few-shot and LLM-based settings it's important to understand the stability of the results obtained.\n\n\n\n*EDIT* - I've read the rebuttal and the authors have cleared out most of the rejection points through subsequent experiments. Keeping the rating as Strong Accept. Thanks for the work. Nice read !", "questions_to_authors": "One question would be do you think PromptMix would perform worse in cases of task like Hate Speech detection ? \nBecause it's already shown in this paper (https://aclanthology.org/2023.findings-eacl.9/) that for tasks like Hate Speech there is no concept of mixup like 70% non-hate + 30% hate. Any sentence which has >0% hate is hateful irrespective of whether it's majority or minority class.\n\nSo it would be interesting to see this kind of analysis in the paper as well or maybe cite & mention the same in Limitations section.", "paper_topic_and_main_contributions": "This paper introduces PromptMix where the authors propose generating borderline class samples using a LLM and then re-labeling them again leads to better learned student models trained on the augmented data. They select a subset of classes and k(=2) examples per class to create the instruction prompt. Then they instruct gpt-3.5-turbo to generate samples which contain mix of classes and get them re-labeled back using LLMs again to fix the ambiguous generation cases. This leads to better performance compared to other data augmentation techniques in few-shot & zero-shot settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://aclanthology.org/2023.findings-eacl.9/ - mentioned above already"}, "extra_text": {}}, {"review_id": "gnrU2upf5g", "reviewer": "Reviewer_ivSt", "summary": "", "questions": "1) Did you follow any specific methodology for writing the prompts and the class descriptions? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your detailed comments on our paper. We are glad you find our approach interesting and are satisfied with our analysis. You raise an excellent point about the feasibility of open-source models in our framework. We've addressed that comment and others in detail below. We would also like to mention that we will release our codebase upon acceptance for reproducibility.\n\n> *It could have been interesting to see how well the approach work with open source models which are much more sensitive to the prompts as well as the classification dataset used for the task.*\n\nGreat question! We have tried the open-source models GPT-Neo (1.3B), GPT-J (6B), and the more recent instruction-tuned Vicuna and Stable Vicuna (13B) models but they achieved low performance in comparison to GPT-3.5.\n\nHowever, new open-source models have emerged recently in the LLM space, like Llama-2. As seen in the Table below, we ran different sized Llama-V2 (7b, 13b, and 70b) on TREC 6 and saw that Llama-V2 70b achieves close results to GPT-3.5, suggesting that current open-source models are also feasible for our proposed method. We have added these results to the paper.\n\n\n**Exp 1. Results on TREC 6**\n\n| Model |  Test Acc (A2) | \n| --- | --- | \n|**PromptMix Results**|\n| Llama-2-7b-chat-hf | 55.6$^\\dagger$ | \n| Llama-2-13b-chat-hf |  66.6$^\\dagger$ | \n| Llama-2-70b-chat-hf |  70.8$^\\dagger$ | \n| GPT-3.5-turbo (as in the paper)  |  73.7 | \n|**Upper Bound Results**|\n|NN+GPT-3.5-turbo (as in the paper)  |  74.4 |\n||\n\n$^\\dagger$ used Llama-70b-chat-hf for relabeling.\n\n> *It is not discussed how easy and whether is possible to adapt the approach simply by using prompts to diverse set of datasets such as legal domain, medical domain.*\n\nThanks for raising this point. For each domain we had around 3-4 iterations of writing the class descriptions. We chose the descriptions that gave us the most reasonable synthetically generated sentences on a very small set of examples, which makes it practical in the sense that it requires little human effort. Since this approach worked for a variety of diverse domains from banking to social media, we believe that this method will work on domains like legal and medical. We have added this detail in the paper.\n\n> *Did you follow any specific methodology for writing the prompts and the class descriptions?*\n\nWe tried multiple class descriptions starting with the most general definitions of the classes and then finetuning them around 3-4 times (manually) as described in the previous answer.\n\nFor constructing the prompts, we chose the one we saw generated the most diverse sentences on a very small set of examples for a single domain and replicated the same prompts template for the other domains. This construction of prompts was inspired by [A] as the starting point of our prompt construction and [B] which performs mixup on the embedding space rather than in the text space like we do.\n\n[A] Sahu, G., Rodriguez, P., Laradji, I., Atighehchian, P., Vazquez, D., & Bahdanau, D. (2022, May). Data Augmentation for Intent Classification with Off-the-shelf Large Language Models. In Proceedings of the 4th Workshop on NLP for Conversational AI (pp. 47-57).\n\n[B] Kim, Y., Jeong, S., & Cho, K. (2021). Linda: Unsupervised learning to interpolate in natural language processing. arXiv preprint arXiv:2112.13969.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is very interesting and shows to work without the need of additional unlabelled data or fine-tuning the generative model. The novelty of the paper is well justified and shows that generating class border samples and relabling them using the same model lead to improvements over baseline methods. The authors have performed extensive analysis.", "reasons_to_reject": "The OPEN AI models work quite well in few and zero shot settings using only prompts, however these services are paid. It could have been interesting to see how well the approach work with open source models which are much more sensitive to the prompts as well as the classification dataset used for the task. The authors do not discuss the methodology they used for writing the instructions or how dependent the model is to the dataset and the classes used for the task. It is not discussed how easy and whether is possible to adapt the approach simply by using prompts to diverse set of datasets such as legal domain, medical domain.", "questions_to_authors": "1) Did you follow any specific methodology for writing the prompts and the class descriptions? ", "paper_topic_and_main_contributions": "The paper proposes a data augmentation approach (PromptMix) for generating additional training data for text classification that utlizes the abilities of large language models to follow instructions and perform few shot classification. The authors hypothesise that training a robust text classifier requires the training data to have a good mix of borderline examples. Therefore, they propose a two-step prompting-based method which first instructs GPT3.5-turbo to generate new examples by mixing information from multiple classes in order to produce borderline examples. Then, in the second step, the method uses a relabling scheme based on GPT3.5-turbo but this time instructing the model to relabel those generations to improve reliability of the class labels. When constructing the classification prompt, the authors select the top-5 closest classes according to the similarity between the SBERT sentence embedding.  The authors evaluate their method accross four diverse text classification datasets. The authors focus on 2-shot and zero-shot cases and fine-tune DistilBERT-base and BERT-base models for performing text classification. The results show that borderline examples aid knowledge distillation and show that 2-shot PromptMix beats multiple 5-shot or higher data augmentation baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a data augmentation approach using pre-trained LLMs, where they generate borderline (challenging) examples using a model GPT3, then prompting the model again to re-label the examples for correcting potential labeling errors. They show experimentally that augmenting this kind of data leads to improved student models, including in challenging few and zero-shot settings. The paper is well written and organized, with extensive analyses and ablation studies supporting the claims of the paper, and demonstrating the validity of the proposed data augmentation approach across various classification tasks.
Decision: Accept-Main