Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models
Authors: Ruida WANG
Wangchunshu Zhou
Mrinmaya Sachan
Venue: EMNLP 2023 Findings (2023)
Keywords: Dataset Synthesis
large language models
efficiency
TL;DR: This paper proposes Synthesis Step by Step (S3), a dataset synthesis framework that gives a better pseudo labelled dataset by synthesizing data with LLM rationales and iteratively optimizing the dataset by extrapolating errors of small models.

Abstract: *Data Synthesis* is a promising way to train a small model with very little labeled data. One approach for data synthesis is to leverage the rich knowledge from large language models to synthesize pseudo training examples for small models, making it possible to achieve both data and compute efficiency at the same time. However, a key challenge in data synthesis is that the synthesized dataset often suffers from a large distributional discrepancy from the *real task* data distribution. Thus, in this paper, we propose *Synthesis Step by Step* (**S3**), a data synthesis framework that shrinks this distribution gap by iteratively extrapolating the errors made by a small model trained on the synthesized dataset on a small real-world validation dataset using a large language model. Extensive experiments on multiple NLP tasks show that our approach improves the performance of a small model by reducing the gap between the synthetic dataset and the real data, resulting in significant improvement compared to several baselines: 9.48% improvement compared to ZeroGen and 2.73% compared to GoldGen, and at most 15.17% improvement compared to the small model trained on human-annotated data.

Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models
Ruida Wang"*
Wangchunshu Zhou
Mrinmaya Sachan E
ETT Zürich rwangbr@connect. ust. hk chunshu@aiwaves.. cn msachan@ethz.cch ch
HKUST
AAWaaee Inc.
Abstract
fewer parameters, are easy to deploy, and perform well in specific downstream tasks (Xu et al., 2021).
Data Synthesis is promising way to train small model with very little labeled data. One approach for data synthesis is to leverage the rich knowledge from large language models to synthesize pseudo training examples for small models, making it possible to achieve both data and compute efficiency at the same time. How- ever, key challenge in data synthesis is that the synthesized dataset often suffers from large distributional discrepancy from the real task data distribution. Thus, in this paper, propose Synthesis Step by Step (S3), a data syn- thesis framework that shrinks this distribution gap by iteratively extrapolating the errors made by a small model trained on the synthesized dataset small real-world validation dataset using large language model. Extensive ex- periments on multiple NI.P tasks show that our approach improves the performance of small model by reducing the gap between the syn- thetic dataset and the real data, resulting in significant improvement compared several baselines: 9.48% improvement compared to ZeroGen, 2.73% compared GoldGen, and 15.17% improvement compared to the small model trained on numan-annotater data.!
105
100
Accuray (%6
90
80
Epoch
-eZerrGen Train Acc.
ZeroCien Test Acc. GoldData Test Acc.
G-GoldData Train Acc. -
Training and testing accuracy of DistilBert with ZeroGen (Ye et al., 2022b) on the IMDb dataset with 200k training datapoints. Also shown are the train- ing and testing accuracy of the model trained on Gold- Data. We can see here that ZeroGen's 's training accuracy quickly reaches nearly 100%, but testing accuracy re- mains low.
Figure 1:
However, fitting small model for specific task may require large amounts of human-labeled data, which is not available in many downstream tasks and is expensive to annotate. This data ineffi- ciency problem makes it challenging to fine-tune small model. Therefore, a number of distinct re- search approaches attempt to reduce the amount of data required for fine-tuning small models spe- cific tasks, including knowledge distillation (Hin- ton al., 2015; Beyer et al., 2022; Hsich et al., 2023; Xu et al., 2020; Zhou et al.. 2020; Shrid- har et al., 2023), data augmentation VDeVries and Taylor, 2017; Shorten and Khoshgoftaar, 2019; Li et al., 2022), module replacing (Xu et al., 2020; Zhou et al., 2023), semi-supervised learning (Chen et al., 2020; Wang et al., 2021; Smith et al., 2022), and data synthesis (Anaby-Tavor et al., 2020; Puri et al., 2020).
1 Introduction
Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023; OpenAl, 2023) have shown promising zero-shot performance wide range of tasks, demonstrat- ing their potential of serving as generalist models. However, L.MS suffer from efficiency issues due to large model sizes and high inference latency, making them hard to deploy real-world appli- cations. Therefore, small models trained on task- specific data are still favored in many resource- constrained scenarios because they have much
Work done while at exchange ETH Zürich 'The code and generated data can be found at St-
In this work, we focus on data synthesis, which generates data and corresponding labels from scratch. Unlike semi-supervised learning, which
Step_Officie

---

Inditienal Zanlka''s Pronett
novie reviev it protinainggeee untimen
Procupt
Symthesized
Train
Small model
Gold data
model
Small
Traditional Shot Synthesize
(b) Train small model with gold data
Ratoral peoept
imagine
Sood Data Synthesis with Rationalies
Error extrapolation by LLM
novie consiter
kad aurthwloggahh
Rationale synthesize prompt
Synthesis prompe with sampled rationales
Small
arposssii marvic
Misclassified gold data
Synthema prompt raberall
New tha wekdaed a posic that las great acting, beorgguing glat and besutiffd cincestograpt.. Now you dould wiihe posiiive rexiow about movis
Sample rationales
Evalustion
Trolning
Synthesis on prompt
LLM
LLM
LLM
gold val. data
Systhens poonpt with
Whike puslive msvic sesiow
cerilar cuppour abagre
Synthesized
Soed data
Add to train
Addicional data
(c) Symthesis Step by Step
Figure 2: Both (a) traditional zero-shot dataset synthesis methods and (b) training small models directly on gold data do leverage feedback from the small model trained on the synthesized dataset. In contrast, (c) our approach. 53, first synthesizes seed dataset in zero-shot fashion with rationales (left-hand side). Then, we iteratively reduce the gap between the synthesized data distribution and the gold data distribution by extrapolating the emros a small model trained on the currently synthesized data on small gold validation The additional synthesized data can, therefore, be considered to be sampled from the difference between the currently synthesized data distribution and gold data distribution. By mixing with the currently synthesized data, we can recover the gold data distribution and therefore improve the performance of small model trained on the data mixture.
relies on unlabeled data, this approach is simpler and more efficient, especially when unlabeled data is scarce. Most existing methods in data synthe- sis for NLP utilize LLMs to generate an unlimited amount of training data for training small model. Existing dataset synthesis methods typically re- quire massive amount of synthesized data to achieve relatively good performance with small model, like ZeroGen (Ye et al., 2022b), which sometimes needs as much as IM records of synthe- sized data. However, this often results in additional data synthesis cost and computation costs when training the small task-specific model.
efficiency in dataset synthesis, we propose Syn- thesis Step by Step (S3), novel dataset synthesis framework that reduces the distribution gap in a data-efficient way by dynamically optimizing the synthesized dataset. As illustrated in Fig. 2, S3 first synthesizes seed dataset with explain-thert generate method that first prompts LLMs to gen- erate rationales for each label and then combines the generated rationale and task-specific prompts to generate data points. S3 then refines the seed dataset by iteratively synthesizing more data by extrapolating the errors of model trained on the seed dataset made on a small validation set, which assume is sampled from the real task data dis- tribution.
Intuitively, the quality of the synthesized data, the extent to which the synthesized data resembles the gold task data, is crucial for the small model's performance. However, due to the complexity of specific tasks in the real world, the synthesized data often suffers from distribution gap from the real- world data distribution. This can be clearly seen in 1ig.1. The small model's 's training accuracy on synthesized data is close to 100% but the testing accuracy on real-world data is still low. In contrast, the gap between training and testing accuracy is much smaller when trained on human-annotated data.
We summarize our contribution as follows: (1) We propose a novel point of view for dynamic dataset synthesis, which allows for the creation of training data for smaller models and can be opti- mized by adding more data; based on this point of view, we propose the S3 framework that can syn- thesize and optimize pscudo dataset using LLM that can efficiently shrink the distribution gap in dataset synthesis. (2) We perform theoretical analysis for the effectiveness the distribution gap. (3) We perform extensive ex- periments on three major NLP tasks and obtain
on reducing
To reduce the distribution gap and improve data

---

an average 9.48% improvement compared to Ze- roGen (Ye et al., 2022b), representative baseline for dataset synthesis, using only 30.43% of data on average.
is a computationally efficient model that will be trained on our synthesized datasct. In general, the small model contains much fewer parameters and is easy to train and deploy in real-world applica- We denote a small model trained by dataset D Dtrain as (trrai).
tions.
2 Methodology
We describe the proposed S3 framework in detail in this section. The key idea of S3 is to first synthesize a seed datasel by prompting LLMs and then to iter- atively reduce the distribution gap by extrapolating errors the small model makes on a small validation set from the gold data distribution. S3 comprises the following steps:
2.2 Seed Data Synthesis with Rationales
Seed Data is defined as the basic zero-shot synthe- sized dataset for our S3 framework.
Algorithm 1: Seed data synthesis with ra- tionales
Input: V, Trationn Tquery, P PLLM, K, k, N seed
1. Seed data generation: We utilize LLM to analyze the task we are working on, then synthesize list of possible rationales for such a task. If the task is hard to analyze, we can skip this step. Then, we combine the synthe- sized rationales, possible context sentences, and labels in one prompt to guide the LLM to synthesize the dataset.
Output: Dsoed
for each yi E y do
2 r; + topK((P1..a (PI.LM( (-TAr-(-TTattti (y:))
D seeed t
for in range( Nsced) do
5
Yeurr ~ U1)))
Teurr ~ Un(r;)
6
Teurr ~ P PLLas (-1 Tgueryy (Feurr Meurr)) 8 Dseed t- D Dseed U (Feurr yeurr )
2. Small model training: Train the small model with the synthesized dataset, then validate the small model on real-world validation data, and attain misclassified data of the small model, use them as errors.
We present the algorithm for seed data synthe- sis with rationales in Alg. 1. Here, y' denotes the of all possible labels in the task we are work- ing on; Tration (y) denotes label and task descrip- tive prompt for rationales synthesis; (1) (r, y) is the data synthesis prompt that wraps the ratio- nales in r and the label y together to query LLM for a data point; topK means top-K sampling from the LLM outputs to obtain the rationale list for a specific label; U.(S) means uniformly sample non-repeating elements in set S. The resulting seed dataset is denoted as Desed [Keeed, Veeed)- For instance, for the IMDb (Maas et al., 2011) dataset, a sentiment analysis dataset on movie reviews, Tration (y) positive /negttiuee is: "What is the reason that may lead to a positive/negative movie review." and the Tquery (F (osrrposs is: "Now imagine that you just watched movie that has great acting, intrigu- ing plot, and beautiful cinematography. Now you should write positive review about this movie." " We use the prompt as an input to the LLM and obtain the target output as the synthesized pseudo example. This enables us to generate more diverse, informative, and realistic examples.
3. Error extrapolation: Use the LLM to extrap- olate the errors of the small model and synthe- size additional data using the information in errors.
4. Combine and Repeat: Combine the addi- tional dataset and original dataset as a new synthesized train dataset for the small model, then repeat steps 2 and 3 for multiple rounds until the performance of the small model con- verges.
We first introduce some background and key notations in Section 2.1. We then describe the al- gorithms for seed data synthesis and iterative error extrapolation- based synthesis in Section 2.2 (point 1. above) and Section 2.3 (points 2, 3, 4 above), respectively. Finally, we give theoretical interpre- tation of the proposed method in Section 2.6.
2.1 Background
Following Sharp et al. (2017), we denote the dis- tribution of human language for the LLM under prompt input T as IP (..M(())) The small model
approach

---

2.3 Dataset Refinement with Error Extrapolation
gold validation dataset in the q-th round of EES, (1)) (Tmis, Vmise denotes the prompt used for error extrapolation. The prompt asks the LLM to syn- thesize data point similar to miss with label Ymiis In our implementation, we use the prompt: "Write positive movie review like The movie great." DD (9+1) denotes the + 1-th additional dataset we add synthesized on LLM based on extrapolating D)9)
We then describe the Error Extrapolation barrppllation-basedd Synthesis (EES) framework that attempts itera- tively reduce the distribution gap by extrapolating the errors of small model trained on the currently synthesized dataset on a small validation set. This is different from conventional data synthesis meth- ods, where the synthesized dataset is fixed after finishing the synthesis process and is used for train- ing the small model. Specifically, the EES process extrapolates errors made by small models the real-world validation datasets to synthesize some additional data to fix the error.
The key steps of the EES algorithm mis" train the small model with the current synthesized dataset (line 6) and utilize the LLM to extrapolate the misclassified data to generate more training data (lines 8-10). This creates a dataset that better reflects the underlying truth.
We use two different data sources in the EES pro- cess: the seed datasct (Dseed), and small human- labeled, real-world dataset referred to as gold data, denoted as D gold. In EES, we first divide the gold data into validation dataset ((aa)) and testing
In sum, the EES process reduces the distribution gap by using the misclassified data to model the distribution gap and using the LLM sample ad- ditional data points from This idea is similar doing optimization on the residuals in the gradient boosting literature (Friedman, 2002).
dataset D D(test). We use D ((md)
goid to find and fix the
distribution gap and use D gold
(test)
to judge the perfor-
2.4 Special process for multi-sentence task
mance of the small model.
For clarity, we focus on single-sentence tasks in our algorithm discussed before. When transition- ing to multi-sentence tasks, small modifications are necessary. Specifically, for complex tasks such as question answering. the context sentence can be excessively long, preventing our prompt from fit- ting LLM's input limit. Even when the prompt fits, generating rationales for each context sentence can be prohibitively costly. Hence, for these situations, resort to a more traditional seed data synthesis approach.
Algorithm 2: Algorithm for Error Extrapo-
lation
Input:
Dsced; Dloral) (ittcss)
gold gold .S PLLM, R, T(())
Output: Derain
D
+
add
for q in range(It) do
init(f); // reinitialize f (clear last round's train)
D(g)
train Deeed (UL=)D add)
tr ((1) )
Specifically, we perform dataset synthesis given set of conditional contexts C -1 , C. (e.g., premise in NLI and context & answer in QA task). We perform dataset synthesis follows:
D(g) + misclasss gold mis
(D(vva)) ((o) )
D ((++1) odd + 0
for each (Imis Ymis) E D(9)) do
Ludd ~ ( (Imis Umis))
1. Uniformly sample the current context Courr sentence from C, and current target label Yourr from all possible labels y. Com- bine them into seed data synthesis prompt T queryy (Courr: yeurr).
add + D 11++)
U {FFadds Vemis))
10
add
D Drrain Dseed ( add)
We present the whole process EES Alg. 2. One round in the for-loop beginning at line denotes one round of EES. R denotes the number of rounds of EES we want to perform; in our imple- mentation, typically do 2 rounds of experiments. f denotes the small model; (()) examples mis-classified by the small model on the
2. Synthesize the target sentence (c.g.. hypothe- sis in NLI and question in QA) from LLM by (nueyy(Courr. (Ceurr: (eeur). ). The synthesized data is
T queryy denoted as (Courr, Tays yeurr). ).
mis denotes the set of
Repeat the above steps until we have enough seed data seed = (Cyends X Y Vsred)

---

Dataset
We denote the probability space of the data ex- ample P (S,E): here, for simplicity, we wrap all possible elements variable s S, and the components in 8 can be varied depending on the specific task, for example, in the text classification task, i.e., = (x,y) where is a piece of text and is the corresponding label. Prompt Prompt
Label
Type
word (Y)
Tration
Imagine you are watching a movie; consider <X> reasons that may lead to <Y>
positive/
IMDb
impression of the movie.
negative
Tquery
Now imagine that you just watched movie that has <X>, Now you should
positive/
write <Y> review about this movie.
negative
T mis
Write a <Y> movie similar to: <X>
positive/
negative
Tquery
Given an information paragraph: <X> in Please ask question that has answers in/ not in
QNLI
<Y> the information paragraph
Given premise: <x["premise"]> 'n And here is question: <Xesquesio in/ not in that the amswer of question is <Y> the premise.InPleas write another question
mis
similar to the given question and have answers <Y> the premise.
Tquery
AnBasd on the above description, the following sentence is definitely <Y>; correct/ <x>
RTE
wrong
T
<X[" X'"premise]>> \nBased the above description, the following sentence:
correct/
<X| X[""yyothesis"">> is definitely <Y>, Now write a sentence similar to the given
wrong
sentence and is definitely <Y> based on the given description.
Tquery
Given context: <X[[context") \nX=["answer"] is the answer the following NA
AdQA
question:
T
Given a context:
<X["context">> "XX["e"s""r") is the answer to: NA
<X[""qeettinn">>ln question that has the same answer in the context is:
Table 1: Designed prompts for the four datasets. Tration denotes the prompt for the LLM to generate rationales. (1/2) Tquery denotes the prompt for seed data synthesis, and <X> denotes the rationale list context sentences for the current seed data example. T((1/2) denotes the prompt for EES, where <X> is the full misclassified example.
For the EES process, multi-sentence tasks, we only need to modify the for-loop beginning at line 8 in Alg. 2 to fit the multi-sentence task. The changed version of line 8 is shown in Alg. 3.
giving an analysis of the distribution gap problem and the effectiveness of our S3 framework.
data example into one
Algorithm 3: Multi-sentence EES, inner for-loop
for each (Cmis; Tmiis Vmica E D(9)
mis do
Radd m FLLM(- (- (2) (Cmis: Tiiss Umis))
D(u+1) add add U ((Cmiss Rodds }mnis)
We assume that the gold dataset (denoted { Pgoldd is obtained by i.idd sampling ngoodd times from a real-world distribution PD E P. Then, also assume the process of obtaining syn- thesized data example
2.5 Prompt engineering
i.i.d sampling from IP FLLM P. In the analysis section, for simplic- ity, we define IP ILLM as distribution over the data example set instead of the space of human lan- guage. This distinction is important because while text data is in natural language, for many tasks, labels may not be.
The design of prompts have huge impact the quality of the synthesized dataset. We present the prompt templates used for generating rationales, data points, and error extrapolation Table 1.
2.6 Theoretical Analysis
In this section, we give detailed analysis of why our S3 framework can shrink the distribution gap between zero-shot synthesis and real-world distri- bution by first clarifying the analysis setup and then
Similarly, we assume that the process of attain- ing the seed dataset (denoted as (S.) where n1 is the number of seed data points, is to draw ny i..d. samples from our seed data distribution

---

POO)
(Section 3.4), transferability study (Section 3.5) for the framework, and study on additional data quality (Section 3.6).
LLM
Let us first recall the origin of the distribution gap problem in dataset synthesis methods: conven- tional data synthesis methods, as well as the seed dataset synthesis stage in our approach, sample data points from fixed distribution PLLAs Since the distribution is fixed and different from the task data distribution Pp, the synthesized dataset suf- fers from fixed distribution gap no matter how much data we synthesize. Therefore, the testing performance of the small model trained on the syn- thesized dataset real task data is bounded by this gap. Our approach, S3, aims resolve this limitation.
3.1 Setup
3.1.1 Datasets
In this study, we evaluate our S3 on three major NLP tasks: text classification, Natural Language Inference (NLI), and Question Answering (QA). For text classification,
use the IMDb (Maas the et al., 201 1) dataset; for the NLI task, QNLI (Rajpurkar al., 2016; Wang et al., 2018) and the RTE (Bentivogli et al., 2009; Giampiccolo et al., 2007; Haim et al., 2006) dataset; for the QA task, we use the Adversarial QA (Bartolo et al., 2020) dataset.
Let us assume that the small model perfectly learns the synthesized dataset distribution. In this case, the error that the small model makes on the small gold validation dataset can represent the dis- tribution gap between PD and (00)
3.2 Baselines
We compare our S3 framework with the following baselines:
LLMM
Finally, we argue that a good LLM can perfectly extrapolate from the errors. This means that the LLM can synthesize samples from the difference between two distributions PD p(0) LLM Formally,
ZeroGen: ZeroGen is the basic data synthe- sis method proposed by Ye et al. (2022b). It neither uses rationales for data synthesis nor attempts to reduce the distribution gap. Note that ZeroGen also uses the same small valida- tion set for tuning hyperparameters.
the additional data synthesized in each round of the EES process follows:
P Padd PLLM( P0 LLLM)
(1)
2. GoldGen:
This bascline extrapolates the en- tire gold validation data instead of the errors made by the small model. We further use this baseline to test the effectiveness of the error extrapolation idea in the S3 framework. We keep the scale of synthesized datasets the same in order to make a fair comparison with S3.
Therefore, by sampling the same number of data points from Pald and combining them with the original seed data distribution LLLM> the mixed dataset shall follow the distribution:
P(1) LLM == p-
Padd + P-
p)D
LLM PD
(2)
where C [0 1] is the ratio of combination, can be intuitively understood as the portion of the additional dataset and seed dataset. This suggests that, thcoretically, we recover the gold data distribution by simply combining the original seed data and the additional data synthesized via BES. However, please note that we cannot guarantee the LLM and the training of the small model are perfect in real-world scenarios. Therefore, 53 re- peats this process iteratively to gradually reduce the distribution gap and optimize the mixed dataset until convergence.
ProGen: This baseline was proposed by Ye et al. (2022a), like the EES, it also considers training feedback. However, this framework is only available for text classification tasks, and it does not use LLM rationales for data synthesis.
4. Gold Data:
We also include baseline that trains the small model on the original gold data for reference.
3.2.1 Implementation details
3 Experiments
This section gives full implementation details of S3 in our experiments. We apply GPT3.5 derived from (Brown et al., 2020) as the LLM for all the synthe- sis work, and we use nucleus sampling (Holtzman
We conduct experiments to test the effectiveness our approach across three major NLP tasks over four datasets. We also do a thorough ablation study

---

Method
Data Size Results
IMDb
QNLI
RTE
Adversarial QA (EM/F1)
Average
Data Size
25k
105k
2.5k
30k
40.63k
Gold Data
Results
87.93
88.05
58.12
18.6/29.85
56.51
Data Size
100k
ProGen
Results
84. 12
Data Size
200k
200k
200k
200k
200k
ZeroGen
Results
84.28
71.19
59.93
6.33/9.96
46.34
Data Size
25k
150k
30k
80k
61.25k
GoldGen
Results
87.93
78.31
64.25
11.63/23.33
53.09
Data Size
21.2k
168k
33.6k
81.5k
76.08k
Results
89.00
79.92
73.29
12.50/24.38
55.73
Table 2:
Main experimental results. All compared methods evaluated by fine-tuning DistilBERT. The perfor- mance of fine-tuning the small model on gold data is in gray because it is not directly comparable with other results.
al., 2019) with a temperature of 0.9 for decod- ing. We use DistilBERT-base-nee
stead of the entire validation We find that S3 outperforms GoldGen with an average absolute per- formance improvement of 2.73%. This confirms the advantage error extrapolation over directly extrapolating gold data.
(Sanh et al.. 2020) provided by the Hugging Face Transform- ers library (Wolf et al., 2019) as the small model. We perform hyperparameter tuning on the batch size, learning rate, weight decay, and the number of epochs for fine-tuning the small model.
It is also noteworthy that S3 yields competitive results compared to directly fine-tuning the small model the full gold training data. Specifically, S3 even outperforms gold data performance on IMDB and RTE. This confirms the potential of applying S3 real-world applications.
3.2.2 Evaluation Method
For text classification and NLI tasks, we use the accuracy rate as the evaluation method. For QA tasks, we use Exact Match (EM) and Fl score as evaluation methods. To implement the experiment of S3 method, we utilize the training data from the original dataset as the gold evaluation data dataset EES (i.e.. D ((coal))
3.4 Ablation Study
3.4.1 Ablation of EES
We first ablate the error extrapolation- syn- thesis (EES) framework of S3, using only the seed data synthesized based on Section 2.2. We make that the scale of the training dataset is approx- imately the same for fair comparison. The result can be seen in Table This result proves the ef- fectiveness of view of the dynamic dataset and BES, We find that for more complex tasks like QA and NLI, our EES framework can give a larger improvement, which proves the distribution gap problem and our EES framework's 's ability to shrink this gap.
gold ). And we use testing data from the original dataset to test our model's perfor- mance.
3.3 Experimental Results
We present our main experimental results in Table 2. We observe that our S3 framework has huge improvement (an average improvement of 9.48%) compared to ZeroGen. The performance gap is especially large NLI and QA tasks. Moreover, we only use an average of 30.43% amount of data compared to ZeroGen, which can be considered as a significant improvement. Such an improvement proves the effectiveness of the initial seed data syn- thesis method and the idea to keep on optimizing the data in our S3.
3.4.2 Ablation of Seed Data Synthesis with Rationales
We then ablate the use of rationale for dataset syn- thesis in the S3 framework on the IMDb dataset. The results are shown in Table We find that us- ing rationale for dataset synthesis enables the LLM to generate datasets of higher quality that leads to
We then compare with the GoldGen base- line to test the effectiveness of extrapolating the errors of the small model on the validation set in-

---

Method
IMDb
QNLI
RTE
Adversarial QA
3.6 Additional data quality study
89.00
79.92
73.29
12.50/24.38
We perform this experiment to check the quality of the additional dataset synthesized by EES. Note that for carlier LLMs like GPT2 (Radford et al., 2019) or T5 (Raffel et al., 2020), there used to be tendency to repeat the prompt. If the LLM just repeats the misclassified data, then there is extrapolation. Thus, we composed experiments follows test the quality of the additional dataset: Sentence Encoding: For both misclassified data D mis and additional data Dadd. we use Distil- BERT to encode each Emis and Todd. This results in encoded sentences represented as Emis and Zadd respectively, and each encoded sentence is in IRd (with d = 768 in DistilBERT)
w/o EES
86.86
73.70
65.71
8.70/20.03
Table 3: Ablation test results (%) on iterative error ex- trapolation. The baseline w/o error extrapolation is fine-tuned on the same amount of data compared to S3.
better performance of the small model with lower budget, i.e., fewer synthesized examples.
with Rationale
w/o Rationale
Dataset Size
15k
40k
Results (%)
86.86
85.34
Cosine Similarity: Then, by comparing the cosine similarity between 2mis and zadd. we gauge their semantic similarity. High cosine similarity indicates substantial semantic overlap.
Table 4: Experiment result of ablation of rationales anal- ysis in seed data synthesis. The section with Rationale means we synthesize seed data guided by of LLM synthesized rationales, and w/o Rationale means the seed data is synthesized by the descripece prompt without rationale.
Edit Distance: Further, to understand textual distinctiveness, we compute the edit distance be- tween sentences Tiis and Fadd- If the edit distance approaches the sentence length, we infer that the texts differ significantly in their composition. The results are shown in Table 6.
3.5 Transferability of EES Data
We then test the transferability of the EES- synthesized data. The results are shown in Table 5.
Label
IMDb
QNLI
RTE
AdQA
In this test, we replace the seed dataset of our frame- work with the data synthesized by Ye et al. (2022b).
Data Num
6.173
51,100
1.522
51,332
Avg. Cos Sim
0.9497
0,9537
0.9380
0.9468
Avg. Edit Dist.
273.92
14.64
16.38
13.99
We do two sets of testing. We compare the variants where we directly add the EES data synthesized in S3 (+ourAdd) and that with the small model trained on the data synthesized by Ye et al. (2022b). We see that the two variants both lead to similar performance improvements. This shows that the EES synthesized data can effectively transfer to other zero-shot synthesized datasets. We believe this is because the distributional gap for different zero-shot data synthesis methods is similar. There- fore, the data synthesized by the EES method can be universally helpful, which further demonstrates the potential of S3.
Avg. Triis len
288 04
14.17
13.91
13.73
AVG. Fadd len
218.72
19.97
24.61
18.70
Table 6: Quality study of Additional Data
The average misclassified data length (avg Tmis len) and average generated data length (avg Fadd len) provide context to interpret edit distances. This result shows that while there is high semantic sim- ilarity among the misclassified data and the ad- ditional generated data (evidenced by the cosine similarity scores), the generated sentences are not mere copies of the misclassified samples (as their edit distance is almost the length of the whole sen- tence). This result provides extra evidence in favor of the quality of the newly generated data.
Method IMDb
QNLI
AdQA
ZeroGen 84.28
68.60
4.60/9.62
+ourAdd
87.50
73.51
9.70/20.10
Related work
+synAdd
87.41
72.21
10.27/19.92
4.1 Dataset Synthesis
Table 5: Transferability test result (%): where +ourAdd is ZeroGen dataset as seed data and S3 synthesized data as additional data, and +synAdd using EES on ZeroGen trained small model's misclassified data
The vast quantity of data required by the majority of Machine Learning methodologies has prompted numerous researchers to explore the concept of Dataset Synthesís. This aims to generate a dataset

---

from large pre-trained models, such as LLMs, in order transfer rich knowledge from large mod- els to small models. Initial attempts to achieve this used fine-tuned generative models to gener- ate data (Anaby-Tavor et al., 2020; Kumar et al., 2020). These efforts involved first fine-tuning the LLMs with small amount of human-annotacee data (gold data), then combining the generated data with gold data to train small models. Other re- searchers sought synthesize copious amounts of data for semi-supervised learning (Chen al., 2020; Wang et al., 2021). Nonetheless, these meth- ods are only suitable for straightforward text classi- fication tasks, proving data inefficient and ineffec- tive for more complex tasks like NLI or QA.
for dataset synthesis. S3 is novel dataset syn- thesis framework that shrinks the distribution gap between purely LLMs synthesized datasets and the real underlying data distribution. S3 achieves this by first using seed data synthesis with rationales to have a low distribution gap in seed data. It shrinks this distribution gap by iteratively extrapolating errors of the small model on small amount of real- world data. Extensive experiments on three major NLP tasks over four commonly used datasets show that compared with a representative baseline, S3 significantly improves the performance of a small model with averagely only one-third of synthesized data. S3 has high practical potential in many real- world applications because it can effectively (i.e, with better performance) and efficiently (i.e., with improved data efficiency) transfer knowledge in extremely large model (e.g.. GPT 3.5) to a small model (e.g., DistilBert), achieving data efficiency and computation efficiency at the same time.
The potential of zero-shot performance offered by LLMs has led some researchers to consider zero-shot dataset synthesis based on non-finetuned LLMs (Meng et al., 2022; Ye et al., 2022b). How- ever, as indicated by 1ig1, direct querying of non- fine-tuned LLMs often results in data that suffers from large distribution gap and is typically in- efficient. Thus, some studies have attempted data selection (Gao et al., 2023) or data augmentation (Ye al., 2022a). However, their capacity to rectify the distribution gap leaves room for improvement.
Acknowledgments
We thank the anonymous reviewers for their feed- back on our paper. MS acknowledges support from the Swiss National Science Foundation (Project No. 197155), Responsible Al grant by the Hasler- stiftung; and an ETH Grant (ETH-19 19 21-1).
4.2 In-context Learning
Brown et al. (2020) suggests LLMs can better learn the task they are working on by conditioning on a few examples in the prompt. This paradigm, known In-context learning, is particularly appealing as it negates the necessity of updating the parame- ters of LLM. Subsequent research has focused on optimizing the choice of prompt templates and in- context examples (Liu al., 2021; Wang et al., 2023; Lu et al., 2021), and learning with in-context objective descriptions (Chen et al., 2021). The key idea for in-context learning is to learn from analogy (Dong et al., 2022), which aligns with our idea of extrapolating error to synthesize additional data to fill the distribution gap. However, most in-context learning methods are designed for a few-shot set- ting, whereas our research, the LLM does not need to be trained. We explore the LLM's ability to directly extrapolate from errors, providing a crucial example for creating a more effective dataset.
Limitations
Although S3 achieved promising results, there are still several limitations of our work. The first lim- itation is that in the experiments, we spotted that a tiny change in the synthesis prompts can lead to a significant performance drop, which means our framework is not prompt-stable. A possible future direction is to develop systematic way to compose prompts that can perform stably well by fine-tuning LLM using good prompts. The sec- ond limitation is that S3 assumes that the LLM has rich knowledge of the specific task. But in the actual application of the approach in the real-world, there is no such guarantee. A possible solution to mitigate this limitation is to ask the LLM to divide the previously unseen task into multiple simple tasks that the LLM has a good understanding of, but it also requires the LLM to have good ability to understand the subtasks. The third limitation is that S3 is task-specific. Future work may try extend the method to cross-task settings to further improve the computational and data efficiency of the method.
5 Conclusion
This paper proposes the Synthesis Step by Step (S3) approach based on a dynamic dataset viewpoint

---

References
Liang, Zhenguo Li, and Lingpeng Kong. 2023. Self- guided noise-free data generation for efficient zero- shot learning. In The Eleventh International Confer- ence on Learning Representations.
Ateret Anaby-Tavor Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 7383-7390.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paruphrasing, pages 1-9.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebas- tian Riedel, and Pontus Stenetorp. 2020. Beat the ai: Investigating adversarial human annotation for read- ing comprehension. Transactions of the Association for Computational Linguistics, 8:662-678 678.
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Damilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop on Recognising Textual Entailment, volume 7.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC. Citeseer.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.02331.
Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Mar- keeva, Rohan Anil, and Alexander Kolesnikov. 2022. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEF/CVF Confer- ence Computer Vision and Paltern Recognition, pages 10925-10934
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv: 1904.09751.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1
Cheng- Yu Hsich, Chun-Liang Li, Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprini 2305:2230002300
Ting Chen, Simon Kornblith, Kevin Swersky, Moham- mad Norouzi, and Geoffrey Hinton. 2020. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243-22255.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained trans- former models. arXiv preprini arXiv:2003.02245
Bohan Li, Yutai Hou, and Wanxiang Che. 2022. Data augmentation approaches in natural language pro- cessing: A survey. AT Open, 3:7 1-90
Yanda Chen, Ruiqi Zhong. Sheng Zha, George Karypis, and He He. 2021. Meta-lcaming via language model in-context tuning. arXiv preprint arXiv: 10.007811.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06800
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 023331
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint 1xXiv:2144008788
Terrance DeVries and Graham W Taylor. 2017. Dataset augmentation in feature space. arXiv preprint
arXiv:1702.05538.
Andrew Maas, Raymond Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Qingxiu Dong, Lei Li. Damai Dai. Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint urXiv:2301.002.
Jerome Friedman. 2002. Stochastic gradient boosting. Compulational statistics & data andlysis, 38(4):367- 378.
Meng. Jiaxin Huang, Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. arXiv preprint arXiv:2202.04538.
Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong WU, WEIZHONG ZHANG, Xiaodan
OpenAL. 2023. Gpt-4 technical report.

---

Raul Puri. Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2020. Training ques- tion answering models from synthetic data. arXiv preprint (tt::20000099999
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5788.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. 2019. Language models unsupervised multitask learners. OpenAl blog. 18)::
Xinyi Wang, Wanrong Zhu, and William Yang Wang. 2023. Large language models are implicitly topic models: Explaining and finding good demon- strations for in-context learning, arXiv preprint arXiv:230.111916.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, (1(1):5485-5555
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et 2019. Huggingface's 's transformers: State-of- the-art natural language processing, arXiv preprint 191.03791..03771.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Perey Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.02250.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020. BERT-of-theseut Com- pressing BERT by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859-7869, Online. Association for Computa- tional Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Bernadette Sharp. Florence Sedes, and Wieslaw Lubaszewski. 2017. Cognitive approach natural language processing. Elsevier.
Jingjing Xu, Wangchunshu Zhou, Zhiyi Fu, Hao Zhou, and Lei Li. 2021. A survey on green deep learning.
Connor Shorten and Taghi M Khoshgoftaar. 2019. survey image data augmentation for deep learning. Journal of big data, 6(1):1-48.
Jiacheng Ye, Jiahui Gao, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022a. Progen: Pro- gressive zero-shot dataset generation via in-context feedback. arXiv preprint arXiv:2210.12322
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. In Findings of the Associa- tion for Computational Linguistics: ACI. 2023, pages 7059-7073.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng. Zhiyong Wu, Tao Yu, and Lingpeng Kong, 2022b. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint arXiv:2202.072: 07922...
Ryan Smith, Jason A Fries, Braden Hancock, and Stephen H Bach. 2022. Language models in the loop: Incorporating prompting into weak supervision. arXiv preprint 2205::2205.02318.
Wangchunshu Zhou, Ronan Le Bras, and Yejin Choi. 2023. Modular transformers: Compressing trans- formers into modularized layers for flexible efficient inference. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 10452-10463 Toronto, Canada, Association for Computational Lin- guistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 18330-18344 Curran Associates, Inc.
Laurens Van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-sne. Journal of machine learning research, 9(11).
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint DoXiv:180.00445
A Intuitive understanding to EES
Since the pseudo-code of EES may be somewhat non-intuitive to understand, this part aims to pro- vide an intuitive understanding of the EES method on single-sentence tasks.
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help. a08 13487.
arXiv preprint

---

dataset D We repeat the Attain Error and Synthesis on extrapolating emor steps for multi- ple rounds until the error converges. With such a method, we can optimize our synthesized dataset step by step to attain a dataset with lower distri- bution gap by utilizing the information provided by extrapolating errors that represent the distribution gap.
A.1 Attain Error
The first step for EES is to attain the error made by the small model on the gold validation dataset, which is, to a certain extent, the representation of the distribution gap between LLM's seed data syn- thesis distribution and the real-world distribution. To attain the error, we must first train the small model with currently synthesized data. This in- cludes the seed data Dseed; and additional datasets D D(9) D(9)) odd
B Computation complexity comparison between S3 and ZeroGen
add where q the current round of iteration. Then we have D(0) add 0. Thus, the training dataset for q-th iteration is:
This section studies the total computation cost of the S3 framework. We compare the number of floating-point operations (FLOPs) involved in fine- tuning the model with S3 and ZeroGen synthesized dataset. For the BERT family of models, accord- ing to Brown et al. (2020), they cost 6 FLOPs per token per parameter (i.e., Proken.para = 6) in train- ing- The DistilBERT model (Sanh al., 2020) has Ppparo 66 10 parameters and the typical input length for one record is numrre Therefore, the training FLOPs per record of data per epoch is:
Deeed U (Ul-oD add) D(g) train
(3)
Then, we train the small model with D(() train. We denote the fitted small model as (( train). Then, we evaluate the fitted small model on the gold val- idation dataset and obtain the data samples with high error in the validation dataset:
(token) 512.
(19) = misclasss fD (eval) mis
gold (Derain)) }
(4)
where the misclass function denotes the function that attains the data samples that have been misclas- sified. For instance, for the QA task, this can mean data samples that do not have exact match with the answer or data samples with low Fl scores. We represent the distribution gap between the underly- train by the misclassified gold (4)
Frec F Ftokon.para numm (token) * Ripara
=2.03 x 10
The ZeroGen method typically uses 200k records of data and trains for average of 10 epochs to achieve the best results based on our ex- periments. Thus, the total fine-tuning cost terms of FLOPs for ZeroGen is:
ing truth and the D (g)
evaluation dataset miss which is the distribution gap in g-th round of EES.
A.2 Synthesis on extrapolating error
FzeroGen Free 200k 10 4.06 10
After having ((1) for all the misclassified data mis:
(Imis, Umie) E ()9) we query the LLM again using a prompt that wraps information of the mis- classified data. The prompt T (Imiir Mmis) intu- itively asks the LLM extrapolate the misclassi- fied data and synthesize a new data example. For example, in the movie classification problem, if the current misclassified data is: (The move is great, positive); our original (((1D()) ( labeled such review
In S3, in the first round of fine-tuning (using only the seed data), the dataset size is 51.2k records on average (i.e., seed dataset is about 2/3th size of final dataset). After round of EES, the total dataset size becomes 64.0k (i.e., 5/6th size of the final dataset). The final round of fine-tuning with two EES additional datasets and the seed dataset that have a total size of 76.8k records of data. On average, our method needs 8 epochs to achieve its best result. Therefore, the total number of FL.OPs of fine-tuning DistilBRRT for the 3 iterations (2 for getting misclassified data, for final fine-tuning) in our S3 is:
negative one. In this case, mis (Imis; Vmiss) can be something like Generate a positive movie review like The move is great.
T
We query the LLM with mis (Imisr Mmis), to obtain another data example similar to the error.
This process is repeated for every misclassified data point. Thus, we obtain the + 1-th additional
Fs3 = =F * (51.2k + 61.0%: + 76.8k)
=3.11 17

---

label Generated Movie Review
Negative If you're looking for film that will make you cringe in disbelief, look further than "Zombie Cheerleader Apocalypse." This movie is the epitome of a low-budget, poorly-made production. The acting atrocious, and the jokes are so bad that they will make you want to gouge your eyes out. The story revolves around a group of cheerleaders who are turned into zombies after a failed science experiment. They then proceed to wreak havoc on their school and town, killing anyone in their path. The supposed "hero" of the film is a nerdy guy who has crush on the head cheerleader and tries to save her from the zombic apocalypse. The special effects are laughable, and the makeup for the zombies is so poorly done that it's hard to tell if they'r re supposed to be scary or comical. The dialogue is cringeworthy, and the attempts humor fall completely flat. The film tries to be parody of horror movies, but instead comes off as cheap knockoff. Overall, "Zombie Cheerleader Apocalypse" is complete waste of time. It's not even bad a fun way, it's just plain bad. The only reason to watch this movie is you want to torture yourself for 90 minutes, otherwise, avoid it at all costs.
Positive
"The Family Man" is a thrilling and heartwarming movie that will keep you at the edge of your seat. The movie revolves around Srikant Tiwari (Manoj Bajpayee), a senior analyst in the National Intelligence Agency, who is trying balance his demanding job and his personal life with his wife Suchitra (Priyamani), and their two children. The movie showcases the struggles of maintaining normaley in Srikant's 's life, while also juggling intense missions. The action sequences are brilliantly choreographed, with twists and turns that keep you guessing till the very end. Bajpayee's performance is outstanding, as he brings a depth and complexity to his character. The chemistry between him and Priyamani is natural and heartwarming, adding a layer of emotion to the story. The supporting cast is equally impressive, with Sharib Hashmi and Shreya Dhanwanthary delivering memorable performances. The movie also explores important themes such as the impact of terrorism, family values, and duty towards one's country. The director, Raj Nidimoru and Krishna D.K., have done remarkable job of balancing the action with the human story, making it a compelling watch for all audiences. Overall, "The Family Man" is must-watch for anyone looking for a thrilling and entertaining ride, with a strong message of family and patriotism."
Positive If you're re a rom-com fan, then you'll love this movie! It's a fun, light-hearted take on the classic boy-meets-girl trope, with plenty of laughs and heartwarming moments to keep you entertained. The chemistry between the two leads is great, and the supporting cast is equally charming. While some may criticize the predictable plot and lack of originality, I found it to be comforting and enjoyable watch. The soundtrack is also a standout, with catchy songs that perfectly capture the mood of each scene. highly recommend this movie for anyone looking for a feel-good, easy watch. 8/10.
Negative This movie is complete waste of time. The story is nonexistent and feels like the director just slapped together random scenes. The characters are forgettable and lack any depth or development. The pacing is so fast that it's hard to keep up with what's 's going on. It's like watching poorly made music video repeat. The only redeeming quality of this film is the cinematography, but even that can't 't save it from being complete mess. It's clear that the director has some talent behind the camera, but they need to work on their storytelling wouldn't 't recommend this film to anyone. Save your money and go watch something else. Let's hope the director can learn from their mistakes and create something better in the future.
Table 7: Examples of generated IMDb data

---

LoN gnd Duloet
LGBE AADA Deiseet
Figure 3: t-SNE result for QNLI (left), RTE (center), AdQA (right) for dataset diversity analysis. ZeroGen data's points are plouted in Yellow, S3's in Green, and Gold data in Purple.
Dataset
S3 Coverage
ZeroGen Coverage
Maaten and Hinton, 2008). The final steps of our analysis are as follows:
QNLI 76.35
63.03
RTE 73.59
14.90
AdQA 51.02
46.00
1. Uniformly sample a similar amount of data from gold data, S3 synthesized data, Zero- Gen synthesized data. We have D
Table 8: Coverage rate (%) of S3 and ZeroGen
gold
1 ygold and D DeroGen
(n (i) De
D's (k)
{-s)
(1)
Method IMDb
QNLI
RTE
AdQA
353 1=1
Gold Data
92.30
91.00
71.50
22.97/36.55 59
ag {ar ZeroGene VeeroGen k=1
ZeroGen
83.66
70.11
72.2
5.07/10.74
where 721, 122 1& should be similar.
89.55
85.20
76.17
20,50/34.40
2. Encode the sentences using DisTiLBERT. Then, have the sentence encodings:
Table 9: Apply S3 framework on MiniLM
{ (6) 18g { {253 f==1:
[ 1n2 { [ZerrGGen
(R) 1 S Rd.
"gold
To conclude, due to fewer rounds of fine-tuning epochs and the lower need for data, S3 uses only 3/4th the number of FLOPs compared to the Zero- Gen baseline, even though we fine-tuned the model multiple times.
where d is the hidden state's dimension (in our case, it is 768).
3. Perform t-SNE the encoded data (k)
12 {2(0) ss U {2s3 J==1 ZezerGGen / k= i=
reduce the dimension from
2. We have: 1 I-SNE(2) = = {p gold mn U (pss Jj=1
Dataset Diversity analysis for S3
This section analyzes the diversity the synthe- sized sentences. Such an analysis necessary as the LLMs may generate sentences with similar meanings, rendering the dataset lacking in diver- sity. As there is no universally approved method for analyzing dataset diversity, titative and qualitative methods to analyze dataset diversity:
{p 1PzeraGen mg
R
4. Draw the reduced dimension points scat- ter plot to directly see the overlap of our syn- thesized dataset and the Gold data. We show the results in Fig. 3. We can see that the green region significantly aligns with the purple re- gion, which indicates that S3 results in similar data diversity as the gold data.
both quan-
C.1 Quantitative Analysis:
Data diversity can also be quantified by count- (k)
For short synthesized sentences, such as the QNLI, RTE, and AdQA datasets, we approach the dataset analysis quantitatively. Given the high hidden di- mension of the sentence encoding (e.g.. 768 for Dis- tiBERT)) direct analysis can be inefficient. Hence, we used t-SNE for dimension reduction (Van der
ing how many points of p gold are in the area of Asi
U and AzeroGen k=1 (ZZeoGGG))) where By(p) represents a solid circle with center and radius 7. The results for QNLI, RTE, and AdQA are shown Table 8.

---

The results further demonstrate the superior cover- age and diversity of our S3 framework compared to ZeroGen.
C.2 Qualitative Analysis:
For tasks that require the generation of longer texts, the text encoding approach not amenable to t- SNE dimension reduction and interpretation. Thus, in such settings, we conduct qualitative analysis. We show examples of the generated data for the case of sentiment classification of IMDB reviews in Table We can observe that these examples exhibit rich contexts and diverse patterns, which supports the superiority of our S3 framework. For more qualitative results, please refer to the dataset in our project repository.
D Additional Results for S3 with MiniLM
addition DistilBERT, we also evaluated the performance of the Synthesis Step by Step (S3) framework using MiniLM (Wang et al., 2020) as the small model. The results of this experiment are presented in Table 9. Notably, there is a substantial enhancement in performance when compared to the ZeroGen baseline in all the tasks. Moreover, in tasks like RTE which lack data, method even surpasses the performance of the model trained gold data. These results provide robust evidence that the effectiveness S3 is not limited to a spe- cific model. Instead, it offers consistent improve- ments across different small models, underscoring its broad applicability and efficacy.


Reviews:
[{"review_id": "4Jnb4ESHmh", "reviewer": "Reviewer_UzEm", "summary": "", "questions": "1. Won’t the model still perform better without the initial seed dataset? We directly start with evaluating the model on the eval gold set and then iteratively create samples similar to {x_mis, y_mis} and finally keep synthesizing the samples very close to the distribution of misclassified samples to improve performance.\n2. Are there any checks for ensuring the quality of new samples generated? How different are the{x_add, y_mis} from the {x_mis, y_mis}?\n3. It would be beneficial if there were a section about dataset diversity. How similar are the samples to each other? There has been a growing concern that human-not-in-loop data synthesis can lead to samples that are not an exact match but are very similar. For example: “I really enjoyed the new XXX movie” vs “I liked the XXX movie”.\n4. Will there be an overlap in the new samples created owing to the constant buffer of rationales r_i that was queried from the LLM? Since we uniformly sample from the same rationale set, there could be significant overlap. Are there checks in place to ensure such scenarios do not occur?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UzEm\n\nThanks a lot for your review. Here are the answers to your question.\n\n#### Response to the High Cost of S3:\n\nThank you for your feedback regarding the perceived computational cost of our method. You have highlighted concerns about the fine-tuning of DistilBERT multiple times, suggesting it might be resource-intensive. Let me address this concern with a detailed breakdown:\n\nWe compare the floating-point operations (FLOPs) involved in fine-tuning the model with the S3 and ZeroGen synthesized datasets. For the DistilBERT model we used, based on OpenAI's report [1], the BERT family model costs $6$ FLOPs per training per token per parameter (i.e., $FLOP_{token, para} = 6$.). And based on [2], the DistilBERT has $numPara = 66 \\times 10^6$ parameters. And the typical token length for one record is $ numToken_{rec} = 512$. Therefore, the training FLOP per record of data per epoch is:\n\n$\nFLOP_{rec} = FLOP_{token, para} * numToken_{rec} * numPara = 2.03 * 10^{11}\n$\n\nFor ZeroGen, it typically uses $200k$ if data records and trains for an average $10$ epochs to achieve the best results based on our experiment. Thus, the fine-tuning cost for ZeroGen is:\n\n$\nFLOP_{ZeroGen} = FLOP_{rec} * 200k * 10 = 4.06 * 10^{17}\n$\n\nFor our S3 method, in the first round of fine-tuning (using only seed data) dataset size is typically in $2/3$ size of final dataset (i.e. $51.2k$ records), and the second round of fine-tuning (using seed data + misclass generated data of first round) is in $5/6$ size of final dataset (i.e., $64.0k$), and the final round of fine-tuning (using seed data + 2 rounds of misclass generated data) have $76.8k$ data. And on average, our method needs $8$ epochs to achieve its best result. Therefore, the total FLOPs of fine-tuning DistilBERT for 3 times (2 for getting misclassified data, 1 for final fine-tuning) in our S3 frameworks are:\n\n$\nFLOP_{S3} = FLOP_{rec} * (51.2k + 64.0k + 76.8k) * 8 = 3.11 * 10^{17}\n$\n\nTo conclude, we can see that our S3 methods use only 3/4 FLOPs compared to the ZeroGen baseline, even though we fine-tuned the model with multiple times.\n\n\n#### Answers to Questions For The Authors:\n\n##### Response to the effect of the model without seed dataset (Question 1):\n\nRegarding the question, \"Won't the model still perform better without the initial seed dataset...\":\n\nWe have indeed experimented with this approach, and unfortunately, the results were suboptimal. When we rely exclusively on training the smaller model without an initial seed dataset, it tends to overfit rapidly. We therefore conclude that the model needs the seed dataset to warmup. Otherwise it tends to significantly overfit the gold validation data.\n\n\n##### Response to the quality check of $x_{add}$ generated by $x_{mis}$ (Question 2):\n\nRegarding your inquiry about the quality assurance of newly generated samples and their distinction from misclassified samples:\n\nWe undertake a comprehensive post-hoc assessment to elucidate the distinctions between ${x_{add}, y_{mis}}$ and ${x_{mis}, y_{mis}}$. The objective is for the LLM-generated data to retain semantic similarity, while differing substantially in terms of word composition. Our analysis follows these steps:\n\n1. **Sentence Encoding**:\n   For both misclassified data $D_{mis}$ and the additional generated data $D_{add}$, we employ DistilBERT to encode each $x$. This results in encoded sentences represented as $z_{mis}^{(i)}$ and $z_{add}^{(i)}$ respectively, where each encoded sentence resides in $\\mathbb{R}^{d}$ (with $d = 768$ for DistilBERT).\n2. **Cosine Similarity**:\n   By comparing the cosine similarity between $ z_{mis}^{(i)} $ and $ z_{add}^{(i)} $, we gauge semantic similarity. High cosine similarity indicates substantial semantic overlap.\n3. **Edit Distance**:\n   To understand textual distinctiveness, we compute the edit distance between sentences $ x_{mis}^{(i)} $ and $ x_{add}^{(i)} $. If the edit distance approaches the sentence length, we infer the texts differ significantly in their composition.\n\nThe results from our evaluation, split across two rounds of data generation, are tabulated below:\n\n| Rounds  | Label         | IMDb   | QNLI   | RTE    | AdQA   |\n| ------- | ------------- | ------ | ------ | ------ | ------ |\n| Round 1 | data num      | 3,291  | 28,532 | 883    | 26,605 |\n|         | cos sim       | 0.9445 | 0.9536 | 0.9367 | 0.9469 |\n|         | edit distance | 289.07 | 14.60  | 16.44  | 13.98  |\n|         | avg mis len   | 282.45 | 14.20  | 13.75  | 13.76  |\n|         | avg gen len   | 258.42 | 19.88  | 24.54  | 18.71  |\n| Round 2 | data num      | 2,882  | 22,568 | 639    | 24,927 |\n|         | cos sim       | 0.9557 | 0.9538 | 0.9397 | 0.9467 |\n|         | edit distance | 265.61 | 14.68  | 16.29  | 14.01  |\n|         | avg mis len   | 294.42 | 14.14  | 14.13  | 13.69  |\n|         | avg gen len   | 173.40 | 20.09  | 24.70  | 18.69  |\n\nThe supplementary information on average misclassified data length (avg mis len) and average generated data length (avg gen len) provides context for the edit distances. This data unequivocally shows that while there's high semantic similarity (evidenced by the cosine similarity scores), the generated sentences are not mere copies of the misclassified samples. Thus, ensuring the quality of the newly generated data.\n\n##### Response to check of dataset diversity (Question 3):\n\nWe agree that dataset analysis is beneficial for dataset synthesis. However, there is no clear definition of \"dataset complexity\", which led us to exclude such an analysis. Nevertheless, we conduct analysis on the dataset diversity both quantitatively and qualitatively:\n\n**1. Quantitative Analysis:**\n\nFor short synthesized sentences, as seen in QNLI, RTE, and AdQA datasets, we approached dataset analysis quantitatively. The brevity of these sentences means less dense information in the sentence encoding. Considering the high hidden dimension for encoding (e.g., 768 for DistilBERT), direct analysis can be inefficient due to the \"curse of dimensionality\". Hence, we used t-SNE [3] for dimension reduction:\n\n1. Uniformally sample similar amount of data from gold data, S3 synthesized data, ZeroGen synthesized data. We have $ D_{gold} ' = \\{(x_{gold}^{(i)}, y_{gold}^{(i)}) \\}$ ($i$ from 1 to $n_1$), $D_{S3}' =  \\{(x_{S3}^{(j)}, y_{S3}^{(j)}) \\}$ ($j$ from $1$ to $n_2$), and $D_{ZeroGen}' =  \\{(x_{ZeroGen}^{(k)}, y_{ZeroGen}^{(k)}) \\}$ ($k$ from 1 to $n_3$), where $n_1, n_2, n_3$ should be similar.\n   \n2. Then encode the sentences by sentence encoder, we have $\\{z_{gold}^{(i)} \\}$, $\\{z_{S3}^{(j)} \\}$, $\\{z_{ZeroGen}^{(k)} \\}\\subseteq \\mathbb{R}^d$, where $d$ is the hidden state's dimension\n   \n3. Perform t-NSE on dataset $\\{z_{gold}^{(i)} \\} \\cup \\{z_{S3}^{(j)} \\} \\cup \\{z_{ZeroGen}^{(k)} \\}$ for all $i, j, k$ to reduce its dimension from $d$ to $2$. Name the processed data as $\\{p_{gold}^{(i)} \\}$, $\\{p_{S3}^{(j)} \\}$, $\\{p_{ZeroGen}^{(k)} \\} \\subseteq \\mathbb{R}^2$\n   \n4. Draw the reduced dimension points on a scattered plot to directly see the area of cover of our synthesized data. Because in the rebuttal section of openreview, we can not show the plots, you can check the plots of these datasets at this anonymous github https://anonymous.4open.science/r/S3_DiversityAnal-4AC4/README.md. Another way to show the plot's result is count how many points of $\\{p_{gold}^{(i)}\\}$ is in area of $A_{S3} = \\cup_{j = 1}^{n_2} B_\\gamma (p_{S3}^{(j)})$ and $A_{ZeroGen} = \\cup_{j = 1}^{n_3} B_\\gamma (p_{ZeroGen}^{(j)})$ where $B_\\gamma (z)$ means the solid circle with center $z$ and radius $\\gamma$. The results of the QNLI, RTE, and AdQA are as follows:\n\n| Dataset | S3 Coverage Rate | ZeroGen Coverage Rate |\n| ------- | ---------------- | --------------------- |\n| QNLI    | 76.35%           | 63.03%                |\n| RTE     | 73.95%           | 14.90%                |\n| AdQA    | 51.02%           | 46.00%                |\n\nThis demonstrates the superior coverage and diversity of our S3 framework compared to ZeroGen.\n\n**2. Qualitative Analysis:**\n\nFor longer text classification tasks, where sentence encoding results in dense information vectors that aren't amenable to t-SNE reduction, we conducted qualitative analysis. Here are a couple of diverse examples of generated data:\n\n- Example 1: \"By chance, I stumbled upon this gem of a film and was completely entranced from beginning to end. As someone who appreciates good acting, I was blown away by the performances which felt so natural and unforced that I had to question whether it was scripted or improvised. The story itself was captivating and kept me invested throughout, leaving me wanting more even after the credits rolled. After a quick search on IMDb, I discovered that the writer-director has a strong track record with their previous film, Everyday People, which I am now eager to watch. It boggles my mind that a movie of this caliber can go unnoticed in the vast sea of blockbusters. This film deserves recognition and a wider audience. I implore everyone to seek it out and experience the magic for themselves.\"\n- Example 2: \"While the cinematography in Mann's photographs of the Alberta Rocky Mountains is impressive, the rest of the film is sorely lacking. The acting, particularly from Jimmy Stewart and Walter Brennan, feels flat and uninspired. But what really ruins the entire experience is the absurd plot. A Mountie suggesting the people of Dawson City elect a marshal? And then gunfights breaking out on the streets? It's a laughable attempt to recreate the American Wild West in the Canadian North. Anyone with even a cursory knowledge of the Klondike gold rush will be shaking their head in disbelief at the ridiculousness of it all. Save yourself the time and skip this disaster of a film.\"\n\nThese examples exhibit rich contexts and diverse patterns. For an exhaustive list, please refer https://anonymous.4open.science/r/S3_Datasets-7421/README.md.\n\n##### Response to potential overlap problem of rational-based synthesized data (Question 4):\n\nThe process we use for dataset synthesis with rationales is designed to minimize the potential for overlap. Let's break down the reasoning:\n\n1. **Theoretical Analysis**: Each label's rationale set comprises 300 distinct rationales. For every dataset synthesis, we select 3 rationales from this set. This results in a total of $\\binom{300}{3}$ (which equals 4,455,100) unique combinations of rationales. The vastness of this search space makes it improbable to generate a dataset with significant overlap.\n\n2. **Randomness in Generation**: Our synthesis method uses a high temperature for generation, introducing a significant degree of randomness. This ensures that even if we use similar rationales, the generated outputs will differ considerably.\n\nConsider the following example where we used the exact same list of rationales: \"low quality cinematography\", \"lack of attention to detail\", and \"too derivative of other movies or TV shows\".\n\n- *Review 1*: \"The movie review critiques the film's unoriginality and reliance on imitating existing content. The movie doesn't offer anything unique, drawing heavily from other cinema and TV shows.\"\n\n- *Review 2*: \"A major flaw in the movie is its lack of innovation. The movie leans on existing plots and themes from previous content, resulting in a viewing experience that feels recycled and predictable.\"\n\nBoth reviews convey similar sentiments, but the composition and phrasing are distinct. This illustrates our point: even with the same rationale, the generated data is diverse in both semantics and wording.\n\n#### Answers to low reproducibility:\nSince there are some concerns about our works' reproducibility, we decided to publish our synthesized dataset on anonymous git via https://anonymous.4open.science/r/S3_Datasets-7421/README.md.\n\n\n[1] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... \\& Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.\n[2] Sanh, V., Debut, L., Chaumond, J., \\& Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.\n[3] Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of machine learning research, 9(11).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposed a novel approach for synthetic data generation by identifying the rationale of the labels.\n2. Proposed a novel error extrapolation-based synthesis to lower the distribution gap.\n3. Thorough experimental evaluation has been conducted on real-world datasets along with ablation studies.", "reasons_to_reject": "1. As shown in Algorithm 2 and Table 2, the paper suggests fine-tuning the small model, R (=2) number of times to extrapolate the error and reduce the dataset distribution gap for ~9% performance gain. This seems to be a costly approach in itself even though DistilBERT is being used.", "questions_to_authors": "1. Won’t the model still perform better without the initial seed dataset? We directly start with evaluating the model on the eval gold set and then iteratively create samples similar to {x_mis, y_mis} and finally keep synthesizing the samples very close to the distribution of misclassified samples to improve performance.\n2. Are there any checks for ensuring the quality of new samples generated? How different are the{x_add, y_mis} from the {x_mis, y_mis}?\n3. It would be beneficial if there were a section about dataset diversity. How similar are the samples to each other? There has been a growing concern that human-not-in-loop data synthesis can lead to samples that are not an exact match but are very similar. For example: “I really enjoyed the new XXX movie” vs “I liked the XXX movie”.\n4. Will there be an overlap in the new samples created owing to the constant buffer of rationales r_i that was queried from the LLM? Since we uniformly sample from the same rationale set, there could be significant overlap. Are there checks in place to ensure such scenarios do not occur?", "paper_topic_and_main_contributions": "The paper proposes a novel strategy to synthesize data that follows a distribution close to the gold dataset using LLMs by determining the rationale of a task and further using the error extrapolation synthesis to bring the distribution of the seed dataset closer to the gold dataset. From their experimental evaluation on several datasets, the proposed approach exhibits significant performance gains by smaller models as compared to previous synthetic data generation work. They also perform a theoretical analysis of the effectiveness of the proposed approach.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Missing_References": "1. GeneratingTraining Data with Language Models: Towards Zero-Shot Language Understanding (https://arxiv.org/pdf/2202.04538.pdf) - SuperGen\n2. SELF-GUIDED NOISE-FREE DATA GENERATION FOR EFFICIENT ZERO-SHOT LEARNING (https://arxiv.org/pdf/2205.12679.pdf) - SunGen"}}, {"review_id": "xHP5JH4Ags", "reviewer": "Reviewer_s5eU", "summary": "", "questions": "1. Did you test with R>2 rounds of training? (line 221)\n1. Why is there such a difference in dataset size for GoldGen and S3 for QNLI (Table 1), when the authors say the size is kept comparable (line 377)? How is the performance of these models at equal size of data? (given that scores are very close)?\n1. (Line 396) Could you specify the range of hyper-parameter search? Was the same search carried out for all the methods?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer s5eU:\n\nThanks a lot for your review, here is some further explaination of this paper:\n\n#### Answers to Reasons To Reject:\n\n##### Response to the redundancy of the theoretical analysis section: \n\nWe sincerely value the reviewer's perspective regarding our theoretical analysis section. While we acknowledge the merit in some of the concerns raised, our primary objective with this section was to offer an intuitive foundation for our methodologies, articulated through a mathematical lens. We did not intend to present these as unequivocal proofs, but rather as a means to provide readers with a deeper, high-level understanding of our approach and its underlying rationale. Given this intent, we believe that such a section enhances the comprehensiveness of our work rather than rendering it redundant. But as the reviewer points out, changing the name of this section can make the paper clearer, also, we will make this section more concise and may consider to move it to Appendix of the review consider it’s appropriate.\n\n##### Response to similarity between rationals method and chain-of-thought:\n\nWe recognize the comparison drawn between our \"rationales\" approach and the chain-of-thought style prompting as discussed in [1]. However, the core intent and execution of these methods are distinct. In our approach, the LLM is prompted to cultivate a comprehensive understanding of the overarching task set before it (as grounded by the rationale list). This acquired knowledge then informs subsequent data synthesis. In contrast, the chain-of-thought technique directs the LLM to tackle specific problems in a step-by-step progression.\n\nThat’s to say, while our \"rationales\" method emphasizes a holistic grasp of the task, the chain-of-thought focuses on step-by-step problem-solving. We therefore believe they are different approaches. That being said, using chain-of-thought prompting for dataset synthesis is also not done before and is also a novel approach.\n\n##### Response to experiments only done with one small model:\n\nWe acknowledge the reviewer's concern regarding the restricted scope of our experiments by focusing solely on DistilBERT. To provide a more comprehensive evaluation of the S3 framework and to address this limitation, we have extended our experiments to encompass MiniLM [2], a model that shares a similar parameter count with DistilBERT but is better pre-trained. The following table presents the results of this additional experimentation, with the same dataset settings compared to our previous DistilBERT experiments:\n\n| Method/Dataset | IMDb | QNLI | RTE  | AdQA           |\n|----------------|------|------|------|----------------|\n|Gold Data       | 92.30| 91.00| 71.50| 22.97/36.59    |\n|ZeroGen         | 83.66| 70.11| 72.2 | 5.07/10.74     |\n|S3              | 89.55| 85.20| 76.17| 20.50/34.40    |\n\nThese results clearly indicate that the S3 framework consistently outperforms ZeroGen across various datasets, irrespective of the model in use. Notably, similar to our DistilBERT findings, the S3 framework surpasses the performance of the gold data in the smaller RTE dataset.\n\nIn light of these results, we hope the reviewer appreciates the broader efficacy of the S3 framework across different model architectures.\n\n#### Answers to Questions For Authors\n\n##### Response to the R > 2 results (Question 1):\n\nWe appreciate the reviewer's inquiry regarding the extension of our training to R>2 rounds. Indeed, we conducted experiments to explore this, specifically with the RTE dataset set for R=3. However, we observed that increasing the rounds of training beyond two does not lead to consistent improvement. Furthermore, when analyzing the results across different training rounds for the S3 method, the incremental benefits from the first to the second round of error extrapolation are quite marginal. Below are the detailed results to provide clarity:\n\n| Method/Dataset          | IMDb | QNLI | RTE  | AdQA           |\n|-------------------------|------|------|------|----------------|\n|SeedData                 | 86.86| 73.79| 59.93| 8.70/24.08     |\n|SeedData + 1 Round EES   | 88.47| 78.86| 71.12| 11.73/24.08    |\n|SeedData + 2 Rounds EES  | 89.00| 79.92| 73.29| 12.50/24.38    |\n\nWe therefore we opted to a maximum of two rounds in our final experiments.\n\n##### Response to difference of dataset size of GoldGen and S3 in QNLI dataset (Question 2):\n\nSorry, that is a mistake we made during the paper writing, the actual dataset size for QNLI in GoldGen baseline is **$150k$** rather than $110k$. We have fixed this typo in the revised version; thanks a lot for pointing this out!\n\n##### Response to parameter searching space (Question 3):\n\nWe appreciate the reviewer's request for clarity on our hyper-parameter search. For your reference, the specific hyper-parameter search space we explored is outlined below:\n\n- Batch Size: 8, 16, 32\n- Learning Rate: 1E-5, 8E-6, 5E-6, 1E-6, 1E-7\n- Weight Decay: 1E-2, 1E-5, 1E-8\n- Epoch: 6, 8, 10, 15, 20\n\nTo ensure a consistent and fair comparison, **we conduct the same hyperparameter search process for all the baselines as well as our approach**.\n\n#### Responses to other reviews:\n\nWe deeply thank the reviewer's attentive feedback on typographical errors, grammar, and overall presentation. We have fixed them in the revised version.\n\nRegarding the concerns about reproducibility, we are committed to transparency and have therefore made our datasets available on an anonymous GitHub repository. You can access them at https://anonymous.4open.science/r/S3_Datasets-7421/README.md. We invite you to review the data at your convenience.\n\n[1] Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., ... & Wei, J. (2022). Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.\n[2] Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33, 5776-5788.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is simple, intuitive and logical. It builds upon a line of prior work on pseudo-labelling and self-training, which the authors adapt to LLMs.\n1. The authors method achieves large improvements across a range of datasets, using fewer synthetic samples than prior methods.\n1. Ablations show the importance of their iterative error refinement method, which generates", "reasons_to_reject": "1. The theoretical analysis section is more of \"theoretical intuition\", as it makes several assumption which are the whole backbone of the author's method - such as \"the error the small model makes on the small gold validation set can represent the distribution gap\", \"if an LLM is good enough it can perfectly extrapolate the errors\", \"theoretically, we can recover gold data distribution by simply combining the original seed data with additional data\". All of these (and more), make this section completely redundant.\n1. The \"rationales\" method is very similar to chain of thought style prompting.\n1. Experiments are only done with one small model (distilbert) - perhaps other small models should also be considered to show the effectiveness for other models.", "questions_to_authors": "1. Did you test with R>2 rounds of training? (line 221)\n1. Why is there such a difference in dataset size for GoldGen and S3 for QNLI (Table 1), when the authors say the size is kept comparable (line 377)? How is the performance of these models at equal size of data? (given that scores are very close)?\n1. (Line 396) Could you specify the range of hyper-parameter search? Was the same search carried out for all the methods?", "paper_topic_and_main_contributions": "This paper proposes a method for progressively-improved multi-round synthetic data augmentation. The authors prompt LLMs to generate data samples given only labels/classes (and prompts), use synthetic data to train downstream model. Thereafter in each round, they generate more data specifically for labels which the downstream model misclassified. The authors method achieves large improvements across a range of datasets, using fewer synthetic samples than prior methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision, Denis Gudovskiy, Alec Hodgkinson, Takuya Yamaguchi, Sotaro Tsukizawa; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 9041-9049", "Typos_Grammar_Style_and_Presentation_Improvements": "1. \"Extrapolating the errors of the small model\" doesn't seem to be the ideal phrase for the author's method - while I understand that the author's are trying to say the LLM generates examples which target areas of the distribution the small model has errors at, perhaps something clearer will be better.\n1. Table 3 could perhaps be merged with table 2.\n1. Line 403 \". And\" typo\n1. Line 215 D eval should D val.\n1. Line 340 full stop.\n1. Line 312 \"if/and\""}}, {"review_id": "fAIjG5wM8C", "reviewer": "Reviewer_fvWF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fvWF\n\n#### Answers to Reasons To Reject:\n\n##### Response to the SSE synthesized data quality check:\n\nWe agree that more sophisticated prompt design is required to apply rationale-based data synthesis on datasets beyond classification and QA. We will acknowledge this in the discussion and limitation section and will leave the application of our approach on more general tasks for future work.\n\nAs for the checking of quality of synthesized data, we perform post-hoc checking to ensure the quality of the EES synthesized data. Here are the detailed steps: \n\n1. For misclassified data $D_{mis} = \\{(x_{mis}^{(i)}, y_{mis}^{(i)})\\}$, $i$ is from 1 to n and additional data generated on the misclassied data $D_{add} = {(x_{mis}^{(i)}, y_{mis}^{(i)})\\}$, $i$ is from 1 to n, we perform sentence encoding by DistilBERT on every $x$'s. We have encoded sentence $z_{mis}^{(i)} = enc(x_{mis}^{(i)}), z_{add}^{(i)} = enc(x_{add}^{(i)}) \\in \\mathbb{R}^{d}$ where $d$ is the hidden dimension, for DistilBERT $d = 768$.\n2. After we have the encoded sentence for $ z_{mis}^{(i)}, z_{add}^{(i)}$ , we compare their cos similarity. If they have high cos similarity, it means they are similar in meaning.\n3. Then, we compare the editing distance between sentence $x_{mis}^{(i)}, x_{add}^{(i)}$; if their editing distance is close to their sentence length, we say that they are significantly different in composing of text (i.e., they are not similar in text)\n\nBy conducting the upon steps on the two rounds of generated on misclassification data, the result are as follows:\n\n| Rounds | Label | IMDb | QNLI | RTE | AdQA|\n|--------|-------|------|------|-----|-----|\n|Round 1 |data num | 3,291 | 28,532 | 883 | 26,605|\n|        |cos sim | 0.9445 | 0.9536 | 0.9367 | 0.9469 |\n|        |editing distance| 289.07 | 14.60 | 16.44 | 13.98 |\n|        |avg mis len| 282.45 | 14.20 | 13.75 | 13.76 |\n|        |avg gen len| 258.42 | 19.88 | 24.54 | 18.71 |\n|Round 2 |data num | 2,882 | 22,568 | 639 | 24,927 |\n|        |cos sim | 0.9557 | 0.9538 | 0.9397 | 0.9467|\n|        |editing distance | 265.61 | 14.68 | 16.29 | 14.01 |\n|        |avg mis len | 294.42 | 14.14 | 14.13 | 13.69 |\n|        |avg gen len | 173.40 | 20.09 | 24.70 | 18.69|\n\nWe add the average length of misclassified data (avg mis len) and average length of generated data (avg gen len) for reference of the editing distance. In the chart, we can see that for all our extrapolation-based synthesis data, the cos similarity is high, which means they are close in meaning of the misclassified data. And their deiting distance is close to the length of data, which means the LLM does not simply copy from the misclassified data. That shows the new data generated is in high quality.\n\n##### Responses to our framework’s fine-tuning cost:\n\nThanks a lot for pointing out the potential high computation cost for our S3 framework caused by fine-tuning the small model iteratively. Let me address this by showing the fine-tuning cost of our framework is less than the ZeroGen baseline in sense of floating-point operations (FLOPs) involved in fine-tuning.\n\nFor the DistilBERT model we used, based on OpenAI's report [1], the BERT family model costs $6$ FLOPs per training per token per parameter (i.e., $FLOP_{token, para} = 6$.). And based on [2], the DistilBERT has $numPara = 66 \\times 10^6$ parameters. And the typical token length for one record is $ numToken_{rec} = 512$. Therefore, the training FLOP per record of data per epoch is:\n\n$$\nFLOP_{rec} = FLOP_{token, para} * numToken_{rec} * numPara = 2.03 * 10^{11}\n$$\n\nFor ZeroGen, it typically uses $200k$ if data records and trains for an average $10$ epochs to achieve the best results based on our experiment. Thus, the fine-tuning cost for ZeroGen is:\n\n$$\nFLOP_{ZeroGen} = FLOP_{rec} * 200k * 10 = 4.06 * 10^{17}\n$$\n\nFor our S3 method, in the first round of fine-tuning (using only seed data) dataset size is typically in $2/3$ size of final dataset (i.e. $51.2k$ records), and the second round of fine-tuning (using seed data + misclass generated data of first round) is in $5/6$ size of final dataset (i.e., $64.0k$), and the final round of fine-tuning (using seed data + 2 rounds of misclass generated data) have $76.8k$ data. And on average, our method needs $8$ epochs to achieve its best result. Therefore, the total FLOPs of fine-tuning DistilBERT for 3 times (2 for getting misclassified data, 1 for final fine-tuning) in our S3 frameworks are:\n\n$$\nFLOP_{S3} = FLOP_{rec} * (51.2k + 64.0k + 76.8k) * 8 = 3.11 * 10^{17}\n$$\n\nTo conclude, we can see that our S3 methods use only 3/4 FLOPs compared to the ZeroGen baseline, even though we fine-tuned the model with R(=2) more times.\n\n#### Answers to other comments:\n\nWe thank the reviewer a lot by detaily pointing out the problem we have in Alogrithm 1 and the subscript of template in the caption of Table 1. We have fixed them in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A simple and effective data synthesize framework that mitigates the demand of data generation (associated with number of queries to large language models) using rational prompt templates.\n- Some qualitative analysis to demonstrate why the iterative process of S3 can mitigate the distribution gap between synthesized data and real data.", "reasons_to_reject": "- The incorporation of rationals can be challenging for tasks beyond classification of standard QA tasks (fluency, consistency, factuality etc.), the work does not address the quality or includes a gating step to check the quality of synthesized data generated from misclassified examples.\n- Since the small model needs to be finetuned iteratively from the synthesized dataset, it can be expensive by a large number of iterations, the information about the finetuning choices in this regard is not well discussed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a data synthesize framework S3 that uses a large language model to generate prompt templates with rational information and construct a seed dataset, which is then used to train a smaller model. The seed dataset is iteratively augmented by generating additional data that are misclassified from a small real dataset. Experimental results show that S3 achieves better improvement over previous data synthesize frameworks like ProGen and ZeroGen. And the template with rational information achieves similar performance compared to augmentation without the rational template and using only 30% of the data. Moreover, S3 has better transferrability to other synthetic dataset.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Redundant parenthesis on the last line of Algorithm 1.\n- The superscript of template in the caption of Table 1 is confusing."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A simple and effective approach to do data generation. Written well and easy to understand. It's technically sound and show good performance. However, for originality, it's not a very novel idea that would surprise people.

Pros:

1. A simple and effective method to do data generation.

2. Great performance on some benchmark datasets.

Cons:

1. It's not a particularly interesting method. None reviewer showed the excitement high enough to support the paper. It reads like reviewers all feel it's reasonable to get this performance and there is no analysis showing why pervious(or simple heuristic) methods would fail and why the problem is extremely difficult.
Decision: Accept-Findings