Making Large Language Models Better Data Creators
Authors: Dong-Ho Lee
Jay Pujara
Mohit Sewak
Ryen W White
Sujay Kumar Jauhar
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model
Data Creation

Abstract: Although large language models (LLMs) have advanced the state-of-the-art in NLP significantly, deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security.
As such, trainable models are still the preferred option in some cases.
However, these models still require human-labeled data for optimal performance, which is expensive and time-consuming to obtain.
In order to address this issue, several techniques to reduce human effort involve labeling or generating data using LLMs.
Although these methods are effective for certain applications, in practice they encounter difficulties in real-world scenarios.
Labeling data requires careful data selection, while generating data necessitates task-specific prompt engineering.
In this paper, we propose a unified data creation pipeline that requires only a single formatting example, and which is applicable to a broad range of tasks, including traditionally problematic ones with semantically devoid label spaces.
In our experiments we demonstrate that instruction-following LLMs are highly cost-effective data creators, and that models trained with these data exhibit performance better than those trained with human-labeled data (by up to 17.5\%) on out-of-distribution evaluation, while maintaining comparable performance on in-distribution tasks. These results have important implications for the robustness of NLP systems deployed in the real-world.

Making Large Language Models Better Data Creators
Dong-Ho Lee'; Jay Pujara', Mohit Sewak?, Ryen W. White", Sujay Kumar Jauhar?
Information Sciences Institute, University of Southern California
Microsoff Research
(ateeeeeeuuuujjaaaaaaaaa {dongho. Teooghholeeppee edu, (jpujara)@isi.. edu, {mohit. sewak, rywnk, s sjauhar yemicrosoft.
Abstract
(4) GPT as Labeler
Although large language models (LLMS)) have advanced the state-of-the-ar art in NLP signifi- cantly, deploying them for downstream appli- cations is still challenging due to cost, respon- siveness, control, or concerns around privacy and security. As such, trainable models still the preferred option in some cases. How- ever, these models still require buman-labeled data for optimal performance, which is ex- pensive and time-consuming to obtain. In or- der address this issue, several techniques to reduce human effort involve labeling or generating data using LLMs. Although these methods are effective for certain applications, in practice they encounter difficulties in real- world scenarios. Labeling data requires care- ful data selection, while generating data ne- cessitates task-specific prompt engineering. In this paper, we propose unified data creation pipeline that requires only single formatting example, and which is applicable to broad range of tasks, including traditionally prob- lematic ones with semantically devoid label spaces. In our experiments we demonstrate that cnstruction-followwe
by hurricane?
Unlabeled Instances
Input
Output
Gonerator
Generate "oppot" ober yeu
Has the UK beer hit by hurscanc?
Output
L.M
Inpat
(c) GPT with Single-formatting Example
Questiun: Dows France have a Prime Minlster?
Filhening 'ersmple' to genemate diverse complles
Option: [yes,
Question: the LK been hit by a hurricane?
Option: [yes, Auswer. yes
Question: Have the San Jow Sharks won Sturley Cup?
Input/ Ceput/OOugy Oulput
Option: [yes, noj Answer: yes
(Single-formatting Example)
Figure Existing LLM-based data augmentation needs unlabeled examples (labeler) or label-specine prompts (generator), while our framework generates examples for variety tasks in a unified way.
Rae al., 2021; Thoppilan al., 2022; Chowdhery et al., 2022). However, despite their effectiveness, there is a continued demand for the deployment of smaller trainable or tunable models in real-world scenarios due to cost constraints, existing service- level agreement response times, or privacy and security concerns around using black-box APIs. Unfortunately, application-speeifee custom models sometimes require large amounts of high-quality human-labeled data, in order perform well. Thus, in order to reduce time and cost in the model de- ployment cycle, recent work has focused on trying to obtain training data by leveraging LLMs as ei- ther labelers to annotate unlabeled data (Yoo et al., 2021; Wang al., 2021a; Lang et al., 2022), generators to generate new data samples (Meng et al.. 2022; Ye al., 2022: Gao al., 2022).
LLMs highly cost- effective data creators, and that models trained with these data exhibit performance better than those trained with human-labelee data (by up to 17.5%) on out-of dfstrriibbttii evaluation, while maintaining comparable performance on in- distribution tasks. These results have important implications for the robustness of NLP systems deployed in the real-world.
1 Introduction
Large language models (LLMs) have revolution- ized the field of NLP, yiclding impressive perfor- mance on various conventional natural language understanding (NLU) and generation (NLG) tasks. They are able to do this with only a handful (i.e., few-shot) or sometimes even training examples (i.e., zero-shot) (Brown et al., 2020; Du et al., 2022; Work done during Microsoft Research Intersship.
Despite initial successes, constraints for these techniques continue to hinder their applicability in broader real roal-world settings. First, in the context of using LLMs as labelers, it is essential to have raw

---

data that closely resembles the distribution of data in the predictive task. Most previous research has assumed access to a training dataset from which the labels are elided; however, for cold-start prob- lems in the real-world, no such assumptions can be made. Curating raw data for tasks in specialized domains, such as those in the biomedical or legal fields, can be particularly challenging. Conversely, sampling a large volume of data at random can re- sult in an imbalanced label distribution due to rare events (Markov et al., 2022).
yield multiple examples that vary in content but are formatted uniformly (Section 3.1-3.2). Further- more, as an efficient means of diversifying the gen- erated data, we propose a "self-reference" strategy, which iteratively samples from the pool of newly created examples to seed the prompt for the next round of generation (Section 3.4). Specifically, we outline 4 distinct instantiations of "selffreference" including random, contrastive, similar, and tree sampling for controlled diversification of data.
We evaluate our data creation pipeline on a bat- tery of tests involving three distinct types of tasks, namely multiple-choice question answering (QA), open-book yes/no QA, and closed-book yes/no QA. The datasets for these tasks range across a variety of domains, including specialized ones such as the biomedical domain. Furthermore, for each cate- gory of task, we use minimum of two datasets in order to compare the out-of-distribution (OOD) generalization of models using original data to syn- thetically generated LLM data. Our results demon- strate that leveraging LLMs as generators using our formatting baaend creation approach is a highly cost- effective way of creating data that can be effectively used to train models for a variety of downstream tasks, including those in specialized domains, and ones where labels are devoid of semantic meaning or vary across the data. For in-distribution (ID) settings, naturally having access to large amounts of high-quality manually curated and labeled data is still ideal. However, when only a small amount of
Meanwhile, leveraging LLMS as generators requires careful curation of few-shot exam- ples (Hartvigsen et al., 2022), or composition of prompts that highlight the semantic meaning of la- bels (Wang et al., 2021b; Meng al., 2022; Ye al., 2022; Gao et al., 2022), such as positive v. negative in sentiment classification. The latter has been a bottleneck to the broader applicability of LLMs as generators, however, since not all tasks have labels that are semantically meaningful, or are enumerable. Consider, for example the label yes v. no, which have meaning when taken without context; the options of multiple choice QA (see Figure 1), which are an effectively open-ended label-set that varies from instance to instance. For these kinds of problems LLMs as generators con- tinue to be inadequate.
In this paper, we first present a formal frame- work for characterizing different approaches for LLM data creation. Specifically, we use graphical models way to characterize and unify disparate approaches that include LLMs as either labelers or generators (Section 2). Next, we propose a novel data creation pipeline that only requires a single for- matting example to generate heterogeneous labeled data for various downstream applications, includ- ing those that focus specialized domains. In con- trast current methods that require dataset-specifie components (e.g., label description, example selec- tion), our pipeline serves as unified solution that can be applied to a wide range of tasks, includ- ing those where the label set either semantically devoid of meaning, or unenumerable.
data is available, our approach yields results that are often comparable, and some- times even better than the original datasets. This highlights the potential role LLMs can play in the model development cycle, especially in resource- poor or specialized domains. Further, for the OOD settings, models trained on data generated by our pipeline consistently, and by large margins, outper- form their counterparts trained on data from human sources. This robustness and generalizability has important implications for the deployment of real- world systems that deal with data that are variable, chaotic and often very different from curated aca- demic datasets. We are realeasing our code and prompts to the community to spur future research in the arca!.
Our data creation pipeline leverages an instruction-following LLM generator in con- junction with single formatting example as simple yet effective way of imposing structured constraints. Specifically, our approach iteratively conditions the generator on an instruction and a unique formatting example in a JSON format to

---

2 Formalization of LLM-based data creation
(a) Labeler
(x,311)
e1
E EDr
In this section, we attempt draw different data creation strategies using LLMs into a unified frame- work, and discuss related research using this frame- work.
Xu
M
y
E Dy
2.1 LLM-based data creation
(b) Generator
Assume a large language model M (e.g., GPT- 3) that has been pre-trained to maximize the like- lihood of generating each token in sequence Fnl by conditioning on previous tokens. Then, .M capable of generating new text through recursive sampling of tokens from its out- put probability distribution. Given such a model M and label space V, the goal of data creation is to induce samples (x,y) where y )/. Based on different instantiations of this general framework, other inputs may be included, such scriptive prompt Wy for each y E D, in-domain unlabeled example x, Dú, or a small number of example pairs (x) Dr along with their corresponding explanation eq.
Wy
Xx
E Dy
x [1, .2. ................................ ........,.
y
M
Figure 2: Graphical models for using LLM M (a) labeler which outputs label y for unlabeled data Xy Dy using instructive prompt Wy few-shot examples (xi, ) E Dr. with or without explanation et, and as (b) generator which generates multiple data x for label y with label-descriptive prompt W or using in-domain unlaheled example x E DD
label de-
have shown that curating diverse and representa- tive samples is critical to the ability of the model to label new samples (Liu et al., 2022; Rubin et al., 2022; Su et al., 2022). This can be challenging, particularly in specialized domains such as, for ex- ample, the legal or biomedical domain where only small number of samples may be curated. Our paper proposes pipeline (Section 3 capable of tackling these challenges, and is particularly useful in resource-poor domains (Section 5.1). Further- more, providing intermediate reasoning steps (i.e., -fains-of-thoughht) as explanations e, in prompts enables better labeling in both few-shot (Wei et al., 2022; Zhou et al., 2022; Lampinen et al., 2022) and zero-shot setting (Kojima et al., 2022).
2.2 Formal Framework and Related Work
Given these basic components, there are two broad strategies for LLM data creation, namely using an LLM labeler as generator. Graphical mod- els for each of these two strategies is presented in Figure summarize the conditional interac- tions and independence assumptions that describe the relationships between common framework con- stituents. The rest of this section discusses existing work using the unified language of these graphical models.
Using LLMs as labelers. M can be used labeler for unlabeled data (See Figure (a)). Here, approaches assume that unlabeled data Du pro- vided as input and the distribution of Du similar to the distribution for the target task. Labeling can be achieved either by conditioning M on a few labeled examples (t) E Dr (Brown et al., 2020; Yoo et al., 2021; Wang et al., 2021a; Lang et al., 2022), or by leveraging instructive prompts W describing the task without any labeled exam- ples (Brown al., 2020). When providing the model M with a small number of labeled examples (1) E D,,- often referred to as the few-shot setting (and contrasted with the zero-shot setting, where no examples are provided) - recent studies
Using LLMs as generators. An altogether dif- ferent data creation approach uses the LLM M directly generator (See Figure (b)). While labeler predicts y by conditioning on an input X, generator does the reverse by generating x given y. However, like with LLMs as labelers, a small number of relevant samples can be used to condition M for generating data. Hartvigsen et al. (2022) feeds human-curated examples of the target label (e.g., implicit hate speech) into M to generate human-like examples for the target label. In con- trast, Wang al. (2021b) conditions M on both in-domain unlabeled examples and target label y to generate domain- domain-related data for y. Mcanwhile a number of different efforts (Meng et al., 2022;

---

Ours
Instruction
x7.911
exapple provided, You are creating (number_nf saaamler__f_raaarene scaampee)))) examples that follow
but with different content.
The created examples AMBuste all have different answers.
The output sAuust be in unnunbered IS0N fornat. [finedoooyl] The created examples
W
(x,y)
the options the provided example.
Figure 3: Framework Overview of example-based data creation which outputs multiple complete data (x, starting with an initial seed formatting example (x) and the instruction Wi.
Table I: Instruction Wi used in the paper.
Example-based Data Creation
This paper proposes a unified data creation ap- proach using LLMs, which does not need in- domain unlabeled examples X E Dr or data- specific label-descriptive prompts Wy. As illus- trated in Figure 3, our framework iteratively cre- ates data De beginning with single initial for- matting example (xy.yy) and an instruction Wi (Section 3.1). The process begins by converting (Xy:yy) into a structured prompt Wy (Section 3.2- 3.3). After conditioning M on [W;; Wy] to gener- ate data, we sample an instance from the pool of newly created data to serve as formatting example for the next iteration (Section 3,4). We continue to iterate in this manner until the required number of instances k is obtained, after discarding dupli- cate and ill formatttee data. This is done by caching data and checking newly created candidates against previously generated ones for duplicates, and by using the python json. loads method for veri- fying the validity of the json output. The resulting data creation pipeline is generally suitable for most classification tasks. Although, in this paper we specifically focus on tasks where the label set is potentially open-ended (e.g., multiple-choice QA). or lacks inherent semantic meaning (e.g., binary QA) - problem spaces that have posed challenges to past work in LLM data creation.
Ye et al., 2022; Gao et al., 2022) condition M on in-domain unlabeled example and well-formatted descriptive prompts Wy for target label to gen- erate data. One important caveat with all these approaches is that the label y, upon which outputs are conditioned, needs be inherently meaningful way
in order for instructions to formulated that prompt the model M into generating coherent outputs. For example, when y is an entailement relationship, a corresponding prompt Wy might include "x,, C Du. In other words...": "; or when y is the sentiment of movie review, Wy might include "The movie review is...". Contrast this with the scenario where y is an index binary response, which has no semantic meaning without context and is therefore difficult to condition on. In this pa- per, we devise a unified approach to tackling these scenarios (Section 3), yielding a method for data creation that is broadly applicable.
Differences with Instruction-follovin Data Generation. Recent studies use M to create data models (Wang et al., 2023b; Taori et al., 2023; Xu et al., 2023; Chiang et al., 2023; Mukherjee et al., 2023), Such studies leverage M as labeler, producing coher- ent response y for given instruction æ. The main focus is on covering a wide range of inputs x and their corresponding responses y to encompass the diversity of instructions potentially encountered in real-world interactions with users. While recent studies have indicated that such models trained with auto gettereede instruction-respone pairs can yield logical and coherent responses to user instruc- NLU tasks remains
for training eestruction-folloviii
3.1 Instruction
The goal of our framework is have the model M generate a diverse set of examples in the same format as the input formatting example Yy). To ensure format consistency and ex- ample diversity, we use the system instruction Wi in Table 1. We generate data in batches of {neumber_og cattbber_oo_eamppple ezamples), not only to account for the token generation limits of LLMs, but also encourage content diversity through subsequent sampling of W, (Sec 3.4). In this paper, we set (number__d) _arrsooo___aeee [ezamples] to and do not vary it. In order to mitigate label bias, we encourage mod-
tions, their performance sub-par (Wang et al., 2023a). Meanwhile, the focus of our work the generation of data specifically tailored for natural language understanding (NLU) tasks with a focus on accurate responses.

---

els strive for maximal variance in their generated responses, avoiding repetitions in data where the answer consistently "yes", for example.
"Question": 'I am black when you buy red when you use me. When turn white, you know it's time throw me away.
12", "Options": l'charcoal", "rose flower", , "ink", "fruit", "shoe"], "Answer": "charcoal"
3.2 Formatting Example
The only assumed input to our example-base based data creation pipeline is a single formatting example (x) and its corresponding label space y: This example is formatted as JSON-structured prompt W as shown in Figure 4. Given the one-shot JSON structured format prompt, it is expected that the model yields syntactically correct output that con- forms the ISON schema. While generating a complex structured output like JSON can be chal- lenging, its easy parsing acts as a way to validate outputs at creation time and for effective usage in training downstream models.
(a) Variant (multiple-choice QA)
"Options": : ["yes", "no", "maybe"],
"Answer": "yes".
"Question" "Is batman and robin a sequel to balman
forever?",
"Context": "With the box office success Batman Forever
in June 1995, Warner Bros. immediately commissioned sequel... ..."
(b) Fixed (yes-no QA)
Figure 4: Example of formatting example prompt W, where "options" contain the label space ) of the task, "answer" contains y and "question" contains xy. "Context" is optional and depends on the task.
3.3 Structure of Formatting Example
Recall that our focus in this paper is on data cre- ation for tasks that are challenging because their output label space is open-ended, or because they lack inherent semantic meaning. We refer these distinct settings using the shorthand variable and fixed, and note that the input formatting example yr) is different for each of these label space settings. Specifically, the major difference is the order of presentation of prompt components.
from the outputs (x E DG,- generated at iteration 1- We experiment with four different sampling strategies.
Random selection. During each iteration, a for- matting example for the next step is randomly cho- sen from the output of the current step.
Contrastive selection. For each iteration, we lect the example that displays the greatest semantic contrast the preceding formatting example. In this approach, we use a pre- pre-trained bidirectional- encoder (Reimers and Gurevych, 2019) to generate embeddings for examples, and compute the cosine similarity between Xy and X9:-1 selecting the in- stance Xg. with the lowest similarity.
Variable Option. The variable option format is structured in a logical sequence beginning with the question xy, followed by list of answer candidates V, and finally the correct answer y--
Fixed Option. In contrast, for the variable op- tion, the expected format consists of the answer candidates }/ first, followed by the correct answer Yf- and finally the question Xy- This inversion of prompt components added ensure that the rego-segress model creates questions with pre- determined optionssince the model, as a free gen- erator, can produce inconsistent output, resulting in answer options y that do not belong to the fixed pre- deeereeer set.
Similar selection. This sampling approach works analogously to Contrastive selection, except that instead of selecting the X9.- with the lowest cosine similarity to xf. we select the one with the highest similarity.
Tree selection. Iterative sampling of data may result in significant domain drift from the first seed example to data generated in later steps of the gen- eration pipeline, due to unexpected content varia- tions produced by the model. To avoid this issue, use all the generated outputs from one step as formatting examples for subsequent iterations. This approach can be viewed as breadth-first tree traversal over generated examples, and is in con- trast with the other three sampling approaches that use a depth- first exploration strategy. Our hypoth-
3.4 Self-Reference
Relying a single formatting example (xy: (xy.y)) yr) reference point for all iterations of data ere- ation may limit the ability of the pipeline yield data that broad coverage, diverse and balanced. To overcome this, we propose "self-reeerrece". wherein the formatting example f, = v,) for all subsequent generation steps i > 0 are sampled

---

esis is that the minimum height of the exploration yields samples that are more topically coher- ent.
Task
Label Space
Domain
Dafaset
PIQA
mulkiplo- choice QA
Variamt (2)
Commoneense
Winogrande
CommonscnecQA
mulliple-choice QA
Variant (5)
Commonsense
RidcleSense
4 Experimental Setup
Knowledge
BoolQ (w/ context)
open
open-book book yes/no
Fixed (2)
Biomedical
PuhMedQA
Biomedical
BloASQ
In this section we describe the experimental setup that we use to evaluate our single-shot example- based data creation framework.
Knowlodge
RoalQ (wlo context)
closed-hook yes/no
Fixed (3)
Knowloddge
StrategyQA
Knowledge
CREAK
Table 2: Datasets used in the paper. The numbers enclosed parentheses indicate the number of labels within the label space.
4.1 Datasets
We evaluate on three types of different tasks: multiple-choice question answering (QA), open- book yes/no QA, and closed- book yes/no QA as shown in Table 2. Multiple-choice QA is used evaluate our data creation pipeline in variable label space setting, while the other tasks are used for fixed label space settings. order to demon- strate the domain generalization capability of our approach, we additionally use a minimum of two datasets for each category of task. The datasets range broadly in the reasoning abilities they de- mand from models, requiring them solve diverse problems such filling in the blank in sentence (PIQA (Bisk al., 2020), WinoGrande (Sakaguchi et al., 2021)), choosing the most suitable option among multiple choices (CommonsenseQA (Tal- mor et al., 2019), RiddleSense (Lin et al., 2021)), comprehending a given passage to make pre- diction (BoolQ with context (Clark et al., 2019), PubMedQA (Jin et al., 2019), BioASQ (Tsatsa- ronis al., 2015)), and answering based in- herent knowledge (BoolQ without context (Clark et al., 2019), StrategyQA (Geva et al., 2021), CREAK (Onoe et al., 2021)). Details of the various datasets are presented in Appendix A.I.
4.3 Implementation Details
Throughout the entire process of data creation, we use gpt-3. 3ptt3.5-turbb. 5-turbo language model, of June 2023, with specific settings of lemperalure and topp set to When conducting fine-tuning experi- ments, we leverage the Adam optimizer (Kingma and Ba, 2014) with a maximum sequence length 256. In each experiment, we perform a grid search on development data for the optimal learning rate in [3e-4, le-4, 5e-5, 2e-5, le-5, 5e-6, 3e-6, 1e-6, 5e-71, and batch size in [4, 8, 16]. All experiments conducted RTX A5000 with FP32.
5 Experimental Results
We conduct comprehensive set of experiments that seek evaluate the effectiveness of data cre- ated by LLMs for focused downstream applica- tion modeling. Our first evaluation involves exam- ining the performance of models trained on data generated by variants of our single-shot data cre- ation pipeline and comparing them against man- ually curated training data. We investigate both in-distributed (ID) and out-of-distribution test data settings, each of which are explained in greater detail below.
4.2 Evaluation Details
5.1 Performance Comparison
order to demonstrate the efficacy of our data creation framework, we present a comparison of a downstream model when it is trained on (1) the original train dataset denoted by Dr; and (2) an LLM created dataset, denoted by Da. where single seed formatting example is randomly se- lected from Dí. We unable to conduct comparative analysis with other LLM-based data generation methods they do provide solu- tions or prompts engineered for the tasks listed in Table The base model used this paper is RoBERTa-large (Liu et al., 2019).
ID Performance. the ID setting the test data is drawn from the same distribution as the train- ing data; specifically, a portion of the full human- sourced dataset is held out as a test set to evaluate models trained on either human-labeled or LLM created data. Table 3 summarizes the performance of models trained two types of datasets: the orig- inal dataset, denoted by Dr, and datasets created using the different "self-reference" variants of our single-shot pipeline the corresponding rows the table marked as DG. In both settings, and for a

---

MCQA (2)
MCQA (5)
Open Yes/No
Closed Yes/No
Trained on.
PIQA
WinoGirande
CommonsenseQA
RiddleSense
BoolQ
Pub
BioASQ
Strategy
CREAK
# Exaampes in D
14.113
160
8,500
3,510
9,427
450
670
9,427
2,061
10.176
Dr
80.95
51.41
68.17
56.48
85.62
55.20
87.14
65.68
49,56
81.19
Do (Random)
51.26
42.06
59.80
80.71
52.23
53.04
67.93
Do (Contrastive)
66.15
52.36
41.57
38.43
66.66
59.33
67.14
61.28
49.56
67.93
Do (Similar)
67.15
52.05
47.62
42.09
69.60
60.60
K3.57
61.28
49.56
69.24
Do (Trea)
68.35
48.50
69.66
85.71
61.28
72.74
(Po Be--Dul DLj/Ps
- 18.43%
+2.65%
-40.55%
-33.64%
-22991%
+10.38%
-1,66%
-7.18%
+12.31%
-11611%
Table 3: ID Performance (Accuracy) comparison between models trained on original train dataset Dz (First group) and LLM-created train dataset De (Second group). The optimal variant for data-creation in the second group is shown in bold, and the second best is underlined. The third group of the table presents the percentage difference between the best variant in the second group and the first group.
MCQA (2)
MCQA EE)
Open Ya/No
Closed YeuNo
Thii--
PIQA Winokinnde
WinCinnde PIQA
Commonommaa RikSieSense
RoblleSeme Commomeensst@A
BoxQQ PubMedi2A
PrbMailQA Bool)
BioASQ PunMedi2A
PriMMedQA BloASQQ
StraegyQA CREAK
CREAK
Trained I'lest
StateggQ4
$2,15
44.65
40.53
62.80
58,65
67.14
56.20
49.27
(Kandom)
43.33
09.27
(Cintrasive)
81.25
43,87
62.59
40.20
40.55
43,35
64.60
61.25
81.42
54.78
(Po- D,y/0.
-14.4%
+5.58%
+2.75%
+6.16%
+17.8%
+14.0%
Table 4: OOD Performance (Aceuracy) comparison between models trained on original train dataset Dr, (First train dataset Dz (Second group). The best dataset for each OOD experiment is shown in
group) and bold, and the second best is underlined. The third group the table presents the percentage difference between the best variant in the second group and the first group.
given task, the number of samples for each dataset is identical. We also compute the percentage dif- ferential between the two types of datasets, shown in the table as (Do Dr,)/Dr. These findings confirm that while there is no substitute for large amounts of hand-crafted data, demonstrated by drops of up 40.55% when using synthetically created data- LLMs can play an important role when access is only available to very little data, and in specialized domains. This is demonstrated by the similar or often better performance of Dc models on WinoGrande, PubMedQA, BioASQ and Strate- gyQA. Meanwhile, comparison between different "self-reference" sampling strategies demonstrates the importance of mitigating domain drift in our single-shot approach, where the sole true format- ting example is the only anchor point to the data distribution we seek to generate. The Tree-based exploration strategy limits the semantic distance between the seed sample and instances created later in the generation process and therefore yield higher performance on ID data.
Specifically, since we have used at least two eval- uation datasets for each category of downstream application, we use one dataset for training and evaluate on a different dataset. Note that in this setting, while the training data can either be manu- ally curated (i.e. Dr,), or generated by LLM (i.e. Dc), the test dataset is always manually curated (i.e., original test data). Table 4 presents a compre- hensive analysis of the OOD performance of mod- els trained on Dr and Dg. The results show that models trained on LLM data are consistently and sometimes significantly better at OOD predictive performance than their hand-crafted counterparts. This has important implications for the robustness and generalizability of real-world systems that of- ten deal with inputs that are very different from carefully curated academic datasets. We note that a combination of human and LLM created data may yield even higher gains in OOD performance and leave a comprehensive evaluation of this to fu- ture work. Finally, a comparison of "self-reference" strategies on the OOD setting shows that while the Tree-based exploration approach is still consis- tently strong one, other sampling approaches are sometimes comparable or better. This is under- standable since some degree of controlled noise is helpful, and can be viewed as a regularizer, when trying to generalize to OOD test data.
While ID data is useful to gain insights of system in controlled settings, real- world applications must deal with data that is often far more variable and chaotic. Therefore, we com- pare manually curated data (i.e., original training data) to LLM generated data in an OOD setting.
OOD Performance.

---

Riddiesense
Dataset
Train
Random
Diverse
Similar
Tree
Human
Random
PIQA
14,113
3.60
2.82
3.62
3.97
WinoGrande
160
0.02
0.02
0.02
40
Synfar
CommonsenseQA
8,500
2.73
2.71
2.77
1.73
Tree
RiddleSense
3,510
0,95
0.95
1.00
1,05
BoolQ
9,427
5.13
2.24
4,95
4.2
PUbMedQA
450
0.17
0.15
0.17
0.17
BioASQ
670
0.24
0.23
0.22
BoolQ
9,427
3.13
4,10
3.22
3.11
10
StrategyQA
2,061
CREAK
10,176
3.24
3,20
4.14
3.50
0.10%
0.205
0.5%% 0.60% 0.90%
0508 0-100%
Table 5: API Usage Cost (USD) of data creation strat- egy. The cost of utilizing API is calculated in USD, based on the current pricing of gpt-3. 2ott3.55turte 5-tubo as of June 2023, with rate of 0.002 USD per IK tokens. The cheapist strategy shown in bold.
Figure 5: Performance (Accuracy) on RiddleSense us- ing cumulative data splits of the full data.
5.2 Distribution shift during creation.
One natural question about the different "self- reference" strategies is whether the domain drift that they induce from iterative sampling detrimen- tally impacts samples generated later in the cre- ation process. In other words does inclusion of parts of the dataset generated later lead to perfor- mance drops plateaus? In order to answer this question perform evaluation of cumulative data splits on one of our benchmark datasets (Rid- dlesense). Specifically we use incremental per- centages of training data in 10% blocks for all human labeled and synthetically generated datasets, and evaluate the performance of these models on the corresponding test set. The results of this ex- periment are shown in Figure 5.
data creation pipeline for each datasct in our evalu- ation benchmark. The results demonstrate that data creation with LLMs is highly cost-effective, and costs well under $5 USD for every single dataset we created. Factored into this is the cost for data rejected because it was duplicated or ill-formed. Furthermore, our Tree-base rreee "self-reference" strat- egy - which was the most performant on quantita- tive analyses was also among the more econom- ical ones. It was the most economical on half the datasets, while the Contrastive strategy incurred the lowest cost on the others. These expenses based the pricing of gpt-3.5-ture from Ope- nAI as of June 2023.
There are several interesting insights to be gained from these results. Firstly, using human- labeled data leads to much faster convergence; this makes sense since the evaluation is performed on ID test data. Random and Contrastive sampling strategies both of which performed less well on main evaluation do exhibit drops in perfor- mance with later cumulative splits. Meanwhile, Similar and Tree approaches- which were consis- tently better sampling strategics demonstrate a steady rise in performance with added data. Jointly these results indicate that judicious selection of ex- amples for subsequent prompts is needed to counter domain drift. Lastly, the final upward trend of all datasets is meaningful because they indicate that models trained all the datasets do generally ben- efit from more data. While this additional data is likely difficult and expensive to obtain from human annotators, LLMs can create arbitrarily more data at a fraction of the cost.
Conclusion
In this paper, we have presented formal frame- work for data creation using LLMSs and proposed a single-shot formatting example- data creation pipeline that leverages an ciatucttion-oolooiing LLM. Specifically, we showed how multiple var- ied examples can be generated from single seed example in machine-friendly JSON format by conditioning the LLM structured prompt sisting of instructions and a formatted example. We further expand the diversity of the output by intro- ducing "self-refereece" mechanism that selects formatting examples for subsequent iterations of generation from newly created data, and present four different instantiations of the sampling strat- egy. While prior efforts at LLM data creation in the literature have avoided domains where the la- bel space is open-ended and varies from instance to instance, is semantically devoid of inherent meaning, our structured prompts are able to tackle both. On a battery of evaluations our findings indi- cate that LLMs can act as highly coft-effective data
5.3 Data creation cost
Table presents the expenses incurred by lever- aging an fnstruction-folloiinng LLM APIs in our

---

generators for the development of small trainable or tunable models in downstream applications. For example, a budget of $5 USD is enough to generate 2M tokens with gpt-3.5- 5-turbo, and depending on the task can yield several thousand data samples. These models exhibit noteworthy predictive abili- ties in generalizing to desiderata in real-world systems where data can be messy, variable and evolving. The impact of these findings are meaningful in a number of enterprise scenarios, including for applications that require strict governance on predictive models due to pri- vacy and security concerns, or due to response- eiseonse-tmme time service-level agreements, or indeed for small busi- nesses where human annotation costs, especially from domain experts, can be prohibitively high.
herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ARO or the U.S. Government.
References
data, key
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 18:7-17-1901.
7 Limitation
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Despite the capability of our pipeline to inte- grate with a variety of other inslroctionnfollloiing LLMs, our testing is restricted ChatGPT (i.e.. 5-tturbo)) due to performance bottlenecks and time-out issues with LLM APIs in general. While we also experimented with several recent open-source cestruction-oolloiing models that dis- till ChatGPT responses, their abilities to generate well-formatted JSON and comprehend instructions were limited. We expect that the integration of our pipeline with other open-source LLMs will be possible time, as the open-source community attains a performance standard commensurate with commercial products.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint Sttiv220440221111
Christopher Clark, Kenton Lee, Ming-Wo Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, 2019. BoolQ: Exploring the surprising difficulty of natural yes/uo questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics.
An important consideration in our single-shot example-based data creation pipeline is the selec- tion of the initial seed formatting sample. We did not perform exhaustive analysis understand the impact of this seed selection on data creation quality, again due to performance bottlenecks with LLM APIs. While we select this seed at random in this paper, it is possible that a more carefully con- sidered approach for crafting or selection of this example may yield better results.
Nan Du, Yanping Huang. Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of ex-eeess. In International Conference on Machine Learning, pages 5547-5569, PMLR.
Jiahui Gao, Renjie Pi, LIN Yong. Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2022. Self- guided noise-free data generation for efficient zero- shot learning. In The Eleventh International Confer- ence on Learning Representations.
8 Acknowledgement
This work was funded in part by the Defense Ad- vanced Research Projects Agency (DARPA) and Army Research Office (ARO) under Contract No. N660011922 1924033, Contract No. W911NF-21-C- 0002 and Contract No. HR001 12390061, and with support from the Keston Exploratory Research Award. The views and conclusions contained
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.

---

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generater dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Cumputational Linguisties (Volume 1: Long Papers), pages 3309-3326, Dublin, Ireland. Association for Computational Linguistics.
Todor Markow, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2022. A holistic approach to un- desired content detection in the real world. arXiv preprint urXiv:220..03274
Yu Meng. Jiaxin Huang. Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. arXiv preprint arXiv:2202.0453 04538.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLL pages 2567- 2577. Hong Kong, China. Association for Computa- tional Linguistics.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023, Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707.
Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. 2021. Creak: A dataset for com- monsense reasoning over entity knowledge. arXiv preprint arXiv:2109.01653.
Diederik P Kingma and Jimmy Ba. 2014, Adam: method for stochastic optimization. arXiv preprint 1eXiv:1412.698))
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205. 19916.
Andrew K Lampinen, Ishita Dasgupta. Stephanic CY Chan, Kory Matthewson, Michael Henry Tessler. Antonia Creswell, James I. McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02322
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert- In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Hunter Lang, Monica N Agrawal, Yoon Kim, and David Sontag, 2022. Co-training improves prompt- based learning for large language models. In Inter- national Conference on Machine Learning, pages 11985-12003. PMLR.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2655-2677 Seattle, United States. Association for Computational Linguistics.
Bill Yuchen Lin, Ziyi Wu, Yichi Yang. Dong-Ho Lee, and Xiang Ren. 2021. RiddleSense: Reasoning about riddle questions featuring linguistic creativ- ity and commonsense knowledge. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 1504-1515, Online. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin. Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2022. Selec- tive annotation makes language models better few- shot learners. arXiv preprint 222.v:2209
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (Deel10 2022); The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114, 11,, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 11692.

---

Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: llama model. https:// An instruction-folloot http:/staffoddadappm gi thub. com/ waton//tttu-la///att
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang. Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Lever- aging large-scale language models for text augmen- tation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 2225-2239. Punta Cana, Dominican Republic. Association for Computational Linguistics.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apporv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.082.9
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei. Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.1000
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers. Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answer- ing competition. BMC bioinformatics, 16(1))1-228
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021a. Want reduce la- beling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNIP P 2021, pages 4195 4205, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023a. How far camels go? exploring the state of instruction tuning on open re- sources. arXiv preprint arXiv:2306.0475!
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021b. Towards zero-late label language learning, arXiv preprint 109.vv2109.09199
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201.1 1903.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large lan- guage models to follow complex instructions. arXiv prepriat arXivv2304.12. 12244.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu. Tao Yu, and Lingpeng Kong. 2022. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint arXiv:2202.07922.

---

A Appendix
largets open-domain questions where the nec- essary reasoning path explicitly stated in the question, and needs to be inferred through a strategic approach. The answers to these questions are either "yes" or "no""
A.1 Details of Dataset
PIQA (Bisk al., 2020) is binary-choice question answering task, which chooses the most suitable solution for questions related to physical commonsense.
CREAK (Onoe et al., 2021) has been specifi- cally formulated for the purpose common- sense reasoning pertaining to entity knowl- edge. The dataset comprises assertions of en- tities, for which the answers need to be speci- fied either True False
WinoGrande (Sakaguchi et al., 2021) a task that involves selecting the correct binary option to fill in a given sentence that requires commonsense reasoning.
CommonsenseQA (Talmor et al., 2019) is multiple-choice question answering task, which picks the most uppropriate answer general commonsense questions.
A.2
Data Statistics
Dataset
Train
Valid
# Test
PIQA
14,113
1,838
2,000
Riddlesense (Lin et al., 2021) is multiple- choice questions answering task, which picks the most appropriate answer on riddle-style questions that need cognitive process.
WinoGrande (XS)
160
633
634
CommonsenseQA
8,500
1,221
1,241
Riddlesense
3,510
1,021
1,202
BoolQ
9,27
1,35
1,365
PubmedQA
450
50
500
BoolQ (Clark et al., 2019) is question an- swering task that answering questions with simple "yes" or "no" response. Questions naturally occurring queries sourced from the Google search engine. In open-book setting, the model must comprehend the given context in order to provide an answer, whereas in a closed- booo setting, the answer must be provided directly without any context.
BioASQ
670
75
140
StrategyQA
2,061
114
115
CREAK
10,176
685
686
Table 6: Data statistics. Each dataset. We use in-house set which randomly splitted from the train set for those dataset that do not provide test set.
PubmedQA (Jin et al., 2019) is a task that involves answering research questions pertain- ing to the corresponding abstracts of biomedi- cal research papers, and the answers pro- vided in the form of ""yes", "no", "maybe". In our study, we treat "maybe" "no" to ensure consistent output format with other datasets.
BioASQ (Tsatsaronis et al., 2015) offers a range of question answering tasks, covering various categories such as factoid, list, sum- mary, and yes/no questions based on the con- tent of biomedical research papers that have been reviewed by experts in the field. For the purpose of this study, our focus will be re- stricted questions that have binary answers of "yes"" or "no".
StrategyQA (Geva et al., 2021) is a bench- mark for question-answerine that specifically


Reviews:
[{"review_id": "arvX1s6kTW", "reviewer": "Reviewer_47zu", "summary": "", "questions": "1. It would be beneficial to provide the exact prompts used for generating data for the mentioned 10 datasets to facilitate reproducibility and understanding for researchers.\n2. Including some examples of LLM-generated data would allow for an evaluation of the quality and usefulness of the generated data.\n3. It would be valuable to explore how generation tasks are seamlessly integrated into the proposed framework. Additionally, providing experiment results related to the generation tasks would further support the efficacy of the approach.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your feedback.\n\n#### **R1. Training LLMs with the generated examples**\nWhile LLMs are extremely capable, they also have some shortcomings (expensive, slow, black-box nature) and are thus not ideally suited to every scenario. Thus the main goal of our paper was to explore methods to leverage the strengths of LLMs for benefitting smaller models that have complementary strengths to LLMs, by generating diverse training data samples to solve natural language understanding (NLU) tasks effectively. The question of using our framework’s generated data to fine-tune an LLM is an interesting proposal that we leave to future work.\n\n#### **R2 & Q3. Data creation for natural language generation (NLG) tasks**\nWe definitely agree that natural language generation (NLG) tasks like machine translation and text summarization are crucial, and incorporating LLM-based data creation can be beneficial for these tasks as well. However, the focus of our paper is on natural language understanding (NLU), - specifically where the label-space is semantically devoid - which we believe is, if not more difficult, at least an orthogonal problem for data generation. This is because there is a hard requirement on preserving a strong association between generated text and a correct predictive label, whereas NLG allows for a degree of freedom in generated text. Additionally, there is already a body of work exploring LLM data augmentation for NLG [1-3], whereas its application to NLU with labels that are open-ended or lack semantic meaning (the focus of our work) have been underexplored.\n\n[1] Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing., Jung et al., 2023 \\\n[2] PLASMA: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning., Brahman et al., 2023 \\\n[3] AUGESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation., Zheng et al., 2023\n\n#### **Q1&2. Providing exact prompt and examples of generated data**\nWe appreciate your suggestion and would like to assure you that the final version will include the exact prompt and examples of the generated data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a formal framework for LLM-based data generation, providing a structured approach.\n2. Several mechanisms are designed to improve the quality of generated examples, demonstrating the authors' efforts in enhancing the data generation process.\n3. The experiments include a comprehensive analysis of OOD settings and a cost analysis, which contribute to a comprehensive evaluation of the proposed approach.", "reasons_to_reject": "1. The paper lacks results on training LLMs with the generated examples, which would have provided insights into the effectiveness of the generated data in training LLMs.\n2. The focus of the current LLM is primarily on decoder-only LLM's performance on text generation tasks, without specific results on LLM data creation for generation tasks.", "questions_to_authors": "1. It would be beneficial to provide the exact prompts used for generating data for the mentioned 10 datasets to facilitate reproducibility and understanding for researchers.\n2. Including some examples of LLM-generated data would allow for an evaluation of the quality and usefulness of the generated data.\n3. It would be valuable to explore how generation tasks are seamlessly integrated into the proposed framework. Additionally, providing experiment results related to the generation tasks would further support the efficacy of the approach.", "paper_topic_and_main_contributions": "This work addresses the challenge of utilizing Large Language Models (LLMs) as data generators by presenting a formal framework. Additionally, the paper highlights the advantages of formatting generation tasks in JSON format and introduces several sampling strategies to enhance the diversity of generated examples. The experiments conducted thoroughly analyze the benefits of LLM-generated examples in out-of-distribution (OOD) settings.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aTDDb09O15", "reviewer": "Reviewer_6qx1", "summary": "", "questions": "N/A", "limitations": "", "rating": 1, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your feedback.\n\n#### **R1. Self-Instruct (Wang et al., ACL 2023)**\nWhile we appreciate the connection you make with the work of Wang et al., and will follow your recommendation to cite them in our paper, we would like to emphasize that our work is different from the task of creating data for training instruction-following LMs (e.g., Self-Instruct). There have been several recent studies that utilize LLMs to generate instruction-answer pairs (e.g., Alpaca, Vicuna, etc.); we highlight the unique aspects of our work and contrast them to that body of work along 3 axes:\n\n- **Purpose**: The main focus of creating instruction-answer pairs (e.g., Self-Instruct) is to generate a diverse set of instructions that people might use in real-world interactions along with coherent responses. The evaluation metric for models trained on this data emphasizes assessing the coherence, helpfulness, and harmlessness of the responses. **However, ensuring correctness is a separate challenge altogether.** Several studies using such instruction-answer pairs have demonstrated poor performance on NLU tasks (See Table 4 in [1] which shows a model trained on Self-Instruct shows the worst performance on MMLU). On the other hand, our primary focus is generating data specifically for NLU tasks. The data we create for NLU tasks must encompass diverse input distributions that are **specific to each task**. At the same time, it is crucial to ensure the **correctness** of the corresponding labels.\n\n- **Methodology**: We contend that our methodology is not an advanced version of the Self-Instruct methodology, but rather a different approach that may share some similarities. (1) Unlike Self-Instruct, which generates instructions and then uses them to generate responses in a two-step process, our approach directly generates responses in a single step; (2) Additionally, Self-Instruct randomly selects 8-shot instruction examples from a pool of 175 instructions, similar to the approach used in ToxiGen [2] mentioned in our paper (line 231) while our work starts with only one seed example and generates data from there.\n\n- **Model**: The main objective of generating instruction-answer pairs (e.g., Self-Instruct) is to train models that have a high number of parameters, like decoder-only or encoder-decoder models, to understand and follow instructions accurately. By using a large parameter size, these models can provide coherent and meaningful responses that reflect their vast knowledge base. However, our goal is to generate training data suitable for smaller models to solve very specific natural language understanding (NLU) tasks effectively. A smaller model does not have the representational capacity of an LLM, and therefore the desiderata for data generation in our scenario are both qualitatively and quantitatively different from a Self-Instruct scenario.\n\n[1] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources., Wang et al., 2023 \\\n[2] TOXIGEN: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection., Hartvigsen et al., 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors made some incremental improvements over self-instruct by considering more complicated self-reference strategies including contrastive selection, similar selection, and tree selection.\n\nThe distribution shift during data creation is an important issue and is analyzed in this work.", "reasons_to_reject": "The primary reason to reject is that, the authors proposed an improved variant of self-instruct (Wang et al.) but did not even cite it. It is impossible to consider this as concurrent work with self-instruct so it is unacceptable to miss such an important prior work. It would be okay for me to claim something like \"We improve the prior methods and demonstrate its effectiveness on other datasets\".\n\nIn contrast to the self-instruct paper, which generates diverse tasks, the tasks considered in this work seem rather limited.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "The paper aims to propose a **data creation** method based on LLM, in particular, `gpt-3.5-turbo`.\n\nThe proposed methods can be regarded as an incremental improvement over Self-Instruct (Wang et al.). In particular, the authors improve the \"self-reference\" by considering contrastive selection, similar selection, and tree selection.", "excitement": 1, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Wang et al. Self-Instruct: Aligning Language Models with Self-Generated Instructions\n", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "qfo6VCw2hc", "reviewer": "Reviewer_zQ9a", "summary": "", "questions": "A. How to do human evaluation in Figure 5?  \nB. How to implement \"discarding duplicate and ill-formatted data\" in section 3?  \nC. How to avoid the LLM-related faithfulness issue during data creation?  \nD. What is the optimal setting of your grid search? And are they generally the same in different evaluation dataset?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your feedback.\n\n#### **R1. Comparison to other LLM-based data generation**\n\nThank you for your suggestion, we definitely agree with your comment. Unfortunately, the lack of publicly available prompt templates from related work have made it difficult to yield a fair comparison. We have, in fact, tried to replicate those papers by creating prompts based on their descriptions, but the resulting generated data have generally been poor and difficult to control.\n\nThe poor quality could be because: a) we have imperfectly translated the authors’ descriptions to effective prompts; and/or b) because our method is in fact superior. While we would have certainly loved to claim the latter, there was no way for us to tell that this was the only contributing factor, and therefore we decided to not include those results.\n\nHowever, if it helps to provide context, we are certainly happy to follow your recommendation and include some of the results you have proposed in our final paper.\n\n#### **R2&Q4. Optimal setting of grid search**\n\nWe appreciate your question and recognize the importance of clearly explaining our optimal hyperparameter settings. First, we set the `number_of_examples` in a batch (line 289). For this, we found setting to five is the maximum number for efficient generation without encountering formatting errors. Additionally, we utilize temperature and top_p as hyperparameters for the language model (LLM) generation settings. For this, we found that the setting that yields the most randomness (i.e. temperature and top_p set to 1) is the most efficient way to avoid duplication and enables faster generation of the desired number of examples. Finally, the random seed used in our paper for the initial data selection is set to 42.\n\nWe will detail and explain all these choices and findings in our final paper.\n\n#### **Q1. Human evaluation in Figure 5**\n\nModel training with human evaluation is training the model using the original training data. In this method, humans are considered as oracle (100% accuracy) and annotate the training data in a sequential manner. To simulate this process, the training data is shuffled, and a certain percentage of the data (N%) is randomly selected for the model training. We will clarify this in the paper.\n\n#### **Q2. Discarding duplicate and ill-formatted data**\nThanks for your question. We keep track of all previously created inputs and check whether the newly created input has already been previously generated. Also, we filter out ill-formed JSON data by checking whether it can be parsed using the python `json.loads()` function. We will provide more details about these in the final version.\n\n#### **Q3. Faithfulness issue of LLM**\nThank you for your great question; we agree that LLM faithfulness is a very important issue. For our paper we think it makes sense to break down that measure into two axes: consistency and correctness.\n\nTo validate the created data for consistency, we ran an experiment where we asked the LLM to generate predictions on our generated data, and found that those predictions match our generated labels 100%. This indicates that the LLM, whether correct or not, at least maintains label consistency. Meanwhile, correctness of generated outputs is a more challenging metric to gauge, especially at the level of individual data points. However, we contend that downstream performance of the smaller model trained on generated data is a proxy measure of correctness, and as a result our approach - which yields improved results - does generate data that is generally more correct than comparative baselines.\n\nThis is, of course, not to say that faithfulness is a solved problem, and there are several ways we could try to improve the quality of our generations. For example, we could add a verification step that checks the factuality and correctness of every generated output as it is produced by the model (at the cost of more LLM API calls), perhaps even augmenting the verification with retrieval from the web to provide supporting evidence. We leave these explorations to future work.\n\nWe will make sure to include a detailed discussion of faithfulness in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Addressed the important problem about how to use LLM as data creator efficiently. Given the facts that the examples in few-shot are not always available, and the prompting methods in data creation pipeline may require specific label description, they proposed a unified iterative method that is general and efficient, avoiding these two issues \n2. Creatively proposed the self-reference method evaluated by random, similarity, or tree search, is added to guarantee the diversity and balance of generation.\n3. Solid experiment with various domains under in-distribution (ID) and out-of-distribution (OOD) setting, showing that under the ID setting, such method may be worse in colossal common domain but useful in specialized domain, and the LLM-generated data leads to more robust performance than using the original data in the OOD setting.", "reasons_to_reject": "1. The paper didn’t provide the comparison to other LLM-based data generation methods. The authors did argue in section 4.2 that it is due to the lack of off-the-shelf prompt template / solution of the tasks listed in this paper. Also, the workload of such comparison seems to be stressful. However, a selling point of this paper is the advantage of their LLM-based data creation method compared to other counterparts, therefore at least some quick, sub-optimal performance comparisons (e.g. by new customized prompt) are desirable. \n2. Miss some technique details, e.g. the optimal setting of the grid search (which matters the generalizability of the method), the random seed of the first data selection (which matters the reproducibility), the consistency of such setting (which matters the actual cost since otherwise you need to do grid search each time).", "questions_to_authors": "A. How to do human evaluation in Figure 5?  \nB. How to implement \"discarding duplicate and ill-formatted data\" in section 3?  \nC. How to avoid the LLM-related faithfulness issue during data creation?  \nD. What is the optimal setting of your grid search? And are they generally the same in different evaluation dataset?", "paper_topic_and_main_contributions": "This paper aims to make large language models (LLMs) as better data creators. The authors first unify the frameworks of current LLM data creation methods using the graphical model paradigm. Then, they propose a single yet general format for various, challenging data creations. In addition, self-reference method is added to guarantee the diversity and balance of generation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Wrong symbol when citing “Self-guided high-quality data generation in efficient zero-shot learning”\n2. It is recommended to explain in the caption of Figure 4 that which part is generated directly, and add some more detail generated examples in the appendix to give readers a better sense of the data generation.\n3. It is encouraged to add the ethics section to mention the inherited bias of LLM when generating the data"}}]

Author rebuttal:

Meta-review: The paper addresses the challenge of efficiently utilizing Large Language Models (LLMs) as data creators for natural language understanding tasks. It introduces a unified data creation pipeline that requires only a single formatting example and applies to a wide range of tasks. The paper emphasizes the cost-effectiveness of instruction-following LLMs in generating data and demonstrates that models trained with LLM-generated data outperform those trained with human-labeled data on out-of-distribution evaluation while maintaining comparable performance on in-distribution tasks.

Reviewer 1 raised concerns about the lack of results on training LLMs with the generated examples and the paper's focus on NLU tasks. The authors responded by explaining their focus on NLU tasks and leaving the use of generated data for LLM fine-tuning as future work.
Reviewer 3 expressed the need for comparisons with other LLM-based data generation methods, additional technical details, and clarification on various aspects of the paper. The authors addressed these concerns by agreeing to include sub-optimal performance comparisons, provide detailed hyperparameter settings, and explain various processes in the final paper.

**Note:** The review and scores of reviewer "@6qx1" need to be ignored due to the low-quality review.
Decision: Accept-Main