TarGEN: Targeted Data Generation with Large Language Models
Authors: Himanshu Gupta
Kevin Scaria
Ujjwala Anantheswaran
Shreyas Verma
Mihir Parmar
Saurabh Arjun Sawant
Chitta Baral
Swaroop Mishra
Venue: COLM (2024)
Keywords: Synthetic Data Generation, Target Data Generation, Seedless Generation, Self Correction
TL;DR: We propose a new method to leverage LLMs to generate high-quality, seedless, diverse synthetic data boosting performance on various NLP tasks.

Abstract: We present TarGEN, a multi-step prompting strategy for generating high-quality synthetic datasets using LLMs. An advantage of TarGEN is its seedless nature; it does not require specific task instances, broadening its applicability beyond task replication. This differentiates it from other data generation techniques, as it can be leveraged for novel or highly domain-specific tasks with no existing data instances.
We augment TarGEN with a self-correction module that enables LLMs to rectify inaccurately labeled instances during dataset creation, ensuring reliable labels. To assess our technique’s effectiveness against existing baselines, we emulate eight tasks from the SuperGLUE benchmark to create a "synthetic" version and finetune various language models on both synthetic and original training sets. Evaluation on the original test set reveals that models trained on the synthetic datasets perform ∼ 1 − 3% points higher than those trained on original datasets. Finally, when pre-finetuned on our "synthetic" SuperGLUE dataset, Llama2 (7B) yields impressive results on the OpenLLM leaderboard, surpassing the model trained on the Self-Instruct dataset by 2.62% points. Our analysis reveals that the synthetic data generated by TarGEN not only improves model learning, but also has comparable or higher levels of complexity, diversity, and similar levels of bias in comparison with the original data.

Published as a conference paper at COLM 2024
TarGEN: Targeted Data Generation with Large Language Models
Himanshu Guptalt Kevin Scaria 14
Ujjwala Anantheswaran 16+ Shreyas Verma
Mihir Parmar Saurabh Arjun Sawant
Chitta Baral Swaroop Mishra
Arizona State University Georgia Institute of Technology (hgupta35, kscaria] @asu.edu
Abstract
We present TarGEN, a multi-step prompting strategy for generating high- quality synthetic datasets using LLMs. An advantage of TarGEN is its seedless nature; it does not require specific task instances, broadening its applicability beyond task replication. This differentiates it from other data generation techniques, as it can be leveraged for novel or highly domain- specific tasks with no existing data instances. We augment TarGEN with a self-correction module that enables LLMs to rectify inaccurately labeled instances during dataset creation, ensuring reliable labels. To assess our technique's effectiveness against existing baselines, we emulate eight tasks from the SuperGLUE benchmark to create a "synthetic" version and fine- tune various language models both synthetic and original training sets. Evaluation on the original test set reveals that models trained on the syn- thetic data perform ~ 1 3% higher than those trained original datasets Finally, when pre-finetuned on our "synthetic" SuperGLUE dataset, Llama2 (7B) yields impressive results on the LpenLLM leaderboard, surpassing the model trained on the Self-Instruct dataset by 2.62%. Our analysis reveals that the synthetic data generated by TarGEN not only improves model learning, but also has comparable or higher levels of complexity, diversity, and similar levels of bias in comparison with the original data
1 Introduction
Large Language Models (LLMs) like ChatGPT, Llama (Touvron et al., 2023a;c), and Mistral (Jiang et al., 2023) have showcased impressive results across plethora of tasks (OpenAT, 2023; Brown et al., 2020). As LLM capabilities advance, the tools test the extent of these capabilities become insufficient (Liu et al., 2022b; He et al., 2023; Valmeekam et al., 2022; Chen et al., 2021). This is particularly true for domain-specific datasets, as the creation of expertly curated evaluation benchmarks is time and labor-intensive (Clark et al., 2018; Suzgun et al., 2022; Wang et al., 2022). Several synthetic dataset creation methods such as Self-Instruct (Wang et al., 2023), AttrPrompt (Yu et al., 2023) and ZeroGen (Ye et al., 2022a) have been proposed primarily for text classification tasks. These approaches employ in-context learning to generate synthetic data points that resemble their prompt exemplars, thereby inherently constraining their ability to produce diverse examples. Additionally, these approaches are often uniquely tailored to certain downstream tasks or datasets, reducing their adaptability. Consequently, they may be unsuited to synthesizing domain- specific tasks/ benchmarks evaluating novel tasks where there is lack of sufficient data samples to use prompt exemplars.
To mitigate these issues, we introduce TarGEN, multi-step prompting strategy (Fig 1). This approach consists of four steps. First, we initialize a set of contexts to inject semantic diversity, followed by the generation of task-specific elements we call "instance seeds"-
!Codebase: https: thg con/ hwtp//vvinscaraa/eeee TarGGEN * Currently in Microsoft Cur- rently in Google Meed 1 Currently in Amazon (The work was done prior to joining Amazon) Equal Contribution
1

---

Published as a conference paper at COLM 2024
Generate list 50 verbe or noune which have more than meaning
list of fewer distinct definitions of the word.
(WORD),
Original dataset
Task description
LLM shot
Input:
She lost her keys. His key. doeen't work. Label:
label TRUE
label FALSE
TRUE
given definitions, print any one definition,
of possible
Given word
definitions:
Input: His key. broke.
For each definition, print sentence containing word that definition.
Discipline is key for Label: FALSE
tollowed by sentences containing the
word in this delinition.
Synthetic instances
Label-constrained generation pipeline
Task description
task evaluate whether, based on these instructions and input, the output is correct or incorrect. Provide explanation for your ressoning.
Synthetic dataset
LLM zero shot
Evaluation examples
Input: rock my shoe. I threw a rock. Label:
Now evaluate the input and oulput below based task instructions. and print whether the output is correct or incorrect. Remember to provide explanation for your evaluation. Remember, the actual answer based on. output.
TRUE
Input:
Don't rock the boat. I play in rock band. Label: FALSE
input and the task instructions,
Self-correction meta-prompt
Figure 1: An overview of using TarGEN to generate instances for the WIC task. We first create set of prompts (boxes 2 in figure) to generate instance seeds, linguistic com components unique to task instance. Next, we create label-specific prompts (box 3) that generate instances based on instance seeds and the relationship implied by the label for this task. We use zero-shot LLM inference to generate an initial set of synthetic instances. The instances are then passed to our self-correction module consisting of single meta-prompt which allows us to re-label mislabeled data instances, helping us reduce noise. Hence, based on the task description, we obtain high-quality synthetic instances to evaluate a task.
elements that form the unique basis of each instance. These seeds can be sentences, passages, or more atomic elements but are not input exemplars. Then, for each "instance seed", we formulate a label constraint that uses these seeds to generate data instance attributable to the constrained label.
Finally, we leverage our evaluator model for self-correction over the generated instances, and re-label them wherever necessary (SE). The self-correction module integrated into the pipeline contributes to noise reduction and enhances the overall quality of generated content. Using TarGEN, we can produce high-quality diverse datasets with precise labels. This method offers the benefit of not necessitating pre-existing task instances starting points for generation. Consequently, novel benchmarks, relying only on the provided task descriptions.
suitable for domain-specific tasks and the generation of
We use TarGEN in conjunction with ChatGPT to generate data for natural language tasks (textual entailment, question answering) used in benchmarks like SNLI, MNLI, and SciTail. To ensure fair evaluation against existing baselines, we pick tasks from the comprehensive and widely-recognized SuperGLUE benchmark and generate synthetic instances for them based solely on task descriptions. We treat each task as a standalone language understanding challenge, without directly referencing the benchmark or dataset name during generation. We train several models from different families (encoder only, encoder-decoder, and decoder only) on the synthetically generated train set and the original SuperGI.UF train set and evaluate these models on the original test set. We find that models trained on the synthetic train set perform at par as opposed to models trained on the original train set. Instruction tuning results in 3.42% increase for Flan T5 models and 3.24% improvement for Pythia GPT models. We also observe that training on synthetic data
multi-task fashion yields average improvement of 4.73%, 3.21% %, and 2.94% for T5-3B, Llama2-7B, and Mistral-7B respectively, over multi-task training on the original data (64) The self-correction module plays vital role in the improved performance; multi-task training with T5-3B on the self-
2

---

Published as a conference paper at COLM 2024
corrected synthetic data improves performance over self-corrected synthetic data all tasks by 5.9% on average. We also conduct a comparative study between TarGEN and Self-Instruct (Wang et al., 2023) by pre-iineuuning Llama2 (7B) (Touvron et al., 2023c) both datasets and evaluating them on the Op benchmark Our findings indicate that Llama2 rreffinetuned on our dataset outperforms Llama2 trained using Self-Instruct by
2.62% points.
systematic human evaluation was conducted appraise the quality of generated instances. Our thorough analysis of the data reveals the robustness of our synthetically generated SuperGLUE dataset in terms of dataset difficulty, diversity, and bias. exhibits equivalent or higher dataset difficulty (lower V-usable information (Ethayarajh et al., 2022)), showcasing the adequate complexity of our datasets. Furthermore, our datasets have comparable lexical diversity (Yu et al., 2023) and consistently display lower cosine similarity between intra- dataset text pairs, highlighting the datasets' rich and distinct content. Additionally, we measure the datasets' lexical diversity in terms of vocabulary count. Our results indicate that the datasets generated using TarGEN consistently exhibit higher lexical diversity than their original counterparts. In terms of bias, our datasets align closely with the original datasets, demonstrating balanced representation of various named entities such as Geopolitical Entities (GPE), Person Entities, and Nationalities Religious or Political groups (NORP). A detailed analysis is present in $5.
2
Related work
Recent research has witnessed the emergence of various methods harnessing LLMs as synthetic data generators (Long et al., 2024; Lu et al., 2023; Anantheswaran et al., 2024; Li et al., 2023a). Specifically, in the context of few-shot classification tasks where labeled data is scarce, several approaches have been introduced. SuperGEN (Schick & Schütze, 2021) and ZeroGEN (Meng et al., 2022a) leverage LLMs to produce synthetic data. For zero-shot tasks, SunGen (Gao et al., 2023) employs noise-filtering techniques, while ProGEN (Ye et al., 2022c) utilizes model feedback to ensure generated data quality. Similarly, (Chia et al., 2022) introduces structured prompts for tasks like relation triplet extraction. Moreover, (Liu et al., 2022a) and (Wiegreffe et al., 2022) propose synthetic data generation methods for natural language entailment (NLT) tasks and free-text explanations in human-AT collaborative setting. There also have been approaches proposed to generate tabular data (Borisov et al., 2023) and instruction data (Peng et al., 2023; Sun et al., 2023).
The research direction in synthetic data generation predominantly focuses on zero/ pero/few-shol few-shot classification or entails finetuning (Chen et al., 2023) or iterative finetuning of open-source LLMs (Yu et al., 2023). In contrast, our method is simple, lightweight, and adaptable even to closed-source L.Mss. Additionally, the aforementioned methods often rely on seed samples where expert annotators are employed to curate and label sample set of considerable size. These seed samples are used to generate additional synthetic samples for various downstream tasks.
We exploit the generative capabilities of LLMs to completely generate seed instances that drive the diverse synthetic dataset generation. Furthermore, our approach uses multi- step prompting strategy along with self-correction to for targeted data generation while preserving quality in terms of diversity, bias, noise, and mislabelling. Finally, existing dataset generation methods often are limited by reliance on seed tasks from the original dataset (Wang et al., 2023) whereas our seedless pipeline leverages high-level dataset characteristics for the generation process. We compare our framework with other popular data generation approaches along various axes in SA.1. For an empirical comparison with these approaches, see SA.2 of the Appendix.
3

---

Published as a conference paper at COLM 2024
3 TarGEN
3.1 Problem formulation
Given dataset for language task f, its data points can be expressed as (d,1) such that there
exists a function f: :
fi(d) = 1, D
where ft is mathematical representation of the task f, D is the instance input, and is the instance label from L, the label space for the given task. We formalize dataset gen- eration as sequence of aabel-contrrained text generation problems, where the genera- tion of an instance is constrained by its label value. This allows us to control label dis- tribution in our synthetic dataset and craft high-quality instances by clearly defining the relationships between instance and la- bel.
Datasets
Task Type
Instances Split
AX-g
NLI
Not Ent: 146 | Ent: 138
BoolQ
Bin. Class.
True: 2535
False: 1764
CB
NLI
Cont: 119
Ent: 115 Neut: 16
COPA
Bin. Class.
Choice 1: 195
Choice 2: 107
ReCoRD
MCQ
1778 MCQs
RTE
NLI
Not Ent: 1241
Ent: 1249
WiC
Bin. Class.
True: 2433
False: 2410
WSC
Bin. Class.
True: 259
False: 285
Table 1: Statistics of the dataset. Following ab- breviations are used: Bin. Class: Binary Classi- fication; Ent: Entailment, Cont: Contradiction, Neut: Neutral; NLI: Natural Language infer- ence
This circumvents the need for any seed in- stances from the original dataset; i.e. for any given task, a dataset can be generated from scratch by formulating its generation function from the task description. The task- specific label-constrained dataset generation approach is given below:
Mr U (Lin,, IeCx=1
where N; is the number of samples for the label 1, is the task, and Gr,1 is an inverse function and in the n instance seed. Formulating these task- and label-
such that (l) specific prompting strategies forms the crux of our simplified data synthesis pipeline. While these individual prompts are task-specific, the nature of these prompts, and the sequence they occur in, follow framework engineered to create diversity and improve coverage. The stages of this framework are as follows:
Step We generate set C of "contexts", or "settings" that provide unique semantic scope, such as "geopolitical news", "book review", or "movie script". These provide contexts within which model can simulate a naturally occurring excerpt, to maintain semantic diversity.
Step 2 We generate set of passages, sentences, task-specific elements, which we call "instance seeds". These seeds form the basis for each instance of the task.
Step 3 We use the instance seeds inputs to the generation prompt. This prompt is descriptive formulation of the inverse generation function GJ allowing us to generate data instances for the task.
Step 4 We pass all generated data instances through a single-step self-correction process, where
task-specific prompt to reinforce the task instructions and identify and correct any mislabeled instances. This helps reduce noise and improve overall dataset quality.
3.2 Task-specific prompting strategies
The generation function G1 for each task is included in ŞE. We generate a common set of contexts (Step 1) for almost all tasks.
4

---

Published as a conference paper at COLM 2024
AX-g
BoolQ
Og
Syn
Og-I
Syn-I
Og
Syn
Og-I
Syn-I
Cerebras
73.23
76.00
72.80
77.63
84.12
86.11
84.77
88.61
Pythia
74.90
78.43
74.57
79.80
83.29
84.02
82.61
85.05
T5
77.12
79.14
77.65
79.84
84.43
85.64
84.16
86.87
Flan
78.03
80.23
76.20
81.70
84.43
86.01
85.08
88.21
RoBERTa
77.10
78.17
76.35
82.14
84.43
84.56
84.07
87.56
СОРА
RTE
Cerebras
80.56
81.98
80.01
82.11
81.12
83.25
81.02
85.42
Pythia
79.98
80.74
80.54
83.27
82.98
83.65
82.85
87.87
T5
82.12
83.34
82.32
85.91
86.18
86.10
86.79
88.92
Flan
81.78
82.41
82.36
86.24
88.91
90.19
89.86
89.22
RoBERTa
82.12
83.98
80.99
83.16
88.20
89.01
88.32
88.84
CB
ReCoRD
Cerebras
88.21
89.15
88.60
88,93
67. .8.8
68.12
68.42
71.45
Pythia
91.93
89.63
89.74
93.03
68.83
69.19
68.61
73.36
T5
90.32
92.43
89.84
92.05
71.13
72.23
70.44
72.07
Flan
89.02
93.21
88.06
94.32
70.21
70.11
71.58
76.83
RoBERTa
87.86
90.12
92.36
90.48
69.88
70.21
70.29
70.02
WiC
WSC
Cerebras
66.78
68.72
66.70
70.22
82.12
85.28
86.31
90.66
Pythia
68.33
71.77
70.53
70.84
86.71
87.32
86.44
88.09
T5
68.01
71.13
68.35
69.41
85.56
85.95
85.91
88.43
Flan
70.12
72.45
69.29
71.43
86.23
88.13
88.08
88.35
RoBERTa
69.90
70.06
70.16
71.23
87.21
85.65
82.37
83.21
Table 2: Performance of various models over SuperGLUE tasks. For each dataset, we compare the performance of these models (Cerebras, Pythia, T5, FLAN, RoBERTa) when trained on 4 distinct variants: Og (original train data), Syn (synthetic train data), Og-I (Instruction tuning on original data), Syn-I (Instruction tuning on synthetic data). For each task, denote the highest performing model trained on original train data in green, and the highest performing model trained on synthetic train data blue. All results are presented in %. We measure performance in accuracy, except in the case of ReCoRD, where we use the Rouge-L score.
Dataset Statistics: We choose the following tasks from the SuperGLUE benchmark: 1. CommitmentBank (CB). 2. Choice of Plausible Alternatives (COPA). 3. Recognizing Textual
Entailment (RTE). 4. Word-in-Context (WiC). 5. Winograd Schema Challenge (WSC). 6. BoolQ. 7. Reading Comprehension with Commonsense Reasoning (ReCoRD). and 8. Wino- gender Diagnostics (AX-g). Given the challenges posed by each dataset, we employ TarGEN to create synthetic instances that can be used to evaluate language model performance. Table 1 showcases the instances split used in the original and synthetic datasets. Since label generation is controlled, synthetic SuperGLUE was created to match the exact number of original instances, while maintaining a balanced label distribution". The exact data synthesis pipeline and prompts used for each dataset can be found in ŞE.
CB (De Marneffe et al., 2019): We generate sample pairs that share the relationship between label E {entailment, neutral, confradiction), and based
given context E C.
CОРA (Roemmele et al., 2011): We generate instances for CAUSE and EFFECT relations. In this case, Step 3 involves generating premise and hypotheses for given context. For each relationship r, we generate (1) sentence pairs (P,C) such that the premise and hypothesis
"Due to ChatGPT budget constraint, ReCoRD dataset was truncated to have 1778 instances, BoolQ dataset had 4299 instances and MultiRC dataset was skipped.

---

Published as a conference paper at COLM 2024
Model
BoolQ
WiC
CB
AX-g
ReCoRD
RTE
WSC
COPA
T5-3B Og
89.02
68.42
84.54
41.66
68.52
87.72
72.11
94.65
T5-3B Syn
90.04
74.78
93.22
49.13
76.59
92.41
73.07
95.20
Llama2-7B Og
88.51
72.26
87 37,66
48.71
72.34
88,76
75,43
94.85
Llama2-7B Syn
90.12
73.21
94.87
51.32
78.81
92.89
77.32
95.72
Mistral-7B Og
89.91
74.32
88,92
50.34
76.87
89.97
74.34
91.92
Mistral-7B Syn
90.81
75.95
94.21
54.19
80.17
93.09
78.21
93.43
Table 3: Results 15-3B, Llama2-7B, and Mistral-7B, trained in multi-task fashion. Og: Using combined set of original datasets to train the model. Syn: Using synthetic versions to train the model. All numbers in %. For each task, denote the highest performing model trained on original train data in green, and the highest performing model trained on synthetic train data in blue.
BoolQ
WiC
CB
AX-g
ReCoRD
RTE
WSC
COРA
T5-3B w/o 85.32
69.94
91.09
35,28
72.38
87.65
63.26
T5-3B with 90.04
93.22
49.13
76.59
92.41
73.07
95.26
Table 4: Results using a T5-3B model without and with the self-correction (SC) module. We observe that using self-correction improves performance over all datasets by 5.9% average.
Task: AXG
Task: RTE
Task: CB
Task: COPA
ent
126
122
ent
947
470
800
ent
1
100
choice 2 choice
137
54
125
600
15
110
10
-50
100
hon
12
-50
302
770
contr
-400
11
5
ent
ent
non
ent
neutr
contr
choice
choice
Task: BOOLQ
Task: WIC
Task: WSC
2000
taise
491
120
2000
false
1120
217
false
1500
1000
true
1267
2413
1290
2216
true
100
faise
true
faisa
tuee
false
true
Figure 2: Matrices showing the effect of the self-correction step across various datasets of SuperGLUE. The row values show the number of labels that were originally assigned to that label (ent, non: entailment, non-entailment; neutr, contr: neutral, contradiction). The number in cell (i,j) reflects the number of labels originally assigned to label which were re-labeled label j after self-correction. While the majority of the instances had their labels reaffirmed by self-correctin,, significant number of instances were re-labeled as result of this step.
share the relationship specified, and (2) an alternate hypothesis Call which explicitly does not share the specified relationship with the premise P. Thus (P, C) e (P, Catt) & The label space for this task is defined as L = (C1, To ensure an even label split, we alternatively select attribute instances of C as C1 and Call C2 and vice versa.
RTE: A collection of textual entailment challenges. For this task, step 2 consists of generating set of premises P instance seeds. For each p E P, we then generate hypotheses that are either logically sound (I = entailment) or logically unsound, i.e. do not follow the premise notentailment).

---

Published as a conference paper at COLM 2024
WiC (Pilehvar & Camacho-Collados, 2019): We curate a list of homonyms (5) and all their defi- nitions (M,Vs E 8). These act instance seeds. For each m E Ms, given the label True, we gen- erate a pair of sentences (d1, d2) containing the word S, such that the definition of s in dl and d2 m. For the label False, we randomly choose m1, m2 E Ms, ml + m2 and generate d'1, d2 such that the definitions of in di and d2 are ml and m2 respectively, making them distinct in word sense.
Og
Syn
Og-I
Syn-I
Cerebras
78.64
79.83
78.80
81.88
Pythia
79.48
80.59
79.49
82.66
T5
80.72
82.00
80.83
82.94
81.12
82.84
81.49
84.54
RoBERTa
79.98
81.47
80.08
82.08
Table 5:
Avg. single-task performance across tasks, models, and data variants. Og, Syn, and I represent Original, Syn- thetic, and Instruction tuning respec- tively.
WSC (Levesque et al., 2012): For each con- text, we generate pairs of distinct noun phrases N2) with identical plurality. This pronoun
and noun phrases act as instance seeds. For each pair (N1,N2) we then generate text con- taining N1 and N2 with all pronouns labeled with coreferred noun phrases. We randomly select an seeguuuusly-corffereee pronoun from the text. Based on this pronoun and the label constraint, we affix noun phrase to the input instance.
BoolQ (Clark et al., 2019): For this task, we generate passages with multiple entities and inter-entity relations which act as instance seeds in Step 2. In Step 3, we gen- erate query, based on the passage p E P and the label constraint / E {Yes, No). In the case of ! = Yes, the query information that can be inferred from the passage. For I 1= No, we generate query that contradicts the passage.
L2SSG
L2SI
ARC
54.65
52.30
HellaSwag
79.25
77.09
MMLU
45.63
41.60
TruthFulQA
42.34
37.58
Winogrande
71.37
69.46
ReCoRD (Zhang et al., 2018): This dataset evaluates un- derstanding implied entailment relationships. In Step 2, we generate set of passages A act as instance seeds. Next, for each article a A, we generate complex, context-relevant sentence, and subsequently obscure single entity reference.
GSMSK
1.98
1.44
Average 49.20
46.58
Table 6: Comparison of Syn- thetic SuperGLUE, Self Instruct Dataset, and AttrPrompt. T5-3B pre-finetuned on both datasets individually and finetuned on OpenLLM datasets.
AX-g (Rudinger et al., 2018): For this task, we generate 10 subject pairs that are from the same domain space, in Step 2. For each subject pair, we then generate an independent clause containing these subjects. These in- dependent clauses are then used to generate dependent clauses coreferent with each subject, act as instance seeds. In step 3, we use these subject-specific dependent clauses and the subject pairs to generate gender-agnostic hypotheses based on label constraints.
3.3 Self-correction
Despite learning capabilities, LLMs demonstrate inconsistent reasoning (Ye & Durrett, 2022). We remedy them by implementing self-correction, an evaluation strategy that corrects inconsistent labels in the data synthesis process. We leverage LLM as an evaluator model (ChatGPT in this case) to verify the alignment between the generated instances and their labels, as well as the alignment between these instances and the task description. Self- correction consists of a single meta-promp! that is common to all tasks. The task instructions and task-specific validation examples are used to augment the meta-prompt and tailor it to each generated dataset. This meta-prompt and the task instructions are in ŠG. Based on the provided input, the meta-prompt helps evaluate the correctness of its attributed label. this label is deemed not correct, the evaluator model generates the correct label based on the instructions and instance input. Notably, from Fig it it can be seen that a significant number of instances require relabeling especially for high complexity tasks such as AX-g and WSC.
7

---

Published as a conference paper at COLM 2024
Estimated Density
axg
Estimated Density
boolg
Estimated Density
cb
Estimated Density
copa
Orignal Synthatic
100
Organal
Griginal
Synthetic
Santhheti
Synthetic:
10
40
50
5
20
0.5
1.0
0.950 0.975 1.000
Cosine Similarity
Cosine Similarity
Cosine Similarity
Cosine Similarity
Estimated Density
record
Estimated Density
Estimated Density
wic
Estimated Density
WSC
Orighnl
Orginel
10
Orginal
Synthedi:
30
Syntheti
20
Synthetic
Syntheti:
20
20
10
10
10
0.9 Cosine Similarity
1
Cosine Similarity
Cosine Similarity
Cosine Similarity
Figure 3: Comparison of semantic diversity across datasets among the original and the synthetically generated dataset. It can be seen that the original datasets' cosine similarity is higher for most tasks as compared to the synthetic datasets' which has a consistently lower cosine similarity indicating higher semantic diversity.
Dataset
BoolQ
WiC
CB
AX-g
ReCoRD
RTE
WSC
CОРA
Original 251.8k
65.8k
6.4k
3.2k
160.3k
79.1k
8.8k
5.1k
Synthetic
190.6k
76.4k
16.6k
4.4k
236.7k
49.4k
8.9k
5.2k
Table 7: Lexical diversity of the dataset. Figures in red correspond where diversity of synthetic dataset is limited. Blue correspond where the synthetic dataset has more diversity than original.
4 Experiments and results
We train five models in the single-task learning (STL) setting for each original and synthetic dataset separately. We evaluate them on the original test set. The above experiments are repeated with instruction tuning as well. We also perform multi-task learning (MTL) experiment where three models are finetuned in multi-task fashion on the original and synthetic datasets separately (Mishra et al., 2021). The results in $4.1 are the average of five runs. Models: We use RoBERTa Large (354M) (Liu et al., 2019), Pythia GPT (410M) (Biderman et al., 2023), Cerebras GPT (590M) (Dey et al., 2023), Flan T5 Large (780 M) (Chung et al., 2022), T5 Large (780M) (Raffel et al., 2020) in STL setting, and T5-3B, Llama2- 7B (Touvron et al., 2023b), and Mistral-7B (Jiang et al., 2023) in the MIL setting.
Evaluation metric: Following the SuperGLUE benchmark, we use accuracy to measure performance over all tasks, except for ReCoRD, which we evaluate on Rouge-L score.
4.1 Results
Table showcases the results of models trained on both original and synthetic datasets for each task. Models finetuned on synthetic data consistently match or outperform variants finetuned on the original data. Furthermore, performance improves by 3% by instruction tuning (Mishra et al., 2021; Wei et al., 2021; Scaria et al., 2023; Gupta et al., 2023).
Table 5 gives the average model-wise results for the same. Table 3 denotes the results when the datasets are trained in multitask fashion with larger models. Table 4 highlights the effect of self correction module when T5 3B is trained without self correction module.
8

---

Published as a conference paper at COLM 2024
5 Analysis
In this section, we present a comprehensive anal- ysis of the synthetic data generated, examining both quantitative and qualitative aspects.
Original BOOLQ
Country
United States
Human Evaluation As part of the quality eval-
GPE
United Kingdam
uation, a systematic human evaluation was car- ried out over all the generated datasets the best of our abilities. To ensure full coverage we performed human evaluation over 20% of all synthetic datasets. The evaluation was con- ducted equitably by teams evaluators each. These teams comprised the authors, and thus required no external labour. The generated in-
Oman
Australia
Synthetic BOOLQ
Country
United States
GPE
Canada
Oman
stances were graded on scale of A (excellent) to D (poor). The evaluators graded the gener- ated instances within the context of the origi- nal label constraint, as well as within the con- text of the updated labels after self-correction. We notice that as the instances go through self- correction, the number of samples rated "excel- lent" increases and the number of "unaccept- able" samples reduces. To gauge the reliability of our human evaluation approach, we measure
Australla
France
Figure 4: Comparison of dataset bias for the BoolQ dataset and the synthetically generated BoolQ dataset. This represents the distribution for GPE (Geo Political En- titiy) named entity tag. The distribution for other entities can be found in SC
0.79) and Spearman's 's correlation coefficient (obtaining 0.85, 0.83, 0.91). Details can be found in SI. We also found that the synthetic data contained no instances from the original SuperGLUE dataset, preventing any data leakage.
inter-evaluator agreement using the Cohen's 's Kappa (resulting in K= 0.71, 0.67,00777
Comparison with Self-Instruct: To compare the synthetic SuperGLUE dataset with other synthetic instruction following benchmarks, we choose Self-Instruct (Wang et al., 2023) popular synthetic dataset generation framework. We choose Llama2 (7B) and pre-finetune using synthetic SuperGLUE. We refer to this tuned model L2SSG. We refer the Llama2 model trained on the Self-Instruce dataset as L2SI. Both models are evaluated on OpenLLM benchmark. Table shows that L22SG scores 2.62% higher than L2ST, underscoring the quality of targeted dataset generation by the TarGEN framework over Self-Instruct.
Lexical diversity: We analyze the dataset diversity, along the lines presented in (Yu et al., 2023). We initiate our exploration with straightforward vocabulary-based examination to assess lexical diversity, as summarized in Table 7. Notably, our TarGEN synthetic data exhibits an average lexical diversity that is 25% higher across various dataset tasks.
Semantic diversity: To analyze the semantic diversity, we examine the cosine similarity distribution of SentenceBERT embeddings of within- sample pairs as presented in Fig 3. Across most SuperGL.UE tasks, the TarGEN datasets consistently display lower cosine similarity than the original dataset, indicating reduced semantic similarity of within-dataset samples and, consequently, higher semantic diversity. This observation underscores our approach's 's intrinsic capability generate diverse samples. Moreover, our findings of higher cosine similarity of the original datasets align with those of (Parmar et al., 2023), where the authors highlight the propensity for crowdsourced datasets exhibit high bias and low diversity. This phenomenon arises as crowdsourced workers often adhere to patterns provided by dataset creators. The dataset generated using our approach can generate high-diversity samples.
Dataset difficulty: To estimate the dataset difficulty, we use V-usable information (Etha- yarajh et al., 2022), which estimates the information input X holds for predicting the target across a family of models V. Lower V-usable information indicates higher dataset difficulty for V. Fig. 5 offers a comparative view of dataset difficulty between the original and synthetically generated datasets by TarGEN. Notably, the synthetic datasets exhibit diverse range of samples with varying pointwise V-usable information, showcasing their

---

Published as a conference paper at COLM 2024
diversity in terms of difficulty. Furthermore, the absence of mislabeled samples, indicated by positive V-usable information in the synthetically generated datasets, underscores the effectiveness of self-correction prompts.
To estiamte the dataset bias, visualize the distribution of to- kens related to named entities, belong- ing to Geopolitical Entities (GPE), Prod- ucts, and Nationalities or Religious or Political groups (NORP). The distribu- tion of input tokens for the original and synthetically generated BoolQ dataset is presented in Fig. 4. Our results re- veal that the distribution of GPE, Prod- uct, and NORP entities in the TarGEN dataset closely resembles that of the original dataset. 3, Additional experi- ments and extended analysis (includ- ing dataset comparison plots) of our approach can be found in SB and $C of the Appendix.
Dataset bias:
1.0
s42,000
ang sonth
0.5
Proportion of Examples
tooky.yythh
0.5
0.0
-22
0.4
1.0
wi.org
0.0
6 Conclusion
-35
-14
61
0.r
PVI (higher <-> easler)
In this work, we introduced TarGEN, a multi-step prompting strategy for gen- erating high-quality and diverse syn- thetic datasets utilizing LLMs without any human supervision.
Figure 5: Comparison of PVI (V-usable infor- mation) for AXG, BoolQ, and WiC original and the synthetically generated dataset. Synthetic data seems to have better quality as the original datasets' PVI is concentrated around -0.1 to 0.1 whereas the synthetic data generated has diverse mix of difficulty level among the samples. The dataset difficulty comparison for all datasets can be found in $6
We described a step-by-step methodol- ogy for TarGEN to synthesize dataset from instructions without any task ex- emplars. To evaluate our proposed framework, we emulated eight tasks from the SuperGLUE benchmark and compared with the original Super- GLUE by training different families of models. Experimental results reveal that models finetuned on our synthetic SuperGLUE outperform models finetuned on the original SuperGLUE. A comprehensive analysis of synthetic benchmark with respect to the original benchmark resulted in several interesting findings such the fact that data instances in our synthesized benchmark are more difficult and diverse compared to the original benchmark, and also exhibit similar dataset bias.
Limitations
Comparison with Self-Instruct and AttrPrompt revealed that synthetic SuperGLUE served as better pre-finetuning corpora when evaluated on the OpenLLM benchmark resulting in an impressive performance when using T5-3B and Llama2-7B. Though TarGEN facilitates high- quality data generation, we believe that it is important to assess our proposed frameworks within a multi-lingual context context and also on additional benchmarks, including BigBench, LILA, and HELM. Furthermore, TarGEN currently relies on the ChatGPT model for synthesizing the benchmark, but our plans involve exploring the impact of other LLMs such as GPT-4, Llama2, and Falcon when employed with TarGEN. We believe that TarGEN can serve valuable tool for enhancing the quality of data generation, thus reducing human effort.
TTo ensure a broad-scale examination across all samples, we utilized the Spacy library and its en_core_weeh smm model to extract named-entity tags.
10

---

Published as a conference paper at COLM 2024
References
The claude 3 model family: Opus, sonnet, haiku. htttpsss/ppp..smmmt.ccchhh.... org/CorpusID: 268232499.
Ujjwala Anantheswaran, Himanshu Gupta, Kevin Scaria, Shreyas Verma, Chitta Baral, and Swaroop Mishra. Investigating the robustness of Ilms on math word problems. arXiv preprint ar Xix:22466.15444 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Derek Chen, Celine Lee, Yunan Lu, Domenic Rosati, and Zhou Yu. Mixture of soft prompts for controllable data generation. CoRR, /bbb//330001880 2023. doi: 10.48550, 10.48550/arXiv.2000 arvvv.23XX 01580. URL https: / //w.i. org/10. 48550 00550//axxi... www.v. 2303. 01580.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Vosss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. ArXiv, abs/ /2td//0333374, 2021. URL https: /tppaap..../// semant ieschola org/Corpus ID: 35775572.
Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. RelationPromptt Leveraging prompts to generate synthetic data for zero-shot relation triplet extraction. Findings of the Association for Computational Linguistics: ACL 2022, pp. 45-57, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v/ 1tt//2222ffnnnnn.. URL https::/accaatthaaat
wwtp:oy...ggg22000 findingsfiiiinnnnnin.an acl.5.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Xiv preprint arXio:2210.11416 2022.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/nu noo questions. ArXiv, abs/ 1905.10044, 2019, URL peatps://apt: semanticscholar.ore Corpus TD: 651 63607.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
11

---

Published as a conference paper at COLM 2024
Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gpt: Open compute-optimal language models trained the cerebras wafer-scale cluster. Xrv preprint ar xio:23304..03208 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint or Xio: 217883 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset diffi- culty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Ma- chine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, Pp. 5988 6008. PMLR, 2022. URL https: //proceedings mlr setess/vv62//e ht
Chandra Kiran Reddy Evuru, Sreyan Ghosh, Sonal Kumar, S. Ramaneswaran, Utkarsh Tyagi, and Dinesh Manocha. Coda: Constrained generation based data augmentation for low- resource nlp. ArXiv, abs//404.a... 2404400415, 2024. URL https: /ts:://aaga /api. semant inmaaniicsenn icscholar. org/Corpus ID: 67888869995
Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-guided noise-free data genera- tion for efficient zero-shot learning. In The Eleventh International Conference on Learning Representations.
Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong, Self-guided noise-free data genera- tion for efficient zero-shot learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL net ddddpidd-dssppeddd
https:://ppnrzet
Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, Mutsumi Nakamura, Arindam Mitra, Santosh Mashetty, and Chitta Baral. Instruction tuned models are quick learners. arviv preprint ar orXio:2306.05539 2023.
Qi He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Lida Chen, Xintao Wang, Yuncheng Huang, Haoning Ye, Zihan Li, Shisong Chen, Yikai Zhang, Zhouhong Gu, Jiaqing Liang, and Yanghua Xiao. Can large language models understand real-world complex instructions? ArXiv, abs/2309.090 2309909155 2023. URL https: /appi. wwwpollar..rgoooouuuooeeeoo Cotg://oooss::::200 :220043772227
semant www.anntssccoaa....ommmmo cet.aattccccolaa..o///oooorpuuop
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arv preprint ar Xrie:2310.06825, 2023.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large language models for text classification: Potential and limitations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10443-10461, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.18653/vv /v15 2003..emllp-mine //aclanthology org/ 2023. emnlp main. 647.
URL https:
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large language models for text classification: Potential and limitations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10443-10461, 2023b.
12

---

Published as a conference paper at COLM 2024
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLE Worker and Al collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6826-6847, Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational Linguistics. doi: 2222..
/ www.fiinnnnnnnnnl...... sandinngseemnlp...
URL htttps://cclanthho....
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zet,lemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. aiv preprint ar arXie:1907.11692 2019.
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir R. Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evalua- tion. ArXiv, abs /2t.//222..9,, 2022b. URL https: www...ssssasccccc.......o CorpusID:2546835
Lin Long, Rui Wang, Rui Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: sur- vey. ArXiv, abs/2406.15126, 2024. URL https://apl semantccscholar. CorpusID:27068833:
Ying-Cheng Lu, Huazheng Wang, and Wenqi Wei. Machine learning for synthetic data gener- ation: review. ArXiv, abs/ / 23022.4402 2023. URL httpps:/app. semanticscholar.
org/Corpus 256662279.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. In NeurlPS, 2022a. URL http://papers.
httpppperrrllepppppeeeepeeeeeeee html..
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. Advances Neural Information Processing Systems, 35:462 35:462-477, 2022b.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https: //api. semanticschonn
pog//orrpsII:::0 23742137
OpenAL. Gpt-4 technical report, 2023.
Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. Don't blame the annotator: Bias already starts in the annotation instructions. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1779-1789, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/200 130l-main.130. URL https: anlloooooooo org/ /rgg .eacl maacllmmaii ma iaaimmii.... 130.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1267-1273, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653 (0665/vv//1// v1/N19-1128. URL https: http///alaaaatthhaae
org/ /NN//1155 9-28.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-535 2020.
13

---

Published as a conference paper at COLM 2024
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume 2 (Short Papers), Pp. 8 8-14, New Orleans, Louisiana, June 2018. Association for Computational Lin- guistics. doi: 10. (0.18653vv1/N16-1 2002 URL / http:///aaaaaaaoooooooooooo org/N1 80202..
Kevin Scaria, Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, and Chitta Baral. Instructabsa: Instruction learning for aspect based sentiment analysis. ArXiv, 0bbs/330.08824 2023. URL https:/ /api. semant www.nnicccchlar. wwwaaarrrrg/ooooo ooggCCrrpusDD 257020097.
Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6943-6951 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: (0.18653/v1/2000 http://11121111llp111pm m51pm.55...55 URL https:
//aclanthology wow..ooooooooooooooooooooooo wrg//0200 emnlp pamlllppm-in.. 555.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-aliigmentt of language models from scratch with minimal human supervision, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Huai hsin Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether Maain-of-though can solve them. In Annual Meeting of the Association for Computaaional Linguistics, 2022. URL https: //api.
semanti cscho 1 ar orgg Co Corpus ID: 215917664
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiz, abs/ (DS/222...1971 2023a. https: //api. www.hnnttttccoaarrr....cca
25:7119940
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint (kki::2307.09288, 2023b.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned mooels
chat models. ArXiv, abs/ / /t////00000022202 009288, 2023c. URL https::// appi. semant ics aescholr.
org/Corpus ID: 25 9950998.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). ArXio, hbbs/2206.1042 2022. URL https:/ ///pp/ap.a /ppi. semanticschoiad org/
CorpusTD:249889477
14

---

Published as a conference paper at COLM 2024
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Ku- mar Pal, M. Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Ro- hitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sam- pat, Savan Doshi, Siddharth Deepak Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hanna Hajishirzi, and Daniel Khashabi. Super-naturallnsttrctooor Generalization via declarative instructions on 1600+ nip tasks. In Conference on Empirical Methods in Natural Language Processing, 2022. URL https:/ / //ppi//pppppp wwwentiisccoooo..ooo... ar hwtpsschhhaar.roooeeeeeeo rpus ID: 2530 98274.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 13484-13508. Associa- tion for Computational Linguistics, 2023. doi: 10. 10.18653/vl 754.
URL
attpps://oii
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. ar arxiv preprint ar xrX:::2109.01633 2021.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. Reframing human-Al collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties:
Human Language Technologies, pp. 632-658, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1 naw.2...aaclmmaaaa
URL https:
//aclanthology.o. org/ 2022.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Zerogen: Efficient zero-shot learning via dataset generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 11653-11669. Association for Computational Linguistics, 2022a. doi: 110866///112222emmlpppp v emn w-w..mmmmmmmmm000
URL https: / doi. org /wtgg/..... /ttg///.........
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Z Zhiyong Wu, Tao Yu, and Ling- peng Kong. ZeroGen: Efficient zero-shot learning via dataset generation. In Yoav Gold- berg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pp. 11653-11669, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10. 10.18653/vil 2022.emnlp-main.90 URL httpt: /aacl:://lllaaaaa
ac woccannthooogy org/ 2222 w- memaain.c. 801.
Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Progen: Progressive zero-shot dataset generation via in-context feedback. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 3671-3683. Association for Computational Linguistics, 2022c. doi: 21116653/112222.... fing:ig--nnnnngeemmllp. 269. URL htwph:ttp///ett //ts::////o..eee org 10. 11w1311111/// http////22//2 findingssemm..... 269.
Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. ProGen: Progressive zero-shot dataset generation via in-context feedback. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Compulational Linguistics: EMNLP 2022, pp. 6671-3683, Abu Dhabi, United Arab Emirates, December 2022d. Association for Computational Linguistics. doi: 10.18653/v1 /22//2222..2mm h0022..indings-emm/. 269. URL https:t///// htt:/aaaaaaa httaaaaaaanttooooooooooo org/ 2022. f findings madings--mmlp.. .269.
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378-30399 2022.
15

---

Published as a conference paper at COLM 2024
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. CoRR, abs/ /tss//330.18955 2023, doi: URL httpt::t////// org/ 1 0 48550/arXiii 2306. 15895.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. Xrv preprint arXic:1810.12885, 2018.
16

---

Published as a conference paper at COLM 2024
Appendix
A Comparison with other data generation frameworks
A.1 Qualitative comparison
Training required as part of pipeline
Ability to generate samples for a complex task
Noise
Focused diversity
Method
Seedless
mitigation
strategy
Self Instruct Wang et al. (2023)
No
No
No
No
Yes
CODA Evuru et al. (2024)
No
No
No
No
No
Synthetic Data
Generation Li al. (2023b)
Yes
No
No
No
No
ZeroGEN Ye et al. (2022b)
Yes
Yes
No
No
No
SuperGEN Meng et al. (2022b)
Yes
Yes
No
No
No
SunGEN Gao et al.
Yes
Yes
No
No
Yes
ProGEN et (2022d)
Yes
No
Yes
Ours (TarGEN)
Yes
No
Yes
Yes
Table Qualitative comparison with previous works along different aspects of methodology.
We compare framework with other popular data generation approaches in the table 8.
Axes of comparison:
Seedless: Whether the approach requires labeled task samples.
Focused diversity: Whether the approach actively injects diversity, incentivizing samples generated across various contexts.
Noise mitigation strategy: Whether the approaches actively mitigate noise, either algorithmically or through the inclusion of module like our Self-Correction part of their pipeline.
A.2 Empirical comparison
To show a quantitative comparison with other data generation methods, we generate syn- thetic versions of specific tasks from the SuperGLUE dataset using recent data generation frameworks: The tasks we generate are RTE (textual entailment) and AXg (gender disam- biguation). We then train Llama-2 model on these synthetic task variants and evaluate them on original test sets of the datasets. The results are present in the Table
Method
RTE
AXg
CODA
64.56
23.46
ZeroGEN
58,54
19.01
SuperGEN
77.94
44.67
SunGEN
75.11
38.16
ProGEN
78.43
37.94
Ours
92.89
51.32
Table 9: Results on RTE, Axg for Llama-2 models trained on synthetic data generated by existing data generation frameworks.
B Additional Experiments
B.1 Prompt variations on different downstream tasks
To understand the effect prompt variations on different downstream tasks, we generated
2 additional variations of the Step prompt for CommitmentBank dataset. I.ama-2 (7B)
17

---

Published as a conference paper at COLM 2024
was fine-tuned on these alternative synthetic datasets alongside our synthetic dataset and evaluated on the original test set. We use the following prompts for this experiment:
Variation
The imput contains a premise and hypothesis.
If the hypothesis logically follows the premise and the kypothesis can be derived from the information in the premise, Output "entailment"
If the hypothesis directly contradicts information in the premise, Output "contradiction" If the hypothesis is unrelated to the premise, cannot be sufficiently proven from the information the premise, Output "neutral"
Variation
The imput consists of premise and a hypothesis.
Output "neutral" if the hypothesis is either unrelated to the premise or camnot be conclu- sively derived from it.
Output "contradiction" if the hypothesis directly opposes the information in the premise.
Output "entailment" if the hypothesis logically follows from and can be deduced from the premise.
Variation
The imput consists of premise and a hypothesis.
If the hypothesis directly opposed the information in the premise, output "contradic- tion."
If the hypothesis logically follows from the premise and can be inferred from its information, output "entailment."
If the hypothesis unrelated to the premise lacks sufficient evidence from the premise, output "neutral."
Mean Cosine Similarity
Accuracy Llama-2 (7B)
Dataset
Variation 1
0.56
94.85
Variation 2
0.42
93.27
Variation 3
0.47
94.14
Table 10: Accuracy of inference results for the original Step 3 prompt and variations. Mean cosine similarity: Average of cosine similarity between given prompt and the other variations.
We see from the results in Table 10 that there slight variation in the final results and similar mean cosine similarity. We would also like to highlight that the motivation behind proposing a clearly-defined, task-agnostic guiding framework such as TarGEN is constrain the prompt generation to series of simpler, clearly defined subtasks. The cascaded stepwise approach has the following advantage: At each step, the subtask for which we prompt the model is straightforward with very little room for misinterpretation and errors. This reduces the prompt engineering effort significantly, especially compared to the human cost of engineering single prompt that completely and accurately models complex task. Any prompt that adheres to the instructions for that particular step in the framework is likely to generate similar outcomes, making this framework task- and prompt- agnostic.
Additionally the constrained, straightforward nature of the task prompt at each step makes it less likely that any variance in prompt would cause significant change in outcome,
18

---

Published as a conference paper at COLM 2024
given the narrower scope of the objective at each step and the fact that the outcomes grounded in the response to the previous step. Furthermore, any slight deviances could be attributable to the randomness inherent in inferring with LLMs, but would not expect to see significant difference in quality or diversity of the output samples.
B.2 Effect of seed instances
For the seed instances, we experimented to see the impact of reducing the number of semantic contexts. Table 11 shows the effect of the generating samples over proportion of the original semantic contexts:
Scores with
BoolQ
WiC
CB
AX-g
ReCoRD
RTE
WSC
COPA
100%
90.12
73.21
94.87
51.32
78.81
92.89
77.32
95.72
50%
85.43
72.65
90.33
49.86
75.42
89.75
76.21
92.11
25%
83.65
71.78
88.47
48.32
73.78
87.81
76.02
91.21
Table 11: Accuracy of models trained over samples generated over the full (100%), half, and a quarter of the original semantic contexts
B.3 Comparison with other generative models
We run the data generation pipeline with Claude Sonnet and Llama-3 (70B) create more variants of synthetic SuperGLUE with each model keeping the task set, prompts, and pipeline steps constant. Llama-2 (7B) was fine-tuned on the above-mentioned datasets and evaluated on the original test set. The results of the experiments are attached in Table 12. We see nearly the same or slightly improved scores when using Llama-3 (70B) (Dubey et al., 2024) and Claude Sonnet (The).
Train data Models
BoolQ
WiC
CB
AX-g
ReCoRD
RTE
WSC
COРA
ChatGPT (current)
90.12
73.21
94.87
51.32
78.81
92.89
77.32
95.72
Claude Sonnet
90.65
73.88
95.23
51.88
79.22
93.01
77.89
95.72
Llama-3 (70B)
89.56
73.33
93.65
51.88
79.01
92.23
77.65
95.23
Table 12: We observe that the data generation pipeline adaptable to different generative models with negligible performance degradation.
C Extended Analysis
C.1 Dataset difficulty comparison
The section contains plots pertaining dataset difficulty and bias for all datasets.
C.2 Dataset bias comparison
C.3 Generalizability of the framework
The basic framework proposed in this work is task and model-agnostic. While RTE relatively straightforward task, AXg is complex NLU task. Despite the differences in the prompts of these tasks, the underlying TarGEN framework delineating the steps remains common. This proposed framework can be adapted to any task, based on the problem being evaluated. Given the parameters of a task the label schema, and the problem statement, this framework is easy to adapt for multiple tasks of varying complexities. We would like to highlight that the simple nature of our framework allows for greater flexibility and control, making it well-suited to domain-specific happy to generate any other tasks that the reviewer deems necessary for the verification of this approach.
particularly complex tasks. We are
19

---

Published as a conference paper at COLM 2024
1
exg_orig
exg_synth
-0.5
0.0
0.5
-0.5
0.0
0.5
1
_oig
boold_synie synth
Proportion of Examples
1
cb_ ab_orig
cb _ysythhhhh
0
-0.5
1
copa_orig
copa copa_synth
0
-0.5
0.0
0.5
-0.5
0.0
0.5
rte_orig
rte synth
-0.5
0.0
0.5
-0,5
0.0
0.5
1
wic_orig
wic sic_syythh
-0.5
1
wsc_oriq
wsc_synth
0
-0.5
0.0
0.5
0.5
0.0
0.3
PVI (higher < easier)
Figure 6: Comparison of PVI (V-usable information) across datasets for the original dataset and the synthetically generated dataset. Synthetic data seems to have better quality as Original datasets PVI is concentrated around -0.1 to 0.1 whereas the synthetic data generated has a diverse mix of difficulty level among the samples.
C.4 Regarding V-Usable Information
HY( - (Y|)))), where H,(Y), is the predictive entropy from a null model LM (trained empty inputs), and (Y| is the conditional entropy from model LMx (trained on input samples and labels). Entropies are measured in bits using log, (P), where P is the probability distribution of the labels assigned by the specific model. Higher V-usable information indicates harder samples, while lower values indicate easier samples for model family V. Original datasets have V-usable information concentrated in narrow range of easier samples. Our strategy generates a broader range of V-usable information, from easy to hard, leading to better finetuning results.
D Hyperparameters
We use 6xNvidia Tesla P40 GPUs. Batch Size: 16 for STL and 1 MTL setting. Gradient Accumulation Steps: 1, Learning rate: 5e-5, Num of Epochs: 5 for STL and 1 MTL setting.
D.1 Additional Results
Task-specific instance generation pipelines
This section details the task-specific TarGEN strategy used to generate instances.
20

---

Published as a conference paper at COLM 2024
Original BOOLQ
Synthetic BOOLQ
Country
Country
GPE
united kingdom
canada
canada
australa
franze
Product
Product
PERSON
one
xbox 360
playstation
playstation
crown
playstation
playstation
Norp
Norp
american
american
NORP
britsh
british
canadian
canadi
german
japanese
Figure 7: Comparison of dataset bias for the BoolQ dataset and the synthetically generated BoolQ dataset.
Original CB
Synthetic CB
Country
Country
france
GPE
france
italy
spain poland
germenyy
china
Product
PERSON
Product
frame
mais
dlever
italo-norman
cruseders
Norp
Norp
american
franch
NORP
southh korean
byzantinee
europeans
Figure 8: Comparison of dataset bias for the CB dataset and the synthetically generated CB dataset.
E.1 COPA
Step Generate list domains settings in which events can take place.
21

---

Published as a conference paper at COLM 2024
Original Record
Synthetic Record
Country
Country
GPE
united states
australia
china
brazil
india
china
Product
Product
PERSON
twitter
twitter
m1
cole
bronces
gepra
napoli
Norp
Norp
NORP
amreicaen
brijish
british
american
chiness
spaniish
Figure 9; Comparison of dataset bias for the Record dataset and the synthetically generated Record dataset.
Original RTE
Synthetic RTE
Country
Country
ukraine
GPE
oman
goorgia
germany
Product
Product
PERSON
ataants
taitter
saturn
my car win top prizes
a380
discovery
magic mike
Norp
Norp
NORP
brilish
russiem
ukrainian
tailen
regublicass
democrats
rrai
ame ican
Figure 10: Comparison of dataset bias for the RTE dataset and the synthetically generated RTE dataset.
Step Generate sentences describing events that could take place in the domain (DOMAIN).
Step 3: query CAUSE. Add "What was the CAUSE of this?" to the premise during post- processing For the given sentence, generate 2 hypotheses (Hypothesis 1, Hypothesis 2) such that Hypothesis is probable cause of the sentence. Hypothesis is very unlikely to be the cause of the
22

---

Published as a conference paper at COLM 2024
Original WIC
Synthetic WIC
Country
Country
GPE
united staces
australle
china
Ireland
cubta
Product
PERSON
becker
centimetre
Product
frisbee
Norp
Norp
NORP
etnerican
polish
french
republican
geeze
sparrish
latin
Figure 11: Comparison of dataset bias for the WIC dataset and the synthetically generated
WIC dataset.
Dataset
Boolq
WIC
CB
AXg
Record
RTE
WSC
Copa
Original
0.026
0.011
0.18
0.21
0.34
0.04
0.13
0.01
Synthetic
0.025
0.012
0.05
0.58
0.70
0.11
0.37
0.06
Table 13: Lexical diversity the dataset. Figures in red correspond where diversity of synthetic dataset is limited. Blue correspond where the synthetic dataset has more diversity than original.
sentence.
Example:
Sentence: cast a long shadow.
Hypothesis 1: The sun was low in the sky.
Hypothesis 2: The grass was tall.
Explanation: Hypothesis 1, the low position of the sun is more likely to cause long shadow. The height of the grass has nothing to do with the long shadow, and thus is unlikely to be a cause. Sentence: [SENTENCE]
query RESULT. Add "What was the RESULT of this?" to the premise during post-processing For the given sentence, generate hypotheses (Hypothesis 1, Hypothesis 2), such that Hypothesis is probable result of the sentence. Hypothesis 2 is very unlikely to be the result of the sentence.
Example:
Sentence: fell down the stairs.
Hypothesis 1: injured myself.
Hypothesis 2: My mother bought a new car.
Explanation: Hypothesis 1, the injury is more likely to be result of the fall. The buying of a car is not implied by the sentence which talks about falling down stairs hence less likely to be the result of the sentence.
23

---

Published as a conference paper at COLM 2024
Sentence: [SENTENCE]
Mathematical formulation of Step 3:
G result) (P,
P= C1, -1
C2, C2, C2 -
(P,CI,C2)
==
c2 P,C P =
The relationship result) is affixed the premise P query. The generated triplet (P, C1, C2) and the constrained label I E {C1, C2} that answers the query, form an instance of this dataset.
Self-correction
Instructions:
You are given a premise and 2 possible hypotheses (Choice and Choice as input. Select the hypothesis which is more likely to have causal link to the sentence.
the premise asks for CAUSE: If the premise more likely to the result of Choice 1, output Choice
1. Otherwise, output Choice 2.
If the premise asks for RESULT: If Choice is more likely to be the result of the premise, output Choice 1. Otherwise, output Choice 2.
E.2
CommitmentBank
Step 1: Generate a list of domains or settings in which events can take place.
Step N/A
Step 3: label entailment
For the given domain (DOMAINI, generate [N] pairs of sentences (Sentence 1, Sentence 2) such that Sentence logically follows, or is implied by, Sentence 1.
Example:
Sentence 1: The singer was very nervous.
Sentence 2: The singer saw critics in the front row.
Now generate such sentence pairs. Generated sentencesllabel neutral For the given domain (DOMAINI, generate [N] pairs of sentences (Sentence 1, Sentence 2) such that Sentence 2 has no relation with Sentence 1, and cannol be derived from it.
Example:
Sentence 1: made coffee.
Sentence 2: My assignment is due tomorrow.
Now generate N such sentence pairs.
Generated sentences: label contradiction
For the given domain [DOMAIN), generate [N] pairs of sentences (Sentence 1, Sentence 2) such that Sentence is explicitly contradicted by Sentence 1.
Example:
Sentence 1: The musician was not on time for the gig.
Sentence 2: The promoter praised the musician for being on time.
Now generate N such sentence pairs.
Generated sentences:
24

---

Published as a conference paper at COLM 2024
Mathematical formulation of Step 3:
P H
entailment
-1)(1)== =
P p H = neutral
P HH contradiction
Each generated pair HP of premise and hypothesis, along with its constrained label E fentailment, cecutral,contaaiitiinn constitutes synthetic instance for this dataset.
Self-correction
Instructions:
The input contains a premise and a hypothesis. Assume the premise always true.
1. the hypothesis logically follows the premise and the hypothesis can be derived from the information in the premise, Output "entailment"
2. If the hypothesis directly contradicts information in the premise, Output "contradiction"
If the hypotheesi unrelated the premise, or camnot be sufficiently proven from the information in the premise, Output "neutral"
E.3 MultiRC
Step 1: List of categories: News* Wikipedia"; History and anthropology" Society, law, and justice Elementary school science textbooks" 9/11 reports" Fiction literature movie plots" art computer science; biological processes physics chemistry; linguistics psychiatry and psychology supernatural phenomena prepetttnt in original dataset)
Step 2: Generate 50 unique topics or titles in the category [CATEGORY] Generate short paragraphs on the following topic: [TOPIC]
Step 3: Given a paragraph, frame a question that requires information from multiple sentences of the paragraph to be answered correctly. Then, generale set of options. The correct answer may be combination of one, some, or all options. Also include options that do not answer the above question. Finally, output the combination of options that form the correct answer. Paragraph: [PARAGRAPH]
Mathematical formulation of Step 3: Not applicable in this case
Self-correction
Instructions:
You are given a passage followed by question and list of options. Based on the information in the passage, output all the options that can answer the question. The output must not include options that do not answer the question. The output must not lack any options that answer the question.
E.4 RTE
Step Generate a list of topics or domains to talk about.
Step 2: For the given domain [DOMAIN), generate N complex sentences containing information relevant to this domain.
Step 3: label = entailment
Given a sentence as premise, add a logically sound hypothessi concerning the information in the premise.
Premise: A place of sorrow after Pope John Paul died became a place of celebration as Roman Catholic faithful gathered in downtown Chicago to mark the installation of new Pope Benedict XVL. Hypothesis: Benedict XVI is the new Catholic Pope. Premise: [SENTENCE]
25

---

Published as a conference paper at COLM 2024
label = non-entailment
Given a sentence as premise, add a logically unsound hypothesis concerning the information in the premise.
Premise: A place of sorrow after Pope John Paul died became a place of celebration as Roman Catholic faithful gathered in downtown Chicago to mark the installation of new Pope Benedict XVI.
Hypothesis: Benedict XVI died recently. Premise: [SENTENCE]
Mathematical formulation of Step
{P H == True
Gisp(hip) = (P HP,
P H = Falee
The generated premise and hypothesis pairs (P, as well as their associated label I {True, False} form the binary labeled instances for this task.
Self-correction Instructions: You are given a premise and hypothesis. If the hypothesis logically follows from the premise, output entailment. If the hypothesis cannot be logically derived from the information the premise, output not entailment.
E.5 WSC
Step 1: Generate a list of domains or settings in which events can take place and where people can Interact.Examplee 'A reslaurant, 'a museum', 'a lively rock concert', 'An opera house'
Step 2: Identify a pair of subjects (Subject 1, Subject 2) in the conlext of [DOMAIN]. For example,
in the context of "a classroom", pair of subjects could be (Subject1: teacher, Subject 2: student), or (Subject1: students, Subject2: rivals). For the given domain, generate [10] such subject pairs. For pairs, Both subjects must singular For 5 pairs, both subjects must be plural. Both subjects must be humans or groups of humans. Output all 10 pairs in a numbered list.
Step For given subject pair (Subject Subject 2), generate 2 passages, S1 and S2. Each passage must be 2 sentences fewer. You are also given the gender of both subjects. For every pronoun in the sentences, identify which subject is being referred to. Example:
Imput:
Subject 1: Teacher, Subject 2: Student
Pronouns: He/him
Output:
S1: The teacher was disappointed in the student because [he=teacher] had high hopes for [him=student].
S2: The teacher and the student are not on good terms. [He=student]] is very rebellious, and does not show up classes.
Explanation: In S1: It is clear that the student has disappointed the teacher, who had high hopes for the student. Hence, "he had high hopes" "he" refers the teacher, "for him" "him" refers the student. In S2: The student is rebellious and does not show up to classes hence the "he" in "he is very rebellious" refers to the student Imput:
Subject 1: |SUBJECT 1] Subject 2: [SUBJECT 2] (randomly chosen) Output:
Pronouns:
26

---

Published as a conference paper at COLM 2024
Mathematical formulation of Step 3:
(s) N1 + P | = True
(s, :): N2 P 1 True
G
Ps) N2 P == False
Ps) N2 + 1 False
where contains N1, N2, and We denote coreference with the ++ operator.
Self-correction Instructions: The given input talks about noun phrases (Subject and Subject
2). You are given Subject and Pronoun that occurs in the input.
Rules:
1. If the pronoum refers to the noun phrase Subject 1, the task output is TRUE.
If the pronoun refers to the noun phrase Subject 2, the task output is FALSE.
E.6 BoolQ
Step 1: Generate list of domains to write article in.
Step 2: Generate 50 unique topics or titles the category [DOMAIN] Generate 7 short paragraphs on the following lopic: [TOPIC]
Step 3: label TRUE
You are given passage. Generate a boolean query. The answer to this query, based on the passage, must be YES. Example:
Passage: The Millennium Falcon, a legendary starship piloted by Han Solo and Chewbacca, has become an iconic symbol of rebellion and hope in the struggle against the oppressive Galactic Empire. May the Force be with you, as the epic adventures of Luke Skywalker, Princess Leia, and Darth Vader remind us that even in the darkest times, there always glimmer of light and chance for redemption in the Slar Wars universe.
Query: Does Han Solo work with Chewbacca? Answer: YES, Han Solo and CHewbacca pilot the Falcon together.
Similarly, generate a query for the following passage.
Passage: [PASSAGE]
Query:
label FALSE
You are given passage. Generate a boolean query. The answer to this query, based on the passage,
must be NO.
Example:
Passage: The Millennium Falcon, a legendary starship piloted by Han Solo and Chewbacca, has become an iconic symboo of rebellion and hope in the struggle against the oppressive Galactic Empire. May the Force be with you, as the epic adventures of Luke Skywalker, Princess Leia, and Darth Vader remind that even in the darkest times, there is always glimmer of light and chance for redemption in the Star Wars universe.
Query: Does Han Solo work alone? Answer: NO, Han Solo works with Chewbacca Similarly, generate a query for the following passage. Passage: [PASSAGE]
Query:
Mathematical formulation of Step 3:
[P= Q
I I Yes
Gr,p(,p) (P :.Q):
{ > QQ
1 1 No
Self-correction
27

---

Published as a conference paper at COLM 2024
Instructions: You are given passage followed by question. The questions ask for confirmation of fuct that may or not be present in the passage. If the passage explicitly confirms the fact being asked in the question, output TRUE as the answer to the question. If the passage offers no information that explicitly conftrms the fact, and the fact has no logical basis or strong evidence in the passage, oulput FALSE as the answer to the question.
E.7 ReCoRD
Step 1: Generate list of domains to write an article in.
Step 2: Generate 50 unique topics or titles in the category [DOMAIN] Generate 7 short paragraphs on the following topic: [TOPIC]
Step 3: Example:
As a spring breeze wafted into his trench French commander Georges Lamour saw something surreal drift his way- a yellow-green cloud. 'All my trenches are choked,' he cried into the field telephone to headquarters. 'I am falling myself!' Chlorine gas carried by favourable winds over Flanders Fields from German positions- - had been used for the first time. It was April 22, 1915. Scroll down for video Chlorine gas carried by favourable winds over Flanders Fields from German positions sowed terror and agony for the first time on April 22, 1915. Above, German Red Cross workers carry bottles of water to help revive troops. German forces launched first attack using gas on April 22, 1915. 150,000 tons of gas were used by German and Allied forces in WW1.
Query: Had they been able to peer a bit further across no-man's land they would have seen how [X] troops had dug in, under cover of night, more than 5,000 gas cylinders with tubes pointing their way. Answer: German
Explanation: The query fits the context of WW1 and talks about both entities involved in the war. The paragraph mentions German forces using gas. German is the entity replaced by [X] in the query. Instruction: Generate a complex sentence that fits the context of the given paragraph. The generated query must be a statement about the events in the paragraph. Put [X] in place of any one entity mention. The query must not contain any events mentioned in the paragraph. The answer must contain the entily mention that can replace [X].
Query:
Mathematical formulation of Step 3: (a) (A= -,s.,,
where is the article, and s,e are the paraphrased sentence (with one entity e obscured) as well as the entity e obscured acting as the label.
Self-correction Instructions: You are given bassage,foloovee by query. The query contains [X] in place of any one entity mentioned. Output the entity that could logically replace [X] in the query.
E.8
AXg
Step 1: Generate a list of domains or settings in which events can take place and where people can interact. Examples: 'A restaurant, 'a museum', 'a lively rock concert', 'An opera house'
Step 2: Identify a pair of subjects (Subject 1, Subject 2) in the context of [DOMAIN]. For example, in the context of "a classroom", pair of subjects could be (Subject1: teacher, Subject 2: student), or (Subject1: students, Subject2: rivals). For the given domain, generate [10] such subject pairs.
For a given pair subjects, Generate 4 sentences with the following specifications: Initial clause: A clause containing Subject 1 and Subject 2 Sentence 1: Initial clause with dependent clause containing a gendered pronoun. Dependent clause should refer to Subject 1, not Subject 2. Sentence 2: Completely identical to sentence 1 but with different gendered pronoun Sentence 3: Initial clause with a dependent clause containing a gendered pronoun. Dependent clause should refer to Subject not Subject 1.
28

---

Published as a conference paper at COLM 2024
Sentence 4: Identical to sentence 3 but with different gendered pronoun Ensure the dependent clauses in Sentence and Sentence refer to Subject and Subject 2 respectively. Sentences 1 and 2 must be identical in terms of dependent, with nothing but the pronoun changed. Similarly, Sentences 3 and 4 must be identical, with only the pronoun changed. Sentences 1 to 4 must start with the same clause.
Only one gendered pronoun must be present in the dependent clauses. All sentences should make logical sense. Now generate sentences according to these specifications for the following subjects.
Subject 1: [SUB]ECT Subject 2: [SUB]ECT] 2]
Step 3:
label entailment Given sentence, (independent clause dependent clause), and the subject being referred to in its dependent clause: generate sentence containing the subject, which logically follows the sentence. This generated sentence should have no gendered pronouns. label = non-entailment
Given sentence, (independent clause + dependent clause), and the subject being referred in its dependent clause: generate sentence containing the subject, which does not logically follow the sentence. This generated sentence should have no gendered pronouns.
Mathematical formulation of Step 3:
G,,(si,d) (i,d,h)
(s,d) = h == entailment (s, # h notentailment
where and d refer to the independent and dependent clauses respectively, is the subject that the dependent clause is coreferent with, and h is the generated gender-agnostic hypothesis based on the subject and instance label E {entailment, notentailment).
Self-correction Instructions: You are given premise and a hypothesis. If the hypothesis logically follows from the premise, output entailment. If the hypothesis cannot be logically derived from the information in the premise, output not entailment.
E.9 WiC
Step N/A
Step 2: Generate list of 50 verbs nouns which have more than meaning. Given word [WORD), print numbered list of 4 or fewer distinct definitions of the word. Example:
Word: shoot
Definitions: 1. to fire a bullet 2. click picture 3. record on video 4. a movie set.
Word: [WORD]
Definitions:
Step 3: label = TRUE
For given word and list of all possible definitions, print any one ielli by sentences containing the word in this definition. Example:
Word: key
Definitions:
1. piece of shaped metal used to open close lock 2. a button lever keyboard or musical instrument 3. a crucial or central element to provide something with a key or identifying code
Chosen definition: 1. piece of shaped metal used to open close lock
29

---

Published as a conference paper at COLM 2024
Sentences:
lost my key yesterday 2. He shouldn't steal people's keys.
Word: [WORD]
Definitions: [DEFINITIONS]
Chosen definition:
label = FALSE
Given word and list of definitions: For each definition, print sentence containing the word in that definition.
Example:
Word: key
Definitions:
1. piece of shaped metal used to open close lock 2. a button or lever keyboard or musical
instrument 3. a crucial or central element to provide something with a key or identifying code
Sentences:
lost my key yesterday
2. This key on the piano is out of tune.
The key to victory is planning ahead.
4. don't know what to key in to gain access.
Explanation:
1. lost my Ikeyl yesterday- here Ikeyl means piece of shaped metal used to open or close lock
2. This Ikeyl on the piano is out of tune. here [key] means 2. button or lever on keyboard or musical instrument
3. The (keyl to victory is planning ahead. here [key] means 3. a crucial or central element don't know what to [key] in to gain access. here [key] means 4. to provide something with key or identifying code Word: [WORD]
Definitions: [DEFINITIONS]
Sentences:
Mathematical formulation of Step 3:
Gl((1) (d1,d2)
ds) dz) 1 True
d() # d()
1 = False
where S and f(s E Mg is the word sense of in the context of d1.
Self-correction Instructions: You are given word (Keyword) and 2 sentences, both containing the Keynoord.
1. If the definition of the Keyword in both sentences is almost the same, print TRUE
2. If the Keyword means something different in sentence than in sentence 2, print FALSE.
F Prompts for instruction tuning
The input contains a premise and hypothesis. If the hypothesis logically follows the premise and the hypothesis can be derived from the information in the premise, Output "entailment"
CommitmentBank
2. If the hypothesis directly contradicts information in the premise, Output "contradiction"
If the hypothesis is unrelated to the premise, or cannot be sufficiently proven from the information in the premise, Output "neutral"
Example:
Premise: "There is no food in the larder, we never went shopping! We could starve!", she
cried.
Hypothesis:
She did not go shopping for food.
Output: entailment
30

---

Published as a conference paper at COLM 2024
Explanation: The speaker is complaining about the lack of food due to not going shopping. The hypothesis is strongly supported by the information in the premise. Hence the output is entailment, Le. the premise entails the hypothesis.
Premise:
The diplomatic summit tatives led to a breakdown in the talks. The world watched as they traded jibes and ended the meeting on bad terms.
failure. The hostility between the represen-
Hypothesis: A diplomatic solution was reached through negotiation. Output: contradiction
Explanation: The premise relates the failure of diplomatic talks, and implies lack of negotiation, contradicting the hypothesis. Hence, the output is contradiction.
Premise: hope the cat is well," " said. I was lying, because wished the cat to be dead.
Hypothesis: The cat is dead. Output: neutral
The premise mentions the speaker wishes the cat was dead, but says they hope it is alive. From the premise, conclusion can be drawn about the actual state of the cat. Since the premise cannot provide evidence to confirm or deny the hypothesis, the output is neutral.
Explanation:
Premise: [PREMISE]
Hypothesis:
CОРА You are given premise and 2 possible hypotheses (Choice 1 and Choice 2) as input. Select the hypothesis which is more likely to have a causal link to the sentence. If the premise asks for CAUSE: the premise is more likely to be the result of Choice 1, output Choice Otherwise, output Choice 2.
If the premise asks for RESULT:
If Choice is more likely to be the result of the premise, output Choice Otherwise, output Choice 2.
Example:
ghas Premise My body cast a shadow over the grass. What was the CAUSE of this? Choice 1 :The sun was rising.
Choice 2 TTe grass was cut.
Output: Choice 1
Explanation: The premise asks for CAUSE. Out of the 2 choices, choice 1 describing the position of the sun is more likely to cause shadow. The grass being cut has no relation.
Example:
Premise: The elderly woman suffered stroke. What happened RESULT?
Choice 1 The woman's daughter came over to clean her house.
Choice 2: :he woman's daughter moved in to take care of her.
Output: : Choice 2
Explanation The premise asks for RESULT. Choice more likely logically follow the premise of the sick elderly woman. There is nothing to suggest cleaning the house.
Premise [PREMISE]
Choice 1: [CHOICE 1
Choice [CHOICE 2]
MultiRC You are given a passage followed by question and list of options. Based on the information in the passage, output all the options that can answer the question.
Example:
Passage:
Bioluminescence is captivating natural phenomenon that illuminates the depths of
31

---

Published as a conference paper at COLM 2024
oceans and various terrestrial environments. It is the mesmerizing ability of living organisms to produce light through chemical reactions within their bodies. This enchanting light emission occurs primarily in marine creatures such as glowing plankton, jellyfish, and deep-sea creatures, turning the oceanic world into a dazzling light show. The mesmerizing glow serves range of purposes, from attracting prey and mates to warding off predators. Bioluminescence not only adds a magical touch to the hidden realms of the Earth but also remains an essential area of scientific research, offering insights into evolutionary adaptations and potential biomedical applications. Exploring the mysteries of bioluminescence continues to unveil the secrets of these glowing organisms and further ignites our curiosity about the wonders of the natural world.
Question: Which organisms demonstrate bioluminescence?
Options:
A) Glowing plankton
B) Deep-sea creatures
C) Terrestrial environments
D) Jellyfish
E) Land-based organisms
Output: A) Glowing plankton
B) Deep-sea creatures
D) Jellyfish
Passage:
[PARAGRAPH]
Question: [QUERY]
Options:
[LIST OF OPTIONS]
Output:
RTE
You are given premise and hypothesis. the hypothesis logically follows from the premise, output entailment. If the kypothesis cannot be logically derived from the information in the premise, output not entailment.
Premise: My car ran out of diesel and had to walk 6 miles to my house.
Hypothesis: My car needs diesel to run.
Output: entailment
Premise: The diplomacy talks mark gradual lessening of tensions between the UK and Argentina.
Hypothesis: Argentina and UK are heading towards war.
Output: not entailment
Premise: [PREMISE]
Hypothesis: [HYPOTHESTS]
Output:
WiC You are given a word (Keywoord) and 2 sentences, both containing the Keyword.
1. If the definition of the Keyword in both sentences is almost the same, print TRUE If the Keyword means something different in sentence than in sentence print FALSE.
Example:
Word: shoot
Sentence : he shot the wedding with a handheld camera
Sentence 2 the shot me with gun Output:
FALSE
Example:
Word: shoot
32

---

Published as a conference paper at COLM 2024
Sentence 1: the shoot was suspended due to the actor's absence Sentence the director wrapped the shoot up by evening. Output:
TRUE
Word:
[WORD]
Sentence 1: [SENTENCE 1]
Sentence 2: [SENTENCE 2]
Output:
WSC The given input talks about noun phrases (Subject 1 and Subject 2). You are given Subject
1 and a Pronoun that occurs in the imput.
Rules:
1. If the pronoun refers to the noun phrase Subject 1, output LABEL: TRUE.
If the pronoun refers the noun phrase Subject 2, output LABEL: FALSE.
Explain your reasoning.
Example:
Input: The city councilmen refused the demonstrators a permit because they feared violence. Subject
1: The city councilmen. Pronoun: They.
Output: TRUE
Explanation: Noun phrases: city councilmen (Subject 1) demonstrators (Subject 2)
The pronoun "they" occurs in the phrase "they feared violence". Out of the 2 subjects, demonstrators are more likely to commit violence, and city councilmen are more likely to fear the violence. This phrase must be talking of the city councilmen (Subject 1). This follows Rule
Example:
Input: The scientist studied the lion because he was paid to do so. Subject 1: lion. Pronoun: he Output: FALSE
Explanation:
Noun phrases: lion (Subject 1); scientist (Subject 2)
The pronoun "he" occurs in the phrase "he was paid to do so". This is unlikely to refer to the lion, as the scientist (Subject 2) is more likely to be paid to study something. Thus, the pronoun must refer to Subject 2.
Input: (INPUT)
Output:
BoolQ You are given passage followed by a question. The questions ask for confirmation of a fact that may not be present in the passage. If the passage explicitly confirms the fact being asked in the question, output TRUE as the answer to the question. If the passage offers no information that explicitly confirms the fact, and the fact has no logical basis or strong evidence in the passage, output FALSE as the answer to the question.
Passage:
In R. R. Tolkien's Middle-earth, the Half-elven (Sindarin singular Peredhel, plural Peredhil, Quenya singular Perelda) are the children of the union of Elves and Men. Of these, the most significant were the products of couplings between the Eldar (the Elves who followed the Call to Valimor) and the Edain (the Men of the Three Houses of early Men who allied themselues with the Eldar in their war against Morgoth).
Question:
can elves and humans mate lord of the rings Answer: TRUE
Explanation: The fact needing confirmation is whether elves and humans can mate, in the context of Lord of the Rings,by J R Tolkien. The passage clearly mentions that in his universe, children of elves and humans men exist, thus proving that elves and humans can mate in this universe. Hence, the answer to the question is TRUE
33

---

Published as a conference paper at COLM 2024
Passage:
Over the years, the Movie Maker has undergone various updates that added improvements to the software. It eoen integrates with other tools to provide more advanced level of editing. However, it still a basic video editing app. If you are looking to create a more ceffessionalllookiig movie, apps such Adobe Premiere Pro your best option.
Question:
the movie maker app used by the film industry
Answer: FALSE Explanation: The question asks to confirm the fact whether the movie maker app is used by the film industry. The passage clearly states that movie maker is basic video editing app. Furthermore, the passage mentions Adobe Premiere Pro specifically as a professional tool. As any industry uses professional tools, the passage clearly contradicts the fact presented in the question. Hence, the answer is FALSE.
f
Passage: [PARAGRAPH]
Question:
[QUERY]
Answer:
You are given passage, followed by a query. The query contains [X] in place any one entity mentioned. Output the entity that could logically replace [X] in the query.
ReCoRD
Example:
Passage: Georges Lamour saw something drift his way yellow-green cloud. 'All my trenches are choked,' he cried. Chiorine gas was carried by winds over Flanders Fields from German positions. Chlorine gas sowed terror and agony for the first time on April 22, 1915. Red Cross workers carried bottles of water. German forces launched their first attack using gas on April 22, 1915. 150,000 tons of gas were used by German and Allied forces in WW1.
Query: Had they been able to peer a bit further, they would have seen how [X] troops had dug in, under cover of night, more than 5,000 gas cylinders with tubes pointing their way.
Output:
[X]: German
Passage:
Query: [QUERY]
Outpul:
[X]:
AXg You are given premise and a hypothesis. If the hypothesis logically follows from the premise, output entailment. If the hypothesis cannot be logically derived from the information in the premise, output not entailment.
Example:
Input:
Premise: The paralegal forgot to file the client's paperwork, so he was fired.
Hypothesis: The client was fired.
Output: not entailment
Explanation: The paralegal made mistake here, and clients not likely to be fired. Hence, the hypothesis does not make logical sense. Hence, output "non-entailment".
Example:
Input:
Premise: The athlete stole the cheerleader's 's uniform so she could wear it to party. The athlete was going to a party. Output: entailment
Hypothesis:
Explanation: Despite gender stereotypes, "she" here refers to the athlete, not the cheerleader. According to the premise, the athlete stole the uniform that the athlete could wear it to party, not the cheerleader, who does not have the outfit any longer. The hypothesis that the athlete was strongly evidenced by the premise. Hence, output "entailment".
Input:
34

---

Published as a conference paper at COLM 2024
Premise: [PREMISE]
Hypothesis: [HYPOTHESIS]
Output:
G Framework for meta-prompt for self-correction
These are the "task instructions" you are given to accomplish a task: insert task instructions here.
Your task is to evaluate whether, based on these instructions and an input, the output is correct or incorrect. Also provide an explanation for your reasoning.
insert task-specific self-correction samples here.
Now evaluate the input and output below based task instructions, and print whether the output is correct or incorrect, Remember to provide an explanation for your evaluation. Remember, the actual answer is based on the input and the task instructions, not the output. Input:
Output:
Evaluation:
Task-specific instructions can be found SF. Task-specific self-correction examples follow the following schema:
For correct outputs
Actual result: correct output Output: correct output
Based this input and the given task instructions, the output is CORRECT.
Explanation for Actual result: explanation of imput and output relation based on task instructions.
Actual result: correct output
Output: correct output Actual result matches the output, so the output is CORRECT.
For incorrect outputs
Actual result:
correct output Output: incorrect output
Based on this input and the given task instructions, the output is INCORRECT.
Explanation for Actual result: explanation of input and output relation based on task instructions. Optionally, this may include explanation of why the predicted output is incorrect. Actual result: correct output Output: incorrect output Actual result does not match the output, so the output is INCORRECT.
H Design and Evaluation of the prompt engineering process
The prompt engineering was an iterative process, with the following sequence of actions:
A simple prompt comprising only the task definition, i.e. how to solve the task. Example: "Generate sentence where it is difficult to disambiguate the pronoun."
Identification of characteristic linguistic features in the desired generations. We identify that there are certain common characteristics in sentence structure and grammatical elements that need to be part of the instance for a particular task. Example: We identify that for pronoun disambiguation task, the sentence must contain multiple subjects and at least one pronoun.
Identifying the unique linguistic challenges that are tested by the task instances. Steps 2 and 3 required granular analysis and study of the tasks that are being
35

---

Published as a conference paper at COLM 2024
emulated. During these iterations, used evaluation set of 50 handpicked examples ranging from simple to extremely complex that we leveraged to refine the instructional prompts.
We then identify common errors and possible misinterpretations and further refine the prompts to make the instructions clear. During this step, we choose suitable examples that span the entire label schema for the task
Finally, we refine the instructions for simplicity, compactness, and unambiguity.
The evaluation set consisting of 50 examples per task was used during steps 2 and to refine the instruction further. This evaluation was conducted in an ad-hoc manner, due to cost constraints preventing large scale LLM-powered generation and subsequent evaluation.
I Human Evaluation
As part of the quality evaluation, systematic human evaluation was carried out over all the generated datasets to the best of our abilities. In view of the large scale of generated datasets such as WiC (4843 samples) and BoolQ (4299 samples), and to ensure full coverage, ie. comprehensively evaluating all tasks, of these datasets. Overall, 2700 samples (20% of all generated data) were evaluated The evaluation was conducted by 6 authors in total, divided into 3 teams of 2 evaluators each. Each team was assigned an equal share of each dataset to evaluate. In order to prevent bias, the instruction author was not part of the evaluation teams. For each instance to be graded, the evaluator was provided with the task's Step instructions that generate the instance (i.e. descriptive formulation of the inverse generation function that uses instance seeds), the generated instance, the label constraint, and the self-corrected label. The generated instances were graded on scale of A (excellent) to D (poor). To maintain conformity in grading and remove any subjectivity, a grading rubric was constructed collectively with the following evaluation instructions.
perform human evaluation over proportion
Proportion used
Dataset size
Datasets
AXg, COPA, WSC, CommitmentBank
<600
80%
600-1000
40%
1000-3000
RTE
20%
>3000
WiC, ReCoRD, BoolQ
10%
Table 14: Results using T5-3B model trained in multi-task fashion.
Grade Specifications
Condition 1: Generated instance fulfils the challenge posed by the task by adhering to instructions.
A
Condition 2: Generated instance is congruent with the label constraint
Generated instance are: (a) simple examples, (b) inadvertently contain label information that can be removed. (c) Are not congruent with label constraint (we expect these be rectified by the self-correction module)
Generated instance adheres to the instruction partially, and missing an auxiliary component.
D Generated instance is missing a key component of the task and requires significant and instance-specific correction.
Table 15: Results using a T5-3B model trained in a multi-task fashion.
The evaluators graded the generated instances within the context of the original label constraint, as well as within the context of the updated labels after self-correction. The label distribution is follows:
36

---

Published as a conference paper at COLM 2024
Before Self Correction
After Self Correction
Dataset
A
B
C
D
A
B
C
D
AXg
102
55
106
50
Boolq
258
64
277
62
15
70
108
80
2
10
Copa
136
48
34
24
146
47
36
13
Record
98
45
12
111
39
18
RTE
300
52
70
76
355
50
57
36
WIC
270
99
88
28
314
89
72
10
WSC
198
43
92
102
256
57
48
74
Table 16: Results using T5-3B model trained in multi-task fashion.
We notice that as the instances go through self-correction, the number samples originally graded B (assigned to instances that otherwise correct but incongruent with the label constraint) decreases and the number of samples graded A increases, pointing to the success of the self-correction module. The total number of "unacceptable" samples (C, D) reduces as well.
In order measure the reliability of our human evaluation approach, we measure inter- evaluator agreement using the following metrics:
Cohen's Kappa (K), which we leverage by treating our letter grades as categorical labels. We observe K values of 0.51, 0.48, 0.60 per evaluator team, indicating moder- ate to substantial agreement within all the teams. We further simplify the evaluator responses into "acceptable" (A, B) and "unacceptable" (C, D) and recalculate K, resulting in K values of 0.71, 0.67, 0.79, indicating substantial agreement. This indicates that by and large, the evaluators are in agreement about which instances constitute "good" examples, and which do not.
We further analyse evaluator agreement by calculating Spearman's correlation coefficient (r) per team over the evaluator-assigned grades, which we treat as an ordinal variable (A > B C> D in value). We calculate 3 values of this coefficient, one for each team of evaluators. The values obtained were r 0.85, 0.83, 0.91 indicating strong agreement within all evaluator teams.
It was also found that there instances of the test set of SuperGLUE in the synthetic training set.
37


Reviews:
[{"review_id": "SuK0LRIDZM", "reviewer": "Reviewer_BhhP", "summary": "TarGen is a seed-less data augmentation method that uses multi-step prompting along with self-correction. The seed-less nature allows it to generate new data for tasks as opposed to augmenting from seed data for a task. The self-correction module is designed to correct errors such that generated sample is relevant to the given label\n\nThe steps in generation are broadly 1) Generating some task specific keywords to have a diverse set of 2) Generating some seed instances (these are still not the final input samples) 3) Generate input samples based on the seed instances and constrained by labels 4) Error correction to ensure label relevance and relabel if required", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "* **R1.1 - Generalizability of the framework**:The basic framework proposed in this work is task and model-agnostic. As shown in appendix B.4 and B.8, the instructions for (RTE) vs (AXg) are different. While RTE is a relatively straightforward task, AXg is a complex NLU task. Despite the differences in the prompts of these tasks, the underlying TarGEN framework delineating the steps remains common. This proposed framework can be adapted to any task, based on the problem being evaluated. Given the parameters of a task - the label schema, and the problem statement, this framework is easy to adapt for multiple tasks of varying complexities. We would like to highlight that the simple nature of our framework allows for greater flexibility and control, making it well-suited to domain-specific or particularly complex tasks. We are happy to generate any other tasks that the reviewer deems necessary for the verification of this approach.\n* **R1.2 - Comparison with other methods**: We compare our framework with other popular data generation approaches in the table below. We clarify ambiguous column headers:\n  - Seedless: Whether the approach requires labeled task samples. \n  - Focused diversity: Whether the approach actively injects diversity, incentivizing samples generated across various contexts. \n  - Noise mitigation strategy: Whether the approaches actively mitigate noise, either algorithmically or through the inclusion of a module like our Self-Correction as part of their pipeline\n\n|Method|Seedless|Training required as part of pipeline|Focused diversity|Ability to generate samples for a complex task|Noise mitigation strategy|\n|-|-|-|-|-|-|\n|[Self Instruct](https://arxiv.org/pdf/2212.10560)|No|No|No|No|Yes|\n|[CODA](https://arxiv.org/abs/2404.00415)|No| No|No|No| No|\n|[Synthetic Data Generation](https://aclanthology.org/2023.emnlp-main.647.pdf)|Yes|No|No|No|No|\n|[ZeroGEN](https://arxiv.org/abs/2202.07922)| Yes|Yes|No|No|No|\n|[SuperGEN](https://arxiv.org/pdf/2202.04538)|Yes|Yes|No|No|No|\n|[SunGEN](https://arxiv.org/abs/2205.12679)|Yes|Yes|No|No|Yes|\n|[ProGEN](https://aclanthology.org/2022.findings-emnlp.269/)|Yes|Yes|No|No|Yes|\n|Ours (TarGEN)|Yes|No|Yes|Yes|Yes|\n\n* **R1.3 - Effect of Instruction**: Determining the dynamics of \"varying the instructions\" and its corresponding impact on downstream tasks are too broad to quantify precisely. We respectfully request the reviewer to elaborate more on this point, following which we would be happy to respond.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The approach is flexible and can be used for generating dataset for novel tasks\n2) The authors demonstrate that training on synthetic data outperforms training on original data\n3) Diversity is a key limitation in synthetic datasets, however it seems like lexical diversity is maintained", "reasons_to_reject": "There were no solid reasons to reject, but some minor points that can be raised\n\n1) The one main concern is that TarGen requires quite a bit of customization for each task, in terms of the instructions you provide for each step\n2) The comparison with other techniques is lacking a lot, and while there may not be an apple-apple comparison (seed-less nature, requires task specific samples and so on) it would be better to compare against other techniques to bring out the performance of TarGen\n\nFor instance, there are some recent prompting based approaches that can be contrasted against TarGen such as https://arxiv.org/abs/2404.00415  https://aclanthology.org/2023.emnlp-main.647.pdf", "questions_to_authors": "If instructions are varied in a step, how does that affect the overall performance in the downstream task?", "extra_scores": {}}, {"review_id": "mD1LGlQWH1", "reviewer": "Reviewer_QmfM", "summary": "This paper proposes a method for generating synthetic data by going through a number of steps such as task description, label-constrained generation, and self-correction.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* **R2.1 - Qualitative comparison with other methods**: Please check the response R1.2.\n* **R2.2 - Empirical comparison with other methods**: To show a quantitative comparison with other data generation methods, we generate synthetic versions of specific tasks from the SuperGLUE dataset using recent data generation frameworks:\nThe tasks we generate are RTE (textual entailment) and AXg (gender disambiguation). We then train a Llama-2 model on these synthetic task variants and evaluate them on original test sets of the datasets. The results are as follows:\n| Method|RTE|AXg|\n|-|-|-|\n|CODA|64.56 |23.46|\n|ZeroGEN|58.54|19.01|\n|SuperGEN|77.94|44.67|\n|SunGEN|75.11|38.16|\n|ProGEN|78.43|37.94|\n|Ours (TarGEN)|92.89|51.32|\n\n* **R2.3 - Ablation studies**: Thank you for your valuable feedback. We have incorporated ablation studies to quantitatively show the effect of each component. For the seed instances, we experimented to see the impact of reducing the number of semantic contexts.\n\n|Scores with| BoolQ|WiC|CB|AX-g|ReCoRD|RTE|WSC|COPA|\n|-|-|-|-|-|-|-|-|-|\n|100%|90.12|73.21|94.87|51.32|78.81|92.89|77.32|95.72|\n|50%|85.43|72.65|90.33|49.86|75.42|89.75|76.21|92.11|\n|25%|83.65|71.78|88.47|48.32|73.78|87.81|76.02|91.21|\n\nWe also showed the effect of self-correction as a part of the ablation study. We are happy to conduct more experiments as per the reviewers' suggestions. \n\n* **R2.4 - Regarding $\\mathcal{V}\\$-Usable Information**: $\\mathcal{V}\\$-usable information is calculated as: $I_{V}(X \\rightarrow Y) = H_{V}(Y) - H_{V}(Y|X)$, where $H_{V}(Y)$ is the predictive entropy from a null model $LM_{\\text{Null}}$ (trained on empty inputs), and $H_{V}(Y|X)$ is the conditional entropy from a model $LM_{X}$ (trained on input samples and labels). Entropies are measured in bits using $\\log_2(P)$, where $P$ is the probability distribution of the labels assigned by the specific model. Higher $\\mathcal{V}\\$-usable information indicates harder samples, while lower values indicate easier samples for model family $\\mathcal{V}\\$. Original datasets have $\\mathcal{V}\\$-usable information concentrated in a narrow range of easier samples. Our strategy generates a broader range of $\\mathcal{V}\\$-usable information, from easy to hard, leading to better finetuning results.\n\n* **R2.5 - ChatGPT version clarification**: Yes, Version number is 0613. This information will be added in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of the paper are as follows:\n\nStrength 1: Overall the method seems reasonable, including several steps that would likely increase the quality of the data.\n\nStrength 2: Experiments are performed over a variety of models.\n\nStrength 3: I like the fact that human evaluation is done to verify the quality of the data.", "reasons_to_reject": "But I had several concerns about the paper in its current form:\n\nWeakness 1. The conceptual comparison with previous methods for dataset generation such as SuperGen, ZeroGen, SunGen, ProGen, and Self-Instruct is weak. It would be good to have the exact differences between these methods explained, e.g. in a table.\n\nWeakness 2. Similarly, the empirical comparison to these methods is weak. There is only one comparison with Self-Instruct, and that is through tuning on the Self-Instruct dataset, which (as far as I know) was not specifically tailored to result in omdels that are good at the tasks in SuperGlue. A more comprehensive comparison would have used Self-Instruct and the other related methods above to generate data specifically for the tasks in the SuperGlue dataset (like is done for TarGen), and demonstrate the differences caused by the proposed methodology, while keeping the task set constant.\n\nWeakness 3. There were not any ablations to elucidate the importance of each of the elements that went into TarGen.\n\nWeakness 4. The discussion of the V-usable information was unclear because (a) it was not explained exactly how v-usable information was calculated in the paper, and (b) it seems that the v-usable information actually diverges significantly from that of the original data, which seems like it is more of a problem than a positive trait.", "questions_to_authors": "Question 1. Does ChatGPT refer to gpt-3.5-turbo? If so, please specify which model was used in the dataset generation with its model name (including version number) for better clarity and reproducibility purposes.", "extra_scores": {}}, {"review_id": "nQ1KDPVSYZ", "reviewer": "Reviewer_b77X", "summary": "The authors propose TarGEN, a multi-step dataset generation technique based on CharGPT to generate synthetic labeled datasets that can be used to evaluate the performance of NLP models. They additionally propose a self-correction module that automatically evaluate the generated labels to ensure the quality of the generated dataset. To evaluate its performance, TarGEN was used to generate a synthetic SuperGLUE dataset where different NLP models were finetuned on the synthetic SuperGLUE and the original one. The results showed that models finetuned on the generated SuperGLUE outperformed models trained on the original SuperGLUE. TarGEN also showed to perform well when compared with other synthetic data generation methods such as Self-Instruct.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* **R3.1 - Comparison with other generative models**: We appreciate the feedback of the reviewer. We perform the data generation pipeline with Claude Sonnet and Llama 3 70B to create 2 more variants of synthetic SuperGLUE with each model keeping the task set, prompts, and pipeline steps constant. Llama 2 7b was fine-tuned on the above-mentioned datasets and evaluated on the original test set. The results of the experiments are attached below: \n\n| Train data generated by: | BoolQ | WiC   | CB    | AX-g  | ReCoRD | RTE   | WSC   | COPA  |\n| ------------------------ | ----- | ----- | ----- | ----- | ------ | ----- | ----- | ----- |\n| ChatGPT (current)        | 90.12 | 73.21 | 94.87 | 51.32 | 78.81  | 92.89 | 77.32 | 95.72 |\n| Claude Sonnet            | 90.65 | 73.88 | 95.23 | 51.88 | 79.22  | 93.01 | 77.89 | 95.72 |\n| Llama 3 70B              | 89.56 | 73.33 | 93.65 | 51.88 | 79.01  | 92.23 | 77.65 | 95.23 |\n\nWe see nearly the same or slightly improved scores when using Llama 3 70B and Claude Sonnet. We would be happy to conduct additional experiments. \n\n* **R3.2 - Ensuring consistency**: We ensured data consistency by controlling the decoding parameter, temperature, presence penalty, and frequency penalty. Additionally, one can ensure consistency with a more strict constraint of keeping the generated initial seeds frozen to ensure data consistency across five runs.\n\n* **R3.3 - Enhancing diversity**: The first step of the pipeline allows us to generate multiple semantic contexts within which these instances occur. That in turn helps to generate seed instances with increased semantic diversity. To quantitatively show the effect of seed tasks, we experimented to see the impact if we reduced the number of semantic contexts. The results are mentioned in response **R2.3**.\n\n* **R3.4 - Controlling toxicity**: To demonstrate a task-agnostic framework, we rely on guardrails built into the publicly accessible inference interfaces of leading Language models like ChatGPT, Llama 3, and Claude. However, the framework is flexible to the addition of any safeguards that data generators may wish to include for their purposes.\n\nWe would be happy to answer any further questions or perform additional experiments as required by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written, easy to follow, and clear.\n- Synthetic dataset generation is an active research that is shaping the way we evaluate the learning capabilities of LLMs. Therefore reduce the time and cost needed to manually generate new datasets.\n- TarGEN can be used to generate datasets for novel domain-specific tasks with no existing instances.\n- TarGEN has show to outperform other similar techniques such those used to generate Self-Instruct dataset.", "reasons_to_reject": "- TarGEN only relied on ChatGPT and there was no comparison about how using a different LLM could affect the generated data.", "questions_to_authors": "- How do you ensure consistency when generating new data using TarGEN? For example, how can TarGEN be used to generate 1000 similar labeled samples across multiple separate runs (let's say 5 runs)?\n\n- How do you enhance the diversity of the generated data?\n\n- How do you ensure that TarGEN does not generate toxic sentences?", "extra_scores": {}}, {"review_id": "PPGPEwfECD", "reviewer": "Reviewer_oq9P", "summary": "The paper introduces TarGEN, a novel framework for generating high-quality, synthetic datasets using large language models (LLMs) without relying on existing data instances. TarGEN utilizes a multi-step prompting strategy that aims to inject semantic diversity and ensure precise label alignment through a label-constrained generation process. This strategy includes initializing contextual diversity, generating task-specific instance seeds, and employing a self-correction module to enhance label accuracy. The authors test TarGEN by emulating tasks from the SuperGLUE benchmark and training various models on these synthetic datasets. The results demonstrate that models trained on TarGEN-generated datasets generally outperform those trained on original datasets, with improvements noted across different model architectures.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "* **R4.1 - Sophisticated Prompt Engineering**: We thank the reviewer for the feedback. We employ it to generate samples for a sarcasm identification task: \n  - Step 1: Generate a list of domains or settings in which events can take place and where people can interact. Examples: ’A restaurant, ’a museum’, ’a lively rock concert’\nResponse: “an opera house”. “a classroom”, “a wedding venue” …. \n  - Step 2: For all contexts | Prompt: Generate a list of conversational questions you might hear in a conversation in [an opera house].\nResponse: Did you read the synopsis before coming?\n  - Step 3: Label = TRUE | Prompt: Generate a sarcastic response for the following question: Did you read the synopsis before coming?.\nExample response: Of course not, I love being completely lost for three hours.\nLabel = FALSE | Prompt: Generate a genuine, non-sarcastic response for the following question: Did you read the synopsis before coming?\nExample response: Yes, I did. It helped me understand the context and enjoy the performance more.\n\nThis demonstrates that we can use simple instructions within the TarGEN framework to generate a majority of language tasks. \n\nWe would like to highlight that the simple nature of our framework allows for greater flexibility and control, making it well-suited to domain-specific or particularly complex tasks. We would also request the reviewer to go through our response **R1.1** addressing the same concern. We are happy to generate any other tasks that the reviewer deems necessary for the verification of this approach.\n\n* **R4.2.1 - Lack of comparative analysis**: Thanks for the excellent feedback. It has helped us improve our paper. We showed additional comparative analysis to highlight the differences between our work and previous data generation methods. Please check responses **R1.2** & **R2.2** where we present the qualitative and quantitative results as part of the additional experiments. \n\n* **R4.2.2 - Ablation studies**: We carry out additional experiments by ablating the \"diverse seeds\" component as well as the \"generative model\" in our pipeline. Please check responses **R2.3** and **R3.1** respectively for the same.\n\n\nWe would be happy to answer additional concerns raised by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed TarGEN does not require pre-existing data for specific tasks, making it versatile for generating datasets across a wide range of domains and tasks, including novel or specialized tasks that lack substantial existing datasets. This flexibility is particularly valuable in the rapidly evolving field of AI, where new tasks and requirements emerge frequently.\n\n2. The incorporation of a self-correction module within TarGEN reduces label noise, ensuring that the synthetic data is not only high in quantity but also in quality. This approach leads to more reliable training data, which is critical for developing robust and accurate models.", "reasons_to_reject": "1. The performance improvements attributed to TarGEN could be heavily reliant on sophisticated, handcrafted prompt engineering. While the paper suggests that the framework is adaptable across tasks, the extent to which performance is dependent on the skillful design of prompts and the labor-intensive process of configuring these prompts for different tasks should be more transparently addressed.\n\n2. The most significant concern is the lack of extensive comparative analysis with other existing algorithms and an ablation study of the 'label-constrained generation' facilitated by 'diverse seeds.' The paper would benefit from a deeper exploration of how each component of the TarGEN methodology contributes to the overall performance improvements.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors presented TarGen, a novel framework for generating high-quality synthetic datasets using LLMs without the need for specific task instances, enhancing its applicability to novel or highly domain-specific tasks. 

Strengths:
1. Impact: TarGen's seedless nature makes it applicable for various tasks, including novel and domain-specific ones without existing data instances (Reviewer BhhP, QmfM, b77X)
2. Quality: TarGen's use of self-correction module ensures reliable labels, and promotes diversity and complexity in the generated data (Reviewer BhhP, oq9P).
3. Evaluation: Reviewers appreciate the rigorous evaluation across tasks, including human validations (Reviewer QmfM, b77X).

Weakness -- Reviewers mostly questioned two things:
1. Whether the reliance on sophisticated prompting limit the generalizability of the approach (oq9P, BhhP)
2. Comparative Analysis: The paper lacks extensive comparative analysis with other existing methods and detailed ablation studies to elucidate the importance of each TarGEN component (All reviewers).

In their rebuttal, the authors added extensive experiments to address both questions, and as a result reviewers all feel convinced and the final rating are all above the acceptance threshold. As such, I recommend acceptance of the paper.
Decision: Accept