Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
Authors: Zhuoyan Li
Hangxiao Zhu
Zhuoran Lu
Ming Yin
Venue: EMNLP 2023 Main (2023)
Keywords: Synthetic Data Generation
Data Augmentation
Large Language Models
TL;DR: In this paper, we explore the potential and the limitations of the synthetic data generated by large lanuage models for text classificiation tasks.

Abstract: The collection and curation of high-quality training data is crucial for developing text classification models with superior performance, but it is often associated with significant costs and time investment.  Researchers have recently explored using large language models (LLMs) to generate synthetic datasets as an alternative approach. However, the effectiveness of the LLM-generated synthetic data in supporting model training is inconsistent across different classification tasks.  To better understand factors that moderate the effectiveness of the LLM-generated synthetic data, in this study, we look into how the performance of models trained on these synthetic data may vary with the $\textit{subjectivity}$ of classification.  Our results indicate that subjectivity, at both the task level and instance level, is negatively associated with the performance of the model trained on synthetic data.   We conclude by discussing the implications of our work on the potential and limitations of leveraging LLM for synthetic data generation.

Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
Zhuoyan Li', Hangxiao Zhu?, Zhuoran Lu', Ming Yin'
Purdue University
2 Washington University in St. Louis
(1i4178, lu800, mingyin]@purdeee edu
hangxiao@wustl. edu
Abstract
tasks and augmenting the training data in low- resourced data settings (Kumar al., 2020; Yoo et al., 2021; Hartvigsen et al., 2022; Sahu et al., 2022). Most recently, a few studies also investi- gate into the feasibility of generating a synthetic dataset from scratch using LLMs to support zero- shot learning (Ye et al., 2022; Wang et al., 2021; Tang al., 2023; Gao et al., 2023). While LLM- based data augmentation is often found outper- form other data augmentation methods in boosting the model performance, mixed results are reported regarding whether the LLM-generated synthetic data can effectively support model training to en- able level of model performance that is compara- ble to models trained on the data collected in the real world and carefully annotated. This leaves uncertainty for researchers and practitioners in de- ciding whether rely on LLMs for synthetic data generation or to proceed with the traditional data collection and curation pipeline when they need to construct a text classification model for a new task. Naturally, one may wonder what factors might mod- erate the effectiveness of LLM- generated synthetic data in facilitating successful model training.
The collection and curation of high-quality training data is erucial for developing text clas- sification models with superior performance, but it is often associated with significant costs and time investment. Researchers have recently explored using large language models (LLMs) to generate synthetic datasets as an alternative approach. However, the effectiveness of the LLM-generated synthetic data in supporting model training is inconsistent across different classification tasks. To better understand fac- tors that moderate the effectiveness of the LLM-- generated synthetic data, in this study, look into how the performance of models trained on these synthetic data may vary with the subjec- rivity of classification. Our results indicate that subjectivity, at both the task level and instance level, is negatively associated with the perfor- mance of the model trained synthetic data. We conclude by discussing the implications of our work on the potential and limitations of leveraging LLM for synthetic data generation
1 Introduction
Today, pachine-leaminng-ooweer text classifica- tion models have been widely applied in diverse applications such as detecting biased or toxic lan- guage on online platforms (Wiegand et al., 2019) and filtering spam emails (Jindal and Liu, 2007). However, the performance of these models largely depends on the quality of the training data. This poses substantial challenge in practice, especially when models need to be built for novel task do- main to incorporate new classification categories, as the training data collection and curation process is often costly, time-consuming, and complex. Mcanwhile, with the recent advancements in large language models (LLMss, researchers have started to explore the potential of utilizing LLMs for generating synthetic data tailored to specific
We conjecture that one such factor could be the subjectivity of classification tasks. Indeed, lan- guage is inherently subjective and interpretive (Ben- veniste, 1971; Wiebe et al., 2004). Previous re- search has showed that people often perceive the same text in different ways because of their per- sonal biases and perspectives (Sap et al., 2021; Li et al., 2022; Gordon et al., 2022). Thus, achiev- ing high model performance for classification tasks with high subjectivity seems to impose greater demand the training data in reflecting the rich- and nuances present in human language, and the extent which L.e--grrrateed synthetic data can acompolish this objective is unclear.
Thus, in this paper, we formally evaluate the effectiveness of LLM (i.c., the cutting-edge GPT- 3.5- Turbo model) in generating synthetic data to
'The collected human annotations are available at http://gaacccoooaaatttttteeeeeee htw.ttsssssstttttce///mmmmm annotation 1.mollttoo_emmlp23...

---

support model training for different text classifica- tion tasks. We adopt two approaches for synthetic data generation-a zero-shot setting in which the LLM is directly prompted to generate text instances with different labels of interests, and a few-shot setting in which a few real-world data instances are provided as examples to guide the LLM in generating the synthetic data. We conduct two evaluation studies, each corresponding to one di- mension of subjectivity- the first study examines the effectiveness of the synthetic data on 10 types of classification tasks and explores how varies with the task-level subjectivity (i.e., whether this type of classification task is subjective); the second study concerns that given a specific classification task, how the performance of model trained on synthetic data changes with the instance- level sub- jectivity (i.e., whether people tend to disagree with each other on the label of this task instance). Our findings suggest that across the 10 types of classifi- cation tasks that we have considered in this study, models trained on the LLM-generated synthetic data generally perform worse than those trained on the real-world data, yet guiding LLM's synthetic data generation process with a small amount of real-world data (i.e., as done in the few-shot data generation setting) can improve the effectiveness of the data generated. Moreover, we find that the per- formance of models trained on the -LM-gcrcrated synthetic data is very close to those trained on the real-world data for tasks with low subjectivity (e.g.. news topic classification, spam email detection), while the performance decrease is much bigger on tasks with high subjectivity (e.g., humor sar- casm detection). Finally, even within the same type of classification task, models trained on the LLM- generated synthetic data tend to exhibit a higher level of performance on those task instances with lower subjectivity, for which human annotators ex- hibit higher level of agreement in their annotation. Together, our study provides important experi- mental evidence regarding the potential and limi- tations of using LLMs to generate synthetic data for text classification tasks. We conclude by dis- cussing the implications, limitations, and future work of our study.
aging generative models create synthetic data for training machine learning models, especially for computer vision (CV) and natural language processing (NLP) tasks. In the realm of CV, sev- eral works have utilized GAN-based models (Kar- ras et al., 2019) or diffusion models (Nichol et al., 2021) to generate synthetic data for image recogni- tion (Besnier et al., 2020; He et al., 2022) or object segmentation (Zhang et al., 2021). Similarly, in the NLP field, researchers have also probed into the ca- pacity of language models in generating synthetic data for various text classification tasks (Kumar et al., 2020; Chung et al., 2023; Sahu et al., 2022; Yoo et al., 2021; Ye et al.. 2022; Wang et al.. 2021), with mixed results reported regarding the effective- ness of the synthetic data generated. In this study, aim to obtain better understanding of when the synthetic data generated by language models can lead effective model training, and we focus on exploring the role of task subjectivity in moderating the effectiveness of the synthetic data.
Large language models. Based on the Trans- former architecture (Vaswani et al., 2017), large language models (LLMs) have facilitated remark- able progress in the field of natural language pro- cessing. The utilization of bidirectional contexts in the BERT model (Devlin et al., 2018) has re- sulted in superior performance across a wide range of tasks. Building on this, OpenAl's GPT series, comprising of models like GPT-2 (Radford et al., 2019), the colossal GPT-3 (Brown et al., 2020) with an impressive 175 billion parameters and the most recent GPT-4 (OpenAI, 2023), pushed the boundaries of possibilities of LLMs. These mod- els exhibit remarkable proficiency in generating high-quality human-like text (Clark et al., 2021; Dou et al., 2021; Zhou et al., 2023), showcasing capabilities in rudimentary reasoning (Wei et al., 2021), translation (Brown et al., 2020), scientific synthetic data generation (Hämäläinen et al., 2023), and code generation (Menutt et al., 2023). In this study, we focus on leveraging the cutting-edge GPT- ..r-Turb model to explore its capabilities and linitations in synthesizing data for text classifica- tion tasks with different subjectivity levels.
2 Related Work
Generative AT in synthetic data generation. Re- cent advancements in generative AI have motivated numerous studies to explore the potential of lever-
We used GPT-3.5-Turbo as the foundational model to generate synthetic data because at the time of this study, an official API for the more advanced GPT-4 model was not yet available from OpenAl.

---

3 Methodolgy
for the large language model in the data generation process, which can potentially provide LLM with insights of the patterns exhibited in the real-world data. We again start the data generation process by using a context prompt to set the context. However, different from that in the zero-shot setting, here, each time before we instruct the LLM to generate piece of text, we first provide the model with a few randomly sampled real-world data instances (including both the text and the label) as the exam- ples. To keep the LLM from merely rephrasing the provided examples, an additional prompt is used to impose a constraint on the LLM in generating the synthetic data (i.e.. "You should imitate the exam- ple I have provided, but you cannot simply modify or rewrite the example have given.").
In this section, we outline the procedure we have followed when leveraging the large language model to generate the synthetic training data for text clas- sification. We consider two data generation settings in this study, i.c., the zero-shot setting and the shot setting.
3.1 Zero-shot Synthetic Data Generation
Under the zero-shot synthetic data generation set- ting, given a text classification task, we assume that the real-world data in the form of "text-label pairs" do not exist. Thus, in order obtain syn- thetic training data for the text classification task, two sequential prompts are constructed and sup- plied to the pretrained large language model (i.c., the GPT-3.5- -urbo model). First, a customized "context prompt" relevant to the targeted domain of interest is used to set the context. For example, in the case of the IMDB movie review classification task (Maas et al., 201 I), the customized context prompt used is "Imagine you are movie reviewer on the IMDB platform". This prompt aims en- courage the LLM to generate synthetic data that resemble the real texts produced in the targeted domain. After the context is set, a second prompt, i.e., the "data generation prompt", is provided to the LLM, instructing the model to generate texts with specific style, label (with respect to the clas- sification task of interest), and word limit. For example, for the IMDB movie review classification task, the style of the text movie review, and the label is a targeted sentiment conveyed by the review (i.e., "positive" or "negative"). To further enhance the diversity of the generated data, after the generation of every data points (i.c., texts of targeted styles along with their labels)3. we pro- vide "diversity prompt" to the LLM-"Can you provide something more diverse compared to the previously generated aimiigg to increase the diversity of the synthetic data generated.
For more details about prompts used for gener- ating data for each type of text classification task, please refer to the App. D.
Evaluation I: Comparison Across Different Types of Tasks
In our first evaluation study, we investigate into how well the synthetic data generated by LLM under both zero-shot and few-shot settings can support effective model training for different types of text classification tasks. We are especially interested in comparing the model performance between those trained on the real-world data and on the LLM- generated synthetic data, and in understanding how the performance of those models trained on the LLM- -LM-eeeertted synthetic data varies with the sub- jectivity of the text classification task.
4.1 Datasets and Tasks
We experiment with 10 representative datasets covering a variety of text classification tasks: AG's news (Zhang et al., 2015b), IMDB reviews (Maas al., 2011), SMS spam (Almeida et al., 2011), Financial phrase bank (Malo al., 2014), Reddit emotion (Demszky et al., 2020), Rela- tion classification (Gao et al., 2019), Tweet irony speech (Van Hee et al., 2018), Tweet emotions (Mo- hammad et al., 2018), Sarcasm news (Misra and Arora, 2023, Misra and Grover, 2021), and Humor speech (Annamoradnejad and Zoghi, 2020). Sec App. A. for detailed descriptions of datasets and the corresponding text classification tasks. These datasets are selected with the goal of spanning a wide range of task subjectivity in mind. For exam- ple, we conjecture that classifying the news topic
3.2 Few-shot Synthetic Data Generation
Under the few-shot synthetic data generation set- ting, we assume that a small amount of real-world data are available for the text classification task. These data points can then serve as the examples
'To increase data diversity while maintaining a reasonable data generation speed, n is set to 10 for generating short texts (i.c., texts with maximum length of 30 words), and for generating longer paragraphs.

---

category (e.g., as that in the AG's news dataset) is relatively objective, while determining whether texts are humorous (e.g., as that in the Humor speech dataset) is quite subjective (Veatch, 1998).
to obtain a linear ordering of the nodes, If a cycle was detected within the graph, the corresponding tasks were considered to have the same level of subjectivity and were merged into a single meta- node before re-runing the algorithm. Our final task subjectivity ranking results are shown in Table 1.
4.2 Task-level Subjectivity Determination
To formally determine the subjectivity levels of dif- ferent text classification tasks, we first conduct a crowdsourced study to collect subjectivity judge- ments from the crowd.
4.3 Model Training
Given text classification task, following the pro- cedures outlined in Sections 3.1 and 3.2, 3,000 syn- thetic data points were generated for each candidate label under both zero-shot and few-shot settings, We then trained classification models using the real- world training data provided by the original dataset, the synthetic data generated under the zero-shot scttings, and the synthetic data generated under the few-shot settings", respectively. Specifically, we utilized the pre-trained BERT (Devlin al., 2018) and RoBERTa (Liu et al., 2019) models from Hug- gingface's transformers library (Wolf al., 2020) as the encoders, and used the representation em- beddings from the last layer of these models as the input to our classification models. The classifica- tion model itself comprised a hidden layer of 768 units and an output layer, and it was fine-tuned with learning rate of 5c 5 and a batch size of 64. For datasets that provided official partitions for training and test sets, we directly evaluated the classifica- tion model's performance on the test sets. Other- wise, we randomly divided the dataset into training (70%), validation (5%), and test (25%) sets Mod- els' performance was evaluated via Macro-F1 and Accuracy scores, and they were computed by com- paring the model's predictions with the gold labels provided in the test sets. To ensure the robustness of our results, all experiments were repeated three times, and the average performance across these repetitions was reported.
Study procedure. We adopt a comparative ap- proach to collect crowdsourced subjectivity judge- ments in this study. Specifically, we recruited crowd workers from Amazon Mechanical Turk (MTurk), and each worker was asked to complete a sequence of 10 subjectivity judgement tasks. In each task, we randomly sampled pair of text clas- sification tasks from the 10 tasks that we considered in this evaluation, and we presented to the worker the task description, label description, and task ex- amples for each task in the pair. Then, the worker was asked to determine which text classification task in the pair was more objective, with "objec- tivity" of a task defined as "the classification of piece of text is based on clear, identifiable features in the text (e.g.. keywords or phrases), and can be done without being affected by any personal inter- pretation of the text resulted from personal biases, emotions or beliefs." The study was restricted to U.S. workers. Each worker was allowed to partic- ipatc only once and received S1.2 payment. An attention check question was included in the study to validate the worker's engagement, and only the data from workers who successfully passed the at- tention check were considered valid.
Ranking task subjectivity. After excluding re- sponses from inattentive workers, total of 540 pairwise subjectivity comparisons for the 10 tasks were obtained from 54 workers. For each pair of tasks, we aggregated relative subjectivity judg- ments made on this pair to determine which task was perceived as more subjective (i.e., less objec- tive). To produce a ranking of the subjectivity of the 10 tasks, we constructed a directed graph based on the pairwise subjectivity comparisons- each task was node in this graph, and directed edges were added between each pair of tasks, pointing from the one that was deemed as more subjective (on the aggregate level) to the one deemed as less subjective. The topological sort algorithm (Cormen et al., 2022) was then applied to this directed graph
4.4 Evaluation Results
Table summarizes the comparative performance of classification models trained with different data. Below, we highlight few key observations we get from this comparison.
"Under the few-shot setting, we randomly sampled 10% of the data points from the real- world training data provided the original dataset the example pool to guide the LLM's synthetic data generation process, but only sythetic data generated were used to train the models.
rerl-world
'To ensure a fair comparison, we maintained an equal size for both the real- world and synthetic training data by downsampling the dataset with larger size.

---

karao
Dewer
Zere
Fee-bat
$5.25
15.3%
(417)
9236
V126
C375
44.55
$4555
18.66 (075)
(3.95
12.9 (1753
14.90
68.1%
47.08 (6.70
90.4% (5500
57.66
05.92
11.0 (5
(1399)
1903
$7.03
$3.55
11.29 14-13
1158 (4338)
91.8 66253
11.59
18.65
13.35
81.54
07200
1159 (3.50
(2466)158)
12/9 (1633
97.23
18.55
15.99
13-93
58225 63.55
9429 12293
$4.9 (425)
0738
53.95
10.5
5099 (275)
64296325)
1578 (519)
8.35
66.99
(42.89)
73.8%
75.95
71.99
08.360035)
72.74
Saviin
51.00 190
66.00
01.66
07.90
5136-010 97
51.96 91/9
Teacil
0325
5158
0078 (23993
13 13:59
74.2% (1646)
35.65
$5.00
58.99 (3690)
7639/11635
75005 (1009)
75.9 (7753)
Haner
09.5
56.09
(41.00
41.79 (3309))
STARS (16.5)
06.76
53.75
3425 (41.90
4089 (30.5))
(408811290)
14.55 (1277)
Table I: Comparing the performance of classification models trained on the L.L I.L.--enerated synthetic data under the zero-shot or few-shot settings, with those trained with the original real-world data, in terms of Macro-Fl (%) and Accuracy Score (%). In the ""uujjectivtty" column, more "x" symbols indicate higher level of task subjectivity.
Models trained on the real-world data consis- tently outperform those trained on the synthetic data. Our results indicate that models trained on the original real-world data consistently outper- form their counterparts trained on the synthetic data generated under either zero-shot or few-shot set- tings, almost for every task. In particular, with the RoBERTa model, we observe that the average im- provements of the model trained on the real-world data over the models trained on zero-shot synthetic data and few-shot synthetic data are 16.9% and 6.7% in terms of Macro-F1, and 14.9% and 6. % in terms of accuracy. Similar trends are observed with the BERT model as well.
the performance decrease resulted from the usage of the synthetic data is more significant-ffor in- stance, across the cluster of 6 tasks with the highest level of subjectivity in our evaluation, there is an average decrease of 27.4% and 24.2% in Macro-FI and accuracy, respectively, comparing the BERT models trained on the zero-shot synthetic data with those trained on the real-world data. In other words. for text classification tasks that are highly objective, there is great potential in training high-performing models simply based on synthetic data generated by LLMs, but the same method falls short in gen- crating synthetic data that effectively support model training for highly subjective classifications. For more robustness check of this finding (e.g.. on more datasets, when using different LLMS for data generation, when using alternative data generation pipelines), see App. B.4-B.10 for details.
Guiding LLM with real-world data examples can boost the effectiveness of the synthetic data. We also observe that models trained on those syn- thetic data generated under the few-shot settings almost always outperform those trained on the syn- thetic data generated under the zero-shot settings. For instance, for the BERT model, we see an aver- age increase of 10.6% and 8.8% in Macro-FI and accuracy scores, respectively, across the 10 tasks in the few-shot setting, as compared to the zero-shot setting. Similarly, with the RoBERTa model, there is an average increase of 10.3% in Macro-Fi and 8.9% in accuracy scores across the 10 tasks when the real-world data are used as examples for LLM to mimic in the synthetic data generation process. For more analysis of the few-shot synthetic data, please see App. B.2 and B.3.
4.5 Exploratory Analysis: Data Diversity
To explore the potential reasons underlying the model performance difference, we conducted an exploratory analysis on the diversity of the training data. Following Rhys Cox et al. (2021), we used the Remote Clique Score (i.c., the average mean distance of a data instance to other instances) and the Chamfer Distance Score (i.c., the average mini- mum distance of a data instance other instances) to quantify the diversity of a set of data. For both metrics, higher values indicate greater data diver- sity. As shown in Figure 1, we find that in general, the world data appear to be more diverse than the synthetic data generated under the few-shot set- tings, which in turn seem to be more diverse than the zero-shot synthetic data. This might partially explain why models trained the real- weal-world data and the few-shot synthetic data tend to outperform those trained on the zero-shot synthetic data. In addition, we also notice that compared to that on the low subjectivity tasks (i.e., AG, Relation,
Synthetic data support more effective model training for tasks that are less subjective. Finally, we notice that for classification tasks with relatively low levels of subjectivity (c.g., those in the AG's news, Relation classification, IMDB reviews, and SMS spam datasets), the performance difference between models trained on the synthetic data and those trained on the real-world data is remarkably small. However, for tasks with high subjectivity,

---

Relution
IMDB
Spum
Emotion
Hlumor Speech
Iromy
Sarcusm
Tweet Emotions
Average Agreement iT
(1.2)
0.78 (4.5)
(7.6
(8.5)
0.69 (6.6)
0.68 (7.1)
0.68 (6.7)
0.51 (7.7)
0.64 (1.6)
(.67
Krippendortt's a. 0.51
0,43
0.19
0.27
0.30
0.06
0.03
0.01
0.17
0.03
Subjectivity Lovel
A6e
Table 2: The average instance-level annotation agreement for different types of tasks, alongside the corresponding task-level subjectivity. Numbers in parentheses in the first row represent the average number of annotations received per task instance. Higher values for both the average agreement ā and Krippendorff's a indicate a higher degree inter-annotator agreement.
5.1 Instance-level Subjectivity Determination
0.0
Chamny Distance
Given text classification task and specific text in- stance, we consider the degree of agreement among annotators on the label of this text proxy for the subjectivity of this instance- lower level of agreement means that annotators hold more diver- gent views, hence the task may have a higher level of subjectivity. Thus, to formally quantify the sub- jectivity of different instances for different tasks, we again conduct crowdsourced study to collect instance-level annotations.
Ramote Clique
2.4
(a) Remote Clique
(b) Chamfer Distance
Figure 1: Comparing the diversity of the real-world data and the synthetic data.
Study procedure. We again considered the 10 types of text classification tasks as that the first evaluation study. For each type of task, we ran- domly sampled 50 text instances per category from the test set to compose our "evaluation dataset" for that task. We then recruited U.S. workers from MTurk to complete annotation tasks for those in- stances in our evaluation dataset. Specifically, each worker was randomly assigned to one type of text classification tasks. After going through a brief in- struction of the assigned task, the worker was asked to complete 20 classification tasks of the assigned type to get a payment of $1.2, where the texts pre- sented in these 20 tasks were randomly sampled from the evaluation dataset for the assigned type of task. Again, we included two attention check ques- tions in our study to filter out inattentive workers. We ensured that each task instance received at least three annotations from unique MTurk workers.
IMDB, Spam), the differences in data diversity between the real-world data and the synthetic data seem to be more salient on the high subjectivity tasks (i.c., the other 6 tasks), especially in terms of the Chamfer Distance Score. In fact, a t-test shows that the decrease of the Chamfer Distance Score in the zero-shot synthetic data compared to the real data is significantly larger for the high subjectivity tasks than for the low subjectivity tasks (p < 0.01). This suggests that for tasks with high subjectivity, such as interpreting humor or sarcasm in language, LLMs may not be able to generate data instances that can cover the full spectrum of real- life scenarios, which may limit the performance of models trained on the synthetic data.
5 Evaluation II: Comparison Across Different Task Instances
Computing instance subjectivity. Based on an- notations we obtained from attentive workers, we quantify the subjectivity level of cach task instance using the fraction of annotators who agree with the majority label for the task instance, that is:
the previous section, we have discovered that the subjectivity of a task can adversely affect the performance of classification models trained on the LLM -eMereeert synthetic data. However, even for the same type of task, the classification for each individual task instance may exhibit different levels of subjectivity as well. Naturally, one may won- der whether models trained on the LLM-ggnerated synthetic data may show different performance on task instances of different subjectivity. We aim to explore the answers to this question in this section.
Ki 1 = =8)
a4 maxxey
(I)
Ki
where } {1.-... Y) is the set of all possible labels, K; the total number of annotators who la- beled instance i, and r is the k-th annotator's anno- tation on instance i. Intuitively, a lower value of a;

---

Dwta
Fitted Line
Date
Fiited Uie
Pitted Line
Conffident 0.51
Coeffieer 0.22
Conticient 0.43 Raguured b.54
Besquared: 0.92
R-squared: 0.50
Bisquaredi 6.85
R-sq jereu 0.81
Sprarmtet's 5: 0.97
Speamer's pi 0.85
Speemark p:
Spearmen's p. 8.82
Spearman' a 0.91
06
Agresment Threshold
Agreement Threshald
Agreament Threshold
Agreement Threshold
Agresment Threshold
(a) AG
(b) Relation
(c) IMDB Reviews
(d) SMS Spam
(c) Reddit Emotion
Data
Fitted Line
Data
Fitted Line
Data
Fitted Line
Data
Fitted Line
Data Pitted Line
Coofficien: 0,21
Conffiiint 5 0.62
Coeffiient B 0:19 Speerman's
Coeffident B 0.35
Contiirent
0.93
Sowsman'' 0.00
Accueet Soore
Spearman 0.88
Speamman' 0.22
Sowngg
Agreament
(f) Sarcusm News
Humor Detection
(h) Tweet Emotions
(i) Tweet Irony Speech (i) Financial Phrasebank
Figure 2: Changes in the accuracy of the BERT model trained on zero-shot synthetic data as the instance-level annotation agreement threshold varies. The solid blue line in each plot is the linear regression fitted on the data, and the ft-squared score quantifies the goodness of fit, The Spearman's i p assesses the strength of rank correlation between the instance-level agreement threshold and the model accuracy for each task. Higher values for both R- squared and Spearman's 's p. ideally close to 1,, indicate a stronger monotonic relationship between the instance-level subjectivity and the model accuracy.
suggests that consensus is less likely to be reached among annotators on instance i, thus instance i may have a higher level of subjectivity. In Table 2, we report the average values of a (i.e., στ) for instances in the evaluation datasets of different types of tasks, along with the average inter-annotate agreement on each task instance (as measured by the Krip- pendorff's a) as well as the task-level subjectivity level for different types of tasks. We can see that a closely aligns with the Krippendorff's a, and tasks with higher levels of subjectivity also exhibit higher value of 77 in general, indicating that a; potentially serve reasonable proxy for the subjectivity of each task instance.
threshold y for different types of tasks, For most tasks (except for the tasks in the Scarcasm News and Finanical Phrasebank datasets), we observe a strong monotonically increasing relationship be- tween y and the model accuracy, with correlations between them (i.c., (3) being positive and values of the Spearman's 's rank correlation coefficient p often exceeding 0.85. Since increasing the instance-level annotation agreement threshold ?y effectively filters out task instances with high subjectivity, this ob- servation suggests that models trained on synthetic data indeed tend to have varying performance on different instances- even within the same type of tasks, these models still perform better on those task instances with low subjectivity.
5.2 Evaluation Results
As a comparison, we also investigate into whether models trained on the real-world data ex- hibit similar behaviors. The detailed results are reported in App. C. On the high level, while we also observe the trend that these models' perfor- mance appears to increase as the instance-level task subjectivity decreases, such relationship usually weaker than that illustrated in the models trained on the synthetic data (c.g., B and p are smaller).
We now look into whether models trained on the LLM--gnnrated synthetic data exhibit different per- formance on instances with different levels of sub- jectivity, and we focus on the models trained on zero-shot synthetic data in this evaluation. Specifi- cally, given classification task, we trained a BERT model using the zero-shot synthetic data and com- puted its accuracy on the subset of task instances in the evaluation dataset whose instance-level an- notation agreement (i.e., a;) exceeds a threshold 7, and we repeated this computation for many times as we varied the value of 7
Conclusions and Discussions
In this paper, we present an initial exploration into factors that moderate the effectiveness of L.M-- generated synthetic data for facilitating the training of text classification models. Our results show that
Figure/ 2 illustrates how the model accuracy varies with the instance-level annotation agreement

---

the performance of the models trained on synthetic data decreases both for classification tasks with higher levels of subjectivity and on task instances with higher subjectivity. In this section, we provide some potential explanations for the observations of our study, and discuss the implications, limitations, and future directions of our work.
on the real-world data. One plausible explanation is that the nature of generating irony texts for so- cial media involves a creative writing task with few language formality constraints, and recent research suggests that LLMs have the potential to exhibit comparable creativity with human writers in such task (Franceschellli and Musolesi, 2023). Another exception we find is in Section for the Fi- nancial Phrasebank and Scarcasm datasets, unlike other tasks, the effectiveness of the models trained on the synthetic data do not vary much with the instance-level task subjectivity. We conjecture that this can be caused by some task-specific proper- ties. On the Financial Phasebank dataset, accurate sentiment analysis requires the understanding of specialized terminology related to finance. Simi- larly, the Sarcasm detection task aims at identifying sareasm in news headlines from selected sources and requires the comprehension on political top- ics. Thus, on these tasks, LLMs might not be fully equipped with the necessary domain knowledge to create effective synthetic data under the zero- shot setting. In fact, as shown in Figure 2, models trained on the zero-shot synthetic data have very low performance these two datasets, regardless of the subjectivity levels of task instances.
6.1
Why subjectivity adversely impacts the effectiveness of the synthetic data?
We provide a few explanations for why task sub- jectivity is found to be negatively associated with the performance of models trained on the LLM- generated synthetic data. First, highly subjective tasks often require a deep understanding of nuanced human emotions and contextual subtlcties, well as the ability to discern and accurately interpret dif- ferent perspectives, As such, LLMs may encounter limitations in generating data that can capture the extensive range and complexity of real-life use of language. Indeed, as shown in our exploratory analysis in Section 4.5, the diversity of the LLM- generated synthetic data appears to be particularly limited on tasks with high subjectivity, when com- pared the real-worke data. This implies that one potential way to improve the effectiveness of syn- thetic data on high subjectivity tasks is to increase the data diversity and ensure the synthetic data can better reflect real-world data distributions.
6.3 Limitations and future work
We acknowledge that task subjectivity may not be the only factor that moderates the effectiveness of the LLM LLM- generated synthetic data. Future studies can look into the potential moderating role of other factors, such as language formality and the require- ment for domain-specifet knowledge, Our reliance on crowd workers determining task subjectivity may introduce some variability due to their lack of linguistic expertise. Most of our evaluation is also based on the GPT-3.5-Ture model only. It is important to note that the conclusions we get here may not generalize other LLMs (e.g., the more advanced GPT-4), considering the continuous improvements of LLMs in generating human-like texts.
Second, specific to the relationship between the instance-level subjectivity and model performance, we note that the "gold label" of task instance is usually decided by a majority vote within a group of annotators. This means that the gold label may not represent the perspective of each individ- ual (Goyal et al., 2022), and they are sometimes "biased" themselves depending on the annotator decomposition (Li et al., 2022). Thus, it may be challenging for LLMs to generate synthetic data recover such potentially biased "majority view." especially if the LLMs are trained to maintain neu- trality. Alternatively, one may ask for subjective task instances that humans can hardly reach any consensus on, whether the "gold label" is really the only "correct" label? If not, rethinking of how to develop and evaluate models for these task instances is urgently needed.
Our findings suggest that incorporating real- world data examples into the synthetic data genera- tion process can increase the data diversity and boost the performance of the resulting models. Thus, future work can explore strategies that lever- age human intelligence, such as feedback or direct intervention in the generation process, to further enrich the diversity of synthetic data (Chung et al., 2023) and to identify the most "informative" type
6.2 Explaining a few exceptions
Table 1, we surprisingly find that on the Tweet irony detection tasks, models trained on the few- shot synthetic data even outperform models trained

---

of data instance to generate. Finally, the signifi- cant correlation between the subjectivity of tasks or instances and the performance of models trained on synthetic data also suggests the potential to uti- lize the performance of such models as a proxy for approximating task or instance subjectivity, or to estimate the reliability of gold labels.
2021. All that's' human' is not gold: Evaluating hu- man evaluation of generated text. arXiv preprint 1xXiv:210700000
Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2022. Introduction to algorithmss MIT press.
Dorottya Demszky, Dana Movshovitz-Atias, Jeongwoo
Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: Dataset of Fine-Grained Emo- tions. In 58th Annual Meeting of the Association for Compudational Linguistics (ACL).
References
2022. Chatgpt review. Accessed on Kaggle, Avail- able from: https://www. kaggle. com/datasetss salom11712/chacptt--p--evv
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre- Pre-training deep bidirectional transformers for language understand- ing, arXiv preprint A.1.::880000005.
2022. Tweet sentiment emotions. Ac-
cessed on Kaggle. Available from: https:
//ww. kaggle. www/daaasets//aakee..cc. 2635a sentimentt-ndd-emttioo----fff tweets.
Yao Dou, Maxwell Forbes, Rik
Noah A Smith, and Yejin Choi. 2021. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. arXiv preprint arXiv:2107.011 07.24.44.
all MiniLM- 2023. cuntencetrannsfommrrsal
minilm-16-v2. Accessed on Hugging Face Model Hub. Available from: https: https://hugginggf httg://gggggnggceeeee co/
sentence- trr.ssrfrrrees/lll-iiiiiiiiaa
Paul Ekman et 1999. Basic emotions. Handbook of cognition and emotion, 98(45-60):16.
Tiago A. Almeida, Jose Maria Gomez Hidalgo, and Akebo Yamakami. 2011. Contributions to the study of sms spam filtering: New collection and results. In Proceedings of the 2011 ACM Symposium on Docu- ment Engineering (DOCENG'1) 17).
Giorgio Franceschelli and Mirco Musolesi. 2023. On the creativity of large language models. arXiv preprint 2XXx:2334400003
Jiahui Gao. Renjie Pi, LIN Yong. Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023, Self- guided noise-free data generation for efficient zero- shot learning. In The Eleventh International Confer- ence Learning Representations.
Issa Annamoradnejad and Gohar Zoghi. 2020. Colbert: Using bert sentence embedding for humor detection. arXiv preprint arXiv:2004. 12765.
BBC. 2022. Accessed on Hugging Face. Avail- able from: https://huggingfaee htt.faacccoo/aaaaseeeeee co/datasess/ SetFit/bbc-news.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classi- fication. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing DEMNLP-IJCNLA pages 6251-6256 -2566, Hong Kong, China. Association for Com- putational Linguistics.
Emile Benveniste. 1971. Subjectivity in language. Problems in general linguisties, 1:223-30.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. 2020. This dataset does not exist: training models from generated images. In ICASSP 2020-2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael Bernstein. 2022. Jury learning: Integrat- ing dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems, pages 1-19.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:7811-1901.
Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. Is your toxicity my toxicity? ex- ploring the impact of rater identity toxicity annota- tion. Proceedings of the ACM on Human- Human-Compune Computer Interaction, 6(CSCW2):1-28
John Joon Young Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing diversity while main- taining accuracy: Text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04144 04140.
Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in gener- ating synthetic hei research data: A case study. Proceedings of the 2023 CHI Conference on Human
Elizabeth Clark,
Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith.

---

Factors Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval- 2018 task I: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1-17.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi. Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated
generated dataset for adversarial and implicit hate speech detection. arXiv preprint 200.00000..
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Jus- tifying recommendations using distandly- re- views and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), pages 188-197.
Ruifei Hc, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. 2022. Is synthetic data from generative models ready for im- age recognition? arXiv preprint arXiv:2210.074.
Nitin Jindal and Bing Liu. 2007. Review spam detection. In Proceedings of the 16th international conference on World Wide Web, pages 1 1189-1190
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin. Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: To- wards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rXiv2212..10444
Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative ad- versarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 4401 4410.
OpenAL. 2023. Gpt-4 technical report.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188)9
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained trans- former models. arXiv preprint Xrxiv:2003.00 02245.
Zhuoyan Li, Zhuoran Lu, and Ming Yin. 2022. Towards better detection of biased language with scarce, noisy, and biased annotations. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. pages 41 1-423.
Samuel Rhys Cox, Yunlong Wang, Ashraf Abdul, Chris- tian von der Weth, and Brian Y. Lim. 2021. Directed diversity: Leveraging language embedding distances for collective creativity in crowd ideation. In Pro- ceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA. Association for Computing Machinery.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 7907.1 16692.
Gaurav Sahu, Pau Rodriguez, Issam H. Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bah- danau. 2022. Data augmentation for intent classi- fication with off-the-shed large language models.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah Smith. 2021. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. arXiv preprint arXhv:2111.07997.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014. Good debt or bad debt: Detecting se- mantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65.
Andrew M Mcnutt, Chenglong Wang, Robert A De- line, and Steven M. Drucker. 2023. On the design of ai-powered code assistants for notebooks. Pro- ceedings of the 2023 CHI Conference on Human Factors in Computing Systems. CHI '23, New York, NY, USA. Association for Computing Machinery.
Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. 2023. Does synthetic data generation of lims help clinical text mining? arXiv preprint 2rXiv:2303.044 043600
Rishabh Misra and Prahal Arora. 2023. Sarcasm detec- tion using news headlines dataset. Al Open, 4:13-18.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in en- glish tweets. In Proceedings of The 12th Interna- tional Workshop on Semantic Evaluation, pages 39- 50.
Rishabh Misra and Jigyasa Grover. 2021. Sculpting Data for ML: The first act of Machine Learning.

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023. Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions. CHI '23, New York, NY, USA. Association for Computing Machinery.
Thomas Veatch, 1998. A theory of humor.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021. Towards zero-label language learning, arXiv preprint 2xxh:2209009133
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.0165.
Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learn- ing subjective language. Computational linguistics, 30(3):77--308.
Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of abusive language: the problem of biased datasets. In Proceedings of the 2019 conference of the North American Chap- ter of the Association for Computational Linguistics: human language technologies, volume / (long and short papers), pages 602-608.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Pier-
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprini arXiv:220..07922.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Woo Lee, and Woomyeong Park. 2021. Gpt3mix: Leveraging large-scale language models for text aug- mentation. arXiv preprint arXiv: 104.004000000 10400886..
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015a. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text clas- sification. In NIPS.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean- Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. 2021. Datasetgan: Efficient labeled data factory with minimal human effort.

---

A Appendices
(5%), and test (25%) sets. our evaluation, this process was repeated three times, with the average performance reported.
A.1 Descriptions of tasks in Main Study
AG's News: This task involves classifying news articles from the subset of AG's News Topic Classification dataset into one of thee categories: World, Sports and Sci/Tech. The AG's News Topic Classification dataset, collected from over 2.000 news sources by the academic news search engine, ComeTToMyHese consists of a training set of 120,000 instances and test set of 7,600 instances. Relation Classification: This task requires the identification of the relationships between two entities within given sentence. In this study, we focus on four relations: 'country', 'league', "screenwriter', and 'tributary'. The dataset comprises English text sourced from Wikipedia and supplemented with crowdsourced English annotations, Each relation has 700 instances. As the dataset does not provide an official division into train, validation, and test sets, we randomly allocated the dataset into train (70%), validation (5%), and test (25%) sets. In our evaluation, this process was repeated three times, with the average performance reported.
Reddit Emotion: The Reddit Emotion is the subset of the Go Emotions dataset. The Go Emo- tions dataset is comprised of 58,009 comments collected from Reddit, and each comment has been annotated with respect to 28 emotion categories. In this task, we focus on three basic emotions (Ekman et al., 1999): joy, sadness, and surprise. Tweet Irony Speech: The task involves classifying tweets into two categories: irony, non-irony. The dataset, which is composed of Singish-languun tweets, has been manually annotated for these specific categories. The distribution of the data includes a training set of 2,862 instances and test set of 784 instances.
Tweet Emotion: The task involves classifying tweets into four emotion categories: anger, joy, optimism, sadness. Each tweet in this Sangssgg dataset has been annotated by human reviewers with respect to these emotional categories. The dataset is partitioned into training set of 3,257 instances and a test set of 1,421 instances.
IMDB Reviews: This task requires classifying the sentiment of movie reviews from the IMDB platform into one of two categories: positive (pos) negative (neg). The dataset comprises 50,000 movie reviews evenly split, with 25,000 designated for training and 25,000 for testing.
Sarcasm News Headlines: This task requires distinguishing between sarcastic and non-sarcastic news headlines. The dataset comprises 26,709 headlines from two news sources: TheOnion, representing sarcasm, and HuffPsst, representing non-sarcasm. As the dataset does not provide an official division into train, validation, and test sets, we randomly allocated the dataset into train (70%), validation (5%), and test (25%) sets. In our evaluation, this process repeated three times, with the average performance reported.
SMS Message Spam: This task involves the classification of SMS messages from the SMS Spam Collection v.1 dataset into either 'ham' (legitimate) or 'spam' categories. The training dataset contains 5,574 English messages, each labeled according to its legitimacy. As the dataset does not provide an official division into train, validation, and test sets, we randomly divided the dataset into train (70%), validation (5%), and test (25%) sets. In our evaluation, this process was repeated three times, with the average performance reported.
Humor Speech Detection: This task involves discerning humorous from non-humorous content for short texts. The dataset, specifically curated for humor detection, is composed of 200,000 instances, balanced between humorous and non- humorous data. It is divided into training set of 160,000 instances and test set of 40,000 instances.
Financial Phrasebank: This task entails the classification of finance-related sentences into one of three categories positivee negative, or neutral-baste on the sentiment expressed by the sentence. The dataset comprises 4,840 English sentences sourced from financial news articles. As the dataset does not provide an official division into train, validation, and test sets, we randomly allocated the dataset into train (70%), validation
A.2 Descriptions tasks in Robustness Check
BBC News: This task involves classifying BBC news articles into one of 5 categories: business. entertainment, polities, sport or tech. The dataset contains 2225 articles. As the dataset does not provide an official division into train, validation,

---

and test sets, we randomly allocated the dataset into train (70%), validation (5%), and test (25%) sets. our evaluation, this process was repeated
B Evaluation I: Comparison Across Different Types of Tasks (Additional Results)
three times, with the average performance reported. Amazon Review: This task contains classifying the customer review text in the Amazon into one of two categories: positive (pos) or negative (neg). Given the size of original dataset, we randomly sample 10000 instances for evaluation.
B.1 Convergence Analysis
Figure B.1 illustrates the training curves of classifi- cation models across the 10 types of tasks. We find that compared to the training curves derived from the real-world data, models trained on the synthetic data exhibit faster convergence rate and greater propensity to overfit. This indicates that under both zero-shot and few-shot settings, the synthetic data generated by the LLM may lack a degree of diver- sity and falls short fully capturing the complex patterns found in the real world language contexts.
SST-2: This task involves classifying single sentences extracted from movie reviews into one of two categories: positive (pos) or negative (neg). The dataset partitioned into a training set of 67,349 instances and test set of 1821 instances. Yelp Review: The objective of this task is to classify Yelp app reviews into either positive (pos) negative (neg) sentiments. The focus is particularly on the extreme reviews: those rated with five stars (pos) and those with just one star (neg). The dataset is split into a training set with 260,000 instances and a test set comprising 20,000 instances.
B.2 Potential of Few-shot Synthetie Data for Data Augmentation
In the main text, the model performance we report for the "few-shot synthetic data" are based on mod- cls that are trained only the synthetic data. As assume that small amount of real-world data are available under the few-shot data generation setting, natural question to ask whether the few-shot synthetic data can be used to augment the real-world data (which are used as the exam- ples in the synthetic data generation process) and improve the model performance. Answering this question, Table B.1 compares the performance of classification models trained only the limited of real-world data (i.e., those used as example to guide LLM in generating synthetic data), only on the few-shot synthetic data generated, and on the combination of both data. We find that the comparison between the performance of models trained exclusively on the limited real-world data and models trained exclusively on few-shot syn- thetic data is task-dependent. However, when the few-shot synthetic data is combined with the small set of real-world data, the resulting model can out- perform the model trained only on the real-world data for many tasks. This highlights the potential of the few-shot synthetic data for data augmentation.
ChatGPT Review: The objective of this task is to classify user reviews from the ChatGPT mobile app iOS into either positive (pos) negative (neg) sentiments. The focus is particularly the extreme reviews: those rated with five stars (pos) and those with just one star (neg). The dataset is comprised of 2292 instances. As the dataset does not provide an official division into train, validation, and test sets, we randomly allocated the dataset into train (70%), validation (5%), and test (25%) sets. In our evaluation, this process was repeated three times, with the average performance reported.
The task involves classi- fying tweets into four emotion categories: anger, joy, optimism, sadness. The tweets in this task are posted during first three quarters of 2022 (Jan 1-Sept 30). The dataset is comprised of 25,000 instances. As the dataset does not provide an official division into train, validation, and test sets, we randomly allocated the dataset into train (70%), validation (5%), and test (25% sets. In our evaluation, this process was repeated three times, with the average performance reported.
2022 Tweet Emotion:
B.3
Similarity between the Synthetic Data and the Real Data
In the few-shot setting, utilized real-world data examples to guide the generation of synthetic data. To quantify the similarity between the real-world data examples and the few-shot synthetic data gen- erated, we employed pre-trained Sentence Trans- former model (all MiniL.M-L--v2 2023) to convert

---

Free Sead
Loos EmDrgy 16.
Pran Dany
0.75
Beal Setal
laserb Cote
Read letal
0.50
0.22
522
1093
-033
200
400
600
002
535
2000
2500
500
1000
1500
0.00
522
1030
3922
Ienstion
Peeration
(a) AG's News
(b) Relation
(c) IMDB Reviews
(d) SMS Spam
(e) Financial Phrasebank
ara Fres basd
sa Loxs
eatten tro
eaaies from der fame
0.
Cross
0.0
Itaration
Iteration
Starntion
Ptaration
Itarasinn
(1) Reddit Emotion
(g) Sarcasm News
(b) Humor Detection
(i Tweet Emotions
i) Tweet Irony Speech
Figure B.1: The training curves for classification models trained with the real-world data, the zero-shot synthetic data, and the few-shot synthetic data.
BERT
RaRERT
Taak
AG
11..%
$0.25
91.5%
11..
91.85 11.09
51.5% (4005)
MOLES (4.1%)
10,05
21.5%
10,95 (4177))
10.95 (4475)
$2.4% (4L25)
$3.5% (ALES)
Ralakon
95.5%
06.55
94.4%
96.65 13.1
96.75
05.8% (40275)
97.05
47.5%
91.15 (3.99)
91.55 63.99)
67.155 (4.95))
67.75 (4.95)
5M5 Spare
15.2%
19.25
14.7%
16.85 1:34
16.15 (4335)
18.2% (40753
16.155
95.1%
9405 (411)
10.75 (2245)
V8315 (-0.99)
Reddi Erretion
02.5%
02.55
(1095)
91.85
91.8% (479)
91.75
41.8%
17.52
87.25 (4.1))
(8./5/01.353
(1055
Trayy
Toed Bextias
62.50
64.59
64.00 (1)
69.19 (64.69)
30.49 (15.9))
50.5% (1605)
73.26
72.5%
66.99 (599)
72.78 (6026)
73,48 (.12
73.9
Sausics
76.99
15.5%
61.599
64.89 633399
73.5% (11.49)
76,49 (1.99)
12.49
72.%%
01.39 (-3899)
60090 (8099
1224% 2.029)
(1..9)
7696
60:
Haser Speck
0..89
04.76
9696 (7.99)
60.08 (7.38)
08.98 59)
08.98.149)
06.96
8539
$100 39)
$489
(43339)
54.66-4075
(4.09-47795)
Table I:I: Comparing the performance of classification models trained using three types of data: a small amount of the real-world data used as the examples for guiding LLM in synthetic data generation (i.e., "real"), few- shot synthetic data generated by the LLM (i.e., "syathetic"), and combination of both ("real+synthetic"). The performance is measured in terms of Macro-F1 (%) and Accuracy Score (%)
texts into vector embeddings. We then computed the cosine similarity between the embeddings of real-world examples and the embeddings of the the synthetic texts. The consine similarity metric ranges from 11 to 1, and we rescaled to the in- terval of [o, 1]. with I representing the highest level of similarity. Then, for each real-world ex- ample, we obtained its mean similarity with the top 5 most similar synthetic texts in the synthetic data and then computed the average mean simi- larity scores across all real-world examples within each type of classification tasks. As reference, we also conducted the same computation between the real-world examples and the synthetic data gener- ated under the zero-shot settings, and results of the similarity comparisons are shown in Figure B.2.
for the zero-shot and few-shot synthetic data is significant. The results are shown in Table B.2, which indicates that the difference is statistically significant for all but the IMDB review classifica- tion task. In other words, the few-shot synthetic data is more similar to the real-world data than the zero-shot synthetic data, which may partly explain why models trained on the few-shot synthetic data tend to outperform models trained on the zero-shot synthetic data.
B.4 Additional Results on Additional Tasks with Low Subjectivity
In the main paper, six out of the ten text classi- fication tasks we examined turn out to have the highest level of subjectivity based on the crowd- sourced annotations we collected. To further vali- date our observation that LLM-generated synthetic data appear to be more powerful for training accu- rate models for less subjective tasks, we conducted additional experiments on a few more datasets which represent less subjective text classification
Visually, we find consistent trend that the few- shot synthetic data has a higher level of similarity with the real-world examples compared the zero- shot synthetic data. We then performed t-tests on each classification task to determine whether the difference of the average cosine similarity scores

---

10-
Datuset
BBC news
Amazon review
SST-2
Yelp
Example Pool vs Zero- Zero-Shot
Real data
93.6
91.8
89.2
94.3
Example Pool vs Few Shot
Mean of Top-5 Similarity Score
Zero-shot data
91.2
87.7
86.4
87.8
Table B.3: Comparing the performance classification models trained on the LLM LLM-generated synthetic data under the zero-shot with those trained with the original real-world data, in terms of Macro-F1 (%)
0.6
Dataset
ChatGPT App Review 2022 Tweet Emotion
0.-
Zero-shot 79.4
Real
68.9
53.5
Few-shot
76.5
58.8
0.0
Ao News
Retation
IMDI
oparn
Financial
Emotions
Barranor
Hamor
Tweet Emotion
trany
Table B.4: Comparing the performance of Bert classifi- cation models trained on the GPT-3.5 turbo-generated synthetic data under the zero-shot or few-shot settings, with those trained with the original real-world data, in terms of Macro-FI (%).
Figure B.2: Average top cosine similarity between the real and synthetic data
Dataset
p-value
that the LLM we used (i.c., OpenAI's GPT-3.5- turbo) may have been exposed to these datasets in its training process, thus it may simply memo- rize some of the data instances and provide them as the senteettiially-eeerrtue an additional study on two post-2022 datasets, i.c., ChatGPT App reviews (cha, 2022) (to reflect less subjective tasks) and the 2022 Tweet Emo- tion dataset (twe, 2022) (to reflect more subjective tasks). On these two datasets, we repeated the zero- shot and few-shot data generation processes with the GPT-3.5-turbo model, used the resulting data to train Bert-based models, and compared the models' performance the test datasets using Macro-Fl scores. As shown in Table B.4, the results confirm our earlier findings: synthetic data is more effec- tive less subjective tasks like ChatGPT reviews, and using real examples to guide synthetic data generation in the few-shot setting can improve the efficacy of the synthetic data generated.
AG News
p < 0.001
Relation
< 0.001
IMDB
< 0.1
Spam
P 0.001
data, conducted
Financial
P < 0.001
Reddit Emotion
< 0.001
Sarcasm
< 0.001
Humor
p 0.001
Tweet Emotion
P 0.001
Tweet Irony
p < 0.001
Table B.2: T-test results for the similarity comparison.
tasks. This includes the BBC News (BBC, 2022), SST-2 movie review (Socher et al., 2013), Ama- zon US review (Ni et al., 2019), and Yelp review dataset (Zhang et al., 2015a). We compared the performance of BERT models trained on real data with those trained zero-shot synthetic data. As indicated in Table B.3, the average performance difference between real-world data and shoo-shot synthetic data is only 4.2%. This gap is notably smaller than what is observed for tasks with greater subjectivity, reinforcing the finding that the subjec- tivity of a task is correlated with the effectiveness of LLM-generated synthetic data.
B.6 Additional Results When Changing the Volume of Synthetic Data
To see whether training models with more synthetic data can enhance classification performance, we conducted a study on several datasets, using GPT- turbo-3.5 as the LLM to generate synthetic data and Bert as the base model for classification. We varied the size of synthetically generated training dataset from half the real data. As shown in Table B.5, our observa- tions indicate that, unlike in low-resource settings, simply varying the training data size between 0.5 and 3 times of that of the real data using unfiltered
B.5 Additional Results of Two Post-2022 Datasets
maximum of three times as that of
In the main paper, we evaluated the effectivness of L.L.-generated synthetic data for text classifi- cations on 10 datasets, and all these datasets are collected before 2022. To address the concern

---

Ratio
2
the synthetic data produced by the GPT-3.5-turbo model in the zero-shot setting the guiding exam- ples in the few-shot setting to generate data. As illustrated Table B.8, we found that for tasks with varying levels of subjectivity, using synthetic data as examples in the prompt for further synthetic data generation (referred to as "second "second-prompt") leads to larger performance degradation compared data generated in single zero-shot round.
SMS
92.9
93.6
93.4
93.2
93.1
91.8
Relation
92.5
92.1
91.6
91.4
92.2
91.8
Tweet Emotion
52.6
57.8
58.9
57.4
56.3
56.5
Sarcasm
56.2
51.4
49.6
47.2
45.8
43.7
Financial
45.1
55.3
51.2
49.5
48.7
46.4
Table B.5: Comparing performance of Bert classi- fication models trained varying size of the GPT-3.5 turbo-generated synthetic data under the zero-shot set- ting terms Macro-F1 (%).
B.10 Additional Results for Directly Prompting LLMs for Text Classification
synthetic data does not consistently enhance the modell's performance across different tasks.
While LLMs are capable of generating high-quality synthetic data through prompting, their direct clas- sification performance can sometimes lag behind that of smaller models trained on this synthetic data. As shown in Table B.9, for many tasks, directly prompting the GPT-3.5-turbo model for classifi- cation often yields poorer results compared to smaller model trained the synthetic data. This discrepancy might arise because the prompt con- straints defining the label space for the LLM can sometimes too lax, making accurate classifica- tion challenging.
B.7 Additional Results When Using Other LLMs
To examine whether our findings hold true for decoder-based models well as models that are reasonably large, conducted the same evaluation studies using the GPT2-large (774M) and Llama2 (7B) models. We conducted this evaluation on selected datasets from the entire set of datasets which covered different levels of subjectivity. As indicated in Table B.6, we observed that models trained on the LLM-generated synthetic data only exhibits slight variations among different LLMs for each respective task. The overall trend remains consistent: the effectiveness of synthetic data tends be higher for tasks with lower subjectivity.
C Evaluation I:: Comparison Across Different Task Instances (Additional Results)
In order to investigate how models trained on the real-world data perform across task instances of varying subjectivity, we used BERT the foun- dational model for training classification model with the real-world data. As depicted in Figure C.1,, we observed that compared to models trained on zero-shot synthetic data, the performance of mod- els trained on the real real-world data is less affected by the subjectivity of the task instance (i.e., / and p are smaller), except for that on the Scareasm News and Financial Phrasebank datasets.
B.8
Additional Results of Improved Data Generation Pipeline
To see how adopting different data generation pipelines may affect the effectiveness of the syn- thetic data for text classification tasks with dif- ferent subjectivity levels, we conducted an addi- tional study in which we follow the data generation pipelines, SunGen (Gao et al., 2023), to collect syn- thetic data. As shown in Table B.7, while SunGen does offer improvement compared to directly prompting LLMs for zero-shot synthetic data gener- ation, the effectiveness of SunGen compared to the real data is still influenced by the task subjectivity level.
D Additional Details on the Generation of Synthetic Data
The prompts we used to generate synthetic data un- der both the zero-shot setting and the few-shot set- ting are shown in the Table D.1 and the Table D.2.
B.9 Additional Results of Using Synthetic Data as Examples Few-shot Setting
Given the effectiveness of the synthetic data gen- erated in the few-shot settings, one may wonder ways to make few-shot data generation possible even without worldworld textual examples. To this end, we conducted an additional study by using

---

Dataset
AG
IMDB
SMS
Tweet Emotion
Humor Speech
Tweet Irony
Subjectivity Level
***
****
*
Real data
95.3
87.6
97.2
77.7
97.0
72.2
GPT2-L lrre
86.5
80.9
86.4
52.2
51.5
60.88
Llama 2
88.7
82.4
88.5
59.1
57.2
63.1
GPT:3,5 turbo
89.3
81.2
93.8
58.5
56.0
63.4
Table B.6: Comparing the performance of Bert classification models trained on synthetic data generated by various LLMs within zero-shot setting using Macro-FI (%) as the metric.
Dataset
AG
IMDB
SMS
Tweet Emotton
Humor Speech
Tweet Irony
Subjectivity Level
**
Real data
95.3
87.6
97.2
77.7
97.0
72.2
SunGen
91.7
84.7
94.5
61.8
59.9
64,6
Zem shot
89.3
81.2
93.8
58.5
56.0
63.44
Table B.7: Comparing the performance of Bert classification models trained on synthetic data generated by the SunGen pipeline and our zero-shot pipeline using Macro-F1 (%) as the metric.
Dataset
AG
IMDB
SMS
Tweet Emotion
Humor Speech
Tweet Irony
Subjectivity Level
***
wwwke
Real data
95.3
87.6
97.2
77.7
97.0
72.2
Zero-shot
89.3
81.2
93.8
58.5
56.0
63.4
Sccond-Pesd Second-Prompt
87.1
86.9
81.1
55.9
53.8
61.9
Table B.8: Comparing the performance of Bert classification models trained on the zero-shot synthetic data and the few-shot synthetic data where the synthetic data is used as the guiding examples ("second-promad generated by the GPT-3.5 turbo in terms of Macro-F1 (%)
Dataset
AG
IMDB
SMS
Tweet Emotion
Humor Speech
Tweet Irony
Subjectivity Level
*
***
**t**
Real data
95.3
87.6
97.2
77.7
97.0
72.2
Direct Prompt
86.5
82.8
89.4
54.3
59.2
61.1
Zenu-shol
89.3
81.2
93.8
58.5
56.0
63.4
Table B.9: Performance comparisons in terms of Macro-FI (%) between "direct prompt" and "zero-shot data generation" using GPT-3.5 turbo. For the zero-shot synthetica data and real data, we adopted the Bert model as the base for classification.
Date
Fitted Line
Data
Fitted Line
Pitted Line
1.0
Coofficent 0.06
Cocfficion: 0.001
Coofficion: B: 0.27
Cowflickerd R-spwared 0.74 Speaman's 5 0.85
Acciyy Sror
R-squared 0.59
Spearman's 2: 2,83
Speamar's p: 0.43
squared: 0.36
R squaredi
Raqqunet: 2:35 23:
Spcama 15 09)
Agreament Thrashold (a) AG
Agrasment Threshald
Agreament Threshald
Agreamenn Threshold
Agreament Threshold
(b) Relation
(c) IMDB Reviews
(d) SMS Spam
(e) Reddit Emotion
Data Fitted Une
Data
ritted Line
Line
Pittedl
Conflicant 2: 0.19
Conffcient R: 0.035
Coufficent 0.35
Hisquare:
sowareed
Seqqaardd 0.4E Sphermen'' 0.72
B-squamet: 0.00
Rusqpared: 081
0.57
Acounger Soore
Sperrnnt's p: 6.28
Spearman's p. 0.8
Agreement Threshald
Agrnement Thrnchold
Agreament Threshald
Agreament Threshald
o4 Threshold
(f) Sarcusm News
(g) Humor Detection
(h) Tweet Emotions
(i) Tweet Irony Speech
(j) Financial Phrasebunk
Figure C.I: Changes in the accuracy of the BERT model trained on real-world data as the instance-level annotation agreement threshold varies. The solid blue line in each plot is the linear regression fitted on the data, and the R-squared score quantifies the goodness of fit. The Spearman's p assesses the strength of rank correlation between the instance-level agreement threshold and the model accuracy for each task. Higher values for both R- R-squared and Spearman's 's p. ideally close to I, indicate a stronger monotonic relationship between the instance-level subjectivity and the model accuracy.

---

Task
Zero-shot/Few-shot
Now you journalist writing news articles. You are given lopic and must write corresponding news article for it. You are also given length requirement. You must ensure your news meets the length requirement.
Context Prompt:
AG
Data Generation Prompt:
Can you write news report with the topic (label)? The length requirement fnum_words: words. Please be creative and write unique news articles.
Now you are a Wikipedia editor. You need to generate new records for describing the relation between entities. You are given a relation type, as well as a sentence describing the relationship. You must write a sentence to deseribe the specified relationship between the two entities that you came up with.
Context Prompt:
Relation
Data Generation Prompt: Give one pair of entities, which have the relation: (label), and generate sentence which contains the pair of entities that have the relation: (label). The description of the relation (label, description).
Now you are a movie critic. You need to have delicate emotions, unique perspectives, and distinctive style. You are going to write a highly polar review for a movie and post it on IMDB, You are given a movie sttre//tyle and a length requirement. You must come up with a movie that corresponds to the genrelstyle and write a review that meets the length requirement.
Context Prompt:
IMDB
Data Generation Prompt: Write film review for a (genre) movie to express (pos neg) feedback. Each review should have (num_ of words) words. Be sure express your personal insights and feelings. Please be creative and write unique movie reviews.
Context Prompt (Spam): Now you are a person who planning to send a spam SMS message. You must as creative as possible to diversify your messages. Ensure your language is conversational and colloquial. Notice that scammers, in order to make people believe them, will make their spam SMS messages lonk like people's ' daily conversations or very formal and serious content. You also need to imitate these contents. Context Prompt (Ham); Now you are a person who is planning to send SMS message. You must be as creative as possible to diversify your messages. Ensure your language is conversational and colloquial. Notice that in people's 's daily communication, sensitive topics may accasionally be involved, which may sometimes make these contents look like spams but actually not. You also need to imitate these contents.
SMS spam
Data Generation Prompt: Now write SMS messages as required. Be creative and write unique SMS messages.
Now you are a Reddit user and you are going to write a comment express your emotions. You have delicate emotions, unique perspectives, and distinctive style. You are given length requirement. You must write one comment that meets the length requirement.
Context Prompt:
Reddit emotion
Data Generation Prompt: Write one Reddit comment to express your [label] emotion. Your comment should have |num_nt _aam_____a___ words] words. Be sure to express your personal insights and feelings. Be creative and write comments that are different from each others.
Table D.I: Detailed prompts for each task under the zero-shot and few-shot settings for data generation.

---

Task
Zero-shot/Few-shot
Now you person using twitter. You asked write an irony non-irony tweet to express your feelings. Your writing style must consistent with texts in the tweet. You must ensure that your language is colloquial, casual, and Twitter-iike You are given a length requirement. You must ensure your tweet meets the length requirement.
Context Prompt:
Tweet irony
Data Generation Prompt: Write a tweet expressing (label) feeling and ensure that the length of the tweet is ahout [num_ of words| | words. Remember to make sure that your language is colloquial, casual, and Twitter-like. Be creative and write unique tweets.
person using twitter. You are provided with emotion, and you need write tweet expressing that emotion. Your writing style must be consistent with the tweets twitter. You must ensure that your language colloquial, casual, and Twitter-like. You are given length requirement. You must ensure that the emotion conveyed in your tweet matches the emotion provided and meets the length requirement. This used for anything that violates the law or social ethics.
Context Prompt:
You
Tweet emotions
academic study and the content you generate will be
Data Generation Prompt: Write a tweet expressing the [label] emotion and ensure that the length of the tweet is about [num_ot of words] words. Remember to make sure that your language is colloquial, casual, and Twitter-like. Be creative and write unique tweets.
................................
journalist write the sarcastic news headlines. Here few characteristics that might help understand what sarcastic news hcadline: 1) Sarcasm often involves saying something different from what is intended. 2) Sarcasm might involve play words or puns. 3) It may involve exaggeration irony. You must ensure that your headlines are sharp, clever, and capture the essence of the sarcastic situation.
Context Prompt:
You
Sarcasm
Data Generation Prompt:
Write a news headline expressing [label) and ensure that the length the news headlines is about (num_ of [fwordss) words. Be creative and write unique news headlines. Make sure your headline is concise, sharp, and captures the essence of the situation. Please be creative and write unique headlines.
Context Prompt: You are now a journalist writing financial news. You need to write some financial news that express polar sentiments. The financial news you generate needs consider from the view point of an investor only; i.c. whether the news may have positive, negative or neutral influence on the stock price. As result, sentences which have sentiment that not relevant from economic financial perspective considered neutral. You given of the polar sentiments and length requirement. You must write financial news that express the corresponding sentiment and meets the length requirement.
Financial
Data Generation Prompt: Write financial news with (label) sentiment and ensure that the length of the financial news is about [num_ot of_________ words) words. Be creative and write unique financial news.
Context Prompt: You are now creating dataset containing humor and non- non-bumor texts. Here are few characteristics that might help understand what is humorous text: 1) Sarcasm and Irony: Sarcasm and irony involve stating one thing and meaning another, often the opposite. 2) Double Entendre: A double entendre is figure of speech which one is typically obvious, while the other often carries risqué or ironic connotation. 3) Parody and Satire: Both involve imitating and exaggerating the features of a particular language style, genre. piece of content to humorous effect. 4) Absurdity and Nonsense: Language that describes absurd nonsensical scenarios can often he funny. This includes non-sequiturs, in which conclusions do not follow from their premises, and other forms of illogical statements.
particular way of wording that is devised to have double meaning, of
Humor speech
Data Generation Prompt: Write a [label] short text and ensure that the length the short text is about [num_of_ words] words. Be creative and write unique short text.
Table D.2: Detailed prompts for each task under the shro-shot and few-shot settings for data generation (Continued).


Reviews:
[{"review_id": "cXgsKS9MQs", "reviewer": "Reviewer_M41S", "summary": "", "questions": "1. The authors propose a “diversity prompt”. What measures have been taken to ensure the new data is diverse enough? Is it done using Remote Clique Score?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review. Below we address your main questions.\n\n> The paper does not conduct extensive experiments using decoder-based models as well as models that are reasonably large such as the 500M-750M parameters range. If these findings hold good for slightly larger models, it would be empirically conclusive proof of the findings. \n\nTo examine whether our findings hold true for decoder-based models as well as models that are reasonably large,  we conducted the same evaluation studies using the GPT2-large (774M) and Llama2 (7B) models. \nDue to the time constraints, we conducted this evaluation on 6 randomly selected  datasets from the entire set of 10 datasets that we considered in our paper, and the selected 6 datasets covered different levels of subjectivity. As indicated in Table1 below,\n\n**Table 1:** Comparing the performance of Bert classification models trained on synthetic data generated by various LLMs within a zero-shot setting using Macro-F1 (\\%) as the metric.\n|                    |  AG  | IMDB |  SMS | Tweet Emotion | Humor Speech | Tweet Irony |\n|:------------------:|:----:|:----:|:----:|:-------------:|:------------:|:-----------:|\n| Subjectivity level |   *  |  *** | **** |     *****     |     *****    |    *****    |\n|      Real Data     | 95.3 | 87.6 | 97.2 |      77.7     |     97.0     |     72.2    |\n|    GPT2-large (774M)    | 86.5 | 80.9 | 86.4 |      52.2     |     51.5     |     60.8    |\n|    Llama 2 (7B)    | 88.9 | 82.4 | 88.5 |      59.1     |     57.2     |     63.1    |\n|    GPT-3.5 turbo   | 89.3 | 81.2 | 93.8 |      58.5     |      56.0      |     63.4    |\n\nwe observed that models trained on the LLM-generated synthetic data only exhibits slight variations among different LLMs for each respective task.  The overall trend as seen in our paper remains consistent: the effectiveness of synthetic data tends to be higher for tasks with lower subjectivity. \n\n> Although the tasks are slightly different, the following papers mentioned empirically show the effectiveness of the LLM-generated dataset in a zero-shot setting which raises the question, if whether better querying/prompting of LLMs knowledge bases or synthetic data creation pipelines can, in fact, improve the performance of small models using zero-shot synthetic data.\n\nThanks for pointing out these papers, we will add them in the reference and discuss them in the paper. To see how the improved data generation pipeline performs in tasks with different subjectivity levels, we conducted an ablation study by evaluating how SunGen [1] performs in tasks with varied subjectivity levels. \nAs shown in Table2 below, \n\n**Table 2:** Comparing the performance of Bert classification models trained on synthetic data generated by the SunGen pipeline and our zero-shot pipeline using Macro-F1 (\\%) as the metric.\n|                    |  AG  | IMDB |  SMS | Tweet Emotion | Humor Speech | Tweet Irony |\n|:------------------:|:----:|:----:|:----:|:-------------:|:------------:|:-----------:|\n| Subjectivity level |   *  |  *** | **** |     *****     |     *****    |    *****    |\n|      Real Data     | 95.3 | 87.6 | 97.2 |      77.7     |     97.0     |     72.2    |\n|       SunGen       | 91.7 | 84.7 | 94.5 |      61.8     |     59.9     |     64.6    |\n|  Zero-shot (Ours)  | 89.3 | 81.2 | 93.8 |      58.5     |      56      |     63.4    |\n\nwhile SunGen does offer an improvement compared to directly prompting LLMs for zero-shot synthetic data generation, the effectiveness of SunGen's synthetic data compared to real data is still influenced by the task's subjectivity level.\n\n\n> The authors propose a “diversity prompt”. What measures have been taken to ensure the new data is diverse enough? Is it done using Remote Clique Score?\n\nThe diversity prompt serves as a constraint to instruct the LLM to produce more diverse data. We didn't compute the Remote Clique Score or similar metrics between newly generated data instances and the preceding $n$ instances in the generation process with a threshold to determine whether to accept the new generated data or prompt the LLM to regenerate it based on feedback.  The Remote Clique Score and Chamfer distance is used for the post-hoc analysis of diversity of data. We empirically found that without this diversity prompt, the Remote Clique Score  or Chamfer distance between new generated data instances and the previous $n$ instances often tend to be very low.\n\n#### We will add the additional experimental results into the later paper or the appendix.\n\n[1] SELF-GUIDED NOISE-FREE DATA GENERATION FOR EFFICIENT ZERO-SHOT LEARNING (https://arxiv.org/pdf/2205.12679.pdf) - SunGen", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written with a comprehensive explanation of how the subjectivity analysis was done. \n2. Paper experimentally confirms the intuition of zero-shot synthetically generated data performs worse than few-shot synthetic data generation.\n3. Conducted sound evaluation studies and presents acceptable explanations for the various findings as well as the few anomalies in the trends.", "reasons_to_reject": "1. The paper does not conduct extensive experiments using decoder-based models as well as models that are reasonably large such as the 500M-750M parameters range. If these findings hold good for slightly larger models, it would be empirically conclusive proof of the findings.\n2. Although the tasks are slightly different, the following papers mentioned empirically show the effectiveness of the LLM-generated dataset in a zero-shot setting which raises the question, if whether better querying/prompting of LLMs knowledge bases or synthetic data creation pipelines can, in fact, improve the performance of small models using zero-shot synthetic data.\n* Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (https://arxiv.org/pdf/2202.04538.pdf) - SuperGen\n* SELF-GUIDED NOISE-FREE DATA GENERATION FOR EFFICIENT ZERO-SHOT LEARNING (https://arxiv.org/pdf/2205.12679.pdf) - SunGen", "questions_to_authors": "1. The authors propose a “diversity prompt”. What measures have been taken to ensure the new data is diverse enough? Is it done using Remote Clique Score?", "paper_topic_and_main_contributions": "The paper conducts a detailed analysis of the effect of subjectivity on classification tasks and its effect on zero-shot and few-shot-based synthetic data generated by LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "UkDuH4PHVV", "reviewer": "Reviewer_wx6Z", "summary": "", "questions": "While it is not a major issue, I would like to understand from authors their thoughts about using GPT-3.5 as the LLM here. GPT-3.5 has likely been trained on all of internet before 2021 atleast, and all the datasets used in this study are pre-2020. This means it has likely seen the data, including the test data of these datasets. I have three questions on this:\n\nA. Is using GPT-3.5 really zero shot/few shot if it has seen these datasets already? Given how well LLMs memorize examples, I would be particularity be concerned about the few shot setting. \n\nB. What's being thought of as subjectivity/complexity of the dataset, is the artifact of just the training data generator has seen?\n AG is news, IMDB are reviews -- one of the most common data types on the internet. Finance news and humor news are relatively niche and low in volume on internet (I would presume, I don't have data to back it up) compared to other datasets - perhaps needing special knowledge to understand. FPB in particular is a simple dataset but does need specialized knowledge to get sentiment; for example, does increasing interest rates is a negative event? I would doubt a MechTurk worker would had that perspective but someone with a knowledge of finance could have been easy to interpret that event easily. So what is being thought of subjectivity could be just about a) relative occurrence in LLMs training data; b) not as subjective for someone who knows the domain. \n\nC. I don't think these undermine the study, but I would be interested in knowing what authors thing about this and should it be a part of limitations. Would you have seen different results with something like GPT-2 or BloombergGPT?\n\nD. There is no limitation on amount of data one can generate using synthetic data generators. Studies (Aggarwal, 2023) have shown that using more synthetic data (2-10x real data) especially in low resource settings can outperform real data. Further, using weak supervision and simple data selection strategy (Chen, 2022) one can select complex examples which can outperform real data. Did you try generating more than just 1x real data to see if it can help in classification performance and perhaps other measures used in the study?\n\n\nAggarwal, K., Jin, H. and Ahmad, A., 2023. ECG-QALM: Entity-controlled synthetic text generation using contextual Q&A for NER.\n\nChen, M., Papangelis, A., Tao, C., Rosenbaum, A., Kim, S., Liu, Y., Yu, Z. and Hakkani-Tur, D., 2022. Weakly supervised data augmentation through prompting for dialogue understanding. arXiv preprint arXiv:2210.14169.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback. Below we address your main questions.\n\n> Is using GPT-3.5 really zero shot/few shot if it has seen these datasets already? Given how well LLMs memorize examples, I would be particularity be concerned about the few shot setting.\n\nThank you for raising concerns that GPT-3.5 could potentially just memorize the training data examples. To address this concern, we conducted  an additional study on two post-2022 datasets: ChatGPT App reviews [1] (less subjective) and the 2023 Tweet Emotion dataset [2] (more subjective). We trained an Bert-based model on these two datasets. As shown in Table 1 below,\n\n**Table 1:** Comparing the performance of Bert classification models trained on the GPT-3.5 turbo-generated synthetic data under the\nzero-shot or few-shot settings, with those trained with the original real-world data, in terms of Macro-F1 (\\%).\n\n| Dataset   | ChatGPT APP review | 2022 Tweet Emotion |\n|-----------|--------------------|--------------------|\n| real      | 79.4               | 68.9               |\n| zero-shot | 73.3               | 53.5               |\n| few-shot  | 76.5               | 58.8               |\n\n\nthe results confirm our earlier findings: synthetic data is more effective in less subjective tasks like ChatGPT reviews, and adding real examples in the few-shot setting can improve zero-shot data efficacy.\n\n[1] ChatGPT App Reviews. https://www.kaggle.com/datasets/saloni1712/chatgpt-app-reviews\n\n[2] 2022 Tweet Emotion Reviews. https://www.kaggle.com/datasets/ankitkumar2635/sentiment-and-emotions-of-tweets\n\n> What's being thought of as subjectivity/complexity of the dataset, is the artifact of just the training data generator has seen? AG is news, IMDB are reviews -- one of the most common data types on the internet. Finance news and humor news are relatively niche and low in volume on internet (I would presume, I don't have data to back it up) compared to other datasets - perhaps needing special knowledge to understand. FPB in particular is a simple dataset but does need specialized knowledge to get sentiment; for example, does increasing interest rates is a negative event? I would doubt a MechTurk worker would had that perspective but someone with a knowledge of finance could have been easy to interpret that event easily. So what is being thought of subjectivity could be just about a) relative occurrence in LLMs training data; b) not as subjective for someone who knows the domain.\n\nThanks for your constructive thought. \n1. We do not think  ''subjectivitiy'' of tasks reflects simply the relative occurrence of different types of data in LLMs' training data. \nFor instance, compared to its effectiveness in AG news or IMDB reviews, synthetic data generated by LLMs appears less effective for sentiment analysis of posts on online platforms like Tweet or Reddit (Tweet Emotion/Reddit Emotion), which are also common types of data on the Internet and likely be included in the LLMs' training data. Moreover, if the model performance degradation for those types of tasks with higher subjectivity are entirely caused by the limited occurrence of relevant data in LLMs' training data, one would expect that the performance degradation occurs on all task instances regardless of their (instance-level) subjectivity. However, this is different from what we actually observed in our second evaluation study, which suggests that the performance degradation on those types of tasks with high subjectivity mostly occurs on task instances with higher subjectivity. \n\n2. We do not think the task subjectivity is just an artifact of that human subjects recruited in our study lack the domain expertise. In fact, even domain experts may find tasks that are deemed as subjective in our study truly ''subjective''. Take a real instance from the Finanical Phrase Bank--''Sales in Finland rose by 3.9\\% and international growth was 0.7\\% .''-- as the example. This event would be also very subjective for domain experts to determine the sentiment. For example, professionals overseeing the Finnish market might deem this positive as an affirmation of robust domestic strategies while global strategists could view it as the negative due to the modest 0.7\\% international surge as a sign of market penetration challenges. Similarly, judging whether ``interest rates increase'' is positive or negative could also be subjective. For companies or investors grappling with substantial debt, an increase in interest rates could be perceived as negative, given that it would amplify their borrowing costs. On the other hand, those with cash reserves could see it as positive which is an opportunity to earn more interest. \n\n> I don't think these undermine the study, but I would be interested in knowing what authors thing about this and should it be a part of limitations. Would you have seen different results with something like GPT-2 or BloombergGPT?\n\nTo examine whether our findings hold true for decoder-based models as well as models that are reasonably large,  we conducted the same evaluation studies using the GPT2-large (774M) and Llama2 (7B) models. \nDue to the time constraints, we conducted this evaluation on 6 randomly selected  datasets from the entire set of 10 datasets that we considered in our paper, and the selected 6 datasets covered different levels of subjectivity. As indicated in Table 2 below,\n\n**Table 2:** Comparing the performance of Bert classification models trained on synthetic data generated by various LLMs within a zero-shot setting using Macro-F1 (\\%) as the metric.\n|                    |  AG  | IMDB |  SMS | Tweet Emotion | Humor Speech | Tweet Irony |\n|:------------------:|:----:|:----:|:----:|:-------------:|:------------:|:-----------:|\n| Subjectivity level |   *  |  *** | **** |     *****     |     *****    |    *****    |\n|      Real Data     | 95.3 | 87.6 | 97.2 |      77.7     |     97.0     |     72.2    |\n|    GPT2-large (774M)    | 86.5 | 80.9 | 86.4 |      52.2     |     51.5     |     60.8    |\n|    Llama 2 (7B)    | 88.9 | 82.4 | 88.5 |      59.1     |     57.2     |     63.1    |\n|    GPT-3.5 turbo   | 89.3 | 81.2 | 93.8 |      58.5     |      56.0      |     63.4    |\n\nwe observed that models trained on the LLM-generated synthetic data only exhibits slight variations among different LLMs for each respective task.  The overall trend as seen in our paper remains consistent: the effectiveness of synthetic data tends to be higher for tasks with lower subjectivity. For the Bloomberg-GPT model or other domain-specific LLMs , due to the access limit, we can only conjecture that it can generate more effective synthetic data in the domains that require specific financial knowledge. However, this effectiveness might also be moderated by the subjectivity of the task or the task instance itself. For example, even experts might differ in their interpretation of a financial event such as \"the interest increases\". \n\n> There is no limitation on amount of data one can generate using synthetic data generators. Studies (Aggarwal, 2023) have shown that using more synthetic data (2-10x real data) especially in low resource settings can outperform real data. Further, using weak supervision and simple data selection strategy (Chen, 2022) one can select complex examples which can outperform real data. Did you try generating more than just 1x real data to see if it can help in classification performance and perhaps other measures used in the study?\n\nThanks for pointing out these papers, we will add them in the reference. To see whether the more synthetic data can enhance classification performance, we conducted an ablation study based on GPT-3.5 turbo on several tasks with Bert as our base model for classification. Due to the time limit, we selected a subset of tasks from our main study and augmented the size of synthetic data to a maximum of three times that of the real data. As depicted in Table 3 below, \n\n**Table 3:** Comparing the performance of Bert classification models trained on varying size of the GPT-3.5 turbo-generated synthetic data under the\nzero-shot in terms of Macro-F1 (\\%).\n| Ratio of synthetic data to real data size | 0.5  | 1    | 1.5  | 2    | 2.5  | 3    |\n|-------------------------------------------|------|------|------|------|------|------|\n| SMS Spam                                  | 92.9 | 93.6 | 93.4 | 93.2 | 93.1 | 91.8 |\n| AG                                        | 89.4 | 88.7 | 88.2 | 88.3 | 88.2 | 88.4 |\n| Tweet Emotion                             | 52.6 | 57.8 | 58.9 | 57.4 | 56.3 | 56.5 |\n| Financial                                 | 45.1 | 55.3 | 51.2 | 49.5 | 48.7 | 46.4 |\n| Sarcasm detection                         | 56.2 | 51.4 | 49.6 | 47.2 | 45.8 | 43.7 |\n| Relation                                  | 92.5 | 92.1 | 91.6 | 91.4 | 92.2 | 91.8 |\n\nour observations indicate that, unlike in low-resource settings, simply augmenting data by 0.5-3 times using unfiltered synthetic data does not consistently enhance performance across general tasks.\n\n#### We will add the additional experimental results into the later paper or the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is very well written and easy to follow. \n- This work tries to analyze the important problem of how good is synthetic data versus real data, which has been often overlooked. They have done an extensive study involving human evaluation to make their conclusions. This would be really valuable to researchers working in the synthetic data generation community.", "reasons_to_reject": "- I do not see major flaws in this paper except using GPT-3.5 as the LLM. Please see questions for authors. Experiments are well motivated and authors have done a great job in adding requisite details in the paper with sufficient caveats, when required.\n- The only major concern is that what authors are correlating with the complexity of the dataset might just be an artifact of familiarity in the training data of LLMs?", "questions_to_authors": "While it is not a major issue, I would like to understand from authors their thoughts about using GPT-3.5 as the LLM here. GPT-3.5 has likely been trained on all of internet before 2021 atleast, and all the datasets used in this study are pre-2020. This means it has likely seen the data, including the test data of these datasets. I have three questions on this:\n\nA. Is using GPT-3.5 really zero shot/few shot if it has seen these datasets already? Given how well LLMs memorize examples, I would be particularity be concerned about the few shot setting. \n\nB. What's being thought of as subjectivity/complexity of the dataset, is the artifact of just the training data generator has seen?\n AG is news, IMDB are reviews -- one of the most common data types on the internet. Finance news and humor news are relatively niche and low in volume on internet (I would presume, I don't have data to back it up) compared to other datasets - perhaps needing special knowledge to understand. FPB in particular is a simple dataset but does need specialized knowledge to get sentiment; for example, does increasing interest rates is a negative event? I would doubt a MechTurk worker would had that perspective but someone with a knowledge of finance could have been easy to interpret that event easily. So what is being thought of subjectivity could be just about a) relative occurrence in LLMs training data; b) not as subjective for someone who knows the domain. \n\nC. I don't think these undermine the study, but I would be interested in knowing what authors thing about this and should it be a part of limitations. Would you have seen different results with something like GPT-2 or BloombergGPT?\n\nD. There is no limitation on amount of data one can generate using synthetic data generators. Studies (Aggarwal, 2023) have shown that using more synthetic data (2-10x real data) especially in low resource settings can outperform real data. Further, using weak supervision and simple data selection strategy (Chen, 2022) one can select complex examples which can outperform real data. Did you try generating more than just 1x real data to see if it can help in classification performance and perhaps other measures used in the study?\n\n\nAggarwal, K., Jin, H. and Ahmad, A., 2023. ECG-QALM: Entity-controlled synthetic text generation using contextual Q&A for NER.\n\nChen, M., Papangelis, A., Tao, C., Rosenbaum, A., Kim, S., Liu, Y., Yu, Z. and Hakkani-Tur, D., 2022. Weakly supervised data augmentation through prompting for dialogue understanding. arXiv preprint arXiv:2210.14169.", "paper_topic_and_main_contributions": "This work investigates the potential of synthetically generated data in lieu of real data. They investigate effectiveness of synthetic data across a variety of classification tasks versus the \"subjectivity\" of the task. They use a variety of experiments and human evaluation to conclude that LLMs are good at generating data for \"easier\" datasets like news, reviews versus more complex datasets like humor.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EusYN3EQzc", "reviewer": "Reviewer_PuaF", "summary": "", "questions": "I only have suggestions what else to consider in future revisions rather than specific questions:\n\nA. How sensitive the results are to the prompts asking LLM to generate training data? I did not see any ablations.\n\nB. It may be interesting to try (or to comment) on a setup with multiple generation chains where synthetic examples are used in the prompt rather than real example to generate more synthetic examples. If the overall quality degrades even further, that will be additional evidence that synthesizing examples can be considered a noisy channel.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for thoughtful review. Below, we address your main questions.\n\n> The setup is a bit artificial: if we believe that LLM (e.g. GPT3) is a good model that is capable of synthesizing training data, then why we only use it for synthesizing but not for the classification directly? The authors use a smaller Bert-size model to perform classification. So, this work may be better positioned as distillation work rather than classification improvement.\n\n Directly applying LLMs for text classification may present several concerns:\n\na)  __Significant Computational Cost__: Directly Using LLMs for text classification demands considerably higher computational resources compared to smaller models trained on the synthetic data. This becomes especially evident in mobile scenarios, where deploying LLMs is often impractical due to inherent resource constraints.\n\nb) __Data Privacy and Security__: In contexts with stringent data security or privacy requirements, organizations may opt for their own smaller, bespoke models rather than relying on external LLM APIs, which they can't directly control or oversee.\n\nc) __Potential Performance Gap__: While LLMs are capable of generating high-quality synthetic data through prompting, their direct classification performance can sometimes lag behind that of smaller models trained on this synthetic data. As illustrated in Table1 below,\n\n**Table 1:** Performance comparisons in terms of Macro-F1 (%) between ''direct prompt'' and ''zero-shot data generation'' using GPT-3.5 turbo. For the \"zero-shot\" setting and \"real data\", we adopted the Bert model as the base for classification.\n\n|                    | AG   | IMDB | SMS Spam | Tweet Emotion | Humor | Tweet Irony |\n|--------------------|------|------|----------|---------------|-------|-------------|\n| Subjectivity level | *    | ***  | ****     | *****         | ***** | *****       |\n| Real data          | 95.3 | 87.6 | 97.2     | 77.7          | 97.0  | 72.2        |\n| Directly prompt    | 86.5 | 82.8 | 89.4     | 54.3          | 59.2  | 61.1        |\n| Zero-shot          | 89.3 | 81.2 | 93.8     | 58.1          | 56.0  | 63.4        |\n \nfor many tasks, directly prompting GPT-3.5  for classification often yields poorer results compared to a smaller model trained on the synthetic data. This discrepancy might arise because the prompt constraints defining the label space for the LLM can sometimes be too lax, making accurate classification challenging.\n\n> How sensitive the results are to the prompts asking LLM to generate training data? I did not see any ablations.\n\nTo understand the sensitivity of the prompt used for generating synthetic data in a zero-shot setting, we performed an ablation study based on GPT-3.5 turbo. We remove the context prompt and diversity prompt from our original prompt to form what we term a``bad prompt'' for the data generation. A shown in the Table 2 below, \n\n**Table 2:** Comparing the performance of Bert classification models trained on the GPT-3.5 turbo-generated  zero-shot synthetic data with different prompts in terms of Macro-F1 (\\%).\n| Dataset            | AG   | SMS Spam | IMDB | Tweet Emotion | Humor detection | Tweet Irony |\n|--------------------|------|----------|------|---------------|-----------------|-------------|\n| Subjectivity level | *    | ***      | **** | *****         | *****           | *****       |\n| real data          | 95.3 | 97.2     | 87.6 | 77.7          | 97.0            | 72.2        |\n| Original prompt    | 89.3 | 93.8     | 81.2 | 58.1          | 56              | 63.4        |\n| bad prompt         | 88.9 | 85.9     | 80.7 | 56.9          | 55.1            | 62.8        |\n\nwe noticed that this ``bad prompt'' led to a slight decline in the effectiveness of the zero-shot synthetic data. Nevertheless, the overall trend in effectiveness of synthetic data remains consistent: synthetic data appears more effective for tasks with lower subjectivity, which is in align with the results from our original prompt.\n\n> It may be interesting to try (or to comment) on a setup with multiple generation chains where synthetic examples are used in the prompt rather than real example to generate more synthetic examples. If the overall quality degrades even further, that will be additional evidence that synthesizing examples can be considered a noisy channel.\n\nThank you for your suggestion. We conducted an addition study by using the zero-shot synthetic data produced by the GPT-3.5 turbo model as the guiding example in a few-shot setting to generate data. As illustrated in Table 3 below,\n\n**Table 3:** Comparing the performance of Bert classification models trained on the zero-shot synthetic data and the few-shot synthetic data where the synthetic data is used as the guiding examples (``second-promot'') generated by the GPT-3.5 turbo in terms of Macro-F1 (\\%). \n| Dataset            | AG   | SMS Spam | IMDB | Tweet Emotion | Humor | Tweet Irony |\n|--------------------|------|----------|------|---------------|-------|-------------|\n| Subjectivity level | *    | ***      | **** | *****         | ***** | *****       |\n| real data          | 95.3 | 97.2     | 87.6 | 77.7          | 97.0  | 72.2        |\n| Zero-shot          | 89.3 | 93.8     | 81.2 | 58.1          | 56.0  | 63.4        |\n| Second-prompt      | 87.1 | 86.9     | 81.1 | 55.9          | 53.8  | 61.9        |\n\nwe found that for tasks with varying levels of subjectivity, using synthetic data as examples in the prompt for further synthetic data generation (referred to as ``second-prompt'') leads to a more  performance degradation compared to data generated in a single zero-shot round. This suggests that synthesized examples can act as a noisier channel in comparison to real examples.\n\n#### We will add the additional experimental results into the later paper or the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Timely topic. The paper is well written and empirical findings are convincing.", "reasons_to_reject": "1) The setup is a bit artificial: if we believe that LLM (e.g. GPT3) is a good model that is capable of synthesizing training data, then why we only use it for synthesizing but not for the classification directly? The authors use a smaller Bert-size model to perform classification. So, this work may be better positioned as distillation work rather than classification improvement.\n2) The findings are not surprizing: more subjective tasks or training examples results in more noise in the synthetic training data.\n", "questions_to_authors": "I only have suggestions what else to consider in future revisions rather than specific questions:\n\nA. How sensitive the results are to the prompts asking LLM to generate training data? I did not see any ablations.\n\nB. It may be interesting to try (or to comment) on a setup with multiple generation chains where synthetic examples are used in the prompt rather than real example to generate more synthetic examples. If the overall quality degrades even further, that will be additional evidence that synthesizing examples can be considered a noisy channel.", "paper_topic_and_main_contributions": "The paper demonstrates empirically that more subjective text classification tasks or individual training examples are less effective for synthetic training data generation with Large Language Models.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None found."}, "extra_text": {}}, {"review_id": "0JAckO7UqP", "reviewer": "Reviewer_uLMj", "summary": "", "questions": "Section 4.2. Why not also rank datasets on \"instance-level subjectivity\", as you defined it, using a sample from each dataset and measuring inter-annotator (dis)agreement?  Also, Table 1 subjectivity ranking are, in some cases, counter-intuitive and makes one question the effectiveness of the described crowd-sourcing approach.\n\nSection 6.2 The explanation for synthetic-data training outperforming real data on the dataset Tweet irony is not very convincing. Have you analyzed examples from the real vs the synthetic dataset?\n\nWhy not compare results to zero-shot/few-shot classification based on the same LLM, vs synthetic data + trained classification model? Does the LLM still perform better on less subjective datasets? \n\nThe more \"subjective\" (per your measure) datasets in your experiments significantly outnumber the  less \"subjective\" datasets. It is unclear that the difference in performance is, therefore, really due to the level of subjectivity or due to the data used by the LLM. Similarly, since a single LLM is used to generate synthetic data, the generalizability of the results is questionable.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful feedback. Below, we address your main questions.\n\n> Section 4.2. Why not also rank datasets on \"instance-level subjectivity\", as you defined it, using a sample from each dataset and measuring inter-annotator (dis)agreement? Also, Table 1 subjectivity ranking are, in some cases, counter-intuitive and makes one question the effectiveness of the described crowd-sourcing approach.\n\nIn Section 4.2, we explore subjectivity at the task level as different instances within the same task can exhibit varying degrees of subjectivity. That said, as illustrated in Table 2 in the original paper, the ranking of task-level subjectivity computed based on the average instance-level disagreement for each type of task closely aligns with the task-level subjectivity that we obtained from our crowdsourced study. We will include the rank based on average instance-level disagreement for each dataset in Table 1 in the original paper to facilitate the understanding. \n\n>Section 6.2 The explanation for synthetic-data training outperforming real data on the dataset Tweet irony is not very convincing. Have you analyzed examples from the real vs the synthetic dataset?\n\n We observed that the enhanced performance in the few-shot synthetic data primarily stems from a decrease in the false positive rate when classifying the non-irony tweets as irony. Upon examining real tweets in the dataset, a small subset was identified where, despite being labeled as non\\_irony, the tweets included hashtags  #irony and #sarcasm. Here are a couple of examples: 1) @user please note #sarcasm in last tweet before you kill yourself ;) many good pros including @user can show you best usage. 2) @user @user republicans suck more at identifying #irony than alanis morissette. In addition, a significant proportion of the tweets labeled as irony in the real data also contain the hashtags #irony and #sarcasm.  When models are trained using real data,  they only encounter a small number of instances that are labeled as non\\_irony but contain hashtags like \\#irony or \\#sarcasm. As a result, when tested, these models exhibit a high false positive rate for those cases. However, in few-shot scenarios, introducing such samples to the LLM effectively augments their presence in the training set by several folds. This augmentation seems to facilitate more accurate predictions for such instances in the test set. This observation aligns with Aggarwal's 2023 study [1], which demonstrated that leveraging a larger volume of synthetic data (2-10x real data) in low-resource situations can improve the performance on real data.\n\n[1] Aggarwal, K., Jin, H. and Ahmad, A., 2023. ECG-QALM: Entity-controlled synthetic text generation using contextual Q&A for NER.\n\n> Why not compare results to zero-shot/few-shot classification based on the same LLM, vs synthetic data + trained classification model? Does the LLM still perform better on less subjective datasets?\n\nDirectly applying LLMs for text classification may present several concerns:\n\na) __Significant Computational Cost__: Directly Using LLMs for text classification demands considerably higher computational resources compared to smaller models trained on the synthetic data. This becomes especially evident in mobile scenarios, where deploying LLMs is often impractical due to inherent resource constraints.\n\nb) __Data Privacy and Security__: In contexts with stringent data security or privacy requirements, organizations may opt for their own smaller, bespoke models rather than relying on external LLM APIs, which they can't directly control or oversee.\n\nc)  __Potential Performance Gap__: While LLMs are capable of generating high-quality synthetic data through prompting, their direct classification performance can sometimes lag behind that of smaller models trained on this synthetic data. As illustrated in Table 1 below,\n\n**Table 1:** Performance comparisons in terms of Macro-F1 (%) between ''direct prompt'' and ''zero-shot data generation'' using GPT-3.5 turbo. For the \"zero-shot\" setting and \"real data\", we adopted the Bert model as the base for classification.\n\n|                    | AG   | IMDB | SMS Spam | Tweet Emotion | Humor | Tweet Irony |\n|--------------------|------|------|----------|---------------|-------|-------------|\n| Subjectivity level | *    | ***  | ****     | *****         | ***** | *****       |\n| Real data          | 95.3 | 87.6 | 97.2     | 77.7          | 97.0  | 72.2        |\n| Directly prompt    | 86.5 | 82.8 | 89.4     | 54.3          | 59.2  | 61.1        |\n| Zero-shot          | 89.3 | 81.2 | 93.8     | 58.1          | 56.0  | 63.4        |\n\nfor many tasks, directly prompting GPT-3.5 turbo for classification often yields poorer results compared to a smaller model trained on the synthetic data. This discrepancy might arise because the prompt constraints defining the label space for the LLM can sometimes be too lax, making accurate classification challenging.\n\nFurthermore, we have observed that using large language models (LLMs) directly for classification also yields the smaller performance gap between the performance on models trained on real data on tasks with lower subjectivity.\n\n>The more \"subjective\" (per your measure) datasets in your experiments significantly outnumber the less \"subjective\" datasets. It is unclear that the difference in performance is, therefore, really due to the level of subjectivity or due to the data used by the LLM. Similarly, since a single LLM is used to generate synthetic data, the generalizability of the results is questionable.\n\nThanks for your concern.  \n\nA) To validate our observations regarding ``subjectivity'' in the data, we conducted additional experiments on a few more datasets which represent less subjective text classification tasks: the BBC News dataset, SST-2 movie review, Amazon Us review , Yelp review, and ChatGPT App review. As depicted in Table 2 below,\n\n**Table 2:** Comparing the performance of classification models trained on the LLM-generated synthetic data under the zero-shot  with those trained with the original real-world data, in terms of Macro-F1 (\\%).\n\n| Dataset   | BBC  | Amazon | SST-2 | Yelp | ChatGPT app review |\n|-----------|------|--------|-------|------|--------------------|\n| Real data | 93.6 | 91.8   | 89.2  | 94.3 | 79.4               |\n| Zero-shot | 91.2 | 87.7   | 86.4  | 87.9 | 73.3               |\n \n\nthe average performance difference between real-world data and synthetic data is only 4.2%. This gap is notably smaller than what is observed in tasks with greater subjectivity, reinforcing the finding that the subjectivity of a task can indeed diminish the effectiveness of synthetic data. \n\n\nB) To examine whether our findings hold true for decoder-based models as well as models that are reasonably large,  we conducted the same evaluation studies using the GPT2-large (774M) and Llama2 (7B) models. \nDue to the time constraints, we conducted this evaluation on 6 randomly selected  datasets from the entire set of 10 datasets that we considered in our paper, and the selected 6 datasets covered different levels of subjectivity. As indicated in Table 3 below,\n\n**Table 3:** Comparing the performance of Bert classification models trained on synthetic data generated by various LLMs within a zero-shot setting using Macro-F1 (\\%) as the metric.\n|                    |  AG  | IMDB |  SMS | Tweet Emotion | Humor Speech | Tweet Irony |\n|:------------------:|:----:|:----:|:----:|:-------------:|:------------:|:-----------:|\n| Subjectivity level |   *  |  *** | **** |     *****     |     *****    |    *****    |\n|      Real Data     | 95.3 | 87.6 | 97.2 |      77.7     |     97.0     |     72.2    |\n|    GPT2-large (774M)    | 86.5 | 80.9 | 86.4 |      52.2     |     51.5     |     60.8    |\n|    Llama 2 (7B)    | 88.9 | 82.4 | 88.5 |      59.1     |     57.2     |     63.1    |\n|    GPT-3.5 turbo   | 89.3 | 81.2 | 93.8 |      58.5     |      56.0      |     63.4    |\n\nwe observed that models trained on the LLM-generated synthetic data only exhibits slight variations among different LLMs for each respective task.  The overall trend as seen in our paper remains consistent: the effectiveness of synthetic data tends to be higher for tasks with lower subjectivity.\n\n#### We will add the additional experimental results into the later paper or the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A very well written and thorough paper.\n\nResults are reproducible.", "reasons_to_reject": "Both the novelty, the conducted experiments, and meaningful insights are limited.  The paper could be of interest to a small group of researchers and is more suitable for a short paper.\n\nThe measurement of datasets by subjectivity (obtained via crowdsourcing) could be questioned. Similarly, the drawn conclusion (LLM synthetic data training on subjective datasets performs worse than on less subjective datasets) could be questioned because of 1) very few \"non\" subjective datasets used in experiments and 2) a single LLM model.", "questions_to_authors": "Section 4.2. Why not also rank datasets on \"instance-level subjectivity\", as you defined it, using a sample from each dataset and measuring inter-annotator (dis)agreement?  Also, Table 1 subjectivity ranking are, in some cases, counter-intuitive and makes one question the effectiveness of the described crowd-sourcing approach.\n\nSection 6.2 The explanation for synthetic-data training outperforming real data on the dataset Tweet irony is not very convincing. Have you analyzed examples from the real vs the synthetic dataset?\n\nWhy not compare results to zero-shot/few-shot classification based on the same LLM, vs synthetic data + trained classification model? Does the LLM still perform better on less subjective datasets? \n\nThe more \"subjective\" (per your measure) datasets in your experiments significantly outnumber the  less \"subjective\" datasets. It is unclear that the difference in performance is, therefore, really due to the level of subjectivity or due to the data used by the LLM. Similarly, since a single LLM is used to generate synthetic data, the generalizability of the results is questionable.", "paper_topic_and_main_contributions": "This longer paper describes the use LLMs to generate synthetic data for text classification. Experiments show that LLM-generated synthetic data is very close to those trained on the real-world data for tasks with low subjectivity while the performance decrease is much bigger on tasks with high subjectivity (e.g., humor or sarcasm detection).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **quality, clarity, originality**

The paper is well written, as also stated by the reviewers. It is comprehensive and high quality. The authors set out the problem well and work to provide different views on the synthetic data generation and classification problems. The inclusion of a wide array of NLP classification tasks works in the favour of this paper. There were a number of reviewers who raised the challenge about the "subjective" vs "less subjective" datasets (the latter having less examples in the paper). The authors have addressed this by adding more experiments with more datasets that would be classified (by their measure) as "less subjective". This is welcome as it allowed better evaluation and understanding of the overall contribution of this paper. The final result might be something that many researchers had an intuition about, but this work provides an empirical and scientific study that confirms it and will form the basis (as I see) of how others also build on evaluating such approaches to data generation for their use cases. 

** significance**

As stated earlier, the work gives very strong evidence about zero shot vs few shot performance for nlp classification data generation. It is a significant result that will be helpful in this LLM era. There was a concern about the LLMs covered for the experiments, but these have also been expanded and this adds to the pros of this paper. 

**Notes to authors**

Will the generated data be made available for further evaluation by other groups?
Decision: Accept-Main