Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias
Authors: Yue Yu
Yuchen Zhuang
Jieyu Zhang
Yu Meng
Alexander Ratner
Ranjay Krishna
Jiaming Shen
Chao Zhang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: training data generation
large language model
text classification
TL;DR: We investigate the use of attributed prompts to generate training data from large language model for NLP classification tasks.

Abstract: Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. While previous research has explored different approaches to training models using generated data, they generally rely on simple class-conditional prompts, which may limit the diversity of the generated data and inherit systematic biases of LLM. Thus, we investigate training data generation with diversely attributed prompts (e.g., specifying attributes like length and style), which have the potential to yield diverse and attributed generated data. Our investigation focuses on datasets with high cardinality and diverse domains, wherein we demonstrate that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance. Additionally, we present a comprehensive empirical study on data generation encompassing vital aspects like bias, diversity, and efficiency, and highlight three key observations: firstly, synthetic datasets generated by simple prompts exhibit significant biases, such as regional bias; secondly, attribute diversity plays a pivotal role in enhancing model performance; lastly, attributed prompts achieve the performance of simple class-conditional prompts while utilizing only 5\% of the querying cost of ChatGPT associated with the latter. The data and code are available on {\url{https://github.com/yueyu1030/AttrPrompt}}.

Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias
Yue Yu'; Yuchen Zhuang's Jieyu Zhang"; Yu Meng", Alexander Ratner", Ranjay Krishna", Jiaming Shen², Chao Zhang' Georgia Tech 2 University of Washington 3 UICC 4 GGoogl Research fyueyu, yezhuang, chaozhang)@gatech. edu, yumeng5eillinois.. edu (fieyuz2, ajratner, ranjay)8cs wwsssinttsnnnnn edu, jushen@google. com
Abstract
Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. While previous research has explored different approaches to training models using generated data, they generally rely on simple class-conditional prompts, which may limit the diversity of the generated data and inherit systematic biases of LLM. Thus, investigate training data generation with diversely attributed prompts (e.g-, specifying attributes like length and style), which have the potential to yield diverse and attributed generated data. Our investigation focuses on datasets with high cardinality and diverse domains, wherein we demonstrate that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance. Additionally, we present a comprehensive empirical study on data generation encompassing vital aspects like bias, diversity, and efficiency, and highlight three key observations: firstly, synthetic datasets generated by simple prompts exhibit significant biases, such as regional bias; secondly, attribute diversity plays pivotal role in enhancing model performance; lastly, attributed prompts achieve the performance of simple class-conditional prompts while utilizing only 5% of the querying cost of ChatGPT associated with the latter"
Introduction
Large language models (LLMs) have demonstrated exceptional performance across broad range of NLP tasks [3, 39, 28, 37, 38, 58]. In recent research, LLMs have been proposed as training data generators, particularly for text classification, aiming to alleviate the need for task-specific data and annotations [54, 15, 55, 31, 57, 6]. While these efforts have showcased the effectiveness of LLMs as data generators, the focus has primarily been advancing the training stage. where the generated data are utilized to train task-specific models, leaving the upstream data generation process relatively unexplored. Notably, the prevailing approach employs simple class- conditionn prompt for querying LLMs during data generation, potentially limiting the diversity of the generated data [6, 51] and inheriting systematic biases inherent in LLMss [13, 60, 24). We refer to this simple conditonntii prompt as SimPrompt, providing an example in Table
In this work, we ground the LLM to ChatGPT [38] for its ability to generate high-quality, human-like text, and consider four challenging topic classification tasks with high cardinality from various domains. Our investigation primarily revolves around assessing the bias and diversity present within the generated training set through the lens of data attributes. In particular, data attributes encompass multiple attribute dimensions and their corresponding attribute values, where the latter represent
'These authors contributed equally to this work.
2The data and code are available on https: /ttps://github.. httpaa/.......oyyyyooo////ttttta
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Suppose you news writer.
generate affordable care act news in following requirenents below:
Mhich sttribute dimensions consider vital in deternininng the toplc of news article?
1. Should focus on rale of state governments;
subtopics, longth, location, rsader group. style, Aaae.
2. Should be length betreen 30 and 80 words;
3. The writing style of the news should be news analysis:
location
Oceania.
AttrPrompt
affordable
subtopiick....
lengths...
Public Opinson,
Enroll Numbers,...
Generated Text
New Zealand's governments implerent Affordable Act, towards
Dainion
100
Organizations. success of Prinary nodel proven effective in providing corprehensive care and reducing casta for putients. llowever, challenges rewii with coordination and equity axong different regions. With ongating
Role of state governments
30-80
states will play cruesal rule shaping the future of healtheare
Inglifications
Generation
Figure 1:
The overall workflow of AttrPrompt.
Table I: Prompt template for the NYT news dataset.
Method
Prompt
SimPrompt
Suppose you
writer. Please generate (topic-clasa)
NYT.
writer. Pleuse generate (topic-claza) NYT following the requirements below: 1. Should focus on (aubtopic):
Suppose you
AttrPrompt
2. Should in length between flength: flength:min-wwot min-words> } and (length: max-xordss words;
3. The writing style of the news should be (style);
4. The location of the news should be in (location).
possible instantiations of the former. For example, an attribute value such as "shorter than 200 words" could serve as an instantiation of the attribute dimension "length".
On one hand, we employ trained attribute classifier to examine the attribute bias present in the dataset generated using SimPrompt. When analyzing the "location" attribute in the NYT news dataset, we observe a striking bias towards "North America" the predicted values of the generated data, accounting for significant majority (68.01 %). In contrast, instances associated with "Africa" are remarkably rare, comprising only 0.69% the dataset (100 times less prevalent than "North America"). This regional bias exhibited in the generated dataset can pose substantial challenges when constructing reliable machine learning models [27, 4].
On the other hand, explore the influence of attribute diversity the downstream model per- formance. Specifically, we leverage ChatGPT to generate attributed data by incorporating desired attributes constraints in the prompts. By comparing the performance of models trained on datasets generated using prompts with random attributes against those with fixed attributes, we observe a substantial underperformance of the latter, uncovering the importance of attribute diversity of the generated dataset. The workflow of AttrPrompt is shown in Figure 1.
To alleviate attribute biases and enhance the attribute diversity of the generated data, we propose to generate data with diversely attributed prompts. For a given classification task, we start by identifying attribute dimensions and their corresponding attribute values in an interactive, semi-automated process facilitated by the LLM. Subsequently, we generate diverse prompts by combining attributes randomly, replacing the simple class-conditional prompt typically used for querying data from the LLM. We refer to these diversely attributed prompts as AutrPrompt. An example of such prompts can be found in Table where the LLM is instructed to generate training data based attributes such as location and style.
On the four classification tasks, we empirically evaluate the generated datasets by measuring the performance of models trained using two scenarios: 1) solely on the generated dataset, and 2) on a merged dataset comprising the real training set and the generated set. In both scenarios, the dataset generated with AttrPrompt significantly outperforms counterpart generated with SimPrompt. Furthermore, demonstrate the superiority of AttrPrompt over SimPrompt in terms of data/budget

---

efficiency and compatibility with different model sizes/various ALM-as-training data generator approaches. Notably, AttrPrompt achieves the performance of SimPrompt while utilizing only 5% of the querying cost of ChatGPT associated with SimPrompt. Lastly, we extend the L.MM-as-traiiing data-generator paradigm to the more challenging multi-label classification tasks for the first time, and AttrPrompt outperforms SimPrompt across all evaluation metrics.
2
Related Work
LLMs Training Data Generators. With the remarkable success of large language models (LLMs), researchers have recently attempted to leverage them as training data generators. Such applications include generating tabular data [2], medical dialogue [7], sentence pairs [47], instruction data [40, 50, 53, 48], etc.. Among these applications, we anchor on training data generation for topic classification in a zero-shot setting where no labeled data is available. In this direction, existing approaches typically use simple class-conditional prompts while focusing on mitigating low-quality issues after generation. In particular. SuperGen [31] and ZeroGen [54] took initial steps explore using LLM as a training data generator for text classification with simple class-conditional prompts, and leveraged additional noise robust learning techniques [26, 36, 52] to deal with the low-quality issue of the generated data. SunGen [15] reweights the generated data during training with learned data quality weight, and ProGen [55] leverages the model feedback to select highly influential generated data which then serve as labeled examples for generation. In this work, we instead explore attributed prompts to reduce the issue of low informativeness and redundancy, which can be readily incorporated into the existing systems mentioned above. Notably, Chen et al. [6] also explores prompts to advance the data generation process, yet it adopts soft prompts and requires a white-box LLM and seed examples to tune them. In contrast, our method is applicable to black-box LLMs and even LLM APIS (e.g.. ChatGPT) and does not rely on any labeled examples. Similarly, WANLI [29] also considers human-AI collaboration for creating more challenging training data, but requires an initial dataset and a strong task model. Instead, we aim to generate training data without any initial dataset pre-existing task model, which allows us effectively handle -imoiee scenarios.
Discrete Prompt Optimization. Several works attempt to optimize discrete prompts for querying LLMs, such as prompt paraphrasing [22]. prompt editing [42], or prompt searching [11]. Recently, [45, 59, 35] use large language models to optimize prompts. More related to us, [34] reframe prompts by decomposing complex task instruction into multiple simple ones. However, these approaches mainly focus on the inference stage for directly predicting the answer and may rely on additional labeled examples for validation. Our focus is on an orthogonal setting, optimizing prompts for LLMss with attributes to diversify the generated training data. This approach improves the model's 's overall performance without the need for additional labeled examples.
Large Language Model as Attributed Training Data Generator
3.1 Datasets
While previous research has primarily focused on binary classification datasets [54, 31, 55] or datasets
containing maximum of 14 classes [15, 57], the performance of LLM as a data generator for topic classification with high cardinality (i.e., many topic classes) remains unclear. Thus, we consider the following datasets from various domains with the number of topics ranging from 23 to 50:
NYT [32]: The NYT dataset comprises news articles that were authored and published by The New York Times. These articles are categorized into 26 fine-grat grained categories.
Amazon [I]: The Amazon dataset contains customer reviews on products from Amazon's 's online
store. It covers products from 23 different categories.
Reddit [17]: The Reddit dataset consists of a vast collection of user-generated content from the popular social media platform Reddit. It encompasses a wide range of topics, discussions, and interactions among users across numerous communities.
StackExchange [17]: The StackExchange dataset is rich collection of structured data encom- passing various online communities and noowledgessharing platforms. It contains a vast array of questions, answers, comments, tags, and user interactions about specific technical problems.
3

---

Table 2: Statistics of datasets.
Dataset
Domain
Task
-Train
Valldd
# Test
Class
Imbalance Ratio
Amazon [1]
Reviews
Multi-class
15.0K
0.2K
1.2K
23
155.6
NYT [32]
News
Multi-class
0.2K
1.2K
357.0
Reddit [17]
Web
Multi class
26.6K
0.2K
2.3K
45
447.4
StackFxchange [17]
Web
Multi-class
27.0K
0.3K
2.5K
50
1283.7
Table Attribute dimensions and values. Attributes with an asterisk* are class- dleendee attributes.
Dataset
f configurations class
Attribute dimenstoo
Attribute value
Subtupic
Appendix I.I.I
Location
Asia North America, South America, Africa, Oceania, Eurupe
NYT
Writing Style
Investggative journalism, Op-Edss Feature writing, News andlysis Profiles and interviows
Length
sheet (30-80 wordyy long (100-150 woeds)
Product Brands
Appendix H.2.1
Preduct Names*
Appendis H.2.2
Amazon
1000
Usage Experience
Wons, Bad, Average, Good, Esceilent
Writing Style
Detailed Review; Comparative Review; Pros and Cous Review; Recommendiation Review
Length
shoot (30-00 words); long (100-150 woods)
Resources*
Appendix II3.1
Reddit
Experience*
Appendix
Wriling Style
Informativeeaccctivnenl FnterainingFuuny; Discuvsion: Storywlling: HelgvAdlvice (30-80 words);; long (100-150 woeds)
Length
Secnasio
Appendix H.A.1
StackFschunge
400
Technical Depth
Beginner; Intermediite;; Advanced: Expert
Writing Style
Specific; Comparative Problan-Solutiou: Trubleeshoting: Turorial
Lengrh
short (30-80 wordaj, long (100-15) 150 woeds)
We summarize the statistics of used dataset in Table 2, from which we can see that the involved datasets not only have high cardinality but also come with high imbalance ratio, iee, the ratio of the sample size of the majority class to that of the minority class, which reflects the long-tail class issue in real applications.
3.2 Attributes
Our initial step involves identifying various types of data attributes (or metadata) that can be manip- ulated to generate attributed data samples. To facilitate this process, we employ ChatGPT to help establish both attribute dimensions and attribute values. Specifically, we begin by engaging ChatGPT in generating essential attribute dimensions. This is achieved by posing questions such as "Which attribute dimensions do you consider vital in determining the topic of a news article?" for the NYT dataset, resulting in responses like "subtopics, length, location, reader group, style, time". Then, manually select the attribute dimensions of the highest quality that best suit the dataset. Similarly, we prompt ChatGPT (the prompt format is listed in Appendix G) to suggest potential attribute values within each attribute dimension and choose high-quality candidates.
Attribute dimensions and values. There are two types of attribute dimensions: class- ciass-indeeeneet attributes and class- dlussdeeendden attributes. Class-independent attributes, such as "length". remain unchanged across different classes, while class-dependent attributes, like "subtopic", have varying attribute values for each class. We list attribute dimensions and values for all datasets in Table 3. These data attributes provide a human-manin manaan--mmmiulltablle interface for generating attributed data. this study, we explore the potential of leveraging attributes to enhance the data generation process, while leaving the search for the optimal data attributes for a specific task to future work.
Class-dependent attribute value filtering. When dealing with class- deiss-depnndent attributes, it is crucial to ensure that their attribute values are specifically associated with the corresponding class to avoid ambiguity and potential connections to multiple classes. For example. in the case of the "economy" class in the NYT dataset, a candidate attribute value generated by ChatGPT for the "subtopic" could be "effect of trade tariffs on manufacturing companies", which is also relevant the "international business" class in the NYT. This overlap may introduce ambiguity in the generated data. To address this issue, we employ a filtering process called Class-Dependent Attribute Value Filtering (CAF). First, we query ChatGPT for the top-5 similar classes and then check with ChatGPT whether each class-dependent attribute value related to these top-5 similar classes. Then, if the answer positive which indicates a potential ambiguity, we remove that attribute value for the specific class.
4

---

3.3 Data generation and model training
Given the data attributes, one could prompt LLMs to generate data samples with diverse attribute configurations. For example, an attribute configuration for the "federal budget" class of the NYT dataset could be ("subtopic"""deffns"" spending"" "rnyyle"""ivestiguuu" journalism", "location"= Northh America"). In Table 3, we list the number of configurations per class, and one can further expand the number of configurations by adding more attribute dimensions and values. To generate attributed data samples, we prompt ChatGPT with random configurations. In particular, each time we generate random configuration, complete a prompt template (see Table 1) with the generated configuration, and query ChatGPT with the completed prompt to collect generated data samples.
max-words=80",
Throughout the experiments, we compare our method (AttrPrompt) against simple caasscoonditinal prompt (SimPrompt, [54]) and the original training set of each dataset (Gold). For fair comparison, we set the number of generated data the same as Gold for both AttrPrompt and SimPrompt. In principle, the generated dataset can be combined with any classifier (Sec. 5.4) and training techniques (Sec. 5.5); if not otherwise specified, we choose to fine-tune BERT- base--bas-uncassee uncased [12] as the backbone and use the standard cross-entropy loss by default. For hyperparameter selection, adhere to the recommendation in [41, 31] for SimPrompt and AttrPrompt, and do not use the validation set for model selection. Detailed hyperparameter configurations can be found in Appendix C.
4 Analysis of the Generated Data
4.1 A glimpse of the generated data
We begin by presenting examples of data generated by AutPPrompt and SimPrompt, and real data from the Gold set for the "federal budgef" class in the NYT dataset (Table 4). It is evident that the data generated by ChatGPT exhibit high quality. Particularly, when comparing AttrPrompt to SimPrompt, we observe that AttrPrompt renders more diverse samples. This is because SimPrompt tends to generate news focused on the U.S., while AttrPrompt has the capability to generate news from various locations around the world.
Table 4: Data examples of different datasets: the "federal budget" class of the NYT dataset.
Method
Example
1. The einergency musserr who was seat to revens fhr focanees this financiall ucabled city aked sume of is eraliors or Frilay to sceept pestics us the dellee so be isid out his plun for tackling Denoi's is enggerng detr. tick strting regetiatoon that could devertire whether the city headed lunkripryy coun. could resch billio 2017 it Cooures does not ast, Fostal Service on Wednesday cilled lommakes to give it the Beuibilly to change
Suyiss that seop
geserdl employees and retuees, modily Congrssiinaa mendai tha requires lee agcncy pay..
Govemmncual Affais Commitee. Punick Dosabes,
Congress
Fostal Service pemissin
1. Wiskington D.C. The United States govemnsent has soceatly passel is foderal budge for the acat focal seus setling soide a whopping 4.5 tnilion for speraling
This the largad budgu is the county's hiamy and reflors the peverments i commiment imposing the cneryy' i common and intrerrccioal dewokomem fodend goverment or Monday, culling
WASHINGTON D,C. Trung adnincttiion releard its proposed bodast
to progrum that provide spending bringing
SimPromp
while incnsding miltory spesinng
proprrual hudget
ponamn
57c billid..
Waalinpten D.C. The U.S. fodkral pocemmena las procettal iss badget for the focal yea 2012 which inclades a peal 6 inilionn in spending and intisaiios simed
naking the country more competinee and anatinbble The budgei plan inchulos significat instaiments in inframmatus, heallhaam, olucation and sencoaale energy.
Gemmnny public opirius on fideral hadgei beurs has boz inflrrecd by The Fodural Agrny Civic Education Tiing in depth eoryadling this syroc communicats hudger policies cltznss enjtussiingg their impect on society. etfont huve grester engagemeent awareeess of budget issues amonng Op yaass
AttePromp
lac mie the Bancn de Mesiao
Spending Pricetes Mesico As Mesican tederal budget drows nea: all finacces Mury exprr that the conral bank should play mine sairve mole hulancing
the
prinities.
budgei and simulaing covnonii powch..
'OKYO The recere federal budget pmoposal has iriggeed wave of apency budgei cuts and resmactorigg in Iopas, with the Japan Iroodation for Regionsl Develogment being onz the look camtalies. The foundation, which aims to promote resional develoyment and reciulization. sei to have budpet stashed by 20 sou year
Then, we show the vocabulary size of the generated dataset and the Gold dataset, which is a natural way to check the lexical diversity of datasets (Table 5). From the table, we can see that AttrPrompt has higher lexical diversity than SimPrompt in terms of both vocabulary size of the whole dataset (All in the table) and the averaged vocabulary size across classes (Class Avg. in the table). Yet, both have much smaller vocabulary sizes than the Gold, indicating there is still room for improvement of the ChatGPT's 's lexical diversity.
Lastly, we visualize the diversity of datasets via the distribution of cosine similarity of same-class text pairs (Figure 2), where the cosine similarity is calculated based on the embedding of Sentence- BERT [44], as well as including two additional metrics, namely average pairwise sample similarity (APS) and inter-sample N-gram Frequency (INGF) [33], as shown in table 6. We can see that the Gold dataset has the lowest cosine similarity, indicating that real data has the largest diversity. In contrast, the similarity between samples generated by SimPrompt is high Compared to SimPrompt,
5

---

Table 5: Comparison of the vocabulary size of different datasets.
NYT
Amazon
Reddit
Stack Exchange
Method
All
Class Avg.
All
Class Avg.
All
Class Avg.
All
Class Avg.
Gold
70.8k
11.3k
44.7k
6.64k
50.8k
4.62k
52.3k
3.60k
SimPrompt
20.6k
3.13k
11.6k
2.50k
19.9k
3.06k
13.3k
2.20k
AttePrompt
21.4k
3.50k
14.0k
2.76k
25.4k
3.64k
17.8k
2.93k
Table 6: Comparison of two quantitative metrics on diversity: the average pairwise sample similarity (APS) and inter-sample N-gram frequency (INGF) of different datasets. For APS, the lower stands for better diversity. For INGF, the higher stands for better diversity.
NYT
Amazzon
Method
Inter-Class Class APS
Intra Class APS
APS
INGF
Inter- Clar-CCaas APS
Intra Class APS
APS
INGF
Gold
0.098
0,358
0.122
7618.1
0.101
0.251
0.114
4992.1
SimPrompt
0.101
0.568
0.135
5277.2
0.207
0.620
0.241
2266.5
AttiPrompt
0.159
0.182
6688.6
0.225
0.483
0.246
2605.5
Reddit
StackExchange
Inter-Classy
Intra-Class
APS
INGF
mltes-Class
Intra-Class APS
INGF
Gold
0.044
0.261
0.054
9079.6
0.056
0.196
0.063
5492.4
SimPrompt
0.173
0.818
0.201
2697.8
0,282
0.804
0.302
2259.8 8
AtrPrropp
0.106
0.474
0.122
3994.5
0.105
0.375
0.114
2464.3
Estmmeed Densry
Gold Mttrtrompt
Extimatte Density 1o
x. t2
Extimreedd Density
4-
4-
8.25 0. 0.25 0.5 0.75 1.0
0. 0.0 0.25 0.5 0.75
1.0
8.25 0.0
025 05 0.75 1.0
8.25 0.0
0.25 05 0.75 1.0
Cosine similartt
Cosine Similarity
Cosine Similntyy
Cosine Simiarity
(a) NYT
(b) Amazon
Reddit
(d) StackFxchange
Figure 2: The distribution of cosine similarity of text pairs sampled from the same class.
dataset generated with AttrPrompt exhibits lower cosine similarity and the distribution is close to that of the Gold, which shows AttrPrompt could render more diverse data. Apart from the above automatic evaluation processes, also conduct human study in Appendix E to manually evaluate the quality of the generated training data.
4.2 Is there attribute bias real and generated data?
We study the attribute bias in both real dataset (Gold) and generated dataset of SimPrompt using dataset generated by AurPPommpt as probe. In particular, we leverage the attributes associated with each data of AttrPrompt to train attribute classifier, which is in turn used to make attribute predictions on Gold and SimPrompt dataset. Note that the attribute values associated with each not necessary the ground truth, yet since ChatGPT has shown remarkable
data of AttrPrompt performance in following instructions [39], the generated data could decently reflect the desired attributes and therefore the attribute classifier trained with them could partially reveal the underlying attribute distribution of tested dataset, i.e., Gold and SimPrompt.
We pick the "location" attribute of the NYT data and visualize the distributions of the predicted "location" of Gold and SimPrompt in pie charts (Figure 3). One can see that both the Gold and SimPrompt dataset are largely biased towards "North America" in terms of the whole dataset (subfigures (e) in Figure 3). As to the "location" distribution for specific classes, we can see that Gold and SimPrompt are biased towards continents other than "North America". In contrast, with attributed prompts, the generated dataset AttrPrompt comes with relatively balanced attribute distributions (Appendix F.2). While existing works using LLMs data generators usually overlook the bias embedded in the generated data, we hope that this preliminary analysis could raise the attention of the community to the attribute bias behind the generated data of LLMs such ChatGPT.
4.3 How important the attribute diversity is?
We investigate the impact of attribute diversity within AttrPrompt on model performance. Specifically, we conduct experiments by fixing one attribute dimension candidate value while keeping other

---

(a) All
(b) Tennis
(c) Soccer
(d) International Business
SimPrompt
Europe
Asia
Africa
(h) Inlintttmatinall
North America
South America
Oceania
Gold
Figure Pie charts of the distributions of "location" predicted by an attribute classifier for the NYT SimPrompt and Gold dataset. (a) and (c) are "location" distribution over the whole dataset, while others are for specific classes. Illustration on types of biases on other datasets can be found in Appendix F.I.
65
Random Configuration
Scores
60
Composing individually
Best Configurattin
Product Name
Style
Length
Experience
45
Brand
Product Name
Style
Experinne
(a) Amazon
Random Configuration Individually
Best Configuration
Location
70
Styli
Length
65
Subtopics
Locatonn
Style
Length
(b) NYT
Figure 4: Bar charts of model performance with different attribute configurations of AttrPrompt.
attribute values random. Then, we generate 50 data class using such one-fixed-orhhes--rnddom configuration to compose a dataset and evaluate the performance of the trained model. Note that for class-dependent attributes, we sample one value for each class and repeat it times, since is computationally prohibitive to enumerate all combinations of class-dependens attribute values, In Figure 4, each bar stands for specific one- www---iedd-oohs---eeeeee random configuration; compared random configurations, most of one- nre-fixed-others-raad -raddom configurations result in a performance drop.
To further reduce the attribute diversity, we pick the attribute value with the best performance for each attribute dimension (the highest bar within each attribute dimension) and compose them to single configuration (the dashed blue line). We can that the dashed blue line is significantly worse than the random configuration, even though composed individually best attribute values. This illustrates the importance and necessity of designing prompts with diverse attributes.
Experiments on the Trained Models
5.1 Training with generated data
We quantitatively evaluate the quality of generated datasets via the test performance of models trained with them. Apart from AttrPrompt and the direct baseline SimPrompt, we include an additional baseline MetaPrompt [45] which leverage LLM generate additional guidance information for improving upon SimPrompt. The details MetaPrompt are shown in Appendix We also directly ChatGPT zero-shot predictor for comparison. The results are in Table 7. Besides the test

---

performance, we include the cost of querying the ChatGPT per 1000 data in the table. From the results, we can draw the following conclusions. First, the AttrPrompt consistently renders better performance compared to the SimPrompt with margin of 6-10 points'. Second, the class-dependent attribute value filter (CAF) is beneficial since the AttrPrompt outperforms its variant without CAF Third, although MetaPrompt [45] attempts to optimize the class-dependent prompt, they are less effective than AttrPrompt as they do not resolve the unique bottleneck (i.e. limited diversity) of the current prompting strategies for the training data generation task. Fourth, out of the four datasets, the AturPrompt outperforms the LLM zero-shot method on three datasets in terms of accuracy, while for the Fl score, the AttrPrompt surpasses the LLM zero-shot on all the datasets; combined with the observation that the LLM zero-shot inference incurs much higher costs compared to data generation and the fact that the generated data is re-usable for training any model, argue that for topic text classification generating training data could be a better practice of leveraging L.L.M than direct zero-shot inference. Lastly, in most cases, the generated data underperform the original training set, indicating that there is still room for future improvement. We conduct further studies in Appendix D.5 to illustrate the performance over different classes.
Table 7: Performance of the models trained with created datasets and the cost of constructing the datasets. The results are averaged over five runs. The gain of AttrPrompt has passed the statistical test with p < 0.05. We also include the performance and cost of using LLM as a zero-shot predictor.
NYT
Amazon
Reddit
StackExchange
Method
Acc.
F1
Price/Ik
Acc.
F1
Pricollk
Acc.
F1
Price/1k
Acc
F1
Price/Ik
LLM Zero-Shot
74.16
69.84
5.44
59.55
54.56
2.11
67.00
56.66
2.89
44.70
43.80
3.12
Gold SimPrompt
83.80
81.02
82.23
83.38
67.56
63.28
75.47
76.22
0.76
57.34
56.96
0.77
53.48
53.81
0.65
42,88
41.30
0.69
MetaPrompt
79.58
79.83
0.87
56.35
55.98
0.84
54.61
54.30
0.74
44.81
44.02
0.83
AttrPrompt wlo CAF
80.40
80.92
0.91
61.67
61.57
0.82
61.22
60.18
0.72
45.90
44.84
0.81
AlrPPrmpi
81.30
82.26
1.05
66.08
65.65
0.87
63.33
63.10
0.84
48.99
47.42
0.90
5.2 Augmenting existing dataset with generated data
Here, we merge the generated dataset and the original training set into a single training set, and then test the model performance when it is trained with the merged dataset to see whether the generated dataset can further improve model performance with the original training set available. We present the results in Table 8. From the table, we can see that the generated dataset is an effective complement to the original training set, since most of the generated datasets introduce performance gain when combined with the original training set, especially our AttrPrompt which leads to improvement for all the cases. This notable improvement with simple dataset merge may motivate future studies of more advanced ways of using the generated data as augmentations to boost existing dataset.
Table 8: Performance of the model trained with the original training set/augmented with the generated dataset. We present the performance gain/drop compared to using the original training set in green/red.
NYT
Amazon
Reddit
StackExxhangg
Method
Acc.
F1
Acc.
F1
Acc.
F1
Acc
SimPrompt
85,56
+1.76
86,34
+5.32
81.85
-0.38
80,23
-0.99
85.
+0.89
84.88
+1.50
74.53
+6.97
74.23
+1095
Metaprompe
87.14
87.33
46.31
82.12
.0.11
80.14 -0.88
84.71
40.49
84.62
+1.24
76.02
48.46
75:70
+12.42
AurPrompt wio CAF
85.71
.1.91
87.18 46.16
82,24
.0.01
80,76
-0.6
85.86
1..64
85.65
12:27
75.16 17.00
74.64 :11.36
AllPrompt
+3.87
88.06
+7.04
+1.72
+2.81
+1.66
+2.60
+9:30
+13.25
5.3
The budget and sample efficiency of the generated data
Here, we aim to study two types of efficiency of the generated dataset, i.e., budget efficiency and sample efficiency, on the model performance. We use Amazon and NYT as examples, and the results for other datasets are deferred to Appendix D.3. First, in Figure 5(a) and 5(b), we compare the budget efficiency of Attr AtrrPropt against that of SimPrompt. Surprisingly, AttrPrompt only requires 5% of budget to be on par with or outperform SimPrompt with 100% of budget across all the datasets. This observation highlights the significance of diverse prompts in the training data generation process.
IIn Appendix D.4 we show that simply increasing the temperature for SimPrompt does not significantly improve its performance.
"ome of the filtered attributes exhibited in Appendix
8

---

22
Riwae
SimPrompt
Attifrompt
AttrPrompt
SimPrompt
SimPrompt
Altifrompt wi 5% Budget
AltrPrompt wi 10% Budget
AttPrrompe
AirPrompt
Proportion (90)
Proportion (5)
Proportion Data
Proportion (10)
(a) Amazon Budget
(b) NYT Budget
(c) Amazon Data
(d) NYT Data
Figure 5: The comparisons on budget efficiency and data efficiency on Amazon and NYT datasets.
Amacon
90
NYT
70
Redbit
StackExcharge
F1-Score
F1-Score
F1-5core
SimPromot
3
SimPrompt
SimPromat
SimPrompt AttrPrompt
AttrPrompt
AstrPromot
AttrPrompt
AR Bastage Cade mer
4r Tellogg cure opiie wrr
Tastage Ber o0r
(a) Amazon
(b) NYT
(c) Reddit
(d) StackExchange
Figure 6: The barplot of performance with models of different parameter sizes. Note that due to budget limit, for GPT-4 model, the size the generated dataset is only 10% the full set thus the result is not directly comparable with other models.
Secondly, we examine the sample efficiency of Gold, SimPrompt, and AttrPrompt in Figure 5(c) and 5(d). While both SimPrompt and AttrPrompt exhibit better sample efficiency than Gold in the low-data regime, with superior performance when the dataset size is relatively small, Gold data shows better sample efficiency the high-data regime. Overall, AttrPrompt renders better sample efficiency than SimPrompt, which suggests that increasing the diversity of the prompts could be an effective way improve the unsatisfactory data scaling trend of using LLM as data generator [55].
5.4
The performance with respect to model parameter size
Effect of the model size for LLM Generators. To study the effect of different LLMs on Attr- Prompt, we other eeneetiion-fiettueed GPT models the generator, namely text-ada-001 [39], text-babbage-001 [39], text-curie-O0
[39]. and GPT-4 [37] (due to budget constraints, we only generate subset with 10% size of the original dataset). Under all settings, our model out- performs the direct baseline SimPrompt by great margin. Besides, the performance is generally better with larger models, they often have better mnslloition-folloiingg capabilities. In addition, an interesting finding is that for SimPrompt (but not for AttrPrompt), the average performance of using ChatGPT is worse than cext-curre-O00 This suggests that straightforward class dependent prompts might not exploit the capabilities of LLMs as effectively as our proposed approaches.
Effect of the model size for classifiers. We experiment with other model choices in addition the BERT-base-uncases [12] used throughout the paper. They are TinyBERT [23], DistillBEs [46], DeBERTa-V3-base, and DeBERTa-V3-Lan
[21] with parameter size from 14M to 435M², We visualize the results in Figure 7. Overall, AttrPrompt outperforms SimPrompt by a large margin yet underperforms the Gold across different model choices. With liightee backbone of 66M parameters, AttrPrompt can often outperform the SimPrompt trained with the model containing 435M parameters. This indicates that diversely attributed prompts could help close the performance gap between the Gold and simple class flass-ooniiiins prompts, and such an improvement is robust to model parameter size.
5.5 Plugging AttrPrompt in existing approaches
In this section, we demonstrate that AttrPrompt can be painlessly integrated with prior zero-shot training data generation techniques. Table shows the results for several recently proposed methods, which design additional techniques based on the noisy-robust loss to further reduce the effect of noisy
Additionally, we also evaluate under the setting of linear probing [25] which trains linear classifier over the frozen pretrained features. The results deferred Appendix D.I.

---

Flazor
Simitompt
Atur Ptompoppp
Gold
Airrhompp
Airrhompp
Mirrhompt
Parsmeters
Parameters
Parameters
Parameters
(a) Amazon
(b) NYT
(c) Reddit
(d) StackExchange
Figure 7: The performance curves with models of different parameter sizes.
labeled data [15, 31, 55], and leverage in-context examples for data generation [55]. Despite these approaches achieving notable performance gains on simple binary classification tasks, their gains become more marginal for fine-grained classification: the performance gain is less than 2% for all methods on two datasets. Instead, using AttrPrompt lead to consistent performance boosts (more than 5% in all cases) for those approaches, indicating that compared with label noise, data diversity is more crucial bottleneck for existing dataset generation methods.
More interestingly, AttrPrompt even benefits dataset generation approaches that do not use LLMs. To demonstrate this, we use the LLM-generated contents (subtopics for NYT and product name for Amazon) to enrich the label names used in ReGen [57], strieval-bbaed approach for training data generation. With the expanded label names, AttrPrompt largely improves (14%-26% absolute gain) the performance of ReGen fine-grained classification tasks. These results justify the advantage of AttrPrompt for serving as a generic plug-in module for existing training data generation approaches.
Table Performance comparison when AttrPrompt serves as a plug-in for existing approaches.
SuperGen
SuperGen Att:Prompt
SunGen
SunGen w/ AttrPrompt
ProGen
ProGen AttrPrompt
ReGen
ReGen
AttPPrompt
Acc.
76.11
82.05
75,82
81.65
77.05
80.93
70.01
82.18
NYT
F1
76.80
82.62
76.52
82.70
76.70
81.32
68.14
82.50
Acc.
58.17
66.76
54.30
63.89
58.40
66.43
34.70
58.40
Amazon
F1
56.06
66.33
53.50
63.76
56.95
66.02
30.93
56.00
Table 10: Multi-label classification performance on the arXiv dataset.
Method
Macro Fl
Micro F1
Precision
Precision@5
NDCG@5
MRR
Price/1k
Gold
27.34
58.22
73.71
27.00
79.70
82.16
SimPrompt
21.03
26.75
37.00
15.08
42.49
49.60
1.41
AttrPrompt
27.10
37.88
49.27
18.79
54.74
61.23
1.53
5.6 Extension multi-label classification
In this section, we take the first attempt extend the paradigm of using the LLM training data generator to the more challenging multi-label classification setting. In particular, we adopt the arXiv dataset [8] consisting of 98 fine-grained classes, which we apply both SimPrompt and AttrPrompt. Following [18, 49]. we consider different evaluation metrics including Micro/Macro-Fi Precision k nDCG@K, k, and MRR. The experimental details are in Appendix B. We present the results in Table 10. Similar to our findings for single-label classification, AtuPPrompt largely outperforms SimPrompt across all the metries, which not only strengthens the superiority of AttrPrompt but also opens the door to using LLM as a training data generator for multi-label classification for future work.
Conclusion
We delve into the realm of training data generation using complex, attributed prompts, which possess the potential to produce a wide range of diverse and attributed generated data. Specifically, we focus on datasets characterized by diverse domains and high cardinality and class-imbalance, and results demonstrate the superior performance of attributed prompts compared to simple class-conditional prompts. Furthermore, we present a comprehensive empirical study on training data generation that covers essential aspects such as bias, diversity, and efficiency.
10

---

References
[1] John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440-447, 2007.
[2] Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelezyk, and Giergji Kasneci. Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultuce Considerations in NLP (C3NLP), pages 53 33-67, 2023.
[5] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
[6] Derek Chen, Celine Lee, Yun-Yun Lu, Domenic Rosati, and Zhou Yu. Mixture of soft prompts for controllable data generation. ArXiv, abs/2303.01580,, 2023.
[7] Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. Medically aware gpt-3 data generator for medical dialogue summarization. In Machine Learning for Healthcare Conference, pages 354-372. PMLR, 2021.
[8] Colin Clement, Matthew Bierbaum, Kevin P O' Keefffe and Alexander A Alemi. On the use of arxiv dataset. arXiv preprint arXiv: 1999900000075. 2019.
[9] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsu- pervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[10] Alexandra DeLucia, Aaron Mueller, Xiang Lisa Li, and João Sedoe. Decoding methods for neural narrative generation. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 202/), l), pages 166-185, 2021.
[11] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with rein- forcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369-3391, 2022.
[12] Jacob Devlin, Ming-Wast Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4188 2019.
[13] Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. ArXiv, abs/2304.03738, 2023.
[14] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, et al. Datacomp: In search the next generation of multimodal datasets. In Neural Information Processing Systems, 2023.
[15] Jiahui Gao, Renjie Pi, Lin Yong. Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-guided noise-free data generation for efficient zero-shot learning. In International Conference Learning Representations, 2023.
11

---

[16] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-69 6910, 2021.
[17] Gregor Geigle, Nils Reimers, Andreas Rückléé and Iryna Gurevych. Tweac: transformer with extendable qa agent classifiers. arXiv preprint arXiv:2104.0708/ 2021.
[18] Chuan Guo, Ali Mousavi, Xiang Wu, Daniel N Holtmann Moltmann-Rice, Satyen Kale, Sashank Reddi, and Sanjiv Kumar. Breaking the glass ceiling for embedding- bbaaddd classifiers for large output spaces. Advances in Neural Information Processing Systems, 32, 2019.
[19] Laura Hanu and Unitary team. Detoxify, 2020. https:// http::///tttttbccoo wwt.cooo/mc......... /oammumitarraa///// detoxify.
[20] Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1689-1701, 2019.
[21] Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with diieeent-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, 2023.
[22] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 2020.
[23] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of EMNLP, pages 2020.
[24] Hannah Rose Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frédéric A. Dreyer, Aleksandar Shtedritski, and Yuki M. Asano. Bias out-of-the-box An empirical analysis of intersectional occupational biases in popular generative language models. In Neural Information Processing Systems, 2021.
[25] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang, Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022.
[26] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Interna- lional Conference on Learning Representations, 2017.
[27] Yizhi Li, Ge Zhang, Bohao Yang, Chenghua Lin, Anton Ragni, Shi Wang, and Jie Fu. HERB: Measuring hierarchical regional bias in pre-trained language models. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 334-346, 2022.
(28) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09111 2022.
[29] Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of EMNLP, pages 6826,6847, 2022.
[30] Ilya Loshchilov and Frank Hutter. Conference on Learning Representations, 2019.
Decoupled weight decay regularization. In International
[31] Yu Meng. Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. In Advances in Neural Information Processing Systems, 2022.
[32] Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. Weakly-supervised hierarchical text classification. In Proceedings of the AAAI conference on artificial intelligence, pages 6826-6833. 2019.
12

---

[33] Swaroop Mishra, Anjana Arunkumar, Bhavdeep Sachdeva, Chris Bryan, and Chitta Baral. Dqi: Measuring data quality in nlp. arXiv preprint arXiv:2005.008/6. 2020.
[34] Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannanch Hajishirzi. Refram- ing instructional prompts to GPTK's language. In Findings of the Association for Computational Linguisties: ACI. 2022, pages 589-612, 2022.
[35] Swaroop Mishra and Elnaz Nouri. HELP ME THINK: simple prompting strategy for non-experts to create customized content with models. In Findings of the Association for Computational Linguistics: ACI. 2023, pages 11834-11890, 2023.
[36] Rafael Müller, Simon Korbblith, and Geoffrey E. Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[37] OpenAl. Gpt-4 technical report. arXiv, 2023.
[38] OpenAl. Introducing chatgpt, 2023.
[39] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[40] Baolin Peng, Chumyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. ArXiv, abs/2304.03277, 2023.
[41] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In Advances in Neural Information Processing Systems, 2021.
[42] Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-freee ddit-based instruction search for prompting large language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3827-3846. 2023.
[43] Charvi Rastogi, Yunfeng Zhang, Dennis Wei, Kush R Varshney, Amit Dhurandhar, and Richard Tomsett. Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making Proceedings of the ACM Hlaman-Compuln Interaction, 6(CSCW1)) 11) 2022.
[44] Nils Reimers and Iryna Gurevych. Centence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 3982-3992, 2019.
[45] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHIl Conference on Human Factors in Computing Systems, pages 1-7, 2021.
[46] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[47] Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. In Proceedings of the 2 Conference on Empirical Methods in Natural Language Processing, pages
2021.
[48] Zhihong Shao, Yeyun Gong. Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: Generating chain-of-thouu -f-toughtt demonstrations for large language models. arXiv preprini 2022v:2302.00618 2023.
[49] Jiaming Shen, Wenda Qiu, Yu Meng, Jingbo Shang, Xiang Ren, and Jiawei Han. Taxoclass: Hierarchical lulti-label text classification using only class names. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages
2021.
13

---

[50] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principledrrvven self aligmeettment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.03047. 2023.
[51] Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326-346, 2021.
[52] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 322-330, 2019.
[53] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprint arXiv:2212.10560, 2022.
[54] Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11653- 11669, 2022.
[55] Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. ProGen: Progressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3671-3683, 2022.
[56] Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. COCO-DR: Combating distribution shifts in zero-shot dense retrieval with contrastive and distributionally robust learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1479-1479, 2022.
[57] Yue Yu, Yuchen Zhuang, Rongzhi Zhang, Yu Meng, Jiaming Shen, and Chao Zhang. ReGen: Zero-shot text classification via training data generation with progressive dense retrieval. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11782-11805. 2023.
[58] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
[59] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023.
[60] Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring ai ethics of chatgpt: A diagnostic analysis. ArXiv, abs/2301. 12867, 2023.
14

---

A Limitation, Social Impact, and Future Work
We acknowledge that AurPPromp, while being effective on several tasks we studied, is still far from being perfect. Here we give honest discussion on the limitation, societal impacts, as well as potential avenues for future works.
A.1 Limitation
One limitation of this research is that the investigation focuses primarily text classification. While this provides valuable insights into the performance of attributed prompts compared to simple class- conditional prompts in these specific contexts, the generalizability of the findings to other data types and tasks remains to be explored.
Besides, the proposed approach assumes the label name of the target task following prior works [31, 54] and requires some extent of human efforts in creating the data attributes, as it is still challenging to automatically determine the optimal number/property of attributes without any labeled data and task-specific information. Human-A AI collaboration, while beneficial, may inevitably introduce certain human biases into the model [43, 5J. We would like to clarify that in this work, the human raters are blinded to the information for task-specific data during attribute selection. To further mitigate the bias, it is crucial to involve multiple team raters, validatii,,,,, and additional review achieve a balanced and unbiased selection of attribute dimensions that contribute positively to the dataset's quality and diversity.
In addition, one issue with LLM-based training data generation the phenomenon of hallucination, wherein the model generates information that may sound plausible but is, in fact, not accurate or grounded in reality. This can lead to the propagation of misinformation, particularly in sensitive areas such news generation where accuracy is of utmost importance. To alleviate this issue, it is possible to leverage additional fact-checking mechanisms to cross-verify the generated text with a reliable knowledge base or dataset. Furthermore, incorporating additional layer of human review can also help in mitigating hallucinations and ensuring the faithfulness of generated outputs.
A.2 Societal impact
The findings of this research have significant implications for the development of trustworthy machine learning models. By addressing the limitations of simple class-conditional prompts and exploring the potential of attributed prompts, the research aims to mitigate biases and enhance the diversity of generated training data. This has a positive social impact as it contributes to the development of more robust and fair machine learning models, reducing the potential for biased decision- -akiing systems and promoting inclusivity in various domains.
During the attribute selection process, we did not use any task-specific data to avoid data feature leakage and minimize human bias. Other than this, we do not involve human subjects research and do not contain any personally identifiable information. Possible misuse may lead to negative outcomes, such as intentionally introducing biases to the generated dataset via attributes.
A.3
Future work
Broader Coverage of Task and Modalities. In future work, it would be valuable to extend the investigation to other types of classification tasks beyond text classification. Exploring the effectiveness of attributed prompts in image classification, audio classification, or other modalities could provide insights into the broader applicability and effectiveness of AttrPrompt.
Automatic Prompt Optimization. In this work, we aim to demonstrate the usefulness of attributes and why they are useful through the lens of diversity and bias, and we believe this opens the door for future work to further explore the methodology for automatic attribute discovery and selection for optimal performance, such as exploring automated or anttmaetomtted methods for identifying high-quality attribute dimensions and values. One promising direction is to incorporate few labeled examples as demonstrations for generating better training data. Under this scenario, leveraging Language Models can aid in automatically detecting diverse attributes, modeling the high- interactions among attributes as well as estimating attribute value distributions for each class, which further reduce the human efforts for selecting key attributes.
15

---

B Dataset Preprocessing
B.1 Multi-class Classification
For Reddit and StackOverflow, we select the classes with more than 65 examples from the original corpus as the target set of topics. For each dataset, we use 50 examples per class for the test set and no more than 10 examples for the validation set (10 for NYT/Amazon and for Reddit/StackOverllo The remaining data is used to compose the gold training set. It is worth noting that, some of the class names on Reddit may contain toxic information. To eliminate their effects, we filter our label names with Detoxify [19]. tool that leverages the multilingual XLM- RoMBRaa [9] for toxic comment identification. We follow [14] to use a threshold of 0.1 to filter out potentially toxic topic classes.
B.2 Multi-label Classification
For the arXiv multi-label classification, we use the arXiv labels from the website as the label names. We select topics under the category of Physics, Mathematics, Computer Science, Quantitative Biology, and Quantitative Finance. Besides, we remove the topic of General Physics, General Mathematics, General Literature, Other Computer Science, Other Quantitative Biology they only contain generic topics without concrete semantics. In total, there are 98 classes in general.
To generate examples using the SimPrompt approach, we follow specific procedure. For each document, we begin by sampling from the distribution of label numbers, utilizing prior knowledge obtained from the original training set. This enables us to determine the precise number of labels, denoted as n, that should be associated with the document. Once we have determined n, we proceed to randomly select classes from the original set of 98 labels. These chosen classes are then utilized as the input for prompting the LLM, forming a foundation for generating the examples.
To generate examples using the AttrPrompt approach, we employ distinct methodology. For each document, we consider set of attributes, namely subtopics, technique, writing style, and length. It is important to note that, unlike multi-class classification, where additional step is typically taken to filter out irrelevant subtopics, our multi-label classification process follows different approach. In our method, we introduce additional step called merging to handle the different subtopics.
To determine the merging of subtopics, we utilize the fuzzy matching score?, which calculates the similarity between the subtopics. Setting the threshold at ? 90, we proceed to merge subtopics whenever their similarity score exceeds this threshold. Consequently, specific subtopics may be associated with multiple classes, providing a more comprehensive representation of the document content. When generating text utilizing these subtopics, the corresponding labels are determined by the classes associated with the specific subtopics. This ensures that the generated examples accurately reflect the relevant classes based on the subtopic associations. The prompt format is deferred in section H.
C Implementation Details
C.1 Hardware information
All experiments are conducted on CPU: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz and GPU: NVIDIA GeForce RTX A5000 GPUs using python 3.8, Huggingface 4.6.0 and Pytorch 1.10.
C.2 Parameter configuration
We keep the parameter lop_p- 1.0 and temperature / 1.0 for calling ChatGPT APIs [38] for the training data generation part. For finetuning the classifier, we optimize the model using Adam W [30] with linear warmup of the first 5% steps and linear learning rate decay. The maximum number of tokens per sequence is 128. Table 11 lists the hyperparameters used for AttrPrompt and SimPrompt. For the generated synthetic dataset, we stick the strict zero-shot learning setting [31], train all
"nttps://arxi..r /rrg///gxxi.o..//////// chttps://githin httppp///gatubb.oomm/ettttt////
16

---

the models for 6 epochs and use the model from the last epoch without using the validation set for evaluation. For the original clean dataset, we train all models for epochs and use the model with the best performance on the validation set for evaluation.
Table 11: Hyperparameters for fine-tuning classifiers on different tasks.
Backbone
Learning Rate Ir
Batch Size
Training Epochs E
Weight Decay
Warmup Ratio
MERT-base-upe uncased [12]
5e-5
6%
TicyBERT [23]
1c-4
32
1c-4
6%
Diat11BERT -a--BETT-aa--uuaasde [46]
5e-5
6%
DaBERTa- base [21]
Se-5
32
le.4
6%
DeBERTa-V3 V3-large [21]
2e:5
Table 12: The reference of model checkpoints used in this study.
Model
Link
BERT base- uncased [12]
httpe://huggingtee
co/bert base uncased
TinyBERT [23]
https://hugginggte co/ ho/hawsei-- warl--naaaT/nnnente General_41.3311
Dist11BBRT-base-uncae [46]
httpe htwggngggggggggggeeeeggccgcacace co/distilbertt
wtt.drrrihbtt--b-e--maaaeee uase--
DeBeRTe-rS-base [21]
https: http:///wwaaagaaceooccaaeeeeeeee so//alcreeeettttberrra----------
DeBERTae-r3-Iarg [21]
https: httpp://wwggggggacc.cccooorrrers ota.
SentenceBERT [44]
https: htaps::/hugginggfae..o co/sentence-
SimCSE [16]
htttp:: https://bugg https////ggggggggeeeee wttpc/eee.oppprieeeeeeeeieeeeece co/princeton
caiinmnnp//mnn nip/sup sincse bert base uncased
COCO-DR [56]
https://huggingface ctt..co//pp/// co/OpenMatchh/ cocodr- bas--aaeeee------------eeee-eee-
C.3 Model checkpoint
All the checkpoint used in fine-tuning and linear probing experiments are listed table 12.
D Additional Experimental Results
D.I Linear probing with generated datasets
In this section, we evaluate the generated dataset under the linear probing setting, where linear
classifier is trained over the frozen pretrained features. To make a comprehensive evaluation, we employ four text embedding methods, namely BERT-CLS [12], SentenceBERT [44], SimCSE [16] and COCO-DR [56], and present the results in Table 13. We can see that AtrPrrmpt consistently outperforms SimPrompt, MetaPrompt and surpasses the Gold the NYT dataset. This further indicates the datasets generated with AttrPrompt has higher quality than two baselines and this superiority is robust to the model choices.
D.2 Additional results on attribute selection
Table 14 shows the result of AttrPrompt without human selection of attributes on two datasets. While slight performance drop observed, the result of AtuPProppt without attribute selection still significantly outperform the SimPrompt. This demonstrates the robustness of AttrPrompt performance to potentially irrelevant attributes.
D.3 Additional results on data/budget efficiency
In Section 5.3, we discuss the data/budget efficiency of AttrPrompt and SimPrompt NYT and Amazon datasets; here, we provide additional results on the remaining datasets in Figure We draw conclusions similar to what is stated in Section 5.3 that AttrPrompt exhibits better data/budget efficiency than SimPrompt.
D.4 Different temperature parameters for ChatGPT
Temperature (!) is one crucial hyperparameter of LLMs that controls the diversity of the generated text [20, 10], while the studied attributed prompts are also for diversifying the generated data. We are then curious about the effectiivnees of the temperature and how it compares to the AttrPrompt. We
17

---

Table 13: Performance of linear probing with different pre- pre-trained encoders.
Sequence Encoder
Method
NYT
Amazon
Reddit
StackExchange
Acc
F1
Acc
F1
Acc
F1
Acc
Fl
Gold
65.39
70.26
66.73
68.76
55.73
58.00
17.17
17.68
SimPrompt
60.69
62.26
37.95
38.07
27.41
27.60
12.27
13.71
BERT-CLS [12)
MctaPrompt
58.43
57.05
27.52
24.43
23.93
23.99
11.40
12.60
AurPrompt
62.38
64.17
48.18
49.46
45.03
45.95
17.33
17.64
Gold
81.16
84.43
80.95
83.30
80.76
81.37
55.13
54.25
SimPrompt
84,32
85.91
67.55
68.29
69.94
70.62
45.37
47.79
SentenceBERT [44]
MctaPrompt
87.65
87.11
67.16
65.78
79.26
69.15
45.71
47.54
AttrPrompt
88.77
88.36
68.10
71.62
71.73
46.80
48.35
Gold
78.02
80.17
73.20
75.13
73.75
74.26
45.26
41.86
SimCSE [16]
SimPrompt
78.93
78.86
52.40
52.83
54.78
53.86
30.98
32,92
MetaPrompt
78.17
78.27
53.53
52.55
51.30
48.93
30.54
31.50
AttPrropt
81.41
81.04
60.66
61.32
59.22
59.15
30.31
31.43
Gold
80.66
82.60
76.13
78.34
79.63
80.04
61.02
63.77
SimPrompt
84,37
84.17
55.66
55.28
63.66
64.71
43.22
41.50
COCO-DR [56]
MetaPrompt
84,51
84.26
57.96
56.58
65.36
65,91
44.30
42.97
AtrPrompt
86.74
86.43
60.47
62.03
66.73
66.71
45.44
43.76
Gold
76.31
79.37
74.25
76.38
72.47
73.42
44.65
44,39
SimPrompt
77.08
77.80
53.39
53.62
53.95
54.20
32.96
33.98
Average
MctaPrompt
77.19
76.67
51.54
49.84
52.00
33.65
AttrPrompt
79.83
80.00
59.35
60.48
60.65
60.89
34.97
35.30
NYT
Amazon
GPT-3.5 Zero-shot
69.84
54.56
SimPrompt AttrPrompt AttrPrompt w/o selection
76.34
56.96
82.26
65.87
81.08
63.76
Table 14: The Performance of AutrPrompt with/without attribute selection.
22 2000
SimPromgt
SimPrompt
Gold
Gold
Attrfrommt
Attrfrompt
SimFrompt
SimPrmptt
Banget
Budget
Proportion (3) Budgets (5)
Proportion 196 of Budgets
Proportion of Data
Proportion (9) Data
(a) Reddit Budget
(b) StackExchange Budget
(c) Reddit Data
(d) StackExchange Data
Figure 8: The comparisons on budget efficiency and data efficiency on Reddit and StackExchange.
study different values the temperature using the NYT dataset and present the results in Table 15. From the results, can that compared with the temperature, AtrPrrmpt brings more significant performance gain, demonstrating its superiority over temperature tuning.
Table 15: Study of the temperature.
1.0
1.5
Method
Acc.
F1
Acc.
F1
Acc.
F1
SimPrompt
AtrPrromp
81.30
82.26
81.47
82.23
79.47
79.86
D.5 The performance with respect to long-tail classes
As we have seen in Table classes issue since the imbalance ratio is high, yet the generated dataset are class-balanced, we are then curious how the class balance in the generated dataset benefits the model performance on long-tail classes. We take the NYT dataset as example and plot the per-class Fl score of Gold, SimPrompt,
original training sets involved datasets have severe long-tail
18

---

and AttrPrompt in Figure 9, where the x x-axis is classes sorted by their number of data in the Gold dataset in descending order. From the figure, we can see that out of 26 classes, AturPrompt renders the best per-class F1 score on 10 classes, which is 13 for Gold and 3 for SimPrompt. Notably, for classes with few examples in the Gold set (the rightmost classes in the figure), AutrPrompt better than the Gold and SimPrompt, especially for the class "abortion" with the fewest examples. This suggests a data-centric way to handle the long-tail class issue in topic classification: one may use LLMs to generate class-balanced training set augment the existing training set with the LLM-generated data such that the augmented dataset is class-balanced, the in-depth study of which is left as future work.
ddata (Gala) (Gold)
0.6. seae
1000
2 0.44
500
SimPrompt
Anttoompp
0.0
oberbe
Figure 9: Per-class F1 F1-score of the NYT dataset.
D.6 Experiment results on other datasets
To provide comprehensive evaluation of AttrPrompt, we conducted additional empirical studies using datasets commonly employed in prior works, which typically exhibit low cardinality. Consistent with [55, 54, 57], we focused on four specific tasks: IMDB, SST-2, Yelp, and AG News. The IMDB, SST-2, and Yelp datasets involve binary sentiment classification, while the AG News dataset entails news classification with only four classes. This selection allows for diverse evaluation across different classification scenarios.
To ensure fair comparison, we utilized smaller pre-trained language model (PLM) called Distill- BERT [46] as the classifier, in line with [55, 54, 15]. For the purpose of fine-tuning the classifier, we generated total of 6000 examples, evenly distributed among the classes. For fine-tuning, we use learning rate of 2 x 10- with 5 epochs and use the model after the final epoch for evaluation.
The results of our evaluation are presented in table 16. Overall, we observe that AttrPrompt demon- strates competitive performance across all four tasks. It outperforms our direct baseline (SimPrompt) by an impressive margin of 3.8%. Moreover, when compared to other baselines incorporating noisy-aware learning techniques, AttrPrompt still achieves promising performance, despite using the standard cross-entropy loss. This suggests that in addition to leveraging noisy-robust learning objectives, enhancing the diversity of input prompts serves as an effective alternative to improving downstream performance. It is interesting yet important to combine these different learning paradigms together to further boost the performance.
E Regional Bias with Manual Annotations
In the main body of the paper, we study the regional bias in the NYT dataset via an attribute classifier trained with the location attribute of the prompt associated with each generated data. Here, we manually annotate location for 100 samples from each of the Gold, SimPrompt, and AttrPrompt dataset, and compare manual annotations with the prediction of the attribute classifier in Table 17 (the first three rows). Note that we include "unkown" as an option in manual annotation to absorb text without clear location specification. From the results, we can see that the attribute classifier largely aligns with manual annotations, since the accuracy is greater than 0.95 on the Gold and AttrPrompt and 0.82 on SimPrompt; and the result on SimPrompt is lower since SimPrompt tend to generate more unknown location data.
With manual annotations, we also check how well ChatGPT can follow the required location attribute in the prompts. We compute the accuracy between the location attribute in the prompt associated with
19

---

Table 16:
Results with recent baselines using DistilBERT [46]. *: Results copied from the previous papers [55, 54, 15, 57].
Method/Dataset
IMDB
SST-2
Yelp
AG News
Avg.
Simple fine-tunnen tuning with standard cross-entropy loss
SimPrompt
85.64
79.38
88.39
74.30 82.74
81.93
AttrPrompt
85.92
83.64
90.60
85.73
Other results with dedicated training techniques (for reference only)
Zero-shot Prompting*
77.31
82.63
78. 10
73.20
77.81
ZeroGen*
82.77
80.41
87.84
76.48
81.88
SuperGen*
85.05
84.28
89.88
77.40
84.15
ProGen*
87.20
84.12
89.39
80.81
85.38
SunGen*
89.45
89.38
89.19
80.49
87.12
ReGen
85.32
87.84
80.18
85.58
each data and that annotated by human on the 100 AttrPrompt samples (the last of Table 17), which is 0.96. This means that ChatGPT is able to faithfully generate news with desired location specifications.
Table 17: Accuracies based manual annotation of the location attribute of the NYT dataset.
Comparison
Data Source
Accuracy AttrPrompt
Gold
0.96
Attribute Classifier Human
SimPrompt
0.82
0.95
Attribute in Prompt v.s. Human
AttrPrompt 0.96
We also plot the pic charts using manual annotations of location Figure 10, and obtain similar findings in Section 4.2, Le., the data in Gold and SimPrompt are largely biased towards "North Amarica", while that of AttrPrompt is relatively balanced.
Gold
SimPrompt
AttrPrompt
Africa
Asia
Europe
North America
Oceania
South America
Unknown
Figure 10: Pie charts of the "location" attribute based on manual annotations.
Additional Bias Analysis
F.1 Biases other than regional bias
In addition to the regional bias study in the main body of the paper. here investigate bias in other datasets (Figure 11). From the results, we can see that the dataset generated with Gold and SimPrompt exhibit different level of biases, which shows that in general the biases exists in both generated and real dataset, while dataset generated with AttrPrompt has relatively balanced attribute distribution.
F.2 Bias analysis in AttrPrompt
Here, we compare the distribution of predicted "location" in the dataset generated with AttrPrompt to that with the Gold and SimPrompt in Figure 12. We can that AttrPrompt renders relatively balanced "location" distribution.
20

---

Gold
SimPrompt
AttrPrompt
worst
bad
average
good
excellent
(a) Amazon-Experience.
Gold
SimPrompt
AttrPrompt
informatiie/eeducatiinnal
entertaining/fuunny
thought-prrovokinj/dlscsssssss
personal/storytellingg
help/advice
(b) Reddit-Style.
Gold
SimPrompt
AttrPrompt
specific
comparative
problem-solution
troubleshooting
tutorial-based
(c) StackFxxcange-Style.
Figure 11: Pie charts of predicted attribute distributions on different datasets.
(a) All
Tennis
Soccer
(d) International Business
Europe
Asia
Africa
North America
South America
Oceania
Figure 12: Pie charts of the distributions of "location" predicted by an attribute classifier for the NYT AttrPrompt dataset. (a) is "location" distribution over the whole dataset, while others specific classes.
21

---

G The Prompt Format for LLM-Generated Attributes
G.1 Prompts for NYT Dataset
G.I.1 Key Attributes
Prompts to ChatGPT. What do you think are important attributes to generate some diverse news from NYT under specific topic, such as Football, Federal Budget. Examples: subtopics, writing style..
Response from ChatGPT. Important attributes of news from the New York Times under a specific topic could include:
Subtopics: exploring different angles and subtopics within the main topic can lead to wider range of news stories.
varying the writing style can help to make the news stories more interesting and engaging for readers. This can include using different tones, perspectives, and sentence structures.
Writing style:
Sources: using a variety of sources, such experts, eyewitnesses, and affected individuals, can provide different perspectives and add diversity to the news stories.
Geographical locations: covering the topic from different geographical locations help show how affects different communities and provide global perspective.
Timeliness: covering current events and breaking news related to the topic can add diversity and relevance the news stories.
Multimedia: using range of multimedia elements, such as images, videos, and infographics, can help to add variety and appeal to the news stories.
Length: varying the length of the news stories, from short and concise to longer and more in-depth, can help provide wider range of information and cater different readers' preferences.
Selected Attributes
1. Subtopies
2. Writing Styles
3. Geographical locations (determined by six continents by default)
4. Length (determined by {30-80} and (100-150) by default)
G.1.2 Attributes Details
Prompts ChatGPT for Similar Classes. List similar classes for (class-name) news on NYT,
The set of classes is listed as: [[a list of class-names]
Prompts ChatGPT for Subtopics.
List 10 diverse subtopies for (class_name] news on NYT.
Response from ChatGPT. See section H for details.
G.1.3 Writing Styles
Prompts to ChatGPT. List different writing styles of NYT news.
Response from ChatGPT.
Investigative reporting
2. Opinion/editorial
3. Human interest/profiles
4. Long-form narrative
5. Breaking news
22

---

G.2
Prompts for Amazon Dataset
G.2.1 Key Attributes
Prompts to ChatGPT. What do you think are important attributes to generate some diverse product reviews. Examples: writing style...
Response from ChatGPT. consider:
When writing review for product, here are five important attributes
Product Type:
Clearly mention the type of product you are reviewing, such as a smartphone, laptop, fitness tracker. This helps readers understand the category and purpose of the product.
Specify the brand of the product as often influences quality, reputation, and customer support. Discuss the brand's overall credibility and whether it aligns with your expectations.
Brand:
User Experience: Evaluate the overall user experience of the product. Discuss its speed, accuracy, reliability, and efficiency in performing its intended tasks. Highlight any exceptional or lacking performance aspects.
Quality and Durability: Assess the quality of the product, including the materials used, construction, and overall durability. Discuss whether feels well-made, solid, and likely to withstand regular over time.
Features and Functionality: Describe the specific features and functions of the product. Highlight any unique or standout features that enhance its usability or set apart from similar products the market.
G.2.2 Selected Attributes
1. Product Name
2. Brands
3. User Experience
4. Writing Styles
5. Length
G.2.3 Attributes Details
Prompts ChatGPT for Similar Classes. List similar classes for (product-ccass) products on Amazon. The set of classes is listed as: {[a list of class-names)
Prompts to ChatGPT for Product Name/Brands. List 20 diverse product names/brands for (product-class) on Amazon.
Response from ChatGPT. See section H.2 for details.
G.2.4 Usage Experience
Prompts to ChatGPT. List 5 different usage experiences of Amazon reviews.
Response from ChatGPT.
1. Worst: A highly unsatisfactory or problematic experience. Users may encounter multiple issues, such usability problems, errors, and frustrations.
2. Bad: below-average experience. Users may encounter some difficulties or frustrations while interacting with the product or service, but the overall experience falls short of meeting their expectations.
3. Average: neutral middling experience. Users may not encounter major issues, but the experience lacks any exceptional aspects standout features.
23

---

4. Good: A positive and satisfactory experience. Users have smooth and enjoyable experience with minimal issues or obstacles.
5. Excellent: The highest level of satisfaction. Users have seamless, delightful, and memo- rable experience.
G.2.5 Writing Styles
Prompts ChatGPT. List 5 different writing styles of Amazon reviews.
Response from ChatGPT.
Detailed Review: A detailed review provides a comprehensive analysis the product, covering various aspects such as features, performance, durability, and user experience.
2. Comparative Review: A comparative review compares the product with similar options available on Amazon.
3. Pros and Cons Review: A pros and cons review presents balanced assessment of the product by listing its positive and negative aspects.
4. Personal Experience Review: A personal experience review shares the reviewer's firsthand experience with the product.
5. Recommendation Review: A recommendation review offers clear recommendation or endorsement of the product.
G.2.6 Prompt Template
See table 18 for details.
Table 18: Prompt template for the Amazon Product Review dataset.
Method
Prompt
SimPrompt
Suppose you review writer. Please write review for (product cclass) Amazon.
Suppose you are a review writer. Please write a review for (prodact-clesa) product in Amazon following the requirements helow:
1. The review should be about the product [aubtopie]s
AurPrompt
2. The brand the product should be [brand];
3. Should in length between (Lengta sat---dddd and (Length mart maxx- words;
Should deseribe the usage experience (usage- experience)
5. The writing style of the review should be (style];
(product-classs) and
[ateiTar-class].
G.3 Prompts for Reddit Dataset
G.3.1 Key Attributes
Prompts ChatGPT. List 5 important attributes generate some diverse reddit posts under specific topic, like build a pc. premed, etc. Examples: subtopics, keywords, writing style...
Create posts that provide step- by-step guides, tutorials, informative content. These can include detailed instructions, tips, or recommendations help readers in their journey. For example, "build_a_pe. you could create guide on assembling gaming PC or optimizing system performance.
Response from ChatGPT. Informative Guides and Tutorials:
Writing Style: Utilize different writing styles create diverse posts. For instance, you can write informative guides or tutorials, opinion pieces, comparison reviews, troubleshooting tips, or personal stories related the topic.
questions or statements to initiate discussions and engage the Reddit community. These can be open-ended queries, asking for opinions or experiences, or presenting a controversial viewpoint to encourage debate and dialogue.
Discussion Starters: Pose
Personal Experiences: Share personal stories experiences related to the topic. This allows to connect with the community on a more personal level and provide insights based on your firsthand experiences. It can include successes, challenges, lessons learned, interesting anecdotes.

---

Table 19; Prompt template for the Reddit dataset.
Method
Prompt
SunPrompp
Suppose you are a writer for posis Reddit. Please give a posi for [post-clase in Reddit.
Suppose you review writer, Please give post Reddit (post-cllass) community following the requirements below:
1. The posi should locus on [expecience]:
AttPrompp
2. Should in length hetween [langth and (Tangth:maz-uordd)) words;
3. The writing style of the post should be [style]:
4. Should mention the
(zesource];
5. the post must be relevant to (post-class) irrelevant to: (similar-clas claessss).
Resources and Recommendations: Provide valuable resources such as articles, websites, tools, or recommended products/services related to the topic. These resources can help the community access additional information, useful tools, or trusted recommendations enhancing their understanding and engagement with the subject.
G.3.2 Selected Attributes
1. Personal Experiences
2. Resources
3. Writing Styles
4. Length
G.3.3 Attributes Details
Prompts to ChatGPT for Similar Classes. List 5 similar classes for (post-class) products on Reddit. The classes is listed as: [[a list class- saas-na
Prompts to ChatGPT for Personal ExperiencessRessorcce. List 20 diverse Personal Experi- ences/Resources for posts about (post-class) Reddit.
Response from ChatGPT.
See section H.3 for details.
G.3.4
Writing Styles
Prompts ChatGPT. List 5 different writing styles of Reddit Posts.
Response from ChatGPT.
Informatiie/Edducctionnl:: Create posts that aim to provide valuable information or educate the community on particular topic. Share informative articles, research findings, guides, tutorials, or explain complex concepts in a simplified manner.
2. Satertaining/Funney Craft posts that meant entertain bring humor the community, Share funny stories, jokes, memes, amusing videos, or any form of content that can elicit laughter and enjoyment.
3. Thought-Provokingg// Provoking/Discuusion Initiate posts that spark thoughtful discussions and invite users to share their opinions or experiences. Pose open-ended questions, present philo- sophical dilemmas, discuss current events, or explore societal issues foster engaging conversations.
4. Personal/Storytelling:: Share personal experiences, anecdotes, or narratives that others can relate to. Tell compelling stories, share achievements, challenges, life lessons that can inspire or resonate with the community.
5. Help/Advice: Create posts where you seek help or offer advice the community. Ask for guidance on a specific problem, seek recommendations, provide support, or share expertise in a particular field.
G.3.5 Prompt Template
See table 19 for details.
25

---

G.4
Prompts for StackExchange Dataset
G.4.1 Key Attributes
Prompts ChatGPT. What do you think are important attributes to generate diverse posts under specific topic on the stackexchange forum, like 3D meshes, server, etc. For example, one important attribute could be writing style...
Response from ChatGPT. When generating diverse posts under a specific topic on the Stack Exchange forum, several important attributes can contribute the variety and richness of the discussions:
Participants with different levels of expertise offer insights and perspectives from beginner to advanced, allowing for a broader range of questions and answers.
Depth of Knowledge:
Diverse Perspectives: Encouraging participants from various backgrounds, industries, and experiences can bring fresh viewpoints and approaches the topic.
Encouraging participants to state their objectives or goals in their posts can help generate diverse content. Some posts may aim seek advice, share experiences, provide step-by-step guides, analyze specific problems, or discuss broader concepts.
Objectives:
Problem Scenarios: Encourage users to discuss real-world problem scenarios related to the topic. This helps generate diverse posts by exploring different use cases, challenges, and approaches to problem-solving.
Problem Solving:
Promoting problem-son solvenn skills and encouraging users to share their challenges, roadblocks, and solutions can foster collaborative and dynamic environment.
Clear Communication: Emphasizing clear and concise communication helps to ensure that questions are well-articulated and that answers are easily understood by wide range of users.
G.4.2 Selected Attributes
1. Problem Scenarios
2. Depth of Knowledge
3. Writing Styles
4. Length
Prompts ChatGPT for Problem Scenarios. List 20 diverse Problem Scenarios for posts about [post-class] in StackExchange.
Response from ChatGPT.
See section H.4 for details.
G.4.3 Writing Styles
Prompts ChatGPT. List 5 different writing styles of StackExchange Posts.
Response from ChatGPT.
1. Specific: Be specific in your question by including relevant details, such error messages, code snippets, or specific scenarios, to help the community provide accurate and targeted answers.
2. Comparative: Compare different approaches, technologies, or methodologies, and ask for insights opinions from the community their strengths, weaknesses, and suitability for your specific situation.
3. Problem-Solution Present problem challenge you encountering and outline any attempted solutions, seeking feedback or alternative approaches from the community to help you find an optimal solution.
4. Troubleshooting: Describe specific issue error you are encountering and provide any relevant error messages or logs, seeking assistance in identifying the cause and potential solutions.
26

---

Table 20: Prompt template for the StackExchange dataset.
Method
Prompt
SimPrompe
Suppose you writer posts StackEiccinge. Pleuse give example of question post StuckExchange about (poas--llux))
Please give an example question post Sinck Exchunge about [poss-clase) following the requirenentss below:
1. Should focus on the secnario of [acenarto]:
AuPPrmppt
Should in longth between (length words) and (Lengthrmax words) words;
writing style
question
(asyla];
The question should he in idepsh];
review be relevant clase) and incelovnnt to: [sinslar clase).
5. Eitorial-Based Ask for guidance step-by-step instructions on how to achieve a specific task or goal, seeking expertise and advice from the community.
G.4.4 Depth of Knowledge
Prompts to ChatGPT. List 5 different levels of depth of knowledge for StackExchange Posts.
Response from ChatGPT.
1. Beginner Level: Questions that address basic concepts and seek introductory explanations or guidance. These questions are suitable for individuals who are new to the topic.
2. Intermediate Level: Questions that require moderate understanding of the topic and seek more specific information or solutions to intermediate-level problems.
3. Advanced Level: Questions that delve into complex concepts or advanced techniques, requiring deeper understanding of the topic and seeking advanced solutions insights.
4. Expert Level: Questions that involve specialized or advanced knowledge within a specific field, targeting experts or professionals seeking detailed and intricate solutions.
G.4.5 Prompt Template
See table 21 for details.
G.S Prompts for arXiv Dataset
G.S.1 Key Attributes
Prompts to ChatGPT. What do you think are important attributes generate diverse papers under specific topic on arXiv. For example, one important attribute could be writing style...
Response from ChatGPT. Generating diverse papers under specific topic on platforms like arXiv involves considering various important attributes. Here are a few attributes that can contribute to generating diverse papers:
Writing Style:
Encouraging papers with different writing styles, such as technical, expository, theoretical, or empirical, can bring diversity to the presentation and appeal to wider range of readers.
Subtopics: Promoting papers that explore different subtopies within the broader topic can provide comprehensive coverage and delve into specific areas of interest.
Techniques: Encouraging papers that employ different research methodologies, such experimental, computational, or analytical, can bring diverse approaches to studying the topic.
Data Sources: Promoting papers that utilize diverse data sources, such as surveys, simulations, real-world datasets, or case studies, can offer different perspectives and insights into the topic.
Interdisciplinary Perspectives: Encouraging papers that incorporate interdisciplinary perspectives, drawing insights from multiple fields combining methodologies from different disciplines, can contribute to a richer understanding of the topic.
G.5.2 Selected Attributes
1. Subtopies
2. Techniques
27

---

Table 21: Prompt template for the arXiv multi-label dataset.
Method
Prompt
SimPPropt
Please give synihetic paper abstract about [paper-classed]
Please write abstract of (pepar-clansae paper arXiv following the requirements below:
1. Should focus on the [aubtapie];
ArtrPrompt
2. Should in length between [length:nin--nnm and (langth:maz-zerdn words;
3. The paper should use the technique relevant to (tachnigua]:
4. The style the puper should be in (style];
3. Writing Styles
4. Length
Prompts to ChatGPT for Subtopics/Techniqes. about (paper-class] in arXiv.
List 20 diverse subtopics/techniqes for paper
Response from ChatGPT. See section H.5 for details.
G.5.3 Writing Styles
Prompts to ChatGPT. List 5 different writing styles of arXiv papers.
Response from ChatGPT.
Experimental:
This style focuses on presenting empirical research with a strong emphasis experimental design, data collection, and statistical analysis.
2. Theoretical: This style emphasizes the development of new theoretical frameworks conceptual models.
3. Review: This style involves synthesizing and analyzing existing literature on a specific topic.
4. Case Study: This style presents an in-depth examination of specific case or situation, aiming to provide insights into complex phenomena or phenomena that context-specific.
G.5.4 Prompt Template
See table 21 for details.
G.6 Prompts for AG News Dataset
The prompts are the same as the NYT dataset. The only difference is the subtopics for different labels. The subtopic information is listed in H.7 for details.
G.7 Prompts for Yelp/SST22/MDB Dataset
G.7.1 Key Attributes
Prompts to ChatGPT. What do you think are important attributes generate some diverse reviews with sentiments? E.g. writing styles
Response from ChatGPT. Writing Styles: Utilizing different writing styles such as descriptive, narrative, persuasive, or humorous can create diverse range of tones and emotions in the reviews.
Language and Tone: Varying the language and tone used in the reviews can evoke different sentiments. This can include formal or casual language, positive or negative tone, even incorporating elements of storytelling.
Personal Experience: Sharing personal experiences and anecdotes in the reviews can add authenticity and emotional depth, allowing readers connect with the sentiments expressed.
Descriptive Details: Including specific and vivid descriptions the ambiance, decor, flavors, smells, or textures of the food can create sensory experience that evokes various sentiments.
28

---

Table 22: Prompt template for the Yelp dataset.
Method
Prompt
SinProopp
Suppose you are a review writer. Please give an example of (senciment) review in Yelp.
Please write [sentiment] review restaurant, following the requirements below:
The overall review should be [sectimet]];
AttPPrompt
It should be
restaurant;
Should focus on [personal- serrona---rppiinee experience]:
Should be length between (length wagggthiiio-weeeee uxto-werds] and (Length max- words] words;
The writing style of the review should be (etylo);
G.7.2 Selected Attributes
1. Personal Experience
2. Descriptive Details
3. Writing Styles
4. Length
G.7.3 Personal Experiences/Descrritiieee Details
Prompts to ChatGPT.
Details are listed in Appendix H.7 and H.8.
G.7.4 Writing Styles
Prompts to ChatGPT. List 5 different writing styles of Restaurant Reviews (for Yelp).
Response from ChatGPT.
1. Descriptive Style:
Focuses on providing detailed descriptions of the restaurant's 's ambiance, decor, and overall atmosphere. aims paint vivid picture of the dining experience.
2. Personal Narrative Style: Incorporates personal anecdotes and experiences from the visit to the restaurant. It adds personal touch to the review and allows the reader to connect with the reviewer's perspective.
3. Analysis Style: Takes a more analytical approach by evaluating various aspects of the restaurant, such as food quality, service, cleanliness, and value for money.
4. Comparative Style: Compares the restaurant to similar establishments in terms of cuisine, price range, service quality, other relevant factors.
5. Gastronomic Style: Focuses primarily on the food and beverage offerings of the restaurant.
Prompts to ChatGPT. List 5 different writing styles of Movie Reviews (for IMDB/SST-2).
Response from ChatGPT.
1. Descriptive: Focuses providing detailed and vivid description of the movie and its elements.
2. Analytical: cinematography, and direction.
Analyzes and evaluates various aspects of the movie, such as the plot, acting,
3. Persuasive: Aims to convince the reader to watch or avoid the movie by presenting strong arguments and opinions.
4. Comparative: Compares the movie other films within the same genre or by the same director.
G.7.5 Prompt Template
See table 22 and 23 for details.
29

---

Table 23; Prompt template for the SST-2/IMDB dataset.
Method
Prompt
SimPrompt
Suppose you are a review writer. Please give an example a [sentiment: review a movie.
Please write (sentinent) review mavie, following the requirements below;
The overall review should be [sentiment]:
AtrPrompt
It shnold he (deacrips ive-dataila] movie
Should focus on perrasal-expprieeee experiences;
Should be length between (Lengthinda words) and (Longth maxx-wordee words;
The writing style of the review should be (style):
H Details for LLM-Generated Attributes
H.1
NYT Attributes
H.1.1 Subtopics
We randomly select 5 categories in NYT dataset and display the corresponding subtopic attributes for each category:
astronomy:
Discoveries of exoplanets
Black holes and their role in shaping galaxies
The search for extraterrestrial life
Gravitational waves and the study of the universe's origins
The use of telescopes to explore the universe
The mysteries of dark matter and dark energy
Solar flares and their impact Earth
The history of the universe and its evolution over time
Exploring the possibility space tourism
The exploration of our neighboring planets, such as Mars and Venus.
baseball:
Recent controversy surrounding sign-stealing scandal MLB
Breakdown top prospects in minor league baseball
Analysis of new rule changes for upcoming baseball season
Coverage of recent World Series champions and their success
In-depth profile of influential baseball figures, such managers players
Updates on retired players and their post-baseball careers
Highlighting standout performances by individual players or teams in recent games
Coverage of international baseball leagues and their top players
economy:
Job market and employment rates
Interest rates and monetary policy
Inflation and deflation
Economic growth and GDP
Consumer spending and retail sales
Income inequality and poverty
GDP growth and contraction
Labor market trends
Economic impacts of natural disasters and pandemics
Housing market and real estate
Rederal_budger
Deficit reduction strategies
Government spending priorities
30

---

Tax reform proposals
Agency budget cuts and restructuring
Budget negotiations and debates
National debt projections
Fiscal policy decisions
Congressional budget proposals
Infrastructure spending plans
Public opinion federal budget issues
movies:
Box office records and trends for Hollywood blockbusters Pioneering techniques in film-making and special effects
Representation and diversity in casting and storytelling Reviews and analysis of anghly-anticipateed new releases
The impact of streaming services on movie distribution and consumption
The intersection of politics and film, from socially-conscious storytelling to politically- charged controversies
Profiles of notable actors, directors, and producers shaping the industry
The changing landscape of film festivals and awards season
Spotlight independent and international cinema
The legacy of classic films and their enduring cultural impact.
H.2
Amazon Attributes
H.2.1
Product Brands
We randomly select 5 categories in Amazon dataset and display the corresponding product brand attributes for each category:
camera :omerr_photoo::
Canon
Nikon
Leica
Hasselblad
Fujifilm
Lumix (Panasonic)
GoPro
Polaroid
Yashica
Mamiya
jewelry_ane arr__ndd_aatheee
Rolex
Cartier
Tiffany & Co.
Bulgari
Omega
Patek Philippe
Swarovski
Gucci
Michael Kors
Pandora
magazines.:
31

---

Rolling Stone
Vogue
National Geographic
The New Yorker
GQ
Cosmopolitan
People
Time
Sports Illustrated
Forbes
health_ang Bailth_and_paee hand_personal__cm care.: care..
Johnson & Johnson
Dove
Colgate
Head & Shoulders
Merck
Pfizer
Merck
Abbott Laboratories
GlaxoSmith
says_games..
Mattel
Fisher-Price
Hasbro
Lego
Nerf
Barbie
Hot Wheels
Playmobil
MGA Entertainment
Paw Patrol
H.2.2 Product Names
We randomly select 5 categories in Amazon dataset and display the corresponding product name attributes for each category:
sports sports_outdent outdoors.t
Trekking poles
Kayak
Stand-up paddleboard Treadmill
Bike
Yoga mat
Weightlifting gloves
Athletic training cones Ab wheel
Resistance bands
Jump rope
Rollerskates
Boxing gloves
32

---

Basketball
Football
Golf clubs
Tennis racquet
camera photo.:
SnapShooter
FocusReady
ClickCapture
MetteiesMatter
FlashFinder
PicturePerfect
ShotSnap
Vision Vibe
PixelPal
FraezeFrame
AngleAttack
SnapStash
FrameFlair
SmileSnaps
Imagelmpact
ZoomZest
ClickCraze
PixelPassion
ShootSmart
CaptionChamp.
grocery_ande seccery_and_gooee aand_gourmet_fodd.
Nutella
Doritos
Hellmann's Mayonnaise
Camphell's Soup
Ritz Crackers
Quaker Oats
Ben & Jerry's Ice Cream
Tostitos Salsa
Goldfish Crackers
Red Bull Energy Drink
McCormick Spices
Crystal Light Drink Mix
Funyuns Onion Rings
Skippy Peanut Butter
Heinz Ketchup
Tabasco Hot Sauce
Hershey's Chocolate Syrup
Nescafe Coffee
Kraft Macaroni & Cheese
Gatorade Sports Drink
baby.:
Baby Swing
Diaper Genie
Milk Warmer
33

---

Baby Carrier
Car Seat
Baby Monitor
Baby Food Maker
Nursery Glider
Changing Table
Baby Bouncer
Playpen
Teething Rings
Baby Wipes Dispenser
Baby Bath Tub
Bibs
Baby Blankets
Pacifier Clip
Baby Sling
Baby Napper
Moses Basket
outdoor_ living.:
Sunbrella
Weber
Keter
Trex
Solaira
Tropitone
Bimini Solutions
La-Z-Boy Outdoor
Suncast
Beltwood
Quikrete
Cosco Outdoor Living
Anova Furnishings
Duramax
US Polymers
Ostrich Chairs
Carefree of Colorado
Tuff Coat
Fire Sense
Heritage Patios.
H.3 Reddit Attributes
H.3.1
Resources
We randomly select categories in Reddit dataset and display the corresponding resource attributes
for each category:
buddy_ _rtissing_osng..
Meetup.com website that allows you find and join groups of individuals with similar interests in your local area, including hiking, book clubs, and social events.
The Buddy System: Understanding Mental Illness and Addiction book that explores the biology of addiction and provides guide for friends and family members of individuals struggling with these issues.
34

---

Lynda.com subscription based online learning platform that provides courses variety of subjects including computer programming, business, web design, and more. - Codecademy.com an interactive online platform that teaches coding skills for free or through a subscription.
KKaa Academy nonprofit organization that provides free online courses in wide range of subjects including math, science, and humanities to learners of all ages.
-Duolingo language-learning app that is available for free the App Store and Google Play, offering courses in a variety of languages including Spanish, French, and German.
Mind App mobile app that helps users find and book local fitness, wellness, and beauty activities such as yoga classes, massages, and haircuts.
Headspace meditation app that offers guided meditation courses to help users reduce stress and improve focus.
The Knot a website that provides tools and resources for wedding planning, including Wedding Website Builder, guest list tracker, and registry management.
Khan Academy nonproñit organization that provides free online courses in wide range of subjects including math, science, and humanities to learners of all ages.
Others resource for buddy_crossing
divediii::
Division Builds A subreddit dedicated to sharing and discussing various builds used The Division.
Division Zone A website with extensive information on game mechanics, gear, and other important gameplay aspects.
The Division Discord community-rum Discord server where players can connect and find groups to play with.
The Division Wiki A comprehensive wiki with guides, tips, and information everything related to The Division.
Skill-Up's YuuTube channel A popular YouTuber who provides detailed analysis and reviews of The Division's 's updates and patches.
MarcoStyle's YouTube channel Another popular YouTuber who provides in-depth analysis and guides for The Division's 's gameplay and mechanics.
The Division LFG website where players can find groups to play with, organize events, and share their experiences.
The Division Zone Map An interactive map that allows players to find important points of interest, loot, and other useful resources.
The Division 2 subreddit A community- subreddit for the sequel, The Division 2, where players can share their experiences and discuss the game.
Others resource for the division.
roblox.:
Roblox Wiki
Roblox Developer Forum (https://defforumroohh www.:umm.ooooo.oooooooo.o
Ultimate Guide
First (https:/mmcdiummccom @ hatthaarritiiiatt--uu---t------- your-fist-ggame-nn
to Making Your
Game on Roblox
rubbx-ppart---111c33abbfc77
Roblox Blog
Roblox Studio Tutorials htth::/wwww...ooouoccccmo///hhhh
- The Roblox Developer Hub (ttps://deeedpeerrrrbbooocccc///
Top 10 Roblox Games
Roblox Support http://www..ccuuuncccco///////eo
Roblox Discord Server (htts::/ddiccord.gg//ooooo)
Top Roblox Youtubers to Follow http::/wwww.ggamrrrraa.coooooooo
to-follow/)
35

---

Others resource for roblox.
whats that book.:
Goodreads A social platform for book lovers where users can search for books, create bookshelves, and write reviews,
LibraryThing A community-drvve driven cataloging website where users can create and share their personal book collections.
AbeBooks An online marketplace for rare and out-of-print books, as well as other antique or collectible items.
Shelfari An online book club where users share book recommendations and read reviews from others.
Project Gutenberg- A digital library freely available public domain books. Paperback Swap A book trading community where users can exchange books with others across the US.
Goodreads Librarians Group community of Goodreads
who help with book cataloging, including identifying books from incomplete information.
Book Riot A website featuring book reviews and book-related news, with an emphasis on diverse and underrepresented voices.
The New York Times Book Review renowned weekly publication featuring book reviews, author interviews, and literary criticism.
Others resource for whats
pokemongo_friends::
Pokemon GO Hub: comprehensive website dedicated news, guides, and analysis on Pokemon GO.
The Silph Road Subreddit: A community-run subreddit dedicated to research and analysis of Pokemon GO mechanics.
Poke Assistant: A website that offers a range of tools to help you optimize your Pokemon GO experience, including IV calculators and gym battle simulations.
The Trainer Club: A YouTube channel that provides daily updates, news, and tips for Pokemon GO trainers.
Gotta Catch 'Em All: A Facebook group where you can connect with other Pokemon GO players and coordinate raid battles and other activities.
Reddit's /PokemonGOFrriends Subreddit: A community of players looking for friends to exchange gifts and share invites for raids.
The PokeMap: A website that allows you to find nearby Pokemon on a map in real-time.
Poke Genie: An app that automatically calculates IVs and other stats for your Pokemon, saving you time and headaches.
Pokemon GO Gamepress: website that offers detailed breakdowns and analysis of Pokemon, movesets, and other game mechanics.
The Go Ranger App: An app that helps you plan your raids and battles, with intuitive mapper tools and filters to help you find the Pokemon you're looking for.
Others resource for pokemongo_fricee friends.
H.3.2 Experience
We randomly select categories in Reddit dataset and display the corresponding experience attributes for each category:
build pc.:
DIY PC Builds: Sharing personal experiences and success stories of building custom PCs, discussing component choices, troubleshooting, and performance optimizations.
BBuget--riendd: Builds: Discussing experiences with building PCs on a tight budget, sharing cost-saving tips, and recommendations for budget-friendy, components.
Cable Management: Sharing personal experiences and tips for effective cable manage- ment in PC builds, discussing cable routing techniques and showcasing clean build aesthetics.
36

---

RGB Lighting: Discussing experiences with RGB lighting setups in PC builds, sharing recommendations for RGB components, software customization, and lighting effects. Troubleshooting Builds: Sharing experiences and tips for troubleshooting common issues in PC builds, helping fellow builders diagnose and solve hardware or software problems.
Silent and Quiet PC Builds: Discussing experiences and recommendations for building silent quiet PCs, focusing noise reduction techniques and quiet component choices.
Workstation Builds: Sharing experiences and insights into building PCs for professional workloads, such as video editing, 3D rendering, programming, and graphic design. Water-Cooling Adventures: Sharing experiences and insights into custom water-cooling loops, discussing the challenges, benefits, and performance improvements achieved. Unique and Custom Builds: Showcasing and discussing unique and custom PC builds, including themed builds, custom cases, or exotic cooling solutions.
Build Planning and Component Selection: Discussing experiences with planning PC builds, researching and selecting components, considering compatibility, and balancing performance and budget.
Modding and Case Customization: Sharing experiences with PC case modding and customization, discussing techniques, materials, and showcasing personal projects.
- Compact and Small Form Factor Builds: Discussing experiences with building compact small form factor PCs, sharing recommendations for mini-ITX cases, cooling solutions, and component choices.
Home Server and NAS Builds: Sharing experiences and insights into building home servers and network-attached storage (NAS) systems, discussing storage options, soft- ware, and data management.
Multimonitor Setups: Discussing experiences with multimonitor setups, sharing tips for optimizing productivity and gaming experiences across multiple displays.
PC Gaming Peripherals: Sharing experiences and recommendations for gaming pe- ripherals, such as keyboards, mice, monitors, and headsets, discussing features and personal preferences.
summon_sipg sign.:
Embracing the Sunbro covenant and assisting fellow players with jolly cooperation, earning sunlight medals and praising the sun together.
Sunbro Covenant:
Fashion Souls: Sharing and showcasing unique and fashionable character builds, armor sets, and weapon combinations for aesthetic enjoyment.
Cooennatt Experiences: Sharing experiences and strategies related various in in-game covenants, such as the Darkwraiths, Blades of the Darkmoon, or Forest Hunters.
-Community Creations: Showcasing community-creater content, such fan art, videos, or fan fiction, celebrating the creativity and talent within the Summon Sign community.
- -Lore-freedll Builds: Discussing and sharing character builds that are aligned with specific characters factions within the game's lore, adding immersion and roleplaying elements.
Community Appreciation: Expressing gratitude and appreciation for the community, developers, and the overall enjoyment derived from the Dark Souls series and the cooperative multiplayer experiences.
xbox.:
Xbox One exclusive games such as Halo 5, Forza Horizon 4, and Gears War Xbox One media and entertainment apps such as Netflix and Hulu
memorable gaming moments or achievements on the Xbox console.
Purchase Xbox One online.
Xbox Kinect motion sensor accessory
Xbox Play Anywhere program
Other Experience of Xbox
pittsburgh.:
37

---

Visit the Andy Warhol Museum
Watch a Steelers football game at Heinz Field
Explore the Carnegie Museum of Natural History
Ride the top of Mount Washington on the Duquesne Incline
Take a leisurely stroll through Phipps Conservatory and Botanical Gardens
Experience the history the city at the Senator John Heinz History Center
Tour the University of Pittsburgh campus
Attend performance at the Benedum Center for the Performing Arts
Take a walk along the Three Rivers Heritage Trail
Taste pierogies, kielbasa, and other traditional Pittsburgh foods
Admire the architecture of the Cathedral of Learning
Explore the Pittsburgh Zoo and PPG Aquarium
Gaze in awe at the exhibits in the Carnegie Science Center
Visit the National Aviary and get up close with tropical birds
Check out the local art scene on Penn Avenue the Garfield neighborhood
Attend the Three Rivers Regatta, Pittsburgh's 's largest annual summer event
Take a bike ride on the Great Allegheny Passage trail
Ride the roller coasters Kennywood Amusement Park
Discover the nightlife in the South Side neighborhood
Go shopping at the Strip District markets for locally-made goods and fresh produce. Others experience for pittsburgh.
metal gear solid.:
gear_solid.:
Tactical Weapon Customization: Experimenting with various weapons, attachments, and equipment to tailor loadouts to different mission objectives and playstyles.
Character Development: Witnessing the growth and development of iconic characters such Solid Snake, Big Boss, Raiden throughout their respective story ares.
Stealthyy Takedowns: Executing silent and non-lethal takedowns. utilizing tranquilizer darts, chokeholds, or sneaking up on enemies from behind.
Tactical Planning: Strategizing mission approaches, analyzing enemy patrols, setting traps, and utilizing distractions to gain the upper hand.
Memorable Characters: Developing connections with unique and memorable characters, such Otacon, Meryl, Revolver Ocelot, or The Boss.
Stealthy Gadgets:
Utilizing gadgets and tools, such as the cardboard box, thermal goggles. or the iconic Solid Eye, to gain advantages during missions.
Emotional Storytelling: Experiencing emotionally impactful moments within the narrative, exploring themes of loss, betrayal, loyalty, and the human cost of warfare.
Espionage Tactics: Participating in undercover missions, gathering intelligence, infil- trating enemy bases, and sabotaging their operations.
LLore and Mythology: Delving into the intricate lore, conspiracies, and historical events within the Metal Gear Solid universe, including topics like The Patriots or the Philosopher's 's Legacy.
Groundbreaking Game Design: Appreciating the innovative gameplay mechanies, cinematic presentation, and attention to detail that have made the Metal Gear Solid series a beloved and influential franchise in the gaming industry. Others experience for _etal_gear gear solid.
H.4
StackExchange Attributes
H.4.1 Scenarios
We randomly select 5 categories in StackExchange dataset and display the corresponding scenario attributes for each category:
multiplayer.:
38

---

Cheating/hacking in online games
Inappropriate player behavior
Unbalanced game mechanics
Difficulty connecting to multiplayer servers Matchmaking errors
Unresponsive or laggy gameplay
Glitches in gameplay affecting online matches
Difficulty finding players to match with Balancing player skills in matchmaking
Disconnects and dropped connections mid-game
Cross-platform compatibility issues
In-game communication problems
Difficulty managing and moderating game servers
Addressing griefing and trolling in multiplayer games
Managing player accounts and login systems
Implementing improving anti-cheat measures
Community feedback and feature requests
Addressing game-breaking exploits
Ensuring fair and accurate reporting of player statistics
Addressing server crashes and downtime.
terrain.:
Error in generative algorithms for creating terrain
Difficulty in implementing procedural terrain generation in a specific game engine
Inconsistencies in terrain generation across different devices
Issues with realism in terrain generation algorithms
Difficulty in implementing terrain physics and collision detection
Terrain rendering issues low-spec hardware
Incompatibility between terrain generation and map or level editors
Optimization of terrain generation algorithms for speed and memory usage
Unwanted artifacts and glitches in terrain mesh generation
Compatibility issues between terrain generation algorithms and game engine systems Difficulty in creating realistic terrain textures and materials
Inacouracy of terrain clevation generation in certain geographic regions Difficulty in implementing terrain deformation mechanics
Poor performance with large-scale terrain rendering and generation Unwanted noise and roughness generated terrain meshes
Compatibility issues between terrain generation and asset importation pipelines Inaccuracy of terrain heightmap data due to low-quality input data sources
Difficulty in handling multi-layer terrain materials and textures
Poor performance with dynamic terrain generation and updates Issues with biome and dimate-based terrain generation. rendering.:
Difficulty creating realistic hair and fur in rendering software.
Debugging issues with transparent materials in 3D rendering engine.
Crashes slow performance when rendering large scenes in real time.
Trouble with anti-aliasing and other graphics optimization techniques.
Struggle with optimizing rendering quality on lower-end hardware.
Difficulty incorporating custom shaders into a game engine or rendering pipeline.
Figuring out how to use the latest rendering features in legacy project.
Issues with rendering dynamic lighting in real time, without pre-baking.
39

---

Optimizing flicker or aliasing issues caused by fast-moving objects in scene.
Solving glitches crashes caused by malfunctioning GPU drivers.
Difficulty in rendering complex ocean water simulations.
Troubleshooting issues with volumetric rendering in a 3D engine.
Finding the optimal rendering settings for particular 3D model or scene.
Figuring out optimal texturing and lighting in photorealistic rendering.
Creating procedural textures and materials in a game engine.
Debugging flicker issues caused by overlapping or intersecting geometry. Difficulty in rendering realistic motion blur in 3D animation.
Solving imbalanced lighting in a photorealistic rendering of a room or scene.
Finding the optimal rendering settings for VR or AR applications.
Debugging issues with inaccurate or glitchy global illumination in a scene.
procedural-geneeratin.
Improving the efficiency of procedural generation algorithms in Python. Troubleshooting issues with memory usage in large-scale procedural generation projects.
Debugging issues with randomized content generation in procedural levels. Implementing procedural generation techniques in C++ for game development.
Exploring the potential of machine learning in procedural content generation. Optimizing the generation of complex 3D models using procedural algorithms.
Managing complexity and maintaining consistency in procedurally generated game worlds.
Addressing issues with procedural generation of text-based content, such as dialogue item descriptions.
Developing tools to aid in the creation and testing of procedural generation algorithms. Balancing the need for randomness with player expectations for fairness and balance. Addressing issues with the procedural generation of music and sound effects. Improving the visual quality of procedurally generated game assets.
Exploring ethical concerns around the use of AI in procedural content generation. Developing procedural generation techniques for non- con-linear narratives or branching storylines.
Improving the procedural generation of inhabited environments, such as procedurally generated NPCs.
Addressing issues with the procedural generation terrain features such as rivers and mountains.
Implementing procedural generation techniques for user-generated content. Supporting multithreaded execution in procedural generation algorithms. Ensuring procedural generation techniques are compatible with various game engines and frameworks.
Improving the scalability of procedural generation algorithms for use in multiplayer games.
networking..:
Difficulty in troubleshooting network connection issues on Linux system Configuring wireless access point for a large office space Implementing load balancing across multiple servers in network
Optimizing network throughput reduce latency in gaming environment Implementing firewall rules to block unauthorized access to network Troubleshooting DNS resolution issues on Windows server Designing and implementing secure VPN connection
Setting up network file server for shared access among multiple clients Configuring SNMP to monitor network traffic and utilization
40

---

Designing network topology for large enterprise with multiple locations Troubleshooting issues with Ethernet switches in a data center environment Implementing QoS to prioritize network traffic for critical applications Configuring NAT and PAT enable internet access for multiple devices Setting up and configuring VLANs to segment a network
Troubleshooting issues with network printers an office environment Configuring routing protocols in a large network
Securing wireless access points to prevent unauthorized access Troubleshooting issues with VPN connection stability and speed Implementing network virtualization with virtual LANS and virtual switches Designing and implementing an effective network security strategy to prevent data breaches.
H.S
ArXiv Attributes
H.S.1
Subtopies
We randomly select 5 categories in arXiv dataset and display the corresponding product brand attributes for each category:
hardware architecture.:
Processor design
Memory architecture
Input/output (I/O) systems
Computer networks
Digital signal processing
Microcontrollers
Embedded systems
Real-time systems
Graphics processing units (GPUs)
Hrigh-performante computing (HPC)
Computer architecture for mobile devices
Wearable technology hardware design
Secure hardware systems
Fault-tolerant hardware design
Quantum computing architecture
Bio-inspired computing systems
Nanoscale computing systems
Reconfigurable hardware systems
Internet of Things (IoT) devices
Green computing and sustainability in hardware design
genomics.:
Genome editing and CRISPR technology
Epigenomics and epigenetic modifications
Pharmacogenomies and personalized medicine
Functional genomics and transcriptomies
Metagenomies and microbiome analysis
Genomic epidemiology and infectious disease
Cancer genomics and tumor heterogeneity
Structural genomics and protein structure prediction
Plant genomics and crop breeding
Animal genomics and livestock improvement
41

---

Comparative genomics and evolution
Genomic diversity and population genetics Next-generation sequencing and high-throughput analysis
Single-cell genomics and cellular heterogeneity
Ethical, legal, and social implications of genomics Cryptography and secure data sharing in genomics Big data analytics and data mining in genomics
Immune system genomics and immunotherapy Genomics and aging research
Gene therapy and gene regulation.
machine baabhine_larning...
Text generation
Natural language understanding for chatbots
Sentiment analysis and opinion mining
Text summarization and keyword extraction
Machine translation
Named entity recognition and entity linking
Dialogue systems and conversational agents
Cross-lingual and Multilingual NLP
Text-to- speechh systems
Phonetics and phonology in computational linguistics
Grammatical error detection and correction
Speech recognition and acoustic modeling
Semantic role labeling
Discourse analysis and coherence modeling
Lexical semantics and word sense disambiguation
Computational lexicography and machine-readable dictionaries
Language Modeling
question answering
Language resources and corpora
Computational sociolinguistics and dialectology.
number number_theory.
Prime numbers
Diophantine equations
Modular arithmetic
Cryptography
Continued Fractions
Pell's Equation
Fermat's 's Last Theorem
Algebraic Number Theory
Riemann Hypothesis
Arithmetic Geometry
Quadratic Forms
L-Functions
Automorphic Forms
Galois Theory
Ramsey Theory
Distribution of Prime Numbers
Number Theory in Cryptography
Summation Formulas
42

---

Gaussian Integers
The Goldbach Conjecture
geophysics.:
Seismic imaging
Earthquake prediction
Geothermal energy
Volcanic eruptions
Plate tectonics
Geomagnetism
Palcomagnetism
Geophysical surveying
Geophysical fluid dynamics
Gravity measurements
Rock physics
Crustal deformation
Geomorphology
Mineral exploration
Earth structure modeling
Geodetic techniques
Hydrogeophysies
Earth modeling
Electrical geophysics
Remote sensing geophysics
H.5.2 Techniques
We randomly select 5 categories in the arXiv dataset and display the corresponding attributes for each category:
hardware aarchitectuceee:
Binary Decision Diagrams (BDDs) for circuit optimization
Probabilistic CMOS (PCMOS) for energy-efficient computing
Boundary Sean Architecture (BSA) for testing and fault diagnosis
Clock gating for power reduction in synchronous circuits
Memristive networks for neuromorphic computing
Approximate computing for energy-efficient digital processing
Verilog Hardware Description Language (HDL) for circuit design and simulation
Reversible logic circuits for low power consumption
High- synthesis (HLS) for synthesizing hardware from software descriptions
Phase Change Memory (PCM) for high-density non-volatile storage. genomics.:
Genome assembly and annotation using hybrid approaches.
Comparative genomics for analyzing evolutionary relationships between genomes.
Differential gene expression analysis using RNA sequeneing data.
Metagenomics for studying the microbial communities in different environments.
Epigenetic analysis for understanding gene regulation.
Network analysis for identifying gene interactions and pathways.
Structural variation analysis for detecting genomic rearrangements.
Functional genomics for studying gene function and pathway regulation. Genome-wide association studies for identifying genetic variants associated with complex traits.
43

---

screening methods for identifying genes involved in specific biological
processes.
computation and and_language.
Word Embeddings
Recurrent Neural Networks (RNNs)
Attention Mechanism
Transformer Model
Dependency Parsing
Topic Modeling:
Machine Translation
Corpus Analysis
Discourse Analysis
Pragmatics
number number_theory.
Primality testing using elliptic curves
Continued fraction factorization method
Algorithm for solving Diophantine equations
Quadratic sieve algorithm for integer factorization
Pollard rho algorithm for integer factorization
Digital sum subtraction method for computing discrete logarithm
Fermat's 's method for factorization of primes
Chinese remainder algorithm for solving modular equations
Exponential-sume algorithm for computing in algebraic number fields
Generalized Hamanujan-Selbery formula for counting integer points on algebraic varieties.
geophysics.:
Seismic attribute interpretation
Full waveform inversion
Gravity inversion
Spherical geometries
Ground penetrating radar imaging
Time-lapse reservoir monitoring
Electrical resistivity tomography
Joint inversion of geophysical data
Radiometric dating
Geomagnetic field modeling
H.6 AG News Attributes
H.6.1 Subtopics
We randomly select categories in AG News dataset and display the corresponding subtopic attributes for each category:
business:
Corporate earnings and financial reports
Stock market updates and analysis
Mergers and acquisitions
Business regulations and policies
Startups and entrepreneurship
Industry trends and forecasts
44

---

Economic indicators and market trends
Business strategies and management practices
Corporate governance and ethics
Consumer behavior and market research
Business leadership and executive profiles
Banking and finance industry updates
Energy and sustainability in business
Retail and e-commerce trends
Real estate and property market updates
Business disruptions and crisis management
Corporate social responsibility and sustainability initiatives
sci_tech:
Artificial intelligence
Robotics
Quantum computing
Biotechnology
Nanotechnology
Internet of Things
Renewable energy
Virtual reality
Augmented reality
Cybersecurity
Genetic engincering
Big data
Autonomous vehicles
3D printing
Blockchain technology
Bioinformatics
Machine learning
Biomedical engineering
Clean technology
sports:
Soccer
Basketball
Baseball
Tennis
Golf
Cricket
Rugby
Athletics
Formula
Olympics
Boxing
Swimming
Volleyball
Ice hockey
American football
Cycling
Motorsports
Martial arts
45

---

Horse racing
Surfing
world:
International politics and diplomacy
Global conflicts and war
Terrorism and security threats
Human rights issues and social justice movements
Migration and refugee crises
Climate change and environmental policies
Global health crises and pandemics
Natural disasters and emergencies
Cross-border crime and corruption
Cultural and social developments worldwide
Geopolitical tensions and territorial disputes
International aid and development efforts
Humanitarian crises and relief efforts
Cultural heritage preservation and promotion
International collaborations and partnerships
H.7
SST-2 Attributes
H.7.1
Subtopics
We randomly select 5 categories in SST-2/IMDB (movie review) dataset and display the corresponding subtopic attributes for each category:
positive:
Compelling Storyline: A strong and engaging narrative that captures the audience's 's attention from beginning end.
Well-Developed Characters: Memorable and relatable characters that evoke emotions and drive the story forward.
Skillful Direction: Effective direction that showcases the filmmaker's vision, ensuring cohesive storytelling and engaging visual elements.
Excellent Acting: Convincing performances from the cast that bring the characters to life and immerse the audience in the story.
Cinematography: Expertly captured visuals, including the use of framing, lighting, and camera movements, enhance the storytelling and create visually appealing experience.
Engaging Dialogue: Well-written dialogue that is natural, meaningful, and contributes to character development and plot progression.
Sound Design and Music: Thoughtful and immersive sound design, including sound effects and a well-curated soundtrack or original score, that enhances the overall cinematic experience.
Production Design: Attention to detail in creating visually appealing and authentic sets, costumes, and overall aesthetics that contribute to the film's atmosphere and world-building
Editing: Skillful editing that maintains good pace, effectively transitions between scenes, and enhances the overall flow and impact of the story.
Emotional Impact: A movie that evokes emotions, whether be through humor, drama, suspense, or other means, leaving a lasting impression on the audience.
negative:
Weak Plot: A poorly developed or uninteresting storyline that fails engage the audience.
46

---

Lackluster Performances: Unconvineing or uninspired performances by the actors that fail bring the characters life.
Poor Production Quality: Subpar production values, including low-quality visuals, amateurish cinematography, and weak special effects.
Incoherent Storytelling: Confusing disjointed narrative structure that makes it difficult to follow or understand the plot.
Unmemorable Characters:
Underdeveloped or forgettable characters that fail to resonate
with the audience.
Weak Soundtrack: A forgettable or poorly composed soundtrack that fails to enhance the mood or add depth to the movie.
Poor Dialogue: Uninteresting or poorly written dialogues that fail to engage or resonate with the audience.
Disjointed Aumosphere: A lack of coherence consistency in creating immersive and believable world for the viewers.
Unresolved Plotlines: Loose ends or unresolved plotlines that leave the audience feeling unsatisfied confused.
LLck of Entertainment Value: A movie that fails deliver enjoyable engaging experience for the audience, leaving them feeling bored or uninterested.
H.7.2 Descriptive Details
We movie genres the characteristics of movies, and the attributes are listed as follows:
Action
Drama
Comedy
Thriller
Romance
Horror
Adventure
Science Fiction
Fantasy
Animation
H.8
Yelp Attributes
H.8.1 Subtopics
We randomly select 5 categories in Yelp review dataset and display the corresponding subtopic attributes for each category:
positive:
Quality of Food: The taste, flavor, and presentation of the dishes.
Fresh Ingredients: The use of fresh and high-quality ingredients in the preparation of the food.
Menu Variety: A diverse range of options catering to different dietary preferences and restrictions.
The visually appealing presentation of the dishes. Service: Attentive, friendly, and prompt service from the restaurant staff.
Presentation:
Value for Money: Offering good quality and portion sizes at reasonable prices.
Cleanliness: A clean and well-maintained dining area, including tables, utensils, and restrooms.
Special Dietary Accommodations: Catering specific dietary needs such as vegetarian, vegan, gluten-free, etc.
47

---

Unique and Creative Dishes: Offering innovative and creative dishes that stand out. Efficient Operations: Smooth and well-coordinated operations to minimize waiting times and delays.
negative:
Poor Service: Slow or inattentive service from the restaurant staff. Unfriendly Staff: Rude unhelpful behavior from the restaurant staff.
Long Waiting Times: Excessive waiting times for a table or food.
Incorrect Orders: Receiving incorrect or poorly prepared food orders.
Unappetizing Presentation: Dishes that are poorly presented or lack visual appeal.
Unpleasant Ambience: Uncomfortable uninviting atmosphere the restaurant.
Dirty Unhygienic Conditions: Lack of cleanliness the dining area, restrooms, or utensils.
Limited Menu Options: A limited selection of dishes or lack of variety.
Poor Food Quality: Dishes that are poorly cooked, tasteless, or low quality. Overpriced: Excessive prices for the quality and portion sizes of the food.
H.8.2
Descriptive Details
We use cuisine types as the characteristics of restaurants, and the attributes are listed as follows:
Turkish
Spanish
Greek
Italian
French
American
Mexican
Canadian
Cajun
Tex-Mex
Brazilian
Peruvian
Argentinean
Colombian
Venezuelan
Ethiopian
Moroccan
South African
Nigerian
Egyptian
Chinese
Japanese
Indian
Thai
Korean
Australian
New Zealand
Polynesian
Hawaiian
Singaporean
48

---

I Examples for Filtered Attribute Values
Here we give some examples of the filtered attributes.
For the Amazon product review dataset, some filtered attributes are listed as follows.
beauty:
Hair Dryer (close to health and personal care)
Hair Straightener (close to health and personal care)
electronics:
Car dashcam (close to automotive) Wireless earbuds (close to cell pwwnessssrviieee service)
office products:
Mouse pad (close to electronics)
For NYT dataset, some filtered attributes are listed as follows:
american american_ football:
The economic impact of football on local communities and businesses.
The role of nutrition and hydration in optimal football performance
The history of football's ' integration with America's s armed forces
stocks _as__nd__hag bonds:
The role of artificial intelligence in stock market prediction
The effects of climate change on bond performance
movies:
The future of cinema in a post-COVID world
- The role of international markets in Hollywood's global success
For Reddit dataset, some filtered attributes are listed as follows:
ame stock:
Working in the stock market industry.
Impact of COVID-19 amc_stock.
Having background in finance or business.
what _arrt_aaaaaaa earr_aar___oae should i buy:
Towing capacity and trailer options
Financing and leasing options
Road trip and travel experiences
pittsburgh:
Attending a game at PNC Park
Watch a Steelers football game at Heinz Field
For StackExchange dataset, some filtered attributes are listed as follows:
algorithm:
Debugging an optimization algorithm for a complex logistics problem.
Improving the robustness of an encryption algorithm against attacks.
inding way to better handle overflow errors a cryptographic algorithm.
iPhone:
Unable to use certain features certain regions/countries
Trouble with GPS working properly
opengl:
DDifficulty setting up a custom camera in OpenGL
Inconsistent rendering of 3D models in OpenGL
Difficulty integrating OpenGL with cross game engine
49

---

Examples for Merged Attribute on the arXiv dataset
Here we give some examples of the filtered attributes.
optimization techniques for computer vision:
computer vision and pattern recognition
machine learning
black holes:
classical physics
popular physics
evolving networks for financial trading:
neural and evolutionary computing
computational finance
multi lulti-lingual information retrieval:
information retrieval
computation and language
iot operator systems and their security challenges:
operating systems
cryptography and security
complexity of social networks:
computational complexity
social and information networks
K
The Prompt Format for MetaPrompt
In this section, we provide more details about the implementation of MetaPrompt [45]. It first lever- ages LLMs provide additional task-specific information, and then augments the class-dependent prompt with the task-specific knowledge.
Prompts to ChatGPT: [Original Prompt]. What does this task ask us to do?
The task of generating an example of a NYT news asks the model to create high-quality, informative, and engaging news articles that mimic the writing style and structure commonly found in The New York Times (NYT).
Response from ChatGPT for NYT dataset:
The task of generating an example of a product review asks the model to create detailed and informative review for particular product, highlighting its features, performance, pros and cons, and overall user experience. This review should be helpful for potential buyers, providing them with valuable insights to make an informed purchasing decision.
Response from ChatGPT for Amazon dataset:
Response from ChatGPT for Reddit dataset: The task of generating an example of a post similar to Reddit asks the model to craft well-written and engaging post that adheres to the guidelines and community standards of the specific subreddit it's intended for. The post should spark discussions, share valuable information, or showcase interesting content that resonates with the subreddit's audience.
Response from ChatGPT for StackExchange dataset: The task of generating an example of post similar to StackExchange asks model to compose clear and concise question or answer related to specific topic within the given StackExchange community. The post should be well-researched and supported by relevant sources or personal expertise, aiming to provide valuable insights or solutions to other users' queries.
50

---

L Dataset Information
L.1 Dataset Documentations
The dataset is provided in json format; there are three json files corresponding to the original train, validation and test split. We also include two additional file, attrprompt and simprompt, which is generated by AttrPrompt and PimPrompt as the synthetic training data.
Each data point contains the following fields:
label: the label for the example. For multi-class classification, the label field is an integer, while for multi-label classification, the label field is a 1ist [int] containing one or multiple integers as each example may refer multiple classes;
text: content of each example.
L.2 Intended Uses
AutrPrompt and SimPrompt are intended for researchers machine learning, natural language processing, and related fields to innovate novel methods for training data generation problems.
L.3 Hosting and Maintenance Plan
The codebase is hosted and version-tracked via GitHub. will be available under the link https: //gi thub. hoomm/ueyy100000 attrrrptrp..mp.. The download link of all the datasets can be found in the Github repository.
Note that it is a community-driven and open-source initiative. We are committed and have the resources maintain and actively develop it for at minimum the five years. We plan grow the GitHub repo by including new tasks and datasets and warmly welcome external contributors.
L.4 Licensing
We license our work using Apache 2.0% All of the train/vallddation/teen data are publicly released by previous work [1, 32, 17].
L.5
Author Statement
We the authors will bear all responsibility in case of violation of rights.
https://www..ppooooooocccse/ll/l 51


Reviews:
[{"review_id": "PN9RZ9pRXd", "reviewer": "Reviewer_t81z", "summary": "", "questions": "", "limitations": "1. The appendix link are not working properly, it link to incorrect place. \n2. It will be better to add diversity metric to quantify the AttrPrompt performance over SimPrompt and Gold.\n3. Since there are multiple ways to design prompt, it will be better to explore more about the prompting ways and add more baselines in evaluation part. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper evaluated different data prompting methods and utilize generated data for model training. The experiments demonstrated utilizing AttrPrompt to generate synthetic training data can improve model performance in general. \n2. The topic raised in this paper is interesting and the paper is well written. \n3. The proposed approach may have more applications and impact. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VWHhmv98Re", "reviewer": "Reviewer_rTXq", "summary": "", "questions": "", "limitations": "Yes, See \"Opportunities for Improvement\" for making this part better.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- interesting and timely problem statement:  diversity of the generated data and inherit systematic biases of LLM. \n- investigation focusing on datasets with high cardinality and diverse domains.\n- interesting results over the baselines.\n- interesting analysis and observations on bias, diversity, and efficiency.\n- well written paper.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "1d42D1ubV0", "reviewer": "Reviewer_KfWS", "summary": "", "questions": "", "limitations": "The authors discuss their limitations and potential negative societal impact  in appendix.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Using language models to generate data for further use is an interesting and useful problem. How to design more powerful prompts that unlock the strong capability of LLMs for more useful applications is a decisive direction.\n\n-  The distribution of cosine similarity of text pairs is very informative and intuitive, clearly showing how having attributes in the prompts help to improve the generated text.\n\n- The experiments are", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cI3DCuXxyw", "reviewer": "Reviewer_mJxq", "summary": "", "questions": "", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "\n1. [Novel meta-info leveraging] The paper explores an innovative approach - using diversely attributed prompts instead of standard class-conditional prompts for generating training data, which is a relatively less explored area in the domain of large language models.\n\n2. [Performance improvement] It demonstrates that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance. \n\n3. [Cost efficiency] The study shows that attributed prompts achieve the same performance as class-conditional prompts while reducing the querying cost by 95%.\n\n4. [Comprehensive experiments and analysis] It offers a comprehensive empirical study on attributes such as bias, diversity, and efficiency in the generated data, providing a more in-depth understanding of potential issues and solutions in training data generation.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "O7sIHNpnKX", "reviewer": "Reviewer_upz4", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "It applies the paradigm of LLM-as-training-data-generator to multiple challenging datasets with higher cardinality compared to datasets used in prior work; it also takes the first step to apply such paradigm on multi-label classification task\nThis study delves into two important aspects of data generated by LLM, ie, diversity and bias, and reveal that existing class-conditional prompt could lead to severe bias in the generated data, which can be alleviated by attributed prompt\nEmpirical results show that attributed prompts not only have better model performance, but also better sample/budget efficiency\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors investigates the effect of diversely attributed prompts on LLMs for generating training data, in contrast to simple class-conditional prompts (simple prompts) which are widely used in practice. The authors suggest that the simple prompts could introduce bias and limit diversity. Through experiments, the paper demonstrates that attributed prompts outperform simple prompts in terms of trained model performance, bias mitigation, and data efficiency. 

Reviewers hold the consensus that the new ways of writing attributed prompts seem to be 1) data-efficient and promising, 2) address challenges such as bias and diversity in LLM-generated training data in a timely manner, and 3) analyses are informative for understanding the data-generation process.

Reviewers requested additional metrics, different types of comparisons, more baselines, and suggestions for other LLMs as data generators. It seems that the authors have faithfully responded to these in their rebuttals and reflected them already in the manuscript.
Decision: Accept (Poster)