SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To determine the merging of subtopics, we utilize the fuzzy matching score<sup>7</sup>, which calculates the similarity between the subtopics. Setting the threshold at  $\gamma = 90$ , we proceed to merge subtopics whenever their similarity score exceeds this threshold. Consequently, specific subtopics may be associated with multiple classes, providing a more comprehensive representation of the document content. When generating text utilizing these subtopics, the corresponding labels are determined by the classes associated with the specific subtopics. This ensures that the generated examples accurately reflect the relevant classes based on the subtopic associations. The prompt format is deferred in section H.
C.1 Hardware information
All experiments are conducted on CPU: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz and GPU: NVIDIA GeForce RTX A5000 GPUs using python 3.8, Huggingface 4.6.0 and Pytorch 1.10.
C.2 Parameter configuration
We keep the parameter top_p = 1.0 and temperature  $t = 1.0$  for calling ChatGPT APIs [38] for the training data generation part. For finetuning the classifier, we optimize the model using AdamW [30] with a linear warmup of the first 5% steps and linear learning rate decay. The maximum number of tokens per sequence is 128. Table 11 lists the hyperparameters used for AttrPrompt and SimPrompt. For the generated synthetic dataset, we stick to the strict zero-shot learning setting [31], train all
$^{6}$ https://arxiv.org/  ${ }^{7}$ https://github.com/seatgeek/thefuzz
the models for 6 epochs and use the model from the last epoch without using the validation set for evaluation. For the original clean dataset, we train all models for 6 epochs and use the model with the best performance on the validation set for evaluation.
Table 11: Hyperparameters for fine-tuning classifiers on different tasks.
Backbone	Learning Rate lr	Batch Size	Training Epochs E	Weight Decay	Warmup Ratio
BERT-base-uncased [12]	5e-5	32	6	1e-4	6%
TinyBERT [23]	1e-4	32	6	1e-4	6%
DistilBERT-base-uncased [46]	5e-5	32	6	1e-4	6%
DeBERTa-V3-base [21]	5e-5	32	6	1e-4	6%
DeBERTa-V3-large [21]	2e-5	32	6	1e-4	6%
Table 12: The reference of model checkpoints used in this study.
Model	Link
BERT-base-uncased [12]	https://huggingface.co/bert-base-uncased
TinyBERT [23]	https://huggingface.co/huawei-noah/TinyBERT_General_4L_312D
DistilBERT-base-uncased [46]	https://huggingface.co/distilbert-base-uncased
DeBERTa-v3-base [21]	https://huggingface.co/microsoft/deberta-v3-base
DeBERTae-v3-large [21]	https://huggingface.co/microsoft/deberta-v3-large
SentenceBERT [44]	https://huggingface.co/sentence-transformers/all-mpnet-base-v2
SimCSE [16]	https://huggingface.co/princeton-nlp/sup-simcse-bert-base-uncased
COCO-DR [56]	https://huggingface.co/OpenMatch/cocodr-base-msmarco
C.3 Model checkpoint
All the checkpoint used in fine-tuning and linear probing experiments are listed in table 12.
D.1 Linear probing with generated datasets
In this section, we evaluate the generated dataset under the linear probing setting, where a linear classifier is trained over the frozen pretrained features. To make a comprehensive evaluation, we employ four text embedding methods, namely BERT-CLS [12], SentenceBERT [44], SimCSE [16] and COCO-DR [56], and present the results in Table 13. We can see that AttrPrompt consistently outperforms SimPrompt, MetaPrompt and surpasses the Gold in the NYT dataset. This further indicates the datasets generated with AttrPrompt has higher quality than two baselines and this superiority is robust to the model choices.
D.2 Additional results on attribute selection
Table 14 shows the result of AttrPrompt without human selection of attributes on two datasets. While a slight performance drop is observed, the result of AttrPrompt without attribute selection still significantly outperform the SimPrompt. This demonstrates the robustness of AttrPrompt performance to potentially irrelevant attributes.
D.3 Additional results on data/budget efficiency
In Section 5.3, we discuss the data/budget efficiency of AttrPrompt and SimPrompt on NYT and Amazon datasets; here, we provide additional results on the remaining datasets in Figure 8. We draw conclusions similar to what is stated in Section 5.3 that AttrPrompt exhibits better data/budget efficiency than SimPrompt.
D.4 Different temperature parameters for ChatGPT
Temperature  $(t)$  is one crucial hyperparameter of LLMs that controls the diversity of the generated text [20, 10], while the studied attributed prompts are also for diversifying the generated data. We are then curious about the effectiveness of the temperature and how it compares to the AttrPrompt. We
Table 13: Performance of linear probing with different pre-trained encoders.
Sequence Encoder	Method	NYT	Amazon	Reddit	StackExchange
Acc	F1	Acc	F1	Acc	F1	Acc	F1
BERT-CLS [12]	Gold	65.39	70.26	66.73	68.76	55.73	58.00	17.17	17.68
SimPrompt	60.69	62.26	37.95	38.07	27.41	27.60	12.27	13.71
MetaPrompt	58.43	57.05	27.52	24.43	23.93	23.99	11.40	12.60
AttrPrompt	62.38	64.17	48.18	49.46	45.03	45.95	17.33	17.64
SentenceBERT [44]	Gold	81.16	84.43	80.95	83.30	80.76	81.37	55.13	54.25
SimPrompt	84.32	85.91	67.55	68.29	69.94	70.62	45.37	47.79
MetaPrompt	87.65	87.11	67.16	65.78	79.26	69.15	45.71	47.54
AttrPrompt	88.77	88.36	68.10	69.11	71.62	71.73	46.80	48.35
SimCSE [16]	Gold	78.02	80.17	73.20	75.13	73.75	74.26	45.26	41.86
SimPrompt	78.93	78.86	52.40	52.83	54.78	53.86	30.98	32.92
MetaPrompt	78.17	78.27	53.53	52.55	51.30	48.93	30.54	31.50
AttrPrompt	81.41	81.04	60.66	61.32	59.22	59.15	30.31	31.43
COCO-DR [56]	Gold	80.66	82.60	76.13	78.34	79.63	80.04	61.02	63.77
SimPrompt	84.37	84.17	55.66	55.28	63.66	64.71	43.22	41.50
MetaPrompt	84.51	84.26	57.96	56.58	65.36	65.91	44.30	42.97
AttrPrompt	86.74	86.43	60.47	62.03	66.73	66.71	45.44	43.76
Average	Gold	76.31	79.37	74.25	76.38	72.47	73.42	44.65	44.39
SimPrompt	77.08	77.80	53.39	53.62	53.95	54.20	32.96	33.98
MetaPrompt	77.19	76.67	51.54	49.84	54.96	52.00	32.99	33.65
AttrPrompt	79.83	80.00	59.35	60.48	60.65	60.89	34.97	35.30
NYT	Amazon
GPT-3.5 Zero-shot	69.84	54.56
SimPrompt	76.34	56.96
AttrPrompt	82.26	65.87
AttrPrompt w/o selection	81.08	63.76
Table 14: The Performance of AttrPrompt with/without attribute selection.
(a) Reddit Budget
Figure 8: The comparisons on budget efficiency and data efficiency on Reddit and StackExchange.
(b) StackExchange Budget
(c) Reddit Data
(d) StackExchange Data
study different values of the temperature using the NYT dataset and present the results in Table 15. From the results, we can see that compared with the temperature, AttrPrompt brings more significant performance gain, demonstrating its superiority over temperature tuning.
Table 15: Study of the temperature.
Method	t=1.0	t=1.5	t=2.0
Acc.	F1	Acc.	F1	Acc.	F1
SimPrompt	76.00	76.34	76.78	77.31	76.55	77.42
AttrPrompt	81.30	82.26	81.47	82.23	79.47	79.86
D.5 The performance with respect to long-tail classes
As we have seen in Table 2, the original training sets of the involved datasets have severe long-tail classes issue since the imbalance ratio is high, yet the generated dataset are class-balanced, we are then curious how the class balance in the generated dataset benefits the model performance on long-tail classes. We take the NYT dataset as an example and plot the per-class F1 score of Gold, SimPrompt,
and AttrPrompt in Figure 9, where the x-axis is classes sorted by their number of data in the Gold dataset in descending order. From the figure, we can see that out of 26 classes, AttrPrompt renders the best per-class F1 score on 10 classes, which is 13 for Gold and 3 for SimPrompt. Notably, for classes with few examples in the Gold set (the rightmost 4 classes in the figure), AttrPrompt is better than the Gold and SimPrompt, especially for the class "abortion" with the fewest examples. This suggests a data-centric way to handle the long-tail class issue in topic classification: one may use LLMs to generate class-balanced training set or augment the existing training set with the LLM-generated data such that the augmented dataset is class-balanced, the in-depth study of which is left as future work.
Figure 9: Per-class F1-score of the NYT dataset.
D.6 Experiment results on other datasets
To provide a comprehensive evaluation of AttrPrompt, we conducted additional empirical studies using datasets commonly employed in prior works, which typically exhibit low cardinality. Consistent with [55, 54, 57], we focused on four specific tasks: IMDB, SST-2, Yelp, and AG News. The IMDB, SST-2, and Yelp datasets involve binary sentiment classification, while the AG News dataset entails news classification with only four classes. This selection allows for diverse evaluation across different classification scenarios.
To ensure a fair comparison, we utilized a smaller pre-trained language model (PLM) called Distill-BERT [46] as the classifier, in line with [55, 54, 15]. For the purpose of fine-tuning the classifier, we generated a total of 6000 examples, evenly distributed among the classes. For fine-tuning, we use a learning rate of  $2 \times 10^{-5}$  with 5 epochs and use the model after the final epoch for evaluation.
The results of our evaluation are presented in table 16. Overall, we observe that AttrPrompt demonstrates competitive performance across all four tasks. It outperforms our direct baseline (SimPrompt) by an impressive margin of  $3.8\%$ . Moreover, when compared to other baselines incorporating noisy-aware learning techniques, AttrPrompt still achieves promising performance, despite using the standard cross-entropy loss. This suggests that in addition to leveraging noisy-robust learning objectives, enhancing the diversity of input prompts serves as an effective alternative to improving downstream performance. It is interesting yet important to combine these different learning paradigms together to further boost the performance.
E Regional Bias with Manual Annotations
In the main body of the paper, we study the regional bias in the NYT dataset via an attribute classifier trained with the location attribute of the prompt associated with each generated data. Here, we manually annotate location for 100 samples from each of the Gold, SimPrompt, and AttrPrompt dataset, and compare manual annotations with the prediction of the attribute classifier in Table 17 (the first three rows). Note that we include "unknown" as an option in manual annotation to absorb text without clear location specification. From the results, we can see that the attribute classifier largely aligns with manual annotations, since the accuracy is greater than 0.95 on the Gold and AttrPrompt and 0.82 on SimPrompt; and the result on SimPrompt is lower since SimPrompt tend to generate more unknown location data.
With manual annotations, we also check how well ChatGPT can follow the required location attribute in the prompts. We compute the accuracy between the location attribute in the prompt associated with
Table 16: Results with recent baselines using DistilBERT [46].*: Results are copied from the previous papers [55, 54, 15, 57].
Method/Dataset	IMDB	SST-2	Yelp	AG News	Avg.
Simple fine-tuning with standard cross-entropy loss
SimPrompt	85.64	79.38	88.39	74.30	81.93
AttrPrompt	85.92	83.64	90.60	82.74	85.73
Other results with dedicated training techniques (for reference only)
Zero-shot Prompt*	77.31	82.63	78.10	73.20	77.81
ZeroGen*	82.77	80.41	87.84	76.48	81.88
SuperGen*	85.05	84.28	89.88	77.40	84.15
ProGen*	87.20	84.12	89.39	80.81	85.38
SunGen*	89.45	89.38	89.19	80.49	87.12
ReGen*	85.32	87.84	89.00	80.18	85.58
each data and that annotated by human on the 100 AttrPrompt samples (the last row of Table 17), which is 0.96. This means that ChatGPT is able to faithfully generate news with desired location specifications.
Table 17: Accuracies based on manual annotation of the location attribute of the NYT dataset.
Comparison	Data Source	Accuracy
Attribute Classifier v.s. Human	Gold	0.96
SimPrompt	0.82
AttrPrompt	0.95
Attribute in Prompt v.s. Human	AttrPrompt	0.96
We also plot the pie charts using manual annotations of location in Figure 10, and obtain similar findings as in Section 4.2, i.e., the data in Gold and SimPrompt are largely biased towards "North Amarica", while that of AttrPrompt is relatively balanced.
Figure 10: Pie charts of the "location" attribute based on manual annotations.
F Additional Bias Analysis
F.1 Biases other than regional bias
In addition to the regional bias we study in the main body of the paper, here we investigate bias in other datasets (Figure 11). From the results, we can see that the dataset generated with Gold and SimPrompt exhibit different level of biases, which shows that in general the biases exist in both generated and real dataset, while dataset generated with AttrPrompt has relatively balanced attribute distribution.
F.2 Bias analysis in AttrPrompt
Here, we compare the distribution of predicted "location" in the dataset generated with AttrPrompt to that with the Gold and SimPrompt in Figure 12. We can see that AttrPrompt renders a relatively balanced "location" distribution.
SimPrompt
(a) Amazon-Experience.
AttrPrompt
informative/educational
entertaining/funny
thought-provoking/discussion
personal/storytelling
(b) Reddit-Style.
help/advice
problem-solution
(c) StackExchange-Style.
Figure 11: Pie charts of predicted attribute distributions on different datasets.
tutorial-based
(a) All
(b) Tennis
(c) Soccer
(d) International Business
Figure 12: Pie charts of the distributions of "location" predicted by an attribute classifier for the NYT AttrPrompt dataset. (a) is "location" distribution over the whole dataset, while others are specific classes.
G The Prompt Format for LLM-Generated Attributes
G.1 Prompts for NYT Dataset
G.1.1 Key Attributes
Prompts to ChatGPT. What do you think are important attributes to generate some diverse news from NYT under a specific topic, such as Football, Federal Budget. Examples: subtopics, writing style...
Response from ChatGPT. Important attributes of news from the New York Times under a specific topic could include:
Subtopics: exploring different angles and subtopics within the main topic can lead to a wider range of news stories.
Writing style: varying the writing style can help to make the news stories more interesting and engaging for readers. This can include using different tones, perspectives, and sentence structures.
Sources: using a variety of sources, such as experts, eyewitnesses, and affected individuals, can provide different perspectives and add diversity to the news stories.
Geographical locations: covering the topic from different geographical locations can help to show how it affects different communities and provide a global perspective.
Timeliness: covering current events and breaking news related to the topic can add diversity and relevance to the news stories.
Multimedia: using a range of multimedia elements, such as images, videos, and infographics, can help to add variety and appeal to the news stories.
Length: varying the length of the news stories, from short and concise to longer and more in-depth, can help to provide a wider range of information and cater to different readers' preferences.
Selected Attributes
1. Subtopics
2. Writing Styles
3. Geographical locations (determined by six continents by default)
4. Length (determined by  $\{30 - 80\}$  and  $\{100 - 150\}$  by default)
G.1.2 Attributes Details
Prompts to ChatGPT for Similar Classes. List 5 similar classes for {class-name} news on NYT. The set of classes is listed as: {{a list of class-names}}.
Prompts to ChatGPT for Subtopics. List 10 diverse subtopics for {class_name} news on NYT.
Response from ChatGPT. See section H for details.
G.1.3 Writing Styles
Prompts to ChatGPT. List 5 different writing styles of NYT news.
Response from ChatGPT.
1. Investigative reporting
2. Opinion/editorial
3. Human interest/profiles
4. Long-form narrative
5. Breaking news
G.2 Prompts for Amazon Dataset
G.2.1 Key Attributes
Prompts to ChatGPT. What do you think are important attributes to generate some diverse product reviews. Examples: writing style...
Response from ChatGPT. When writing a review for a product, here are five important attributes to consider:
Product Type: Clearly mention the type of product you are reviewing, such as a smartphone, laptop, or fitness tracker. This helps readers understand the category and purpose of the product.
Brand: Specify the brand of the product as it often influences quality, reputation, and customer support. Discuss the brand's overall credibility and whether it aligns with your expectations.
User Experience: Evaluate the overall user experience of the product. Discuss its speed, accuracy, reliability, and efficiency in performing its intended tasks. Highlight any exceptional or lacking performance aspects.
Quality and Durability: Assess the quality of the product, including the materials used, construction, and overall durability. Discuss whether it feels well-made, solid, and likely to withstand regular use over time.
Features and Functionality: Describe the specific features and functions of the product. Highlight any unique or standout features that enhance its usability or set it apart from similar products in the market.
G.2.2 Selected Attributes
1. Product Name
2. Brands
3. User Experience
4. Writing Styles
5. Length
G.2.3 Attributes Details
Prompts to ChatGPT for Similar Classes. List 5 similar classes for {product-class} products on Amazon. The set of classes is listed as: {{a list of class-names}}.
Prompts to ChatGPT for Product Name/Brands. List 20 diverse product names/brands for {product-class} on Amazon.
Response from ChatGPT. See section H.2 for details.
G.2.4 Usage Experience
Prompts to ChatGPT. List 5 different usage experiences of Amazon reviews.
1. Worst: A highly unsatisfactory or problematic experience. Users may encounter multiple issues, such as usability problems, errors, and frustrations.
2. Bad: A below-average experience. Users may encounter some difficulties or frustrations while interacting with the product or service, but the overall experience falls short of meeting their expectations.
3. Average: A neutral or middling experience. Users may not encounter major issues, but the experience lacks any exceptional aspects or standout features.
4. Good: A positive and satisfactory experience. Users have a smooth and enjoyable experience with minimal issues or obstacles.
5. Excellent: The highest level of satisfaction. Users have a seamless, delightful, and memorable experience.
G.2.5 Writing Styles
Prompts to ChatGPT. List 5 different writing styles of Amazon reviews.
1. Detailed Review: A detailed review provides a comprehensive analysis of the product, covering various aspects such as features, performance, durability, and user experience.
2. Comparative Review: A comparative review compares the product with similar options available on Amazon.
3. Pros and Cons Review: A pros and cons review presents a balanced assessment of the product by listing its positive and negative aspects.
4. Personal Experience Review: A personal experience review shares the reviewer's firsthand experience with the product.
5. Recommendation Review: A recommendation review offers a clear recommendation or endorsement of the product.
G.2.6 Prompt Template
See table 18 for details.
Table 18: Prompt template for the Amazon Product Review dataset.
Method	Prompt
SimPrompt	Suppose you are a review writer. Please write a review for {product-class} in Amazon.
AttrPrompt	Suppose you are a review writer. Please write a review for {product-class} product in Amazon following the requirements below:1. The review should be about the product of {subtotal};2. The brand for the product should be {brand};3. Should be in length between {length:min-words} and {length:max-words} words;4. Should describe the usage experience {usage-experience};5. The writing style of the review should be {style};6. the review must be relevant to {product-class} and irrelevant to: {similar-class}.
G.3 Prompts for Reddit Dataset
G.3.1 Key Attributes
Prompts to ChatGPT. List 5 important attributes to generate some diverse reddit posts under a specific topic, like build_a_pc, premed, etc. Examples: subtopics, keywords, writing style...
Response from ChatGPT. Informative Guides and Tutorials: Create posts that provide step-by-step guides, tutorials, or informative content. These can include detailed instructions, tips, or recommendations to help readers in their journey. For example, in "build_a_pc," you could create a guide on assembling a gaming PC or optimizing system performance.
Writing Style: Utilize different writing styles to create diverse posts. For instance, you can write informative guides or tutorials, opinion pieces, comparison reviews, troubleshooting tips, or personal stories related to the topic.
Discussion Starters: Pose thought-provoking questions or statements to initiate discussions and engage the Reddit community. These can be open-ended queries, asking for opinions or experiences, or presenting a controversial viewpoint to encourage debate and dialogue.
Personal Experiences: Share personal stories or experiences related to the topic. This allows you to connect with the community on a more personal level and provide insights based on your firsthand experiences. It can include successes, challenges, lessons learned, or interesting anecdotes.
Table 19: Prompt template for the Reddit dataset.
Method	Prompt
SimPrompt	Suppose you are a writer for posts in Reddit. Please give a post for {post-class} in Reddit.
AttrPrompt	Suppose you are a review writer. Please give a post in Reddit on {post-class} community following the requirements below:1. The post should focus on {experience};2. Should be in length between {length:min-words} and {length:max-words} words;3. The writing style of the post should be {style};4. Should mention the resource of {resource};5. the post must be relevant to {post-class} and irrelevant to: {similar-class}.
Resources and Recommendations: Provide valuable resources such as articles, websites, tools, or recommended products/services related to the topic. These resources can help the community access additional information, useful tools, or trusted recommendations, enhancing their understanding and engagement with the subject.
G.3.2 Selected Attributes
1. Personal Experiences
2. Resources
3. Writing Styles
4. Length
G.3.3 Attributes Details
Prompts to ChatGPT for Similar Classes. List 5 similar classes for {post-class} products on Reddit. The set of classes is listed as: {{a list of class-names}}.
Prompts to ChatGPT for Personal Experiences/Resources. List 20 diverse Personal Experiences/Resources for posts about {post-class} in Reddit.
Response from ChatGPT. See section H.3 for details.
G.3.4 Writing Styles
Prompts to ChatGPT. List 5 different writing styles of Reddit Posts.
1. Informative/Educational: Create posts that aim to provide valuable information or educate the community on a particular topic. Share informative articles, research findings, guides, tutorials, or explain complex concepts in a simplified manner.
2. Entertaining/Funny: Craft posts that are meant to entertain or bring humor to the community. Share funny stories, jokes, memes, amusing videos, or any form of content that can elicit laughter and enjoyment.
3. Thought-Provoking/Discussion: Initiate posts that spark thoughtful discussions and invite users to share their opinions or experiences. Pose open-ended questions, present philosophical dilemmas, discuss current events, or explore societal issues to foster engaging conversations.
4. Personal/Storytelling: Share personal experiences, anecdotes, or narratives that others can relate to. Tell compelling stories, share achievements, challenges, or life lessons that can inspire or resonate with the community.
5. Help/Advice: Create posts where you seek help or offer advice to the community. Ask for guidance on a specific problem, seek recommendations, provide support, or share expertise in a particular field.
G.3.5 Prompt Template
See table 19 for details.
G.4 Prompts for StackExchange Dataset
G.4.1 Key Attributes
Prompts to ChatGPT. What do you think are important attributes to generate diverse posts under a specific topic on the stackexchange forum, like 3D meshes, server, etc. For example, one important attribute could be writing style...