AutoGeTS: Automated Generation of Text Synthetics for Improving Text Classification

Authors: Chenhao Xue, Yuanzhe Jin, Adrián Carrasco Revilla, Joyraj Chakraborty, Min Chen
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Text Classification, Synthetic Data, Data Augmentation, Large Language Model, Text Analysis, Optimization

Abstract:
When developing text classification models for real world applications, one major challenge is the difficulty to collect sufficient data for all text classes. In this work, we address this challenge by utilizing large language models (LLMs) to generate synthetic data and using such data to improve the performance of the models without waiting for more real data to be collected and labelled. As an LLM generates different synthetic data in response to different input examples, we formulate an automated workflow, which searches for input examples that lead to more "effective'' synthetic data for improving the model concerned. We study three search strategies with an extensive set of experiments, and use experiment results to inform an ensemble algorithm that selects a search strategy according to the characteristics of a class. Our further experiments demonstrate that this ensemble approach is more effective than each individual strategy in our automated workflow for improving classification models using LLMs.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730527598148

thread_type:
official_review

<thread_object>
creator: reviewer_qnux

<content>
summary:
The paper introduces a framework for using synthetic data generated by LLMs to improve text classification tasks, particularly for low-frequency classes, without the need for human intervention. They address the challenges of limited data for underrepresented classes with the framework of automatic selection. They explored three methods for selecting inputs, which are sliding window (SW), hierarchical sliding window (HSW), and genetic algorithm (GA). The methods target specific objectives, for example, class recall, class-based balanced accuracy, overall balanced accuracy, or F1-score. Experimental results showed that HSW is effective for larger classes and GA performs well for smaller and mid-sized classes. They also proposed an ensemble algorithm that combines these methods to further improve performance.

strengths:
- The paper introduces a framework that addresses the real-world challenges of class imbalance in text classification, a common issue in industrial applications, by leveraging synthetic data from LLMs to improve performance in low-frequency classes. The framework automates the selection of inputs for synthetic data generation, reducing the need for domain expertise.

- The paper tested three different strategies to gain insights into which methods work well under various conditions. Also, they considered ensemble algorithm for further improving performance.

weaknesses:
- The paper evaluates their proposed framework on only a single dataset, which limits the demonstration of AutoGets's effectiveness in other domains.  Additionally, the absences of baseline comparisons, for example, traditional data augmentation techniques, makes it challenging to fully understand the advantages of AutoGeTs. While HSW and GA methods are intended to reduce computation costs, further analysis is needed to determine how computational demands scale with larger datasets, as real-world industrial applications typically involve substantial data volumes.

questions:
.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659406619

thread_type:
official_review

<thread_object>
creator: reviewer_dzcv

<content>
summary:
This paper studies the text classification task by exploring text data synthesis, which is especially useful to solve the data scarcity issue for long-tail classes. This paper specifically studies how to select demonstrations for generating synthetic data. Three approaches are proposed to select examples from a embedding space of existing data samples, including a naive traversal of embedding space with static sliding windows, hierarchical top-down search, and a generic algorithm to iteratively update the selection. Experiments results mainly shows that the data synthesis method can significantly improve model performance on the infrequent classes.

strengths:
- This paper proposes three alternative methods for example selection from an embedding space to synthesize text data. Experiments compare these methods and their combinations to show their strength
- The proposed data synthesis method significantly improves classification performance on small-size classes, which shows the motivation of the paper
- Comprehensive experiments are conducted to compare the proposed three alternative methods and their combinations on different types of classes
- The paper overall is clearly written and easy to follow

weaknesses:
- Only one dataset with skewed distributed classes are used. More datasets could be beneficial to show the generalizability of the method
- In the experiments, the proposed data synthesis methods are only compared with the baseline classifier. However, there should be a baseline with a "naive" example selection method such as random or using local density. I expect such a method is also effective to improve performance on rare classes and the performance increment of the proposed method may be marginal.
- The presentation of experiment results is hard to read. The numbers are too compact and hard to directly see the comparison. The figures have too many small fonts and cannot easily see the content. Some major revision here should be necessary.

questions:
See the first two weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685253114

thread_type:
official_review

<thread_object>
creator: reviewer_pqjl

<content>
summary:
This paper explores the use of synthetic generations for retraining classification models with augmented data. The authors’ main contribution is through a selection pipeline for examples from the training set to be used in synthetic augmentation. They demonstrate their methodology through controlled experiments on class imbalanced settings.

strengths:
- The paper demonstrates a good level of technical descriptions for experiments, definitions and results.
- Experiment setups are elaborate and controlled across different aspects of the selection methodology and throughout the retraining pipeline.
- Elaborate presentation of overall and class-specific performance results.
- The appendix is appreciated, especially the inclusion of baseline results and PCA experiments for synthetic correctness.

weaknesses:
- A major issue for me is the lack of detail about synthetic generation in the main paper, this is an integral part of your contribution and yet it seems to be lacking in experimentation and presentation. 
- There seems to be a major focus on the selection process but not as much exploration for the improvement of generation prompts/pipelines given these selections, in fact, according to the appendix, the synthetic generation prompt seems to be very vanilla, with little prompt engineering or any measures to maintain diversity or even any experimentation on the later. The lack of entropy injection or verification in the prompt could present an issue with generalizability and scale, these issues need to be addressed as part of synthetic pipelines not only with post-hoc checks.
- Section 5.2 could use further detail on how you obtain the order of classes optimized for R1 and R2 measures as well as relevant classes for ICs in R3, this is a difficult and important part of the problem and it would’ve been helpful if it was discussed further in the main paper.

questions:
- Have you experimented with more models for generation?
- Have you performed any human annotations on a sample of your synthetic data in order to identify potential issues with diversity and correctness (is new data in fact in line with prompt examples)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732772857367

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Interim Version Revisions Update

comment:
Dear Reviewers,

We sincerely thank you for your time, effort, comments, and suggestions. While we continue improving the paper, we are submitting an interim version to report what has been done so far. The text revisions are marked in blue. Below, we outline the important revisions made.

1. In Section 3.1, we expanded the description of synthetic data generation (SDG) and provided further details in Appendix B.1 (including the SDG process, prompt template, format quality assurance of LLM responses, and parameter experiment results) to address Question 1 (W1) from Reviewer PqjL(1). Additionally in Appendix B.1, we mentioned the early experiments using Llama 3 for synthetic sample generation yielded comparable results as using GPT-3.5, addressing Question 4 (Q1) from the Reviewer PqjL(1).

2. In Section 3.3, we briefly described the limited performance of random example selection in the AutoGeTS workflow. We further detailed it in Appendix B.1.3, including its comparison with the three proposed example selection strategies (SW, HSW, and GA) under the same experimental setting. It can be considered as an additional baseline, addressing Q8 (W2) raised by Reviewer dzCV(2). The results show that random selection mostly underperforms the strategic methods, particularly for larger classes. Moreover, it yields only marginal overall improvements over the original model for classes T7, T13, T14, and T15, and even degrades the performance for class T10.

3. In Section 4.1, we briefly described the test of AutoGeTS workflow, in conjunction with the Easy Data Augmentation (EDA) tool [Wei & Zou,2019] for generating synthetic samples. EDA is a traditional data augmentation method. The testing results show that the EDA-based workflow underperforms in comparison with our LLM-based AutoGeTS workflow. We provided further comparative analysis in Appendix E, examining both the best improvements (overall and class-specific) and the temporal progression of overall balanced accuracy improvement across all 15 classes. The results show the LLM-based approach is always superior to the EDA-based approach, with this gap becoming larger when using strategic example selection methods (SW, HSW, GA) compared to random selection.

4. We have increased the line spacing in Table 2 in Section 4.2 and Table 3 in Section 5.1, and enlarged the font size in Figure 4 in Section 4.3 to enhance readability, addressing Question 9 (W3) raised by Reviewer dzCV(2). We have also checked the font size in all figures in appendices and improved those figures that had small fonts previously.

5. In Section 5.2, we clarified the class order determination for our ensemble algorithm based on Figure 6, and provided the detailed class orders for all three requirements (R1, R2, and R3) in Appendix C.2 with an illustrative Figure 28, addressing Question 3 (W3) from Reviewer PqjL(1).

6. In Section 5.3, we enhanced the presentation of the experimental results of ensemble AutoGeTS for improving class T13, and the discussion on the computational efficiency of HSW and GA methods. Accordingly, in Section 6 (Conclusions), we improved our presentation of the ensemble AutoGeTS algorithm and its contribution, addressing Question 12 (W3) from Reviewer qNuX(3).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733305964971

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
AutoGeTS Generalizability: New Results on TREC-6 Dataset

comment:
Dear Reviewers,

We greatly appreciate your thorough review and valuable feedback. While we continue to enhance our paper based on your suggestions, we would like to share the results of our additional experiments using the TREC-6 dataset, which demonstrate the generalizability of our AutoGeTS workflow. These experiments specifically address the suggestions by Reviewer dzCV(2) (Q7-W1) and by Reviewer qNuX(3) (Q10-W1). The complete results can be found in a new appendix, Appendix F, at https://anonymous.4open.science/r/ICLR_2025_Submission_10435-8BB7/ICLR_AutoGeTS_Appendix%20F_TREC-6%20Results.pdf.

The TREC-6 dataset contains 5542 fact-based questions distributed across six semantic classes, with varying class sizes from 86 to 1250 questions. Likely, because the overall size of the dataset is much smaller than the ticketing dataset used in our main experimentation, both EDA-based and LLM-based workflows achieved improvements compared to the baseline M0 model across different testing conditions, i.e.,

- selection strategies (random, SW, HSW, GA),
- objectives (6 classes, overall)
- metrics (global OBA, local CBA)

Meanwhile, our three selection methods (SW, HSW, and GA) showed better performance than random selection, with the LLM-based workflow performs consistently better than the EDA-based approach across all testing conditions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors introduce Automated Generation of Text Synthetics (AutoGeTS), an approach to synthetic data generation for text classification tasks. 

While this seems to represent a reasonable approach, the technical contribution is rather limited. This could be sufficient if the empirical evaluation were more thorough, but as is the evaluation relies on a single dataset (though a second was added in rebuttal, this is still only two datasets, which seems inadequate to me), and there is a lack of appropriate augmentation baselines (a point partially addressed in rebuttal, although only EDA was considered). These aspects greatly limit the scope of the contribution.