Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification

Authors: Hsun-Yu Kuo, Yin-Hsiang Liao, Yu-Chieh Chao, Wei-Yun Ma, Pu-Jen Cheng
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: data weighing, data augmentation, distillation, data-efficient training, nlp in resource-constrained settings, fine-tuning, weighted loss

Abstract:
Synthetic data augmentation via Large Language Models (LLMs) allows researchers to leverage additional training data, thus enhancing the performance of downstream tasks, especially when real-world data is scarce. However, the generated data can deviate from the real-world data, and this misalignment can bring about deficient results while applying the trained model to applications. Therefore, we proposed efficient weighted-loss approaches to align synthetic data with real-world distribution by emphasizing high-quality and diversified data generated by LLMs using merely a tiny amount of real-world data. We empirically assessed the effectiveness of our methods on multiple text classification tasks, and the results showed that leveraging our approaches on a BERT-level model robustly outperformed standard cross-entropy and other data weighting approaches, providing potential solutions to effectively leveraging synthetic data from any suitable data generator.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729630494390

thread_type:
official_review

<thread_object>
creator: reviewer_e1ke

<content>
summary:
This paper proposes a modification to the loss function to improve the performance of models trained on synthetic data, making it comparable to models trained on real-world data. This approach addresses the challenge of limited availability of real-world data, where models trained solely on synthetic data often underperform. The authors introduce two new loss functions—IMP-Loss and DIMP-Loss—which incorporate mechanisms for quality-checking and diversity-checking the synthetic data. IMP-Loss prioritizes synthetic samples that closely resemble real-world distributions, while DIMP-Loss emphasizes samples that guide the model toward the performance of a small base model trained on limited real-world data. Experimental results demonstrate that these loss functions enhance the performance of models trained on synthetic data, outperforming traditional baseline methods.

strengths:
Overall, this paper is well-written, addressing a compelling and practical problem for researchers working with synthetic data. The issue of model performance degradation when using synthetic data is aligned with recent research findings. The proposed loss functions IMP-Loss and DIMP-Loss are creative and effective solutions.

The clarity of the paper is a strong point, with a well-defined problem formulation and a meticulous derivation of the proposed loss functions. Each step in the reasoning process is clearly outlined, making the theoretical contributions easy to follow.

The paper includes extensive experimentation, which demonstrates the efficacy of the proposed methods. The analysis of the runtime complexity further adds the practical applicability of the algorithm in real-world scenarios.

weaknesses:
The core idea of weighting synthetic data samples is not entirely novel. A similar concept was explored in the paper "Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning" published last year. Although the formulation and implementation in this paper differ, the underlying principle of prioritizing certain synthetic data points is conceptually similar, which somewhat limits the novelty of the contribution.

Another concern lies in the strong reliance on the assumption that a model trained on a small real-world dataset P can approximate the performance of a model trained on a larger, complete dataset. The paper would benefit from a more in-depth discussion or experimentation addressing this assumption. Specifically, it would be helpful to explore the minimum number of real-world examples required for this assumption to hold and to provide guidelines on the amount of data practitioners should aim to collect.

questions:
In some formulas, it's unclear whether the pairs (xi, yi) are drawn from the real-world distribution P or the synthetic data distribution Q. For instance, in Appendix C, it is clearly noted that (yp', xp') is from distribution P. Could you use similar notation consistently throughout the main body to help readers differentiate between real and synthetic data sources?

To improve clarity, consider starting a new paragraph with the sentence: "Thus, inspired by the objective of Hu et al. (2019), we introduce two novel, efficient, automatic weighted-loss approaches: Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP-Loss)..." This will make the contributions of this paper stand out more clearly, preventing them from being mixed with discussions of prior work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730104756033

thread_type:
official_review

<thread_object>
creator: reviewer_eilk

<content>
summary:
This paper proposes a reweighting scheme for synthetic data generated by LLMs. The focus is on the distinction between the distributions of real and synthetic data. To address this, the authors suggest IMP-Loss and DIMP-Loss to align the distribution of synthetic data with that of real data. These two loss functions assign weights to the cross-entropy loss for each data point. The empirical results support the effectiveness of this approach.

(Note: my review has been revised by LLM for improving the grammar.)

strengths:
- The use of synthetic data has been of significant interest to researchers for years. It is also important to note that noise in synthetic data has been a long-standing concern. While several studies have proposed methods to address this issue, they come with their own limitations, as the current manuscript notes. This work is timely in addressing those previous limitations.
  - While the proposed method requires some additional computation for assigning weights, I agree that it is more efficient compared to methods based on meta-learning, at least.
  - As a result, this method seems more practical than previous approaches for mitigating the noise in synthetic data.

weaknesses:
- **Most of my concerns are resolved during the rebuttal process.**

- ~~I see several issues with the introduction and related work section.~~
  - The authors primarily build their argument by citing studies focused on generating synthetic data for *instruction tuning of LLMs*, such as Alpaca.
  - However, the method proposed in this paper is aimed at mitigating noise in synthetic data for *text classification*, which might create some confusion.
  - The confusion arises because there are many studies focused on generating synthetic data specifically for text classification, which are more relevant to this work, but the authors do not cite them.
    - Meng et al., Generating Training Data with Language Models: Towards Zero-Shot Language Understanding, NeurIPS 2022.
    - Meng et al., Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning, ICML 2023.
    - Gao et al., Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning, ICLR 2023.
    - Choi et al., UniGen: Universal Domain Generalization for Sentiment Classification via Zero-Shot Dataset Generation, arXiv Preprint 2024.
    - Zou et al., FuseGen: PLM Fusion for Data-Generation-Based Zero-Shot Learning, arXiv Preprint 2024.
  - I believe discussing these studies would be more relevant compared to the current organization of the manuscript.
- ~~Accordingly, the authors should have discussed these works and used them as baselines.~~
  - For instance, SunGen (Gao et al., 2023), UniGen, and FuseGen propose methods for mitigating the noise in synthetic data.
  - SunGen, in particular, is very similar to this work, as it suggests reweighting synthetic data based on quality. While SunGen is based on meta-learning and may require more computation than the proposed method, the current organization of the manuscript does not adequately address this overlap.
  - Furthermore, FuseGen suggests a reweighting scheme that does not require additional computation, unlike the proposed method.

~~Given these points, I do not believe this paper is ready for publication at ICLR in its current form.~~

questions:
- I am still uncertain whether the proposed diversity checker functions as intended. What happens if unrelated data is fed into the diversity checker? SunGen has already demonstrated that noise in synthetic data is not limited to data with noisy labels but also includes data unrelated to the desired domain.
- Some comments regarding the paper's formatting:
  - Line 030: Use a capital letter for the abbreviation of LLM. Currently, it reads 'Large language models,' which is inconsistent with other terms like 'Natural Language Processing.'
  - Line 063: Ensure proper spacing between ‘ers’ and ‘highly.’
  - Footnote 1: Start the sentence with a capital letter, and ensure there is a period at the end.
  - Please ensure consistent use of \citet and \citep, as this inconsistency currently reduces the overall quality of the paper. For example:
    - Line 033: Use \citet for Taori et al.
    - Line 050: Use \citet for Hu et al.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730185808292

thread_type:
official_review

<thread_object>
creator: reviewer_r1bf

<content>
summary:
Since the development of LLMs, LLM-generated data are widely used for training models. However, the distribution of such artificial datasets may be misaligned from real-world datasets, creating concerns. 
Therefore, during the training process, to prioritize data that is more diverse and has higher quality, the authors develop an approach to weighing LLM-generated data according to its quality and diversity. Compared to previous data-filtering techniques that only use a subset of generated data, this approach leverages all the training data while giving nonuniform weights to the loss of each data point.
Specifically, the authors theoretically define two losses: IMP-Loss and DIMP-Loss. IMP-Loss applies WCE-Loss, where the weight function is compounded by a quality checker (approximated with P, estimated by fitting a model on D_P, the real-world dataset) and a diversity checker (approximated with the inverse of Q, estimated by fitting a model on D_Q, which is the synthetic dataset). DIMP-Loss simplifies the calculation by estimating the diversity checker with the current model in training.
As a result, the performance using the newly devised loss improves compared to the baselines, with DIMP Loss falling slightly behind IMP loss, as a compromise for lower costs.

strengths:
1. The theoretical derivation of IMP and DIMP loss is clear and sufficient.
2. The empirical improvement shown in table 1 is consistent.
3. There are multiple studies to verify the robustness of the method: different datasets in figure 1, different model sizes in table 2, and using various percentages of the training set in figure 2.
4. Section 3.2 shows the motivation for employing diversity and quality in the weighting strategy, laying the foundation for the methodology developed later in the section. 
5. The writing and structure is clear to follow.

weaknesses:
1. The results are derived using a BERT-Based model, which is quite small compared to contemporary LLMs. How would the method scale to larger LLMs? How would the costs grow as the model size grows?

questions:
1. For the metrics "diversity" and "quality", would it be possible to verify that they actually correspond to what they refer to? I.e., does higher quality data actually get higher quality metrics? Currently, it seems like a term for the mathematical expression. While I understand it intuitively, it would also be nice to verify them empirically.
2. In the tweet irony case in Figure 1, DIMP loss start out with worse performance than other baselines for the first and second epoch. Could you explain why such a scenario would occur? Does this mean that these methods introduce instability to the training process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731266866865

thread_type:
official_review

<thread_object>
creator: reviewer_gcsy

<content>
summary:
This paper introduced IMP-Loss and DIMP-Loss as novel weighted-loss objectives to enhance the performance of models trained on LLM-generated data. Paper showed results demonstrating that both methods outperformed traditional loss functions CE-loss, focal loss, Meta-learning weighting on 3 NLP tasks Financial Phrasebank, Tweet Irony and MRPC.

Both IMP-Loss and DIMP-Loss include quality over diversity as weighting per training example. Quality weight is the predicted conditinoal probability from a BERT-base model trained on a small real world data of size 200-400 samples. Diversity for IMP-Loss is similarly trained model but on synthetic data. Diversity for DIMP-Loss is the current model's predicted conditional probability. IMP-Loss is based on importance sampling, DIMP-loss is motivated by online batch selection methods.

For experiments, paper compared training on synthetic data alone, noisy synthetic data (label flipped, duplicated data entry). In both case, IMP-Loss and DIMP-Loss work better than the quality checker and three other loss functions. Paper also applied the two weighted loss function to real data, where the results showed better than training on real data using the other loss functions too.

strengths:
- The  IMP-Loss and DIMP-Loss  approaches are novel contributions. The approaches are clearly motivated. Derivation are clearly demonstrated step by step. Approximations are used to make the implementation efficient.
- The experiments covered several scenarios, including small real data, synthetic data, noisy data and real data. In all cases, the introduced methods work best, showing the robustness of the approaches. Several oblations are interesting: increasing real data for quality checker increases final performance. Diversity check is very important for final performance.

weaknesses:
The paper introduces two weighting scheme for synthetic data. The experiments can be made more convincing:
- The three benchmarks can be expanded to more NLP tasks. Eg nn the ZEROGEN paper cited here there are several benchmarks: sQuAD, QNLI and AdversarialQA, which are particularly challenging for synthetic data generated from LLM. The performance was far below supervised results. It will be interesting to apply the weighting method to see if the gap can be reduced for these challenging benchmarks.
- The paper used 5 epochs for training the downstream models by Line 1101. It is unclear whether the models have converged. It is interesting to not only see weighting proposed here can speed up learning, but what happens at convergence.
- The weighting method does not seem to achieve SOTA for the two benchmarks. It would be more convincing if the paper adopted a strong baseline rather than BERT-large to build on top and compare with SOTA results.
  - mrpc https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc, tinybert is at 86.4%.
  - financial phrasebank https://paperswithcode.com/sota/text-classification-on-financial-phrasebank. 96.7% for distillbert.

questions:
- Paper assumes Q(x) ~ P(x) on line 171, but in line 154-164, the synthetic data has high diversity and entropy. Wondering how to interpret the two together?
- Synthetic data includes the label for the data. For Q(x) ~ P(x) on line 171, does that means Q(x, y) ~ P(x, y)? Seems the paper mix Q(x, y), Q(y|x) eg at line 942 too.
- Line 999 missing ', so it should be x_{p'}, right?
- The table 1 shows Focal-loss and Hu et al.'s method did not work better than CE-Loss, esp for the "large real world" case, could you comment on the reason?
- For the DIMP-Loss at line 274, if we apply it to large real data, where P' is also trained on the same large real data. In this case, the weighting will converge to 1, is that correct? In that case, the final performance at convergence should be the same as CE-Loss, right? How does this case affect table 1?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces IMP-Loss and DIMP-Loss, two novel weighted-loss functions designed to improve the performance of models trained on LLM-generated synthetic data. The approach addresses challenges such as the misalignment between real-world and synthetic data distributions and the limited availability of real-world data. There are a few key contributions. First, the Loss Function Design: author proposed IMP-Loss based on importance sampling, to assign weights using both quality and diversity metrics; authors also proposed DIMP-Loss, inspired by online batch selection methods, to simplify diversity estimation by using the current model’s predictions, trading slight performance for reduced computational cost. Both loss functions assign nonuniform weights to each data point instead of discarding data, unlike traditional filtering techniques. Quality and Diversity Metrics is another contribution. Derived from the conditional probabilities predicted by a BERT-base model trained on a small real-world dataset (200–400 samples), Quality Weights are obtained. Both losses are also showed to obtain Diversity Weights. Solid experimental Results are provided to show efficacy of proposed methods. Both IMP-Loss and DIMP-Loss consistently outperformed traditional loss functions (e.g., cross-entropy loss, focal loss, and meta-learning weighting) and even improved performance on real-world data. This study highlights how prioritizing quality and diversity in synthetic data training can bridge the performance gap between models trained on synthetic versus real-world data, making synthetic data more viable for machine learning tasks. 


Strength of this paper

Novelty and Motivation: the introduction of IMP-Loss and DIMP-Loss. The motivations for incorporating quality and diversity in the weighting strategy are well-articulated and grounded in prior challenges with synthetic data.
Clarity and Structure: The paper is well-written and clearly structured, with a well-defined problem formulation and step-by-step derivation of the proposed methods.
Efficiency and Practicality: The use of approximations improves computational efficiency, making the approach more practical than meta-learning-based alternatives. Runtime complexity analysis demonstrates the applicability of the methods in real-world scenarios.
Empirical Validation: Comprehensive experiments validate the efficacy of the methods, with consistent performance improvements shown in results tables and figures. The empirical results are compelling, showing the superiority of the proposed methods over baselines across multiple scenarios.
Timeliness and Relevance: The work addresses a long-standing issue of noise in synthetic data, building on prior research while mitigating its limitations.

Weakness of this paper

Several reviewers raised few concerns/limitations of this paper. By addressing these limitations, the paper could strengthen its experiment and expand impact.
1. Convergence Uncertainty: The experiments use only five training epochs, and it is unclear whether the models have fully converged. Analyzing the proposed weighting method's impact on convergence speed and performance at full convergence would add depth to the results.
2. Scalability to Larger Models: The experiments rely on a BERT-base model, which is relatively small compared to contemporary large language models (LLMs). It is unclear how the method scales to larger models in terms of performance improvements and computational cost.
3. Novelty Concerns: The core idea of weighting synthetic data is not entirely new, as similar concepts have been explored (e.g., in "Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning"). While the formulation and implementation here are distinct, the conceptual similarity somewhat limits the novelty.