Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification

## Abstract

Abstract Synthetic data augmentation via large language models (LLMs) allows researchers to leverage additional training data, thus enhancing the performance of downstream tasks, especially when real-world data is scarce. However, the generated data can deviate from the real-world data, and this misalignment can bring deficient outcomes while applying the trained model to applications. Therefore, we proposed efficient weighted-loss approaches to align synthetic data with real-world distribution by emphasizing high-quality and diversified data generated by LLMs with using merely a little real-world data. We empirically assessed the effectiveness of our method on multiple text classification tasks, and the results showed leveraging our approaches on a BERT-level model robustly outperformed standard cross-entropy and other data weighting approaches, providing potential solutions to effectively leveraging synthetic data from any suitable data generator for model training.

## 1 Introduction

The quantity and quality of data play a significant role in many tasks of Natural Language Processing (NLP). However, due to the scarcity of data in a particular domain for a specific task, we may need expertise to collect such data, resulting in budget limitations. Fortunately, Large language models (LLMs) provide a practical solution to this problem. LLMs, such as GPT series (Brown et al., ; OpenAI, ; OpenAI et al., ), can be leveraged to generate synthetic data that mimics real-world examples, thereby enriching the training set (Wang et al., ). (Taori et al., ), and other works have shown the capability of using LLM-generated data for the downstream tasks, and it seems to be a new cut-in solution to any NLP downstream tasks. However, training models with LLM-generated data can lead to drawbacks such as model collapse (Shumailov et al., ; Dohmatob et al., ), tail phenomena, reinforcing LM biases (Wang et al., ). Moreover, based on our empirical study, the performance of models trained on synthetic data without proper processing can be lower than models trained on much smaller real-world data (Sec. 3.1), highlighting the uncertainty of using LLM-generated data.

Previous works took data filtering strategy to get high quality or variant data (Dubey et al., ; MetaAI, ; Chiang et al., ; West et al., ). Still, this strategy was mostly human-crafted or needed efforts to train a judge model, and most importantly, filtering strategies abandoned the potential of the filtered data that may contribute to the final performance.
In contrast, data weighting approaches leverage all the training data, including augmented and biased data, but prioritize data by giving nonuniform weights to the loss of each data point. For example, Focal-Loss (Lin et al., ) prioritized more diverse data, and Hu et al. () optimized the weights of training samples so that the model performs best on a small real-world dataset. It is worth noting that while Hu et al. () steered the training toward higher performance, using this method in a large-scale scenario seems infeasible because it took the weights as learnable parameters, the number of which increased as the training set scale. Thus, inspired by the objective of (Hu et al., ), we introduce two novel, efficient, automatic weighted-loss approaches: Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP-Loss), which are designed to closely align the distribution of synthetic data with that of real-world data. Furthermore, both IMP-Loss and DIMP-Loss incorporate mechanisms as quality-checkers and diversity-checkers, assigning higher quality and more diverse data points with greater weight. In other words, these methods prioritize data points that are more relevant and more informative to target downstream tasks, thereby reducing the impact of less valuable data on the fine-tuning model.

To validate our approaches, we conduct comprehensive empirical studies focusing on various text classification tasks by comparing the performance of models trained with our novel weighted-loss objectives under different conditions: 1) models trained exclusively on LLM-generated datasets using few-shot prompts from a limited real-world dataset (Sec. 5.1); 2) models trained on large, real-world datasets (Sec. 5.2); and 3) models trained on noisy datasets (Sec. G). Our findings indicate that using a small real-world dataset to build the quality checkers and incorporating diversity checkers highly enhances performance, even surpasses the few-shot prediction accuracy of the tremendous data generator (Sec. 5.1). This demonstrates the efficacy of our methods in leveraging little real-world data to improve models trained on LLM-generated datasets. Notably, DIMP-Loss is efficient in terms of model size (Sec. 5.1), data requirements (Sec. 2), and computational resources (Sec. 4.4), making it a practical solution to enhance downstream performance.

## 2 Preliminaries

### 2.1 Cross Entropy Loss on Real-World Dataset

In supervised learning for text classification tasks, we consider a real-world dataset $D_{P}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{M}$ comprising $M$ samples. Each pair $(\mathbf{x}_{i},y_{i})$ is drawn independently and identically distributed (i.i.d.) from the joint distribution $P(\mathcal{X},\mathcal{Y})$, where $\mathbf{x}_{i}$ represents the input sample and $y_{i}\in\mathcal{Y}=\{1,2,\ldots,C\}$ is the corresponding class label.

This setup forms the basis for training models using the empirical cross-entropy (CE-Loss) as the loss, a standard objective function in such tasks. The CE-Loss over the entire dataset $D_{P}$ is calculated as follows:

$$ $\displaystyle\mathcal{L}_{\text{CE}}(\theta,D_{P})=-\frac{1}{M}\sum_{i=1}^{M} \log\hat{P}(y_{i}|\mathbf{x}_{i};\theta)\overset{p}{\to}\mathbb{E}_{P}\left[- \log\hat{P}(y|\mathbf{x};\theta)\right]$ (1) $$

where $\hat{P}(y_{i}|\mathbf{x}_{i};\theta)$ is the predicted probability of the model with parameters $\theta$ for the true class label $y_{i}$ given input $\mathbf{x}_{i}$. The CE-Loss converges in probability to the expected version of conditional cross-entropy under the true joint distribution $P(\mathcal{X},\mathcal{Y})$ by the law of large numbers. This convergence is crucial because the minimizer of the CE-Loss occurs if and only if the predicted distribution $\hat{P}(y|\mathbf{x};\theta)$ matches the true distribution $P(y|\mathbf{x})$.

### 2.2 Weighted Cross Entropy Loss (WCE-Loss)

WCE-Loss is a modification of the standard CE-Loss that assigns different weights to each data point. It is defined as:

$$ $\displaystyle\mathcal{L}_{\text{WCE}}(\theta,D_{P},w)$ $\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}w_{i}\log\hat{P}(y_{i}|\mathbf{x}_{i};\theta)$ (2) $$

Here, $w_{i}$ represents the weight assigned to the $i$-th data point $(\mathbf{x}_{i},y_{i})$. A higher weight $w_{i}$ assigned to a data point $(\mathbf{x}_{i},y_{i})$ indicates that the data point is considered more important for the training process, thereby having a greater influence on the model’s learning or adjustment of parameters.

There have been several variants of the weight function, such as Focal Loss $\mathcal{L}_{\text{FL}}$ (Lin et al., ). It addressed class imbalance and reduced the impact of easily classified examples as its weight function was defined as $w_{i}=(1-\hat{P}(y_{i}|\mathbf{x}_{i};\theta))^{\gamma}$, where $\gamma\geq 1$ was a focusing parameter that adjusted the rate at which easy examples were down-weighted. Research has shown that models trained with Focal Loss were better calibrated under the i.i.d. assumption and performed well under distribution shifts (Mukhoti et al., ). This made Focal Loss a promising baseline for evaluating our proposed weight function in the context of LLM-generated synthetic data training.

Additionally, a series of meta-learning approaches addressed these challenges by leveraging bi-level optimization to dynamically adjust weights based on each instance’s contribution to the meta-set from the real world. These methods handled class imbalance, noisy labels, and augmented data by reweighting these instances based on their gradient direction or model outputs, providing a flexible mechanism for weighting data points (Ren et al., ; Hu et al., ). While effective, meta-learning-based approaches were time-consuming and computationally expensive, making them difficult to scale up to larger datasets or complex models. In contrast, our methods share the same objective of optimizing performance on real-world data but achieve it without meta-learning. This makes it more computationally efficient and scalable while still maintaining high performance.

## 3 Optimization on LLM-generated Dataset

LLMs are capable of generating synthetic datasets (Lu et al., ; West et al., ; Li et al., ), denoted as $D_{Q}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N}$, sourced from the distribution $Q(\mathcal{X},\mathcal{Y})$. This distribution is shaped by specific prompts comprising instruction prompts, system prompts, and few-shot examples that guide the LLM’s output. This method offers a valuable alternative for acquiring training data, especially when access to real-world data is limited. Moreover, the relevance of $Q$ can be further refined by incorporating few-shot examples from a small real-world dataset $D_{P^{\prime}}$, enhancing the utility and applicability of the synthetic data (Li et al., ).

The CE-Loss on the LLM-generated dataset converges to the expected cross-entropy under $Q$:

$$ $\mathcal{L}_{\text{CE}}(\theta,D_{Q})\overset{p}{\to}\mathbb{E}_{Q}\left[-\log \hat{P}(y|\mathbf{x};\theta)\right]$ (3) $$

A significant distributional shift between $Q$ and $P$ may lead to suboptimal predictive performance on real-world data.

### 3.1 Uncertainty of LLM-Generated Data Performance

Our empirical study, shown in Table 1, demonstrates notable variability in the performance of CE-Loss on LLM-generated datasets. Specifically, on the Financial (Malo et al., ) and MRPC (Wang et al., ) benchmarks, CE-Loss on large LLM-generated datasets ($>$ 3k samples) performs worse than training on small real-world datasets, which contain only around 200-400 samples. Conversely, for the Twitter Irony (Van Hee et al., ) benchmark, CE-Loss on LLM-generated data improves accuracy. This variability underscores the uncertainty associated with using CE-Loss on LLM-generated data. These findings are consistent with another research (West et al., ), showing that when using CE-Loss, without proper filtering, LLM-generated data may lead to decent results on downstream tasks, even though its size is considerably larger than that of real-world data.

### 3.2 Potential of LLM-Generated Data: Model-Based Information Measurement

We employ information-theoretic metrics to evaluate the uncertainty within the conditional distributions of the real-world data and LLM-generated data. A higher conditional entropy indicates more significant uncertainty given an input $\mathbf{x}$, suggesting various outcomes. We estimate this by fine-tuning a BERT model (Devlin et al., ) on both datasets separately. Higher conditional entropy is often associated with greater diversity within the dataset, reflecting a broader range of information that the model must learn to predict accurately. The conditional KL divergence(^1^11it is also called conditional divergence or conditional relative entropy) quantifies the difference between two conditional distributions, $P(y|\mathbf{x})$ and $Q(y|\mathbf{x})$, showing how well a model trained on one dataset describes another.

We show these metrics for a Financial benchmark scenario. The real-world dataset $D_{P}$ exhibits significantly lower conditional entropy ($\mathbb{H}(P)=0.0365$) compared with the LLM-generated dataset $Q$ ($\mathbb{H}(Q)=0.2299$), indicating that $D_{Q}$ is more diverse. Furthermore, the conditional KL divergence from $P$ to $Q$ ($D_{\mathrm{KL}}(Q||P)=1.8781$) is much greater than it from $Q$ to $P$ ($D_{\mathrm{KL}}(P||Q)=0.444$), suggesting that models trained on real-world data struggle to capture the complexity of the synthetic dataset. Those models trained on the synthetic dataset are relatively efficient, requiring fewer additional nits on average to encode samples from the real-world distribution $P$. This difference, along with the results from Sec. 3.1 highlights that, although the synthetic dataset contains some points that are less representative of the real-world distribution, it still includes a substantial proportion of relevant data points. This analysis indicates potential for improving modeling techniques to utilize the rich, informative content of LLM-generated data.

### 3.3 Problem Formulation

In this study, we devise a weighted loss function that transforms CE-Loss from an LLM-generated data distribution $Q$ to match the real-world data distribution $P$. We assumed the dataset $D_{Q}$ is i.i.d. and the LLM can approximate the real-world input distribution $P(\mathbf{x})$ through strategic prompting, effectively simulating $Q(\mathbf{x})\approx P(\mathbf{x})$. For instance, by using system prompts like ”Now you are a journalist writing news articles,” it can produce synthetic texts that closely mimic authentic news articles. Lastly, We use a small set $D_{P^{\prime}}$, approximately 200-400 samples from real-world datasets, to facilitate the alignment process. These samples are i.i.d. from the distribution $P$. We use $P^{\prime}$ as the probability function representing this small real-world dataset.

This approach leverages the rich diversity of LLM-generated data to bridge the distributional gap between $Q$ and $P$. By creating an effective weighted loss function, we aim to enhance model performance on real-world tasks by better aligning synthetic data with real-world distributions.

## 4 Methodologies

In this section, we present our Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP-Loss) methods, which transform the CE-Loss to align with the real-world distribution $P$ from the LLM-generated distribution $Q$.

### 4.1 IMP-Loss: Transformation from Q 𝑄 Q italic_Q to P 𝑃 P italic_P

In order to achieve convergence to the real-world data distribution $P$, we applied WCE-loss. Inspired by the Monte Carlo method of Importance Sampling (Hesterberg, ), used to estimate expectation values from a source distribution to a target distribution, we design the weight function as follows:

$$ $w_{i}=\frac{P(y|\mathbf{x}_{i})}{Q(y|\mathbf{x}_{i})}$ (4) $$

By applying this weight function to WCE-Loss, the asymptotic convergence is approximately the expectation under $P$ (details in Appendix B):

$$ $\displaystyle\mathbb{E}_{Q}\left[-\frac{P(y|\mathbf{x})}{Q(y|\mathbf{x})}\log \hat{P}(y|\mathbf{x};\theta)\right]$ (5) $\displaystyle=-\sum_{\mathbf{x}\in\mathcal{X}}Q(\mathbf{x})\sum_{y\in\mathcal{ Y}}P(y|\mathbf{x})\log\hat{P}(y|\mathbf{x};\theta)$ $\displaystyle\approx-\sum_{\mathbf{x}\in\mathcal{X}}P(\mathbf{x})\sum_{y\in \mathcal{Y}}P(y|\mathbf{x})\log\hat{P}(y|\mathbf{x};\theta)$ $\displaystyle=\mathbb{E}_{P}\left[-\log\hat{P}(y|\mathbf{x};\theta)\right]$ $$

The approximation in the penultimate step is based on the assumption stated in Sec. 3.3: the LLM can simulate the real-world input distribution through careful and appropriate prompting. This transformation ensures that the WCE-Loss function effectively aligns the synthetic distribution $Q$ with the real-world distribution $P$.

Further, $Q$ can be estimated by fitting a neural model $\hat{Q}$, such as BERT, on the LLM-generated dataset $D_{Q}$ using the CE-Loss; however, estimating the weight function is challenging because the real-world distribution $P$ is unknown. To address this,
we fit a model $\hat{P^{\prime}}$ on small real-world dataset $D_{P^{\prime}}$.
Using $\hat{P^{\prime}}$ and $\hat{Q}$, we define the Importance Loss $\mathcal{L}_{\text{IMP}}(\theta,D_{Q})$ as follows:

$$ $\mathcal{L}_{\text{IMP}}(\theta,D_{Q})=-\frac{1}{N}\sum_{i=1}^{N}\frac{ \overbrace{\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})}^{\text{{\color[rgb]{ 0.724,0.3,0.956}Quality Checker}}}}{\underbrace{\hat{Q}(y_{i}|\mathbf{x}_{i})} _{\text{{\color[rgb]{0.028,0.644,0.696}Diversity Checker}}}}\log\hat{P}(y_{i}| \mathbf{x}_{i};\theta)$ (6) $$

Algorithm 1 outlines how we use IMP-Loss.

Figure: Algorithm 1 Training with Importance Loss

### 4.2 DIMP-Loss: Which data point causes the model to be closest to P 𝑃 P italic_P ?

In this section, drawing inspiration from online batch selection methods (Deng et al., ; Mindermann et al., ), we investigate which data point in $D_{Q}$, when used for training, will most effectively bring the distribution of the model closer to $P$ in the subsequent optimization step.
In optimization formulation, this can be expressed as:

$$ $\displaystyle(\mathbf{x}^{*},y^{*})=\operatorname*{arg\,min}_{(\mathbf{x}^{ \prime},y^{\prime})\in D_{Q}}\mathbb{E}_{P}\left[-\log\hat{P}\left(y|\mathbf{x };\theta_{t},\{(\mathbf{x}^{\prime},y^{\prime})\}\right)\right],$ (7) $$

where $\theta_{t}$ represents the model parameters at optimisation step $t$. Consider a one-step optimization algorithm $f$ (e.g. SGD), where $\theta_{t+1}\leftarrow f(\theta_{t},\{(\mathbf{x}^{\prime},y^{\prime})\})$. The algorithm updates the model parameters $\theta_{t}$ using $(\mathbf{x}^{\prime},y^{\prime})$ to obtain the new parameters $\theta_{t+1}$ after one optimization step.
The Eq. 7 means the data point $(\mathbf{x}^{*},y^{*})$ is the optimal data point in $D_{Q}$ that leads to the lowest conditional cross-entropy after one update step. Specifically, it identifies which data point is used for training results in the model parameters that yield the model closest to the real-world distribution $P$.

In empirical settings, we may not have access to the complete real-world distribution $P$, but we can approximate it by a small real-world dataset $D_{P^{\prime}}$, also denoted as $(\mathbf{y}_{P^{\prime}},\mathbf{x}_{P^{\prime}})$ in the perspective of labels and inputs. This allows us to rewrite the objective as maximizing the probability:

$$ $\displaystyle\operatorname*{arg\,max}_{(\mathbf{x},y)\in D_{Q}}\hat{P}(\mathbf {y}_{P^{\prime}}|\mathbf{x}_{P^{\prime}};\theta_{t},\{(\mathbf{x},y)\})= \operatorname*{arg\,max}_{(\mathbf{x},y)\in D_{Q}}\frac{\hat{P}(y|\mathbf{x}; \theta_{t},D_{P^{\prime}})}{\hat{P}(y|\mathbf{x};\theta_{t})}$ (8) $$

Eq. 8 aims to maximize the joint likelihood of all data points in $D_{P^{\prime}}$. The joint likelihood involves inferring all data points in $D_{P^{\prime}}$ and multiplying their prediction probabilities (due to the i.i.d. assumption). However, this optimization is infeasible, as it requires updating the model for each data point in $D_{Q}$, resulting in $|D_{Q}|$ models, and each needs evaluation on the whole $D_{P^{\prime}}$.

Notably, by applying Bayes’ rule, we derive the second line of Eq. 8 (see Appendix C for details), showing a more feasible calculation approach. This requires evaluating only two models for each data point in $D_{Q}$: the denominator $\hat{P}(y|\mathbf{x};\theta_{t})$ is the current model in step $t$, and the numerator $\hat{P}(y|\mathbf{x};\theta_{t},D_{P^{\prime}})$ would require additional training on $D_{P^{\prime}}$. To simplify, we approximate $\hat{P}(y|\mathbf{x};\theta_{t},D_{P^{\prime}})$ with $\hat{P^{\prime}}(y|\mathbf{x})$, the probability estimated from $D_{P^{\prime}}$ as in (Deng et al., ).

The approximation of Eq. 8 is then utilized as the weight in our loss function. Consequently, if a data point brings the model closer to the real-world distribution $P$, its corresponding weight will be higher, thus having a greater impact on the model’s training. Thus, we define the Dynamic Importance Loss (DIMP-Loss) $\mathcal{L}_{\text{DIMP}}(\theta_{t},D_{Q})$ as:

$$ $\mathcal{L}_{\text{DIMP}}(\theta_{t},D_{Q})=-\frac{1}{N}\sum_{i=1}^{N}\frac{ \overbrace{\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})}^{\text{{\color[rgb]{ 0.724,0.3,0.956}Quality Checker}}}}{\underbrace{\hat{P}(y_{i}|\mathbf{x}_{i}; \theta_{t})}_{\text{{\color[rgb]{0.028,0.644,0.696}Diversity Checker}}}}\log \hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})$ (9) $$

The approximation of Eq. 8 simplifies the calculation of the weight function, making the implementation of DIMP-Loss practical. We can observe this weight function dynamically changes at each optimization step and adjust the weights based on the current parameters $\theta_{t}$, thereby continually refining the alignment between the model $\hat{P}_{\theta}$ and the real-world data distribution $P$. Algorithm 2 outlines how DIMP-Loss is used in training a model.

Figure: Algorithm 2 Training with DIMP-Loss

To better understand the properties of DIMP-Loss, we derived a lower bound for it (details can be found in Appendix D). Precisely, we have:

$$ $\mathcal{L}_{\text{DIMP}}(\theta_{t},D_{Q})\geq\underbrace{-\frac{2}{N}\sum_{i =1}^{N}\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})\log\hat{P}(y_{i}|\mathbf{x}_{i}; \theta_{t})}_{\text{Empirical distilled cross-entropy loss}}+\underbrace{\frac {1}{N}\sum_{i=1}^{N}\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})\log\hat{P}(y_{i}| \mathbf{x}_{i};\theta_{t})}_{\text{Maximum entropy regularizer}}$ (10) $$

DIMP-Loss can be interpreted as an upper bound on the regularized empirical distilled risk (Menon et al., ; Wang et al., ), where the ”teacher” model is the quality checker. The regularizer is a maximum entropy term (Pereyra et al., ) designed to prevent overconfidence in output distribution. In this context, DIMP-Loss can also be viewed as a form of knowledge distillation, where the knowledge from a model is trained on a small amount of real-world data. The objective is to align the predicted distribution $\hat{P}_{\theta}$ with $P^{\prime}$ while promoting higher entropy in $\hat{P}_{\theta}$ to avoid overly confident predictions.

### 4.3 Quality and Diversity Checkers in IMP-Loss and DIMP-Loss

According to both Eq. 6 and 9, a higher weight value in either loss means that the data point will have a greater influence on the model.

the Quality Checker ($\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})$) assesses the likelihood of a data point sampled from the real-world distribution $P$. Higher values indicate the data point is, highly relevant, and unambiguous for the real-world distribution $P$.

The Diversity Checker differs in the two losses, $\hat{Q}(y_{i}|\mathbf{x}_{i})$ for IMP-Loss, and $\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})$ for DIMP-Loss. In the context of IMP-Loss, a low Diversity Checker value ($\hat{Q}(y_{i}|\mathbf{x}_{i})$) suggests the data point contains a high amount of information within the LLM-generated dataset $D_{Q}$, because a redundant data point in $D_{Q}$ will have a high probability, indicating less diversity. Hence, the Diversity Checker serves as an indicator of diversity from the perspective of the LLM-generated distribution. In contrast, for DIMP-Loss, a low Diversity Checker value ($\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})$) implies the data point is challenging to be learnt in previous steps, departing from the data points the model has already learnt. Thus, Diversity Checker of DIMP-Loss reflects diversity from the perspective of a model. This distinction highlights how each loss function prioritizes different aspects of data diversity during training. We simulated defect and redundant situations for further exploration in the Appendix. G.3.

### 4.4 Computational Cost of Training with IMP-Loss and DIMP-Loss

The analysis covers computational requirements and practical run-time detailed in Appendix E.

#### IMP-Loss.

According to Algorithm 1, the computational cost of training with IMP-Loss is approximately (ignore the cost on $D_{P^{\prime}}$) twice training plus twice forward pass on $D_{Q}$. First, we estimate $P^{\prime}(y|\mathbf{x})$ by fitting a model on $D_{P^{\prime}}$, and $Q(y|\mathbf{x})$ by fitting a model on $D_{Q}$, respectively. Second, we compute the weights for each sample in $D_{Q}$ by $\hat{Q}(y|\mathbf{x})$ and $\hat{P^{\prime}}(y|\mathbf{x})$. Although the estimation of $P^{\prime}(y|\mathbf{x})$ incurs minimal cost due to the small size of $D_{P^{\prime}}$, the primary additional overhead comes from the repeated training on $D_{Q}$ and the additional forward passes needed to compute the weights.

#### DIMP-Loss.

According to Algorithm 2, the computational cost of training with DIMP-Loss is approximately (ignore the cost on $D_{P^{\prime}}$) one training plus one forward pass on $D_{Q}$. On the one hand, we need to fit a model on the small real-world $D_{P^{\prime}}$ to estimate $P^{\prime}(y|\mathbf{x})$, the numerator of the weight coefficient, for each data point in $D_{Q}$. On the other hand, we compute the weights for each sample in $D_{Q}$ using the DIMP-Loss formulation, which involves evaluating the computed $\log\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})$ and hence getting $\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})$. Without the estimation to $Q(y|\mathbf{x})$, DIMP-Loss is more efficient than IMP-Loss. The computational overhead is only slightly higher than that of CE-Loss, because of the additional step of estimating $P^{\prime}$ from the small dataset $D_{P^{\prime}}$ and performing a single inference pass on $D_{Q}$, as the values of the quality checker for each data point remain constant in training.

## 5 Experiments

**Table 1: Performance metrics across datasets and methods. The table showcases the accuracy (Acc) and macro F1 score (F1) for each method applied on three benchmarks. The methods include GPT-3.5 few-shot, CE-Loss, Focal-Loss, ’s method, IMP-Loss, and DIMP-Loss. Bold entries denote the performance within 0.5%, comparing to the best performance of each training source.**
| Dataset | Method | Financial | Tweet Irony | MRPC |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Acc | F1 | Acc | F1 | Acc | F1 |  |  |
|  | GPT-3.5 few-shot | 79.46 | 81.6 | 63.39 | 69.39 | 69.28 | 71.75 |
| Small real world | CE-Loss (quality checker) | 78.05 | 75.26 | 62.5 | 62.38 | 73.16 | 68.69 |
| Focal-Loss | 78.47 | 76.2 | 67.73 | 62.32 | 73.10 | 66.64 |  |
| DIMP-Loss (Ours) | 79.87 | 77.05 | 69.01 | 67.05 | 74.84 | 66.80 |  |
| GPT-3.5 generated | CE-Loss | 77.39 | 74.01 | 76.91 | 76.8 | 72 | 65.47 |
| Focal-Loss | 79.29 | 75.32 | 74.87 | 74.82 | 72.17 | 62.77 |  |
| ’s | 71.7 | 61.93 | 71.42 | 70.18 | 67.13 | 50.08 |  |
| IMP-Loss (Ours) | 82.09 | 79.40 | 81.89 | 81.71 | 75.83 | 70.52 |  |
| DIMP-Loss (Ours) | 82.67 | 79.53 | 78.44 | 78.14 | 75.83 | 70.04 |  |
|  | - w/o diversity checker | 81.35 | 77.94 | 77.68 | 77.62 | 74.72 | 69.34 |
| Large real world | CE-Loss | 84.74 | 82.69 | 68.75 | 68.41 | 80.92 | 77.73 |
| Focal-Loss | 84.98 | 81.98 | 67.6 | 67.19 | 80.35 | 76.28 |  |
| ’s | 80.19 | 76.58 | 60.33 | 37.63 | 71.36 | 67.78 |  |
| IMP-Loss (Ours) | 85.3 | 83.27 | 70.15 | 70.08 | 81.33 | 78.3 |  |
|  | DIMP-Loss (Ours) | 85.4 | 82.79 | 69 | 68.78 | 82.84 | 80.49 |

We assessed our proposed methods by comparing them with standard loss functions across several text classification benchmarks, including Financial Phrasebank (Financial) (Malo et al., ), irony detection (Tweet Irony) (Van Hee et al., ), and the MRPC dataset from GLUE (Wang et al., ). Detailed descriptions and specifications of these datasets are provided in Appendix H. In our experiments, we referred the large real-world data $D_{P}$ to the original training set from each benchmark and the small real-world data $D_{P^{\prime}}$ to the original development set, with the sizes from approximately 200 to 400, as shown in Table 5. Our experiments explored three different scenarios: training solely on synthetic data (Sec. 5.1), real-world data (Sec. 5.2), and noisy data (Sec. G). We evaluated Accuracy (Acc) and Macro F1 score (F1) for every benchmark. These metrics were computed by comparing the model’s predictions with the gold labels provided in the test sets. We used a BERT-based model for fine-tuning and building the checkers. The Appendix F details the configurations.

#### Baselines.

CE-Loss, Focal Loss, and Hu et al. ()’s method are our baselines, detailed in Sec. 2.1 and Sec. 2.2. Focal Loss addressed class imbalance and mitigated easily classified data’s impact, preventing overconfidence. The weight function for Focal Loss was defined as $w_{i}=(1-\hat{P}(y_{i}|\mathbf{x}_{i};\theta))^{\gamma}$ where $\gamma$ controled the downweighting of easy examples. Mukhoti et al. () showed models trained with Focal Loss exhibited better calibration under the i.i.d. assumption and performed robustly under distribution shifts. This made Focal Loss a strong baseline for evaluating our proposed approaches. Hu et al. () used meta-learning to dynamically adjust weights training data by maximizing the likelihood on a small real-world dataset, similar to IMP-Loss and DIMP-Loss; however, our methods directly adjust weights based on quality and diversity checkers, while Hu et al. ()’s method relied on meta-learning to optimize weights indirectly. (^2^22We implemented Focal-Loss and Hu et al. ()’s method by using their official code.)

### 5.1 Training on LLM-generated Data

We compared our proposed methods with standard loss functions on LLM-generated data.

#### Data Generation.

We used GPT-3.5-turbo-1106 (OpenAI, ). A system prompt, 8-shot examples from the development set $D_{P^{\prime}}$, and the corresponding labels to generate the input text were provided. For the Financial and Tweet Irony benchmarks, our generation prompt based on previous research (Li et al., ). Similarly, for the MRPC benchmark, the prompt included pairs of sentences with answers, which automatically guided the LLM in generating the answers. The Appendix I.1 provides each benchmark’s prompt.

#### IMP-Loss and DIMP-Loss Outperform on LLM-Generated Data.

As shown in Table 1, our methods outperformed all baselines in all benchmarks. For instance, in Financial Phrasebank, IMP-Loss achieved 82.09% in accuracy and 79.40% in F1 score, and DIMP-Loss reached 82.67% and 79.53% respectively, while the CE-Loss reached 77.39% / 74.01%. The result showed if we use the baselines, CE-Loss, Focal-Loss, and Hu et al. ()’s method to train a classifier on $D_{Q}$, the performance could be worse than that of using CE-Loss on much smaller $D_{P^{\prime}}$ (quality checker). In contrast, our methods consistently outperformed the small quality checker, encouraging the usage of abundant LLM-generated data. Notably, even when the quality checker performs poorly, such as on the Tweet Irony dataset, where the accuracy was 62.5%, which was lower than the 76.9% achieved by directly training on generated data using CE-Loss, our methods still delivered strong performance. This suggested that a high-performance quality checker was not a prerequisite for the effectiveness of our methods.

#### IMP-Loss and DIMP-Loss Surpass the Accuracy of the Data Generator.

The GPT-3.5 few-shot predictor generated predictions using 8 examples from the small real-world dataset in the input prompt. GPT-3.5 achieved 79.46% in the Financial dataset and 68.82% in the MRPC dataset. Our approaches consistently surpassed the GPT-3.5 few-shot prediction in accuracy. The parameter size of the fine-tuned models using our methods was significantly lower than that of the GPT-3.5 data generator, yet they delivered higher performance.

#### Superior and Robust Accuracy Across Epochs.

Figure: (a)
Refer to caption: extracted/5952991/figures/financial_dynamic.png

The training dynamics in Figure 1 revealed our methods outperformed CE-Loss and Focal-Loss across all benchmarks. Notably, both IMP-Loss and DIMP-Loss achieved low variation by the end of training, indicating stable performance. Moreover, DIMP-Loss showed higher variation in the initial epochs compared with IMP-Loss. This increased variability could be attributed to the order of sampled data, which caused initial fluctuations. Nevertheless, the Acc ultimately converged at a higher value than the baselines.

#### Quality Checkers are Data Efficient.

**Table 2: Accuracy of methods on benchmarks when training a larger model with smaller Quality Checkers. ”base DC” and ”large DC” denote smaller and larger Diversity Checkers, respectively. Bold entries highlight the top value of metrics within each dataset.**
| Model Size | Method | Financial | Tweet irony | MRPC |
| --- | --- | --- | --- | --- |
| Base | Quality checker | 78.05 | 62.5 | 73.16 |
| Large | CE-Loss | 80.45 | 78.83 | 74.2 |
| IMP-Loss (base DC) | 80.94 | 74.23 | 75.36 |  |
| IMP-Loss (large DC) | 81.93 | 78.83 | 76.41 |  |
| DIMP-Loss | 83.25 | 81.25 | 77.04 |  |

Figure 2 illustrates the test accuracy on the Financial benchmark with quality checkers trained on various proportions of the original training set. As seen in this figure, even a small number of data points, e.g., 10%, was sufficient to enhance the performance of both IMP-Loss and DIMP-Loss. This suggested that a small amount of real-world data was effective for building a quality checker, making our approach efficient and practical.

#### Diversity Checkers are Important.

The results in Table 1 highlighted the importance of Diversity Checkers in our proposed methods. When training on GPT-3.5 generated data, the performance of the model trained with IMP-Loss without Diversity Checkers dropped compared with IMP-Loss with Diversity Checkers. For instance, in the Financial dataset, the accuracy drops from 82.09% to 81.35% and the F1 score from 79.40% to 77.94%. These results indicated that incorporating Diversity Checkers helped effectively use LLM-generated data.

#### Smaller Quality Checker Still Enhances Performance by DIMP-Loss.

The results in Table 2 illustrated the performance of each method on the benchmarks when training a larger classifier (BERT-large) with smaller Quality Checkers (BERT-base). Notably, DIMP-Loss consistently performed well even when the Quality Checker was small. This demonstrated the robustness of DIMP-Loss in adapting to different model sizes for Quality Checkers. In contrast, IMP-Loss showed inconsistent performance when using a smaller Diversity Checker compared with its training model, indicating the choice of the Diversity Checker in size significantly impacted its efficacy. In short, using a smaller Quality Checkers to guide the model was efficient in terms of both space and time.

### 5.2 Training on Real World Data

#### Robust Performance of IMP-Loss and DIMP-Loss on Real-World Data.

As shown in Table 1, IMP-Loss and DIMP-Loss outperformed other baselines even when applied directly to real-world data. Although the performance improvements are less than that of using GPT-3.5-generated data, the results indicated our methods were versatile and able to handle multiple sources of training data effectively. Specifically, in the Financial dataset, IMP-Loss achieved 85.3% Acc and 83.27% F1 score, while DIMP-Loss reached 85.4% Acc and 82.79% F1 score, surpassing both CE-Loss, Focal-Loss, and ’s method baselines. From our perspective, the reduced improvement in this scenario was due to the lack of a requirement to shift the training data distribution. Regarding the asymptotic viewpoint, the optimal solution of cross-entropy is already the best solution when training on real-world data. Nonetheless, our methods demonstrated robust performance across various conditions.

Figure: Figure 2: Test accuracy on the Financial benchmark with varying percentages of the training set for the quality checker. The graph shows the performance of IMP-Loss, DIMP-Loss, and their Quality Checker.
Refer to caption: extracted/5952991/figures/quality_checker_ablation.png

## 6 Related Works

We list some essential related work in this section and others in A.

#### Synthetic Data Generation from LLMs.

Recent advancements in generative AI have spurred interest in using LLMs to generate synthetic data for specific tasks, particularly in low-resource settings. Studies have explored zero-shot and few-shot settings for data generation, where LLMs directly generate instances with distinct labels or use little real-world data as examples to create relevant and diverse data (Li et al., ; West et al., ). LLMs have the unique ability to generate both labels and diverse input instances, significantly enhancing the variety and quality of synthetic datasets. Approaches like ZEROGEN synthesized data by pre-trained LMs to train smaller task models, achieving competitive performance in NLP tasks (Ye et al., ). Additionally, (Wang et al., ; Taori et al., ) have fine-tuned GPT-3 and LLaMA on augmented data, respectively, to follow human instructions, while domain-specific tuning before generation has been effective for tasks like text classification (Stylianou et al., ).

#### Weighting for Misalignment Data

Importance weighting (IW) serves as a classical strategy for addressing the issue of shifts between data distributions (Hesterberg, ). In traditional applications of IW, weights are derived by evaluating the degree of similarity between training and testing distributions through various statistical techniques. These techniques include maximum mean discrepancy (Schölkopf et al., ) and the estimation of KL divergence (Sugiyama et al., ). Although effective in linear model contexts, the efficacy of these methods seems to significantly diminish when applying IW to more complex deep learning frameworks (Byrd & Lipton, ). Besides traditional methods, recent studies have explored approaches such as Focal Loss (Lin et al., ) and meta-learning techniques (Hu et al., ), which address issues like class imbalance and dynamic reweighting, as discussed in Sec. 2.2.

## 7 Conclusions and Discussions

IMP-Loss and DIMP-Loss are novel weighted-loss objectives that further enhance the performance of models trained on LLM-generated data. Our empirical results demonstrated that both methods outperformed traditional loss functions across various benchmarks. Notably, DIMP-Loss was particularly computationally efficient, requiring subtly additional resources while increasing performance. These findings emphasized the potential of IMP-Loss and DIMP-Loss in effectively leveraging synthetic data for training machine learning models. In the future, we will extend our methods on question answering, text generation, LLM pertaining, and other potential tasks, further exploring how quality and diversity matter for learning.

## 8 Reproducibility Statement

To ensure reproducibility, we provided the source code and generated dataset in supplementary materials, prompts used for generation in I.1, hyper-parameters and other training details in F, testing datasets’ descriptions in H, and theoretical results in B, C, D. In addition, for the baselines, we implemented the Focal Loss as in the source code and used the publicly available code on (Hu et al., )’s Github but replaced the input data. We hope one can smoothly reproduce our results via these materials.

## Appendices

## Appendix A Other Related Work

#### Online Batch Selection

Online batch selection (Loshchilov & Hutter, ; Katharopoulos & Fleuret, ; Mindermann et al., ; Deng et al., ) is a method to speed up training convergence by dynamically prioritizing the most informative data points, from the perspective of the minimizing loss function. This technique evaluates how informative a data point is and selects a batch $B_{t}$ of informative data during each training step.
Unlike online batch selection methods substituting uniformly sampled batches during training, this paper focused on developing a weight function to enhance performance on downstream tasks by aligning the LLM-generated data distribution with real-world data distribution.

#### Data Pruning.

Data pruning approaches filter out noisy text data. Traditional methods took rule-based filtering for high-quality data (Bane et al., ; Wenzek et al., ), while recent approaches focused on diversification (Marion et al., ; Ankner et al., ), which used perplexity to build a diverse dataset.
In advance, our methods considered both quality and diversity, and this dual focus made our weighting mechanism a possible pruning scorer. Our methods did not conflict with existing pruning methods, thus becoming a potential complement.

## Appendix B Asymptotic Convergence of IMP-Loss

In this section, we provide the formal proof of the asymptotic convergence of IMP-Loss using Chebyshev’s inequality. Specifically, we show that this approximately converges in probability to the expected conditional cross-entropy under $P$.

###### Definition B.1 (Convergence in Probability)

A sequence of random variables $\{X_{n}\}$ converges in probability to a random variable $X$, denoted as $\{X_{n}\}\overset{p}{\to}X$, if for any $\epsilon>0$,

$$ $\lim_{n\to\infty}P(|X_{n}-X|\geq\epsilon)=0$ (11) $$

###### Theorem B.1 (Chebyshev’s Inequality)

Let $X$ be a random variable with finite expected value $\mathbb{E}[X]$ and variance $\text{Var}(X)$. For any $\epsilon>0$,

$$ $P\left(\left|X-\mathbb{E}[X]\right|\geq\epsilon\right)\leq\frac{\text{Var}(X)} {\epsilon^{2}}$ (12) $$

###### Definition B.1 (Convergence in Probability)

###### Theorem B.1 (Chebyshev’s Inequality)

### Applying Chebyshev’s Inequality to IMP-Loss

Following the definition of IMP-Loss from Eq. 6 and considering the situation without using small real-world data to approximate. Let

$$ $\mathcal{L}_{\text{IMP}}(\theta,D_{Q})=-\frac{1}{N}\sum_{i=1}^{N}\frac{P(y_{i} |\mathbf{x}_{i})}{Q(y_{i}|\mathbf{x}_{i})}\log\hat{P}(y_{i}|\mathbf{x}_{i};\theta)$ (13) $$

Assume that all data points $(\mathbf{x}_{i},y_{i})$ are i.i.d. samples from the joint distribution $Q(\mathcal{X},\mathcal{Y})$. Define

$$ $Z_{i}=-\frac{P(y_{i}|\mathbf{x}_{i})}{Q(y_{i}|\mathbf{x}_{i})}\log\hat{P}(y_{i }|\mathbf{x}_{i};\theta)$ (14) $$

The empirical mean of $Z_{i}$ over $N$ samples is given by:

$$ $\overline{Z}=\frac{1}{N}\sum_{i=1}^{N}Z_{i}=\mathcal{L}_{\text{IMP}}(\theta,D_ {Q})$ (15) $$

The expected value of $Z_{i}$ under the distribution $Q$ is:

$$ $\mathbb{E}_{Q}[Z]=\mathbb{E}_{Q}\left[-\frac{P(y|\mathbf{x})}{Q(y|\mathbf{x})} \log\hat{P}(y|\mathbf{x};\theta)\right]$ (16) $$

Applying Chebyshev’s inequality to the sequence $\overline{Z}$:

$$ $P\left(\left|\overline{Z}-\mathbb{E}_{Q}[Z]\right|\geq\epsilon\right)\leq\frac {\text{Var}_{Q}(\overline{Z})}{\epsilon^{2}}=\frac{\text{Var}_{Q}(Z_{1})}{N \epsilon^{2}}$ (17) $$

As $N$ grows large, the right-hand side converges to zero, implying that $\overline{Z}$ converges in probability to $\mathbb{E}_{Q}[Z]$.

Therefore,

$$ $\displaystyle\mathcal{L}_{\text{IMP}}(\theta,D_{Q})\overset{p}{\to}\mathbb{E}_ {Q}\left[-\frac{P(y|\mathbf{x})}{Q(y|\mathbf{x})}\log\hat{P}(y|\mathbf{x}; \theta)\right]$ (18) $$

### Transforming from Q 𝑄 Q italic_Q to P 𝑃 P italic_P

Next, we show that:

$$ $\displaystyle\mathbb{E}_{Q}\left[-\frac{P(y|\mathbf{x})}{Q(y|\mathbf{x})}\log \hat{P}(y|\mathbf{x};\theta)\right]$ (19) $\displaystyle=-\sum_{\mathbf{x}\in\mathcal{X}}\sum_{y\in\mathcal{Y}}Q(\mathbf{ x},y)\frac{P(y|\mathbf{x})}{Q(y|\mathbf{x})}\log\hat{P}(y|\mathbf{x};\theta)$ $\displaystyle=-\sum_{\mathbf{x}\in\mathcal{X}}Q(\mathbf{x})\sum_{y\in\mathcal{ Y}}P(y|\mathbf{x})\log\hat{P}(y|\mathbf{x};\theta)$ $\displaystyle\approx-\sum_{\mathbf{x}\in\mathcal{X}}P(\mathbf{x})\sum_{y\in \mathcal{Y}}P(y|\mathbf{x})\log\hat{P}(y|\mathbf{x};\theta)$ $\displaystyle=\mathbb{E}_{P}\left[-\log\hat{P}(y|\mathbf{x};\theta)\right]$ $$

Given that $Q(\mathbf{x})\approx P(\mathbf{x})$ by the assumption that the LLM is capable of simulating the real-world input distribution through careful and appropriate prompting, we have:

$$ $\displaystyle\mathcal{L}_{\text{IMP}}(\theta,D_{Q})$ (20) $\displaystyle\overset{p}{\to}\mathbb{E}_{Q}\left[-\frac{P(y|\mathbf{x})}{Q(y| \mathbf{x})}\log\hat{P}(y|\mathbf{x};\theta)\right]$ $\displaystyle\approx\mathbb{E}_{P}\left[-\log\hat{P}(y|\mathbf{x};\theta)\right]$ $$

Thus, the asymptotic convergence of IMP-Loss ensures that the weighted loss function effectively aligns the LLM-generated data distribution $Q$ with the real-world data distribution $P$.

## Appendix C Derivation of DIMP-Loss

In this section, we provide the formal derivation to address the question: Which data point in $D_{Q}$, when used for training, will most effectively bring the model distribution closest to $P$? Following the optimization formulation in Eq. 7, we can empirically apply Monte Carlo estimation using a small real-world dataset $D_{P^{\prime}}$, denoted as $(\mathbf{y}_{P^{\prime}},\mathbf{x}_{P^{\prime}})$. This allows us to reformulate the problem by maximizing the joint probability of the data points in $D_{P^{\prime}}$, which leads to the following optimization problem. This derivation is similar to the online batch selection techniques discussed in previous research (Deng et al., )

$$ $\operatorname*{arg\,max}_{(\mathbf{x},y)\in D_{Q}}\hat{P}(\mathbf{y}_{P^{ \prime}}|\mathbf{x}_{P^{\prime}};\theta_{t},\{(\mathbf{x},y)\})$ (21) $$

This formulation leverages the joint probability of the dataset $D_{P^{\prime}}$, ensuring that the selected data points in $D_{Q}$ are those that, when used for training, most effectively align the model’s distribution with the small real-world distribution $P^{\prime}$. This also implies that the chosen data point leads the model to perform well on $D_{P^{\prime}}$, enhancing the likelihood of better generalization to real-world data.

### Applying Bayes Rule

By applying Bayes’ rule to the joint probability of the dataset $D_{P^{\prime}}$, we obtain:

$$ $\displaystyle\hat{P}(\mathbf{y}_{P^{\prime}}|\mathbf{x}_{P^{\prime}};\theta_{t },\{(\mathbf{x},y)\})$ (22) $\displaystyle=\frac{\hat{P}(D_{P^{\prime}},\mathbf{x},y,\theta_{t})}{\hat{P}( \mathbf{x}_{P},\mathbf{x},y,\theta_{t})}$ $\displaystyle=\frac{\hat{P}(\mathbf{y}_{P^{\prime}}|\mathbf{x}_{P^{\prime}}, \mathbf{x},\theta_{t})\hat{P}(y|\mathbf{x},D_{P^{\prime}},\theta_{t})}{\hat{P} (y|\mathbf{x},\mathbf{x}_{P^{\prime}},\theta_{t})}$ $\displaystyle=\frac{\hat{P}(\mathbf{y}_{P^{\prime}}|\mathbf{x}_{P^{\prime}}, \theta_{t})\hat{P}(y|\mathbf{x};D_{P^{\prime}},\theta_{t})}{\hat{P}(y|\mathbf{ x};\theta_{t})}$ $$

The final equality holds because $\mathbf{x}$ alone cannot perform a model update, leading to the conditional independence assumption. Since $\hat{P}(\mathbf{y}_{P^{\prime}}|\mathbf{x}_{P^{\prime}},\mathbf{x},\theta_{t})$ is a constant for this optimization problem and does not influence the result, we can further simplify the optimization as follows:

$$ $\displaystyle\operatorname*{arg\,max}_{(\mathbf{x},y)\in D_{Q}}\hat{P}(\mathbf {y}_{P^{\prime}}|\mathbf{x}_{P^{\prime}};\theta_{t},\{(\mathbf{x},y)\})=$ (23) $\displaystyle\operatorname*{arg\,max}_{(\mathbf{x},y)\in D_{Q}}\frac{\hat{P}(y |\mathbf{x};\theta_{t},D_{P^{\prime}})}{\hat{P}(y|\mathbf{x};\theta_{t})}$ $$

Similar to the online batch selection work, we use $P^{\prime}(y|\mathbf{x})$ to approximate $\hat{P}(y|\mathbf{x};\theta_{t},D_{P^{\prime}})$. This approximation is then utilized as the weight in our loss function. Consequently, if a data point brings the model closer to the real-world distribution $P$, its corresponding weight will be higher, thus impacting the model’s training.

## Appendix D Lower Bound of DIMP-Loss

#### The Lower Bound

$$ $\displaystyle N\mathcal{L}_{\text{DIMP}}(\theta_{t},D_{Q})$ $\displaystyle=-\sum_{i=1}^{N}\frac{\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})}{ \hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})}\log\hat{P}(y_{i}|\mathbf{x}_{i}; \theta_{t})$ (24) $\displaystyle\quad\text{(By *AM-GM inequality: }\frac{1}{a}\geq 2-a\text{ for }a=\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})\text{)}$ $\displaystyle\geq-\sum_{i=1}^{N}\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})\left(2- \hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})\right)\log\hat{P}(y_{i}|\mathbf{x}_{i };\theta_{t})$ $\displaystyle=-2\sum_{i=1}^{N}\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})\log\hat{P }(y_{i}|\mathbf{x}_{i};\theta_{t})-\left|\sum_{i=1}^{N}\hat{P^{\prime}}(y_{i}| \mathbf{x}_{i})\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})\log\hat{P}(y_{i}| \mathbf{x}_{i};\theta_{t})\right|$ $\displaystyle\quad\text{(By Hölder's Inequality }\|fg\|_{1}\leq\|f\|_{\infty} \,\|g\|_{1}\text{)}$ $\displaystyle\geq-2\sum_{i=1}^{N}\hat{P^{\prime}}(y_{i}|\mathbf{x}_{i})\log \hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})-\max_{i}\hat{P^{\prime}}(y_{i}| \mathbf{x}_{i})\sum_{i=1}^{N}\left|\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t}) \log\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})\right|$ $\displaystyle\geq\underbrace{-\sum_{i=1}^{N}\hat{P^{\prime}}(y_{i}|\mathbf{x}_ {i})\log\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})}_{\text{Empirical distilled cross-entropy loss}}+\underbrace{\sum_{i=1}^{N}\hat{P}(y_{i}|\mathbf{x}_{i}; \theta_{t})\log\hat{P}(y_{i}|\mathbf{x}_{i};\theta_{t})}_{\text{Maximum entropy regularizer}}$ $$

#### *AM-GM Inequality

The derivation shown illustrates the application of the Arithmetic Mean - Geometric Mean (AM-GM) inequality, which states that for any two positive numbers $x$ and $y$, the arithmetic mean is greater than or equal to the geometric mean, i.e., $\frac{a+b}{2}\geq\sqrt{ab},\quad\forall a,b>0.$
In this specific case, $b$ is set to $\frac{1}{a}$, simplifying the inequality to:

$$ $\frac{a+\frac{1}{a}}{2}\geq\sqrt{1}=1.$ $$

Multiplying both sides by 2 yields:

$$ $a+\frac{1}{a}\geq 2,$ $$

and rearranging the inequality gives:

$$ $\frac{1}{a}\geq 2-a.$ $$

This result is a classic application of the AM-GM inequality, demonstrating that the sum of a number and its reciprocal is always greater than or equal to 2 for any positive $x$.

## Appendix E Computational Time Comparison

Figure: Figure 3: Total running time (in seconds) for CE-Loss, IMP-Loss, and DIMP-Loss on the LLM-generated Financial benchmark.
Refer to caption: extracted/5952991/figures/computation_time_financial.png

**Table 3: Total running time of each component (in seconds) for CE-Loss, IMP-Loss, and DIMP-Loss on the LLM-generated Financial benchmark. The table breaks down the time spent building the Quality Checker (QC), building the Diversity Checker (DC), precalculating weights, and training. The total time combines all these components.**
| Method | Build QC | Build DC | Precalculate weights | Training | Total |
| --- | --- | --- | --- | --- | --- |
| CE-Loss | - | - | - | 333.242s | 333.242s |
| IMP-Loss | 8.824s | 333.516s | 57.695s | 333.328s | 733.363s |
| DIMP-Loss | 8.824s | - | 29.274s | 333.426s | 371.524s |

In this computational time experiment, we evaluated the running times on the LLM-generated Financial benchmark dataset, which includes 10,012 training samples ($D_{Q}$) and 242 development samples (small real-world data, $D_{P^{\prime}}$). Our comparison focused on three methods: CE-Loss, IMP-Loss, and DIMP-Loss. We have broken down the total process into four components: building the Quality Checker (QC), building the Diversity Checker (DC), precalculating constant weights, and the actual training time for each respective loss function. For all experiments, the downstream models and checkers were trained for 5 epochs with a batch size of 32. The batch size was set to 64 during the precalculating constant weights phase. The results of this breakdown are presented in Table 3, and the total time is in Figure 3. The results indicate that IMP-Loss requires approximately 2.2 times the running time of CE-Loss. In contrast, DIMP-Loss requires only about 1.1 times the running time of CE-Loss. This demonstrates that training with DIMP-Loss is highly efficient, adding only a slight overhead while significantly enhancing downstream performance.

## Appendix F Training Details and Hyperparameters

For our experiments, we used a pre-trained BERT-base model (Devlin et al., ) from Huggingface’s transformers library (Wolf et al., ) as the encoder, utilizing the representation embedding from the last layer as input to our classification models. We fine-tuned the model with hyperparameters selected from the following ranges: learning rate {6e-6, 6e-5}, epochs {5, 7}, and batch size {32, 64}. Other hyperparameters were set to the default values provided by Huggingface’s trainer for text classification. The best checkpoint was selected based on the accuracy of the development set. We repeated each experiment with four random seeds. We reported the best accuracy run on tables (Table 1), while also providing the minimum, maximum, and average in the training dynamics section (Sec. 5.1). To train the quality checker, we used the small real-world dataset (development split) not included in the training data and trained the quality checker for five epochs. Similarly, the diversity checker of IMP-Loss was also trained for five epochs. This approach aligns with our setup, where access to real-world data is limited, and thus, we reuse the development set to build the quality checker and perform model selection. All experiments were conducted using PyTorch (Paszke et al., ) and Huggingface (for models and datasets) on V100 GPUs with 32GB memory.

## Appendix G Training on noisy data

**Table 4: Performance metrics on the noisy data. The table showcases the accuracy (Acc) and macro F1 score (F1) for each method applied on three distinct datasets: Financial, Tweet Irony, and MRPC. The methods include CE-Loss, GPT-3.5 few-shot, Focal-Loss, IMP-Loss, and DIMP-Loss. Notably, bold entries indicate the best-performing metrics within each training dataset category.**
| Dataset | Method | Financial | Tweet Irony | MRPC |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Acc | F1 | Acc | F1 | Acc | F1 |  |  |
|  | GPT-3.5 few-shot | 79.46 | 81.6 | 63.39 | 69.39 | 69.28 | 71.75 |
| Small real world | CE-Loss (quality checker) | 78.05 | 75.26 | 62.5 | 62.38 | 73.16 | 68.69 |
| Noisy Data | CE-Loss | 79.95 | 77.41 | 60.46 | 60.14 | 74.03 | 67.5 |
| Focal-Loss | 80.2 | 78.26 | 62.11 | 61.12 | 74.72 | 69.59 |  |
| IMP-Loss (Ours) | 81.85 | 78.93 | 64.8 | 64.51 | 76 | 70.46 |  |
| DIMP-Loss (Ours) | 82.51 | 80.22 | 64.16 | 64.09 | 76.58 | 71.32 |  |

In this section, we evaluate the robustness of our proposed methods, IMP-Loss and DIMP-Loss, by training on noisy datasets. We aim to simulate real-world scenarios where LLM-generated data may be imperfect due to labeling errors (low quality) or duplicate entries (low diversity). This allows us to analyze the effects of the Quality Checker and Diversity Checker in IMP-Loss and DIMP-Loss.

### G.1 Experimental Setup

To create noisy datasets, we start with the original training set from each benchmark (Financial, Tweet Irony, and MRPC) and split it into three parts:

- 1.
Original Data: This part remains unchanged and serves as the control set.
- 2.
Random Swapped Label Noise: In this part, the labels are randomly altered, introducing label noise and reducing data quality.
- 3.
Duplicated Data: In this part, each data point is duplicated once, introducing redundancy and reducing data diversity.

### G.2 Performance Results

The results in Tabel 4 indicated that our proposed methods, IMP-Loss and DIMP-Loss, consistently outperform the baselines across all benchmarks, even when the training data is noisy. Specifically, in the Financial dataset, IMP-Loss achieves 81.85% Acc and 78.93% F1 score, while DIMP-Loss reaches 82.51% Acc and 80.22% F1 score, surpassing the CE-Loss and Focal-Loss baselines. In the Tweet Irony dataset, the performance improvement is more pronounced, with IMP-Loss achieving 64.8% Acc and 64.51% F1 score, and DIMP-Loss achieving 64.16% Acc and 64.09% F1 score, significantly higher than CE-Loss and Focal-Loss. For the MRPC dataset, IMP-Loss and DIMP-Loss show robust performance with 76% Acc and 70.46% F1 score and 76.58% Acc and 71.32% F1 score, respectively, outperforming the GPT-3.5 few-shot approach, which achieves 69.28% Acc and 71.75% F1 score.

### G.3 Analysis of Checker Scores and Weights

Figure: Figure 4: Average Quality Checker Score, Diversity Checker Score, and Weights of IMP-Loss for Financial Dataset: Comparison between Original, Swapped Label, and Duplicated Data.
Refer to caption: extracted/5952991/figures/noise_analysis_importance_financial.png

Figure: Figure 5: Average Quality Checker Score, Diversity Checker Score, and Weights of IMP-Loss for Tweet Irony Dataset: Comparison between Original, Swapped Label, and Duplicated Data.
Refer to caption: extracted/5952991/figures/noise_analysis_importance_tweet_irony.png

#### IMP-Loss

Figure 4 and Figure 5 illustrate the Quality Checker Score $P^{\prime}(y|\mathbf{x})$, Diversity Checker Score $Q(y|\mathbf{x})$, and the corresponding weights of the IMP-Loss for the Financial and the Tweet Irony dataset, across three different data conditions: original, swapped labels, and duplicated entries.

The Quality Checker Score (blue bar) is highest for the original data and significantly lower for the swapped label data, indicating that the model correctly identifies the labels as lower quality. The Diversity Checker Score (where lower values are better) is lower for the original data than the duplicated data, indicating the impact of duplication on diversity. Additionally, according to the diversity checker, the swapped label data exhibits the highest diversity. This is because the data points with changed labels are significantly different from other data in the dataset, thus enhancing their diversity score. Consequently, the weights assigned to the original data are higher compared to the swapped label data and the duplicated data, demonstrating the effectiveness of IMP-Loss in recognizing and appropriately weighting high-quality, diverse data.

#### DIMP-Loss

Figure: Figure 6: Average Diversity Checker Score of DIMP-Loss for Original, Swapped label, and Duplicated Data on the Financial Dataset across Epoch.
Refer to caption: extracted/5952991/figures/noise_analysis_dimp_financial_diversity_checker.png

Figure: Figure 7: DIMP-Loss Weight for Original, Swapped label, and Duplicated Data on the Financial Dataset across Epoch.
Refer to caption: extracted/5952991/figures/noise_analysis_dimp_financial_diversity_weight.png

In contrast, Figure 6 shows the diversity scores $\hat{P}(y|\mathbf{x};\theta_{t})$ and the weights for the DIMP-Loss method on Financial benchmark across epoch, where the diversity checker is the training model itself. The Diversity Checker Score (lower is better) is also lower for the original data than the duplicated data. In the end, The weights (Figure 7) assigned by the DIMP-Loss method are consistently higher for the original data than the swapped label and duplicated data across epochs. This pattern aligns with the results observed for the IMP-Loss method.

## Appendix H Dataset Descriptions

The size of each split and the generated data in Table 5.

**Table 5: Data size of each split**
|  | Financial | Tweet Irony | MRPC |
| --- | --- | --- | --- |
| Train | 3392 | 2862 | 3668 |
| Dev | 242 | 200 | 408 |
| Test | 1212 | 784 | 1,725 |
| Generated | 10012 | 3000 | 3005 |

The description of each dataset is following:

#### Financial Phrasebank:

This benchmark involves categorizing finance-related sentences into positive, negative, or neutral sentiment categories. These sentences, numbering 4,840, are extracted from financial news articles. Since the dataset does not come with predefined training, validation, and testing splits, we randomly divided it into training (70%), validation (5%), and testing (25%) sets like the previous work (Li et al., ).

#### Tweet Irony:

This task requires sorting tweets into two groups: ironic and non-ironic. The dataset containing tweets in English has been explicitly annotated for these categories. It comprises 2,862 instances for training and 784 instances for testing. Initially, there were 955 instances in the validation set, but due to limited access to real-world data in our scenario, we have randomly selected 200 instances for our validation sets.

#### MRPC:

The Microsoft Research Paraphrase Corpus (MRPC) consists of 5,801 sentence pairs sourced from news articles. Human annotators manually labeled each pair to determine whether the sentences were paraphrased from each other. We employ the official MRPC dataset available through Huggingface’s datasets library, segmented into training, validation, and testing sets containing 3,668, 408, and 1,725 instances, respectively.

## Appendix I Data Generation

### I.1 Prompt

The prompts used for data generation across different benchmarks are provided in Table 6. The prompts for Tweet Irony and Financial datasets are based on those used in previous work (Li et al., ).

### I.2 Data Generation Budget

We used OpenAI GPT-3.5-turbo-1106 (OpenAI, ) to generate a dataset for the three benchmarks, adhering to OpenAI’s terms of service and usage policies. The total cost is $38.74.

Table: Table 6: Detailed prompts for each task for data generation.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: - Paper assumes Q(x) ~ P(x) on line 171, but in line 154-164, the synthetic data has high diversity and entropy. Wondering how to interpret the two together?
- Synthetic data includes the label for the data. For Q(x) ~ P(x) on line 171, does that means Q(x, y) ~ P(x, y)? Seems the paper mix Q(x, y), Q(y|x) eg at line 942 too.
- Line 999 missing ', so it should be x_{p'}, right?
- The table 1 shows Focal-loss and Hu et al.'s method did not work better than CE-Loss, esp for the "large real world" case, could you comment on the reason?
- For the DIMP-Loss at line 274, if we apply it to large real data, where P' is also trained on the same large real data. In this case, the weighting will converge to 1, is that correct? In that case, the final performance at convergence should be the same as CE-Loss, right? How does this case affect table 1?
soundness: 3
strengths: - The  IMP-Loss and DIMP-Loss  approaches are novel contributions. The approaches are clearly motivated. Derivation are clearly demonstrated step by step. Approximations are used to make the implementation efficient.
- The experiments covered several scenarios, including small real data, synthetic data, noisy data and real data. In all cases, the introduced methods work best, showing the robustness of the approaches. Several oblations are interesting: increasing real data for quality checker increases final performance. Diversity check is very important for final performance.
summary: This paper introduced IMP-Loss and DIMP-Loss as novel weighted-loss objectives to enhance the performance of models trained on LLM-generated data. Paper showed results demonstrating that both methods outperformed traditional loss functions CE-loss, focal loss, Meta-learning weighting on 3 NLP tasks Financial Phrasebank, Tweet Irony and MRPC.

Both IMP-Loss and DIMP-Loss include quality over diversity as weighting per training example. Quality weight is the predicted conditinoal probability from a BERT-base model trained on a small real world data of size 200-400 samples. Diversity for IMP-Loss is similarly trained model but on synthetic data. Diversity for DIMP-Loss is the current model's predicted conditional probability. IMP-Loss is based on importance sampling, DIMP-loss is motivated by online batch selection methods.

For experiments, paper compared training on synthetic data alone, noisy synthetic data (label flipped, duplicated data entry). In both case, IMP-Loss and DIMP-Loss work better than the quality checker and three other loss functions. Paper also applied the two weighted loss function to real data, where the results showed better than training on real data using the other loss functions too.
weaknesses: The paper introduces two weighting scheme for synthetic data. The experiments can be made more convincing:
- The three benchmarks can be expanded to more NLP tasks. Eg nn the ZEROGEN paper cited here there are several benchmarks: sQuAD, QNLI and AdversarialQA, which are particularly challenging for synthetic data generated from LLM. The performance was far below supervised results. It will be interesting to apply the weighting method to see if the gap can be reduced for these challenging benchmarks.
- The paper used 5 epochs for training the downstream models by Line 1101. It is unclear whether the models have converged. It is interesting to not only see weighting proposed here can speed up learning, but what happens at convergence.
- The weighting method does not seem to achieve SOTA for the two benchmarks. It would be more convincing if the paper adopted a strong baseline rather than BERT-large to build on top and compare with SOTA results.
  - mrpc https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc, tinybert is at 86.4%.
  - financial phrasebank https://paperswithcode.com/sota/text-classification-on-financial-phrasebank. 96.7% for distillbert.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. For the metrics "diversity" and "quality", would it be possible to verify that they actually correspond to what they refer to? I.e., does higher quality data actually get higher quality metrics? Currently, it seems like a term for the mathematical expression. While I understand it intuitively, it would also be nice to verify them empirically.
2. In the tweet irony case in Figure 1, DIMP loss start out with worse performance than other baselines for the first and second epoch. Could you explain why such a scenario would occur? Does this mean that these methods introduce instability to the training process?
soundness: 3
strengths: 1. The theoretical derivation of IMP and DIMP loss is clear and sufficient.
2. The empirical improvement shown in table 1 is consistent.
3. There are multiple studies to verify the robustness of the method: different datasets in figure 1, different model sizes in table 2, and using various percentages of the training set in figure 2.
4. Section 3.2 shows the motivation for employing diversity and quality in the weighting strategy, laying the foundation for the methodology developed later in the section. 
5. The writing and structure is clear to follow.
summary: Since the development of LLMs, LLM-generated data are widely used for training models. However, the distribution of such artificial datasets may be misaligned from real-world datasets, creating concerns. 
Therefore, during the training process, to prioritize data that is more diverse and has higher quality, the authors develop an approach to weighing LLM-generated data according to its quality and diversity. Compared to previous data-filtering techniques that only use a subset of generated data, this approach leverages all the training data while giving nonuniform weights to the loss of each data point.
Specifically, the authors theoretically define two losses: IMP-Loss and DIMP-Loss. IMP-Loss applies WCE-Loss, where the weight function is compounded by a quality checker (approximated with P, estimated by fitting a model on D_P, the real-world dataset) and a diversity checker (approximated with the inverse of Q, estimated by fitting a model on D_Q, which is the synthetic dataset). DIMP-Loss simplifies the calculation by estimating the diversity checker with the current model in training.
As a result, the performance using the newly devised loss improves compared to the baselines, with DIMP Loss falling slightly behind IMP loss, as a compromise for lower costs.
weaknesses: 1. The results are derived using a BERT-Based model, which is quite small compared to contemporary LLMs. How would the method scale to larger LLMs? How would the costs grow as the model size grows?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - I am still uncertain whether the proposed diversity checker functions as intended. What happens if unrelated data is fed into the diversity checker? SunGen has already demonstrated that noise in synthetic data is not limited to data with noisy labels but also includes data unrelated to the desired domain.
- Some comments regarding the paper's formatting:
  - Line 030: Use a capital letter for the abbreviation of LLM. Currently, it reads 'Large language models,' which is inconsistent with other terms like 'Natural Language Processing.'
  - Line 063: Ensure proper spacing between ‘ers’ and ‘highly.’
  - Footnote 1: Start the sentence with a capital letter, and ensure there is a period at the end.
  - Please ensure consistent use of \citet and \citep, as this inconsistency currently reduces the overall quality of the paper. For example:
    - Line 033: Use \citet for Taori et al.
    - Line 050: Use \citet for Hu et al.
soundness: 4
strengths: - The use of synthetic data has been of significant interest to researchers for years. It is also important to note that noise in synthetic data has been a long-standing concern. While several studies have proposed methods to address this issue, they come with their own limitations, as the current manuscript notes. This work is timely in addressing those previous limitations.
  - While the proposed method requires some additional computation for assigning weights, I agree that it is more efficient compared to methods based on meta-learning, at least.
  - As a result, this method seems more practical than previous approaches for mitigating the noise in synthetic data.
summary: This paper proposes a reweighting scheme for synthetic data generated by LLMs. The focus is on the distinction between the distributions of real and synthetic data. To address this, the authors suggest IMP-Loss and DIMP-Loss to align the distribution of synthetic data with that of real data. These two loss functions assign weights to the cross-entropy loss for each data point. The empirical results support the effectiveness of this approach.

(Note: my review has been revised by LLM for improving the grammar.)
weaknesses: - **Most of my concerns are resolved during the rebuttal process.**

- ~~I see several issues with the introduction and related work section.~~
  - The authors primarily build their argument by citing studies focused on generating synthetic data for *instruction tuning of LLMs*, such as Alpaca.
  - However, the method proposed in this paper is aimed at mitigating noise in synthetic data for *text classification*, which might create some confusion.
  - The confusion arises because there are many studies focused on generating synthetic data specifically for text classification, which are more relevant to this work, but the authors do not cite them.
    - Meng et al., Generating Training Data with Language Models: Towards Zero-Shot Language Understanding, NeurIPS 2022.
    - Meng et al., Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning, ICML 2023.
    - Gao et al., Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning, ICLR 2023.
    - Choi et al., UniGen: Universal Domain Generalization for Sentiment Classification via Zero-Shot Dataset Generation, arXiv Preprint 2024.
    - Zou et al., FuseGen: PLM Fusion for Data-Generation-Based Zero-Shot Learning, arXiv Preprint 2024.
  - I believe discussing these studies would be more relevant compared to the current organization of the manuscript.
- ~~Accordingly, the authors should have discussed these works and used them as baselines.~~
  - For instance, SunGen (Gao et al., 2023), UniGen, and FuseGen propose methods for mitigating the noise in synthetic data.
  - SunGen, in particular, is very similar to this work, as it suggests reweighting synthetic data based on quality. While SunGen is based on meta-learning and may require more computation than the proposed method, the current organization of the manuscript does not adequately address this overlap.
  - Furthermore, FuseGen suggests a reweighting scheme that does not require additional computation, unlike the proposed method.

~~Given these points, I do not believe this paper is ready for publication at ICLR in its current form.~~

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: In some formulas, it's unclear whether the pairs (xi, yi) are drawn from the real-world distribution P or the synthetic data distribution Q. For instance, in Appendix C, it is clearly noted that (yp', xp') is from distribution P. Could you use similar notation consistently throughout the main body to help readers differentiate between real and synthetic data sources?

To improve clarity, consider starting a new paragraph with the sentence: "Thus, inspired by the objective of Hu et al. (2019), we introduce two novel, efficient, automatic weighted-loss approaches: Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP-Loss)..." This will make the contributions of this paper stand out more clearly, preventing them from being mixed with discussions of prior work.
soundness: 3
strengths: Overall, this paper is well-written, addressing a compelling and practical problem for researchers working with synthetic data. The issue of model performance degradation when using synthetic data is aligned with recent research findings. The proposed loss functions IMP-Loss and DIMP-Loss are creative and effective solutions.

The clarity of the paper is a strong point, with a well-defined problem formulation and a meticulous derivation of the proposed loss functions. Each step in the reasoning process is clearly outlined, making the theoretical contributions easy to follow.

The paper includes extensive experimentation, which demonstrates the efficacy of the proposed methods. The analysis of the runtime complexity further adds the practical applicability of the algorithm in real-world scenarios.
summary: This paper proposes a modification to the loss function to improve the performance of models trained on synthetic data, making it comparable to models trained on real-world data. This approach addresses the challenge of limited availability of real-world data, where models trained solely on synthetic data often underperform. The authors introduce two new loss functions—IMP-Loss and DIMP-Loss—which incorporate mechanisms for quality-checking and diversity-checking the synthetic data. IMP-Loss prioritizes synthetic samples that closely resemble real-world distributions, while DIMP-Loss emphasizes samples that guide the model toward the performance of a small base model trained on limited real-world data. Experimental results demonstrate that these loss functions enhance the performance of models trained on synthetic data, outperforming traditional baseline methods.
weaknesses: The core idea of weighting synthetic data samples is not entirely novel. A similar concept was explored in the paper "Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning" published last year. Although the formulation and implementation in this paper differ, the underlying principle of prioritizing certain synthetic data points is conceptually similar, which somewhat limits the novelty of the contribution.

Another concern lies in the strong reliance on the assumption that a model trained on a small real-world dataset P can approximate the performance of a model trained on a larger, complete dataset. The paper would benefit from a more in-depth discussion or experimentation addressing this assumption. Specifically, it would be helpful to explore the minimum number of real-world examples required for this assumption to hold and to provide guidelines on the amount of data practitioners should aim to collect.

Meta-review: This paper introduces IMP-Loss and DIMP-Loss, two novel weighted-loss functions designed to improve the performance of models trained on LLM-generated synthetic data. The approach addresses challenges such as the misalignment between real-world and synthetic data distributions and the limited availability of real-world data. There are a few key contributions. First, the Loss Function Design: author proposed IMP-Loss based on importance sampling, to assign weights using both quality and diversity metrics; authors also proposed DIMP-Loss, inspired by online batch selection methods, to simplify diversity estimation by using the current model’s predictions, trading slight performance for reduced computational cost. Both loss functions assign nonuniform weights to each data point instead of discarding data, unlike traditional filtering techniques. Quality and Diversity Metrics is another contribution. Derived from the conditional probabilities predicted by a BERT-base model trained on a small real-world dataset (200–400 samples), Quality Weights are obtained. Both losses are also showed to obtain Diversity Weights. Solid experimental Results are provided to show efficacy of proposed methods. Both IMP-Loss and DIMP-Loss consistently outperformed traditional loss functions (e.g., cross-entropy loss, focal loss, and meta-learning weighting) and even improved performance on real-world data. This study highlights how prioritizing quality and diversity in synthetic data training can bridge the performance gap between models trained on synthetic versus real-world data, making synthetic data more viable for machine learning tasks. 


Strength of this paper

Novelty and Motivation: the introduction of IMP-Loss and DIMP-Loss. The motivations for incorporating quality and diversity in the weighting strategy are well-articulated and grounded in prior challenges with synthetic data.
Clarity and Structure: The paper is well-written and clearly structured, with a well-defined problem formulation and step-by-step derivation of the proposed methods.
Efficiency and Practicality: The use of approximations improves computational efficiency, making the approach more practical than meta-learning-based alternatives. Runtime complexity analysis demonstrates the applicability of the methods in real-world scenarios.
Empirical Validation: Comprehensive experiments validate the efficacy of the methods, with consistent performance improvements shown in results tables and figures. The empirical results are compelling, showing the superiority of the proposed methods over baselines across multiple scenarios.
Timeliness and Relevance: The work addresses a long-standing issue of noise in synthetic data, building on prior research while mitigating its limitations.

Weakness of this paper

Several reviewers raised few concerns/limitations of this paper. By addressing these limitations, the paper could strengthen its experiment and expand impact.
1. Convergence Uncertainty: The experiments use only five training epochs, and it is unclear whether the models have fully converged. Analyzing the proposed weighting method's impact on convergence speed and performance at full convergence would add depth to the results.
2. Scalability to Larger Models: The experiments rely on a BERT-base model, which is relatively small compared to contemporary large language models (LLMs). It is unclear how the method scales to larger models in terms of performance improvements and computational cost.
3. Novelty Concerns: The core idea of weighting synthetic data is not entirely new, as similar concepts have been explored (e.g., in "Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning"). While the formulation and implementation here are distinct, the conceptual similarity somewhat limits the novelty.

Decision: Accept (Spotlight)