Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification
Authors: Hsun-Yu Kuo
Yin-Hsiang Liao
Yu-Chieh Chao
Wei-Yun Ma
Pu-Jen Cheng
Venue: ICLR 2025 Spotlight (2025)
Keywords: data weighing
data augmentation
distillation
data-efficient training
NLP in resource-constrained settings
fine-tuning
weighted loss
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Weighted loss for training on data generated by LLM

Abstract: Synthetic data augmentation via Large Language Models (LLMs) allows researchers to leverage additional training data, thus enhancing the performance of downstream tasks, especially when real-world data is scarce. However, the generated data can deviate from the real-world data, and this misalignment can bring about deficient results while applying the trained model to applications. Therefore, we proposed efficient weighted-loss approaches to align synthetic data with real-world distribution by emphasizing high-quality and diversified data generated by LLMs using merely a tiny amount of real-world data. We empirically assessed the effectiveness of our methods on multiple text classification tasks, and the results showed that leveraging our approaches on a BERT-level model robustly outperformed standard cross-entropy and other data weighting approaches, providing potential solutions to effectively leveraging synthetic data from any suitable data generator.

Published as a conference paper at ICLR 2025
NOT ALL LLM- DATA ARE EQUAL: RETHINKING DATA WEIGHTING IN TEXT CLASSIFICA- TION
Pu-Jen Hsun- Yu K..51.2.3., Yin-Hsiang L.20 Yu-Chich Chao', Wei- Yun Mat.f.t, Pu-Jen Cheng2.1 Academia Sinica National Taiwan University
Swiss Federal Institute of Technology in Lausanne (EPFL)
hsun-yu. www.yuukuoupp....omm
{zenonliao, vpj870331, majeiis. majeiis.sinica.
pjcheng@csie.ntu..dee
ABSTRACT
Synthetic data augmentation via Large Language Models (LLMs) allows re- searchers to leverage additional training data, thus enhancing the performance of downstream tasks, especially when real-world data is scarce. However, the gen- erated data can deviate from the real-world data, and this misalignment can bring about deficient results while applying the trained model to applications. There- fore, we proposed efficient weighted-loss approaches to align synthetic data with real-world distribution by emphasizing high-quality and diversified data generated by LLMs using merely a tiny amount of real-world data. We empirically assessed the effecttiveness of our methods multiple text classification tasks, and the re- sults showed that leveraging our approaches on a BERT-level model robustly out- performed standard cross-entropy and other data weighting approaches, providing potential solutions to effectively leveraging synthetic data from any suitable data generator.
INTRODUCTION
The quantity and quality of data play significant role in many tasks of Natural Language Processing (NLP). However, due to the scareity of data in a particular domain for a specific task, we may need expertise collect such data, resulting in budget limitations. Fortunately, Large Language Models (LLMs) provide a practical solution to this problem. LLMs, such as GPT series (Brown et al.) 2020 OpenAl 2022; OpenAl et al. 2024), can be leveraged to generate synthetic data that mimics real-world examples, thereby enriching the training set [Wang et al. 2023). Taori et al. (2023). and other works (Ye al. [022] West et al.) [202]] Li al. (023) have shown the capability of using L.M-ggenrrted data for the downstream tasks, and it seems to be new cut-in solution to any NLP downstream tasks. However, training models with LLM --neerred data can lead to drawbacks such as model collapse (Shumailov et al. 2023 Dohmatob et al., 2024), tail phenomena, reinforcing LM biases (Wang et al.J (003).. Morcover, based on our empirical study, the performance of models trained synthetic data without proper processing can be lower than models trained much smaller real-world data (Sec. 3.1. highlighting the uncertainty of using L.M.-generated data.
Previous works took filtering strategy to get high quality or variant data (Dubey et al. 2024 MetaAl 2024 Chiang et al.] 2023) West et al.) [2022] Meng et al. [2022] 2023). Still, this strategy was mostly human-crafted needed efforts to train a judge model, and most importantly, filtering strategies abandoned the potential of the filtered data that may contribute to the final performance. In contrast, data weighting approaches leverage all the training data, including augmented and biased data, but prioritize data by giving nonuniform weights to the loss of each data point. For example, Focal-Loss
Lin et al. 2017) prioritized more diverse data; Hu et al. (2019) and SunGen (Gao al. 2023)
"Joint first authorship: Data Science Degree Program, National Taiwan University and Academia Sinica; Work done while Hsun-Yo Yu Kuo and Yu-Chich Chao were at Academia Sinica; #Corresponding author; Code: htts://gihuh..om/HHaao Yu/DIMP-Loos

---

Published as a conference paper at ICLR 2025
optimized the weights of training samples that the model performs best small real-world dataset. It is worth noting that while et al. (2019) and SunGen steered the training toward higher performance, using these methods in a karge-scale scenario seems infeasible because the weights are regarded as learnable parameters, the number of which increases as the training set grows.
Thus, inspired by the objective of Hu et al. (2019), introduce two novel, efficient, automatic weighted-loss approaches: Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP- Loss), which are designed closely align the distribution of synthetic data with that of real-world data. Furthermore, both IMP-Loss and DIMP-Loss incorporate mechanisms as quality-checkers and diversity-checkers, assigning higher quality and more diverse data points with greater weight. In other words, these methods prioritize data points that are more relevant and more informative to target downstream tasks, thereby reducing the impact of less valuable data on the fine-tuning model.
To validate our approaches, we conduct comprehensive empirical studies focusing on various text classification tasks by comparing the performance of models trained with our novel weighted-loss objectives under different conditions: 1) models trained exclusively
LLM-generated datasets using few-shot prompts from a limited real-world dataset (Sec. b.1 2) models trained on large, real- world datasets (Sec. (.2): and 3) models trained on noisy datasets (Sec. Gb. Our findings indicate that using a small real-world dataset to build the quality checkers and incorporating diversity checkers highly enhances performance, even surpasses the few-shot prediction accuracy of the tremendous data generator (Sec. 5. This demonstrates the efficacy of our methods in leveraging little real- world data to improve models trained on L.L datasets. Notably, DIMP-Loss is efficient in terms of model size (Sec. 5.1), data requirements (Sec. [5.1), and computational resources (Sec. 4.4
making a practical solution enhance downstream performance.
PRELIMINARIES
2.1 CROSS ENTROPY LOSS ON REAL- WORLD DATASET
In supervised learning for text classification tasks, we consider a real-world dataset Dp ((x,,)))) 1 comprising M samples. Each pair (x, yi) is drawn independently and identically distributed (i.id.) from the joint distribution P(X' V)). where x; represents the input sample and y: y = 21,2, .,,,....7 C} is the corresponding class label. This setup forms the basis for training models using the empirical cross-entropy loss (CF-Loss), standard loss function in such tasks. The CE-Loss over the entire dataset Dr is calculated as follows:
Σ log P(ys|x;;; 0) 2 Er CCE(9, (cE(O,Dr)= M
[-log log P(v|x;
where (x;; 0) is the predicted probability of the model with parameters for the true class label y given inputx; x,. The CE-Loss converges in probability to the expected version of conditional cross- entropy under the true joint distribution P(X,)) by the law of large numbers. This convergence
is erucial because the minimizer of the CE-Loss occurs if and only if the predicted distribution P(y|x; 0) matches the true distribution P(y]x).
2.2 WEIGHTED CROSS ENTROPY Loss (WCE-Loss)
WCE-Loss is a modification of the standard CE-Loss that assigns different weights to each data point. It is defined as:
N Σ w, log |
Lw w)
Here, w; represents the weight assigned to the i-th data point (x;, y/). A higher weight w, assigned to a data point [x;, yi) indicates the data point has a greater influence on the model's learning or adjustment of parameters, thereby being considered more important for the training process.
There have been several variants of the weight function, such as Focal Loss LF1. (Lin et al. 2017).
addressed class imbalance and reduced the impact of easily classified examples its weight function
2

---

Published as a conference paper at ICLR 2025
was defined as wo (1 P [],[xx; 0)); where y 2 1 focusing parameter that adjusted the rate at which easy examples were down-weighted. Research has shown that models trained with Focal Loss were better calibrated under the iid. assumption and performed well under distribution shifts (Mukhoti et al. 2020). This made Focal Loss a promising baseline for evaluating our proposed weight function in the context of L.M-ggnnrated synthetic data training.
Additionally, a series of meta-learning approaches addressed these challenges by leveraging bi-level optimization to dynamically adjust weights based each instance's contribution to the meta-set from the real world. These methods handled class imbalance, noisy labels, and augmented data by reweighting these instances based on their gradient direction or model outputs, providing a flexible mechanism for weighting data points (Ren et al. [2018] Hu et al.) 2019; Gao et al.] 2023). While effective, meta-learning-basee -aaedd approaches were computationally expensive, making them difficult to scale up to larger datasets or complex models. In contrast, our methods share the same objective of optimizing performance on real-world data but achieve it without meta-learning. This makes more computationally efficient and scalable while still maintaining high performance.
3 OPTIMIZATION ON LLM-GENERATED DATASET
LLMs are capable of generating synthetic datasets (Lu et al. 2023 West et al. 2022 Li et al. 2023), denoted as Do = [(x,, }:)} sourced from the distribution Q(2,)) V). This distribution is shaped by specific prompts comprising instruction prompts, system prompts, and few-shot examples that guide the LLM's output. This method offers a valuable alternative for acquiring training data, especially when access to real-world data is limited. Moreover, the relevance of Q can be further refined by incorporating few-shot examples from small real-world dataset Dp', enhancing the utility and applicability of the synthetic data (Li et al. (202)).
The CE-Loss on the -LM-generated dataset converges to the expected cross-entropy under Q:
(0 D2) 1s Eq
| log P()]x;
(3)
A significant distributional shift between Q and P may lead to suboptimal predictive performance on real-world data.
3.1 UNCERTAINTY OF LLM-GENERATED DATA PERFORMANCE
Our empirical study, shown in Table I. demonstrates notable variability in the performance of CE-
Loss on LL.--genrraed datasets. Specifically, on the Financial (Malo et al. 2014) and MRPC (Wang al., 2018) benchmarks, CE-Loss on large L.LM-generated datasets (> 3k samples) per- forms worse than training on small real-world datasets, which contain only around 200-400 sam- ples. In contrast, CE-Loss in -LM-eeeerated data improves accuracy for the Twitter Irony (Van Hee et al. 2018) benchmark. This variability underscores the uncertainty associated with using CE- Loss on LLM--eneraate showing that when using CE-Loss, without proper filtering, LLM -eaeerttdt data may lead to decent results on downstream tasks, even though its size considerably larger than that of real-world data.
data. These findings are consistent with another research (West et al., 2022).
3.2 POTENTIAL OF LLM-GEEERTTTDD DATA: MODEL-BASED INFORMATION MEASUREMENT
We employ Eiformation-theoreiee metrics to evaluate the uncertainty within the conditional distribu- tions of real-world data and L.LM-generated data. A higher conditional entropy indicates more significant uncertainty given an input x, suggesting various outcomes. We estimate this by sepa- rately fine-tuning BERT model (Devlin al., 2019) on both datasets. Higher conditional entropy is often associated with greater diversity within the dataset, reflecting a broader range of informa- tion that the model must learn to predict accurately. The conditional KL divergence quantifies the difference between two conditional distributions, P(ylx) and Q x), showing how well model trained on one dataset describes another.
It is also called conditional divergence or conditional relative entropy.
3

---

Published as a conference paper at ICLR 2025
We show these metrics for financial benchmark scenario. The real-world dataset Dp exhibits significantly lower conditional entropy (H/() = 0.0365) compared with the LLM-ggnerrted dataset Q (Ho tional KL, divergence from P to Q (DRL(Q) P) = 1.8781) is much greater than it from Q to 0.444), suggesting that models trained real-world data struggle capture the complexity of the synthetic dataset. Those models trained on the synthetic dataset are rela- tively efficient, requiring fewer additional nits on average encode samples from the real-world distribution P. This difference, along with the results from Sec. [.1] highlights that, although the synthetic dataset contains some points that are less representative of the real-world distribution, still includes substantial proportion of relevant data points. This analysis indicates the potential to improve modeling techniques to utilize LLM -.M--nnrrated data's rich, informative content.
0.2299), indicating that DQ is more diverse. Furthermore, the condi-
P (Pki(P
3.3 PROBLEM FORMULATION
In this study, we devise weighted loss function that transforms CE-Loss from LLM-generated data distribution Q to match real-world data distribution P. We assumed the dataset Dq is i.i.d. and the LLM can approximate the real-world imput distribution P(x) through strategic prompting, effectively simulating Q(x) se P(x). For example, by using system prompts like "Now you are a articles,"
journalist writing
produce synthetic texts that closely mimic authentic news articles. Lastly, we use a small set Dr', approximately 200-400 samples from real-world datasets, facilitate the alignment process. These samples i.i.d. from the distribution P. We use P" as the probability function representing this small real-world dataset.
This approach leverages the rich diversity of LLM-generated data to bridge the distributional gap between Q and P. By creating an effective weighted loss function, we aim to enhance model performance on real-world tasks by better aligning synthetic data with real-worke distributions.
METHODOLOGIES
In this section, we present our Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP- Loss) methods, which transform the CE-Loss to align with the real-world distribution P from the LLM-generated distribution Q.
4.1 IMP-LOSS: TRANSFORMATION FROM Q TO P
To achieve convergence to the real-world data distribution P. we applied WCE-loss. Inspired by the Monte Carlo method of Importance Sampling (Hesterberg) 1995), used to estimate expectation values from a source distribution to target distribution, we design the weight function as follows:
P(y[x;)
0;
Q(y(x;)
(4)
By applying this weight function to WCE-Loss, the asymptotic convergence is approximately the expectation under P (details in Appendix
B
Eq
log P()xx:0)
Q(ylx)
Σ ((x)
P(ylx) log P()|x
xEX
NEy
(5)
Σ P(x)
lo ()
XEX
NEY
Ep
log P(y|x;
The approximation in the penultimate step is based on the assumption stated in Sec. 3.3 the LLM simulate the real-world input distribution through carcful and appropriate prompting. This trans- formation ensures that the WCE-Loss effectively aligns Q with P.
Further, Q can be estimated by fitting a neural model Q. such as BERT, on the LLM -LM-generated dataset D2 using the CE-Loss; however, estimating the weight function challenging because the
4

---

Published as a conference paper at ICLR 2025
real-world distribution P is unknown. To address this, fit model p on small real-world dataset Dp. Using P and Q, we define the Importance Loss LiMP(0, DQ) as follows:
Quality Checker
1
P'(va Xi)
(0 DQ)
log P(ys [x;]
(6)
(x)
Diversity Checker
Algorithm outlines how we use IMP-Loss.
Algorithm Training with Importance Loss
Require:
Small real-world dataset Drr, synthetic dataset Di, model P, initial parameters
Step 1: fr < Estimation of P' by fitting model with CE-Loss on Dp
Step 2: Q t Estimation of Q by fitting model with CE-Loss on DQ
P'(y(x) for each training sample (x. v) E DQ
Step 3: Compute the weights w; Qu(xx
Step 4: Optimize model parameters to minimize Li D2) by SGD
4.2 DIMP-LOSS: WHICH DATA POINT CAUSES THE MODEL TO BE CLOSEST TO P?
In this section, drawing inspiration from online batch selection methods (Deng et al. 2023 Min- dermann et al. 2022). we investigate which data point in Do. when used for training, will most efjectively bring the distribution of the model closer to P in the subsequent optimizution step. In optimization formulation, this can expressed as:
(x" ".y" arg min Ep llog P (y]xx O, ((x ((xx,1))))
(xf (EDg
where 0, represents the model parameters at optimization step t. Consider a one-step optimization algorithm f (e.g. SGD), where 0.+1 + f(O,, (x'','')) The algorithm updates the model param- eters Og using (x'; 3x) to obtain the new parameters 0.+1 after one optimization step. The Eq. 7 means the data point (x" y²) is the optimal data point Do that leads the lowest conditional cross-entropy after one update step. Specifically, it identifies which data point is used for training
results in the model parameters that yield the model closest to the wallworld distribution P.
In empirical settings, we may not have access to the complete real-world distribution P. but we can approximate by a small real-world dataset Dp., also denoted as (yp', Xp.) in the perspective of labels and inputs. This allows us to rewrite the objective as maximizing the probability:
P(u|x; Dpr) 0())x;0))
arg max P(y Xp:; 62, ((x,)))) arg max (x.))EDg
(x.j)eDo
Eq. aims to maximize the joint likelihood of all data points Dp. The joint likelihood involves inferring all data points in Dp and multiplying their prediction probabilities (due to the i.id. as- sumption). However, this optimization is infeasible, as it requires updating the model for each data point in Do. resulting in Dol models, and each needs evaluation the whole Dp
Notably, by applying Bayes' rule, we derive the right-hand side of Eq. g (see Appendix C for details), showing for each data point in DQ: the denominator P(y|xx; 0,) is the current model in step and the nu- merator P(y] x; Ot, Dp.) would require additional training on Dp.. To simplify, we approximate Ply|x; 0,, ) with ('(y)x), the probability estimated from Dp as in Deng et al. (2023).
feasible calculation approach. This requires evaluating only two models
The approximation of Eq. is then utilized as the weight in loss function. Consequently, if data point brings the model closer to the real-world distribution P. its corresponding weight will be higher, thus having greater impact on training. Thus, we define the Dynamic Importance Loss
5

---

Published as a conference paper at ICLR 2025
(DIMP-Loss) LUIM (0z, D2)
Quality Checker
1
('(y((x)) log P(u](x;;00)
CDIMP(Or;
(9)
P(y 0c)
Diversity Checker
The approximation of Eq. simplifees the calculation of the weight function, making the implemen- tation of DIMP-Loss practical. We can observe this weight function dynamically changes each optimization step and adjust the weights based on the current parameters O,, thereby continually refining the alignment between the model Pe and the real-world data distribution P. Algorithm outlines how DIMP-Loss is used training model.
Algorithm Training with DIMP-Loss
Require: Small real-world dataset Drs, synthetic dataset Do, model P, initial parameters 0
Step 1: pr + Estimation P' P'ylx) by fitting a model with CE-Loss on Dp
Step 2: Compute the ("(xxx) for each training sample (x, y) E DQ
Step 3: Optimize model parameters minimize LDMMP (0, Dq) by SGD
To better understand the properties of DIMP-Loss, we derived lower bound for (details can be found in Appendix D). Precisely, we have:
(0,, DQ) 2 P'(yi(x;))og P(yl(x;;0))
P(y) 0x) Po) (10)
Empirical distilled cross-entropy loss
Maximum entropy regularizer
DIMP-Loss can be interpreted as an upper bound on the regularized empirical distilled risk (Menon al. 2021| Wang et al. 2022), where the "teacher" model is the quality checker. The regularizer is a maximum entropy term designed to prevent overconfidence in output distribution (Pereyra et al.) 2017). In this context, DIMP-Loss can also be viewed as form of knowledge distillation, where the knowledge from model is trained on small amount of real-world data. The objective is to align the predicted distribution Po with p' while promoting higher entropy in Po to avoid overly confident predictions.
4.3 QUALITY AND DIVERSITY CHECKERS IN IMP-LOSS AND DIMP-LOSS
According both Eq. G and high weight means the data point significantly influences the model. The Quality Checker (u::)x)) assesses the likelihood data point sampled from the real-world distribution P. Higher values indicate the data point is, highly relevant, and unambiguous for the real-world distribution P.
The Diversity Checker differs in the two losses, Q (x)) for IMP-Loss, and P(y. x,; 0,) for DIMP-Loss. In the context of IMP-Loss, low Diversity Checker value Q( xx) suggests the data point contains high amount of information within the LLM-generated dataset DQ. because a redundant data point in DQ will have a high probability, indicating less diversity. Hence, serves as indicator of diversity from the perspective of the i.LM-generated distribution. In contrast, for DIMP-Loss, a low Diversity Checker value P(y.|x, x;; 0t) implies the data point is challenging to be learned in previous steps, departing from the data points the model has already learned. Thus, Di- versity Checker of DIMP-Loss reflects diversity from the perspective of model. This distinction highlights how each loss function prioritizes different aspects of data diversity during training. We simulated defect and redundant situations for further exploration in the Appendix. G.3
4.4 COMPUTATIONAL. COST OF TRAINING WITH IMP-LOSS AND DIMP-LOSS
The analysis covers computational requirements and practical run-time detailed in Appendix

---

Published as a conference paper at ICLR 2025
IMP-Loss. According Algorithm[ the computational cost of training with IMP-Loss is approx- imately (ignore the cost on P twice training plus twice forward pass on DQ. First, we estimate ('(y)) by fitting a model on Dr, and Qz/xx) by fitting model on DQ. respectively. Second, compute the weights for each sample Do by (y)xx) and ('(y)x). Although the estimation of (' x) incurs minimal cost due to the small size of Dpr, the primary additional overhead comes from the repeated training on Dę and the additional forward passes needed to compute the weights.
DIMP-Loss. According to Algorithm 2 the computational cost of training with DIMP-Loss is approximately (ignore the cost on Drr) one training plus one forward pass on Dq. On the one hand, we need to a model on the small real-world Dr estimate ("(yl x), the numerator of the weight coefficient, for each data point in DQ- On the other hand, compute the weights for each sample in Do using the DIMP-Loss formulation, which involves evaluating the computed log P(yi 0e) and hence getting P(u: x;; 0t). Without the estimation to Q(y(x), x). DIMP-LoSS more efficient than IMP-Loss. The computational overhead is only slightly higher than that of CE-Loss, because of the additional step of estimating P from the small dataset Dp and performing a single inference pass on Do. as the values of the quality checker for each data point remain constant in training.
5 EXPERIMENTS
Financial
Tweet Irony
MRPC
Dataset
Method
Acc
F1
Acc
F1
Acc
F1
GPT:3.5 few-shot
79.46
81.6
63.39
69.39
69.28
71.75
CE-Loss (quality checker)
78.05
75.26
62.5
62.38
73.16
68.69
Small real world
Focal-Loss
78.47
76.2
67.73
62.32
73.10
66.64
DIMP-Loss (Ours)
79.87
77.05
69.01
67.05
74.84
66.80
Ch-LLoss
77.39
74.01
76.91
76.8
72
65.47
Focal-Loss
79.29
75.32
74.87
74.82
72.17
62.77
Hu et al.
71.7
61.93
71.42
70.18
67.13
50.08
GPT-3.5 generated
SunGen
80.45
76.87
78.96
75.06
71.65
66.08
IMP-Loss (Ours)
82.09
79.40
81.89
81.71
75.83
70.52
DIMP-Loss (Ours)
82.67
79.53
78.44
78.14
75.83
70.04
w/o diversity checker
81.35
77.94
77.68
77.62
74.72
69.34
CE-Loss
84.74
82.69
68.75
68.41
80.92
77.73
Focal-Loss
84.98
81.98
67.6
67.19
80.35
76.28
Large real world
Hu et al.
80.19
76.58
60.33
37.63
71.36
67.78
SunGen
84.65
82.51
63.9
62.66
80.81
78.78
IMP-Loss (Ours)
85.3
83.27
70.15
70.08
81.33
78.3
DIMP-Loss (Ours)
85.4
82.79
68.78
82.84
80.49
Table 1: Performance metrics across datasets and methods. The table showcases each combination's accuracy (Acc) and macro Fl score (FI). The methods include GPT-3.5 few-shot, CE-Loss, Focal- Loss, Hu et a method, SunGen, IMP-Loss, and DIMP-Loss. Bold entries denote the performance within 0.5%, compared to the best performance of each training source.
We assessed our proposed methods by comparing them with standard loss functions across several text classification benchmarks, including Financial Phrasebank (Financial) (Malo et al. 2014). irony detection (Tweet Irony) (Van Hee et al., 2018), and the MRPC dataset from GLUE (Wang et al.) 2018). Detailed descriptions and specifications are provided in Appendix H In our experiments, we referred the large real-world data Dp to the original training set from each benchmark and the small real-world data Dp. to the original development set, with the sizes from approximately 200 to 400, as shown in Table 5 Our experiments explored three different scenarios: training solely on synthetic data (Sec. 5.1 real-world data (Sec.55 5.2), and noisy data (Sec. G We evaluated Accuracy (Ace) and Macro Fl score (F1) for every benchmark. These metrics were computed by comparing the model's predictions with the gold labels provided in the test sets. We used BERT-baaee model for fine-tuning and building the checkers. The Appendix FaannddxxFFettaa details the configurations.
Baselines. CE-Loss, Focal Loss, SunGen and Hu et al. (0019) are our baselines, detailed in Sec. 2.1 and Sec. 2.2 Focal Loss addressed class imbalance and mitigated easily classified
7

---

Published as a conference paper at ICLR 2025
data's impact, preventing overconfidence. The weight function for Focal Loss was defined as Wy; (1 Ply 0))7 where 7 controled the downweighting of easy examples. Mukhoti et al. (2020) showed models trained with Focal Loss exhibited better calibration under the i.i.d. assump- tion and performed robustly under distribution shifts. This made Focal Loss strong baseline for evaluating our proposed approaches. Both SunGen and Hu et al. (2019) are bilevel optimization methods but differ in their weight update mechanisms and the objective functions of their outer loops. They used meta-learning to dynamically adjust weights training data by maximizing the like- lihood on small real-world dataset, similar IMP-Loss and DIMP-Loss; however, our methods directly adjust weights based quality and diversity checkers, while Hu et al. method and SunGen relied on meta-learning to optimize weights indirectly
5.1 TRAINING ON L.M-GENNEATED DATA
We compared our proposed methods with standard loss functions on L..M-generteed data.
Data Generation. We used GPT-3.5-turbo-11 106 (OpenAl 2022), given a system prompt, 8-shot examples from the development set Dr, and the corresponding labels to generate the input text. For the Financial and Tweet Irony, our generation prompt based on previous research (Li et al. _023). Similarly, for the MRPC benchmark, the prompt included pairs of sentences with answers, which automatically guided the LLM in generating the answers. See Appendix I.I for details.
IMP-Loss and DIMP-Loss Outperform LLM-Generated Data. As shown in Table our methods outperformed all baselines in all benchmarks. For instance, in Financial Phrasebank, TMP- Loss achieved 82.09% in accuracy and 79.40% Fl score, and DIMP-Loss reached 82.67% and 79.53% respectively, while the CE-Loss reached 77.39% / 7..01%. The result showed if we use the baselines, CE-Loss, Focal-Loss, SunGen and Hu et al. (0T9)) to train a classifier on Dg. the performance could be worse than that of using CE-Loss on much smaller Dr (quality checker). In contrast, our methods consistently outperformed the small quality checker, encouraging the usage of abundant LLM-generated data. Notably, even when the quality checker performs poorly, such as on the Tweet Irony dataset, where the accuracy was 62.5%, which was lower than the 76.9% achieved by directly training generated data using CE-Loss, methods still delivered strong performance. This suggested that a high- peeefperffrmmece quality checker was not a prerequisite for SunGen the effectiveness of our methods. Although the performance of the meta-learning-baase method was, in some cases, close to that of our methods (though still slightly below), our approaches have significant advantages in computational efficiency, making them more practical for large-scale applications. Further details on computational efficiency are shown in the Appendix
IMP-Loss and DIMP-Loss Surpass the Accuracy of the Data Generator. The GPT-3.5 few- shot predictor generated predictions using 8 examples from the small real-world dataset in the input prompt. GPT-3.5 achieved 79.46% in the Financial dataset and 68. 18.82% in the MRPC dataset. Our approaches consistently surpassed the GPT-3.5 few-shot prediction in accuracy. The parameter size of the fine-tuned models using our methods was significantly lower than that of the GPT-3.5 data generator, yet they delivered higher performance.
Superior and Robust Accuracy Across Epochs. The training dynamics in Figure I revealed our methods outperformed CE-Loss and Focal-Loss across all benchmarks. Notably, both IMP-Loss and DIMP-Loss achieved low variation by the end of training, indicating stable performance. Moreover, DIMP-Loss showed higher variation in the initial epochs compared with IMP-Loss. This increased variability could be attributed to the order of sampled data, which caused initial fluctuations. Nev- ertheless, the Acc ultimately converged at a higher value than the baselines.
Quality Checkers are Data Efficient. Figure illustrates the test accuracy on the Financial benchmark with quality checkers trained on various proportions of the original training set. As seen in this figure, even a small number of data points, e.g., 10%, was sufficient to enhance the performance of both IMP-Loss and DIMP-Loss. This suggested that small amount of real-world data was effective for building a quality checker, making our approach efficient and practical.
We implemented Focal-Loss, SunGen and Hu al. 2019 by using their official code.
8

---

Published as a conference paper at ICLR 2025
072
0.00
0.75
0014
0.7
0.72
(0mm)
(Sant)
CSiom
(Dun)) Vihaie
Figure 1: Training dynamics shows the testing accuracy over five epochs for benchmarks. This chart displays the minimum, maximum, and average accuracy observed across four runs with different random seeds, comparing our proposed methods with the standard CE-Loss and Focal-Loss.
Model Size
Method
Financial
Tweet irony
MRPC
Base
Quality checker
78.05
62.5
73.16
CE-Loss
80.45
78.83
74.2
IMP-Loss (hase DC)
80,94
74.23
75.36
Large
IMP-Loss (large DC)
81.93
78.83
76.41
DIMP-LosS
83.25
81.25
77.04
Table 2: Accuracy of methods on benchmarks when training a larger model with smaller Quality Checkers. "base DC" and "large DC" denote smaller and larger Diversity Checkers, respectively. Bold entries highlight the top value of metrics within each dataset.
Diversity Checkers are Important. The results in Table highhihtiel the importance of Diversity Checkers our proposed methods. When training on GPT-3.5 generated data, the performance of the model trained with IMP-Loss without Diversity Checkers dropped compared with IMP-Loss with Diversity Checkers. For instance, in the Financial dataset, the accuracy drops from 82.09% to 81.35% and the Fl score from 79.40% to 77.94%. These results indicated that incorporating Diversity Checkers helped effectively use LLM-ggnerated data.
Smaller Quality Checker Still Enhances Performance by DIMP-LosS. The results in Table 2 illustrrated the performance of each method the benchmarks when training larger classifier (BERT-large) with smaller Quality Checkers (BERT-base). Notably, DIMP-Loss consistently per- formed well even when the Quality Checker small. This demonstrated the robustness of DIMP- Loss in adapting to different model sizes for Quality Checkers. contrast, IMP-Loss showed inconsistent performance when using smaller Diversity Checker compared with its training model, indicating the choice of the Diversity Checker in size significantly impacted its efficacy. In short, using a smaller Quality Checkers guide the model efficient in terms of both space and time.
5.2 TRAINING ON REAL WORLD DATA
Robust Performance of IMP-Loss and DIMP-Loss Real-World Data. As shown in Table I IMP-Loss and DIMP-Loss outperformed other baselines even when applied directly to real-world data. Although the performance improvements less than that of using GPT-3.5-generated data, the results indicated our methods were versatile and able to handle multiple sources of training data effectively. Specifically, in the Financial dataset, IMP-Loss achieved 85.3% Acc and 83.27% Fl score, while DIMP-LoSS reached 85.4% Acc and 82.79% F1 score, surpassing CE-Loss, Focal-Loss, and (Hu et al. 2019). From our perspective, the reduced improvement in this scenario was due to the lack of requirement to shift the training data distribution. Regarding the asymptotic viewpoint, the optimal solution of cross- entropy is already the best solution when training on real-world data. Nonetheless, our methods demonstrated robust performance across various conditions.
RELATED WORKS
We list some essential related works in this section and others in A

---

Published as a conference paper at ICLR 2025
0.86
Treacal Ted: Aeaaay
0.84
0.78
16%
3014
50%
70%
90%
Lage Saaurrorza www.ytyyyyc............ claccyy
+quaiiy
Figure 2:
Test accuracy on the Financial with varying percentages of the training set for the quality checker. The graph shows the performance of each loss and the Quality Checker.
Weighting for Misalignment Data Importance weighting (IW) serves as a classical strategy for addressing the issue of shifts between data distributions (Hesterberg) [995). In traditional applica- tions of IW, weights are derived by evaluating the degree of similarity between training and testing distributions through various statistical techniques. These techniques include maximum mean dis- crepancy (Schölkopf ct al., 2007) and the estimation of KL divergence (Sugiyama et al. 2007).
Although effective in linear model contexts, the efficacy of these methods seems to significantly diminish when applying IW to more complex deep learning frameworks (Byrd & Lipton 2019)
Besides traditional methods, recent studies have explored approaches such as Focal Loss (Lin al. 2017) and meta-learning techniques (Hu et al. 2019, Meng et al.) [2033] Gao et al. 2023). which
take the weights samples as trainable hyperparameters as discussed Sec. 2.2
Synthetic Data Generation from LMs Recent advancements in generative AI have spurred inter- est in using LMs to generate synthetic data for specific tasks, particularly in low-resource settings. Studies have explored zero-shot and few-shot settings for data generation, where LMs directly gen- erate instances with distinct labels or use little real-world data as examples to create relevant and diverse data (Li al. 2023 West al. 2022 Ye al. 2022; Wang et al. 2023 Taori al. 2023). LMs have the unique ability to generate both labels and diverse input instances, significantly en- hancing the variety and quality of synthetic datasets. Approaches like ZEROGEN synthesized data by pre-trained LMs to train smaller task models, achieving competitive performance in NLP tasks,
such as text classification (Ye et al. 2022
LM-Generated Data for Training Text Classifier Several studies have investigated leveraging
LM-generated data for text classification. Some works maintain data quality by filtering strategy
Stylianou et al.) 2023 Meng et al.) 2022 2023 Li et al. 2023) West al. 2022 Ye et al. 2022).
Another common approach is data reweighting. For example, SunCien (Gao et al. 2023) adopted a
bilevel optimization approach to learn weights for synthetic data, incorporating a noise-robust loss in the outer loop to improve the reliability, and this benefited SunGen to outperform counterparts using meta-learning for data reweighting, such as Hu al. (2019). Despite its advantages, the bilevel optimization process remains computationally expensive. making our approaches outstand by their efficiency. Moreover, there exists novel research further enhancing the use of synthetic data. For instance, UniGen (Choi et al.] (204)) utilized contrastive learning to improve generalization capabilities, but required a open-source pretrained LM, while FuseGen (Zou et al., 2024) combined synthetic data from multiple L.MSs to enhance performance. It is worth noting our approaches are compatible with UniGen or FuseGien, and potential complement and enhancement of these works.
CONCLUSIONS AND DISCUSSIONS
IMP-Loss and DIMP-Loss are novel weighted-loss objectives that further enhance the performance of models trained on LLM- data. Our empirical results demonstrated that both methods outperformed traditional loss functions across various benchmarks. Notably, DIMP-Loss was partic- ularly computationally efficient, requiring subtly additional resources while increasing performance. These findings emphasized the potential of IMP-Loss and DIMP-Loss in effectively leveraging syn- thetic data for training machine learning models. In the future, will extend methods on question answering, text generation, LLM pretraining, and other potential tasks, further exploring how quality and diversity matter for learning.
10

---

Published as a conference paper at ICLR 2025
8 REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provided the source code and generated dataset in supplementary materials, prompts used for generation in [.] hyper-parameters and other training details in[F] testing datasets' descriptions in H and theoretical results in B D In addition, for the baselines, we implemented the Focal Loss as the source code and used the publicly available code provided by
Hu al. (2019) but replaced the input data. We hope one smoothly reproduce our results via these materials.
ACKNOWLEDGMENTS
We sincerely appreciate the insightful and valuable feedback from the anonymous reviewers. This work is primarily supported by the National Science and Technology Council, Taiwan, under grant number 1TT211122221---00--22 and partially supported by E.SUN COMMERCIAL BANK. We extend our special thanks to Chien-An Chen and Yi-Ren Yeh at E.SUN for their valuable input dur- ing the preliminary discussions. Additionally, we thank the National Center for High-performance Computing (NCHC), Taiwan, for providing essential computational and storage resources.
REFERENCES
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, and Man- sheej Paul. Perplexed by perplexity: Perplexity-ban data pruning with small reference models, 2024.
Fred Bane, Celia Soler Uguet, Wiktor Stribizew, and Anna Zaretskaya. A comparison of data filter- ing methods for neural machine translation. In Janice Campbell, Stephen Larocca, Jay Marciano, Konstantin Savenkov, and Alex Yanishevsky (eds.), Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pp. 313-325, Orlando, USA, September 2022. Association for Machine Translation in the Americas. htthhtts::// acl www.aanttooooggggooooo.ooooo 2022 amta-upg. 22
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, 1877-1901. Curran Associates, Inc., 2020. URL Inttps://proceeednnggg wttp://ps...c//pppeeeeeee////eee fites/aape//ee////// Ele22590ccbccc307110833322222222
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 872-887288888811 PMLR, 09-15 Jun 2019. URL hattps://pprceeee byra19a. html
www.mmrr.ps.s. press//97
Wei-Linn Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
Juhwan Choi, Yeonghwa Kim, Seunguk Yu, JungMin Yun, and YoungBin Kim. UniGen: Univer- sal domain generalization for sentiment classification via zero-shot dataset generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 1-14, Miami, Florida. USA, Novem- ber 2024. Association for Computational Linguistics. URL https: wtttcps://acaaathooo htt////aaaaannnoaoa.aaaaa. oow.aatooooooooooooooooooooo 2024 emn 1p-main.1
11

---

Published as a conference paper at ICLR 2025
Zhijie Deng, Peng Cui, and Jun Zhu. Towards accelerated model training via bayesian data selec- tion. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 8513-8527, Curran Associates, Inc., htspr///eennnnoooo////////eeeaaa
2023.
file/1a T300000000007777711363666161-860
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 1..18653//1/N191 1423. URL
N19-1423
Elvis Dohmatob, Yunzhen Feng, Pu Yang. Francois Charton, and Julia Kempe. A tale of tails: Model collapse as change of scaling laws, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El- El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Rearala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Seialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujiwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus. Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh
12

---

Published as a conference paper at ICLR 2025
Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti. Bran- don Spence, Brani Stojkovic. Brian Gamido. Britt Montalvo, Carl Parker. Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Monigomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptinn Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- Bawa, Nayan Singhal, Nick Egebo, Nicolas Usumier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Cherogguz Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha. Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary De Zef Rosnbrick, Zhaoduo Wen. Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL /attps://arxxt org. abs /ts.o///444 2407 783
Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-guided noise-free data generation for efficient zero-shot learning. In International Conference on Learning Representations, 2023.
URL /attps://opeereeeee
net wwt.:neet/oouneeee wtt.///rummioooooo wow.umidddshpppeee 1ob
Tim Hesterberg. Weighted average importance sampling and defensive mixture distributions.
Technometrics, 37(2):1851944, 1995. doi: 10. 10.1080000401006 19t.:/000000099999...1111 10484303. URL https: 1995. 01.95.048800 10484303
www. www.ttanffollinee ctmee.com/oae htt.://wwnaaa.aeooooooo////////a
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learn ing data manipulation for augmentation and weighting. In H. Wallach, H. Larochelle,
13

---

Published as a conference paper at ICLR 2025
A. Beygelzimer. F. d'Alché-Bus Buc, E. Fox, and R. Gamettt (eds.), Advances in Neu- ral Information Processing Systems, volume 32. Curran Associates, Inc.. 2019. URL
www.oo/www...................... newrips. cc/paper Files/paperr2019/1111 https: http://ppceeee.eeccccccoo
Angelos Katharopoulos and Francois Fleuret. Not all samples are created equal:
Deep learning with importance sampling. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. pp. 2525-2534 PMLR, 10-15 Jul 2018. URL lattss://proceedinn .mlr.
press/ wwtaa/vv0// htt00//ahharopoooo os .tmml
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large lan- guage models for text classification: Potential and limitations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 10443-10461, Singapore, December 2023. Association for Computational Linguisties. doi: 18.18/ 00000/3//1/2023..3 http:///2222/m////. http:/33.mmmmmmmaiiia mann.64mnn URL httph::/accaarrreee hww.aaanannntoooo..oooooaooooooa org/
2023. www.emnnpppmmmm pa-llppmmainnn w.w.emennmmmminnm
647
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV) pp. 2999-3007, 2017. doi: 10. 10911CC2220111133244
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks, 2015.
Yingzhou Lu, Minjie Shen, Huazheng Wang. Xiao Wang, Capucine van Rechem, Tianfan Fu, and Wenqi Wei. Machine learning for synthetic data generation: A review, 2023.
Maggie, Phil Culliton, and Wei Chen. Tweet sentiment extraction. https://wwe /www kaggle. www.mmmetttttttttittioo 2020. Accessed: 2024-11-18.
Com/ conpetitions/twwett
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol,, 65(4): 782-796, apr 2014. ISSN 2330-1635. doi: 10.1002/asi 23062.. URL 10. 10.1002/asi 23062
Max Marion, Ahmet Üstün,, Luiza Pozzobon, Alex Wang, Marzieh Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale, 2023.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language mod- els: Towards zero-shot language understanding. In Advances in Neural Information Processing Systems, 2022.
Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han. Tuning language models as training data generators for augmentation--nnaacee few-shot learning. In International Conference Machine Learning, 2023.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7632-7642, PMLR, 18-24 Jul 2021. URL https: /tpp::///rroe http::///ooceedeeeeee httLlhhtts://ppoooot
wlr.ppesss/vll http//319//aaa mttesss/v339/eeoon2aaa. html
MetaAI. Introducing meta llama 3: The most capable openly available Ilm to date, https://aa 3, 2024.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15630-15649, PMLR, 17-23 Jul 2022. URL https::/proccest ngs. wtt.ssssl..p..rsss.mm pwwsss////eee..... v1 62, mi wwwennderaannn html
14

---

Published as a conference paper at ICLR 2025
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philipp Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 15288-15299. Curran Associates, Inc., 2020. URL httpps://proceedd nwurips cc cc/paper_ files/paper/20000
OpenAl. Openai. introducing chatgpt.
httpp:://pppeeee http:/ppaaaa...oolooooooeeeoocce hwc.m//llg//chagpptt..// 2022.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha GontijoLopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo. Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright. Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff. Dave Willner, Clemens Winter, Samuel Wolrich. Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URI. https:// org/abss////ooooooooooooooo.o /2003 08774
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed-
15

---

Published as a conference paper at ICLR 2025
ward Yang. Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
Gabriel Pereyra,
George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions, 2017.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4334-4343 PMLR, 10-15 Jul 2018. URL bttps://proceedinng mmlr.
press/v80) ren 1aa htmml
Bernhard Schölkopf. John Platt, and Thomas Hofmann. Correcting Sample Selection Bias by Unla- beled Data, pp. 601-608. 2007.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget, 2023.
Nikolaos Stylianou, Despoina Chatzakou, Theodora Tsikrika, Stefanos Vrochidis, and Joannis Kom- patsiaris. Domain-aligned data augmentation for low-resource and imbalanced text classification. In Advances in Information Retrieval, pp. 172-187, Cham, 2023, Springer Nature Switzerland. ISBN 978-3-031-28238-6
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adap- tation. In Proceedings of the 20th International Conference on Neural Information Processing Systems, NIPS'07, pp. 1433-1440, Red Hook, NY, USA, 2007. Curran Associates Inc. ISBN 178160600320..
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followin llama model. https: htttt::///ithubb /gghhub..... com /att su- 1ab/ stanford alpaca) 2023.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. SemEval-2011 8 task 3: Irony detection in En- glish tweets. In Marianna Apidianaki, Saif M. Mohammad, Jonathan May. Ekaterina Shutova, Steven Bethard, and Marine Carpuat (eds.), Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 39-50, New Orleans, Louisiana, June 2018. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1/3 10S/////1/S1//// 1005. URL https:://cllamthhooogg... S18 618-1005]
Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 1/.186//11/118-5446 URL
https::/aclanthoot wwg.ellooooooooooooooooooo -
Huan Wang, Suhas Lohit, Michael N Jones, and Yun Fu. What makes a "good" data augmentation in knowledge distillation a statistical perspective. In S. Koyejo. S. Mo- hamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 13469, Curran Associates, Inc., 2022. URL htttss://rrcceed wwwpeeedddinnnnnndnneeeeeeee neurips cc cc/pcc/p.p.......ccceeece fitps//apppssep//2////1//1 Ei e
Conaereeeeooofeeeeeeeec .pdf
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484- 13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18653/ acl v1/2023.acl acl-- URL https: http:////aaaaaaao.oooooooooee w0t.222...llooo.. 754
16

---

Published as a conference paper at ICLR 2025
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari. Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceed- ings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4012, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 URL https: /tlps://aac ac http//aaannttooyy...oooomo org 2020. www.-1.ee.... 494
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Marine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 4602 4625, Seat- tle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022. maacl-main.341. URL https://ac wttetaaaannttootoooooooooo www. 2022 nwac .
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2220. emnlp-demos.6. htthst::::/ //aps:///aaaaaaa anw.ooooonoooooooooooooooooooooe org/ 2020 -wn.p-demos.eeeeee
ofte-oof-thearr natural
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. ZeroGen: Efficient zero-shot learning via dataset generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empiricul Methods in Natural Language Processing, pp. 1 11653-11666 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 18653/v 222022.mmnlp--.. main.801. URL org/ 2022. emnlp- ma main. 801
Tianyuan Zou, Yang Liu, Peng Li, Jianqing Zhang, Jingjing Liu, and Ya-Qin Zhang, FuseGen: PLM fusion for data-generation based zero-shot learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2172-2199 Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https:ttp///. settps:/accaantho// www.hhoooo........o........ 2024. emn p-mmiin.
130
APPENDICES
A OTHER RELATED WORK
Online Batch Selection Online batch selection [Loshchilov & Hutter 2015 Katharopoulos & Fleuret 2018 Mindermann et al. [002] Deng et al. (023) is a method to speed up training conver- gence by dynamically prioritizing the most informative data points, from the perspective of the min- imizing loss function. This technique evaluates how informative a data point is and selects a batch B, of informative data during each training step. Unlike online batch selection methods substituting uniformly sampled batches during training, this paper focused on developing a weight function to enhance performance on downstream tasks by aligning the L.L.-generated data distribution with real-world data distribution.
Data Pruning. Data pruning approaches filter out noisy text data. Traditional methods took rule- based filtering for high-quality data (Bane et al.) 2022 Wenzek et al., (2020), while recent approaches focused on diversification (Marion et al. 2023] Ankner et al. (024). which used perplexity to build diverse dataset. In advance, methods considered both quality and diversity, and this dual focus made our weighting mechanism a possible pruning scorer. Our methods did not conflict with existing pruning methods, thus becoming a potential complement.
17

---

Published as a conference paper at ICLR 2025
B ASYMPTOTIC CONVERGENCE OF IMP-LOSS
In this section, provide the formal proof of the asymptotic convergence of IMP-Loss using Chebyshev's ' inequality. Specifically, we show that this approximately converges probability to the expected conditional cross-entropy under P.
Definition B.1 (Convergence in Probability) A sequence of random variables {Xn} converges in probability to random variable X, denoted as {Xn} 3 X, if for any e >> 0,
lim P( - X 2 c) =0 n-000
(11)
Theorem B.1 (Chebyshev's Inequality) Let X be random variable with finite expected value B[X] and variance Var(X). For any e > 0,
Var(X)
P( - [[X] 20) S
(12)
2
APPLYING CHEBYSHEV'S INEQUALITY TO IMP-LOSS
Following the definition of IMP-Loss from Eq. 6 and considering the situation without using small real-world data to approximate. Let
N
xx) X;)
(0 DQ) = N
log P(ys xx;
(13)
Assume that all data points (x,, M1) are i.idd samples from the joint distribution Q(X,))) Define
(x))
Z. (xx log (2 0)
(14)
The empirical mean of Z; over samples is given by:
7
Zj (0 DQ)
(15)
N
The expected value of Z, under the distribution Q is:
P(vxx) Q(vxx) log P(y|x;
Eq [2]
Eq
(16)
Applying Chebyshev's inequality to the sequence Z:
Varo Zt) Varq Ne2
P P( ([Z-Eq|Z] 2
(17)
As N grows large, the right-hand side converges to zero, implying that Z converges in probability to Eq[2] [Z].
Therefore,
P (x(xx
(0MP D2) 1 Eg
log P(y|x:
(18)
18

---

Published as a conference paper at ICLR 2025
TRANSFORMING FROM Q TO P
Next, we show that:
P(y|x)
EQ
log 0())x;
Quixx
P()
Σ E
Q(x, v)
log 0)
(uixx
XEX NEV
Σ ((x)
Σ
P(y]x) log P(y)x;0)
(19)
XEX
yEY
Σ P(x)
Σ
P(y|x) log P())x:
XEX
vEV
Ep
log P(u)x;0)
Given that (x) P(x) by the assumption that the LLM is capable of simulating the real-world input distribution through careful and appropriate prompting, we have:
LIMP(O, DQ)
P
4 EQ
log P(v|x; 0)
Q(ulx)
(20)
log P(y]xx;)) 0]
2 Ep
Thus, the asymptotic convergence of IMP-Loss ensures that the weighted loss function effectively aligns the LLM-generated data distribution Q with the real-world data distribution P.
C DERIVATION OF DIMP-LOSS
In this section, provide the formal derivation to address the question: Which data point in Do- when used for training, will most effectively bring the model distribution closest to P? Following the optimization formulation in Eq. we can empirically apply Monte Carlo estimation using small real-world dataset Dp. denoted as Xp)). This allows us to reformulate the problem by maximizing the joint probability of the data points in Dp,, which leads the following optimization problem. This derivation similar to the online batch selection techniques discussed in previous research (Deng et al. 2023).
arg max P(yp: XP:; {(x, v))) = arg max (x. Da
II
P(y' (x,,
(21)
(x. (x' D'
This formulation leverages the joint probability of the dataset Dr,, ensuring that the selected data points Do are those that, when used for training, most effectively align the model's distribution with the small real-world distribution P'. This also implies that the chosen data point leads the model to perform well on Drr, enhancing the likelihood of better generalization to real-world data.
APPLYING BAYES RULE
By applying Bayes' nule to the joint probability of the dataset Dp, obtain:
19

---

Published as a conference paper at ICLR 2025
P(yp'|Xp Oz ((x,))))
P(Dp,X,,) N,
P(Xp,, x, Uo 0c)
Plyp X X. 01) P Dp,, 0c) 0.)
(22)
Plu|x, p'
Plyp Xpr, 0 P(y)x Dpr,
P(y|x;0; 0z)
The final equality holds because x alone cannot perform model update, leading to the conditional independence assumption. Since Plypr |xpx,x, 02) is constant for this optimization problem and does not influence the result, we can further simplify the optimization as follows:
arg max Plyp X Oz, v)) = (x
(23)
P(u|x; Ou, Dpr
arg max (x
P(y]x; 0,)
Similar to the online batch selection work, we P" (1())) to approximate P(y]x; 0τ, Dp.). This approximation then utilized as the weight in our loss function. Consequently, a data point brings the model closer to the real-world distribution P, its corresponding weight will be higher, thus impacting the model's training.
D LOWER BOUND OF DIMP-LOSS
The Lower Bound
P'( (xx)
N DDMMP D2) =
log P(vi(x;;).)
P(yi 0c)
(By " AM-GM inequality:
2 for P(y xx; 0.))
((())(x))) (2- P(un)xx;)))) log P(ye(x;;0e)
N
2 P'(y x.) log P(yx(x;;0.)
Σ
(()x 0,) log P(v: xi; 0z)
(By Hölder's Inequality [[fgll1 I 1l 11
N
N P(uilx;; 0e) log P(us 01)
('(vxx,) log P(ux|x;; 02) max (vx
(()))(x:) log P(yx(x;;
P(us 0c) log P(yxxx;00 0.)
Empirical distilled entes-entropy loss.
Maximum entropy regularizer
(24)
*AM-GM Inequality The derivation shown illustrates the application of the Arithmetic Mean
Geometric Mean (AM-GM) inequality, which states that for any two positive numbers and y, the arithmetic mean is greater than or equal the geometric mean, i.e., 2 2 Vab, Va, b 0.
a+b
In this specific case, b is set to a' simplifying the inequality
2 -
2
20

---

Published as a conference paper at ICLR 2025
Multiplying both sides by 2 yields:
1 a - 2 2,
a
and rearranging the inequality gives:
2 2 a.
This result is a classic application of the AM-GM inequality, demonstrating that the sum of a number and its reciprocal always greater than equal for any positive I.
E COMPUTATIONAL TIME COMPARISON
LLM-generated Financial Data
2500
Total Total Secend
2000
1300
CE-Loss
SunGen
IMP-Loss
DIMP-Loss
Figure 3: Total running time (in seconds) for CE-Loss, IMP-Loss, and DIMP-Loss on the LLM- generated Financial benchmark.
Method
Build QC
Build DC
Precalculate weights
Training
Total
CE-Loss
333.242s
333.242s
SunGen
2680s
2680s
IMP-Loss
8.824s
333.516s
57.695s
333.328s
733.363s
DIMP-Loss
8.824s
29.274s
333.426s
371.524s
Total running time of each component (in seconds) for CE-Loss, SunGen, IMP-Loss, and DIMP-Loss on the -Ln-gererated financial benchmark. The table breaks down the time spent building the Quality Checker (QC), building the Diversity Checker (DC), precalculating weights, and training. The total time combines all these components.
Table 3:
In this computational time experiment, evaluated the running times the LLM- LLM-generated Finan- cial benchmark dataset, which includes 10,012 training samples (Dq) and 242 development samples (small real-world data, Dy=). Our comparison focused four methods: CE-Loss, SunGen, IMP- Loss, and DIMP-Loss. We have broken down the total process into four components: building the Quality Checker (QC), building the Diversity Checker (DC), precalculating constant weights, and the actual training time for each respective loss function. For all experiments, the downstream mod- els and checkers were trained for 5 epochs with batch size of 32. The batch size was set to 64 during the precalculating constant weights phase. The inner loop epochs were set to 1 for SunGen. The results of this breakdown are presented in Table and the total time is in Figure The results indicate that IMP-Loss requires approximately 2.2 times the running time of CE-Loss. In contrast, This demonstrates that DIMP-LosS highly efficient, requiring only slight overhead compared to CE-Loss, while SunGen's 's computational time is approximately 7 times higher, further underscoring the efficiency of our methods for large-scale applications.
F TRAINING DETAILS AND HYPERPARAMETHRR
For our experiments, we used a pre-trained BERT-base model (Devlin et al. 2019) from Hugging- face's 's transformers library (Wolf al. 2020 the encoder, utilizing the representation embedding from the last layer as input to our classification models. We fine-tuned the model with hyperparam- eters selected from the following ranges: learning rate (6e-6, 6e-5), epochs (5, 7), and batch size
21

---

Published as a conference paper at ICLR 2025
{32, 64). Other hyperparameters were set to the default values provided by Huggingface's trainer for text classification. The best checkpoint was selected based on the accuracy of the development set. We repeated each experiment with four random seeds. We reported the best accuracy run on tables (Ta while also providing the minimum, maximum, and average in the training dynamics section (Sec. 5. To train the quality checker, used the small real-world dataset (development split) not included in the training data and trained the quality checker for five epochs. Similarly, the diversity checker of IMP-Loss was also trained for five epochs. This approach aligns with our setup, where access to real-world data is limited, and thus, we reuse the development set to build the quality checker and perform model selection. All experiments were conducted using Py PyTorch (aaszke et al. (2019) and Huggingface (for models and datasets) on V100 GPUs with 32GB memory.
G TRAINING ON NOISY DATA
Financial
Tweet Irony
MRPC
Dataset
Method
Acc
F1
Acc
F1
Acc
F1
GPT-3.5 few-shot
79,46
81.6
63.39
69.39
69.28
71.75
Small real world
CE-Loss (quality checker)
78.05
75.26
62.5
62.38
73.16
68.69
CE-Loss
78.38
73.44
60.46
60.14
74.03
67.5
Focal-Loss
78.55
74.97
62.11
61.12
74.72
69.59
Noisy Data
IMP-Loss (Ours)
81.6
78.24
64.8
64.51
76
70,46
DIMP-Loss (Ours)
82.59
80.28
64.16
64.09
76.58
71.32
Table Performance metries on the noisy data. The table showcases the accuracy (Acc) and macro F1 score (F1) ) for each method applied on three distinct datasets: Financial, Tweet Irony, and MRPC. The methods include CE-Loss, GPT-3.5 few-shot, Focal-Loss, IMP-Loss, and DIMP-Loss. Notably, bold entries indicate the best-performing metrics within each training dataset category.
In this section, we evaluate the robustness of our proposed methods, IMP-Loss and DIMP-Loss, by training on noisy datasets. We aim simulate real-world scenarios where --e-erreertted data may be imperfect due to labeling errors (low quality), duplicate entries (low diversity), and unrelated inputs (low quality). This allows us to analyze the effects of the Quality Checker and Diversity Checker in IMP-Loss and DIMP-LOSS.
G.1 EXPERIMENTAL SETUP
To create noisy datasets, we start with the original training set from each benchmark (Financial, Tweet Irony, and MRPC) and split it into three parts:
1. Original Data: This part remains unchanged and serves as the control set.
2. Random Swapped Label Noise: In this part, the labels are randomly altered, introducing label noise and reducing data quality.
3. Duplicated Data: In this part, each data point is duplicated once, introducing redundancy
and reducing data diversity.
4. Unrelated Input Data (Only for Financial):
For the financial benchmark, we introduce out-of-domain input noise by randomly selecting 452 data points from the Tweet Senti- ment Extraction benchmark (Maggie et al. 2020). While this dataset is also sentiment classification task, it is unrelated to the financial domain.
G.2 PERFORMANCE RESULTS
The results in Tabel 4 indicated that our proposed methods, IMP-Loss and DIMP-Loss, consistently outperform the baselines across all benchmarks, even when the training data is noisy. Specifically, in the Financial dataset, IMP-Loss achieves 81.6% Acc and 78.24% Fl score, while DIMP-Loss reaches 82.59% Ace and 80.28% Fl score, surpassing the CE-Loss and Focal-Loss baselines. In the Tweet Irony dataset, the performance improvement is more pronounced, with IMP-Loss achieving 64.8% Acc and 64.51 % F1 score, and DIMP-Loss achieving 64.16% Acc and 64.09% F1 score,
22

---

Published as a conference paper at ICLR 2025
significantly higher than CE-Loss and Focal-Loss. For the MRPC dataset, IMP-Loss and DIMP- Loss show robust performance with 76% Acc and 70.46% Fl score and 76.58% Acc and 71.32% score, respectively, outperforming the GPT-3.5 few-shot approach, which achieves 69.28% Acc and 71.75% F1 score.
G.3 ANALYSIS OF CHECKER SCORES AND WEIGHTS
IMP-Loss Figure 4 and Figure illustrate the Quality Checker Score ())xx), Diversity Checker Score Qulxx), and the corresponding weights of the IMP-Loss for the Financial and the Tweet Irony dataset, across three different data conditions: original, swapped labels, and duplicated entries.
The Quality Checker Score is highest for the original data and significantly lower for the swapped label data, indicating that the model correctly identifies the labels as lower quality. The Diversity Checker Score (where lower values are better) is lower for the original data than the duplicated data, indicating the impact of duplication highest diversity because the altered labels create data points that are substantially distinct from the rest of the dataset. Similarly, the unrelated input data exhibits relatively high diversity due to its out-of-domain nature. However, data points from both the swapped label and unrelated input cate- gories have low Quality Checker Scores, resulting in their lower assigned weights. Consequently. the weights assigned to the original data are higher compared to the swapped label data and the dupli- cated data, demonstrating the effectiveness IMP-Loss in recognizing and appropriately weighting high-quality, diverse data.
diversity. Additionally, the swapped label data achieves the
Financial
Original
Soop
Duplicate
Untelattd
Figure 4: Average Quality Checker Score, Diversity Checker Score, and Weights of IMP-Loss for Financial Dataset: Comparison between Original, Swapped Label, Duplicated Data and Unrelated Input Data.
Tweet Irany
Quality Checker Scrre
Chersty BI
Weight
Orggral
Soap
Duplicat
Figure 5: Average Quality Checker Score, Diversity Checker Score, and Weights of IMP-Loss for Tweet Irony Dataset: Comparison between Original, Swapped Label, and Duplicated Data
In contrast, Figure 6 shows the diversity scores (()] 0z) and the weights for the DIMP-Loss method on Financial benchmark across epoch, where the diversity checker is the training model itself. The Diversity Checker Score (lower is better) is also lower for the original data than the duplicated data and Unrelated input data. In the end, The weights (Figure 7 assigned by the DIMP-Loss method are consistently higher for the original data than the swapped label, Unrelated
DIMP-Loss
23

---

Published as a conference paper at ICLR 2025
Financial Noisy Data
Ortginal
Swap
Dupiicte
Unnelated
Figure 6: Average Diversity Checker Score of DIMP-Loss for Original, Swapped label, Unrelated input data, and Duplicated data on the Financial Dataset across Epoch.
Financial Noisy
Figure 7: DIMP-Loss Weight for Original, Swapped label, Unrelated input data, and Duplicated Data the Financial Dataset across Epoch.
input data, and duplicated data across epochs. This pattern aligns with the results observed for the IMP-Loss method.
H
DATASET DESCRIPTIONS
The size each split and the generated data Table
Financial
Tweet Irony
MRPC
Train
3392
2862
3668
Dev
242
200
408
Test
1212
784
1,725
Generated
10012
3000
3005
Table 5: Data size each split
The description of each dataset is following:
Financial Phrasebank: This benchmark involves categorizing finance-related sentences into pos- itive, negative, or neutral sentiment categories. These sentences, numbering 4,840, are extracted from financial news articles. Since the dataset does not come with predefined training, validation, and testing splits, we randomly divided it into training (70%), validation (5%), and testing (25%) sets like the previous work (Li et al. 2023).
This task requires sorting tweets into two groups: ironic and non-ironic. The dataset containing tweets in English has been explicitly annotated for these categories. comprises 2,862 instances for training and 784 instances for testing. Initially, there were 955 instances in the valida- tion set, but due to limited access to real-world data our scenario, we have randomly selected 200 instances for our validation sets.
Tweet Irony:
24

---

Published as a conference paper at ICLR 2025
Task
Prompt
System Prompt:
Now you are a person using Twitter. You are asked to write an irony or non-irony tweet to express your feelings. Your writing style must be consistent with the texts in the tweet. You must ensure that your language is colloquial, casual, and Twitter-like, You are given length require- ment. You must ensure your tweet meets the length requirement.
Tweet irony
Data Generation Prompt: Write tweet expressing [label] feeling and ensure that the length of the tweet is about mnum_o__worde words. Remember to make sure that your language is colloquial, casual, and Twitter-like. Be creative and write unique tweets.
For example:
(Examples of the label from small-real world dataset)...
Can you provide something more diverse than the previously generated data?
You are now a journalist writing financial news. You need to write some financial news that expresses polar sentiments. The financial news you generate needs to be considered from investor's 's viewpoint only, i.e., whether the news may have positive, negative, the stock price. As result, sentences with sentiment irrelevant from an economic or financial perspective are considered neutral. You are given one the polar sentiments and a length requirement. You must write financial news that expresses the corresponding sentiment and meets the length require- ment.
Context Prompt:
neutral influence
Financial
Data Generation Prompt: Write financial news with {label} sentiment and ensure that the length of the financial news is about [num_ of words] words. Be creative and write unique financial news.
For example:
(Examples of the label from small-real world dataset)...
Can you provide something more diverse than the previously generated data?
Context Prompt: Generate jrum_of_examples } data points like the following examples. A label of means they are semantically similar, and label of 0 means they of each category (Please just output the format like what provide, and the output MUST be different from input):
Try to balance the number
Data Generation Prompt: For example:
MRPC
sentencel: Amrozi accused his brother, wh-on om he called the witness ", of deliberately distorting his evidence
sentence2: Referring to him as only 'the witness ", Amrozi accused his brother of deliberately distorting his evidence - label: sentencel: They had published an advertisement on the Internet on June 10, offering cargo sale, he added Internet, offering the explosives for sale. (Other examples from small-real world dataset)... Can you provide something more diverse than the previously generated data?
sentence2: On June 10, the ship's owners had published an advertisement on the
label: 1
Table 6: Detailed prompts for each task for data generation.
The Microsoft Research Paraphrase Corpus (MRPC) consists of 5,801 sentence pairs sourced from news articles. Human annotators manually labeled each pair to determine whether the sentences were paraphrased from each other. We employ the official MRPC dataset available through Huggingface's 's datasets library, segmented into training, validation, and testing sets contain- ing 3,668, 408, and 1,725 instances, respectively.
MRPC:
I DATA GENERATION
I.1 PROMPT
The prompts used for data generation across different benchmarks provided in Table 66 The prompts for Tweet Irony and Financial datasets are based on those used in previous work (Li et al.) 2023
1.2 DATA GENERATION BUDGET
We used OpenAl GPT-3.5-turbe 106 (OpenAl, 2022) to generate a dataset for the three bench- marks, adhering to OpenAl's terms of service and usage policies. The total cost is $38 73.
25


Reviews:
[{"review_id": "9QtZyvKpiZ", "reviewer": "Reviewer_gCSy", "summary": "This paper introduced IMP-Loss and DIMP-Loss as novel weighted-loss objectives to enhance the performance of models trained on LLM-generated data. Paper showed results demonstrating that both methods outperformed traditional loss functions CE-loss, focal loss, Meta-learning weighting on 3 NLP tasks Financial Phrasebank, Tweet Irony and MRPC.\n\nBoth IMP-Loss and DIMP-Loss include quality over diversity as weighting per training example. Quality weight is the predicted conditinoal probability from a BERT-base model trained on a small real world data of size 200-400 samples. Diversity for IMP-Loss is similarly trained model but on synthetic data. Diversity for DIMP-Loss is the current model's predicted conditional probability. IMP-Loss is based on importance sampling, DIMP-loss is motivated by online batch selection methods.\n\nFor experiments, paper compared training on synthetic data alone, noisy synthetic data (label flipped, duplicated data entry). In both case, IMP-Loss and DIMP-Loss work better than the quality checker and three other loss functions. Paper also applied the two weighted loss function to real data, where the results showed better than training on real data using the other loss functions too.", "questions": "- Paper assumes Q(x) ~ P(x) on line 171, but in line 154-164, the synthetic data has high diversity and entropy. Wondering how to interpret the two together?\n- Synthetic data includes the label for the data. For Q(x) ~ P(x) on line 171, does that means Q(x, y) ~ P(x, y)? Seems the paper mix Q(x, y), Q(y|x) eg at line 942 too.\n- Line 999 missing ', so it should be x_{p'}, right?\n- The table 1 shows Focal-loss and Hu et al.'s method did not work better than CE-Loss, esp for the \"large real world\" case, could you comment on the reason?\n- For the DIMP-Loss at line 274, if we apply it to large real data, where P' is also trained on the same large real data. In this case, the weighting will converge to 1, is that correct? In that case, the final performance at convergence should be the same as CE-Loss, right? How does this case affect table 1?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The  IMP-Loss and DIMP-Loss  approaches are novel contributions. The approaches are clearly motivated. Derivation are clearly demonstrated step by step. Approximations are used to make the implementation efficient.\n- The experiments covered several scenarios, including small real data, synthetic data, noisy data and real data. In all cases, the introduced methods work best, showing the robustness of the approaches. Several oblations are interesting: increasing real data for quality checker increases final performance. Diversity check is very important for final performance.", "weaknesses": "The paper introduces two weighting scheme for synthetic data. The experiments can be made more convincing:\n- The three benchmarks can be expanded to more NLP tasks. Eg nn the ZEROGEN paper cited here there are several benchmarks: sQuAD, QNLI and AdversarialQA, which are particularly challenging for synthetic data generated from LLM. The performance was far below supervised results. It will be interesting to apply the weighting method to see if the gap can be reduced for these challenging benchmarks.\n- The paper used 5 epochs for training the downstream models by Line 1101. It is unclear whether the models have converged. It is interesting to not only see weighting proposed here can speed up learning, but what happens at convergence.\n- The weighting method does not seem to achieve SOTA for the two benchmarks. It would be more convincing if the paper adopted a strong baseline rather than BERT-large to build on top and compare with SOTA results.\n  - mrpc https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc, tinybert is at 86.4%.\n  - financial phrasebank https://paperswithcode.com/sota/text-classification-on-financial-phrasebank. 96.7% for distillbert.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIySvY254L", "reviewer": "Reviewer_R1bf", "summary": "Since the development of LLMs, LLM-generated data are widely used for training models. However, the distribution of such artificial datasets may be misaligned from real-world datasets, creating concerns. \nTherefore, during the training process, to prioritize data that is more diverse and has higher quality, the authors develop an approach to weighing LLM-generated data according to its quality and diversity. Compared to previous data-filtering techniques that only use a subset of generated data, this approach leverages all the training data while giving nonuniform weights to the loss of each data point.\nSpecifically, the authors theoretically define two losses: IMP-Loss and DIMP-Loss. IMP-Loss applies WCE-Loss, where the weight function is compounded by a quality checker (approximated with P, estimated by fitting a model on D_P, the real-world dataset) and a diversity checker (approximated with the inverse of Q, estimated by fitting a model on D_Q, which is the synthetic dataset). DIMP-Loss simplifies the calculation by estimating the diversity checker with the current model in training.\nAs a result, the performance using the newly devised loss improves compared to the baselines, with DIMP Loss falling slightly behind IMP loss, as a compromise for lower costs.", "questions": "1. For the metrics \"diversity\" and \"quality\", would it be possible to verify that they actually correspond to what they refer to? I.e., does higher quality data actually get higher quality metrics? Currently, it seems like a term for the mathematical expression. While I understand it intuitively, it would also be nice to verify them empirically.\n2. In the tweet irony case in Figure 1, DIMP loss start out with worse performance than other baselines for the first and second epoch. Could you explain why such a scenario would occur? Does this mean that these methods introduce instability to the training process?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical derivation of IMP and DIMP loss is clear and sufficient.\n2. The empirical improvement shown in table 1 is consistent.\n3. There are multiple studies to verify the robustness of the method: different datasets in figure 1, different model sizes in table 2, and using various percentages of the training set in figure 2.\n4. Section 3.2 shows the motivation for employing diversity and quality in the weighting strategy, laying the foundation for the methodology developed later in the section. \n5. The writing and structure is clear to follow.", "weaknesses": "1. The results are derived using a BERT-Based model, which is quite small compared to contemporary LLMs. How would the method scale to larger LLMs? How would the costs grow as the model size grows?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xHkjveXycI", "reviewer": "Reviewer_eiLk", "summary": "This paper proposes a reweighting scheme for synthetic data generated by LLMs. The focus is on the distinction between the distributions of real and synthetic data. To address this, the authors suggest IMP-Loss and DIMP-Loss to align the distribution of synthetic data with that of real data. These two loss functions assign weights to the cross-entropy loss for each data point. The empirical results support the effectiveness of this approach.\n\n(Note: my review has been revised by LLM for improving the grammar.)", "questions": "- I am still uncertain whether the proposed diversity checker functions as intended. What happens if unrelated data is fed into the diversity checker? SunGen has already demonstrated that noise in synthetic data is not limited to data with noisy labels but also includes data unrelated to the desired domain.\n- Some comments regarding the paper's formatting:\n  - Line 030: Use a capital letter for the abbreviation of LLM. Currently, it reads 'Large language models,' which is inconsistent with other terms like 'Natural Language Processing.'\n  - Line 063: Ensure proper spacing between ‘ers’ and ‘highly.’\n  - Footnote 1: Start the sentence with a capital letter, and ensure there is a period at the end.\n  - Please ensure consistent use of \\citet and \\citep, as this inconsistency currently reduces the overall quality of the paper. For example:\n    - Line 033: Use \\citet for Taori et al.\n    - Line 050: Use \\citet for Hu et al.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The use of synthetic data has been of significant interest to researchers for years. It is also important to note that noise in synthetic data has been a long-standing concern. While several studies have proposed methods to address this issue, they come with their own limitations, as the current manuscript notes. This work is timely in addressing those previous limitations.\n  - While the proposed method requires some additional computation for assigning weights, I agree that it is more efficient compared to methods based on meta-learning, at least.\n  - As a result, this method seems more practical than previous approaches for mitigating the noise in synthetic data.", "weaknesses": "- **Most of my concerns are resolved during the rebuttal process.**\n\n- ~~I see several issues with the introduction and related work section.~~\n  - The authors primarily build their argument by citing studies focused on generating synthetic data for *instruction tuning of LLMs*, such as Alpaca.\n  - However, the method proposed in this paper is aimed at mitigating noise in synthetic data for *text classification*, which might create some confusion.\n  - The confusion arises because there are many studies focused on generating synthetic data specifically for text classification, which are more relevant to this work, but the authors do not cite them.\n    - Meng et al., Generating Training Data with Language Models: Towards Zero-Shot Language Understanding, NeurIPS 2022.\n    - Meng et al., Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning, ICML 2023.\n    - Gao et al., Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning, ICLR 2023.\n    - Choi et al., UniGen: Universal Domain Generalization for Sentiment Classification via Zero-Shot Dataset Generation, arXiv Preprint 2024.\n    - Zou et al., FuseGen: PLM Fusion for Data-Generation-Based Zero-Shot Learning, arXiv Preprint 2024.\n  - I believe discussing these studies would be more relevant compared to the current organization of the manuscript.\n- ~~Accordingly, the authors should have discussed these works and used them as baselines.~~\n  - For instance, SunGen (Gao et al., 2023), UniGen, and FuseGen propose methods for mitigating the noise in synthetic data.\n  - SunGen, in particular, is very similar to this work, as it suggests reweighting synthetic data based on quality. While SunGen is based on meta-learning and may require more computation than the proposed method, the current organization of the manuscript does not adequately address this overlap.\n  - Furthermore, FuseGen suggests a reweighting scheme that does not require additional computation, unlike the proposed method.\n\n~~Given these points, I do not believe this paper is ready for publication at ICLR in its current form.~~", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eOHF5mktHd", "reviewer": "Reviewer_e1Ke", "summary": "This paper proposes a modification to the loss function to improve the performance of models trained on synthetic data, making it comparable to models trained on real-world data. This approach addresses the challenge of limited availability of real-world data, where models trained solely on synthetic data often underperform. The authors introduce two new loss functions—IMP-Loss and DIMP-Loss—which incorporate mechanisms for quality-checking and diversity-checking the synthetic data. IMP-Loss prioritizes synthetic samples that closely resemble real-world distributions, while DIMP-Loss emphasizes samples that guide the model toward the performance of a small base model trained on limited real-world data. Experimental results demonstrate that these loss functions enhance the performance of models trained on synthetic data, outperforming traditional baseline methods.", "questions": "In some formulas, it's unclear whether the pairs (xi, yi) are drawn from the real-world distribution P or the synthetic data distribution Q. For instance, in Appendix C, it is clearly noted that (yp', xp') is from distribution P. Could you use similar notation consistently throughout the main body to help readers differentiate between real and synthetic data sources?\n\nTo improve clarity, consider starting a new paragraph with the sentence: \"Thus, inspired by the objective of Hu et al. (2019), we introduce two novel, efficient, automatic weighted-loss approaches: Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP-Loss)...\" This will make the contributions of this paper stand out more clearly, preventing them from being mixed with discussions of prior work.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, this paper is well-written, addressing a compelling and practical problem for researchers working with synthetic data. The issue of model performance degradation when using synthetic data is aligned with recent research findings. The proposed loss functions IMP-Loss and DIMP-Loss are creative and effective solutions.\n\nThe clarity of the paper is a strong point, with a well-defined problem formulation and a meticulous derivation of the proposed loss functions. Each step in the reasoning process is clearly outlined, making the theoretical contributions easy to follow.\n\nThe paper includes extensive experimentation, which demonstrates the efficacy of the proposed methods. The analysis of the runtime complexity further adds the practical applicability of the algorithm in real-world scenarios.", "weaknesses": "The core idea of weighting synthetic data samples is not entirely novel. A similar concept was explored in the paper \"Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning\" published last year. Although the formulation and implementation in this paper differ, the underlying principle of prioritizing certain synthetic data points is conceptually similar, which somewhat limits the novelty of the contribution.\n\nAnother concern lies in the strong reliance on the assumption that a model trained on a small real-world dataset P can approximate the performance of a model trained on a larger, complete dataset. The paper would benefit from a more in-depth discussion or experimentation addressing this assumption. Specifically, it would be helpful to explore the minimum number of real-world examples required for this assumption to hold and to provide guidelines on the amount of data practitioners should aim to collect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)