Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory

Authors: Aymane El Firdoussi, Mohamed El Amine Seddik, Soufiane Hayou, Reda ALAMI, Ahmed Alzubaidi, Hakim Hacid
Primary area: learning theory
Keywords: Synthetic Data, RLHF, Generative Models, Statistical Models, Random Matrices

Abstract:
Synthetic data has gained attention for training large language models, but poor-quality data can harm performance (see, e.g., Shumailov et al. (2023); Seddik et al. (2024)). A potential solution is data pruning, which retains only high-quality data based on a score function (human or machine feedback). Previous work Feng et al. (2024) analyzed models trained on synthetic data as sample size increases. We extend this by using random matrix theory to derive the performance of a binary classifier trained on a mix of real and pruned synthetic data in a high dimensional setting. Our findings identify conditions where synthetic data could improve performance, focusing on the quality of the generative model and verification strategy. We also show a smooth phase transition in synthetic label noise, contrasting with prior sharp behavior in infinite sample limits. Experiments with toy models and large language models validate our theoretical results.

Decision: accept (poster)
Reviewer scores: 8, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730571803929

thread_type:
official_review

<thread_object>
creator: reviewer_qhab

<content>
summary:
This paper builds upon Feng et al. that examined how using synthetic data with selection impacts final performance. Here, the authors extend that research to a setting that considers both distribution shifts in the feature space and the combination of real and synthetic data. This approach provides precise results without requiring an infinite number of synthetic samples. The experiments corroborate some theoretical insights.

strengths:
1. The theoretical section is well-written, offering strong motivations and clear explanations.
2. Analyzing distribution shifts in the feature space is a valuable contribution compared with the previous work.
3. The paper provides a precise analysis of performance when training on mixed real and (selected) synthetic data with random matrix theory, expanding beyond previous work.

weaknesses:
My primary concern is the applicability of the theoretical insights in practical settings. The theoretical predictions are based on binary Gaussian mixture data, with synthetic data generated by fitting another Gaussian mixture. While the authors provide performance predictions for models trained on mixed data, it is unclear how well these predictions, derived from Gaussian mixture models, will generalize to real-world data. Specifically, how can we estimate some of the key theoretical variables? How can we ensure that the supervision quality is sufficient to mitigate the negative effects of distribution shift?

Currently, the experimental section could be improved by drawing more explicit connections to the theoretical insights. It is generally expected that better supervision and a more accurate generator would lead to improved performance with selected data. However, what practical insights does the theory provide? Can the theoretical findings be leveraged to inform the design of good data selection methods?

The experiment part is not well-written, with lots of details missing. I will elaborate more in the questions.

questions:
1. In Equation (2), the generative model assumes a prior with symmetric data sharing the same $\hat{\mu}$. How would the theory change if one Gaussian per class was fit instead?
2. On line 129, label noise is introduced as purely random noise. What if the label error depended on the input? For instance, in safety applications, a generative model might introduce label errors in specific regions of the distribution due to bias. Could such input-dependent noise be analyzed?
3. It would be helpful to discuss how label noise and feature noise correspond to various types of errors and shifts in real settings.

Missing details from the experiment:

4. What value of $\hat{n}$ is used across all experiments? The authors mention $\hat{n}$ varies in the MNIST experiment (line 400) but don’t provide further details.
5. In the LLM Safety experiment (line 411), it is stated, "we focus only on label noise." However, since data is generated by another language model, feature noise is likely present. 
6. In Figure 7 (left), both feature noise and label noise should be included, as data are generated by a Gaussian fitted to the distribution. Could the authors provide error bars here? Given the simplicity of the MNIST setup with two-layer networks, this should be feasible.
7. In Figure 8, why does using $(\rho, \phi) = (0.5, 0.5)$ as weak supervision? According to the definition in Equation (5), the data selection remains random regardless of label accuracy, which seems equivalent to no supervision.
8. In the LLM Q&A Safety Generation experiment, is accuracy evaluated by the Llama-Guard-3.1 model? What weak supervision method is employed here? Why do the authors use two models to annotate labels? It would be helpful to include the evaluation results comparing the two labeling models with the “ground truth” from Llama-Guard-3.1 to assess the quality of synthetic labels. Additionally, how is supervised fine-tuning conducted with data containing both safe and unsafe answers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636254708

thread_type:
official_review

<thread_object>
creator: reviewer_bes2

<content>
summary:
This paper extends important work by Feng et al. 24 and studies the effect of synthetesized data as part of the training data of models. It has two parts, a theoretical part that significantly extends the analysis of a mixture of two isotropic Gaussians from Feng et al. 24 from the case of infinite data at finite dimension to the case where both dimension and data tend to infinity with fixed ratio. Using tools from RMT they thus uncover a richer structure of transitions in a parameter space defined by the quality of synthetic data, the generator, label noise and verifier strength. A key novel element (which facilitates the analysis) is the assumption that the synthetic data comes from a model that estimates the mean and covariance matrix from initial (real) data (as opposed to a more general estimator). This allows to nicely study distribution shift of the features (not just the labels). Another extension over Feng et al.24 is the analysis of mixtures of real and synthetic data (though this is also done in "Strong Model Collapse" by Dohmatob et al., in a different and more general setting). 4 experiments nicely support the theory.

strengths:
The paper lays out a coherent program and goes ahead to prove it with relatively sophisticated tools. The message is conveyed very clearly, the extensions over prior work, and in particular Feng et al 24 are clearly demarcated. 
The consequences of theorems are explained well, despite the necessarily heavy math. This paper thus provides further evidence to what was laid out in Feng et al 2024: synthetic data can only boost performance when coupled with verification (pruning).
The paper does a masterful job of presenting the results and the experiments are a nice addition. The paper is also well written and is fun to read.

weaknesses:
The main weakness is that a few crucial references are missing. In particular there are three papers mixing real and synthetic data that should be cited, [1] and [2], and [3], a more recent one (there is no fault in not citing the latter, as it is recent, 
but the former date back over 8 months). In particular, a careful brief comparison to the [1] setting and results should be added, and [2] should be mentioned. Another paper that should be cited among the theoretical works for model collapse is [4]. On the other hand, 
your reference to [5] should probably be completely removed as it is misleading (the analysis of combination of real and synthetic data doesn't compare to adding the same amount of real data properly - this has been debunked on several occasions and should not be propagated).

Specifics (mostly minor):
1. line 252: typo "usefull"
2. line 402: "emphasize on" - sounds wrong, reformulate
3. line 406 use \citep for Malartic
4. line 408: "from the the Antropic's" --> "from Antropic's"
5. line 483-484: discuss Llama versus Gemma to make clear that you regard one as a weaker supervisor than the other
6. line 518: "Conclusionn" - typo
7. line 527-28: you might want to discuss that label pruning can induce a distribution shift in the features (because the pruning could be biased towards keeping certain features alive) - see (and cite) the discussion in Feng et al. 2024 (https://arxiv.org/pdf/2406.07515 version bottom of p. 10 - top p 11)

I will be happy to reconsider my score once these have been addressed (and the questions).

[1] Ayush Jain, Andrea Montanari, Eren Sasoglu: Scaling laws for learning with real and surrogate data, https://arxiv.org/abs/2402.04376

[2] Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data, ICLR 2024 spotlight

[3] Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, Julia Kempe: Strong Model Collapse, https://arxiv.org/abs/2410.04840

[4] Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, Julia Kempe: A Tale of Tails: Model Collapse as a Change of Scaling Laws, ICML 2024

[5] Gerstgrasser et al. 2024

questions:
1) How does your work compare to Jain et al. [1] specifically? (It’s a different data model, of course, but qualitatively, are there similarities/differences and how do the methods differ?)
2) I am confused about the "No supervision" case in Fig. 6 (the Amazon data). While in all other experiments more synthetic data doesn't help, here the U-shape of the curve stands out. Can you discuss a little more why in this case the red curve starts going up?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730656154794

thread_type:
official_review

<thread_object>
creator: reviewer_m5rf

<content>
summary:
Synthetic data has been used in training large language models, but concerns about its quality have emerged in practical applications. This paper works on data pruning to select high-quality data and analyzing the performance of trained on a mixture of real and synthetic data in a high-dimensional setting. The authors provide both theoretical and experimental results for the benefits of using and verifying synthetic data.

strengths:
1. This work presents analysis and insights with theoretical justification. 
2. Through multiple tasks, the approach is shown its empirical effectiveness.

weaknesses:
1. The paper lacks a clear definition of the quality of synthetic data. 
2. The derived theory in the paper is strongly assumed based on Gaussian distribution, however, non-Gaussian data are common in text or image data used for LLMs.
3. The paper only focuses on label verification for synthetic data. In real-world applications, synthetic data generation often struggles with feature fidelity, addressing feature consistency or the quality of synthetic data in feature space is more beneficial.

questions:
1.Figure 1: $\hat{\eta}$ = 0.1, $p$ = 500, and $\hat{n}$ = 500, given that $\hat{\eta}$ = $\frac{p}{\hat{n}}$, is this a typo?              
2. Figure 3, in the figure of weak supervision, when the proportion of synthetic data is higher than 0.8, the performance increases, what is the reason for this? This trend is opposite to that when the proportion of synthetic data is lower than 0.8.  Similar observations also happen in Figure 6.      
3. Can authors explain why $\hat{\eta}$ can represent a distribution shift?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731485059303

thread_type:
official_review

<thread_object>
creator: reviewer_kxsn

<content>
summary:
This work considers the problem of maximizing the usefulness of synthetic data in the modern era of generative modelling whereby at least part of the training data is not from the real data distribution, but synthesized from another AI model. In the solvable setting of a mixture of isotropic Gaussians, the authors consider a linear model whose weights vector is a regression-style ridge estimator. The synthetic data distribution is taken as yet another Gaussian mixture whose parameters (i.e shape) is given by empirical estimates of the ground-truth Gaussian mixture: the amount of data on which these parameters are estimated serves as a proxy for the quality of the synthetic data. Optiinally, this synthetic data is mixed with data from the real data distribution.

In the "proportionate" high-dimensional scaling limit, the authors use classical tools from random matrix theory (RMT) to obtain exact analytic expressions for the classifier and its accuracy. In particular, they paper recovers the theoretical results of Feng. et al (2024) as special case (corresponding to training only on synthetic data).

strengths:
- A nice theoretical setup for analysing the impact of synthetic data and pruning strategies is introduced.
- A complete analytic theory for this toy setting is obtained and its phenomological clearly discussed.
- Theoretical results from previous work Feng et al. (2024) are recovered as special cases.
- The calculations provided in the appendix could be of independent usefulness, beyond the specific problem considered in the paper. I should however immediately nuance this praise by noting that similar calculations have been done in  Liao and Couillet (2019) "A Large Dimensional Analysis of Least Squares Support Vector Machines", for more general kernel-based models.
- The paper is very clearly written.

weaknesses:
- **The setup.** Fitting a regression model to use its weights as a linear classifier, nobody does this in practice. The authors should justify why such a model is reasonable / relevant, beyond the fact that it leads to a tractable analysis. Note that the underlying estimator is a linear version of the so-called least-squares support vector machine (LS-SVM) which was analyzed in Liao and Couillet (2019) "A Large Dimensional Analysis of Least Squares Support Vector Machines".
- **(Ir)relevance to practice.** In the asymptotic regime considered in this paper, mixing real and synthetic data only helps when the proportion of real data is bounded-away from zero. Also, the practitional needs to know which samples are real and which samples are synthetic. These constraints might not match what happens in practice. This observations have also been made in Dohmatob et al. (2024) "Strong Model Collapse".
- **Non-optimal mixing.** The weights in mixing strategy considered is a bit naive / sub-optimal. The right thing to do would be to consider a general mixing constant $\alpha \in (0,1)$ (the papers uses $\alpha=1-\alpha=1/2$), and derive a theory as a function of $\alpha$, alongside the other  constants in the theory, namely $\pi$, etc.. In general the optimal value of $\alpha$ will depend on the other constants. See Jain et al. (2024) "Scaling laws for learning with real and surrogate data" and Dohmatob et al. (2024) "Strong Model Collapse"

questions:
Though I think the paper is potentially a very good paper, I still have a few worries as outlined in the **Weaknesses** section of this review and also the questions listed below (I'm open to changing my mind if addressed).

- Clarify relevance of the consider model (see **Wicknesses** section above)
- Clarify connection to literature, especially Jain et al. (2024) "Scaling laws for learning with real and surrogate data" and Dohmatob et al. (2024) "Strong Model Collapse" (see **Wicknesses** section above)
- Can you instantiate your results on a concrete setup and show the corresponding scaling laws (perhaps with some approximations of Gaussian CDF) ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732923975548

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment

comment:
We thank all the reviewers for their active engagement and interest in our work. We are also grateful for their thoughtful and constructive feedback, as well as their valuable suggestions to improve our manuscript.

In this rebuttal, we have carefully addressed all the reviewers' questions and comments, and we hope to have resolved all their concerns.

Additionally, we have provided an updated version of the manuscript, which includes references to additional related work, corrections of typos (highlighted in blue), and updated figure subtitles (Figure 7).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission considers the impact of using synthetic data (e.g., samples from a pre-trained generative model) to train a model, and how one must incorporate a verification/pruning mechanism. This is investigated first from a theoretical point of view using a Gaussian mixture model, and then experimentally with a more realistic data setup. The analysis is more general than previous work, and recovers the result of Feng et al. (2024) as a special case. Moreover, the empirical analysis corroborates and extends the claims made about Gaussian mixture analysis to the more practical real-world setting. Reviewers also note that the presentation is excellent and more approachable than one might expect given the sophistication of the tools employed.

The main concerns outlined by the reviewers were related to the Gaussian mixture model used in the theoretical analysis. However, I agree with the authors and some of the reviewers that this is not a massive limitation, given the experimental validation and the possibility that the results can be generalised to the sub-Gaussian case.