Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
Authors: Zeyu Gan
Yong Liu
Venue: ICLR 2025 Poster (2025)
Keywords: large language models; synthetic data; information bottleneck
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper explores the critical role of synthetic data in enhancing the post-training performance of large language models (LLMs) from a novel reverse-bottleneck perspective.

Abstract: Synthetic data has become a pivotal resource in post-training tasks for large language models (LLMs) due to the scarcity of high-quality, specific data. While various methods have been developed to generate synthetic data, there remains a discernible gap between the practical effects of synthetic data and our theoretical comprehension. To address this challenge, we commence by presenting a detailed modeling of the prevalent synthetic data generation process. Building upon this modeling, we demonstrate that the generalization capability of the post-trained model is critically determined by the information gain derived from the generative model, as analyzed from a novel reverse-bottleneck perspective. Moreover, we introduce the concept of Generalization Gain via Mutual Information (GGMI) and elucidate the relationship between generalization gain and information gain. This analysis serves as a theoretical foundation for synthetic data generation and further highlights its connection with the generalization capability of post-trained models, offering an understanding about the design of synthetic data generation techniques and the optimization of the post-training process. We open-source our code at https://github.com/ZyGan1999/Towards-a-Theoretical-Understanding-of-Synthetic-Data-in-LLM-Post-Training.

Published as a conference paper at ICLR 2025
TOWARDS A THEORETICAL UNDERSTANDING OF SYN- THETIC DATA IN LLM
A REVERSE-BOTTEE BOTTLENECK PERSPECTIVE
Zeyu Gan, Yong Liu"
Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
{zygan, 1 liuyonggsai e rruc. edu.cn
ABSTRACT
Synthetic data has become pivotal resource in post-training tasks for large lan- guage models (L..Ms) due to the scarcity of high-quality, specific data. While various methods have been developed to generate synthetic data, there remains a discernible gap between the practical effeets of synthetic data and our theoretical comprehension. To address this challenge, we commence by presenting detailed modeling of the prevalent synthetic data generation process. Building upon this modeling, demonstrate that the generalization capability of the post-trained model is critically determined by the information gain derived from the generative model, as analyzed from novel reverse-bottleneck perspective. Moreover, in- troduce the concept of Generalization Gain via Mutual Information (GGMI) and elucidate the relationship between generalization gain and information gain, This analysis serves as theoretical foundation for synthetic data generation and further highlights its connection with the generalization capability of post-trained models, offering understanding about the design of synthetic data generation techniques and the optimization of the post-training process. We open-source our code Theoretical
com http:///ggttaaooccooo///aa//////
LLM Post Training
INTRODUCTION
The efficacy of large language models (LLMs) extensively influenced by both the volume and quality of the training data, as established by the widely acknowledged scaling laws (Kaplan et al. 2020). Given the inherent sparsity data available during the post-training phases of LLMs, syn- thetic data plays critical role, particularly during fine-tuning and alignment processes. Over the past decades, the LLM community has increasingly employed synthetic data to augment training in scenarios where real data is scarce. As of September 2024, there are over 1,000 datasets labeled as "synthetic" on the Hugging Face platform! Several leading-edgs large language models, includ- ing LLaMA (Dubey et al. (0024). Falcon (Almazrouei et al., (023), Qwen (Bai et al. (023). and GPT-4 (OpenAl et al.) 20024), have also reported utilizing synthetic data during their post-training stages. These instances underscore the pivotal role of synthetic data in enhancing the post-training of L.LMs.
Numerous methodologies for synthetic data generation have been advanced (Patel et al. 2024 Møller et al. 2023 Park et al., (024), yet the most prevalent and efficacious approach within the community involves generating synthetic data through sampling from a proficiently trained gener- ative model, often another LLM tailored for specific domain tasks. To delincate this process more precisely, Long et al. (2024) describe the generation of synthetic data as follows: a well-trained gen- erative model M is utilized, and synthetic data Syen is produced by sampling from M, conditioned on a set prompts p. just as illustrated in the lower part of Figure] (a).
"Corresponding Author. https//huggingffcc.cel

---

Published as a conference paper at ICLR 2025
Synthetic data with such generation manner is widely recognized and has been verified to be effective in LLM post-training practice. However, several challenges persist that compromise its potential benefits. First. the quality and diversity of synthetic data vary significantly depending on the generation method and the underlying model parameters (Koo et al., 2023). This variability lead inconsistent training outcomes and may not fully address the sparsity in real data. Ad- ditionally, while synthetic data offers a promising solution to enrich the limited real data, ensuring that it sufficiently mimics real-world complexities without carrying over biases or errors from the original data is still a daunting task (Villalobos et al. 2022). Addressing these challenges requires a nuanced understanding of both the generation processes and their interaction with model training dynamics.
Unfortunately, there remains a significant gap the rigorous modeling of synthetic data, which in turn limits deeper understanding of its inherent mechanisms [Liagg et al. 2024). This lack of a comprehensive theoretical framework hinders our ability to predict the effectiveness of syn- thetic data across different LLM applications and constrains the optimization of generative models for more targeted data synthesis (Giles et al. 2022). Consequently, advancing our knowledge on how synthetic data interacts with LLMs during training phases is crucial for enhancing model per- formance and reliability, and can enable the development of tailored synthetic datasets that more effectively address specific gaps in training data, thereby enhancing the overall performance and generalization capabilities of large language models.
In this paper, we endeavor examine the influence of synthetic data the post-training phases of large language models (LLMs) through an analytical lens focused on data distribution and informa- tion content. Our investigation seeks address the following theoretical questions:
What underlies the effectiveness of synthetic data? How can model the data generation process and connect it with the generalization capabilities of post-trained models?
What is the reason for the effectiveness of synthetic data in LLM post-training?
In response to these inquiries, we introduce a theoretical framework designed to dissect the impacts of synthetic data on L.M post-training. The principal contributions of our study are outlined as follows:
1. We develop modeling of synthetic data generation from distributional perspective, pro- viding a theoretical foundation for understanding the generation process and its implica- tions on LLM post-training.
2. Drawing on this modeling. propose reverse-bottleneck framework that elucidates the mechanisms through which synthetic data influences LLM post-training.
3. We perform a theoretical analysis from an information -hhoorettct standpoint, delivering sev- eral upper bounds that quantifies the expected generalization capabilities of LLMS when trained with synthetic data.
The remainder of this paper is structured as follows. In Section 2 we provide a comprehensive review of literature pertinent to our research. In Section 3 we first delineate the symbols and foun- dational concepts critical to our analysis, then introduce the modeling for synthetic data generation and bridge its connection with generalization capability of post-trained models. (fattiounjitt framework, designed to assess the effects of synthetic data post- training stages of LLMs, and to establish generalization error upper bounds. The paper concludes summarizing findings and discussing potential avenues for future research.
novel
with Section 5
RELATED WORK
2.1 GENERATIVE DATA AUGMENTATION
Generative models constitute a category of machine learning models that are specifically trained to create data points mimicking the original data distribution. Various types of generative models have been developed, each suited to particular data types and model architectures. Notable among these are Variational Autoencoders (Kingma 2013), Generative Adversarial Networks (Goodfellow
2

---

Published as a conference paper at ICLR 2025
Distribution Compression
Duta Cuestin
D
(P)
Prorosia K(())
Conation Divergence Dry Epenl
DR
@
Prompt
+
Sutcker
Output
2gen
Synthetic Data Generation
Drr Ip)
(a) Overview synthetic data generation
(b) relationships between distributions
Figure I: overview of the synthetic data generation modeling and the relationships between the distributions. (a) The synthetic data generation process and the corresponding distribution compres- sion process. (b) The relationships between the distributions in the generation process.
et al. 2014), Normalizing Flows (Rezende & Mohamed 2015), and, more recently, diffusion mod- els (Rombach et al. 2022). Building on this premise, generative data augmentation has emerged promising approach to bolster machine learning model performance (Yamaguchi et al. 2020 This technique involves scaling up the available training dataset by generating new data points from limited pool of labeled data using generative models. Empirical evidence suggests that generative data augmentation is particularly effective across various tasks, including knowledge graph reason- ing (Maharana Bansal 2022), text-to-image generation (Yin al. 2023), and relation extraction from natural language texts (Hu et al. 2023). Theoretical investigations have also been conducted elucidate the underlying mechanisms through which generative data augmentation delivers these benefits (Zheng et al. (2023a). Collectively, these advancements highlight generative data augmen- tation as highly promising avenue for improving machine learning model performance, especially in scenarios characterized by scarcity of labeled data.
2.2 SYNTHETIC DATA IN LLMS
Similar to traditional generative data augmentation, synthetic data produced by LLMs is increasingly utilized to enhance post-training phases. Given the scarcity of labeled data in specific domains, syn- thetic data plays crucial role in boosting the performance of LLMs across variety of downstream tasks, including text classification (Li et al. (023), clinical text mining (Tang et al. 2023), and code generation (T sai al. 2024). However, unlike classic generative data augmentation, synthetic data within LLMS is typically generated by the language models themselves and often predominates the training data in post-train post-training stages. This predominance stems from the high-quality demands of synthetic data in LLM contexts, which necessitates alignment with human intent. Efforts to en- hance the quality of synthetic data LLMMs have included integrating methodologies such active learning (Wagner et al. 2024) and reinforcement learning (Setlur et al. 2024). Despite these ad- vancements, the theoretical understanding how synthetic data influences the learning process in L.LMs remains limited. Key questions persist regarding the mechanisms through which synthetic data impacts LLM training and the optimal strategies for designing synthetic data to maximize LLM performance (Long et al.] (024). Addressing these questions is essential for furthering our compre- hension and utilization of synthetic data in enhancing large language model efficacy.
2.3 INFORMATION BOTTLENECK THEORY & GENERALIZATION CAPABILITY
The information bottleneck (IB) theory, as introduced by (Tishby et al. 2000), serves as a theo- retical construct designed to elucidate the learning processes within neural networks. In essence, for a given Markov chain - Z Y, the IB theory aims to optimize the learning process by maximizing the mutual information between Y and Z while minimizing the mutual information be- tween and Z (Hu et al., 2024). IB theory has been widely adopted across various deep learning fields, such text classification (Slonim al. 2001), sentence summarization (West et al. 2019
3

---

Published as a conference paper at ICLR 2025
and image clustering (Hu al. 2019). Expanding upon these foundations, further research has plored generalization error upper bounds that incorporate mutual information (Russo & Zou 2019 Xu & Raginsky 2017). These studies have established a connection between the generalization capabilities of deep neural networks (DNNs) and IB theory (Alquier et al. 2024). More recent ad- vancements have also highlighted the links between mutual information bounds and the PAC-Bayes framework (Banerjee & Montúfar 2022). This type of bound suggests that the generalization error intrinsically limited by the relevance between the training data and the learned model parameters.
PRELIMINARIES
3.1 NOTATIONS & EXPERIMENTAL SETUP
Let Sanchor represent the real data utilized for generation, and Sgen denote the synthetically gener- ated data. The LLM employed in the generation process is designated as M, with the input prompt labeled as p. The distribution of the post-training target task T is referred to as D, while the output distribution of the LLM is denoted by Das- Additionally, the distribution corresponding to the syn- thetic data is represented as geen- The generalization error associated with the under-alis under-aligned LLM on the synthetic data Sgen is expressed as BIr(T:Sm., and the generalization error related to the anchor data is indicated by Frr(x:Snatrr We define (.) as the entropy of a random variable, ((: divergence,
as the mutual information between two random variables, DKL. as the Kullback-Leein and Drv as the total variation distance. The detailed definitions are listed in Appendix A.
To provide a more intuitive demonstration, we use an example in the Gaussian mixture model (GMM) setting during the explanation. In simple terms, we assume that the target of the post- training task contains K + J Gaussian distribution components, and set up a corresponding ground- truth GMM (gt- GMM G) to represent the target of the post-training task. After that, randomly sample from the first K components of the gt-GMM as anchor data. To simulate the generative model M, added L. random components to the gt-GMM, which may include extra distributions, making M a GMM with total K + J + L components. Finally, we randomly sampled data from M to obtain the simulated synthetic data. The detailed experimental setup is listed in Appendix
3.2 MODELING SYNTHETIC DATA GENERATION
Long et al. (2024) provided a brief summary for the synthetic data generation, the overall process of synthetic data generation can be modeled Sgen <- Mp Suncorr ). where Sgen is the gener- ated synthetic data, M is the generation model (usually a well-trained L.LM), p is the prompt for generation, T is the downstream task, and Sanchor is the anchor data (real data). More specifically, the prompt p is derived from the generation task T and the anchor data Sanchor and consists of three crucial elements: p(T, Sanchor) < E (Crusk; Coundition Cdemo), where E the prompt template, Cask is the task element, Ccondition is the condition element, and Edemo is the anchor data element. The conceptual framework of this modeling is straightforward. Sgen essentially constitutes modifica- tion of the output generated by M in response to the prompt p, where the prompt p is defined by the downstream task T and the anchor data Sanchor. The specifics of the generation process are thus governed by the prompt p and M.
We enhance our understanding of synthetic data generation by reevaluating the distributional rela- tionships among the anchor data Sundurr the prompt p. and the synthetic data Syen produced. We postulate that the anchor data Sanchor is sampled from distribution D associated with the down- stream task, and the generation process is influenced by both the prompt p and the generative model M. Consequently, Sgen represents a modification of M's output in response to the prompt p: Sgen M(p) + e, where is noise term for the measurement of revision, such as the data curation process.
The prompt p intricately linked to the downstream task T and the anchor data Sanchor- We postulate that Sanchor forms the core of the prompt p. upon which a task-specific transformation function ØT applied. Consequently, the prompt p can be mathematically modeled as p Φ (Sanhar)) where Φv is a function that maps the anchor data to the prompt, consists of all the task-relevan transfor- mation, like the template and other customized settings for more faithful and diverse generation.
4

---

Published as a conference paper at ICLR 2025
For simplicity, note that Sanchor D, p Ancher Data Synthetic Data
Dp. (p) ~ (-p), and Sgen ~ gen and comprchensive details about the relation- ships between the distributions are listed in Ap- pendix The overall synthetic data generation process In our modeling is depicted in Figure
(a). This illustration enhances our understand- ing of the connection between the generation process and distributions.
Dimension
The lower part of Figure ! (a) details the spe- cific stages of data generation. Initially, the anchor data Sanchor undergoes transformation via the function @T to constitute the prompt p, which in turn is used by the LLM . to gener- ate the synthetic data S Semn incorporating noise I (a) delineates the corresponding process of distribution shift. Sunchor is derived from distribution D, and the prompt p emerges from distribution Dp con- ditioned on ④T. The LLM M produces the output M (p) from the conditional distribution D (-lp), and the final synthetic data Sen is sampled from DDen gen, representing convolution of DM and D. also conditioned on p.
The upper portion of Figure
-2
Dimension
Figure 2: The simulation of the distribution rela- tionships with GMMs. "e" represents the anchor data sampled from distributions colored blue, and represents the synthetic data sampled from distributions colored orange.
Given that D, relates solely to σT and D (or Sanchor), and D yen related only to M and p. the transition from Sanchor to to Sgens (i.e. Sanchur - Sgen). constitutes a Markov chain. Figure (b) provides a comprehensive view of the distributions and the nature of the distribution shift discussed. Specifically, D denoted as the orange circle, and Da is denoted as the blue circle. After M being prompted p, the conditioned distribution DM (-Ip) denoted as all blue areas, and the final Deen is represented as the deep blue area after the compression
This illustration aids in understanding that the generation process essentially compresses the output distribution of M, Da, towards the post-training target distribution D, based on the conditions imposed by the prompt p and noise
To provide a clearer visualization, we simulate the distribution relationships using GMMs, the re- sult is depieted in Figure 2 The distributions of Sgan are visualized
effort to encompass the distributions of Sanchor However, since Sgen is derived from the model M, which incorporates more complex distribution components, the distribution of Sgen only attempts to mirror Sanchor but also extends beyond, covering broader areas.
3.3 BRIDGING THE GENERALIZATION CAPABILITY
Subsection 3.2 offers an exhaustive examination of the synthetic data generation process, which is pivotal for elucidating the generalization error associated with the under-aligned LLM π when applied synthetic data Sgen- This subsection endeavors correlate the generalization error of on the synthetic data Sgen with the synthetic data generation process as previously delineated.
Given the focus on the alignment task performance, and considering that π is a pre-trained LLM, then is subsequently trained on synthetic data sampled from D Dgens the generalization error of post- trained LLM Spo is delineated as ( = ( - Rs (TT/0)) where D is the real distribution of the post-training task. (D EEND [e(mSSn, 2)] denotes the true error of Sm on the distribution D, and Rs (mr = Lyes Se0 2)] denotes the empirical error of S on the synthetic data. Similar like Zheng et al. (2023b), and by the definition of the synthetic data generation process, we can simplify the above upper bound as the following lemma:
Lemma 3.1. Assume that π with loss function É bounded by C, given an i.i.d. synthetic dataset Sgen generated the above defiee, then the following synthetic data training generalization error
5

---

Published as a conference paper at ICLR 2025
Forward Process
Information Flow
(latent)
reverse-bottleneck
M
Synthetic Data Generation
Sacchor
P
Sgen
S anchor
S gen
(latent)
enc
dec
Classic ML
X
Z
bottleneck
Figure 3: Illustration about the reverse bottleneck effect and comparison with classic ML process. Left: the similarity between the forward process of synthetic data generation and classic ML. Right: the difference between the information flow of the two process, where synthetic data generation gains information from M, constituting a reverse-bottlenek.
upper bound holds:
Err(z S C (Drv(D, + Drv(DM; Deon)) (om))++ +
Ro (1 (T Ro (n (m))
Distributions Divergence
Generalization Error syntheric data
The proof is referred the Appendix D The divergences can be defined the task divergence (Drv Das) and the generation divergence Drr((Da, D D),,)), which is denoted in Figure 1 (b). The task divergence is determined by the ability of the LLM M and the relevance with the task T. The generation divergence is determined by the generation process including the prompt engineering and the data curation. In the training practice, the two divergences are controlled by either the strong ability of M or the strict prompt engineering, this partially explains why synthetic data is effective.
4
MAIN RESULT
In this section, we delves deeper into the implications of the synthetic data generation process on the generalization capabilities.
4.1 INFORMATION GAIN & EYPERSE-BOTTLENCKE
To enhance understanding of the synthetic data generation process, delineate suite of cepts pertaining to the information-flon within this process. Initially, we introduce the notion of synthetic factors, which represent the fundamental elements that influence the formation of Sgen.
Definition 4.1. fantthetic.actorss Assume that the synthetic data Sgen M() is derived from two factors, i.e. M(p) = h((p) + g(ea). The represents the factor w.r.t. prompt and the eM represents the factor w.r.t. applied LLM M.
With the synthetic factors established, we posit that the synthetic data Sgen is primarily governed by two distinct factors: ep and eM, which are actually assumed random variables related the prompt p and the LLM M respectively. Following this framework, proceed to introduce the concept of information gain within the context of the synthetic data generation process.
Definition 4.2. (Information gain.) ) The information gain in the synthetic data generation process is defined as:
Al
I (b(c))
The information gain, denoted as AI, serves metric for assessing the enhancement of informa- tion the synthetic data generation process. It quantifies the incremental information content from the prompt p to the synthetic data Syen. specifically, the information introduced by the L.M M.

---

Published as a conference paper at ICLR 2025
In alignment with the classical information bottleneck theory, also introduce the concept of compression bottleneck, which is defined in the context of synthetic factors.
Definition 4.3. (Compression bottleneck.) We consider the compression bottleneck of the synthetic data towards the post- traitrd model parameter W as:
yyn I [eM, W) + [ep, W).
Having delineated the concepts of information gain and compression bottleneck, we now advance our discussion to clarify the information flow within the synthetic data generation process. introduc- ing the notion of reverse-bottleneee effect. This framework acknowledges that the distribution Dp directly influenced by ΦT and D anchor (or Sanchor), while Dgen pertains solely to M and p. Conse- quently, the sequence Sanchor - p Cp - W constitutes Markov chain. Similarly, the process (p) CM + W also forms a Markov chain.
The former Markov chain, as depicted in the left part of Figure 3 parallels classical machine learning (ML) process, in which the input is transformed into latent representation Z via an encoder, and then Z is further decoded into the output Y through decoder. Similarly, in the synthetic data generation process, the input Sanchor is converted to p (which is often assumed as latent in practical applications) via ΦT, and subsequently p is transformed into Sgen by M. However, the presence of the latter Markov chain introduces a crucial distinction between the two processes from an information flow perspective due the prior knowledge embedded by M. As illustrated in the right part of Figure unlike classic ML process, the synthetic data generation process leverages M to facilitate information gains, thereby enriching the informational content of Sgen-
This perspective emphasizes the distinctive dynamics and augmented capabilities of the synthetic data generation process in terms of capturing and utilizing information. Subsequently, we aim to analyze the relationship between the information gain and the generalization error of the model after training on the synthetic data.
4.2 INFORMATION- FLOW GENERALIZATION ERROR UPPER BOUND
In this subsection, we endeavor to derive the upper bounds of the generalization error from an information-Aee flow perspective, employing the concepts previously defined. We initiate our analy- sis with classical information upper bound applicable to deep neural networks, as elaborated in Lemma 4.4 Zhhang et al.) 2018).
Lemma 4.4. For a deep neural network with L hidden layers, input S, and parameters W. The loss function is -asu-uuuussia with respect to (W, Z) given any w, if all L hidden layers are contraction layers, the expected generalization error can be bounded as follows,
L E. (R(W)- Rs)) exp (-2 log !)
V 202
200 n I(S,W).
Lemma 4.4 establishes connection between the expected generalization error and the mutual in- formation between training data S and learned model parameters W. Despite network depth L and instance volume n. the principal constraint is imposed by the mutual information term.
Accordingly, in scenarios where post-training is with synthetic data, the generalization error is in- herently constrained by the mutual information between the synthetic data Sgen and LLM parameters after training, denoted as (Sgen W). Characterizing this term presents significant challenge due the difficulty measuring mutual information accurately. To address this, we introduce an ana- lytical upper bound for I(Sgen: W) in Lemma 4.5 to facilitate a more comprehensive understanding of the dynamies influencing model performance in post-training.
Lemma 4.5. (Information-loen flow upper bound.) Given synthetic dataset S Seen defined above, and model parameters learned from Sgen the mutual information term I(Sgen, W) can be bounded by the following inequality:
I(S gen W) < --- Byn H(EM) 5
(5)
where 8z, indicates the efficiency during the data curation and model prompting process, which detailed in the proof in Appendix E Together with Lemma 4.4 we can further derive an upper bound for training procedure with relation to the synthetic data defined above in Lemma 4.6
7

---

Published as a conference paper at ICLR 2025
Lemma 4.6. (Generalization upper bound w.r.t synthetic data.) For deep neural network with L hidden layers, the parameters W are optimized from synthetic data Sgen described aboved. The loss function is r-sub-Gaussiie with respect to (W, Z) given any if all L hidden layers are contraction layers, the expected generalization error can be bounded as follows:
(-2 L
1
20
AT + Bayn + H(ess + Sc.pl
E Ro (g)) ) (T S <xp
log
(6)
Lemma 4.6 delineates quantifiable upper bound for the expected generalization enror in relation to synthetic data. Beyond basic configuration parameters such as network depth L. and data size n, this upper bound determined by four key factors outlined in the corresponding remarks.
Remark 1. AT quantifies the information gain during the data generation process. This bound demonstrates that an increase in information extracted from the model M enhances the quality of the generated data.
Remark 2.
Beyn denotes the compression bottleneck, which is defined the mutual information between synthetic factors and the model parameters W. A more pronounced compression of this term leads to improved generalization performance.
Remark 3. (eM)) ) represents the entropy associated with the synthetic factor relative to the model M. Intuitively, reducing this entropy by choosing model M more aligned with the specific tasks can substantially enhance downstream generalization.
Remark 4.
dep concerns the efficiency during the data curation and model prompting process, highlighting the impact of noise and other data degradation factors on the overall data utility.
These factors collectively influence the generalization performance, indicating that better general- ization ability can be achieved by enhancing the information gain, reducing the compression bottle- neck, minimizing the entropy, and balancing the efficiency. Finally, by integrating the insights from Lemma 3.1] the overall upper bound of the expected generalization error in the LLM post-training with synthetic data can be derived as a comprehensive boundary in Theorem
Theorem 4.7. (Synthetic data post-training upper bound.) For the same condition as lemma 4.6 and a synthetic data generation process described above, the generalization error of the model post-trained on the synthetic data can be bounded as:
E(Err(Z5(m)) < (Drv(D, Dws) DTv(DM: D
Distributions' Divergence
2x AT B + H H(MM) Se,pl
exp
log
n
Genevalization Eroor wet. synthetic data
4.3 GENERALIZATION GAIN WITH SYNTHETIC DATA
Theorem 4.7 establishes a general upper bound for the generalization error of LLMs post-trained with synthetic data. In this section, our objective to analyze the generalization gains achieved by using synthetic data compared to scenarios devoid of synthetic data.
We commence our analysis with the anchor data Smetor- Analogous the definition of Err( the generalization error of an LLM that has been post-trained on Sanchar is defined as Err(x Sunder)
Smnter) Riswa (7. It is logically sound to assume that Sunchor is sampled from the distribution D. Building upon Lemma 4.4 and assume that Sanbor comprises m instances, we can
derive the subsequent result in Lemma 4.8
Lemma 4.8. (Anchor data post-training upper bound.) For the same condition lemma 4.6 the generalization error of the model π post-trained on the anchor data can bounded as:
(
L
1
20² T(S ([Sanchor W
E(Frr( B(Err(I=Sutrr) & exp
(
log
(8)
m
8

---

Published as a conference paper at ICLR 2025
where r and W' are the variables of the model trained with Sanchor. noted different from that of model trained with Sgan 77 is constant depending the information loss and W is the model pa- rameters. Given that m << n typically applies in real-world scenarios, Lemma (.8 offee represents less stringent upper bound compared to Lemma this results in potentially poorer generalization when relying solely on Sanchur rather than utilizing synthetic data.
But pertinent question arises: do other aspects of synthetic data generation, beyond the influ- ence of data size, also contribute to improvements in generalization performance? Our focus is on examining how various elements within the synthetic data process impact generalization during post-training. It is inappropriate, however, to directly compare other components across these two bounds due variations in loss and training data specifics, which affect the parameters r and W differently, where represents a measure of information compression and is challenging to quan- tify accurately (Zhang et al. 2018). Thus, our analysis primarily centers on the mutual information terms [[Sachhr, W) and (SSgen: W). To systematically evaluate the generalization capabilities con- ferred by synthetic data in relation to these mutual information metrics, we introduce a definition for generalization gain measurement as Definition 4.9
Definition 4.9. (Gemeralization Gain via Mutual Information, GGMI.) GGMI is defined as the dif- ference between the mutual information terms in the two generalization upper bounds:
GGMI = [[Sanchor: W
g(Sgen> W).
A larger upper bound for the GGMI signifies greater potential generalization benefits when utilizing synthetic data. To elucidate the impact of synthetic data on model generalization, we isolate the influence of W' and establish that the GGMI can be effectively bounded.
Theorem 4.10, (Upper bound of GGML)) Given the synthetic data generation above, W' is param- elerized by training with Sanchor, and W is parameterized by training with Syon. the GGMI can be bounded as follows:
GGMI S AT (ar + 1)H (W) + 2AH + H((See) gen (W) + CW.p
(10)
where AH- H (Sanchor) H (enn), eW.p H( W) (Sanhorr M(p)), it is assumed that H (Sanchor W') = H (Samchor W), a 20.
The proof is referred to Appendix F Consequently, we proceed to conduct a thorough analysis of each component specified in Theorem 4.10
Al represents the information gain derived from the model M. An increase in this information gain typically leads improved generalization capability for Sec compared -Saa as the model leverages additional insights to enhance performance.
Remark 1.
Remark 2. H (Sanchor (W) indicates the conditional entropy between the anchor data Sanchor and the model parameters W. For larger upper bound of GGMI, it is encouraged to decrease this value by strengthen the relevance between model parameters W and anchor data Sanchor.
Remark 3. AH denotes the entropy decrease when generating synthetic data Spen from anchor data Sanchor It implies that more uncertainty is eliminated during synthetic data generation leads to more generalization ability.
Remark 4. H (S) reflects the conditional entropy between the synthetic data Sgen and the model parameters W. Weakening the relevance between these two entities is encouraged to ensure that the model learns the general pattern of synthetic data thus leading to better generalization.
Remark 5. CW.p denotes the effect of information compression by the training algorithm. A more pronounced compression effect typically results in higher value, suggesting that efficient data representation contributes positively to model efficacy.
As emphasized in (Long al., 2024), the generation of synthetic data typically focuses on two primary objectives: faithfulness and diversity. These objectives are associated with AHl and AT, respectively. Specifically, AH, which quantifies the entropy decrease during synthetic data genera- tion, presented in Theorem 10 encourages the model to eliminate uncertainty during synthetic data generation, thereby enhancing the faithfulness of the synthetic data. In addition, Al serves as measurement of the additional information introduced by the generative model M. Given that M is typically pre-trate pre-trained on a more extensive dataset, A/ in Theorem 4.10 promotes the objective of diversity by facilitating greater information gain from M.

---

Published as a conference paper at ICLR 2025
KL Gap vs K
KL Gap vs
KL Gap vs
KL Gap
KL Gap
Gap
K
Figure 4: KL Gap with different components settings. By default, we set K = J L = 2, and vary each of them from 2 15 to observe the corresponding change of KL Gap. An increase of KL Gap is observed when J increases, while decrease is observed when K and L increase. The shading indicates the standard deviation of 100 rounds of random settings.
4.4 VERIFICATION WITH GMM SIMULATION
Building upon the simulation settings, we offer a straightforward validation of the theoretical results discussed above. Specifically,
first fit a GMM comprising K+J+ componetts to both Sanchur and Sgen. yielding π Sasderr and π Sez respectively. We then introduce a metric termed KL Gap, defined Smater [G) - k (T Sp |[G), which represents the difference of KL-divergence between the fitted GMMs (w Santee and x -00)) and the ground-truth GMM G. A larger KL Gap corresponds to greater GGMI, indicating enhanced generalization benefits from synthetic data.
as ( Santer
To control the variables outlined in Theorem 4.10 we adjust the number of components in the GMM M and the ground-truth GMM G. The result is illustrated in Figure 4 Generally, increasing larger upper bound for GGMI. contrast, larger K amplifies the influence of anchor data within the post-training target distribution, thereby increasing the (Snchor (W) term and tightening the upper bound of GGMI. Additionally, while an increase in L enhances H (Seen) W), it concurrentyy leads to reduction in AH. As result, we observe trade-off manifested as a decrease in the KL Gap in our simulation outcomes.
facilitates the scaling of AT, resulting
CONCLUSION
In this paper, we have conducted a detailed analysis of synthetic data utilization post-training large language models (LLMs). We present comprehensive modeling of the current synthetic data generation process, focusing on its distributional aspects, which further connects the generalization capabilities of post-trained models. We introduce a novel breesse-bottlenek framework, allowing us to derive a measurable upper bound on generalization errors. Our analysis reveals that the pivotal constraint on generalization ability is influenced by the information gain from the generative model M. Additionally. we present the Generalization Gain via Mutual Information (GGMI). showing that larger information gains enhance the generalization capability post-trained models. We empha- size the importance of balancing faithfulness and diversity during post-training stages, providing a theoretical foundation for existing methodologies. Unfortunately, due to limitations in computa- tional resources, are unable validate our findings within real-world LLM settings. Looking ahead, future research should focus on developing adaptive models that respond to the evolving char- acteristics of synthetic data. This includes enhancing generative models and fine-tuning parameters for specific learning scenarios, as well as exploring various generative models to better replicate real-world data complexities while improving model performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by National Natural Science Foundation of China (No.62476277), Na- tional Key Research and Development Program of China (No. (4.22024YYE00032000 CCF-ALIMAMA TECH Kangaroo Fund NNL.MAAAA OF 2024008), and Kuawei-Renmin University joint program on Information Retrieval. We also acknowledge the support provided by the fund for build- ing worldclass universities (disciplines) of Renmin University of China and by the funds from Bei- jing Key Laboratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, from Engineering Research Center Next-Generation Intelligent Search and Recommendation, Ministry of Education, from Intelligent Social Governance Interdisciplinary Platform, Major Innovation Planning Interdisciplinary Platform for the "Double- First Class" Initiative, Renmin University of China, from Public Policy and Decision-making Re- search Lab of Renmin University of China, and from Public Computing Cloud, Renmin University of China.
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, and al. The falcon series of open language models, 2023. URL /ttp:p/:////axx.oor/////////////
Pierre Alquier et al. User-friendly introduction to pac-bayes bounds. Foundations and Trends@ in Machine Learning, 17(2):174-303, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, and et al. Qwen technical report, 2023. URL https://arxi
v. org/ abs 22099. 16609
Pradeep Kr Banerjee and Guido Montúfar. Information complexity and generalization bounds. In 2021 IEEE International Symposium on Information Theory (ISIT), pp. 676-681. IEEE, 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tumed Ilm, 2023. URL |tt.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and al. The llama herd of models, 2024. URL
2ttp.//..rrr/aa///77.1...... 21783
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--23) 2024.
Oscar Giles, Kasra Hosseini, Grigorios Mingas, Oliver Strickson, Louise Bowler, Camila Rangel Smith, Harrison Wilde, Jen Ning Lim, Bilal Mateen, Kasun Amarasinghe, et al. Faking feature importance: A cautionary tale on the use of differentialll--rriee synthetic data. arXiv preprint arXiv:2203.01363, 2022.
Ian Goodfellow, Jean Aouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farkes Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Shizhe Hu, Xiaoqiang Yan, and Yangdong Ye. Multi-task image clustering through correlation propagation. IEEE Transactions Knowledge and Data Engineering, 33(3):1113-1127 2019.
Shizhe Hu, Zhengzheng Lou, Xiaoqiang Yan, and Yangdong Ye. A survey on information bottle- neck. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Xuming Hu, Aiwei Liu, Zeqi Tan, Xin Zhang, Chenwei Zhang, Irwin King, and Philip S Yu. Gda: Generative data augmentation techniques for relation extraction tasks. arXiv preprint arXiv:2305.1666 2023.
11

---

Published as a conference paper at ICLR 2025
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXIv:2001.08361, 2020.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint
2013.
Sconmin Koo, Chanjun Park, Scolhwa Lee, Jachyung Seo, Sugyeong Eo, Hyeonseok Moon, and Heuiseok Lim. Uncovering the risks and drawbacks associated with the use of synthetic data for grammatical error correction, IEEE Access, 2023.
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large lan- guage models for text classification: Potential and limitations. arXiv preprint arXiv:2310.07849, 2023.
Hao Liang, Linzhuang Sun, Jingxuan Wei, Xijie Huang, Linkun Sun, Bihui Yu, Conghui He, and Wentao Zhang. Synth-empathy: Towards high-quality synthetic empathy data. arXiv preprint arXiv:2407.2 2024.
Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On Ilms-driven synthetic data generation, curation, and evaluation: A survey, 2024. URL https:
//arxiv. org. /206///2406 15126
Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5085-509,, 2020.
Adyasha Maharana and Mohit Bansal. Grada: Graph generative data augmentation for common- sense reasoning. In Proceedings of the 29th International Conference on Computational Linguis- fics, pp. 4499-4516, 2022.
Meta. Introducing Ilama 3.2, 2024. http://tttttt///w.aa.........o/.
cards
Anders Giovanni Møller, Jacob Aarup Dalsgaard, Arianna Pera, and Luca Maria Aiello. The parrot dilemma: Human-labeled vs. llm-augmented data in classification tasks. arXiv preprint arXiv:2304.13861, 2023.
OpenAl. Gpt-4o system card, 2024. URL
OpenAl, Josh Achiam, Steven Adler, and al. Gpt-4 technical report, 2024. URL https:
Jeiyoon Park, Chanjun Park, and Heuiseok Lim. Chatlang-8: An llm-based synthetic data generation framework for grammatical error correction. arXiv preprint arXiv:2406.032202 2024.
Ajay Patel, Colin Raffel, and Chris Callison-Burch. Datadreamer: tool for synthetic data genera- tion and reproducible IIm workflows. arXiv preprint arXiv:2402.10339 2024.
Chen Qian, Jie Zhang, Wei Yao, Dongrui Liu, Zhenfei Yin, Yu Qiao, Yong Liu, and Jing Shao. Towards tracing trustworthiness dynamics: Revisiting pre-training period of large language mod- els. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 4888, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.184533 http:///ww.1122......nn.gee..000 finiiii.fiidigggg aclpsss-cll..m .20
URL
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Interna- tional conference machine learning, pp. 1530-1538. PMLR, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory, 66(1):302-32) -0,33,233,,, 2019.
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. arXiv preprint arXiv:2406. 14532, 2024.
Noam Slonim, Naftali Tishby, et al. The power of word clusters for text classification. In 23rd European Colloquium on Information Retrieval Research, volume 1, pp. 200, 2001.
Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. Does synthetic data generation of Ilms help clinical text mining? arXiv preprint XXiv:2303.04360 2023.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followii llama model. battps://githae com/tat su- 1ab/ stanford alpaca, 2023.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. Code less, align more: Efficient Ilm fine-tuning for code generation with data pruning. arXiv preprint arXiv:2407.05040, 2024.
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and An Chang Ho. Will we run out of data? limits of Ilm scaling based on human-generated data. 2022.
Stefan Sylvius Wagner, Maike Behrendt, Mare Ziegele, and Stefan Harmeling. Sqbe: Active learn- ing using llm-generated synthetic data for stance detection in online political discussions. arXiv preprint arXiv:2404.. 08078, 2024.
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer/-ppreeerence Complementing ratings with preferences, 2024.
URL /ttp:://axxivvvi
01257
Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. Bottlesum: Unsupervised and self- supervised sentence summarization using the information bottleneck principle. arXiv preprint arXiv:190..07405. 2019.
Aolin Xu and Maxim Raginsky. Information- analysis of generalization capability of learn- ing algorithms. Advances neurul information processing systems, 30, 2017.
Shin'y ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi- domain learning gans. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 34, pp. 6566-6574, 2020.
Yuwei Yin, Jean Kaddour, Xiang Zhang, Yixin Nie, Zhenguang Liu, Lingpeng Kong, and Qi Liu. Ttida: Controllable generative data augmentation via text-to-text and ext-tto-image models, 2023.
URL https::/aaxxi......:/ org. abs/ /20ss/2300 2304. 08821
Jingwei Zhang, Tongliang Liu, and Dacheng Tao. An information-theorrticc view for deep learning. arXiv preprint arXiv: 0.00000, 2018
Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmen- tation. Advances in neural information processing systems, 36:54046-540600 2023a.
Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmen- tation, 2023b. BtLhhtts::/ http://///aaaaaavoroaaa/////////
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 66:46595-46623 2023c.
13

---

Published as a conference paper at ICLR 2025
A DEFINITION AND INTRODUCTION ABOUT INFOMATION BOTTLENECK THBORY
A.1 DEFINITION OF NOTATIONS
We summarize the notations used in subsection 3.1 and provide their specific definitions.
First, define the notations about the concept related to information entropy.
Definition A.I. (Entropy of a random variable.) The entropy of a random variable X is defined as: logp(æ),
(()) =
- -
For continual random variable, the entropy is defined as:
/ p(x)) log p(x)dr.
H(X) = H(X)=-
The entropy is measurement of the uncertainty of the random variable, and the larger the entropy, the more uncertain the random variable is. It can also considered as the average information content of the random variable.
Definition A.2. (Conditional entropy of a random variable.) The conditional entropy of a random variable X given another random variable Y is defined as:
H(X = > p(x, logp(zly)
For continual random variable, the conditional entropy defined as:
H(XY)) / p(z, y) log logy(rly)
The conditional entropy is measurement of the uncertainty of the random variable X given the information of the random variable Y. It can also be considered as the average information content of the random variable X with Y given.
Building upon the definitions above, we can further define the concepts we used in the main text with relation information theory, including relative entropy, total variation distance, and mutual information.
Definition A.3. (Relative entropy or Kullback-Leibler divergence.) The relative entropy or Kullback-Leibe divergence between two probability distributions and is defined as:
p(x)
Dxu(pllg) = - -p(z))og log
g(x)
The relative entropy serves as measurement of the difference between two probability distributions. Definition A.4. (Total variation distance.) The total variation distance between two probability distributions and finite or countable set E is defined as:
Div(p, q) supAeE Ip(A) q(A))
1
Σ - g(x))
2
ZEE
The total variation distance is also measurement of the difference hetween two probability distri- butions.
Definition A.5. (Mutual information.) The mutual information between two random variables X and Y is defined
I(X,Y) H(X) (X
The mutual information is measurement of the amount of information that one random variable contains about amother random variable. The larger the mutual information, the more information the random variables share.
14

---

Published as a conference paper at ICLR 2025
gt- GMM
L
anchor sample part
unsampled part
lask-irrelevant part
M
sampling
Synthelic Data
Ancher Cata
S gen
Sas
Figure Illustration of the setup of the GMMs for simulation.
A.2 THE INFORMATION BOTTLENECK THEORY
The information bottleneck (IB) theory is theoretical construct designed elucidate the learning processes within neural networks. In essence, for given Markov chain X + Z + Y, the IB theory aims optimize the learning process by maximizing the mutual information between and Z while minimizing the mutual information between X and Z. The optimization objective in IB theory generally expressed as:
= [(2
(11)
Originally developed within the context of information compression, IB theory has been widely adopted across various deep learning fields, and further research has explored generalization error upper bounds that incorporate mutual information (Russo & Zou 2019: Xu & Raginsky (017). These studies have established a connection between the generalization capabilities of deep neural networks (DNNs) and IB theory (Alquier al., 2024). representative formulation of general- ization error upper bound from the perspective of mutual information as follows:
20 I(S,W),
genErr
(12)
where S and W are training data and model parameters respectively, with the assumption that the loss is o-subgaussian. This type of bound suggests that the generalization error intrinsically limited by the relevance between the training data and the learned model parameters.
B DETAILS OF EXPERIMENTAL SETTINGS
We utilize Gaussian Mixture Models (GMMs) to simulate the data generation process, as illustrated in Figure 5 Overall, we use a gt-MMM simulate the ground- or the post-training target distribution, and a GMM M to simulate the generative model applied in the data generation process.
Their are three parts of components the GMMs: the anchor sample part with K components, the unsampled part with J components, and task-irrelevan part with L components. It is assumed that the post-training target distribution is combination of the anchor sample part and the unsampled part, thus the gt-GMM contains K components from the anchor sample part and the unsampled part, which denoted blue in Figure However, the anchor data only sampled from the anchor sample part. This is reasonable assumption for the real-world scenario, since the anchor data is sparse and hard to cover the whole post-training task distribution.
Additionally, the generative model M is assumed to be GMM with K+J- L components. Except for the post-training target distribution, M also contains a task-irrelevant part, which denoted as orange in Figure 5 This is due to the fact that the generative model is always pre-trus pre-trained larger scale of data. and may not be perfectly aligned with the post-training target distribution, and may introduce task-ineleevant components in the synthetic data generation process.
15

---

Published as a conference paper at ICLR 2025
Building upon the settings above, we sample from the anchor sample part components of the gt- GMM to generate the anchor data mentors and sample from the generative model M to generate the synthetic data S gen In the experiment, we set the dimension of the data to be d = 2, and K = = I. 2 by default, to facilitate the visualization and analysis of the data generation process.
For simulation in the main text, we set the number of initial anhcor data = 50 for each anchor sample part component, and resample the 1000 data points for both GMM fitted Sanchor and Sgen- For the simulation to evaluate the KL Gap, the results are averaged over the 100 rounds, where for each round, we also resample the final data points for 100 rounds.
C DETAILS OF SYNTHETIC DATA GENERATION MODELING
In this section, we elaborate on the modeling aspects of synthetic data generation, particularly fo- cusing the distributions of the prompt p and synthetic data Sgn- which are central to the process of generating synthetic data for training large language models (LLMs).
Distribution of p: The prompt p is is derived from the transformation function ΦT. applied the anchor data Sanchor. This function is assumed to be reversible, allowing us to explore its properties the context of data generation:
GT (Sanchur))
where oT integrates various task-specific and conditional elements, defined as Cask and Ccondition- Assuming that Φ reversible, we can derive the distribution p through the probability density function (PDF) of Danch (denoted as )DDnn)) the distribution of p can be modeled as follows:
p Dp(oT) =D Ø7
where the PDF of D #7
is expressed as:
f f - f Damneo (x)) det
(x))
0x
which indicates how changes in Danchor influence the distribution p through the transformation function, taking into account the Jacobian determinant of the inverse transformation.
Distribution of Sge: The synthetic data Sgen is the output of the large language model M when prompted with p. typically augmented with noise to introduce variability and improve robustness. Assuming that the output of M follows a specific distribution DM, based on the conditioning on p. represent the distribution of (p)
M PM (-1 p),
The distribution of Sgen then combines the model's output with noise, which is mathematically characterized by the convolution of D (-P)) and D.:
Syen gen (M, p) Das(- D..
where * the convolution operator, integrating the noise distribution D. into the output distribution of the model. This convolution reflects how noise impacts the precision and variability the gener- ated synthetic data, thus affecting the overall utility and effectiveness of the synthetic data in model training.
Through these detailed formulations, we aim to provide more granular understanding of how syn- thetic data is modeled and generated, facilitating better integration and utilization LLM training processes. This deeper insight into the synthetic data generation mechanics enables more targeted and effective training strategies, optimizing the performance of large language models in diverse applications.
D PROOF OF LEMMA 3.1
Proof. Similar like Zheng et al. 2023b), we can further decompose the generalization error into the following three components:
Err( Spm) ) < (TD(SSm)) RDA Sza + RD Sta) - RD (T Sea)
(13)
+ RD (π (m)) Roym (T (78
16

---

Published as a conference paper at ICLR 2025
For the first item in lemma, have:
/ ((T 2) (PD(z) -PaNr (2)) dz
RD( RD (TT (T
||=
|((rSen 22) (PD(z) -Da (z)) dz
(14)
C / (Fz(()-Pox (z)l
X CDrv(D,DMi)
Similarly, for the second item in lemma, we have:
- | ((TSm, 2) (PDr (2) -P (2)) dz
RDi ( (TS
RR ( (TT))m))
/ |(TTS 2) (PDis (2)
- -IP (2)) dz
(15)
$ C / |PD (2) -D (z)
2 CDDvvDDs Dgea).
Together with Eq.
13.. Eq. 14). and Eq. 15) we have:
Err( Saa
- RD( (T )mm) RD ( Sma) + RD (T - RD (m))
+ RD (T (mn) - -Rs (TT (m55m)))
(16)
- CDvv (D.DM) CDvv Dzen) + RD (I R.Sp (x))
CDrrv(DD DMS) + Drv(DM, Dgen))
Ro ( (IS Ri ( (TT
This finishes the proof.
E PROOF OF LEMMA 4.5
Proof. Considering the Markov chain M (p) + Sgen + W, according to the properties of mutual information, we have:
(SSgen) I (p))))).
(17)
Furtherly, the following inequality can be derived:
I I(S gen: W) < I (MM W).
(18)
Building upon equation (18) we can derive the following equations:
[Sgen W) = [[M (p), W) de I (p), W),
(19)
where & is the information loss due to the noise in the data curation process.
Since h(-) and g(-) are deterministic functions which decrease the entropy of random variables, we have:
Hhh(e))) < H(e,). H(g(em)) < H(em).
(20)
Accordingly, the following inequalities can be derived:
T(h(ep), W) = H(h(ep)) H(h(ew))W)
H((e)) (ee,lWW)
(21)
= 1(e,, W).
17

---

Published as a conference paper at ICLR 2025
Similarly, we have:
I(((ex), W) = H(g(ems) H(g(ex)))W)
S {f(ee)) - (e
(22)
= =(eM, W).
This is because the deterministic functions h(.) and g(-) decrease the information content, and make the information a subset of the original random variables.
Then consider the upper bound of (pM),, W) according the result above:
(p) W) = (h((e + g(Em), W)
I [h((ep), W) + [(g(ex), W)
(23)
s I(e,, W) + I(eM,
For further analysis, we consider the following assumption related to the efficiency of the model utilizing the prompt:
Lemma E.1. (Efficiency of the model prompting.) For the model M utilizing the prompt p, with
I, we have:
H(ep) S ((
(24)
Lemma E.I indicates that the entropy of Ep upper bounded by the mutual information between the synthetic factor ep and the model output M (p) by factor of X. In other words, the efficiency of the model utilizing the prompt is reflected in the value of X, which quantifies the extent to which the model can leverage the information contained in the prompt. For cxample, larger À indicates smaller T (cp, (p)), which implies that the (p) contains more information from the prompt p.
Building upon Lemma E.I can further derive the deduction following equation 23
I(M(p), W) = T(ep. W) + I(eM, W)
H(ep) - H(e + T(eas, W)
H(ep H(M())) H(M()) + H(ep) H(e)))H((p)WW) + T(eM, W)
S 5-H(M(W) + Iee (()) - [(ep M())
(25)
+ +Aleep M(p) + H(M(p)) HW) + T[eM,, W)
S -AT + T(EM, W) + H(M())) H(e, W) + (2 1)7(ep: M(p))
< -I- + Bay + H(M)).
Lemma E.2. (Entropy gap upper bound) The difference between the entropy of (p) and ep can be upper bounded by the following inequality:
H(M(p)) H(ep) & H(eM))
(26)
The proof of Lemma E.2 is listed in equation
(27:
H(M(p)) H(e,) H(h(e,) + g(em)) H(ep)
< H(e(p)) + (g(em)) (1pp))
(27)
H(ee)) + ((em))(((H)))))
= H(eM))
Building upon Lemma E.2 we can further deduce the following inequality following equation (5)
I(M(p), W) < -AT + I(eM, W)+ + H(M())) - H(e, + (A - 1)T M(p)
< - + B.yn I(ep. W) H (M) H(e,/W)(()))))())))) (A 1),(e,,
(28)
= - + Bsyn + (p)(p)) - (e,) ( ( 1),(e,,
< - AT + Byn H(EM) + (A ()T M(r)).
Together with equations 19 and 28 we have:
I(S gen: W
[(M(p), W) - 6,
Al + Byn + H(EM) (A (), 8.
(29)
< A + Byn + H ((EM) + +d.mp
18

---

Published as a conference paper at ICLR 2025
where &2. = (A (), M()) de.
This finishes the proof.
F PROOF OF THEOREM 4.10
Proof. Considering the Markov chain h(ep) + Mpp) Sgens we have:
(p) 2 (Sgan).
(30)
In addition, according to the properties of mutual information, have:
[SSchor M (p)) 2 I (h())),
(31)
Building upon the inequalities above, we can derive the following equations:
A = H(M(p)) 7 (h(ep), M())
(Sege)) aSachor (p))
(32)
T(S gen: W) + H genl ( T(S M()).
Based on the assumptions mentioned above, we also have:
anancor W (Snnchor) ( IW
H (Sunctor)) H ( Sancchon W) -Sachor W) + (1 a H)HH( (Q)HH(Snceee (Sachoo W..
(33)
Furthermore, based on the definitions, we have:
I Sanchoo Mp))
H (Sanchorr) (S (M)))
=I (Sanchor W) + (Sancoor (W) (Saneebar M (p))
(34)
[ auchor, W) + EW.p.
By the definition of GGMI, and with equation
33 the following result can be deduced:
GGMI =1 =I W') 1(Sgen W)
=I(Sanchor W) (1 a) (SS.nhoo W) T I(S W)
= =[(See gen: W) + (.Seen W) - I(Saacbonn M (p))
(35)
(Sggen; W) (S gen (W) + SSaacchr M (p))
+ / (SSnchor W) + (1 at) (S | (W) - (Spen W).
Subsequently, together with equations
32 and
further deduce that:
GGMI <AI - 2I(SSen, W) H(S gen (W) + I W)
+ a:) ( W
+ ((Sanchurs Mp))
= AI (S
W) H (S WW)
(36)
+ 2I(Sunchor:
W) + H (Sanhoor WW) + CW.p
AT - 22 gen)
+ H (Syenll
+ 25
- (a + (Sanchll W) + 6W.p-
Finally, together with all the deduce and definition above, we have:
GGMI < Al - (a + (S Sanchor [W) + 2AH + H W) + Ew.ps
(37)
This finishes the proof.
19

---

Published as a conference paper at ICLR 2025
G EXPERIMENTS: EXPLORING BETTER SYNTHETIC DATA IN PRACTICE
To further investigate the process of synthetic data generation in real-world settings, conduct experiments to evaluate the quality of synthetic data produced under different conditions and aim to identify the factors that contribute its effectiveness in enhancing model performance.
The experiments follow the same setup as described in the main text, with the synthetic data Sgen generated from generative model M prompted by p. We utilize standard in-context learning (ICL) framework to determine p using anchor data Sunchors and we then evaluate the performance of the model trained on the synthetic data. Additionally, we estimate key components from our theoretical analysis in the main text, including information gain A/ and the entropy of the synthetic data gSen).
In the remainder of this section, commence by introducing the experimental setup and the eval- uation metrics. We then present the results of the synthetic data generation, focusing on the per- formance of the model trained synthetic data to assess its quality. Furthermore, we estimate the key components from our theoretical analysis and analyze the factors that contribute to the effec- tiveness of synthetic data in improving model performance. Finally, provide brief conclusion and discuss potential principles for generating higher-quality synthetic data in practice.
G.1 EXPERIMENTAL SETUP
We conducted experiments to evaluate the effectiveness of synthetic data generated by generative model M prompted by p in enhancing model performance. Our experimental setup follows the synthetic data utilization process described in the main text, including selecting benchmark dataset, determining prompt p. generating synthetic data Sgen training the model on the synthetic data, and evaluating the trained model.
G.1.1 BENCIMARK DATASET
The benchmark dataset is utilized sample Sanchor- Specifically, we adopt Dolly-15K (Conover et al.] 2023) as our benchmark dataset, which contains 15,000 lines of text data designed for instriction-following tasks. split the benchmark dataset into training and testing sets with ratio of 8:2, and Sancher is sampled from the training set. For each data instance, retain the keys "instruction", " "context" and "response" and combine them using the following template.
G.1.2 DETERMINING PROMPT
Consistent with the methodology described in the main text, we employ standard In-Context Learn- ing (ICL) framework to determine the prompt p. Specifically, p E(Sancber) where is a prede- fined template for the prompt. We follow the settings of Alpaca TTooi al. 2023) and modify the template to better suit the benchmark dataset used in our experiments. The modified template is as follows:
20

---

The language Published as a conference paper at ICLR 2025
You asked to come up with set of 20 diverse task instructions. These task instructions will be given language model and we will evaluate the language model for completing the instructions. Try not to repeat the verb for cach instruction to maximize diversity. The language used for the instruction also should be diverse. For example, you should combine questions with imperative instrucitons. 3. The type of instructions should be diverse. The list should include diverse lypes of tasks like open-ended generation, classification, editing, example, do not ask the assistant create any
Here are the requirements:
model should able complete the instruction. visual or audio output. For another example, do not ask the assistant to wake you up at 5pm or set a reminder because it cannot perform any action. 5. The instructions should be in English. 6. The instructions should be to sentences long. Either imperative sentence You should generate an appropriate input the instruction. The input field should contain specific example provided for instruction. should involve realistic data and should not contain simple placeholders. The input should provide substantial content to make the instruction challenging but should ideally not exceed 100 words. 8. Not all instructions require input. For example, when general information, "what is the highest peak in the world", it
question is permitted.
instruction asks about necssary to provide specific context. In this case, we simply put "noinput" in input field. The output should an appropriate response to the instruction and the input. Make sure the output is less than 100 words.
Your output should consist of 3 parts: instruction, context and reference response. "Instruction" is the task instruction which the language model should complete. "Context" is information related the instruction,
need, you can set as empty. "Reference response" the correct answer to instruction your recommend.
Your output must be in the following json form like: ("instruction": [the instruction generate], "context": [the context you generate]. "reference response": [the reference response you generate]] Here are some examples you should emulate:
{anchor data}
List of 20 tasks:""
We then sample Sanchor from the benchmark dataset and populate the "(anchor data}" placeholders the prompt template with these samples. This completes the process of determining the prompt y.
G.1.3 GENERATING SYNTHETIC DATA SOEN
After determining the prompt p. generate synthetic data Spen by prompting the generative model M with p. In our experiments, we primarily utilize GPT-4o (OpenAl 2024) as the generative model M. Additionally, wo also employ the latest Llama 3.2 models (Meta) 2024) including Llama-3.2-
IB- Instructtt and Llama-3.2- 3.2-3B.. 1-2-3B--Insruu Instructt for comparison experiments.
G. TRAINING ON SYNTHETIC DATA
We fine-tune a GPT-2 (Rafford et al. 2019) model using both the synthetic data Sgen generated by the generative model M and the traming set T of the benchmark dataset. The training procedure follows the standard instruction tuning process, where we fine-tune the model on the synthetic data for fixed 20 epochs.
G.1.5 EVALUATING FINE-TUNED MODEL
We assess the performance of the fine-tuned model the testing set of the benchmark dataset. Following the evaluation procedure of ZZheng et al.] (2023c), we evaluate the model's 's ability by rating the generated responses using LLM. To better align our evaluation with datasets, modify the original evaluation prompt to ensure that the judge LLM compares the output with the ground-truth answer. The evaluation prompt we adopt is as follows:
21

---

Published as a conference paper at ICLR 2025
Synthetic Data Fine-Tuned
Real Data
Base
3-ins
10-ins
20-ins
Fine- Tunee 0.2745
Rating
0.1409
0.1863
0.1965
0.2015
Table 1: Average ratings of the fine-tuned model on the testing set. The ratings were normalized using softmax function. The synthetic data were generated by GPT-4o with varying numbers of instances in In-Context Learning (ICI.) (denoted as æ-ins). The unfine-tuned base model (Base) and the model fine-tuned real data marked gray.
impartial judge and evaluate the quality of the response provided by Al assistant the user question displayed below. You are provided with parts of the text, [Question] the question asked by the user, [Context] is information related to the question, [Reference Answer] is the correct answer for your reference, and Assistant's Answer which is surrounded by [The Start of Assistant's Answer] and [The End of Assistant's Answer] is the answer given by the assistant. Your evaluation should consider factors such the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing short explanation. possible. After providing your explanation, you must rate the Assistant's ' Answer on scale to 10 by strictly following this format: "[[rating]]] for example: "Rating: [[5]]". [Question] {instruction]
Please
as objective as
[Context] {context)
[Reference Answer] {reference response]
(The Start of Assistant's Answer] (generated response) [The End of Assistant's Answer]
We then populate the placeholders "(instruction)". "(context)", "[reference response)", and "[generated response}" in the evaluation prompt with the corresponding text. We adopt Llama- L---eeotroon-P00 -on-roo-n------tuutt------- its output. The final rating is averaged over the testing set to evaluate the performance of the fine- tuned model.
(Wang et al.] 2024) as the judge LLM and extract the ratings from
G.2 SYNTHETIC DATA QUALITY
We assess the quality of synthetic data generated by the generative model M prompted by p in terms of its effectiveness in enhancing model performance. Specifically, we utilize GPT-4o generate synthetic data with varying numbers of instances in ICL, corresponding to different sizes of Sanchor denoted as 3-ins, 10-ins, and 20-ins, respectively. We then fine-tune a GPT-2 model on both the synthetic data and the training set of the benchmark dataset. The performance of the fine- tuned model on the testing set is used as a measure of the quality of the synthetic data. For better presentation, apply softmax function to normalize the ratings. The results shown in Table [
to
The results demonstrate that the synthetic data effectively enhances the performance of the fine- tuned model, with the rating positively correlated with the number of instances in ICI., This finding indicates that appropriately increasing the number of instances in ICL can improve the quality of the synthetic data. This phenomenon may be attributed to the fact that increasing the number of instances in the ICL prompts provides the generative model with richer and more diverse context. This enhanced context allows the model to capture a broader range of patterns present in the anchor data, thereby generating synthetic data with richer content.
G.3 ESTIMATING THEORETICAL COMPONENTS
Building upon the results of synthetic data quality, we further estimate the key components from our theoretical analysis, including information gain A and the entropy of the synthetic data (Sge)) We aim to analyze the factors that contribute to improving the quality of synthetic data.
G.3.1 ESTIMATING INFORMATION GAIN
Given the definition of information gain A/ in Definition is difficult to directly estimate A/ in practice. However, it is possible to estimate T(T, Sgen), the mutual information between the synthetic
22

---

Published as a conference paper at ICLR 2025
Sgen HSIC w/ T (010
3-ins
10-ins
20-ins
7.8703
7.8668
7.8502
Table 2: The HSIC value between the synthetic data gen and training set 'T' for different numbers of instances in ICL setting.
Sen
3-ins
10-ins
20-ins
Semantic Entropy
1.0739
1.0503
1.0005
Table 3: The semantic entropy of the synthetic data Sgen in different numbers for instances in ICL setting.
data Sgea and the training set 7 of the benchmark dataset where Sancbor is sampled. Since the crucial part of prompt p is Sanchor- [T,, Sgea) has a negative correlation with A certain extent.
To measure [(T, Sgen). we follow the setting of existing works Qian et al. 2024 Ma et al. 2020 and utilize HSIC (Gretton et al. 2005) as an estimator. The result shown in Table
It is supurising that more instances doesn't increase the HSIC value, but even lead to lower HSIC value, indicating reduced mutual information between the synthetic data and the training set. This phenomenon suggests that enlarging the sizes of Sancher does not significantly increase the depen- dency between the synthetic data and the training set, and may even enhance the diversity of the synthetic data. This may be attributed to the fact that when LLM with wide range of knowledge is employed as M, leverages its broad understanding to generate synthetic data that less reliant on the specific instances in aachor- As the number of instances in the ICL setup increases, the LLM interprets this as a richer and more varied context, thereby increasing the output diversity instead.
A smaller HSIC value indicates lower mutual information between the synthetic data and the training set, which leads to larger information gain A/. With Theorem 4.7 guarantees tighter upper bound of the generalization error and higher GGMI, which contributes to the quality of synthetic data and increase the generalization capabilities.
(4.7and and Theorem 10 this
G.3.2 ESTIMATING ENTROPY OF SYNTHETIC DATA
As another important component in Theorem 4.10 (Sgen) crutial for determining the value of AH. We use semantic entropy (Farquhar al. 2024) as estimator to measure the entropy of the dataset and estimate the value of H (Sgen. The result is shown in Table 3
The results indicate that the semantic entropy of the synthetic data Spen also negatively correlated with the number of instances in ICL. This suggests that increasing the sizes of Sanchor when utilizing LLM as generative model M can help reduce the entropy of the synthetic data. This reduction in entropy may be attributed to the richer and more varied context provided by larger Sanchors which enables M to generate more accurate and informative synthetic data, thereby increasing the faithfulness of the synthetic data.
smaller semantic entropy indicates a lower entropy of the synthetic data Sgen which leads a larger AH. With Theorem 4.10 this benifts increasing the upper bound of GGMI, and contributes to the generalization capabilities the model trained on the synthetic data.
G.4 ESTIMATING ON DIFFERENT MODEL ARCHITECTURES
To further investigate the impact of different model architectures and parameters on the quality of synthetic data, conduct experiments to evaluate the HSIC value and semantic entropy of the synthetic data Sgen generated by different models. Due to computational resource limitations, we utilized GPT-4o, Llama-3.2-33 --sttuutttt and Llama-3.2- 3.2-1BBIInttruc as the generative model M to generate synthetic data with 3 instances ICL setting. The results are presented in Table
Note that under the prompt determined in the experimental setups, the Llama-3.2- Instrrctct model did not adhere to the format requirements and failed to produce meaningful synthetic data. Conse- quently, the estimators are not available for this model. This observation underscores fundamental
23

---

Published as a conference paper at ICLR 2025
GPT-40
Lamma-32-3B-Instruue
Llama-3.2-1h I--ssruuttt
HSIC w/ T (x10-9) Semantic Entropy
7.8703
11.4306
1.0739
2.9427
Table 4: The HSIC value and semantic entropy of the synthetic data Sgen generated using different model architectures. All the synthetic data are generated with instances in ICL setting. Note that the Llama-3.22 3.2-11 In-Isructt model did not adhere to the format requirements and thus failed to produce meaningful synthetic data.
premise that the generative model M must possess sufficient instruction-followie capabilities to generate synthetic data that be effectively utilized enhance model performance.
On the other hand, although 1aama-3.2-3B-Isttie insufficient for fine-tuning GPT-2, and the HSIC value and semantic entropy were significantly higher than those GPT-4o. This may be attributed to the smaller model size of Llama-3.2 ---3BB- Instruct compared GPT-4o, resulting in diminished capacity generate synthetic data that is both faithful to and diverse from the anchor data. For instance, we provide some examples of the synthetic data generated by Llama-3.2-3 1lama-3.2-3B-mmm 2--tuuutttt the following as case study:
produced usable synthetic data, its quality was
Instructions generated by Llama-3.. Instruct::
"instruction": "Explain the concept of blockchain in simple terms."
"instruction": "Explain the concept of artificial intelligence in simple terms."
"instruction": "Explain the concept of climate change simple terms."
"instruction": "Identify the type of music genre: classical or jazz: 'Moonlight Sonata' or "Take Five"" "instruction": "Identify the type of literary device used in the following sentence: "The eyes that fixed upon her were like two bright stars in the night sky."
"instruction": "Identify the type music instrument: string or woodwind: 'Violin' "Flute""
"instruction": "Write short story about character who discovers hidden world within their reflec- tion." "instruction": "Write review of the movie "The Shawshank Redemption"."
"instruction": "Write poem about the beauty of nature."
The examples demonstrate that the synthetic data generated by Llama-3.2 3.2ma--.--3B----- 2-3t--nsttucct is highly ho- mogencous, even within single generation cycle. Morcover, it is highly dependent the specific instances the anchor data, leading to a higher HSIC value. Furthermore, although the synthetic data lacks diversity in form, the semantic entropy remains high. This indicates that the generated synthetic data lacks sufficient faithfulness. Collectively, these factors contribute to the poor quality of the synthetic data produced by Llama-3.2-33 I--tuunnt
G.5 CONCLUSION
Building upon the experiments, we can derive some brief conclusions about how to guarantee the synthetic data quality and estimate the key factors in real-world LLM practice.
The quality of synthetic data is mainly reflected in two aspects: diversity and faithfulness. Diversity makes the synthetic data contain richer contents and thus increase the information gain AT. With our theoretical analysis, this will benifit the generalization ability of the model post-trained synthetic data. Faithfulness makes the synthetic data semantically continuous, and thus decrease the entropy of the synthetic data Sgen which also strenghten the generalization capabilities.
practice, the diversity and the faithfulness can estimated by HSIC value and the semantic entropy, respectively, demonstrated in the experimental settings of this section. It is also important to highlight that employing a generative model with stronger instruction -owouuiioooooooiinng capabilities and more diverse knowledge can enhance the quality of synthetic data in both aspects.
24


Reviews:
[{"review_id": "MZsFEyZ9Zr", "reviewer": "Reviewer_qaT5", "summary": "The paper introduces the framework of understanding the LLM synthetic data generation process, by incorporating the reverse-bottleneck perspective. Using a reverse-bottleneck perspective, the authors model the generation of synthetic data and propose an information-theoretic metric called Generalization Gain via Mutual Information (GGMI) to assess the generalization performance of post-trained models. It also reports the information gain and generalization bound from previous literature.", "questions": "See above", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is easy to follow and the figure is well represented.\n\n2. The equation derived from previous literature, such as Lemma 3.1 and 4.4 is clear. \n\n3. The paper tries to inject the LLM synthetic generation perspective using previously derived framework, showing a potential direction to understanding the synthetic data.", "weaknesses": "1. The paper lacks the experimental support, which cannot convince the readers of their assumption and derivations.\n\n2. The paper is not organized in a good shape for guiding the readers for understanding their contributions. Instead, it enumerates the equations of generalization errors and gains, but lacking sufficient experimental results to support this.\n\n3. Some derivations may be challenging for readers unfamiliar with advanced information theory, which may requires providing backgrounds about this. Since the authors did not conduct real-world experiments, it is unsure whether this can generalize to real-world settings. The framework assumes ideal conditions for synthetic data and LLMs, which may not generalize well to complex real-world scenarios with noisy data or biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPDgivgRuA", "reviewer": "Reviewer_sWKh", "summary": "The paper does a good job of establishing a theoretical rationale behind using synthetic data for LLM post-training processes like SFT. The modeling of synthetic data, focusing on distributional aspects is elaborate and properly justified as well. It establishes an intuitive information-theoretic based upper bound on the generalization error for an under-aligned LLM fine-tuned on synthetic data, and also delineates how this upper bound compares to the one when the LLM is tuned on just the real world samples (anchor data) - showcasing how the former proves to be a more stringent upper bound, leading to better generalization capabilities. Definitely helps bridge a gaping gap in LLM research!", "questions": "Here are suggested experiments that could help validate the paper's hypotheses and theoretical frameworks:\n1. LLM-based Validation Experiments :\n     - ⁠Compare different sizes of anchor data and their corresponding synthetic data generation\n     - ⁠Measure the relationship between model size and synthetic data quality\n     - ⁠Test various prompting strategies and their impact on synthetic data diversity\n     - Track information gain across different LLM architectures \n2. Practical Application Questions:\n    - How can practitioners use your theoretical bounds to improve their synthetic data generation process?\n    - What specific guidance would you give for prompt engineering based on your theoretical findings?\n3. Validation Questions:\n    - How would your bounds change with different LLM architectures or sizes?\n    -  ⁠What metrics would you recommend for measuring the quality of synthetic data in practice?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical modeling of synthetic data and how it interacts with the output distribution of the LLM is intuitive and complete. Great job there!\n2. The math behind generalization error bounds is solid.\n3. The connection to the classical information bottleneck theory is novel and very intuitive, clearly justifies why synthetic data helps better performance on downstream task generalization.", "weaknesses": "1. The paper presents good theoretical foundations of \"why\" it is important to use synthetic data in post-training alignment and \"how\" it aids better generalization capabilities of the model. But it fails to address some really important \"hows\" that it promises in the Introduction section :\n   - How does this study help in developing tailored synthetic data generation - that more effectively address specific gaps in training data, thereby enhancing the overall performance and generalization capabilities of large language models\n   - How does this translate to better post-training practices involving synthetic datasets, from a practical perspective.\n2. The assumption that the transformation function φT is reversible (Section 3.2) is unclear for real LLM prompting scenarios, as task-to-prompt relationships are often many-to-many.A better delineation of this point would be helpful.\n3. ⁠The relationship between \"synthetic factors\" and actual LLM generation processes is not well defined.\n4. The connection between the theoretical \"information gain\" concept and practical improvements in LLM performance is not clearly established. \n5. I think its very crucial to address how the \"diversity\" and \"quality\" in synthetic data samples are a big part of why task generalization actually works when models are post trained on synthetic data. These should be adequately quantified and incorporated in the upper bound equations as well, for a more complete picture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGailkzL0y", "reviewer": "Reviewer_Re6N", "summary": "This work provides a theoretical framework to analyse the nature and impact of synthetic data on the post-training process of LLMs. This work also proposes a \"reverse-bottleneck\" framework to visualize information flow in the synthetic generation process. On the basis of these theoretical principles, this work proceeds to mathematically derive useful upper bounds for model generalization when trained with synthetic data, in order to better quantify performance expectations.", "questions": "- How does task difficulty affect generalization capability and the derived upper bounds? Are there any factors in the documented theoretical expressions that model or are associated with task complexity?\n- Lemma 4.8: η' and W' are not defined prior to their usage in the expression; for clarity, it would be useful to define them as other variables have been.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This work has a sound, well-researched theoretical basis\n- The provided mathematical proofs are comprehensive and well-structured.\n- The experimental setup is well-described and the theory is supported by some experimental results.", "weaknesses": "- This work could benefit from additional explanation of how diversity of generated data influences generalization capabilities, as well as a clearer explanation of the intuition behind how ∆I (the information gain derived from the model M) is associated with diversity (lines 483-485) and not just more unique but similar data points, possibly with examples.\n-  The dimension of data on which these experiments are performed is 2; however, real-world data is often more complex and higher-dimensional. It would be illustrative to conduct the experiments documented in this paper on data of higher dimensions to reinforce that the theory holds for datasets of varying dimensionality.\n- Additionally, given that this analysis is aimed at studying the behaviour of LLMs, the theoretical proofs presented in this work would benefit from conducting the same experiments either with LLMs or other non-GMM models to corroborate the theoretical inferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xXEk4Uyt8z", "reviewer": "Reviewer_7KR4", "summary": "This paper explores the use of synthetic data in post-training tasks for large language models. It presents a theoretical framework to analyze the effects of synthetic data on model generalization, focusing on the reverse-bottleneck effect. The authors introduce key metrics like Generalization Gain via Mutual Information (GGMI) and propose a detailed modeling of the synthetic data generation process. They also provide upper bounds on generalization error based on information theory.", "questions": "1. How would synthetic data that has not been processed by an LLM impact the proposed theory? For instance, data generated through specific rules, such as template-based concatenation or converting game-playing sequences into text.\n2. Does this theoretical analysis still hold for non-text, multimodal data? Would the same principles apply, or are there limitations in extending the theory to such data types?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper provides a detailed theoretical analysis of the introduction of synthetic data, clearly explaining the information gain that synthetic data can bring.\n2. The authors present thorough and well-supported mathematical proofs to substantiate their claims, offering a solid foundation for their arguments.\n3. The reverse-bottleneck theory mathematically captures the essence of synthetic data's impact, offering valuable insights and guidance for the development of synthetic data methodologies.", "weaknesses": "1. The paper primarily focuses on theoretical analysis with minimal experimental validation. It is recommended to include empirical experiments to support the proposed theory.\n2. The use of GMM models and non-real numerical data raises questions about whether the findings can effectively reflect real-world LLM outcomes.\n3. The theory mainly emphasizes the performance benefits of introducing synthetic data, but lacks analysis in other areas. For example, it could explore potential drawbacks of over-reliance on synthetic data in LLMs, as highlighted in studies like https://www.nature.com/articles/s41586-024-07566-y.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)