How to Synthesize Text Data without Model Collapse?
Authors: Xuekai Zhu
Daixuan Cheng
Hengli Li
Kaiyan Zhang
Ermo Hua
Xingtai Lv
Ning Ding
Zhouhan Lin
Zilong Zheng
Bowen Zhou
Venue: ICML 2025 poster (2025)
Keywords: synthetic data
model collapse
Primary area: deep_learning

Abstract: Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the proliferation of AI models, synthetic data will fundamentally reshape the web data ecosystem. Future GPT-$\{n\}$ models will inevitably be trained on a blend of synthetic and human-produced data. In this paper, we focus on two questions: what is the impact of synthetic data on language model training, and how to synthesize data without model collapse? We first pre-train language models across different proportions of synthetic data, revealing a negative correlation between the proportion of synthetic data and model performance. We further conduct statistical analysis on synthetic data to uncover distributional shift phenomenon and over-concentration of n-gram features. Inspired by the above findings, we propose token editing on human-produced data to obtain semi-synthetic data. As a proof of concept, we theoretically demonstrate that token-level editing can prevent model collapse, as the test error is constrained by a finite upper bound. We conduct extensive experiments on pre-training from scratch, continual pre-training, and supervised fine-tuning. The results validate our theoretical proof that token-level editing improves data quality and enhances model performance.

How to Synthesize Text Data without Model Collapse?
Xuekai Zhu 12 Daixuan Cheng² Hengli Li Kaiyan Zhang* + Ermo Hua Xingtai Ly
Ning Ding
Zhouhan Lint Zilong Zheng?? 2 Bowen Zhou 45
Abstract
lent in research and industry, synthetic data will proliferate throughout the web data ecosystem. Consequently, future training of GPT-(n) on a mixture of synthetic and human- produced data will be inevitable. Thus, model collapse is a critical concern that must be considered when training models on synthetic data.
Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the pro- liferation of AI models, synthetic data will funda- mentally reshape the web data ecosystem. Future GPT-(n) models will inevitably be trained a blend of synthetic and human arodun-r data. In this paper, we focus on two questions: what is the impact of synthetic data on language model train- ing. and how to synthesize data without model col- lapse? We first pre-train language models across different proportions of synthetic data, revealing negative correlation between the proportion of synthetic data and model performance. We fur- ther conduct statistical analysis on synthetic data to uncover distributional shift phenomenon and over cvercooncettattoon of n-gram features. Inspired by the above findings, we propose token editing on human- human-produced data obtain eynt-synthetic data. As proof of concept, we theoretically demonstrate that token-level editing can prevent model collapse, as the test error is constrained by a finite upper bound. We conduct extensive ex- periments on pre-training from scratch, continual pre-training, and supervised fine-tuning. The re- sults validate our theoretical proof that token-level editing improves model performance.
Model collapse refers to degenerative process in which the output data of learned generative models contaminates the training sets of subsequent generations. As shown in Fig- ure I, iterative training coupled with data synthesis induces a progressive accumulation of test errors (Shumailov et al., 2024; Dohmatob et al., 2024a). Consequently, generative models increasingly overfit synthetic data distributions, failing capture the complexity
cumaan-roduued data. Through successive iterations in Figure these distortions accumulate, finally undermining the model's 's capacity.
Recent studies focus on two aspects. First, theoretical foun- dations of model collapse. Shumailov et al. (2024) and Dohmatob et (2024a) identify the model collapse phe- nomenon and formalize theoretical framework based on linear regression. Gerstgrasser et al. (2024) demonstrate that if synthetic data is accumulated while retaining the ini- tial real data, the test error will be bounded, thus breaking model collapse. Dohmatob et al. (2024c;b) indicate that missing long tails of synthetic data lead to scaling law cut- off. Second, practical implementations on synthetic datasets by diverse prompting. Synthetic datasets (Trinh et al., 2024; Zhang al., 2024) have been proven boost the capabil- ities of language models. Cheng et al. (2024a;b); Maini et al. (2024) rephrase text into more formal styles, thereby improving the data quality. There are still two key questions that require further investigation: (Q1) What is the impact of synthetic data on language model training? (Q2) How can data be synthesized without causing model collapse?
1. Introduction
As generative artificial intelligence (AT) (Rombach al., 2021; Achiam et al., 2023) becomes increasingly preva-
'LUMIA Lab, Shanghai Jiao Tong University SSttte Key Laboratory of General Artificial Intelligenee BIGAI 'Institute for Artificial Intelligence, Peking University 'Department of Electronic Engineering, Tsinghua University 'Shanghai Artifi- Zhouhan Lin <lin.zhouhan@ @ gmail.com>. com>, Zilong Zheng <zlzheng bigai.ai>,, Bowen Zhou <zhoubowen tsinghua.cdu.com edu.en..
In this paper, we address the first question by training lan- guage models on varying mixtures of synthetic and human- produced data, demonstrating non-iterative model collapse. Unlike the original model collapse setting which iteratively trains on self- ref-reeerated data, we directly mix synthetic and produced data to create training datasets with differ- ent mixing ratios. The results show a negative correlation between performance and the proportion of synthetic data.
cial Intelligence Laboratory. Correspondence
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

How to Synthesize Text Data without Model Collapse?
o²d T-d-1
Model Collapse Setting - Eest
x
Trained Model
Source Real Data: Datas
Datda
fo
Datas
f1
Datan
fn
Training
Synthesizing
Training
Synthetic Data: Dataoo
Iterations (2.... R)
o'd T-d-1
Test Error Exest
Token-Level Editing --> E Best S
2 Avoiding Model Collapse
Data Size
Input
Datag
ill. fo
M,Data, + (1 Ma)
fn
Noise Scalar
Training
Editing
Training
Editing Operation Matrix
Figure 1. Model collapse of synthetic data. ① The model continuously trains on its previously generated data, leading to gradual decline in model performance, i.e. model collapse. Starting from real data Dalas, the test error Ecest increases as fo undergoes iterative training on synthetic data Dataso. @ ToEdit (ours). we use trained model for token-level editing rather than purely synthesizing data, Leveraging fo and operation matrix M, edit the data, the test error is constrained within fixed upper bound. Therefore, we preserve the distribution coverage avoid model collapse.
Subsequent statistical analysis on distributions and features indicates coverage narrowing-synthoe data covers only a small portion of the human-produced data distribution and over-concentration of synthetic n- n-gram features. Based on the above findings, proposing token-level editing (ToEdit), resamples and re- places data points with relatively high model confidence. As illustrated in Figure 1, ToEdit preserves distribution cov- erage and theoretically constrains test error within a fixed upper bound. Extensive experiments across pre-training from scratch, continual pre-training, and supervised fine- tuning confirm its positive impact on model performance.
earlier versions of themselves can result in performance degradation, ultimately rendering the AI model completely useless. This process can be formulated as follows:
address the second question by
o²d
Ecest
T d
This indicates that the error will continuously increase with the number of iterations n. The detailed theoretical nota- tion is provided in $ 4.2. Dohmatob et al. (2024c) further point out that synthetic data also contribute to a truncation of the scaling law. Gerstgrasser et al. (2024); Seddik et al. (2024) further adjust the data iteration setting to data accu- mulation or real data mixing. They demonstrate that data accumulation can prevent model collapse. Inspired by the above work, we further explore the impact of synthetic data in pre- pre-training and analyze its differences from real data. Building on our findings, we propose token editing as a method to prevent model collapse during data synthesis. Further comparisons are in Appendix B and D.
Contributions. We summarize the key contributions of this work as follows?:
We demonstrate non-iterative model collapse by pre- training language models on a mixture of synthetic and human-produced data ($ 3.1): directly mixing pure syn- thetic data, without iterative training, leads to performance degradation. Furthermore, we perform a distributional statistical analysis, revealing that synthetic data leads to coverage narrowing and over-concentration of n-gram fea- tures. Even subsequent data selection struggles to correct the distribution ($ 3.2).
3. Non-Iterative Model Collapse
We propose token-level editing with a theoretical proof to prevent model collapse ($ 4) and validate effectiveness through experiments spanning pre-training from scratch, continual pre pre-training, and supervised fine-tuning of lan- guage models ($ 5).
Prior studies (Shumailov et al., 2024: Dohmatob et al., 2024a) investigate the curse of recursion, where iterative training on self-generated data leads to a degenerative pro- cess known as iterative model collapse. However, we often face direct data mixing of numan-produced and synthetic data, and pre-training from scratch. We attempt to ana- lyze model collapse in this more general scenario, called non-iterative model collapse. Specifically, we conduct pre- training on synthetic data mixtures and explore the reasons behind non-iterative model collapse through data distribu- tion and characteristics. A more detailed comparison of iterative and non-iterative settings Appendix G.1.
2. Background
Shumailov et al. (2024); Dohmatob et al. (2024a;c) demon- strate AI models trained recursively on data generated by
Work done during internship BIGAL. "odee repository available at https: httpg/gttbbb.oooo Xuekai- Soekki-ght/tcaae 2ht/toedit.
2

---

How to Synthesize Text Data without Model Collapse?
Table 1. SSuhdomain PPL. evaluation results for GPT-2 Small (124M) pre-trained data mixture. The PPT. increases as the proportion of
synthetic data grows, providing further confirmation of Figure 2.
ArXiv
Books2
Books3
Math
Earon
EuroParl
FreeLLaw
Gitllub
PG-19
HackerNews
NIII
Avg
Human data
22.26
25.39
22.87
10.84
23,50
30.73
12.04
4.15
16.88
32.54
23.53
20.99
25%
Synthetic Data
21.86
26.32
23.87
11.05
24.85
35.02
12.84
17.99
33.80
23.76
22.06
50% Synthetic Data
22.50
28.01
25.75
10.84
26.56
41,99
14.02
4.67
19.70
36.
24.61
23.48
75%
Synthetic Data
24.35
31.19
28.98
11.81
30.30
56.32
16.03
5.30
22.75
40.44
26.19
27.60
Synthrii Data
35.60
43.72
47.72
17.25
66.97
129.75
29.62
12.00
87.95
39.48
OpenSubts
OWT2
Phil
Pile-CC
PubMed-A
PubMed-C
StackEx
Ubantu
USPTO
Wilkipdiaa
Youtube
Avg
Human data
28.08
25.77
33.56
26.78
18.97
15.49
10.81
20.86
19.32
24.31
21.54
22.59
25%
Synthetic Data
29.25
26.94
34.63
27.83
19.55
15.38
11.03
22.32
19.58
25.88
22.63
23.91
50%
Synthetic Data
31,00
28.76
37.48
29.36
20.51
15,89
11.54
23.53
20.51
27.57
24.91
25.09
75%
Synthetic Data
34,18
32.04
42.39
32.17
22.33
16,92
12.55
26.54
22.21
30.68
28.98
28 64
Synthetic Data
57.83
53.94
78.18
54.69
34.82
23.87
20.47
51.78
37.24
46.12
65.49
47.87
Wikitext-103
Falcon
Redpajama
o4-en
122
140
100
Humur Data
Hunur Data
Human Data
100
Haman Data
39%
25%
53% Sprthast
199 Seithets
15% Swntharic
Synthesi
53%
75% 5o ebc
00
75% Syrt sits
06
Thkens km))
Toanss Bonel
Tolers
Bsons 18880
Figure 2. Non-iterative model collapse. Training language models from scratch on Al-synthesized data or mixture of human and synthetic data leads to performance degradation. This degradation is negatively correlated with the proportion of synthetic data used in training. Setting: We pre-train GPT-2 Small (124M) on human data (Dolma (Soldaini et al., 2024)) and synthetic data (Cosmopedia (Ben Allal et al., 2024)) and evaluate the PPL. on the Paloma benchmark (Magnusson et al., 2023). Training loss in Figure 7. Further validations on 22 subdomains and general downstream tasks are presented in Table 1 and Table respectively.
3.1. Pre-training on Data Mixture
Table 1, respectively. These results demonstrate that PPL on real-world validation sets increases as the proportion of synthetic data grows, indicating degraded model perfor- mance. When training from scratch, synthetic data does not benefit the model and may even hinder its learning pro- cess. However, incorporating numan-producad data into the training mixture mitigates model collapse to some ex- tent. Further results on general downstream tasks in Table 9 and 11 also corroborate the above findings. The overall trend shows decline as the proportion of synthetic data increases, with models trained on purely synthetic data per- forming the worst. Compared to previous research it- erative model collapse (Shumailov et al., 2024; Dohmatob et al., 2024a;c), the non-iterative damage caused by syn- thetic data is more concerning and directly relevant to the training of gexeraenent language models.
In this section, we investigate the impact of synthetic data on pre-training. Compared with studies on SFT and RLHF, we examine synthetic data integration in more fundamental stage of the language model.
Setup We pre-train GPT-2 (Radford et al., 2019) and OLMo (Groeneveld et al.. 2024) from scratch, using data mixtures containing 50B tokens each. We define the mixing ratio between human-produced and synthetic data as a, where 0 S a < 1. The total amount of train- ing data Desaa is a combination of human-produced data Dhuman and synthetic data Daymbeticc represented by the formula: Drotal aDDuman + (1 syythetie We use Dolma (Soldaini et al., 2024) as source human- produced data. We use Cosmopedia (Ben Allal et al., 2024) as the source synthetic data, which is distilled from (icral--x7B-Inn ruct-vO 1 (Jiang et al., 2024). To ensure rigorous validation and prevent data leakage, we construct three-tier evaluation: (1) the Paloma bench- mark (Magnusson et al., 2023), including carefully de- contaminated test sets for Dolma; (2) comprehensive PPL. evaluation across 22 subdomains from the Pile (Gao et al., 2020b); and (3) seven general downstream tasks as outlined in (Maini et al., 2024).
3.2. Why Does Synthetic Data Fail in Pre-training?
We conduct three statistical analyses: (1) sample-level distribution, (2) feature-based overlap, and (3) distribution- reference data selection. The experimental results reveal that, compared human-produced data, synthetic data lacks the long-tail tail samples and suffers from coverage narrowing. The limited diversity and concentrated features in synthetic data make using numan-producee data as a reference to select synthetic data particularly challenging.
Finding I: Incorporating synthetic data harms the lan- guage models pre-training. result of Paloma bench- mark and 22 subdomains are presented in Figure 2 and
Setup We conduct statistical and feature-based analyses to explore why synthetic data fails in pre-training. (1) We lever-
3

---

How to Synthesize Text Data without Model Collapse?
110
3.0
Dolma
Cosmopedia
2.5
2.0
Count
Count
1.5
3
1.0
2
1
0.5
0.0
20
40
60
80
100
2
6 8
10
12
14
A. Human Data PPL [0, 100+]
B. Synthetic Data PPL [0, 14]
Figure 3. PPL. distribution of human and synthetic data estimated by Llama-3-8B. The synthetic data lacks the long tail of the human- produced data and is also concentrated within the first 25% of the human-produced data distribution. A. Distribution of numan-produced data is sharp with long tail, spanning a wide range from 0 to over 100. B. The values are concentrated within a much narrower range. mostly between and 14. The same trend estimated by StableLM-3B demonstrated in Figure 10.
age a prior distribution to estimate the human pronan-roduued and synthetic data. We use Llama-3-8B (AI @ Meta, 2024) and StableLM Zeephrr-3 (Bellagente et al., 2024). Different priors consistently yield the same results. (2) We analyze the n-gram features of human-produced and synthetic data from a feature- based perspective, such as n-gram response values. (3) Based on the features of human-produced data, we apply importance sampling (Xie et al., 2023) to filter syn- thetic data that closely aligns with human- producdd features. More details of importance sampling are in G.5.
we hash uni-gram and bi-gram features into 10,000 hash buckets. As illustrated in Figure 11, human-produced data displays noticeably broader response range, while syn- thetic data features are concentrated in a few specific buck- ets. This indirectly supports our earlier observation of fea- ture coer-ooncettattoon We then expanded the hash bucket range to 1,000 x 20,000 buckets and used a locality-sensitive hashing method to differentiate the features more precisely. The results remain consistent. As shown in Figure 14, most response values are near zero. Distinguishing features in synthetic data remains challenging.
Finding II: Synthetic data distribution not only lacks long tails but also exhibits significant coverage narrow- ing. Figure 3 illustrate that the PPL of synthetic data is confined to the lower 25% of the human-produced data, failing to capture the full range and complexity of human- produced data distributions. Specifically, as illustrated in human-produced Figure 3A, human- produced data exhibit wide distribution in the range [1, 100+], characterized by a sharp peak and pronounced long tail. In contrast, as shown in Figure 3B, the synthetic data is confined to narrower range of [o, 14], displaying a smoother distribution. Additional results of StabLM are shown in Figure 10. While the absolute PPL ranges estimated by different models may vary, the relative shapes and proportional ranges of these two distributions remain consistent. This phenomenon demonstrate that when scaling up to larger synthetic datasets, there will be no- table absence of the long tail, leading to severe coverage narrowing. This limited coverage reduces the generalization ability and contribute to model collapse.
Finding IV: Distribution shifting cannot be mitigated through data selection. Inspired by recent data selection works (Xie et al., 2023; Albalak et al., 2024), we try to human-producce produced data features as a reference distribution for selecting synthetic samples. We apply im- portance sampling from DSIR (Xie al., 2023) to filter synthetic data. As illustrated in Figure 4A, the training re- sults of selected synthetic samples still fluctuates around the original performance of the synthetic data, indicating that even biased sampling cannot correct the distributional shift. As shown in Figure 4B, the sampled data still fails to align with numan-producee data in the embedding space, even at the boundary regions of the synthetic data.
leverage the human
3.3. Proposed Strategy
Following these lessons so far, due to the coverage and fea- ture over-concentration properties of synthetic data, the best approach is rely entirely on human-produced data and avoid including synthetic data. However, we are still won- dering how synthetic data can be used to enhance human- produced data. This leads to a general guideline for syn- thetic data: relying solely on synthetic data leads to model collapse, so preserving the primary human-pree pronuced data distribution is essential, In that case, we propose teken-level
Finding III: Synthetic data over-concentrates N-gram features. Based on the above distribution estimate, we further analyze why synthetic data fails at the feature level. Figure 12 and 13 demonstrate that synthetic data exhibits higher frequencies of certain bi-grams compared to human- produced data. To further examine feature-level differences,
4

---

How to Synthesize Text Data without Model Collapse?
Stathetic Dara Ratio
80
1.700
5%
29%
60
1300
90%
J00S
USII (1M)
1200
20
PPL
1000
-20
-40
250
-80
Synthetic dara josemopacisi
Wiittex
Red Iejma
Paloon
M&P2 W
M2D2-8
60
PPL Results for OLMo-237M
B. Embedding Visualization using t-SNE
Figure 4. A. Pre-training results for selected synthetic data and other data mixtures on OLMo-237M. B. Embedding visualization between hummanproduued, synthetic, and DSIR-selected data using
1.0
This conditional probability formulation outlines the gen- eration of synthetic data conditioned on the given token sequence. Then the synthetic data is used to train models.
0.8
Count
0.6
Inspired by previous studies of data selection (Mindermann et al., 2022; Ankner et al., 2024; Lin et al., 2024), prior distri- butions can serve as pointers indicating useless or learnable samples. In this case, we use pre-trained language model to infer the pre-training corpus. As illustrated in Figure even a model pre-trained on trillions of tokens can fit the pre- tre-training corpus perfectly. Specifically, 75% is under 0.6 probability. The tokens with both high and low probabilities are the most concentrated, suggesting the potential for data filtering. We leverage this U-shape distribution as a pointer to resample tokens. Specifically, we use a language model as prior distribution compute each token's 's conditional probability P(-lx) the probability exceeds certain thresh- old P(-xx) p, it indicates that the token is easy leam, and we proceed with resampling that point. The filtering potential of the U-shaped distribution is discussed in $ G.2.
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Probability
Figure 5. U-shape token probability distribution of Dolma-sampled V6 estimated by Qwen-0.5B- ISB-Instruct (qwe, 2024).
editing, which leverages a prior distribution to adjust the data. Our method can maintain the source distribution while improving the source data, called semi-synthetic data.
4. Token-Level Editing
We introduce token-level editing as a method for generating semi-synthetic data. Furthermore, we present a theoretical analysis and proof demonstrating that the test squared error of our method has finite upper bound, regardless of the number of iterations. This ensures prevention of model collapse while enhancing performance.
Token-level Editing doesn't generate the entire sequence but leverages conditional probability P(z; x Ii,...,;; 2;-1) to revise the input sequence. In this way, we can avoid using purely synthetic data while modifying the dataset to preserve long-tail features of human cunaun-podduced data, aiming to obtain higher-quality semi-synthetic data. Token-level Editing can be formulated as follows:
4.1. Method
We formulate data synthesis follows: assuming P is prior distribution, given sequence of tokens .e....,,,.), the full synthetic data is y = (y,...., Vn). The synthesis process is derived as:
Dir if P(æs I1,...,,
x4-1) <
1 =
(2)
Ti. if P(ay $1...., ri 1) 2 p
Where of is the final token in the edited sequence. ã; is a token resampled from prior distribution. We can adjust the threshold p to balance retaining the structure of human- produced data while avoiding overfitting to synthetic data.
12
P(y ... Un x) = II P(y. 1 ....
,..,, Ve.x
5

---

How to Synthesize Text Data without Model Collapse?
4.2. Theoretical Analysis
where Y is the data after editing in the n genera- tion, and Yn is the synthetic data from the n-th generation. This process can be described as: firstly, synthesizing la- bels for all inputs; secondly, the M matrix determining which data is edited and which is retained. For a matrix A with full column rank, its Moore-Penrose pseudo-inverse is A+ = A)- The noise terms Ei, Ez,..., En are independent of each other and the covariatessfeatures. Since X has full column rank, X, retains this property for all n 2 1.
To gain deeper mathematical insights, we utilize an analyt- ical framework of the linear model and adopt notations in prior research (Mobahi et al., 2020; Dohmatob et al., 2024a; Gerstgrasser et al., 2024). This theoretical framework pri- marily considers a linear model that iteratively trains on its own generated data, similar to pipelines like self-play and self-distillation, but without complex constraints. The process involves training continuously on the data generated by the previous generation of the model. (Dohmatob al., 2024a) point out that with iterative training, test enrors accu- mulate progressively, eventually leading to model collapse. Building on this theoretical framework, we integrate our proposed token editing method and analyze whether our method can prevent model collapse.
Test Error Model collapse is ultimately reflected through test error. Following previous work, we adopt the stan- dard test error definition as presented in (Gerstgrasser et al., 2024). For any linear estimator tú derived from the training data, we evaluate the test error using the standard method:
Notation and Preliminaries For given distribution os, the data (x. y) ~ a on Rd R. where x is drawn from multivariate normal distribution e is an independent noise term sampled from ((0 o²), and the label y is given by the linear model y = x = +
(wns(w) def E [(ehaaw Meeas) )²] 0² = =B[lw w [13).
(0, E),
(4)
where the expectation is computed with respect to the train- ing data, while the test pair (Trest, Mest) is sampled indepen- dently from ps .²2 of the training set.
Iterative Data Editing Process We utilize the model ob- tained from the previous round of training to make lim- ited number of modifications. Specifically,, we resam- ple and replace data points with relatively high confi- dence. The editing operations are defined by the matri- ces {M1, M,, M, The iterative data synthesis and model-fitting process is formalized as follows:
4.3. Test Error Under Data Editing
Our goal is to derive an analytical expression for the test error of the n-th model in the data editing setting. As indi- cated by the test error in Eq. 4, this process involves two main steps: (1) establishing the relationship between the fitted linear parameters wn, and the true parameters w", and (2) simplifying the test error expression. We begin by for- mulating the relationship between Wn, and w². Proofs are detailed in Appendix A.
PE .02 + Po + ..... -...-
oz,
where n is the number of iterations. The detailed process of data editing and iterations is described as follows:
For n = 1, we begin by initializing the covariates/features as X, = X. The target values are defined as Vi = 9- = Xw" + E1, where Ej ~ NN0, o2 Ir)). The linear model is then fitted by solving for w = Xty. To proceed to the next iteration, we resample the data, obtaining Y/2 Xú0 + Ez, with E, ~ NN
Theorem 1. In the data editing setting, Vn 2 1, the fitted linear parameters Wn+1 can be derived as:
W. = w + (X xx
E +
M.E.-1
(5)
where, w" is the true parameter, X is the original design matrix, E, is the extra noise added at the i'th iteration, and M, is an idempotent diagonal matrix, defining the editing operation.
For n 2 2, the input covariates/feature remain as XI = X, while the target values are updated using the edited targets, following the equation V Mn-1 P 11-- The linear model is then fitted by computing vi, XjY.. Finally, the data is resampled for the next iteration, yiclding Yz+1 iú, + E where Ex+1 ~ o0)
Theorem 2. Consider an n + 1 fold data editing process with T d 2 samples per iteration and isotropic features ( def Ia), the test error for the ridgeless linear model vn,. learned on the edited data up to iteration n. + is bounded by:
The matrix M,, is a diagonal matrix, where some diagonal elements are 1, while others are O. The multiplication by M can be interpreted as an operation that selectively mod- ifies certain data points (those corresponding to Is) while retaining others (those corresponding to Os). Then, the data editing process can be formulated as follows:
20
(ie4(( $ T - d-1
(6)
Furthermore, assuming the editing operation satisfies
Y.T Mn-18, + (1 - 1)
(3)
|| = [|Ma-1lln with n E (0, 1), the test error can be
6

---

How to Synthesize Text Data without Model Collapse?
Table 2. General performance of the pre-trained base models. PT indicates we pre-train OLMo- 1B from scratch.
PIQA
BoolQ
OBQA
ARC-c
ARC-e
HellaSwag
SIQA
Winogrande
Avg
OLMo-IB (PT)
53.97
38.26
12.20
17.23
28.36
26.02
34.80
51.14 52,49
32.75
ToEdit
54.13
38.65
12.80
18.43
27.48
25.94
34.95
33.11
Table Performance on domain-specific tasks for continual pre-training models. CPT indicates continual pre-training. denotes training with our edited data. Our method demonstrates consistent improvements across three domains on both OLMo- IB and Llama-3-8B.
Biomedicine
Models
MQP
ChemProt
PubMedQA
RCT
USMLE
Avg
OLMo-IB
52.59
17.2
51.40
32.70
28.90
36.63
CPT
52.29
21.00
58.50
34.90
27.49
38.83
ToEdit
54.59
22.40
65,00
34.50
27.96
40.89
LJama-3-8B
66.80
28.59
60.8
73.85
40.61
54.13
CPT
72.29
29.4
69.1
72.65
36.76
56.04
ToEdit
76.39
30.2
65.3
73.30
37.23
56.48
Finance
Models
HcadLine
FPB
FiQA-SA
ConvFinQA
NER
Avg
OLMo-IB
69.00
47.03
48.05
4.83
62.19
46.22
CPT
70.31
49.78
40.36
18.72
60.44
47.92
ToEdit
71.77
51.39
46.06
18.85
62.97
50.21
Llama-3-SB
81.28
63.58
81.60
52.88
72.53
70.37
CPT
85.68
54.22
81.88
67.78
67.43
71.40
A ToEdit
83.83
61.61
80.82
67.31
67.62
72.24
Math
Models
ARC-c
GPQA
GSM8K
MATH
MMLU
Avg
OLMo-IB
28.67
1.67
0.00
26.56
16.23
CPT
28.41
24.03
1.52
0.10
27.23
16.26
ToEdit
28.92
28.12
2.20
0.10
23.63
16.59
further bounded by:
conclusions. They indicated mixing real data with synthetic data will break model collapse and provide upper bound under data accumulation. Different from their work, our data editing aims to yield better data, enabling synthetic data to perform well both in theory and practice.
odd
Eres((Wxx4)) S T-d d-1
B (rr(Ar
+ + B
1-n
5. Experiments
We provide supporting evidence for the assumption in G.3. Recalling model collapse (Dohmatob et al., 2024a): training iteratively on synthetic data leads to an accumulation of enror over iterations, as shown in the following equation:
We validate method in three language model train- ing stages: pre-training from scratch (PT), continual pre- training (CPT), and supervised fine-tuning (SFT).
pecillapee (w:..
T -
5.1. Implementation
Comparing Eq. with Eq. 7, the test error under data editing bounded by a fixed value, preventing continuous error ac- cumulation and thus avoiding model collapse. Based on the theoretical derivations and statistical analysis of synthetic data ($ 3.1), the underlying reason is that our approach tains the coverage of the initial distribution. We move away from pure data synthesis toward token-level editing, which allows us to obtain better data while avoiding model col- lapse. Moreover, remarkable previous studies (Dohmatob al., 2024c; Gerstgrasser al., 2024) pointed out similar
We use the Llama-3-8B (AI@ Meta, 2024) prior distri- bution to estimate the token distribution in each text sample. The modification probability is set to 0.99. This means that we resample tokens positions where the probabil- ity exceeds p. and the resampling is based on the condi- tional probability given the preceding context. The entire process requires only single forward pass, without auto- regressive generation. We integrate the fast inference engine vLLM (Kwon et al., 2023), allowing the entire data editing

---

How to Synthesize Text Data without Model Collapse?
Table 4. Performance of the SFT tasks. We fine-tune 3- 8B using instruction tuning and code reasoning tasks, comparing performance with the edited version produced by our method. HS and WG are short for HellaSwag and Winogrande respectively.
Table 5. Ablations on
resampled token
condition
(p)
in
biomedicine domain.
PuhMedQA
MQP
RCT
USML.E
ChemPro
Avg
Models
PIQA BoolQ
HS
SIQA
WG
Avg
0.99
64.50
55.73
30.95
27.65
14.60
38.69
2 0.999
63.60
55,40
29.09
28.12
16.20
38.48
Instruction Troning
S 0.1
62.40
51.47
25.60
29.14
10.00
35,72
Natural
Lluma-3
79.82
87.06
58.32
46.83
74,66
69.34
P 0.01
65.40
54.91
28.19
27.80
11.00
37.46
bus.
A ToEdit
80.58
87,80
58.27
46.93
74.90
69.70
Llama-3 A ToEdit
79.87
81.28
59.72
49.69
74.51
69.01
CoT
80.25
81.16
59.74
50.56
74.59
69.26
Table 6. Token distribution across different probability intervals in the biomedicine domain dataset.
Llama-3
80.79
84,04
59.98
51.43
74.66
70.18
FLAN22
A ToEdit
80.69
85,20
59.99
52.00
75.37
70.65
Interval
Percent
# Tokens
Interval
Percent
# Tokens
Open
Llama-3
79.65
83,18
60.51
48.52
74.11
69.19
Assist.
A ToFdiit
79.98
83.91
60.34
48.31
74.66
69.44
0.00 0.1)
34.7%
389M
0.55, 0.6)
3.6%
40M
0.2)
8.1%
91M
0066, 0.7)
3.7%
41M
Models
ARC-c
GPQA
GSM8K
MMLU
Avg
0.3)
5.4%
60M
[0.7.0.8]
4.0%
44M
Code Reasoning
0.4,
4.4%
49M
[..8
5.2%
58M
0.5)
3.8%
43M
1.0)
27.1%
303M
OSS-
Llama-3
51.28
27.46
49.58
62.14
45.76
Inst.
A ToEdit
51.79
28.79
49.36
62.04
46.13
Evol-
Llama-3
52.90
27.90
50.87
62.40
46.62
Ins.
A ToEdit
52.22
29.69
50.87
62.60
46.92
5.4. Ablation Studies
We conduct experiments on hyper-parameter p, including: (I) ablation studies on different values, (2) token percentage statistics, (3) comparisons of sampling strategies, and (4) an ablation study on sampling size.
Table shows the impact of different p values on the model performance, with fluctuations observed across various set- tings. Table 6 presents the distribution percentages across different probability value ranges. As mentioned above, we need to refine the data while preserving mainly source dis- tribution. As shown in Figure 5, a larger p indicates fewer tokens will be resampled, while a smaller p results in more tokens being resampled. To balance model performance and data distribution preservation, we set p 0.99 as threshold for experiments. Table 7 presents the results of different sampling strategies. Specifically, to control variables, we set k = 8 for top-k sampling and p 0.99 for top-p sam- pling. We use rejection sampling implementation in (Liu et al., 2023). The results of reject sampling, top-p, and top-k are comparable. However, top-p involves a dynamic sam- pling range, and reject sampling requires multiple rounds of computation, leading increased overhead. Consider- ing computational efficiency, we choose top-k for sampling, Table 8 shows the ablation study on sampling size of top-k. The performance gain from increasing & is relatively small. Therefore, we set k = 8 in our experiments. And a detailed case for token editing is provided in Table 12.
process to be completed on a single 4090 GPU. We use top-k as the sampling strategy with k = 8. We also incorpo- rate top-p sampling and rejection sampling in our ablation studies.
5.2. Datasets and Models
We provide an overview of our experimental setup, more details are in Appendix F. For pre-training, we pre-train the IB OL.o model (Groeneveld et al., 2024) from scratch using Dolma-sampled V6 (6B tokens) and evaluate on gen- eral tasks. For continual pre-training, we follow (Cheng et al., 2024a;b;c) continual pre-train OLMo-1B (Groen- eveld et al., 2024) and Llama-3-8B (AI@ Meta, 2024) on corpora of Biomedicine, Finance, and Math, evaluating on 5 downstream tasks per domain. For supervised fine-tuning, we fine-tune Llama-3-8B on instruction tuning and code reasoning tasks, evaluating on 9 downstream tasks.
5.3. Main Results
Table 2, 3, and 4 respectively demonstrate the effectiveness of our method in pre-training from scratch, continual pre- training and fine-tuning tasks. Across these three stages of language model training, our method consistently enhances downstream tasks without in-
6. Conclusion
the model performance creasing the data size. This consistency is validated across two models. This indicates that our method unlocks the potential of existing data, demonstrating that semi- -ynihhytthetic data is a viable path to improving model performance. A further numerical analysis is provided in $ C.
With the growing prevalence of generative Al models, when training next-generation AI models, it will be inevitable to use a mixture of synthetic data and numan--pooduce Therefore, we focus on two key questions: (1) What is the impact of synthetic data on language model pre-training,
data.
8

---

How to Synthesize Text Data without Model Collapse?
N., Wang, X., Muennighoff, N., Hou, B., Pan, L.,, Jeong,
Table 7. Ablations sampling strategy.
H., et al. A survey on data selection for language models. arXiv preprint arXiv:2402.1688 2024.
Strategy
PubMedQA
MedMCQA
MedQA
Top-k
64.50
26.13
24.82
Top-p
63.80
27.11
25.61
Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Hu- mayun, I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. Self-consuming generative models go mad. arXiv preprint wrXiv:237..01855 4:14, 2023.
Rejection Sampling
64.50
28.90
28.20
Table 8. Ablations on sampling size & for top-k.
Sampling Size (k)
PubMedQA
MedMCQA
MedQA
Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity- based data pruning with small reference mod- cls. arXiv preprint arXiv:2405.20541, 2024.
==8
64.50
26.13 28.
24.82
= 64
63,80
27.34
and what are the underlying causes? (2) How can we pre- vent model collapse and synthesize high-quality data? We found that synthetic data can impair the effectiveness of pre-training when mixed with buman-produced data, lead- ing to non-iterative model collapse. Statistical analysis re- veals that synthetic data suffers from significant distribution gaps and overly concentrated n-gram features. We propose token-level editing instead of relying purely on synthetic data. Specifically, perform token resampling guided by trained prior. Theoretically, our method prevent model collapse. Our approach demonstrates improvements over the source data across pre-training, continual pre-training, and supervised fine-tuning.
Bai, F. Zhang, H., Tao, T., Wu, Z., Wang, Y., and Xu, B. Picor: Multi-task deep reinforcement learning with policy correction. Proceedings of the AAAI Conference Artificial Intelligence, 37(6):6728 6736, Jun. 2023.
Bai, F., Wang, M., Zhang, Z., Chen, B., Xu, Y., Wen, Y., and Yang, Efficient model-agnostic alignment via bayesian persuasion. arXiv preprint uxXiv22405.1877 2024.
Bai, R. Liu, R., Du Y., Wen, Y., and Yang, Y. Rat: Ad- versarial attacks on deep reinforcement agents for tar- geted behaviors. Proceedings the AAAI Conference on Artificial Intelligence, volume 39, pp. 15453-15461, 2025.
Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravin- skyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., Datta, A., al. Stable Im 2 1.6 b technical report. arXiv preprint ar axXiv:2402.1783, 2024.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work. none which we feel must be specifically highlighted here.
Ben Allal, I.., Lozhkov A., Penedo, G., Wolf, T. and von Werra, L. Cosmopedia, 2024. URL https:
co datasets/
Acknowledgement
#uggingFFaceTT//coo cowpeeeeeeeeoooooooooooooo
This work is sponsored by the National Key Research and Development Program of China (No. 2023ZD0121400 X.Z., H.I. and Z.Z.. are supported by the National Natural Science Foundation of China (62376031).
Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data. arXiv preprint arXiv:2310.00429, 2023.
Briesch, M., Sobania, D., and Rothlauf, F. Large lan- guage models suffer from their own output: An anal- ysis of the self-consuming training loop. arXiv preprint arXiv:2311 1.6622,,.1.. 2023.
References
Qwen2 technical report. 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv.2303.08774 2023.
Cheng, D., Gu, Y., Huang, S., Bi, J., Huang, M., and Wei, F. Instruction pre-training: Language models are supervised multitask learners. In Conference on Empirical Methods in Natural Language Processing, 2024a. URL https:ttttoccmmmccco //pii. semanticscholar: org/ / Corpus :700000002
AetMe Llama 3 model card. 2024.
URL https: //tp:://ghttt//// www.gitb..o///moeeeeeeemeeee 1ale-llamm /1mmm// blob/ma i / /MRDEE_CCEDDD...... md.
Cheng, D., Huang, S., and Wei, F. Adapting large lan- guage models via reading comprehension. In The Twelfth
Albalak, A., Elazar, Y., Xie, M., Longpre, S., Lambert,

---

How to Synthesize Text Data without Model Collapse?
International Conference on Learning Representations, 2024b. URL wttp:////ppeweeeeeeeeeeeeeeeeeee forum? Cd-Y886UXXEZO.
Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T.,, Agrawal, R., Pai, D., Gromov, A., et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint 24X1:22444001413 2024.
Cheng. D., Huang, S., Zhu, Z., Zhang, X., Zhao, W. X., Luan, Z., Dai, B., and Zhang, Z. On domain- -ppeciffe post- training for multimodal large language models. arXiv preprint T.V:22111.199300 2024c.
Groeneveld, D., Beltagy, I,, Walsh, P., Bhagia, A., Kin- ney, R., Tafjord, O., Jha, A., Ivison, H.,, Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, Khot, T., Merrill, W. Morrison, J. D., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell. E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Sol- daini, L., Smith, N. A., and Hajishirzi, H. Olmo: Acceler- ating the science of language models. arXiv preprint, 2024. URL https: appi. wwmant.cccholll.. org pegg/CorpusD:::00 2673554488
Dohmatob, E., Feng, Y., and Kempe, J. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07712. 2024a.
Dohmatob, E., Feng. Y., Subramonian, A., and Kempe, Strong model collapse. arXiv preprint arXiv:2410004800, 2024b.
Dohmatob, E., Feng, Y., Yang, P., Charton, F., and Kempe,
J. A tale of tails: Model collapse as a change of scaling laws. arXiv preprint arXiv:2402.07043 2024c.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.00 08999, 2023.
Eldan, R. and Li, Tinystories: How small can lan- guage models be and still speak coherent english? arXiv preprint arXiv:2305.07759 2023.
Feng, Y., Dohmatob, E., Yang, P., Charton, F., Kempe, J., and Meta, F. Beyond model collapse: Scaling up with syn-thesized data requires verification. arXiv preprint arXiv:2406.07515, 2024.
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306. 2023.
Ferbach, D., Bertrand, Q., Bose, A. J., and Gidel, G. Self-consuming generative models with curated data provably optimize human preferences. arXiv preprint arXiv:2407. 0994999999999.... 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.000 066696 2021.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. ArXiv, abs/2101.00027, 2020a. URL https: //api. swwanteeeiiiee semanticscholar. wtw.choolar...r/crrpuss.... 230435736.
Jia, X., Yang, Z., Li, Q., Zhang, Z., and Yan, J. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877, 2024.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F,, et al. Mixtral of experts. arXiv preprint urXiv:2401.04088 2024.
Gao, L., Biderman, S. Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima. N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020b.
Kazdan, J., Schaeffer, R., Dey, A., Gerstgrasser, M., Rafailov, R., Donoho, D. I.., and Koyejo, S. Col- lapse or thrive? perils and promises of synthetic data a self self geleraeneg world, 2025. URL https:// open rpeineeiew net/ forfmrrmmmm/ id=xr5i TNA3ZU.
Gao, L., Tow, Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, I.., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation,
Kopf, A., Kilcher, Y., von Rutte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., Shahul, E., Suri, S., Glushkov, D., Dantu- luri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations. democratizing
07 2024. URL https: zenodo. org/ wwwg///eooooooooooo records/ 12608602.
10

---

How to Synthesize Text Data without Model Collapse?
large language model alignment. ArXiv, abs/2304.07327, 2023. URL https: http:::://app.ssmmttiiishiheee semanticschola::
of the Association for Computational Linguistics, 2024. URL https://aaat org/ CowppsoD:2673111000.
seemanticschooar.
Kwon, W. Li, Z., Zhuang, S., Sheng, Y., Zheng. L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Ef- ficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Martínez, G., Watson, L.., Reviriego, P., Hernández, J. A.,
Juarez, M., and Sarkar, Towards understanding the interplay of generative artificial intelligence and the inter- net. In International Workshop Epistemic Uncertainty in Artificial Intelligence, pp. 59-73. Springer, 2023.
Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649. PMLR, 2022.
Li, L., Lyu, J., Ma, G., Wang, Z., Yang, Z., Li, X., and Li, Z. Normalization enhances generalization in visual re- inforcement learning. arXiv preprint arXiv:2306.0066 2023.
points
Li, Q., Jia, X., Wang, S., and Yan, J. Think2drive: Effi- cient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla- v2). arxiv preprint arXiv:2402. 16720, 2024.
Mobahi, H.,, Farajtabar, M., and Bartlett, P. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3366 2020.
Li, X., Yang. Z., and Wu, H. Face detection based on re- ceptive field enhanced multi-task cascaded convolutional neural networks. IEEE access, 8:174922-174930 2020.
Niu, Y., Pu, Y., Yang. Z., Li, X., Zhou, T., Ren, J., Hu, S., Li, H., and Liu, Y. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., et al. Rho-1: I: Not all tokens are what you need. arXiv preprint arXiv:2404.07965 2024.
Pu, Y., Niu, Y., Yang, Z., Ren, Li, H., and Liu, Y. Unizero: Generalized and efficient planning with scalable latent world models. arXiv preprint 1xXi:22406.1066 2024.
Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., Zhou, D., et al. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503, 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Salch, M., Liu, P. J., and Liu, J. Statistical rejection sampling im- proves preference optimization. ArXiv, ab6/2309.06657, 2023. URL https: wwt..ssstatttcsco....o.o.... org/CorpusII:2551 :2:
Rombach, R., Blattmann, A., Lorenz, D.,, Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2021.
Rudin, W. Principles of Mathematical Analysis. McGraw- Hill, New York, 3rd edition, 1976.
Longpre, S., Hou, L.,, Vu, T., Webson, A., Chung, W., Tay, Y., Zhou, D., Le. Q V. Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. arXiv preprint reXiv:2301.1366 2023.
Seddik, M. E. A., Chen, S.-W., Hayou, S., Youssef, P., and Debbah, M. How bad is training on synthetic data? a statistical analysis of language model collapse. arXiv 2024.
preprint arXiv:2404.0500
Magnusson, I., Bhagia, A., Hofmann, V., Soldaini, L., Jha,
A., Tafjord, O., Schwenk, D., Walsh, P., Elazar, Y., Lo, K., Groeneveld, D., Beltagy, I., Hajishirzi, H., Smith, N. A., Richardson, K., and Dodge, J. Paloma: A benchmark for evaluating language model fit. ArXiv, abs/2312.10523,, 2023. URL https: wwwpp.ssaaatiiscccccceo org/ Corpoorr 266348815
Shumailov, I., Shumaylov, 7.,, Zhao, Y., Papernot, N., Ander- son, R., and Gal, Y. Ai models collapse when trained on recursively generated data. Nature, 63 (8122)2))7555758 2024.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023.
Maini, P., Seto, S., Bai, R. H., Grangier, D., Zhang, Y., and Jaitly, N. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Annual Meeting

---

How to Synthesize Text Data without Model Collapse?
Soldaini, L.., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Beltagy. I., Groeneveld, D., Dodge. J., and Lo, K. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, arXiv preprint, 2024. URL https: 2402. 00159.
Yang, Z., Jia, X., Li, H., and Yan, LIm4drive: A survey of large language models for autonomous driving. arXiv e-prints, pp. arXiv-2311, 2023.
Zhang. K., Zeng, S., Hua, E., Ding, N., Chen, Z.-R., Ma, Z., Li, H., Cui, G., Qi, B., Zhu, X., al. Ultramedical: Building specialized generalists in biomedicine. arXiv 2024.
preprint arXiv:2406.0390
Zhang. M., Zhang, S., Yang. 7., Chen, I.., Zheng. J., Yang, C., Li, C., Zhou, H., Niu, Y., and Liu, Y. Gobigger: scalable platform for cooperative-compettive multi-agent interactive simulation. In The Eleventh International Conference on Learning Representations, 2023.
org/ abs/
Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhat- tacharjee, A., Karami, M., Li, J., Cheng, L.,, and Liu, H. Large language models for data annotation and synthesis: A survey. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 930-957, Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguistics. doi: 10. 10.188663/1120224.cmmmmmmm ac wellanthoooy..orggg/ooo int..///........................ 54/.
Zheng, Y., Zhang, R., Zhang, J., Ye. Y., Luo, Z., Feng, Z., and Ma. Y. Llamafactory: Unified ef- ficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. As- sociation for Computational Linguistics. URL http:
URL https://
//arxiv. org/ abs 2403. 13372.
Zhu, X., Guan, J., Huang, M., and Liu, J. Storytrans: Non-parallel story author-style transfer with discourse representations and content enhancing. arXiv preprint Stlv:2008.134233. 2022.
Trinh, T., Wu, Y., Le, Q. He, H., and Luong, T. Solv- ing olympiad geometry without human demonstrations. Nature, 2024. doi: 10. 10.10888441586--2200664444
Zhu, X., Fu, Y., Zhou, B., and Lin, Z. Critical data size language models from a grokking perspective. arXiv preprint arXiv:2401.10463 2024a.
Ulmer, D., Mansimov, E., Lin, K., Sun, J., Gao, X., and Zhang, Bootstrapping Ilm-based lask-oriented dialogue agents via self-talk. arXiv preprint arXiv:2401.0500 .050033, 2024.
Zhu, X., Qi, B., Zhang, K., Long, X., Lin, Z. and Zhou, B. PaD: Program- aided distillation can teach small models reasoning better than thain-of-thought fine-tuning. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1571-2597, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10.18663/11/2222mmaalll aclanthology. wwt/oogg//0022 http//2222........... naa.laccl--ooooooo hhacclllngg...... 142.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self- generaene instructions. arXiv preprint 221: 10560, 2022.
Wei, Wang, X., Schuurmans, D., Bosma, M., hsin Chi, E. H., Xia, F., Le, Q. and Zhou, D. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903, 2022. URL https: https://ap...emmtiiissss semanticscoollee... orr//CerpusDD:24111222
142. URL https:
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for tar- geted instruction tuning. ArXiv, abs/2402.004333, 2024. URL http:///app..eeaancccch.ooooooo brg/CorpusID::2652222
Xie, S. M., Santurkar, S., Ma, T., and Liang, P. S. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201 -34227, 2023.
12

---

How to Synthesize Text Data without Model Collapse?
A. Proof
A.1. Proof of Theorem
For n = I, we have:
11 = (x ""X""""""""" + +Ei = w" + - 7xTE, E
(Xw"
For n 2 1, we have:
Wa+1 81 x 1n+1
Y
(X x ) Xx Yn+1
(x x) X Yn+1
Recalling that:
y, Xw + Ei,
ifi=1
E,) + +(1 (1- M1--)- i
Substituting this Y into the expression for un+1:
We begin the data editing data process:
V2 = -M F2) +11+(11 (1)+(1-MM))
(8)
Then:
Y,- = M2(X + Ea) + +(1 (1- -M2)
(9)
We have:
Y33 M2( + FF)) + (1 - M2) (X + F2) + +11 M))B)
M2 + E3) + (1 ( + + + M1) (1)
We can expand Ya+1 by recursively substituting the previous expressions:
y n+1 M2 ún + Fn+1) (1 Mn
(10)
.M,(Xix, úÚ, + En+1) (1 MM.) M Mn Em) (1 Mn-1))
(11)
Mr(X th + En+1) + (1 -
En) + (1 - Mn)(1 - -Mn
(12)
(13)
II (1 M,) MI(Xx + E M j=i+1
+ [N M,) 81
(14)
Recalling properties of Mj:
M1(1 - M1) = 0 and (1 - (-MM)M =
(15)
M,M,- = for
i i j
(16)
(1-M)(1 M2)= M₁ Mj for i i
(17)
(18)
13

---

How to Synthesize Text Data without Model Collapse?
Then have:
Yn+1
M(Xi; + + 1 Σ
Mr
Y
(19)
i=1
MiXii + E5-1)+ +
1 -1-_u0
Mi
(Xw" + E1)
(20)
i=1
n
Xw" + E +
Σ
Mi (X(ix) w²) (Ei+1 -E1))
(21)
Substituting this back into the expression for uz+1:
Wn+1 = (X
Xw Xw*+E +E+
Σ Mi (X(u;; w²) + (Bi+1 Bil)
(22)
w =w"+ +x (X XT
E1 +
M,X (w; =w'++ Σ Mi(E++1 -Ei)
(23)
We observe:
W1 (XTX)- X (Xx'""XX""
(Xw" + =)) (XX E
(24)
U2 = w² + +(X - (MX((X))) E M, + (1 - M,)
(25)
= + X (E1 M2
(26)
We prove this Theorem 1 by induction.
Inductive Step:
Assume the formula holds for n, we have:
Wn+1 w² + +XX T (E1 + M, M2F3 +...
+ M, Fm+1)
(27)
= = - + X) T
Ei + Σ M,E++1
(28)
Substitute úi into W
Then we can get:
0n+1 w² x(
E1 +
M,P
E + Σ
M₁Ef+1
Σ M,
M,(E,+1 -E1)
(29)
=w- =x=xxxxx) -X
E1 +
Σ
Mi
E +
Σ
M, Ej+1
(30)
1=1
1=1
=w- - + xx
Ei +
Σ
M, Ei+1
(31)
1=1
where P P=X(X x) x=
(32)
The above derivation aligns with Theorem 1, and the proof is complete.
14

---

How to Synthesize Text Data without Model Collapse?
A.2. Proof of Theorem 4.3 We substitute the Eq. 28 into Test Error Eq.
Etest(un =
(X x
Ej + Σ
M1E
(33)
T
n
n
Ei Σ
MrBi+1
x(X X)2Xx
-2X
Ei + Σ
M, Bi+1
(34)
i=1
i=1
O'E tr ((XTx)-)) x)
0²
Σ E tr (M (xxxxXXXXX x M.)l
(35)
= o (IxXxxx) + 0² E tr ((X (x-
(36)
Further, by applying the Cauchy-Schwarz inequality (Rudin, 1976), we obtain:
Etest(unn+l O [tr ((XTx)=)) ²² (I Σ [t(
(37)
We refer the following lemma (Dohmatob et al., 2024a), which is essential for proving Theorem
Lemma 3. Let T and d be positive integers with T 2 d + 2, and let X E R xd be random matrix with i..d. rows from (0, Σ) with D positive definite. Then, X has full rank a.s. Moreover, it holds that:
Ex ((XX))+
T-d-1 E
(38)
Using Lemma 3, we have:
d
Frsst [tr (xxxx)
(39)
T T-d-1
Then, we have:
Erest = 8²E [tr ((X (x-]))-))] + 0² Σ E [tr ((
(40)
Fdd S T 1-1 BIr x) x)
(
[tr M.)]
(41)
In our setting, the data is incrementally modified over iterations and modifications decreases progressively. This behavior can be modeled by the sum geometric series, where the amount of modified data decreases by fixed ratio y with each iteration. Then, we assume the editing operation as ||M||| ||MM, for 1,2,.... n. Therefore, the test error for data editing be bounded:
o²d
E [r(
Ecest (w)+1) S T o² E ((X ((XTX)-2)
(42)
Additionally, since M, is full-rank, as seen from Eq. 36, we can apply more relaxed and simplified bound, follows:
202d
Etest (wes((@n)))))) < T
(43)
Thus, the above derivation satisfies the Theorem 4.3.
15

---

How to Synthesize Text Data without Model Collapse?
Algorithm 1 Token-level Editing
Input: Sequence of tokens x (21 (2....... 2t), prior distribution P. probability threshold p
2: Output: Edited sequence x' = (1/ (
3: for each token z; in sequence x do
4: Compute conditional probability P(x;) x1, .1...., .3-1)
5: if P(æ; 21,,..,, T;-1) 2 then
6: Resample token é; from prior distribution
7: Set xet f I;;
8: else
9:
Set x' 6 I2
10: end if
11:
end for
12:
Return: Edited sequence x' =
x==(x''...>>>> (z1
Table 9. Comparison of human and synthetic data performance across downstream tasks in (Maini et al., 2024), based on training with GPT-2.
TruthfulQA
LogiQA
Wino.
PIQA
ARC-E
BoolQ
OBQA
Avg
Human Data
32.68
23.03
51.3
64.42
44.4
60.98
15
41.69
25% Synthetic Data
27.91
21.37
50.12
63.93
43.94
62.29
15.4
40.71
50% Synthetic Data
30.84
22.58
52.41
63.33
44.02
62.14
16
41.62
75% Synthetic Data
29.5
22.65
49.8
63.44
44.53
61.56
17.2
41.24
Synthetic Data
28.89
22.58
49.72
63
46.3
54.53
16.8
40.26
B. More Related Work
Phi-1/2 (Gunasekar et al., 2023) demonstrate that the synthetic data can boost training efficiency and performance compared to raw data in language model pre-training. Furthermore, Feng et al. (2024) introduce verifier to filter synthetic samples, theoretically avoiding model collapse. Liu et al. (2024); Tan et al. (2024) highlight that synthetic data will play a crucial role in the development AI. For example, synthetic data can be used construct highly specialized datasets, enhancing the performance of downstream tasks. Trinh al. (2024) utilize synthetic math data train a 125M language model, which successfully solved 25 out of 30 selected problems from the International Mathematical Olympiad (IMO) problem set. Zhang et al. (2024) develop a biomedical instruction dataset that was used to train specialized bio-models, enabling them to excel in answering questions related to medical exams and clinical scenarios. Eldan & Li (2023) introduce a novel synthetic dataset and evaluation paradigm that enables small language models to generate coherent, diverse, and grammatically sound stories. As outlined above, in the post-training stages of LLMs, synthetic data enhances downstream task performance and aligns foundation models with humans. And Maini et al. (2024) propose rephrasing the pre- pre-training data into Wikipedia or Q/A style achieve better alignment with downstream tasks. Synthetic data is powerful tool for training. Our approach is also based on synthetic data methods. Instead of sampling data solely based on this prior, modify the data using the prior as guide.
Bertrand et al. (2023) develop rigorous framework to demonstrate the importance of real data in maintaining the stability of iterative training. Ferbach et al. (2024) theoretically demonstrate that the impact of data curation can be formalized as an implicit preference optimization mechanism. Kazdan al. (2025) reveal the detailed training dynamics of model collapse under three different training workflows. Of course, there are also some remarkable studies that successfully used synthetic data. Wang et (2022) propose the Self-Instruct data generation framework, enhancing cesructioo-followinn capabilities. Ulmer et al. (2024) employ the self-talk method to generate high-quality data. ReST (Gulcehre et al., 2023) uses policy model to generate datasets and then employs offline RL to fine-tune L.MMs on generated datasets. Singh et al. (2023) demonstrate that self-training with binary feedback filtering can reduce reliance on real data. Alemohammad et al. (2023) demonstrate that without enough fresh real images, future generative models will gradually decline. Briesch et al. (2023) illustrates that real data the iterative training process can slow the decline of LLMs, but cannot fully prevent it. Martínez et (2023) shows that the quality and diversity of generated images degrade over time.
16

---

How to Synthesize Text Data without Model Collapse?
0.9-1.0
0.8 0.9
5.2%
0.7-0.8
4.0%
0.6-0.7
306
34.7%
3,80
0.0.0.1
0.4-0.5
1.0%
5.4%
8.1%
0.2-0.3
0.1-0.2
Figure 6. Token distribution across different probability ranges in BioMed dataset.
C. More Discussion of Main Results
As shown in Table 3, our method shows consistent improvements over the source data across OLMo-IB and LLaMA-3-8B. For instance, in the Biomedicine domain, the average score for OLMo-1B increased from 36.63 to 40.89 with ToEdit, while LLaMA-3-8B saw an increase from 54. to 56.48. Table 2 further supports the effectiveness of our approach in pre-training. The average performance of OLMo-1B increases from 32.75 33.11, reflecting improved generalization capabilities. While the improvement is modest, the consistent trend across tasks like PIQA, BoolQ, and ARC-c highlights the broader applicability of our method. As for SFT results in Table 4, using both the original and edited data, the results indicate small but consistent improvement. Specifically, ToEdit improves original FLAN v2, with average performance increasing from 70.18 to 70.65. For Natural Instructions, the average performance of LLaMA-3-SB improves from 69.34 to 69.70, with gains in tasks like Winogrande and SIQA. These improvements demonstrate the adaptability of our method to instruction-tuning tasks. For code-related tasks, the improvements demonstrate better reasoning and code comprehension.
D. Comparison with Pure Synthetic Data and Reformat Methods
Definition and Characteristics of Synthetic Data Synthetic data (Da) can be categorized based on its relationship with the distributions of language model (PLM) and human- data (Panaa during the generation process, quantified as (L[(I] )
Drre PLM: if (L.(PMM)
| Pitasa) > 6,
D. =
(44)
Drmi Psemis
if (L(Pscmil
Pdata) S
where Pure Synthetic Data Dpure; Generated entirely from the language model (D.tre ~ PLM). with KL divergence KL Pasa) exceeding a threshold c. This implies a significant deviation the language model's distribution from the human- producedddddddde data distribution. Semi-Synthetic Data D Derived from limited modifications to human- human-produced data (Pama), ensuring that the resulting distribution (Psem) has KL divergence KL (ssmi Paata) bounded by e. This reflects
closer alignment of semi-synthetic data with human produced data.
From the generation process, pure synthetic data Dere This data is induced by a language model through prompts and does not modify human-produced data, resulting in low overlap content with human-produced data. For example, Cosmopedia (Ben Allal et al., 2024) expands numan--roduuee Semi-Synthetic Data D This data generated by directly modifying human- data, such as paraphrasing or token-level editing. It derives from transformations human- produced data. For example, WRAP (Maini et al., 2024) generates paraphrases of human- producdd data. ToEdit (ours) performs token editing on human muman-produced data.
data and generates data without human human-producce produced data.
17

---

How to Synthesize Text Data without Model Collapse?
Human Data
10
Mix 25% Synthetic Data
Mix 50% Synthetic Data
8
Mix 75% Synthetic Data
Synthetic Data
Loss
6
4-
2
0
5 10
15
20
25
30
35
40
Tokens (billions)
Figure 7. Pre-training loss of GPT-2 Small (124M) on human (Dolma (Soldaini et al., 2024)) and synthetic (Cosmopedia (Ben Allal et al., 2024)) data, As the proportion of synthetic data increases, the model's loss decreases.
Specifically, both Rephrasing the Web (Maini et al., 2024) and our token-level editing aim to refine data while preserving the original distribution, producing semi-synthetic data. In contrast, purely synthetic data in Cosmopedia lacks the long-tail distribution and overly concentrates on n n-gram features. Ultimately, semi- mymi-synthhtic data enhances training performance, whereas purely synthetic data results in model collapse. Morcover, replacing a whole real sample with synthetic data can damage the performance.
The primary distinction between Cosmopedia, Rephrasing the Web (Maini et al., 2024), and approach lies in how much of the original human data distribution is preserved. We provide detailed comparison of these synthetic methods in Table 10.
Table 10. Comparison of different synthetic data methods.
Method
Data Type
Approach
Result
Cosmopedia (Ben Allal el al., 2024) Rophrasing the Web (Maini et al., 2024)
Pure synthetic Semi-synthetic
Using prompt to induce data from LLMs. Using prompt and source content to guide LLMs to reformat souree content.
Reveal non-iterative model collapse. Improve training performance.
ToFdit (Ours)
Semi-syntheric
Using the distribution of source content estimated by M. (single forward pass) to replace tokens.
Improve training performance.
E. More Results of Human and Synthetic Data Mixture Training
We provide more training results for the human and synthetic data mixture. The main results and analysis can be found in Sec 3.1. Except for GPT-2 pre-training, we also use the OLMo models (Groeneveld et al., 2024) for further experiments.
As shown in Figure the training loss continues to decrease as the amount of synthetic data increases. which is consistent with GPT-2 pre-training Figure 2. More synthetic data can lead to better fitting. However, lower loss does not necessarily mean a better model. As illustrated in Figure 2B and 9, models that fits better perform worse in real world tasks.
Furthermore we follow (Maini et al., 2024) to conduct more experiments including PPL results on 22 validation sets of Pile (Gao et al., 2020a) and general understanding tasks. The additional results in Table 9, 11 and 1 are consistent with our findings. Specifically, the PPL increases as the proportion of purely synthetic data grows, while the performance on downstream tasks similarly exhibits a gradual decline with the increase in synthetic data.
18

---

How to Synthesize Text Data without Model Collapse?
Table 11. Comparison of human and synthetic data performance across downstream tasks in (Maini al., 2024), based training with
OLMo-237M. indicates the standard error.
TruthfulQA
LogiQA
Wino.
PIQA
ARC-E
OBQA
Avg
Human Data
26.81 + 1,550
21.06 + 1.028
52.01 i: 1404
56.69
1.156
31.73 +: 0.9550
13.80
1.543
33.68
25% Synthetic Data
26.44 + 1.543
21.25 = 1.032
52.64 = 1.403
57.02
1.155
31.78 = 0,9582
12.40 1.475
33.59
50%
Synthetic Data
25.95 + 1.534
20.04 = 1.099
52.25 # 1.408
56.64
1.126
31.82 = 0,9557
12.80 1.495
33.25
75% Synthetic Data
25.34 + 1.522
20.87 = 1.025
50.43
+ 1.405
55.60
1.159
32.74 + 0.9629
12.00 1.454
32.83
Synthetic Data
23.01 + 1.473
20.29 + 1.014
49.33 + 1.405
55.93
1.158
33.33 + 0,9673
14.20
1,562
32.68
Pre-Training Loss Curves on OLMo 237M
GPT-2 Perplexity on mc4-en
Human Data
10
Mix 25% Synthetic Data
160
Mix 50% Synthetic Data
140
Mix 75% Synthetic Data
Human Data
Loss
Synthetic Data
120
25% Synthetic
50% Synthetic
100
75% Synthetic
Synthetic Data
80
60
10
15
20
25
30
40
0
5
10
15
20
25
Tokens (billions)
Tokens (billions)
Figure 8. OI.Mo-33MM pretraining with mixed human and syn- thetic data proportions. We pretrain the MLMo-237M model using mixture of human data (Dolma (Soldaini al., 2024)) and syn- thetic data (Cosmopedia (Ben Allal et al., 2024)).
Figure GPT-2 perplexity (PPI.) on validation sets, trained from scratch.
F. Experiment Settings
In this section, we describe our experiments settings in detail.
F1. Training
Pre-training We utilized both GPT-2 and OLMo models. The pre-training datasets included Dolma, representing real data, and Cosmopedia, representing synthetic data. For GPT-2, we employed the official FSDP (Fully Sharded Data Parallel) framework provided by Torch for training. For OLMo², we used the official open-source computational code, which also incorporates the FSDP framework alongside Flash Attention for acceleration.
Continual Pre-training We follow (Cheng et al., 2024b) conduct continual pre-training on biomedicine, finance, and math domains. Specifically, PubMed Abstracts from the Pile are utilized as the pre-training corpora for the biomedicine domain. For the finance domain, financial news data covering over 7,000 stocks from May 2022 to May 2023 is collected using the FinGPT framework. We continue pre-training OLMo-1B and LLAM--3-8B on each domain. For implementation, we utilized the official training framework for OLMo-1B, leveraging Fully Sharded Data Parallel (FSDP) for continual pre-training. For LLaMA, we adopted the LLaMA- LaMM--Fattryy framework to carry out the continual pre-training process. Our experiments was primarily conducted OLMo- B and Llama-3-8B models, with Llama-3-8B utilizing LoRA (Low-Rank Adaptation) for parameter-efficien fine-tuning. The data and evaluation given in this repo*. We conducted the continual pre-training on total of IB tokens.
Supervised Fine-tuning We used the Llama-Factory (Zheng et al., 2024) framework to fine-tune Llama-3-8B, As for general instruction tuning tasks, we adopt instruction tuning datasets from (Xia et al., 2024) $ including CoT (Wei et al., 2022), , FLAN v2 (Longpre al., 2023), and Open Assistant (Kopf et al., 2023). As for code-related reasoning tasks,
pettps://githie wttpta/..coo/aaaa///ooaa
htt.:s////gtttoooo...ccoooorr/t/ i 1m wttp:hhwww......toooototeteeeeee
weeeeasnnnnse/nneeeeeeeeeeeeeeee les__datt
19

---

How to Synthesize Text Data without Model Collapse?
Table 12. Case Study.
Before (source)
After (edited)
Changes
Construct a function using PHP lan- guage that applies lexical analysis on a provided text string analyze the individual, non-repeated words ele- ments present.
Construct function using PHP lan-
"analyze" + "quantify"
guage that applies lexical analysis a provided text string quantify unique words.
Test with provided string, Sstr Greetings, Planet Earth!'.
Test with provided string, $str 'Greetings, Planet Earth!'.
No changes.
Implements wordCount to remove punctuation, convert text to lower- case, split into words, and count unique words.
Implements wordCount to remove punctuation, convert text to lower- case, split into words, and calculate unique words.
"count" +
"calculate"
Returns ('greetings': 1, Returns l'greetings': 1, No changes. pplanet 1, 'earth':
'planet': 1, 'earth':
1}.
1}.
we utilize DSS-Instruct-75K and Evol- Evol-Instruct-11 10K 7. These datasets provide sufficient diversity for verification on fine-tuning. We apply LoRA (Hu et al., 2021) Llama-3-8B experiments.
F22 Evaluation
Pre-training We use PPL and downstream tasks conduct analysis and performance test. As for PPL., stands for perplexity, commonly used metric in NLP to evaluate the quality of language models. It measures how well probabilistic model predicts given dataset, with lower values indicating better performance. Formally, the perplexity of a language model is calculated as:
PPL =2 2- EN loga P(n:)
Alternatively, can also be expressed as:
N log P(x:)
PPL exp
Σ
N
Where N is the number of tokens in the dataset, and P(x;)) is the predicted probability of the i-th token. Perplexity essentially represents the exponential of the average negative lig-likelihood of the predicted tokens, indicating how "perplexed" the model is when making predictions.
As for downstream tasks, we use general understanding tasks in (Maini et al., 2024) to analyze model collapse in Table 9 and general test tasks in (Cheng et al., 2024a) to test our methods in Table 2. All downstream tasks we used can be found (Gao et al., 2
Continual Pre-training We use the test data and code in (Cheng et al., 2024b) to test domain specific task performance after CPT.
Supervised Fine-tuning performance and reasoning tasks to assess reasoming capabilities. All downstream tasks we used can be found in (Gao et al.,
We utilize the general downstream tasks from (Cheng et al.. 2024a) to evaluate instruction-tune
'https: http:///wwggggacccccc..ee/////es http:://whw.....ooeooeoeee/////a
"nttps: httpt/////aaaaaaacccc///sattttas
/
20

---

How to Synthesize Text Data without Model Collapse?
-10
1.4
Dolma
Cosmopedia
2.0
1.2
1.0
1.5
Count
Count
0.8
1.0
0.6
0.4
0.5
0.2
0.0
50
100
150
200
250
300
350
400
5
10
15
20
25
30
Perplexity (PPL)
Perplexity (PPL)
A. Human Data PPL Distribution
B. Synthetic Data PPL Distribution
Estimated StableLM-3B Dolma-sampled v6(6B Tokens)
Estimated StableLM-3B cosmopedia-sampled (6B Tokens)
Figure 10. PPL distribution of human and synthetic data estimated by StabLM-ZZephr--3B This indicates that different prior distributions yielded the same result, which is consistent with Figure The synthetic data lacks long tail and concentrated within narrow portion of the distribution.
2024) 10
Table 13. PPL results of GPT-2 124M pre-training on mixture of human and synthetic data.
Synthetic Data Ratio
25%
50%
75%
Tokens Size
8.4B
16.8B
25.2B
33.6B
42B
8.4B
16.8B
25.2B
33.6B
42B
8.4B
16.8B
25.2B
33.6B
42B
Epochs
1
2
3
4
5
2
3
4
5
Wikitext- 103
45.97
39.87
37.65
36.91
36.32
50,29
43.15
40.46
39.43
38.65
58.66
48.75
45.20
43.42
42,95
RedPajama
42.28
37.62
35.72
34.66
34.24
46.89
41.42
39.37
38.21
37.72
55.72
49.26
46.27
44.81
44.30
Falcon-RefinedWebb
56.40
50.62
48.26
47.13
46.66
61.06
54.34
51.72
50.39
49.87
69.32
61.50
58.28
56.77
56.19
c4-en
48.15
43.14
40.98
39.91
39.41
51.79
46.06
43.90
42.73
42.23
58.60
52.22
49.26
47.87
47.27
mo4-en
62.46
56.80
54.35
53.06
52.71
70.43
62.48
59.61
57.66
57.07
80.37
71.77
67.90
65.31
64.82
Table 14, PPL results of OLMo-237M pretraining mixture of human and synthetic data.
Symbheic Habo: 0%%
25%
3155
1077
USIR (IM)
1950 (30M)
Edts Clancfir (IM)
Clachhr (10M)
PIt. Fiker (IM)
191 (1IM)
Deraity Sumping (IM)
Denuiyy Sumpling (10M)
1488
5.48
0.88
7,1B
Trome
3-40
5-411
1.10
8.41
8-41
1.401
5.411
10:31
3.5
7.30
Speche
14
Wikiiees- 303
1ST.35
185.5
25006
367.46
1605.73
1300.5
1757.03
1111.20
1612.6
738.36
1198.25
1188.00
1753.89
RodPijazra
175.35
153.98
236.35
301,9
007.0
549.36
916.51
$11.34
1104.77
370.30
645.22
759.57
$96. 15
Falore Wihh
M200-
G. Discussion
G.1. Non-Iterative Iterative Model Collapse.
We define non-iterative model collapse the performance degradation caused by directly mixing general synthetic data with human-produce data, without iterative training. Theoretically, without additional regularization constraints to guide data generation, the variance of the model-generated data gradually decreases during this process. The diversity of the generated data diminishes over time, ultimately leading to the collapse of the model itself.
The difference between the two lies their scope. Non-iterative model collapse is not confined to training on self celf-generated data, allowing uncover broader properties of synthetic data. For instance, in our experiments, we train GPT-2 on the Cosmopedia dataset in a single generation, which was generated by Mixtral 8x7B Instruct v0. 1. In contrast,
Phttps:////t thub. com aeeeuhherAA/// eva nat on hon-harness
21

---

How to Synthesize Text Data without Model Collapse?
iterative model collapse focuses on training the model over multiple generations using self-generated data.
Furthermore, the non-iterative model collapse emphasizes the gap between human data and general purely synthetic data, particularly regarding distributional properties and n-gram features. In contrast, the iterative model collapse illustrates the iterative evolution of the model, resembling self-play process, This process illustrates the gradual evolution of self-generated data. does not involve an analysis of the differences in nature between self-generated and human-produced data. They both ultimately lead to model collapse, driven by the same underlying cause- synth-iy data, though they investigate different aspects of synthetic data. We often face that training model on a mixture of human and synthetic data, where the synthetic data is not generated by the model itself, and its exact origin may be unknown.
G.2. Why Does the Observed Probability Distribution Exhibit Filtering Potential?
From the perspective of information theory, we can analyze the filtering potential of the U-shape distribution as follows: We utilize the U-shape distribution in Figure 5 to re-sample tokens in the high-probability region, adjust the U-shaped distribution toward uniform distribution. By doing so, we can maximize the information entropy. According information theory, maximizing information entropy is achieved when the distribution is uniform.
Lemma 1: Let X be a discrete random variable with n possible outcomes, If the probability of each outcome is uniform, iee. P(x)) = for all E (1,2,...., n.. the Shannon entropy is maximized, given by:
n H(X) = Σ 12
log log n.
(45)
72
This represents the maximum uncertainty achievable, implying that the dataset carries the maximum possible information content. Thus, the uniform distribution, which assigns equal probability to all outcomes, possesses the maximum information entropy. To leverage this property, we utilize the U-shape distribution to re-sample tokens in the high-probabilityy region, adjusting the U-shaped distribution toward a uniform distribution. By doing so, we can maximize the information entropy.
From the perspective of language model learning, our method emphasizes the importance of poorly learned data. Specifically, we resample easy tokens and encourage the model to focus learning more challenging ones. Our method can enhance the learning of underrepresented data by resampling high-probability tokens.
G.3. Gradual Decline in Editing
Table 15, Percentage of tokens requiring edits in the
dataset. The total number of tokens is 4,671,834.
Gen 1 (source)
Gen 2
Gen 3
Tokens (p > 0.99) Percentage
584,103 12.5%
549,519 11.76%
517,433
11.08%
We present the percentage statistics of edited tokens in Table 15 and performance in an iterative process in Table 16, demonstrating that the edited tokens indeed exhibit progressive decrease. Specifically, We observe that the percentage of edited tokens (above the threshold p > 0.99) decreases as the generation number increases. Theoretically, this is a process of distribution shifting. When tokens (p > 0.99) are resampled, randomness is introduced. The sampling process can select tokens with lower probabilities. Then, tokens (p 0.99) is replaced, leading to a reduction of edited tokens in subsequent generations. The Table 15 provides empirical evidence for this pattern of decay.
Table 16. Performance in an iterative process on Instruction tuning data.
PIQA
BoolQ
HS
SIQA
WG
Avg
Gen 0
79.87
81.28
59.72
49.69
74.51
69.01
Gen 1
80.25
81.16
59.74
50.56
74.59
69.26
Gen 2
80.14
82.69
59.82
50.51
73.80
69.39
22

---

How to Synthesize Text Data without Model Collapse?
G.4. What is Coverage Narrowing?
"coverage narrowing' refers phenomenon in which the distribution of synthetic data covers a significantly narrower range of values compared to human data, even when the data sizes are identical. For instance, shown in Figure 3, the PPL range of synthetic data is limited to [0, 14), whereas the PPL range of human data extends from [0. 100+]. Despite this disparity, the overall coverage, represented by the area under the distribution curves, remains the same. This significant distribution gap is what we define as 'coverage narrowing."
G.5. How Does the DSIR Work?
DSIR (Xie et al., 2023) works by estimating importance weights for each data sample to measure its relevance to the target distribution. This involves three main steps: first, we leverage n-gram models to estimate two distributions of human and synthetic data, ( and Pfeat. which represent the target and raw distributions, respectively. We use them to compute the likelihood ratio for each sample. Next, calculate the importance weight for each sample 2; w; w, quantifies how well the sample aligns with the target distribution. Finally, we perform without replacement to select examples, ensuring that the selected data is more representative of the target distribution.
Pea (+.) Aew (z,) The weight
sampling
We use DSIR in our data analysis as it allows for principled and computationally efficient selection of synthetic data points that align with the target distribution. Moreover, the importance weight also reflects the alignment between the n-gram features of synthetic data and human data. Using DSIR, we can analyze the differences between synthetic and human data across n-gram feature distributions and data matching. As shown Figure 11, is challenging to select synthetic data that matches human data characteristics under the significant distribution difference. To obtain high-quality synthetic data, it essential focus on improving the data synthesis methods.
G.6. Non-autoregressive Token Replacement May Compromise Text Coherence.
When designing data synthesis algorithms, we must balance synthesis efficiency and effectiveness, considering both autoregressive and non-autoregressive approaches. Autoregressive methods leverage the inherent capabilities of language models to generate coherent text sequentially. In contrast, non-autoregressive methods resample individual tokens based on their probability distributions. Since data synthesis is a prerequisite for model training, we aim to ensure that the cost of data synthesis does not exceed the cost of training itself.
Specifically, our ToEdit modifies data using the probability distribution in a single forward pass. For instance, the generated sequence length is 1024, the computational cost of autoregressive methods would 1024 times higher than ours. This efficiency advantage is why our method can run effectively on GPUs like the 3090 or 4090 series.
However, this efficiency may come the cost of coherence, resampled tokens may not fit seamlessly into given sentence. To address this issue, we introduce hyperparameter, resampling probability p, control the resampling threshold. We perform sampling in high-probability regions, focusing on tokens that are relatively easier to predict. We manually verify and tune on a small validation set before applying it across all experiments. In our experiments, we set p = 0.99.
Additionally, we supplement more experiments and discussion about peper-parameter As Table 5 shows, different values of p influence BioMed performance, leading to fluctuations in data quality. Table 6 presents the distribution percentages of the token probabilities across different value ranges. We need to refine the data while primarily preserving the source distribution. A larger p indicates fewer tokens will be resampled, while smaller p results in more tokens being resampled. Balancing performance and the preservation of data distribution, we set p 0.99 as the threshold for our experiments.
G.7. Must We Assume the Data is 100% Human-authored?
We do not need to assume that the data is 100% human authored; In experimental settings, some datasets used in experiments include partially synthetic data:
Datasets used in continual pretraining (e.g., Biomed, Finance) include partially synthetic data, which has been reformatted into a reading comprehension structure (Cheng et al., 2024b).
OSS InStIInstttt--55 and Evol-Instruct-I 10K also contain samples synthesized by ChatGPT.
In the theoretical framework, synthetic data is generated iteratively through an n-generation process. (1) If the starting point a real distribution, our method preserves most of the initial distribution to generate higher-quality data. (2) If the starting
23

---

How to Synthesize Text Data without Model Collapse?
Table 17. PPL. results of GPT-2 124M pretraining on pure Human Synthetic data.
Data Type
Human Data (Dolma)
Synthetic Data (Cosmopediaa)
Tokens Size
8.4B
16.8B
25.2B
33.6B
42B
8.4B
16.8B
25.2B
33.6B
42B
Epochs
3
3
5
Wikitext- 103
43.62
38.57
36.11
34.89
34.55
169.38
147.73
135.23
131.78
128.05
RedPajama
40.18
35.84
33.97
32.74
32.34
116.37
103.25
99.27
96.81
96.03
Falcon-RefinedWee
54.85
49.10
46.93
45.43
44,90
146.97
132.60
127.68
124.32
122.69
c4-en
45.87
41.00
39.10
37.95
37.56
128.25
114.41
109.73
107.53
106.55
mc4-en
61.00
54.44
52.11
50.38
49.74
171.44
153.70
150.28
145.44
144.99
point is mixture of synthetic and real data, the modifications are minimal, ensuring the original distribution remains largely unaffected. Therefore, applying our method in any generation i, we can further avoid issues, such as reduced variance and diminished diversity, which are key factors contributing to model collapse.
In other words, whether the current data is fully real or a mix of real and synthetic, using it as anchor data to synthesize data, our method builds upon the current data distribution achieve improvements, rather than causing model collapse.
In summary, we aim improve the data synthesis method, specifically focusing how obtain higher-quality data from the existing datasets. We do not need to assume that the data at hand is 100% human-generated. Our algorithm is designed to minimize excessive distribution truncation of the original data.
G.8. Can ToEdit Help with Already Strongly Collapsed Data or is Minimum Quality of the Data Necessary?
The ToEdit algorithm was initially designed to preserve the long-tail distribution during the data generation process, thereby avoiding model collapse. For already collapsed data, the variance is typically very small, and enhancing diversity is crucial. We can also adjust the threshold to introduce more randomness in data. Through this operation, we can inject randomness into the collapsed data. However, this a theoretical scenario, and as we know, data situations are highly complex. In practice, there will be many more challenges to address.
G.9. Could We Generate Tons of Data Using LLMs and Select Only the Long-Tail Ones?
For now, generating long-tail samples is currently difficult for language models. The reason lies in the sampling strategy of LLMs. Current LLMS adopt top-p, top-k or other sampling strategy for better performance. However, these sampling strategy will lead cut-off distribution. When the data synthesizing scale up. this drawback will finally scaling law cut-off on synthetic data. However, human corpus data follows Zipf distribution. The truncated output distribution causes the LLMs to nearly fail to sample long-tail samples. In other words, it is currently difficult to induce long-tail samples from LLMs that are as diverse human data.
On the other hand, if we force the language model to generate long-tail samples, these may contain both noisy and high- information samples, which are like two sides a coin, both distributed in the long tail of the data. This necessitates further filtering of the high-information samples. Unfortunately, such samples are challenging to automatically identify in practice and may require extensive human annotation.
H. Potential Applications and Future Work
Based on the above discussion, our approach can be applied to optimize the current data, even it is mixture of real and synthetic data. From the findings and proposed method in our paper, we can influence future research in the following aspects:
Potential applications of our work: (1) Data optimizations. We can quickly modify and optimize the current data, using a trained language model with a single forward pass. (2) Regularization the data synthesizing process. When synthetic data becomes excessive, we can introduce real data as an anchor to balance the issues excessive homogeneity and tail distribution cut-off in synthetic data, thereby preventing mode collapse.
24

---

How to Synthesize Text Data without Model Collapse?
0.22
luman Teats Poes
2,000
Synthetle Tatte Fmibe
Proze
0.20
1.605
0222
Probabitty
0.23
0.20
10000
Hash Bucket Indox
HosT Bucket Indox
Hosh Bucket Index
Figure 11. Uni/Bi-gram feature distribution across 10,000 hash buckets.
Lessons from our work: The key to improving the quality of synthetic data lies in balancing long-tail distribution preservation and optimizing synthetic data approaches. In other words, should focus two questions: how generate more informative synthetic data and how to integrate it with real data effectively. Building on this foundation, future improvements can focus on two aspects: first, obtaining more information gain by designing more efficient generation mechanisms to inject valuable information into the synthetic data; and second, optimizing methods to reduce noise during the synthesis process. This approach ensures that synthetic data retains its authenticity while enhancing its utility in practical tasks.
Extended related applications A broader range of recent works across domains also explore synthetic data generation and usage for diverse applications (Bai al., 2025; 2024; 2023: Jia et al., 2024; Yang et al., 2023; Li al., 2020; Niu et al.,
2024; Zhang et al., 2023; Li al., 2024; Pu et al., 2024; Li al., 2023; Zhu al., 2024a;b; 2022).
Top 40 2-grams Distribution Across Dalasels
Dataset
DSIR Sampled 1M
50000
Dolma 1M
Cosmopedia 1M
40000
Count
30000
20000
and
the
tne
me
The
the
WB
This
OR the
R
buf
you
for the wwhch and
But
Waey
B8
We
with the
However
Figure 12. The 40 bi-grams from separately sampled IM subsets of Dolma, Cosmopedia, and DSIR-selected datasets.
25

---

How to Synthesize Text Data without Model Collapse?
Top 2-grams Distribution Datasets
Dataset
DSIR Sampled 1M
50000
Delma
Cosmapedia 1M
40000
Count
30000
20000
10000
of the
Thre
the Thie, on
but
Bia whicls and t!g! Uhep-
uith the such an
at tife
from the Howwykhi Howera
therie
into Wille
Prr
mha the
be
Figure 13, The top 64 bi-grams from separately sampled 1M subsets of Dolma, Cosmopedia, and DSIR- Seleetele datasets.
Table 18. Dolma dataset statisties (v1.6), quoted from source (Soldaini et al., 2024).
Source
Doe Type
UTF-8 bytes (GB)
Documents (millions)
Unicode words (billions)
Llama tokens (billions)
Common Crawl
web pages
9,022
3,370
1,775
2.281
The Stack
code
1.043
210
260
411
web pages
198
Reddit
social media
339
377
72
89
PeS2o
STEM papers
268
38.8
50
70
Project Gutenberg
books
20.4
0.056
6.0
Wikipedia, Wikibooks
encyclopedic
16.2
6.2
3,7
4.3
Total
11,519
4,367
2,318
3,059
Distriburion of Festure Volues
3.0
0.2
0.
3.5
18
12
3.A
1.0
Figure 14. Density sampling response values. This result further confirms the issuc of feature collapse in synthetic data.
26


Reviews:
[{"review_id": "vIDlzxbdrd", "reviewer": "Reviewer_KJd9", "summary": "The issue that is addressed by this paper, using synthetic data during pretraining, is a very important and timely one. Going forward, pretraining will use a higher, and eventually dominant, proportion of synthetic data. The main findings are in 3.2, the three failure modes of Cosmopedia, when evaluated using perplexity as the main metric. These failure modes teach us how to inspect synthetic data to find out what's wrong, and inspirs ways to adjust the synthetic data generation pipeline. For this reviewer, the token level editing method is interesting, but not as important as the identification of the failure modes. Better to address the problem at the source, not after the synthetic data are generated.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your critical feedback and valuable suggestions. Below, we will strive to address your concerns and refine our paper accordingly.\n\n> [Q1] : I think the problem with \"synthetic data\" is with a particular synthetic data (Cosmopedia) only. This paper can lead to a false impression that all synthetic data suffer from the same issues.\n\nWe would like to clarify that our analysis is not meant to imply that all synthetic datasets suffer from the same issues. The analysis and theoretical validation are grounded in previous research on model collapse [1,2,3,4], a potential risk associated with training using synthetic datasets. Specifically, we use Cosmopedia as a representative example to highlight these concerns. \n\nTo avoid any potential misunderstanding, we will include the following explanation and clarification in the revision:\n\n1. Cosmopedia is currently the largest open-source synthetic dataset, and it comes with detailed statistical information.  We use Cosmopedia as a representative example to illustrate failure modes when using synthetic data. We agree that issues identified may vary depending on the specific dataset or generation pipeline used.  However, the results obtained on Cosmopedia should also have a certain level of representativeness.\n2. While our analysis primarily focuses on Cosmopedia, we do not intend to imply that all synthetic datasets suffer from identical problems. Rather, we aim to highlight general principles and cautionary lessons about synthetic data generation and evaluation.\n3. The tremendous success of synthetic data does not conflict with potential issues, such as model collapse. We provide several examples of the success of synthetic data in the Introduction and Related Work. In Line 33-36 and 799-812, there are listed numerous high-quality synthetic datasets, such as UltraChat, UltraMedical, and so on. Furthermore, famous Phi-1/2 series models are basiclly on synthetic data.\n\n[1] AI models collapse when trained on recursively generated data. *Nature* 631, 755–759 (2024).\n\n[2] A tale of tails: model collapse as a change of scaling laws. ICML'24\n\n[3] Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating, arXiv, 2024\n\n[4]  Position: Model Collapse Does Not Mean What You Think. 2025.", "claims_and_evidence": "I think the problem with \"synthetic data\" is with a particular synthetic data (Cosmopedia) only. This paper can lead to a false impression that all synthetic data suffer from the same issues.", "methods_and_evaluation_criteria": "Yes they make sense", "theoretical_claims": "I apologize that I did not have time to check the equations. The ideas are sound and practice-able, so I didn't check.", "experimental_designs_or_analyses": "They are sound", "relation_to_broader_scientific_literature": "No comment", "essential_references_not_discussed": "I don't know of any", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "No questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dfJbHgo44f", "reviewer": "Reviewer_3Snu", "summary": "This paper investigates the issue of model collapse. Model collapse happens when training models on synthetic data cause performance degradations or, in some scenarios, complete model breakdown. The authors discuss the negative correlation between the proportion of synthetic data and model performance, even without iterative training. \n\nTo understand this decline, the authors perform statistical analyses revealing that synthetic data suffers from distribution narrowing and over-concentration of n-gram features. As a solution, instead of focusing on synthetic data generation, they propose token-level editing on human-produced data to generate semi-synthetic data. They utilize a pretrained LLM to find the 'easy' data points, then edit them for better final training performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. In the following, we will address your concerns accordingly.\n\n> [Q1] : References Not Discussed\n\nWe will include a discussion on all the provided references as follows:\n\n[1] develops a rigorous framework to demonstrate the importance of real data in maintaining the stability of iterative training. [2] theoretically demonstrates that the impact of data curation can be formalized as an implicit preference optimization mechanism. [3] reveals the detailed training dynamics of model collapse under three different training workflows. Of course, there are also some remarkable studies that successfully used synthetic data. [4] proposes the Self-Instruct data generation framework, enhancing instruction-following capabilities. [5] employs the self-talk method to generate high-quality data. ReST [6] uses a policy model to generate datasets and then employs offline RL to fine-tune LLMs on generated datasets. [7] demonstrates that self-training with binary feedback filtering can reduce reliance on real data.\n\n[1] On the stability of iterative retraining of generative models on their own data.\n\n[2] Self-consuming generative models with curated data provably optimize human preferences.\n\n[3] Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World.\n\n[4] Self-instruct: Aligning language models with self-generated instructions.\n\n[5] Ulmer, Dennis, et al. \"Bootstrapping llm-based task-oriented dialogue agents via self-talk.\n\n[6] Gulcehre, Caglar, et al. \"Reinforced self-training (rest) for language modeling.\n\n[7] Beyond human data: Scaling self-training for problem-solving with language models. \n\n>  [Q2] : The first finding is already established information.\n\nWe agree with you that the potential for synthetic data to cause model collapse has been demonstrated in prior notable works [8]. However, our contribution of statistical  experiments provides a more fine-grained analysis of textual data features, specifically examining the underlying reasons behind the failure of synthetic data, e.g., overconcentration of n-grams.\n\n[8] Position: Model Collapse Does Not Mean What You Think\n\n> [Q3] : The authors focus on improving the quality of the real-data training via editing.\n\nYes, we agree with you on this point. Based on insights from prior works [1], model collapse is associated with multiple factors, including self-generated processes, data quality, and others. Our proposed method is inspired by the statistical findings on synthetic data (as stated in Sec. 3). Therefore, we attempt to prevent model collapse by improving data quality.\n\nFurthermore, we will clarify in the paper that we are not directly addressing model collapse, but rather indirectly preventing it by improving data quality. Additionally, in our experiments, the data we edited is not entirely real data. We also conduct experiments with synthetic data, such as (1) datasets used in continual pretraining (e.g., Biomed, Finance) , (2) OSS-Instruct-75K and Evol-Instruct-110K also contain samples synthesized by ChatGPT. \n\nPlease refer to  Appendix G, Q7 for more discussion.\n\n[9] Instruction Pre-Training: Language Models are Supervised Multitask Learners (Cheng et al., EMNLP 2024)\n\n>  [Q4] : The authors should elaborate further on the results.\n\nAs shown in Table 3, our method improves performance across both OLMo-1B and Llama-3-8B in successful tasks. In Biomedicine, OLMo-1B’s average score improves from 38.83 to 40.89, and Llama-3-8B from 56.04 to 56.48. Similar improvements are seen in Finance and Math domains. Additionally, Table 2 shows the effectiveness of our approach in general pre-training, with OLMo-1B’s average performance rising from 32.75 to 33.11. In Table 4 (SFT), ToEdit enhances FLAN v2 from 70.18 to 70.65 and boosts task performance in Natural Instructions. However, in more challenging tasks, such as Math, the improvements are limited or negligible. This indicate that data modification alone has limited impact on harder reasoning tasks.\n\nPlease refer to Appendix C for detailed discussion. \n\n> [Q5] :\n>\n> 1. Some of the experiment settings are missing.  \n> 2. The information about the details of the ToEdit experiments is missing.\n\nPlease refer to:\n\n1. Appendix F (Line 412) for detailed experimental settings, including pre-training and fine-tuning.  \n2. Section 5.1 (Line 381) for the ToEdit experiment settings and cost.\n\nAdditionally, we add iterative experiments on instruction tuning data in Rebuttal to Reviewer Qb5k.\n\n> [Q6] : How does the editing mechanism ensure the quality of the semi-synthetic data?\n\nWe conduct pre-ablation on experiments on throhold $p$ to control the editing process and ensure quality of output data. As shown in Table 5, we choose best parameters for main experiments.\n\n> [Q7] : How are algorithm metrics selected?\n\nWe select our evaluation metrics based on established practices. Please refer to Appendix F.2.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I've checked the theoretical results of the main papers and appendix, and they seem correct.", "experimental_designs_or_analyses": "Yes, this is explained further in the weakness and question sections.", "relation_to_broader_scientific_literature": "The underlying problem, model collapse, is critical since real data has become more scarce, and models do need high-quality synthetic data in their training. Some of the findings of the paper regarding the characteristics of synthetic data are especially important in guiding better synthetic data generation methods. However, the proposed method does not do much about solving the model collapse. Instead, the authors focus on improving and enhancing the quality of the real-data training via editing.", "essential_references_not_discussed": "Papers showing the model collapse phenomena: \n\n[1] Bertrand, Quentin, et al. \"On the stability of iterative retraining of generative models on their own data.\" ICLR 2024.\n\n[2] Ferbach, Damien, et al. \"Self-consuming generative models with curated data provably optimize human preferences.\" NeurIPS 2024.\n\n[3] Kazdan, Joshua, et al. \"Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World.\" ICML 2024.\n\nSince this paper shows a method to alleviate model collapse, it should also mention some of the known methods that have successfully generated useful synthetic data. [some of them are listed below]\n\n[1] Wang, Yizhong, et al. \"Self-instruct: Aligning language models with self-generated instructions.\" ACL 2023.\n\n[2] Ulmer, Dennis, et al. \"Bootstrapping llm-based task-oriented dialogue agents via self-talk.\" arXiv preprint arXiv:2401.05033 (2024).\n\n[3] Gulcehre, Caglar, et al. \"Reinforced self-training (rest) for language modeling.\" arXiv preprint arXiv:2308.08998 (2023).\n\n[4] Singh, Avi, et al. \"Beyond human data: Scaling self-training for problem-solving with language models.\" TMLR 2024\n\netc.", "other_strengths_and_weaknesses": "Strength: \n\n1- The statistical findings about the differences between synthetic and real data are interesting. \n\n2- The problem description and observations are well-written\n\n3- The paper presents the importance of the problem in various settings and experiments. \n\nWeakness:\n1- The first finding is already established information, and multiple prior works have observed it before. \n\n2- The proposed method is not a direct solution for model collapse but focuses on improving the quality of real data. \n\n3-  Some of the improvements in Tables 2 and 3 are minor. The authors should elaborate further on the results, as they are currently show that the proposed method does not help in several tasks.\n\n4- Some of the experiment settings are missing (for example the training parameters for pertaining and fine-tuning).\n\n5- The information about details of ToEdit experiments is missing. How many iterations are required to generate the semi-synthetic data? What are the computational costs?", "other_comments_or_suggestions": "Please check out the weakness, question, and citation sections.", "questions_for_authors": "1- How does the editing mechanism ensure the quality of the semi-synthetic data? \n\n2- How are algorithm meters selected?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cBMzSTKKbI", "reviewer": "Reviewer_Qb5k", "summary": "The paper is twofold: The first part of the paper focuses on the effects of mixing real and synthetic data and what the authors call non-iterative model collapse. The second part of the paper proposes ToEdit, a method to adjust synthetic text data by resampling those tokens that have high probability to be generated. They argue that this method can prevent the negative effects of synthetic text data.\n\n### update after rebuttal ###\n\nAfter reading the other reviews and corresponding rebuttals, I think the authors addressed most of my (and the other reviewers) concerns. Therefore, I think this paper can be accepted and I will raise my score. I still suggest that the authors spend some more space on explaining their method to make it easier for the reader.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your enthusiastic feedback. Below, we give detailed responses to your questions.  \n\n> [Q1] : References Not Discussed\n\nWe will include a discussion in the revised version as follows:\n\n[1] demonstrates that without enough fresh real images, future generative models will gradually decline. [2] develops a rigorous framework to demonstrate the importance of real data in maintaining the stability of iterative training. [3] illustrates that real data in the iterative training process can slow the decline of LLMs, but cannot fully prevent it. [4] shows that the quality and diversity of generated images degrade over time.\n\n[1] Self-Consuming Generative Models Go MAD.\n\n[2] On the Stability of Iterative Retraining of Generative Models on their own Data. \n\n[3] Large language models suffer from their own output: An analysis of the self-consuming training loop.\n\n[4] Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet. \n\n> [Q2] : Maybe a small example to illustrate their method.\n\nWe add an input-output example below. The complete workflow is also provided in Lines.\n\nCase 1: code reasoning sample in Magicoder-Evol-Instruct-110K:\n\n| **Before** (source)                                          | **After** (edited)                                           |                        |\n| ------------------------------------------------------------ | ------------------------------------------------------------ | ---------------------- |\n| Construct a function using PHP language that applies lexical analysis on a provided text string to analyze the individual, non-repeated words elements present. | Construct a function using PHP language that applies lexical analysis on a provided text string to quantify unique words. | \"analyze\" → \"quantify\" |\n| Test with provided string, `$str = 'Greetings, Planet Earth!'`. | Test with provided string, `$str = 'Greetings, Planet Earth!'`. | No changes.            |\n| Implements `wordCount` to remove punctuation, convert text to lowercase, split into words, and count unique words. | Implements `wordCount` to remove punctuation, convert text to lowercase, split into words, and calculate unique words. | \"count\" → \"calculate\"  |\n| Returns `{'greetings': 1, 'planet': 1, 'earth': 1}`.         | Returns `{'greetings': 1, 'planet': 1, 'earth': 1}`.         | No changes.            |\n\n> [Q3] : Does ToEdit change properties of data in a desired way ?\n\nYes, we present the statistical analysis of edited data below. **As shown in Supplemented Tables 1 and 2, ToEdit meets our expectations by preserving the original long-tail distribution.** Additionally, Table 14 (Line 1100) illustrates that tokens above the threshold gradually decrease as the iterations progress.\n\n1. KL Divergence Between Distributions (gen_0, gen_1, gen_2)\n\n| Distribution Comparison | gen_0   | gen_1   | gen_2   |\n| ----------------------- | ------- | ------- | ------- |\n| **gen_0**               | 0       | 5.56e-6 | 1.34e-5 |\n| **gen_1**               | 5.56e-6 | 0       | 9.61e-6 |\n| **gen_2**               | 1.34e-5 | 9.61e-6 | 0       |\n\n2. Sample Distribution Across PPL Intervals (gen_0, gen_1, gen_2)\n\n| PPL Interval   | gen_0 | gen_1 | gen_2 |\n| :------------- | ----: | ----: | ----: |\n| 6.04～25.43    | 49087 | 49101 | 49132 |\n| 25.43～44.83   | 42548 | 42532 | 42510 |\n| 44.83～64.23   |  5147 |  5149 |  5148 |\n| 64.23～83.62   |  1993 |  1993 |  1984 |\n| 83.62～103.02  |   762 |   763 |   763 |\n| 103.02～122.42 |   339 |   338 |   340 |\n| 122.42～141.81 |    98 |    98 |    95 |\n| 141.81～161.21 |    18 |    18 |    19 |\n| 161.21～180.6  |     4 |     4 |     4 |\n| 180.6～200.0   |     4 |     4 |     4 |\n\n> [Q4] : Does ToEdit help in an iterative process?\n\nYes, the following  supplemented table shows effectiveness in an iterative process, with slight improvements in performance across generations.\n\n3.  Performance in an iterative process on Instruction tuning data.\n\n|       | PIQA  | BoolQ | HS    | SIQA  | WG    | Avg       |\n| ----- | ----- | ----- | ----- | ----- | ----- | --------- |\n| Gen 0 | 79.87 | 81.28 | 59.72 | 49.69 | 74.51 | **69.01** |\n| Gen 1 | 80.25 | 81.16 | 59.74 | 50.56 | 74.59 | **69.26** |\n| Gen 2 | 80.14 | 82.69 | 59.82 | 50.51 | 73.80 | **69.39** |\n\n> [Q5] :  Can ToEdit help with already strongly collapsed data or is a minimum quality of the data necessary?\n\nThis is a very interesting and insightful question. The ToEdit algorithm was initially designed to preserve the long-tail distribution during the data generation process, thereby avoiding model collapse. For already collapsed data, the variance is typically very small, and enhancing diversity is crucial. We can also adjust the threshold $p$ to introduce more randomness in data.  Through this operation, we can inject randomness into the collapsed data. However, this is a theoretical scenario, and as we know, data situations are highly complex. In practice, there will be many more challenges to address.", "claims_and_evidence": "The claims of the paper are supported by sound experiments and theoretical analysis.", "methods_and_evaluation_criteria": "The methods to validate their ToEdit approach seem fitting and convincing.", "theoretical_claims": "The authors provide theoretical evidence for their ToEdit method. However, I did not check the extend proof in the Appendix for correctness.", "experimental_designs_or_analyses": "The experimental design seems fitting.  Nevertheless, I have some further questions (see below).", "relation_to_broader_scientific_literature": "The first part about non-iterative model collapse is nothing else than a single iteration of model collapse like it happens in the real world. The authors claim that non-iterative model collapse is different as it is not data generated by the same model as in the related work. I would argue that this is just an experimental choice for tractability in other papers.\n\nThe main contribution to literature is the ToEdit approach to help mitigate model collapse for textual data.", "essential_references_not_discussed": "There are several other studies on model collapse both theoretical and empirical. I suggest that the authors extend their related work section to give proper credit.\n\nFor example, but not limited to:\n\nAlemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., ... & Baraniuk, R. Self-Consuming Generative Models Go MAD. In The Twelfth International Conference on Learning Representations. (2024)\n\nBertrand, Q., Bose, J., Duplessis, A., Jiralerspong, M., & Gidel, G. On the Stability of Iterative Retraining of Generative Models on their own Data. In The Twelfth International Conference on Learning Representations. (2024)\n\nBriesch, M., Sobania, D., & Rothlauf, F. (2023). Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv preprint arXiv:2311.16822.\n\nMartínez, G., Watson, L., Reviriego, P., Hernández, J. A., Juarez, M., & Sarkar, R. (2023, August). Towards understanding the interplay of generative artificial intelligence and the internet. In International Workshop on Epistemic Uncertainty in Artificial Intelligence (pp. 59-73). Cham: Springer Nature Switzerland.", "other_strengths_and_weaknesses": "The ToEdit method is not trivial to understand on the first read of the paper. Maybe the authors could use some more space to better illustrate their method (as it is the core contribution) so the reader can easier understand the edit operations (maybe a small example can help here).", "other_comments_or_suggestions": "none", "questions_for_authors": "-\tQ1: Does ToEdit change the properties of synthetic data in a desired way as specified in the first part of the paper? Could the authors provide similar statistics as in section 3 for the edited data?\n-\tQ2: Does ToEdit help in an iterative process as well? If I understand it correctly the theoretical proof says so but there is no experimental evidence for that in the paper?\n-\tQ3: Can ToEdit help with already strongly collapsed data or is a minimum quality of the data necessary?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UH0q9q3rjm", "reviewer": "Reviewer_QsnQ", "summary": "This paper introduces a novel approach to generating semi-synthetic text data to address the issue of model collapse when trained with synthetic data. The method is supported by a solid theoretical framework under a simplified linear model setting. Extensive experiments validate the effectiveness of the approach, showing improvements in the stability of the training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive feedback and insightful comments. Below, we give detailed responses to your questions.  \n\n> [Q1] : The solution authors provided is to use semi-synthetic data instead, which seems deviated from what the title suggests.\n\nWe will revise the wording and provide further clarification to avoid misunderstandings. An example is provided below:\n\nBased on prior works, model collapse is related to multiple factors, including self-generated processes, data quality, and others. Our proposed method is inspired by the statistical analysis on synthetic data in Sec. 3. Specifically, we focus more on the model collapse phenomenon caused by data quality. Therefore, we try to improve data quality then further indirectly prevent model collapse. In other words, we are not directly addressing model collapse, and thereby indirectly prevent it by improving data quality.\n\nThese statements will be included in the Introduction and Related work sections to better clarify our method.\n\n> [Q2] : If the issue of synthetic data is the coverage and lack of long tails, could we generate tons of data using LLM and only preserve those long tail ones? Will this method be better than token editing because token editing still require human data?\n\n` 1. Could we generate tons of data using LLM and only preserve those long tail ones? `\n\nFor now, generating long-tail samples is currently difficult for language models. The reason lies in the sampling strategy of LLMs [1]. Current LLMs adopt top-p, top-k or other sampling strategy for better performance. However, these sampling strategy will lead to cut-off distribution. When the data synthesizing scale up, this drawback will finally scaling law cut-off on synthetic data. However, human corpus data follows a Zipf distribution, $ p_i \\propto i^{-\\beta}, \\quad i = 1, 2, \\dots $ [2]. The truncated output distribution causes the LLMs to nearly fail to sample long-tail samples. In other words, it is currently difficult to induce long-tail samples fromLLMs that are as diverse as human data.\n\nOn the other hand, if we force the language model to generate long-tail samples, these may contain both noisy and high-information samples, which are like two sides of a coin, both distributed in the long tail of the data [3]. This necessitates further filtering of the high-information samples. Unfortunately, such samples are challenging to automatically identify in practice and may require extensive human annotation [4].\n\n[1] Dohmatob E, Feng Y, Yang P, et al. A tale of tails: Model collapse as a change of scaling laws[J]. arXiv preprint arXiv:2402.07043, 2024.\n\n[2] Zipf G K. The psycho-biology of language: An introduction to dynamic philology[M]. Routledge, 2013.\n\n[3] Swayamdipta S, Schwartz R, Lourie N, et al. Dataset cartography: Mapping and diagnosing datasets with training dynamics[J]. arXiv preprint arXiv:2009.10795, 2020.\n\n[4] Lin Z, Gou Z, Gong Y, et al. Rho-1: Not all tokens are what you need[J]. arXiv preprint arXiv:2404.07965, 2024.\n\n` 2. Will this method be better than token editing because token editing still require human data?`\n\nYes, we agree with you on this idea. If we had sufficiently powerful LLMs, we could generate a large amount of long-tail data and quickly filter out the irrelevant ones, which would significantly boost the performance of current LLMs. However, these conditions are difficult to meet at present, and much work remains to be done.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I didn't check the proofs closely", "experimental_designs_or_analyses": "I checked the experiments- they look sound to me.", "relation_to_broader_scientific_literature": "According to my knowledge, the key contribution, token editing method, is novel.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The proposed token editing method to avoid model collapse is interesting. It is simple and useful. The experiments demonstrate the effectiveness of the method.\n\nHowever, the proposed method is generating semi-synthetic data rather than pure synthetic data. In this sense, it does not address the model collapse when training with pure synthetic data, which is a more important problem. Because the motivation of using synthetic data is for the situation where we ran out of human data; this semi-synthetic data still requires human data and is more like a data augmentation method, not a data synthesis method.", "other_comments_or_suggestions": "The title is \"How to Synthesize Text Data without Model Collapse?\", but the solution authors provided is to use semi-synthetic data instead, which seems deviated from what the title suggests.", "questions_for_authors": "If the issue of synthetic data is the coverage and lack of long tails, could we generate tons of data using LLM and only preserve those long tail ones? Will this method be better than token editing because token editing still require human data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel approach to text data synthesis that effectively addresses model collapse issues. The proposed token-level editing method demonstrates technical innovation and promising empirical results. For the camera-ready version, the authors should expand their experimentation across different dataset sizes and provide more comprehensive ablation studies to better demonstrate the method's advantages.
Decision: Accept (poster)