ToEdit: How to Synthesize Text Data to Avoid Model Collapse?

Authors: Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, Ermo Hua, Xingtai Lv, Ning Ding, Zhouhan Lin, Zilong Zheng, Bowen Zhou
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: synthetic data, model collapse

Abstract:
We explore model collapse caused by synthetic data, where AI models trained on such data experience a gradual decline in performance. 
Our initial analysis examines language model pretraining on mixed human and synthetic data, highlighting performance degradation. Further statistical analysis reveals distributional shifts and an over-concentration of n-gram features caused by synthetic data. Inspired by these insights, we propose token-level editing on human data, to obtain semi-synthetic data instead of fully using model outputs. As a proof of concept, we theoretically demonstrate that token-level editing can prevent model collapse, as the test error is constrained by a finite upper bound. We conducted extensive experiments on pretraining, continual pretraining, and supervised fine-tuning of language models. The results validate our theoretical proof that token-level editing improves data quality and enhances model performance.

Decision: reject
Reviewer scores: 3, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730278209847

thread_type:
official_review

<thread_object>
creator: reviewer_1bd4

<content>
summary:
The paper investigates the problem of model collapse when training with synthetic data, and provides theoretical and empirical evaluations for training under different mixtures of human and synthetic data. They introduce token-level editing, which is a straightforward technique to introduce noise to human-written data by resampling a token every time a prior LLM probability assigned to the next token is larger than p=0.99. Authors show that using token-level editing in pretraining/continual pretraining/supervised finetuning they can achieve comparable performance to using standard human-authored data across a number of downstream tasks.

strengths:
- Interesting setup with both theoretical and empirical evaluation.
- Empirical evaluation is extensive, covering many downstream tasks and (pretraining / continual pretraining) settings.
- Token-level editing works well in practice for continual pretraining setting, and does not damage learning for pretraining and SFT settings.

weaknesses:
- Overall, although results with continual pretraining seem good, I would add way more nuance to results with pretraining and SFT. Differences in results obtained are very narrow and oftentimes the proposed method is numerically worse (by a very small margin).
- I find that some important points are not well addressed, for instance how did authors choose the p=0.99 as their threshold for resampling with token-level editing and what happens if one varies this threshold. This is especially important in light of the positive-to-mixed results (in the reviewer's opinion) obtained in the experiments.

questions:
- In Figure 1, you introduce a lot of notation that you do not explain neither in a legend nor in the caption, like $\sigma$, $d$, $T$, $f_0$, $m_1$, $y_1$, $m_0$, $y_0$, etc. Please introduce all these terms in the Figure caption (or in a legend in the figure itself). If that will make things too complicated, simplify the figure by removing the notation.
- In lines 89-92, you briefly describe iterative and non-iterative model collapse. I think this needs more clear language / more detail. What are the key differences between the two? Where are they similar? Why is one more realistic than the other and why? Please focus on that.
- One stark difference in Fig.2 you do not discuss is a bimodal distribution of ppl on human data. Or is that an artifact from the binning strategy you were using (if any)?
- Not sure I fully agree with the discussion around Fig.2 (lines 190-198). I find the plots for 1M human texts and 1M synthetic texts very similar. Why did you choose to use Figures to motivate your work instead of using more objective metrics, like those that measure the similarity between probability distributions?
- I do not think that a line plot is the best setting for Fig.5B, you would probably be better off with a bar plot here. The lines hint at the wrong intuition that they have meaning, whereas only the dots have meaning.
- Applying token-level editing by using top-k sampling with an LLM as the prior when p>0.99 means you discard the tokens where the LLM is too certain. One important question here is how did you choose this number of 0.99, and what would happen if this number were different? How do we generalise any findings (or can propose any future work) based on your results?
- In line 211, you may want to explain how the DSIR sampling works and why you use it in your data analysis.
- You may have written this somewhere, but what percentage of the original tokens are edited (on average per dataset) when you use token-level editing with p=0.99?

Typos worth mentioning:
- Line 211: "important sampling"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658268518

thread_type:
official_review

<thread_object>
creator: reviewer_pcas

<content>
summary:
The paper explores the impact of synthetic data on language model training, particularly the phenomenon of model collapse. Model collapse occurs when models, trained extensively on synthetic data, experience performance degradation due to distributional shifts,  lack of full distributional range of human language, and over-concentrates certain linguistic features, leading to "coverage collapse." The authors propose a novel solution called Token-Level Editing  to counteract model collapse by editing human data at the token level, creating "semi-synthetic" data that retains critical human data distribution characteristics. They demonstrate, both theoretically and experimentally, that this method helps prevent model collapse by bounding test error within a finite range. Through extensive pretraining, continual pretraining, and supervised fine-tuning experiments, the results indicate that token-level editing improves the quality of synthetic data and enhances model performance across tasks.

strengths:
1. The paper is well written and consistently maintains the readers interest by presenting core findings around synthetic data issues and the proposed token editing approach initially and later go on to support those claims.

2. The paper presents a novel strategy for avoiding model collapse, a critical issue as human data becomes scarce and reliance on synthetic data grows. The proposed Token-Level Editing (ToEdit) approach offers a novel way to mitigate collapse by creating semi-synthetic data, modifying only certain tokens in human data instead of relying entirely on synthetic data. This targeted editing approach preserves key distributional characteristics, reducing the risk of collapse while leveraging the benefits of synthetic data.

3. The authors support their method with a robust theoretical framework, calculating bounded test error as evidence that model performance degradation can be prevented.

4. Empirical results are provided with experiments involving pretraining, continual pretraining, and fine-tuning stages, showing that token-level editing yields consistent performance improvements across diverse learning approaches and domains, such as biomedicine, finance, and mathematics.

weaknesses:
1. Token-Level Editing relies heavily on a pre-trained language model to estimate token probabilities, which may introduce biases based on the pre-existing model's characteristics. Did the authors try an ensemble of models with varied pretraining sources could be used to reduce single-model bias in the token-level editing process. This could enhance the diversity of the generated semi-synthetic data.

2. Although the study focuses on text, with recent LLMs heavily focused on solving tasks such as code generation,  I wonder if ToEdit would work on other data types (e.g., code, tabular data)? Changing tokens in code sequences might make the code prone to erros. It will be interesting to see experiments on the generalizability of ToEdit across different data domains and will help the community understand its broader applicability.

3. The experiments are limited to GPT-2 and OLMo models, which may not reflect the broader applicability of Token-Level Editing across diverse architectures. Testing on models like BERT (with bidirectional attention), T5 (with encoder-decoder architecture), and maybe more recent LLMs with sparse attention or mixture-of-experts layers could reveal whether ToEdit’s effectiveness is architecture-agnostic or if it needs adaptation for specific configurations.

questions:
Questions & Suggestions:
1. It would help if the paper provided further clarification on the specific threshold values used for token-level editing (e.g., the probability threshold $p=0.99$. Explaining why this threshold was chosen and whether it varies by dataset or model could add depth to the methodology section.

2. Although common, it might still be useful to expand that PPL stands for perplexity. It will be helpful for new researchers.

3. The paper uses terms like "non-iterative model collapse" and "coverage collapse." It would be helpful broader audience to better understand these terms if it was clearly defined.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670218480

thread_type:
official_review

<thread_object>
creator: reviewer_4xbz

<content>
summary:
The paper addresses the problem of model collapse in language models trained on synthetic data. The authors propose Token-Level Editing (ToEdit), a method that creates semi-synthetic data by selectively editing tokens based on their probability estimates from a pre-trained model. Key contributions include:

1. Demonstration of non-iterative model collapse when mixing synthetic and human data during pre-training
2. Statistical analysis showing synthetic data suffers from distribution coverage collapse and feature over-concentration
3. A token-level editing approach that theoretically prevents model collapse while improving data quality
4. Experimental validation across pre-training, continual pre-training, and supervised fine-tuning scenarios

strengths:
- Well-structured paper with clear progression
- The authors identify a critical issue in modern LLM training (model collapse with synthetic data)
- Provides thorough statistical analysis of synthetic data's limitations
- Demonstrates non-iterative model collapse, extending beyond previous work focusing on iterative collapse
- Proposed Method is computationally efficient (single forward pass)
- Tests across multiple training scenarios (pre-training, continual pre-training, fine-tuning)
- Uses various downstream tasks for validation

weaknesses:
1. Parameter Sensitivity: The paper mentions using a probability threshold p=0.99 for token resampling but doesn't explore how sensitive the results are to this parameter choice. There's no ablation study showing how different threshold values affect performance. The paper doesn't explore or justify why this specific threshold works best. No ablation studies are provided to show the impact of different threshold values

2. Token Resampling Strategy (Limited Justification):
- The choice of top-k sampling with k=8 seems arbitrary
- The paper mentions they tried other sampling strategies (top-p and rejection sampling) but doesn't provide comparative results

3. Progressive Decrease in Editing 
- The assumption that editing operations decrease with a fixed ratio η seems mathematically convenient but lacks empirical validation
- No real-world evidence is provided to support this pattern of decay

4. Distribution Analysis:
- The U-shaped distribution in Figure 6 needs better explanation of why it's a good indicator for token editing
- The "coverage collapse" phenomenon mentioned multiple times could be better defined and explained
- The relationship between "feature over-concentration" and model performance isn't clearly explained

4. Experimental Setup:
- Figure 5 shows results about "DSIR-Selected Data" but details about this selection process aren't well explained
- Details about the continual pre-training setup across different domains (Biomedicine, Finance, Math) are sparse

5. Results Interpretation:
- The improvements shown in Tables 1-3 could use more discussion about statistical significance
- 
- Compare different sampling strategies (top-k, top-p, nucleus sampling) with ablation studies

questions:
1. Can you please provide justification for choosing p =0.99 and k =8 
2. Can you please provide more evidences for Progressive Decrease in Editing 
3. Can you elaborate the relationship between perplexity improvements and actual model capabilities.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730847197725

thread_type:
official_review

<thread_object>
creator: reviewer_wnzi

<content>
summary:
This paper investigates the role of synthetic data for LM pretraining.

1. The main observation/finding is:

> The mixture of synthetic data and real data is worse than real data only, which is evaluated with perplexity. (trained with Dolma + x% of Cosmopedia, and eval with wikitext-103, Redpajama, RefineWeb, C4-en)

2. A new data synthesis method, TOEDIT, which resamples tokens with lower perplexity using a language model. The authors claim TOEDIT outperforms real data in continual pretraining and SFT scenarios.

strengths:
The analysis of Cosmopedia synthetic data is comprehensive and provides valuable insights for future research

The theoretical framework is interesting and well-developed

The paper addresses an important question in the field about the utility of synthetic data in pretraining

weaknesses:
**Major Concerns**

1. Significant Literature Gap and Contradictory Findings 
- The paper overlooks a crucial published work [1] that directly contradicts its main findings
- [1] demonstrates that mixing synthetic and real data improves perplexity across various domains in the Pile and enhances downstream task performance
- This fundamental contradiction needs to be addressed and explained

Figure2 in [1] shows that by mixing the synthetic data with the real C4 data, LM can get lower perplexity on various domain corpus of the Pile. [1] also shows the introduce of synthetic data gave better results on downstream tasks. However, this submission claims the mixture of data is worse than real data only, by evaluating the perplexity. (I suspect the evaluation is incorrect which lead to this contrast observation. My 2nd concern below will detail more)

2. Flawed Experimental Design for Main Claims 

The pretraining real data is Dolma, which consist of wikipedia, C4. Hence, the more real data you use, the lower perplexity on Wikitext-103 and C4-en you should expect. So it is domain mismatching problem, instead of saying synthetic data is not useful. 
On the other hand, the synthetic pretraining data Cosmpopedia was build from LLM's rephrase of web-crawled data, which means Cosmopedia's data distribution differs significantly from web-crawled data, making the comparison with RedPajama and RefineWeb potentially unfair.  I would suggest to use proper out of domain text for perplexity evaluation, and also introduce the downstream task evaluation.  I would suggest to follow the evaluation framework from [1] for better comparison

3. TOEDIT Methodology Concerns
- The non-autoregressive token replacement approach may compromise text coherence and grammatical correctness
- No discussion of how grammatical consistency is maintained during token resampling

4. Weak Empirical Results (Baseline performance is concerningly low)
- MMLU: baseline (26.56) vs. proposed (23.63) vs. random (25)
- WinoGrande and Hellaswag results near random guessing. Improvements are marginal (1.3, -0.08) and not statistically significant


**Minor Concern**

1.  The conclusion drawn from "75% words under 0.6 probability" (Lines 265-267) ignores fundamental linguistic entropy
- English typically has 10-11 bits per word entropy
- The observed probability distribution may be natural rather than indicating filtering potential


[1][Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://aclanthology.org/2024.acl-long.757) (Maini et al., ACL 2024)

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1732528993384

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal

comment:
Dear Reviewers,

Thank you once again for your valuable comments and suggestions on our paper. We have carefully revised our manuscript and conducted additional experiments based on your feedback. For the reviewers’ convenience, **we have highlighted the changes in the revised manuscript in blue.** We present a summary of the modifications as follows:

1. **More validation of findings (@Reviewer wNZi and 4XBZ):**  We have supplemented below experiments, which aligns with our original findings. Additionally, we provide a detailed comparison with other methods in Table 10 and Appendix F.7. 
   - 22 PPL validation sets and 7 downstream tasks on GPT-2 and OLMo models in Table 5 and 6.
2. **Ablation Study on Hyper-Parameter $p$ (@Reviewer wNZi, 4XBZ, PCAs, and 1Bd4)**: We have supplemented our analysis with (1) various values of $p$, (2) percentage statistics, (3) sampling strategies, and (4) sampling sizes, as detailed in Tables 4, 7, 8, 9, and Appendix E.
3. **Additional Empirical Results (@Reviewer wNZi, 1Bd4, 4XBZ, and PCAs):** We include 3 general instruction tuning tasks and 2 code reasoning SFT tasks in Table 3 to further validate our method.
4. **Domain Statistics for Dolma (@Reviewer wNZi):** We present the domain statistics for Dolma in Table 15, demonstrating that it is not a domain mismatch problem.
5. **Evidence of Progressive Decrease in Editing (@Reviewer 4XBZ and 1Bd4):** We have supplemented the evidence of the progressive decrease in editing, as presented in Table 11 and Appendix F.6.
6. **Explanation of U-Shape Distribution Filtering Potential (@Reviewer wNZi and 4XBZ):** We analyze the filtering potential of the U-shape distribution through the perspectives of information theory and model learning, as detailed in Appendix F.4.
7. **Explanation of Experimental Setup and DSIR (@Reviewer 4XBZ and PCAs):** We provide more detailed descriptions of the experimental setups in Appendix D and how DSIR works in Appendix F.3.
8. **Definitions of Non-Iterative Model Collapse and Coverage Collapse (@Reviewer PCAs and 1Bd4):** We provide comprehensive definitions of non-iterative model collapse and coverage collapse in Appendices F.1 and F.2, respectively.
9. **Improvements of Figure 1 (@Reviewer 1Bd4)**:  We have annotated all the processes and symbols at the bottom of Figure 1 to facilitate quick reference for readers.
10. **Impact on Future Work (@Reviewer 1Bd4):** We discuss the potential applications and influence on future research in Appendix G. And, we explain that our method needn't assume the initial data is 100% human-authored in Appendix F.8.

Finally, we sincerely thank all reviewers and ACs for their critical feedback and valuable suggestions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes to mitigate model collapse by creating "semi-synthetic" data. ToEdit resamples tokens in human data based on probabilities from a pre-trained model, with a threshold parameter controlling the extent of modification. The paper provides a multi-faceted investigation, including statistical analysis of synthetic data, a theoretical framework for ToEdit, and empirical evaluation. 

Reviewer wNZi highlighted a critical flaw in the experiments presented in Figure 2, arguing that the observed performance degradation could be attributed to domain mismatch rather than the inherent properties of synthetic data. The authors addressed this by adding more downstream evaluations. The rationale behind the final parameter selection could be more thoroughly explained in the revised version. Reviewer 1Bd4 also expressed concerns about the practical implications of the paper's findings, arguing that the assumption of having 100% human-authored data to start with limits the applicability of ToEdit in real-world scenarios where data is often a mix of human and synthetic content. A more detailed explanation of the DSIR sampling method and its role in the data analysis can be added.

While the paper focuses on a timely problem, the reviewers argue that the paper has fundamental flaws, especially Reviewer wNZi with high confidence. Considering the weaknesses detract from the paper's overall quality and impact, the submission can be improved by overcoming the critical concerns raised by reviewers.