Reformulation for Pretraining Data Augmentation
Authors: Xintong Hao
Rui-Jie Zhu
Ge Zhang
Ke Shen
Chenggang Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Data Augmentation
Synthetic Pretraining Data
Primary area: foundation or frontier models, including LLMs

Abstract: Despite the impressive capabilities of large language models across various tasks, their continued scaling is severely hampered not only by data scarcity but also by the performance degradation associated with excessive data repetition during training.
To overcome this critical bottleneck, we introduce the Massive Genre-Audience (MGA) reformulation method, a framework designed to augment corpora in a way that supports more effective model performance scaling. 
Instead of relying on complex, predefined seed systems, MGA systematically reformulates existing corpora into diverse, contextually-rich variations by adaptively generating genre-audience pairs. 
We present this framework and the resulting 770 billion token MGACorpus, created as a practical instantiation of our methodology.
We experimentally validate MGA's core benefits by demonstrating superior scaling properties, in terms of both model size and data budget, against data repetition and upsampling (up to 13B parameters).
Furthermore, our comprehensive analysis investigates the role of synthesis principles in generation quality and reveals nuances in evaluating model capabilities using standard loss metrics.
Our work shows that a systematic framework like MGA provides a reliable pathway to substantially augment training datasets, effectively alleviating repetition bottlenecks and enabling more efficient scaling of large language models.

Published as a conference paper at ICLR 2026
REFORMULATION FOR PRETRAINING DATA AUGMEN- TATION
Xintong Hao', Ruijie Zhu²; Ge Zhang', Ke Shen' Chenggang Li!
BytteDncce Seed University of Califormia, Santa Cruz
(hacxintong, shenke) (henke)@byteddanl abyeeancceeeeeeceee...aa
ABSTRACT
Despite the impressive capabilities of large language models across various tasks, their continued scaling is severely hampered not only by data scarcity but also by the performance degradation associated with excessive data repetition during training. To overcome this critical bottleneck,, we introduce the Massive Genre- Audience (MGA) reformulation method, a framework designed augment cor- pora in a way that supports more effective model performance scaling. Instead relying complex, predefined seed systems, MGA systematically reformulates existing corpora into diverse, contextually-rich variations by adaptively generat- ing genre-audience pairs. We present this framework and the resulting 770 billion token MGACorpus]) created as practical instantiation of our methodology. We experimentally validate MGA's core benefits by demonstrating superior scaling properties, in terms of both model size and data budget, against data repetition and upsampling (up to 13B parameters). Furthermore, our comprehensive anal- ysis investigates the role of synthesis principles in generation quality and reveals nuances in evaluating model capabilities using standard loss metrics. Our work shows that a systematic framework like MGA provides reliable pathway to sub- stantially augment training datasets, effectively alleviating repetition bottlenecks and enabling more efficient scaling of large language models.
INTRODUCTION
The remarkable success of Large Language Models (LLMs) heavily relies on the scale of model parameters and training data (Kaplan et al. [2200] Hoffmann al. (0022.. Scaling laws demonstrate that improvements in model performance are increasingly dependent on data quantity and quality. However, the growth rate of available natural language corpora significantly lags behind the increas- ing demand for training data (Villalobos et al. (022). While data repetition is a standard tool in traditional deep learning, it backfires in LLM pre-training, where it degrades performance and cre- ates a major scaling bottleneck. This raises a critical question: how can we fully utilize the potential of existing data in data-constrained situations?
Leveraging LLMs to synthesize high-quality training data has emerged frontier approach (Su cet al. 2024; Abdin et al.) 2024). theory, data synthesis can generate limitless training material, expanding datasets without the negative consequences of repetition. However, while the promise of synthetic data is clear, the specific methodologies, the 'how' behind successful large-scale data synthesis often remain opaque, existing as black-box processes within large industrial labs rather than as systematic, reproducible science. Many prevailing methods depend on large-scale models for generation, effectively creating "distillations" rather than data augmentations,
require phisticated, pre-defined seed curation systems (Abdin et al.) 2024 Ben Allal et al.) 2024). These dependencies introduce substantial computational bottlenecks and limit their accessibility and scal- ability for the broader research community.
In this work, we propose MGA (Massive Genre-Audience reformulation), transparent and prin- cipled framework designed to directly address the data repetition challenge by augmenting the raw
"Wor done ByteDance Seed
'Corresponding author
-tw.ee/eeeen--iieeeeeeeeee neweb-edu

---

Published as a conference paper at ICLR 2026
Stage
Stage 2
(Genre, Audience) generation
reformulation
SOWRCE TEXT
Story
cheaning
SLM-2
Piologue
Cyandia
cleaning
SLM-2
SLM-I
SLM-2
clarning
Generate pairs per inference pass
Reforrulsse target inferenice pass
Figure I: Overview of the MGA framework. Our framework expands the original corpus through two-stage synthesis followed by cleaning stage process. Each document is reformulated new documents, finally achieving a 3.9x token number expansion while maintaining diversity through adaptively generated (genre, audience) pairs.
text and creating more unique tokens. As illustrated in Figure I the MGA framework is efficiently implemented using a lightweight 3.3B MoE model. Crucially, avoids complex external seed sys- tems by adaptively generating diverse genre- audience pairs directly from input documents. This design makes the data generation process highly efficient and applicable to web-scale corpora.
However, proposing framework is only the first step. To establish its scientific merit and provide actionable insights for the community, deeper investigation is essential. We contend that thorough understanding requires answering three fundamental questions: 1) How does MGA reformulation complement existing synthetic data strategies? 2) What is the core mechanism, specifically the role of diversity, that drives its effectiveness in data-scarce scenarios (especially high repetition scenarios)? 3) Why does reformulation fundamentally benefit the model's 's learning process? By addressing these questions, our main contributions are:
We introduce the MGA framework, systematic and reproducible methodology for corpus reformulation. To validate framework and ensure full reproducibility, will release the MGACorpus (a 770B token dataset) and open-source all key artifacts, including prompts, tool- model finetuning data, and cleaning scripts. Our experiments demonstrate that models trained on MGACorpus significantly outperform those trained on the original corpus it expands upon.
We experimentally validate MGA's superior scaling properties in terms of both model size and data budget, revealing a widening performance gap over standard data repetition and upsampling and data budget (up to 700B tokens).
across wide range of model sizes 3377//1.B/7B/113.
We analyze synthetic data collapse from two key perspectives: first, we characterize how syn- thesis principles (manifested through prompt engineering) mitigate collapse, and second, reveal the limitations of validation loss as a collapse detection metric. This analysis provides key insights for future synthetic data optimization.
2 RELATED WORK
Data Curation While web-crawled data contains hundreds of trillions tokens, stringent quality filters typically remove the majority of this content. Popular datasets like C4, Gopher, Dolma, and RefinedWeb (Raffel et al.) 2020: Rae et al., 2021 Penedo et al.) 2023) Soldaini et al.) 2024) non-learned heuristics. More recently, aggressive model-based based and retrieval-based filtering has become prominent datasets like Fine Web- Edu (Penedo et al.) 2024), DCLM (Li et al., 2024). and FineFine Web (Zhang et al. 2024). Such heavy filtering results removal of over 90% of tokens,
2

---

Published as a conference paper at ICLR 2026
which has led some researchers focus on balancing accuracy and data quantity (Su et al. 2024). However, this does not alter the fact that the total amount of high-quality data remains limited.
Repetition Training Studies subset repetition training have revealed that model divergence tends to occur earlier as model parameters increase (Hernandez et al. 2022). For scenarios where entire datasets are repeated for training. limiting 4 epochs or fewer results in minimal efficiency degradation (Muennighoff' et al., 2023; Taylor et al. 2022). Researchers have explored regulariza- tion techniques to mitigate repetition degradation, but this highlights the critical need for careful hyperparameter tuning. For example, while some work shows that increasing weight decay can yield better metrics (Fang et al. 2025), the same technique can also destabilize training. In set of ablation studies, Xue et al. (2023) found that models with weight decay failed to converge, whereas using dropout proved to be an effective alternative. This sensitivity underscores the challenge of applying regularization in repetition scenarios. Overall, this topic remains understudied across dif- ferent hyper- paraaeearamesess data distributions, and repetition ratios.
Synthetie Pretraining Data Data synthesis for pretraining has rapidly evolved, with two primary approaches: seed-based synthesis and raw-text rephrasing. Seed-based methods, exemplified by Phi models (Abdin et al. 2024), Cosmopedia (Ben Allal et al. (222), use predefined seed systems and templates to precisely control the generated content. In contrast, rephrasing methods, such as WRAP (Maini al. 2024) and Nemotron-CC (Su et al.) 2024), rewrite existing web content into different formats. Recent state-of-the-art models have validated the effectiveness of rewriting at an unprecedented scale. The Kimi K2 model applied rewriting to knowledge data as part of its training corpus (Kimi et al. 2025), and Nemotron Nano 2 (NVIDIA et al. 2025) employs Qwen3- 30B-A3B (Yang et al.) 2025) for rewriting. Concurrently, recent analysis has begun to codify the high-level insights for success and highlight the importance of the "methodology' behind data syn- thesis (DatologyAl et al.) 2025). These efforts confirm that synthetic data is a key component in training frontier models.
While these works validate the general approach, they often do not provide the detailed information and ablation of a successful synthesis implementation. Our work bridges this gap by providing concrete instantiation of these principles. Inspired by Maini et al. (2024); Ge et al. (2024). our framework adaptively generates diverse 'Genre' and 'Audience' pairs from high-quality source text. This approach systematically enhances diversity in a scalable manner, without requiring complex external seed systems large-scale generator models.
MASSIVE GENRE-AUDIENCE REFORMULATION
The central challenge of data reformulation is balancing two competing goals: generating novel, diverse content (variance) while preserving the source document's core factual information (invariance). To resolve this tension, we introduce the Massive Genre-Audience (MGA) frame- work, a principled pipeline designed for systematic corpus expansion. The framework operates on our central principle of "Limited Consistency"', which seeks to balance stylistic diversity with factual fidelity. This approach is implemented efficiently using lightweight small language models (Tool SLMs) fine-tuned for specific sub-tasks, ensuring both quality and scalability.
3.1 LIMITED CONSISTENCY
We define "Limited Consisteney" as guiding principle that seeks to maximize the stylistic and structural variance of reformulated content while maintaining strict invariance of the source docu- ment's core factual information. This principle directly addresses the risk of generating data that is either too repetitive or factually incorrect. The primary mechanism for implementing this principle is through careful Prompt Engineering, which steers the generative process.
To identify an optimal balance, we explored the design space of PE strategies. Prompts that are too strict ('SLM-Strict') enforce high fidelity, leading to a distribution that closely mirrors the original corpus but lacks diversity. Conversely, prompts that are too relaxed ('SLM-Relaxed")) encourage excessive deviation, resulting in significant distributional shift and high risk of factual degrada- tion. Our final approach ( ('SLM-Basse is calibrated to strike a balance, expanding the original data distribution without losing topical coherence. The distinct distributional impacts of these strategies
3

---

Published as a conference paper at ICLR 2026
Original
Original Striet
Relawed
Figure 2: t-SNE visualization illustrating our "Limited Consistency" principle. Our Base PE strategy (left) achieves balanced expansion of the original data distribution, while the Strict (middle) and Relaxed (right) variants are overly conservative or deviate excessively, respectively.
A detailed quantitative analysis validating the superiority of our balanced approach is presented in the ablation studies in Section 4.3.2
visualized in Figure 2
3.2 FRAMEWORK IMPLEMENTATION
The MGA framework is operationalized as two-stage synthesis pipeline: maximizin stage for generating diverse directives, followed by an invariance-enforcing stage for controlled reformulation. Each stage powered by specialized Tool SLM, which is finetuned on task -pecifcec data generated by larger language mode This implementation choice is validated in Table 1 where our final Tool SLM achieves performance nearly identical to the original LLM labeler.
Table 1: Reformulation quality comparison between the Tool SLM and its LLM teacher. All outputs were scored on a 1-5 scale by the LLM itself to evaluate the SLM's alignment. Scoring reliability was verified via human-in-the-logg cross-checking, yielding alignment rate of over 90%.
Models
Total Examples
5
4
3
2
Rate 3)
Diff
Labeler LLM
15,355
4,120
7.143
3,034
661
214 93. 03.11%
Tool SLM
15,355
3,788
7,124
3,224
736 285
92.06%
-1.05%
Stage 1: Adaptive GA-Pair Generation. The primary objective of this stage is to maximize diversity by generating wide array of creative reformulation directives. Our choice of Genre- Audience (GA) pairs as the core mechanism is deliberate. While simple rephrasing can generate stylistic variants, it often lacks structured diversity. GA pairs provide robust framework for mean- ingful content adaptation:
Genre dictates the structural and stylistic format of the content (e.g., an analytical report, a step-by-step tutorial, blog post), controlling how information is organized and presented.
Audience defines the intended reader's 's profile (e.g., a university student, an industry expert, curious teenager), guiding the tone, vocabulary, and conceptual depth.
Crucially, MGA moves beyond using small, fixed set of styles. Instead, it adaptively generates multiple, contextually relevant GA pairs for each source document. To achieve this, we prompted the laheler I.LM to produce five distinct GA pairs and curated this data through rigorous rule-based validation process (c.g., validating JSON structure and pair count). This filtered dataset explicitly strategy, mitigating the risk of mode col-
trains the 'GA-SLM' execute lapse, where repeated sampling requests to a model can yield highly similar outputs.
Stage 2: Controlled Reformulation. This stage aims to balance variance and invariance, di- rectly implementing our "Limited Consistency" principle. The core design is finetuning strategy that, instead of narrowly optimizing for perfect outputs (c.g., a score of 5), relaxes the quality thresh- old to ensure high proportion of broadly acceptable generations (a score of 3 or higher).
Tool model training details presented in Appendix B
4

---

Published as a conference paper at ICLR 2026
To formalize this, let D be source document and G be a generated GA-pair. The teacher LLM first produces an initial set of synthetic reformulations Daynth = {(D, D)). However, training the Tool SLM directly on this full dataset would cause it to replicate the teacher's suboptimal outputs. To circumvent this, we leverage the teacher L.M as a quality judge, using its scoring function ((P)) E (I,...,5) filter for high-quality subset, DSrT:
SSFT = ((D,,, E Dayath SSD') 2 3}
The 'Reformulation-SL.. parameterized by 0, is then trained exclusively on this curated subset using a standard supervised fine-tuning (SFT) objective:
(srr(0) = DID.G. |- log Po(D'ID, G)]
This targeted alignment process imbues the 'ReformulationSLMM with the nuanced ability gen- erate novel content that remains faithful to the source material.
After the stage synthesis, final heuristic cleaning process is applied the generated corpus. This stage filters out high-frequency generative patterns (c.g., 'Please note that ...') and removes documents with extremely low keyword coverage with the source document, ensuring final data quality. Finally, we achieve 3.9x token expansion while maintaining high quality and diversity.
EXPERIMENTS
We now empirically validate the MGA framework. We begin by establishing its core effectiveness in data-constrained scaling scenarios (Section 4.2). Following this validation, we provide a deeper analysis by addressing three key research questions (Section 4.3):
RQI: How does MGA reformulation complement existing synthetic data strategies?
RQ2: What role does reformulation diversity play in high-repetition training?
RQ3: Why does reformulation fundamentally benefit the model's learning process?
4.1
SETUP
Models and Hyperparams The architecture of pretraining model follows that of Llama 3 (Dubey) al., 2024). Experiments across various sizes (134M/377M/117B) were running with Warmup- Decay Ir scheduler (Hu et al. 2024) where 0.1 % warmup steps, 75% stable and final 25%
decay phase. Detailed model specifications are provided in Appendix C.2
Datasets We build MGACorpus based on SmolLM-Corpus (Ben Allal et al. 2024), which con- tains four subsources (finwwb-eeuu-dedup) / cosmopedia/ / python-edu open-web--aane We refor- mulated the 195B tokens finewebeedu-dedup source and finally got 770B cleaned synthetic tokens.
Evaluation We evaluate the models on a comprehensive suite of benchmarks include ARC-Easy
/ Challenge (Clark et al. 20188), HcllaSwag (Zellers et al. 2019), Winogrande (Sakaguchi et al.) 2020), MMLU (Hendrycks et al., 2021b. GSMSK (Cobbe et al. 22221) 1). etc., following popular prac- tice LIGHTEvAL. (Fourrier et al. 2023) and LM-HARNESS (Gao et al. 2023). For the directly effectiveness validation of MGA, evaluate MGACorpus aligned with Sineweb/SmolLL/Coann For training dynamics, we report the average of 12 benchmarks and validation losses
pedia settings @
on held-out fineweb-edu-dedun dataset.
4.2 MAIN EXPERIMENTS
To directly evaluate MGA's potential as solution for data scarcity and repetition, present comprehensive analysis in two parts. First, we benchmark MGA's performance against recent SOTA small LMs to establish a comparative baseline. Subsequently, constrained scaling scenarios, specifically situations where the training budget exceeds the available unique high-quality data, a common limitation in practical applications.
investigate its behavior under data-
https: / / thub. com //mmg/gggii. ng face / www.oppdii....ccccooooo a/bl ob/ wb/main//ve n/eva luat i on Details on data recipes and comparisons with other models provided in Appendix and D.1
5

---

Published as a conference paper at ICLR 2026
Table 2: Benchmark MGA with SOTA SmolLM series. Models of similar size are grouped. All
results are obtained via LILHTEVAL (Fourrier et al. 2023). The best result within each fair compar- ison is highlighted in green. Note that Sm models, trained with substantially more compute, are included for reference only.
Moddl
Paranns.
*Tokems ARC(C+E)
Wino.
Hella,
MMLU
MMLU-PRO
CSQA
OpenBaokQA
PIQA
TrivisQA
GSMSK
Avg.
SmolL 12-13M
135M
2T
44.12
$1.07
42.03
31.27
11.06
33.82
35
68.23
1.91
1.52
32.00
SmeLLM-133MM
135M
600B
42:47
$1.51
11.08
29.93
11.4
32.51
13.2
68.17
1.08
0.99
31.24
SmolL (ours)
134M
600B
41.21
52.41
40.69
30,03
11:37
34,32
35.4
67.85
0.02
1.29
31.51
MGA-Pxpansion
134M
603B
43.01
31.7
41.25
30.1
11.76
32.68
36.4
67.3
2.08
1:44
31.77
Mmm1LM2-366M
53.4
52.33
11.17
37.6
16.73
37.37
Mm6LLM-360M
360M
600B
49.99
52.96
51.67
33.84
11.42
34,81
37.6
71.87
2.27
1.97
34.84
M.30LM (ours)
377M
600B
48.57
52.64
33.63
11.25
36.77
39
0.29
1.52
34.57
MGA--Fxpansion
377M
600R
49.39
52.64
51:34
33409
11:35
57.1
38
72.31
7.28
1.74
35.52
SmelLM2-LL77 L.7B
1.7B
11T
60:42
59.59
68.73
41.4
19.61
43.65
12.6
77.53
36.68
29.04
47.93
SmoILM--L.BB
1.7B
IT
59,95
54.7
62.83
39.35
10.92
38
42.6
75.9
13.14
40.20
Smoll M-1.7B (ours)
1.78
TT
59.63
$7.38
65.19
39.4
12.11
42.59
45.6
76.88
4,95
7.81
41.15
MGA--Expunsio
57.46
14.1
42.8
20.42
43.4
Performance training on MGACorpus We evaluate whether incorporating MGA data enhances model performance compared to a baseline trained solely on the original sources, using fixed training budgets and model sizes ranging from 134M to I.7B. As shown in Table 2 MGA-Expansion shows consistent improvements across different model sizes, with larger performance gains model size increases, +0.26/40.95/42.11 for 134M/377M/1. 7B models respectively. Notably, MGA-Expansion achieved substantial gains in reasoning-intensive intensive tasks such TriviaQA 22.00//6..9//15447 and GSM8K ++0115/0..22/66.00) and shows strong performance on MMLUM/MMU-PPr We hypothe- size that MGA's data reformulation, by exposing the model diverse phrasings of the same underly- ing information, fosters more robust generalization. This enhanced generalization, in turn, improves the model's reasoning capabilities, leading to the results observed on these specific benchmarks.
Scaling Dynamics We further investigate MGA's behavior under data-constrained scaling scenar- ios. Models of 377M/1. TBMM/..B/BB//3D are trained using a learning rate scheduler with only warmup and stable phases, which allows for direct performance comparison across repetition epochs.
Benchmariss Avg.
FineWeb EDU Vall Lasses
Benchmarks Aug
FineWeb EDU Vol Lasses
0.55
0.55
0.50
0.45
0.45
0.40
22-
2.0
0.35
2.0
100
230
300
600
500
100
200
30C
430
200
400
600
200
420
600
of tolkens 181
of tokers (B)
of takens repwall hi = 450b ln citta 1.4 apocches with iq apply MSA reformddation get 2006 expersion
tokens
repasst high quality (nc] dltts 10 spachs
collect more
(106it) reckune repellioo apply MGA informdation get 2006 eccansion
upoammple
Figure 3: Training dynamics of two common scenarios under cata-constr conditions: (I) ex- panding 50B high-quality dataset to a 500B training budget (entire set repetition), (2) expanding a 500B -ixed-quality dataset to a 700B training budget (subset repetition). Data recipe and bench- mark details are provided in Appendix C.1 and
Scaling Results As shown in Figure MGA demonstrates favorable scaling properties with both data budget (D-scaling) and model sizes (N- -caaling under two common fata-constrained scenarios.
In the entire set experiments, simply increasing unique token count by collecting more high qual- ity data (195B via hall-Fineweb-Eddee Edu) shows marginal improvements (+0. (+0.2/+0. 15/-0. 16/+0. 11) at 200/300/400/500 billion token steps (13B size). In contrast, MGA, through a 200B reformulation as expansion of the original 50B data, demonstrates consistent gains highlighting effective D-scaling.
Similarly, in the subset experiments, both upsampling the high-quality sub data portion (5x) and MGA (via a 200B expansion) improve upon the bascline. However, their N-scaling prop- erties with model parameters differ significantly: the performance advantage of upsampling remains relatively constant across model sizes 1.0.89/+1,53/++ 1.23/4 whereas MGA

---

Published as a conference paper at ICLR 2026
pansion exhibits superior N-scaling, its performance gains amplifying with increasing model scale (+4144642266674435544333333
These scaling experiments confirm that MGA is a powerful data augmentation strategy that aids both model (N) and data (D) scaling in constrained scenarios. Notably, MGA's performance advantage emerges from the very first epoch, well before significant data repetition occurs, and this gap widens as training progresses. The dual observation leads directly to our core research questions, which we will investigate in Section 4.3 How does MGA's inherent diversity mitigate the degradation from high- reghhrreptition training (RQ2), and what fundamental learning benefits does the reformulation provide from the outset (RQ3)?
Validation Losses Although MGA demonstrates superior benchmark performance, we observe increasing validation losses compared to baseline models. While higher validation losses might seem concerning at first glance, it's important to note that validation loss may not fully reflect model performance, as token-level perplexity is inherently biased by the frequency distribution of the vali- dation set, and in-domain validation metrics may not necessarily correlate with out-of-domain gen- eralization capabilities. This observation, combined with recent studies linking loss degradation to model collapse (Dohmatob et al. 2024b Zhu et al. 2024), calls for more nuanced analysis, which we will also provide in Section 4.3.3
4.3 DISCUSSIONS
The strong empirical results in our main experiments validate MGA as a powerful data augmen- tation strategy. However, these outcomes naturally lead to deeper questions about its positioning. mechanics, and underlying benefits. In the following section, we move from empirical validation to analytical discussion, addressing the key research questions outlined at the beginning of this section.
4.3.1 How DOES MGA COMPLEMENT EXISTING SYNTHETIC DATA STRATEGIES?
Our main experiments demonstrated MGA's value relative to data repetition, but how does it stand within the diverse and rapidly evolving ecosystem of synthetic data? To address RQ1, this section positions MGA not as a standalone replacement, but as complementary approach to other prevalent strategies, such as Wemotron-CC-Syntten this popular open-source synthetic datasets to highlight its unique contribution.
(Su et al. 2024). We compare MGA-enhanced data with
To conduct fair comparison, designed a controlled experiment with four distinct data mixtures, each training a 1.7B parameter model for 800B tokens. The data blend setups were as follows:
Baseline: high-quality real dataset fineweb-edu.
Exp A: 35% token budget replaced by Nemotron-CC-HO synthetic corpus (+Nemotron-Syn)
Exp B: 35% token budget replaced by MGACorpus (+MGA).
Exp C: 70% token budget replaced by an equal combination of Nemotron-Syn and MGACorpus
data (+Nemotron-Syy +MGA).
Benchmarks Aug.
Knowledge
Roasoning
Math
Bisunde ail
0.31
0.55
0.031
0.03
0.025
0.40
0.34
8:37
200
4.0
6.0
032
200
420
620
832
200
4.00
630
852
200
4300
630
352
Labutta
Lakuna
Lakuna
lakuna
Figure 4: Benchmark results demonstrating the complementary nature of different synthetic data strategies. While both MGA and Nemotron-Syy individually improve over the baseline, their com- bination (Exp C) yields significant synergistic boost in performance. For a detailed breakdown of performance by each synthetic task, as well as supplementary cross-min moxs-ng experiments that further validate MGA's role in creating generalizable base model, please refer to Appendix D.3
As illustrated in Figure 4, the results reveal clear performance hierarchy: Exp C > Exp A > Exp > Baseline. The strong performance of Exp A is understandable, as Nemotron-Sy is a high- quality and diverse synthetic corpus composed of five diverse subsets. The inclusion of various
7

---

Published as a conference paper at ICLR 2026
data formats, some of which (like QA pairs) align well with common evaluation structures. While our MGA-enhanced mix (Exp B) also surpasses the baseline, the most compelling finding comes from Exp C, which significantly outperforms all other configurations. This demonstrates clear synergistic effect, where the structural and stylistic diversity from MGA's reformulation enriches the high-quality, task-aligned data from Nemotron-Syn.
Therefore, we answer RQ1 by concluding that MGA is not in competition with but is complemen- tary to other synthetic data methodologies. The path resolving data scarcity does not lie in single synthesis technique, but in the thoughtful combination of diverse strategies. MGA provides foundational, general- peneral-purpose purpose enhancement through reformulation that benefits even further when combined with specialized, task lask-eligned synthetic data.
4.3.2 DOES REFORMULATION DIVERSITY HELP TO MITIGATE REPETITION ISSUE?
To address RQ2, this section examines how different design choices in prompt engineering influence the effectiveness of the MGA framework, particularly under high-repetition conditions. By compar- ing SLM variants (introduced in Section 3.1 using different consistency requirements (see Table identify optimal strategies for balancing information preservation with content diversity.
Table 3: Performance comparison of different SLM variants on reformulation quality metries.
Models
Total Examples
5
4
3
2
Rate(S 2) Rate(2 4) Rate(= 5)
SLM-Base 15,355
3,788
7,124
3,224
736
285
6.65%
71.06%
24.67%
SLM-Strict 15,355
6,814
5,220
2,384
520
227
4.86%
78.37%
14.38%
SLM-RRlaxxd
15,355
408
1,685 3,889
4,156
5,086
60.19%
13.63%
2.66%
We sample an additional 20B tokens from real data and generate three synthetic datasets: 80B tokens using SLM-Base, 80B tokens using SLM-Strict, and 40B tokens using SLM-Relaxed. As mentioned before, SLM-Base expands the original corpus to 3.9x more tokens, while -eeleel makes only 2x tokens as we only require basic topical relevance. Similar to experimental setup in early sections, we set high-repetition baseline on a smaller data scale (replicating the original 20B tokens 10 times) to more clearly demonstrate the potential impact of SL.M-Strict compared to SLM-Base in high-repetition scenarios.
As shown in Figure 5, our experiments reveal distinct patterns across training configurations. Both SLM-Base and SLM-Strict show performance improvements, while the SLM-Relaxed configuration leads to significant collapse. More supplementary experiments could be found in Appendix D.2
D.2
Average
Knowledge
Reasoning
Math
Validation Losses
0.200
0.55 sevalic
150
0.22
2.8
0.000
2.4
0.75
0.30
0.50
1.25
2.4
0:25
1.000
tobern
#oltckanx (1)
Iolaens
Figure 5: Benchmark results and validation losses. The sensitivity to data repetition varies across capability domains, with knowledge dimension showing greater resilience.
Despite the apparent effectiveness of strict information preservation, can it fundamentally address the challenges posed by data repetition? Our examination of validation loss trajectories reveals a critical distinction: SLM-Base maintains healthy optimization characteristics throughout training. whereas SLM-Strice exhibits degraded scaling behavior at higher iteration steps, reminiscent of the limitations observed with data repetition.
Therefore, this investigation into different prompt engineering strategies concludes that a balanced 'Limited Consistency' approach (SLM-Base) yields the best reformulation quality and subsequent model performance answering to RQ2.
4.3.3 WHY DOES REFORMULATION BENEFIT THE MODEL'S 'S LEARNING PROCESS?
Having explored the impact of diversity in addressing data repetition (RQ2), now turn to RQ3: this section investigates the underlying mechanisms by analyzing learning characteristics and vali-
dating against potential issues like model collapse (Dohmatob et al. 2024b Zhu et al. 2024).
8

---

Published as a conference paper at ICLR 2026
Multi Multi-perspectivv Validation Analysis Our analyses across different validation sets reveal vary- ing patterns in model behavior (Figure 6). As expected, MGA groups' substitution of fineweb-edu data results in adverse effects corresponding loss, with similar deterioration observed in open- web-math. Interestingly, the synthetie dataset cosmopedia demonstrates improved loss metrics. A notable contrast emerges in python-edu: while MGA exhibit negative impact the 134M and 377M
parameter, this trend reverses at 1.7B, suggesting scale-dependent effects on model behavior.
coomopidtoooo
cdu Scdup
apon
pychon
144
3.3
24
3900
480
#.2 13cora E1
offisionn 1VC
# tolorn 18)
0:0 (oloaaa 341
Figure 6: validation losses of experiments in Section 4.2
Fine-grained Pattern Analysis To better understand whether increased validation loss truly in- dicates model collapse, we conduct tie-grained analysis of loss patterns. Specifically, we com- pare token-level losses of 800B checkpoint between models trained on real data and synthetic data (Baseline and MGA-Expansion in Section 4.2 respectively). The document samples are from both Fineweb-Edd and MGACorpus. As illustrated in subfigures I and 3 of Figure 7 each point repre- sents sample's average token loss, consistent with the overall loss discrepancy shown in Figure 6
Fineweb Edu Examples
Distri. of First Anormaly Pos.
MGACorpus Examples
Distri. of first Anomell Pos.
23
400
(ea) losa
100
300
0-
25
75
100
0
2
2
25
50
75
100
Position
Position
Figure 7: Losses pattern analysis. Subfigures 1 and shows comparison between models trained on different data settings, with lossreal on y-axis and losssynt on x-axis. Subfigures 2 and 4 track the position where loss -oss' real (loss first becomes significantly higher than the sequence's average difference (detailed definition in Appendix D.4.
The distribution of first anomaly positions (subfigures 2 and 4) reveals crucial insight: when processing real data, models trained on synthetic data show performance degradation (measured by lossar) that predominantly manifests in later sequence positions, which intensifies as lossaff increases. However, this positional bias disappears when evaluating on synthetic data.
The systematic pattern suggests that rather than experiencing model collapse, the synthetic-trained model may have developed a different learning strategy (examples shown in Appendix D.4). While shows higher validation losses on certain real-world datasets, its strong performance our main experiments indicates a potential trade-off: the model may prioritize learning generalizable patterns from context over memorizing specific sequence dependencies. This shift in learning process could explain both the improved performance on benchmark tasks and the increased losses on validation sets that potentially require more memorization-base processing.
These findings indicate that the performance characteristics associated with MGA data likely stem from altered learning strategies, potentially prioritizing generalizability, rather than representing model collapse, which addressing RQ3,
5 CONCLUSION
In this work, we introduced MGA, a principled framework that leverages genre- cenre-audience reformu- lation to systematically expand and augment existing corpora with diverse, synthetically generated variations. Our core finding highlights MGA's effectiveness as a data augmentation strategy specifi- cally targeting the repetition challenge: in data-constrained scaling experiments, MGA significantly

---

Published as a conference paper at ICLR 2026
outperformed naive data repetition and simple upsampling, enabling more effective model training beyond unique data limits. Furthermore, the quality of the MGA was confirmed by consistent per- formance improvements when incorporated into standard training mixtures across various model sizes. Crucially, our experiments also revealed MGA's role as a complementary strategy, demon- strating a powerful synergistic effect when combined with other prominent synthetic datasets. In essence, MGA's effectiveness demonstrates that the key to overcoming data limits is generating rel- evant diversity, not just raw volume. Therefore, our work offers more than a tool; it provides new roadmap for the community, where the thoughtful combination of different approaches becomes the cornerstone of sustainable progress in the continue scaling of LLM development.
REPRODUCIBILITY STATEMENT
We are committed to reproducibility and will release the 770B-token MGACorpus and all associated resources. Our MGA framework implementation, including tool model details, is documented in Appendix B The complete pretraining and evaluation setup, with data recipes and hyperparameters, detailed Appendix C
Cases and prompts used to curate the dataset is provided in Appendix E
ACKNOWLEDGEMENTS
We are grateful to Chao He, Zhixin Yao, Yue Chen and Runyu Shi for their help with prompt templates and case studies, and to Seed-Foundation team for providing the stable training/inference platform, which enabled us to build the synthetic pipeline and corpus within a reasonable timeframe.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi. Piero Kauffmann, al. Phi-4 techni- cal report. ArXiv preprint, abs/2412.08905, 2024.
URL /Bl.:tttsss//axx...rrr///a//////
08905
Loubna Ben Allal, Anton Lozhkov. Guilherme Penedo. Thomas Wolf. and Leandro von Werra. Smollm-corpus, 2024.
URL https://nuggiggaacecco http:/aaaa....ca/aaaaaaaaaaoeaae co/dattsets/
MuuginngFaceTR.. smol 1m-corpus,
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
ArXiv preprint, abs/1803.05457, 2018. URL (tttttts:://ara //tp:v/aaivv..//.a org/
wttgg/s///////o/ooo/////////ael/ abs /100/ 05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/21 14168, 2021. URL https://arxiv.
erg/abs. 2110.1100000 14168
Aat :, Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Ur- banek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Charvi Bannur, Christina Baek, Darren Teh, David Schwab, Haakon Mongstad, Haoli Yin, Josh Wills, Kaleigh Mentzer, Luke Merrick, Ricardo Monti, Rishabh Adiga, Siddharth Joshi, Spandan Das, Zhengping Wang, Bogdan Gaza, Ari Morcos, and Matthew Leavitt. Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining, 2025. URL https:/ /tt////axvvvv... org.orgg//....
2508.10977
Elvis Dohmatoh, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. ArXiv preprint, abs/2410.00840., 2024a. URL URt/pssss://axxi wttp:aawwwi..r.as//aa/aoaaaaaasa org/abs//4110 04840
Elvis Dohmatob, Yunzhen Feng, Pu Yang, François Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. In Forty-first International Conference Machine Learning, JCML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL https:
//openzeview..o. wwtp::enetttttiieee www.or/www..w.uuuaa... KVvku47sha
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783. 2024. URL httpts:///arttt.. wwwg/abs......... /240 .21783
10

---

Published as a conference paper at ICLR 2026
Alex Fang, Hadi Pouransari, Matt Jordan, Alexander Toshev, Vaishaal Shankar, Ludwig Schmidt, and Tom Gunter. Datasets, documents, and repetitions: The practicalities of unequal data quality, 2025. UtLtttps:://// http:p:::///xi....o/ org/ abs http:////aa/////0777777
Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for Ilm evaluation, 2023. URL httpp:///ittub.com htwps//hhhggiigggg
lighteval
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023. URL htttps:/ wrt./ecords///////oooooooo.o.s 10256836
Tao Ge, Xin Chan, Xiaoyang Wang. Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. ArXiv preprint, 0bs/22406200994 2024. URL https:
7/ //arxiv.ooo wwtpii.rrrgabba. haarg//ba///4//
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URL
net www.fooddddddaoooo jmI 3CmQ
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodee Tom Henighan, Tristan Hume, et al. Scaling laws and interpretabillity of learning from repeated data. ArXiv preprint, abs/2205. 104877, 2022. URL
https://arx:: org/ wwt./ass////oooooooeooo/o/saaa.a 2205. 10487
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Au- relia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December ni 9, 2022, 2022. URL / nips. ps. wttppcce_paaeeeeeeeeeee www.s//laeeee///eeeeaaaaaaaaaeea p2tppppprrr2222e/ppppp / c 2122aaf611809951111111
Absttaattt--bbstttttttaaaaaaa-ee Contraaat-Coofeeencee. htmel
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small lan- guage models with scalable training strategies. ArXiv preprint, abs/2404.06395, 2024. URL
hittps://arxiv orgo
arrg/abs abs 22tp://444..335.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv preprint, abs/2001.08366 2020. URL
08361
Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, An- gang Du, Chenzhuang Du Dikang Du Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang. Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin. Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tian- wei Liu. Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi,
11

---

Published as a conference paper at ICLR 2026
Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang. Dinglu Wang, Feng Wang. Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang. Shengjie Wang. Shuyi Wang, Yao Wang, Yejie Wang. Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jin- jing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang. Zhen Yang, Zhilin Yang, Zonghan Yang. Haotian Yao, Xingcheng Yao, Wenjie Ye. Zhuorui Ye, Bohong Yin. Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang. Yizhi Zhang. Yongting Zhang. Yu Zhang, Yutao Zhang, Yutong Zhang. Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL https: / //tp/i/aaaivv../ai org aataaaa/22510 20534
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muen- nighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yr Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M. Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Raghavi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade. Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alazeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Con- ference on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024. httpa//ttttpp//ppes.......pooooe wwt.nniisscccpppecmmm filrrfieee/aae /papr/ http://pppesp.ppipppccccceeeeaap
and Baw___nnhharaas__eaaa html
Pratyush Maini, Skyler Seto, He Bai, David Grangier. Yizhe Zhang, and Navdeep Jaitly. Rephras- ing the web: A recipe for compute and data-efficient language modeling. ArXiv preprint, abs/2401.16380, 2024. URL
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A. Raffel. Scaling data-constrained language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::ttpppppptpt www.errsssppsssss... nips cc wwppaper.pppppppppeppeeeeeee. hash/ 94894488944448 8b 1aa1://88555.aeeee011127766666e3
html
NVIDIA, :, Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinay Khattar, Adithya Renduch- intala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, Alex Kondratenko, Alex Shaposhnikov, Alexander Bukharin, Ali Taghibakhshi, Amelia Bar- ton, Ameya Sunil Mahabaleshwarkar, Amy Shen, Andrew Tao, Ann Guan, Anna Shors, Anub- hav Mandarwal, Arham Mehta, Arun Venkatesan, Ashton Sharabiani, Ashwath Aithal, Ashwin Poojary. Ayush Dattagupta, Balaram Buddharaju, Banghua Zhu, Barnaby Simkin, Bilal Kartal, Bita Darvish Rouhani. Bobby Chen, Boris Ginsburg, Brandon Norick, Brian Yu. Bryan Catan- zaro, Charles Wang. Charlie Truong, Chetan Mungekar, Chintan Patel, Chris Alexiuk, Christian Munley, Christopher Parisien, Dan Su, Daniel Afrimi, Daniel Korzekwa, Daniel Rohrer, Daria Gitman, David Mosallanezhad, Deepak Narayanan, Dima Rekesh, Dina Yared, Dmytro Pykhtar, Dong Ahn, Duncan Riach, Eileen Long, Elliott Ning, Eric Chung, Erick Galinkin, Evelina Bakh- turina, Gargi Prasad, Gerald Shen, Haifeng Qian, Haim Elisha, Harsh Sharma, Hayley Ross, Helen Ngo, Herman Sahota, Hexin Wang, Hoo Chang Shin, Hua Huang, lain Cunningham, Igor Gitman, Ivan Moshkov, Jachun Jung. Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jian Zhang, Jiaqi Zeng, Jimmy Zhang, Jinze Xue, Jocelyn Huang, Joey Conway, John Kamalu, Jonathan
12

---

Published as a conference paper at ICLR 2026
Cohen, Joseph Jennings, Julien Veron Vialard, Junkeun Yi, Jupinder Parmar, Kari Briski, Kather- ine Cheung, Katherine Luna, Keith Wyss, Keshav Santhanam, Kezhi Kong, Krzysztof Pawelec, Kumar Anik, Kunlun Li. Kushan Ahmadian, Lawrence McAfce. Laya Sleiman, Leon Derczyn- ski, Luis Vega, Maer Rodrigues de Melo, Makesh Narsimhan Sreedhar, Marcin Chochowski, Mark Cai, Markus Kliegl, Marta Heepniewska-Dziubinska Matvei Novikov, Mehrzad Samadi, Meredith Price, Meriem Boubdir, Michael Boone, Michael Evans, Michal Bien, Michal Za- walski, Miguel Martinez, Mike Chrzanowski, Mohammad Shoeybi, Mostofa Patwary, Namit Dhameja, Nave Assaf, Negar Habibi, Nidhi Bhatia, Nikki Pope, Nima Tajbakhsh, Nirmal Ku- mar Juluru, Oleg Rybakov, Oleksii Hrinchuk, Oleksii Kuchaiev, Oluwatobi Olabiyi, Pablo Rib- alta, Padmavathy Subramanian, Parth Chadha, Pavio Molchanov, Peter Dykas, Peter Jin, Piotr Bialecki, Piotr Januszewski, Pradeep Thalasta, Prashant Gaikwad, Prasoon Varshney, Pritam Gun- decha, Przemek Tredak, Rabeeh Karimi Mahabadi, Rajen Patel, Ran Yaniv, Ranjit Rajan, Ria Cheruvu, Rima Shahbazyan, Ritika Borkar, Ritu Gala, Roger Waleffe, Ruoxi Zhang. Russell J. Hewett, Ryan Prenger, Sahil Jain, Samuel Kriman, Sanjeev Satheesh, saori Kaji, Sarah Yurick, Saurav Muralidharan, Sean Narenthiran, Seonmyeong Bak, Sepehr Sameni, Seungju Han, Shan- mugam Ramasamy, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shelby Thomas, Shizhe Diao, Shreya Gopal, Shrimai Prabhumoye, Shubham Toshniwal, Shuoyang Ding, Siddharth Singh, Sid- dhartha Jain, Somshubra Majumdar, Soumye Singhal, Stefania Alborghetti, Syeda Nahida Akter, Terry Kong, Tim Moon, Tomasz Hliwiak. Tomer Asida, Tony Wang, Tugrul Konuk, Twinkle Vashishth, Tyler Poon, Udi Karpas, Vahid Noroozi. Venkat Srinivasan, Vijay Korthikanti, Vikram Fugro, Vineeth Kalluru, Vitaly Kurin, Vitaly Lavrukhin, Wasi Uddin Ahmad, Wei Du, Wonmin Byeon, Ximing Lu, Xin Dong, Yashaswi Karnati, Yejin Choi, Yian Zhang, Ying Lin, Yong- gan Fu, Yoshi Suhara, Zhen Dong, Zhiyu Li, Zhongbo Zhu, and Zijia Chen. Nvidia nemotron nano An accurate and efficient hybrid mamba-traasformee reasoning model, 2025. URL
attps://aaxxt org/ abs//ab///// 2508 14444
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouci, and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data only. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 nips cc wttpeeerfileeepppee//2//e 16, 2023, 2023. URL ottraaaeeeeeetttttttaaaaaaaaaaaa Datasess,
Abstract
Benchmarks html
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Amir Globersons, Lester Mackey, Danielle Bel- grave, Angela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference Neural Informa- tion Processing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December 10 nips cc/paper files/paper/2024 -bea---bstrtttt Datasets and
15, 2024, 2024. URL Benchmarks, Track.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. ArXiv preprint, 12.22112.111466 2021. URL
https://arxiv. wttp://wwww.pp///////////////// 11446 11444
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:7, 2020. URL http:://jmmm
paaprrs/v21/2--072 mttml
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. In The Thirty- Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thiony- Innovative Applications of Artificial Intelligence
13

---

Published as a conference paper at ICLR 2026
Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8732-8740, AAAI Press, 2020.
BRLL:t/:::/// http://aaaii...... org/ /s/ httpo//s/nnniinnn//ie index. httppp//...//atttieeeeaaaaaaae/e
6399
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraiming research. ArXiv preprint, abs/2402.00159, 2024.
//tp:ii//axxviv/// org. abs 20000//24220 24402000130
Dan Su, Kezhi Kong. Ying Lin, Joseph Jennings. Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-ce: Transforming common crawl into a refined long-horizon pretraining dataset. ArXiv preprint, abs/2412.02595, 2024. URL https:
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science. ArXiv preprint, abs/2211.09085, 2022. URL https::/arxiv. org. http///xxxvv.or//b///2211
09085
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. ArXiv preprint, 01/222211043325 2022. URL https:: / org/ abs 2211. 04325
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not repeat: Insights from scaling LLM under token-crisis. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, nips ctt.pps.ccppppppp wttps//stppee///////////////
2023, 2023. URL http: 000472265773333226344445556.2322
html
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL bttts:://aaatt.
org/ abs/2505. 09388
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1472. httpsttss: : www.e/aattoooo....gg.ooooooo
1472
Ge Zhang, Xinrun Du, Zhimiao Yu, Zili Wang. Zekun Wang, Shuyue Guo, Tianyu Zheng, Kang Zhu, Jerry Liu, Shawn Yue, Binbin Liu, Zhongyuan Peng, Yifan Yao, Jack Yang, Ziming Li, Bingni Zhang, Minghao Liu, Tianyu Liu, Yang Gao. Wenhu Chen, Xiaohuan Zhou, Qian Liu, Taifeng Wang, and Wenhao Huang. Finefineweb: A comprehensive study on fine-grained domain web corpus, 2024.
Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, Ermo Hua, Xingtai Lv, Ning Ding, Zhouhan Lin, Zilong Zheng, and Bowen Zhou, How to synthesize text data without model collapse? ArXiv preprint, abs/2412, 2024. URL htthsts:: awwpvvivvvvv..////v/ wrg. org/ abs 2412. 14689
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE WORK
While our experimental results demonstrate the effectiveness of MGA in both quality validation and scaling scenarios, several important aspects warrant further investigation. We identify two key areas for future research:
Our current experiments demonstrate effectiveness up to 13B parameters and 1,000B tokens of training budget. Extending this approach to long-horizon training and larger-scale models requires additional validations, particularly for next-generation models which require hundreds of trillions of training tokens.
Regarding data repetition strategies, we present preliminary explorations under computational resource constraints. The underlying patterns and their sensitivity to various factors, such as repetition ratio, data distribution, and even model hyperparameters, require systematic investi- gation. Future research should examine how these factors collectively determine optimal data strategies across different training scenarios.
Broader Impact This paper explores the use of LLMs as a data expansion method for pretraining large language models. We introduce the MGAframework to mitigate data repetition issues, which holds potential for positive societal impact, particularly in synthetic data generation for training language models. Nonetheless, the use of synthetic data generated by LLMs is not without risks; for instance, LLM hallucinations, even after filtering, could introduce novel errors biases into models trained on such data, a factor that warrants careful consideration in future research and deployment.
B
MGA FRAMEWORK IMPLEMENTATION DETAILS
B.1 TOOL MODEL. TRAINING & RESOURCES
Corpuct Sämple
Tein part Egenns cud enco) paint generate
Tea port document refornuiation
Vol
IUM
SUn
Texl Pakei
FCT Grlevise
Leleene
Selerrie
Tolun
Laroe Model 2E2 Infernce
"ood
Sanpiing
Popel Troin Sat
Shoge Train
Large Hotd
Micel
Tex Hakel
Heunetic
Reformulated
High Quality Corpus
Roult
Poout
MSA Corpus
Figure 8: Implementation details. From a high-quality corpus, we sample a subset to serve as input for the LLM labeler and judger. Through data filtering, we train and quantize tool SLMs for each stage to improve inference efficiency, which are used to generate the reformulated corpus.
High Quality Corpus We conduct our reformulated corpus based on SmolLM- Ben Allal al. 2024), expanding fineweb-edu-dedun source from 195B tokens to 770B tokens. Then we setup additionally experiments FineWeb and FineWeb-Edu (Penedo al. 2024), which constitute a solid foundation for research on data scaling approaches. Prior to these experiments, we have validated our approach on our in-house datasets. The results demonstrate consistent performance across both datasets, suggesting broad applicability of our method.
15

---

Published as a conference paper at ICLR 2026
Tool Models Training Initialized from a pretrained SLM (a 3.3B MoE model), collect 50,000 training samples through LLM teacher, where 15,000 of raw text to genre-audience pairs, 35,000 of raw text to reformulated output. Each model's validation responses judger, that ensures the SL.s achieve comparable synthesis quality to the LLM labeler as shown in Table 1 The sequence length is 8192 with maximum prompt/response length 4,096 tokens, each model is trained 3 epochs on the samples with cosine Ir scheduler.
scored by capable LLM
We also finetuned a public model, Qwen3-30B-A3B Base(Yang et 2025). serve the 'Open Tool SLM', which will also be released. The results confirm that its performance is highly compara- ble our internal models, proving our method is not dependent on any specific proprietary tool, as shown in Table 4 The result provides conclusive evidence that our core methodology is robust and reproducible technique that can be readily implemented by other researchers using public models.
Table 4: Reformulation quality comparison between finetuned qwen3-a3b and its LLM teacher.
Models
Total Examples
3
2
Rate 3)
Diff
Labeler LLM
15,355
4,120
7,143
3,034
661
214
93. 11 %
Tool SLM
15,355
3,788
7,124
3,224
736
285
92.06%
-1.05%
Qwen3-30B-A3B (Finctuned)
15,355
3,322
7,154
3,614
805
249
91.76%
-1.35%
Quantifying Generation Diversity To systematically validate our design choices in preventing mode collapse and ensuring synthetic data heterogeneity, we introduce two metries: Reference Sim- ilarity (RefSim), which measures the average cosine similarity between synthetic documents and their original sources, and Heterogeneity Score, defined as [ Σie sim(pair,, pair,) to evaluate internal corpus diversity.
As shown Table 5| our base MGA strategy strikes an optimal balance. While a relaxed prompt strategy (free LLM sampling) leads to severe homogenization (lowest Heterogeneity Score), and strict prompt policy limits diversity below that of the original data, our approach maximizes internal diversity while preserving relevance to the source text.
Table 5: Trade-off between source relevance and internal diversity across different prompt strategies.
Setting (Prompt Strategy)
RefSim (vs. Original) Heterogeneity Score
Original Data
1.0000
0,3355
Base (Our Method)
0.8979
0.3390
Strict (Enforce high similarity)
0,9392
0,3326
Relaxed (LLM free-sampling)
0.8319
0.3063
Furthermore, we evaluate the impact of our "one-pass-for-many" generation strategy against a stan- dard "one-pass-for-one" baseline. As detailed Table rather than introducing conditioning bias, our strategy acts as a diversity catalyst. It yields a significantly higher number of unique genre- audience pairs and achieves superior lexical and semantic diversity, confirming its effectiveness in scaling data synthesis without collapsing into repetitive patterns.
Table 6: Diversity metrics comparing our "one-pass-for-many" strategy against standard sampling.
Metric
One-pass-for-many (Ours)
One-pass-for-one
Total Examples
1000
1000
Unique Genres
913
779
Unique Audiences
871
715
Unique Pairs
987
951
Type-Token Ratio (TTR)
0.1213
0.0839
TF-IDF Dissimilarity
0.9751
0.9572
Resource Analysis
To generate 770B synthetic tokens, it takes 256x64 and 1024x130 NVIDIA H100 GPU hours to process two stages.
16

---

Data Recipe The training token budgets are 600B/600B/00008 for size of 134M/377M/1.75 mod- els, which are aligned with SmolLM1 series (Ben Allal et al. (024). Our baseline is trained on unique token number from each Published as a conference paper at ICLR 2026
C PRETRAINING AND EVALUATION SETUP
C.I PRETRAINING DATA RECIPES
MmolLM-Corpus dataset, in contrast to SmolLM's recipe, source as the mixing ratio shown in Table 7 This ensures that different sources have consistent repetition epochs during training. For fair comparison, the mixing ratios of other data sources kept constant across experiments. We specifically adjusted the proportions of fineweb-edu-deduy and MGACorpus isolate the impact of the MGA reformulation.
Table 7: MGACorpus experiments data recipe: source weight (%) and #unique x #epochs.
experiments
fineweh dedup cosmopedia v2 python-edh open weh MGACorpus
Baseline
weight
80.89
11.65
1.66
5.80
funique tokens #epochs
195 4.15
28 x 4.15
4 4.15
14 x4.15
MGA-Expansion
weight
16.29
11.65
1.66
5.80
64.59
#unique.tokee tokens #epochs
195 0.84
28 4.15
4.15
14 4.15
770
0.84
The experiment design for different strategies is presented in Table 8 which involves three datasets: (1) a 50B-token random sample from fineweb-edu-dedup- (2) a corresponding filtered subset from MGACorpus, and (3) 450B-token deduplicated corpus obtained from Fineweb.
Table 8: Scaling experiments data recipe, values represent #unique tokens #epochs.
Training Budget
fineweb-edu dedup
MGA
fineweh random
Repetition
Experiments
Design Rationale
corpus
Baseline
500B
50 x 10
EntireSet
Full-Fineweb-Räi
500B
195 2.56
What if could collect more unique data. Add MGA to reduce the repetition num.
MGA Expansion
500B
x0 2
200 x
Baseline
700B
50x 1.4
450 1.4
Subset
Upsample-FDU
700B
50
450 x
Upsample to get 200B more budget. Add MGA to achieve the same target.
MGA Expansion
700B
50
200 x
450 x
C.2 MODEL HYPERPARAMETERS
We sample 100 million tokens from SmolLM-Corpus the validation dataset. The hyperparams presented in Table 9, These hyperparameters are determined by scaling laws to ensure an optimal baseline and are kept consistent across all experimental groups. The tokenizer used for training and computing token counts is the same as SmolLM with a vocab size of 49,152.
Table 9; Hyperparams of different model size.
model size
batch size
learning rate
weight decay
hidden
fîn
num heads
num layers
shared
seq
tie
total
size
inner
head
len
emb
params
134M
128
3e-3
0.1
1,204
4.096
8
8
8,192
false 134M
377M
320
1.5e-3
0.1
1,536
6,144
12
10
8,192
false
377M
1.7B
512
5e-4
0.1
2,560
10,240
20
16
8.192
false 1,68B
7B
1,024
4e-4
0.1
4,096
8,192
32
32
8.192
false
6,98B
13B
1.024
4e-4
0.1
4,096
12,288
32
48
8,192
false 12.9B
C.3 EVALUATION DETAILS
The LightEval results provided in (cction|4.2]ooll SmolLM setting, that with GSMSK/MMLU 5- shot and all the others 0-shot. The benchmarks presented in Figure 9 and Figure 10 follow few-shot evaluation settings, specifically ARC(8-shots), TriviaQA(5-shhn shots),, Winogrande(S-shot) and similar configurations for other tasks.
Figure
httts::/huugiinggface
17

---

Published as a conference paper at ICLR 2026
D SUPPLEMENTARY EXPERIMENTAL RESULTS & ANALYSIS
D.1 BENCHMARK COMPARISONS WITH MORE SOTA MODELS
While model performance is influenced by multiple factors, we list some recently SOTA small lan- guage models reference.
Table 10: Benchmark MGA with SOTA small LMs. Models of similar size are grouped. All results are obtained through LIGHTEVAL (Fourrie et al., 2023). Best results in each group are highlighted in bold, the second in underline, and in green for that MGA wins under fair comparison.
Model
AParaus.
4Tokems ARC(C+E)
Wino.
Hella,
MMLU
MMLU-PRO
CSQA
OpenBookQA
PIQA
TriviaQA
GSMSK
Avg.
SmolLM2: M000LM2--33MM
135M
2T
44.12
51.07
42.03
31.27
11.06
33.82
35
68.23
1.91
1.52
32.00
Smoll M-135M
135M
600R
42.47
51.54
1144
32.51
33.2
1.08
31:24
SmoILM-135M (ours)
134M
6038
41.71
52.41
40.69
30.03
11.37
34.32
35.4
67.85
0.02
1.29
31.51
Exposssoooo
43.01
31.77
Qwen2.5-0.
360M
18T
45.16
53.99
51.16
33.51
11.97
31.61
37.6
69.97
3.96
32,9
37.18
MM00LM2-360M
37.6
SmolLM-3600 M-360M
360M
600B
49.99
52.96
51.67
33.84
11.42
34.81
37.6
71.87
2.27
1.97
34.84
Smoll 3-360M (ours)
377M
600B
48.57
52.64
$1.02
33.63
11:25
36.77
39
0.29
1.52.
34.57
MGA-Eue Exponsion
577M
600B
49.39
52.64
511.3
34,09
11:35
37.1
38
72.31
7.28
1.74
35.52
Qwen2 5-1.5
1.3
18T
58.36
58.64
06.30
40.23
13.85
34.4
39.6
75.95
20.51
60.8
46.87
Smoll M2-1.7B
611.42
59.59
42.6
77.53
36.68
47.93
LJaaa-3 3.2-1B
1.26
9T
49.2
$7.8
61.2
36.63
11.7
41.2
38.4
74.8
28.1
7.2
40.62
1B-0721
SmoILM-L.7B
1.7B
IT
59.95
54.7
62.83
39.35
10.92
38
42.6
75.9
13.14
4.82
40.20
Smoll M-1.78 (ours)
$9.63
57.38
12.11
42.59
45.6
4.95
41.15
MGGA-Expinsion
1.78
TT
66.36
57.46
65.52
40.79
14.1
41,11
42.8
77.53
20.42
13.87
43.4
In our experimental observations (Table 10), notable performance improvements are demonstrated in both TriviaQA and GSM8k benchmarks, warranting detailed examination of these score variations. The enhanced TriviaQA performance exhibited by SmolLM1-1.77B relative to our baseline can be attributed to the larger proportion of Cosmopedia its training configuration. Both MGACorpus and Cosmopedia employ synthetic methodologies, which contribute to improved learning efficiency. The observed gains in GSM8K performance can be traced to the target genres, including teaching schemas and problem-solving exemplars, embedded within the Reformulation component. This early exposure to structured problem-solvin approaches facilitates more effective performance on analogous mathematical reasoning tasks.
commmnaeers
0.3
04m11k
Telkcang
777u
02eebzokge
0.22
0.25
635
0.50
250
655
302
82
182
300
230
6375
0332
0.37%
3600
3310
2.5
Figure 9: Detail evaluation results of EntireSet described in Table MGACorpus group demonstrats advantages over other groups across most evaluation sets, consistently across models of sizes.
18

---

Published as a conference paper at ICLR 2026
2.41
arc chalengs
rodat Commenseres
6'1p
0.84
1.50
6.50
0.1
640
035
1.2
600
210
0.75
0.75
0423-
0150
0.3
9174
625
630
0380
0.45
0.0
040
Pio
9.2
winugrande de
6.09
0.8
3.24
0.2
1.24
048
2.2
0.4
048
400
750
200
220
Figure 10: Detail evaluation results of Subset described in Table 8 As the model size increases, the performance gap between the upsampling group and MGACorpus gradually widens in ARC, DROP, GSM8K, RACE, but with some variations observed in TriviaQA and WinoGrande.
D.2 "MGA-ONLY" EXPERIMENT
Our primary goal with MGA is efficient dataset expansion, typically achieved by mixing the gener- ated corpus with existing real data, aligning with current best practices for leveraging synthetic data. However, to better characterize the properties of the MGACorpus itself and understand the impact of training exclusively on reformulated content, we also investigate an experimental setting where MGACorpus completely replaces its source data (fiewwb-eeuu-deuup)
Table 11: MGACorpus experiments data source weight (%).
experiments
fineweb fneweb-edu-deee edu-p cosmopedia-v2 python-edu open mat--web-math MGA-corpus
Baseline
80.89
11.65
1.66
5.80
MGA-Expanse
16.29
11.65
1.66
5.80
64.59
MGA-Only
11.65
1.66
5.80
80.89
the absence of real data leads to performance degradation across most tasks (average -0.95), particularly in two tasks, Bellaswag(-1.23// 1.22//11666)--2.85 and Commonsense@AA 3. 1/4488//-4.50) This decline can be attributed to our design choice, which focuses on diversity and overall quality rather than requiring the preservation of all information from each raw documents.
As shown in Table 12
Table 12: Comparison between MGA-Expanan and MGA-Only
Model
Paraams.
#Tokens ARC(C+E)
Wino.
Hella
MMLU
MMLU-PRO CSQA
OpenBookQA PIQA
TriviaQA
GSMBK
Avg.
MKGA-Espansion
134M
6008
43.01
51.7
41.25
30.1
11.76
32.68
36.4
67.3
2.05
1.44
31.77
MGA-Only
134M
1:1.03
[:0.32
[-1.23
L-0.23
[-0.26
1:3.11
1:3.40
70.71
10:31
1-0.38
[-0.90
MGA Expansion
377M
600B
49.39
$2.64
51.34
34,09
11.35
37.1
38
72.31
7.28
1.74
35.52
MGA44nly
49.65
33.31
1:1.14
10.71
1:1.69
L-0.78
10.03
1-1.83
1:1.36
1:0:45
[-0.15
1-0.99
MGA-Expanan Expansion
L7B L7B
IT
60.36
57.46
65.52
40.79
14.1
41.11
42.8
77.53
20.42
13.87
43.40
MGA-Only
59.02
57.06
62.67
40.34
13.51
36.61
45.2
76.71
19.78
13.57
42.45
1-1.34
(-0.40
1-2.85
L-0.4
C40.59
4-4.0
12.40
1-0.82
J-0.64
(-0.30
[-0.95
MGA-Only Setting of PE Ablation Upon relaxing the information preservation requirements for
PE objectives in the MGA-Only setting, we observe a complete collapse in knowledge- baseed di- mensions while maintaining modest improvements in reasoning and mathematical capabilities. This divergence suggests that different cognitive capabilities have distinct requirements for the richness and nature of training data content.
19

---

Published as a conference paper at ICLR 2026
Average.
Knowledge
Ressoning
Validation Losses
0.210
proce part
0.31
6.40
0.225
and
0.00
Basocs
2.
0.15
0.25
0.30
2.
0.50
6125
2.4
57
100
1:0
230
50
302
156
200
100
230
202
200
22
100
350
250
250
# of tocerss (8)
4 #oftchena (5)
if Losers (5)
#olithkana (8)
of tolsens (8)
Figure 11: Corresponding benchmark results described in Section 4.3.2
D.3 COMPARISON WITH OPEN SYNTHETIC DATA
This section provides crucial empirical support for the central thesis of our Discussion (Section 4.3 the path to resolving data scareity lies not in single synthesis technique, but in the thoughtful combination of diverse strategies. We argue for an approach that breaks down the walls between different synthetic data paradigms, demonstrating that a model's robustness comes from being trained on rich variety of data formats. The following experiments designed to validate this philosophy, showing how MGA acts as a synergistic component within a broader data ecosystem, rather than a standalone solution.
D.3.1 HEAD-TO-HEAL PERFORMANCE AGAINST OPEN SYNTHETIC DATASETS
This first experiment aims to benchmark MGA's effectiveness against other well-known synthetic data generation methods when each is used as the primary training corpus. We trained 377M param- eter models for 300B tokens several distinct datasets to establish a clear performance baseline. For a fair comparison with Cosmopedia, the MGA corpus was sampled to 28B unique tokens, and both datasets were repeated approximately 10.7 times during training.
Table 13:
Comparative benchmark performance of 377M models trained on MGA reformulations versus other synthetic datasets for 300B tokens. For a fair comparison with Cosmopedia, MGA is sampled to 28B unique tokens, with both datasets then repeated 10.7 times during training. All benchmarks are O-shot evaluations (obtained through LiGHTEVAL), except for MMLU (5-shot).
Category
Document Sources
Synthctic Target
ARC(C+E) Wino. Hella, MMLU
CSQA OpenBookQA
PIQA TriviQQA
Avg.
Cosmopedia
Textbook High quality webs
Story/Textbool/W// mix
42.15
50:43
45.06
29.17
30.38
33.2
68.77
0.23
35.57
Diverse Genre-Audiece
42.31
31.42
32.19
68.39
3.79
37.28
Low quality wehs
Wrap medium (Wils style)
29.01
50.83
38.36
26.29
29.32
32
67.03
0
31.72
Extmat knowledge
40.42
53.2
44.68
30.57
28.99
35
69:42
0.96
35.72
Nemotron-CC CC
Knowledge
42.08
52.17
42.7
30.71
32.51
35.4
70.08
High quality webs
Concise and clear passage
42.22
52.01
43.99
30,96
31.53
69.7
0.06
36.21
Wrap-medium (Wiki style)
42,95
52.17
43.72
31.06
31.53
36-2
70.13
0.82
36.63
Diverse pairs
52,57
38.82
38.8
70.84
40.7:
MGA
High quality webss
Diverse Genre-Audiens:
45,33
52.41
42.42
31,33
31.45
38
68.61
4.24
37.34
The results in Table 13 highlight MGA's competitive performance general-purposs augmenta- tion strategy. MGA (average 37.28) surpasses Cosmopedia (35.57), which blend of story, text- book, and wiki formats. When compared against the various synthesis strategies from Nemotron- CC, MGA (average 37.34) again shows strong performance, outperforming most alternatives such as 'extract knowledge" (35.72) and 'wrap-medium (Wiki style)" (36.63).
Notably, while Nemotron's 's diverse QA slice achieves the highest average score (40.72), this advan- tage is likely attributable to its format aligning directly with the question-answent lent in our 0-shot evaluation benchmarks. Despite this format specific advantage, MGA's broader reformulation approach proves its robust utility by outperforming five of the six Nemotron strategies. This underscores MGA's value in building a well-rounded and capable base model.
structure preva-
D.3.2 INVESTIGATING SYNERGISTIC EFFECTS VIA CONTINUED PRE-TRAINING
Moving beyond static head-to-head comparisons, this second experiment directly tests our hypoth- esis about the importance of data diversity. We investigate the dynamic interplay between different
The predominantly (-shot evaluation particularly benefits datasets like Nemotron 'diverse QA pairs' whose format directly aligns with many evaluation tasks.
20

---

Published as a conference paper at ICLR 2026
data types to see how model trained on corpus adapts when another is introduced. This di- rectly probes the synergistic potential discussed in our main paper. To do this, we took checkpoints of models pre-trained for 300B tokens MGA-Corpus and the Nemotron-CC (QA-slice), spectively. We then continued pre-training each model for an additional 30B tokens, mixing in data from the other corpus 1:1 ratio.
Table 14: Model Performance with Mixed-Corpus Continued Pre-training
Experiment
Tokens
Wino.
C-QA
Hella.
MMLU
OBQA
PIQA
TriviaQA
ARC
AVG
mga mga mixet Change
300B
52.4
31.5
42.4
31.3
38.0
68.6
4.2
45.3
37.3
330B
52.8
37.6
47.5
31.6
37.8
70,2
45.5
39.2
+0.4
+6.1
+5.1
+0.3
-0.2
+16
+0.2
+0.2
+1.9
nemotron qa
300B
52.6
38.8
49.0
31.4
38.8
70.8
9.2
47.0
40.7
nemotron_gg mixet
330B
51.8
37.8
47.8
31.8
36.6
70.6
3,5
45.9
39.1
Change
-0.8
-1.1
-1.2
+0.4
-2.2
-0.2
-5.7
-1.1
-1.6
This cross-mixing experiment revealed two interesting and complementary phenomena:
Synergy and Receptiveness MGA. When Nemotron QA'ss structured data was mixed into the MGA-trained model, the model's average performance on downstream benchmarks significantly improved by 1.9 points. This suggests that the diverse, rich foundation built by MGA is highly effective and "receptive", readily absorbing the benefits of more specialized, format-aligned data.
Distributional Path Dependence. Conversely, when MGA's diverse data was mixed into the Nemotron-QA-ttraince model, its performance decreased by 1.6 points. This suggests that pre- training on stylistically monolithic dataset can create a "path dependence", making harder for the model adapt when different data distribution is introduced.
In conclusion, this experiment provides compelling evidence for our core argument. The strong per- formance of task-aligned data like Nemotron QA undeniable, but it does not represent complete solution. Our findings show that the most effective approach is one that embraces variety. MGA's primary value lies in creating robust, generalist foundation that is highly receptive to other data types. It helps build a model that is not locked into a single stylistic mode, effectively breaking down the data walls and paving the way for a more adaptable and capable AI. This synergy the key to building next-generation base models.
D.4 FURTHER ANALYSIS OF MODEL COLLAPSE
we define the token loss difference loss loss syni loss where i is the token index, synt/real is dataset used for model training. Note that we consistently use synthetic minus real, where a positive value indicates poorer prediction performance by the synthetic model given sample.
Further Discussion of Section 4.3.3
For our analysis method in Figure 7
Since next token prediction computed based on preceding context, we define the first anomaly position to identify where model's prediction for tokens within the window begins significantly deteriorate. The definition is as follows:
p+w
first first_anomaly_pon = min(p)
loss
> |cg| + koJ.
where w = max(0.05 x seq_length, 1), mean (lossaa)) a= sid(loss Here, we employ the absolute value of the windowed average loss identify sigmificant performance degradation in either model. This approach enables the detection of notable prediction quality drops regardless of which model (synthetic or real) experiences the deterioration.
Finally, we define the normalized position, enabling fair comparisons across various sequence lengths:
first_amomaly sengen
saalamooaly_pocttion position x 100% if anomaly found
normalized position
otherwise
21

---

Published as a conference paper at ICLR 2026
Below example cases from English and Chinese documents. Figure 12 presents the token loss difference on each position. Example and Example 3 show similar anomaly pattern, we can get the reason in Figure 13 that they are from the same website source contain identical boilerplate text about region selection and website localization at the end of their content.
Example
Example
Example
Example 4
Loos Diference
7.5
5.0
2.5
0-
0.0
0.0
11
-7.
-25
60
80
100
20
D0
100
100
20
60
100
Position d1 Sequence (56)
Position
Sequence (9))
Positton in Secg requannce (96)
Posiition Sequuence (86)
Figure 12: Random examples sampling from where mean mean(lossann model fail to predict the tokens later sequence positions.
0.5, the synthetic-trained
This suggests potential noise in the data preprocessing pipeline, specifically in handling website navigation elements and localization prompts that should have been removed during content extrac- tion.
While these examples demonstrate clear patterns of model behavior differences in handling noisy web data, acknowledge that this analysis is limited selected cases with apparent preprocessing artifacts. A more comprehensive evaluation across diverse data sources and quality levels would be necessary to fully understand the impact of synthetic training data on model performance.
Web design ancompasses many different skilss and disciplines in the and maintenence of webeites The Bifferent web design incluce graphle lintortaca design Butnorinng Ing Inoluding standars code broprictory sortware optimization.
Web design partislly overl web engineer ing the broader acope web divelopment Web designers expecced have awareness of usability creating mark then they also expected
tole
with web cccossibility guidelines.
Fiet design the new trend in web design focuses on minimal istli sporeach for usability spec fhe to dimenssonnll and flat loons and images fresh
considered distraction. Flat designs are also oasy designs to corvort to mobile and other devices which makes funct orality benehcial.
to
Compute the product of all elemants in ar array.
wife
Energy photon
s this triangle right =langled
ireepective equares
Besld metl leb numeric and string marip uletions
ios counting
Rep ettioon matricas
generate eapital anglich pl
web
get Ianalated contiant peherallwalal wallable Son inventa
Breundion your location, will racommend that you sekeet
You can alsc sprect wab sito from the following list
the
Chinese English
country optimized
Contact your local office
Olven the lotter, the outiput will show the comneponding ASCI code Addd
select the primes vector
Find the area the four walls
Mat
unvsnted parts vactor zero
Divide by
equetion
Log of number
Vector with repeated
Square root number
Choose web site get Iranaistod content enhorolovlllble and Sen locol oventta and effers, Bosed on location, wi rocommend that select
web Select the Shinaleite ( Chineee or English) for beet site performance. Otherr Merni Works country sites are not optimized for Veite from your locerion.
following list:
Dontact your local
Figure 13: Corresponding cases sampled from Fineweb-Edu, which align with the loss patterns
shown in Figure 12 with higher loss by synthetic-trained model highlighted in red.
22

---

Published as a conference paper at ICLR 2026
x 100
d MO.
MA 10
A
BU
SERE
IM I175 10-4824
SE- 2811 ESO-E SET
14.
TRI
WETH WO IY MEI
L
644 RR. AN B.N0
Figure 14: Chinese corpus samples with higher loss by synthetic-trained model in red.
23

---

Published as a conference paper at ICLR 2026
E PROMPTS AND CASES
E
CASES
Table 15: Example outputs of SLM variants.
Decument
choonirry teaching, corieulaa ghilosopee goding sndeats
dbilifilly captuingg sprial nethed thet combises
etillee tomsiiion key
Bocause under
stillees" trosstomms their thinking and helps there leam through activiiles Being ahle skitifill sagture the optinal timinn frasstito betwess "movemstt and silless" key to success selting these peritct cioments for
inmviiion loads to grster officiency with Ice offiet.
GeuroAudiesee Pais
Tool Model
Refonmultted Document
deppy inportates and Praniic allu Optinal Thelgee "devennt Silues Thaakaen Mikkk Scood Cheauary teackin sereee homv
oo fighl racking an this for Mt schaok clrniity cothin it iCy repriccaai by cat by and pervent honer reder cadomnan diig arideeneer
Acstamic teaching paper ssing
chorary soddiag bote maanhoa Opeasl Trading
rigoros atstenii style to desply
SIM-Ilao:
dall en hime rine lrving reder loo ho contees hervere an st mon' his groe repprted Optinal their of MMenmamesd
analys ofeapturing opiinal timing of inveenne-atillnss innotione in chemistry taaching
Thoough referesees to relevimn
oo da lags perisic radal abued clecarty toad up de lass of munces conccconli laaad usatica hulp etalcas concistas
aducatural thourtes and sctsol vaching expesianees, provlde meatingfi traching teight chemiery sashams
Teercccc caskine susciees and dee ley paiit clss and
Vniier
Miille school chamiery unchers, aqad 25-45, prodorinarrly female with
cuseiis flgm with mesiliie lelpe apandioss
prfeestunal chemitty cdandiion
ploeest drrugh
baziground They am pattent and steady in characerr beca lesdung mikdi: mony yoars inpeeve teadista
SI M-Sarcc
loan dig rediale miciica cas an detugh spallle whin the led suakky prppatd oo iiadia
pluck de
thrugh anarcmatvve leea" mutrios
musec tran je equit propersive bets do poia
help sturkons betker uncemtten chenstry.
drvegler dn enre
key bein thee beo weiiige
crichicsss scied suary sacinn ace ciiiatate cocule achopet doas uairy chote lanning rac racling color of the up monely aetenid be sld
SLM Reiased
lommd hnoeicile mare deppl
gridas
the
wothe This coegle thip compleitt.
Populic science story, using lively and introting turrative tell sury of how moddle school studeers realinee the importasee capturing optinal trunitioss their
SLM-Base
an wo the bus
loasing Threagh spacifle intoreiing oxpetimenss help student usderstand chemany knswehatge an relased and engoyatle ommospthere sperking interes in chemisery loaning.
oo aoeund anvornnd aeplaaeeolee and bookez
yru
Thy diveene Dayan brew the ver ford world dhnmarr then's roaly imporant grod cptirall
contt and and now whsch torefiin may criinaiin and tod os loon toegth aarbor tamn doinaery lornag comaagppccn procoss Pir Bocplty
SLM-Strict
ararnting aquriness kudder tu godd
-Middo school andente who find chanistr diffieul, spad 12-15, botl moir and torals, win trkon-aviing ceadeni perforaace. They are gencrally liucly and setive chimaccs interes chenisrry but stady ways improve their chemisny gride.
the
bree"
Fay daveiso, donh you thin in diniiig nod Lefk scrver ear too tigoter su denitryy comning joaeet
Aniig ti wis or wire an Naa we so nemeer in or very tu haat aidyy in and was thips toding mg the meys repenctti
tibrroees Me wartect tocher magiral rpperivess cherhaher forr pu ontrrmm pokkngpped mrspperee and
liee
ad bas
SLM Relased
jae played
wani sat doia
yourrr
toru spuiiil in likig bose't
ta dely
and
an in marep
ploci
Pareninn edocation gaide, combroed wilh seal-life secaarios cuplaia the consept of copturing optimel rining of insvenent-aillee aillacss bassiiios is chemaary teaching Prosire emple porest interaction netieds help uncersandingg
toundeeend
aboa dh la dajae Eae qdicty keap De cashet proctiss and clanorc die cunaaaa dill the beil othee
SLM-Blas
youtack anee deowiaee is del It Birriia i the ciscoel The reall lnorac golio dilics they Iellaca' is "poeciical "Whlo your
muy sn
apoly die "rarrnati kuniig dely lioe carkig plocess, you parar-chill
this mothed daily lif, while improving pweet-child rebaiunsiitys.
dull qurntry the apdraal of muncecivaa Encee ssad in toy your dilt vascus forning curtary-- bocds ronchitgro and
dell
(Paenis aged 10-50, male and fanale with occepations They shildnare
SLM Striet
abooret southe Netven deogee Dating thes precess and Fine mgg in proeess you tiite will
copurncco topeler with
clicd
unde what Seek chithen and leamiin achool betier rosst with their studies, but hove linited chanstry knowlodes diemaiivo..
"hre Meel te ae
colled
mutted Bicc
SLM-Relaaa
tall
didy 1if
wil
die ingees ded parco dl alaieaa i abcoppag Ag Bikk caldra is builing and kapp glodd.
24

---

Published as a conference paper at ICLR 2026
E.2 PROMPTS
Although the term "rewrite" is used in some prompt templates as the editing instruction, serves the same function "reformulate" discussed in sections above, which aims to maintain the core meaning of the documents while only optimizing its expression.
# strict version
# relaxed version
You are text polishing expert. You will polish text based on the given [Genre] and [Audience].
You creative expert skilled transforming materials into creative inspiration and building independent, complete, and highly original texts.
When polishing, you must follow these rules: 1. Read through the entire text and polish it according the requirements of the given [Genre] and [Audience]
Requirements:
Read through the original text thoroughly, extract several key themes/keywords. transform abstract or universal concept inspiration, then
2. The degree of polishing should not be too heavy just aim to satisfy the requirements of [Genre] and [Audience] much as possible
generate entirely new text constructions.
2. Extract content from [Audience] and [Genre] sections, but don't he constrained by them directly, just use them as creative inspiration.
3. Double-check that the polished text is suitable for the audience described in [Audience]:
4. Pay attention to the frequency of modal particles the text should not contain too many modal particles
Create and reformulated text around points 1/2, and build new meaning from details to the whole structure.
Figure 15: two different prompt templates, we keep the input aligned with MGA strategy, using raw text, genre, audience to fill the template.
#*
#ldentity and Capabilities#
You are a content creation expert, specializing in text analysis and rewriting, capable adapting content based on varying "genres" and "audiences" to produce "diverse" and "high-quality" texts. Your English writing at native editor level, and you will output your rewritten texts in English. International audiences particularly enjoy your work, which receives widespread readership and circulation, earning unanimous acclaim from the industry for your capahilities!
#Workflow#
Please utilize your analytical and writing abilities rewrite the based the original content and given genre" and "'audience" Before beginning the rewrite, you will consider the following requirements:
1. First, read through the original text thoroughly, identify its information content and value, and consider how prevent any loss of information points and value in the rewritten text Focus on original content, combine it with the given "genre" requirements, rewrite the text following the descriptions, content modules, language requirements, and other stylistic elements specified in the "genre",, to form an initial draft
Polish the initial draft according the given "audiiece" requirements, and generate the final rewritten text English
Refine rewritten text to match native English speakers' reading habits and expression patterns
#Detailed Requirementsit
Please ensure you follow the three workflow requirements above, then generate the final English rewritten text according to these detailed requirements.
The given "audiienee" is <audienncc
The given "genre" is < <<<ngge>>>>>>
#Raw Text# (raw.text) text)
Prompt 1: reformulation prompt template.
25

---

Published as a conference paper at ICLR 2026
#ldentity and Capabilitiistf
You are content creation expert, specializing in text analysis and rewriting, skilled at adapting content based varying [genres] and [audiences] to produce "diverse" and "high-quality" texts. Your rewriting approaches consistently transform original texts into remarkable content, carning acclaim from both readers and industry professionals!
#Workflow
Please utilize your imagination and creativity to generate pairs of [genre] and [audience] combinations suitable for the original text. Your analysis should follow these requirements:
First, analyze the characteristics of the text, including writing style, information content, and value Then, consider how to preserve the primary content and information while exploring possibilities for broader audience engagement" and "alternative genres"
#Detailed Requirements#
Ensure adherence to workflow requirements above, then generate pairs of [genre] and [audience] combinations according to these specifications:
Your provided [genres] should meet the following requirements:
Clear Genre Definition: Demonstrate strong diversity; include genres you encountered, read, or can envision
Detailed Genre Description: Provide 2-3 sentences describing each genre, considering but not limited to
type, style, emotional tone, form, conflict. rhythm, and atmosphere. Emphasize diversity to guide knowledge adaptation for specific audiences, facilitating comprehension across different backgrounds. Note: Exclude visual formats (picture books, comics, videos): use text-only genres.
Your provided [audiences] should meet the following requirements:
Clear Audience Definition: Demonstrate strong diversity; include both interested and uninterested parties, those who like and dislike the content, overcoming bias toward positive audiences only
Detailed Audience Description: Provide sentences describing each audience, including not limited age, occupation, gender, personality, appearance, educational background, life stage, motivations and goals, interests, and cognitive level
нннннлвннннн #Responsc#
"audienee_1 1":: audiencel,
"genre_"" 1": genrel,
"audience 2": audience2, "genre_2"; "genree2"":geree" genre2,
"audience_3". audience3, :
"audience_4": audience4, "genre_4"; genre4, "audience_5"": audience5, "genre 5" genre5
#laputit
(raw_text)
Prompt 2: genre-audience pairs prompt template.
26

---

Published as a conference paper at ICLR 2026
#ldentity and Capabilitiest
You Content Reviewer, skilled at analyzing and keenly identifying and analyzing the relationships, similaritics, and differences between two texts. Your thorough analysis cach pair of texts, with attention to every detail, provides great convenience for subsequent review work!
#Thinking Process#
Please fully utilize your analytical abilities, review capabilities, and deep thinking skills to analyze the Rewritten Text" against the "Original Text" as benchmark, ultimately providing analysis and scoring for [
A]. You will follow these steps for detailed consideration:
First, you will read through original text thoroughly, identifying the information points in the "Original Text"
2. You will also read through the rewritten text thoroughly. identifying the information points in the Rewritten Text"
Compare the information both texts' content. The "Rewritten Text" is allowed to have new information points, different writing styles, expression styles, order, and focus from the "Original Text",, As long as is created based on some information points from the "Original Text", it is considered good for [A] 4. After careful analysis and review, please clearly list the connections and differences between the two texts, and based this, provide final analysis and scoring for [A]
#Detailed Requirementsit
The scoring judgment for [A] must follow these standards:
The "scoring range" is 1-5 points. You need to analyze and grasp each aspect mentioned in #Thinking Process#, and differentiate scores accordingly. Be strict, don't too lenient with scoring!
The "Rewritten Text" is allowed to differ from the "Original Text" in writing style, expression style, and focus! This cannot be basis for deducting points! The "Rewritten Text" is allowed to omit that all information from "Original Text" appears for deducting points! this is the only issue, please give a full score 5 points.
information from the "Original Text"! It is required
"Rewritten Text"! This also cannot basis
In scoring [A]. the following situations will NONOT reduces* the score for [A]: The "Rewritten Text" can include information points present in the "Original Text" added content the "Rewritten Text" significantly deviates the core information the Original Text"
3. The expression style, order, and focus of the "Rewritten Text" differ from the "Original Text"
scoring [A]. the following situations slILL reduce+s score [A]: The information points in "Rewritten Text" differ so greatly from the "Original Text" that not recognizable as being rewritten from the "Original Text""
2. The "Rewritten Text" contains none of the information points from the "Original Text"
#Original Text# (raw_text)
#Rewritten Text# (rewritten.text)
#Response Format#
"A":(
"analysis": "xxx", provide reasons for point deductions "score" 2, 2, 3, 4, or
Prompt 3: Full LLM judger prompt.
27

---

Published as a conference paper at ICLR 2026
F USE OF LARGE LANGUAGE MODELS
During the preparation this manuscript, we utilized large language model (LLM) as a writing assistant. The LLM's role was primarily focused on improving the clarity, precision, and readability of the text. This included tasks such as correcting grammar and spelling, refining sentence structure for better flow, and suggesting alternative phrasing enhance the academic tone.
The core scientific contributions- including the initial research ideation, the design of the MGA framework, the experimental methodology. and the interpretation results-were conceived and executed entirely by the human authors. The LLM did not contribute to the research ideas or the analysis presented. The authors have meticulously reviewed, edited, and validated all L-s--asssteed text to ensure its scientific accuracy and take full responsibility for the final content of this work.
28


Reviews:
[{"review_id": "4fkT3nWEUX", "reviewer": "Reviewer_W91b", "summary": "This paper introduces a synthetic data generation pipeline for pretraining language models. Their pipeline, MGA (Massive-Genre Audience), asks a lightweight 3.3B MoE model to rewrite source text for different “genres” (e.g. story, dialogue, textbook) and audiences “girl”, “grandpa”, and “teacher”. MGA provides better scaling properties for training language models than training on the original data alone.", "questions": "The authors have a nice analytical discussion section around data reformulation diversity, learning characteristics, and mode collapse – it does make me curious if some of the conceptual findings here generalize to other synthetic data generation processes. \n\nI may have accidentally missed this somewhere, but from where did you source the range of possible “genres” (structural/stylistic formats, e.g. dialogue) and “audiences” (reader profiles, e.g. teacher) for your pipeline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Most of the experiments in this paper are done very thoroughly, and results are presented in a very neat and orderly manner. I appreciate that the authors release an open-source dataset and also release tooling and other artifacts that make this work reproducible.", "weaknesses": "A key thing I wanted to know in this paper was whether the author’s proposed approach, MGA, outperforms alternative approaches for generating synthetic data. Even though the authors say that “MGA is not in competition with but is complementary to other synthetic data methodologies” (line 370), the authors should properly demonstrate MGA’s utility in the current landscape of synthetic data generation strategies. I am not sure if I got a clear answer to this question from the current version of the paper. \n\nSection 4.3.1. Claims that MGA can complement existing synthetic data approaches. There, a plot shows the following, as I quote: \n> - Baseline: A high-quality real dataset fineweb-edu.\n> - Exp A: 35% token budget replaced by Nemotron-CC-HQ synthetic corpus (+Nemotron-Syn).\n> - Exp B: 35% token budget replaced by MGACorpus (+MGA).\n> - Exp C: 70% token budget replaced by an equal combination of Nemotron-Syn and MGACorpus data (+Nemotron-Syn +MGA).\n\nThe authors’ results show that though A outperforms B, C outperforms A and C outperforms B. However, one key experiment missing here is whether a 70% token budget replaced by only Nemotron-Syn outperforms C, because it if it does, then it makes the inclusion of any MGACorpus/MGA data not necessary and potentially detrimental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXeQstJu8T", "reviewer": "Reviewer_HLbn", "summary": "This paper proposes a new way of synthetically augmenting pretraining data through Genre Audience reformulation. \nThis paper works, and its numbers are solid. The science is also thorough.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The greatest strength is the scaling plots, which demonstrate consistent performance improvement in the token-matched regime. Really nice! Figure 5 was particularly great. The training noise seems relatively less which is good for trusting the downstream evals. \n2. Really nice overall benchmark numbers. Grounds the work with good numbers to help set this up relative to other methods.  Achieving a MMLU score of 40.7 is great for the 1.7B model class. \n3. Beating finding more hq data is really good result.  Particularly Figure 3 is strong! The graphs showed consistent improvement across all scales, beating the use of a more original dataset. \n4. Also, a good set of benchmark coverage. LightEval and LM-Harness are both great tools for eval and using the set of evals like MMLU is great.", "weaknesses": "My main gripe is that I would have liked to see a few more baselines of synthetic data/rephrasing. WRAP, for example, would have been nice to show. I understand that this would have been expensive at the higher token counts, but at the lower token counts, it would have been nice. There are great synthetic pretraining data techniques. Could you add WRAP or just adding other synthetic post-training datasets (nemotron for example) would be great?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rlYzVLQTq8", "reviewer": "Reviewer_eWsn", "summary": "This paper introduces Massive Genre Audience (MGA), a reformulation method to augment training corpora based on generating multiple diverse genre-audience pairs per inference pass through a prompting-based method. This is done with the intention of mitigating the influence of data repetition during training that often leads to performance degradation particularly in data scarcity scenarios. The authors proceed to analyze the effectiveness of their method (including whether the diversity introduced is actually beneficial) and how it synergizes with alternate synthetic data strategies. Models at different scales are evaluated across a wide range of datasets.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well organized and clear. Specific research questions are laid out and experiments are directly designed to support each one in a clear manner.\n2. Comprehensive results across many datasets and models at 5 different scales.\n3. The paper emphasizes and shows the synergistic nature of their method. \n4. Authors state they will release MGACorpus and all artifacts for reproducibility.\n5. The motivation of addressing negative influences of data repetition during training is important.", "weaknesses": "1. The method is based on prompt engineering and as such there are some potential issues with bias introduced by using a solely LLM based system. For example, when the LLM is used for judging, what sorts of bias is being introduced? Was some form of human evaluation done or some alternative test to determine whether the scores produced by the LLM align with the actual quality of the text. Furthermore there should be some evidence that actual diverse outputs are being generated by the method. I believe that there should also be some results on text heterogeneity quantification to pinpoint the actual heterogeneity introduced by the sampling method which would allow us to concretely correlate with the performance.\n\n2. Tied to the above, some discussion on specific design decisions is warranted, namely the choices behind the genre audience pairs. Also it is stated that the SLM \"executes a “one-pass-for-many” strategy ... since repeated sampling requests to a model can yield highly similar outputs\" but doing so can lead to bias since later generated pairs are conditioned on the early ones compared to random sampling (although as mentioned some means to stop potential redundancy is needed). If heterogeneity could be quantified and these two methods compared this assertion could be validated.\n\n3. Given the similarities in the idea, some discussion of previous works on synthetic data creation using personas [1] is warranted in the paper.\n\n4. Only one model family is tested (SmolLM) for MGA-Expansion which presents a limitation in determining how well the method will work on alternative architectures.\n\n5. Table 3 is not referred or discussed anywhere in the text. \n\n[1] Tao Ge et al. Scaling Synthetic Data Creation with 1,000,000,000 Personas. Arxiv. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)