Synthetic Bootstrapped Pretraining
Authors: Zitong Yang
Aonan Zhang
Hong Liu
Tatsunori Hashimoto
Emmanuel Candes
Chong Wang
Ruoming Pang
Venue: ICLR 2026 Poster (2026)
Keywords: language model
pretraining
synthetic data
Primary area: foundation or frontier models, including LLMs
TL;DR: New pretraining paradigm that bootstraps model performance via synthetic data -- no external teacher needed.

Abstract: We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training.
While the standard pretraining teaches LMs to learn causal correlations among tokens within a single document, it is not designed to efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance.
We validate SBP by designing a compute-matched pretraining setup and pretrain a 3B-parameter and a 6B-parameter model on up to 1T tokens from scratch.
We find SBP consistently improves upon a strong repetition baseline and delivers up to 60% of performance improvement attainable by an oracle upper bound with access to 20x more unique data.
Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases -- SBP first abstracts a core concept from the seed material and then crafts a new narration on top of it.
Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.

Published as a conference paper at ICLR 2026
SYNTHETIC BOOTSTRAPPED PRETRAINING
Zitong Yang 122 212 Aonan Zhang"' Hong Liu?
Tatsunori Hashimoto Emmanuel Candès", Chong Wang', Ruoming Pang
Apple Staanford University 'Equal contribution
ABSTRACT
We introduce Synthetic Bootstrapped Pretraining (SBP), language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize vast new corpus for joint training. While the standard pretraining teaches LMs to learn causal cor- relations among tokens within single document, it is not designed efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance. We validate SBP by designing a compute-matches pretraining setup and pretrain a 3B- 3B-parameter and 6B 1B--arameter model on up to IT' tokens from scratch. We find SBP consistently improves upon a strong repetition baseline and delivers up to 60% of performance improvement attainable by an oracle upper bound with access to 20x more unique data. Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases SBP first abstracts core concept from the seed material and then crafts a new narration on top of it. Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.
INTRODUCTION
Pretraining on the diverse internet texts is now seen to be bottlenecked by the rapid depletion of high-quality text data [64]. This imminent "scaling wall" motivates us utilize existing data more effectively. Re-examining the conceptual foundation of pretraining, its success originates from the rich causal correlation among tokens within a document. However, this is not the only source of correlation pretraining datasets contain: code document implementing the attention mechanism is derived from the arXiv preprint of the transformer paper; The book of Harry Potter structurally similar to the screenplay of its movie production. Such connections suggest weaker form of inter- document correlation derived from an underlying joint distribution of pretraining documents. We hypothesize that this additional signal, which is missed by standard pretraining, can be captured by synthetic data, presenting an underexplored avenue for improving performance.
To leverage this opportunity, introduce Synthetic Bootstrapped Pretraining (SBP), LM pre- training procedure that operates in three steps (Figure 1). First, SBP identifies semantically similar document pairs (d1,d2), such the transformer paper and its code implementation, from the pre- training dataset. Second, SBP models the conditional probability of d2 given d1, creating a "data synthesizer" that can synthesize new, related document given a seed document. Finally, SBP ap- plies the trained conditional synthesizer to the pretraining corpus itself, creating vast text corpus that encodes the rich inter-document correlations that were previously missed ($2). By training a data synthesizer from the pretraining dataset itself, SBP avoids the pitfall of "bootstrapping" model performance using an external, readily available teacher L.M, demonstrating clean setup where the source of improvement stems from better utilization of the same pretraining corpus.
To test our hypothesis, we design a compute-matchedd data-constrained experimental framework under which we pretrain a 3B -a-paaamettr and a 6B- dB--paraeeter model on up to IT tokens from scratch [34, 72], demonstrating the potential applicability of SBP for advancing frontier LMs. We compare SBP's performance against two crucial references: strong repetition baseline, which represents the
Correspondence z tong@be rkebbrrreeeeyy edu

---

Published as a conference paper at ICLR 2026
transformer paper
synthesized notes on the paper
embeddings
the Harry Potter book
movie review HP
language modeling tutorial
a lesser
pytorch code of attention
known paper
synthesized
blogpost about the paper
Step nearest neighbor pairing Step 2: synthesizer-tuning
Step synthesis scale
Figure 1: Data synthesis illustration of Synthetic Bootstrapped Pretraining (SBP): It first identifies semantically similar documents (Step 1) and then trains conditional model that generates one element of the pair from the other (Step 2). Finally, SBP applies the conditional model to the pretraining corpus itself to synthesize new, vast corpus for joint training (Step 3).
standard approach in data-constrained settings, and oracle upper bound, which has access to an unlimited pool of unique internet data ($3). Our results show that SBP consistently surpasses the strong repetition baseline across different pretraining scales and closes up to 60% of the performance gap to the oracle with 20x additional unique data access ($4.1).
Besides strong benchmark performances, qualitative analysis of the synthesized documents reveals that they went beyond mere paraphrases of the real documents ($4.2). We postulate that the SBP syn- thesizer first abstracts latent concepts from the real document and then synthesizes that expands upon the abstracted concept, incorporating diverse genres and content. We formalize this intuition through Bayesian hierarchical concept model, where documents are related through shared concepts. From this perspective, we argue that the synthesizer implicitly learns a posterior likelihood model that abstracts latent concepts from the document a mechanism not present in the standard LM pretraining ($5).
document
In summary, our contributions are threefold:
New pretraining framework: We propose the Synthetic Bootstrapped Pretraining (SBP) algo- rithm that explicitly models inter-document correlations missed by standard pretraining practice and encodes those correlations into training via synthetic data.
Large-scale empirical validation: We design compute- compute-matched pretraining setup that enables rigorous measurement of LM self-improvement and empirically validate SBP on 3B and 6B parameter models trained up IT tokens from scratch.
Principled statistical interpretation: We offer a natural Bayesian interpretation of SBP as implicitly learning a posterior for the latent concepts in text document and concretize the intuition via qualitative analysis of synthesized documents.
In the remainder of the paper. we will first define the cata-connt pretraining problem we ad- dress and introduce the SBP technique we propose in $2. Then, we present the compute-mateee matched experiment setup in $3 and results in $4. Finally, we conclude with Bayesian interpretation of SBP that sheds light on the origin of the improved performance $5.
1.1 RELATED WORK
Before we proceed, we review related work that highlights our contribution in three broad areas of research: LM pretraining, synthetic data for LM, and retrieval-augmented LM.
LM pretraining. The concept of pretraining, closest its modern form, originates from series of works including ELMo [47], ULMFiT [27], BERT [15], that propose to pretrain a neural network via an unsupervised objective and subsequently finetune for wide range of downstream tasks. The GPT-series [48, 49, 8, 42] cemented the practice of using next-token prediction as the pretrain- ing objective and applying it to large-scale crawled webpages as opposed to task-specific datasets (e.g., -hgeeshhto-Frenchh translation). In recent years, the size of the pretraining corpora has grown rapidly, driven by the availability of massive web-crawled datasets, leading to a successful stream of dataset and pretrained model artifact: BERT [15, 37], GPT-2 WebText [49], CommonCrawi [12], CCNet [67], T5 C4 [51], the Pile [18], Gopher Massive Text [50], Llamda series [62, 16), Refined- Web [46]. Dolma [56]. CCLM-baseline [34], NemotronCC [57]. etc. While pretraining has been tremendously successful, the rapid depletion of available internet text motivates us to shift our focus
from acquiring more data using the existing data more effectively.
2

---

Published as a conference paper at ICLR 2026
Synthetic data. A natural way to overeome the limitations of scarce high-quality web data is to pretrain [20, 1, 2, 3, 61] or continually pretrain [69, 52, 71, 40] LMs on synthetic data. Existing approaches to data synthesis rely distillation from powerful "teacher" LM that generates com- pressed knowledge representation for the "student" LM to learn [26]. These teacher models must first undergo a human alignment process, which requires extensive human annotations and pref- erence data [43]. Synthetic data from the teacher LM hints at a limited scaling trend: whilst the synthesized data from the teacher LM can be impressive [13] as 7x more effective than real data, the performance improvement quickly converges to that of the teacher LM [9]. We instead con- sider the scenario where the sole source of world knowledge comes from a fixed set of pretraining documents (e.g., the internet) and algorithmically learn data synthesizer with minimal human in- tervention (e.g., generative teacher models or human writing prompts). Therefore, our experiment setup simulates situation where the LMs can self-boost their pretraining capability by refining their understanding of the fixed collection of pretraining documents.
Retrieval augmented LMM natural class of methods that incorporates multiple documents to- gether is retrieval augmented generation (RAG) [32, 33]. While originally introduced as a tech- nique to be used at test-time for a domain-specific downstream task [6, 35], retrieval augmented approaches have been extended in scope: [30] and [68] implement RAG at pretraining scale and show improved test perplexity; [24] incorporates RAG at pretraining time by jointly training a re- triever and the model itself for improved QA performance. [55] groups related documents into the same context window for improved long-contex: capability. In general, while the RAG-related ap- proach enables the model to utilize rich inter-document correlations, it is fundamentally limited by the context window of the LM. In contrast, SBP encodes correlations into synthetic data that can be iteratively learned by the LM one document time. Prior to the advancement of embedding mod- els that allow retrieving the entire document, [23] proposed retrieving neighboring pairs of sentences using Jaccard similarity and modeling the conditional distribution between them, similar our con- ditional data synthesizer objective; however, they did not perform any pretraining experiments.
2
OUR METHOD
In this section, we introduce the data-constrained pretraining setup ($2.1) and then present the SBP procedure in three detailed steps ($2.2). We will present SBP as a general pretraining recipe by introducing generic setup that includes pretraining dataset, an LM architecture, and collection of evaluation benchmarks. We defer the concrete compute-matched experiment design to $3.
2.1 DATA-CONTRAINED PRETRAINING SETUP
We consider a conts-conttrineed setup where the goal is to train the best-performing LM given ac- cess to fixed document collection Prretrain (c.g., snapshot of the entire internet). To establish a controlled experimental framework, we also choose transformer architecture with parameters and collection of held-out evaluation benchmarks Perf (e.g.. perplexity, few-shot QA accuracy). Recall that a transformer takes in a sequence of tokens and outputs a sequence of conditional prob- abilities of each token given all previous tokens. Applying the chain rule for joint probability, we can use transformer to calculate the probability (3) of observing a particular text input y, or the conditional probability po(y)æ:) of one piece of text y followed by x.
Under such setup defined by (D (Dpretrains Pa, Perf), pretraining searches for the best persfoeerfrnnn transformer weights by maximizing the sum of the log-likelihood of pretraining documents arg max, Ede log Pa (dd) and then evaluates the performance through Pfr Statistically, this objective treats each document as an independent sample from a hypothetical distribution of all documents and attempts to learn this marginal distribution. However, this modeling assumption overlooks the structural similarities shared between natural language texts (e.g.. Figure 1). We next present the SBP procedure that fills this gap.
2.2 SYNTHETIC BOOTSTRAPPED PRETRAINING
At a high level, SBP finds related document pairs (di, d2) from the pretraining dataset D Prretrai and trains conditional synthesizer po (dl d1) using the same transformer architecture parametrized by
3

---

Published as a conference paper at ICLR 2026
8. It then uses it to synthesize large collection of documents Speerain to perform joint pretraining (P pretrain Speerain- } The fact that SBP trains a data synthesizer from Preerain itself also distinguishes it from extensive existing work that relies on readily available "teacher" LM.
Nearest neighbor pairing. In preparation for training the conditional data synthesizer, SBP first curates pairs of related documents. To efficiently perform similarity search at pretraining scale, adopt the Approximate Nearest Neighbor (ANN) methodology [38], which embeds each document as a quantized vector normalized to the unit sphere and then performs massively paral- lelizable linear algebraic operations. In implementation of SBP, we inner-product similarity, which we denote by (d1, d2). Then, we select a subset of pairs whose similarity score exceeds a certain threshold a: Dst {[d], d2) E pretrin x 'D pretrains (d,, > a)- We provide the implementation details of paired data curation in $A.2.
Step I:
Step 2: Synthesizer-tuning. SBP exploits the correlation between pairs of related documents by maximizing the conditional probability of d2 given d1: ASt arg max, da) log (d2 ) which we obtain by summing over the log conditional probabilities corresponding tokens from document d2. We refer to this step as "synthesizer-tuninng" as we are training a conditional proba- bilistic model that synthesizes related d2 from given d1. When performing synthesizer-tuning we initialize pe at the transformer weights that has gone through normal pretraining. As result, the model is equipped with the knowledge of individual documents at initialization, but not the conditional relation between them. Importantly, each document d1 can be associated with multiple instances of d2, encouraging the synthesizer to produce diverse, high-entropy outputs rather than deterministic synthesis.
Step 3: Data synthesis at scale. Finally, SBP synthesizes Spretraio through a hierarchical sampling process: (I) First sample the seed document d1 from peetrin uniformly at random; (II) Then sample synthesized document d2 from (- This process achieves synthetic data diversity utilizing two sources of variation: first through the variation of the seed documents d1, which comes from the diversity of the pretraining document D Pretrain itself, and second through the entropy of the con- ditional distribution (-(d), which stems from the diverse inter-document correlations captured in DST- While the procedure is empirically motivated, it actually admits a statistically principled Bayesian modeling of the distribution of natural language texts, which we explain in $5. For now, we focus demonstrating the empirical effectiveness of SBP.
EXPERIMENT SETUP
In this section, we present our concrete experimental implementation of SBP. In a nutshell, we cu- rated pretraining dataset of 582M high-quality documents totaling 482B tokens from DCLM [34], designed 3B and 6B transformer architectures modified from the Llama 3 implementation [16], and chose nine commonly used benchmarks targeted at general world knowledge and commonsense rea- soning (S..1). We propose a compute- matched comparison scheme to validate SBP against natural reference methods at compute scale of up to IT total training tokens in our largest experiment 1)3,1), bringing validation at a scale relevant for fronticr LM development.
3.1 COMPUTE-MATCHEES COMPARSION
We propose a compute-matched experimentation framework to rigorously compare SBP against two natural references: repetition baseline where we repeat D prrttrin multiple times to utilize the available training compute and an oracle upper bound that enables the model to access as many unique documents possible. Operationally, total tokens seen during training, which is proportional to the training FLOPs given a fixed batch size and context window. We validate SBP across three different settings:
control the training compute by controlling the
200B-scale: In this setting, we cap the training compute to be 200B tokens and cap the data access at [|Dpeerain| =10B tokens.
1T-scale (3B): We also consider a larger scale closer to frontier model training, where we cap the training compute at IT tokens and data access | pretrain =50B tokens.
4

---

Published as a conference paper at ICLR 2026
IT-scale (6B): To validate SBP on larger models, we additionally train 0B-parameter model with the same IT token budget and 50B unique data access.
For each training scale, pretrin with different sizes is sampled uniformly at random from the 582M documents pool. Given the ence methods against which we compare SBP.
comparison scheme, we next introduce two refer-
Repetition baseline. Since the compute budget typically exceeds the total number of unique to- kens |P I, natural baseline to use the additional compute is to repeat prettain over multiple epochs. By design, in both 200B-scale and IT-scale, we repeat the pretraining dataset 'D preetrain 20 times to exploit the available compute budget. In practice, when the pretraining dataset comes from a mixture different sources, higher-quality documents can be seen as many as 30 times during pretraining, while lower-quality texts may appear only once. [39] systematically evaluates the rep- etition baseline as proposal to scale LMs under data constraints and finds that repeating D Prretain up to 4 times yields nearly performance degradation compared with having access unlim- ited fresh data, but after around 40 times, repetition yields rapidly diminishing returns. Therefore, choice of 20 times repetition with camputte-macched comparison strikes a reasonable balance between efficient execution and maximizing performance gains from a fixed 'D pretrai via repetition.
Oracle upper bound. Besides showing improvement against the repetition baseline, we also eval- uate an oracle upper bound with unlimited data access. The motivation behind this is to contextualize the numerical improvement delivered by SBP. As we shall see in the next section, because differ- benchmarks respond differently data size changes, SBP deliver an improvement large as 3.74% on some benchmarks but only 0.14% on others (Table 1). Also, as performance on LM benchmarks tend scale logarithmically [44, 29] against data improvement, the numerical differ- ence quickly caps out as we move from the 200B scale to the IT-scale. By introducing this oracle upper bound, we can contrast the SBP improvement against this "oracle" improvement.
For the 200B-scale experiment, we implement the oracle upper bound as having access to 200B unique tokens from our document pool of size 482B tokens. For the IT-scale experiment, we unfor- tunately do not have IT unique documents due to the large fraction of duplicates from DCLM. As a surrogate, we utilize all 482B unique tokens as the dataset for training the oracle upper bound at the IT-scale. We provide partial justification for this by performing scaled-down comparison at 400B training tokens, with one model having 400B unique tokens and the other one having 200B unique tokens repeated twice (SC.1). We find that the two models (400B unique and 200B repeated twice) yield nearly identical performance.
Training recipe. For both the repetition baseline and oracle upper bound at all scales, we use a batch size of 2,048 and context window of 4,096, resulting in a throughput of 8M tokens per step. We apply a cosine learning rate scale with 5% warmup to a peak learning rate of le-2, followed by subsequent decay to 5e-5 towards the end. Under this setup, pretraining costs 11K v5p-TPU hours at 200B-scale 59K v5p- TPU hours at 1T-scale (3B), and 265K vSp-TPU hours at IT-scale (6B). We adhere to this hyperparameter throughout the paper, including the SBP experiment presented next.
EXPERIMENT RESULTS
We perform SBP experiments under the manputte-matched framework outlined in $3 three levels of training compute budget: 200B-scale, IT-scale (3B), and IT-scale (6B). After joint training real and synthetic data { (Dpretrain Sprerain)) we find SBP consistently improves upon the repetition baseline across all scales (Table 1). In this section, focus presenting the performance of SBP and evaluating the quality of the synthesized pretraining data. We defer the implementation details of SBP to $A.2.
4.1 MAIN BENCHMARK PERFORMANCE
At the 200B-scale, start with the source dataset of D Prretraill 10B and curate SBP dataset of ||Spretmai|| =75B tokens (detailed ablation in SA.3). We perform joint training on {D (Dprttrinns Speetriin) with the principle that we do not repeat any synthetic documents during training. This means that
5

---

Published as a conference paper at ICLR 2026
Table I: Computed-matcher comparison of Synthetic Bootstrapped Pretraining (SBP) and oracle performance gains over the repetition baseline. On average, SBP delivers roughly 43% of the per- formance improvement in QA accuracy for the 3B model and 58% for the 6B model, attainable by an oracle with access to 20x more unique data.
200B-scale SBP
IT-scale (3B)
IT-scale (6B)
Benchmark
Baseline
Oracle
Baseline
SBP
Oracle
Baseline
SBP
Oracle
Perplexity held-out data L
OpenWebTex12
5.74
-0.53
-1.02
-0.12
-0.21
LAMBADA
6.87
-0.85
-1.86
4.33
-0.03
-0.22
3.63
-0.06
-0.25
Five-shot MMLU
3.83
-0.36
-0,51
3.17
-0.06
-0.05
3.08
-0.08
-0.13
Q4 accuracy
ARC-Challenge ob-
35.32
+1 +1.28
+2.82
42.66
+1.62
+3.84
47.44
+0.77
+0.17
ARC-Easy (0-ba0)
68.94
+2.65
+4.29
75.63
+0,42
+2.11
78.70
+0.51
+0.85
SciQ (isbx)
90.50
+1.00
+2.40
93.20
+0.80
+0.50
92.90
+1.90
+1.80
Winogrande (bobor)
60.14
+1.90
+5.53
65.19
+1.42
+2.92
70.
+0.47
+2.36
TriviaQA (1-m0t)
22.51
+3.36
+7.37
36.07
+0.25
+0.59
40.64
+0.49
+3,19
WebQS (1:00)
8.56
+3.74
+10.83
19.34
+0.54
+0.44
19.88
+3.79
+5.22
Average QA accuracy
47.66
+2.32
+5.54
55.35
+0.84
+1.73
58.29
+1.32
+2.26
out of 200B token training budget, we spent 37.5% of it on the 75B synthetic tokens from Spretrain without any repetition, and the remaining 62.5% on the real dataset prettain repeated 12.5 times. As shown in Table 1, SBP consistently decreases test loss and improves QA accuracy. On average, SBP captures 2.32/5.54 2.54 =42% the improvement in QA accuracy delivered by the oracle run with 20x additional data access.
The training dynamics SBP partly reveal its core mech- anism. As can see in Figure initially, the baseline performs similarly to the oracle, since their training data
Test loss (OpenWebText)
2.7
2.6
share the same distribution, and when the number of to- kens seen is small, there is no distinction between the two. Then gradually, the oracle becomes a better model than the baseline, as it has access to unlimited unique training data. For the SBP dynamics, initially performs worse than both the baseline and the oracle, which is expected since the quality of the synthesized data at most matches that of the real data. However, gradually, SBP continues to scale while the baseline has plateaued. This suggests that Spretrii offers signal D pretrain alone cannot capture.
2.4
Baseline
SBP
2.3
Oracle
25
50
100
Compute billions of tokens)
Figure 2: Training dynamics (200B-scale).
Lastly, to validate the benefit of SBP across different training scales, we implement a larger ex- periment with [| [Dpettrii|] =50B unique tokens under compute budget of 1T total training tokens using both 3B and 6B 68-parameter models. Based on the ablation studies presented $A.3, we in- clude ||Speerain!! || =125B synthetic tokens for the 3B model and |[Spenrinn =250B synthetic tokens for the 6B model, adhering to the principle of no repetition for synthetic data. Examining the results Table we observe that while perplexity- based measurements plateau for the 3B model [36], benchmarks like ARC-Challenge and Winogrande continue to show significant gains. On average, SBP recovers 1..41.73 =48% of oracle's QA accuracy improvement for the 3B model. The results are even more pronounced for the 6B model, where SBP delivers relative improvement of 1.32/2.26 58% compared to the oracle. This demonstrates that SBP's effectiveness scales fa- vorably with model size. The increased optimal synthetic data ratio for the 6B model suggests that larger models possess greater capacity to exploit the additional information in the synthetic corpus.
4.2 ANALYSIS OF SYNTHETIC DATA
In this section, we provide some qualitative and quantitative analyses of the synthesized documents gain insight into the SBP procedure beyond what is measurable by the benchmark performance.

---

Published as a conference paper at ICLR 2026
Real document
Synthesis
Synthesis II
HOME
> Chowhoand > California >
Coffee Love
Discuss
Coffechouses in SD
recest tap San Diego, thing wanted to find was coffee. I was wocried it might be dif- licult with the huge decrease in Asian flavors, but Mexican.
Coffectouses: San Diego. best?
I always looking for guod colfeehouses Sun
Café Lestat, by for, stand-out. (photo by Thomas Moriatty)
Diego. am looking for coffechouees with lobvi- ousdy) excellent coffeebrans, burisaas who know their business (good microfosm, good cremu, they should know the diffioeett rutios steused milk, cappuccinu, latte etc., esc), good snacks ppaninis, cakes esc). good (cold) flavored coffeedrinks and good ambiencc. So haven't I found would to hear more suggestion.
pleassatly surpised.
Italian, Cuban flavors that are peecty common in Califoraia are now un the mix. While I did find a low highly recommended spots, the majority of locations checked out nothing write house about. the they were not being sold out, which I consider a plus.
If the coffee soene in San Diege sounds like the one in New York, there's resson. The upper cehelonss of NYC's drink scese
foamed milk,, espresso
who have taken over the majosity of coffee houses in San Diego. This probably partly die the fact that both cities have large stu- cennt populaations,
who
but
home large drinker hases, boch have a large number expresso shops, and both offer top-zotch, hand- crafted eappaccinos. But if there is one city America that deserves have
One major facoor that really needs be taken into the type of machine being used. There
Cafle Calabris By far the best coffeeetouse SD.
Excellent coflee and baristas (best cappuccimo in
two types of expresso machines: the cheap ones that spit out lois of pressure und doa't it have any control over the shot, and the expensive ones types
SDJ, great puninis. Only downside that they very carly.
thove
of New York, it's San Disgo.
amazinng shoes and utlizz superfine grinds you usually find in Italy. The lat- ter types are much harder to find and cost signili- cantly more.
that
Café Lestat: Good example what huppens with
There are just under
coffee shops San Diego, with almost half of them located on Uni- versiyy Ave. alone. So finding the perfect coffee shop crucial. We speat whole just ing around the area, hunting the of voffee isself, it's hard Lestst. The buristas are amaxiing and their methh- ods are pristine
good coffee and average baristas. Lesiat also
coffeebeuns from Callé Calabria but make
sad see they don't know cellent espresso, cappuccino.
Another factor consider the type of beans be- ing used. they are done correctly, great expresso can be made from any type bean, but it a experienne to find a place that does...
E. Street Café: Averago expresso but stragge fla- vored...
Figure 3: Comparison of original text with synthesized text variations.
Qualitative examples. We first show some samples of synthesized documents from the 200B- scale experiment (Figure 3) with more samples from IT-scale (3B) presented in $B.4. On the left, we display a real document about a practical, first-person guide to the coffee houses in San Diego. Then, we present two synthesized texts that exhibit notable differences in both framing and depth, with varying degrees of fidelity to the seed document. Synthesis sticks to the same topic but shifts toward an expository essay on espresso machines and bean quality, with little mention of specific coffee shops. Synthesis II adopts promotional, comparative style, linking San Diego's coffee culture to New York's s and praising Café Lestat in way that departs from the original's 's balanced assessments. Note that SBP provides no instructions on how the synthesizer should use the seed texts to write new documents. The model spontaneously learns to introduce new content and style into the discussion while staying on topic. In fact, is challenging to manually craft prompt to an instruction- -uned model that would output either Synthesis I or II with the real document as input. This example highlights how SBP differs from existing paradigms of data synthesis - the output somehow first abstracts the seed document and then synthesizes new text with more generalized narratives, genres, and intent. We provide more extensive analysis of this observation in $5.
Quantitative Analysis. In addition to qualitative examples, we also conduct quantitative evalua- tions to assess the quality of the generated texts. We measure text distributions for the synthesized document at 200B-scale and IT-scale. To establish a reference, we also conduct the same evaluation on the real documents. We measure five basic quality indicators:
Repetition: A document may contain too many repeated sentences or patterns. Repetition rate refers to the fraction of documents that exhibit this problematic behavior.
Duplicate @ 1M: Another failure mode of synthesis is when the documents sampled from the synthesizer distribution are nearly duplicates of each other. Duplicate@ IM refers to the frac- tion of unique documents (determined by Jaccard similarity documents are sampled from the text distribution.
threshold of 0.6) when 1M
Non-factual: A common failure mode of synthesis is the generation of content that contradicts established knowledge or facts. Non-factual rate refers to the fraction of documents that contain verifiable factual errors, as determined by automated fact-checking tools.
The synthesized d2 is considered relevant to di if they pertain the same topic, event, entity, person, place, or object. Pair-irrelevance refers to the fraction of synthesized d2 that is not relevant to d1. indicating the synthesis is not rightly using information from d1.
Pair-irrelevance:
Pair-copying: d1 and de are considered near-duplicates if they are almost identical, except for some extra white spaces, line breaks, or punctuation. Pair-copying refers to the fraction of synthesized d2 that is near duplicate of dj.
7

---

Published as a conference paper at ICLR 2026
Operationally, we implement Repetition, Pair-irrelevance, and Pair-copying using LLM-s-judge (prompts and more implementation details given in $B.3) by sampling 1,000 examples from each distribution and estimating the fraction of documents satisfying each criterion. For Non-factual (prompts and details given in $B.2), we sample 10,000 examples and conduct a comprehensive ex- amination of factual errors ensure broader coverage of the generated data. For Duplicate @ M, we use rule- based filtering to detect the fraction of duplicates based on IM documents sampled from each distribution. We present the result in the table below, All metries are lower for better data.
Table 2: Quantitative evaluation of documents sampled from the synthesizer at 200B-scale and IT- scale. We can see that the synthesized documents preserve topics and are not are simple duplicates.
Repetition J
Duplicate@ @ 1M
Non-factual
Pair-irrelevance J
Pair-copying
200B-scale
4.3%
0.8%
15.1%
25.6%
0.1%
1T-scale (3B)
3.9%
0.8%
8.7%
7.8%
0.9%
IT-scale (6B)
2.6%
0.3%
6.5%
6.0%
0.3%
Real data
1.8%
0.7%
1.8%
n.a.
n.a.
At high level, Repetition and Duplicate @ IM measure a basic text quality that independent of the specific pair-synthesis strategy employed by SBP. They aim to detect two simple failure modes: text repetition, common failure pattern in generations from small language models (3B in our case), and the lack of diversity, a common issue with synthetic data that relies on variation induced by the sampling temperature. From Table 2, we find that both 200B-scale and IT-scale synthesis match the quality of real data as captured by these two metrics. We note that the absence of repetitions and duplicates is not, in itself, indicator of high-quality or educational text, but rather basic sanity check that ensures the synthesized texts are diverse.
Non-factual failure stems from hallucinations that introduce non-existent entities relations incon- sistent with reality. We find that synthesis at the IT-scale (3B) significantly reduces these errors compared the 200B-scale. Furthermore, with the 6B-parameter model, the non-factual rate fur- ther decreases to 6.5%. This indicates that as the data synthesizer is trained on more data and with larger models, the factuality of the generated outputs tends to converge toward that of real data.
Pair-irrelevance and Pair-copying, on the other hand, measure how synthesized d2 relates to the seed d1. There are two failure modes we would like to detect: first, when d2 completely irrel- evant to d1, and second, when d2 merely copies the content of d1. We observe that both 200B- scale and IT-scale synthesis avoid simply copying and pasting dj. More interestingly, we observe that the I'T-scale demonstrates substantially higher relevance than the 200B-scale, which intuitively makes sense as the synthesizer learns more diverse relations among [D |Ppretraill -00 documents than [D PPeertrnnl = 12M corpus.
At this point, we have shared the main results of the experiment. the appendix, we present the implementation details of SBP in $A.2, ablations involving synthetic data mixture ratio in $A.3, additional analysis of synthesized documents in $B, and comparsion with using a larger 6B model
in SC.2.
STATISTICAL FOUNDATIONS OF SBP
In this section, we present Bayesian interpretation of the SBP procedure, offering one potential explanation for the origin of the SBP improvement. We will formulate a hicrarchical model of natural language texts 1$5.1) and demonstrate that SBP implicitly enables LMs to learn posterior standard pretraining cannot capture. We conclude by connecting our findings from this idealized model to the reality of LM ($5.2). We begin with the observation that the pretraining objective models the marginal likelihood of documents:
arg max log po ( (Dpretrain) arg max
log (d).
Dpmmni
However, different natural language documents share structural similarities (Figure 1), which sug- gests potentially more complex underlying joint distribution that will explore next.
8

---

Published as a conference paper at ICLR 2026
5.1 A HIERARCHICAL CONCEPT MODEL FOR NATURAL LANGUAGE
In the transformer example from Figure I, both the arXiv preprint of the transformer paper and its code implementation derived from the abstract concept of "transformer neural network". From this perspective, we can view the generation process of natural language documents as a hierarchical sampling process where we first sample a collection of abstract concepts c(0) (e.g.. the idea of transformer) from semantic space of all concepts C and then generate new documents d(--) conditional on c(a), If we adopt this view, we can think of the pretraining document as follows.
Concept sampling: Sample fixed concept collection {c(0), } P(c).
Document generation: For each concept c( generate docuemnts from {dit))), ~ P(d)c())
constituting one part of the pretraining dataset.
Under such model, the structural similarity between documents generated from the same concept is modeled as probabilistic dependence. The standard pretraining objective (1) then neglects inter- document correlation and only learns the marginal distribution P(d) - Jesc P(d)d))(eee In this view, the model learns to generate plausible text by first generating a core concept and then per- forming the generation P(dle). In contrast, the synthesizer-tuning objective models posterior of c given d. To see this, we additionally assume that the curated pairs (d1. d2) come from the same un- derlying concept c. Then, the wythesszer-tuning objective ($A.2) forces the LM to perform distinct task: (1 = Seec P(dzlc)PPcld, )dc. Here, we use Bayes' rule and the conditional indepen- dence assumption P(d2lc, d1) = P(dzlc), which says that the documents from the same concept are conditionally independent given that concept. As a result, to successfully model P(d2(d1), the synthesizer must first perform posterior inference to infer the latent concept given the document d1, and then use this inferred concept to synthesize a new document d2, a signal that is ignored by the standard pretraining objective. To illustrate this concretely, we perform post-hoc analy- sis by prompting an LM to identify the shared concepts between the synthesized document and its seed (Table 4 in $B). We can see that while it is difficult describe a synthesized document the outcome of a simple transform, such as paraphrase or summarization, it always share common underlying concept with its seed origin.
The additional signal from the posterior then enables a form of self-distillation. The syathesizer, by learning more complex conditional objective, becomes more knowledgeable "teacher" model that has learned to infer the latent structure of data. The synthetic data it produces is then the knowledge "distilled" from this teacher [26]. The final LM training acts as a "student" learning from real and synthetic data, enabling it to uncover information that real data alone cannot reveal.
5.2 FROM IDEALIZED MODELS TO LANGUAGE MODEL REALITY
For real text documents, we do not know the true data-generating process, and any parametric as- sumption would be incorrect. This is where the power of the transformer neural network shines. A transformer a mapping-first [7] approach. It does not require explicit modeling of the underlying parametrie model. Instead, as universal function approximator [10], it directly learns the complex conditional distribution ps (d2 (d) from paired data alone.
In this context, the transformer's ignorance of an explicit hierarchical model is its blessing. It by- passes the impossible step of modeling the true hierarchical distribution of language and instead brute-forces the learning of the exact transformation required: the end-to-end process of posterior inference and subsequent synthesis. The self-distillation framework - synthesizing data from this conditional model and then training it is all that is needed. We never need to introduce an explicit hierarchical model to perform the forward P(d[c) and backward pass P(cld) in the latent space. The entire procedure implicitly carried through the synthesizer-tuning update with the latent concept e integrated, demonstrating a powerful insight for scaling LMs in the real world.
6 DISCUSSION
Before the prevalence of pretraining [49, 15], we would need 40M pairs of English and French sen- tences [60] to grant LM the ability to translate from one language to another. In contrast, any modern LM [19, 42, 21] can easily achieve this task via single prompt. Such impressive advancement owes its origin to the rich correlations between words within a document LMs have learned during pre- training. This shift from hard-to-collect, task-specific dataset to the readily available, generically

---

Published as a conference paper at ICLR 2026
crawled internet data marks a transition from relying on strong but scarce supervision to weak self-supervision that is easy to scale. As we gradually deplete this weak source of self-supervision by exhausting the available internet data, many have called for stronger forms of supervision, such as reinforcement learning [14, 41]. We instead offer an alternative perspective that continues to search for form of self-supervision weaker than next-token prediction. SBP offers a particular in- stantiation of such an effort by examining the correlations between documents missed by the current pretraining paradigm. It remains explore other forms of supervision not currently utilized.
The fact that SBP could provide any improvement originates from the poor inductive bias of the transformer neural network. For example, transformers trained on the text "A is B" can not gener- alize to "B is A" [5], requiring the user to curate synthetic data that explicitly narrates the converse relation [69]. Similarly, one can imagine an architecture with very good inductive bias such that the LM trained individually on d1 and d2 can automatically internalize the relation between the two. Despite such poor inductive bias, transformers have the impressive advantage of being more par- allelizable and scalable than their alternatives [63, 54]. Given this trade-off, SBP offers unique solution that preserves the system benefits of the transformer architecture while also enabling the learning of missed correlations by encoding such additional signals via synthetic data.
6.1 FUTURE DIRECTIONS
Document embedding with activations of pretrained LM In our implementation of SBP, we use Qwen3-0. Diven--06B--Embiee EB-Embedding [70] to obtain embeddings of DCLM [34] documents. An ideal imple- mentation of SBP would only rely on the 3B 3B-parameter model and the pretraining dataset itself to curate the paired synthesizer-tuniis
dataset. To achieve this, we can use the activations of the self- attention layer from intermediate transformer block as learned representation of documents. [30] and [68] implemented this at the much smaller scale of ~ 300M parameters and ~ 3B tokens. However, our experiments operate at much larger scale with a customized model. As result, we utilize the optimized vLLM [31] inference infrastructure for @wwen30066B 6.6B embedding models to effi- ciently index the pretraining corpus. Since the SBP procedure only requires coarse binary decision of relevant vs. not relevant, which is much weaker than fine-grained document ranking embedding models are optimized for, we leave the more involved inference infrastructure for future work.
Parametric fit of SBP scaling law One experimental consequence of LM pretraining is a clean scaling law [29, Equation 1.4] that relates the held-out test loss L(N, D) to the number of LM parameters N and the size of the pretraining dataset D. natural scientific question is how the scaling law of SBP compares to the scaling law of pretraining. In our experiments, we evaluate LNN, D) at three distinct points: (N = 3B, D 10B), (N = 3B, D = 50B), and (N = 6B, 50B). We observe that SBP delivers larger relative improvements with the 6B model, suggesting a favorable scaling behavior. There are two obstacles to a full scaling law: first, SBP is inherently large-scale algorithm that cannot be scaled down. Since SBP first uses the pretrained LM to generate synthetic text and then trains on it, if the model and dataset sizes are too small, the generated text may not even be coherent. In contrast, the scaling experiments in [29] involve model sizes ranging from 768M to 1.5B and dataset sizes ranging from 22M to 23B, which allows for efficient experimentation. Second, varying N or D implies redoing the synthesizer-tuning and subsequent data synthesis over billions of tokens. Additionally, varying D also implies redoing the nearest neighbor matching, as the neighbors are only defined given a fixed document pool. Obstacles aside, it would be interesting to see whether the SBP scaling law differs from the normal scaling law by a smaller multiplicative factor or better exponent. Since SBP additionally utilizes inter-document correlations, a form of long-range interactions, its scaling law not only helps us understand SBP but also potentially helps us better understand natural language data itself [17].
6.2 CONCLUSION
In conclusion, we introduce Synthetic Bootstrapped Pretraining (SBP) as a new LM pretraining framework that leverages inter-document correlations missed by the standard pretraining objective. We demonstrate the effeectivnness of SBP by pretraining 3B and 6B-paareeerr models from scratch for up to IT tokens under a rigorously designed compute-matched experiment setup. Qualitative and quantitative analyses of the synthesized text reveal rich variation that cannot be captured by simple paraphrases. SBP offers natural Bayesian interpretation, implicitly learning posterior that infers the latent concept in a document while being practically effective. Its bootstrapping nature grants SBP the possibility of scaling the pretraining capability of the LM beyond its current limit.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We would like thank Zonglin Li for guidance on the implementation of ScaNN, Percy Liang for discussion regarding the curation of paired data, and Andrew Ilyas, Bailin Wang, Barry Theobald, Xiang Kong, Xin Cheng, Yizhe Zhang, Zhiyun Lu for their feedback on the early draft of this paper. ZY was supported by the Albion Walter Hewlett Stanford Graduate Fellowship. EC was supported by the Office of Naval Research grant 100014-4-1-2305. the National Science Foundation grant DMS-203201 4, and the Simons Foundation under award 814641. TH was supported by a grant by HAL DSO labs, gifts from Schmidt Sciences and grant under the NSF CAREER IIS-2338866, ONR 400014-24-1 1-2--99 and DARPA Cooperative Agreement HR001 I 2520013. This work does not necessarily reflect the position or policy of the government, and no official endorsement should be inferred.
REFERENCES
[1] Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Gioro, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Mo- jan Javaheripi, Piero Kauffmann, Yin Tat Lee, Yuanzhi Li, Anh Nguyen, Gus- tavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, Harki- rat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel Ward, Philipp Witte, Cyril Zhang, and Yi Zhang. Phi-2: The surprising power of small language models, 2023. URL attps://wwet wwt.:wwmcrosof...o
[2] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari. Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Ja- cobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis. Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee. Yuanzhi Li, Yun- sheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majer- cak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song. Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang. Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang. Jian- wei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi- 3 technical report: A highly capable language model locally on your phone, 2024. URL org/ abaaabbsoo/0 14219.
[3] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL https:
[4] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Free- base from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: htaa:p:s//aaaattoono //waaattooooggy...///// .org/ D13-1160
11

---

Published as a conference paper at ICLR 2026
[5] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on "a is b" fail to learn "b is a", 2023.
[6] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean- can--aptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang. Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. CoRR, abs/2112.04422 abs 2222. 04426.
2021. URL bttps://arxiv..r
[7] Leo Breiman. Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author). Statistical Science, 16(3):199 231, 2001. doi: 10.1214/s//100221377 URI. https:: wwwpboo.orrg1.... 121 ss 000213726.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpp::/pppoceeeenggg..... neurips cc cctppppr_filllceeeeeeeeeeeeeeppe wwles/paperr////// file/ (ttl://6577c000ff664441111188888
[9] Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russell Webb. Distillation scaling laws. In Forty-second International Conference on Machine Learn- ing, 2025. URL https:tt//// http//ppeeeevveeeeeee http://wwwwee///ouumreeo worum? Le-rrummd--1EEEAppBSS..
net/
[10] Emmanuel J Candès. Ridgelets: Theory and applications. Department of Statisties, Stanford University, 1998.
[11] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint 1XX3::8803005555 2018.
[12] Common Crawl. Common crawl. https:/ wwt.//ccwwowoocooooooooooooc ortp///..w..///000......
[13] DatologyAl, Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Charvi Bannur, Christina Back, Darren Teh, David Schwab, Haakon Mongstad, Haoli Yin, Josh Wills, Kaleigh Mentzer, Luke Merrick, Ricardo Monti, Rishabh Adiga, Siddharth Joshi, Spandan Das, Zhengping Wang, Bogdan Gaza, Ari Morcos, and Matthew Leavitt. Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining, 2025. URL https://
wwxivoorg//ee abs 2508. 10975.
[14] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen. R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting
12

---

Published as a conference paper at ICLR 2026
Pan. S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao. Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng. Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, 7. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https: : / arxiv. org./...orr.....oooooo.......... org/ abs 22../251..11 12948.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. Computational Linguistics. doi: 10. 10.18653/v1/N19- 1423. URLhttps: http///aaaaattoctco.oooo org 1og/N19-1423.
Minneapolis. Minnesota, June 2019, Association for
[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chun- yang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Olanno Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Fric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Syn- naeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen. Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jen- nifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yean Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jemkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kar- das, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Niko- lay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Kr- ishnan, Punit Singh Koura, Puxin Xu, Qing He. Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Railcanu, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie. Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodin- sky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan
13

---

Published as a conference paper at ICLR 2026
Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whimey Meers, Xavier Martinet, Xi- aodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie. Xuchao Jia, Xuewei Wang, Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen. Yiwen Song. Yuchen Zhang, Yue Li, Yun- ing Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, 7oe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiane Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide. Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, I Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Pa- tel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Niko- lay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz. Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchan- dani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Ya- mamoto, Sharadh Ramaswamy, Shaun Lindsay. Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Sub- ramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vitor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xi- aocheng Tang. Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL https: ttapp:///rxxt /arxiv. org/ abs///////// abss/407 2407 21783.
14

---

Published as a conference paper at ICLR 2026
[17] W Ebeling and T Pöschel. Entropy and long- long-range correlations in literary english. Europhysics Letters (EPL). 26(4))241-246 May 1994. ISSN 1286-4854. doi: 10. 10.1000025555075/2266 001. URL http: / dx. doi org/ 10. 1209/ 5055//005555555////2 4/00..
[18] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprini arXiv:2101.00027, 2020.
[19] Team Gemini. Gemini: A family of highly capable multimodal models, 2024. URL https:
// //arxiv.cgg/abbll...... htt.g///bb//////// /2322.22222222 11805.
[20] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https: / arxiv....v....////////aa//////v/ org/ abs 2306. 11644.
[21] Tom Gunter, Zirui Wang, Chong Wang. Ruoming Pang. Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap. Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebroek, Sam Wise- man, Syd Evans, Tao Lei, Vivek Rathod, Xiang Kong, Xianzhi Du Yanghao Li, Yongqiang Wang, Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, Al Rashid, Albin Madappally Jose, Alec Doane, Alfredo Bencomo, Allison Vanderby, Andrew Hansen, Ankur Jain, Anu- pama Mann Anupama, Areeba Kamal, Bugu Wu, Carolina Brum, Charlie Maalouf, Chinguun Erdenebileg, Chris Dulhanty, Dominik Moritz, Doug Kang, Eduardo Jimenez, Evan Ladd, Fangping Shi, Felix Bai, Frank Chu, Fred Hohman, Hadas Kotek, Hannah Gillis Coleman, Jane Li, Jeffrey Bigham. Jeffery Cao, Jeff Lai, Jessica Cheung, Jiulong Shan, Joe Zhou, John Li Jun Qin, Karanjeet Singh, Karla Vega, Kelvin Zou, Laura Heckman, Lauren Gardiner, Margit Bowler, Maria Cordell, Meng Cao, Nicole Hay, Nilesh Shahdadpuri, Otto Godwin, Pranay Dighe, Pushyami Rachapudi, Ramsey Tantawi, Roman Frigg, Sam Davarnia, San- skruti Shah, Saptarshi Guha Sasha Sirovica, Shen Ma. Shuang Ma, Simon Wang, Sulgi Kim, Suma Jayaram, Vaishaal Shankar, Varsha Paidi, Vivek Kumar, Xin Wang, Xin Zheng, Walker Cheng. Yacl Shrager, Yang Ye, Yasu Tanaka, Yihao Guo, Yunsong Meng, Zhao Tang Luo, Zhi Ouyang, Alp Aygar, Alvin Wan, Andrew Walkingshaw, Andy Narayanan, Antonic Lin, Arsalan Farooq, Brent Ramerth, Colorado Reed, Chris Bartels, Chris Chaney, David Riazati, Eric Liang Yang, Erin Feldman, Gabriel Hochstrasser, Guillaume Seguin, Irina Be- lousova, Joris Pelemans, Karen Yang. Keivan Alizadeh Vahid, Liangliang Cao, Mahyar Na- jibi, Marco Zuliani, Max Horton, Minsik Cho. Nikhil Bhendawade, Patrick Dong, Piotr Maj, Pulkit Agrawal, Qi Shan, Qichen Fu, Regan Poston, Sam Xu, Shuangning Liu, Sushma Rao, Tashweena Heeramun, Thomas Merth, Uday Rayala, Victor Cui. Vivek Rangarajan Sridhar, Wencong Zhang. Wenqi Zhang, Wentao Wu, Xingyu Zhou, Xinwen Liu, Yang Zhao, Yin Xia, Zhile Ren, and Zhongzheng Ren. Apple intelligence foundation language models, 2024. URL. https: / a arxiv org/ abs /2007 21075.
[22] Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Ku- mar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference Machine Learning, pp. 3887-3896. PMLR, 2020.
[23] Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences abs 110./ 08878.
by editing prototypes, 2018 URL
[24] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wer Chang. Realm: Retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020.
[25] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0338 2020.
[26] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, abs/a/ss/// /1003. 02531.
2015. URL wwtph::tt///atteeeee
15

---

Published as a conference paper at ICLR 2026
[27] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifi- cation, 2018. URL https::/p/...m.... http:::/wwww.......e.eeooo wwt.aa////..o.... 06146.
[28] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 10.555 2017.
[29] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL httpt: wwtp:/iv.org///////////
08361.
[30] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. In International Con- ference on Learning Representations, 2020. URL https: wwwpneeeeewww.../o///p net forum? 1d=Hk1BjCEKvy
[31] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Sym- posium on Operating Systems Principles, 2023.
[32] Guillaume Lample, Alexandre Sablayrolles, Marc Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8546-8557, 2019. URL wwtptttp///t/. htttt::://procceeee http:pppocennggg.cooompppp wwwprcceedingsneeeriieee nwwripu.pp.... cc/ ptcpppppe/2219 hash/ 1wt.////aaceeeeeeeeeeeeee wtt.:/EESSSSDTT0ccc00000220. Absttrttbbbtat........ html.
[33] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmentt ings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
generation for knowledge-intensee intensive nlp tasks. In Proceed-
[34] Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Rein- hard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Al- balak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel II- harco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song. Sujay Sanghavi, Fartash Faghri, Se- woong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-N El-Nouby, Hadi Pouransari, Alexander To- shev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2024.
[35] Zonglin Li, Ruiqi Guo, and Sanjiv Kumar. Decoupled context processing for context augmented language modeling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL http::///ppeervvieeee net http://mmmmmmmmmmmmmeee
[36] Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: implicit bias matters for language models. In Proceedings of the 40th Interna- tional Conference on Machine Learning, ICML'23. JMLR.org, 2023.
[37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Ro(bert)a: A robustly optimized [bert] pre- training approach, 2020. URL /wwpnepeeviieeeeeeeeoeeeeeeeeeee net forum ttw.rorrmmmnnsssSSSSsssss d=Syx5004tvv..
16

---

Published as a conference paper at ICLR 2026
[38] Yu. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs, 2018. URL https:/::// httppss/////xvvva//aaa/ wrg.iiooorrooooooo / 1 1603. 09320.
[39] Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Alek- sandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained lan- guage models. In shirty-seventh Conference on Neural Information Processing Systems, 2023.
https:/ /oppereeeeeieeeeee net. forem?tormmtt tot.oooommmdddduuiioddd J3R
[40] Thao Nguyen, Yang Li, Olga Golovneva, Luke Zettlemoyer, Sewoong Oh, Ludwig Schmidt, and Xian Li. Recycling the web: A method to enhance pre-training data quality and quantity for language models, 2025. URL /ttetps://arxiee org/ awss/500 04689.
[41] OpenAl. Learning to reason with Ilms, September 2024. URL https: /ttttp::///peaa...om com/
[42] OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bermadett-hhapiro, Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dumning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou. David Farhi, Liam Fe- dus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lppes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin. Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Eukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Med- ina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkín, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk. David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeon- woo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ash- ley Pantuliano, Giambatiista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Hen- rique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sas- try, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng. Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone. Arun vegayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt
17

---

Published as a conference paper at ICLR 2026
Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo. Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv.crm wwwpp/wwwwooooooooooo.oooo.. 2303. 08774.
[43] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Infor- mation Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. URL ht http::/pprceeeennn.eeerreeeeeeee http://reepp....ppppppeeeeeeeee. httpee//iiieespppe///222/ pdf.
[44] David Owen. How predictable is language model benchmark performance?, 2024. URL
https: wwwpaiiiv..gggbbooe.
04757.
[45] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.0603// 2016.
[46] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cap- pelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The re- finedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint urXiv:2306.011 116, 2023.
[47] Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, 2018. URL https: //arxiv.org/ wwwpxw..org/abs/1822..2. 05365.
[48] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018
[49] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners, 2018. URL https:ttt///tc /tt////famffffff///////// wawerr-aannnagggggaaea models /llagaaggggeeeee modeeee-ooeellll..le .pp...
cloudfront... wewndffott.net/beeee
[50] Jack W Rae, Sebastian Borgeaud, Trevor Cai. Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 1 2021.
[51] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res,, 21(1), January 2020. ISSN 1532-4435.
[52] Yangjun Ruan, Neil Band, Chris Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts. arXiv preprint 2rXiv:2503.18866 2025.
[53] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[54] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of-expprrs layer, 2017. URL htttts:///aaattt
06538.
[55] Weijia Shi, Sewon Min. Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Tweifth International Conference on Learning Representations, 2024. URL https: /ppp//eeeeeeeeeeeeeeeeeeeee net forum? d=t..VswTTHHH..
18

---

Published as a conference paper at ICLR 2026
[56] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159 2024.
[57] Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-ce: Transforming common crawl into a refined long-horizon pretraining dataset. arXiv preprint arXi::2412.02295,, 2024.
[58] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023, URL wwtthhtss////tttie orgg/ abs/ 221422144.0 09864.
[59] Philip Sun, David Simcha, Dave Dopson, Ruiqi Guo, and Sanjiv Kumar. Soar: improved indexing for approximate nearest neighbor search. Advances in Neural Information Processing Systems, 36:3189-3321 189-3204, 2023.
[60] Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks, 2014. URL
absgg//bb//ia /109/ 3215.
[61]
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao. Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhe- jun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li. Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li. Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma. Xinyu Ma. Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung. Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang. Feng Wang, Haiming Wang. Jianzhou Wang, Jiaxing Wang, Jinhong Wang. Shengjie Wang, Shuyi Wang. Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang. Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu. Xinran Xu, Yangehuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang. Zhen Yang. Zhilin Yang. Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang. Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL https: / arxiv org/ wtwg/bs//////ooooeeo...oo...eese abs//bbs//2000 22007. 20534.
[62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas. Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tume tuned chat models, 2023. URL https: / arxiv. www//b////////...////..//.eeeeee 200.. 09288.
19

---

Published as a conference paper at ICLR 2026
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http///ooeedddeeeooeoooeooe ngs. nwwrips cttppsercppppppeppppeeeeep wtt.ss/ppppeee/111/// Pappe...
[64] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of Ilm scaling based on human auman-generated data, 2024.
[65] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Inter- national Conference Learning Representations, 2022. URL https: httpthts/ppteeeeeeeeeeeeee// /openrevie.. net/ www.mforrriiooooo 1
[66] Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In NUT@ EMNLP, 2017.
[67] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.0035, 2019.
[68] Zitong Yang, MICHAL LUKASIK, Vaishnavh Nagarajan, Zonglin Li, Ankit Rawat, Manzil Zaheer, Aditya K Menon, and Sanjiv Kumar. Resmem: Learn what you can and memorize the rest. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 60768-60799 Curran Associates, Inc., 2023. URL bttps://proceediine neurips. cc wwwpppppppp..eeeeeee..eeee wtt.:e/e/paaeee///////ee..//
pdf.
[69] Zitong Yang. Neil Band, Shuangping Li, Emmanuel Candes, and Tatsunori Hashimoto. Syn- thetic continued pretraining. In The Thirteenth International Conference on Learning Repre- sentations, 2025. URL https: httptt::://pppeeeeeeeeee oppprevieweweeeee////e net/ Forum? wawpummmidd0777 1d77mmm001.11
[70] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https:
//w.xiv.rggaabs////// 250.. 05176.
[71] Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self- adapting language models, 2025. URL https : http//xxvvvvooo/ /
www. /tt.b////5....m. 10943.
[72] Zyphra. Zyda-2, a 5 trillion token high-quality dataset, 2024.
URL https://
huggingface co/ co/datased dazasets / /2yphha/dddmmmdddddu.
20

---

Published as a conference paper at ICLR 2026
CONTENTS
Additional details on synthetic bootstrapped pretraining
22
A.1 Data, model, and evaluation
22
A.2 SBP implementation details
22
A.3 Ablation data mixture ratio
24
A.4 Random pairs and embedding analysis
25
Additional analysis of synthesized samples
26
B.1
Analyze concepts in documents...
26
B.2
Factuality analysis
................................
29
B.3 Mideval prompts
31
B.4 Synthesized documents from the I'-scale experiment
................................
34
Additional pretraining results
34
C.1 Two epochs validation
34
C.2 Model Scaling
34
21

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DETAILS ON SYNTHETIC BOOTSTRAPPED PRETRAINING
A.1 1 DATA, MODEL, AND EVALUATION
In this section, we present the complete details of the experiment set referenced in $3.
Dataset. A typical pretraining dataset is a mixture of different sources (e.g., GitHub, arviv, Com- monCrawl, etc.) with distinct sampling weights assigned to each constituent. We simplify this real- ity by considering a fixed document collection, which is a customized version of the DCLM dataset split contains roughly 80% duplicates, as reported by [72]. Therefore, we begin with the de-duplicated dataset, which consists of 769B tokens. We clean the raw Zyphra de-duplicated data by normalizing repeated line breaks, removing long URL links, and fixing malformed Unicode characters. For efficiency reasons, we cap the context window of the synthesizer-tuning ($3) step at 8,192 tokens. As result, we additionally filter out the documents whose length is above 4,096 tokens, allowing both di and d2 to fit into the context window in the worst case when both documents are 4,096 tokens long. After all the de-duplication, cleaning, and filtering procedures, we end up with a collection of 582M high-quality documents Doretrai totaling 482B tokens. We use the notation [D prretrain| to denote the number of documents in the pretraining dataset and [ pretraial to denote the total number of tokens.
[34]. The original 4T token DCLM-
Architecture. We use the Llama 3 transformer architecture [16] to model the probability pe with the notable exception of implementing QK-norm on top of the existing design, which we empir- ically find to stabilize training. Our resulting model is 3B- 1B-parameter 26-layer transformer model with a hidden dimension of 3,072. Each layer employs grouped query attention with 24 query heads and 8 key/value heads. To validate the scalability of SBP, we also train 6B 18-parameter model with 32 layers, a hidden dimension of 4,096, 32 query heads, and a feedforward dimension of 13,056 (detailed in Table 12). The position embedding is RoPE [58] for queries and keys, with frequency Se+5. The feedforward network (FFN) has hidden dimension 8,064, and we apply prenorm to both the attention and FFN blocks. For tokenization, we implement customized BPE tokenization with a vocabulary size of 19,152. To match the 8, 192 context window design for synthesizer-tumi have mentioned, we use context window 4,096 for pretraining, so that every document in D Ppretrain can fit into the context window.
we
Benchmarks. To assess the pretraining capability of LM, we measure pretraining test loss and general world knowledge benchmarks. We evaluate held-out test perplexity (exponential of nega- tive log-probability) on 1) OpenWebText from EleutherAI [49]; 2) Narrative understanding with LAMBADA [45] and 3) Broad domain multiple-choice with MMLU [25]. We evaluate QA ac- curacy on 4) Hard scientific reasoning with ARC-Challenge [1I]: 5) Easy scientific reasoning with ARC-Easy [11]: 6) Scientific QA with SciQ [66]; 7) Common sense reasoning with Wino- grande [53]; 8) Reading comprehension with TriviaQA [28]; 9) Openbook QA with WebQS [4]. We directly evaluate the pretrained model with either zero-shot or few-shot prompts. Although MMLU is more commonly known as QA benchmark, we find that evaluating MMLU accuracy for weak models yields a highly non-smooth readout. As a result, for each MMLU test question, we prepend the question with 5-shot example of QA pairs and postpend it with the correct answer. Then, we treat each such sample as a text corpus and evaluate LM's perplexity on such a text sample. Em- pirically, find that this perplexity-basee underlying model is large enough to yield a stable readout, and also delivers smooth performance changes for smaller models. Note that those benchmarks are known to improve significantly with instruction finetuning [65]. However, we stick to our data-constrained setup and do not introduce any additional data that may confound the comparison.
MMLU correlates well with MMLU accuracy when the
A.2 SBP IMPLEMENTATION DETAILS
In this section, we present the implementation details of SBP outlined in $2.
Nearest neighbor pairing Recall from $3 that we work with 3B and 6B-parameter transformer
architectures and pretraining dataset at || pPpretrain| =10B and [2 pprrttriin || =50B scale. To take advan- tage of efficient ANN search at pretraining scale, embed the documents from D Prearain as 1,024
22

---

Published as a conference paper at ICLR 2026
Then, ScaNN [22, 59] with 8-bit quan- tization to perform efficient similarity search. We adopt an asymmetric sharding to keys and value vectors. For each value vector, we build a ScaNN search tree with VN leaves where is the num- ber of vectors in each value shard. To distribute the key shards across each search tree, employ "salting" strategy, where we create multiple copies of the ScaNN searcher and assign one key shard each salted copy of the searcher (Figure 4). This design enables us perform top-200 nearest neighbor search over PPretrrin| -00 documents within 155M CPU hours.
dimensional vectors using Siven--Embeddinee
Key sharding 1024
SCaNN searcher
ScaNN searcher
Distributed search searcher.
ScaNN searcher
Pretraining data 60M documents
8-bit quantized index tree
Salting
Value sharding Split into 32 shards
ScaNN searcher 8-bit quantized Index tree
Aggregation by key Only keep the top 200 neighbors
Figure 4: ScaNN system design for efficient distributed search.
At all scales, after obtaining the top 200 neighbors for each sample, we select the pairs whose similarity score is greater than 0.75. We chose this cut-off it would later lead to tractable size of synthesizer-tuning dataset DST. To access the effect of choosing a different threshold, we provide quantitative analysis of the fraction of relevant documents around each bin of similarity threshold in Figure using the same metric defined in $4.2. We can see that larger similarity score yields pairs with higher relevance but also more duplicates. Finally, we climinate near-duplicates using rule- based filtering approach. The dedup process involves first normalizing text by removing punctuation, converting to lowercase, and eliminating numbers, followed by tokenization using SentencePiece We then generate "shingles" using 13-token sliding windows within d1. Training pairs are discarded any shingle from di appears d2.
Pair-rreeevantee (%6)
3000
0.75
2000
0.5
1000
0.25
0.0
0.00
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
Similarity score
Similarity bin
Similarity bin
(a) Similarity histogram
(b) Duplicate rate
(c) Relevance rate
Figure 5: Analysis paired data at 200B-scale. Figure 5(a): a histogram of 100K subsampled pairs grouped by their similarity score. Figure 5(b): the fraction of duplicate pairs when subsample 1K pairs around a specific similarity score. Figure 5(c): same as 5(b) but showing the fraction of relevant documents.
Synthesizer-tuning After we collected the cleaned pair data DST (previous step), we perform the synthesizer-tuning with the objective ($3). We initialize the 3B-parameter at the baseline checkpoint and finetune the model with constant learning rate of 5e-6 and batch size 16M tokens per step. Before we settled on this learning rate schedule, we first attempted the cosine decay schedule with larger learning rate. We found that the generated text has lower quality than our final design with a small, constant learning rate. We measure the Pair-novelty score (defined in $4.2) of the synthesized example for different checkpoints synthesizer tuning, and find that longer training results in better Pair-novelty.
23

---

Published as a conference paper at ICLR 2026
Synthesis at scale Finally, perform the hierarchical sampling procedure defined in $2 with temperature of 1.0 and top-p threshold 0.9. We apply a rule-based filtering that removes synthe- sized documents containing repeated occurrences of 13-token shingles. This effectively removes texts with repetition failure. We use vL..M [31] and obtain a throughput of 8.3K tokens per B200 second. This amounts 2.5K B200 hours for the 200B-scale synthesis, 4.2K B200 hours for the IT-scale (3B) synthesis, and 8.4K B200 hours for the IT-scale (6B).
A.3 ABLATION ON DATA MIXTURE RATIO
When performing joint training on a mixture of real and synthesized documents for the final SBP run, natural question arises: how much fraction of synthesized documents to include. In $4, we discussed that we utilized ||Spretrin|| =75B for the 200B-scale experiment, ||Spretraiall = =125B for the IT-scale (3B) experiment, and ||ppecraill =250B for the IT-scale (6B) experiment. In this section, we present ablation experiments for this design choice.
200B-scale At this smaller scale, we perform a comprehensive sweep over five possible values of ||S ||ppetmaill E (OB, 25B, 50B, 75B, 100B]. As seen in Figure 6, different benchmarks exhibit varying behavior when more synthetic data is included during training: the perplexity (Open WebText2 and LAMBADA) decreases monotonically with increasing synthetic data, while most QA benchmarks display a peak around |Spetrrin| 75B.
OPENWEBTEXT2
LAMBADA
MMLU-5S
Oracle
Oracle
3.0
Oracle
Perplestyy
3.4
5.6
SBP
Parplexiity
6.5
SBP
Pusplexity
3.7-
SBP
6.0
5.2
3.5
5.0
5.5-
3.4-
4.8
5.0
20
20
60
Synthetic tokens (B)
Synthetic takens
Synthetic tokens (B)
ARC
SCIQ
93.0
30
92.5-
Accuracy (%)
Accuracy (%)
92.0
Accuracy
Oracie
91.5
SBP
71
36-
91.0
70
90.5-
Oracle
Oracle
69
SUP
90.0
SUP
0
20
40
60
80
1000
0
20
40
60
80
100
20
40
60
80
100
Synthotic tokens (B)
Synthetic tokens (H)
Synthetic tokens (8)
WINOGRANDE
TRIVIAQA-15
WEBQS-1S
30
65
18
Ancuracy (96
Accuracy (%)
16.
Accuracy
63-
Oracle
SBP
26
Oracle SRP
12
Oracle
10
20
40
60
80
100
20
60
80
100
100
Synthetic tokens (B)
Synthetic tokens
Syntbetic tokens (B)
Figure 6: SBP performance with varying synthetic tokens at 200B-scale
IT-scale (3B) At the IT-scale, both data synthesis and subsequent joint pretraining become signif- icantly more expensive. Therefore, we evaluate SBP three different values of the synthetic data ||Sprerain|]] E (OB. 125B, 250B). As shown in Figure 7, we find that ||Speraall =125B produces the best-performing model across all benchmarks except LAMBADA perplexity.
24

---

Published as a conference paper at ICLR 2026
OPENWEBTENTZ
LAMBADA
MMLU-55
3.17
Oracle
4.50
4.30
3.16
SDP
Perplecity
4.2
Pemplesity
3.15
4.46
Oradle
Oracle
3.14
4.44
4.20
3.13
4.42-
4.15
3.12
4.40
0
50
100
150
200
250
50
100
150
200
250
0
50
160
150
200
250
Synthotic tokens (0)
Syathetic tokens
Synthetic tokens (B)
ARC
ARC E
SCIQ
90.0
Oracle SDP
Accuracy (%)
Accuracy (%)
92.8
77.0
Accuracy
SBP
76.5
SBP
03.6-
44
93.4
76.0-
43-
93.2
200 Synthetic tokens (B)
30
150
200
Synthetic tokens (B)
Synthetic tokens (D)
WINOGRANDE
TRIVIAQA-1S
WEBQS-1S
19.9
68.0
36.5
19.8
67.5
36.0
19.7
Accurssy
35.5
66.5
35.0
19.6
66.0
34.5
19.5
Oracle
Oracie
Oracle
63.3
SBP
34.0
SBP
19.4
SBP
0
50
1000
150
200
250
50
100
150
200
250
0
50
100
150
200
250
Synthetic tokens (8)
Syathelic tokens (B)
Syothetic tokens (B)
Figure 7: SBP performance with varying synthetic tokens at 1 I'T-salee (3B).
IT-scale (6B) We also performed the mixture ratio sweep for the 6B model at the IT-scale, eval- uating Spretaaal E {OB, 125B, 250BJ. As shown in Figure 8, we find that the optimal amount of synthetic data is around 250B, which is higher than the optimal 125B observed for the 3B model.
Discussion From this analysis, we can observe general pattern: the best-performing model is achieved when pretraining is conducted on mixture of real and synthetic data. Furthermore, the optimal ratio of synthetic data appears to increase with model size (from 12.5% for 3B to 25% for 6B). Real internet data has higher quality and therefore merits more repetition. However, as repetition yields diminishing returns, synthetic data could offer another source of signal that real research typically finds that training purely on
data cannot capture. In contrast, distillation-base synthetic data yields significantly higher training efficiency. However, this finding is obscured by the fact that such a model eventually converges to the capability of the teacher LM. This contrast reveals that the SBP mechanism does not generate compressed and denoised representation of knowledge that is more efficient for LM to learn. Instead, it offers an additional source of improvement that real data alone cannot capture.
A.4 RANDOM PAIRS AND EMBEDDING ANALYSIS
To verify that SBP relies learning specific inter-document correlations rather than generic data augmentation, we conduct an ablation study where we train the synthesizer on random document pairs. We measure the semantic similarity between the seed document d1 and the target document (either d2 or the synthesized output) using the Saeenn3-Embedding-O... model.
As shown in Table 3, the paired documents used in SBP exhibit high semantic similarity (0.79), whereas random documents have minimal similarity (0.15). Crucially, the documents generated by the SBP synthesizer maintain high degree of relevance (0.66) to the seed document. In contrast,
25

---

Published as a conference paper at ICLR 2026
OPENWEBTEXTZ
LAMBADA
MMLU-5S
3.65
2.075
Oracle
3.60
3,050
SDP
Perplesity
Purpleeity
Puplexity
4.15
Oracle
2.55
Oracle
3.025
3.50
4.10
3.45-
4.05
3.40
2.950
50
160
150
2000
250
50
100
1.50
200
250
50
100
150
260
250
Synthetic tokens
Synthetic tokens (D)
Synthetic tokens
ARC
ARC E
SCIQ
48.2
Oraale
95.0
SBP
79.4
(90
48.0
Accuracy (%6)
Accuracy (%6
94.5
79.2-
70.0
93.5
47.6
Oracle
Oracle
93.0
SBP
100
200
Synthetic tokens (B)
Synthetic tokens (D)
Synthetic tokens (D)
WINOGRANDE
TRIVIAQA-1S
WEBQS-1S
72.0
Accurayy (%)
43-
24
71.5
Oracle
Accursoy
23
42-
22
Oracle
21
Oracle
SBP
20
SBP
0
50
100
150
200
250
0
50
100
150
200
250
50
100
150
200
250
Syntbetic tokens (B)
Syuthetic tokebs (B)
Synthetic tokens (8)
Figure 8: SBP performance with varying synthetic tokens for the 6B model at IT-scale.
Table 3: Embedding similarity statistics. "Paired documents" refers to the SBP training pairs found by nearest neighbor search. "Random documents" refers to randomly paired documents. "Generated documents (SBP)" refers to the synthetic data generated by the SBP model at 200B-scale (3B). "Generated documents (Random)" refers the synthetic data generated by the model trained on random pairs. All comparisons are based on the 10B dataset.
Statistic
Paired docs
Random docs
Generated (SBP) Generated (Random)
Mean
0.79
0.15
0.66
0.32
the model trained random pairs produces outputs with significantly lower relevance (0.32). This result confirms that the SBP synthesizer successfully learns to preserve semantic relevance from the training pairs, a property that is absent when training on random associations.
B ADDITIONAL ANALYSIS OF SYNTHESIZED SAMPLES
B.1 ANALYZE CONCEPTS DOCUMENTS
In this section, we further examine the intermediate hidden mechanisms underlying the document synthesis process Specifically, we classify the hypothesized concepts inferred from real documents (see Table 4 for details) along two complementary dimensions: concept domains, which denote the broad subject areas or fields concept belongs to (c.g., science, psychology, health, culture), and concept types, which capture the abstract role or nature of the concept itself (e.g., theory, method, comparison, symbol).
26

---

Published as a conference paper at ICLR 2026
Table 4: Examples of latent concepts с inferred by an external LM (prompts provided in $B.1). From left to right, we provide summary of the real document, the inferred latent concept, and summary of the synthesized document.
Real document summary
Concepts
Synthesized document summary
Examination of Twitter's impact journalism
Opportunitics arise from Twitter
Guide Twitter user monctization
Family story about kids and doughnuis
Parenting kids' Food catering
Emotional anecclotes of parents treating kids
Minor parties" challenges in the U.S. Congress
Minor political parties in the U.S
Explains U.S. minor parties' history
Personal storieskquestinn about swollen eyes
Causes/reeameets swollen eyes
Non-personal guide treating swollen eyes.
Antarctic carbon Exation mochanisms
How life survives in Antarctic
Antarctic geography and survival adaptations
Profile of a belly dancing teacher in the U.K.
Belly doncing as a dance form
General introduction to belly dancing
Anxicty about creative work judged in dream
Dream as personal self-reflectinn
Description and reflection of personal dreams
NYC (yearly/monthly) climate extremes
NYC weather and temperature
QA NYC July heat and related topics
Tutorial for Minecraft block modding
Block ediring in Minecraft
Minecraft forum question on removing blocks
Cosmic airburst destroys Tall el Hammam city
Destruction of amcient cities
Tall Holleel-Hammee Hammam excavation as news event
Table 5: Categorize extracted concepts into domains,
Concept Domains
Examples
Culture (38.74%)
Inter-communityy conflict in Nigeria, Family-based based immigration policy, Reac- tions to Horrid Henry books, Interracial dating and bias
Health (11.8%%
Cosmetic dental appliance. Colistin toxicity in infections. Hair health tips, Portable/home medical diagnostics, Vitamin D and pregnancy outcomes
Technology (6)
Recovering deleted phone data, Video editing app review, Flash platform pros and cons, HTML. 2.0 draft process, Email attachment processing speed
Politics (3.69%)
Iran nuclear negotiations, Student loans policy reform, Democratic primary candidate choice, Catalan independence aftermath
Psychology (3.42%)
Differences in personality disorders, Exploring the strange in daily life, Aging and nostalgia, Toxic relationship breakup, Psychology research paper topics
The distribution of concept domains and types in Table 5 and 6 underscores the multidimensional nature of the knowledge space under consideration. The domains encompass macro-level socio- cultural phenomena, such as Culture, where topics range from inter-community conflict in Nigeria to immigration policy and interracial dating and bias, alongside micro-level issues of individual health and wellbeing, as exemplified in Health. In parallel, the typological elassification reveals not only subject matter but also modes of conceptual engagement: Methods comprise formalized pro- cedures (multidimensional poverty measurement, commercial real estate appraisal), Events capture historically situated crises (Mediterrancan migrant crisis, BP oil spill nationalization), and Com- parisons and Analyses facilitate interpretive framing through juxtapositions (cancer suffering: indi- vidual vs. family) and evaluative inquiries (Manchester United player analysis). Collectively, this taxonomy illustrates not only topical diversity but also a spectrum of cognitive orientations.
While real and synthesized documents share the same underlying concept, they differ in multiple ways that merit closer examination. We categorize these differences into a taxonomy of relations using a small ontology. Table 7 illustrates several relationship types, highlighting how synthesized data can reflect multiple facets that vary from real data. These relations range from scope-based distinctions (c.g., specific vs. general), to causal connections (c.g., corruption leading to reform), and to contrastive contrasts (e.g.. Constitution articles vs. Articles of Confederation). This diversity demonstrates the rich variation structure that the synthesizer captures and learns.
27

---

Published as a conference paper at ICLR 2026
Table 6: Categorize extracted concepts into abstract types.
Concept Types
Examples
Method (9.17%)
Multidimensional poverty mcasurement, Commercial real estate appraisal, Stop words search duplicates, DAT chemistry preparation
Event (6.98%)
Mediterrancan migrant crisis, BP spill nationalization, Paula Abdul stalked, Eminem-Apple music rights lawsuit, Presidents Cup U.S. golf
Comparison (5.54%)
Hobhit film adaptation length/cost, Biking superior transport, Cancer suf- fering, individual vs. family, Progress critique: 4G vs. alternatives
Analysis (5.20%)
Health effects of substances, Thai massage benefits, Scrabble word break- down, Relationship roles and challenges, Manchester United player analysis
Phenomenon (4.95%)
Secret pain; self-destruction, Car-related online humor/pranks, Transnational corporations in globalization, Hippic identity and lifestyle
Table Categorize relations between real documents d, and synthesized documents d2.
Relation Categories
Examples
Scope relation (8.14%)
d: Probiotics' possible effects HINI infection
d2: Probiotics' general digestive and immune benefits
Relation: specific application vs general health benefits of probiotics
Perspectival relation (5.51%)
d:: Personal, humorous struggles of new bloggers
dz: Objective guide pros and of blogging
Relation:
subjective experiences vs objective guidance about blogging
Functional relation (4.70%)
dj: Reviews and feedback on "Space Bound" game
d2: Forum troubleshooting for bugs in "Space Bound"
Relation: reviews/fceddbac vs troubleshooting for the same game
Causal relation (2.05%)
d1: DTEK faces corruption probe, financial risk
d2: DTEK nationalized for state-driven energy reform
Relation: corruption/tnaaccal issues nationalizaaioon/ennry reform
Contrastive relation (1.65%)
d:: Detailed summary of Constitution articles
d2: Overview, flaws of Articles of Confederation
Relation: U.S. Constitution articles Articles of Confederation: different
foundational documents
28

---

Published as a conference paper at ICLR 2026
Document Summarize and Concept Analysis Instructions
In following, you are given two documents, docl and doc2. Doc2 generated from docl.
The principle of generation is to first abstract a concept from docl, and then starting from this concept, generate doc2. Can you guess what this concept is and how doc2 was generated?
Please keep the summary and concepts to be LESS OR EQUAL TO 10 WORDS and format your answer as follows. Highlight the difference between doc2 and docl in your doc2 summary:
<docl unmary? sumnary of doel //docl_sanmaryy summary? <concept_c> abstract concept frem doel </concept_e>
summary of doc2 built on doal given the concept </doc2_aurmary"
Example I:
<doo1 <doc1_aummn aummaryy recommenaation of local caffee shopa in San Diego </deol </docl_aumma aunmarys sooncept_ee coffee San Diego s/concept_c>
comparison of coffee culture in SD and NCC c/doc2_sunmary
Example 2:
edoci_sunmery Patient with sxollen eye discusses pain causes & symptoms and seeks for advice </doci_sunmary>
<concept_co c> mediccal sympton of swoller eye </concept_
cures swollen eye sunmary> wixi-style acticle introducing
Now, give your answer for the following documents:
<docl>
</doc1>
<doc2>
</doc2>
B.2 FACTUALITY ANALYSIS
Table 8: Estimation of the ratio of non-factual documents. We can see that the occurrence factuality error decays as the SP scales up.
Factuality undefined
No factual error
Factual error
Real data
31 31.44%
66.74%
1.81%
Synthetic data (200B-scale)
34.43%
50.47%
15.09%
Synthetic data (1T-scale)
31.91%
59.43%
8.65%
All LM generated synthetic data face the limitation of potentially generating non-factual content due to their probabilistic modeling nature. Moreover, because the internet inherently contains factual inaccuracies, LMs are likely to absorb these errors unless the data is carefully cleaned. During post-training, factuality must also be recalibrated alongside other objectives such data safety.
SBP relies solely on document-level correlations and does not incorporate human intervention to filter non-factual content. As a result, the generated outputs are also expected contain factual errors. Interestingly, we observe that the frequency of such errors correlates with the amount of data used the SBP pipeline. We define a document having undefined factuality if it is primarily subjective or -drive,, or if it concerns personal, obscure, or unverifiable entities. In all other cases, the document's factuality is considered well-defined and verifiable.
In Table 8, we analyze both the real data and the synthesized data used in the main experiment presented in Section 4.1. Specifically, we consider two types of synthetic datasets: a smaller-scale set initialized with 10B seed tokens, and a larger-scale set initialized with 50B seed tokens. From each source: real data, smaller-scale synthetic data, and larger-scale synthetic data, we randomly sample 10k documents. Each document is then categorized into three bins: factuality undefined, no factual error, and factual error, using judge. Our analysis shows that synthetic data contains more factual errors than real data. However, the amount of seed data increases, the factuality of synthetic data improves significantly. approaching that of real data. This finding is consistent with our mideval results in Table 2, where greater seed data availability enables the LM
29

---

Published as a conference paper at ICLR 2026
Table 9: Factuality undefined synthetic text.
Synthetic Text
Sunday, December 28, 2008 Tante Leni
Tante Leni is not only my Aunt Leni; she is my Eternity. When my Aunt Leni died few years ago, she was deeply saddened and was devastated. She was not the first family member to die, but she was the first felt so strongly about. Tante Leni was all my parents really had to show for the 25 years they worked as public school teachers and she was the one who had been with them the longest. There was a special place in her heart for my parents. In addition to all that, she the kind of person who always had a smile and a funny story to share. She was kind and funny and generous. The story that always comes to mind when think about her is the time she bank and someone dropped something from the top floor. was a very large parcel and the workers on the ground floor didn't have the tools open it. She jumped down see what was in it. A very large package of champagne appeared and she began gulping it down. Tante Leni and my mother Leni and my parents in a family portrait she took for my mom at 22. Tante Leni and my dad home when he was working as dance instructor. When my mom died, she had all the people who had known her since she was child living in the house. Tante Leni was the oldest, but she was also the best at cleaning, cooking and taking care of the house. When my mom passed away, she went to rehab center and Tante Leni stayed in the house.
working the
portrait they took when my mom was 20. Tante
to capture more factual knowledge and the synthesizer tuning to generate more relevant documents, thereby reducing hallucinations and producing more realistic outputs.
We extend our analysis of factuality errors in synthesized data in Table 10, highlighting the inaccu- racies present in the synthetic texts. These include false transfer and timeline claims in football, as well as incorrect institutional, company location, and certification details in the ecolabel example. This underscores the importance rigorous fact-checking, particularly in areas such as historical events (e.g., sports) and certification standards (e.g., eco-labels).
Factuality detection instructions
You helpful AI assistant. Your task is to evaluate whether the given document has well- defined factuality.
Definitions:
Not well-defined factuality: The document is primarily subjective or opinion-based (e.g., express
disapproval of politician in social media). The document discusses personal, unknown, unverifiable entities (c.g., a private diary).
Well-defined factuality: The document refers to well-known, identifiable entities (e.g., famous people, historical events, popular movies). Its factual claims can be checked or verified.
Output format:
If the document's factuality is not well-defined, output:
<not well defined></not well defined>
If the document's factuality is well-defined and factual, output:
<well detined>True</vel defined>
If the document's factuality is well-defined but non-factual, output:
<well defLned>Falsc</ver dedined>
Now, analyze the following document and provide your answer:
document
30

---

Published as a conference paper at ICLR 2026
B.3 MIDEVAL PROMPTS
Before each large-scale synthesis (on the order of billions of tokens), begin by synthesizing a small subset of data to evaluate its overall quality, a step we refer to as "mideval", The goal is to maximize the Pair-relevance of the generated data while monitoring Pair-novelty and Non-repetition rates. Although near-duplicates may not directly degrade quality, they reduce the data's overall utility, so we aim minimize their occurrence. While self-repetition can be removed via rule-based filtering, we still track it as an indicator of the synthesizer's 's quality. The quality of both the paired training data for synthesizer-tunins and the synthesized document influences the performance of the final model.
We have cited mideval results many sections throughout the paper. this section, we present the prompt that was used for Pair-novelty, Pair-relevance, and Non-repetition.
Pair-relevance detection
You are helpful Al assistant helping the user to determine if two provided texts are relevant to each other.
The user will provide you two texts in the following format: ## Text [textl)
## Text [text:2)
Your job to determine if two texts are relevant enough to considered a pair. Relevance means that the two texts are about the same topic, event, entity, person, place, or thing. If two texts talks about completely unrelated topics, they not relevant.
Please explain your reasoning your response. and conclude the response line with either "Yes" or "No". not with any other text including punctuation.
Pair-novelty detection
You are helpful Al assistant helping the user to determine if two provided texts are near dupli- cates.
The user will provide two texts in the following format:
## Text [text]]
Text [text2)
Your job determine the two texts are near duplicates, which means they are almost identical, except for some extra white spaces, line breaks, or punctuation. Two texts are not near duplicates they talk about the same topic but use different language, words, or style.
Please explain your reasoning in your response, and conclude the response in line with either "Yes" or "No". not end with any other text including punctuation.
31

---

Published as a conference paper at ICLR 2026
Non-repetition detection
You are helpful AI assistant helping the user to determine the provided text has repetition issues.
The user will provide you text in the following format: Text [text)
Your job is to determine if the text has repetition issues, which means some particular sentence or pattern are repeated more than three times. Some examples of problematic text:
* Example 1 of problematic text
have list of users
SharePoint 2010
want to send en amzil to all of them.
have tried the following code:
Mai (1 email.From new MsilAddress ("*); emall. Subject "Test"; email.Hodg "Test"; var smop new SntpClient sntp. "*; smtp.Port 25;
email
MallAditeess"");
smtp.Credentiall new ketworkkredentiall ("", *");
smtp. Smtp.Send(emmlll))
get the follewing error: server could not send the message.
The server could not send the mesaage.
The server could not send the message.
#
problemalic
my Profile
Product Reviews Send Message You responding the following review:
Submitted: 02-11-2006 by mikeschmid
been paddling for 10 years and have owned kayaks.
I aave been paddling in the ocean for a years and have owned 3 kayaks, been paddling 2 kayaks.
ocean
Pscific Northwest
years
have been paddling the ocean in the Caribbean for years and have owned 1 kayak.
have been paddling the
in the Mediterranesn for year and have exned 1 kaysk. have been paddling in the ocean in the South Pacific for 1 month and have owned kayak. owned
paddling
in the South Atlantio for month and
1 kayak,
ocean in Indian Ocean have Seen paddling in the ocean in the Arctic ocean for 1 month and have owned 1 kaysk. have been paddling the ocean the Ancarctic Ocean for month and have kaysk
have been paddling
nonth and have owned kayak.
Please explain your reasoning in your response, and conclude the response in new line with either "Yes" (which means the text has repetition issues) or "No" (which means the text does not have repetition issues). Do not end with any other text including punctuation.
32

---

Published as a conference paper at ICLR 2026
Table 10: Factuality errors detected in synthetic text.
Synthetic Text
So just how much has Chelsea been prepared to pay for the 34-year-old midfielder? Realistically, the clubs involved should be keeping in the region of E25 million ($38.8 million) and around f30 million (S45.5 million) being bandied about for the player in Italy. With the Blues expected to complete the sale of Cesc Fabregas to Arsenal this week, Lampard appears the logical replacement in midfield, but his bid to extend his contract has hit a roadblock with Chelsea's owners Roman Abramovich and the club being unable to agree to an increase in salary.
Lampard, who played in the Champions League final in Lisbon for Chelsea in 2007, has been linked with a move away from Stamford Bridge this summer, after having his contract with the club indefinitely extended 2010. There were rumours that he was to leave on a free transfer in the summer of 2011, but it appears that those again being placed to the side.
Miguel Delaney, from Sky Sports, believes that Lampard could be the highest paid player in the Premier League, and puts the bid that Chelsea are offering him in the region of 96 million (S8.9 million) per year. If Lampard is to join a team Italy, seems that this sum would be realistic, but the Premier League is a different beast, and it would be unrealistic for Chelsea to splash out that amount. As the speculation around his future continues, you can watch the pair discussing the topic in the video below.
Factual Errors
Cesc Fabregas to Arsenal - Fabregas moved from Arsenal to Barcelona in 2011, then to Chelsea in 2014, 4, not back to Arsenal.
2007 - The 2014 Champions League final in Lisbon did not involve Chelsea or Lampard;
Chelsea won in 2012.
indefinitely extended in 2010 LLmpard did not get an indefinite contract extension in 2010 with Chelsea.
Synthetic Text
Swanee Glue Brand: First Glue to be Awarded the Swan Ecolabel Published:27 July 2022
The global glue stick market is expected to reach USD 3.45 billion by 2028. Adhesives are the first choice of manufacturers in all industries such as food, pharmaceuticals, automotive, acrospace. construction, and packaging. As consumers are increasingly conscious of their carbon footprint and environmental issues, glue manufacturers are aiming to produce products that comply with environmental standards and are effective and cost-effective in their applications. This is why the Swan Ecolabel was established by the Swedish Environment Agency as certification for sustainable adhesive products. Swanee Glue one of the world's leading glue brands in glue sticks, and this year its brand received the Swan Ecolabel. UHU is adhesive brand owned by Bolton Adhesives in the Netherlands, and part of the Italian Bolton Group with a strong agenda for sustainability.
Glue sticks, specifically glue sticks with a wider applicator and swan neck applicators, have the most impact the environment because they are consumable item and their impact is greatest when thrown away. Therefore, the Swance Swan Ecolabel ensures that UHU is part of the solution to the growing demand for sustainable adhesive products. In order to obtain the Swan Ecolabel, the adhesive must have at least 50% renewable content. Besides this, the glue stick should also contain a higher percentage of recyclable content. UHU meets all these criteria and has permanent and multi-use applicator. For further information, you can contact UHU receives the Swan Ecolabel
Factual Errors
Swan Ecolabel was established by the Swedish Environment Agency The Nordic Swan Ecolabel was established by the Nordic Council of Ministers, not only Sweden.
Netherlands - UHU is based in Germany, not the Netherlands. 50% renewable content- The Swan Ecolabel requires least 20% renewable content in adhesives, not 50%.
33

---

Published as a conference paper at ICLR 2026
B.4 SYNTHESIZED DOCUMENTS FROM THE IT-SCALE EXPERIMENT
In this section, we present additional examples of synthesized documents at the IT-scale to comple- ment the example given at the 200B-scale in Section 4.2.
Real document
Synthesis
Synthesis II
The Karale Kid
The Kanil Kid (2010)
The Kunsle Kid
From Wikipedia, the free encyclopedia
The Kid
movie poster
drink- ing ice cream and talking about not tilking to them, Duniel turns his mom and says "We'r re family. We're a family good away,
When Daniel and
(Redirected from Karute Kid)
115 minutes Roted PG-13 Directed by Harakl Zwartt + Starrin: Will Smith, Jackie Chan, RZA. Jaden Smith More info IMDB
Jump fot navigation. search
Far other uses, see Karate Kid (diambiiguzzoe).
The Karate Kid Karate kid.pp
Directed by John Avildsen
LaRusso (Ralph Maschiu) and Johnny Lawrence (Julen Smith) appear The Kornte Kid in their middle ages, because what's 's moot insercsting about this remake is the way it updites old, famillir storyline new generations moviegoers.
irunic
front his underwear (that was on fire). it said "Daniel don't 1 quii on family"" The script for this movie oviginally ssid that Daniel's mom was in the film was cut because she would not look like Elisaheth Shuc and they thought the characters could have a less motherly character. Elisabeth Shue actually called the di- noctor John Avildsen, film. John Avildsen replied that he didn't it know whether or not he wamied to add less mocherly chorecter the film. Shue replied "If you don't. the film was also impressed that she did this and he was going to hire her in any case.
ple" Daniel
Produced by Jerry Weintraub
Written by Rubert Mark Kamen
Music by Bill Conti
film makers thouught
Cinettatngrapl Jumes Crabe
Distrbured by Columbia Pictures
Relesse dates + June 22, 1984 (1984-06-22)
Pechaps I shouldn't say "updates" - bocause that
Running 127 musutes!
implies that the new version is significant depar- 1984 classic. you're
being the
Country United
Language English
Karate Kid sacrosanct, don't 'I expeci thing different
folks thinks the 1984 version
Budget 58 million]21
Box office $90.8 million[3]
here. Jest watch both movies and compare the staries characters.
not playing the movie."
a 1984 America martial arts drmma film prodcced by Jerry Weintraub, directed by John G. Avildsen, wimen by Robert Mark Ka- Ralph Macchio, Noriyuki "Pat" Moeita and Blissheth [4][] It is an undes- dog story in the mold of previous success. Avild- son's a 1976 film Rocky.....
The Karate
The stary is bosically same before - young Daniel Lakasso moves to Los Angeles, and soon finds himself beset by Johnny Lnwrence and gang students from the Cabon Kai schuol, which bullies Damiel whenever gets the <hinee....
Revealing mistake: When Daniel is throwing the bucket Johnny's head while on bus, supposed du it several times, yet when be hits Johany the first time, you can see Johnny fall on the ground
that
Real document
Synthesis
Synthesis
Skip main content
Beach Vibe
Hene's a playlist to worm your hesst and ger you ready to leave for the beach in few weeks. So many classics from your youth, this playlist packos ensotionaal punch.
Are you craving some beach lime? you're
Any naaelller will tell you that if you get on a plane and octuailly arrive somewftere where the weather is the you' made mistake. I've experienced
escaping down south for speing break, we have
some tunes for you. But even if you uren't licky enough sprawil in the sand back in golden
weeks
you
It's kind of lame playlist. sure better one exists soneewfere on the intemet. But this most northern of days. like get warm chill go- positive
we'll bring beach you! Until your nest break, enjoy listening to The Beach Roys and lack Johnson.
sunshine right
this time and again- it's one the many things
that you just can't fully appreciste ce appreciate fullest until you feel your
before going with my dags
If you're wo stuck in the cold-or womewhere where
My latest trip was Viernam
week. had
ousside. This wark
the sun indocisive (I'm looking you, Mid- west) use this playlist take you back to sun- nier days and remind you that the winter is almost ower, soon the sun will lest!
been plansing this tip for little over a year now and I really hoped it would be everything had imagined.
and they are the best caddlerrs in the world, so that makes it perfect for me. This playlisr will make your winter that
hearable and
exactly what wanted. The
people, the calture. the food, semuery - was all perfectly dieveed inte. I'd never been to an Asian comntry before and so the whole trip was just like discovering
then hopefullly you head to beach!
You know you want it!
Here's the playlist:
contitent.
(credit poovider" "You Eube" url="|
It one the hest places I've ever been...
free mobile app
Figure 9: Comparison of original text with synthesized text variations. On the first row, the real document provides factual information about the 1984 film's production and release. In contrast, the synthesized documents offer subjective commentary, opinions, and behind-the-scene anecdotes about both the 1984 film and its 2010 remake. On the second row, the synthesized documents are continuations of the real document.
C ADDITIONAL PRETRAINING RESULTS
C.1 TWO EPOCHS VALIDATION
When designing the oracle experiment for IT-seale, we noted that we use 482B tokens repeated twice proxy for training IT unique tokens. This is because the contains 80% duplicates, which hinders our evaluation. We validate our choice by scaling down the experiment to a 400B scale, where we had sufficiently many unique tokens. As seen in Table 11, 200B tokens repeated twice yield nearly identical performance to 400B unique tokens. This finding is consistent with the observation from [39] where repetition up to 4 times yields nearly no performance degradation.
[34] dataset
C.2 MODEL SCALING
An alternative approach to leveraging additional compute is to use a larger model. In this section, we examine the benefits of fixing a training token budget, but using a 6B-parameter model (Table 12).
34

---

Published as a conference paper at ICLR 2026
Table 11: Performance comparsion with 200B tokens repeated twice vs. 400B unique tokens for the 3B model. We can see that the two models yield similar performance.
Benchmark
2x200B
1x400B
Perplexity on held-out data |
Open Tex2
4.55
4.54
LAMBADA
4.49
4.46
Five-shot MMLU
3.19
3.17
QA accuracy
f
ARC-Challenge (l-shot)
38.31
41.47
ARC-Easy (0-sho
73.11
75.29
SciQ (U-shot)
93.80
93.30
Winogrande (0-ahor)
64.96
63.93
TriviaQA (l-shot)
32.51
34.35
WebQS (1-shot)
18.75
13.58
Average QA accuracy
53.57
53.65
Table 12: 6B-66B-pppm 38-parameter model setup.
We conduct a pretraining experiment in a 200B-scale setting, replacing 3B 3B-parameter model with 6B- parameter model. In Table 13, we observe that the 6B-parameter model consistently outperforms the base- line method, indicating that effectively utilizes the ad- ditional computational resources available. Comparing SBP with the 6B-parameter model, we see that one per- forms better on some benchmarks while the other per- forms better on others. This suggests the benefits offered by SBP are orthogonal to the benefits provided by hav- ing a larger model, offering the potential to combine both approaches to obtain an even better model.
Total Params.
3B
6B
Loontext
4096
4096
Tvoccb
49152
49152
Thayers
26
32
dmodel
3072
4096
d
8064
13056
Theads
24
32
Fkyy heads
8
8
Table 13: 200B-scale experiments with model scaling. The first three columns are identical to Table1. The last column shows the performance of training 6B model under 200B training token budget with 10B unique tokens.
Benchmark
Baseline
SBP
Oracle
6B-model
Perplexity on held-out data J
Open WebText2
5.74
-0.53
-1.02
-0.36
LAMBADA
6.87
-0.85
-1.86
-1.10
Five-shot MMLU
3.83
-0.36
-0.51
-0.13
QA accuracy f
ARC-Challenge (D-shon)
35.32
+1.28
+2.82
+3,42
ARC-Easy (0-bx))
68.94
+2,65
+4.29
+0.67
SciQ (D-abor)
90.50
+1.00
+2.40
+0.80
Winogrande (0-sho))
60.14
+1.90
+5.53
+2,92
TriviaQA (1-shot)
22.51
+3.36
+7.37
+3.11
WebQS (1-ab0)
8.56
+3.74
+10.83
+5.22
Average QA accuracy
47.66
+2.32
+5.54
+2.69
35


Reviews:
[{"review_id": "8GiF2bRJOE", "reviewer": "Reviewer_Vkk4", "summary": "This paper proposes a method to generate synthetic data for pre-training in constrained data settings. The method relies on a document generation model that learns to generate new diverse documents from given ones. Concretely, an LM is pre-trained on a dataset D_pretrain (standard next token prediction) and then fine-tuned on the document generation task. To fine-tune the document generator, pairs of input-output documents are created by pairing related documents in D_pretrain. The document generator is then used to generate a synthetic corpus of documents D_synthetic. An LM is then pre-trained on D_pretrain+D_synthetic. The authors also discuss a Bayesian interpretation of the standard data pre-training objective versus the document generator (synthesizer)fine-tuning one; the former marginalizes out latent concepts from which documents are generated while the second models the posterior over these concepts.", "questions": "- Line 135 DATA-CONTRAINED > CONSTRAINED\n\n- For the Quantitative analysis in Section 4.2. It would make sense to add metrics based on novel n-grams, e.g.,  percentage of new n-grams appearing in the generated documents or those used in [2], to better reflect the deviation of the generated documents from the seed ones.\n\n[2] https://aclanthology.org/N18-1065/\n\n- Statistics about the length of synthetically generated documents?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- An intelligent idea to augment with synthetic data that carries new signals (compared to the repetition baseline).", "weaknesses": "- There are known issues related to biases set up in pre-training (e.g., [1]). The authors should probably report analysis with respect to this. In particular because of the document generator.\n\n[1] https://openreview.net/pdf?id=KQhUEoPmJy\n\n- From the experiments it is not clear how the improvements brought by the data synthesizer would generalize to larger models. From Table 2 it seems to indicate that with scale, the generated documents might be closer to the input documents. Still the idea is interesting and could be useful for the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8EELzff3yF", "reviewer": "Reviewer_s1yS", "summary": "This paper introduces Synthetic Bootstrapped Pretraining (SBP), a novel framework aimed at improving data efficiency by explicitly modeling inter-document correlations within the pretraining dataset. The authors conduct rigorous compute-matched large-scale experiments (3B model, up to 1T tokens) demonstrating that SBP significantly outperforms the traditional repetition baseline and effectively reduces the performance gap to the \"Oracle\" upper bound (a model with access to more unique data). The concept is novel, the experiments are solid, the results are convincing, and the inclusion of a theoretical explanation based on a Bayesian hierarchical concept model makes this a valuable and highly relevant research direction for addressing the imminent \"data scaling wall.\"", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. SBP creatively addresses the issue of exhausting high-quality text data. By learning a conditional generator $P(d_2|d_1)$ from document pairs to synthesize new data, it serves as a powerful complement to traditional token-level autoregressive training, leveraging latent structural signals in the data.\n\n2. The experiments utilize a 3B-parameter model, making the scale relevant for frontier LM development. Crucially, the compute-matched setup ensures a fair comparison across SBP, the repetition baseline, and the Oracle.\n\n3. The results (Table 1 in the original paper) show SBP bridges 42% to 48% of the QA accuracy gap to the Oracle, confirming that the synthesized data $S_{pretrain}$ provides a real and effective signal.", "weaknesses": "1. The quantitative analysis (Tables 2 and 7 in the original paper) shows an alarmingly high non-factual error rate of 15.1% for the 200B-scale synthetic data (compared to 1.81% for real data). Although this rate decreases to 8.7% at the 1T-scale, it remains a significant concern. By the way, the caption in Table 2, `are not are' should be `are not'?\n\n2. The paper acknowledges in Section 6 that the initial nearest neighbor search relies on the external Qwen3-0.6B Embedding model. Why choose such a model, while efficiency is a valid reason, this limitation detracts from the \"Bootstrapped Pretraining\" claim. \n\n3. The $d_1 \\to c \\to d_2$ relationship captured by SBP is fundamentally a statistical dependency (or association) based on a shared latent concept $c$. While this process involves an \"implicit posterior inference,\" framing this learned relationship as \"Causality\" in a statistical context is inaccurate and can be misleading, as it does not imply a strong intervention relationship.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rfkyAxhPGe", "reviewer": "Reviewer_9CYh", "summary": "This paper introduces Synthetic Bootstrapped Pretraining (SBP), a new large-scale pretraining procedure that aims to capture inter-document correlations that standard LM pretraining overlooks. SBP identifies semantically similar document pairs, trains a conditional “synthesizer” model to generate one document from another, and then uses it to synthesize new text for joint training. The method is evaluated using compute-matched experiments on a 3B model trained up to 1T tokens. Results show consistent improvement over repetition-based baselines, recovering about 40–50% of the performance gains achievable by an oracle model with access to 20× more unique data. The authors also present a Bayesian interpretation of SBP and provide qualitative and quantitative analyses of the generated data.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Introduces a new dimension of pretraining that models inter-document relations rather than intra-document token dependencies.\n2. Clear structure and logical flow from motivation.", "weaknesses": "1. Can a 1.5B model trained on a 200B dataset generate data that is both relevant to the source documents and diverse? \n2. What is the computational resource consumption across the three stages?  \n3. The related evaluation metrics are highly dependent on the LLM used for judgment. What is the judged LLM?\n4. The \"pair copying\" metric should not be compared solely with the source documents but rather with the entire training dataset. This is because the model might output documents that are similar to others in the dataset, which would contribute little to overall improvement. \n5. Comparing Figure 6 and Figure 7 reveals that as the dataset size increases, the proportion of synthetic data required decreases, which to some extent also illustrates this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXoX8czvFw", "reviewer": "Reviewer_6oUP", "summary": "The paper proposes Synthetic Bootstrapped Pretraining (SBP), a framework that improves language model pretraining by generating synthetic documents from existing data.\n\nSBP first finds semantically related document pairs $(d_1, d_2)$, trains a conditional model to predict $d_2$ from $d_1$, and then uses this model to synthesize a large corpus for joint training.\n\nExperiments with a 3B-parameter model under compute-matched setups (200B and 1T tokens) show consistent improvements over repetition baselines and recover up to ~50% of the oracle gain from 20× more unique data.\nQuantitative analysis confirms that SBP-generated data maintains diversity and factuality comparable to real text.\n\nThe paper also offers a Bayesian view, interpreting SBP as implicitly learning latent “concept” structures across documents, providing a principled explanation for its data-efficiency benefits.", "questions": "1. The paper argues that SBP captures inter-document structure ignored by standard pretraining. \n\n    - Could the authors provide a more direct analysis or evidence of such cross-document dependencies (e.g., similarity shifts in embedding space, conditional generation tests, or document-pair prediction tasks)?\n\n2. The 200B-scale experiment uses a 37.5% synthetic and 62.5% real data ratio.\n\n    - How sensitive are the results to this ratio?\n\n    - Would a smaller amount of synthetic data (e.g., 10–20%) still bring noticeable gains?\n\n3. The paper presents SBP as compute-matched to repetition baselines, but it remains unclear whether the synthesizer training and data generation costs are included in the total compute budget.\n\n    - Could the authors clarify how these costs were accounted for, and whether SBP remains more efficient when end-to-end compute (including synthesis) is considered?\n\n\n4. The synthesizer is initialized from a pretrained LM, which may introduce prior knowledge not attributable to SBP itself.\n\n    - Could the authors clarify how much this initialization contributes to the overall improvement?\n\n    - Have they tried training the synthesizer from scratch or using weaker initialization to isolate SBP’s effect?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. SBP introduces a principled way to exploit inter-document relations for self-synthetic pretraining without external teachers.\n\n2. Compute-matched experiments at 200B/1T scales show consistent gains and high-quality synthetic data, recovering about half of the oracle improvement.", "weaknesses": "1. While SBP claims to capture inter-document correlations, the experiments mainly report aggregate metrics (loss, accuracy) without direct analysis of cross-document dependencies or representation changes. The authors should add the following experiments:\n\n    - Evaluate whether the SBP-trained model can generate $\\hat{d}_2$ from $d_1$ that semantically aligns with the true $d_2$ , proving it learns document-to-document conditional structure.\n\n    - Compare embedding distances of paired vs. random documents, see if SBP pulls semantically related but stylistically different pairs closer than the baseline, which proves structural alignment in representation space.\n\n2. The repetition baseline may not reflect realistic industry data-mixing or quality-weighted sampling strategies, possibly inflating SBP’s relative gains.\n\n3. Add ablation: use random document pairs to train the synthsizer.\n\n4. The paper does not clearly state whether the synthesizer-tuning and data generation costs are included in the compute-matched budget, leaving uncertainty about true efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)