Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
Authors: Xinran Gu
Kaifeng Lyu
Jiazheng Li
Jingzhao Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: knowledge acquisition
memorization
scaling laws
LLMs
Primary area: deep_learning
TL;DR: We show that knowledge acuiqistion under data mixing can exhibit phase transitions with respect to the mixing ratio and model size.

Abstract: Large Language Models (LLMs) are typically trained on data mixtures: most data come from web scrapes, while a small portion is curated from high-quality sources with dense domain-specific knowledge.
In this paper, we show that when training LLMs on such data mixtures, knowledge acquisition from knowledge-dense datasets—unlike training exclusively on knowledge-dense data—does not always follow a smooth scaling law but can exhibit phase transitions with respect to the mixing ratio and model size. Through controlled experiments on a synthetic biography dataset mixed with web-scraped data, we demonstrate that: (1) as we increase the model size to a critical value, the model suddenly transitions from memorizing very few to most of the biographies; (2) below a critical mixing ratio, the model memorizes almost nothing even with extensive training, but beyond this threshold, it rapidly memorizes more biographies. We attribute these phase transitions to a capacity allocation phenomenon: a model with bounded capacity must act like a knapsack problem solver to minimize the overall test loss, and the optimal allocation across datasets can change discontinuously as the model size or mixing ratio varies. We formalize this intuition in an information-theoretic framework and reveal that these phase transitions are predictable, with the critical mixing ratio following a power-law relationship with the model size. Our findings highlight a concrete case where a good mixing recipe for large models may not be optimal for small models, and vice versa.

Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
Xinran Gu
Kaifeng Lyu'* Jiazheng Li
Jingzhao Zhang!-
Institute for Interdisciplinary Information Sciences, Tsinghua University Coolege of AI, Tsinghua University 'Shanghai Qizhi Institute guxr240ma ils wes.s..sssghuaa edu. en, klyuetsinghi edu en foreverlastt look. com, jingzhaozétsinggne edu.cn
Abstract
Large Language Models (LLMs) are typically trained on data mixtures: most data come from web scrapes, while a small portion is curated from high-quality sources with dense speciffc knowledge. In this paper, we show that when train- ing LLMs on such data mixtures, knowledge acquisition from knowledge-dent training exclusively on knowledge-dense data [Allen-Zht and Li, 2024a] does not always follow a smooth scaling law but can exhibit phase transitions with respect to the mixing ratio and model size. Through controlled experiments synthetic biography dataset mixed with web-scraped data, we demonstrate that: (1) as we increase the model size to a critical value, the model sud- denly transitions from memorizing very few to most of the biographies; (2) below a critical mixing ratio, the model memorizes almost nothing even with extensive training, but beyond this threshold, it rapidly memorizes more biographies. We attribute these phase transitions to a capacity allocation phenomenon: a model with bounded capacity must act like knapsack problem solver to minimize the overall test loss, and the optimal allocation across datasets can change discontinuously as the model size mixing ratio varies. We formalize this intuition in information- theoretic framework and reveal that these phase transitions are predictable, with the critical mixing ratio following power-law relationship with the model size. Our findings highlight concrete case where good mixing recipe for large models may be optimal for small models, and vice versa.
Introduction
The pre-training data of large language models (LLMs) can be categorized into two major types. The first type consists of large-scale corpora scraped from the web [Raffel et al., 2020, Penedo et al., 2024, Li al., 2024], often spanning billions trillions of tokens across diverse topics and styles. Due to the scale, it is inherently hard to ensure the information density of the dataset and its relevance to downstream tasks. Hence, second type of data, smaller-scale datasets curated from high-quality sources, is incorporated. This type of data usually contains very dense knowledge on tasks or domains with significant practical value. For example, Wikipedia and Stack Exchange cover a wide range of world knowledge. Open Math [Paster et al., 2024] and StarCoder [Li et al., 2023, Kocetkov et al., 2022] provide valuable data for improving model performance on mathematics and coding tasks.
The second type of data, which we refer to as knowledge-dense data, typically accounts for only small fraction of the entire corpus. In the pre-training data of recently released model family,
"Equal contribution
I Work done while the Simons Institute for the Theory of Computing, UC Berkeley.
I Conrespondiig author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

100
Acc. on SynBio SynBlo-320K 320k
60
Acc. on SynBi-1 2..2888 (30
60-
Mixing ratio
on Syntho- (rr8ho-32004 320k
1%. oB scall
0.1
0.2
0.3
40
0.4
20
0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
14
31 50
90 160 256 410
Mbding rator
Mixing ratio
Model size (M, log scale)
(a) 70M models.
(b) 410M models.
Figure 1: Phase transition in model size. For each mixing ratio, model size in- creases, accuracy initially remains zero, Once model size surpasses some thresh- old, accuracy rapidly grows to over 60%.
Phase transition in mixing ratio. For each model size, as mixing ratio increases, accuracy initially remains zero. Only when r exceeds some threshold does accuracy quickly improve.
Figure 2:
OLMo 2 [OLMo et al., 2025], over 95% of the tokens are from web data, and only less than 5% are from knowledge-dense data. The proportion of each individual knowledge-dent dataset is even smaller, e.g. only less than 0.1 of the tokens are from Wikipedia. This naturally raises question: How much knowledge can L.MSs really acquire from this small amount of knowledge- dense data?
If LLMs were exclusively trained on knowledge-dense data without any data mixing, the amount of knowledge acquired after sufficient training should scale linearly with model size. Although quantifying knowledge in natural data is non-trivial, Allen-Zhu and Li [2024a] sidestep this issue and provide strong empirical evidence for this linear scaling law through extensive pre-training experiments synthetically generated biographies. In their setting, the amount of knowledge stored by a model is quantified by evaluating its memorization of the biographies using coormatioon-hhoreete metrics. Similar linear scaling laws also observed in memorizing Wikidata fact triples by Lu et al. [2024], and analyzed theoretically by Nichani et al. [2025]. Based on these results, one might naively expect similar linear relationship between model size and acquired knowledge when knowledge-dense data is mixed with web data.
However, in this paper, we show that the linear scaling no longer holds under data mixing. We consider the setup where a knowledge-dense dataset focused on a single domain constitutes a small fraction r of the pre-training corpus-redt referred to as the mixing ratio-and and the rest is large-scale web text (see Appendix E. for our implementation of data mixing). We demonstrate via quantitative study that knowledge acquisition from the knowledge-dens, data exhibits a more intricate behavior with notable phase transitions with respect to the mixing ratio and model size.
More specifically, we study factual knowledge acquisition. We follow the approach of Allen-Ziin and Li [2024a] to curate synthetic dataset of biographies, where each individual's information is embedded into natural text descriptions using diverse templates. Due to the uniform data format and content of this dataset, we can quantify how much knowledge the model has stored simply by counting the number of memorized biographies. We then mix this synthetic biography dataset with large-scale web corpus FineWWe-Edu [Penedo al., 2024] or the Pile [Gao al., 2020] create the pre-training mixture. We pre-train or continually pre-train Pythia models [Biderman et al., 2023] ranging from 14M 6.9B parameters on these mixtures.
While setting r closer to will make the model learn more from the knowledge-dense data, in practice, is typically set to a small value either because the knowledge-dense data has limited amount or increasing r may hurt the model's capabilities acquired from other domains. Therefore, the essence of our study is to understand whether models can still memorize a decent number of biographies for relatively small r. Our experiments reveal two interesting findings (Section 3):
Finding 1: Phase Transition in Model Size (Figure 1). Fixing the mixing ratio r and varying the model size M, we observe that when M is smaller than critical model size Mthress the number of memorized biographies can be nearly zero Only when M > Mthres the model suddenly memorizes most biographies. Moreover, the threshold Mehres is higher for smaller
Phase Transition in Mixing Ratio (Figures and 9). When varying the mixing ratio r while keeping the model size M fixed, we find that below a critical mixing ratio Fthress the model memorizes almost nothing even after significantly longer training, during which each biography appears hundreds of times or more (Figures 3(a) and 4). But when > Tthres, the number of memorized biographies grows rapidly with r. We further find that as we gradually decrease r, the number of steps needed to memorize fixed number of biographies initially grows linearly with
Finding 2:

---

1/r (Figure 3(b),, but soon becomes exponential and even superexponential (Figure 3(c)), making it impossible or practically infeasible for the model to memorize a non-trivial number of biographies.
In Figure 10, we further show that the observed phase transitions are not limited to discrete metrics like accuracy, but also persist in validation loss, a continuous metric.
Theoretical Analysis. In Section 4, we attribute the observed phase transitions to a capacity allocation phenomenon: a model with bounded capacity must act like knapsack problem solver to minimize the overall test loss, and the optimal allocation across datasets can change discontinuously as the model size or mixing ratio varies. To formalize this intuition, LLM as the best model that minimizes the test loss under a fixed capacity constraint M. We develop an information-theowe
model sufficiently trained
framework and show that, when trained on mixture of knowledge-don and web-scraped data, the model should allocate its capacity across the two datasets based on their respective "marginal values" that is, the reduction in test loss achieved by assigning one additional unit of capacity to that dataset. We rigorously prove that only when the mixing ratio r or the model size M is above a certain threshold does the knowledge-dense dataset become worth learning, thus leading to the observed phase transitions. Assuming that the optimal test loss on web-scraped data follows power law in model size, we further show that these phase transitions are in fact predictable, with the critical mixing ratio following a power-law relationship with the model size. Empirically, we validate this power-law relationship on both synthetic biographies and a set of real-world knowledge extracted from Wikipedia (Section 5).
Strategies to Enhance Knowledge Acquisition Under Low Mixing Ratios (Section 6). Inspired by our theory, we propose two strategies to enhance knowledge acquisition at low mixing ratios: (1) randomly subsampling the knowledge-dense dataset; (2) rephrasing knowledge into more compact forms and augmenting the original dataset with the rephrased versions. The key idea is to increase the "marginal value" of the knowledge-denss dataset by increasing the exposure frequency of each single fact. We validate on both synthetic and real-world Wikipedia biographies that these strategies help models memorize significantly more biographies while preserving models' general capability.
Takeaways. The key takeaways of our paper are as follows:
1. The mixing ratio should be set with care for different model sizes: mixing in knowledge-dense datasets with small mixing ratios can offer no benefit at all, especially when training small LMs.
2. Naively measuring the performance of small models on small data domain may provide little to no predictive signal on how well larger models perform, revealing potential limitation of using small proxy models for data curation, as also evidenced by Kang et al. [2024], Jiang et al. [2024], Ye et al. [2024], Magnusson et al. [2025]. Mizrahi et al. [2025].
3. Slightly improving the "marginal value" of knowledge-dense data can offer large gain in performance, as evidenced by our proposed strategies.
2 Experimental Setup
The SynBio Dataset. We follow Allen-Zhu and Li [2024b] to create a synthetic biography dataset, with each individual characterized by five attributes: birth date, birth city, university, major, and employer. For each individual, the value of each attribute is randomly and independently sampled from a predefined domain. These (name, attribute. value) triplets are then converted into natural text using sentence templates. For instance, (Gracie Tessa Howell, birth city, St. Louis, MO) is converted into "Gracie Tessa Howell's birthplace is St. Louis, MO." Following [Allen-Zhu and Li, 2024b), every time the model encounters a biography, the five sentences are randomly shuffled, and a new sentence template is selected for each attribute from set of five possible templates. We denote the dataset containing N biographies as SynBio-N. See Appendix E.2. for full details.
Evaluation. Denote knowledge triplet (name, attribute, value) as (n, v) and let |e| represent the number of tokens in v. For evaluation, the model is prompted with the sentence prefix containing n and a and is tasked to generate |v] tokens via greedy decoding. We then check whether the output exactly matches v. For example, given the triplet (Gracie Tessa Howell, birth city, St. Louis, MO), the prompt "Gracie Tessa Howell's birthplace is" is provided. We say the model has memorized the fact if it generates "St. Louis, MO." We report the accuracy averaged over all individuals, attributes, and templates in the main text and defer the detailed results to Appendix D.6.
3

---

Misho 16
Total training tokens (6)
X
not sttained she 2568
Thtal antattin 40X
1000
--02
0:45
055
-007
250-
SynBio 320k (%)
60--
200
600-
50-
100
400
50-
200
30
0
59
Total Training
100
150
202
(9)
250
500
0.8
0.5
0.4
1/r
0.25
0.2
05
04
3.V
025
(a) Train until acc. 60% or a total of
(b) Required training steps to (c) Fitting required training steps
256B tokens are passed.
achieve target accuracy v.s 1/r. to attain 40% accuracy against 1 1/r.
Figure 3: Training longer barely helps for low mixing ratios, with the required training steps to reach target accuracy grow exponentially or even superexponentialy with 1/r. We train 70M models on the mixture of FineWeb-Eae Edu and SynBio-3200 with r ranging from 0.2 to 0.8.
Training Setup. Our experiments the Pythia architecture [Biderman et al., 2023]; with model sizes ranging from 14M to 6.9B. The default setup involves pre-training from scratch on a mixture of Web--ue-Edu and SynBio. Since FineWeb-Edi is large (> IT tokens) and SynBio is small (<1B tokens), our typical training runs involve the model seeing SynBio for multiple epochs but Webe Edu for less than one epoch. For instance, in 32B-token run with the mixing ratio for SynBio-320k set as 0.1, the model passes SynBio ~ 100 times. We also study the continual pre-training setup in Section 6 and Appendix D.1. Full experimental details are provided in Appendix E.
3 Phase Transitions of Knowledge Acquisition within Data Mixtures
3.1 Phase Transition in Model Size
We first investigate how knowledge acquisition is affected by model size given the data mixture. For each E (0.1, 0.2, 0.3, 0.4). we train models with sizes from 14M to 410M on the mixture of Feb-WebFFdu and SynBio-320k for a sufficiently long horizon of 32B tokens, which is approximately four times the compute-optimal training tokens for 410M models according Hoffmann et al. [2022]. As shown in Figure 1, as the model size increases, accuracy on SynBio initially remains near zero. Once the model size surpasses some threshold, accuracy rapidly grows above 60%. The transition is consistently sharp across different mixing ratios while larger r leads to a smaller critical point.
3.2 Phase Transition in Mixing Ratio
We now study how knowledge acquisition under data mixing scenario is affected by mixing ratios.
Performance on knowledge-dense data undergoes a phase transition as mixing ratio increases. We begin by training models of the same size with different mixing ratios r. Specifically, we train 70M models on the mixture of Fine Web-Edu and SynBio-320K, varying r from 0.1 to 0,45 (stepsize 0.05), and 410M models on the mixture of FineWeb-Edu and SynBio-1.28M, varying from 0.1 to 0.4 (stepsize 0. All models are trained for a total of 32B tokens. As shown in Figure 2(a), for 70M models, as r increases from 0.1 to 0.25, its accuracy on SynBio remains near zero. Only when r > 0.3 does the accuracy begin to steadily improve. In Figure 2(b), the accuracy for 410M models exhibit similar trends where it remains near zero for r S 0.3 and suddenly attains 80% when grows to 0.4. In Figure 9, we replicate the experiments on Pythia 2.8B and 6.9B models to show that similar phase transition mixing ratio persists for larger models. In Table 1, we report the mean and standard deviation of accuracy for experiments in Figure 2(a).
Training longer barely helps for low mixing ratios. Given the observed phase transition, one may raise the following counter-argument: if models are trained for a sufficiently long horizon-suce that even a small mixing ratio r would eventually result in each biography being encountered hundreds or even thousands of times-then the phase transition might no longer exist. To test this counter- argument, we extend the training horizon for r 0.2 512B tokens for the 70M and 410M models by 16 and 4 times respectively. Under this extended training, each biography appears ~ 3000 times for the 70M model and 200 times for the 410M model. As shown in Figures 3(a) and 4, the accuracy on SynBio remains near zero even after such extensions.
Required training steps increase exponentially or even superexponentially with 1/r. To further refute this counter-argument, we quantify how the required training steps to reach a target accuracy, denoted as T, scales with 1/r. Specifically, we train 70M models with ranging from 0.2 to 0.8. For

---

Aoc. on knowladge (SG)
Makig nater
Acc. on Slope Calculation
100
Acc. on Slope Cal culauttion
60
(%% log scaie)
60
40-
Mixing ratio 0.1
20-
0.2
20
10
10-
training takens
14 31 50 507095 160256410
0.0
0.1
0.2
0.3
0.4
Model size (M, log scale)
Mixing ratio
Figure 4: For 410M mod- els trained FineWeb-Edu SynBio-1, 1,28M, 0.2 remains near zero even with 4x more training.
(a) Phase transition in model size.
(b) Phase transition in r.
for
Figure 5: Similar phase transitions the slope calculation subtask persist when we mix the modified OpenWehMait FineWeb-Fduu The model size for (b) is 70M.
each mixing r, we evaluate 20 training horizons, approximately evenly spaced on a logarithmic scale with factor of 1.2 ranging from 0 to 256B tokens. Training continues until the model reaches 60% accuracy or exhausts 256B tokens. As shown in Figures 3(a) and 3(b), when r decreases from 0.8, T initially increase linearly with 1 /r for
0.4 and quickly deviates from the linear trend for < 0.4.
We further fit a scaling law the required training steps reach 40% accuracy against l/r, modeling T' as a power-law or exponential function of 1/r. Specifically, we fit T against 1/r for 2 0.3 and examine whether the extrapolation can predict I' for smaller т. As shown in In Figure 3(c), the actual T' is more than 2.9 times the power-law prediction for r = 0.25, and more than 1.9 times for r 0.2. Moreover, the actual T for 0.25 even more than twice the exponential prediction. These significant deviations suggest exponential or even superexponential growth of 'T' with respect to 1/r. See Appendix E.5 for the detailed fitting process.
We also conduct ablation studies on hyperparameters in Appendix D.2.
3.3 Phase Transitions on Reasoning Tasks
In this subsection, we show that the phase transition phenomenon is not limited to factual knowledge, but also extends datasets related reasoning. Such datasets are often multi-task in practice. For example, Open WebMath [Paster et al., 2024] covers diverse math topics. We show that phase transitions can occur for each single subtask within this dataset. Inspired by Ruis et al. [2024]. we consider the slope calculation task between two points (I1, 31) and (æ22 32). To explicitly control the frequency and format of the slope calculation examples, we replace all the documents containing the word "slope" in OpenWeeMath with our clean and high-quality slope calculation demonstrations. We then mix the modified Open WebMath with Fine FineWeb-Edu and train Pythia models on this mixture from scratch. Similar to the setup of SynBio, every time the model sees slope calculation example, we uniformly sample 31, /1, 22. 1/2 from {0, 1.. .99} (ensuring #1 + x2), and apply randomly chosen question-answer templates. For evaluation, we randomly generate Ik questions for slope calculation and check the model produces the correct final answer. Results in Figure show similar phase transitions as factual knowledge acquisition (see details in Appendix E.3). Appendix D.3 presents further discussions and experiments on another reasoning task with larger input space.
4 Theoretical Analysis
In this section, we take an information-theerrtic view to explain the observed phase transitions. The key challenge in developing theory is that training LLMs can involve a lot of tricks, making it hard to identify the most important factors inducing the phase transitions. In our paper, we consider an ideal case where the model is sufficiently trained, allowing us to focus on the key factor model capacity-aae and abstract away all other complexities.
4.1 Hive-Level Intuition
We model a sufficiently trained language model with capacity M as an optimal bounded-capatee learner, which minimizes loss much as possible under the capacity constraint M. The high-level intuition can be framed as a fractional knapsack problem (see Figure 6 for an illustration).
When training solely on knowledge-dent optimal learner seeks to store as much knowledge as possible within its capacity. As a result, the total amount of memorized knowledge scales proportionally with the model's capacity M (Section 4.3).
data, where each fact appears with equal probability, the
5

---

Train exclusively random facts
Train Model caparity
should allocato my oxpucity she "marginal value" of each dataset!
X random facts (D:)
"Marginal, value" test lees rediation per extra unit "capacity" assigned to a datases
Model capacity M Knowlolyy Senowooggoddmie (facte with uniform prub. p)
(1 F) web datte (D.)
Thcorctioal Result (Efformal)
Marphall wike of
Magrisal of
[C(M)
knowlegee
The model stores a maximall number of facts without excerding iie "capocity"
dara becomme worth leariig
sloper C. (M
Kay Exomplec 4,- A M + (Powrr Lax: Web Doto)
Knuwledge stured ot capacity
Miros
Pr.
Figure 6: An illustration of the intuition behind our theory.
However, the situation changes when the knowledge-dense data is mixed with web-scraped data. In this case, the optimal learner should allocate its capacity across the two datasets based on their respective "marginal values" that is, the reduction in test loss resulting from assigning one additional unit of capacity to dataset. Only when M exceeds certain threshold does the knowledge-dense data become worth learning.
4.2 Problem Formulation
The essence of language modeling is to model the distribution the next token y for given context æ containing all previous tokens. We take Bayesian view, assuming latent variable 0 E @ governing the distribution of (z, y), denoted as (ær, y) ~ DD. Conceptually, 0 encodes knowledge about the world. For example, person may be born in 1996 in one universe but 1999 in another. Or, in different universe, popular Python libraries may feature a different set of functions. We assume the universe first draws 6 from prior P before we observe the data distribution Dg.
Data distribution.
learning algorithm A procedure that takes samples from a data distribution 'D of (ar, v) and outputs predictor h A(D), which maps to distribution over y. The performance of h is measured by the expected cross-entropy loss [(;; D) == ((,,))>D[- log ply h, x)]. where ply x)) denotes the predicted distribution of y given x by the predictor h, and log in base 2 for convenience. We measure the performance of a learning algorithm A by its expected loss over all data distributions Da with respect to the prior P:
Learning Algorithm.
(P( Ee~ (C(A()))C)(()))))) Da)].
In practice, predictor h can be a transformer, and A can be the pre-training algorithm.
Model Capacity and Mutual Information. We measure model's "effective" capacity the amount of information model, produced by some learning algorithm A, stores about the data distribution Da- the mutual information (MJ) between the model and the data distribution Do. i.e., ((A(D)); Do). For practical learning al algorithms with bounded capacity, if AA always outputs a model with at most N parameters each represented by a b-bit floating number, then T (A(D)); Do) < bN by information theory. Empirically, Allen-Zhu and Li [2024a] found that T(A(Da); Da) z 2N holds across various training setups by controlled experiments.
We model a sufficiently trained LM with capacity M as an optimal bounded-capacity learner, which minimizes the expected loss much as possible under the capacity constraint M:
Definition 4.1 (Optimal Bounded-Capacity Learner). For given prior P and M > 0, the best achievable loss under the capacity constraint M is defined as
(M( inf (Ep(A) T(A(Do); Do) S M), A
where the infimum is taken over all learning algorithms. An optimal M M--bounded-capacity learner is learning algorithm A such that (AA(),); Do) & M and (x(A) [M(M).
4.3 Warmup: Training Exclusively Mixture of Facts
We start with a simple case where the data distribution D, contains K random facts. Each fact is a pair (X,, yi), where X, is set of input contexts (c.g., paraphrases) and ys is the target token. For

---

instance, the fact "Gracie Tessa Howell born in 1946" can have contexts like "Gracie Tessa Howell's birth year is" or "Gracie Tessa Howell came to this world in the year," all mapping to the target y "1946". We further assume that , are disjoint.
Let Po(y | 1)) be the next-token distribution given context æ. The universe samples y1, U2....: yK independently from fixed distributions ....... .VK and sets 0 = (U1,... DK). The universe further sets (y (;)) as a point mass at yı. Vx; E X; Other inputs x may oceur in Dg, but their target tokens are independent of O. Define the exposure frequency of the i-th fact the total probability that any æ E X; appears D3: Exrxx Po(x = x').
K facts have equal exposure frequency (despite different entropies), a bounded-capacit learner reduces expected loss linearly with capacity M. thus no phase transitions:
Theorem 4.2. For all M 2 O, if all the facts have the same exposure frequency then Fp(M) C + p- max {Hrot M,,}.
(3)
where Hoot DK H(O)) and C == FP(0o).
4.4 Data Mixing Induces Phase Transitions
What if we mix the random facts with data from another domain, say web text? Consider data distribution Da composed of two domains: (1) a mixture of K random facts (as in Section 4.3) and (2) another domain with much more complex structure. Let the latent variable 0 - (01, 02).
where 0 governs the distribution of K random facts, 01 and 02 governs the data distribution of the second domain, ! Assume the universe draws 0₁ and 0g independently from priors P1 and P2, respectively. The overall data distribution Do is Do = rD((1 A: + (1 10--ra -r)DD with mixing ratio r E (0, 1). Let p denote the exposure frequency of each fact in D and Heot := Σ H(),) be the total entropy of the target tokens in the first domain (as in Section 4.3). For simplicity, assume the two domains contain non-overlapping information (see Definition F.5).
82
To measure models' performance on the first domain after training with algorithm A on the data mixture, define (A) Ee-p, [C(A( D as the model's expected loss the first domain. If &, (A) = Fp, (0), then the model learns nothing (random guessing). C (A) = Fr, (oc), the model perfectly learns the facts.
Theorem 4.3 shows that the learner sharply transitions between the two extremes model size increases. This transition is characterized by two functions: Mo (4) == sup M 2 0 : -F (M) > () and Mtt (t) == i 2 0: - (M) 1). By rate-distortion theorem, Fr, (M) is convex and hence F%,, (M) ) is non-increasing. Thus, Mf (t) and Mý (t) mark the last and first model sizes where - FF* exceeds or falls below t. If Fj,, (M) is strictly decreasing, then Mj (t) = Ma (t). Theorem 4.3 (Phase Transition in Model Size). For any optimal M -bounded-capacit learner A,
1. ifM < MT - P)) then Li (A) Fr, (0);
2. if Mo (1= p) + Hoor, then (A) Fp, (oc).
Key Example: When Web Data Loss Follows a Power Law in Model Size. Consider the case where Fr, (M) is a power-law function of M, i.e., Fp, (M) C+A-M Here, (0, 1) and A is a large constant. This is a reasonable assumption since LLM pre-training usually exhibits such power- law scaling behavior in model size [Kaplan al., 2020, Hoffmann et al., 2022]. In this case, taking the derivative of Fp, FM)(M) gives - -F (M) A - Then, M, (t) M, (t) (40 M(++1) Plugging this into Theorem 4.3, we have the critical value for model size:
1/(+11)
Muhres
(4)
rP
This implies that small p may cause the model to learn nothing from the knowledge-dense dataset, even if its capacity is sufficient to learn the entire dataset.
Arranging the terms in (4), we can also obtain the critical value in the mixing ratio r:
1
Tthres
p M-M++1
(5)
7

---

Liamaa2
Gemmar2
Qwen2.5
Rtted
Trresold frequncy (log (21k
Vol. low on FirelWee-d
3.2-
fitted madel scaling law:
10000
C 1.769 + 4.562a/ 4.00s
fitted dope: - 152
R.0-
www-smmbbbiigggg params,
05
120
160
295
410
1C0
200
903
100
10
10'
15
105
Model parameters (log
Mocal size (M)
Trescheld popularity Oog scale)
(a) Threshold frequency of syn- thetic biographics across differ- ent model sizes.
(b) The scaling law for the vali- dation loss on FineWeb-Edd with respect to model size.
(c) The threshold popularity for knowledge tested in PopQA v.s. model size.
Figure 7: Validating the power-law relationship of threshold Frequency and model size. (a) & (b): Experiments on the mixture of SynBio-1Ok-powert lOk-power-law and FineWeb- Web-Edu confirm that (1) the threshold frequency follows power-law relationship with model size, and (2) the power-law exponent is approximately equal to the model scaling exponent plus one. (c): For the three open-source model families we examined, the threshold popularity for knowledge tested in PopQA also follows a power-law relationship with model size.
Threshold Frequency for a Single Fact. For each fact the first domain, its overall probability of being sampled is rp in the data mixture. Again, arranging the terms in (5), obtain that for single fact to be learned by the model, its frequency of appearing in the pre-training corpus should be larger than threshold frequency fihres. which scales with the model size following power law:
fihres
(6)
Mo+1
5 Power-Law Relationship of Threshold Frequency and Model Size
In this section, we validate the predicted power-law relationship between model size and threshold frequency on both synthetic biographies and a set of knowledge extracted from Wikipedia.
5.1 Experiments on Synthetic Biographies
We construct SynBio- 10k- polk-powe--law, where 10k biographies are divided into 100 subsets of 100 individuals, with subset sampling probability following a power-law distribution (exponent 1.5). Within each subset, all biographies have a uniform sampling probability. We then mix this dataset with FineWeb-Edu using r = 0.01 and train models under this setup.
To estimate the threshold frequency fthres: we sort the subsets by sampling probability in descending order and identify the first group where model accuracy falls below a target value Otarget- The frequency biographies in this subset is used to approximate fthres. We use Ottarget 80%.
As shown in Figure 7(a), log fthres and log M exhibit a linear relationship. yielding slope of 1.152. This value is larger than 1, as expected from our theory. Further, we wonder if this slope is indeed close to a + 1. Following the approach of Hoffmann et al. [2022], we fit model scaling function for Fine Web-EEu validation loss in Figure 7(b), obtaining Ra 0.283. This leads to a predicted exponent of 1.283, which close to the observed value 1.152.
5.2 Experiments on Knowledge Extracted from Wikipedia
We further evaluate models PopQA [Mallen et al., 2023], which contains 14k QA pairs derived from Wikidata triplets, along with monthly page view for corresponding Wikipedia articles. Since knowledge tested in PopQA can be structured triplcts, expect them to exhibit similar threshold frequencies for a given model size.
consider them as homogeneous and
Estimating the Threshold Frequency. Counting the frequency of specific knowledge in the pre- training data is challenging due the scale [Kandpal et al., 2023]. Following Mallen al. [2023], we use Wikipedia page views as proxy for popularity, which is assumed roughly proportional to the frequency of the knowledge in web data. To estimate the threshold popularity Pihress we identify the smallest popularity P such that the model's 's accuracy on knowledge with popularity above P meets the target accuracy Qtarget which is set 60% our experiments. See Appendix E.6 for details,
Threshold frequency and model size follow a power law. We examine base models from Llama- 2 [Touvron et al., 2023], Qwen-2.5 [Qwen et al., 2024], and Gemma-2 [Team et al., 2024], which
8

---

subsamptnngg
r=2
m0
SymBlo (86)
m=2
m0
ReTwase 0.02
15000
WC our Wusurataiggg
35%
30
p
12500
O ong mord
0.5
10
20
p-
7500
W/ Wasterrelo
pa50h
5000
10
0:00
2.68
2.70
2.72
2.26
2:20
2.29
2:50
FineWeb Edu val. less
Pile val. loss
(a) 410M, train from scratch Web--Eddd&SyuBBoo.22MM
(b) 410M, continual pre-train on (c) 1B, continual pre-train the the
Pile&SynBio-2.56M.
Figure 8: Our proposed strategies significantly boost knowledge acquisition under low mixing ratios while preserving models' general capability.
are likely trained on similar data mixtures within each family. Figure 7(c) reveals that log Pehres generally decreases linearly as log model size increases, though the slope varies across families due to differences in architecture and training data. We examine more model families in Appendix D.4.
6 Strategies to Enhance Knowledge Acquisition Under Low Mixing Ratios
Inspired by our theory, we propose two simple yet effective strategies to enhance knowledge acqui- sition under low mixing ratios. This setting common in practice, large r may harm general capabilities expected to be acquired from other data sources. The key idea is to raise the frequency of each fact, thereby increasing the "marginal value" of the knowledge-dense data.
Strategy 1: Random Subsampling: Randomly subsample the knowledge dataset.
Compact Knowledge Mixing (CKM): Rephrase the knowledge into compact form and add the rephrased version to the original dataset while keeping the overall mixing ratio fixed. See implementation details in Appendix E.7.
Strategy 2:
We validate these strategies SynBio and WikiBio, curated dataset of Wikipedia biographies. For example, on WikiBio, random subsampling and CKM improve the number of learned facts by 4x and 20x, respectively. The effectiveness of random subsampling is especially surprising, as it yields higher accuracy despite discarding a significant proportion of the knowledge-dense data.
6.1 Real-World Knowledge Data: WikiBio
To extend our study to a more real-world scenario, we curate WikiBio, a dataset containing Wikipedia biographies along with ten paraphrased versions for 275k individuals, totaling 453M tokens. This task is more challenging than SynBio as WikiBio features diverse texts without uniform formats, requiring the model to generalize to queries that rarely have exact matches in the training data. See Appendix E.2.2 for dataset construction details and Appendix E.7 for evaluation details.
6.2 Strategy 1: Random Subsampling
While random subsampling seems counterintuitive at first glance, it becomes reasonable if we consider how the threshold mixing ratio Tthres relates to exposure frequency of each fact within the knowledge-dense dataset, denoted p. For dataset containing only S facts with uniform probability, x 1/S. We can derive from (5) that the threshold mixing ratio Tthres ~ M Subsampling reduces S and thus lowers the threshold mixing ratio, allowing the model to achieve much higher accuracy on the subsampled dataset. We use p to represent the subsampling ratio below.
Experimental Setup. We study both pre-training from scratch and continual pre-training setups. To evaluate the model's general capabilities, we use its validation loss on the web data (the Pile or FineWeb-Edu) and its zero-shot performance on five downstream tasks (see details in Appendix D.7). We compare the validation loss and average downstream performance to the model trained with r in the pre-training-fee -fronngg--------- comm-scaachh setup or to the original Pythia model in the continual pre-training setup. Downstream performance drop of more than 2% considered unacceptable.
Subsampling enables faster knowledge acquisition while maintaining general capability. We train 410M models from scratch Fine Web-Edu mixed with SynBio- 1.28M using E (0,0.1,0.2,0.33

---

for total of 32B tokens. As shown in Figures 8(a) and 14(a), increasing degrades Fine FineWeh-Fdu validation loss and downstream accuracy, with performance becoming unacceptable at r = 0.3 (-2.09% accuracy, +0.05 loss), while SynBio accuracy remains near zero. In contrast, subsampling SynBio-1.28M 2.8M to 25%, 50%, and 56.25% boosts SynBio accuracy to 23.53%, 37.46%, and 39.81% 1%, respectively, while maintaining downstream performance within the acceptable range. Note that further increasing to 62.5% makes the frequency of each biography too low, resulting in SynBio accuracy dropping back to near zero. See more details in Appendix E.7, Tables 2(b) and (aa).
Consistent Results for Continual Pre-training. We continually pre-train the 410M or IB Pythia models from their 100k-step checkpoints on the mixture of the Pile and WikiBio or SynBio-2.56M. The 410M models are trained for 32B tokens and IB models for 64B. When r is large, the Pile validation loss may increase with training due to catastrophic forgetting [lbrahim et al., 2024]. To preserve models' general capabilities, we apply early stopping when Pile validation loss increases by 0.05 (410M model) or 0.03 (IB model), each corresponding to 2% drop in downstream performance. As shown in Figures 8(b) and 14(b), without subsampling, r = 0.1 or 0.15 results in slow learning of WikiBio, while r 0.2 triggers early stopping after 20B tokens, resulting in poor WikiBio performance. By contrast, subsampling WikiBio to 25% or 50% significantly accelerates knowledge acquisition and keeps Pile validation loss acceptable. For example. for r 0.1, setting to 50% improves the number of learned facts by 4 times. Similar trends hold for 1B models: subsampling SynBio to 50%
0.2 outperforms both 0.2 and early-stopped r 0.4 without subsampling by ~ 30%. See more details in Appendix E.7, Tables 2(c), 3(b) and 4.
6.3 Strategy 2: Compact Knowledge Mixing (CKM)
The second strategy rephrases knowledge into compact forms (c.g.. tuples) and adds them to the original dataset. Given that the frequency of each fact f is inversely proportional to its average representation token count, this augmentation reduces the average token count, thereby increasing f's effective frequency and potentially pushing it above the threshold fthres. CKM is in the same spirit as the data synthesis technique in Su et al. [2024], which rephrases high-quality data into condense forms such as QA pairs and knowledge lists.
For WikiBio, we compress the key information name, birth date, and occupation intt the tuple format "Bio: N {name} B (birth date) O foccupation]". We add these tuples until their token count equals proportion F (which we call the CKM ratio) of the original dataset's 's token count.
Experimental Setup. We apply CKM to WikiBio with the same continual pre-training setup as in Section 6.2. We apply carly stopping when Pile validation loss increases by 0.05.
CKM significantly improves knowledge acquisition efficiency while preserving general capa- bility. We fix 0.1 and explore CKM ratios 10.1, 0.3,, 0.6), which correspond to roughly 2x, 3x, and 4x increases in fact frequency, respectively. As shown in Figures 8(b) and 14(c), CKM preserves the general capability and consistently boosts knowledge acquisition. Notably, performance on WikiBio improves by 4x when the short-form augmentation makes up only 10% tokens of the original WikiBio dataset. Increasing T to 30% further boosts the number of learned facts by 20x. See downstream performance in Table 5.
7 Discussions and Future Directions
Extensions to reasoning tasks. Although our experiments mainly investigate factual knowledge, we also identify phase transitions in simple reasoning tasks. This suggests a commonality: memorization is foundational to reasoning, not just to fect-recall. Without basic knowledge, models cannot reason effectively, an observation shared by Ruis et al. [2024], Xie et al. [2024]. For instance, solving math problems requires memorizing theorems, definitions, and techniques. We defer the exploration of more complex reasoning tasks to future work.
Connection to real-world data. Following Allen-Zhu and Li [2024a], we use synthetic biographies as a proxy for knowledge-dense data for controlled and quantitative experiments. In contrast, real- world datasets are more heterogeneous for example, Wikipedia includes both simple facts (e.g.. biographies) and more complex content (e.g., scientific theories). These types of knowledge vary in learning difficulty and may exhibit different threshold frequencies. As a result, phase transitions may not be as apparent when mixing a heterogeneous dataset with web text. Nevertheless, our findings still apply at the level of individual knowledge pieces. That is, specific fact or reasoning procedure may not be acquired at all its frequency or the model size falls below a threshold, as evidenced by the theoretical result in (6) and empirical evidence in Section 5 and Section 3.3.
10

---

Acknowledgements and Disclosure of Funding
J.Z. acknowledges support by the National Key R&D Program of China 2024YFA1015800 and Shanghai Qi Zhi Institute Innovation Program.
References
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.144 /4316, 2023.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws, arXiv preprint arXiv:2404.0544 054405, 2024a.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation, 2024b.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Halla- han, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf. Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in Py Torch, 2023. URL https: / /www gi thub. com/ eleutherai/ggtt
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivan- shu Purohit, and Edward Raff. Emergent and predictable memorization large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. The Eleventh International Conference on Learning Representations, 2023,
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang. Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 05457, 2018.
Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut, Analyzing commonsense emergence in few-shot knowledge models. In 3rd Conference on Automated Knowledge Base Construction, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.

---

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Data mixing made efficient: A bivariate scaling law for language model pretraining. arXiv preprint arXiv:2405. 2024.
Gaurav Rohit Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction. In Proceedings of the 41st International Conference Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 15558. PMLR, 21-27 Jul 2024.
Dirk Groeneveld, Iz. Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradcep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannanch Hajishirzi. OLMo: Accelerating the science of language models. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Bangkok, Thailand, August 2024. Association for Computational Linguistics.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30016-30030. Curran Associates, Inc., 2022.
Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10711-10732, 2024.
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats I Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprini 200:::200300876533 2024.
Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and Zico Kolter, Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11822 2024.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn tail knowledge. In International Conference on Machine Learning. pages 15696-15707. PMLR, 2023.
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Automatic prediction of compute-optimal data composition for training lmss. arXiv preprint arxiv:2407.20177 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jía Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: of permissively licensed source code. arXiv preprint arXiv:2211.1555 2022.
12

---

Jeffrey Li, Alex Fang, Georgios Smymiss Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406. 11794, 2024.
R Li, LB Allal, Y Zá, N Muennighoff, D Kocetkov, C Mou, M Marone, C Akiki, J Li, Chim, et al. Starcoder: May the source be with you! Transactions on machine learning research, 2023.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01 2024.
Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. Scaling laws for fact memorization of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguisties: EMNLP 2024, pages 1 11263-11 11262- Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 1/236/1/22224. 1/00i..iiin.. Hundings-emmlp.66
Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D Hwang. Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, et al. Datadecide: How to predict best pretraining data with small experiments. arXiv preprint arXiv:2504. / 1393, 2025.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non- non-parametric memories. In Anna Rogers, Jordan Boyd-Grale Grabbe,, and Naoaki Okazaki, editors, Proceedings of the s Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 9802-9822. Toronto, Canada, July 2023. Association for Computational Linguistics.
David Mizrahi, Anders Boesen Lindbo Larsen, Jesse Allardice, Suzie Petryk, Yuri Gorokhov, Jeffrey Li, Alex Fang, Josh Gardner, Tom Gunter, and Afshin Dehghan. Language models improve when pretraining data matches target tasks. arXiv preprint arXiv:2507. 12466, 2025.
Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories. In The Thirteenth International Conference on Learning Representations, 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo furious, 2025.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv: 1606.06033 1, 2016.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024.
Guilherme Penedo, Hynek Kydliček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The Web datasets: Decanting the web for the finest text data at scale. In The Thihty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2024.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 1909.0/ 01066 2019.
13

---

Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67 2020.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprinl arXiv:2002.08910 2020.
Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. Procedural knowl- edge in pretraining drives reasoning in large language models. arXiv preprint 1rXiv:2411.22580 2024.
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-ce: Transforming common crawl into refined long-horizon pretraining dataset. arXiv preprint arXiv:2412.02559. 2024.
Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledgeable are large language models (Imss? aka will Ilms replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 311-325, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen. Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard. Piotr Stanczyk. Sertan Girgin, Nikola Momchev, Matt Hoffmann, Shantanu Thakoor, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Plucinfe Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej. Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin. Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya,
Grill, Behnam Neyshabur,
Harleen Batra,
14

---

Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290. 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXivv2307.092288 2023.
Johannes Welbl. Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint 1707:v::700.000
2017.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization large language models in logical reasoning, arXiv preprint arXiv: 2rX1v:2410.23123. 2024.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint 16952,,, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905. 078830, 2019.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et Deepseek-coder--22 Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. /931,, 2024.
Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, and Soham De. How do language models learn facts? dynamics, curricula and hallucinations. arXiv preprint arXiv:2503.. 22...266666 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main contributions of this paper are (I) identifying two phase transitions in knowledge acquisition within data mixtures and (2) providing theoretical understanding of these phenomena. The abstract and introduction are closely aligned with these contributions, providing detailed overview and context for our findings and theory.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other settings.
fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: See Appendix F.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might specific model and empirical evaluation, may be necessary
suffice, or if the contribution to either make it possible for others to replicate the model with the same dataset, provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
17

---

Justification: We provide detailed instructions for how reproduce experiments in Ap- pendix E. Moreover, our experiments are based on the public gpt-neox-library We are currently preparing our codebase for public release and will make it available once the cleaning and documentation process is complete.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: httts//nipps.... cc/ ceuiiii//ddesssss "CodSSubiiss settmmmssssii onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. cc cttppbclclblcccccccc publ public//ee /et//uiiieiee iesi////// CodeSulom ss onPol icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the mean and standard deviation of accuracy in Table for the experi- ments in Figure 2(a). However, it is computationally infeasible for us replicate all experiments with different random seeds. As shown Table 6, our experiments involves training LLMs from scratch, which is computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1- 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric enror bars that would yield results that out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
See Table 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///eurippes http://///...oot/tttccccooeeeeee
Answer:
[Yes]
Justification:
We strictly follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it legitimate to point that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategics (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our paper does not have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the public datasets, code, and models used in this paper. explicitly mention their licenses in Appendix E.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com / datasets has cu- rated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
20

---

Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See Appendix E.2.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mmodll part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.x
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs is an important, original, or non- standard component of the methods in this research? Note that if the LIM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
21

---

Justification:
We only use LLMs for writing, editing, and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / /nnurrip...... cc/Conferences /2025/02222222 /LLM) for what should or should not be described.
22

---

Contents
1 Introduction
2 Experimental Setup
3 Phase Transitions of Knowledge Acquisition within Data Mixtures
3.1 Phase Transition in Model Size
3.2 Phase Transition in Mixing Ratio
3.3 Phase Transitions on Reasoning Tasks
Theoretical Analysis
4.1
High-Level Intuition
4.2 Problem Formulation
4.3 Warmup: Training Exclusively on Mixture of Facts
4.4
Data Mixing Induces Phase Transitions
5
Power-Law Relationship of Threshold Frequency and Model Size
5.1
Experiments on Synthetic Biographies
8
5.2
Experiments Knowledge Extracted from Wikipedia
Strategies to Enhance Knowledge Acquisition Under Low Mixing Ratios
6.1
Real-World Knowledge Data: WikiBio
6.2 Strategy 1: Random Subsampling
6.3 Strategy Compact Knowledge Mixing (CKM)
10
7 Discussions and Future Directions
10
Limitations
25
Broader Impact
25
C Related Works
25
Additional Experimental Results
26
D.1 Additional Results for Phase Transitions
26
D.2
Ablation
27
D.3
Additional Discussions and Experiments Reasoning Tasks
27
D.4
Additional Results for Validating the Power-Law Relationship Threshold Fre- quency and Model Size
29
D.5 Additional Plots for Strategies to Enhance Knowledge Acquisition
29
D.6 Detailed Performance SynBio
30
D.7
Detailed Downstream Performance
30
E Experimental Details
32
23

---

E.1
General Setup
32
E.2
Details of Dataset Construction
................................
33
E.2.1
Constructing the SynBio Dataset
................................
33
E.2.2
Constructing the WikiBio Dataset
.i..............................
34
E.3
Constructing the SlopeQA Dataset
36
E,4
Constructing the Max-over-N Dataset
38
E.5
Details of the Fitting Process
38
E,6 Details of Estimating the Threshold Popularity
38
E.7
Experimental Details for Strategies to Enhance Knowledge Acquisition
40
Proofs of Theoretical Results
Fl1
Convexity of the Best Achievable Loss
41
F2 Proofs for the Warmup Case
42
F3 Proofs for the Data Mixing Case
43
24

---

A Limitations
The high computational costs to conduct all these experiments impede us from replicate all the experiments with different random seeds. These costs include the number of GPU hours. For example, a typical run of training a 410M model for 32B tokens requires 256 A100 GPU hours. Despite these difficulties, we managed to conduct experiments on models up to 6.9B and conduct ablation studies on hyperparameters in Appendix D.2.
B Broader Impact
This paper identifies two phase transitions in knowledge acquisition within data mixtures and provides theoretical understanding of these phenomena. Building on our theory, we propose two strategies to enhance the efficiency of knowledge acquisition. Our findings offer deeper insights into LLM behavior and can be applied to improve the factual accuracy of LLMs.
C Related Works
Knowledge Capacity Scaling Law. LLMs typically trained vast amount of data that are rich in knowledge, and extensive studies have investigated how much knowledge LLMs can acquire from the training data. Pioneering studies [Petroni et al., 2019, Roberts et al., 2020, Da et al., 2021] demonstrate that LLMs can capture a substantial amount of knowledge, suggesting their potential knowledge bases. To quantify the relationship between model size and knowledge storage, Allen-Zhu and Li [2024a] and Lu et al. [2024] discover a linear relationship between models' knowledge capacity and their parameter count by training LLMs on data only containing formaf knowledge for sufficiently long horizons. Later, Nichani et al. [2025] formally proved this linear relationship. contrast, this paper examines the data mixing scenario and demonstrates that this linear scaling can be disrupted when the knowledge-dene dataset is mixed with vast amounts of web-scraped data. Another important factor is the frequency of occurrence for knowledge.
Impact of Frequency on Knowledge Acquisition. This paper identifies phase transitions in knowledge acquisition within data mixtures with respect model size and mixing ratio. Some relevant observations can be found in previous papers, but we takes a more direct and systematic approach. Kandpal et al. [2023], Mallen et al. [2023], Sun et al. [2024] find that LLMs can perform poorly on low-frequency knowledge, Ghosal et al. [2024] show that frequency of knowledge in the pre-training data determines how well the model encodes the knowledge, which influences its extractability after QA fine-tuning. Taking a more microscopic view, Chang et al. [2024] insert few pieces of new knowledge during training and track their loss. By fitting a forgetting curve, they conjecture that the model may fail to learn the knowledge if its frequency is lower than some threshold.
Memorization and Forgetting. Our findings also relate to prior observations on the memorization and forgetting behaviors of LLMs, but we explicitly characterize phase transitions in the context of data mixing. Carlini et al. [2023] show that memorization of training data follows a log-linear relationship with model size, the number of repetitions, and prompt length. Biderman et al. [2024] take a data levet-level perspective and demonstrate that it is difficult to predict whether given data point will be memorized using smaller or partially trained model. By injecting few new sequences into the training data, Huang et al. [2024] find that a sequence must be repeated a non- trivial number of times be memorized. By examining training dynamics, Tirumala et al. [2022] observe that memorization can occur before overfitting and that larger models memorize faster while forgetting more slowly. Zucchet et al. [2025] study the training dynamics governing factual knowledge acquisition of LLMs and find that the performance can undergo plateau before the model acquires precise knowledge, during which the attention-based circuits form. From a theoretical perspective, Feldman [2020] prove that memorization of training labels is necessary to achieve near-optimal generalization error for long-tailed data distributions.
Scaling laws for Data Mixing. LLM performance is significantly influenced by the mixing propor- tions of the training data from different domains. Our paper is related to a line of studies that optimize the mixing proportions by modeling L.LM performance as a function of the mixing proportions [Liu et al., 2024, Kang et al., 2024, Ye et al., 2024, Ge et al., 2024]. However, their datasets can be highly
25

---

heterogeneous within single domain (e.g.. OpenWehText, Pile-CC) while focus mixing a uniform, knowledge-dense dataset into web-scraped data.
D
Additional Experimental Results
D.1
Additional Results for Phase Transitions
Acc. SynBio-208 eo-20k (96)
Acc. SynBio-100 10k (96)
30
40-
20
10
0.00 0.01 0.03 0.04 0.05
0.00
0.04
Mixing ratio
Mixing ratio
(a) Pythia-2.8B.
(b) Pythia-6.9B.
Figure 9: Phase transition in mixing ratio persists for larger models. We train Pythia-2.8B and Pythia-6.98 with 2B and 1B total training tokens, respectively. To ensure sufficient exposure to SynBio within these training horizons, we use smaller SynBio datasets-SSnBio--D for the 2.8B model and SynBio-100 10k for the 6.9B
with FineWWe-Edu
Val. Loss on SynBlo-1.2 1.28M
Val. Lass on SynBio-322 320k
0.8
0.8
0.6
0.6
(lag
0.5
0.4
0.4
0.0
0.2
0.4
0.3
160256 Model size (M, log scale)
Mixing ratio
(a) Validation loss on SynBio-1.288 1.28M for 410M models.
(b) Validation loss on SynBio-320k for
0.1.
Figure 10: In addition to discrete metrics like accuracy, we can also observe phase transitions in validation loss, a continuous metric.
Table We replicate the experiments in Figure 2(a) with three different random seeds and report the mean and standard deviation below. While accuracy varies slightly with random seeds, the phase transition behavior remains consistent and clearly observable across runs.
Mean Acc. (%)
Std. Dev. (%)
0.1
0.4
0.0
0.15
0.4
0.0
0.2
0.4
0.0
0.25
0.8
0.3
7.3
2.8
0.35
27.5
2.8
0.4
40.4
3.1
0.45
58.1
3.1
26

---

Nixdng reto and batch size 0.,0)
Miing mite leaming
Mixing ratio and learning rate schedule (r. sthel (0aucosine)
(0.2.comm
-0-
(0.85113)
- (04.512)
-0-10.25522)
-0-10.8
10.5 1e:3)
-0-(022e3)
-0- (0.5
- (0.4, mid) -Q-(0.2, vsd)
-d- (0.0.1924)
-4- 1044.024] -6-10.2.1624)
-6-
10.8.4e-3)
-a- --.44 -6-(0.24463)
ALC. on knewledge (60
80
80-
Acc. oe knewledge
60
60
40
40
40
20
20
Total tokens
(a) Vary the batch size.
(b) Vary the peak learning rate.
(c) Vary the learning rate schedule. Both schedules use peak learning rate of 10-3
Figure 11: Ablation studies on hyperparameters. The models exhibit consistent trends in knowledge acquisition across different batch sizes, learning rate values and schedules. All experiments are conducted by training 70M models on the mixture of Fine and SynBio-32Ok.
D.2 Ablation Studies
We conduct ablation studies to demonstrate the robustness of our findings with respect to hyperparameters. We explore r {0.2, 0.4, 0.8} and train 70M models for a total of 64B, 32B, and 16B tokens, respectively, ensuring each configuration passes SynBio the same number of times.
Consistent Trends Across Different Batch Sizes. As shown in Figure I I1()), evaluate three batch sizes, B E {256, 512, 1024], for each and observe consistent general trends across all batch sizes. For 0.4 and 0.8, smaller batch sizes yield slightly higher accuracies, likely due the increased number of update steps. These experiments further distinguish between two types of frequency which the model encounters the knowledge dataset: per-token frequency and per-step frequency. For fixed mixing ratio, doubling the batch size doubles the occurrences of each biography per step, while the occurrences per token remain unchanged. The results demonstrate that per-token frequency, rather than per-step frequency, determines training efficiency in knowledge acquisition.
Consistent trends across learning rate values and schedules. In Figure 11(b), explore peak learning rates among {2.5 x 10-4, 10-3, 4 x 10-33 using the WSD scheduler. We observe that the trends are consistent across these values, although the learning process slows down at the lowest value 2.5 x 10 In Figure II(c), results for both cosine and WSD schedulers show similar trends.
D.3 Additional Discussions and Experiments on Reasoning Tasks
Acc. on Max-over-N (96)
60
100
Acc. on Max-over-N
(%, log scale)
40
20
0.4
0.5
90 160256410 Model size (M, log scale)
Mixing ratio
(a) Accuracy on Max-over-N for 70M models.
(h) Accuracy on Max-over-N for r 0.2.
Figure 12: Phase transitions in the Max-over-N task. Here, set N 30.
Discussion on the slope calculation task: model indeed learning the procedure rather than memorizing the training data. We show that the model cannot rely purely memorization to
27

---

solve the slope calculation tasks specified in Section 3.3. Specifically, the total number of possible slope calculation problems our setup is 100 x 100 x 99 x 100 = 9.9 x 10". In contrast, a typical training run Figure 5 with r 0.4 sees fewer than 3.5 10 unique slope calculation examples, less than 5% of the full space. Despite this limited coverage, the model still achieves 60% accuracy test time, where examples are uniformly sampled from the full distribution. This substantial generalization beyond the training data suggests that the model is indeed learning a generalizable computation procedure, rather than memorizing specific input-output pairs.
Experiments on Another Reasoning Task with Larger Input Space: Max-over-N. The slope calculation task suggests that the model learns generalizable procedure. To test this hypothesis in more challenging setting, we introduce another task named "Max-over-N" This task is explic- itly designed with vastly larger input space, rendering memorization computationally infeasible. Specifically, the model is asked to find the maximum number given list of N 30 integers, each randomly sampled from {0, 1, ... }99.. This creates an enormous input space of 1060, The format of the training samples is shown in Table 13.
Following the setup in Section 3.3, we add 3M tokens of such examples to the Open dataset (accounting for less than 0.02% of the original Open WebMath token count) create a modified version. We then train Pythia models on the mixture of Fine Web-Edu and the modified Open WebMath dataset, with r denoting the mixing ratio of the modified Open WebMath. For evaluation, we generate 1,000 test samples of Over-Nve-N" and assess whether the model outputs the correct final answer.
As shown in Figure 12, we observe the same phase transition phenomena with respect to both model size and mixing ratio, consistent with our main findings. The key result is that the 70M model achieves 60% test accuracy after training on fewer than 3,500 unique examples (at 0.5). This training set is an infinitesimal fraction of the 1060 possible inputs. This clearly demonstrates generalization beyond memorization.
28

---

D.4 Additional Results for Validating the Power-Law Relationship of Threshold Frequency and Model Size
In Figure 13, we relax the constraint of training on the same data mixture and investigate the overall trend between model size and Pthres. We add the Llama-3 [Dubey et al., 2024] family, and evaluate both base and instruction-tunct models for all families, totaling 30 models. Interestingly, in Figure 13, log
Owen2 5
GMt (sredicted)
Fiied intenol
(ground
ion-ambbediingg Params (8) log
10
model size and log Pthres also exhibit a linear relationship, with
102
ar 5348
most models falling within the 95% confidence interval. We further use models from the OLMo [Groeneveld et al., 2024]
102
101
family as validation set, where predictions of the fitted power law closely match the ground truth.
10"
10
104
10"
10a
10"
Inferring the Size of Proprietary Models. The identified power-law relationship offers poten- tial method for estimating the size of proprietary models, such GPTs. As a preliminary attempt, we estimate the threshold popularity for GPT-3.5-Turbo, GPT-4, GPT-4o, and GPT--oo mini. Applying the fitted power law yields size predictions of 61B, 514B, 226B, and 24B, respectively. The 95% confidence intervals are 12-314B, 80-3315B, 19-1313B, and 5-118B, respectively.
Potential Application:
Threshold popularity (log scale)
Figure 13: For 410M models trained on the mixture of FineWeb-Ead and SynBio-1.28M, accuracy for r = 0.2 remains near zero even when we extend the training by 4 times.
D.5 Additional Plots for Strategies to Enhance Knowledge Acquisition
subsompiing
Sperstrcam ne0 2
hum nere tesmme
acc knowlsdeg 196
40
2000
BoN
30020
2000
30
1000
20
10
$
2..20-
62.5%
2:20-
35.5
56.0
56.5
57.0
24
Downstream avg. acc. (%)
Bealtmin toxens 383
Total training Cakens 380
(a) 410M, trained from scratch on the mixture of Web-WWe-EBd and
(b) Training trajectory for applying subsampling to WikiBio.
(c) Training trajectory for applying CKM to WikiBio.
SynBio-1.28M.
Figure 14: Additional plots for strategies to enhance knowledge acquisition.
29

---

D.6 Detailed Performance SynBio
In Table 2(a), we detail the accuracy of each attribute for 70M models trained on the mixture of FineWeb-Edu and SynBio-320k with E {0.2, 0.4, 0.8), trained for 64B, 32B, and 16B tokens respectively. We notice that the accuracy for birth date is lower than other attributes. This can be attributed to the complexity of precisely recalling the combined elements of day, month, and year information, which together form a much larger domain than other attributes. To maintain clarity and conciseness, we omit the detailed performance in other 70M experiments, as this pattern persists across them.
Furthermore, present the detailed performance of 410M models on SynBio- 1.28M corresponding to Figure 8(a) in Table 2(b). We also provide the detailed performance of 1B models on SynBio-2.56M corresponding to Figure 8(c) in Table 2(c).
Table Detailed performance on SynBio. We report the accuracy (%) for each attribute averaged over five templates.
(a) 70M model, pre-trained from scratch on the mixture of Web-Edu and SynBio- 320k.
Birth date
Birth city
University
Major
Employer
Avg.
Random guess
0.00
0.50
0.33
1.00
0.38
0.44
0.2
0.00
0.63
0.43
1.12
0.38
0.51
0.4
16.96
45.67
41.03
50.78
43.93
39.68
0.8
79.76
88.64
88.55
90.10
88.30
87.07
(b) 410M model, pre-trained from scratch the mixture of FineWeb-Edu and SynBio- ..28.
N
p (%)
Birth date
Birth city
University
Major
Employer
Avg.
Random guess
0.00
0.50
0.33
1.00
0.38
0.44
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.28M
100
0.1
0.00
0.42
0.33
1.01
0.21
0.39
1. 1.28M
100
0.2
0.00
0.45
0.34
1.09
0.22
0.42
1.28M
100
0.3
0.00
0.49
0.35
1.14
0,25
0.45
320k
25
0,2
22.34
23.98
23.
24.03
23.65
23.53
640k
50
0.2
27.97
39.66
38.51
41.50
39.68
37.46
720k
56.25
0.2
28.02
42.94
42.15
44.07
41.88
39.81
800k
62.5
0.2
0.01
1.16
0.85
3.19
0.89
1.22
(c) IB model, continually pre-trained the mixture of the Pile and SynBio-2.56M. Note that 0.4 early stopped due to its Pile validation loss increasing beyond acceptable range.
Training tokens (B)
Birth date
Birth city
University
Major
Employer
Avg.
Random guess
0.00
0.50
0.33
100
0.38
0.44
Pythia 1h-a-IBBBi 18-1100--ckp
0.00
0.00
0.00
0.00
0.00
0.00
2.56M
100
0.2
64
0.01
0.46
0.33
0.98
0.21
0.39
2.56M
100
0.4
24
0.05
10.95
3.90
4.74
3.64
4.66
1.28M
50
0.2
64
23.95
34.55
35.05
35.96
35.19
32.94
D.7
Detailed Downstream Performance
We employ the In-eval-harneet [Gao et al., 2024] codebase evaluate the zero-shot perfor- mance on five downstream tasks, including LAMBADA [Paperno et al., 2016]. ARC-E [Clark et al., 2018), PIQA [Bisk al., 2020], SciQ [Welbl et al., 2017], and HellaSwag [Zellers et al., 2019], cov- ering core capabilities such text understanding. commonsense reasoning. and question answering. We compute the validation loss on about 50M tokens on holdout set from the Pile Fine Web.FFu. The detailed downstream performance and validation loss for applying the random subsampling
30

---

strategy SynBio and WikiBio presented in Tables 3 and 4, respectively. Additionally, we report the detailed downstream results for applying CKM to WikiBio in Table 5.
Table 3:
Detailed downstream performance and validation loss for applying the random subsampling strategy to SynBio. We report the accuracy (%) and standard deviation (%) in the format (sC. dev.) for each downstream task.
(a) 410M model, train from scratch.
FibeWeb-Edu
N
(%)
LAMBADA
ARC-E
Seig
PIQA
HellaSwag
Avg.
0
38. 25,
(1.83(1.00)
83.60
68. (1 o0)
35.04
57.35
2.067
1.28M
100
0.1
34.56(o.s0 86)
(2.33
83.50(1.17)
68.34(1
35. 35.13 13
56.77(L
0.58)
2. 2f 0.001)
1.28M
0.2
(4.43
62. 62.13(0.99)
(3.80(1.17)
68.12(1.09) (1.0)
35.39 48)
56.77(4 0.58)
2.668(1 0.001)
1.26M
100
0.3
33.91 (6))
60.77 (1.00)
(0.80(1.26)
66.54(1 10)
3/1.33ro.
55.26(L 2.09)
2.722(1 0.054)
320k
0.2
36.70(o.87)
60.35
(82.701.20
67.74(1.00 (00)
34.76,o.4 48)
56.45(L 0.90)
2.686(1 0.019)
610k
50
0.2
36. 58 (7)
60. (1.0))
83.30
66. G5 10)
311.53
56.33(L 1.02)
2. G88 0.021)
720k
56.25
0.2
35.61 (0.87)
60.94(1.08)
83.00 18)
67. (1(1.)))
34.54(0.47)
56.25(L 1.10)
2.687(1 0.020)
800k
62.5
0.2
35.22 (7)
60.48(1.00)
83.40(1.200 20)
66.54(1 10)
34.455o.45
56.01(L 1.34)
2.688(1 0.021)
(b) IB model, continually pre-trained. Note that 0.4 is early stopped due to its Pile validation loss increasing beyond the acceptable
range.
Training tnkens (R)
Pile val. loss
N
LAMBADA
ARC-E
Sciq
PIQA
HellaSwag
Avg.
Pythia- IB- 100k-ckpt
55.66 55.66ç0.c0)
54.50(1..2)
(3.00((..99)
70.78(1.08)
36.97 (6.97(0.48) (0.48)
60.18
2.168
2.56M
100
63.68(0.60)
(1.0))
(1.00(
68.7T
35. 19.911pm (o.48)
58.17(4 2.01)
2.181(1 0.016)
2.50M
100
0.4
24
(2.38(0.70)
61.47(1.03)
80.70(1.255
68. 68.17 ..93
34. (4.95(0-48)
57.53( 2.66)
2.198(1 0.030)
1.28M
50
0.2
64
54.71 (0.60) (o.7))
62.86(1.02)
(1.30
68.99(s.08)
35.48(0.48)
58.67(4 1.51)
2.189(1 0.022)
Table 4: Detailed downstream performance for applying the random subsampling strategy to WikiBio. We use to denote the subsampling ratio. We report the accuracy (%) and standard deviation (%) in the format (std. dev.) for each downstream task. Note that r 0.2 is early stopped due its Pile validation loss increasing beyond the acceptable range.
Training tokens (B)
N
P (%)
LAMBADA
ARC-E
Sciq
PIQA
HellaSwag
Avg.
Pile loss
Pythia IB ckpt
50.86
52. 62.10(1.05)
83.70
67.14 (7.14(1.10)
34.09(o.43)
57.58
2.255
277k
100
0.1
32
50.77 (0))
48.95(1.0s)
60.80(1.25)
66.43(1.10)
33.16 (60o.7)
56.02(L
1.56)
2.286(1 0.031)
277k
100
0.15
32
19. 19.12 12 76)
19. (6(1.0))
81 (1.))2))
66.38(1.10)
32.84
55.96(L 1.62)
2.292(1
0.037)
277k
100
0.2
49.37 (0.70)
(0.87(1.03)
79.70
(1.40
33.08
55.48() 2.10)
2.306 0.051)
69k
25
0.1
32
48.63
50.59(1.om)
81.00(1.24)
(6.49(1.10)
33.16
55.97(L 1.54)
2.286(1 0.031)
137k
50
0.1
32
50.30 ro)
50.38 (1.0)
78.80
66.27(1.10)
33. 13.16
55.78() 1.80)
2.285(1 0.030)
208k
75
0.1
32
(0.34(o.ro)
49.20
80.10(1.26)
66.97(1.10)
33.19(0.47)
55.96(1 1.62)
2.286(1 0.031)
Table 5: Detailed downstream performance for applying the compact knowledge mixing strategy on WikiBio. We use to denote the CKM ratio. We report the accuracy (%) and standard deviation (%) in the format (cc. dev.) for each downstream task. Note that r = 0.2 is early stopped due to its Pile validation loss increasing beyond the acceptable range.
Training tokens (B)
T (%)
LAMBADA
ARC-E
Sciq
PIQA
HellaSwag
Avg.
Pile val. loss
Pythia- IB-IIBB- ckpt
50.86 70)
52.10(1.00)
83.70 17)
67.14(1.10)
34.09ço.ar (7)
57.58
2.255
0.1
32
50.77;0. 70)
18.95(1.00)
60.80(1.25)
66.13(1.10)
33.16 47)
56. 56.02(1 1.56)
2. 2.286(t 0. 0.031)
0.15
32
49.12,o. 70)
49.66
81.80 (22))
(6.38(1.10)
32.84/o 47)
1.62) 65.96(L
2.292(1 0.037)
0.2
20
49.37
(9.87(1.08)
79.70
65.40/1.11)
33.08(0.47)
55.48(} 2.10)
2.306(1 0.051)
0.1
10
32
49.70
49.54
80.40
66.32(1.10)
33.1 47)
55.81(1 1.77)
2.287(1 0. 0.032)
32
50.11 To)
49.12(1.o8)
(1.20
66.54(1.10)
33.11
55.82(1 1.76)
2.285(1 0.030)
0.1
60
32
49.99 70)
(9.41(1.00)
80.00(1.77)
65.78(1.13)
32.99/0.47)
55.63(L 1.76)
2,286(T 0.031)
31

---

E Experimental Details
E.1
General Setup
Code Base and Hyperparameters. Our experiments use the GPT-NeoX library [Andonian et al., 2023]. For all experiments, we the batch size 512 and the sequence length as 2048. For all the experiments in Section 3, we use a Warmup--Stblle-ee -ecay (WSD) learning rate schedule with a peak learning rate of 10-3, We allocate 160 steps for warmup and the final 10% steps for cooldown. We keep other hyperparameters consistent with those used in Pythia.
Hardware. We train models of sizes 70M and 160M using 8 NVIDIA RTX 6000 Ada GPUs, while models of sizes 410M and IB are trained using either 16 NVIDIA RTX 6000 Ada GPUs or 8 NVIDIA A100 GPUs. The estimated runtime required to train each model size IB tokens is detailed in Table Consequently, a typical run training 410M model on 32B tokens takes approximately 32 hours, whereas the longest run, which trains IB model on 64B tokens, exceeds five days.
Table 6: Estimated runtime required to train each model size on IB tokens on our hardware.
Model size
Hardware
Runtime (h) per billion tokens.
70M
0.25
SxNVIDIA RTX 6000 Ada
160M
0.70
410M
16xNVIDIA RTX 6000 Ada or 8XNVIDIA A100
1.0
IB
2.8B
5.8
8xNVIDIA A
6.9B
16.89
Implementation of Data Mixing. Let S denote the total number of training tokens. Then, the model sees rS tokens from the wwwlleggedennss dataset and (1 - -) tokens from the web data. dataset and web data. Since
Let S and S2 denote the total sizes (in tokens) of the knowledge-denee Si is small (< 1B tokens) and S2 is large (> 1T tokens), for the training horizons S considered in our experiments, we typically have rS S and -r) < S2. In this case, replicate the knowledge-dense dataset S/ times, sample random (1 -rrr) Stooken subset from the web data, and then shuffle the combined data.
Licenses for the Public Assets. The FineWeb-Fdu and Open WehMath datasets are under the ODC-BY License. The Pile dataset is under the MIT License. The Pythia model suite and the gpt-neox-library are under the Apache License 2.0. All of them are open for academic usage.
32

---

E.2 Details of Dataset Construction
E.2.1
Constructing the SynBio Dataset
To generate names, we collect a list of 400 common first names, 400 common middle names, and 1000 common last names, resulting in 1.6 10 unique names. To generate SynBio-N, sample N names from this set without replacement. For each individual, the value for each attribute randomly assigned as follows: birth date (1-28 days 12 months x 100 years spanning 1900-2099), birth city (from 200 U.S. cities), university (from 300 institutions), major (from 100 fields of study), and employer (from 263 companies). Each attribute is paired with five sentence templates, which are used to convert (name, attribute, value) triplets into natural text descriptions. A complete list of sentence templates provided in Table 7, and an example of synthetic biography can be found in Table
Table Sentence templates to generate the SynBio Dataset.
Attribute
Template
[name] } was born on {birth datc).
name came into this world {birth date).
Birth date
name 's birth date {birth date).
name
date birth {birth date).
namee] }
celebrates (possessi ve pronoun) hirthday on {birth
date].
[name] }
spent {possessi ve pronoun) early years in {birth city). name was brought up in {birth city).
Birth city
name
's birthplace is (birth city].
name
originates from (birth city).
[name]
was born in (birth city).
[name]
received mentorship
guidance from faculty members
(niivrsityy).
University
[nmme]
graduated from (university).
name spent (possessive pronoun) college years at (university).
[name]
completed [possessive pronoun) degree at {university).
[name] completed (possessive pronoun] academic journey at
[un (university).
[name] completed (possessive pronoun] education with a focus on [major].
Major
name] devoted (possessive pronoun) academic focus to [major): a degree {major).
(ame
[aae focused (possessive pronoun academic pursuits on {major)
[name]
specialized in the field of [major]-
[name]
employed (employer).
name
staff member at {employer)
Employer
name] associated with (employer].
[ame engaged in work (employer).
[name]
part team at (employer).
Table 8: An example of synthetic biography. The values that expect the model to recall during evaluation are underlined.
Gracie Tessa Howell's birth date August 09, 1992. Gracie Tessa Howell's birthplace St. Louis, MO. Gracie Tessa Howell received mentorship and guidance from faculty members at Santa Clara University. Gracie Tessa How- ell has a degree Robotics. Gracie Tessa Howell is engaged in work at Truist Financial.
33

---

E.2.2 Constructing the WikiBio Dataset
To create the WikiBio dataset, we first query Wikidata gather names and birth dates of individuals from 16 common occupations. We then identify each person's 's Wikipedia page by matching their name with the page title. We retain the first paragraph of each page, it typically provides short summary of the person's 's life and contains key biographical information. The detailed composition listed in Table 9. Finally, align with the evaluation setup in Section 6, we filter the dataset to ensure both the person's 's occupation and birth date are explicitly mentioned.
Inspired by Allen-Zhu and Li [2023], we employ LJama-3. 1-100---nsssst -nsstnsttcttt to paraphrase each biogra- phy ten times, thereby simulating the real-world scenario where models encounter different variations of the same person's 's information during training. See Table 10 for the prompt for paraphrasing. An example of the original text and the paraphrased versions are presented below.
Original text: Rebecca Jo Budig (born June 26, 1973) is American actress and television presenter. Her career began in 1993, and in 1995, she was cast in the role of Michelle Bauer on the CBS soap opera Guiding Light. In 1999, she was cast as Greenlee Smythe on the ABC soap opera All My Children; she held the role off-and-on until the network series finale in 2011. In 2015, she was cast in the role of Hayden Barnes General Hospital. In 2019, Budig was cast on L.A.'s Finest, as Carlene Hart, the drug trafficking-socaa soccer mom of two young school-age children, a boy and girl.
Paraphrased versions:
1. Born on June 26, 1973, Rebecca Jo Budig is an American television presenter and actress with career spanning nearly three decades. Her professional journey began in 1993 and has included notable roles such as Michelle Bauer on Guiding Light and Greenlee Smythe on All My Children. After playing the latter role on-and-off until 201 she went on to portray Hayden Barnes in General Hospital in 2015 and Carlene Hart in the 2019 television series
L.A.'s Finest.
2. With a diverse career in television, Rebecca Jo Budig, born June 26, 1973, has established herself talented actress and presenter. Her career milestones include her roles as Michelle Bauer in the CBS soap opera Guiding Light, and Greenlee Smythe in All My Children. Her portrayal of Greenlee spanned several years, concluding with the show's 's finale in 2011. Budig continued to expand her repertoire with roles in General Hospital and L.A.'s Finest.
3. Rebecca Jo Budig is versatile American actress and television host, born June 26, 1973. Since her career began in 1993, she has landed prominent roles in several television series. One of her earliest notable roles was Michelle Bauer in Guiding Light, followed by her portrayal of Greenlee Smythe in All My Children, a character she played until the series' conclusion in 2011. In the years that followed, she appeared in General Hospital and L.A.' Finest.
Rebecca Jo Budig, an American actress and television presenter, was born on June 26, 1973. She began her career two decades later, securing the role of Michelle Bauer on Guiding Light. Budig's s subsequent roles have included Greenlee Smythe on All My Children, a part she played intermittently until the series ended in 2011. Her later appearances include role in General Hospital and as Carlene Hart in the series L.A.'s Finest.
5. American actress Rebecca Jo Budig was born June 26, 1973, Her television career, which began in 1993, encompasses multiple notable roles, such as Michelle Bauer on the soap opera Guiding Light and Greenlee Smythe All My Children. She portrayed the latter character until the series finale in 2011. Budig later appeared as Hayden Barnes in General Hospital and took on the role of Carlene Hart in L.A.'s Finest.
Since launching her career in 1993, Rebecca Jo Budig has established herself as a talented actress and television presenter in the United States. Born on June 26, 1973, she has appeared in a range of notable roles, including Michelle Bauer on Guiding Light and Greenlee Smythe on All My Children. The latter role spanned several years, concluding with the show's 'oo's finale in 2011. Her subsequent appearances include General Hospital and L.A.'s 's Finest.
7. Rebecca Jo Budig, born on June 26, 1973, has enjoyed a successful career in American television as an actress and presenter. Her breakout role came 1995 when she was cast as Michelle Bauer Guiding Light. Later, she played the character Greenlee Smythe on All My Children, a part she held intermittently until the show's finale in 2011, Her more recent roles include appearances in General Hospital and as Carlene Hart in L.A.'s Finest.
34

---

8. Born June 26, 1973, Rehecca Jo Budig is talented American actress and television presenter. Since her career began in 1993, she has appeared in various television series. Notable roles include her portrayal of Michelle Bauer on the soap opera Guiding Light, as well as Greenlee Smythe on All My Children. Budig continued to expand her acting repertoire with roles in General Hospital and L.A.'s Finest, including her portrayal of Carlene Hart.
As an American actress and television host, Rebecca Jo Budig has had a diverse career spanning nearly three decades. Born on June 26. 1973. she began her professional journey in 1993. Her notable roles include Michelle Bauer on Guiding Light and Greenlee Smythe on My Children, a character she played until the series finale in 201 Her subsequent appearances include General Hospital and the series L.A.'s Finest, where she portrayed Carlene Hart.
10. With career television that began in 1993, Rebecca Jo Budig, born June 26, 1973, has established herself as versatile actress and presenter. Her early roles include Michelle Bauer on Guiding Light, while her breakout role came as Greenlee Smythe on All My Children. She continued portray Greenlee intermittently until the show's ' finale in 201 1. Her later roles include appearances in General Hospital and L.A.'s Finest, where she took on the role of Carlene Hart.
Table 9: Detailed Composition of WikiBio.
Occupation Num. Wikipedia biographics
Singer
18.482
Actor
31,846
Politician
38,653
Businessperson
8,068
Mathematician
5,093
Physicist
4,296
Writer
26,746
Football player
56,547
Basketball player
16,956
Sport shooter
3,156
Tennis plater
7,602
Swimmer
9,108
Painter
12,927
Volleyball player
3,556
Composer
13.719
Athlete
18,013
Total
274,768
Table 10: The prompt for paraphrasing the first paragraph of Wikipedia documents.
creating the training data for an LLM. would like to teach it to flexibly extract knowledge from Wikipedia paragraph.
Therefore, I want to diversify the Wikipedia paragraphs as much as possible so that the
model can learn the actual relationships 4 between entities, rather than just memorizing the text. Please assist with the + paraphrasing task. Paraphrase the following Wikipedia paragraph about (Wikipedia document title] 10 times. Aim to make the paraphrased versions as varied possible. Ensure all - essential information is retained,
particularly the information about the birthday and the occupation.
35

---

E.3 Constructing the SlopeQA Dataset
Every time the model sees a slope calculation example, we first uniformly sample 21, 3/1, 42, y/2 from {0, 11, 1,.... .99 (ensuring * 22), and then apply randomly chose question and step-by-step answer templates. We prompt GPT-4o to generate diverse question and answer templates, as shown in Tables 1l and 12. The final answer is expressed the simplified fraction.
Table 1I: Question templates for the slope calculation subtask.
Q: Find the slope of the 1ine passing through ((x1), (yl}) and ( [x2), (y2)).
Think by-
Q: what the slope the line passing through (xx1), (y1}) and (x2),, {
Think step-by-sten
Q: Compute the slope of the line going through (x1))xy1) and (x2), (y2)).
Think
Q: Determine the slope of the line connecting ([x1), {yl}) and ( [x2], (y2)).
Think step-by-step.
Q: If a line goes through ([x1), (y1}) and ((x2), {y2}), what is its slope?
Think step-by-step.
Table 12: Answer templates for the slope calculation subtask.
A: Recall the slope formula:
k= k= (x2-x1)
1. Let (x1, = ((x1), (y1}) and (x2, {y2}).
2. Compute the difference:
xx-x
3. Plug into the formula: k=(x2-x1)/(y2-y1}
### Final Answer:
k=(k)
36

---

A:
1. Identify the coordinates: First point: (x1, (f11 (y1])
Second point: (22 (x2))
2. Compute the difference in y-values: (v2)-(y1)=1y2-y1}
3. Compute the difference in x-values: [x2]-(x1]=(
Divide the differences: x=(y2yy1//xx2-x
### Final Answer: k=(k)
A: We use the slope formula:
(y2-y1)) (x2-x1)
Plug in the coordinates:
k=
2. Simplify the numerator and denominator:
k= (=y2-y11)/(2-xx1
Simplify the fraction:
k=(k)
### Final Answer:
k=(k)
A: To find the slope, we use the slope formula: k= (y2-y11/ (x2-x1)
are given two points: ((x1), (y1)), (x22), {y2})
Plug the values into the formula: k= (
#4# Final Answer:
k=(k)
37

---

A: The slope k between (x1,x2) and (yl,y2) is given by: (y (x2-x1)
1. From the problem: ([1yy))=((x]] (y1)), (x2, (y2})
2. Substituting into the formula:
3. Simplify:
(=yy-y11/(x22-1
### Final Answer:
k=(k)
E.4
Constructing the Max-over-N Dataset
We design a new task named "Max-over-N", where the model is tasked with outputting the maximum number of a list of N integers, each randomly sampled from {0, 1, ,,,,,99)))) In our experiments, we N 30.
Table 13: An example of the Max-over-N subtask.
Q: Find the maximum value of the following list: [47, 83, 38, ...] Think step step-by-sten stee....
Compare 47 and 83. Keep 83. Compare and Keep 83.
E.5
Details of the Fitting Process
We use T to denote the required training steps to reach 40% accuracy and to denote the mixing ratio.
Fitting the exponential function. We fit T' with respect to for all r 2 0.3 using the function A (xp(B/r), where A and B are coefficients to be fitted. Taking logarithmic both sides, we obtain a linear function log T log A + B/r. By fitting log T against 1/r with linear regression, obtain log A = - 0.25512, B 1.5137 with ofoddness-offitt R 0.9980.
T()) =
Fitting the pawer-law function. We fit T with respect to for all {0.3, 0.4, 0.45, 0.5, 0.55) using the function T(r) = Cr-D, where C and D are coefficients to be fitted. Taking logarithmic on both sides, we obtain linear function log T log C D logr. By fitting log T against log r with linear regression, we obtain C' x 0.098158, D R= 3.83878 with goodness-offfit R² 0.9853.
E.6
Details of Estimating the Threshold Popularity
Following Mallen et al. [2023], we evaluate models using 15-shot prompting. We use the prompt presented in Table 14 for evaluatioin and allow models generate up 128 tokens with greedy decoding. To assess answer correctness, we employ Llama-3.1 -lama-3.1-8B-Instnncc as judge. Specifically, instruct the Llama-3.1188 1-.1--BBInntuu model to evaluate the semantic similarity between the model- generated answer and the reference answer provided in PopQA. The prompt used for the Llama judge detailed Table 15.
After judging the correctness of each answer, we use Algorithm to estimate the popularity threshold. In our experiments, we set the target accuracy Otarget 60% and the fault tolerance level Niil 5.
38

---

Table 14: The prompt for evaluating models on PopQA.
You are a helpful assistant. want to test your knowledge level.
Here are a few examples.
(few shot examples text with templates)
Now, I have question for you. Please respond in just few words following the style of the examples provided above.
Table 15: The prompt for testing synonym.
htt___eett><(sstttt__eeeeeeeee beste / end head_heaee____ i
Cutting Knowledge Date: December 2023
Today Date: 19 Dec 2024
You are linguistic expert specializing in synonyms. Your task is to determine whether two given English words are synonyms or not. synonym is word that has a very similar meaning to another word and can often replace it in sentences without significantly changing the meaning.
For each pair of words provided:
1. Analyze their meanings and typical usage.
2. Decide whether they are synonyms (Yes/No).
3. Provide brief explanation for your decision.
Here are some examples to guide you:
Words: "happy" and "joyful"
Yes Explanation:
Both words describe a state of being pleased or content and are often interchangeable in most contexts.
Words: "run" and "jog"
No
Explanation:
While both refer to forms of movement, "run" typically implies faster pace than "jog."
Words: "angry" and "frustrated"
No
Explanation: Although both express negative emotions, angry" implies strong displeasure or rage, while frustrated" conveys annoyance due to obstacles or failure.
< <|eot_id><< http:d/ie..attr.aaaeeeeeeeeeeeee
Words: () and ()
<leot_id|><> staestttareeeeeeteeet header hed idwssidddssssstttttsssaa end id/>
39

---

Algorithm 1: Estimate Threshold Popularity
1: Input:
2: x: A list of popularity values for each data point, where xy represents the popularity of the
i-th data point.
3:
y: A list of binary values indicating the correctness of the model's response, where y/2 = the model answers the i-the question correctly, and y, 0 otherwise.
4: -Itarget: The target accuracy.
5: Niat:: The maximum number of failures before termination, denoting the fault tolerance
level.
6: Output:
7: Pihres: The threshold popularity.
8: Initialize correct count: sum correct 4-
9: Initialize error count: e <- 0
Sort (z. y) by x in ascending order and store the indices in list 7.
11: Initialize loop variable <- len(x))
12: Initialize flag counter flag f-
13: while j 2 do
14: A +
15: while k 2 0 and F4 21, do
16:
+ k 1
17:
end while
18:
for forl=1 ls to j do
19:
i- I
20:
Lum_correct t sum_correct + y:
21:
end for
22:
if sum correct
len())-A-I < set threhhll then c+
23:
end if
25: if e
NEil then
Return: Iı, (Return the threshold popularity]
27: end if
j +
29:
end while
30: Return: Tip {If no such point found, return the smallest popularity value}
E.7 Experimental Details for Strategies to Enhance Knowledge Acquisition
This subsection presents the experimental details for Section 6.
Evaluation Details for WikiBio. For simplicity, we focus on how well the model memorizes one specific type of fact: the birth date of person, which ensured Specifically, for each fact, which can be represented prompt the model with "The (occupation inn) (name) was born on" and consider the response correct if it contains the correct birth year and month. The occupation is included in the evaluation prompt not only to avoid prompts that are exactly identical the training data but also to provide additional context for disambiguation.
mentioned in WikiBio. (name, occupation, birth date) triplet, we
Implementation Details of CKM. When we apply CKM to WikiBio, augment the dataset by adding compact tuple representations. To maintain the same token budget for WikiBio after augmentation (as and the total training tokens fixed), we proportionally reduce the number of epochs. Although the model completes fewer epochs over the dataset, each fact's frequency per epoch is increased, boosting total exposure during training. For example, in Figure 8(b), setting T= 0.1, 0.3 and 0.6 correspond to roughly 2x, 3x, and 4x increases in fact frequency, respectively. Each time models encounter the tuple-form data point, the order of birth date and occupation is randomly flipped.
Experimental Details of Random Subsampling. In Figure 8(a), we train all models from scratch on the mixture of Fine Web-Fdu and SynBio-1.251 1.28M using the cosine learning rate schedule with peak value of 10-3 In Figures 8(b) and 8(c), following Zhu et al. [2024], we continually pre-train
40

---

intermediate checkpoints of Pythia models. This strategy allows without experiencing extreme loss spikes. Specifically, we continually pre-train 410M and 1B Pythia models from their respective 100k-step checkpoint with a constant learning rate of 8.7 10 which corresponds to the original learning rate used at step 100k in the Pythia model training.
larger learning rate
Proofs of Theoretical Results
We follow the notations in Section 4. We use H()) to denote the entropy and [(:; denote the mutual information.
We define a data distribution as a distribution over (æ, v), where x is an input and is a token. data universe U (P, D3) is defined by prior over latent variable 0 and family of data distributions D, indexed by 0.
A predictor is function that maps to distribution over y. A learning algorithm A is procedure that takes samples from data distribution D of (z, y) and outputs a predictor h ~ A(D) in the end. For given predictor h, we measure its performance by the expected cross-entropy loss
[h; D) == (a.,))_ Pog h,,),,
where p(y h, x) denotes the predicted distribution of y given x by the predictor h, and log is in base 2 for convenience. For data universe U (P, Do), measure the performance of learning algorithm A by its expected loss over all data distributions D, with respect to the prior P:
(A(A) Ee Da)].
(8)
We use the mutual information ((A(D)); D2) as measure of the effective model capacity for the predictor picked by A on Dg, where 0 is sampled from Q.
Same as Definition 1, for data universe U (P, De) and M > 0, we define the best achievable loss under the capacity constraint M as
FM( inf (Cp(A): I(A(D)): D,) M) A
where the infimum is taken over all learning algorithms. An optimal Mf-bounded-capacity learner is a learning algorithm A such that 7 T(A(Do); Do) < M and (p(A) = (M).).
Convexity of the Best Achievable Loss
It is easy to see that Fp(M) is non-negative and non-increasing in M. A classic result in rate- distortion theory is that the rate-distortion function is convex. This further implies that F' (M) is convex in M. Here we present as lemma for completeness.
Lemma F.1. For any data universe U = (P, Fp(M) is convex in M.
Proof. Let > 0 be any positive number. Let Ai be learning algorithm that achieves loss S FF(M) + with mutual information 7, (AA(D))) Dp) S M1 and A2 learning algorithm that achieves a loss Fp(M2)
with mutual information (AAo))) De) S M2.
Let A be learning algorithm that outputs the same An with probability p and the same as A2 with probability p. Then the mutual information between A(Do) and D, is
I(A(Do):Do) (1- (A, Do) + pl (A
I (1 - p) PMM-
By linearity of expectation, the expected loss AA can be bounded as
(a-p())[[(((D))) Po)] (1 p) Do)l (Eo.P1o)(((A((())))))
< (1 F)F(() (PP) 2e.
Therefore, we have
Fp Fp(11-))M pM2) < (oo-Poo(((A(D)):Po) S (M) +PF 2c,
taking + 0 finishes the proof.
41

---

F.2 Proofs for the Warmup Case
Definition F.2 (Factual Data Universe). We define fact as a pair (X, where X is set of inputs and y a target token. factual data universe is a data universe U (P,Da) Da) containing K random facts (X1, vi)...., (Xx, yk) in the following way:
1. Xg are K disjoint sets of inputs, and y1..... UK are random tokens;
2. 0 is structured as (y),.... yk) Given 0 = (y),...., ./k) the data distribution Dg satisfies that for all X, Dy | (4)) is point mass at y::
3. For all 0, the input distribution Dz(x) is the same;
4. For all 8, the target distribution Do 22) is the same for all & U X;;
5. The prior distribution P over 0 given by the product distribution P(y .2.....
I (())). where V% a fixed prior distribution over yk-
The exposure frequency of each random fact is defined the total probability that an input x X occurs in Dg.
Theorem F.3 (Theorem 4.2, restated). For a factual data universe U = (P, Do) with K random facts, if all the facts have the same exposure frequency p, then
Fp(M) = =C Cpppp max (Het M,
(10)
where Hot == =E H(V)) and C == Fo(loo).
Proof. First, we prove a lower bound for Fp(M). For any learning algorithm A with D() s M.
(r(A(Da)) Ba-pB(ax))a BRnAA(Do log p(y h, x)]
E,EgwpB Byy ((D)) log pvv h,
K
2 E. 1 1( X. Hamp (D 1
+D Σ (H())) - I(A(Do):uE))
=:Co
Co + [Hlot - ((A((D)):Do))
Co p [Hhor MM
For upper bounds, we first show that (P(M) < Co for all M 2 HEor Let Ai be the learning algorithm that inputs Dg and outputs the predictor h that always outputs the token yi for the input x E X₁. For all the other inputs æ, the predictor just outputs hy h, = Ee-p æ)]. Both
A1 (Dg) and Da can be transformed from 0 with a reversible function, so
I(Ai (Da): - =1 Σ H(AA) Hhot.
is easy to see that (>(Ai) Co. This implies that Fr(M) S Co for all M 2 H
Now, if M < Heot, we construct a learning algorithm Ag that outputs the same as Ag with probability q and outputs h(y h x) Eq- (Do (y x)] with probability q. Setting g Hese we have
1 (A, Do) = HHot = M.
By linearity of expectation, we also have (p(A)(Do)) = (Ai))) + (1- p DK H())). This implies that (M < Lp(A,) + max H M, 0} for all M < Heat-
Putting all the pieces together finishes the proof.
42

---

E3 Proofs for the Data Mixing Case
Definition F.4 (Mixture of Data Universes). Let (PI, ) and U2 (P2, ) be two data universes. We mix them together to form a new data universe U (P,D)):
1. is structured as (01, 02). Given 0 = (01,02), the data distribution Da is formed as Da r D (1-r) r) where r is called the mixing ratio;
2. The prior distribution P over 0 is a joint distribution of Pi and Pz.
In reality, mixing datasets can be seen mixing two data universes first and then sampling data distribution from the mixed data universe. Here we consider the simplified case where the two data universes are different from each other that they convey orthogonal information.
Definition E.5 (Orthogonal Mixture of Data Universes). We say that U is an orthogonal mixture of U1 and U4
1. For any that is in both supports of D and D'? we have D0 (v x) = (y a) for all
0, and 02. In other words, the conditional distribution of the next token given the context remains consistent across both domains and is unaffected by variations in values of 0, and 02.
2. P(0,, (2) Pi (0:) P2(02), i.e., 02 and 0g are independent.
Below, we first establish two lemmas that provide conditions for when the loss on the first domain will be very low or very high for an optimal M-bounded-capacitt learner given an orthogonal mixture of two data universes. Then, we use these lemmas to prove Theorem 4.3.
We use D F=)(t) and D+F(t) to denote the left and right derivatives of a function F at a point t, respectively.
Lemma F6. Let = (P) be an orthogonal mixture of = (/) and U2 = (P2, ( with mixing ratio r. For all r E (0,1) and M 2 0, if the following inequality holds.
D (M, (+)
(11)
T-r
then for any optimal M -bounded-capacity learner A on U1 Eo. (C(h; (hih;;;; D (O).
Intuitive Explanation. We define (z( = Ra-P, (CLA(D)) ). similar to (A). The overall test loss is given by (A) (AA) (1- (2A). Since D*F < 0. we can rearrange (11) obtain rD-Fr, (0) (1-)-(11)---))) (M) > 0. Intuitively, this means that increasing the capacity assigned to learn ii by one unit and reducing the capacity for U2 by one unit will increase the overall test loss, compared fully assigning capacity to l2 and none U1. Alternativeyy,
we can view (I-r)D (M as the ratio of cost-efffettivenes of the knowledge
dataset relative to web data. Hence, the model should prioritize web data and not learn from the knowledge-dens dataset when this ratio is below 1.
rD*FF,,00 (0)
Proof. Let h be the predictor picked by A on Do. Let X1 and Xr be the supports of x in DD and ( respectively. Let my = T Thh],, :Do)) ) and m2 = T(h),, Do,)). By data processing inequality, we have
mi 1 [11x Do,) S I(h:Do,).
m2 1 [(11x, (oo)) S [1h;; Do,),
Further noticing that [h;; Da) = I(h; Da, Da,) 2 I(h; Do,) + I(h; Do,), we have
mi + m2 - [h; De) S M.
Since hl2, and hxx,, are valid predictors on Do, and Doy, respectively, we have B[C(h; Do, )] = E[C(h), x, Do, 2 Fy, (m1), B[c Do = E[[())x De.)l 2 (m²) 2 Fr,(M - m1)-
43

---

Adding the two inequalities with weights and r, have
(A( = (I Do)] 2 rFp, (m1) (1-r) (M - -m1).
By convexity (Lemma F.1), we have
Fp, (m1) 2 Fp, (0) D+F (0)m², Fp. - m1) (M) DD DDFPp (M)mm. Plugging these into the previous inequality, we have
(h; Da)] 2 r Pr, (0) + (1 - Pp. + (rD+F,, (0) (1 (--)) (.>)DF7..
Fr, (M)) mi.
By (1I) and the fact that D+Fp, < O, we have rD =D+FE, (0) > (1 - r) Fr (M). So the right-hamn hand side is strictly increasing in m..
Now we claim that m O. If not, then the following learning algorithm A'' is better than A. Let A be an optimal -bouaded-capacityy learner on U1 and A2 be an optimal M-bounded-capacityy learner on U2- Run the algorithms to obtain 11 ~ A1 (Delxx) and h2 A2 (Dlxx)) Then, whenever seeing an input from X1, output f (x);; otherwise output h2(2)) This algorithm achieves the expected loss r rFp, (0) + (1 P), (M), which is strictly less than (A4) and contradicts the optimality of A.
Therefore, for the optimal algorithm A. (AA) = (0)
Lemma F.7. Let u = (P, Do) be an orthogonal mixture of = (P1, Do, and U2 = (P2, Do₂) with mixing ratio For all r E (0, 1), M 2 0 and B 0, if the following inequality holds,
D+Fp, (M D-Fr,(3)
(12)
then for any optimal M bbuuded-ccapcctt learner A on U1 EL
S Fp, (8).
Intuitive Explanation. Similar to the explanation for Lemma F.6, we can rearrange Equation (12) into -rr-rrDDpp I', (B) + (1 (1=r)D+Fp, (M- - B) < O. Intuitively, this means that reducing the capacity assigned to learn 2i by one unit and increasing the capacity for U4 by one unit will increase the overall test loss, compared to assigning capacity 3 to Ui and M 3 to 42. Therefore, the optimal MMboondddd-capacit learner A will assign at least capacity 33 to learn U1, resulting in a test loss rD=FF, (9)
on 14 that lower than Fr, (3). Alternatively, we can view (I-r)D1FP, (M--J as the ratio of cost- effectiveness of the knowledge-dense dataset relative to web data. Hence, the model should do its best to learn the knowledge-dena dataset when this ratio is above 1.
Proof. Similar to the previous proof, letting m. = I(hlx :D,)) and m2 == T(blx; Do,), we have
mi + m2 S Doh - M,
B[ Do, E[[(h)x Do,)] ) 2 Fr, (m1);
B[C(h; Do,)] B[C(h)x Do,)) 2 Fr, (m2)) 2 Fr, (M m1);
[F[[h:; Da) 2 rFp, (m1) + (1 - Fb, (M m1).
First, we show that mi 2 8. If not, then by convexity (Lemma F.1), have
Fp, (m1) 2 Fb, (8) - D FxF (8) (3 m1), Fp, (M m,) 2 Fy, (M - B) + +D+Fo, (M B) (B - m1).
Plugging these into the previous incquality, we have
E[c(h; Do)1 (F) + (1 -1-11-00 - rp, M B) (-rD (8) + r1-r))DFF,(MM B)) (8- m1).
By (12) and the fact that D Fr, (B) < 0, we have rD Fr, (8) (1- (M - B). So the right-hand side is strictly decreasing in mi.
Next, we prove by contradiction that 2 3. If m. < 3, the following learning algorithm A'' better than A. Let A, be an optimal 3-bounded-capacity learner on Ui and Az be an optimal (M- - -apbedcddddc learner on Uz. Run the algorithms to obtain h Ai (Dolxx, 1x; and h2 Ae(Dolx)) Then, whenever seeing an input from X1, output h (æ); otherwise output h2(2).
44

---

This algorithm achieves the expected loss r (B) (1 Pr, (M B), which is strictly less than (A).
Therefore. we have m 2 3 for the algorithm A. Now we prove that E[ Do,)l < Fp. (8). not, then the following learning algorithm AA" is better than A. Construct A" similarly as A'. but with A1 and A2 replaced by the optimal my-bounded-capacit learner on U1 and the optimal boa-bounded-capaciiin learner on l2, respectively. If [E[ Do, ) > Fp, (8), then A" achieves lower expected loss than A, which contradicts the optimality of A.
Now we consider the case where U1 is factual data universe, and Ul is an arbitrary data universe. Theorem E.8. Let U1 be a factual data universe with K random facts, each with the same exposure frequency p. and the entropies of their target tokens sum to Ho == E HY:).. Let U2 be an arbitrary data universe. Let U = (P,Do) be an orthogonal mixture Uz and ld with mixing ratio r. For all (0,1) and M
1. if - P -D-F (M), then (i(A) = (0);
2. if 1-, p> DDF Hoot), then (i = Fp, (00).
Proof. By Theorem F.3, (0) D-Fp, (Hrot) and Lemma F.7 with B = Hos finishes the proof.
p. Plugging this into Lemma R6
Now we are ready to prove the main theorem we stated Section 4.4. Recall that
Mr (t) sup M 0 Fb,, (M) > t):
MØt (t) == ( 2 0 (M) < t).
Theorem F9 (Theorem 4.3, restated). For any optimal M l--bundeddcapacing learner A,
1. M S MT ( p), then (1 = Fp. (0);
2. if M Mo (T= p) Hoot then Li (AA) FT.(oc)
Proof. This a direct consequence Theorem F.8 by noting that (1) DD--,,, is left continuous and non- increasingngggg in M; (2) - - DDFF,, (M) is right continuous and non- non-increasing in M; (3) Fr, (M)
almost everywhere differentiable.
45


Reviews:
[{"review_id": "XBOL41iacy", "reviewer": "Reviewer_N1QD", "summary": "This paper studies the relationship between model training and dataset mixing, model size, revealing some interesting phenomenon, along with some theoretical analysis.", "questions": "The paper has shown that the knowledge acquisition happens only when mixing ratio larger than a threshold for a fixed model size, what will happen if we use different mixing ratio at different stages of training, will the model forget what it already memorize?\n\nCould the authors provide results (for example Figure 2) with mixing ratio from 0 to 1 rather than from 0 to 0.4?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Some interesting findings are revealed via the experiments in this paper. For instance, given a fixed data mixing, the model memorizes knowledge only when its model size is larger than threshold. Given a fixed model size, data mixing also has a threshold, below which leads to model memorizing nothing even with extensive training.\n\nSome theoretical analysis are provided. The authors also discussed strategies to enhance knowledge acquisition under low mixing ratios.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive feedback and for recognizing the value of our findings. Below, we answer the reviewer's questions.\n\n---\n**Q1:** What will happen if we use different mixing ratios at different stages of training? Will the model forget what it has already memorized?\n\n**A1:** Yes, the model can forget previously memorized knowledge if the mixing ratio of knowledge-dense data is reduced in later stages of training. The extent of forgetting depends on the new mixing ratio.\n\nSpecifically, consider the following setup: we start from a model checkpoint $\\mathcal{M}_1$, trained for $T_1$ tokens with a knowledge-dense mixing ratio $r_1$. We then continue training for an additional $T_2$ tokens using a smaller mixing ratio $r_2 < r_1$, resulting in a new checkpoint $\\mathcal{M}_2$. The performance of $\\mathcal{M}_2$ on the knowledge-dense data depends on $r_2$:\n\n- For moderate $r_2$, the model can retain the knowledge from $\\mathcal{M}_1$ and keeps learning from the knowledge-dense data in the second stage, with $\\mathcal{M}_2$ achieving higher accuracy than $\\mathcal{M}_1$.\n\n- For small $r_2$, the model undergoes catastrophic forgetting, and the accuracy of $\\mathcal{M}_2$ on knowledge-dense data drops below that of $\\mathcal{M}_1$, even returning to near-zero levels when $r_2$ is too small.\n\n\nTo verify this, we use a 70M Pythia model as $\\mathcal{M}_1$, which has been trained for $T_1 = 24$B tokens on a mixture of FineWeb-Edu and SynBio-320k with $r_1 = 0.8$, achieving **62.6%** accuracy on SynBio. We then continue training with $r_2 < r_1$ for $T_2=32$B tokens. Results in Table 1 confirm the above arguments.\n\n\n\n**Table 1: Accuracy (%) on SynBio-320k with different mixing ratios during continual pre-training.** \n*CPT* stands for *Continual Pre-Training*. The initial checkpoint achieves 62.6% accuracy on SynBio.\n\n| Mixing Ratio ($r_2$) |  Acc. on SynBio(%) |\n|:----------------------:|:-------------------:|\n| 0.1                 | 0.4              |\n| 0.15                | 1.8               |\n| 0.2                 | 14.3               |\n| 0.25                | 35.3               |\n| 0.3                 | 50.0               |\n| 0.35                | 59.7               |\n| 0.4                 | 67.4               |\n\n---\n\n\n**Q2:** Could the authors provide results (for example, Figure 2) with mixing ratio from 0 to 1 rather than from 0 to 0.4?\n\n**A2:** Sure! Following the setup in Figure 2(a), we conduct additional experiments for $r>0.4$ using 70M Pythia models. We summarize the results for $r$ from 0 to 1 in Table 2 below. Consistent with our findings in the paper, the accuracy on SynBio remains near zero for r<0.3 and quickly improves as r grows from 0.3 to 1. We will include the additional results for $r>0.4$ in our next version.\n\n**Table 2: Accuracy (%) on SynBio for 70M Pythia models trained on the mixture of FineWeb-Edu and SynBio-320k.**  \n\n| Mixing Ratio $r$ | Acc. on SynBio (%) | \n|:----------------------:|:---------------:|\n| 0.1                 | 0.4           |\n| 0.15                 | 0.4          | \n| 0.2                 | 0.4          | \n| 0.25                 | 0.8          |\n| 0.3                  |8.7           |\n| 0.35                 |29.6          | \n| 0.4                 |40.1         | \n| 0.45                |  58.8        | \n| 0.5                 |   68.6     | \n| 0.6                 |   75.9   | \n| 0.7                 |     84.7     | \n| 0.8                 |     91.1 |\n| 0.9                 |       96.0   | \n| 1                   |       99.2     | \n\n\n---\n\nWe thank the reviewer again for the thoughtful and constructive feedback! We hope our responses have addressed all questions from the reviewer. If so, we kindly ask the reviewer to raise the rating. If the reviewer has any further questions, we will be more than happy to answer them."}, {"review_id": "TDN9KbzAvM", "reviewer": "Reviewer_gxsV", "summary": "This paper studies the dynamics of knowledge acquisition in large language model. Specifically, this work examines how things change with different mixtures of data. It aims to address what is learned from small amounts of knowledge rich data and large amounts of less rich data. The authors propose a scaling law that predicts phase transitions in the learning along the axis of data scaling.", "questions": "My questions are listed with my weaknesses above.\n\nI'm excited to engage with the authors to clear up the aspects I don't fully understand and I'm optimistic that with some iteration this paper can be made stronger.", "limitations": "I'd like to see a bit more discussion on the limitations of experiments. The very controlled setting here allows for nice experimentation, but the authors should better describe and discuss how these findings may or may not translate to real settings where data isn't necessarily broken up so nicely into general data and specific data. Can the authors further explain which of their findings might relate to real world training regimes?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I like the framing and the angle of this paper. I think it addresses a timely and important question. I do have some concerns about clarity and experimental design.\n\nStrengths:\n- This work touches on an important topic and makes a reasonable contribution toward trying to deepen our understanding of knowledge acquisition in large language models.\n- To my knowledge, this work is novel and original. The controlled experiments on data mixture and the isolation of knowledge-dense data from more general data is interesting.\n- The writing is generally good, although I do have some clarity questions below.\n\nWeaknesses:\n- Clarity\n  - The words \"memorized\" and \"learned\" are used in confusing ways in this paper. Can the authors clarify how these things are measured and whether they mean the same thing or not?\n  - In the summary of Finding 1 on the second page, the authors write \"... the model suddenly memorizes most biographies. Can the authors give a little more description of what \"suddenly\" means and what an alternative would look like? \n- Formatting\n  - The margins over the figures on pages 2, 4, 5, 7, and 8 are smaller than the formatting rule allow. Please fix this.\n- Experimental Design\n  - In Section 2, the training setup describes multiple epochs of some data and only one epoch of the web data. Can the authors address how repeating data may be at play here? Is that a confounder? Is there a possibility that repeating the general data or not repeating the biography data would change the results?\n  - Synthetic data poses a potential problem here. This introduces some distillation potential if the synthetic data is generated with an LLM.\n  - The Compact Knowledge Mixing is not so clear to me. Can the authors make a bit more clear how the frequencies are changed? Is the data compressed to make it fewer tokens and thus more frequent in the training set? Adding data but keeping the ratio fixed is confusing to me.", "was_revised": true, "final_justification": "The rebuttal was thorough and convincing. This is a good paper and I recommend it be accepted.", "rebuttal": "We sincerely thank the reviewer for the positive and encouraging feedback. We are glad that the reviewer appreciates the significance and originality of this paper. Below, we address the reviewer's questions and concerns.\n\n---\n**Clarity Concern 1:** The words \"memorized\" and \"learned\" are used in confusing ways in this paper. Can the authors clarify how these things are measured and whether they mean the same thing or not?\n\n**A:** This paper mainly studies factual knowledge acquisition, where the model is tasked with recalling the facts seen during training. We use the words \"memorized\" and \"learned \"interchangeably to refer to this factual recall ability.  We will clarify this terminology in the next version to avoid confusion.\n\n---\n\n**Clarity Concern 2:** In the summary of Finding 1 on the second page, the authors write \"... the model suddenly memorizes most biographies. Can the authors give a little more description of what \"suddenly\" means and what an alternative would look like?\n\n**A:** Thank you for this thoughtful question. Below, we provide a clearer contrast between the phase transition behavior we describe and a more gradual, linear trend.\n\nLet $x$ be the independent variable (e.g., model size or mixing ratio), and $y \\in [0, 1]$ be the dependent variable (e.g., accuracy).\n\n* **Phase transition:** We say that $y$ exhibits a phase transition with respect to $x$ if there exists a threshold $x_\\mathrm{thres}$ and a small constant $\\delta$ such that for $x < x_\\mathrm{thres}$, $y$ remains close to zero (i.e., below $\\delta$) with nearly no improvement. However, once $x$ exceeds the threshold $x_\\mathrm{thres}$, $y$ increases sharply within a narrow range of $x$. **This sharp, non-linear change is what we refer to as \"sudden\"—the model transitions from recalling almost nothing to recalling most facts in just a small increase of $x$**.\n\n* **Alternative: a linear trend:** In contrast, if $y$ increases gradually and roughly proportionally with $x$ before saturating at 1, we consider that a linear or smooth development. The curve would resemble a steady incline, without abrupt jumps in performance.\n\n\nTo illustrate this contrast, we present the exact data points in Fiugure 2 (b) and a hypothetical linear counterpart below:\n\n\n**Table 1 (a): Acc. on SynBio-1.28M for training runs in Figure 2 (b). Note: 0.4% accuracy $\\approx$ random guessing.**\n| Mixing ratio $r$ | Acc. (%)|\n|:------------:|:-------:|\n|0             |  0.4     |\n|0.1             | 0.4     |\n|0.2             |  0.4     |\n|0.3             |0.4     |\n|0.4             | 80.4    |\n\n\n\n\n**Table 1 (b): A hypothetical linear counterpart to Figure 2(b).**\n\n| Mixing ratio $r$ | Acc. (%)|\n|:----------:|:-------:|\n|0                | 0.4    |\n|0.1             | 19.5   |\n|0.2             | 42.3   |\n|0.3             | 57.9    |\n|0.4             | 80.4    |\n\nThe steady progression in Table 1 (b) differs significantly from the actual results in Figure 2(b), where accuracy stays near zero for $r\\leq 0.3$ and then jumps abruptly to 80.4% for $r=0.4$.\n\nWe will revise our wording in the paper to make the use of \"suddenly\" more precise in the next version.\n\n---\n\n**Formatting Concern:** The margins over the figures on pages 2, 4, 5, 7, and 8 are smaller than the formatting rule allow. Please fix this.\n\n**A:** Sorry about that. We will revise the formatting and move many of the details of Section 6 to the Appendix.\n\n\n---\n\n**Experimental Design Concern 1:** In Section 2, the training setup describes multiple epochs of some data and only one epoch of the web data. Can the authors address how repeating data may be at play here? Is that a confounder? Is there a possibility that repeating the general data or not repeating the biography data would change the results?\n\n**A:**  The choice to train on SynBio for multiple epochs while using only one epoch of web data is intended to **better reflect practical training practices**:\n\n- Web data is abundant, typically comprising trillions of tokens, so a single pass is generally sufficient.\n\n- Knowledge-dense data is much smaller. For instance, Wikipedia contains only a few billion tokens. Mdels often struggle to absorb all the knowledge in a single pass [1, 2]. As a result, it is now common to rephrase such data in to diverse forms to expose the model to the same knowledge multiple times [2, 3]. \n\n- Similar to the rephrasing practice, the model does not see exact repetitions of SynBio biographies. Instead, each time a biography is sampled, its five sentences are randomly shuffled, and new sentence templates are sampled on-the-fly for each attribute (as described in Section 2). If we do not expose the model to the biographies multiple times, it fails to memorize any of them.\n\n\n\n\n\n\n[1] Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models.\n\n[2] Kimi Team. Kimi K2: Open Agentic Intelligence.\n\n[3] Xintong Hao, Ke Shen, and Chenggang Li. MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus Expansion.\n\n---\n\n**Experimental Concern 2:** Synthetic data poses a potential problem here. This introduces some distillation potential if the synthetic data is generated with an LLM.\n\n**A:** As described in Section 2, our synthetic biographies are generated by (1) randomly sampling a value for each attribute of a person, and (2) converting each (name, attribute, value) triple into natural text using sentence templates listed in Table 6. While these templates were created via prompting GPT-4o and then manually polished, they are **simple surface-level rephrasings** and **do not introduce any additional knowledge from GPT-4o**. Therefore, **no distillation occurs**. The core challenge lies in learning the correct mapping from (name, attribute) to value, not in modeling or memorizing the template text.\n\n\n---\n\n**Experimental Concern 3:** The Compact Knowledge Mixing is not so clear to me. Can the authors make a bit more clear how the frequencies are changed? Is the data compressed to make it fewer tokens and thus more frequent in the training set? Adding data but keeping the ratio fixed is confusing to me.\n\n\n\n\n**A:** Yes, CKM increases the frequency of each fact by reducing its representation length. In Figure 7(b), $\\tau = 0.1$, $0.3$, and $0.6$ correspond to roughly 2×, 3×, and 4× increases in fact frequency, respectively.\n\nThe “adding data but keeping the ratio fixed” setup is implemented as follows: A mixing ratio \\$r\\$ means that \\$r\\$ fraction of training tokens comes from WikiBio. When we apply CKM, we augment WikiBio by adding compact tuple representations. To maintain the same token budget for WikiBio after augmentation, we reduce the number of epochs over it proportionally. Although the model sees fewer total passes over the dataset, each individual fact appears more frequently per epoch, thereby increasing its exposure.\n\nWe will clarify how CKM changes fact frequency and explicitly describe this implementation detail in the next version.\n\n\n---\n\n**More discussions on limitations:** I'd like to see a bit more discussion on the limitations of experiments. The very controlled setting here allows for nice experimentation, but the authors should better describe and discuss how these findings may or may not translate to real settings where data isn't necessarily broken up so nicely into general data and specific data. Can the authors further explain which of their findings might relate to real-world training regimes?\n\n**A:** We agree that real-world data is not always cleanly separable into “general” and “specific” categories. However, our findings still apply at the level of individual knowledge pieces. That is, a specific fact or reasoning procedure may not be acquired at all if its frequency or the model size falls below a threshold. This core insight generalizes to real-world training scenarios, which is supported by the following evidence: \n\n- Theoretical evidence: Section 4 derives a threshold frequency for the acquisition of each individual fact.\n\n- Empirical evidence 1: Figures 6(c) and 10 show that our predicted power-law threshold holds for real-world Wikipedia facts.\n\n- Empirical evidence 2: Section 3.3 and additional experiments (see response to Reviewer rq2w's Q1) demonstrate that reasoning procedures—such as slope calculation and max-over-n—can also exhibit sharp transitions similar to factual knowledge.\n\nWe appreciate the reviewer’s insightful question and will incorporate this discussion into the next version to better clarify how our findings extend to practical training regimes.\n\n---\n\nWe thank the reviewer again for the thoughtful and constructive feedback! We hope our responses have addressed all questions from the reviewer. If so, we kindly ask the reviewer to raise the rating. If the reviewer has any further questions, we will be more than happy to answer them."}, {"review_id": "h3c49A9VOD", "reviewer": "Reviewer_rq2w", "summary": "This paper investigates how mixing knowledge-dense data with large-scale web data during LLM pre-training can cause *phase transitions* in knowledge acquisition. Using synthetic and real-world datasets, the authors show that knowledge retention does not always scale smoothly with model size or data mix ratio: instead, there are critical thresholds below which models memorize almost nothing, and above which they suddenly acquire much more knowledge. They develop an information-theoretic framework to explain this as a capacity allocation problem, and they validate it with empirical results and scaling laws. The paper also proposes practical strategies—like random subsampling and compact rephrasing—to improve knowledge retention under low mixing ratios.", "questions": "1. Based on Weakness 2, could you design a task that better aligns with the definition of reasoning to more effectively support your claims?\n2. Although your experimental setup follows prior work, I think this setting—consisting of only a single entity and its associated attributes—is insufficient to verify whether your conclusions hold in scenarios involving multiple entities and more complex relationships. Could you consider supplementing your study with some simple experiments under a more complex setting?\n3. The experiments in Section 6 appear to be conducted only on a 410M parameter model, which is not commonly used in current real-world scenarios. Could you reproduce this part of the study on a model with approximately 7B parameters to better demonstrate the generalizability of your applications?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper demonstrates a substantial amount of work, including comprehensive experiments, thorough theoretical analysis, and corresponding application designs.\n2. The work presents many interesting findings, which have clear implications for data curation and training strategies for large language models at different scales.\n\n### Weaknesses\n\n1. The spacing between section titles and the main text is too narrow, which does not conform to the standard template. As a result, the overall content appears overly dense, negatively affecting readability.\n2. I believe the design of the Reasoning Tasks in Section 3.3 is questionable. Although the task appears to assess a slope-related problem involving mathematical reasoning, the evaluation instances frequently appear in the training data. As a result, the model can rely purely on memorization to answer the slope questions correctly, rather than genuinely computing the slope through intermediate reasoning steps. Therefore, in my opinion, this task design does not align with the fundamental definition of reasoning and remains within the scope of factual memorization.", "was_revised": true, "final_justification": "This paper presents extensive theoretical and empirical work revealing phase transitions in LLMs’ knowledge acquisition when mixing data types, offering practical insights and strategies for effective pre-training at scale. During rebuttals, the authors have addressed my main concerns by providing results on more complex tasks and additional models. Therefore, I have raised my score from 4 to 5.", "rebuttal": "We sincerely thank the reviewer for the positive feedback and for acknowledging the thoroughness of our work, the interesting findings, and the clear implications for data curation and LLM training strategies. Below, we address the reviewer's questions and concerns.\n\n\n---\n\n**W1:** The spacing between section titles and the main text is too narrow.\n\n**A:** Sorry about that. We will revise the formatting to improve readability and move many of the details in Section 6 to the appendix.\n\n---\n\n**W2+Q1:** I believe the design of the Reasoning Tasks in Section 3.3 is questionable. The model can rely purely on memorization to answer the slope questions correctly, rather than genuinely computing the slope through intermediate reasoning steps. \n\n**A:** **We respectfully disagree with the concern that the model could rely purely on memorization to solve the slope questions.** The total number of possible slope calculation problems in our setup is $100 \\times 100 \\times 99 \\times 100 = 9.9 \\times 10^7$. In contrast, a typical training run in Figure 5(b) with $r = 0.4$ sees fewer than $3.5 \\times 10^6$ unique slope examples, less than 5% of the full space. Despite this limited coverage, the model still achieves 60% accuracy at test time, where examples are uniformly sampled from the full distribution. \n\nThis substantial generalization beyond the training data suggests that **the model is indeed learning a generalizable computation procedure, rather than memorizing specific input-output pairs.** We will clarify this point in the next version by explicitly reporting the size of the problem space and the number of unique training samples to avoid potential confusion.\n\n---\n\n**Q1:** Could you design a task that better aligns with the definition of reasoning to support your claims more effectively?\n\n**A:** Certainly! We design a new task named \"max over n numbers\". Compared to the slope calculation task, this task has a vastly larger input space, making memorization infeasible.\n\nSpecifically, the model is asked to find the maximum number given a list of $n=30$ numbers, each randomly sampled from $\\{0, 1, ..., 99\\}$. The training samples are formatted as follows:\n\n> Q: Find the maximum value of the following list:\n>\n> [47, 83, 38, ...]\n>\n> Think step-by-step.\n>\n> A: \n>\n> Compare 47 and 83. Keep 83.\n>\n> Compare 83 and 38. Keep 83.\n>\n> ...\n>\n> Final answer: 83. \n\nFollowing the setup in Section 3.3, we add 3M tokens of such examples to the OpenWebMath dataset (accounting for less than 0.02% of the original OpenWebMath token count) to create a modified version. We then train Pythia models on a mixture of FineWeb-Edu and the modified OpenWebMath dataset, with $r$ denoting the mixing ratio of the modified OpenWebMath.\n\nFor evaluation, we generate 1,000 test samples of \"max over 30 numbers\" and assess whether the model outputs the correct final answer.\n\nAs shown in Table 1: \n\n1. The 70M model achieves a test accuracy of 60% with fewer than 3,500 unique training examples (at $r = 0.5$), a negligible fraction of the total sample space, which has size $10^{60}$. This clearly demonstrates generalization beyond memorization.\n\n2. We observe similar phase transitions with respect to both model size and mixing ratio, consistent with our main findings.\n\n**Table 1: We observe similar phase transitions in mixing ratio $r$ and model size for the \"max over 30 numbers\" task.**\n\n**Table 1 (a)：Phase transition in $r$ on 70M Pythia models.**\n\n|Mixing ratio $r$ | Acc. on \"max over 30 numbers\" (%)|\n|:-----------------: | :----------------------:  | \n|0.1                | 0.0                       |\n|0.2                | 0.0                      |\n|0.3                | 0.5                       |\n|0.4                | 34.6                       |\n|0.5                | 61.8                       |\n\n\n\n**Table 1 (b)：Phase transition in model size at $r$=0.2.**\n\n|Model size (M) | Acc. on \"max over 30 numbers\" (%)|\n|:-----------------:  | :----------------------:  | \n|31                | 0.0                       |\n|50                | 0.0                      |\n|70                | 0.0                       |\n|95               | 0.1                     |\n|120                | 0.1                       |\n|160                | 2.6                       |\n|256                | 30.5                      |\n|410               | 87.0                    |\n\n\n\n\n\n\n\n---\n\n**Q2:** The experiment setting is insufficient to verify whether your conclusions hold in scenarios involving multiple entities and more complex relationships. Could you consider supplementing your study with some simple experiments under a more complex setting?\n\n\n**A:** We believe the tasks used in our experiments—slope calculation and \"max over n numbers\"—already involve multiple entities and require learning complex relationships among them. Specifically, each number can be viewed as an individual entity. The \"max over n numbers\" task requires the model to learn pairwise comparisons and reason over a sequence of such relations to identify the maximum. The slope calculation task involves both subtraction and division, which represent structured relationships between three numerical entities. For example, subtraction captures the relationship among the minuend, subtrahend, and difference. These tasks go beyond simple recall and reflect nontrivial multi-step reasoning.\n\n\n\n\n---\n\n**Q3:** The experiments in Section 6 appear to be conducted only on a 410M parameter model, which is not commonly used in current real-world scenarios. Could you reproduce this part of the study on a model with approximately 7B parameters?\n\n\n**A**: Sure! We conduct additional experiments on Pythia 6.9B to show the effectiveness of random subsampling on larger models. Specifically, we follow the continual pre-training setup in Figure 8(b) and apply random subsampling at mixing ratio $r=0.2$. Let  $\\rho$ denote the subsampling ratio. As shown in Table 2, setting $\\rho$ as 0.5 and 0.25 significantly improves the model's accuracy on SynBio from 0.4% to 19.5% and 17.5% respectively. This demonstrates that random subsampling remains highly effective even at the 7B scale.\n\n**Table 2: Random subsampling enables the 6.9B model to learn significantly more from knowledge-dense data.**\n\n| Subsampling ratio $\\rho$ | Acc. on SynBio-10k (%) | \n|:----------------------:|:---------------:|\n| 1 (no subsampling)                | 0.4           |\n| 0.5                | 19.5          | \n| 0.25                 | 17.9          | \n\n\n---\n\nWe thank the reviewer again for the thoughtful and constructive feedback! We hope our responses have fully addressed the reviewer's concerns. If so, we kindly ask the reviewer to raise the rating. If the reviewer has any further questions, we will be more than happy to answer them."}, {"review_id": "X5x7Hjz1bz", "reviewer": "Reviewer_DKRZ", "summary": "In this paper, authors investigate how mixing small amounts of high-quality data such as synthetic biographies with web-scraping data affects knowledge acquisition. Through controlled experiments. authors discover two types of phase transitions: along model size an along mixing ratio. The authors then attribute these sharp transitions to a capacity allocation phenomenon. They also formalize their intuition to derive a power-law relationship that accurately predicts the critical thresholds.", "questions": "1. Authors claim they observe \"phase\" transitions across model scale and data ratio. But what qualifies as a phase transition versus a linear development? Without a frame of reference, how do we know the observe performance change is sharp and considers as a phase transition?", "limitations": "No, authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. Data mixing during pretraining is a critical and under-explored topic, and this paper addresses it using a highly controlled and well-designed experimental setup. By systematically varying the model size and the ratio of knowledge-dense to web data, the authors reveal two sharp phase transitions in knowledge acquisition. The clarity of these findings, and the effective presentation of results (e.g., Figures 1–3), make the empirical contribution compelling.\n\n\n2. In addition to analysis, authors also propose two actionable and well-motivated strategies to improve knowledge retention task. Overall, this paper has a complete analysis of theory, empirical findings and practical insights. \n\n\n**Weakness**\n\n1. Prior work (e.g., [1], [2]) has shown that phase transitions and emergent behaviors in LLMs can be artifacts of (a) metric thresholding and (b) sensitivity to random initialization. However, this paper does not adequately rule out those possibilities. For instance, all key evaluations use discrete accuracy measures (e.g., exact match of factual answers). It would strengthen the case considerably if the authors provided results using more continuous metrics (e.g., token-level F1, log-likelihood) and ran models with multiple random seeds to test robustness against training noise.\n\n\n2. Figures 1, 2, and 5 use a mix of linear and log scales on the y-axis—sometimes for the same metric (accuracy) across different plots. This inconsistency is not explained and raises the concern that log scaling may be exaggerating the sharpness of transitions. A clearer and more principled explanation of axis choices would help ensure the visualizations do not unintentionally bias interpretation.\n\n\n\n3. One of the motivations for data mixing is to enable multitask performance (e.g., reasoning, coding, factual recall). However, in this paper, the “web” data serves more as a placeholder than a concrete \"competing\" task (i.e., learning synthetic biography versus math). As a result, the paper does not deeply analyze trade-offs between capabilities caused by data mixes—for example, whether increasing knowledge-dense data harms performance on reasoning.  \n\n\n4. Although Section 6 introduces two mitigation strategies, both seem most applicable to factoid-style datasets like synthetic biographies, where each data point is short, modular, and isolated. It is less clear how these strategies would extend to richer forms of knowledge or reasoning, where facts are embedded in longer contexts or interconnected. This limits the generality of the proposed solutions.\n\n\n\n[1] Are Emergent Abilities of Large Language Models a Mirage?   http://arxiv.org/abs/2304.15004\n\n\n[2] Distributional scaling for emergent capabilities. https://arxiv.org/abs/2502.17356", "was_revised": true, "final_justification": "Reviewers have thoroughly addressed my concerns during rebuttal and I think overall the study provide a comprehensive analysis on scaling and data mixing.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and for acknowledging that our paper presents a well-controlled study on an important and under-explored topic. We are glad that the reviewer found our empirical contributions compelling and our proposed strategies to be well-motivated and actionable. Below, we address the reviewer's concerns and questions.\n\n---\n**W1:** Prior work has shown that phase transitions and emergent behaviors in LLMs can be artifacts of ... It would strengthen the case considerably if the authors provided results using more continuous metrics (e.g., token-level F1, log-likelihood) and ran models with multiple random seeds to test robustness against training noise.\n\n**A:** 1. **Phase transitions in validation loss**: In addition to accuracy, **we do observe phase transitions in SynBio validation loss**, a continuous metric. As shown in Table 1 (a) below (corresponding to $r=0.1$ in Figure 1), the loss remains nearly flat as the model size increases from 14M to 160M, decreasing by only 0.03. However, a sharp drop of 0.45 occurs between 256M and 410M, aligning with the sudden jump in accuracy from 0.9% to 80.3%. Similar phase transitions in validation loss can be observed in Table 1(b) (from Figure 2(b)). These trends confirm that **the observed phase transition is not merely an artifact of accuracy thresholding, but also reflected in smooth metrics like loss**. \n\n**Table 1 (a) Validation loss and accuracy on SynBio-320k for training runs in Figure 1 with $r=0.1$. Note: 0.4% accuracy $\\approx$ random guessing.**\n\n| Model size (M)| Val. loss on SynBio-320k | Acc (%)|\n|:---:|:---:|:---:|\n|14 | 0.86    |0.4      |\n|31  | 0.86  | 0.4     |\n|70 | 0.85   | 0.4     |\n|120 | 0.85 | 0.4     |\n|160  | 0.83   | 0.4     |\n|256  | 0.78 | 0.9     |\n|410   | 0.33   | 80.3     |\n\n**Table 1 (b) Validation loss and accuracy on SynBio-1.28M for training runs in Figure 2 (b). Note: 0.4% accuracy $\\approx$ random guessing.**\n| Mixing ratio $r$ | Val. loss on SynBio-1.28M| Acc (%)|\n|:------------:|:--:|:-------:|\n|0 | 0.86 | 0.4     |\n|0.1 | 0.86 | 0.4     |\n|0.2  | 0.85  | 0.4     |\n|0.3 | 0.85 | 0.4     |\n|0.4   | 0.33| 80.4    |\n\n\n2. **Robustness to training noise**. We replicate the experiments in Figure 2 (a) with two additional random seeds and report the mean and standard deviation below. While accuracy varies slightly with random seeds, the phase transition behavior remains consistent and clearly observable across runs. \n\n**Table 2: Mean and standard deviations over 3 random seeds for runs in Figure 2 (a).**\n\n| Mixing ratio $r$| Mean acc (%)|Std. dev. (%)|\n|:------------:|:-------:|:-------:|\n|0.1           |0.4      |0.0|\n|0.15          | 0.4     |0.0|\n|0.2           |0.4      |0.0|\n|0.25          |0.8      |0.0|\n|0.3           |7.3      |2.8|\n|0.35          | 27.5     |2.8|\n|0.4           |40.4    |3.1|\n|0.45          |58.1    |3.1|\n\nWe will include validation loss results and report experiments with multiple random seeds in the next version to strengthen our findings.\n\n---\n**W2:** Figs. 1, 2, 5 use a mix of linear and log scales on the y-axis—sometimes for the same metric (acc.) across different plots... A clearer and more principled explanation of axis choices would help ensure the visualizations do not unintentionally bias interpretation.\n\n**A:** Our principle for axis choices in Figures 1, 2, and 5 is to **match the scaling of the x- and y-axes, in order to avoid distorting trends**. \n\n- For phase transitions with respect to mixing ratio $r$, we use a linear x-axis, since $r$ is a tunable parameter with uniform meaning across its range. Accordingly, we keep the y-axis linear.\n\n- For phase transitions with respect to model size, we use a logarithmic x-axis, as model size typically increases exponentially in practice (e.g., via width or depth). To avoid misleading exponential trends due to this scaling, we also apply a log scale on the y-axis, so linear accuracy growth does not appear exaggerated. This ensures that the sharp transition we observe is not an artifact of axis distortion.\n\nAs additional support, we provide the exact data points for $r = 0.2$ from Figure 1 below. The reviewer is welcome to replot the curve and will find that the sharp transition remains clearly visible even when changing the y-axis from log scale to linear. If plotting with a log scale, we recommend using a symmetric log scale with a linear threshold of 1, to avoid exaggeration near zero.\n\n**Table 3: Data points for $r=0.2$ in Figure 1**\n\n|Model size (M) |Acc. (%)|\n|:-------------:|:------:|\n|   14   |    0.4    |\n|   31 |     0.4   |\n|   70 |     0.4   |\n|    95  |      0.5  |\n|     120  |      5.2  |\n|     160 |      82.1  |\n|     256|       93.1 |\n|     410   |       97.6 |\n\nWe will include a clear explanation of our axis scaling choices in our next version to avoid potential confusion.\n\n---\n\n**W3**: One of the motivations for data mixing is to enable multitask performance (e.g., reasoning, coding, factual recall) ... As a result, the paper does not deeply analyze trade-offs between capabilities caused by data mixes, for example, whether increasing knowledge-dense data harms performance on reasoning.\n\n**A:**  We agree that multitask learning is a key motivation for data mixing, and that different capabilities may compete for model capacity. In fact, **Sec. 6 of our paper explicitly discusses this trade-off.**\n\n- As stated at the beginning of Sec. 6, our motivation for proposing the two strategies is to mitigate this dilemma: a small $r$ limits learning from knowledge-dense data, while a large $r$ may degrade the general capabilities acquired from web-scale data (e.g., commonsense reasoning, text understanding). Our strategies aim to enhance knowledge acquisition even at small $r$, thereby preserving overall model capability.\n\n- All experiments in Section 6 not only report the accuracy on knowledge-dense data, but also the validation loss on web data and downstream performance (averaged over five benchmarks) to assess general capabilities. For example, Figure 7 (b) and Tables 3 and 4 show that\n    -  Without our strategies, increasing $r$ worsens validation loss on Pile and degrades downstream performance.\n\n    - Applying either strategy enables the model to acquire significantly more factual knowledge at a small mixing ratio ($r = 0.1$) while maintaining Pile loss and downstream task performance.\n\nIn our next version, we will better highlight this tradeoff between knowledge acquisition and general capabilities and clarify how our strategies help mitigate it.\n\n---\n**W4:** It is less clear how these strategies (in Sec. 6) would extend to richer forms of knowledge or reasoning, where facts are embedded in longer contexts or interconnected. This limits the generality of the proposed solutions.\n\n**A:** We thank the reviewer for pointing out the important direction to test our mitigation strategies on richer forms of knowledge or reasoning. However, we would like to clarify the intended scope and generality of our contributions:\n\n1. Our main contributions are to discover and formulate the phase transition in knowledge acquisition due to data mixing, along with theoretical analyses.\n\n2. The two mitigation strategies in Section 6 are practical implications of our theory: by increasing the marginal value (i.e., cost-effectiveness) of the knowledge we care about, we can significantly improve learning outcomes. While these strategies are demonstrated on factoid-style data, the underlying principle is broadly applicable.\n\n3. For example, random subsampling could be adapted to other domains by selectively reducing the volume of less relevant data. CKM) could be extended by prompting LLMs to rephrase or compress richer knowledge into more concise formats.\n\nTherefore, we leave it as future work to explore the extensions of our mitigation strategies to more complex tasks.\n\n---\n\n**Q1:** What qualifies as a phase transition versus a linear development? Without a frame of reference, how do we know the observed performance change is sharp and considered a phase transition?\n\n**A:** Thank you for this thoughtful question. Let $x$ be the independent variable (e.g., model size or mixing ratio), and $y \\in [0, 1]$ be the dependent variable (e.g., accuracy).\n\n- **Phase transition:** As defined in our abstract and introduction, we consider $y$ to exhibit a phase transition with respect to $x$ if there exists a threshold $x_\\mathrm{thres}$ and a small constant $\\delta$ such that for $x < x_\\mathrm{thres}$, $y$ remains below $\\delta$ with nearly no improvement, but for $x > x_\\mathrm{thres}$, $y$ rises rapidly. \n\n- **Linear development:** In contrast, if $y$ increases gradually and proportionally with $x$ before saturating at 1, we consider it a linear trend. The curve would appear as a steady incline without abrupt jumps.\n\nTo illustrate this contrast, below is an example of what a linear counterpart to Figure 2(b) might look like:\n\n**Table 4: A hypothetical linear counterpart to Figure 2(b).**\n\n| Mixing ratio $r$ | Acc (%)|\n|:-----:|:-----:|\n|0   | 0.4    |\n|0.1 | 19.5   |\n|0.2   | 42.3   |\n|0.3 | 57.9    |\n|0.4 | 80.4    |\n\nWe will revise the paper to explicitly clarify the definition of phase transitions, contrast them with linear trends, and explain how our empirical results (e.g., in Figure 2(b)) fit this definition.\n\n---\n**Response to format concerns:** We are sorry about that. We will revise the format and move many of the details in Section 6 to the appendix.\n\n---\nWe thank the reviewer again for the thoughtful and constructive feedback! We hope our responses have addressed all questions from the reviewer. If so, we kindly ask the reviewer to raise the rating. If the reviewer has any further questions, we will be more than happy to answer them."}]

Author rebuttal:

Meta-review: The paper studies how mixing between different types of data (knowledge-dense expert data vs web scrape) results for pretraining data results in transitions in scaling laws, and connect this transition to memorization. They also propose an information-theoretic model to predict these transitions.

Overall, the reviewers found the results novel and significant. The controlled experimental setup was highlighted as a strength, providing clear and well-presented results. Some minor issues around sensitivity to init, robustness of phase transition claims under a different loss, and clarity of definitions were addressed during the rebuttal phase.
Decision: Accept (spotlight)