Learning Dynamics in Continual Pre-Training for Large Language Models
Authors: Xingjin Wang
Howe Tissue
Lu Wang
Linjing Li
Daniel Dajun Zeng
Venue: ICML 2025 oral (2025)
Keywords: Continual Pre-Training
Large Language Models
Learning Dynamics
Primary area: deep_learning->large_language_models

Abstract: Continual Pre-Training (CPT) has become a popular and effective method to apply strong foundation models to specific downstream tasks. In this work, we explore the **learning dynamics** throughout the CPT process for large language models (LLMs). 
We specifically focus on how general and downstream domain performance evolves at each training step, with domain performance measured via validation losses. 
We have observed that the CPT loss curve fundamentally characterizes the transition from one curve to another hidden curve, and could be described by decoupling the effects of distribution shift and learning rate (LR) annealing. 
We derive a CPT scaling law that combines the two factors, enabling the prediction of loss at any (continual) training steps and across learning rate schedules (LRS) in CPT. 
Our formulation presents a comprehensive understanding of several critical factors in CPT, including the learning rate, the training steps, and the distribution distance between PT and CPT datasets.
Moreover, our approach can be adapted to customize training hyper-parameters to different CPT goals such as balancing general and domain-specific performance.
Extensive experiments demonstrate that our scaling law holds across various CPT datasets and training hyper-parameters.

Learning Dynamics in Continual Pre- Pre-Training for Large Language Models
Xingjin Wang" 12 Howe Tissue X
Lu Wang
Linjing Li Daniel Dajun Zeng' 1.2
Abstract
Continual Pre-Training (CPT) of LLMs aims enhance their abilities in specific downstream domains (c.g. coding, finance, math) while mitigating the substantial costs asso- ciated with re-training (Chen et al., 2023a; Çağatay Yıldız et al., 2024; Ibrahim et al., 2024).
Continual Pre- Tre-Trann (CPT) is a popular and effective method for applying strong foundation models to specific downstream tasks. In this work, we explore the learning dynamics through- out the CPT process for large language models. We specifically focus on how general and down- stream domain performance evolves at each train- ing step, with performance measured by valida- tion losses. We observe that the CPT loss curve fundamentally characterizes a transition from an initial pre-training trajectory to a new, domain- specific one, conceptualized as a shift between two hidden loss curves. This transition can be described by decoupling the effects of distribu- tion shift and learning rate annealing. We derive CPT scaling law that combines these two fac- tors, enabling the prediction of loss at any (contin- ual) training step and across various learning rate schedules. Our formulation presents a compre- hensive understanding of several critical factors in CPT, including loss potential, peak learning rate, training steps, and replay ratio. Moreover, our approach can be adapted to optimize training hyper-paramest for different CPT goals, such as balancing general and domain-specific perfor- mance. Extensive experiments demonstrate that our scaling law holds across various CPT datasets and hlype--araamteess
CPT primarily involves a trade-off between performance on general and downstream domains. It widely observed that improvements on downstream tasks may come the expense of degrading performance on general domain tasks, phenomenon known as catastrophic forgetting (French, 1999; Gupta et al., 2023). Recently, some scaling laws have been proposed for CPT scenarios. For example, Hernandez et al. (2021b) and Barnett (2024) discovered a law describ- ing how data transfer effectiveness scales with fine-tuning dataset size and model size. Que et al. (2024) and Gu et al. (2024) proposed law to find the optimal replay ratio to balance general and downstream performances.
However, very few studies have attempted to quantitatively describe the learning dynamics of CPT, particularly how performance varies on general and downstream domains throughout the CPT process. We have two primary research questions (RQs): (1) Can we derive an accurate law de- scribing the influence of as many variables as possible on the final CPT performance?
(2) Can we trace the perfor- mance of LLMs throughout the entire CPT process, rather than only the final performance? Studying the first RQ will help researchers investigate various factors that affect CPT performance and facilitate hyper- parametarssessss optimization through prediction; studying the second RQ will help the community understand the learning dynamics of LLMs at each step of the CPT process, providing deeper insights and theoretical guidance for subsequent CPT research.
1. Introduction
In recent years, large language models (LLMs) have ex- hibited versatile abilities and garnered significant academic and industrial attention (Dubey et al., 2024: OpenAI, 2023).
Following previous works (Gupta et al., 2023; Ibrahim et al., 2024; Que et al., 2024), we trace performance changes using validation losses on corresponding domains. We find that the CPT loss curve acts transfer curve and can be described by decoupling the effects of distribution shift and learning rate (LR) annealing. Specifically, the distribution shift between the pre- pre-training (PT) and CPT data leads to a deviation in the loss curve, while LR annealing results in a loss decrease both the PT and CPT phases. By analyzing various loss curves, we discover a CPT scaling law that integrates these two factors, enabling accurate prediction of
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing. China "State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Au- tomation, Chinese Academy of Sciences, Beijing, China 'Ritzz- AI. lwzzfzll 8 gma il com. Correspondence to: Howe Tissue (project lcad) sun20un seeeetsinnghhe singhua. org..rr.goo.... .n>>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Dynamics in Continual Pre-Training for Large Language Models
Learning Rate .-10
2.53
Validation LoGS
2.5
No Thriig with
14
1.53
25-
resir and
2.03
34
33-
123
Tacco Noet
Step
Step
(a) Constant PT and CPT LRS.
(b) Dpr (FineWeb) Validation Loss.
(c) Depe (Knowledge Pile) Validation Loss.
Learning Rete X100
2.0
Validatiin Loss
Thcdin By carannal tiving
1.75
Fro Thaing witn
1:35
64-
175
24
3.3
329
10030
20000
23006
43306
$3.00
55083
10003
20000
20300
49300
$9300
50300
24
10000
20043
20083
40003
20083
60643
Step
Step
Step
(d) WSD PT and CPT LRS.
(e) Dau (FineWeb) Validation Loss.
(1) Dept (Knowledge Pile) Validation Loss.
Figure 1. CPT loss curves under different learning rate schedules (LRS): constant (a-c) and warmup-stable-deeas (WSD) (Hu et al., 2024) (d-r). The CPT loss curve acts as transfer curve from the hidden PT curve trained on Dpt (Blue dashed) to the hidden PT curve trained on Dent (Orange dashed). The transfer curve converges to the hidden PT curve trained on Dept.
losses throughout the entire CPT phase.
Experimental Setup. Our main experiments employ LLaMA-like models (Dubey et al., 2024) with 106M to 1.7B non-embedding parameters. We use FineWeb (Penedo et al., 2024) as Dyt and Knowledge-Pile (Fei et al., 2024) as Dept- We leverage different LRS in the PT and CPT phases (see Fig. 1). More details are provided in Appendix B.
Our proposed scaling law provides a comprehensive model of how key variables affect the training dynamics of CPT, such as loss potential (defined in section 3.3), peak LR, training steps, and replay ratio. We demonstrate how these variables jointly affect model performance at each CPT step, and how to optimize these hyper-parameters for better CPT performance. By applying our scaling law, several valuable conclusions emerge. For example: (1) PT models with higher loss potential can better adapt to downstream domains in CPT: (2) The performance degradation on the PT domain during the CPT phase is inevitable if the turning length is infinitely large, which implies that the PT model is adequately trained or the distribution shift between the PT and CPT data is very large; (3) For specific CPT goals, like balancing performance between the PT and CPT domains, or optimizing out-of-domain performance, our scaling law such as the loss potential, peak LR, and PT dataset replay ratio.
Observation. As observed in previous studies (Ibrahim et al., 2024; Gupta et al., 2023), during the CPT process, the Dpt validation loss tends increase (Fig. 1b and Fig. le), whereas the Dept validation loss decreases (Fig. Ic and Fig. If). Morcover, in both PT and CPT phases, the loss curve is significantly influenced by the LRS. For example, the loss decreases rapidly when the LR anneals, which is observed our prior work (Tissue al., 2024).
2.2. CPT Transfer Loss Curve
can predict the optimal training
To enhance our understanding of the CPT training dynamics, we train two additional loss curves: the hidden PT curve trained on Dppt and the hidden PT curve trained on Dept-
2. Pilot Observation
2.1. Task Formulation
Hidden PT Curve trained on Dpm- This curve represents the loss when the model is consistently pre-trained using Dpt with the same LRS as used in the CPT phase.
We investigate the dynamics of performance in both general and downstream domains during the CPT process. Follow- ing previous works (Ibrahim et al., 2024; Que et al., 2024; Gu al., 2024; Hermandez al., 2021a), we assess model performance by examining the validation loss the PT dataset Dpr and the CPT dataset Dept-
Hidden PT Curve trained on Dept. This curve depicts the loss when the model is trained from scratch on D Dept. while adhering to the same training setups (such as LRS) as those applied in the PT and CPT phases.
2

---

Learning Dynamics in Continual Pre-Training for Large Language Models
4.0
Validation Loss
3.6
Validation Loss
3.8
Freer STEE -0.300 12-040 gP-4002
Distribution Shift
3.4
CPT Stop F1
3.2
0-1
3.2-
Oxtritttonn
3.0
Distribution Shift
3.0
0.1-
3
2.5
0.0
2.6
0
10000
20000
30000
40000
50000
80000
10000
20000
30000
40000
50000
60000
Step
Step
(a) p (eb) validation loss shift.
(b) Dept (Knowledge Pile) validation loss shift.
Figure 2. The transfer loss curve in Dpt and Dept validation sets for different transfer starting points with constant LRS. We find that the distribution shift term independent of the transfer starting points and adheres to a power-law form.
Transfer Curve. As shown in Fig. 1, the CPT loss curve acts as a transfer curve between these two hidden PT curves; i.e. the CPT loss deviates from the hidden PT curve trained on Drt and converges towards the hidden PT curve trained on D Dept. The discrepancy between the transfer loss curve and the hidden PT curve trained Dpt is called distribution shift. As the number of CPT steps approaches infinity, the CPT loss is expected to converge to the hidden PT curve trained on Dept
where / denotes the CPT step, and SP' (5%) and STP (5tgt) are the forward (annealing) areas at the PT and CPT stages, respectively.
3.2. Distribution Shift Term
The distribution shift term describes the deviations from the hidden PT curve trained on Dpt- This shift reflects the distributional distance between Dpt and Dept- Many studies (Ibrahim et al., 2024; Wang et al., 2024; Parmar et al., 2024) have highlighted the impact of LRS at the CPT stage, implying that this shift should be also affected by the LRS. We first analyze the form of the distribution shift term with a constant LR to isolate the effects of LRS, then we incorporate the forward area into the equation accurately describe the distribution shift term for different LRS.
Finding 1. The process of CPT is how the loss curve transitions from the hidden PT curve trained on Dpt to the hidden PT curve trained on Dept-
3. Continual Learning Dynamics Law
We quantitatively analyze the transfer curve by modeling the effects of LR annealing and distribution shift.
Constant LRS. We first use a constant LR in both PT and CPT phases. To study the relationship between distribution shift and the PT model state, we continually pre-train the model starting from different transfer points. As shown in Fig. 2, these distribution shift terms tend to overlap regard- less of the transfer starting point. This overlap suggests that the distribution shift term is independent of transfer starting points or PT model checkpoints.
3.1. LR Annealing
Without data transfer, the CPT loss curve would follow the trajectory of the hidden PT curve trained on Dpt- Tissue et al. (2024) introduced scaling law to describe the loss dynamics at each step / as affected by LR annealing:
We compare to fit the distribution shift term using expo- nential and plwer-law forms, and find the best fit to be AL(t) B (1 (E t 1)-9). We do not adopt the simple power-law form AL(!) B 1-8 to ensure that ( = O. We leverage this equation to fit the transfer loss curve of both Dp and Deps validation sets, as shown in Fig. 2.
T(t) Lo + A A.ST C. S2,
where the forward area S = Σi= 7i is the summed LR,
and the annealing area S2 Ei= Ei= (Mh-1 mT)
x is term affected by LR annealing. Lo, A, C, are
constant positive parameters to be fitted. À = 0.999 is a
hyper-parameter related to the momentum term.
The loss in the CPT process without distribution shift (de- noted Liase (1)) follows this law, i.e.,
Other LRS. When considering the effect of LRS, we find that the LR values, i.e., the forward area in Eq. 1, sig- nificantly affects the distribution shift term. The smaller forward area in the CPT results in a smaller distribution
I Lbase (1) = Top+ A-S - -0-----(5" +
SSP (2)
3

---

Learning Dynamics in Continual Pre- Pre-Training for Large Language Models
Learning Rate X110
2.53
Validation LoGS
All Cad bur Paih
14
Caliroll Sastin
1.50
3.5
2.0
64-
36-
23
2.
2-27
123
330ce
20000
Step
Step
(a) WSD PT and CPT L.R..
(b) Dpt (FineWeb) Validation Loss.
(c) Dept (Knowledge Pile) Validation Loss.
Learning Rate X100
2.0
Validation Loss
Selia Pho Tuciing Teck Lane
Dukes Carto taaie
1.7
Car Phating
Validation
1:35
1.5
25
24
0.3
3.0
10030
20000
23006
43006
$3.00
56083
33000
20000
20066
40303
33300
6300
25
10493
20023
33303
40000
sedce
60083
Step
Step
Step
(d) Cosine PT and CPT LRS.
(e) Dps (FineWeb) Validation Loss.
(1) Dept (Knowledge Pile) Validation Loss.
Figure 3. Using Eq. to fit all PT and CPT loss curves with different LRS (WSD and Cosine). For Dyt validation sets, all loss curves (b and e) are described by the same equation: similarly, for Dept validation sets, all loss curves (c and f) follow the same equation.
shift, as shown in different transfer curves in Fig. lb vs. le
across different LRS throughout the training process. We also use the fitted equation to predict loss curves of other LRS, and the prediction accurately matches the observation (see Fig. 10). Furthermore, the batch size and sequence length may change in the CPT phase. However, scaling law equation remains adaptable to these hyper-parametce changes, as demonstrated in Appendix F.
(or Fig. Ic vs. If). Hence, following Tissue et al. (2024), we replace the training steps with the forward area Sp in the CPT phase:
AT() B . (1- (1 + F (1+
(3)
which instead adopts STPt to represent the training amount in CPT stage, considering the impact of LR values.
Finding 2. The CPT loss curve can be decomposed into a hidden PT curve trained on Dpt and a distribu- tion shift term. The hidden PT curve trained on Dp is formalized as a scaling law with LR annealing, whereas the distribution shift term is independent of transfer starting points and adheres power-law form.
3.3. Final Transfer Curve
We combine the effect of LR annealing (Eq. 2) and distribu- tion shift (Eq. 3) to get the equation for the CPT loss:
(tt) = Lbase (t) + At
= Lo + A. (spt S -C1 Sh C2- S
Scaling law with LR annealing
+ B. (1 (1- (1-
E E- Spr)
Transfer Loss Surface. To better understand our formula- tion, follow Tissue al. (2024) to view the loss surface of LLMs as a slide-like transition between surfaces in Fig. 4. The CPT process transitions from one surface to another following power-law form. A larger distributional dis- tance between Dpe and Dept leads to a steeper slope of the transfer surface, and thus a sharper increase in the D,r loss. When the LR anneals, the amplitude of the oscillation on the loss surface decreases, and thus the loss also decreases. In the annealing view, term the "height" of the current model state as its loss potential. We use this concept to capture the potential for future loss drop via LR annealing. Quantitatively, we can define loss potential as the ratio of the final annealed LR of the PT phase to the peak learning rate in the PT phase.
Power-law distribution shift
We adopt different coefficients C1 and C2 for Spd and Sept because the distributions of Dpe and Dept are different, and thus result in different annealing effects.
Our equation can predict the loss at any step with any LRS during both the PT and CPT phases. We conduct experi- ments utilizing the widely adopted WSD (Hu et al., 2024) and cosine (Loshchilov & Hutter, 2016) LRS in the PT and CPT phases (see Fig. 3a and Fig. 3d). We use Eq. 4 to fit all loss curves on the D,m and Dept validation sets. As illustrated in the middle and right panels of Fig. 3, our equation successfully captures the trends in loss variations
4

---

Learning Dynamics in Continual Pre-Training for Large Language Models
the concept of loss potential introduced in section 3.3 to describe the degree of annealing for PT models. Specifi- cally, a PT model trained without annealing has high loss potential, while PT model that anneals to a zero LR value has a low loss potential. We investigate the impact of loss potential on CPT under two different experimental settings: without or with LR re-warmup.
Du Lass Surface
Annesinn Area Directin
W/o Re- Re-warmup. In this setting, we set the initial LR for CPT as the final LR in PT and linearly anneal the LR for CPT to zero. We conduct experiments using PT models with different loss potentials (Fig. 5a). As shown in Fig. 5b, models with higher loss potential achieve lower final losses on Pept- This observation matches the prediction made by our CPT scaling law (Fig. 5c). We also utilize our equation to predict the final loss across various CPT steps, confirming that this trend persists in different settings.
Pre-Training Constant
Continual Pre- Pre-Tjaining
Andeaiing
Foward frea Diroction
(a) Loss surface of Dpt transfer slide.
Foward Loss
Potential
Arnealing Direction
With Re-warmup. A common practice for CPT is to lin- early re-warmup the LR from zero to a certain value, such as 10% of the peak LR in PT, before annealing to zero (Fig. 5d). As shown in Fig. Se and Fig. 5f, models with high loss potential consistently achieve lower final losses.
(b) Forward view.
(c) Annealing view.
Figure 4. The loss surface of the CPT process and two directional views.
3.4. Extension to Model Size and Replay Ratio
We can use our CPT scaling law (Eq. 4) to analyze the impact of loss potentials. Specifically, as the annealing coefficient C2 > C1 often holds for Dept. then allocating a larger annealing area in the CPT phase, i.e. a larger S.g". facilitates a lower loss. Moreover, models with higher loss potential have larger forward areas STT and STT which further contribute with high loss potential usually lead to lower Dept loss. This conclusion is also validated in previous works (Wang et al., 2024).
We attempt to incorporate model size N into our CPT seal- ing law by analyzing the effect of N on both the LR anneal- ing and distribution shift term. Our experiments show that the distribution shift terms remains unchanged across differ- ent model sizes when other settings are fixed (see details in Appendix E). Therefore, we can directly follow Tissue et al. (2024) to integrate an N-related term and use our scaling law to fit and predict CPT loss curves for different model sizes. More discussion is provided in Appendix E.
lower loss. Therefore, PT models
We also integrate the replay ratio into scaling law since replaying some data from Dpc is a common practice in CPT. Our experiments show that the replay ratio influences the distribution shift term in an exponential manner. By adding a single replay ratio related term, our scaling law can predict the entire training dynamic for different replay ratios, while previous studies (Que et al., 2024) can only predict the final loss. More details are given in Appendix H.
Finding 3. PT models with higher loss potential consistently achieve lower Dept validation losses. Hence, we advocate that when releasing open- source models, it is beneficial release high loss potential version facilitate downstream tasks.
4.2. Replay Ratio
4. Factor Analyses and Applications
The distributional distance between Dpt and Dept signif- icantly influences the distribution shift term in Eq. 4. As shown in Fig. 6a, more distinct Dept. Pile of Law (Hen- derson* et al., 2022), leads to sharper transfer curve than more similar Dept (Knowledge Pile (Fei et al., 2024)).
In this section, we analyze various factors for CPT and apply our scaling law to provide insights into these factors.
4.1. Loss Potential
In CPT, it is common practice to mix Due into Dept based on a certain replay ratio to mitigate the increase of validation loss on Dpt. The replay ratio plays a critical role in adjusting the distributional distance between Dyt and Dept since Dept
Most PT models are trained by annealing to a minimum LR for lower PT losses. However, the optimal PT model for CPT is not necessarily a fully annealed model. We use
5

---

Learning Dynamics in Continual Pre-Training for Large Language Models
Learning Rate x10
Lo's Leb moande Morly uee Nandaa
28% Lass dpaana Lous
J8C 5age
Vall dattonn
2,00
50% Last Peceria Vuat Lou Phocele Thin Los
Validationn
d0c CF S0gs
2.04
2.40
2.00
10850
32300
Step
CPT Stop
Lass Patential 56
(a) CPT with different loss potentials (w/o
(b) Pepe true loss CPT step for different loss potentials (w/o re- wa-waruup setting).
(c) Dept predicted loss vs. loss potentials for different CPT steps (w/o re- warmup setting).
re-warmup setting).
2.47
Rale X10-
1.7
100 Lime Roveniaa
Vallddation Loss
seld Las
10% Last mooras but lous
Validation Loes
J0c CPT Saps
1.3
And
1:35
1:5
4.00
244
150
2.06
159
7:40
10000
12000
100:00
L0003
251
Step
CPT Step
Loss Potential %
(d) CPT with different loss potentials (w/ (e) Dejt true loss vs. CPT step for different re- ce-warmu)).
(i) D Dept. predicted loss vs. loss potentials for different CPT steps (w/ re- we-warmup setting).
loss potentials (w/ re-warmup setting).
Figure 5. The impact of the loss potential of PT models. We illustrate the true loss of models with different loss potential in the middle panel. We utilize Eq. 4 to predict the losses of these models across different training steps in the right panel. The red star (+) refers to the models that achieve the lowest Depe validation loss given the number of CPT steps.
is modified to approach Dpe Results in Fig. 6b and Fig. 6c indicate that higher replay ratios lead smaller distribution shifts and thus effectively decelerate the deviation from Dpt- Quantitatively, we find that the replay ratio influences the distribution shift term based on exponential form, which is elaborated in Appendix H.
rise followed by a decline that does not return to the original loss value; or (3) an initial rise followed by a decline that goes below the original loss value.
As shown in Fig. 7c, we define the critical point (indicated by the blue dashed line) as the convergence value of the Dpt loss on the hidden PT curve trained on Dept. When CPT occurs before this critical point, the Dpe loss will first rise and then decline. The final loss may or may not be lower than the original loss. The minimum training steps required to return to the initial loss value are designated as the turning length. Conversely, if CPT occurs after the critical point, achieving lower Dpt loss than the initial value becomes unattainable, regardless of how many steps we train.
4.3. Peak LR
In real scenarios, choosing an appropriate peak LR for re- warmup is important for CPT. Different peak LRs affect the Drt and Dept validation loss. We leverage Eq. 4 to predict the final loss of different peak LRs. Specifically, we assume the PT model is trained using the WSD LRS. As shown in Fig. 7a and Fig. 7b, a high peak LR in the CPT phase accelerates the decrease of the Dept validation loss while leading to increase of the Dye validation loss.
Finding 4. Inadequate pre-training or weak dis- tribution shift can result in lower Drt loss values after sufficient CPT steps compared to the PT model. Otherwise, we are unlikely to achieve a lower Dpt loss than the PT model, regardless of how many CPT steps we train. In this situation, more training often leads to degraded general performance.
4.4. CPT Training Steps
The number of CPT training steps also an important hyper- parameter. A general observation is that more training steps lead to lower D Dept validation loss. However, the Dpe val- idation loss may exhibit three different patterns based on the state of the PT model and the distributional distance between Dpt and Dept: (1) a continuous rise; (2) an initial
5. Balance Between Dpt and Dept Loss
'It is important to note that the Dept undergoes modifications in the presence of replays. For instance, if we mix 0.1 Web with 0.9 KP. the actual Dept distribution is 0.1 FineWeb with 0.9 KP, not 1.0 KPP.
Validation losses on Dpt and Dept typically exhibit a trade- off in the CPT process. Balancing these losses is critical for optimizing the overall performance of the model during CPT. We define the increase Dpt loss as A ALD,, and the
6

---

Learning Dynamics in Continual Pre-Training for Large Language Models
debind Crowidge file Oge drfisst We Laee
Knowledge Pile Loss
FineWeb Loss
FineWebb Lost
Peraaait22 2.2 jif batty
Sreep Darriauten Sa
decriaation
Drann D vit font
Bacio Stit
Step
Step
Step
(a) The difference in distribution shift for (b) The distribution shift the De valida- different Dept datasets.
(c) The distribution shift the Dept valida- tion set for different replay ratios.
tion set for different replay ratios.
Figure 6. We compare the distribution shift for different distributional distances between the Degs and Dyt datasets. Additionally, we examine the impact of different replay ratios on the distribution shifts within both the Dept and Dye validation sets.
CIT Sam Whn
Validation 1055
Validation Loss
2:04
Turing Donga
CIT Mox
ureald
Dy Validation
334
2,00
005
6:5
8ith
300
0.23
2.52
0.50
1.2
3.15
Max learning rate
Max learning rate
Step
(a) Dpt predicted loss vs. peak LRs for dif- ferent CPT steps.
(b) Dept predicted loss vs. peak LRs for
(c) Critical point and turning length in Dps validation loss.
different CPT steps.
Figure 7. (a)-(b) The effect of the peak LR. We utilize Eq. 4 to predict the final loss for different peak LRs. (c) The effeet of different CPT steps. We show the critical point and turning length in the Dre validation loss.
decrease in Dept loss as ALD To balance the loss of Dpt and Dept validation sets, we assign normalized balance coefficient to different validation sets:
Replay Ratio. Based on our scaling law with replay ratio Eq. we can determine the optimal replay ratio for each D1 (Fig. 8c). The same distribution line (dashed line) in Fig. 8c indicates that the optimal replay ratio should be the same the target weight \ if we initialize the CPT model randomly rather than from a pre- rre-trained model. Instead, in practice, the optimal replay ratio shifts because the PT model has already been trained on Dpt. which causes the curve to deviate and exhibit a wave pattern.
min STP SS2
A1 ATD >r
(5)
s.t. A1 +
where A and X2 are coefficients that should be set based on our prior knowledge of the relative importance of general and downstream performance.
CPT Training Steps. As shown in Fig. 13, we can get different turning lengths for different values of λ₁. When A is small, the Dept loss predominates the composite loss A Ln,, + >2 which consistently remains below the initial value. Conversely, with a moderate A₁. there exists a specific step that makes the composite loss equals the inital loss. For large }1, the composite loss is always higher than the initial loss, which means that CPT is not suitable any more in this situation.
5.1. Optimal
Given the different coefficients A₁ and X2, there exist some optimal CPT hyper-parament parameters.
Loss Potential. Fig. 8a shows the optimal loss potential for different values of A1 It can be observed that a small X corresponds to large optimal loss potential. This makes sense since a small A means that the final loss is dominated by the Dept loss, and thus it is necessary to reserve sufficient loss potential for downstream domains.
5.2. Out-of-Domanl Validation Set
Note that our CPT scaling law is designated to predict losses on Dptt and Dept validation sets, while it is not directly applicable to the oft-of-domain (OOD) validation set D Dood- Inspired by previous works (Ye et al., 2024; Zhang et al., 2025) that the OOD validation loss can be represented as a
Peak LR. We can also predict the optimal peak LR in the CPT process when A is given (Fig. 8b). A larger A suggests a preference for minimizing the increase in Dpt loss, thereby necessitating a lower peak L.R
7

---

Learning Dynamics in Continual Pre-Training for Large Language Models
Rieibllio DAA Suden
pur serrodooo pher
1.3
Optinal Loss Potential
1.50
0.8-
Samr dige
Opámal Replay
2.1
0.4
0.2
7g
(a) The optimal loss potential.
(b) The optimal peak LR.
(c) The optimal replay ratio.
Figure 8. Optimizing hyper-parameters for CPT based on different coefficients to balance general and downstream performance. Strong, moderate, weak distribution shift (a) and (b) denote different CPT datasets as Pile of Law, Knowledge Pile, and mixture 67% Web and 33% Knowledge Pile, respectively. The "same distribution" in (c) represents reference line where the target weight (A1) is the same replay ratio. See Appendix I for more details.
34
Out-of-Domain Loss
Sinfwema Los
Out Out-of-Domain Loss
3.6
Stotee Lone
3.4
2.8-
Stackeschargg 18330 6.580
Stsrenthhrgg Lasa
c4 0.152 + 0.620
3.2
2.6-
SimParra 0.300 6.722
2.0
62500
45000
47500
50000
52500
55000
57500
60000
42500
45000
67500
50000
525.00
55000
57500
60000
Step
Step
D aod dataset truth and predicted loss similar Dpt-
(b) Dood dataset truth and predicted loss similar Dept-
Figure 9. The predicted loss curve of Dood validation set, predicted by leveraging a linear combination of Dpt and Dept validation losses. We show some different shapes rising curves similar to Dae on the left and some different shapes of falling curves similar to Dept on the right.
linear combination of losses on several base domains, we hypothesize that the loss on D ood be represented by linear combination based on Dpt and Dept validation losses:
that our approach are quite effective and practical in real scenarios. Moreover, the calculated coefficient represents the "similarity" between OOD datasets and Dpt or Dept- As Fig. 9 shows, there are two kinds of OOD datasets: (1) - one (larger X)) with loss curve upward, and (2) D like one (larger X₂)) with loss curve downward.
(6) We verify this hypothesis and calculate X1 and X₂2 for sev- eral example OOD datasets in Appendix K. Note that the coefficients X1 and X/2 are related only to datasets and not to other training hyper-paramette
DD LDD + X2L
There exists an optimal loss potential, peak L.R and replay ratio designated to balance Dpt and Dept losses. Besides, the turning lengths vary depending on the different balance weights. Predict- ing D. is equivalent to balancing Dat and Dept losses by utilization of linear combination tricks.
Finding 5.
Loss Prediction of Dood. The Dood validation loss does not adhere to the formulation described in Eq. 4. However, by calculating and specifying the coefficients X and X/2 becomes feasible to predict the Dood loss curve using a linear combination of the Dpt and Dept loss curves. It is interesting that this problem reduces the balance between Dpt and Dept loss (Eq. 5). The optimal hyper-parameter such as LR and replay ratio for this setting have been ade- quately discussed in the previous section.
6. Open-Source PT Models
For the majority of LLM communities, the PT models use are usually not trained by ourselves, but from open- source models. Most training details are not reported for those source PT models, i.e., the distribution of Dpm- the loss potential, and the PT training hyper-parameters are usually unknown. This inhibits the direct application
As shown Fig. 9, we first calculate the coefficients in Eq. 6 for several OOD datasets and then predict the corresponding validation losses. The almost perfect prediction suggests
8

---

Learning Dynamics in Continual Pre-Training for Large Language Models
of our CPT scaling law. To solve this issue, we propose the following methods to make our scaling law become applicable again.
Limitations. One main limitation of our work is that our laws are primarily based on empirical analyses and experi- mental verifications. We acknowledge that there is lack of rigorous theoretical analysis and proof because it is difficult to build theoretical deduction in a non-toy environment with thousands of LLM training factors. However, our scaling law can reasonably reflect the learning dynamics of the CPT process, which can be applied in practical CPT scenarios.
(a) Firstly, for the unknown PT dataset distribution, some methods based on probing (Hayase et al., 2024) have been proposed. Instead, we simply utilize an open-source Com- mon Crawl dataset as a proxy Dpt to approximate the dis- tribution of pt. (b) Secondly, when fitting our scaling law, regard some variables unknown parameters to fit. For example, we treat ST as parameter that requires fitting to be close to the undisclosed real ST (c) Thirdly, as most open-source PT models anneal to a minimal LR to get a better performance nowadays, we assume all open-source models anneal their LR to zero when calculating S.p Refer to Appendix G for more details.
8. Conclusion
In this study, we explore the learning dynamics in continual pre-training of large language models. We focus on the evolution of performance across general and downstream domains, with domain performance assessed with valida- tion loss. By observations and analyses, we propose a CPT scaling law that integrates distribution shift and learning rate annealing to predict the validation loss at any interme- diate training step under common learning rate schedules. Our scaling law provides a comprehensive understanding of key CPT factors and helps optimize the nyper-paramette in CPT for different training goals. Further experiments demonstrate that the law can also be extended to more com- plicated scenarios such as out-of-domain datasets and pre- trained models with unknown training details. We believe that our CPT scaling law is promising to reshape the under- standing of researchers for LLM continual pre-training and scaling laws.
To verify our solutions for open-source PT models, we con- tinually pre-train I.L.MA3.... 2- 1B (Dubey et al., 2024) and se- lect the RedPajama (Weber et al., 2024) dataset as an proxy Dpt- As Fig. 18 in Appendix G shows, the almost perfect fitting and prediction for CPT loss curve of LLaMA3.2-IB IBB suggests the effectiivnness of our proposed methods. More- over, this result also indicates that our scaling law can be casily extended to CPT scenarios with unknown PT model information, demonstrating the superiority of our scaling law to capture the learning dynamics of CPT.
7. Discussion
Laws Formulation. The formulation of S2 in Eq. 1 can have other forms. For example, S2 could also be a multi- power form (Luo et al., 2025), which is proposed following the work of Tissue et al. (2024). We adopt the equation form in Eq. 1 because it has fewer parameters and it works more effectively in practice. We also compare some format variates including adding a LR-weighted coefficient and adding power term to S2 (see more details in Appendix J). The experiments show that all formats lead similar re- sults while our formulation has superiority in simplicity (i.e. fewer parameters).
Impact Statement
CPT is effective method to enhance the foundation large language models to specific downstream domains or tasks, Our work provides a scaling law to quantitatively describe the learning dynamics of CPT processes. Our results can be used to optimize the training hyper-parameters for balancing the general and downstream performance.
While there will be important impacts resulting from the of CPT in general, here we focus on the impact of using scaling law to provide explanations for CPT process. There are many benefits for using our method, such as predicting the loss curve dynamics and optimizing The work presented in this aims to advance the field of Large Language Models. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Laws Fitting, In our experiments, we predominantly em- ploy constant, cosine, and WSD LRS to fit data, which are widely used in practical applications. It is worth noting that many other LR schedules could be also modeled. To apply our scaling law, we use common LRS (e.g. constant and cosine) to train a few steps to collect loss values. Af- ter parameters are fitted based on these values, our scaling law is also capable of predicting the loss curve under other specialized LRS for much longer training durations. Our scaling law shares the similar idea of fitting cost conser- vation with Tissue et al. (2024), thanks to our scaling law being able to describe the whole dynamics in CPT rather than only final loss.
A Acknowledgments
This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant XDA0480301 and Fujian Provincial Natural Science Foun- dation of China (No. 2024J08371).
9

---

Learning Dynamics in Continual Pre- Pre-Training for Large Language Models
References
Fei, Z., Shao, Y., Li, L.., Zeng, Z., Yan, H., Qiu, X., and Lin, D. Query of ce: Unearthing large scale domain- specific knowledge from public corpora. arXiv preprint arXiv:2401 1ete224411..62444 14624, 2024.
Balandat, M., Karrer, B., Jiang, D., Daulton, S., Letham, B.,
Wilson, A. G., and Bakshy, E. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems, 33:21524-21538,, 2020.
French, R. M. Catastrophic forgetting in connectionist net- works, Trends in cognitive sciences, 3(4): 10:::128-1333 135, 1999.
Barnett, M. An empirical study of scaling laws for trans- fer, 2024. URL httph::///// //tp:////aiiv.eeeee org/oorg//a... /200. 16947.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser. S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002) 2020.
Bergstra, J. and Bengio, Random search for hyper- parameter optimization. Journal of machine learning research, 13(2), 2012.
Gu, J., Yang, Z., Ding, C., Zhao, R., and Tan, F. CMR scaling law: Predicting critical mixture ratios for con- tinual pre-training of language models. AI-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.). Proceedings of the 2024 Conference Empirical Methods in Nat- ural Lunguage Processing, pp. 16143-16162, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10. 10.11663ww120024ccmmmmmmmm 903. URL https: emn 1p-mal-mnnn.. 903.
Chen, W., Zhou, Y., Du, N., Huang, Y., Laudon, J., Chen, Z., and Cui, C. Lifelong language pretraining with ietributiio-specialiie experts. In Proceedings of the 40th International Conference Machine Learning, ICML'23, JorR.org, 2023a.
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mo- htashami, A., Sallinen, A., Sakhaeirad, A., Swamy, V., Krawezuk, I., Bayazit, D., Marmet, A., Montariol, S., Hartley, M.-A., Jaggi, M., and Bosselut, A. Meditron- 70b: Scaling medical pretraining for large language mod- cls. 2023b. URL https: arxiv. org/ abss/311- 16079.
.rg.
2024/ 2024.
Gupta, K., Thérien, B., Ibrahim, A., Richter, M. L., An- thony, Q., Belilovsky, E., Rish, I., and Lesort, Con- tinual pre-training of large language models: How to (re)warm your model?, 2023. URL htRp::/t:///ar org/ abs / 300 04014.
Hayase, J., Liu, A., Choi, Y., Oh, S., and Smith, N. A. Data mixture inference: What do bpe tokenizers reveal about their training data? arXiv preprint arXiv:2407.16600 2024.
Colombo, P., Pires, T. P., Boudiaf, M., Culver, D., Melo, R., Corro, C., Martins, A. F. T., Esposito, F., Raposo, V. I.,, Morgado, S., and Desa, M. Saullm-7b: A pioneering large language model for law, 2024. URL https:// arxiv. org/ abs 2403.33.22 03883.
Henderson P., Krass*, M. S., Zheng, L., Guha, N., Man-
ning. C. D., Jurafsky, D., and Ho, D. E. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset, 2022. URL https: //arxiv. orooorg////o... /tg/abs//////oao///////////aaeea abs 2207000220
DeepSeek-AI, Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R. Wu, Y., Li, Y., Gao, H., Ma, S., Zeng, W., Bi, X., Gu, Z., Xu, H., Dai, D., Dong, K., Zhang, L., Piao, Y., Gou, Z., Xie, Z., Hao, Z., Wang. B., Song, J., Chen, D., Xie, X., Guan, K., You, Y., Liu, A., Du, Q., Gao, W., Lu, X., Chen, Q. Wang, Y., Deng, C., Li. J., Zhao, C., Ruan, C., Luo, F,, and Liang, W. Deepseek- coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024. URL https: htt.ixxxvvv..../... abs/ 2406 11931.
Hernandez, D., Kaplan, J., Henighan, T., and McCan- dlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021a.
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer, 2021b. URL https: a arxiv. orlvv.rrg/aai abs 2202221122.0 01293.
Dou, L.., Liu, Q. Zeng, G., Guo, J., Zhou, J., Lu, W., and Lin, M. Sailor: Open language models for south-east asia. arXiv preprint PxXx:24444003000 2024.
Hoffmann, J.. Borgeaud, S., Mensch, A., Buchatskaya, E.. Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W. Vinyals, O., and Sifre, L. Training compute-optimal large language models. arXiv preprint arXiv: 2203.15553 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd models. arXiv preprint arXiv: 2024.
10

---

Learning Dynamics in Continual Pre- Pre-Training for Large Language Models
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang. X., Thai, Z. L.. Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J. Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arvi 06395, 2024.
Luo, K., Wen, H., Hu, S., Sun, Z., Sun, M., Liu, Z., Lyu, K., and Chen, W. A multi-power law for loss curve pre- diction across learning rate schedules. In The Thirteenth International Conference on Learning Representations, 2025. URL https:: /wwp//epeeeeee.eeoe//e net forum? Ko-K6999xxIK
Nocedal, J. Updating quasi newton matrices with lim- ited storage. Mathematics of Computation, 35(151): 951-958, July 1980. ISSN 0025-5718. doi: 10.1090/ 111-5771--1980--57285----
Huber, J. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1):73 101, 1964. doi: 10.1214aamm/11177700077 //doi. org / 10.
URL https: /
OpenAL. Gpt-4 technical report.
arXiv preprint
Hui, B., Yang, J., Cui, Z., Yang. J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., Dang, K., Fan, Y.,
arXiv:2303.0877,,200 2023. 2023.
Zhang, Y.. Yang, A., Men, R., Huang, F., Zheng. B.,
Parmar, J., Satheesh, S., Patwary, M., Shoeybi, M., and Catanzaro, B. Reuse, don't retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv: 2407.07263, 2024.
Miao, Y., Quan, S., Feng, Y., Ren, X., Ren, X., Zhou, J., and Lin, J. Qwen2.5-coder technical report, 2024. URL https: //ariv///rvvv.//// org/ abs/ / 2409 12186.
Ibrahim, A., Thérien, B., Gupta, K., Richter, M. L., An- thony, Q. G.. Belilovsky, E., Lesort, T., and Rish, Simple and scalable strategies continually pre-train large language models. Trans. Mach. Learn. Res., 2024, 2024. URL https: http:::://pppeeviiii wwwpeeeeiee.ene/oorouooooo ad-DimPeeCxKO
Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathematical web text, 2023.
Penedo, G., Kydlíček, H., allal, L.. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data scale. In The Thirty-eight Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2024. URL https:////// wwwp/ewpeneeeeeeeepeppe..... net/ forum? 1 dd=n 3S=n6sCn22@G.
Kaplan, J.. McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv: 2001.08361, 2020.
Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference tations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv.ors abs/ / 1412.1120 6980.
Que, H., Liu, J., Zhang, G., Zhang, C., Qu, X., Ma, Y., Duan, F,, ZhiqiBai, JiakaiWang, Zhang, Y., Tan, X., Fu, J., Wang, J., Qu, L., Su, W., and Zheng, B. D-CPT law: Domain-specific continual pre-training scaling law for large language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL ht https: /pp.neeeeeeeeeeeeeeeeee/eeroeeee net/ forum? i J2=K
Learning Represen-
Lange, M. D., van de Ven, G. M., and Tuytelaars, T. Con- tinual evaluation for lifelong learning: Identifying the stability gap. In The Fleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview.comommmmrrer wet.:iiwww..eeeeeeeeeeeeeeeeeeee net
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang. Z., Ebrahimi, S., and Wang. H. Continual learning of large language models: comprehensive survey, 2024. URL https: / arxiv www.:oorgg/bb....../.a http:/abb/2244.166688 http:////wwwww..//////////a
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Snoek, J., Larochelle, H., and Adams, R. P. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Loshchilov. I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. International Conference on Learning Representations, 2016.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B to- ken cleaned and deduplicated version of RedPajama, June 2023. URL https: datasetss/ https////gggggggggggeeege co/ i jama- 627B.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv: 1711.05101 101, 2017.

---

Learning Dynamics in Continual Pre-Training for Large Language Models
Tissue, H., Wang, V., and Wang, L. Scaling law with learning annealing, 2024.
org/ htg//bs//a 2408. 11029.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, Su, G., Perot, V., Dy, and Pfister, T. Learning to prompt for continual learning. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 139-149, 2022. doi: 10. 109/CVPR52688 0TT.5526882202.0000000 2022.00024.
Wang, Z., Liu, S., Huang, J., Zheng, W., Liao, Y., Chen, X., Yao, J., and Su, J. A learning rate path switching training paradigm for version updates of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13581- 13594, 2024.
Weber, M., Fu, D. Y., Anthony, Q. Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Redpajama: an open dataset for training large language models. NeurlPS Datasets and Benchmarks Track, 2024.
Xie, X., Ding, K., Yan, S., Toh, K.-C., and Wei, T. Opti- mization hyper-parameter laws for large language mod- els, 2025. URL https: http//axxvvv////// http://..rr//////40/0000 04777.
Ye, J.. Liu, P., Sun, T., Zhou, Y., Zhan, J., and Qiu, X. Data mixing laws: Optimizing data mixtures by predicting language modeling performance, 2024. URL https: //arxiv.orr .rg/ abs/ 2403. 16952.
Zhang, M., Tissue, H., Wang, L., and Qiu, X. Domain2vec: Vectorizing datasets to find the optimal data mixture without training, 2025. URL https:// //tpr///eee/er.e...eemrom
Çagatay Yıldız, Ravichandran, N. K., Punia, P., Bethge, M., and Ermis, B. Investigating continual pretraining in large language models: Insights and implications, 2024. URL https: / //arxiv... wrggabs/////ooo.oooo.goooobbbebb 2402. 17400.
12

---

Learning Dynamics in Continual Pre-Training for Large Language Models
A. Related Work
Continual Pre-Training Continual Pre-Training (CPT) aims to continuously pre-train LLMs to adapt to new domains, such as code (Hui et al.. 2024; DeepSeek-AL et al., 2024), medicine (Chen al., 2023b) and law (Colombo et al., 2024), while avoiding the need to train domain-specific LLMs from scratch (Shi al., 2024). The primary objective of CPT is to enhance downstream performance while avoiding catastrophic forgetting (Lange et al., 2023; French, 1999; Gupta et al., 2023; Ibrahim et al., 2024). Most existing CPT methods primarily leverage the replay to mix appropriate pre-training data (Que et al., 2024; Gu et al., 2024) or introduce extra model parameters (Wang et al., 2022) to assimilate new domain knowledge. In this work, we comprehensively study the learning dynamics of CPT and propose a CPT scaling law to describe both general and downstream validation loss.
Scaling Laws. Kaplan et al. (2020) empirically discovers a power-law relationship between validation loss L and there factors: model size N, dataset size D, and training compute. Hoffmann et al. (2022) develops Chinchilla, compute-optimal LLM to balance model size and dataset size. Tissue et al. (2024) introduces a scaling law to describe the learning dynamics affected by the learning rate annealing, which can predict loss any training steps under various LRS. However, these scaling laws are limited to pre-training scenarios and do not apply when the training dataset changes.
Regarding continual pre-training, Hernandez et al. (2021b) study scaling law for transfer with respect to model size and CPT data. Barnett (2024) proposes an empirical scaling law that incorporates a transfer gap term to indicate the distribution difference between two datasets. Some methods like D-CPT (Que et al., 2024) and CMR (Gu et al., 2024) introduce scaling laws that account for data replay or mixture ratio in the CPT process. Dou et al. (2024) proposes a quadratic function that considers both learning rate and replay ratio. Nevertheless, these existing scaling laws primarily describe the final loss of given LRS and do not account for all CPT-related factors. Our proposed CPT scaling law integrates all relevant factors and can predict loss at each CPT step, thereby providing a comprehensive description of the complete learning dynamics.
Slyper-Parameee Optimization. Identifying optimal hyper-parametes settings is crucial for achieving robust performance in machine learning (Bergstra & Bengio, 2012; Snoek et al., 2012). The principal hyper-pann parameters in large language models include peak learning rate, learning rate schedules, batch size, and training steps (Kaplan et al., 2020; Hu et al., 2024; Xie et al., 2025). Initial approaches to apper--rrameter optimization primarily utilize model-free techniques such as grid and random search (Bergstra & Bengio, 2012). Subsequently, some methods have employed Bayesian Optimization (Balandat et al., 2020) to predict the performance of various hyper-parameters and select the most effective ones accordingly. Our research focuses on the hyper-parameter in the continual pre-training of larger language models using our proposed CPT scaling law. The hyper-paramedt parameter we optimize include the learning rate schedules, peak learning rate, and replay ratio.
B. Experiment Setups
In this work, we employ multiple experimental setups to validate the effectiveness of our equation. We summarize all experimental setups in Table 1. The majority of our experiments utilize Setting A. Experiments with different replay ratios, batch size, or sequence length are conducted by directly modifying corresponding setups.
C. Fitting Details
We set À = 0.999 in our all experiments, Given the LRS of PT and CPT, we can compute out Spt S and S in advance. We adopt a similar fitting method as Chinchilla scaling law (Hoffmann et al., 2022). We minimize the Huber loss (Huber. 1964) between the predicted and the observed log loss using the L-BFGS algorithm (Nocedal, 1980). We implement this by the utilization of minimize ze in s scipy library. We mitigate the potential issue of local minima of fitting by choosing the optimal fit from a range of initial conditions.
D. A Additional Continual Pre- Pre-Training Results
Prediction of Other LRS. We use the fitted parameters in Fig. 3 to predict the loss of other LRS (Fig. 10a and Fig. 10d). Our equation could effectively predict the loss of other LRS as shown in Fig. 10.
Other Dept Dataset. Besides Knowledge-Piie (Fei et al., 2024), we also use Eq. 4 to fit transfer loss curves of other Dept dataset Pile-of-Law (Henderson* et al., 2022) in the Fig. 11.
13

---

Learning Dynamics in Continual Pre- Pre-Training for Large Language Models
Table 1. Experimental settings adopted in this work. Model size denotes the number of nonembedding parameters. We use Adam W
Optimizer (Kingma & Ba, 2015; Loshchilov & Hutter, 2017). Most experiments adopt LLaMA-3* 3's tokenizer (Dubey et al., 2024).
Setups
Setting (main)
Setting
Setting
Setting D
Model Size
106M
106M
594M
1720M
PT dataset
Wih
FineWeb
FineWeh
Febe
CPT dataset
Knowledge-Pin
Pile-of-Law
Knowledge-Pile
Knowledge-Pin
Peak LR
2 10-4
2 10-
2 10-4
10
PT Batch Size (Tokens)
4M
4M
4M
4M
CPT Batch Size (Tokens)
4M
4M
4M
4M
PT Sequence Length
4096
4096
4096
4096
CPT Sequence Length
4096
4096
4096
4096
Tokenizer
LLMMA-3's
LLaMA-3's
LLaMA-3's
LLaMA-3's
31, 32 in A
0.9, 0.95
0.9, 0.95
0.9, 0.95
0.9, 0.95
Weight Decay
0.1
0.1
0.1
0.1
Gradient Clip
1.0
1.0
1.0
1.0
Setups
LJaMA3.2-1B
Model Size
1B
PT dataset
Unknown Pile-of-Law
CPT dataset
Peak LR
2 x 10-5
PT Batch Size (Tokens)
Unknown
CPT Batch Size (Tokens)
4M
PT Sequence Length
Unknown
CPT Sequence Length
4096
Tokenizer
LLaMA-3's
31,80 82 in AdamW
0.9, 0.95
Weight Decay
0.1
Gradient Clip
1.0
14

---

Learning Dynamics in Continual Pre-Training for Large Language Models
2.00
WSo Resir ding trrat Lest Uneer Cantruu Peo
3.4
Unea Picherin This
wid inelikingg Tuan Los
1.75
Dze Validation Loss
3.7
Uneer Contrnll PosTuring hes ted Lows
Validation Loss
3.3
Unea Corinval Ph-erii Prockcil Less
32-
1.25
3.5
2.1
3.0
Learning
2.4
0.75
2.3
0.50
2.7
Pre-Training LRS
0.25
Continual Pre Pre-Trainng LRS
2.6
0.00
6
10000
20500
33000
40000
50000
t0000
10000
20003
30003
40000
50300
02020
2.5
12090
20200
32000
40:00
50000
60000
Step
Step
Step
(a) Learning Rate Schedule.
(b) Predicted Duu Web) Loss.
(c) Predicted Dgu (Knowledge Pile) Loss.
Who tast
Widt tis variing wit test
Con Contiis Pep
Crat Continal PacTaring Thin
1.75
Validation LOSs
3.7
Cos Contioee ProTuinngg hrd atd Low
Dax Validation Loss
3.3
Ceslre Continal TweThiinng Preclcil Less
32-
1.25
3.5
2.1
2.0
Learning
2.9
0.75
2:3
0.50
2.7
0.2
Pre-Faiiingg LKS
Continaal Pre-iiining LRS
2.8
0.00
0
10000
20500
32000
40000
50000
10000
10000
20003
10000
00005
50000
63030
13650
20300
32000
10000
Sc000
Step
Step
Step
(d) Learning Rate Schedule.
(e) Predicted Dpe FineWeb) Loss.
(I) Predicted Dept (Knowledge Pile) Loss.
Figure 10. Using the fitted parameters in the Fig. 3 predict all pre- pre-training and CPT loss curve of other LRS. (a) is one kind of without re-warmup method and (b) is a more realistic LRS that the learning rate re-warmup to 10% peak PT learning rate and then annealing to zero with cosine method.
W42 Freilssiingg Tuah Lass Contie Frotthiiing Titad
3.50
1.75
Do: Validation Loss
4.0
Cuirs Orecaa beyy flasdd Los
and Aogieieeerg Kired loo.
Validation Loss
1.50
3.02
1.25
2R--30 FUREE spirrs 11.0111
2.294 -1256
2:75
Uo1- 2370 3 43507
-17100 -6
Learning
1.00
3.6
2.52
0.50
2.00
Pre-Thaining LBS
Contt nual The-Thiting LBS
1.75
0.00
5000
10030
2030025000 30003 40000000000000000000000000000000 45000 50000
10:003500:220000 25:03 30000 35090 4500040000055002 50000
5000
10000 2000020000000 30003 35:00 400000000000000000000000000 45000 50000
Step
Step
Step
(a) Learning Rate Schedule.
(b) Dpc (ebneWeb) Validation Loss.
(c) Dept (Pile-of- Law Validation Loss.
WSD fredhti ting Tash Lasa eCD treetaa flaed Loss
3.50
Con Cantin TooTutiin Thuth Loo
Fre taiiing Tualt
Learning x10-
1.75
4:2
Validation Loss
Contercal Feo Phitiingg Wited
Op: Validation
4.0-
2.02
1.25
Lid 2.00 catti .5%% -1,2331
2:75
104 caspet -32700
1.00
2.59
0.25
2.25
0.5
3.4
2.03
ReiThai-nngRS
Curkrual Nc Taising LS
177
0.00
10000
20020
20205
10020
50000
GCOO0
70500
10000
20000
30000
42020
50600
600000
70000
10000
20000
30202
40000
50203
600000
70200
Step
Step
Step
(d) Learning Rate Schedule.
(c) Dp: (FineWeb) Validation Loss.
(f) Deps (Pilc-of-Lav) Validation Loss.
Figure II. Using Eq. 4 to fit all loss curves which are pre-trained with Web and continual pre-trained with law.
15

---

Learning Dynamics in Continual Pre-Training for Large Language Models
377
2,00
WGB Continal hrchh wit Pire
mn cande yor sagin
Dxc Validation Loss
3.6
tho soaiing
1.50
Dax Validation
25
1.00
3.4
2.8
33
0.0
Preirahiinggs Continual Tniring
0.00
Step
Step
Step
(a) Learning Rate Schedule.
(b) Dpt (FineWeb) Loss of 33% Replay.
(c) Dept (KP) Loss of 33% Replay.
2.00
West ein tuts test wal Cononnal Tho wes ene
34
Candinus Fru Fru
Learning x10%
Dxt Validation Loss
3.6
Dent Validation Loss
155
3.5
1.00
3.4-
3.0
238971 (2110
24
0.50
Contrnal Pre-inttinng
24
3:000
Step
Step
Step
(d) Learning Rate Schedule.
(e) Dpc (Fine Loss of 50% Replay.
(1) D (KP) Loss of 50% Replay.
Leamning X10
2.00
pet Validation LOss
3.5
was Consnaal Pe-Ta wil Concied har
wit aur Wais
Lows
Due Validation Loss
3.4
into ug 122nd Lolo
Candincl Pechhining
1:50
2.5
-1255 -0205)
1.00
3.4
3.0
4U1-2667- EAGUE Aone -E33211 -190557 11-4M
2.
050
Continaa Pre-iniring
Pre-hainingg.s
2.6
0.00
35000
Step
Step
Step
(g) Learning Rate Schedule.
(h) Dpt. Web) Loss of 67% Replay.
(i) Dept (KP) Loss of 67% Replay.
Figure 12. Using Eq. to fit different Dpt replay ratio models independently.
Strong Derrbuttor
Tumingg Length
80K
60K
40K
20k
0.4
0.5
0.6
0.7
0.8
0.9
1.0
A1
Figure 13. The CPT turning lengths of different coefficients for halancing general and downstream domain performance.
16

---

Learning Dynamics in Continual Pre-Training for Large Language Models
4.2
4.00
Knowledge Pile Loss
3.75
FineWeb Loss
3.8
2210
3.50
6670 20060
N: 17200
CPT Steptel
3:25
3,4
3.00
2.75
3.0
Distroottenn Sritt
2.50
Disty sattybutiion SHift
2.6
20000
30000
40000
50000
60000
10000
20000
30000
40000
50000
60000
Step
Step
(a) Dpt Distribution Shift of different model size N.
(b) Dept Distribution Shift of different model size
Figure 14, The distribution shift across different model size with constant LRS.
Leaminn x100%
cide Grard Bulh
Knowledge Pile Loss
3.6
1.75
3.4
1.50
1.25
FineWeb
3.4
3.0-
1.00
3:-
3.0
2.6
0.50
2.8
2.4
Pro.Tahiingg LhS
Continual Pre-tmiriing LBS
2.
0.00
20000
30000
20:03
00002
60020
2.0
10000
30000
50000
Step
Step
Step
(a) Learning Rate Schedule.
(b) FineWeb (Dpc) Validation Loss.
(c) Knowledge Pile (Dea)) Validation Loss.
Figure 15. Using Eq. to fit the loss curve of all model size both D, and Dept validation set.
Different Replay Ratio. We use Eq. 4 to fit all loss curves of different Dyt replay ratio independently in Fig. 12.
E. Extension To Model Size Scaling
Distribution Shift Term of Different Model Sizes. We first explore the effect of model size N on the distribution shift term. CPT experiments are conducted across various model sizes-106M, 594M, and 1.7B without embedding- -usin a constant learning rate. As shown in Fig. 14a, the shift terms for different model sizes N nearly coincide. Based these observations, we hypothesize that the distribution shift term is independent of both model size N and transfer starting points. This implies that data transfer results in a consistent loss difference across different models sizes.
Model Size Scaling. Meanwhile, scaling law with LR annealing (Tissue et al., 2024) has demonstrated that the learning rate annealing scales with model sizes N, that S2 N", Building on the experiments and analysis above, we extend our proposed Eq. 4 to incorporate model size scaling:
LSSStt, Scott =To
(S + STPt Ci S No S
C2 S N
+ B (1 E. STPL +
where F,71, 72, 73 the constant parameters. The F N law (Hoffmann et al., 2022). We use Eq. 7 to fit the transfer curves of all model sizes as shown in Fig. 15.
7a is the model size term in traditional Chinchilla scaling
Furthermore, we apply Eq. 4 to fit the CPT loss curves of larger model sizes independently, as illustrated in Fig. 16. This demonstrates the adaptability of our equation across various model sizes.
However, it should be emphasized that the model size in our experiments have not yet reached the scale of mainstream LLMs today, so our experimental conclusions regarding model size are based on the assumptions derived from existing
17

---

Learning Dynamics in Continual Pre- Pre-Training for Large Language Models
2.00
WDD Continall hec was rite Lass
311
Cantincl Pechihing Tuth Los
X10"
Dept Validation Loss
2.-
mi merri rize
1.50
3.2
Dp: Validation
-640300 14021--13300 33607
27
(11-2657-244322 sptyiii
Learning
1.00
3.0
2.6
.420002
33355
2.5
030
24
Continal Pie training LRS
Pre-taiiiing
23
0.00
20500
33000
50300
52000
Step
Step
Step
(a) Learning Rate Schedule.
(b) Dps (FineWeb) Loss of 594M.
(c) Dept (Knowledge Pilc) Loss of 594M.
2.00
West Exedived Aer toeeg terv WGB Prattaiing Fad
3.0
Who col Iro 1ure ong Lod. Prechalin Lom
Dept Validation Loss
2.4
1.50
Dp: Validation
2.2
2.6
Learning
1.00
2.5
2.5
spfle 132211 Homnp'
2.4
0.50
2.3
Proitatting Lhs
Continaal Pre Thiring Lis
22-
0.00
20000
35000
40000
03000
21
50000
Step
Step
Step
(d) Learning Rate Schedule.
(c) Dpe (FineWeb) Loss of 1720M.
(f) Dept (Knowledge Pile) Loss of 1720M.
Figure 16. Using Eq. 4 fit all PT and CPT loss curve of 594M and 1720M model size respectively.
results. If the assumption hold for larger model size (e.g., 7B. 70B), we can conclude that inftuenced by the same absolute distribution shift value, larger models exhibit greater vulnerability in the general domain but demonstrate better adaptability to downstream domains.
F. Batch Size and Sequence Length
In the above experiments, we maintain the same batch size for both PT and CPT phases. However, in the real situation, when computational resources and datasets are limited, practitioners may keep a smaller CPT global batch size than PT phase. Additionally, in other cases, CPT aims to increase the context length of LLMs, requiring increases in both sequence length and RoPE base. We conduct CPT experiments with larger and smaller batch sizes, as shown in Fig. 17. When the sequence length is 8K, we increase RoPE base from 10,000 to 500,000.
Distribution Shift of Different Batch Size We leverage the constant LR to examine whether the distribution shift term for different batch sizes satisfies the same functional form. We using Eq. 4 to fit the loss curves of different batch sizes. As shown in Fig. 17a and Fig. 17b, all loss curves with different transfer steps for both larger and smaller batch sizes can be fitted with single distribution shift term, which demonstrates that Eq. 4 can also accommodate changes in batch size and sequence length.
G. Open-Source Pre- Fre-Training Models
more realistic scenario posits that the PT model is an open-source model, and we do not have access to the exact PT process. Therefore, the distribution of PT dataset, the loss potential, and the PT training amount usually remain unknown.
Unknown PT Training Amount and Loss Potential For the open-source models, we do not know the PT training amount and loss potential to get the PT forward area Spc and the final LR to calculate the CPT annealing area Syot For forward area STd we treat it as a parameter to be fitted. Typically, most open source models will anneal the LR to zero or a minium LR to get a better benchmark performance. We assume that the final LR of all open- source models is zero, which facilitates the computation of the CPT annealing area s.at The learning rate of CPT of open-source models is consider to we-aamup
18

---

Learning Dynamics in Continual Pre-Training for Large Language Models
4.0
3.8
Sedjenu 6824 Sedie:: d8S
PT Seepon 685-1124 1524
Validation Loss
Dept Validation Loss
Seg SK GB:
3.8-
Aur(S 92001 (585
Sealereas 055-4006
Sealerw4k 053#250
(orl5)
0.5811
117625
Seg crres 025
Segerez 085
3.4
3.6
Lon5t -
(xd5)
1.00
1144665 -2)-
3.2
3.0
D A4
3.2
2.8
Lor55 - 0.36
(375
13
3.0
5000
10000
15000
20000
25000
30000
35000
40000
5000
-0.8101
10000
15000
20000
25000
30000
35,00
40000
Step
Step
(a) Dpt (Fine Web) loss curve of different batch size.
(b) Dept (Knowledge Pile) loss curve of different batch size.
Figure 17. Using Eq. 4 fitted loss curve of different batch size in the continual pre-traininig. The smaller batch size is IM tokens with 4K sequence length and the larger batch size is 8M tokens with 8K sequence length. We annotate the different distribution shift terms in the figure.
from zero to the specific peak learning rate and then anneal with specific LRS.
Unknown PT Dataset Distribution The aforementioned equation holds only in the loss curve of Dat and Dept validation dataset. However, we do not know the exact Dpr dataset distribution of open-source models. In this case, we could select an open-source common crawl validation set as proxy Ppt-
To verify the two hypotheses mentioned above are reasonable, We conduct the experiments that continual pre-train the LLaMA3.2-1B (Dubey et al., 2024) with Pile-of-Law dataset (Henderson* et al., 2022). We consider the C4 portion of the RedPajama (Weber et al., 2024) dataset as proxy Dpt- We use fewer training steps to fit the parameters and then predict the loss of longer steps for the proxy Dpt and true Dept. As shown in Fig. 18, Eq. could effectively predict the further loss of proxy ppt and Dept- Based on the proxy Dpt and true Dept. we could describe the performance dynamics and complete the above hyper-parameters optimization for open-source models.
We also conduct experiments with our model pre-trained with Web and continual pre-trained with Pile-of-Law dataset. We treat it as a model with unknown PT information. We still use the portion of RedPajama as the proxy Dpr dataset to predict the loss of longer training steps as shown in Fig. 18.
H. Adding Replay Ratio to Our Formulation
D-CPT law (Que al., 2024) proposed a scaling law integrating with Dpt and Dept data mixture ratio. We have also integrated this data mixture ratio into our formulation. Appendix D demonstrates that loss curves for different data ratios be individually fitted using distinct equations. However, we are currently exploring a unified formulation that incorporates the data mixture ratio to represent all loss curves. Both the distribution shift term and the LR annealing term are influenced by the replay ratio. A higher Dpc ratio leads to weaker distribution shift, and results in a smaller LR annealing term in the Dept validation loss, while increasing the LR annealing term in the Dyt validation loss. We find that the exponential form, which is consistent with the Data Mixing Law (Ye al., 2024), best fits these effects and subsequently incorporate it into both the distribution shift term and LR annealing term:
Ttpt = Ta + A (SP" + STPA) -a - C1 Sgt - C2 . S2 da + B (1 (1 (1+ F. SPPU)
-8) -0) (1 (aaTepe))
(8)
Lept = Lo + A (sp* + STPL) -a C₁ Shh - C2 S e Tege + B. (1 (1- E ST -0) (e aaaten 1)
where rpt and Tept are the data mixture ratio of PT and CPT data respectively, such that Tpt + Tept 1, and and d2 are the additional parameters. To ensure that the distribution shift term is zero when Tept equals zero, we have modified the exponential formulation in the distribution shift term accordingly. The effectiveness of this equation is illustrated in Fig. 19, While the D-CPT law predicts only the final loss across different replay ratios, our method is capable of describing the entire training dynamics for various replay ratios.
19

---

Learning Dynamics in Continual Pre-Training for Large Language Models
2.00
Goord 11ah) Lo14
RedPajama-O jaaa-4 Loss
1.50
2.03
Fitted Predict LRS
RedPajama
Learning
1.00
0.75
2.75
seall -aall-0aay" +
0.50
2,7
20203
32000
40000
50000
1000
3000
5002
10300 12500 Step
Step
Step
(a) Learning rate schedule of fitted and pre- dict of Model I.
(h) Fitted RedPajama- C4 datasct loss curve of Model I.
(c) Predicted RedPajama-C4 dataset loss
curve of Model I.
4.4
2,00
Leaming x10
1.75
41
RedPajama-Ca C4 Loss
1.50
1.25
Fitted Predict LRS
RedPajama-CC
1.00
220--2222-2 3 (100s)
4.0
0.75
3.7
-euras"
3.5
3.5
3.8
0.5
Truch
2500
5050
7500
3000
12500 15300 37500 20000
2000
6.000
6003
8000
10000
30363
23000
30000
40300
50000
Step
Step
Step
(d) Learning rate schedule of fitted and pre- dict of Model
(e) Fitted RedPajama- C4 dataset loss curve of Model
(l) Predicted RedPajama-Ce C4 dataset loss curve Model II.
2.00
2,00
Tuth La3
Crand
1.75
1.75
3.75
Loss
Law Loss
1.70
125
Filted URS Predict LRS
uewaias Aaway a 380 5000
2.05
Lalt FRADUITTN -3 EXCC
Learning
0.75
Pile
Pile
3.55
0.50
1.50
0.2
3.45
0.00
10000
20:03
33000
46000
50000
1.0
1000
20:0
3000
4:00
5020
3.40
2500
5000
7500
15000 37500 20000 Step
10000 12500
Step
Step
(g) Learning rate schedule of fitted and pre- dict of Model I.
(h) Fitted Pile-of-Law dataset loss curve of Model I.
Predicted Pile-of-Law dataset loss curve of Model
Grunc Futh Rited Los Caree
Ground Truch Loss Prodic Loss Curve
Learning x10-4
1.75
Law Loss
Pile of Law Loss
2.0-
1.25
Fitted LRS Predict
2.0
Lut 2434 CaTLu Aag may
1.00
0.25
3.8.
0.0
3.7
0.00
2500
5050
7500
15000 12500 19505 19500 20000
1.7
2000
4000
6002
0000
10850
3.0
10563
20000
100.0
10000
50000
Step
Step
Step
(i) Learning rate schedule of fitted and pre- dict of Model II.
(k) Fitted Pile-of-Law dataset loss curve of Model II.
()) Predicted Pile-of-Law dataset loss curve of Model II.
Figure 18. Using Eq. 4 to fit and predict the proxy Dre and true Dept dataset of open-source PT models. The Model I refers to LLaMA3.2- IB. Model II refers to our model pre- pre-trained with Fine but we regard it as an unknown model and use proxy D,r rather than FineWeb. The eep dataset are both Pile-of-Law, and the proxy Dpt is RedPajama-C4.
20

---

Learning Dynamics in Continual Pre-Training for Large Language Models
I. Optimal Hyper-Parameter"
In the Fig. 8, we show the optimal hyper-parameters based on different coefficients. For optimal loss potential and peak LR, we use FineWeb as Dut and and explore three different Dept dataset: (1) Pile of Law, (2) Knowledge Pile, and (3) a mixture of 67% FineWeb and 33% Knowledge Pile. These three Dept datasets have strong, moderate, and weak distribution shifts comparing to Dpt- The training setting for each Dept dataset is consistent with the Setting A and Setting B in Table 1. The LRS used for fitting these three Dept datastes and the fitted equation coefficients are shown in Fig. 11 11, Fig. 3 and Fig. 12. We directly use these three sets of coefficients to search the optimal saper-parameters.. In the Fig. 8a and Fig. 8b, we assume that the LRS for the PT phase follows WSD schedule with 40k steps, while the continual CPT phase employs cosine schedule with 10k steps, consistent with the configuration shown in Fig. 1la
For optimal replay ratio, we use Web as pt and Knowledge Pile as Dept. In Fig. 8c, we maintain the assumption that the PT phase employs WSD scheduling while the CPT phase uses cosine scheduling with varying CPT step counts. The blue dashed reference line represents the scenario where the target weight (A1) equals the replay ratio. It can be reasonably assumed that if the model were initialized from scratch (rather than from pre-trained model), the optimal replay ratio curve would follow the blue dashed line. However, since our model is pre-trained, this causes the curve to deviate and exhibit a wavy pattern. We directly apply the fitted coefficients presented in Fig. 19 to determine the optimal replay ratio.
J. Other Formats with LR Annealing
Similar with scaling law with LR annealing (Tissue al., 2024), also try the other possible forms of LR annealing.
Adding a Le-weighted Coefficient To solve that when LR anneals to nearly 0, S2 still has historical momentum, making the loss continue to decrease. A revision is that adding a L--weighted coefficient to Sy:
S2 m, nn
(9)
We test the coefficient is 0.1 and 0.2, showing the fitted result in the Fig. 20.
S2 Power Formats Considering that the annealing loss and S2 have a positive correlation, L x Sh might be a more reasonable format than L x S2. We revise our formulation:
L = Lo + A. (SP + 5TPt)) - C1 -C-- - C2 (S
(10)
+ B (1 (1- (1 F. ST" -P)
We add two other fitted parameters in the function for different annealing area of PT and CPT. We also show the fitted effect in the Fig. 20. We show the huber loss and R² of all possible formats in the Table 2. All the fitting effect are really good, but the original format has the fewest parameters which is more effective.
K. Out-of-Domain Validation Set
Data mixing law (Ye et al., 2024) shows that validation loss for some domains can be represented by combination of other domains. In scenarios where the CPT dataset, such as Knowledge-Piie is not highly domain-specitn employing linear combination of Dp, and Dep can serve as a reasonable approximation for certain downstream validation sets. However, it is important to note that this approach may not be universally applicable across all CPT datasets and all Dood validation loss scenarios. We test the validity of using a linear combination of Dpr (FineWeb) and Dep Scrowledge-Piley to estimate the validation loss for certain out-of-domain sets in the Fig. 21 and Fig. 22. These out-of-domain validation sets include StackExchange, arXiv, and C4 in RedPajama (Weber et al., 2024), as well PhilPapers and Books in Pile (Gao et al., 2020). SlimPajama (Soboleva et al., 2023), and Open-Web-Man Web-Math (Paster et al., 2023).
21

---

Learning Dynamics in Continual Pre-Training for Large Language Models
3.6
3.6
CPT Fith Leds
CFT Thuh USS
Validation LoSs
Dept Validation Loss
3.4
Losa
3.0
3.3
2.8
1 3.2
30000
35000
40000
45000
50000
55000
60000
30000
35000
40000
45,000
50000
55000
60000
Step
Step
(a) Dpc loss curve of different replay ratios.
(b) Dept loss curve of different replay ratios.
Figure 19. Using Eq. 8 to fitted all loss curves of different replay ratio the continual pre-training The fitted equation is Lpt
3.067 + 0.480 (S + STOT) -0.10 0.280 S 0.263 S.pt 0.055r + 0.276 (1 99.99.93 ST0e) -P (1 e- -32238repd) and
Teppe = 2.992+0.456- (S STaR)
-0.510 0.285-85555 ST* 0.279-- 2.037ry 0.526 (1 100.34 ST9t) -P) (e 5.000rr 1).
Table 2. The litting effect of different possible equation formats.
Dpt
Huber Loss
R² f
Original
0.0016
0.9944
Adding L.R Cofficient (E (== 0.1)
0.0016
0.9950
Adding LR Cofficient === 0.2)
0.0017
0,9950
Adding S₂ Power
0.0016
0.9952
Dept
Huber Loss
R² f
Original
0.0021
0.9993
Adding LR Cofficient (e 0.1)
0.0025
0.9984
Adding LR Cofficient (e 0.2)
0.0024
0,9983
Adding S2 Power
0.0025
0.9984
22

---

Learning Dynamics in Continual Pre- Pre-Training for Large Language Models
Wil Poctiais caine Tuis tree
344
150 Canti ProTuking Los
x10%
1.75
3.7
wil Concrnd Poola ding
wib Pichi Fiece Lnt
Validation Loss
3.3
Continue Prchetiing Tiind
32
1.25
Dot Validation
2.2- 023007 10320100----1000 -pt e 5403 2065 I1-i
401-2292- EAtLER varyy -82000 1.006
1.00
3.4
28
0.0
27
Pre-PainingLA
Continaa Pre-tmiriin LBS
2.6
0.00
10000
20000
32000
40000
50000
10000
33
10000
20003
10303
<0000
50350
63020
25
15050
70200
12000
40000
50000
r0COn
Step
Step
Step
(a) Learning Rate Schedule used in fitting adding LR- weighttt coefficient = 0.1.
(b) Dzt Validation Loss for adding LR- (c) Dept Validation Loss for adding LR- weighted coefficient = 0.1.
weighted cofficient 0.1.
2.00
West miri saring rusl seet Contrva he-harnn nin Wes contived in
34
Continul Frchhaiing 1no
3.7
Dex Validation Loss
X
1.50
Dxt Valldation
3.5
3.0
1.00
2.9
2,222
2.8
0.0
2.7
Continaal Pre Tuiring Lts
Pro-haiting
3.2
28
0.00
30000
Step
Step
Step
(d) Learning Rate Schedule used in fitting adding LR- weighted coefficient 0.2.
(c) Dpt Validation Loss for adding LR- weighted cofficient 0.2.
(f) Dept Validation Loss for adding LR- weighted cofficient 0.2.
3.6
Predral ny 655 Precral Rind ices
x10
2.00
wao Cononel he ting Tar
Validation Loss
WGD Contrnal
2.0
32
Validation
caded""
3.5
1.00
3.0
-453011
-1005" Linter
2.9
2.
2.7
0.25
32-
2.6-
33000
40000
10000
30000
40000
63000
15000
20300
35000
50000
Step
Step
Step
(g) Learning Rate Schedule used in fitting adding adding power format.
(h) Dp Validation Loss for adding Sa power format.
Dept Validation Loss for adding S2 power format.
Figure 20. Using other possible S2 formats Eq. 4 to fit all PT and CPT loss curve with different LRS.
23

---

Learning Dynamics in Continual Pre-Training for Large Language Models
StackExchange
Open-weD- -a---maaaa
Arxiv
Books
0,5460,z
1,6 0,60MO,
0.07.0,
Pile- Pile-PhllPapers
SlimPajama
Stories
3
20 o.
or 0.0229
-0.5100 1.4090
2.505 0.1780,
Figure 21. The linear combination of Dp and Dept to represent the out-of -o--min Dood validation loss.
0.0030
0.0025
Absolute Errors
0.0020
0.0015
g
0.0010
0.0005
0.0000
StackExchange Arxiv
Books
Math
SlimPajama Stories
PhilPapers
Figure The absolute errors linear combination of Dpt Dept to represent the out-of-doamin Dood validation loss.
24


Reviews:
[{"review_id": "SxItrhmK3q", "reviewer": "Reviewer_nT6s", "summary": "The paper explores learning dynamics in Continual Pre-Training (CPT) for LLMs, focusing on how general and downstream domain performance evolves at each training step, using validation losses. The paper observes that the CPT loss curve represents a transition between hidden loss curves, influenced by distribution shift and learning rate annealing. The paper empirically derives a CPT scaling law combining these factors, enabling loss prediction at any training step and across learning rate schedules in CPT. This scaling law sheds light on key CPT factors, including loss potential, learning rate, training steps, and replay ratio. The proposed approach enables customizing training hyper-parameters for different CPT goals, such as balancing general and domain-specific performance. Lastly, the paper extends the scaling laws to more complicated scenarios such as out-of-domain datasets and models with unknown pre-training details.\n\n## update after rebuttal\n\nMost of my concerns were addressed in the rebuttal response. I hope remaining clarity issues are fixed in the revision. Considering all feedback, I stand by my score and recommend acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work.   \n**Response to \"Limited theoretical foundation for the derived scaling laws\":**  \nOur work is indeed empircial. However, most scaling laws in LLMs are empirical, such as the OpenAI [1] scaling law and the Chinchilla [2] scaling law. We believe that it is common practice in developing scaling laws of LLMs.\n\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).  \n[2] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\" arXiv preprint arXiv:2203.15556 (2022). \n\n**Response to \"Smaller model scales (0.1-1B parameters)\":**  \nThe largerest model size in our experiment is 1.7B parameters without embedding, and we also conduct experiments with LLaMA-3.2-1B in Appendix G. We believe this model size is reansonable in the current context of large language models. Due to resource constraints, we are indeed unable to train larger models. Our scaling law has demonstrated consistent patterns across these model sizes (0.1-1.7B), so we believe our law is also applicable for larger models.\n\n**Response to \"Comments Or Suggestions 1\" and \"Q1\" about \"Loss Potential\":**  \nWe apologize for any poor reading experience caused by organizational issues about these terms in our paper and will address these concerns in the final version.     \nFor loss potential, we define in line 187 \"captures the potential of future loss drop via LR annealing.\", which corresponds to the height in Figure 4(c). For example, if the pre-training implements substantial annealing and end at the valley bottom in Figure 4(c), the remaining annealing room or potential for CPT is diminished. Quantitatively, we define loss potential as the ratio of the final learning rate of the pre-training annealing phase to the initial or maximum learning rate in the pre-training phase: $ loss\\ potential = \\frac{Final\\ LR\\ of\\ PT}{Maximum\\ LR\\ of\\ PT} $, which is same as the legend in our Figure 5(b)(e) and x-axis of Figure 5(c)(f). For example, in Figure 5(a)(d), if the pre-training model anneals from 2e-4 to 1e-4 using a linear method, the loss potential for this model is 50%.  \n\n**Response to \"Comments Or Suggestions 2: typo in figure 5\":**  \nWe apologize for the typo in the legend of Figure 5(a); the correct value is 10% loss potential, not 0%. We will revise it in the final version.\n\n**Response to \"Q2: loss potential is relative to D_pt validation loss or D_cpt validation loss\":**  \nBased on the defination of $loss\\ potential$ above, the loss potential is neither related to D_PT validation loss nor D_CPT validation loss but rather represents the annealing potential of the models. Our findings indicate that pre-trained models with higher loss potential can achieve lower D_CPT validation losses, suggesting that these models could better adapt to the downstream datasets.", "claims_and_evidence": "The paper’s claims are well-supported by the provided evidence.\n\n1. A CPT scaling law (Equation 4) is derived and validated by decomposing the CPT loss curve (Figures 1, 2). Experiments (Figure 3) demonstrate effectiveness for small models (0.1-1B parameters) across various training settings, though generalizability to larger models is a concern.\n\n2. The paper analyzes key CPT factors (loss potential, distribution distance, learning rate, CPT steps) and their impact (Section 4, Figures 5-7).\n\n3. Figure 8 demonstrates the approach’s adaptability for customizing hyperparameters to specific CPT goals, with more detailed discussions in Section 5.\n\n4. The scaling law is shown to extend to OOD datasets (Subsection 5.2, Figure 9) and open-source models with unknown training details (Appendix G). \n\nIn summary, the paper provides substantial empirical support for its claims, including the empirical scaling laws. The analysis is detailed, and limitations regarding model scales are acknowledged.", "methods_and_evaluation_criteria": "Based on the paper, the proposed methods and evaluation criteria make sense for the problem and application at hand. The reason being:\n\n1. The paper analyzes and models learning dynamics in CPT of large language models, focusing on performance evolution across general and downstream domains during CPT, analyzing catastrophic forgetting.    \n\n2. The paper introduces a CPT scaling law combining distribution shift and learning rate annealing to predict loss at any training step and across learning rate schedules, modeling key CPT factors like loss potential, learning rate, training steps, and replay ratio.    \n\n3. The paper uses validation loss of corresponding domains to trace performance changes, a standard practice in language modeling and continual learning, and validates the scaling law using different learning rate schedules, datasets, and model sizes.    \n\n4. The paper uses standard datasets like FineWeb and Knowledge-Pile, enabling comparisons and ensuring relevance.", "theoretical_claims": "The paper derives scaling laws inspired by existing works and validates them by fitting all (continual) pre-training loss curves with different learning-rate schedules. So it is mainly an empirical paper.", "experimental_designs_or_analyses": "The paper generally employs sound experimental designs. The paper considers multiple setups with variations in training settings—learning rate schedules, model size, continual pre-training datasets, and hyperparameters—to test the robustness of the CPT scaling law. The paper uses established datasets (Fineweb, Knowledge-Pile) and standard evaluation metrics (validation loss). The analysis of the CPT loss curve and the validation of the scaling law through fitting empirical loss curves are appropriate (with details about the fitting procedure in Appendix C).\n\nHowever, a key limitation is the lack of rigorous theoretical analysis and proof for the CPT scaling law, as acknowledged in the paper. The conclusions regarding model size are also based on assumptions, as the experiments did not reach the scale of current large language models and are limited to 0.1-1B parameter models.", "relation_to_broader_scientific_literature": "The paper analyzes and models the learning dynamics in Continual Pre-Training (CPT) of large language models, especially understanding how the performance of these models evolves across general and downstream domains during the domain-specific adaptation process. This is a critical issue in continual learning, as models often suffer from catastrophic forgetting, where they lose previously learned information when learning new tasks. So, identifying key operating points like optimal replay ratios, starting loss potentials, and learning rate schedules is very important. Although some of these details are well-known and intuitive, the paper takes a stab at defining them, deriving scaling laws for the same, and employing them even in more realistic scenarios like open-source pre-training language models with unknown training details. This work sets a stage for subsequent research for scaling to larger model scales during continual pre-training.", "essential_references_not_discussed": "The paper is organized well, citing all relevant prior work and detailing how the proposed work draws inspiration from and differs from existing research. To my knowledge, the paper does not omit any important related works or works that would hinder an understanding of the current paper.", "other_strengths_and_weaknesses": "The paper addresses the important and timely problem of understanding continual pre-training, a prevalent technique where a generic pre-trained model is further pre-trained with a domain-specific corpus for particular use cases. The paper is well-written, clearly positions itself within existing work, is easy to follow, and includes extensive experimentation to support the claims.\n\nAs discussed previously, two main weaknesses are the paper’s focus on smaller model scales (0.1-1B parameters), raising questions about the generality of findings to larger models, and the limited theoretical foundation for the derived scaling laws, as they are primarily empirical. However, the latter does not constitute a reason to reject the paper.", "other_comments_or_suggestions": "1. It would help to formally define “loss potential” for better readability and understanding of this introduced concept.\n\n2. In Figure 5, there are few legends that are 0% loss potential while some are 10% loss potential. Is there a typo somewhere?", "questions_for_authors": "1. What do 0-100% loss potential values mean? What is the baseline used to compute this percentage?\n\n2. In Finding 3, the paper notes that “PT models with higher loss potential always achieve lower $D_{cpt}$ validation losses…”. However, \"loss potential\" is relative to $D_{pt}$ validation loss or $D_{cpt}$ validation loss. It is unclear from the context which loss is being referred to in this finding.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tqSpu6iJU5", "reviewer": "Reviewer_oDDW", "summary": "This paper performs an empirical study of loss curves during continual pre-training. The paper seeks to characterize how the training loss on a new dataset will evolve when a pretrained model is subjected to new data in a CPT setup. To that end, the authors present a series of experimental setups showing how loss behaves in different scenarios. CPT loss curve is shown to be decomposable into previously characterized scaling law with learning rate annealing and a novel power-law term for capturing distribution shift between pretraining and continued pre-training data domains. The authors show in many setups that the laws proposed fit nicely the empirical data. The authors discuss the formulation and fitting of the laws.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your suggestions.  We would also like to acknowledge the thorough and constructive feedback provided which help strengthen our work, which we summarize and respond as follows.\n\n**Response to \"Our work is empirical\":**     \nOur work is indeed empirical. However, most scaling laws in LLMs are empirical, such as the OpenAI [1] scaling law and the Chinchilla [2] scaling law. We believe that it is common practice in developing scaling laws of LLMs.\nMoreover, our scaling law can be applied to many practical scenarios\nWe have conducted extensive experiments to demonstrate the validity of our empirical law, including different learning rate schedules in Figure 3 and Figure 10, various model sizes in Appendix E, and different CPT datasets or replay ratios in Appendix D.\n\n**Response to \"No baselines\":**   \nWe are the first to propose the CPT law that traces the learning dynamics of LLMs throughout the CPT process and we model the full loss curve during CPT. Our CPT law considers numerous variables that affect CPT performance, which was not addressed in previous works. Consequently, we cannot compare the accuracy of fitting or prediction with other baselines due to the absence of comparable approaches. Nevertheless, to validate our CPT law, we have compared it with alternative law formulations as baselines, as detailed in Appendix I.\n\n**Response to \"Contribution of laws\", \"Why this may be useful\", \"What problem may be solved\", \"No extant articulated problem\" and \"Questions: what is a practitioner who reads this work and is now familiar with the proposed 'laws' to do differently?\":**  \nAlthough our scaling law is empirical, it could help researchers understand the dynamics of continual pre-training, as noted by several reviewers. Reviewer nT6s states: \"This work sets a stage for subsequent research for scaling to larger model scales during continual pre-training.\" Similarly, Reviewer 5BNQ observes: \"Overall, this is an educational paper to understand how continual pre-training works,\" while Reviewer oHxM acknowledges: \"These contributions are unique and not seen in prior works.\" Our scaling law especially assists researchers in understanding the impact of various hyperparameters on the complete learning dynamics. More specifically, our scaling law could fit by very few CPT data points and accurately predict the loss across diverse hyper-parameters (e.g. LR schedule) as shown in Figure 10, which enables to efficiently optimize the training hyperparameters (Figure 8) without extensive search procedures, thereby conserving computational resources.\n\n**Response to \"Findings seems to be a series of uncontroversial statements which are so general as to be obvious\":**  \nWe believe that some findings in our paper may not be so obvious to certain researchers. Our scaling law validates these obvious findings, which further demonstrates the correctness of our scaling law and reliability of these findings. Most importantly, our scaling law not only provides the qualitative analysis about these findings but also enables quantitative analysis for each finding. For example, it can predict the optimal loss potential, peak LR, and other hyperparameters through the scaling law as shown in Figure 8.\n\n**Response to \"Loss Potential\" and \"Turning Length\":**  \nWe apologize for any poor reading experience caused by organizational issues about these terms in our paper and will address these concerns in the final version.     \nFor loss potential, we define in line 187 \"captures the potential of future loss drop via LR annealing.\", which corresponds to the height in Figure 4(c). For example, if the pre-training implements substantial annealing and end at the valley bottom in Figure 4(c), the remaining annealing room or potential for CPT is diminished. Quantitatively, we define loss potential as the ratio of the final learning rate of the pre-training annealing phase to the initial or maximum learning rate in the pre-training phase: $ loss\\ potential = \\frac{Final\\ LR\\ of\\ PT}{Maximum\\ LR\\ of\\ PT} $, which is same as the legend in our Figure 5(b)(e) and x-axis of Figure 5(c)(f). In Figure 5(a)(d), if the pre-training model anneals from 2e-4 to 1e-4 using a linear method, the loss potential for this model is 50%.  \nFor turning length, we define it in line 315 as \"The minimum training steps required to achieve a lower loss are designated as the turning length.\" We also illustrate the meaning of this term in Figure 7(c). Qualitatively, when the CPT steps once reach the turning length, the validation loss on pre-training dataset reverts to the same loss as the very beginning of CPT as shown in Figure 7(c).  \n\n\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).  \n[2] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\" arXiv preprint arXiv:2203.15556 (2022).", "claims_and_evidence": "The claims of the authors are limited to the well-fittedness of their law-formulation to their empirical data. The data presented seem clear enough. The authors make scant claims of the effectiveness of their method compared to any baseline, and the claims they do make are not substantiated with experimentation, but rather exist as almost side-comments.", "methods_and_evaluation_criteria": "The paper does not really present baselines as it does not really present experiments. There are no figures or tables showing comparison to any other method. Rather, the paper presents empirical findings showing the proposed law fits nicely the data drawn from the various experimental setups.", "theoretical_claims": "No theoretical claims present. The authors note this in limitations.", "experimental_designs_or_analyses": "This is a peculiar paper in that it does not really have experiments per se. The authors present empirical results on the setups they designed and show that their 'law' formulation is a good fit for the empirical data. But they do not apply this 'law' towards solving any extant articulated problem; there is no 'baseline' against which a novelty is compared. A stronger version of the paper might say something like \"this allows us to more efficiently find CPT hyperparams which lead to good performance\" and then show experiments against a baseline. Another potential experiment would be fitting the laws on some setup and extrapolating the fit laws to other setups, showing generalization across datasets or models.", "relation_to_broader_scientific_literature": "CPT is a currently topical method for analysis, as it is widely in use with LLMs. The 'laws' proposed in this work follow a general trend of identifying power laws in various machine learning settings to help make predictions about model performance, oftentimes with the goal of restricting the hyperparameter space of training and thus increasing the efficiency of producing useful models. This work evaluates laws which seem to fit nicely the data presented, though the contribution of those laws towards improving efficiency is left unstated and unexplored.", "essential_references_not_discussed": "Not to this reviewers limited knowledge.", "other_strengths_and_weaknesses": "Clarity leaves much to be desired. The first of two \"Research Questions\" in the intro reads \"can we find such an accurate law containing as many variables that affect the final CPT performance as possible?\". This can be greatly improved. Notions 'loss potential' and 'turning length' are used before being explained in the introduction. Throughout, concepts are poorly articulated, making unclear in sections what the authors are attempting to communicate.\n\nSignificance is somewhat unclear. It may be interesting that the \"laws\" described fit the data nicely, but it is left unstated as to why this may be *useful*. There is not a clear statement of what problem may be solved by the description of the proposed \"laws\" and accordingly there is no experiment which demonstrates that problem as baseline and some methodological solution. Some of the \"Findings\" are of questionable value. #5 seems to be a series of uncontroversial statements which are so general as to be obvious.\n\nMany works already exist fitting power laws to various aspects of machine learning, so originality of the meta-methodology is limited. The presented empirical analysis seems reasonable, if limited. The utility of the method is unclear.", "other_comments_or_suggestions": "The fundamental issue with this work is that it is a empirical study with neither theoretical analysis or demonstrated pragmatic value. The authors propose a law for continual pre-training loss evolution and show that this law fits nicely, but do not explicate or demonstrate why anyone might want to know about it, or how anyone might make use of it. The paper reads almost like an extended analysis section, missing experiments comparing to any baseline or a framing of the method which answers the question 'so what?'. For what the paper is, the analysis seems sound and the law seems to fit the data well. Whether or not this alone is sufficient for publication is dubious. It seems addressing the above concern could feasibly produce a much stronger presentation of this work.", "questions_for_authors": "In the authors view, what is a practitioner who reads this work and is now famliar with the proposed 'laws' to do differently? What problem is being solved, and what evidence is there that is has been?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsT8UBVgLc", "reviewer": "Reviewer_5BNQ", "summary": "This paper explores the learning dynamics of continual pre-training and proposes scaling laws for the same. Based on the proposed scaling laws, the authors discuss several critical factors in continual pre-training. Overall, this is an educational paper to understand how continual pre-training works.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful suggestions and valuable feedback.\n\n**Response to \"Comments Or Suggestions 1 about loss potential and turning length\":**    \nWe apologize for any poor reading experience caused by organizational issues about these terms in our paper and will address these concerns in the final version.     \nFor loss potential, we define in line 187 \"captures the potential of future loss drop via LR annealing.\", which corresponds to the height in Figure 4(c). For example, if the pre-training implements substantial annealing and end at the valley bottom in Figure 4(c), the remaining annealing room or potential for CPT is diminished. Quantitatively, we define loss potential as the ratio of the final learning rate of the pre-training annealing phase to the initial or maximum learning rate in the pre-training phase: $ loss\\ potential = \\frac{Final\\ LR\\ of\\ PT}{Maximum\\ LR\\ of\\ PT} $, which is same as the legend in our Figure 5(b)(e) and x-axis of Figure 5(c)(f). In Figure 5(a)(d), if the pre-training model anneals from 2e-4 to 1e-4 using a linear method, the loss potential for this model is 50%.  \nFor turning length, we define it in line 315 as \"The minimum training steps required to achieve a lower loss are designated as the turning length.\" We also illustrate the meaning of this term in Figure 7(c). Qualitatively, when the CPT steps once reach the turning length, the validation loss on pre-training dataset reverts to the same loss as the very beginning of CPT as shown in Figure 7(c).  \n\n**Response to \"Comments Or Suggestions 2 about Figure 13\":**   \nWe will implement appropriate revisions for Figure 13 in the final version of the paper.\n\n**Response to \"Q1: Why is models with higher loss potential achieve lower final losses\":**   \nThe statement \"models with higher loss potential achieve lower final losses\" indicates that pre-training models with higher loss potential can attain lower CPT validation losses and demonstrate superior adaptation to the CPT dataset. \nThe intuitive understanding is that models with higher loss potential can allocate more annealing to the CPT phase, which enable to achieve lower D_cpt validation loss. The more specific reason of this finding is explained in line 255-265 in our paper. This finding is substantiated by the truth training loss trajectories of different loss potential models, as illustrated in Figure 5(b) and 5(e), as well as by the predicted loss values derived from our CPT law, as depicted in Figure 5(c) and 5(f). \n\n**Response to \"Q2: What do you mean by releasing the high loss potential version\":**  \nBased on the above explanation, the models with higher loss potential could better adapt to the CPT dataset. However, current open-source models, such as Qwen or Llama, typically employ extensive learning rate annealing that reduces the learning rate to near-zero or minimal values, resulting in low loss potential. If releasing high loss potential variants of these models specifically (i.e., those trained without learning rate annealing), researchers and practitioners could adapt these models for CPT or downstream tasks.\n\n**Response to \"Q3: Explain Finding 4\":**   \nThe turning length is affected by: (1) the distribution distance between PT and CPT data; and (2) the sufficiency of pretraining. If the distribution distance is large or the pre-training is sufficient, the turning length becomes larger and even to potentially infinite. The detailed explanation is in the line 308-315 and Figure 7(c). If you have further questions about the Finding 4, feel free to further comment.\n\n**Response to \"Q4: How do you set lambda_1 and lambda_2 in equation 5?\":**     \nIt can be considered in two scenarios. \nIn some situations, we could allocate a percentage ratio for validation losses on PT and CPT datasets based on our prior knowledge with the importance between general and downstream performance. \nOtherwise, for a specific test set that we focus and optimize, we could always precompute the linear coefficients based on dataset similarity, as demonstrated in Section 5.2.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no proofs in the paper. There are several claims about the scaling laws which are derived from other works. I did not check the correctness of the scaling laws since I am not an expert in scaling laws. I will let other reviewers judge this aspect.", "experimental_designs_or_analyses": "Yes. The experimental designs in this paper are well thought out. There is a lot of thorough analysis.", "relation_to_broader_scientific_literature": "The authors have done a good job of relating the work to the existing works.", "essential_references_not_discussed": "Nothing I am aware of.", "other_strengths_and_weaknesses": "This paper explores the effect of several hyperparameters including learning rate, replay ratio during continual pre-training. Overall, this is a great educational paper to learn more about the learning dynamics of CPT.", "other_comments_or_suggestions": "1. \"loss potential\" and \"turning length\" are used in the introduction without proper definition of them. So it is hard to understand the last paragraph of section 1.\n\n2. In page 7, for Fig 13, it is worth mentioning that the figure is in the appendix.", "questions_for_authors": "1. In page 5, you mention \"models with higher loss potential achieve lower final losses\". Why is that the case?\n\n2. Finding-3 is not clear. What do you mean by releasing the high loss potential version?\n\n3. Finding-4 is also not clear to me. Can you please explain it?\n\n4. How do you set lambda_1 and lambda_2 in equation 5? You mention that it is based on practical requirements. But it is not clear what these requirements could be.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NuDvEzzhpx", "reviewer": "Reviewer_oHxM", "summary": "This paper studies the training dynamics during continual pre-training. Specifically, they study the scaling law of the loss curve during continual pre-training (CPT), which considers the scaling law due to learning rate annealing and the scaling law due to the distribution shift. Based on this formulation, they consider several factors in CPT, including loss potential, distribution shift between pre-training and CPT dataset, peak learning rate, and CPT steps. They also discuss how the proposed scaling law can be accurately fitted for open-source models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful suggestions and valuable feedback.\n\n**Response to \"Claims and Evidence about sentences that are hard to understand\":**   \nWe apologize for misunderstanding caused by our use of the term \"overlap.\" Our intended meaning was that the curves from different transfer starting points \"coincide\". In the subgraph of Figure 2, we show the distribution shift curves of different transfer starting points and the horizontal axis represents the number of steps in CPT. These distribution shift curves almost \"coincide\" and could be fitted with a power-law form.\n\n**Response to \"Choosing Knowledge-Pile\":**  \nFirst, we do not exclusively utilize Knowledge-Pile as the CPT dataset. We also employ Pile-of-Law [1] as a domain-specific CPT dataset and demonstrate the fitting and predictive capabilities of our scaling law in Appendix D. Second, the data in Knowledge-Pile [2] is derived and filtered from existing data sources, such as Wikipedia, arXiv, Semantic Scholar, without any atypical components. From the components [2], Knowledge-Pile is a good combination with higher STEM knowledge density, which can be used to enhance domain-specific abilities. Third, Our law is independent with the CPT dataset. When the CPT dataset distribution or quality change (better or worse), the form of our law remains applicable while the coefficients differ.\n\n**Response to \"Weakness\":**   \n**W1: \"LRS\" and \"WSD\"**:  \nWe apologize for the reduced readability caused by the abbreviation of certain concepts in Figure 1. LRS refers to learning rate schedules, and WSD [3] denotes the Warmup-Stable-Decay schedule. We will implement these clarifications in the final version.  \n**W2: Explain \"transfer curve\"**:  \nThe term \"transfer curve\" is actually the CPT curve. As stated in line 96, \"the CPT loss curve is a transitional curve on both D_pt and D_cpt validation set.\" The CPT curves in Figure 1 (b), (c), (e), and (f) illustrate that the curves deviate from the blue dashed line, and finally align with the orange dashed line.  \n**W3: Explain \"transfer point\" and \"Hidden Pre-training\"**:  \nThe \"transfer starting point\" represents the starting step point of CPT. We formally define the \"Hidden Pre-training\" curve in Section 2.2 and represent these trajectories with dashed lines in Figure 1.  \nWe acknowledge that the definations of some terms were not placed appropriately, and we commit to refining these definitions in the final version.  \n**W4: Small font size**:   \nWe apologize for the small font size in the figures and we will increase the font size in the final version.  \n**W5:  Explain \"loss potential\"**:  \nIn the final version of the manuscript, we will incorporate a comprehensive description of the \"loss potential\" concept within the introduction section to establish a clearer theoretical foundation from the outset.\n\n**Response to \"Comments and suggestions: The wrong caption in Figue 5 (b)(c)(e)(f)?\":**  \nWe sincerely apologize for unclear captions of Figure 5 due to our intention to save space during the writing process.\nThe complete caption in the Figure 5 is:  \n(b) D_cpt true loss v.s. CPT step of different loss potentials (w/o re-warmup setting)  \n(c) D_cpt predicted loss v.s. loss potentials of different CPT steps (w/o re-warmup setting)  \n(e) D_cpt true loss v.s. CPT step of different loss potentials (w/ re-warmup setting)  \n(f) D_cpt predicted loss v.s. loss potentials of different CPT steps (w/ re-warmup setting)  \n\n\n**Response to \"Questions: is the hidden pre-training the second epoch of the same pre-training data?\":**  \nNo. The hidden pre-training is not from the second epoch. it is based on the different data from the same distribution. For example, in our experiment, we continually pre-train the model on the remaining PT dataset Fine-web.\n\n\n[1] Henderson, Peter, et al. \"Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset.\" Advances in Neural Information Processing Systems 35 (2022): 29217-29234.  \n[2] Fei, Zhaoye, et al. \"Query of cc: unearthing large scale domain-specific knowledge from public corpora.\" arXiv preprint arXiv:2401.14624 (2024).    \n[3] Hu, Shengding, et al. \"Minicpm: Unveiling the potential of small language models with scalable training strategies.\" arXiv preprint arXiv:2404.06395 (2024).", "claims_and_evidence": "I do not find all the claims to be supported by the evidence. However, I believe this is somewhat related to the presentation issue of the paper, making it hard for me to know what the paper is saying. For example, on page 3, the paper writes, \" As shown in Fig. 2, these distribution shift terms tend to overlap at each transfer starting point.\" I am not sure where the overlap referred to here is. In this case, it would be hard to evaluate how well-supported this claim is.", "methods_and_evaluation_criteria": "This paper uses FineWeb for pre-training and Knowledge-Pile for CPT. Using FineWeb as a pre-training dataset is reasonable. However, only using Knowledge-Pile as the CPT data is a weakness of this paper. CPT can be conducted on coding or mathematical datasets to improve the specific ability. It is unclear whether the observation and conclusion in this paper hold for CPT datasets in other domains or different datasets. Using Knowledge-Pile may also be questionable since this is not a dataset that the research community has a consensus on its quality, at least judging by the review from ICLR 2025.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experiments use multiple models of the same architecture with different sizes. The issue with the evaluation is mostly related to the choice of the CPT dataset, which I elaborated previously.", "relation_to_broader_scientific_literature": "This paper proposes a scaling law for CPT, which has not been discussed in prior literature. They also discuss how the proposed scaling law can help understand the contribution of different factors in CPT and how to choose the best hyperparameters. These contributions are unique and not seen in prior works", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n==\nI found this paper to be quite interesting, and it provided many useful and intriguing observations. The findings highlighted in this paper should be of interest for researchers in the community\n\nWeakness\n==\nThe key weakness of this paper, and the reason I am currently leaning toward rejecting it, is the presentation issue, which makes the paper difficult to read.\n - Acronyms are not properly explained. For example, the term \"LRS\" is never explained. The term \"WSD\" is not clearly explained when it first appeared in the paper.\n  - The term \"transfer curve\" is not explained. This is not a formal term in math, so I am not sure what the author is referring to when using this term.\n   - The explanation of Figure 1 is not sufficient when it is presented. In Line 107 and the \"Observation\" in Section 2.1, the paper refers to Figure 1 without explaining many details, including the LRS, WSD, hidden pre-training, and transfer point. It is not easy for the reader to understand this figure.\n   - The font size in many figures is too small to read. The legends in Figures 1, 2, and 3 are completely unreadible when printed on an A4 paper.\n   - The term \"loss potential\" appears in the abstract and introduction but is explained in Section 3.3", "other_comments_or_suggestions": "- Is the caption in Figue 5 (b) (c) (e) (f) wrong?", "questions_for_authors": "- In Figure 1, is the hidden pre-training the second epoch of the same pre-training data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ### Summary

This paper derives scaling laws for continual pretraining (CPT) scenarios, leveraging the work of Tissue et al (2024) to account for learning rate schedulings. They analysis the loss shift attributed to:
- distribution shift between PT and CPT domains
- change of learning rate (w/ and w/o decay, w/ and w/o warmup)

Important observations:
- the important finding that after a critical point, the moment where the PT->CPT transition is done has little to no impact, because there is a "hidden CPT training curve" that is reached
- the "turning length" for PT -> CPT transition done before critical point
- even when PT domain is not known, and generic PT dataset (e.g SlimPajama) can be used as a good proxy

New tool:
- the "loss potential" quantity that allow to predict how much improvement can be expected by further training, as function of last LR value
- for OOD domains, the law can be applied by finding coefficients lambda1 and lambda2 and express the new loss as linear combination of PT/CPT losses, regression lambda coefficients

### Rebuttal and discussion

Reviewers oHxM and oDDW mainly complained about a lack of clarity, to which authors answered in a satisfactory manner, and committed to improve the final version of the manuscript.

Reviewer nT6s noticed a lack of theoretical foundation, but said "However, the latter does not constitute a reason to reject the paper." to which I agree. They also noticed that the scale was limited to 1B model - which is already consequent.  

Reviewer oDDW wrote: "The fundamental issue with this work is that it is an empirical study with neither theoretical analysis or demonstrated pragmatic value". I disagree and believed the authors did a good job in answering this point. The paper validates many intuitive findings, but also brings new insights, that have an obvious pedagogical value.

Besides that, the extensive experiments, the rigor, were all praised by reviewers. 

### Recommendation

The paper has an obvious pedagogical value. Findings are new, timely, relevant. Experiments are rigorous. Message is conveyed in a clear manner.

If anything, the paper suffers from too many messages and findings, and some concepts were insufficiently explained (e.g turning length and loss potential); but the authors promised to improve the writing in the camera-ready version.  

Overall, I would recommend the reading of this paper for anyone interested in continual pretraining.  Therefore I recommend acceptance.
Decision: Accept (oral)