Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Authors: Louis Béthune
David Grangier
Dan Busbridge
Eleonora Gualdoni
marco cuturi
Pierre Ablin
Venue: ICML 2025 poster (2025)
Keywords: LLM
fine-tuning
scaling laws
Primary area: deep_learning->foundation_models
TL;DR: We study how the loss on pretraining datasets and specialization datasets changes with model size, number of available fine-tuning tokens, and amount of pre-training data injected in the mix during fine-tuning.

Abstract: A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain.
Finetuning presents two challenges: \textit{(i)} if the amount of target data is limited, as in most practical applications, the model will quickly overfit, and \textit{(ii)} the model will drift away from the original model, forgetting the pretraining data and the generic knowledge that comes with it.
Our goal is to derive scaling laws that quantify these two phenomena for various target domains, amounts of available target data, and model scales.
We measure the efficiency of injecting pretraining data into the finetuning data mixture to avoid forgetting and mitigate overfitting.
A key practical takeaway from our study is that injecting as little as $1\%$ of pretraining data in the finetuning data mixture prevents the model from forgetting the pretraining set.

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Louis Bethune' David Grangier' Dan Busbridge'
Eleonora Gualdoni'
Marco Cuturi! Pierre Ablin
Abstract
encompasses the popular supervised finetuning framework (Ouyang et al., 2022), where the target domain data consists of high-quality examples that the LLM should imitate.
A widespread strategy to obtain a language model that performs well on a target domain is to fine- tune a pretrained model to perform unsupervised next-token prediction on data from that target do- main. Finetuning presents two challenges: (i) if the amount of target data is limited, as in most practical applications, the model will quickly over- fit, and (āi) the model will drift away from the orig- inal model, forgetting the pretraining data and the generic knowledge that comes with it. Our goal is to derive scaling laws that quantify these two phenomena for various target domains, amounts of available target data, and model scales. We measure the efficiency of injecting pretraining data into the finetuning data mixture to avoid for- getting and mitigate overlitting. A key practical takeaway from our study is that injecting as little as 1% of pretraining data in the finetuning data mixture prevents the model from forgetting the pretraining set.
Challenges in Finetuning Finetuning runs into two inter- twined issues. Because the target domain data is often avail- able in limited quantity, notably when compared to the large capacity of an LLM, the model will overfit the target do- main during optimization of the training loss. Furthermore, the parameters can move far away from the base models' parameters, which may lead to forgetting: the models' per- formance generic tasks decreases during finetuning.
Pretraining data injection customary approach to miti- gate forgetting during finetuning consists of injecting some pretraining data into the finetuning data mixture (Liu et al., 2022; Kang al., 2024; Ibrahim et al., 2024). The idea is that this data serves stay good on generic tasks. However, it is unclear how to pick the right amount of pretraining data to inject and what precise impact it has on both forgetting and overfitting.
regularizer that helps the model
Our findings We consider the target loss, i.e., the loss on validation version of the target dataset, as measure of performance (relative to the target domain) and the pretrain- ing loss as a measure of forgetting. The main takeaway of our work is that for each target domain considered in this paper, the pretraining loss after finetuning can be pre- dicted accurately from (i) the model scale, (ii) the amount of target data available and (ili) the fraction of pretraining data injected in the finetuning data mixture. We demonstrate that the following scaling law well predicts the pretraining loss after finetuning
1. Introduction
Large Language Models (LLMs) are generalist models that are trained on a large and diverse corpus of data, called the pretraining set. The diversity of the pretraining set is widely credited with giving LLMs general knowledge and versatile applicability (Yu et al., 2024). However, not every user requires a gigantic generalist LLM to answer simple tasks, and one might instead favor more specialized models that are able to work better on a specific subset of tasks. For such scenarios, finetuning is the go-to method to obtain a custom LLM that performs well on comparatively smaller target domain. More precisely, we refer to finetuning in this work as the optimization of the next-token- prediction loss on target domain data, where the starting point of optimization a pretrained reference model. This definition of finetuning
Pretraining Loss Before Finetuning. Unique Finetune Tokens
Dn. 8
Cpt =
C
+
(1)
((1+
B p NN
Parameter Relative Efficiency
*Pretraining Mixture
where A, B, a and 3 are domain-dependent constants, N is the model size, Dr. is the number of available finetun- ing tokens, and p E [0, 1] is the proportion of pretraining data injected in the finetuning data mixture. In particular, we report that, as rule of thumb, as little as = 1% of pretraining data injection is enough to mitigate forgetting.
Equal contribution 'Apple. Correspondence to: Louis Bethune <bcchhnnc apple apple.com>>, com>, Pierre Ablin <p.ablin @ @pple..pppp....ccccc
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
We also study the impact of these three quantities on the

---

Scaling Laws for Finetuning and Forgetting in LLMs
Fraction of injected pre-training data p
Zero
1.0%
5.0%
Pretrained checkpoint
2.8
Loss on the finetune validation set
2.5
3.6
Loss on the finelune train set
2.6
2.0
3.5
Pretraining loss
2.4
1.5
3.4
1.0
2.2
3.3
0.5
2.0
3.2
0.0
2000 Iteration
4000
2000
4000
2000
4000
Iteration
Iteration
Figure 1. As little as p 1% of prefraining data injection shields the model from forgetting on the pretrain dataset. The finetuning validation follows a conventional U-curve. this paper, we always consider the models obtained at the bottom of U-curve, that is, models with the best validation loss on the finetuning set, indicated here by black dot. Github dataset with small model. The minimum validation loss is barely impacted by the amount of injected pretraining data p. and it takes more iterations to reach the minimum as p increases. The loss the training finetuning set converges to zero as training progresses since the network memorizes the datasct. The pretraining loss increases monotonically during finetuning. Injecting pretraining data has regularizing effect that reduces overfitting and forgetting.
target loss, and report that the data mixture coefficient p has little effect on it. In the case where p is small, recover the multiplicative scaling laws of Zhang et al. (2024):
Continual pretraining. In the context of continual pre- training, it has been observed by Ibrahim al. (2024) that mixing small percentage of pretraining data mitigates for- getting (see Hiratani, 2024, for an analytical framework to explore how task similarity can affect knowledge transfer). Unlike finetuning, continual pre-training assumes an infinite data stream for the new task, which plays nicely with learn- ing rate schedulers. The scarcity of data fine-tuning forces repetitions of multiple epochs, which can cause overfitting. Therefore, practitioners typically perform early stopping with small constant learning rate. Our study focuses on this last scenario.
Ca = A N Dn
E,
Parameter Count
Unique Finetune Tokens
Paper organization In section 2, we describe the models used in this study, the different phases of model training and the pretraining data injection method. In section 3, we describe the models and datasets used in this study, as well as the experimental setup. Finally, section 4 describes our findings and the estimated scaling laws.
Alternative approaches in the context of pframeter-efficient finetuning (PEFT) have tried address these challenges by introducing lightweight modules, such as adapters, that are updated with knowledge about the new task (Houlsby et al., 2019; He et al., 2022). In this context, Low-Rank Adapta- tion methods, or LoRA (Hu et al., 2021; Zhang et al., 2024), inject trainable low-rank matrices into the model layers, by adding them to the untouched original model weights, thus reducing the parameter overhead needed to train for new downstream tasks (see Zhu al.. 2024, for a discussion the different roles of LoRA decomposition matrices). Zhang al. (2024) question the use of PEFT methods for fine-tuning, reporting that it usually underperforms in terms of fine-tuning loss compared to -ar-mearreeter tuning. This is why in this work, we focus on full parameter tuning, and defer the study of PEFT on forgetting for future work.
1.1. Related works
Auto-regressive pretraining and fine tuning. Auto- regressive pretraining is paradigm in which a generative model is trained on sequences to predict the next token given a sequence prefix (Sutskever, 2014). This is typically formu- lated as a classification task over a given vocabulary built from a tokenizer. Finetuning consists of pursuing the auto- regressive pretraining on a specific finetuning dataset, orders of magnitude smaller than the pretraining dataset. This fine- tuning dataset typically encompasses data from single domain, exhibiting a significant distribution shift compared to the base dataset. Transferring the model knowledge to the new distributions can lead to significant capability loss on the previous dataset, a phenomenon known as forgetting (L (Luo al., 2023; Kalajdzievski, 2024).
Neural scaling laws were first introduced by the semi-
2

---

Scaling Laws for Finetuning and Forgetting in LLMs
nal work of Hestness al. (2017) to predict the final loss achieved by a model, function of its numher of parame- ters N and the amount of data D seen, Later, this empirical study was scaled to GPT-2 scale models trained on billions of tokens by Kaplan et al. (2020). In their most simple forms, these "additive" laws are typically written as:
Fraction of injected pre-training data p
0%
1 1%
Number of finetuning tokens
Pretrained
900K
9,000K
300K
3,000K
30,000K
A
B D
Loss on the finetune validation set
Tiny
L L
No
3.00
where E, A, B, a, 3 are parameters estimated from measure- ments. Scaling laws allow us to find the best performance at IsoFLOPS, i.e., the lowest loss achievable when the number of FLOPS is fixed. For transformer-lik like architectures, train- ing FLOPS is approximated as 6ND while inference cost is estimated as 2N D. When = B, as found in the setup of Hoffmann et al. (2022b), every parameter is worth a con-
Small
2.75
Medium
2.50
Large *6:
2.25
stant amount of tokens. their study, is a factor x20 but can go as high as 192 for training procedures tailored for small models (Hu et al., 2024). This exact value varies from study to study and typically depends on optimizer hyper- parameters (Porian et al., 2024; Besiroglu et al., 2024) or the quality of data (DeepSeek-AI, 2024). When accounting for the cost of inference (Sardana et al., 2024), smaller mod- els trained for longer should be preferred over bigger ones. Other scaling laws can be baked in, accounting for mixing different modalities (Aghajanyan al., 2023), encompass- ing learning rate scheduling (Tissue et al., 2024) or taking inspiration from statistical physics (An al., 2024).
2.00
1.75
2.5
3.0 Pretraining loss
3.5
Figure 2. pntrraliaationmmmmrrzaaat tradcoff. Arxiv domain. Each point corresponds to the bottom of the U-curve for model trained on datasets of sizes 300K, 900K, 3,000K, 9,000K and 30,000K tokens with mixture parameter p 1%. Forgetting more severe when the model is small and when the finetuning dataset big. As shown Equation 6, this can be attributed to the lack of capacity of the model. More parameters are assigned to training set memorization, and fewer parameters are assigned to the pretraining set performance.
Scaling laws for finetuning. Conventional training laws focus on datasets too big for overfitting, sometimes even too big to repeat any data. In this context, validation and training loss tend to be equal. This setting cannot be ap- plied in the context of finetuning, where the smaller datasets subject overfitting, and where the discrepancy be- tween tasks might induce significant performance gap. This setup of repeating training data has been first studied by Hernandez et al. (2021) and extended by Muennighoff et al. (2023): they apply an exponential "decay" factor 1 -- exp RRp / to tokens D to account for the lack of information provided by further repetitions RD over multi- ple epochs. In Zhang al. (2024), scaling laws are derived for the finetuning loss with full parameter, LORA adaptation, and prompt finetuning. Other studies focus on the diversity of domains or different measures than the loss (Barnett, 2024; Isik et al., 2024). Close to our work, Kalajdzievski (2024) characterizes forgetting during finetuning. The key differences with our work are that Kalajdzievski (2024) uses an instructed pretrained LLama2 model. In contrast, we use several model scales to characterize the behavior of finetuning at different scales. Also, they measure the forget- ting between a pair of datasets, finetuning on one domain and computing the loss on another domain. In contrast, we measure forgetting by looking at the pretraining loss and considering 6 different domains. They also do not mention
pretraining data injection, which is a key contribution to this work. Finally, they consider parameter-efficient finetuning, while we consider full -ala-erreettr finetuning, which is more costly but also more efficient (Zhang et al., 2024).
2. Methods
We describe the training phases a model undergoes in our study: pretraining and finetuning. We also describe pretrain- ing data injection for finetuning.
2.1. Pretraining phase
The model is trained on a significant amount of tokens from pretraining dataset D Prettrin using the next-token prediction loss, until satisfying performance is achieved. The model minimizes the pretraining loss
(4) The model is typically pretrained using Adam and de- caying learning rate, for example, following a cosine scheduling (Loshchilov & Hutter, 2017) or a square root
Cpt En [l(x, 0)].
3

---

Scaling Laws for Finetuning and Forgetting in LLMs
one (Hägele et al., 2024). This pretraining loss is low when the model is able to correctly do sest-token-prediccine on the pretraining set, which, if it is diverse enough, means that the model has general knowledge. In the rest of the paper, we use this pretraining loss as a proxy of general knowledge of a model, and in particular, we measure the forgetting of a model through the increase of pretraining loss.
3. Experiments
3.1. Experimental setup
Models. We train GPT2 style transformers (Radford et al., 2019) of different scales. Table reports the model architec- tures used here. The parameter count follows exponential progression cover different magnitudes. For every model, we fix the vocabulary size to 32, 000 and the sequence length to 1, 024. We use the SentencePiece tokenizer (Kudo, 2018). Every computation is performed in bfloat 16 precision, ex- cept for normalization layers and softmax in self-attention, which are computed in float32 precision, following standard practices (Rabe & Staats, 2021; Wang et al., 2024). The biggest model fits single A100 80GB GPU without sharding, with parameter replication across GPUs to handle large batch sizes. 1 GPU is used for Tiny and Small, 4 for Medium, 8 for Large and XL.
2.2. Finetuning phase
In the second phase of training, the model is finetuned on a smaller task-specific dataset D Dinetune. by resuming training from the pretrained model. The parameters are updated by minimizing the loss on D Dineetuee
La = EEw-Drenn ((x 8)).
(5)
Since the finetuning set is small, the optimization of this loss quickly leads to overfitting, where the loss on a validation finetuning set increases- this yields U-curve as displayed in Figure 1. Furthermore, since the model is now only trained by seeing data from the finetuning set, it drifts away from the base model and forgets some of the pretraining set. The optimization of the finetuning loss is typically carried out until the validation loss starts increasing, and this checkpoint gives the so-called finetuned model.
Datasets. We use RedpajamaV2 (Weber et al., 2024) as the pretraining set. We use several domains from The Pile (Gao et al., 2020) as finetuning sets, covering all 5 categories ("academic", "internet", "prose", "dialogue" and "misc"). We artificially keep a limited number of tokens to imitate data scarcity, following log scale between 300 K and 30,000 K tokens which typically represents between 300 and 60,000 pages from a book, respectively. We believe this accurately reflects realistic use cases where finetuning is performed on specific internal documentation on a specific topic. Data-scarcity means that repetitions of training data will be necessary (multiple epochs), which ensures that overfitting can be observed.
2.3. Pretraining data injection
A simple method to mitigate overfitting and model forget- ting consists of mixing data from the pretraining set and finetuning the set during finetuning.
Pretraining. We pretrain each model on Redpajama V2 using standard hyperparameters and a total count of 100 tokens per parameter. The learning rate follows a linear warmup for 0.5% of the total iterations and then follows a cosine scheduling until the end of the training, with a ter- minal value that is one-hundredth of the maximum value. We use AdamW with a weight decay of 0.1, which yielded better results than without weight decay (Figure 17). A gra- dient clipping of 5 was found sufficient to stabilize training across all model scales.
Formally, give mixture parameter e [0, 1], we define the mixture of domains mix(p) (1 - -D)nnneuuee + pD pretrain In other words, the probability of selecting a datapoint from mix(p) is
P(r]mix(p)) = (1 - p) P(c ( finetune +
We can sample efficiently from mix(p) by first sampling random binary variable i E {0. 1) with probability p that i = 1, and then sampling from Prrttinn if i = 1 and from Daneume otherwise. We can then perform finetuning of the model by minimizing the loss over mix(p):
Finetuning. We then finetune the model on several domains from the Pile, using varying amount of target data and different fractions of injected pretraining data. We perform finetuning for 12K steps, which is sufficient to observe a U-curve on the validation loss in every configuration tested.
mmix = | [(xx,)]] = -1 + pCpt
The learning rate is equal to 1/30 times the peak pretraining LR, which was reached at about 90% of the pretraining stage. Empirically, we observe in the ablations of Figure 5 that this rule of thumb is sufficient to ensure both overfitting well within 12K steps and stable training at all model scales and all mixtures p. This corresponds to anywhere between 13 and 5200 epochs, depending on model and data scale.
This formulation makes it clear that pretraining data injec- tion can be interpreted as adding the pretraining loss as regularizer to the finetuning loss (Hastic et al., 2017). This way, the model still sees some samples from the pretraining set, which acts as a way to mitigate forgetting and overfit- ting.
4

---

Scaling Laws for Finetuning and Forgetting in LLMs
Model size
Optimal fraction
Loss on the finetune validation set
Large
Medium
Tiny
3.25
3.2
3.00
Pretraining loss
3.0
2.75
2.8
2.50
2.6
2.25
2.4
0.0%
0.5%
1.0%
10%
30%
100%
0.0%
0.5%
1.0%
10%
30%
100%
Fraction of injected pre-training data p
Figure 3. Losses function of the fraction of injected pretraining data p on Enron emails with 900K finetuning tokens. Data mixing improves generalization when finetuning data is searce. The diversity of the pretraining dataset biases learning toward features that exhibit higher generalization. The optimal value of p depends on the domain, the dataset size, and the model size. The finetuning loss as a function of p also follows a U-curve: when p is too small, the model overfits too quickly and does not benefit from the regularizing effect of pretraining data injection. When p too large, the model does there is too much tension with learning from the pretraining set. As expected, increasing monotonically decreases the pretraining loss.
enough finetuning data allocate enough capacity,
Model Size
Parameters (N)
Dim
Heads
Layers
Batch Size
Initial LRR
Tokens (D)
FLOPSs
Tiny
41M
512
8
8
32
le-3
5.1B
1.25 x 1018
Small
109M
768
12
12
32
le-3
12.B
7.84 x 1018
Medium
334M
1024
16
24
64
1e-3
33B
6.61 x 10
Large
665M
1536
24
128
3e-4
66B
2.63 x 1020
XL.
1.27B
2048
24
112
3e-4
100B
7.62 x 1020
Table Pretrained model configurations for different sizes of GPT models. All models the same tokenizer with vocabulary size of 32, 000, and the MLP hidden dimension is times the dimension of the model. The output embedding layer is included in the parameter count, as per Porian al. (2024). Training FLOPS are reported GND, following standard practices (Kaplan al., 2020).
We use Adam without weight decay. The parameter p is chosen from the set (0%, 0.1%, 0.5%, 1%, 5%).
ing dataset follows U-shape since the small number of to- kens allows for memorization on the train set. The loss over the train set decreases monotonically, with steeper slope for smaller train sets. The pretraining loss typically follows forgetting pattern, which is mitigated by taking > 0.
In total, we, therefore, have 5 model sizes x 5 number of finetuning tokens x 5 injection fractions x 12 domains, which gives 1500 finetuning runs to study.
4.2. Speed of forgetting
We report validation losses on the pretraining and target domains, and train losses on the target domains. We use these losses as proxy for the model's performance on the finetuning domains.
Empirically, we observe in Figure that as little as p 1% of pretraining data shields the model from forgetting at cost for performance pretraining data plays an "anchoring" role that helps retain useful features.
the finetuning dataset. The
4. Results
On the contrary, in the absence of pretraining data injection, performance can plummet drastically, especially for smaller models. We hypothesize that smaller models must balance their capacity between tasks, whereas bigger ones can "leam new task" while preserving most existing knowledge.
Unless specified otherwise, we report these values at the checkpoint corresponding to the lower validation domain target loss, i.e., the bottom of the "U-curve".
4.1. Mundane observations: overfitting
The loss profile of typical finetuning run is shown in Fig- ure Unsurprisingly, every validation curve for the finetun-
5

---

Scaling Laws for Finetuning and Forgetting in LLMs
Number of finetuning tokens
Fraction of injected pre-training data
300
3,000 KT
9,000
30,000 KT
0.0%
1.0%
Dm matherattcs
Wikipedia
Dm mathematics validation loss
RedPajama V2 validation loss
3.0
Wikipedis en validation loss
3.0
RedPaajaaa V2 validation loss
4.5
2.8
1.3
3.0
1.0
2.6
2.8
1.2
3.5
2.4
2.6
3.0
2.2
2.4
2.5
Tiny (1M)
Small (109M)
Mediom (B3MM)
Large (665M)
XL (1.38)
Tiny (41M)
Small (109M)
Medium (34M)
Large (665M)
XL (1.38)
Tiny (4IM)
Small (109M)
Medium (33M)
Lage (665M)
XL (1.38)
Tiny (41M)
Small (109M)
Medium (34M)
Laree (665M)
XL (.30)
Figure Overfitting and forgetting profiles two domains. On one hand, mathematics (left) a dataset that differs from the pretraining set, benefits little from more parameters, and a lot from more data. On the other hand, Wikipedia En (right) is more similar to the pretraining set, more parameters are more beneficial than more training data. Datasets that are far from pretraining distribution are more prone to forgetting and benefit the most from injecting pre-training data p 0.
Domain
Finetuning Scaling Laws
Forgetting Scaling Laws
Bootstrapped MRE (1)
Bootstrapped MRE (()
8
A
E
B
A
B
Arxiv
0.17
0.10
95.18
1.30
0.91%
0.74
0.34
526
392
0.36%
Dm mathematics
0.06
0.19
16.03
0.88
0.50%
0.58
0.27
202
9847
0,91 %
Enron emails
0.07
0.05
20.21
0.00
1.13%
0.53
0.21
127
1754
0.49%
Github
0.14
0.12
84.55
0.79
1.40%
0.76
0.43
217
647
0.43%
Pg19
0.14
0.02
34.55
1.25
0.65%
0.78
0.60
14
259
0.39%
Wikipedia
0.13
0.02
30.11
0.62
0.53%
0.52
0.11
145
829
0.21% %
Euro parl
0.12
0.17
160.24
1.10
1.86%
0,81
0.39
2511
1107
0.79%
Free law
0.19
0.05
94.06
1.11
0.91 %
0.75
0.45
74
236
0.30%
Openwebtext
0.14
0.01
31.54
0.96
0.35%
0.38
0.23
2
6504
0.25%
Pubmed abstracts
0.17
0.01
46.89
0.94
0.83%
0.76
0.57
948
0.17%
Pubmed central
0.18
0.05
74.37
1.09
0.56%
0.65
0.34
81
574
0.26%
Stackexchange
0.16
0.08
78.23
1.27
1.03%
0.62
0.34
63
1179
0.27%
Table Comparison of Scaling Law Coefficients for Finetuning and Forgetting. The Bootstrapped Estimate of Mean Relative Error | wl/y across domains for finetuning is 0.89%, and for forgetting is 0.40%. These estimates are obtained by resampling each example independently with equal probability to construct set of 125 points per domain. The final results averaged over 128 independent hootstrap repetitions.
4.3. Pretraining data injection improves generalization
4.4. Repeating pretraining tokens during finetuning
A > 0, the bottom of the U-curve can reach lower values than = 0, which suggests that pretraining data injection serves are a regularization and helps generalization on the finetuning dataset. This effect can be observed in Figure As function of p. the finetuning validation loss also follows U-curve albeit less pronounced. This effect is more striking for small models.
In all our experiments, we inject pretraining data that is streamed from the pretraining set without repetition. As reported, injecting only 1% of pretraining data in the training mix is enough to mitigate forgetting significantly, Since p quite small, it means that we use small number of pre-training tokens. For instance, in Figure we see that the bottom of the U-curve is reached around 1800 iterations for the run with y 1% of injected pre-training data. Since we use batch-size of 32 and context length of 1024, it means that, at the bottom of the U-curve, we have only seen
6

---

Scaling Laws for Finetuning and Forgetting in LLMs
Peak Learning Rate Multiplier
consider one domain (arxiv), one model size (tiny), and fixed fraction of pretraining data injection (p 1%). We then artificially limit the number of available pretraining tokens in 32KT, 96KT, 320KT, 920KT,, and 3,200KT. We report the results in Figure 6. We observe that as the number of available pretraining tokens increases, the pretraining loss after finetuning decreases. However, in this regime, the pretraining dataset itself becomes susceptible to overfit- ting. This underlines the pivotal role of pretraining dataset diversity in mitigating forgetting.
x 300
x
100
30
x1
Arxiv validation loss
2.3
retrained choikp
Pretrained checkpoint
2.2
2.1
2.0
1.9
2.4
2.6
2.8
RedpajamaV2 loss
Arxiv train loss
4.5. Scaling laws
Figure 5. Finetuning learning rate fraction of peak pretrain- ing learning rate. Ablation on Arxiv with XL Model (1.3B). The model's learning rate (LR) is reduced by factor of 100 during pretraining using cosine scheduling. For finetuning, we employ constant IRR, defined a multiple of the peak I.R. Our obser- vations indicate that setting the finetuning I.R to 1 1/30 times the terminal value strikes an optimal balance between effective adap- tation to the finctuning dataset and stability with respect to the pretrained features. Notably, this factor of 1/30x corresponds to the LR reached at 90% of the pretraining phase, near convergence.
We display the loss as function of model size and dataset size for two domains in Figure 4. We observe highly predictable behavior that is amenable to the fitting of a scaling law.
We consider different model sizes, which in turn impact the number of pretraining tokens used to train the model, using 100 tokens per parameter. Hence, we do not explore the full pretraining (N, D) space, restricting ourselves to the isocurve D 100N. We have two main reasons for this approach. First, while analyzing the behavior of losses in relation to both N and the total number of pretraining tokens is insightful, using an isocurve aligns with the most com- mon practice, which is to train only one model per model size. Second, pretraining a large grid of models would be computationally expensive, and we prefer to allocate our computing budget toward a more detailed understanding of finetuning mechanisms.
Unique finetune tokens
300 KT
900 KT
3,000 KT
9,000 KT
4.25
Pretrain loss
4.00
3.75
We fit a different scaling law for the finetuning and forget- ting loss for every domain. Per domain, we have model sizes dataset sizes x mixture values = 125 points in total. This is large in front of the number of degrees of freedom of the scaling law (4 or 5 in our case).
3.50
3.25
10-2
10a Unique pretrain tokens per unique finetune token
10
10"
10
10a
Figure 6. Influence of the number of available pretraining to- kens on the pretraining loss after fine-tuning with p 1% of pretraining data injected to the mix. We use the Arxiv domain and a model of size "tiny". With little finetuning data, finetuning is short and hence we can use very few pre-training tokens to reach the optimal forgetting. When there is a lot of finetuning tokens, the optimization takes a long time to reach the bottom of the U-curve. Hence, when the number of pretraining tokens available is also limited, these tokens repeated many times, which leads to over- fitting on the pretraining set as well, and increases the pre-training loss. Remarkably, only 0.3 unique pretraining tokens per unique finetuning token are sufficient to avoid forgetting.
Method. To fit the coefficients, we follow conventional approach (Muennighoff et al., 2023), i.c., we rely on the Huber loss and we perform optimization in log space for improved numerical stability. See Appendix C for a detailed discussion.
Scaling law for finetuning loss the Pile, we fit the multiplicative law proposed by Zhang et al. (2024) for finetuning:
For every domain D of
Ln A Na DH
EE,
(6)
which yields a Bootstrapped (n = 128) Mean Relative Error (MRE) of 0.89% across domains. As depicted in Figure 3, the fine-tuning loss is barely impacted by the fraction of injected pre-training data p. which is why we chose to treat the loss as a constant of p. We report scaling
1% x 1800 x 32> x 1024 = 600K pretraining tokens.
We now turn to a study of how limiting the amount of avail- able pretraining tokens that are used for pretraining data injection impacts finetuning and forgetting. To do so, we
7

---

Scaling Laws for Finetuning and Forgetting in LLMs
Scaling law prediction
Empirical measurement
Scaling law prediction
Empirical mensurement
Unique finetune tokens
Model size
300K
900K
3,000K
9,000K
30,000K
Tiny
Small
Medium
Large
XL
Loss on the finetune validation set
Loss on the finetune validation set
3.0
3.0
2.8
2.8
2.6
2.0
2.4
2.4
2.2
2.2
Tiny
Small
Medium
Large
0.0%
0.5%
1.0%
5.0%
Model Size
Fraction of injected pre-training data
1%
300K Tokens
Figure Scaling laws for finetuning loss. Wikipedia domain. We extend multiplicative laws Zhang et al. (2024) to take into account the fraction of injected pretraining data. At first order, this scaling law is independent of the mixture Left Agreement between the observed behavior and behavior predicted by the scaling law. Right Evaluated and predicted loss as a function of
laws coefficients in Table 2. The results for fixed dataset size or for a fixed mixture p are given Figure 7. also tried to fit additive law of the form Lit = N + Dr." + E, but despite the additional degree of freedom through coefficient B, it yields an higher MRE of 1.36%. The superiority of the multiplicative scaling law compared to the additive is consistent with the findings of Zhang et al. (2024).
2020; Hoffmann et al., 2022a). We propose a multiplica- tive law where D on the numerator with 3 > 0. since increasing the finetuning dataset size leads to more itera- tions to reach the bottom of the U-curve, which results in model that drifts further from the base model and higher pretraining loss. We use factor (1 + Bp) in front of the N parameter. It accounts for the fact that fraction of the pa- rameters are allocated to the pretraining task rather than the finetuning one, and they are times more efficient this context. Typically, we observe B > since the features of the pretrained model are already aligned with the pretraining task, which translates into more efficient parameter alloca- tion. Coefficients are given in Table 2. Since tin-ttunng is performed with learning rate slightly higher than the ter- minal LR of pretraining, the value of Cp corresponds to the rewarmed pretrained model, i.c, the model pretrained with the LR used for fine-tuning, and not the terminal LR the cosine scheduling. Differences are highlighted in Table 3. The average Bootstrapped MRE across domains is 0.40%.
Analysis. The value of E R 0 for Enron emails stands out the table. This pattern suggests that memorization of the corpus is possible, firstly thanks to its modest size, and secondly because has been made available online more than 20 years ago and used in several NLP publications ever since then'. Besides that, the value of oscillates between 0.12 and 0919, which is consistent with previous studies (Hoffmann et al., 2022a). The value of 3 seems to capture the difficulty of the dataset, with the high-entropy ones like Wikipedia or Openwebtext standing out.
Scaling law for forgetting For every domain of the Pile, we want to predict Cpt. the value of the pretraining loss after finetuning, as measure of forgetting. We propose the (modified) multiplicative law:
Other laws. An additive law to predict the difference Cpt - C. cp. has an error greater than 0.82% despite having more degree of freedom. We also note that it is critical to add the base pretraining loss Cp to the scaling law, as D. fitting scaling law of the form Lpt = A (1)+B) + E leads to MRE of 1.05%. Finally, we observe that the "cost of rewarming" the model must be accounted for when computing Cp.- as observed in Table 3. Otherwise, that an additive corrective factor of E Rs 0.05 is necessary to account for forgetting induced by optimization, which yields a higher MRE of 0.49%. Indeed, even in the setting = 1, further pretraining with a constant LR (which is typically higher than the terminal LR as shown in Figure 5)
DA
Lpt = co. A ((1 Bp)
where Cgp is the pretraining loss before fine-tuning, i.e. the pretraining loss of the pretrained model, Dp. is the number of available fine-tuning tokens, N the model size. The variables of the scaling law are A, B, 3 and a, and they are all positive numbers. Note that the value Cot exactly what the original scaling law papers try estimate (Kaplan al.,
find
See http:/ www. enron marronn-maat com example.
8

---

Scaling Laws for Finetuning and Forgetting in LLMs
Scaling law prediction
Empirical measurement
Scaling law prediction
Empirical mensurement
Unique finetune tokens
Model size
300K
900K
3,000K
9,000K
30,000K
Tiny
Small
Medium
Large
XL
3.2
3.5
Pretraining loss
Pretrain loss
3.0
3.0
2.5
2.4
Tiny
Small
Medium
Large
XL
0.0%
0.5%
1.0%
5.0%
Model Size
Fraction of injected pre-training data
0%
300K Tokens
Figure Scaling laws for forgetting, Github domain. propose model the increase pretraining loss as a multiplicative scaling law (Equation 1), that takes into account model size, number of finetuning tokens available, and fraction of pretraining data injected in the data mixture p. Left Agreement between evaluated and predicted loss on one domain. Right Evaluated and predicted loss as function p.
Pretraining loss Lpe with
require more compute, so forgetting is more expensive for them. Results are summarized in Figure 14.
pretrained model
rewarmed model 1/30 peak LR
Model Size
1/100 peak LR
Scaling laws are mainly useful when they enable performance predictions based small-scale exper- iments, to assess if bigger models and datasets are worth the cost. To test this, we design a synthetic experiment where scaling law coefficients are fitted using only a subset of model sizes and fine-tuning dataset sizes. The results, presented in Table 4, demonstrate that models trained on no more than Medium (334M parameters) and 3,000K tokens can accurately predict the performance of Large and XL models trained on 9,000K and 30,000K tokens. Averaged over all domains, the prediction error remains within 2.01 % on the finetuning set and 0.83% on the pretraining set. On GPUs, the Medium model fine-tuned on 3,000K tokens reaches the bottom of the U-curve in under 30 minutes, whereas the model requires up hours on GPUs. The cost acquiring additional data scale from 3,000K 30,000K tokens varies depending the context, but be significant in certain environments. In this context, scaling laws can lead to substantial computational savings.
Extrapolation.
Tiny
3.13
3.19
Small
2.84
2.92
Medium
2.55
2.60
Large
2.34
2.39
XL.
2.22
2.27
Forgetting
0.49%
0.40%
MRE (L)
Table 3. Effect of rewarming the model on the pretraining loss. Finetuning performed with learning rate (LR) which is 1/50 of the peak LR, slightly higher than the terminal LR of the pretraining stage. This "rewarms" the model, incurring a loss increase that we measure and must be accounted for.
cancels some of the benefits of the cosine cooldown used during pretraining.
Analysis. The value of B indicates how much pretraining data injection helps mitigate forgetting. For Dm mathemat- ics, which was observed in Figure 4 to be highly subject to forgetting, B x 10 On the contrary, for Wikipedia, on which pre- pre-training training data injection was less useful, we only have ~ 829.
Conclusion
We showed that one can accurately predict the finetuning performance and the forgetting of the pretraining of large language models, number of available tokens, and the fraction of pretraining data injected into the data mixture. This behavior is very consistent across datasets and can be explained by simple scaling laws. A key takeaway of our work that injecting even 1% of training data during fine-tuning helps mitigate pretraining set forgetting.
function of the model size, the
An interesting consequence of our functional form is that forgetting is primarily attributed network capacity. This is confirmed by the fact that smaller models suffer the most: they lose up to 95% (!) of the pretraining progress when forgetting (i.e, the pretraining validation loss reverts to a point reached at 5% pretraining). while bigger models only lose 20% of the progress. However, bigger models

---

Scaling Laws for Finetuning and Forgetting in LLMs
Acknowledgments
Hastie, T., Tibshirani, R., and Friedman, J. The elements of statistical learning: data mining, inference, and predic- tion, 2017.
The authors heavily relied on a codebase that Awni Hannun kickstarted. The authors thank Federico Danielli for his careful proof-reading. Finally, the authors warmly thank Alaaeldin EI Nouby for his judicious suggestions in the design and organization of the figures and tables.
He, J., Zhou, C., Ma, X., Berg-- T., and Neubig, G. Towards unified view of parameter-efficient transfer learning, 2022. URL https: watpir.iiivv..... org/ /aag/////// abs//
2110.04365.
Impact Statement
Hendryeks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer, 2021. URL https: // arxiv. wrg.abs////oo....oooooo........a 2202.00022200 01293.
References
Aghajanyan, A., Yu, L., Conneau, A., Hsu. W.-N.. Ham- bardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-modal language models. In International Confer- ence on Machine Learning, pp. 265-279. PMLR, 2023.
Hestness, J., Narang, S., Ardalani, N., Diamos, Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.00409 2017.
An, H., Song, Y., and Li, X. Physics in next-token predic- tion, 2024. URL htt.tttss//aaiiiii httpa////aaaiiooor/a//////1//// 00660.
Hiratani, N. Disentangling and mitigating the impact task similarity for continual learning, 2024. URL https: //arxiv. org. abs 220...245..222.
Barnett, M. An empirical study of scaling laws for transfer. arXiv preprint arXiv:2408. 16947, 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models. CoRR, abs/2203, 15556, 2022a. doi: 10.48550AARXI1:203..55555 URL https: /doi. org 10. 483448330000 /0t/555//XXiii/
Besiroglu, T., Erdil, E., Barnett, M., and You, Chin- chilla scaling: replication attempt. arXiv preprint arXiv:2404. 10102, 2024.
Clark, P., Cowhey, I., Etzioni, O.,, Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. 2018.
15556.
arXiv preprini
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbi, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J., and Sifre, L. An empirical analysis compute- optimal large language model training. In Advances in Neural Information Processing Systems, 2022b.
DeepSeck-Al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint urXiv:2401.02954 2024. URL https: github. com/deepseek- hee//DDeeeeeetaee
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00033 2020.
Houlsby, N., Giurgiu, A., Jastrzebski, S.. Morrone, B., de Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efffiieet transfer learning for nip, 2019. URL http:: /atp::///axvvviia www.iiioorr/a/o wttpv/..rrg/bb//192222 00751.
Hägele, A., Bakouch, E., Kosson, A., allal, L. B., Werra, L. V., and Jaggi, M. Scaling laws and compute-optinal training beyond fixed training durations. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. URL https::pp/... openreview. net/ forum? d= d
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https: //arxiv. org/abs, htggabs/2100 /2006 09685.
10

---

Scaling Laws for Finetuning and Forgetting in LLMs
Hu, S., Tu, Y., Han, X., Cui, G., He, C., Zhao, W., Long, X., Zheng, Z., Fang, Y., Huang. Y., Zhang, X., Thai, Z. L., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G.. dahai li, Liu, Z., and Sun, M. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024. URL https: //openreview net/forum forum? id= 3X212TFr0f.
Mayilvahanan, P., Wiedemer, T., Mallick, S., Bethge, M., and Brendel, W. Lims on the line: Data determines loss- to-loss scaling laws. arXiv preprint arXiv:2502.12120, 2025.
Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-28999 2022.
Ibrahim, A., Thérien, B., Gupta, K., Richter, M. L., An- thony, Q., Lesort, T., Belilovsky, E., and Rish, I. Simple and scalable strategies to continually pre-train large lan- guage models, 2024. URL https: /aapss///rxvv... httpi.xivvvvooo... org/ abs/2403. 08763.
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. Sealing data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50376 2023,
Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vas- silvitskii, S., and Koyejo, S. Scaling laws for down- stream task performance of large language models. arXiv preprint arXiv:2402.04177 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C.,, Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2771 2022.
Kalajdzievski, D. Scaling laws for forgetting when fine- tuning large language models, 2024. https:// arxiv. org/ abtaa/ass//4///1 05605.
Porian, T., Wortsman, M., Jitsev, J., Schmidt, L.,, and Car- mon. Y. Resolving discrepancies in compute-optimal scaling of language models, 2024. URL https:// arxiv. orgg/bb......ooooooo..eee.eeee.e 2200 19146.
Kang. F., Just, H. A., Sun, Y., Jahagirdar, H., Zhang, Y., Du, R., Sahu, A. K., and Jia, R. Get more for less: Principled data selection for warming fine-tuning in Ilms. arXiv preprint ar 2l0:2400500774, 2024.
Rabe, M. N. and Staats, C. Self-attention does not need o(n2) memory. arXiv preprint arXiv:2112.05682 2021.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836) 2020.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. 2019.
Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., al. Multitask prompted training enables zero-shot task generalization. In International Conference Learning Representations, 2022.
Kudo, T. Sentencepiece: A simple and language indepen- dent subword tokenizer and detokenizer for neural text processing, arXiv preprint arXiv: 1808. 06226, 2018.
Liu, Z., Xu, Y., Xu, Y., Qian, Q., Li, H., Ji, X., Chan, A., and Jin, R. Improved fine-tuning by better leveraging pre- training data. Advances in Neural Information Processing Systems, 35:32568 13332568-32581, 2022.
Sardana, N., Portes, J., Doubov, S., and Frankle, J. Be- yond chinchilla-optimal: Accounting for inference in language model scaling laws, 2024. URL https:
/ wwtp:/s///aab..aooop//////
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Sutskever, I. Sequence to sequence learning with neural networks, arXiv preprint arXiv:1409.3222 2014.
Loshchilov, and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023. URL.
htt.gaaa..../.aaaeeee co/datasets/ teknium/ www.hu/mmppeeeeees.....
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint X0v:22300 08747, 2023.
Tissue, H., Wang, V., and Wang, L. Scaling law with learn- ing rate annealing. arXiv preprint arXiv:2408.11022 2024.

---

Scaling Laws for Finetuning and Forgetting in LLMs
Wang, H., Liu, Q., Du, C., Zhu, T., Du, C., Kawaguchi, K., and Pang. T. When precision meets position: B breaks down rope in long-context training, arXiv preprint arXiv:2411.13476. 2024.
Weber, M., Fu, D. Y., Anthony, Q. Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M.. Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
Wei, J., Bosma, M., Zhao, V. Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2022.
Yu, X., Zhang, Z., Niu, F., Hu, X., Xia, X., and Grundy, J. What makes a high-quality training dataset for large language models: A practitioners' perspective. In Pro- ceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp. 656-668, 2024.
Zhang, B., Liu, Z., Cherry, C., and Firat, O. When scaling meets LLM finetuning: The effect of data, model and finetuning method. The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview: net net//forumm forum ? dd5HChnDeTWw..
Zhu, J.. Greenewald, K., Nadjahi, K., de Ocáriz Borde, H. S., Gabrielsson, R. B., Choshen, I.., Ghassemi, M., Yurochkin, M., and Solomon, J. Asymmetry in low- rank adapters of foundation models, 2024. URL https: // /arxivv. org/ abs 22022.2222222222 16842.
12

---

Scaling Laws for Finetuning and Forgetting in LLMs
A. Ablations
Learning rate. The learning rate (Figure 5) is a critical parameter. If too low, we might fail to reach the bottom of the U-curve within reasonable time. If too high, it causes the model to diverge away from the pretraining validation loss. Warmup and cosine scheduling cause modelization issues in this context. In particular, they require to know in advance the number of steps necessary for convergence, which is tricky when targeting the bottom of U-curve. Therefore, we focus on constant learning rate.
Anchored AdamW. We propose another approach to reduce forgetting, coined Anchored AdamW, which performs the same updates as AdamW with the difference that the regularization ties the parameters 0, to the pre-trained model 0g. instead of O.
X 110: 0-1
regularization
(8)
2
This contrasts with the default weight decay which simply 3/ 11. The pre- pre-trained parameters 6, hence play the role of anchoring. The difference between Adam, AdamW, and Anchored Adamw illustrated in Figure The results on the Github domain are given in Figure 10. In this scenario, we see that adding only p 1% of pre-training data outperforms fine-tuning with weight decay by a large margin. We see that the higher performance of finetuning with pre-training data can not be attributed to closeness with 8, only. Anchored AdamW fails to prevent forgetting on RedPajama, but at the same time, the regularization is strong enough to hurt performance on Github noticeably. This suggests that having more data diversity is more powerful than simple parameter-spa space regularization.
Cuique finctune
-+0-
sage 3,00K - 9,00K
50,00KK
Adam
A
Anchored AdamW
Optiniier with danay 1 4
Adan (y 0%8
AdanW (p 0% 2 -0.10)
Adan (2 19)
Starting point B
.anWW 0%
AdannW
1.00)
2.4
5.8
RosPPqumaVZ wilditionn loss
3.60
20n
Githob Githihyalldoiion valitotiion lose
3.4
100
2.0
2.8
288
2.
24
They G116
fond (oodl)
Median C3AM0
Large HedtMl
The (HIM)
Bodt nami
Media n9MM)
Lmpge rest
0
X
Figure 9. Comparison of Adam, AdamW, and Anchored AdamW on the objective function f(x y) (x 2)² + (y + 3)² with the starting point 0u = (4, 4). 100 steps, learning rate 0.1.
Figure 10. Comparison of Adam and Anchored AdamW on Github domain. Mixing p 1% of pre-training data with Adam optimizers elearly outperforms Anchored AdamW, both on pre-trained data and fine-tuning data.
Forgetting law. Other laws can be tested, such as A D"(1-p)" E which has the appealing property of going to zero when p- 1. However, the MRE reaches 0.67% for this law, despite having the same degrees of freedom with the new parameter k. Purely additive laws had an MRE superior to 1%.
Additional isocurves. Most experiments of the paper rely on the isocurve D = 100N for the checkpoint of the pre-trained model. We also re-train full models from scratch with cosine decay for the isocurve D = 10.N, and evaluate them on the Freelaw domain. Results are given in Figure 11 and demonstrate that our scaling laws still hold for these checkpoints.
Extrapolation. In Table 4 we evaluate the extrapolation capabilities of the scaling law, by fitting only on a subset of all available tokens and models. Overall, the MRE remains below 2% for the law of (Zhang et al., 2024), and even below 1% for the forgetting law we propose. This demonstrates the feasibility of estimating coefficients at a small scale and evaluating their impact at larger scale.
13

---

Scaling Laws for Finetuning and Forgetting in LLMs
Scaling law prediction
Empirical measurement
Scaling law prediction
Eapirriaa mensurement
Unique finetune tokens
Unique finetune tokens
300K
900K
3,000K
9,000K
30,000K
300K
900K
3,000K
9,000K
30,000K
Loss the finetine validation
3.00
3.50
2.75
Pretraining loss
3.35
2.50
3.00
2.25
2.75
2,00
2.50
Tiny
Small
Medium
Large
XL.
Tiny
Small
Medium
Large
Model Size
Model Size
Figure 11. Scaling laws for finetuning and forgetting in models pre-trained at the D 10N isocurve. Freelaw domain. The law remains robust the choice of checkpoint, with bootstrapped MRE of 0.57% for forgetting and 1.14% for finetuning.
Finetuning MRE
Forgetting MRE
Setup
Predict
XL
A
1.69%
0.73%
30,000K
Large, XL
B
2.01%
0.83%
9,000K, 30,000K
Table Mean Relative Error across all 12 domains in the extrapolation setting, In setup A, scaling law coefficients are estimated on all model sizes except XL, and all finetuning set sizes except 30, 000K tokens. setup the largest model used for fitting Medium (334M) and the maximum finetuning dataset size is limited to 3, 000K tokens.
Instruction finetuning (IFT). Our work focused mainly on the next token prediction task in finetuning, directly on the raw sequence, but variant exists: instruction finetuning (Wei et al., 2022; Sanh et al., 2022; Min al., 2022). In this setup, the "instruction" tokens (sometimes referred as "prompt" or "input") are masked from the loss, and the model is trained to solve specific task conditioned by the prompt. Implementation-wine we leverage that our vocabulary is capped to 32k tokens (ids can be stored on 15 bits), and we store the mask in the (unused) 16th bit of the uint16 types we use to store token ids. At train time, bitwise operations are used to efficiently extract the mask from the ids, and apply the next token prediction loss only on the unmasked tokens. We rely on the OpenHermes dataset (Teknium, 2023), with special "
" and "
" tokens to delimit the instruction part from the prediction part. Results are given in Figure 12. We fit the scaling law coefficients, and see that the bootstrap estimate of MAE is 0.59% for finetuning, and 0.29% for forgetting. Coefficients are given in Table below.
Domain
Finetuning Scaling Law
Forgetting Scaling Law
9
A
E
MRE
8
A
B
MRE
OpenHermes
0.17
0.03
64.28
0.46
0.59%
0.80
0.27
5513
8584
0.29%
B. Practical consequences on finetuning
The hidden cost of forgetting. The Figure 14 shows that small models suffer the most from forgetting, losing up to 95% of the pre-training stage. This makes them unsuitable for sequential adaptation, and they benefit less from pre-training on diverse data when fine-tuning specific data. Overall, this balances some findings of Sardana et al. (2024), which suggested that smaller models should be preferred due to their lower inference cost. While being more expensive, bigger models retain more information from the pre-training stage.
14

---

Scaling Laws for Finetuning and Forgetting in LLMs
Scaling law prediction
Empirical measurement
Scaling law prediction
Empirical mensurement
Unique finetune tokens
Unique finetune tokens
189K
567K
1,890K
189K
567K
1,890K
Loss on the finetune validation
3.2
Pretraining loss
3.0
2.8
2.4
1.6
Tiny
Small
Medium
Large
XL
Tiny
Small
Medium
Large
XL
Model Size
Model Size
0%
p 0%
Figure 12. Scaling laws for finetuning and forgetting with Instruction Finetuning (IFT) OpenHermes dataset. The dataset is much smaller than the domain Pile considered here, so different values of unique tokens are considered. Finally, this dataset is very diverse and challenging by design, which flattens the contribution of the number of tokens, and makes the model size the driving factor behind scaling.
Downstream task performance. It has been documented that training loss correlates with downstream task perfor- mance (Hoffmann et al., 2022a; Mayilvahanan et al., 2025). Therefore, we evaluate the quality of the 1.3B model, before and after finetuning, with and without pre-training data injection, on ARC-easy (Clark et al., 2018) and MMLU tasks (Hendrycks al., 2021). Results are given in Figure 13. We see that the performance on ARC-easy degrades after fine-tuning, but injection of pre-training data mitigates this phenomenon. On MMLU the trend is less obvious, the model does not perform better than random classifier. However, the monotonic improvement across all model scales suggests that dm- mattmmtthmaatcss contains inductive bias that helps some tasks of MMLU.
Furthermore, for ARC-easy in the 0-shot setting, we compare the performance with pretraining data injection (p > 0%) against the baseline at p = 0%, and we report the difference accuracy when it's significant at the 99% threshold, based on the 7-score. Otherwise, we report the difference as "Not Significant" (n.s). We see that injecting pre-training data yields consistent improvement across all model scales and data abundance.
Percentage of pre-training data injection
Model size
Number of tokens D
0.1%
0.5%
1%
5%
Medium
300 KT
ns
ns
INS
ns
Medium
900 KT
Medium
3,000 KT
3,9
4.5
4.0
4.0
Medium
9,000 KT
4.1
4.1
4.0
ns
Medium
30,000 KT
4.8
6.6
4.8
5.4
Large
300 KT
ns
ns
ns
Large
900 KT
Large
3,000 KT
3.9
4.6
5.1
Large
9,000 KT
3.8
5.0
5.3
4.6
Large
30,000 KT
6.9
7.7
7.2
8.4
XL
300 KT
nS
XL
900 KT
BS
XL
3,000 KT
IS
ns
XL
9,000 KT
IS
nS
4.0
ns
XI.
30,000 KT
4.8
5.0
5.5
5.3
15

---

Scaling Laws for Finetuning and Forgetting in LLMs
Fasy
Pretraned
Pretraited
p
with p
Finetuning with 0%
Vineouning with Oi
(01
20-
50-
O-dixx Ascaray
-0
10-
20
10-
Medium
Large
Medium
XI
Figure 13. Downstream tasks performance for pretrained checkpoint before finetuning, after finetuning, and with data injection. checkpoints have been fine-tuned on dm mathhmaattics. Forgetting penalizes model on ARC Easy, but finetuning improves (marginally) the model on MMLU.
C. Scaling law coefficient estimation
We follow the procedure outlined (Hoffmann et al., 2022a; Muennighoff et al., 2023; Besiroglu al., 2024), identify the coefficients in our scaling laws. Restating the supervised law for convenience
A
L(N, D)
+ D8
(9)
Na
To aid numerical stability, we write this expression in log space. First note that for b > 0
log(a + b) = log (exp log exp log b) LSE(log a, log b),
(10)
where L.S. is the log-sum-exp operator. We can now proceed to write the supervised scaling law in log form
B
log I(N, D; A,B,E,,,,,,P log
E+
(11)
No
D
LSE [log E, log A - aN, log B 3D)
(12)
We make assumptions about the relationships between the values (i.e. parameter tying) and optimize
Σ Hubers (log L(N()) ,, L())
(A*, B" B", a= 3*) arg min (,.3,
E
(13)
with a Huber 5 10-4 where N6, ( and [ are the model size, number of training tokens and loss achieved by the i-th run. We fit on 125 samples over grid of L-BFGS-B initializations given by: log {0,3, 12). log B (0, 6, 12). log E {-2, 115, -,, 1, 0.5, 1.5, 2,5,5 (0, 0.5, 1), {0, 0.5, 1). We similar procedure to fit the different forms of the scaling laws.
16

---

Scaling Laws for Finetuning and Forgetting in LLMs
XL
Large
Medium
Small
Tiny
%
0%
Number of finetuning tokens
Pretrained * 900K 9,000K
300K
3,00DK
30,000K
Pretraining cost lost (A100 GPUh)
300
200
con
dsfe
20%
60% Percentage of pretraining lost
100%
Figure 14, Cost forgetting. Points are reported at the bottom the U-curve for the Arxiv domain. Same setup as Figure 2. Smaller models lose significant portion of the progress achieved during pretraining up 80% c0%-cffectively negating much of the initial effort. contrast, larger models retain more knowledge due their greater capacity, allowing them to accommodate both tasks simultancously. However. these models are also the most expensive train, only because of their size but also due the increased number of pretraining tokens required for compute-optimality (Hoffmaan et al., 2022a). Consequently, the GPU-hour cost induced by forgetting substantially higher for larger models.
17

---

Scaling Laws for Finetuning and Forgetting in LLMs
Mixture
0.0%
0.1%
0.2%
0.5%
1.0%
5.0%
Pretrained checkpoint
Enron emails (900KT) Medium Model
5.5
2.5
Enron emails validation loss
Enron emails train loss
2.0
RedPajana V2 loss
1.5
3.0
1.0
0.5
2.5
3.0
0.0
2.5
5000
10000
5000
10000
0
5000
10000
Iteration
Iteration
Iteration
Mixture
0.0%
0.1%
0.2%
0.5%
1.0%
5.0%
Pretrained checkpoint
Github (3,000KT) Tiny Model
2.5
Github validation loss
3.5
2.0
RedPajama V2 loss
5.5
Github train loss
3.0
1.5
5.0
1.0
2.5
4.0
0.5
3.5
2.0
0.0
5000
10000
5000
10000
0
5000
10000
Iteration
Iteration
Iteration
Mixture
0.0%
0.1%
0.2%
0.5%
1.0%
5.0%
Pretrained checkpoint
1g19 (9,000KT) Large Model
3.0
7
5.5
2.5
5.0
Pg19 validation loss
RedPajamaVa V2 loss
Pel 19 train loss
2.0
4.5
1.5
4.0
1.0
3.5
0.5
3.0
2.5
0.0
5000
10000
5000
10000
5000
10000
Iteration
Iteration
Iteration
Figure 15. As little as = 1% of pretraining data shields the model from forgetting on the pretrain dataset.
18

---

Scaling Laws for Finetuning and Forgetting in LLMs
1.0
0106
011
009
035
105
Giitsh
0109
009
Lags
Tay
Lage
Pg19
Wilipolo cn
5.50
100
10%
Figure Losses at bottom U-curve for domains of The Pile with models and dataset sizes, values data mixture
19

---

Scaling Laws for Finetuning and Forgetting in LLMs
XL Model after pretraining
3.0
Adam
A
2.5
Validation Loss
1.5
1.0
0.5
0.0
Anxiv
Dm Mathematics
Enron Emails
Earo Parl
Free Law
Github
Hackernews
Nih Exporter
Openwebtext
Pg19
Phil Papers
Pubmed Abstracts
Pubmed Central
Btackeschange
Ubantu Ire
Uspto Backgrounds
Wikipedia En
Figure 17. Ablation. Weight decay during pretraining improves checkpoint quality. Note that default implementations of Pytorch and Optax do not decouple the weight decay from the learning rate (per this Github issue: https: / //ggttgttb.///////// com/ google-deepminng nd/ apt 1 sssuss/25 /292), unlike the seminal paper suggested (Loshchilov & Hutter, 2019), which implics that weight decay impact diminishes during lcarning rate decay.
20

---

Scaling Laws for Finetuning and Forgetting in LLMs
Eixtaning ST0KT
3,00 KT
Misuec
+0 0.16
+d+ 1.05
3,000
<0 11.7
1.0%
9,000
0.0%
5.00
Arxiv
Dm mathematics
84
1.00
1:35
RoPPajana2 -ildaain
Audie Aa
1.30
8.0
300
0.35
0.30
LU5
(M
Smath (1095))
Mallle (3AM))
Lags desMM)
AL 300
Taey 44M0
Smult (109e))
Medium (11
Thy (414)
Smadl HEMM)
Mediam Chaid
Lare (M/M
(.,00
Smaal reoats
Mediaa (Sild)
Lage A40
NL. 1B1
Faxtaning
3600 KT
M
* 0.16
+d+ 1.05
Factasin size
3,000 XT
Mouur
* 0
1.6%
KT
0.0%
5.00
300
9,000 KT
0.55
0.59
Enron enaill
Wikipedlaa cm
2.8
88
3.0
Rodingamakz valitnee Inee
22
Wilipedi en vddidcioon los
Raalajaaaa whilntoe loe.
122
2.6
13
24
(M))
Small (RP95))
Mallinn (MAa
Lagar (d6M0)
al.r.500
They 4480
Small (lond)
Medium (1186)
They (4MM)
Small (oond
Medien O14a
Laga (666))
(1.00
They (FM
Smaali Lone))
Medium (3B6))
Lagu (eetd))
ML11-19
Xt.
Miouzy
Factasin
5,00
Mouure
0.5
200 KT
0000 KT
0.0%
0.5%
- 3.0%
KT
0,000 XT
0.0%
B+ 0.59
3.0%
500 KT
30,000 KT
900
30,00 KT
Pg19
256
3.4
MedP waaeeenn 1een
NodDbjeme2 wiitetr bee
6.6
Pg vllidatiioos
Gibub salidains
5.2
50
5
Thn prop
Small (1995)
Medion (3AM)
Lagan dell!
XL. m0
Thyy 1100
Bestl (19Rd)
Medso MMM
Lagel Getd
Hi mi
Beall (196))
Mediom DMM
Laps ddl
Al (1.000
The unai
Seall crom
Media (LAB))
28, 30
The
Lager
Figure 18. Losses as a function of model size, datasct size, and data-mixing.
21

---

Scaling Laws for Finetuning and Forgetting in LLMs
Stilig pudiic
Dap
Up
620
50s
prddioss
Wicpelo Up (0
Lawe aaoee
Depici
Depecd
produie
Pa 3250
Beat Govd
prostitnn
Figure 19. Example of finetuning scaling laws for several domains.
22

---

Scaling Laws for Finetuning and Forgetting in LLMs
Stiliy pudiie
Dap
Stilin priddies
Dapsica
Layerd
prddiios
www.ersk..... (p-..
Wicpelo 1p (00
777
Lawe Gtoee
Dipecd
LOOK
Gr
Pg 3250
Nettortind
prostitin
Figure 20. Example of forgetting scaling laws for several domains.
23


Reviews:
[{"review_id": "2ODbzO3ORb", "reviewer": "Reviewer_QR3Z", "summary": "The paper addresses two key challenges in finetuning large language models: (1) overfitting when target domain data is limited and (2) forgetting of pretraining knowledge as the model drifts from its original parameters. The paper studies pretraining data injection as a solution to these challenges, and quantifies its effects through scaling laws.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,  \n  \nWe thank you warmly for your detailed and thorough feedback on our work. We are happy to read that you found that our work is a “simple and effective solution”, that \"claims are supported by the evidence presented\", and that we have \"a central point that is [...] new and relevant for continual LLM pretraining\".\n\n> The paper focuses on continual pre-training scenarios rather than what is commonly called \"fine-tuning\" in modern LLM contexts.\n\nWe will clarify this very important point in the paper. A critical part of our setup is the specialization data scarcity: we only have a small number of tokens to train on, which is not necessarily the case in continual pre-training, while it is a defining aspect of fine-tuning.\n**Our findings also extend to more standard LLM fine-tuning set like openHermes**, please refere to the last part of the answer to rev.R4eh\n\n> How sensitive is the 1% rule to the nature of the pretraining data? \n\nThis is an interesting question; testing it would require using a novel pretraining set and pretraining models on it, which is cumbersome. However, we believe that indeed the diversity of the pretraining data is very important ot counteract forgetting. \n\n> How sensitive is the 1% rule to data quality? \n\nPretraining data injection acts as a regularization. The value of $p$ allows to move along a Pareto front - see [the figure](https://anonymous.4open.science/r/icml2025-figures-3ED2/pareto_front.pdf). We expect that data quality does not have much impact on the fraction p necessary to overcome forgetting, since p only drives the \"regularization strength\". However, we believe that higher data quality in the pretraining set would allows using less of it to fine tune in a scenario where we repeat the pretraining data; we expect improvements in an experiment like fig.6\n\n> Did you observe any qualitative differences in the types of knowledge that were forgotten? Are certain types of knowledge more prone to forgetting?\n\nWe did not look into the details of knowledge that are forgotten; but clarifying how models forget is an extremely interesting future research direction.\n\n> Your scaling law includes parameter B [...]. The values vary significantly across domains. What factors do you believe drive these significant differences?\n\nThis is an interesting point. The parameter B indicates how much pretraining data helps mitigating forgetting. A high B value therefore indicates a strong discrepancy between the pretraining and fine-tuning set, ann indeed B is low for datasets such as free law and wikipedia which are close to the pretraining set, while it is high for dm_mathematics and euro parl, which are far from the pretraining set. We will clarify this in the text. \n\n> Your results suggest that pretraining data injection is a regularizer that improves generalization. How does it compare to standard regularization methods (e.g., L2 regularization) at preventing forgetting?\n\nWe tested an \"anchored\" baseline with a variant of AdamW, using $\\lambda(\\theta\\_t-\\theta\\_0)$ as the weight decay term, instead of the conventional $\\lambda\\theta_t$, where $\\theta\\_0$ are the parameters of the pretrained checkpoint. Note that this is equivalent to using weight decay on the delta between fine-tuned and base model. We added [this illustration](https://anonymous.4open.science/r/icml2025-figures-3ED2/demo_anchored_adamw.pdf) to the paper. For $\\lambda\\in[1e-2, 1e-1, 1]$ the forgetting remains significant (between 15% and 4% more than p=1%, accross all model sizes). For $\\lambda\\geq 1$ the finetuning performance decreases compared to the baseline p=1%. Therefore _more data diversity is superior to regularization in the parameter space_. We added [the plot](https://anonymous.4open.science/r/icml2025-figures-3ED2/github_anchored_adamw.pdf) in appendix.\n\nWe also found that standard weight decay had not impact on forgetting. \n\n> Is there a way you can test some of these ideas on standard Llama/Gemma/Qwen base checkpoints\n\nThank you for the suggestion, this is an interesting future work. We chose to focus on a family of models we had full control over, where we precisely knew the pretraining pipeline and datasets. This allowed us to observe clear trends as a function of scale. In contrast, the precise training pipeline as well as their training data are not available.\n\nHowever, we tested our ideas with other models from the same family, with the **isocurve D=10N**. We finetune on free_law,  and we measure a bootstrapped MRE of **0.57% for forgetting** and **1.14%  for finetuning**. We aded [this curve](https://anonymous.4open.science/r/icml2025-figures-3ED2/forgetting_isocurve10_freewlaw_0.5perc.pdf) and [this curve](https://anonymous.4open.science/r/icml2025-figures-3ED2/finetuning_isocurve10_freelaw_0.5perc.pdf) to the paper.\n\nWe hope that our answers have alleviated your concerns, and we thank you again for your review !", "claims_and_evidence": "The paper's claims are supported by the evidence presented. The central claim that 1% pretraining data injection mitigates forgetting is clearly demonstrated through experiments across multiple model sizes and domains.\n\nScaling law predictions match observed values fairly closely.", "methods_and_evaluation_criteria": "The methods are appropriate for the research question. The choice to measure forgetting via pretraining loss is reasonable. The experimental design systematically varies model size, finetuning dataset size, and pretraining injection proportion.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental setup is sound.", "relation_to_broader_scientific_literature": "The paper's contributions build on work in scaling laws and compute-optimal training. It is also closely related to the literature on catastrophic forgetting and data mixtures. It has a central point that is, to my knowledge, new and relevant for continual LLM pretraining.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The paper addresses a practical problem in language model finetuning with a simple and effective solution (1% pretraining data injection).\n- The experimental design is systematic and thorough, covering multiple model sizes, domains, and dataset sizes.\n- The finding that smaller models are more prone to forgetting than larger ones is interesting.\n- The paper focuses on continual pre-training scenarios rather than what is commonly called \"fine-tuning\" in modern LLM contexts.", "other_comments_or_suggestions": "- The use of the word \"fine-tuning\" throughout the paper may be slightly misleading as it's more commonly associated with post-training (SFT, RLHF...). The paper would benefit from clearer positioning relative to instruction tuning.", "questions_for_authors": "- How sensitive is the 1% rule to the nature of the pretraining data? Would you expect higher or lower optimal injection rates if the pretraining data is highly diverse but the target domain is specialized?\n- How sensitive is the 1% rule to data quality? For example, would it be possible to get away with 0.1% of the pretraining data if it was carefully selected? This question falls outside the scope of this paper, but it could be an interesting direction to take this line of inquiry.\n- Did you observe any qualitative differences in the types of knowledge that were forgotten when p=0% versus preserved when p=1%? Are certain types of knowledge (e.g., factual, linguistic, reasoning) more prone to forgetting?\n- Your scaling law includes parameter B, which indicates the relative efficiency of parameters allocated to pretraining versus finetuning. The values vary significantly across domains. What factors do you believe drive these significant differences?\n- Your results suggest that pretraining data injection is a regularizer that improves generalization. How does it compare to standard regularization methods (e.g., L2 regularization) at preventing forgetting?\n- Is there a way you can test some of these ideas on standard Llama/Gemma/Qwen base checkpoints?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ccgd0mAdfb", "reviewer": "Reviewer_5seN", "summary": "This paper studies a setting (examined previously by Liu 2022, Kang et al 2024, Ibrahim et al 2024) where a small amount of pre-training data is injected in fine-tuning to prevent catastrophic forgetting of the pre-training domain and provide regularization in the target domain. In this setting, the paper develops scaling laws for both pre-training and fine-tuning losses during fine-tuning. The paper builds upon prior work (Kalajdzievski et al 2024), who develop scaling laws for forgetting losses during fine-tuning when pre-training data is not injected during fine-tuning. Results are reported using GPT-2 models of different scales on a number of datasets. The paper provides a scaling law for pre-training loss as a function of model size, number of target tokens, proportion of pre-training data added to the data mixture at fine-tuning, and also a law for the fine-tuning loss. Similar to prior work, the paper finds that a multiplicative form works better than an additive form. The paper shows that injecting even 1% of the pretraining data can prevent a degradation of the pretraining loss and can provide some regularization on the target domain.\n\n## update after rebuttal”\nThe authors have addressed many of my concerns in their rebuttal especially wrt the following:\n* Providing an additional isocurve\n* Reporting the training loss as a function of 'Unique pretrain tokens per unique finetune token'\nHence I am increasing the score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n  \nWe thank you warmly for your detailed and thorough feedback on our work. We are glad to see that you found that \"claims are mostly reasonable\", that \"the methods and evaluation criteria make sense,\" and that we study \"a problem which is valuable to the community\".\n\n\n > Prior work (Kalajdzievski et al 2024) has developed scaling laws for forgetting for fine-tuning when using LoRA. Hence the novel aspect of this work is limited to a) obtaining these laws in a setting where a small proportion of the pre-training data is injected during fine-tuning, b) using full-finetuning rather than LoRA.\n\nAdditionally to those differences, as explained L148, we want to point out that we use many more specialization tasks, model scales, and we measure forgetting through the pretraining loss (which is impossible to do with models like LLAMA 2 for which the training data is unavailable).\n\n> Though the paper presents a large number of results for a single model (Gpt-2), it is unclear if these laws will carry over to alternative LLMs.\n\nIndeed, we acknowledge that we only consider autoregressive decocoder-only transformers. Note that the architecture we use is widely used, up to minor changes (see e.g. sec.2.2 of [1]). Testing the impact of these minor architectural changes like use of RoPE and swiglu on the scaling laws is indeed an interesting future research direction.\n\n> The paper does not explore the full grid of model size and data set size i.e. (N, D) but rather an isocurve D=100N. While this is a reasonable choice for computational reasons, it would have been useful to study the full grid at least for a single domain to understand how far the isocurve is from the optimal value for this domain.\n\nThanks for raising this point ! We tested our idea with another **isocurve D=10N**. We finetune on free_law,  and we measure a bootstrapped MRE of **0.57% for forgetting** and **1.14%  for finetuning**. We aded [this curve](https://anonymous.4open.science/r/icml2025-figures-3ED2/forgetting_isocurve10_freewlaw_0.5perc.pdf) and [this curve](https://anonymous.4open.science/r/icml2025-figures-3ED2/finetuning_isocurve10_freelaw_0.5perc.pdf) to the paper.\n\n> it would be advantageous to discuss whether alternative functional forms of the regression model may fit the data better\n\nWe are looking for a forgetting law that (i) should yield a zero finetuning when $D^{ft}$ is zero (ii) does not exhibit too many arbitrary parameters.  For example, we found that the $+E$ term is entirely explained by the re-warming of the model (since the finetuning LR was x3 times terminal LR of pretraining). As a matter of fact, measuring it that way instead of regressing its value __decreases__ the MRE from 0.48% to 0.40%. See [this table here](https://anonymous.4open.science/r/icml2025-figures-3ED2/estimate_E.png).\n\nWe also tested additive laws, as mentioned in p8. Other laws, like $A\\frac{D^{\\beta}(1-p)^{\\kappa})}{N^{\\alpha}}+E$ had a higher error of 0.67% despite having one more parameter $\\kappa$. We added these results in appendix. \n\nFurthermore, to strengthen the statistical significance of our results, we also compute and report the bootstrapped MRE (we sample 125 measurements with replacement from the pool, and average the result over 128 independent samples). \n\n > It would be useful to report the extent of repetition in the pre-training dataset\n\nGood call! Since that the bottom of the U-curve is typically reached within a few dozen epochs, and that p=1%, the number of repetitions quickly falls to 1. We updated [the figure](https://anonymous.4open.science/r/icml2025-figures-3ED2/ablation_pretraining_size.pdf) with “Unique pretrain tokens per unique finetune token” in x-axis. We discover that 0.3 unique pretrain tokens per unique finetune token are typically sufficient.\n\n\nWe hope that our answers have alleviated your concerns, and we thank you again for your review !\n\n[1] Touvron, Hugo, et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).", "claims_and_evidence": "The claims are mostly reasonable except for the following:\n\n* The paper does not explore the full grid of model size and data set size i.e. (N, D) but rather an isocurve D=100N. While this is a reasonable choice for computational reasons, it would have been useful to study the full grid at least for a single domain to understand how far the isocurve is from the optimal value for this domain. \n* Though the paper presents a large number of results for a single model (Gpt-2), it is unclear if these laws will carry over to alternative LLMs.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense.", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "Yes, I checked. They seem reasonable.", "relation_to_broader_scientific_literature": "* Building upon prior work (Liu 2022, Kang et al. 2024, Ibrahim et al. 2024), this paper investigates a scenario where a small proportion of pre-training data is incorporated in fine-tuning mitigate catastrophic forgetting and improve target domain regularization.\n* Prior work (Kalajdzievski et al 2024) has developed scaling laws for forgetting for fine-tuning when using LoRA. Hence the novel aspect of this work is limited to a) obtaining these laws in a setting where a small proportion of the pre-training data is injected during fine-tuning, b) using full- finetuning rather than LoRA.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n* Studies a problem which is valuable to the community: how to fine-tune while preventing catastrophic forgetting.\n* Develops scaling laws for both fine-tuning and pretraining loss as a function of model size, finetuning data size and proportion of pretraining data injected during fine-tuning. This is a setting that has not been examined previously in the context of developing scaling laws.\n* Proposes a practical solution to prevent catastrophic forgetting and obtain some regularization by injecting a small proportion of pre-training data during fine-tuning.\n* Examines the effect of pre-training data repetitions on forgetting\n\n\nWeaknesses:\n* Prior work (Kalajdzievski et al 2024) has developed scaling laws for forgetting for fine-tuning when using LoRA. Hence the novel aspect of this work is limited to a) obtaining these laws in a setting where a small proportion of the pre-training data is injected during fine-tuning, b) using full-finetuning rather than LoRA.\n* Though the paper presents a large number of results for a single model (Gpt-2), it is unclear if these laws will carry over to alternative LLMs.\n* The paper does not explore the full grid of model size and data set size i.e. (N, D) but rather an isocurve D=100N. While this is a reasonable choice for computational reasons, it would have been useful to study the full grid at least for a single domain to understand how far the isocurve is from the optimal value for this domain. \n* Similar to prior work, the paper reports MRE for the additive and multiplicative scaling laws. However, it would be advantageous to discuss whether alternative functional forms of the regression model may fit the data better.\n* Figure 6: It would be useful to report the extent of repetition in the pre-training dataset.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "x4IawJgToY", "reviewer": "Reviewer_wqv9", "summary": "The paper studies the domain adaptation and forgetting effects of language model finetuning by deriving scaling laws that quantify these two phenomena. It shows that one can accurately predict the finetuning performance and the forgetting of the pretraining set of large language models, as a function of the model size, the number of available finetuning tokens, and of the fraction of pretraining data injected into the finetuning data mixture.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n  \nWe thank you warmly for your detailed and thorough feedback on our work. We are glad to read that \"The claims made in the submission are supported by clear evidence\" and that our method \"makes sense and provides novel insights.\"\n\n> While the proposed curve fits these points well, the paper needs to show the scaling law can accurately predict the loss for a model that is much larger than 1.27B\n\nThis is indeed an important point, and we are happy to report that our scaling laws allow us to *extrapolate*, that is, as you mention, predict large scale behavior from small scale.\n\nWe could predict the forgetting for **650M and 1.3B models**, on **9M and 30M unique tokens**, with a bootstrapped MRE of 0.83% - **using only models no bigger than 350M parameters, and finetuned with no more than 3M unique tokens**. We added [this table](https://anonymous.4open.science/r/icml2025-figures-3ED2/extrapolate.png) in the paper.\n\n> I am unsure whether there is mathematical expression to characterize the U-curve\n\nWe agree that’s a fruitful research question. However, finding how many steps/repetitions are required to find the bottom of the U-curve is by no means trivial, since it depends on both the domain, the model size, and the number of tokens. Nonetheless, assuming fixed number of epochs, we benchmarked the scaling laws of “Scaling Data-Constrained Language Models”. We found it yielded accurate predictions, correct in our setting too. We will clarify this in the paper.\n\nTherefore, it might be possible to predict when the bottom of U-curve will be reached (find the **argmin** over time) based on the performance improvements after a few epochs. Our current work doesn’t need to characterize the full U-curve: it is sufficient to estimate the minimum value reached at the bottom (find the **min** over time).  \n\n> It's unsure what's the use case for the proposed scaling law\n \nWe will clarify two important outcomes from our scaling laws.\n\nA) The proposed scaling law allows to quantify forgetting as function of scale, and notably to extrapolate, as you mentioned before. Testing at small scales allows you to not be surprised with what happens at larger scales. \n\nB) We also believe our work brings some understanding to the phenomenon of forgetting. An interesting consequence of our functional form reveals that the leading cause behind forgetting might be related to the parameters count $BpN$. This suggests that forgetting is primarily due to limitations in network capacity. This is also confirmed by the fact that smaller models suffer the most: they lose up to 95% (!) of the pretraining progress when forgetting (i.e, the pretraining validation loss reverts to a point reached at 5% of pretraining), while bigger models only lose 20% of the progress. We added [this plot](https://anonymous.4open.science/r/icml2025-figures-3ED2/arxiv_forget_hours.pdf) in the paper, which quantifies forgetting with a fraction of pre-training cost lost.\n\nOverall, this balances some findings of “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws”, which suggested that smaller models should be preferred due to their lower inference cost. We will make these points in the paper.\n\nWe thank you again for your review, and we hope that we have alleviated your concerns !", "claims_and_evidence": "The claims made in the submission are supported by clear evidence based on my understanding of scaling law - I would suggest having people familiar with scaling law double-check the claim due to my low confidence on this area.", "methods_and_evaluation_criteria": "Using scaling law to understand the dynamics between the domain adaptation and forgetting effects of language model finetuning makes sense and provides novel insights.", "theoretical_claims": "Yes. I have carefully read the figures related to the claim that there are scaling laws that quantify the dynamics between the domain adaptation and forgetting effects of language model finetuning from various target domains.", "experimental_designs_or_analyses": "How the paper makes the scaling law plot is sound. However, I am wondering whether the current experiments are enough to demonstrate the robustness of the proposed scaling law:\n1. The scaling law is fit with data points coming from model size of 41M, 109M, 334M, 665M, 1.27B. While the proposed curve fits these points well, the paper needs to show the scaling law can accurately predict the loss for a model that is much larger than 1.27B (due to resource constraints, the authors may consider experimenting on a 7B-parameter model).\n2. While the U-curve for generalization-memorization tradeoff (Figure 2) is intuitive to interpret, I am unsure whether there is mathematical expression to characterize the U-curve. This is important for understanding the observed tradeoff can extrapolate to region not covered by the current datapoints.", "relation_to_broader_scientific_literature": "To the best of my knowledge, this is the first paper that studies scaling law in continual learning and catastrophic forgetting.", "essential_references_not_discussed": "I do not have specific reference in mind.", "other_strengths_and_weaknesses": "Strengths: The paper is novel and deepens the understanding of catastrophic forgetting in continual learning.\n\nWeaknesses:\n1. It's unsure what's the use case for the proposed scaling law. For traditional scaling laws, they could help us understand how to better distribute budget (e.g., parameter size, pre-training token size). But for scaling laws proposed in this work, the scenario is different, as the paper also shows that a small percentage of pre-training data could already significantly mitigate the forgetting.\n    - I am willing to adjust my assessment if this point could be addressed.", "other_comments_or_suggestions": "1. Figure 7 caption: \"(Zhang et al., 2024)\" -> \"Zhang et al., 2024\"", "questions_for_authors": "See my comment above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kG4yUrs1fN", "reviewer": "Reviewer_R4eh", "summary": "This paper presents a study of scaling laws for fine-tuning, in the particular case where replay data (in the form of pretraining data) is available. The paper models the forgetting loss as a function of the replay data, fine-tuning data, and number of parameters.\nIt also extends the scaling law of existing work on modeling the fine-tuning loss, showing that the amount of injected replay data barely impacts the final validation loss on the fine-tuning data. They also show that as low as $p$=1% of the pre training data is sufficient to not lose performance on pretraining data, and that the scaling coefficient $p$ depends on the nature of the fine-tuning domain. The experiments are comprehensive and validate the hypotheses.\n\n## Update after rebuttal\n\n I am satisfied with the authors' response, and increased my score to Accept. As mentioned in the discussions below, I do think it is important clarify the distinction between \"fine-tuning\" in the general sense versus training on domain specific data in their updated version to highlight the true setting where this study is useful.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,  \n  \nWe thank you warmly for your detailed and thorough feedback on our work. We are glad to read that \"The experiments are comprehensive and validate the hypotheses\", that \"most claims and the evidence presented are pretty solid\", and that \" experimental design choices are clear\".  Thank you as well for the additional reference. \n\n> I would recommend adding the hyper parameter values for each configuration and other related details for reproducibility.\n\nThanks! We added a summary table in appendix. \n\n> I would also recommend adding downstream task results for these models as a sanity check\n\nPlease find [here](https://anonymous.4open.science/r/icml2025-figures-3ED2/barplot_arceasy.pdf) and [here](https://anonymous.4open.science/r/icml2025-figures-3ED2/barplot_MMLU.pdf) results for the `ARC_easy` and `MMLU` task, for the pretrained checkpoint, and for models finetuned on `dm_mathematics`. Performance degrades on the generalist questions of ARC easy, but improves on MMLU which is more aligned with `dm_mathematics`.   \n\nFurthermore, on the domain `dm_mathematics` we evaluated the quality of 75 checkpoints (all model sizes above 350M) on the  `arc_easy` task, in a 0-shot setting. We take the z-score at 99% significance threshold, and report the best result below (ns = not significant, numbers are accuracy gain over p=0%). **TLDR**; the difference is significant for 15 experiments, oscillating between 3.7% and 8.7% more, and pretraining data injection always helps. As noticed before, the bigger the finetuning dataset the more important the forgetting. For these bigger datasets (>9M tokens), injecting pretraining data is crucial.\n\n|                      |    0.1% |    0.5% |      1% |      5% |\n|:---------------------|--------:|--------:|--------:|--------:|\n| ('medium', 307200)   | ns     | ns     | ns    | ns     |\n| ('medium', 921600)    | ns     | ns     | ns    | ns     |\n| ('medium', 3072000)  |   3.872 |   4.461 |   4.082 |   4.082 |\n| ('medium', 9216000)  |   4.125 |   4.082 |   4.04  | ns     |\n| ('medium', 30720000) |   4.798 |   6.566 |   4.798 |   5.429 |\n| ('large', 307200)    | ns     | ns     | ns    | ns     |\n| ('large', 921600)    | ns     | ns     | ns    | ns     |\n| ('large', 3072000)   | ns     |   3.872 |   4.588 |   5.093 |\n| ('large', 9216000)   |   3.788 |   4.966 |   5.345 |   4.588 |\n| ('large', 30720000)  |   6.944 |   7.744 |   7.239 |   8.375 |\n| ('xl', 307200)       | ns     | ns     | ns    | ns     |\n| ('xl', 921600)       | ns     | ns     | ns    | ns     |\n| ('xl', 3072000)       | ns     | ns     | ns    | ns     |\n| ('xl', 9216000)      | ns     | ns     |   3.998 | ns     |\n| ('xl', 30720000)     |   4.756 |   5.008 |   5.471 |   5.303 |\n\n> From what I understand, in 4.4, the number of unique tokens available varies but they are upsampled such at =1%? If yes, it would be helpful to make this clear.\n\nYes you’re correct - we updated the section and [the figure](https://anonymous.4open.science/r/icml2025-figures-3ED2/ablation_pretraining_size.pdf) x-axis. Each batch is built following the process described in section 3.3: each sequence is picked from the pretraining split with probability 1%, and from the specific domain with probability 99%. The pretrained set is capped to the number of tokens wanted, and then repeated as many times as necessary.\n\n>  Could the authors elaborate on why they chose the Pile as the fine tuning task, as opposed to other SFT/Instruction-Tuning/Alignment datasets which are conventionally used?\n\n\nWe wanted to simulate the setup in which a company fine-tune a model on raw data from internal documentation. The highly-specialized content of The Pile (which is partitioned semantically) allows to simulate the scenario, unlike typical typical IFT datasets that are generalist.   \n\nNonetheless, we agree that the Instruction Finetuning scenario is also of interest; thank you for your suggestion. We perform instruction finetuning on the OpenHermes dataset - 3M tokens in the train split (95% of the total). We finetune the model to perform next token prediction of the output, conditioned on the 
 prompt input 
 prefix. We added [finetuning curves](https://anonymous.4open.science/r/icml2025-figures-3ED2/openhermes_finetuning.pdf) and [forgetting curves](https://anonymous.4open.science/r/icml2025-figures-3ED2/openhermes_forgetting.pdf) in appendix.\n\nWe report the fitted scaling laws parameters here:\n\n\n**Finetuning**:\n\n| Domain     |   alpha |      beta |       A |      E |      HeldOutMRE |\n|:-----------|--------:|----------:|--------:|-------:|-------------:|\n| Openhermes | 0.17582 | 0.0286171 | 64.2788 | 0.4585 | 0.59% |\n\n**Forgetting**:\n\n| Domain     |    alpha |     beta |    A |    B | HeldOutMRE   |\n|:-----------|---------:|---------:|-----:|-----:|:-------------|\n| OpenHermes | 0.793364 | 0.266993 | 5513 | 8584 | 0.29\\%       |\n\n\nWe thank you again for your review, and hope that our answer has alleviated your concerns.", "claims_and_evidence": "I think most claims and the evidence presented are pretty solid. Models between 40M-1B parameter ranges are studied which is pretty comprehensive. There are extensive results on each factors and how they affect the scaling.", "methods_and_evaluation_criteria": "- My main complaint about this paper is the notion of the fine tuning task. The authors consider sub-domain splits of the Pile, which is a dataset that is generally used as a pre-training dataset. In practice, many fine-tuning datasets and instruction tuning datasets, such as Alpaca, contain a range of examples from diverse domains.", "theoretical_claims": "Given the empirical nature of the work, there are no explicit theory or proofs. The scaling law for fine-tuning is drawn from prior work. For forgetting equations, the choices seem fair.", "experimental_designs_or_analyses": "All experimental design choices are clear, apart from the fine tuning dataset as discussed above. The search space over the pretraining data, fine-tuning data, and model scale covers a broad range.", "relation_to_broader_scientific_literature": "To the best of my knowledge, this is the first paper that studies scaling laws for fine-tuning by also considering replay data (i.e pre-training data injection) during fine-tuning. [1] studies scaling laws for forgetting during fine-tuning, but does not consider replay data. Similarly, [2] only studies scaling laws on the fine-tuning task, while this paper also attempts to model the loss on the pre-training data after fine-tuning.\n\n[1] https://arxiv.org/pdf/2401.05605\n\n[2] https://arxiv.org/abs/2402.17193v1", "essential_references_not_discussed": "[1] https://arxiv.org/pdf/2406.01375v1 - proposes scaling laws for domain specific training after pre-training", "other_strengths_and_weaknesses": "Strengths: \n\nOverall, a good paper that the community will find useful.\n\nWeaknesses: \n\nAs mentioned I think the paper can be strengthened further with experiments on actual fine-tuning tasks. As such, this feels more like domain-specific training, whose scaling laws some existing work has studied [1]. I will however note that [1] has only studied domain specific pretraining i.e. in the dataset scale of billions of tokens. But, the fine-tuning setup here is still a little artificial than the type of fine-tuning that is conventionally followed (SFT/Instruction Tuning). Given this observation, I lean towards a weak accept for now.\n\n[1] https://arxiv.org/pdf/2406.01375v1", "other_comments_or_suggestions": "1. I would recommend adding the hyper parameter values for each configuration and other related details for reproducibility.\n2. I would also recommend adding downstream task results for these models as a sanity check. This would give a better sense of how the models perform in downstream applications before and after fine-tuning (eg. MMLU, instruction following tasks etc)", "questions_for_authors": "1. Could the authors elaborate on why they chose the Pile as the fine tuning task, as opposed to other SFT/Instruction-Tuning/Alignment datasets which are conventionally used?\n2. From what I understand, in 4.4, the number of unique tokens available varies but they are upsampled such at $p$=1\\%? If yes, it would be helpful to make this clear.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper derives "scaling laws" on how a finetuned model either overfits to the target task, or "drifts away" from the pre-trained model. One natural idea to "fix" such an issue is to add more of pre-training data; the paper shows that adding a small amount of such data helps prevent drifting and overfitting. The reviewers are generally positive about the evaluation and methodology. The problem itself is an interesting one, of broad appeal.
Decision: Accept (poster)