D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Authors: Haoran Que
Jiaheng Liu
Ge Zhang
Chenchen Zhang
Xingwei Qu
Yinghao Ma
Feiyu Duan
ZhiqiBai
JiakaiWang
Yuanxing Zhang
Xu Tan
Jie Fu
Jiamang Wang
Lin Qu
Wenbo Su
Bo Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Scaling Law
Continual Pre-training
Primary area: natural_language_processing

Abstract: Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model’s fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1\% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.

D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Haoran Que*! Jiaheng Liu Ge Zhang*3.7, Chenchen Zhang", Xingwei Qu Yinghao Ma Feiyu Duan', Zhiqi Bai², Jiakai Wang". Yuanxing Zhang", Xu Tan",
Jie Fu", Jiamang Wang". Lin Qu², Wenbo Su², Bo Zheng'
Taobao & Tmall Group of Alibaba, Alibaba Group, University of Waterloo UUnieersity of Manchester, SQUUL, HKUST, "M-A-P fquehaoran.que qhr, 1jh411989) com, gezhang @umich.edu
Abstract
Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model's fundamental understanding of specific downstream domains (e.g.. math and code). For the CPT on domain-specifie LLMs, one important question how choose the optimal mixture ratio between the general- corpus (e.g., Dolma, Slim-pajama) and the downstream comain-corpus Existing methods usually adopt laborious human efforts by grid-searching on a set mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs different sizes. Specifically, by fitting the D-CPT Law, arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs limited experiments. Moreover, Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1% of the normal training costs) are needed for the target domains. Comprehensive experimental results six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.
easily predict the general and downstream performance of
also extend our standard D-CPT
Introduction
Continual Pre-Training (CPT) is essential part of training better Large Language models (LLMs). In this work, we mainly focus on Domain-specific CPT (D-CPT), which aims to enhance the fundamental understanding abilities of the specific downstream domains and has been widely used in existing works [49 30). In practice, for D-CPT, we usually need to collect high-quality domain-corpus enhance the downstream performance and general-corpus mitigate catastrophic
forgetting on the general abilities 13 40
54
20 53 Therefore, how to determine the data corpus and general-corput plays an important role in
47
32
composition or mixture ratio of the producing well- well-performed domain-specific LLMs. Besides, grid-searching on the mixture ratios requires heavy GPU consumption costs. and cannot always obtain the optimal ratio under limited GPU usage. Recently, Scaling Law has been widely used for performance prediction [3] 27 26
* First three authors contributed equally. Corresponding Author: Jiaheng Liu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

General
Domain sperific
2.355
Mizzo peats
Inlans pelnts
D- 340B
63548
.44
cannes points
frting
producioo core
2- 6.5500
10,48C3
2-320
Dor www.atooorgeeee wwlittatitmm love
General valddrsoon
2.826
.05
2,000
2.77
2.780
2.723
2,700
04
1.0
041 Domaiz-torpus mixture ratio
1.0
General-corpas
Figure 1: Illustration of the performance of D-CPT Law. (Leff): The curves show the relationship between Lg and rg under different dataset sizes D for Qwen1.5-L.8B model. CPT data mixture of code-corpus and general-corpus Here, Lg represents the loss on the general-corpus validation set, while rg indicates the percentage of the general corpus in the training data. The dashed curves denote the curves predicted by D-CPT Law, circular markers and star markers are fitting data points and unseen validation points, respectively. (Right): These curves are the corresponding results between the code-corpus validation loss La and the percentage of the code-corpus data rd.
which can be used to find the optimal dataset size and model size under the given GPU consumption costs. Therefore, for D-CPT, can we find the optimal mixture ratio in the training corpus using the Scaling Law enhance the performance of domain-speciftt tasks?
To address the above question, in this work, we investigate the Scaling Law of D-CPT and propose the D-CPT Law to find the optimal mixture ratio with limited training costs for LLMs with different sizes. Specifically, inspired by the robust predictive ability Scaling Law across various scales, we first perform experiments under diverse mixture ratios and several relatively small model and data scales. Following the Chinchilla Scaling Law, we then introduce the mixture ratio r into the D-CPT Law, where the parameterization is defined as follows:
A B. C L(N, D.N. = No
+
+ p.7) where r' = r
(1)
D
where is used to guarantee the stability of when r near zero. Based on Equation I for a model with model size N, dataset volume D and mixture ratio we can accurately predict the validation loss L. Note that when r denotes the domain-corpus mixture ratio ra. L means domain corain validation loss Lg. Similarly, general-corpus validation loss Lg also follows the law relationship with the general-corpus mixture ratio rg. To illustrate our D-CPT Law clearly, as shown in Figure I we take the code domain an example and provide the fitting results on the general and domain-spectnn settings, where we validate the fitting accuracy on different mixture ratios under model with different dataset sizes D. Our main contributions summarized as follows:
(1). To show the effectiveness and generalizability of D-CPT Law, we perform extensive experiments using model sizes from 0.5B to 4B parameters, dataset sizes from O. to 26B tokens, and mixture ratios from
1. The experiments show that the D-CPT law exhibits high fitting accuracy with Huber loss 28 lower than 0.02 and R² [9] greater than 0.97. Besides, experiments on generalizability show that D-CPT Law not only inherits model size and dataset size generalizability following previous Scaling Law, but also precisely predicts performance for differentt mixture ratios.
(2). Despite the effectiveness in an in-domain setting, where fit the D-CPT Law based on data points from one downstream domain, we also apply our D-CPT Law in the cross-domain setting, which denotes that we use the data points from multiple domains predict the performance of unseen domains. Specifically, we first introduce the Domain-specific Learnable Coefficient (DLC) to denote the doomain-speciffi parameter of each domain and integrate the DLC into the D-CPT Law. We name this new law as a new domain, we can easily derive the D-CPT Law for this new domain. our experiments, we fit the Domain D-CPT Law using data points from 4 domains and apply the
D-CPT Law. In this way, if we can obtain the DLC of

---

In-domain
Small-scale CPT
Preparation
D-CPT Law
Usages
Code
Mediaal
Musk
4(N.0.))
Cross-domain
Cross-Domain D-CPT Law
Limited domain-sseciiie deta
Resource Allocation
Figure 2: Illustration of D-CPT Law and Cross-Domain CPT-Law pipeline. (Upper): In D-CPT Law, first collect domain-corpus and general-corpus, and conduct experiments under small-scale experimental setup to gather empirical data points to fit the D-CPT Law. After that, we can predict the model's performance in large-scale experimental settings. (Lower): In Cross-Domain CPT-Law, for an unseen downstream domain, like Physics, we can calculate its Domain-specific Learnable Coefficient value and incorporate it into the fitted Cross-Domain D-CPT Law derive the D-CPT Law for this new domain. Based on the D-CPT Law, we introduce three application scenarios: optimal mixture the trade-off between general and domain-speotin abilities, optimal mixture for limited domain-specific data, and resource allocation in Section 4.3
D-CPT Law to predict the remaining 2 domains. The results show that DLC can represent the specific information for each downstream domain well, enabling efficient and effective fitting performance for the cross-domain setting and significantly reducing training costs for new domains.
(3). To show the real-world usages of the D-CPT Law, we apply our D-CPT Law three important scenarios: optimal mixture on the trade-off between general and domain-specific abilities, optimal mixture for limited Nomain-specific data, and resource allocation setting in Figure 2 (Details are provided in Section 4.3
Background
Following previous work 433
categorize the objectives of Scaling Law as Allocation and Return. Specifically, (1). Allocation: What is the optimal allocation of model size N and dataset size D given fixed compute budget? (2). Return: What is the expected return on incremental resources?
The first objective on Allocation is as follows:
argmin L(N, D) s.t. FL.PPsNN, D) = C.
N.D
Equation given fixed compute budget C, the objective is to find the optimal model size and dataset size D that minimize the loss. The second objective on Return fundamentally depends on the generalizability of Scaling Law to accurately predict beyond the fitting data points.
Chinchilla Scaling Law Hoffmann et al. [27 propose parameterizatton as follows:
A
B
L=
No
D
where (E,A,B.o,3) are fitting parameters (See Appendix D for more details).
3 Methods
In Figure D-CPT Law aims to investigate the behaviors of the Domain-specife Continual Pre- Training scenario with respect different mixture ratios, and the objective the D-CPT Law is to analyze appropriate parameterization of law that represents the relationship of validation loss L

---

with respect to model size N, dataset size D, and mixture ratio In this section, we first discuss the D-CPT Law in the in-domain setting (Section B.Ip. where the fitting and testing data points are from the same domains. Then, we propose to adapt D-CPT Law to the cross-domain setting (Section 3.2). where the fitting and testing data points are from multiple domains, and introduce the Cross-Domain D-CPT Law, where a new term Domain-specific Learnable Coefficient (DLC) is used.
3.1 D-CPT
As the training data includes mixture of general-corpus and domain-corpus, we introduce two mixture ratios (i.e., general-corpus mixture ratio rg and domain-corpus mixture ratio rg). Correspond- ingly, we define two validation losses (i.c., general-corpus validation loss Lg and domain-corpe validation loss La). Therefore, can derive two D-CPT Laws (i.e., Lg (N, D, rg) and La(N, D. r.)).. For convenience, we directly use r and L(N, D,r) r) as default notations for D-CPT Law. Besides, in Appendix D Chinchilla Scaling Law provides greater interpretabilityy and clarity when compared to OpenAI Scaling Law. Thus, we choose Chinchilla Scaling Law as the foundational parameterization for D-CPT Law. In addition, since Scaling Law aims to fit data points, their parametric forms should be intrinsically related to the observed trends in the data points. Based on previous works and data trends with varying N, D and r, we have summarized essential requirements for D-CPT Law:
Adaptability: D-CPT Law is valid for values of between 0 and 1.
Explicit trends: Based on the results across varying values of N, D, and r, observed the following explicit trends data points:
aL ON
OL 3D <0
dL
<0,
<0. 0.
Dr
The first two trends are consistent with the previous Chinchilla Scaling Law, and the third trend also has intuitive explanation. A larger indicates higher proportion of valid corpus in the training corpus, leading to a lower L. Details are provided Appendix E.
Implicit trends: We further discover inherent connections between r, D and L as follows:
L
0.
aD0r
For detailed explanations, please refer to the Appendix E.2
When is fixed, the D-CPT Law is supposed transform into the Chinchilla Scaling Law. In this way, D-CPT Law can inherit the excellent features of Chinchilla Scaling Law and address the issues resource allocation discussed in Section 2
Consistency:
To satisfy these requirements, have compared multiple parameterizations in Section 4.2 and eventually, we propose the parameterization as follows:
B. D
C 7
DNN - E
where r'r+te.
(6)
No
In Equation 6 L,, A, C, 3, n.e)) are the fitting parameters and we use L-BFGS ) to fit the D-CPT Law due its suitability for large-scale optimizations. As shown Section 4.2, the Equation 6 can accurately fit the trends of data points at any scale and demonstrates strong performance in both effectiveness and generalizability. Besides, it effectively meets the aforementioned requirements. (Please see Appendix E.3 for more details on the mathematical derivation.)
3.2 Cross-Domain D-CPT Law
Apart from the in-domain setting for D-CPT Law, we also investigate the cross-domain setting and extend the D-CPT Law to the Doosa-Domain D-CPT Law, which aims to reduce the training costs of the D-CPT Law significantly. Specifically, although the D-CPT Law collects data points using small LLMs, the GPU resource and time costs are still relatively substantial, which limits the applications of the Scaling Law. Therefore, in our Cross-Domain D-CPT Law, we first define Domain- speciffc Learnable Coefficient (DLC) K for each domain, which measures the learnability

---

for specific domain (See Section 4.4 4 for more details.). Then, we incorporate the K into the D-CPT Law and obtain the Cross-Domain D-CPT Law, which defined as follows:
A No
B. D
C
F K
D,,,,r E+
where -'=+t.
(7)
C, (EF.AB C, are fitting parameters. Thus, for unseen domain, we only need to calculate the DLC with modest costs, which substantially increases the domain generalizability of D-CPT Law. Besides, Cross-Domain D-CPT Law has the following features:
In Equation 7
Uniformity: Once we calculate the K value of an unseen domain, we can convert Cross- domain D-CPT Law into normal D-CPT Law as follows:
A Na
B-r"
C y
F
L(K = Ko) Eo
+
where Es - 1
T' -
D
Therefore, the Cross-Domain D-CPT Law inherits all features of the D-CPT Law,
Monotonicity: K denotes the learnability of specific domain, which aligns with the intuition that more learnable domain yields lower validation loss. Meanwhile, the Cross- domain D-CPT Law confirms a monotonic decrease with respect to K, i.c.
aL
RF = KAtI < O.
(8)
ŌK
After confirming the parameterization Cross-domain D-CPT Law, essential to identify representation for K that accurately quantifies domain's learnability. The representation of K is supposed to be accessible, distinct and robust. Specifically, first, "Accessible" denotes that is easy to obtain for an unseen domain with low costs. Second, "Distinct" indicates that K values must exhibit significant variance across domains to ensure fitting accuracy and maintain clear distinctions between domains. Third, "Robust" means that the representation of K enhances the effectiveness and generalization ability of Cross-domain D-CPT Law. In Section 4.4] we compare several variants of the representations of K, and the final representation is determined follows:
Wi
K
W2 X k2,
where wi and we are fitting parameters, Kj represents the initial validation loss for an unseen domain, and l2 denotes the rate of decline in validation loss.
Experiments
4.1 Experimental Setup
Data Setup verify the effectiveness and generalizability of D-CPT Law and Cross-Domain D-
CPT Law, we have prepared the 6 different downstream domains, which include Code (39). Math (5]
Law (4), Chemistry [10, Music 44 and Medical [33]. For general corpus, we use Dolma [48]
All the tokens of these training datasets are sufficient, the experiments are not performed under data-constrained setting. Besides, we build a high-quality and held-out validation set for each domain. [ for more details.)
(See Appendix
Model Setup We the 1.5 series due to its robust performance in both English and Chinese [6]. Furthermore, 1we has multiple open-sourced and well-performed pre-training base models. Specifically,, select Owen-1.5-0.5B, Qwen-1.5-11 1.5-1.8B, and Qwen-1.5-4PP our base models to perform the continual pre-training for multiple downstream domains.
Note that lower DLC means easier to learn for the corresponding domain.

---

Table 1: Mean performance across five parameterizations over six domains. "G" and "D" denote general and downstream domains. Detailed results on all domains are shown in Appendix
Huber loss
R
fitting parameters
Parameterization
G
L
0.0064
0.0169
0.994467
0,976700
L2
0.0050
0.0166
0,996483
0,978283
La
0,0048
0.0157
0.996750
0.979633
L4 0.0066
0.0160
0,993567
0.978367
Ls 0.0328
0.0438
0.9496
0.9512
Training Setup We follow Chinchilla [27] to fix model sizes and vary the number of training tokens for data point collection. Specifically, we test the validation loss every 1,000 steps 2 and the total training steps are 200k. Then, we establish 9 mixture ratios between general-corpe
and domain- corpus as follows: (0:10, 1:9, 2:8, 3.3:6.7, 5:5, 6.7:3.3, 8:2, 9:1, 10:0]. Note that all experiments conducted with the same learning rate schedule (Hyperparameters can be found in Appendix
we use validation loss as the performance indicator. To compare various parameterizations, we follow 288 [4] to utilize the R² and Huber loss evaluation metrics. Specifically, first, the coefficient of determination (i.e., 1²2) indicates the fitting quality and typically ranges from 0 to 1, where a higher value means better explanatory power of the regression model. Second, Huber loss combines the properties of mean squared error and mean absolute error, which is particularly useful for regression with outliers. Similarly, Huber loss also assesses the fit qualities of different parameterizations, where lower Huber loss shows better fitting performances.
Metrics Following
4.2
D-CPT Law
In Section 3.1 an ideal parameterization should meet four requirements (i.e., adaptability, explicit trends, implicit trends, and consistency), and we define the following five parameterizations:
A
C
A
B
C
Br
C
1
E
No
D8
+
pyy
L2 = E+
No
+
D8
+
rM7
Ls =
E+
No
+
DR
+
mty
A
B.
C
A
B
=+ No
+
Ls = E+
E+
D
Na
(rD+
where (N.D.r} are variables and others are learned parameters fitted by L-BFGS algorithm which is the same Chinchilla Scaling Law.
Effectiveness As shown in Table 1 we present the performance of five different parameterizations. In effectiveness settings, we use entire data points for fitting with the aim of validating the effectiveness of various parameterizations. In Table we observe that although L5 has the fewest fitting parameters, its performance is significantly less impressive compared to the others. L₁ and L4, having relatively fewer fitting parameters, still fall short in performance compared to T2 and Lg. Moreover, T.1 fails to meet the requirements for implicit trends, while LA does not satisfy the explicit trends requirement. Finally, the results of L2 and L3 are comparable, but Le does not meet the requirements for consistency. Therefore, we choose La for D-CPT Law. Moreover, Figure 3 shows the robust effectiveness of L3 across varying dataset sizes, mixture ratios, model sizes, and domains.
Model Size Generalizability: Our main experiments focus on model sizes: 0.5B, 1.8B, and 4B. We use 3-fold cross-vale volssvvaliationn to evaluate the model size generalizability of D-CPT Law, and the average results across domains are shown in Table For example, we fit D-CPT Law with data points from 0.5B, and 1.8B and evaluate the Huber loss and R² for 4B, In Table 2 we observe that D-CPT Law can generalize well across model sizes and L3 shows the best performance. Besides, we conduct experiments the unseen 7B size (i.c.. Qwen- 1.5 7B), and observe that D-CPT Law can accurately predict the general-corput validation loss with general-corpel mixture ratio of 0.2 in Figure 4
Training steps number of training tokens follows: t steps t 64 048 10

---

Dataset Size Generalizability: Our main experiments cover dataset sizes from O. to 26B tokens, and we also utilize 3-fold cross-validation approach. The data points are uniformly divided into three segments, with 2/3 used for fitring the model and the remaining 1/3 for testing. In Table 3 we report the average results across domains, and observe that L shows notably enhanced dataset size generalizability.
Figure 4: Lg with respect to D, domain-corpus is code, rg 0.2, N = 7B.
Table 2: Model Size Generalizability.
Huber loss
It' 1
Parameterization
G
D
D
L
0.0055
0.0172
0.9521
0.9366
4g
220
L2
0.0047
0.0171
0,9663
0.9420
Ls
0.0049
0.0166
0,9711
0,9516
T4
0.0054
0.0168
0.9680
0,9453
Ls
0.0105
0.0578
0.6835
0.8257
Table 3: Dataset Size Generalizability.
Table 4: Mixture ratio Generalizability.
Parameterization
Huber loss,
R² f
Parameterization
Huber loss.
R²a
D
G
D
G
D
G
41 0,0065
0.0098
0,9450
0,9069
h
0.0022
0.00679
0.9950
0.9663
L2 0.0054
0.0123
0,9352
0.8909
L2
0.0021
0.00695
0.9957
0,9672
Lg 0,0038
0.0096
0,9865
0,9126
Ls 0,0019
0.00673
0,9964
0.9717
L1 0,0084
0.0093
0.9126
0.9037
L4 0.0049
0.00670
0.9797
0,9579
L5
0.1212
0.0167
0.8686
0.8783
Lg
0.0094
0.0256
0.9570
0.8434
Mixture ratio Generalizability: We apply the k-fold cross-validatio method across various pa- rameterizations. Specifically, we select 7 out of 9 mixture ratios for fitting and the remaining for testing, resulting in 36 experiments per domain. For simplicity, we show average results across domains in Table and observe that that Lg still shows significantly better performance on mixture ratio generalizability. Besides, in Figure | we observe that our D-CPT Law has well-performed generalizability on unseen mixture ratios.
4.3 Usages of D-CPT Law
Usage 1: Trade-off' between general and domain-specitie abilities For D-CPT, training data is a mixture of general and domain-specific data, where ryy and r' denote the corresponding proportions respectively. In D-CPT Law, when rgg increases, the Lg will decrease and La will increase, indicating a trade-off between the general and Nomainnsseciffi abilities of LLM. Fortunately, D-CPT Law can identify the optimal mixture ratio under any trade-off scenario. Specifically, we assume that an LLM with parameter size No. it exhibits general-corpus validation loss of Ly and domain-corpus validation loss of T. before continual pretraining. After mixing training data size of Do with a ratio ra of domain -omeif- data and 1 - ra of general data, we obtain general-corpe momain-corpus validation loss La after D-CPT. Then, we identify the optimal mixture ratio while limiting the decline in the model's 's general abilities within a threshold T as follows:
validation loss Lg and
N N=0.BB
N N=1.8B
N=0.BB
N=188
1.00
Redl Lane
300
Rellow hon
Les Prderd m0
Prockredoon
Protuce Jea
2.2
2.5
1.40
4g
2.50
4
2.00
Lg
Lg
13.
2.0
3.00
2.58
1.00
1,50
1.55
D
D
D
Figure 3: Effectiveness of D-CPT Law (La). (left two): General-corpus validation loss Lg with
respect to dataset size D across different model sizes N, domain-corpus is code and general-corpus mixture ratio rg = 0.5. (right two): nomain-corpus validation loss La with respect to dataset size D across different model sizes N, domain-corpus is code and domain-corpur mixture ratio r = 0.5.
7

---

lig T.0 9
argmin La(N No, D Da)
s.t.
T,
(10)
where T is the threshold based practical need. In Appendix G.1 given fixed T, unique optimal solution rd is obtained. To validate in real scenarios, by applying D-CPT Law, we calculate the optimal domain- corpuss mixture ratio 'd 0.924 given a dataset size Do 10B, model size No 1.8B, T = 3%. domain-corpp is chemistry, and initial general validation loss value of T.0 2.8602. Table presents the results of real general-corpus validation loss and domain-corpus validation loss with respect to different domain-corpus mixture ratios, we find that the real value exactly matches the predicted values( l g_ppred 2.9458 and _de_pre 1.7284), domain-corpus mixture ratio exceeding 0.924 leads general validation loss that surpasses the 3% threshold of Lg.
Optimal mixture limited domain-specific data Given that domain-corpus is typically limited relative to the abundance of the general corpus, we study how to determine the optimal mixture ratio when domain-corpus is limited and general-corpus sufficient. Specifically, for an LLM with parameter No and limited domain-corpus d' we aim to minimize the domain-corpus validation loss La by selecting the optimal domain-corppa corpus mixture ratio Ta as follows:
Usage 2:
argmin La(N = No, D,ra) S.L. D3 D
(11)
In Equation we can reach the minimum value within the 0 < < 1 discussed in Appendix G.2
To validate real scenarios, we conducted experiments within the music domain by setting the model parameters No 1.8B and the domain-spednn points at a large scale, we fit D-CPT Law using only data where Da < 2B to align with the use case scenarios. After using the D-CPT Law, we find that the optimal domain-corpus mixture ratio is 0.732. Table shows the results of real domain-corpus validation loss of the music domain. We observe that тa 0.732 yields the lowest domain-corpus validation loss. Moreover, our predicted domain-corpus validation loss is 0.7328 when ra = 0.732, which is close to the real value (0.7309).
dataset size D3 5B. As we have data
Usage 3: Resource allocation D-CPT Law is consistent with Chinchilla Scaling Law under the fixed mixture ratio address resource allocation. Specifically, how find the optimal values of N and given a fixed compute budget. Detailed results are shown in Appendix G.3
4.4 Cross-Domain D-CPT Law
In Section 3.2. have mentioned that the learnability of specific domain is measured by DLC (i.e.. K). For Cross-Domaie D-CPT Law, must satisfy 3 core requirements: accessible, distinct, and robust. Based these requirements, different representations of K are defined as follows:
W1 K1
W1 k1
W3
K1
K2
Wy
K3
+ wh
K2,
K4
+ W2
x
122
+
(12)
K3
where [W1, W2, ws] are fitting parameters. In the approximate Taylor expansion for the validation loss function near the initial points, (kg, represent the first three coefficients. Due to the discrete nature of data points in practical scenarios, (2,33) are approximated using the variants of validation loss. Specifically, ki denotes the precise value of the validation loss at the initial point, 22 represents the difference in validation loss close to the initial points, and K3 is an approximation of the second derivative of the validation loss near the initial points, details are provided Appendix H To compare these four representations of K, we have conducted experiments in both effectiveness and generalizability aspects.
Table 5: The real I.g and La with respect to ra usage setting
Fa 0.9
0.91
0.92
0.924
0.93
0.94
1.0
by 2.9052
2.9193
2.9376
2.9445
2.9644
2.9848
3,4667
La
1.7321
1.7312
1.7311
1.7291
1.7279
1.7265
1.7220
8

---

Table 6: The real domain- corpus validation loss with respect to ra when Da is fixed with 5B.
0.2
0.33
0.5
0.67
0.69
0.71
0.732
0.75
0.77
0.8
La 0.7486
0.7495
0.7448
0.7402
0.7387
0.7391
0.7309
0.7339
0.7336
0.7398
Effectiveness We utilize data points from all 6 domains for fitting and then evaluate their perfor- mance using R2 and Huber loss. In Table 7 we find that 4 representations of K yield comparable results in the general domain. However, K1 and K2 demonstrate a noticeable decline in domain- specific aspects. Although Ka slightly outperforms K3 in domain-specift aspects, it requires larger number of fitting parameters. Therefore, considering the balance between fitting efficiency, fitting performance, and accessibility, We consider K3 to be the optimal representation. To further visualize it, Figure 5 illustrates the predicted curves in comparison to the real curves under various settings.
N=0.5B
N=1.8B
N=0.5B
N=1.8B
6.0
1.0
300
0.0
L 2.35
4g 2:5
Ld
Lo
0.4
0.4
340
2.0
2.00
2.40
10
15
15
D
D
D
Figure 5: Effectiveness of Cross-Domain D-CPT Law (Kx). (left nwo): Lg with respect to dataset size D across different model size N, domain-corpus is music and rg is 0.2. (right two): La with respect to dataset size D across different model size N, domain-corpus is music and ra is 0.8.
Generalizability When K is identified, Cross-Domain D-CPT Law can be transformed into D-CPT Law, we believe that the former inherits the latter's generalizability in terms of model size, dataset size, and mixture ratio. In this part, we will specifically focus on discussing the domain generalizability of D-CPT Law. To evaluate domain generalizability. we use data points from 4 out of 6 domains for fitting and assign the remaining 2 domains for testing. For simplicity, we only show the averaged results across 15 combinations in Table 8 Among these 4 representations of K, K3 exhibits the superior performance, further proving its strength.
5 Related Works
Scaling Law Many studies 126 31 16 588 show a power-law relationship between per- formance and the increases in both the num- ber of parameters and the size of the training data [3 27 18). which are crucial for large
Table 8: Domain generalizability.
Huber loss]
Representation
G
K;
0.0231
0.7712
0,9851
0.5855
language models (LLMs [5
51 29 55
35]
K2
0.0222
2.5792
0.9860
0.5865
K₃ 0.0214
0.5335
0.9886
0.8611
59 36 (6)) and provide a predictive structure for determining the most efficient setups for ex-
K,
0.0232
1.1634
0.9849
0.6763
panded models using the insights gained from smaller models 17] Moreover, the extension of
scaling laws to autoregressive generative models widens their relevance to encompass tasks beyond
text [18 25 [7]. Recently, Muennighoff et al. [3) studied the Scaling Law of data-constrained settings by using the full pre-training dataset across multiple epochs. Ye et al. [57] investigate the data-mixing scaling law for the general LLMs to improve the pretraining efficiency.
Table 7: The performance of 4 representations under effectiveness setting.
Huber loss.
R²
fitting parameters
Accessibility
Representation
D
G/D
G/D
K1 0.0675
0,9224
0.9853
0.9462
+
+
K2 0.0612
1.1924
0.9875
0.8526
++
K;
0.0566
0.3682
0.9889
0.9918
0.0671
0.3396
0.9854
0.9928
For Numbers of fitting parameters, more "4" indicates a larger number of fitting parameters, implying lower fitting efficiency. Accessibility denotes the accessibility of K, fewer "4" signifies higher accessibility.

---

Domain-specific Continual Pre- Tre-Trainn Domain-specific Continual Pre- Tra-nrainnn aims to contin- ually pre-train LLMs to adapt them new domains [23]
22
30 19 For example, Gururangan
et al. [33 introduces growing mixture of expert architecture for domain-adaptive continual pre-
training. Cossu et [3] show that continually pre-trained models (RoBERTa [38] and BERT (14) are robust against catastrophic forgetting on downstream tasks. However, the above works only investigate small encoder-decoder models on limited tasks. Recently, Gupta et al. [21] study different warm-up strategies for continual pertraining for better results.
6 Conclusion
In this work, we have investigated the Scaling Law of Domain-specific Continual Pre- Pre-Trainnn (D-CPT), which provides significant step forward in the optimization of training LLMs for specific downstream domains. By developing and validating the D-CPT Law, we can easily predict the optimal mixture ratio of general and nomain-specffic corpora, greatly reducing the previously necessary but costly grid-searching efforts. Besides, we also adapt our D-CPT Law to the cross-domain setting and introduce the Cross-Domar D-CPT Law to further reduce the efforts of fitting the D-CPT Law of new domains. Moreover, we discuss the three practical usages of our D-CPT Law. Finally, we believe our D-CPT Law is an initial investigation into quantitative prediction methods for the domain-specific continual pre-training, With its increasing focus on data engineering, we hope exploration facilitates further quantitative studies and theoretical analyses in this research area.
References
[1] Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.N., Hambardzumyan,, K., Zhang, S., Roller, S., Goyal, N., Levy, O., Zettlemoyer, L., 2023. Scaling laws for generative mixed-modal language models, in: International Conference on Machine Learning, PMLR. pp. 265-279
[2] Agiza, A., Mostagir, M., Reda, S., 2024. Analyzing the impact of data selection and fine-tuning on economic and political biases in llms. arXiv preprint arXiv:2404. 0..699.9........
[3] AI, Young, A., Chen, B., Li, C.,, Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang. S., Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., Dai, Z., 2024.
Yi: Open foundation models by 01.ai. arXiv: 2403.04652
[4] Alabdulmohsin, I.M., Neyshabur, B., Zhai, X., 2022. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems 35, 22300-22312.
[5] Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M.D., McAleer, S., Jiang, A.Q., Deng, J., Biderman, S., Welleck, S., 2023. Llemma: An open language model for mathematics. arXiv preprint ArXiv:2310. 10631.
[6] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F,, et al., 2023. Qwen technical report. arXiv preprint arXiv:2309.1660
[7] Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z.,, et al., 2024. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.022) 1.2954
[8] Cai, 7.,, Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., Fan, Q. Fei. Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo,
Q., He, C., Hu, Y., Huang, T.. Jiang, T.. Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li, Y.. Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma. L.. Ma, R., Ma, Z., Ning, W., Ouyang, L., Qiu, J., Qu, Y., Shang, F., Shao, Y., Song, D., Song, Z.
Sui, Z., Sun, P.. Sun, Y., Tang. H., Wang. B., Wang, G.. Wang. J., Wang, J., Wang, R. Wang.
Y., Wang. Z., Wei, X., Weng, Q. WU, F., Xiong, Y. Xu, C., Xu, R., Yan, H., Yan, Y. Yang, X.,
Ye, H., Ying, H., Yu, J., Yu, J., Zang, Y., Zhang, C., Zhang, L., Zhang, P., Zhang, P., Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q. Zhao,
X., Zhou, F., Zhou, 7.,, Zhuo J., Zou, Y. Qiu, X., Qiao, Y., Lin, D., 2024. Internim2 technical
report. arXiv: 2403. 17297
10

---

[9] Carpenter, R., 1960. Principles and procedures of statistics, with special reference to the biological sciences. The Eugenics Review 52, 172.
[10] Chemrxiv,
[11] Chen, W., Zhou, Y., Du, N., Huang, Y., Laudon, J., Chen, Z., Cui, 2023. Lifelong language pretraining with distribution cestribttionsseciiaiizeed specialized experts, in: International Conference on Machine Learning, PMLR. pp. 5383-54 5395..
[12] Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al., 2022. Unified scaling laws for routed language models, in: International conference on machine learning, PMLR. pp. 4057 4086.
[13] Cossu, A., Tuytelaars, T., Carta, A., Passaro, L., Lomonaco, V., Bacciu, D., 2022. Continual pre-training mitigates forgetting language and vision. arXiv preprint arXiv:2205.099
[14] Devlin, J., Chang, M.W. W.W., Lee, K., Toutanova, K., 2018, Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
[15] Eloundou, T., Manning, S., Mishkin, P., Rock, D., 2023. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130.
[16] Frantar, E., Riquelme, C., Houlsby, N., Alistarh, D., Evci, U., 2023. Scaling laws for sparsely- connected foundation models. arXiv preprint arXiv:2309.0855
[17] Gao, L.., Schulman, J., Hilton, J., 2023. Scaling laws for reward model overoptimization, in: International Conference on Machine Learning, PMLR. pp. 10835-10866.
[18] Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., Cherry, C., 2021. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740)
[19] Guo, D., Zhu, Q. Yang, D., Xie, Z.,, Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al., 2024. Decpseek-coder: When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401.141:1
[20] Guo, H., Yang, J., Liu, J., Yang, I.., Chai, I.., Bai, J., Peng. J., Hu, X., Chen, C., Zhang, D., al., 2023. Owl: A large language model for it operations. arXiv preprint arXiv:2309.09298
[21] Gupta, K., Thérien, B., Ibrahim, A., Richter, M.L., Anthony, Q.. Belilovsky, E., Rish, I., Lesort, T., 2023. Continual pre-training of large language models: How (re) warm your model? arXiv preprint arXiv:2308.0400
[22] Gururangan, S., Lewis, M., Holtzman, A., Smith, N.A., Zettlemoyer, L.., 2021. Demix layers: Disentangling domains for modular language modeling. arXiv preprint arXiv:2108.0555 0..0000.........
[23] Gururangan, S., Marasović, A., Swayamdipta, S., Lo. K., Beltagy. I., Downey, D., Smith, N.A., 2020. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.1000 109644
[24] Henderson, P., Krass, M., Zheng, I.., Guha, N., Manning, C.D., Jurafsky, D., Ho, D., 2022. Pile of law: Learning responsible data filtering from the law and 256gb open-source legal dataset. Advances in Neural Information Processing Systems 35, 29217-29234.
[25) Hernandez, D., Kaplan, J., Henighan, T., McCandlish, S., 2021. Scaling laws for transfer. arXiv:2102.00 01293
[26] Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M.M.A., Yang, Y., Zhou, Y., 2017. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00449.
[27] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., Clark, A., et al., 2022. Training compute-optimal large language models. arXiv preprint 25X5v:2003.5555.

---

[28] Huber, P.J., 1992. Robust estimation of a location parameter, in: Breakthroughs in statistics: Methodology and distribution. Springer, pp. 492-518.
[29] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.I., Bres- sand, F,, Lengyel, G., Lample, G., Saulnier, L., et al., 2023. Mistral 7b, arXiv preprint arXiv:2310.06825
[30] Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.W., Wei, X., Arnold, A., Ren, X., 2021. Life- long pretraining: Continually adapting language models emerging corpora. arXiv preprint urXiv22110.08544
[31] Kaplan, J.. McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D., 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361
[32] Ke, Z., Shao, Y., Lin, H., Konishi, T., Kim, G., Liu, B., 2023. Continual pre-training of language models. arXiv preprint arXiv:2302.03241.
[33] Li, J., Wang, X., Wu, X., Zhang, Z., Xu, X., Fu, J., Tiwari, P. Wan, X., Wang, B., 2023. Huatuo-26m, a large-scale chinese medical qa dataset. arXiv 2305 01526
[34] Liu, D.C., Nocedal, J., 1989. On the limited memory bfgs method for large scale optimization. Mathematical programming 45, 503-528.
[35] Liu, J., Bai, Z., Zhang. Y.. Zhang, C., Zhang, Y., Zhang, G., Wang, J., Que, H., Chen, Y., Su, W., al., 2024a. E2-Ilm: Efficient and extreme length extension of large language models, arXiv preprint arXiv:2401.066
[36] Liu, J., Zhang. C., Guo, J., Zhang, Y., Que, H., Deng, K., Bai, Z., Liu, J., Zhang, G., Wang, J., et al., 2024b. Ddk: Distilling domain knowledge for efficient large language models. arXiv preprint arXiv:2407.16154.
[37] Liu, X., Yan, H., Zhang, S., An, C., Qiu, X., Lin, D., 2023. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209
[38] Liu. Y., Ott, M.. Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis. M., Zettlemoyer, L., Stoyanov, V., 2019. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1Xiv:1907.11692..
[39] Lozhkov, A., Li, R., Allal, L.B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al., 2024. Starcoder and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.
[40] Mehta, S.V., Patil, D., Chandar, S., Strubell, E., 2023. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research 24, 1-50.
[41] Mendieta, M., Han, B., Shi, X., Zhu, Y., Chen, C., 2023. Towards geospatial foundation models via continual pretraining, Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
[42] Motoki, F. Pinho Neto, V., Rodrigues, V., 2024. More human than human: Measuring chatgpt political bias. Public Choice 198, 3-23.
[43] Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T. Raffel, C.A., 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems 36.
[44] Qu, X., Bai, Y., Ma, Y., Zhou, Z., Lo, K.M., Liu, J., Yuan, R., Min, L., Liu, X., Zhang, T.. et al., 2024. Mupt: generative symbolic music pretrained transformer. arXiv preprint arXiv:2404.063933
[45] Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song. F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al., 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 12.
12

---

[46] Rillig, M.C., Agerstrand, M., Bi, M., Gould, K.A., Sauerland, U., 2023. Risks and benefits of large language models for the environment. Environmental Science Technology 57, 3464 3466.
[47] Rongali, S., Jagannatha, A., Rawat, B.P.S., Yu, H. 2020. Continual domain-tuning for pretrained language models. arxiv preprint arXiv:2004002888.
[48] Soldaini, L.., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al., 2024. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159.
[49] Sun, Y., Wang. S., Li, Y., Feng. S., Tian, H., Wu, H., Wang, H., 2020. Ernie 2.0: A continual pre-training framework for language understanding, in: Proceedings of the AAAI conference on artificial intelligence, pp. 8968-8975.
[50] Thakur, V., 2023. Unveiling gender bias in terms of profession across llms: Analyzing and addressing sociological implications. arXiv preprint arXiv:2307.09) 10.662.............
[51] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N.. Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G., 2023a.
Llama: Open and efficient foundation language models. arXiv: 13971
[52] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al., 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288;
[53] Wang. Z.M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Chen, W., Fu, J., Peng. J., 2023, Rolellm: Benchmarking. eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: 2310.00746.
[54] Wu, T., Caccia, M., Li, Z., Li, Y.F. Qi, G., Haffari, G., 2021. Pretrained language model in con- tinual learning: A comparative study, in: International conference learning representations.
[55] Yang, A., Xiao, B., Wang. B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., Yang, F., Deng, F., Wang, F., Liu, F., Ai, G., Dong, G., Zhao, H., Xu, H., Sun, H., Zhang, H., Liu, H., Ji, J., Xie, J., Dai, J., Fang, K., Su, L., Song, L., Liu, L., Ru, L., Ma. L., Wang, M., Liu, M., Lin, M., Nie, N., Guo, P., Sun, R., Zhang, T., Li, T., Li, T., Cheng, W., Chen, W., Zeng, X., Wang, X., Chen, X., Men, X., Yu, X., Pan, X., Shen, Y., Wang, Y., Li, Y., Jiang, Y.,
Gao, Y., Zhang, Y., Zhou, Z., Wu, Z., 2023. Baichuan 2: Open scrge-scale language models.
arXiv:2309 10305
[56] Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., Zhang, Y., 2024. survey on large language model (Ilm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 100211.
[57] Ye, J., Liu, P., Sun, T., Zhou, Y., Zhan, J., Qiu, X., 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.1695.
[58) Zhang, B., Liu, Z., Cherry, C., Firat, O., 2024a. When scaling meets Ilm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17119
[59] Zhang, G. Qu, S., Liu, J., Zhang, C., Lin, C., Yu, C.L., Pan, D., Cheng, E., Liu, J., Lin, Q.. al., 2024b. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405. 193277
[60] Zhou, Z., Liu, J., Dong, 7.,, Liu, J., Yang, C.,, Ouyang, W., Qiao, Y., 2024. Emulated disalignment: Safety alignment for large language models may backfire!. in: Ku, I.W., Martins, A., Srikumar, V. (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Bangkok. Thailand. pp. 15810-15830 URL: httpt:://aclaaaaae http:///aaaattooooo.oooo .wr.loooyoooogg//// org/2024 acl-long. 842 doi/10.
acl long. .o2..442 842
13

---

A Limitations and Future works
Experiments on more downstream domains In our work, main experiments cover six downstream In future works, it is important to experiments on more downstream domains. We will attempt to conduct experiments on CPT in more domains and fit the D-CPT Law well the Cross-Domane D-CPT Law.
domains [39 3 24
T0
44
[3]
Experiments on more LLMs We primarily conduct experiments based on Qwen-1.5 and lack exploration of other Pre-Trained Base LLMs.
Multilingualismm We lack research on multllingualism settings. Although the medical data is in fact in Chinese, the other data are in English. Moreover, the experimental results show that the fitting results in the medical domain are poor compared others. We lack detailed experimental analysis of different language settings. In future research, we hope to realize cross- moos--linguistic and multi-linguistic D-CPT Law and thereby further extend the generalizability of D-CPT Law.
Difficulty fitting parameters We find that when using L-BFGS for fitting, the initialization of the fitting parameters is essential. Different parameter initializations can lead to significantly distinct results. Besides, we find that fitting algorithms also matter, in subsequent works, we hope compare different fitting algorithms and design methods to reduce the dependency on the initialization of the fitting parameters.
Extensive training costs of Scaling Law Although we attempt to ameliorate the training costs and enhance the fitting efficiency of Scaling Law, which are detailed in Section 4.4 and Appendix [I] Scaling Law 22 37] still remains prohibitively expensive for the majority. We hope that future research endeavors will seek to reduce the training costs of Scaling Law, thereby facilitating a wider usage and understanding of these laws within the community.
B Broader Impacts
LLMs, particularly those involving pre-training on massive Internet data, have been identified to carry significant societal impacts and inherent biases [56]] 50 15| 2 For instance, large language models (L.LMs) may generate content that carries political bias 42 With the rise of downstream applications of LLMs, there is growing effort limit their output of offensive content, rendering LLMs more controllable and mitigating their potential negative impacts. We hope that our research to make the downstream applications of LLMs more controllable.
Besides, LLMs have a significant environmental impact due the substantial energy consumption required for their training and inference stages [46]. The extensive computational resources needed result high carbon footprint, thus raising concerns about the sustainability of such models the context of global efforts to reduce greenhouse gas emissions. To this, our research can also partially reduce the consumption of GPU, thereby reducing the environmental impact of LLMs.
C Symbols
To enhance the reader's experience, we have listed the symbols used this paper in Table
D D-CPT Law with constant Mixture Ratio
OpenAl Scaling Law Kaplan et [1] propose parameterization as follows:
an
N₂ N
D
I. =
(13)
D
where (N., De, aN, ap) are fitting parameters.
14

---

Table
List of symbols presented in this paper.
Symbol
Description
Fd
The proportion of the domain-specific corpus within the training dataset.
Tg
The proportion of the general corpus within the training dataset.
F
The proportion of the target corpus within the training dataset. corpus.
La
The validation loss for the
Li
The validation loss for the general corpus. The validation loss for the target corpus.
L.
N
The size of the model parameters. The number of training tokens for the model.
D
Da
The number training tokens of the domain -seeifi corpus for the model. The number of training tokens of the general corpus for the model.
D
L
The validation loss for the domain-specific corpus before continual pre-training. The validation loss for the general corpus before continual pre-training.
Chinchilla Scaling Law Continuing along the trajectory established by OpenAI Scaling Law,
Hoffmann al. [2 propose parameterization follows:
= A B
(14)
No
D8
where (E,A,B,a,3) are fitting parameters. After fitting, the Allocation problem can be resolved by:
Nopt = G (g))
D Dopt G-1
(15)
6
QA
642
8
a a + B
where G
3B
a
a + 3
(16)
where Nopt and D Dopt represent the optimal value of model size and dataset size, respectively.
If we fix the mixture ratio in the training corpus, the D-CPT Law narrows down to the relationship involving only the model size N and dataset size D. Although previous works have proposed Scaling Law to describe the relationship between variables and performance, it has not been validated under our experimental setup. Here, we present the performance of OpenAl Scaling Law and Chinchilla Scaling Law in our experimental setup. For simplicity, we present results only in the code domain, with 1:1 mixture ratio. The experimental results are shown in Figure and Table T0 We find that the Chinchilla Scaling Law is obviously better in experimental setup.
Table 10: The fitting performance of two laws on code-corpus with 1:1 mixture ratio.
Huber loss
R² t
Law
G
OpenAI Scaling Law
0.0026
0.0059
0.9609
0.8888
Chinchilla Scaling Law
0.0002
0.0013
0.9994
0.9925
E
Supplementary Materials of D-CPT Law
E.1
Explicit trends
To provide a clear visualization of Equation we have provided figures under different settings, depicted in Figure Figure 8 and Figure 9 All plots are trends of real data points.
15

---

N = 0.5B
= 1.8B
N = 4.0B
33
Red losa Prodiction
Redl Joss
Real les Preductos
Prediction
50-
L
29
26
15
30
16
15
20
10
Figure 6: Ly with respect to across multiple model sizes N. Blue solid lines stand for real data points and orange dashed lines stand for the predicted curve. Fitting law is Chinchilla Scaling Law.
Domain-specific validation loss on different model size
Domain-specne specific validation loss Ld
D=1.31B
1.70
D=3.93B
D=6.558
1.65
D=9.18B
1.60
D=13.11B
D=19.66B
1.55
1.50
1.45
1.40
1.35
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Model size N(unit B)
Figure 7: Domain-corpus validation loss La with respect to model size N while (Drr) are fixed, domain-corpus law and domain-corpus mixture ratio ra = 0.2.
Domain-specife validation loss different dataset size
Domain-specifie validation loss Ld
N=0.5B
N=1.8B
N=4.0B
1.8
1.7
1.6
1.5
1.3
20
25
Dataset size D(unit B)
Figure 8: Domain-corput validation loss La with respect dataset size D while (N.r) fixed, domain-corpus is law and domain-corpus mixture ratio ra = 0.2.
16

---

Domain-specific validation loss on different domain-specific mixture ratio
Domain- Comain-specifc specific validation loss La
D=1.38
1.60
D=3.8B
D=5.4B
1.55
D=9.28
D=12.18
D=19.7B
1.50
1.40
1.35
0.2
0.4
0.6 0.8 1.0
Domain-specific mixture ratio rd
Figure 9: Domain-corppai validation loss Le with respect to domain domainn-crpus corpus mixture ratio ra while (N,D) are fixed, domain-corpus corpuss is law and model size N 1.8B.
E.2 Implicit trends
In this section, we start from the perspective of experimental observations to illustrate why we can arrive at the conclusions presented in Equation Subsequently, we will briefly analyze the underlying reasons for these implicit trends. For convenience, we replicate here for clarity:
LL
0,
(17)
aD0r
In mathematics, D-CPT Law has continuous second partial derivatives with respect D and Based on Clairaut's s Theorem, we have:
0L 8
(18)
aD0
drHD
which implies that the order partial derivative does not affect the pattern presented in Equation 17 Based on the experiments, we have plotted the approximate values of dhg as function of the an general-corpus mixture ratio, as shown Figure I0 Since data points are discrete, we take the dL difference of every 5k steps as approximate values for 2D. We present the curves of aD with respect ry across multiple dataset sizes D. It is clear that dl, monotonically decreases with rg. Thus, based on the real experimental observations, we can infer Equation]] 17
dE,,
In fact, there exists explicit relationship between and D, which can be represented as:
Dg rg. D)
(19)
Da = Tá D,
(20)
Tg + Fd
(21)
Dg + Da = D.
(22)
where D, represents the general-corpus dataset size and Da represents the domain-corpus dataset size. If we focus on the domain-corpus validation loss, then D, is noisy data to domain-corput and Da valid data domain-corpus. we consider Tat, the domain-corpus validation loss, to be solely dependent on Da and N, then D and ra influence each other and cannot be considered independent.
17

---

Approximation of aLg
with respect to rg
3D
-0.005
Approximation of aLg
ab
-0.010
-0.015
D-0.78
-0.020
D-1.6
D=2.08
D=2.6B
D=3.38
-0.030
D-3.95
D=4.6B
D-5.28
-0.035
D-5.96
Dw6.68
0.6
1.0
General mixture ratio rg
aL, Figure 10: Approximate values of DD with respect general-corpus mixture ratio ry while (N.D) are fixed, domain-corpus is law and model size N = 1.8B.
Previous works have treated N and as independent variables, not influencing each other. However, in our works, D and are not able be independent of each other, both from the perspective of experimental phenomena and the explicit relationship.
Additionally, we can explain Equation 5 by the principle of data efficiency. The term dD can be interpreted as the efficiency of each unit of data. With the increase of the proportion of valid data in each unit of data rises while the proportion of noisy data diminishes, resulting in enhanced efficiency of each unit of data. Given that lower loss signifies improved model performance, dD displays decreasing trend as r increases.
consequently
E.3 Details behind D-CPT Law
In this section, we will first derive and demonstrate that D-CPT law satisfies the requirements mentioned in Section 3.1 Subsequently, we will briefly describe the algorithm's 's setup and some minor improvements.
Adaptability: The newly introduced variable, the mixture ratio r, significantly differs from N and D, in that the range of values for N and D is greater than whereas is limited to the range [0,1]. This means that r should yield valid results at both 0 and 1, and it is crucial ensure that values of near 0+ 1 do not cause L to exhibit infinity. The trend of L with respect to generally exhibits an initially rapid and subsequently slow pattern, behavior that can be accurately modeled by a power function. However, positioning in the denominator leads to an asymptotic increase to infinity as r approaches zero from the positive direction. To mitigate this issue, we have introduced a small positive bias to which is fitting parameter. Typically, the value of lies near 0.1. This adjustment effectively prevents explosive growth near r 0+.
Explicit trends:
0L
a a.A A
<0,
(23)
0N
Na+1
ØL
3 B.
< 0.
(24)
0D
D8+1
0L
B
7.C
where -
(25)
Dr
D
P/+11
18

---

It is important to note that for the third equation, having OL on the fitting parameters, specifically;
< 0 requires certain constraints
8r
n> C>
Bn (1 +6)
where Co
(26)
YD mén
If these two constraints satisfied, have:
0L
B.
V.C
(27)
0r
D)
Bn
1 1
yCD
(28)
D
Bn
Bn
yCDs
(1 (1+
(29)
D&
By
Bn (1+
2 7
C
(30)
y141
D
Buj (1 +6)
7
C
(31)
D
min
WT+ (Co - -)<
(32)
In our experimental setup, has minimum value, with the minimum value Dmin being approximately 0. 0.1 Therefore, as long we set C' greater than Co and 1) greater than 1, the condition 0L 25 < can be satisfied. This effectively imposes constraints on the fitting parameters. In our actual fitting process. we have modified the algorithm to seamlessly incorporate these constraints. Specific details will be mentioned when introducing the algorithm.
Implicit trends:
PL
LL
0
3Br" DF+T
n3Br911
0,
(33)
aDår
dråD
Òr
D8+1
Consistency:
A
B-r
C
L(N, D
(34)
Na
DB
(ro +6)
A
Bo
= Eo +
(35)
Na
D8
C
where
Es = E +
(36)
(ro +)
Bg = B r
(37)
which means that if r a constant ro- then D-CPT Law can be transformed into a conven- tional Chinchilla Scaling Law. This suggests that under specific conditions where assumes a fixed value, D-CPT Law aligns with the more universally recognized Chinchilla Scaling Law.
Constrained L-BFGS
We utilize L-BFGS to data points, with the objective being:
min a,0r,...0.88....
Huber (Lyx log Lreat),
Lyu LSE(e,a a log N, b, (1 exp(m)) logr 3 log D, log(r e), co 7 log(r + +c), where co = log Co, a = log A, b = log B, e₁ log Ci, e = log E,
C = Co C1, + exp(11),
19

---

where LSE is the log-sum-exp operator. Our improvements to the algorithm primarily focus on the third and the last item. Previously, we mentioned that must be greater than Cp to ensure the monotonic decrease of the D-CPT law with respect to r. Without any restrictions and fitting directly, would sometimes lead to fitting results where does not satisfy C 2 Co. Therefore, to ensure that the fitted C' must be greater than Co. we have indirectly imposed certain restrictions on the algorithm. We decomposed the original C' into two parts: Co and C₁ and due to the characteristics of the exponential function, the fitted result of C₁ expei Ci will be greater than O. Consequentl, C will be greater than Co. i.e.,
C Co Ci Co + exp(ci) > Co:
(38)
1 = ++ cxp(n)) > 1,
(39)
Co B(1 + exp(m) (1 + +))++
where
(40)
D min
Following Chinchilla Scaling Law, we find local minima of the objective function, initiating our search on predefined grid of starting points follows:
(-1., -0., 5. b E {-1., 0.,, .... 5. c
{- [-1., O., .... e. E {-1., .... 1.),a a E {-0.5., 0., 0.5), B E { {-0.5., 0., 0.5), E
0.5. O., 0.5). m e 0., 0.5), E {0., 0.5.. Besides, use d 10-3 3 for the Huber
loss.
E.4 The choices of D-CPT Law Parameterizations
In Section 4.2 have proposed five possible parameterizations of the D-CPT Law. After analyzing the experimental results, we have selected L3 as the final parameterization of the D-CPT Law. Could there be other parameterizations better than L We acknowledge that there may exist better parameterization than L3. but we believe that this is not crucial because our core objective is to find one that satisfies the 4 requirements outlined in Section 3.1 and assume that when these 4 requirements are met, the parameterization is considered good option of D-CPT Law. Specifically, regarding the choice of parameterization for the D-CPT Law, our research goal can be understood as finding a parameterization that matches the trend of real (N, D, r) data points and possesses certain mathematical properties. These trends and mathematical properties can be explicitly expressed as the 4 requirements in Section 3.1 Adaptability, Explicit trends, Implicit trends, and Consistency. Besides. in Section we provide 5 parameterizations, only L3 can meet all requirements.
Moreover, as there are also other parameterizations that can meet these 4 requirements, we provide another 2 parameterizations that satisfy these 4 requirements follows:
A
e.
C
LG E +
(41)
Na
D
r'
A
B.r
C
= Na
(42)
DS
The fitting results for Le, Lr, and La on general-corpus and domain-corpus are as listed Table We observe that when 4 requirements are met, the fitting results do not differ a lot. In conclusion, first, as there are many parameterizations that meet the requirements, optimal parameterization. Second, the L3 mentioned in the paper is relatively simple and meets the requirements with good fitting results.
challenging to find the
E.5
Compute resources
Our main experiment requires approximately 150k hours of runtime single A100.
20

---

Table 11: Mean performance across Ls, La, and L, over six domains. "G" and "D" denote general and downstream domains.
Huber loss +
R²
Parameterization
G
D
G
D
L3
0.0048
0.0157
0.9968
0.9796
L6
0.0051
0.0164
0.9963
0.9778
L7
0.0049
0.0160
0.9969
0.9801
F Supplementary Materials of Experiments
F1
Validation datasets collection
Specifically, for each domain, we first randomly select 5,000 samples from the original dataset, and then we use four open-sourced LLMs (i.e., Qwen- 1.5 72B [6] Yi-34B [] LLaMA2-133 [52). IntermL.2-200B [8] to compute the perplexity (PPL) and sort these samples based on the PPL. values. Specifically, lower PPL value denotes higher fluency of the data indicated by the model. If a sample ranks bottom 10% under four open-source LLMs, we consider this sample to noisy and exclude it. Subsequently, as the validation set for each domain. In this way, we can obtain high-quality validation set for all domains.
randomly sample 1,000 samples from the filtered sample pool to serve
F22 Hyperparameters
The hyperparameters for the experiments are listed in Table 12
Table 12: The list of hyperparameters.
Hyperparameters
Value
Warm-up Steps Gradient Accumulation Steps
0
Train Batch Size Per Device
4
Max Sequence Length
2048
Learning Rate
3e-5
Learning Rate Scheduler
cosine
Numbers of GPUs
16
G
Mathematical Derivation behind use case
G.1
Usage
First, will standardize the notation: Mg denotes the proportion of the general corpus, ra represents the proportion of the domain corpus, L, signifies the general-corpus validation loss, La indicates the domain-corpus validation loss, D represents the dataset size, and N denotes the model size. Therefore, we have:
A
B B. (-(1-ral) ra)"
C
Lg =E
+
(43)
Na
D
(1-ra+
A
B B.r
C
La =
+
+
(44)
Na
D
(ra+e)*
Note that we have the loss L monotonically decreasing with respect to r, therefore we have:
21

---

bhg
3Lg
0Lg
<0
<0
> 0,
(45)
drg
0(1 ra)
0rd
OLd
LLd
OLd
< 0
< 0
> 0,
(46)
Dra
0(1 rg)
Or g
Within the context of D-CPT, we focus on Ld the domain-corpus validation loss. As the proportion of domain-corpus ra increases, La is expected to decrease, indicating an improvement in domain- specific performance. Conversely, Lg. the general-corpus validation loss, expected to increase with the growing ra, suggesting decline in general abilities. Therefore, need to strike a balance between general and domain-specific abilities. To be specific, we will revisit the objective function of Usage 1:
Lg L L 9
argmin (2(N No, D = Do, s.t. Tá
T
(47)
where L represents the initial general validation loss. Since Lg monotonically increases with ra. maximal ra will certainly be attained under the constraint. Concurrently, La monotonically decreases with rd, there must exist unique ra that minimizes La-
G.2 Usage
For simplicity, we restate the objective function for usage 2:
Da
argmin LN(N No, D ra) s.t. Da Dà
(48)
ra
where Da denotes the domain-corpus dataset size, for Laa in format of D-CPT Law, we have:
Da
Br
C.
La(N No. rd) = E+
+
+
where r/= = rd +
(49)
No
D3 8
(r'))²
dLa
B(1 + B) 1
C
(D2) -dd (r'))++
(50)
dra
LL
B(n +B + 17
1) I'd
1-8-2
7 1JC (22)
(51)
dră
(D2)
Based on Appendix
E.3 have 1) 1, therefore have:
LaL > 0, dr
>
(52)
dldd (ra = 0)
70
6+4 < 0,
(53)
dra
dT-d (ra 1) B(n 3) G
(54)
dra
(Dy)
(1 (1+e)++1
The derivative dhe is continuously differentiable and monotonically increasing. Given that dLa is dLa is greater than 0 at ra = 1, then follows that Equation 49 attains its negative at ra = 0 and drg minimum within the interval [0 < 1 Therefore, to ensure the existence of a valid minimum for the objective function 48 the following conditions must be satisfied:
drg
dra
Solving
dha dr.
0 relatively complex, in this works, MATLAB to find the roots of this function.
22

---

B(n+ B1 + +)7 +1
D2
(55)
7
G.3 Usage
For convenience, we repeat the objective function of resource allocation as follows:
argmin (IN,)) D) s.t. FLOPs D) C.
(56)
N.D
Following [31] we calculate compute budget C by:
C RJ 6ND.
(57)
To validate its effectiveness in real-world scenarios, we take the law domain as an example and by fixing the mixture ratio at 1:1, fit D-CPT Law. We fix compute budget C = 5e Subsequently. based on the Efficient Frontier of Chinchilla (2], we obtain:
0.6252, 0.3748, G 4.1282, Nopt 15.54B, Dopt 0.536B.
(58)
As the closest available model size to the optimal model size indicated by Qwen1 is 14B, we conducted our experiments using this 14B model. The experimental results are shown in Table
13 The experimental results reveal that the model sizes of 0.5B, 1.8B, and 4B suffer from data insufficiency. The optimal model size (14B) indeed exhibits the best performance.
Table 13: Domain-corpus validation loss with respect to various model sizes and dataset sizes while keeping the same compute budget.
Ld
0.5
16.648
1.4921
1.8
4.588
1.4214
4.0
2.097
1.3552
14.0
0.590
1.3066
H Details behind Domain-speeific Learnable Coefficient
In practice, the data points we obtain are discrete, thus we can only utilize approximate values to express k2 and kg. Specifically, use the difference between the initial validation loss and the validation loss after 5k-step continual pre-training.iee.
kg
Liseeps
La000steps-
(59)
Besides, we define K3 as the difference in the decline values between the intervals of O to 5k steps and 5k to 10k steps.i.e..
Kg (Lostops L5000steps)
55000sspps 110000stps)
LOstops L 11000sttpp
(60)
Lastly, we denote k as the validation loss obtained after training for 1k steps.
NNote that Lessess represent the validation loss steps.
23

---

I Further Analysis
1.1 Fitting Efficiency
As each data point requires computational resourees, we also investigate to improve the fitting efficiency with relatively low computational resources. In Table [4] we have compared different sampling methods for data points and introduced a decay sampling method based on the exponential decay function to enhance fitting efficiency. Specifically, we focus on the fitting efficiency across dataset size while maintaining a constant model size.
Table 14: The fitting performance of different sampling methods.
Huber loss
R² +
Resource consumption
Sampling Method
G
D
G
D
G/D
Mi
0.0041
0.0094
0.9977
0.9937
200
M2
0.0042
0.0103
0.9976
0.9936
40
M3
0.0043
0.0097
0.9978
0.9938
40
M4
0.0042
0.0092
0.9980
0.9941
45
For Resource consumption, focus evaluation costs and storage costs, We have experimented with different sampling methods, follows:
Mj Dense sampling, evaluating validation loss every 1,000 steps.
M2: Sparse sampling, evaluating validation loss every 5,000 steps.
M3: Sectional sampling, evaluating every 4,000 steps in the initial 60% steps, every 8,000
steps the remaining 40% steps.
M4 Sampling basdd exponential decay function, detailed in Appendix L.2
Experimental results show that the performance of M1 is relatively poor. In situations where resource consumption is comparatively high, no significant improvement fitting performance observed, thus indicating that the sampling density in our main experiments is excessively high. The overall performance M3 and M4 surpasses that of M2 because both M3 and M4 adopt strategy of dense sampling in the initial phase and sparser sampling in the later phase. The trend of L with respect to D also shifts from rapid to slow changes, and sampling more points during phases of faster decline can considerably enhance fitting efficiency. However, the sampling setup of M3 is of fixed paradigm and the sampling function follows a step-wise pattern. Of course, the overall performance of M1 points in
is slightly better than Ms, it also offers richer paradigm. In summary, sampling the early phase of D can improve the overall fitting efficiency. practical applications, has the potential to save evaluation costs and storage costs.
1.2
Decay function
In our main experiments, each experiment trains for 200,000 steps, with evaluations every 1,000 steps, resulting in total of 200 data points. The decay function is represented as follows:
f(x) = Ar
(61)
we the decay parameter 0.02 which yields 45 data points sampled. Figure II Iillustrates the decay function.
For M2 Section I.1
1.3 Analysis of near-zero
Interestingly, have found from the experiments that the trends between L and D reversed when r approaches in this section, we will explore it in depth and find that D-CPT Law between L and D has an inflection point r; change its trend.
We take the Law domain for example, the experimental results show that most T. decreases strictly with rg when N and D are fixed, which is consistent with D-CPT Law. However, as Tg approaches

---

Exponential Decay Function
1.0
Sampling Probability
0.6
0.4
0.2
25
150
200
Steps(10²)
Figure 11: Illustration of decay function.
General validation loss with respect to dataset size different mixture ratio
5m0.0
4=0.0
4=000
General Validation Loss Lg
g-0.33
g00.0
90067
"
0=000
4-1.00
Dataset size
Figure 12: General validation loss with respect to dataset size across various mixture ratios, the domain-specific corpus is the law and N 1.8B.
the trend of changes. Through analysis of Figure 12, observe that when ry is greater than 0.1, Lg monotonically decreases with D, which aligns with the findings of D-CPT Law and previous works. However, when rgg less than equal to 0.05, Lg monotonically increases with D. This phenomenon is not limited to just one domain, we find that almost all domains exhibit this kind of behavior. We name the mixture ratio which changes the trends of L as inflection point 7;. Accurately pinpointing is challenging. From an experimental perspective, it requires repeated experiments to approach r; progressively, which requires high experimental costs. Additionally, the exact value of r; changes across different domains, in our experimental setup, we find that for 6 domains all fall between and 0.1.
When the mixture ratio is less than the inflection point, L monotonically increases with D, which inconsistent with the D-CPT Law. Therefore, the D-CPT Law predicts poorly when the mixture ratio is less than r; Fortunately, predictions when the mixture ratio is less than are meaningless in the context our works for two reasons: (1) In practical situations, we may be particularly concerned with cases where the mixture ratio is very small, as the inflection point in most domains less than 0.05. (2) When the mixture ratio lower than r., monotonically increases with D, meaning that the training cost increases, the performance of the model worsens. This is contrary to our initial objective, as we hope that after D-CPT, the domain-specific ability enhanced. Thus, predictions when the mixture ratio is less than T4 considered meaningless.
25

---

Of course, if we collect data points of small mixture ratios which means that the curves for these data points all show increasing with D. then we can fit these data points. that case, the fitting parameter B in D-CPT Law would be negative value. If we know accurately the value of ri, we can express D-CPT Law in form of piecewise function or represent it with a unified equation. However, the problem lies in precisely determining the value of r.. In future works, we hope to propose low-cost method to accurately determine the value of r;. For example, we could conduct experiments with both small and large mixture ratios and fit them separately, then determine the value of r; based on intersection two resulting laws.
26

---

J Supplementary Tables
Table 15: Supplementary Table of Table
Huber loss of parameterizations across domains.
Parameterization
Code
Chemistry
Medical
G
D
G
D
G
D
G
D
G
D
G
D
L
0.0046
0.0191
0.0049
0.0165
0.0058
0.0182
0.0033
0.0291
0.0055
0.0108
0.0141
0.0078
La
0.0035
0.0190
0.0040
0.0165
0.0047
0.0182
0.0027
0.0275
0.0045
0.0109
0.0104
0,0077
Ls
0.0036
0.0190
0.0040
0.0164
0.0066
0.0181
0.0027
0.0224
0.0044
0.0104
0.0092
0.0076
L1
0,0040
0.0195
0,0040
0.0156
0.0047
0.01833
0.0035
0.0249
0.0050
0.0096
0.0183
0,0080
Ls
0.0300
0.0657
0.0302
0.0440
0.0426
0.0364
0.0188
0.0357
0.0248
0.0229
0.0501
0.0582
Table 16: Supplementary Table of Table
R of 5 parameterizations across 6 domains.
Parameterization
Code
Math
Law
Music
Chemistry
Medical
G
G
D
G
D
L
0.9967
0,9775
0,9965
0.9911
0,9959
0.9854
0.9972
0.9596
0.9959
0.9915
0.9846
0.9551
L2
0.9977
0.9784
0.9974
0.9909
0.9971
0.9853
0.9978
0.9655
0.9970
0.9912
0.9919
0.9584
Ls
0,9977
0,9783
0,9974
0,9910
0,9971
0,9853
0.9978
0.9734
0.9971
0.9915
0.9934
0,9583
0.9980
0,9774
0.9980
0.9916
0.9976
0,9852
0.9970
0.9689
0.9973
0.9937
0.9735
0.9534
Lg. 0,9628
0,9104
0.9639
0.9665
0,9431
0.9732
0.9787
0.9542
0.9677
0.9820
0.8814
0.9208
Table 17: Supplementary Table of Table 2
Huber loss of 5 parameterizations across 6 domains, each
unit displays the average value of 3-fold cross-vaus validation.
Parameterization
Chemistry
Medical
G
D
6
D
G
D
G
D
G
D
G
D
L
0.0033
0.01900
0.0049
0.0170
0.0073
0.0175
0.0043
0.0202
0.0052
0.0147
0.0083
0.0145
L2
0.0048
0.0188
0.0046
0.0169
0.0049
0.0176
0.0031
0.0198
0.0049
0.0147
0.0060
0.0145
Ls 0.0039
0.0185
0.0046
0.0167
0.0047
0.0176
0.0051
0.0186
0.0059
0.0144
0.0051
0.0143
L4 0.0036
0.0182
0.0036
0.0170
0.0067
0.0176
0.0054
0.0195
0.0070
0.0144
0.0064
0.0144
Lg 0.0103
0.0237
0.0104
0.0157
0.0108
0.0082
0.0063
0.2523
0.0084
0.0082
0.0168
0.0389
Table 18: Supplementary Table of Table 1t of 5 parameterizations across 6 domains, each unit displays the average value 3-fold cross- validation.
Parameterization
Code
Chemistry
Medical
G
D
G
D
G
D
G
D
G
D
G
D
L
0.9583
0.9472
0.9589
0.9425
0.9549
0.9336
0.9715
0.9130
0.9582
0.9536
0.9108
0.9295
La
0.9694
0.9536
0.9681
0.9521
0.9656
0.9301
0.9774
0.9230
0.9681
0.9529
0.9491
0.9404
Is 0.9686
0.9577
0.9672
0.9551
0.9718
0.9508
0.9811
0.9131
0.9780
0.9706
0.9598
0.9623
La 0.9578
0.9509
0.9760
0.9535
0.9741
0.9304
0.9700
0.9293
0.9725
0.9660
0.9575
0.9419
Ig 0.7411
0.7785
0.7466
0.5661
0.7008
0.8728
0.8146
0.9186
0.7158
0.9307
0.3821
0.8877
Table 19: Supplementary Table of Table 3 Huber loss of 5 parameterizations across 6 domains, each unit displays the average value of 3-fold cross-validation.
Parameterizatin
Code
Math
Law
Music
Chemistry
Medical
G
D
G
D
G
D
G
D
G
D
G
D
0.0029
0.0018
0.0032
0.0076
0.0265
0.0043
La
0.0047
0.0252
0.0039
0.0097
0.0046
0.0072
0.0018
0.0216
0.0040
0.0055
0.0136
0.0049
Lg
0.0129
0.0019
0.0139
L4
0.0066
0,0180
0.0647
0.0093
0.0059
0.0068
0.0024
0.0121
0.0055
0.004]
0.0254
0.0054
Lg
0.0123
0.0114
0.0140
0.0087
27

---

Table 20: Supplementary Table of Table 3 Rt parameterizations across 6 domains, each unit
displays the average value of 3-fold cross validation.
Parameterization
Code
Math
Law
Music
Chemistry
Medical
G
D
G
D
G
D
G
0,9930
0.8001
0.9943
0.9639
0.9927
0.9827
0.9943
0.9310
0.967]
0.9093
0.7084
0.8545
1i
0.9736
0.7848
0.9760
0.9544
0.9683
0.9818
0.9939
0.7847
0.9783
0.9754
0.7212
0.8644
Ls
0,9900
0.8849
0.9863
0.8435
0.9858
0.9814
0.9935
0.9014
0.9879
0.9633
0.9753
0.9012
14 0.9453
0.8489
0.9568
0.9630
0.9296
0.9492
0.9921
0.8740
0.9468
0.9545
0.7048
0.8324
Lg
0.8946
0.8309
0.8959
0.9049
0.8931
0.9142
0.9139
0.8667
0.9028
0.9173
0.7115
0.8356
Table 21: Supplementary Table of 4 4 Huber loss of parameterizations across domains, each unit displays the average value of k-fold cross-validation.
Paramederization
Code
Math
Law
Music
Chemistry
Medical
G
D
G
D
G
D
G
D
G
G
L
0.0014
0,0070
0.0016
0.0064
0.0020
0.0059
0.0010
0.0148
0.0018
0.0041
0.0051
0,0024
L2
0.0077
0.0010
0.0148
0.0017
0.0042
Ls 0.0013
0,0061
0.0015
0.0065
0.0018
0.0059
0.0010
0.0151
0.0017
0.0042
0.0041
0.0027
0.0040
0.0074
0.0046
0.0060
0.0047
0.0104
0.0049
0.0048
Ls. 0,0067
0,0162
0,0071
0,0122
0.0090
0.0089
0.0063
0.0112
0.0087
0.0087
0.0188
0.0964
Table 22: Supplementary Table of Table R 5 parameterizations across 6 domains, each unit
displays the average value of k-fold cross cross-validation.
Parameterization
Code
Math
Law
Music
Chemistry
Medical
G
D
G
D
G
D
G
D
G
L1
0.9978
0,9746
0.9976
0.9892
0.9965
0.9861
0.9983
0.9181
0.9971
0.9912
0.9829
0.9443
Ta
0.9890
0.9985
0.9221
0.9911
La 0.9980
0.9761
0.9977
0.9892
0.9972
0.9861
0.9984
0.9293
0.9974
0.9911
0.9899
0.9585
0,9600
0.9844
0.9803
0.9856
0.9836
0.9404
0.9800
0.9887
Lg
0,9801
0,9106
0,9779
0.9672
0.9670
0.9775
0.9794
0.9491
0.9674
0.9759
0.8702
0.2798
28

---

K
Supplementary Figures
K.1
Effectiveness of D-CPT Law
N-D D.BB
N-1.88
N-4.0B
Eerlan
Caneree Valder LOR
Numbers training
Numbers training tokens
Numbers training tokens
Figure 13: Effectiveness of D-CPT Law(L General-corpus validation loss Ly with respect to dataset size D across different model size N, domain-corpus is code and general-corpus mixture ratio ry is 0.33.
Nm0.56
N=1.66
4.06
Frid Dic
Frud Dic
Fnd Dld
1.55
Numbers of training tokans D
Numbers of trailing tokens
Numbers of traiiing tokens
Figure 14: Effectiveness of D-CPT Law(Ls): Domain corpus validation loss La with respect to dataset size D across different model size N, domain corpus chemistry and domain- corpus mixture ratio is 0.5.
29

---

K.2 Dataset Size Generalizability of the D-CPT Law
N=
N=1 88
N=4.0B
Real ball
Cantre walccattin Lock
Numbers training
Numbers training tokans
Numbers training
Figure 15: Dataset Size Generalizability of the D-CPT Law: General-corpus validation loss Lg with respect to dataset size D across various model sizes N, domain-corpus is math and general-corpis mixture ratio rg 0.8. The experiments use data from the first 2/3 of the steps for fitting, verify whether the D-CPT Law exhibits generalizability across different dataset sizes.
K.3 Domain Generalizability of the Cross-Domain D-CPT Law
N=0.5B
N=1.8B
N=4.00
Redlan Lau
Los
Aos Lous
25
4g 23
Lg
Lg
23
14
D
D
D
Figure 16: Domain Generalizability of the Cross-Domain D-CPT Law: General-corpus validation loss T.y with respect to dataset size across various model sizes N, domain-corpus Music and general-corpus mixture ratio rg 0.8. The experiments use data points from [Code, Math, Law, Medical] domains for fitting, to verify whether the Cross D-CPT Law exhibits generalizability across different domains.
N=0.5B
N=1.8B
N=4.00 .0
Rod
Loc Anised lan
Fedand
Lg
Lg
Lg
27
24
D
D
Figure 17: Domain Generalizability of the Cross-Domaie D-CPT Law: General-corpus validation loss Ly with respect to dataset size D across various model sizes N, domain-corpus is Chemistry and general-corpus mixture ratio rg 0.8. The experiments use data points from (Code, Math, Law, Medical] domains for fitting, to verify whether the Cross D-CPT Law exhibits generalizability across different domains.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly stated our main contributions in our Introduction and Ab- stract. The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [Yes]
Justification:
Certainly, we have corresponding detailed derivations for all theoretical results,
such as Appendix E.3 and Appendix G
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The main experimental results are reproducible. We have mentioned our experimental setup in Section 4 and Appendix F2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
32

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code belongs to company's intellectual property, but the data can be downloaded from open-source repositories. For example, the Dolma dataset can be downloaded from https:://githubb.com wot.:///aoo/alaaaaa//a//// In the main text, we cited all the data sources.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis. cc/
public/guide httpe//guiiide/eeeuuuineeeeeee CoodeSummissionoool1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cilips.cc/publle httee//ullccggguddeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Section and Appendix F2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experimental results have been validated through cross-validation and repeated experiments. The effectiveness and significance of the method have been confirmed by multiple domains and multiple k-fold cross-validation experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have discussed our compute resources Appendix
E.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research respects the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts of our works in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have respected all the licenses and terms of use for CPT data and the Qwen model.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our works do not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our works do not involve crowdsoureing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "rlaw3kEstE", "reviewer": "Reviewer_ycBX", "summary": "For continual pretraining of domain-specific large language models, an important question is how to choose the optimal mixture ratio between the general corpus and the downstream domain corpus. This paper proposes to fit the scaling law for domain-specific continual pre-training (D-CPT Law), using small-scale training costs to predict the general and downstream performance of arbitrary model sizes, data sizes, and data mixture ratios. They also extend it to Cross-Domain D-CPT Law to reduce training costs for the target domain. The paper shows empirical evidence for the effectiveness and applications of the Laws.", "questions": "- Is it possible to fit a set of parameters in the scaling law, such that all independent variables can change their values?\n- How many different $(N, D, r)$ are needed to fit the parameters in the D-CPT Law formula?", "limitations": "The authors addressed the limitations well in Appendix A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Discovered that model performance can be predicted given the data mixture ratio between general and domain-specific corpus and a few other factors in domain-specific language model training.\n- Proposed essential requirements of the scaling law formula, compared multiple parametrization of the scaling law, and proposed a parametrization that extends the Chinchilla Scaling Law.\n- Extended the law to cross-domain scenarios and described its features.\n- Empirically verified the effectiveness, model size generalizability, and dataset size generalizability of the proposed methods.", "weaknesses": "- The paper lacks a comparison between applying the proposed laws and performing a grid search over hyperparameters. How much computation is saved in various applications?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments and questions. I appreciate the time and effort you have put into reviewing our manuscript. Below, I address your concerns and provide further clarifications.\n\n**Weakness**\n\nFirst, the main motivation of our paper is to address the challenge of determining the data mixture ratio between general-corpus and domain-corpus during domain-specific continual pre-training. Specifically, we aim to predict performance for any given N, D, and r using scaling laws. To achieve this, we conduct multiple experiments to collect data points for fitting the D-CPT Law and then use the L-BFGS algorithm to fit the parameters of the D-CPT Law. Once we have fitted the D-CPT Law for a specific domain, we can use it to predict model performance for different N, D, and r settings. Therefore, our D-CPT Law can be considered as an **auxiliary tool**, which can be used to estimate validation loss in various scenarios and real-world usages once we obtain the D-CPT Law. In contrast, grid search typically addresses a single search problem under one setting, and the results of grid search are usually **not reusable** for other settings. Therefore, it would be **unfair** to directly compare the experimental costs of the two methods based on these aspects. However, for clarity, we will show the differences in experimental costs between using the D-CPT Law and grid search for a specific setting, as illustrated in Table 5 of the paper. \n\n**Cost of D-CPT Law**: We take the setting of Table 5 as an example. Specifically, the model size is 1.8 B, and the dataset size is 10B (See Line 214-Line 221). We have prepared data points for 9 different ratios, so the total compute budget required for fitting D-CPT Law is:\n$$1.8 * 10^9 * 10 * 10^9 * 9 * 6/10^{12}=9.72*10^8\\  \\text{TFLOPs}$$ \n**Cost of Grid Search**: In Table 5, both the D-CPT Law and actual experimental data indicate that the optimal data ratio is 0.924. To perform a grid search on this ratio, based on binary search, we start by conducting experiments with r=0 and r=1. Then we would test r=0.5, followed by r=0.75, r=0.875, r=0.9375, r=0.90625, r=0.921875, r=0.9296875, r=0.92578125, and r=0.923828125. Assuming this meets the requirements under the settings of Table 5, a total of 11 experiments on different ratios would be needed. The total compute budget is: \n$$1.8 * 10^9 * 10 * 10^9 * 11 * 6 /10^{12}=11.88 * 10^8 \\ \\text{TFLOPs}$$\nBy comparison, it can be observed that in the single setting of Table 5, the experimental cost of grid search is still higher than that of D-CPT Law. Moreover, a significant feature of scaling laws is that the performance of small-scale experiments can be used to predict the performance of large-scale experiments. Therefore, for a 14B model, D-CPT Law can be experimented on at 1.8B, whereas grid search needs to be experimented on at 14B. This results in a tenfold difference in experimental cost for a single experiment, and the results of the grid search cannot be **reused**. Furthermore, if the grid search needs to evaluate 10 experimental settings, the difference would grow to 100 times (14 / 1.8 *11.88 / 9.72 * 10 = 95.06 times). Therefore, the D-CPT Law significantly reduces the experimental cost required by grid search, and it can be reused across multiple settings.\n\nWe appreciate your suggestion and will include this supplementary information in our new version to visually illustrate the cost difference between grid search and D-CPT Law.\n\n**Q1**\n\nFor clarity, here I first briefly introduce the D-CPT Law formulation and the fitting process as follows:\n\n$$L(N,D,r)=E+\\frac{A}{N^\\alpha}+\\frac{B\\cdot r^\\eta}{D^\\beta}+\\frac{C}{(r+\\epsilon)^\\gamma},$$\nwhere N, D, and r are independent variables and the other parameters as fitting parameters. After collecting multiple data points for (N, D, r), the L-BFGS algorithm is used to fit the D-CPT Law and obtain specific values for the fitting parameters E, A, B, C, $\\epsilon$, $\\gamma\\$, $\\alpha$, and $\\beta$. Then, we can use the D-CPT Law to predict L for any given N, D, and r.\n\nFor your question, I have **2 interpretations**. I will explain each situation separately:\n1. If you mean fitting the relationships between L and N, D, and r separately, as done in the OpenAI Scaling Law(https://arxiv.org/pdf/2001.08361),  I have added the fitting formulation for L versus D and L versus r as follows:\n$$ L(D) = E_D + \\frac{B}{D^\\beta}, \\quad \\text{when N, r are fixed} $$\n$$ L(r) = E_r + \\frac{C}{r^\\gamma}, \\quad \\text{when N,D are fixed} $$\nSimilarly, the fitting parameters of the above formula can be obtained using L-BFGS (for instance, we have fitted the above formula and we get B=2.2075, $E_D$=0.6175585, $\\beta$=0.014580619, C=0.49847686, $E_r$=2.1966128, and $\\gamma$=0.13764255). This allows us to individually explore the relationship between L and D for a given N and r, as well as the relationship between L and r for a given N and D. However, the goal of our work is to explore the relationship between L and (N, D,r) together. The relationships L(D) and L(r) are insufficient for exploring the behavior of L with respect to arbitrary values of N, D, and r.\n2. If you mean fitting the relationship among L and (N, D, r) (as our paper does), each independent variable can change its values independently. However, in terms of the function's trend, D and r are interrelated (see line 106 - line 110 ). Specifically, changes in r will affect the trend of L with respect to D, and changes in D will affect the trend of L with respect to r. This can be seen from the form of Equation 5, and further explanation can be found in Appendix E.2 of the paper. Additionally, when N and D are fixed, L only depends on r; when N and r are fixed, L only depends on D; and when D and r are fixed, L only depends on N. This leads to the above situation where we fit L(D) and L(r) independently.\n\n**Q2**\n\nPlease see **G.Q2** in Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AAMns2Ztpj", "reviewer": "Reviewer_rpUb", "summary": "This work explore the data mixture scaling law ( D-CPT Law) between the general corpus and the downstream domain-corpus for the continual pre-training of large language model.They also extend the fitted standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to  predict the D-CPT law of target domains.  D-CPT Law could be applied on three important scenarios: optimal mixture on the trade-off between general and domain-specific abilities, optimal  mixture for limited domain-specific data, and resource allocation setting, which is useful in practice.", "questions": "1.Could you explain why this scaling law function is for continue pre-training?\n2.In Equation 7, the newly involved parameter K, why is also in a similar power law pattern? The domain-specific learnable coefficient might be a much different dimension with the total number of model parameters and training data.\n3.How many combination did you choose for the experiments? There are many between the data sizes from 0.1B to 26B. The way the portfolio is chosen has an impact on the on the average results.", "limitations": "As the authors have stated comprehensive in the limitation Section, I have no further comments for this part.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This work experimented and analyzed on three scenarios including optimal mixture on the trade-off between general and domain-specific abilities, optimal  mixture for limited domain-specific data, and resource allocation setting, which is useful in practice\n2. D-CPT Law considered both the in-domain setting and the cross-domain setting with a new involved parameter K, which provide a reference for the unseen domain data mixing.\n3. The analysis on the selected fitting function is comprehensive and make sense.", "weaknesses": "This work focus on the data mixing scaling law for the continue pre-training phase of LLM. However, I think this work didn’t specifically have any design for the continue pre-training. The setting for the final scaling Function (Equation 6) have not mentioned any specific for continue pre-training. As the author mentioned , Equation 6  is supposed to transform into the Chinchilla  Scaling Law (which is for a LLM training from scratch) when r is fixed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and constructive suggestions. We will address your concerns shown below in detail.\n\n**Weakness**\n\nFrom the view of validation loss, continual pre-training (CPT) and pre-training (PT) are consistent, with the main difference being the initial validation loss (CPT: <5, PT: >10). This difference is represented by the bias term E in the D-CPT Law, and the fitted E values will differ between CPT and PT.\n\nIn addition, the scenario of our work is Domain-specific Continual Pre-Training (D-CPT). In this setting, we usually need to collect high-quality domain-corpus to enhance the downstream performance and general-corpus to mitigate catastrophic forgetting of the general abilities. There, the mixture ratio (i.e., r) between domain-corpus and general-corpus plays an important role in the D-CPT setting, which is also the key distinction from the Chinchilla Scaling Law. Besides, though r is also introduced in PT, it's typically multidimensional (e.g., different data sources in CommonCrawl). In the D-CPT setting, r is two-dimensional.\n\nMoreover, we think that the consistency between the scaling laws in CPT and PT offers several benefits. First, it enables a seamless connection between PT and CPT. Specifically, by using a step function or other methods, we can jointly represent the scaling laws of both phases in a uniform parameterization. Second, when r is fixed and the D-CPT Law becomes equivalent to the Chinchilla Scaling Law, it ensures that the D-CPT Law can address resource allocation issues (line 235-line 237).\n\nTherefore, we designed such D-CPT Law. In the future, we will continue to investigate how to unify the Scaling Law for both PT and CPT.\n\n**Q1**\n\nThe Scaling Law is derived from fitting data points obtained from real experiments. Therefore, a good parameterization for D-CPT Law must align with the trends observed in actual data, only then can the parameterization effectively fit the data. In Section 3.1 (line 89-line 117), we listed 4 requirements that a good parameterization should meet: Adaptability, Explicit Trends, Implicit Trends, and Consistency. Both Explicit Trends and Implicit Trends refer to the observable trends in actual D-CPT scenarios. The final form of the D-CPT Law (Equation 6) successfully meets these 4 requirements (detailed mathematical derivation is provided in line 521-line 564). We believe that as long as these 4 requirements are met, the parameterization is sufficiently robust. For more information, please see **G.Q1** in Author Rebuttal.\n\n**Q2**\n\nIn Section 3.2, I mentioned K, which represents the learnability of a domain. Unlike N, D, and r, which are real variables, K is an abstract variable for which we wish to establish certain predefined properties, such as Uniformity and Monotonicity, as discussed in Section 3.2 (line 130-line 135). We incorporate this variable into the D-CPT Law using a power-law form (both in OpenAI Scaling Law(https://arxiv.org/pdf/2001.08361) and Chinchilla Scaling Law(https://arxiv.org/pdf/2203.15556), they use a power-law form to represent the scaling law). Our main objective is to first **find a method to represent K (i.e., to model K)**, and then we focus on the form of K in the D-CPT Law, where we directly use the power-law form to ensure it meets the requirements of Uniformity and Monotonicity.\n\nSecondly, the dimensions of K differ from those of N and D. This difference is not only reflected in their value ranges (N: 0.5 to 4, while K: 0.5 to 1) but also in the fitting parameters of their respective power-law forms. For instance, the fitting parameters for N, denoted as A=4.18 and $\\alpha$=1.61. In contrast, the fitting parameters for K, denoted as F=0.35 and $\\mu$=0.55.\n\nFinally, we added a comparison of the fitting results with different K forms. I supplemented 2 parameterizations:\n$$L_2=E+\\frac{A}{N^\\alpha}+\\frac{Br^\\eta}{D^\\beta}+\\frac{C}{r'^\\gamma}+\\frac{F}{\\mu^K}$$\n$$L_3=E+\\frac{A}{N^\\alpha}+\\frac{Br^\\eta}{D^\\beta}+\\frac{C}{r'^\\gamma}+\\frac{F}{(-\\log K)^\\mu}$$\nThe fitting results for L1(original form), L2, and L3 on general-corpus are as follows:\n|Paramterization|Huber loss|$R^2$|\n|-|-|-|\n|L1|**0.0214**|**0.9886**|\n|L2|0.0256|0.9781|\n|L3|0.0296|0.9847|\n\nIt can be observed that the fitting results for different parameterizations do not show significant differences, so we believe it is more important to model the representation form of K.\n\nAdditionally, modeling K with the loss (equation 9) is very intuitive: \n$$K=\\frac{w_1}{k_1}+w_2\\times k_2,$$\nwhere k1 represents the initial loss of the model. A higher value indicates the model's weaker ability in this domain, making it harder to learn, so it is inversely proportional. k2 represents the rate of decline in the loss for this domain. A higher value indicates better mastery of this domain, so it is directly proportional. We use the loss for modeling K because loss is a very direct assessment of model performance (also used in Compression(https://arxiv.org/pdf/2404.09937) to represent compression).\n\n**Q3**\n\nIn the main experiment (Section 4.2), there are 6 domains, 3 model sizes, 200 dataset sizes, and 9 data mixtures.\n- **Effectiveness**:For the Effectiveness experiment, we fit and evaluate using all the data points.\n- **Model Size Generalizability**: we use 3-fold cross-validation. We randomly select 2 model sizes for fitting and then use the fitted D-CPT Law to predict the remaining model size. This yields 3 cross-validation experiments, and we take the average result.\n- **Dataset Size Generalizability**: we divide the 200 dataset size data points into 3 groups (1-67, 66-133, 133-200). Using 3-fold cross-validation, we fit two groups and validate the remaining group. This yields 3 cross-validation experiments, and we take the average result.\n- **Mixture Ratio Generalizability**: we randomly select 7 ratios for fitting and use the remaining 2 ratios for validation. With 36 combinations of selecting 7 out of 9 ratios, we take the average.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JynDDrbRee", "reviewer": "Reviewer_sFHX", "summary": "This work proposes a domain-specific scaling law to optimize mixture ratio between the general domain training corpus and the specific domain corpus. The authors considered several parameterizations and experimented on six different downstream domains such as code, math, and Medical. The result show that D-CPT Law has a high fitting accuracy across various model sizes, dataset sizes, and mixture ratios. Furthermore, the proposed cross-domain D-CPT Law reduced training costs while maintaining high performance in predicting domain-specific continual pre-training results.", "questions": "1. The author stated, \"we test the validation loss every 1,000 steps and the total training steps are 20k.\" How did you determine the step parameters, such as 1,000 and 20,000?\n\n2. You defined five parameterizations and demonstrated that L3 is the best through experiments. Could there be other parameterizations better than L3? In other words, is it possible to theoretically prove that L3 is the optimal D-CPT Law?", "limitations": "The authors state several limitations and will consider addressing them in their future work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a domain-specific scaling that can more efficiently determine thec optimal mixture ratio between general and domain-specific corpora. The experiments demonstrate its effectiveness.\n\n2. The paper presents three applications of D-CPT Law that may be of interest and benefit to the community.", "weaknesses": "1. The models are limited to the Qwen series, which are relatively small in size. This makes it unclear whether similar results would be observed with larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments. I appreciate the time and effort you have put into reviewing our manuscript. Below, I will address your concerns and provide further clarifications.\n\n**Weakness**\n\nWe acknowledge that the current status of our works is limited to the Qwen series (as we have also mentioned in the limitations) and the size of our model is relatively small (0.5 B to 4 B parameters). However, I will explain why we chose the Qwen series and the reasons behind the model size from the following points:\n1. **The Choice of the Qwen Series**: Our work focuses on domain-specific continual pretraining(D-CPT), thus we need open-source base LLMs with strong foundational capabilities. Additionally, as we aim to explore performance across different model sizes, it is necessary that the model series have multiple model sizes. To meet these requirements, we choose the Qwen-1.5 series as this series provides multiple model versions (0.5B,1.8B,4B,7B,14B,32B,72B,110B). In contrast,  LLaMA2/3 only has 3 available model sizes (7B,13B,70B),  and the training costs on these models are large. While Pythia also offers a variety of model sizes, its overall model performance is relatively poor, making it unsuitable for D-CPT in practical scenarios.\n2. **Relevantly Large Model Size**: Regarding the comparatively smaller model sizes mentioned, we believe our model sizes and dataset sizes are already quite large (model sizes: 0.5B,1.8B,4B, and dataset sizes: 0.1B-26B), especially when compared to 2 related works about data mixture (Data Mixing Law(https://arxiv.org/pdf/2403.16952): model sizes: fit on 70M, 160M and validate on 1B, dataset sizes: 30B; Regmix(https://arxiv.org/pdf/2407.01492): model sizes: fit on 1M and validate on 1B, dataset sizes: 1B). Additionally, we have also conducted experiments on 7B model to demonstrate that similar results will be observed with larger models (see Section 4.2, line 183-line 189). Besides, our primary focus is on the data mixture, and increasing the model size would result in a huge rise in experimental costs. Therefore, we choose these model sizes as the training and evaluation computation costs are relatively acceptable.\n\nLastly,  the recent work Observational Scaling Law (https://arxiv.org/pdf/2405.10938) has explored the Scaling Law across different models, they stated that \"model families only vary in their efficiency in converting training compute to capabilities.\" We agree with their perspective and we think the overall experimental patterns and findings observed in the Qwen series can also be extended to other model families.\n\n**Q1**\n\nIn the Introduction section (line 50), we mention \"dataset sizes from 0.1B to 26B tokens\". However, there is a typo in Section 4.1 (line 159) on the training setup: \"20k\" should be \"200k.\" Thank you for your question, which helped me discover this typo. \n\nThen, I would like to explain the conversion between the training step (t) and dataset size (D). The global batch size is 64, and the length of single data is 2048. The unit of dataset size is B, so the relationship between them is:\n$$D=t\\cdot64\\cdot2048\\cdot10^{-9}$$\nThe total training step is 200k, which corresponds to a dataset size of 26.21 B, while 1,000 training steps correspond to a dataset size of 0.131 B.\n\nFor the validation step count of 1000, we analyzed the fitting efficiency of different data sampling methods in Appendix I (line 621- line 643), which includes results for the validation step count of 5000. We conduct supplementary experiments with different validation steps on general-corpus validation loss. The fitting results (validation data points under different settings all correspond to the data points of the 1000 validation steps) are as follows:\n|Validation Step|Huber loss|$R^2$|\n|-|-|-|\n|1000|0.0041|0.9977|\n|2000|0.0043|0.9970|\n|3000|0.0045|0.9968|\n|4000|0.0042|0.9973|\n|5000|0.0042|0.9976|\n|10000|0.0053|0.9953|\n\nWe observe that the results are quite similar across different validation steps, indicating that the final findings of D-CPT Law remain consistent regardless of the chosen validation step. Moreover, selecting 1000 as the validation step ensures a sufficient number of data points.\n\nThe total training steps are set to 200k based on 3 main considerations:\n1. Our experimental setup assumes that all data is not repeated (each data is trained only once). While the general-corpus Dolma is sufficient, the domain-corpus might be insufficient. We have 6 domains, the total training steps must be less than the amount of domain-corpus. Setting the total training steps to 200k ensures no repetition of domain-corpus.\n2. Our total experimental cost is 5.3071 e+10 TFLOPs (compared to 9.864 e+10 TFLOPs for the Data-Constrained Scaling Law(https://arxiv.org/pdf/2305.16264)). Given the already high cost, increasing the total training steps would largely increase the experimental cost. Therefore, we chose 200k steps to maintain a reasonable experimental cost.\n3. We observed that the validation loss stabilizes around 100k-150k, so increasing the validation steps further will not lead to significant changes in the validation loss. Here, I will provide some actual data to support this point. In the music domain, for a 1.8B model, using a 1:1 data ratio, the following table shows the general-corpus validation loss and domain-corpus validation loss at different training steps.\n\n|Training Steps|General-corpus validation loss| Domain-corpus validation loss|\n|-|-|-|\n|10000|2.8191|0.8005| \n|50000|2.7674|0.7526|\n|60000|2.7612|0.7462|\n|70000|2.7556|0.7460|\n|80000|2.7505|0.7447|\n|90000|2.7471|0.7501|\n|100000|2.7432|0.7398|\n|120000|2.7367|0.7338|\n|140000|2.7312|0.7350|\n|160000|2.7253|0.7301|\n|180000|2.7206|0.7298|\n|200000|2.7159|0.7270|\n\nWe can observe that the changes in the model after 100k are not as significant as those before 100k and the loss is relatively stable when the training step is close to 200k.\n\n**Q2**\n\nPlease see **G.Q1** in Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
Thanks a lot for handling/reviewing our submitted manuscript. We would like to thank the reviewers for their thoughtful and constructive comments and suggestions. By addressing each of the issues raised by the reviewers, we believe that the quality and clarity of our D-CPT Law can be improved a lot. The general responses are summarized as follows:

**G.Q1:  Could there be other parameterizations better than L3?**

We acknowledge that there may exist better parameterization than L3, but we believe that this is not crucial because our core objective is to find one that satisfies the 4 requirements outlined in Section 3.1 (Line 100-Line 110), and we assume that when these 4 requirements are met, the parameterization is considered a good option of D-CPT Law. Specifically, regarding the choice of parameterization for the D-CPT Law, our research goal can be understood as finding a parameterization that matches the trend of real (N, D, r) data points and possesses certain mathematical properties. These trends and mathematical properties can be explicitly expressed as the 4 requirements in Section 3.1: Adaptability, Explicit trends, Implicit trends, and Consistency. Besides, in Section 4.2, we provide 5 parameterizations, only L3 can meet all 4 requirements.

Moreover, as there are also other parameterizations that can meet these 4 requirements, we provide another 2 parameterizations that satisfy these 4 requirements as follows:

$$ L_6 = E + \frac{A}{N^{\alpha}} + \frac{B \cdot e^{r}}{D^\beta} + \frac{C}{r'^{\gamma}} $$

$$L_7 = E + \frac{A}{N^{\alpha}} + \frac{B \cdot r'^\eta}{D^\beta} + \frac{C}{r'^{\gamma}} $$

The fitting results for L6, L7, and L3 on general-corpus are as follows:
| Paramterization | Huber loss | $R^2$ |
| ---------------- | ---------- | ----- |
| $L_3$              | 0.0048     | 0.9968  |
| $L_6$             | 0.0051     | 0.9963  |
| $L_7$             | 0.0049     | 0.9969  |

The fitting results for L6, L7, and L3 on domain-corpus are as follows:

| Paramterization | Huber loss | $R^2$ |
| ---------------- | ---------- | ----- |
| $L_3$              | 0.0157     | 0.9796  |
| $L_6$             | 0.0164     | 0.9778  |
| $L_7$             | 0.0160     | 0.9801 |

We observe that when 4 requirements are met, the fitting results do not  differ a lot. 
In conclusion, first, as there are many parameterizations that meet the 4 requirements, it is challenging to  find the optimal parameterization. Second, the L3 mentioned in the paper is relatively simple and meets the 4 requirements with good fitting results. In the future, we will continue to investigate how to find the optimal parameterization for the D-CPT Law.

**G.Q2: How many different (N, D, r) are needed to fit the parameters in the D-CPT Law formula?**

In the main experiment, we have a total of 6 domains, 3 model sizes (0.5B, 1.8B, 4B), 9 data mixtures, and 200 validation loss data points (collected every 1k training steps for a total of 200k training steps). Hence, the main experiment consists of 6x3x9x200 = 32,400 data points. In other words, there are 5,400 data points for each domain. 

Additionally, in Appendix I (see line 621 - line 643), we analyzed the impact of data point sampling methods on the fitting results. We found that sampling more points when D is relatively small, and sampling fewer points when D is relatively large, can greatly reduce the experimental costs. 

Besides, we found that validating every 5000 steps and every 1000 steps resulted in similar fitting results. Therefore, we can actually achieve a good fitting result with only 1/5 of the original data points, which means we only need 5400/5=1080 data points to fit. Therefore, in practical scenarios, considering fitting efficiency, approximately **1000** actual data points are sufficient to reasonably fit a domain's D-CPT Law.

Meta-review: This paper proposes a scaling law called D-CPT Law for domain-specific continual pre-training of large language models, which predicts model performance based on model size, dataset size, and data mixture ratio between general and domain-specific corpora. It has broad applicability to: trade-off between general and domain-specific abilities; find the optimal mixture for limited domain-specific data; allocate compute resources. 

The paper extends the D-CPT Law to cross-domain settings with Cross-Domain D-CPT Law to predict performance on new domains with minimal additional training. The paper conducts experiments on 6 domains showing the effectiveness and generalizability of the proposed scaling laws.


**Strengths**
1) Addresses an important problem of optimizing data mixture for domain-specific LLM CPT training; 2) Proposes a novel scaling law with theoretical justification and empirical validation; 3) Demonstrates practical applications and generalizability; 4) Thorough experiments and analysis across multiple domains and settings.

**Weaknesses**  
1) Limited to relatively small model sizes (0.5B-4B parameters) focused only on Qwen model series; Leaves some open questions about theoretical optimality of the chosen parameterization.


The paper is technically solid with good contributions. The authors provided reasonable justifications for their experimental choices regarding small models in the rebuttal. The authors provided good reasoning for their choices, and demonstrated some results on 7B models. Overall, the strengths outweigh the weaknesses.

Based on the reviews and discussion, I recommend accepting this paper (poster). Given the importance of the problem and the broad applicability, I recommend a spotlight
Decision: Accept (poster)