D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models

Authors: haoran  que
jiaheng  liu
ge  zhang
chenchen  zhang
xingwei  qu
yinghao  ma
feiyu  duan
zhiqi  bai
jiakai  wang
yuanxing  zhang
xu  tan
jie  fu
jiamang  wang
lin  qu
wenbo  su
bo  zheng
taobao & tmall group of alibaba
alibaba  group
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 13:15:32

Abstract: Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model’s fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1\% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.

Main paper: 
 introduction: 
 Continual Pre-Training (CPT) is an essential part of training better Large Language models (LLMs). In this work, we mainly focus on Domain-specific CPT (D-CPT), which aims to enhance the fundamental understanding abilities of the specific downstream domains and has been widely used in existing works [49, 41, 30] . In practice, for D-CPT, we usually need to collect high-quality domain-corpus to enhance the downstream performance and general-corpus to mitigate catastrophic forgetting on the general abilities [13, 40, 54, 47, 32, 20, 53] . Therefore, how to determine the data composition or mixture ratio of the domain-corpus and general-corpus plays an important role in producing well-performed domain-specific LLMs. Besides, grid-searching on the mixture ratios requires heavy GPU consumption costs, and we cannot always obtain the optimal ratio under limited GPU usage. Recently, Scaling Law has been widely used for performance prediction [31, 27, 43, 26 which can be used to find the optimal dataset size and model size under the given GPU consumption costs. Therefore, for D-CPT, can we find the optimal mixture ratio in the training corpus using the Scaling Law to enhance the performance of domain-specific tasks?To address the above question, in this work, we investigate the Scaling Law of D-CPT and propose the D-CPT Law to find the optimal mixture ratio with limited training costs for LLMs with different sizes. Specifically, inspired by the robust predictive ability of Scaling Law across various scales, we first perform experiments under diverse mixture ratios and several relatively small model and data scales. Following the Chinchilla Scaling Law, we then introduce the mixture ratio r into the D-CPT Law, where the parameterization is defined as follows:EQUATIONwhere ϵ is used to guarantee the stability of L when r near zero. Based on Equation 1, for a model with model size N , dataset volume D and mixture ratio r, we can accurately predict the validation loss L. Note that when r denotes the domain-corpus mixture ratio r d , L means domain-corpus validation loss L d . Similarly, general-corpus validation loss L g also follows the law relationship with the general-corpus mixture ratio r g . To illustrate our D-CPT Law clearly, as shown in Figure 1 , we take the code domain as an example and provide the fitting results on the general and domain-specific settings, where we validate the fitting accuracy on different mixture ratios under a model with different dataset sizes D. Our main contributions are summarized as follows:(1). To show the effectiveness and generalizability of D-CPT Law, we perform extensive experiments using model sizes from 0.5B to 4B parameters, dataset sizes from 0.1B to 26B tokens, and mixture ratios from 0 to 1. The experiments show that the D-CPT law exhibits a high fitting accuracy with Huber loss [28] lower than 0.02 and R 2 [9] greater than 0.97. Besides, experiments on generalizability show that D-CPT Law not only inherits model size and dataset size generalizability following previous Scaling Law, but also precisely predicts performance for different mixture ratios.(2). Despite the effectiveness in an in-domain setting, where we fit the D-CPT Law based on data points from one downstream domain, we also apply our D-CPT Law in the cross-domain setting, which denotes that we use the data points from multiple domains to predict the performance of unseen domains. Specifically, we first introduce the Domain-specific Learnable Coefficient (DLC) to denote the domain-specific parameter of each domain and integrate the DLC into the D-CPT Law.We name this new law as Cross-Domain D-CPT Law. In this way, if we can obtain the DLC of a new domain, we can easily derive the D-CPT Law for this new domain. In our experiments, we fit the Cross-Domain D-CPT Law using data points from 4 domains and apply the Cross-Domain 
background: 
 Following previous work [43] , we categorize the objectives of Scaling Law as Allocation and Return. Specifically, (1) . Allocation: What is the optimal allocation of model size N and dataset size D given a fixed compute budget? (2) . Return: What is the expected return on incremental resources?The first objective on Allocation is as follows:EQUATIONIn Equation 2, given a fixed compute budget C, the objective is to find the optimal model size N and dataset size D that minimize the loss. The second objective on Return fundamentally depends on the generalizability of Scaling Law to accurately predict beyond the fitting data points.Chinchilla Scaling Law Hoffmann et al. [27] propose a parameterization as follows:EQUATIONwhere {E,A,B,α,β} are fitting parameters (See Appendix D for more details). 
methods: 
 In Figure 2 , D-CPT Law aims to investigate the behaviors of the Domain-specific Continual Pre-Training scenario with respect to different mixture ratios, and the objective of the D-CPT Law is to analyze an appropriate parameterization of law that represents the relationship of validation loss L with respect to model size N , dataset size D, and mixture ratio r. In this section, we first discuss the D-CPT Law in the in-domain setting (Section 3.1), where the fitting and testing data points are from the same domains. Then, we propose to adapt D-CPT Law to the cross-domain setting (Section 3.2), where the fitting and testing data points are from multiple domains, and introduce the Cross-Domain D-CPT Law, where a new term Domain-specific Learnable Coefficient (DLC) is used.As the training data includes a mixture of general-corpus and domain-corpus, we introduce two mixture ratios (i.e., general-corpus mixture ratio r g and domain-corpus mixture ratio r d ). Correspondingly, we define two validation losses (i.e., general-corpus validation loss L g and domain-corpus validation loss L d ). Therefore, we can derive two D-CPT Laws (i.e., L g (N, D, r g ) and L d (N, D, r d )).For convenience, we directly use r and L(N, D, r) as default notations for D-CPT Law. Besides, in Appendix D, Chinchilla Scaling Law provides greater interpretability and clarity when compared to OpenAI Scaling Law. Thus, we choose Chinchilla Scaling Law as the foundational parameterization for D-CPT Law. In addition, since Scaling Law aims to fit data points, their parametric forms should be intrinsically related to the observed trends in the data points. Based on previous works and data trends with varying N , D and r, we have summarized 4 essential requirements for D-CPT Law:• Adaptability: D-CPT Law is valid for values of r between 0 and 1.• Explicit trends: Based on the results across varying values of N , D, and r, we observed the following explicit trends of data points:EQUATIONThe first two trends are consistent with the previous Chinchilla Scaling Law, and the third trend also has an intuitive explanation. A larger r indicates a higher proportion of valid corpus in the training corpus, leading to a lower L. Details are provided in Appendix E.1.• Implicit trends: We further discover inherent connections between r, D and L as follows:EQUATIONFor detailed explanations, please refer to the Appendix E.2.• Consistency: When r is fixed, the D-CPT Law is supposed to transform into the Chinchilla Scaling Law. In this way, D-CPT Law can inherit the excellent features of Chinchilla Scaling Law and address the issues on resource allocation discussed in Section 2.To satisfy these requirements, we have compared multiple parameterizations in Section 4.2 and eventually, we propose the parameterization as follows:EQUATIONIn Equation 6, {E, A, B, C, α, β, γ, η,ϵ} are the fitting parameters and we use L-BFGS [34] to fit the D-CPT Law due to its suitability for large-scale optimizations. As shown in Section 4.2, the Equation 6 can accurately fit the trends of data points at any scale and demonstrates strong performance in both effectiveness and generalizability. Besides, it effectively meets the aforementioned 4 requirements. (Please see Appendix E.3 for more details on the mathematical derivation.)Apart from the in-domain setting for D-CPT Law, we also investigate the cross-domain setting and extend the D-CPT Law to the Cross-Domain D-CPT Law, which aims to reduce the training costs of the D-CPT Law significantly. Specifically, although the D-CPT Law collects data points using small LLMs, the GPU resource and time costs are still relatively substantial, which limits the applications of the Scaling Law. Therefore, in our Cross-Domain D-CPT Law, we first define the Domain-specific Learnable Coefficient (DLC) K for each domain, which measures the learnability for a specific domain1 (See Section 4.4 for more details.). Then, we incorporate the K into the D-CPT Law and obtain the Cross-Domain D-CPT Law, which is defined as follows:EQUATIONIn Equation 7, {E, A, B, C, F, α, β, η, γ, ϵ, µ} are fitting parameters. Thus, for an unseen domain, we only need to calculate the DLC with modest costs, which substantially increases the domain generalizability of D-CPT Law. Besides, Cross-Domain D-CPT Law has the following features:• Uniformity: Once we calculate the K value of an unseen domain, we can convert Crossdomain D-CPT Law into normal D-CPT Law as follows:L(K = K 0 ) = E 0 + A N α + B • r η D β + C r ′γ, whereE 0 = E + F K µ 0 , r ′ = r + ϵ.Therefore, the Cross-Domain D-CPT Law inherits all features of the D-CPT Law,• Monotonicity: K denotes the learnability of a specific domain, which aligns with the intuition that a more learnable domain yields lower validation loss. Meanwhile, the Crossdomain D-CPT Law confirms a monotonic decrease with respect to K, i.e.,EQUATIONAfter confirming the parameterization of Cross-domain D-CPT Law, it is essential to identify a representation for K that accurately quantifies a domain's learnability. The representation of K is supposed to be accessible, distinct and robust. Specifically, first, "Accessible" denotes that it is easy to obtain for an unseen domain with low costs. Second, "Distinct" indicates that K values must exhibit significant variance across domains to ensure fitting accuracy and maintain clear distinctions between domains. Third, "Robust" means that the representation of K enhances the effectiveness and generalization ability of Cross-domain D-CPT Law. In Section 4.4, we compare several variants of the representations of K, and the final representation is determined as follows:EQUATIONwhere w 1 and w 2 are fitting parameters, k 1 represents the initial validation loss for an unseen domain, and k 2 denotes the rate of decline in validation loss. 
experimental setup: 
 Data Setup To verify the effectiveness and generalizability of D-CPT Law and Cross-Domain D-CPT Law, we have prepared the 6 different downstream domains, which include Code [39] , Math [5] , Law [24] , Chemistry [10] , Music [44] and Medical [33] . For general corpus, we use Dolma [48] .All the tokens of these training datasets are sufficient, so the experiments are not performed under a data-constrained setting. Besides, we build a high-quality and held-out validation set for each domain.(See Appendix F.1 for more details.)Model Setup We use the Qwen-1.5 series due to its robust performance in both English and Chinese [6] . Furthermore, Qwen-1.5 has multiple open-sourced and well-performed pre-training base models. Specifically, we select Qwen-1.5-0.5B, Qwen-1.5-1.8B, and Qwen-1.5-4B as our base models to perform the continual pre-training for multiple downstream domains. Metrics Following [44, 43, 27] , we use validation loss as the performance indicator. To compare various parameterizations, we follow [28, 44] to utilize the R2 and Huber loss as evaluation metrics. Specifically, first, the coefficient of determination (i.e., R 2 ) indicates the fitting quality and typically ranges from 0 to 1, where a higher value means better explanatory power of the regression model. Second, Huber loss combines the properties of mean squared error and mean absolute error, which is particularly useful for regression with outliers. Similarly, Huber loss also assesses the fit qualities of different parameterizations, where lower Huber loss shows better fitting performances.In Section 3.1, an ideal parameterization should meet four requirements (i.e., adaptability, explicit trends, implicit trends, and consistency), and we define the following five parameterizations:L 1 = E + A N α + B D β + C r ′γ , L 2 = E + A N α + B D β + C r ′γ η , L 3 = E + A N α + Br η D β + C r ′γ , L 4 = E + A N α + B • b r D β + C c r , L 5 = E + A N α + B (rD + (1 -r) σ) β ,where {N ,D,r} are variables and others are learned parameters fitted by L-BFGS algorithm [34] which is the same as Chinchilla Scaling Law.Effectiveness As shown in Table 1 , we present the performance of five different parameterizations.In effectiveness settings, we use entire data points for fitting with the aim of validating the effectiveness of various parameterizations. In Table 1 , we observe that although L 5 has the fewest fitting parameters, its performance is significantly less impressive compared to the others. L 1 and L 4 , having relatively fewer fitting parameters, still fall short in performance compared to L 2 and L 3 . Moreover, L 1 fails to meet the requirements for implicit trends, while L 4 does not satisfy the explicit trends requirement. Finally, the results of L 2 and L 3 are comparable, but L 2 does not meet the requirements for consistency. Therefore, we choose L 3 for D-CPT Law. Moreover, Figure 3 shows the robust effectiveness of L 3 across varying dataset sizes, mixture ratios, model sizes, and domains.Model Size Generalizability: Our main experiments focus on 3 model sizes: 0.5B, 1.8B, and 4B. We use 3-fold cross-validation to evaluate the model size generalizability of D-CPT Law, and the average results across domains are shown in Table 2 . For example, we fit D-CPT Law with data points from 0.5B, and 1.8B and evaluate the Huber loss and R 2 for 4B. In Table 2 , we observe that D-CPT Law can generalize well across model sizes and L 3 shows the best performance. Besides, we conduct experiments on the unseen 7B size (i.e., Qwen-1.5 7B), and observe that D-CPT Law can accurately predict the general-corpus validation loss with a general-corpus mixture ratio of 0.2 in Figure 4 .Dataset Size Generalizability: Our main experiments cover dataset sizes from 0.1B to 26B tokens, and we also utilize a 3-fold cross-validation approach. The data points are uniformly divided into three segments, with 2/3 used for fitting the model and the remaining 1/3 for testing. In Table 3 , we report the average results across domains, and observe that L 3 shows notably enhanced dataset size generalizability.Figure 4 : L g with respect to D, domain-corpus is code, r g = 0.2, N = 7B. Mixture ratio Generalizability: We apply the k-fold cross-validation method across various parameterizations. Specifically, we select 7 out of 9 mixture ratios for fitting and the remaining for testing, resulting in 36 experiments per domain. For simplicity, we show average results across domains in Table 4 , and observe that that L 3 still shows significantly better performance on mixture ratio generalizability. Besides, in Figure 1 , we observe that our D-CPT Law has well-performed generalizability on unseen mixture ratios.Usage 1: Trade-off between general and domain-specific abilities For D-CPT, training data is a mixture of general and domain-specific data, where r g and r d denote the corresponding proportions, respectively. In D-CPT Law, when r g increases, the L g will decrease and L d will increase, indicating a trade-off between the general and domain-specific abilities of LLM. Fortunately, D-CPT Law can identify the optimal mixture ratio under any trade-off scenario. Specifically, we assume that an LLM with parameter size N 0 , it exhibits general-corpus validation loss of L 0 g and domain-corpus validation loss of L 0 d before continual pretraining. After mixing training data size of D 0 with a ratio r d of domain-specific data and 1 -r d of general data, we obtain general-corpus validation loss L g and domain-corpus validation loss L d after D-CPT. Then, we can identify the optimal mixture ratio while limiting the decline in the model's general abilities within a threshold T as follows: argminEQUATION)where T is the threshold based on practical need. In Appendix G.1, given a fixed T , a unique optimal solution r d is obtained. To validate it in real scenarios, by applying D-CPT Law, we calculate the optimal domain-corpus mixture ratio r d = 0.924 given a dataset size D 0 = 10B, model size N 0 = 1.8B, T = 3%, domain-corpus is chemistry, and initial general validation loss value of L 0 g = 2.8602. Table 5 presents the results of real general-corpus validation loss and domain-corpus validation loss with respect to different domain-corpus mixture ratios, we find that the real value exactly matches the predicted values(L g_pred = 2.9458 and L d_pred = 1.7284), domain-corpus mixture ratio exceeding 0.924 leads to a general validation loss that surpasses the 3% threshold of L 0 g .Usage 2: Optimal mixture on limited domain-specific data Given that domain-corpus is typically limited relative to the abundance of the general corpus, we study how to determine the optimal mixture ratio when domain-corpus is limited and general-corpus is sufficient. Specifically, for an LLM with parameter N 0 and limited domain-corpus D 0 d , we aim to minimize the domain-corpus validation loss L d by selecting the optimal domain-corpus mixture ratio r d as follows:EQUATIONIn Equation 11, we can reach the minimum value within the0 < r d < 1 discussed in Appendix G.2.To validate it in real scenarios, we conducted experiments within the music domain by setting the model parameters N 0 = 1.8B and the domain-specific dataset size D d = 5B. As we have data points at a large scale, we fit D-CPT Law using only data where D d < 2B to align with the use case scenarios. After using the D-CPT Law, we find that the optimal domain-corpus mixture ratio is 0.732. Table 6 shows the results of real domain-corpus validation loss of the music domain. We observe that r d = 0.732 yields the lowest domain-corpus validation loss. Moreover, our predicted domain-corpus validation loss is 0.7328 when r d = 0.732, which is close to the real value (0.7309).Usage 3: Resource allocation D-CPT Law is consistent with Chinchilla Scaling Law under the fixed mixture ratio to address resource allocation. Specifically, how to find the optimal values of N and D given a fixed compute budget. Detailed results are shown in Appendix G.3.In Section 3.2, we have mentioned that the learnability of a specific domain is measured by DLC (i.e., K). For Cross-Domain D-CPT Law, K must satisfy 3 core requirements: accessible, distinct, and robust. Based on these requirements, 4 different representations of K are defined as follows:EQUATIONwhere {w 1 , w 2 , w 3 } are fitting parameters. In the approximate Taylor expansion for the validation loss function near the initial points, {k 1 ,k 2 ,k 3 } represent the first three coefficients. Due to the discrete nature of data points in practical scenarios, {k 1 ,k 2 ,k 3 } are approximated using the variants of validation loss. Specifically, k 1 denotes the precise value of the validation loss at the initial point, k 2 represents the difference in validation loss close to the initial points, and k 3 is an approximation of the second derivative of the validation loss near the initial points, details are provided in Appendix H.To compare these four representations of K, we have conducted experiments in both effectiveness and generalizability aspects. Effectiveness We utilize data points from all 6 domains for fitting and then evaluate their performance using R 2 and Huber loss. In Table 7 , we find that 4 representations of K yield comparable results in the general domain. However, K 1 and K 2 demonstrate a noticeable decline in domainspecific aspects. Although K 4 slightly outperforms K 3 in domain-specific aspects, it requires a larger number of fitting parameters. Therefore, considering the balance between fitting efficiency, fitting performance, and accessibility, We consider K 3 to be the optimal representation. To further visualize it, Figure 5 illustrates the predicted curves in comparison to the real curves under various settings. Generalizability When K is identified, Cross-Domain D-CPT Law can be transformed into D-CPT Law, we believe that the former inherits the latter's generalizability in terms of model size, dataset size, and mixture ratio. In this part, we will specifically focus on discussing the domain generalizability of Cross-Domain D-CPT Law. To evaluate domain generalizability, we use data points from 4 out of 6 domains for fitting and assign the remaining 2 domains for testing. For simplicity, we only show the averaged results across 15 combinations in Table 8 . Among these 4 representations of K, K 3 exhibits the superior performance, further proving its strength. Scaling Law Many studies [26, 31, 27, 7, 16, 58] show a power-law relationship between performance and the increases in both the number of parameters and the size of the training data [31, 27, 18] , which are crucial for large language models (LLMs [45, 51, 29, 3, 55, 35, 59, 36, 60] ) and provide a predictive structure for determining the most efficient setups for expanded models using the insights gained from smaller models [17] . Moreover, the extension of scaling laws to autoregressive generative models widens their relevance to encompass tasks beyond text [18, 25, 17] . Recently, Muennighoff et al. [43] studied the Scaling Law of data-constrained settings by using the full pre-training dataset across multiple epochs. Ye et al. [57] investigate the data-mixing scaling law for the general LLMs to improve the pretraining efficiency. [23, 11, 22, 30, 19] . For example, Gururangan et al. [23] introduces a growing mixture of expert architecture for domain-adaptive continual pretraining. Cossu et al. [13] show that continually pre-trained models (RoBERTa [38] and BERT [14] ) are robust against catastrophic forgetting on downstream tasks. However, the above works only investigate small encoder-decoder models on limited tasks. Recently, Gupta et al. [21] study different warm-up strategies for continual pertraining for better results. 
conclusion: 
 In this work, we have investigated the Scaling Law of Domain-specific Continual Pre-Training (D-CPT), which provides a significant step forward in the optimization of training LLMs for specific downstream domains. By developing and validating the D-CPT Law, we can easily predict the optimal mixture ratio of general and domain-specific corpora, greatly reducing the previously necessary but costly grid-searching efforts. Besides, we also adapt our D-CPT Law to the cross-domain setting and introduce the Cross-Domain D-CPT Law to further reduce the efforts of fitting the D-CPT Law of new domains. Moreover, we discuss the three practical usages of our D-CPT Law. Finally, we believe our D-CPT Law is an initial investigation into quantitative prediction methods for the domain-specific continual pre-training. With its increasing focus on data engineering, we hope our exploration facilitates further quantitative studies and theoretical analyses in this research area. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have respected all the licenses and terms of use for CPT data and the Qwen model.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our works do not involve crowdsourcing and research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our works do not involve crowdsourcing and research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Experiments on more downstream domains In our work, main experiments cover six downstream domains [39, 5, 24, 10, 44, 33] . In future works, it is important to experiments on more downstream domains. We will attempt to conduct experiments on CPT in more domains and fit the D-CPT Law as well as the Cross-Domain D-CPT Law.Experiments on more LLMs We primarily conduct experiments based on Qwen-1.5 and lack exploration of other Pre-Trained Base LLMs.Multilingualism We lack research on multilingualism settings. Although the medical data is in fact in Chinese, the other data are in English. Moreover, the experimental results show that the fitting results in the medical domain are poor compared to others. We lack a detailed experimental analysis of different language settings. In future research, we hope to realize cross-linguistic and multi-linguistic D-CPT Law and thereby further extend the generalizability of D-CPT Law.Difficulty on fitting parameters We find that when using L-BFGS for fitting, the initialization of the fitting parameters is essential. Different parameter initializations can lead to significantly distinct results. Besides, we find that fitting algorithms also matter, in subsequent works, we hope to compare different fitting algorithms and design methods to reduce the dependency on the initialization of the fitting parameters.Extensive training costs of Scaling Law Although we attempt to ameliorate the training costs and enhance the fitting efficiency of Scaling Law, which are detailed in Section 4.4 and Appendix I.1, Scaling Law [12, 4, 1, 37] still remains prohibitively expensive for the majority. We hope that future research endeavors will seek to reduce the training costs of Scaling Law, thereby facilitating a wider usage and understanding of these laws within the community.LLMs, particularly those involving pre-training on massive Internet data, have been identified to carry significant societal impacts and inherent biases [56, 50, 15, 2] . For instance, large language models (LLMs) may generate content that carries political bias [42] . With the rise of downstream applications of LLMs, there is a growing effort to limit their output of offensive content, rendering LLMs more controllable and mitigating their potential negative impacts. We hope that our research to make the downstream applications of LLMs more controllable.Besides, LLMs have a significant environmental impact due to the substantial energy consumption required for their training and inference stages [46] . The extensive computational resources needed result in a high carbon footprint, thus raising concerns about the sustainability of such models in the context of global efforts to reduce greenhouse gas emissions. To this, our research can also partially reduce the consumption of GPU, thereby reducing the environmental impact of LLMs.To enhance the reader's experience, we have listed the symbols used in this paper in Table 9 .OpenAI Scaling Law Kaplan et al. [31] propose a parameterization as follows:EQUATIONwhere {N c , D c , α N , α D } are fitting parameters. The proportion of the domain-specific corpus within the training dataset. r gThe proportion of the general corpus within the training dataset. rThe proportion of the target corpus within the training dataset.The validation loss for the domain-specific corpus.The validation loss for the general corpus.The validation loss for the target corpus.The size of the model parameters.The number of training tokens for the model.The number of training tokens of the domain-specific corpus for the model.The number of training tokens of the general corpus for the model. Chinchilla Scaling Law Continuing along the trajectory established by OpenAI Scaling Law, Hoffmann et al. [27] propose a parameterization as follows:EQUATIONwhere {E,A,B,α,β} are fitting parameters. After fitting, the Allocation problem can be resolved by:EQUATIONwhere G = αA βBEQUATIONwhere N opt and D opt represent the optimal value of model size and dataset size, respectively.If we fix the mixture ratio in the training corpus, the D-CPT Law narrows down to the relationship involving only the model size N and dataset size D. Although previous works have proposed Scaling Law to describe the relationship between variables and performance, it has not been validated under our experimental setup. Here, we present the performance of OpenAI Scaling Law and Chinchilla Scaling Law in our experimental setup. For simplicity, we present results only in the code domain, with a 1:1 mixture ratio. The experimental results are shown in Figure 6 and Table 10 . We find that the Chinchilla Scaling Law is obviously better in our experimental setup. In this section, we start from the perspective of experimental observations to illustrate why we can arrive at the conclusions presented in Equation 5. Subsequently, we will briefly analyze the underlying reasons for these implicit trends. For convenience, we replicate here for clarity:EQUATIONIn mathematics, D-CPT Law has continuous second partial derivatives with respect to D and r. Based on Clairaut's Theorem, we have:EQUATIONwhich implies that the order of partial derivative does not affect the pattern presented in Equation 17. Based on the experiments, we have plotted the approximate values of dLg dD as a function of the general-corpus mixture ratio, as shown in Figure 10 . Since data points are discrete, we take the difference of every 5k steps as approximate values for dLg dD . We present the curves of dLg dD with respect to r g across multiple dataset sizes D. It is clear that dLg dD monotonically decreases with r g . Thus, based on the real experimental observations, we can infer Equation 17.In fact, there exists an explicit relationship between r and D, which can be represented as: Previous works have treated N and D as independent variables, not influencing each other. However, in our works, D and r are not able to be independent of each other, both from the perspective of experimental phenomena and the explicit relationship.EQUATIONEQUATIONEQUATIONEQUATIONAdditionally, we can explain Equation 5 by the principle of data efficiency. The term dL dD can be interpreted as the efficiency of each unit of data. With the increase of r, the proportion of valid data in each unit of data rises while the proportion of noisy data diminishes, resulting in enhanced efficiency of each unit of data. Given that lower loss signifies improved model performance, dL dD consequently displays a decreasing trend as r increases.In this section, we will first derive and demonstrate that D-CPT law satisfies the 4 requirements mentioned in Section 3.1. Subsequently, we will briefly describe the algorithm's setup and some minor improvements.• Adaptability: The newly introduced variable, the mixture ratio r, significantly differs from N and D, in that the range of values for N and D is greater than 0, whereas r is limited to the range [0,1]. This means that r should yield valid results at both 0 and 1, and it is crucial to ensure that values of r near 0 + or 1 -do not cause L to exhibit infinity. The trend of L with respect to r generally exhibits an initially rapid and subsequently slow pattern, a behavior that can be accurately modeled by a power function. However, positioning r in the denominator leads to an asymptotic increase to infinity as r approaches zero from the positive direction. To mitigate this issue, we have introduced a small positive bias ϵ to r, which is a fitting parameter. Typically, the value of ϵ lies near 0.1. This adjustment effectively prevents explosive growth near r = 0 + .• Explicit trends:EQUATIONEQUATIONEQUATIONIt is important to note that for the third equation, having ∂L ∂r < 0 requires certain constraints on the fitting parameters, specifically:EQUATIONIf these two constraints are satisfied, we have:EQUATIONEQUATIONEQUATIONIn our experimental setup, D has a minimum value, with the minimum value D min being approximately 0.1311B. Therefore, as long as we set C greater than C 0 and η greater than 1, the condition ∂L ∂r < 0 can be satisfied. This effectively imposes constraints on the fitting parameters. In our actual fitting process, we have modified the algorithm to seamlessly incorporate these constraints. Specific details will be mentioned when introducing the algorithm.• Implicit trends:EQUATION• Consistency:EQUATIONEQUATIONwhereEQUATIONEQUATIONwhich means that if r is a constant r 0 , then D-CPT Law can be transformed into a conventional Chinchilla Scaling Law. This suggests that under specific conditions where r assumes a fixed value, D-CPT Law aligns with the more universally recognized Chinchilla Scaling Law.Constrained L-BFGS We utilize L-BFGS to fit data points, with the objective being: where LSE is the log-sum-exp operator. Our improvements to the algorithm primarily focus on the third and the last item. Previously, we mentioned that C must be greater than C 0 to ensure the monotonic decrease of the D-CPT law with respect to r. Without any restrictions and fitting directly, it would sometimes lead to fitting results where C does not satisfy C ≥ C 0 . Therefore, to ensure that the fitted C must be greater than C 0 , we have indirectly imposed certain restrictions on the algorithm. We decomposed the original C into two parts: C 0 and C 1 , and due to the characteristics of the exponential function, the fitted result of C 1 = exp c 1 will be greater than 0. Consequently, C will be greater than C 0 , i.e.,EQUATIONEQUATIONwhereEQUATIONFollowing Chinchilla Scaling Law, we find local minima of the objective function, initiating our search on a predefined grid of starting points as follows: a ∈ {-1., -0., ..., 5.}, b ∈ {-1., 0., ..., 5.}, c ∈ {-1., 0., ..., 5.}, e ∈ {-1., 0.5, ..., 1.}, α ∈ {-0.5., 0., 0.5}, β ∈ {-0.5., 0., 0.5}, γ ∈ {-0.5., 0., 0.5}, η 1 ∈ {-0.5., 0., 0.5}, ϵ ∈ {0., 0.5}. Besides, we use δ = 10 -3 for the Huber loss.In Section 4.2, we have proposed five possible parameterizations of the D-CPT Law. After analyzing the experimental results, we have selected L 3 as the final parameterization of the D-CPT Law.Could there be other parameterizations better than L 3 ? We acknowledge that there may exist better parameterization than L 3 , but we believe that this is not crucial because our core objective is to find one that satisfies the 4 requirements outlined in Section 3.1, and we assume that when these 4 requirements are met, the parameterization is considered a good option of D-CPT Law. Specifically, regarding the choice of parameterization for the D-CPT Law, our research goal can be understood as finding a parameterization that matches the trend of real (N, D, r) data points and possesses certain mathematical properties. These trends and mathematical properties can be explicitly expressed as the 4 requirements in Section 3.1: Adaptability, Explicit trends, Implicit trends, and Consistency. Besides, in Section 4.2, we provide 5 parameterizations, only L 3 can meet all 4 requirements.Moreover, as there are also other parameterizations that can meet these 4 requirements, we provide another 2 parameterizations that satisfy these 4 requirements as follows:EQUATIONEQUATION)The fitting results for L 6 , L 7 , and L 3 on general-corpus and domain-corpus are as listed in Table 11 .We observe that when 4 requirements are met, the fitting results do not differ a lot. In conclusion, first, as there are many parameterizations that meet the 4 requirements, it is challenging to find the optimal parameterization. Second, the L 3 mentioned in the paper is relatively simple and meets the 4 requirements with good fitting results.Our main experiment requires approximately 150k hours of runtime on a single A100. The hyperparameters for the experiments are listed in Table 12 . First, we will standardize the notation: r g denotes the proportion of the general corpus, r d represents the proportion of the domain-corpus, L g signifies the general-corpus validation loss, L d indicates the domain-corpus validation loss, D represents the dataset size, and N denotes the model size. Therefore, we have:EQUATION)EQUATION)Note that we have the loss L monotonically decreasing with respect to r, therefore we have:EQUATIONEQUATIONWithin the context of D-CPT, we focus on L d , the domain-corpus validation loss. As the proportion of domain-corpus r d increases, L d is expected to decrease, indicating an improvement in domainspecific performance. Conversely, L g , the general-corpus validation loss, is expected to increase with the growing r d , suggesting a decline in general abilities. Therefore, we need to strike a balance between general and domain-specific abilities. To be specific, we will revisit the objective function of Usage 1:EQUATIONwhere L 0 g represents the initial general validation loss. Since L g monotonically increases with r d , a maximal r d will certainly be attained under the constraint. Concurrently, as L d monotonically decreases with r d , there must exist a unique r d that minimizes L d .For simplicity, we restate the objective function for usage 2:EQUATIONwhere D d denotes the domain-corpus dataset size, for L d in format of D-CPT Law, we have:EQUATION)EQUATIONEQUATIONBased on Appendix E.3, we have η > 1, therefore we have:EQUATION)EQUATION)EQUATION)The derivative dL d dr d is continuously differentiable and monotonically increasing. Given that dL d dr d is negative at r d = 0 and if dL d dr d is greater than 0 at r d = 1, then it follows that Equation 49 attains its minimum within the interval [0 < r d < 1]3 . Therefore, to ensure the existence of a valid minimum for the objective function 48, the following conditions must be satisfied:EQUATION)For convenience, we repeat the objective function of resource allocation as follows:EQUATIONFollowing [31] , we calculate compute budget C by:EQUATION)To validate its effectiveness in real-world scenarios, we take the law domain as an example and by fixing the mixture ratio at 1:1, fit D-CPT Law. We fix compute budget C = 5e 19 . Subsequently, based on the Efficient Frontier of Chinchilla [27] , we obtain:EQUATION)As the closest available model size to the optimal model size indicated by Qwen1.5 is 14B, we conducted our experiments using this 14B model. The experimental results are as shown in Table 13 . The experimental results reveal that the model sizes of 0.5B, 1.8B, and 4B suffer from data insufficiency. The optimal model size (14B) indeed exhibits the best performance. In practice, the data points we obtain are discrete, thus we can only utilize approximate values to express k 2 and k 3 . Specifically, we use the difference between the initial validation loss and the validation loss after 5k-steps4 continual pre-training,i.e.,EQUATION)Besides, we define k 3 as the difference in the decline values between the intervals of 0 to 5k steps and 5k to 10k steps,i.e.,EQUATIONLastly, we denote k 1 as the validation loss obtained after training for 1k steps.As each data point requires computational resources, we also investigate to improve the fitting efficiency with relatively low computational resources. In Table 14 , we have compared different sampling methods for data points and introduced a decay sampling method based on the exponential decay function to enhance fitting efficiency. Specifically, we focus on the fitting efficiency across dataset size while maintaining a constant model size. We have experimented with 4 different sampling methods, as follows:• M 1 : Dense sampling, evaluating validation loss every 1,000 steps.• M 2 : Sparse sampling, evaluating validation loss every 5,000 steps.• M 3 : Sectional sampling, evaluating every 4,000 steps in the initial 60% steps, every 8,000 steps in the remaining 40% steps. • M 4 : Sampling-based on an exponential decay function, detailed in Appendix I.2.Experimental results show that the performance of M 1 is relatively poor. In situations where resource consumption is comparatively high, no significant improvement in fitting performance is observed, thus indicating that the sampling density in our main experiments is excessively high. The overall performance of M 3 and M 4 surpasses that of M 2 because both M 3 and M 4 adopt a strategy of dense sampling in the initial phase and sparser sampling in the later phase. The trend of L with respect to D also shifts from rapid to slow changes, and sampling more points during phases of faster decline can considerably enhance fitting efficiency. However, the sampling setup of M 3 is of fixed paradigm and the sampling function follows a step-wise pattern. Of course, the overall performance of M 4 is slightly better than M 3 , it also offers a richer paradigm. In summary, sampling more points in the early phase of D can improve the overall fitting efficiency. In practical applications, it has the potential to save on evaluation costs and storage costs.In our main experiments, each experiment trains for 200,000 steps, with evaluations every 1,000 steps, resulting in a total of 200 data points. The decay function is represented as follows:EQUATIONFor M 4 in Section I.1, we set the decay parameter λ to 0.02 which yields 45 data points sampled. Figure 11 illustrates the decay function.Interestingly, we have found from the experiments that the trends between L and D are reversed when r approaches 0, in this section, we will explore it in depth and find that D-CPT Law between L and D has an inflection point r i to change its trend.We take the Law domain for example, the experimental results show that most of L g decreases strictly with r g when N and D are fixed, which is consistent with D-CPT Law. However, as r g approaches 0, the trend of L changes. Through analysis of Figure 12 , we observe that when r g is greater than 0.1, L g monotonically decreases with D, which aligns with the findings of D-CPT Law and previous works. However, when r g is less than or equal to 0.05, L g monotonically increases with D. This phenomenon is not limited to just one domain, we find that almost all domains exhibit this kind of behavior. We name the mixture ratio which changes the trends of L as inflection point r i . Accurately pinpointing r i is challenging. From an experimental perspective, it requires repeated experiments to approach r i progressively, which requires high experimental costs. Additionally, the exact value of r i changes across different domains, in our experimental setup, we find that r i for 6 domains all fall between 0 and 0.1.When the mixture ratio is less than the inflection point, L monotonically increases with D, which is inconsistent with the D-CPT Law. Therefore, the D-CPT Law predicts poorly when the mixture ratio is less than r i . Fortunately, predictions when the mixture ratio is less than r i are meaningless in the context of our works for two reasons: (1) In practical situations, we may not be particularly concerned with cases where the mixture ratio is very small, as the inflection point in most domains is less than 0.05. (2) When the mixture ratio is lower than r i , L monotonically increases with D, meaning that as the training cost increases, the performance of the model worsens. This is contrary to our initial objective, as we hope that after D-CPT, the domain-specific ability is enhanced. Thus, predictions when the mixture ratio is less than r i are considered meaningless.Of course, if we collect data points of small mixture ratios which means that the curves for these data points all show L increasing with D, then we can fit these data points. In that case, the fitting parameter B in D-CPT Law would be a negative value. If we know accurately the value of r i , we can express D-CPT Law in form of a piecewise function or represent it with a unified equation. However, the problem lies in precisely determining the value of r i . In future works, we hope to propose a low-cost method to accurately determine the value of r i . For example, we could conduct experiments with both small and large mixture ratios and fit them separately, then determine the value of r i based on the intersection of two resulting laws. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Note that we assume lower DLC means easier to learn for the corresponding domain.Training steps to number of training tokens is as follows: t steps = t • 64 • 2, 048 • 10 -9 .Solving dL d dr d = 0 is relatively complex, in this works, we use MATLAB to find the roots of this function.Note that we use Ltsteps to represent the validation loss at t steps. 


Decision: This paper proposes a scaling law called D-CPT Law for domain-specific continual pre-training of large language models, which predicts model performance based on model size, dataset size, and data mixture ratio between general and domain-specific corpora. It has broad applicability to: trade-off between general and domain-specific abilities; find the optimal mixture for limited domain-specific data; allocate compute resources. 

The paper extends the D-CPT Law to cross-domain settings with Cross-Domain D-CPT Law to predict performance on new domains with minimal additional training. The paper conducts experiments on 6 domains showing the effectiveness and generalizability of the proposed scaling laws.


**Strengths**
1) Addresses an important problem of optimizing data mixture for domain-specific LLM CPT training; 2) Proposes a novel scaling law with theoretical justification and empirical validation; 3) Demonstrates practical applications and generalizability; 4) Thorough experiments and analysis across multiple domains and settings.

**Weaknesses**  
1) Limited to relatively small model sizes (0.5B-4B parameters) focused only on Qwen model series; Leaves some open questions about theoretical optimality of the chosen parameterization.


The paper is technically solid with good contributions. The authors provided reasonable justifications for their experimental choices regarding small models in the rebuttal. The authors provided good reasoning for their choices, and demonstrated some results on 7B models. Overall, the strengths outweigh the weaknesses.

Based on the reviews and discussion, I recommend accepting this paper (poster). Given the importance of the problem and the broad applicability, I recommend a spotlight

Reviews: review: paper_summary: paper_summary: For continual pretraining of domain-specific large language models, an important question is how to choose the optimal mixture ratio between the general corpus and the downstream domain corpus. This paper proposes to fit the scaling law for domain-specific continual pre-training (D-CPT Law), using small-scale training costs to predict the general and downstream performance of arbitrary model sizes, data sizes, and data mixture ratios. They also extend it to Cross-Domain D-CPT Law to reduce training costs for the target domain. The paper shows empirical evidence for the effectiveness and applications of the Laws.
strength_weakness: strength_weakness: - The paper lacks a comparison between applying the proposed laws and performing a grid search over hyperparameters. How much computation is saved in various applications?
questions: questions: - Is it possible to fit a set of parameters in the scaling law, such that all independent variables can change their values?
- How many different $(N, D, r)$ are needed to fit the parameters in the D-CPT Law formula?
limitations: limitations: The authors addressed the limitations well in Appendix A.
paper_summary: paper_summary: This work explore the data mixture scaling law ( D-CPT Law) between the general corpus and the downstream domain-corpus for the continual pre-training of large language model.They also extend the fitted standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to  predict the D-CPT law of target domains.  D-CPT Law could be applied on three important scenarios: optimal mixture on the trade-off between general and domain-specific abilities, optimal  mixture for limited domain-specific data, and resource allocation setting, which is useful in practice.
strength_weakness: strength_weakness: This work focus on the data mixing scaling law for the continue pre-training phase of LLM. However, I think this work didn’t specifically have any design for the continue pre-training. The setting for the final scaling Function (Equation 6) have not mentioned any specific for continue pre-training. As the author mentioned , Equation 6  is supposed to transform into the Chinchilla  Scaling Law (which is for a LLM training from scratch) when r is fixed.
questions: questions: 1.Could you explain why this scaling law function is for continue pre-training?
2.In Equation 7, the newly involved parameter K, why is also in a similar power law pattern? The domain-specific learnable coefficient might be a much different dimension with the total number of model parameters and training data.
3.How many combination did you choose for the experiments? There are many between the data sizes from 0.1B to 26B. The way the portfolio is chosen has an impact on the on the average results.
limitations: limitations: As the authors have stated comprehensive in the limitation Section, I have no further comments for this part.
paper_summary: paper_summary: This work proposes a domain-specific scaling law to optimize mixture ratio between the general domain training corpus and the specific domain corpus. The authors considered several parameterizations and experimented on six different downstream domains such as code, math, and Medical. The result show that D-CPT Law has a high fitting accuracy across various model sizes, dataset sizes, and mixture ratios. Furthermore, the proposed cross-domain D-CPT Law reduced training costs while maintaining high performance in predicting domain-specific continual pre-training results.
strength_weakness: strength_weakness: 1. The models are limited to the Qwen series, which are relatively small in size. This makes it unclear whether similar results would be observed with larger models.
questions: questions: 1. The author stated, "we test the validation loss every 1,000 steps and the total training steps are 20k." How did you determine the step parameters, such as 1,000 and 20,000?

2. You defined five parameterizations and demonstrated that L3 is the best through experiments. Could there be other parameterizations better than L3? In other words, is it possible to theoretically prove that L3 is the optimal D-CPT Law?
limitations: limitations: The authors state several limitations and will consider addressing them in their future work.

Comments: title: Looking forward to Feedback as Discussion Deadline Approaches
Looking forward to Feedback as Discussion Deadline Approaches
Looking forward to feedback on the Responses
Summarization on the Responses
comment: Thank you for your responses. My questions are addressed and I have raised my rating.
Thank you for your responses. After reviewing the rebuttal, I will maintain the previous score.
Thanks for your feedback. We will carefully address your concerns in our new version.
Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. As the discussion phase will be closed on Aug 13 11:59 pm AoE, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.
Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. As the discussion phase will be closed on Aug 13 11:59 pm AoE, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.
I thoroughly looked at the author's response, and the score remained unchanged.
Dear Reviewers:

Hello! Since the discussion period has started, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address them. We appreciate your time and consideration!
Thanks for handling/reviewing our submitted manuscript: "D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models". We would like to thank the reviewers for their insightful and constructive comments and suggestions. By addressing each of the issues raised by the reviewers, we believe that the quality and clarity of our D-CPT Law can be improved a lot. The major responses are summarized as follows:

(1). We have carefully discussed the advantages and reasons for the choice of the formula of D-CPT Law (See Reviewer rpUb. Q1 and Reviewer sFHX. Q2).

(2). We have additionally compared the additional parameterizations of D-CPT Law and Cross-Domain D-CPT Law and discussed the advantages ours in detail (See Reviewer rpUb.Q1&Q2, Reviewer sFHX. Q2).

(3). We have discussed the advantages and reasons behind the choice of Qwen-1.5 series and the model sizes in detail (See Reviewer sFHX.Weakness).

(4). We have provided more clarification on the experimental settings of our main experiments (See Reviewer ycBX.Q2, Reviewer rpUb. Q3, and Reviewer sFHX.Q1).

(5). We have conducted additional experiments to compare the compute budget of grid search with the D-CPT Law, further demonstrating the efficiency of the D-CPT Law (See Reviewer ycBX.Weakness).

(6). We have provided more clarification on the reasons behind the design of D-CPT Law in the context of domain-specific continual pre-training (See Reviewer rpUb.Weakness).

(7). We have provided a more detailed analysis and clarification on the independent variables in the D-CPT Law (See Reviewer ycBX.Q1).

Again, we would like to sincerely thank you very much for these constructive comments and evaluation for our manuscript.