Title: D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models

Scaling Law

Continual Pre-training
Abstract: Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model’s fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1\% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.

D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models

Abstract

Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model's fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the generalcorpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training ( D-CPT Law ) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.

1 Introduction

Continual Pre-Training (CPT) is an essential part of training better Large Language models (LLMs). In this work, we mainly focus on Domain-specific CPT ( D-CPT ), which aims to enhance the fundamental understanding abilities of the specific downstream domains and has been widely used in existing works [49, 41, 30]. In practice, for D-CPT, we usually need to collect high-quality domain-corpus to enhance the downstream performance and general-corpus to mitigate catastrophic forgetting on the general abilities [13, 40, 54, 47, 32, 20, 53]. Therefore, how to determine the data composition or mixture ratio of the domain-corpus and general-corpus plays an important role in producing well-performed domain-specific LLMs. Besides, grid-searching on the mixture ratios requires heavy GPU consumption costs, and we cannot always obtain the optimal ratio under limited GPU usage. Recently, Scaling Law has been widely used for performance prediction [31, 27, 43, 26],

0.2

0.4

0.6

0.8

1.0

General-corpus mixture ratio

rg

2.700

2.725

2.750

2.775

2.800

2.825

2.850

2.875

General-corpus validation loss

Lg

General

D = 3.932B

D = 6.554B

D = 10.486B

D = 15.729B

fitting points

unseen points

predicted curve

0.2

0.4

0.6

0.8

1.0

Domain-corpus mixture ratio

rd

1.25

1.30

1.35

1.40

Domain-corpus validation loss

Ld

Domain-specific

D = 3.932B

D = 6.554B

D = 10.486B

D = 15.729B

fitting points

unseen points

predicted curve

which can be used to find the optimal dataset size and model size under the given GPU consumption costs. Therefore, for D-CPT, can we find the optimal mixture ratio in the training corpus using the Scaling Law to enhance the performance of domain-specific tasks?

To address the above question, in this work, we investigate the Scaling Law of D-CPT and propose the D-CPT Law to find the optimal mixture ratio with limited training costs for LLMs with different sizes. Specifically, inspired by the robust predictive ability of Scaling Law across various scales, we first perform experiments under diverse mixture ratios and several relatively small model and data scales. Following the Chinchilla Scaling Law, we then introduce the mixture ratio r into the D-CPT Law, where the parameterization is defined as follows:

where ϵ is used to guarantee the stability of L when r near zero. Based on Equation 1, for a model with model size N , dataset volume D and mixture ratio r , we can accurately predict the validation loss L . Note that when r denotes the domain-corpus mixture ratio r d , L means domain-corpus validation loss L d . Similarly, general-corpus validation loss L g also follows the law relationship with the general-corpus mixture ratio r g . To illustrate our D-CPT Law clearly, as shown in Figure 1, we take the code domain as an example and provide the fitting results on the general and domain-specific settings, where we validate the fitting accuracy on different mixture ratios under a model with different dataset sizes D . Our main contributions are summarized as follows:

(1). To show the effectiveness and generalizability of D-CPT Law, we perform extensive experiments using model sizes from 0.5B to 4B parameters, dataset sizes from 0.1B to 26B tokens, and mixture ratios from 0 to 1. The experiments show that the D-CPT law exhibits a high fitting accuracy with Huber loss [28] lower than 0.02 and R 2 [9] greater than 0.97. Besides, experiments on generalizability show that D-CPT Law not only inherits model size and dataset size generalizability following previous Scaling Law, but also precisely predicts performance for different mixture ratios.

(2). Despite the effectiveness in an in-domain setting, where we fit the D-CPT Law based on data points from one downstream domain, we also apply our D-CPT Law in the cross-domain setting, which denotes that we use the data points from multiple domains to predict the performance of unseen domains. Specifically, we first introduce the Domain-specific Learnable Coefficient (DLC) to denote the domain-specific parameter of each domain and integrate the DLC into the D-CPT Law. We name this new law as Cross-Domain D-CPT Law . In this way, if we can obtain the DLC of a new domain, we can easily derive the D-CPT Law for this new domain. In our experiments, we fit the Cross-Domain D-CPT Law using data points from 4 domains and apply the Cross-Domain

Small-scale CPT

Cross-Domain D-CPT Law

Calculate DLC

D-CPT Law

Usages

Trade-off

Limited domain-specific data

Resource Allocation

In-domain

Code

Medical

Math

Chemistry

Music

Law

Cross-domain

Physics

Data Preparation

·X

<>

+O

LIMITED

D-CPT Law to predict the remaining 2 domains. The results show that DLC can represent the specific information for each downstream domain well, enabling efficient and effective fitting performance for the cross-domain setting and significantly reducing training costs for new domains.

(3). To show the real-world usages of the D-CPT Law, we apply our D-CPT Law on three important scenarios: optimal mixture on the trade-off between general and domain-specific abilities, optimal mixture for limited domain-specific data, and resource allocation setting in Figure 2 (Details are provided in Section 4.3).

2 Background

Following previous work [43], we categorize the objectives of Scaling Law as Allocation and Return . Specifically, (1). Allocation : What is the optimal allocation of model size N and dataset size D given a fixed compute budget? (2). Return : What is the expected return on incremental resources?

The first objective on Allocation is as follows:

In Equation 2, given a fixed compute budget C , the objective is to find the optimal model size N and dataset size D that minimize the loss. The second objective on Return fundamentally depends on the generalizability of Scaling Law to accurately predict beyond the fitting data points.

where { E , A , B , α , β } are fitting parameters (See Appendix D for more details).

3 Methods

In Figure 2, D-CPT Law aims to investigate the behaviors of the Domain-specific Continual PreTraining scenario with respect to different mixture ratios, and the objective of the D-CPT Law is to analyze an appropriate parameterization of law that represents the relationship of validation loss L

with respect to model size N , dataset size D , and mixture ratio r . In this section, we first discuss the D-CPT Law in the in-domain setting (Section 3.1), where the fitting and testing data points are from the same domains. Then, we propose to adapt D-CPT Law to the cross-domain setting (Section 3.2), where the fitting and testing data points are from multiple domains, and introduce the Cross-Domain D-CPT Law, where a new term Domain-specific Learnable Coefficient (DLC) is used.

3.1 D-CPT Law

As the training data includes a mixture of general-corpus and domain-corpus, we introduce two mixture ratios (i.e., general-corpus mixture ratio r g and domain-corpus mixture ratio r d ). Correspondingly, we define two validation losses (i.e., general-corpus validation loss L g and domain-corpus validation loss L d ). Therefore, we can derive two D-CPT Laws (i.e., L g ( N,D,r g ) and L d ( N,D,r d ) ). For convenience, we directly use r and L ( N,D,r ) as default notations for D-CPT Law. Besides, in Appendix D, Chinchilla Scaling Law provides greater interpretability and clarity when compared to OpenAI Scaling Law. Thus, we choose Chinchilla Scaling Law as the foundational parameterization for D-CPT Law. In addition, since Scaling Law aims to fit data points, their parametric forms should be intrinsically related to the observed trends in the data points. Based on previous works and data trends with varying N , D and r , we have summarized 4 essential requirements for D-CPT Law:

Adaptability : D-CPT Law is valid for values of r between 0 and 1.

Explicit trends : Based on the results across varying values of N , D , and r , we observed the following explicit trends of data points:

The first two trends are consistent with the previous Chinchilla Scaling Law, and the third trend also has an intuitive explanation. A larger r indicates a higher proportion of valid corpus in the training corpus, leading to a lower L . Details are provided in Appendix E.1.

Implicit trends : We further discover inherent connections between r , D and L as follows:

For detailed explanations, please refer to the Appendix E.2.

Consistency : When r is fixed, the D-CPT Law is supposed to transform into the Chinchilla Scaling Law. In this way, D-CPT Law can inherit the excellent features of Chinchilla Scaling Law and address the issues on resource allocation discussed in Section 2.

To satisfy these requirements, we have compared multiple parameterizations in Section 4.2 and eventually, we propose the parameterization as follows:

In Equation 6, { E , A , B , C , α , β , γ , η , ϵ } are the fitting parameters and we use L-BFGS[34] to fit the D-CPT Law due to its suitability for large-scale optimizations. As shown in Section 4.2, the Equation 6 can accurately fit the trends of data points at any scale and demonstrates strong performance in both effectiveness and generalizability. Besides, it effectively meets the aforementioned 4 requirements. (Please see Appendix E.3 for more details on the mathematical derivation.)

3.2 Cross-Domain D-CPT Law

Apart from the in-domain setting for D-CPT Law, we also investigate the cross-domain setting and extend the D-CPT Law to the Cross-Domain D-CPT Law, which aims to reduce the training costs of the D-CPT Law significantly. Specifically, although the D-CPT Law collects data points using small LLMs, the GPU resource and time costs are still relatively substantial, which limits the applications of the Scaling Law. Therefore, in our Cross-Domain D-CPT Law, we first define the Domain-specific Learnable Coefficient (DLC) K for each domain, which measures the learnability

for a specific domain 1 (See Section 4.4 for more details.). Then, we incorporate the K into the D-CPT Law and obtain the Cross-Domain D-CPT Law, which is defined as follows:

In Equation 7, { E,A,B,C,F,α,β,η,γ,ϵ,µ } are fitting parameters. Thus, for an unseen domain, we only need to calculate the DLC with modest costs, which substantially increases the domain generalizability of D-CPT Law. Besides, Cross-Domain D-CPT Law has the following features:

Uniformity : Once we calculate the K value of an unseen domain, we can convert Crossdomain D-CPT Law into normal D-CPT Law as follows:

Therefore, the Cross-Domain D-CPT Law inherits all features of the D-CPT Law,

Monotonicity : K denotes the learnability of a specific domain, which aligns with the intuition that a more learnable domain yields lower validation loss. Meanwhile, the Crossdomain D-CPT Law confirms a monotonic decrease with respect to K , i.e.,

After confirming the parameterization of Cross-domain D-CPT Law, it is essential to identify a representation for K that accurately quantifies a domain's learnability. The representation of K is supposed to be accessible , distinct and robust . Specifically, first, 'Accessible' denotes that it is easy to obtain for an unseen domain with low costs. Second, 'Distinct' indicates that K values must exhibit significant variance across domains to ensure fitting accuracy and maintain clear distinctions between domains. Third, 'Robust' means that the representation of K enhances the effectiveness and generalization ability of Cross-domain D-CPT Law. In Section 4.4, we compare several variants of the representations of K , and the final representation is determined as follows:

where w 1 and w 2 are fitting parameters, k 1 represents the initial validation loss for an unseen domain, and k 2 denotes the rate of decline in validation loss.

4 Experiments

4.1 Experimental Setup

Data Setup To verify the effectiveness and generalizability of D-CPT Law and Cross-Domain DCPT Law, we have prepared the 6 different downstream domains, which include Code [39], Math [5], Law [24], Chemistry [10], Music [44] and Medical [33]. For general corpus, we use Dolma [48]. All the tokens of these training datasets are sufficient, so the experiments are not performed under a data-constrained setting. Besides, we build a high-quality and held-out validation set for each domain. (See Appendix F.1 for more details.)

Model Setup We use the Qwen-1.5 series due to its robust performance in both English and Chinese [6]. Furthermore, Qwen-1.5 has multiple open-sourced and well-performed pre-training base models. Specifically, we select Qwen-1.5-0.5B, Qwen-1.5-1.8B, and Qwen-1.5-4B as our base models to perform the continual pre-training for multiple downstream domains.

Training Setup We follow Chinchilla [27] to fix model sizes and vary the number of training tokens for data point collection. Specifically, we test the validation loss every 1,000 steps 2 and the total training steps are 200k. Then, we establish 9 mixture ratios between general-corpus and domaincorpus as follows: {0:10, 1:9, 2:8, 3.3:6.7, 5:5, 6.7:3.3, 8:2, 9:1, 10:0}. Note that all experiments are conducted with the same learning rate schedule (Hyperparameters can be found in Appendix F.2).

Metrics Following [44, 43, 27], we use validation loss as the performance indicator. To compare various parameterizations, we follow [28, 44] to utilize the R 2 and Huber loss as evaluation metrics. Specifically, first, the coefficient of determination (i.e., R 2 ) indicates the fitting quality and typically ranges from 0 to 1, where a higher value means better explanatory power of the regression model. Second, Huber loss combines the properties of mean squared error and mean absolute error, which is particularly useful for regression with outliers. Similarly, Huber loss also assesses the fit qualities of different parameterizations, where lower Huber loss shows better fitting performances.

4.2 D-CPT Law

In Section 3.1, an ideal parameterization should meet four requirements (i.e., adaptability, explicit trends, implicit trends, and consistency), and we define the following five parameterizations:

where { N , D , r } are variables and others are learned parameters fitted by L-BFGS algorithm[34] which is the same as Chinchilla Scaling Law.

Effectiveness As shown in Table 1, we present the performance of five different parameterizations. In effectiveness settings, we use entire data points for fitting with the aim of validating the effectiveness of various parameterizations. In Table 1, we observe that although L 5 has the fewest fitting parameters, its performance is significantly less impressive compared to the others. L 1 and L 4 , having relatively fewer fitting parameters, still fall short in performance compared to L 2 and L 3 . Moreover, L 1 fails to meet the requirements for implicit trends, while L 4 does not satisfy the explicit trends requirement. Finally, the results of L 2 and L 3 are comparable, but L 2 does not meet the requirements for consistency. Therefore, we choose L 3 for D-CPT Law. Moreover, Figure 3 shows the robust effectiveness of L 3 across varying dataset sizes, mixture ratios, model sizes, and domains.

Model Size Generalizability : Our main experiments focus on 3 model sizes: 0.5B, 1.8B, and 4B. We use 3-fold cross-validation to evaluate the model size generalizability of D-CPT Law, and the average results across domains are shown in Table 2. For example, we fit D-CPT Law with data points from 0.5B, and 1.8B and evaluate the Huber loss and R 2 for 4B. In Table 2, we observe that D-CPT Law can generalize well across model sizes and L 3 shows the best performance. Besides, we conduct experiments on the unseen 7B size (i.e., Qwen-1.5 7B), and observe that D-CPT Law can accurately predict the general-corpus validation loss with a general-corpus mixture ratio of 0.2 in Figure 4.

Dataset Size Generalizability : Our main experiments cover dataset sizes from 0.1B to 26B tokens, and we also utilize a 3-fold cross-validation approach. The data points are uniformly divided into three segments, with 2/3 used for fitting the model and the remaining 1/3 for testing. In Table 3, we report the average results across domains, and observe that L 3 shows notably enhanced dataset size generalizability.

Figure 4: L g with respect to D , domain-corpus is code, r g = 0 . 2 , N = 7 B .

2

3

4

5

6

D

2.570

2.575

2.580

2.585

2.590

2.595

2.600

2.605

2.610

Lg

Real Point

Predicted loss

Mixture ratio Generalizability : We apply the k-fold cross-validation method across various parameterizations. Specifically, we select 7 out of 9 mixture ratios for fitting and the remaining for testing, resulting in 36 experiments per domain. For simplicity, we show average results across domains in Table 4, and observe that that L 3 still shows significantly better performance on mixture ratio generalizability. Besides, in Figure 1, we observe that our D-CPT Law has well-performed generalizability on unseen mixture ratios.

4.3 Usages of D-CPT Law

Usage 1: Trade-off between general and domain-specific abilities For D-CPT, training data is a mixture of general and domain-specific data, where r g and r d denote the corresponding proportions, respectively. In D-CPT Law, when r g increases, the L g will decrease and L d will increase, indicating a trade-off between the general and domain-specific abilities of LLM. Fortunately, D-CPT Law can identify the optimal mixture ratio under any trade-off scenario. Specifically, we assume that an LLM with parameter size N 0 , it exhibits general-corpus validation loss of L 0 g and domain-corpus validation loss of L 0 d before continual pretraining. After mixing training data size of D 0 with a ratio r d of domain-specific data and 1 -r d of general data, we obtain general-corpus validation loss L g and domain-corpus validation loss L d after D-CPT. Then, we can identify the optimal mixture ratio while limiting the decline in the model's general abilities within a threshold T as follows:

5

10

15

20

25

2.75

2.80

2.85

2.90

2.95

3.00

Lg

N=0.5B

Real Loss

Predicted loss

5

10

15

20

25

2.75

2.80

2.85

2.90

2.95

3.00

Lg

N=1.8B

Real Loss

Predicted loss

5

10

15

20

25

1.75

1.80

1.85

1.90

1.95

2.00

Ld

N=0.5B

Real Loss

Predicted loss

5

10

15

20

25

1.75

1.80

1.85

1.90

1.95

2.00

Ld

N=1.8B

Real Loss

Predicted loss

D

D

D

D

where T is the threshold based on practical need. In Appendix G.1, given a fixed T , a unique optimal solution r d is obtained. To validate it in real scenarios, by applying D-CPT Law, we calculate the optimal domain-corpus mixture ratio r d = 0 . 924 given a dataset size D 0 = 10 B , model size N 0 = 1 . 8 B , T = 3% , domain-corpus is chemistry, and initial general validation loss value of L 0 g = 2 . 8602 . Table 5 presents the results of real general-corpus validation loss and domain-corpus validation loss with respect to different domain-corpus mixture ratios, we find that the real value exactly matches the predicted values( L g _ pred = 2 . 9458 and L d _ pred = 1 . 7284 ), domain-corpus mixture ratio exceeding 0.924 leads to a general validation loss that surpasses the 3% threshold of L 0 g .

Usage 2: Optimal mixture on limited domain-specific data Given that domain-corpus is typically limited relative to the abundance of the general corpus, we study how to determine the optimal mixture ratio when domain-corpus is limited and general-corpus is sufficient. Specifically, for an LLM with parameter N 0 and limited domain-corpus D 0 d , we aim to minimize the domain-corpus validation loss L d by selecting the optimal domain-corpus mixture ratio r d as follows:

In Equation 11, we can reach the minimum value within the 0 < r d < 1 discussed in Appendix G.2. To validate it in real scenarios, we conducted experiments within the music domain by setting the model parameters N 0 = 1 . 8 B and the domain-specific dataset size D d = 5 B . As we have data points at a large scale, we fit D-CPT Law using only data where D d < 2 B to align with the use case scenarios. After using the D-CPT Law, we find that the optimal domain-corpus mixture ratio is 0.732. Table 6 shows the results of real domain-corpus validation loss of the music domain. We observe that r d = 0 . 732 yields the lowest domain-corpus validation loss. Moreover, our predicted domain-corpus validation loss is 0.7328 when r d = 0 . 732 , which is close to the real value (0.7309).

Usage 3: Resource allocation D-CPT Law is consistent with Chinchilla Scaling Law under the fixed mixture ratio to address resource allocation. Specifically, how to find the optimal values of N and D given a fixed compute budget. Detailed results are shown in Appendix G.3.

4.4 Cross-Domain D-CPT Law

In Section 3.2, we have mentioned that the learnability of a specific domain is measured by DLC (i.e., K ). For Cross-Domain D-CPT Law, K must satisfy 3 core requirements: accessible, distinct, and robust. Based on these requirements, 4 different representations of K are defined as follows:

where { w 1 , w 2 , w 3 } are fitting parameters. In the approximate Taylor expansion for the validation loss function near the initial points, { k 1 , k 2 , k 3 } represent the first three coefficients. Due to the discrete nature of data points in practical scenarios, { k 1 , k 2 , k 3 } are approximated using the variants of validation loss. Specifically, k 1 denotes the precise value of the validation loss at the initial point, k 2 represents the difference in validation loss close to the initial points, and k 3 is an approximation of the second derivative of the validation loss near the initial points, details are provided in Appendix H. To compare these four representations of K , we have conducted experiments in both effectiveness and generalizability aspects.

Effectiveness We utilize data points from all 6 domains for fitting and then evaluate their performance using R 2 and Huber loss. In Table 7, we find that 4 representations of K yield comparable results in the general domain. However, K 1 and K 2 demonstrate a noticeable decline in domainspecific aspects. Although K 4 slightly outperforms K 3 in domain-specific aspects, it requires a larger number of fitting parameters. Therefore, considering the balance between fitting efficiency, fitting performance, and accessibility, We consider K 3 to be the optimal representation. To further visualize it, Figure 5 illustrates the predicted curves in comparison to the real curves under various settings.

5

10

15

20

25

2.80

2.85

2.90

2.95

3.00

3.05

Lg

N=0.5B

Real Loss

Predicted Loss

5

10

15

20

25

2.80

2.85

2.90

2.95

3.00

3.05

Lg

N=1.8B

Real Loss

Predicted Loss

5

10

15

20

25

0.70

0.72

0.74

0.76

0.78

0.80

0.82

Ld

N=0.5B

Real Loss

Predicted Loss

5

10

15

20

25

0.70

0.72

0.74

0.76

0.78

0.80

0.82

Ld

N=1.8B

Real Loss

Predicted Loss

D

D

D

D

Generalizability When K is identified, Cross-Domain D-CPT Law can be transformed into D-CPT Law, we believe that the former inherits the latter's generalizability in terms of model size, dataset size, and mixture ratio. In this part, we will specifically focus on discussing the domain generalizability of Cross-Domain D-CPT Law. To evaluate domain generalizability, we use data points from 4 out of 6 domains for fitting and assign the remaining 2 domains for testing. For simplicity, we only show the averaged results across 15 combinations in Table 8. Among these 4 representations of K , K 3 exhibits the superior performance, further proving its strength.

5 Related Works

Scaling Law Many studies [26, 31, 27, 7, 16, 58] show a power-law relationship between performance and the increases in both the number of parameters and the size of the training data [31, 27, 18], which are crucial for large language models (LLMs [45, 51, 29, 3, 55, 35, 59, 36, 60]) and provide a predictive structure for determining the most efficient setups for ex-

panded models using the insights gained from smaller models [17]. Moreover, the extension of scaling laws to autoregressive generative models widens their relevance to encompass tasks beyond text [18, 25, 17]. Recently, Muennighoff et al. [43] studied the Scaling Law of data-constrained settings by using the full pre-training dataset across multiple epochs. Ye et al. [57] investigate the data-mixing scaling law for the general LLMs to improve the pretraining efficiency.

Domain-specific Continual Pre-Training Domain-specific Continual Pre-Training aims to continually pre-train LLMs to adapt them to new domains [23, 11, 22, 30, 19]. For example, Gururangan et al. [23] introduces a growing mixture of expert architecture for domain-adaptive continual pretraining. Cossu et al. [13] show that continually pre-trained models (RoBERTa [38] and BERT [14]) are robust against catastrophic forgetting on downstream tasks. However, the above works only investigate small encoder-decoder models on limited tasks. Recently, Gupta et al. [21] study different warm-up strategies for continual pertraining for better results.

6 Conclusion

In this work, we have investigated the Scaling Law of Domain-specific Continual Pre-Training (D-CPT), which provides a significant step forward in the optimization of training LLMs for specific downstream domains. By developing and validating the D-CPT Law, we can easily predict the optimal mixture ratio of general and domain-specific corpora, greatly reducing the previously necessary but costly grid-searching efforts. Besides, we also adapt our D-CPT Law to the cross-domain setting and introduce the Cross-Domain D-CPT Law to further reduce the efforts of fitting the D-CPT Law of new domains. Moreover, we discuss the three practical usages of our D-CPT Law. Finally, we believe our D-CPT Law is an initial investigation into quantitative prediction methods for the domain-specific continual pre-training. With its increasing focus on data engineering, we hope our exploration facilitates further quantitative studies and theoretical analyses in this research area.

References

Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., Zettlemoyer, L., 2023. Scaling laws for generative mixed-modal language models, in: International Conference on Machine Learning, PMLR. pp. 265-279.

Agiza, A., Mostagir, M., Reda, S., 2024. Analyzing the impact of data selection and fine-tuning on economic and political biases in llms. arXiv preprint arXiv:2404.08699 .

AI, ., :, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y ., Wang, Y ., Cai, Y ., Gu, Z., Liu, Z., Dai, Z., 2024. Yi: Open foundation models by 01.ai. arXiv:2403.04652 .

Alabdulmohsin, I.M., Neyshabur, B., Zhai, X., 2022. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems 35, 22300-22312.

Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M.D., McAleer, S., Jiang, A.Q., Deng, J., Biderman, S., Welleck, S., 2023. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631 .

Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., et al., 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 .

Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al., 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 .

Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., Fan, Q., Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y ., Gui, T., Guo, A., Guo, Q., He, C., Hu, Y., Huang, T., Jiang, T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L., Ma, R., Ma, Z., Ning, W., Ouyang, L., Qiu, J., Qu, Y., Shang, F., Shao, Y., Song, D., Song, Z., Sui, Z., Sun, P., Sun, Y., Tang, H., Wang, B., Wang, G., Wang, J., Wang, J., Wang, R., Wang, Y., Wang, Z., Wei, X., Weng, Q., Wu, F., Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying, H., Yu, J., Yu, J., Zang, Y ., Zhang, C., Zhang, L., Zhang, P., Zhang, P., Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q., Zhao, X., Zhou, F., Zhou, Z., Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., Lin, D., 2024. Internlm2 technical report. arXiv:2403.17297 .

Carpenter, R., 1960. Principles and procedures of statistics, with special reference to the biological sciences. The Eugenics Review 52, 172.

Chemrxiv, . https://chemrxiv.org/engage/chemrxiv/public-dashboard .

Chen, W., Zhou, Y., Du, N., Huang, Y., Laudon, J., Chen, Z., Cui, C., 2023. Lifelong language pretraining with distribution-specialized experts, in: International Conference on Machine Learning, PMLR. pp. 5383-5395.

Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al., 2022. Unified scaling laws for routed language models, in: International conference on machine learning, PMLR. pp. 4057-4086.

Cossu, A., Tuytelaars, T., Carta, A., Passaro, L., Lomonaco, V., Bacciu, D., 2022. Continual pre-training mitigates forgetting in language and vision. arXiv preprint arXiv:2205.09357 .

Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .

Eloundou, T., Manning, S., Mishkin, P., Rock, D., 2023. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130 .

Frantar, E., Riquelme, C., Houlsby, N., Alistarh, D., Evci, U., 2023. Scaling laws for sparselyconnected foundation models. arXiv preprint arXiv:2309.08520 .

Gao, L., Schulman, J., Hilton, J., 2023. Scaling laws for reward model overoptimization, in: International Conference on Machine Learning, PMLR. pp. 10835-10866.

Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., Cherry, C., 2021. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740 .

Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al., 2024. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196 .

Guo, H., Yang, J., Liu, J., Yang, L., Chai, L., Bai, J., Peng, J., Hu, X., Chen, C., Zhang, D., et al., 2023. Owl: A large language model for it operations. arXiv preprint arXiv:2309.09298 .

Gupta, K., Thérien, B., Ibrahim, A., Richter, M.L., Anthony, Q., Belilovsky, E., Rish, I., Lesort, T., 2023. Continual pre-training of large language models: How to (re) warm your model? arXiv preprint arXiv:2308.04014 .

Gururangan, S., Lewis, M., Holtzman, A., Smith, N.A., Zettlemoyer, L., 2021. Demix layers: Disentangling domains for modular language modeling. arXiv preprint arXiv:2108.05036 .

Gururangan, S., Marasovi´ c, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., Smith, N.A., 2020. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964 .

Henderson, P., Krass, M., Zheng, L., Guha, N., Manning, C.D., Jurafsky, D., Ho, D., 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances in Neural Information Processing Systems 35, 29217-29234.

Hernandez, D., Kaplan, J., Henighan, T., McCandlish, S., 2021. Scaling laws for transfer. arXiv:2102.01293 .

Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M.M.A., Yang, Y., Zhou, Y., 2017. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 .

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., Clark, A., et al., 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 .

Huber, P.J., 1992. Robust estimation of a location parameter, in: Breakthroughs in statistics: Methodology and distribution. Springer, pp. 492-518.

Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al., 2023. Mistral 7b. arXiv preprint arXiv:2310.06825 .

Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.W., Wei, X., Arnold, A., Ren, X., 2021. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint arXiv:2110.08534 .

Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D., 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 .

Ke, Z., Shao, Y., Lin, H., Konishi, T., Kim, G., Liu, B., 2023. Continual pre-training of language models. arXiv preprint arXiv:2302.03241 .

Li, J., Wang, X., Wu, X., Zhang, Z., Xu, X., Fu, J., Tiwari, P., Wan, X., Wang, B., 2023. Huatuo-26m, a large-scale chinese medical qa dataset. arXiv:2305.01526 .

Liu, D.C., Nocedal, J., 1989. On the limited memory bfgs method for large scale optimization. Mathematical programming 45, 503-528.

Liu, J., Bai, Z., Zhang, Y ., Zhang, C., Zhang, Y ., Zhang, G., Wang, J., Que, H., Chen, Y ., Su, W., et al., 2024a. E2-llm: Efficient and extreme length extension of large language models. arXiv preprint arXiv:2401.06951 .

Liu, J., Zhang, C., Guo, J., Zhang, Y., Que, H., Deng, K., Bai, Z., Liu, J., Zhang, G., Wang, J., et al., 2024b. Ddk: Distilling domain knowledge for efficient large language models. arXiv preprint arXiv:2407.16154 .

Liu, X., Yan, H., Zhang, S., An, C., Qiu, X., Lin, D., 2023. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209 .

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V., 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 .

Lozhkov, A., Li, R., Allal, L.B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y ., et al., 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 .

Mehta, S.V., Patil, D., Chandar, S., Strubell, E., 2023. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research 24, 1-50.

Mendieta, M., Han, B., Shi, X., Zhu, Y., Chen, C., 2023. Towards geospatial foundation models via continual pretraining, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16806-16816.

Motoki, F., Pinho Neto, V., Rodrigues, V., 2024. More human than human: Measuring chatgpt political bias. Public Choice 198, 3-23.

Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., Raffel, C.A., 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems 36.

Qu, X., Bai, Y., Ma, Y., Zhou, Z., Lo, K.M., Liu, J., Yuan, R., Min, L., Liu, X., Zhang, T., et al., 2024. Mupt: A generative symbolic music pretrained transformer. arXiv preprint arXiv:2404.06393 .

Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al., 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 .

Rillig, M.C., Ågerstrand, M., Bi, M., Gould, K.A., Sauerland, U., 2023. Risks and benefits of large language models for the environment. Environmental Science & Technology 57, 3464-3466.

Rongali, S., Jagannatha, A., Rawat, B.P.S., Yu, H., 2020. Continual domain-tuning for pretrained language models. arXiv preprint arXiv:2004.02288 .

Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al., 2024. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159 .

Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., Wang, H., 2020. Ernie 2.0: A continual pre-training framework for language understanding, in: Proceedings of the AAAI conference on artificial intelligence, pp. 8968-8975.

Thakur, V., 2023. Unveiling gender bias in terms of profession across llms: Analyzing and addressing sociological implications. arXiv preprint arXiv:2307.09162 .

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G., 2023a. Llama: Open and efficient foundation language models. arXiv:2302.13971 .

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al., 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Wang, Z.M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Chen, W., Fu, J., Peng, J., 2023. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: 2310.00746 .

Wu, T., Caccia, M., Li, Z., Li, Y .F., Qi, G., Haffari, G., 2021. Pretrained language model in continual learning: A comparative study, in: International conference on learning representations.

Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., Yang, F., Deng, F., Wang, F., Liu, F., Ai, G., Dong, G., Zhao, H., Xu, H., Sun, H., Zhang, H., Liu, H., Ji, J., Xie, J., Dai, J., Fang, K., Su, L., Song, L., Liu, L., Ru, L., Ma, L., Wang, M., Liu, M., Lin, M., Nie, N., Guo, P., Sun, R., Zhang, T., Li, T., Li, T., Cheng, W., Chen, W., Zeng, X., Wang, X., Chen, X., Men, X., Yu, X., Pan, X., Shen, Y., Wang, Y., Li, Y., Jiang, Y., Gao, Y., Zhang, Y., Zhou, Z., Wu, Z., 2023. Baichuan 2: Open large-scale language models. arXiv:2309.10305 .

Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., Zhang, Y., 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing , 100211.

Ye, J., Liu, P., Sun, T., Zhou, Y ., Zhan, J., Qiu, X., 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952 .

Zhang, B., Liu, Z., Cherry, C., Firat, O., 2024a. When scaling meets llm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193 .

Zhang, G., Qu, S., Liu, J., Zhang, C., Lin, C., Yu, C.L., Pan, D., Cheng, E., Liu, J., Lin, Q., et al., 2024b. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405.19327 .

Zhou, Z., Liu, J., Dong, Z., Liu, J., Yang, C., Ouyang, W., Qiao, Y ., 2024. Emulated disalignment: Safety alignment for large language models may backfire!, in: Ku, L.W., Martins, A., Srikumar, V. (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Bangkok, Thailand. pp. 15810-15830. URL: https://aclanthology.org/2024.acl-long.842 , doi: 10.18653/v1/2024.acl-long.842 .

A Limitations and Future works

Experiments on more downstream domains In our work, main experiments cover six downstream domains [39, 5, 24, 10, 44, 33]. In future works, it is important to experiments on more downstream domains. We will attempt to conduct experiments on CPT in more domains and fit the D-CPT Law as well as the Cross-Domain D-CPT Law.

Experiments on more LLMs We primarily conduct experiments based on Qwen-1.5 and lack exploration of other Pre-Trained Base LLMs.

Multilingualism We lack research on multilingualism settings. Although the medical data is in fact in Chinese, the other data are in English. Moreover, the experimental results show that the fitting results in the medical domain are poor compared to others. We lack a detailed experimental analysis of different language settings. In future research, we hope to realize cross-linguistic and multi-linguistic D-CPT Law and thereby further extend the generalizability of D-CPT Law.

Difficulty on fitting parameters We find that when using L-BFGS for fitting, the initialization of the fitting parameters is essential. Different parameter initializations can lead to significantly distinct results. Besides, we find that fitting algorithms also matter, in subsequent works, we hope to compare different fitting algorithms and design methods to reduce the dependency on the initialization of the fitting parameters.

Extensive training costs of Scaling Law Although we attempt to ameliorate the training costs and enhance the fitting efficiency of Scaling Law, which are detailed in Section 4.4 and Appendix I.1, Scaling Law [12, 4, 1, 37] still remains prohibitively expensive for the majority. We hope that future research endeavors will seek to reduce the training costs of Scaling Law, thereby facilitating a wider usage and understanding of these laws within the community.

B Broader Impacts

LLMs, particularly those involving pre-training on massive Internet data, have been identified to carry significant societal impacts and inherent biases [56, 50, 15, 2]. For instance, large language models (LLMs) may generate content that carries political bias [42]. With the rise of downstream applications of LLMs, there is a growing effort to limit their output of offensive content, rendering LLMs more controllable and mitigating their potential negative impacts. We hope that our research to make the downstream applications of LLMs more controllable.

Besides, LLMs have a significant environmental impact due to the substantial energy consumption required for their training and inference stages [46]. The extensive computational resources needed result in a high carbon footprint, thus raising concerns about the sustainability of such models in the context of global efforts to reduce greenhouse gas emissions. To this, our research can also partially reduce the consumption of GPU, thereby reducing the environmental impact of LLMs.

C Symbols

To enhance the reader's experience, we have listed the symbols used in this paper in Table 9.

D D-CPT Law with a constant Mixture Ratio

OpenAI Scaling Law Kaplan et al.[31] propose a parameterization as follows:

where { N c , D c , α N , α D } are fitting parameters.

Chinchilla Scaling Law Continuing along the trajectory established by OpenAI Scaling Law, Hoffmann et al.[27] propose a parameterization as follows:

where { E , A , B , α , β } are fitting parameters. After fitting, the Allocation problem can be resolved by:

where N opt and D opt represent the optimal value of model size and dataset size, respectively.

If we fix the mixture ratio in the training corpus, the D-CPT Law narrows down to the relationship involving only the model size N and dataset size D . Although previous works have proposed Scaling Law to describe the relationship between variables and performance, it has not been validated under our experimental setup. Here, we present the performance of OpenAI Scaling Law and Chinchilla Scaling Law in our experimental setup. For simplicity, we present results only in the code domain, with a 1:1 mixture ratio. The experimental results are shown in Figure 6 and Table 10. We find that the Chinchilla Scaling Law is obviously better in our experimental setup.

E Supplementary Materials of D-CPT Law

E.1 Explicit trends

To provide a clear visualization of Equation 4, we have provided figures under 3 different settings, depicted in Figure 7, Figure 8, and Figure 9. All plots are trends of real data points.

0

5

10

15

20

25

2.6

2.7

2.8

2.9

3.0

3.1

Lg

N = 0.5B

Real loss

Prediction loss

0

5

10

15

20

25

N = 1.8B

Real loss

Prediction loss

0

5

10

15

20

25

N = 4.0B

Real loss

Prediction loss

D

D

D

Domain-specific validation loss on different model size

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Model size

N

(unit B)

1.35

1.40

1.45

1.50

1.55

1.60

1.65

1.70

Domain-specific validation loss

Ld

D=1.31B

D=3.93B

D=6.55B

D=9.18B

D=13.11B

D=19.66B

Domain-specific validation loss on different dataset size

0

5

10

15

20

25

Dataset size

D

(unit B)

1.3

1.4

1.5

1.6

1.7

1.8

1.9

Domain-specific validation loss

Ld

N=0.5B

N=1.8B

N=4.0B

Domain-specific validation loss on different domain-specific mixture ratio

0.2

0.4

0.6

0.8

1.0

Domain-specific mixture ratio

rd

1.35

1.40

1.45

1.50

1.55

1.60

Domain-specific validation loss

Ld

D=1.3B

D=3.8B

D=5.4B

D=9.2B

D=12.1B

D=19.7B

E.2 Implicit trends

In this section, we start from the perspective of experimental observations to illustrate why we can arrive at the conclusions presented in Equation 5. Subsequently, we will briefly analyze the underlying reasons for these implicit trends. For convenience, we replicate here for clarity:

In mathematics, D-CPT Law has continuous second partial derivatives with respect to D and r . Based on Clairaut's Theorem, we have:

which implies that the order of partial derivative does not affect the pattern presented in Equation 17. Based on the experiments, we have plotted the approximate values of dL g dD as a function of the general-corpus mixture ratio, as shown in Figure 10. Since data points are discrete, we take the difference of every 5k steps as approximate values for dL g dD . We present the curves of dL g dD with respect to r g across multiple dataset sizes D . It is clear that dL g dD monotonically decreases with r g . Thus, based on the real experimental observations, we can infer Equation 17.

In fact, there exists an explicit relationship between r and D , which can be represented as:

where D g represents the general-corpus dataset size and D d represents the domain-corpus dataset size. If we focus on the domain-corpus validation loss, then D g is noisy data to domain-corpus, and D d is valid data to domain-corpus. If we consider L d , the domain-corpus validation loss, to be solely dependent on D d and N , then D and r d influence each other and cannot be considered independent.

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

General mixture ratio

rg

0.040

0.035

0.030

0.025

0.020

0.015

0.010

0.005

Approximation of

Lg

D

Approximation of

Lg

D

with respect to

rg

D=1.3B

D=2.0B

D=2.6B

D=3.3B

D=4.6B

D=5.2B

D=6.6B

Previous works have treated N and D as independent variables, not influencing each other. However, in our works, D and r are not able to be independent of each other, both from the perspective of experimental phenomena and the explicit relationship.

Additionally, we can explain Equation 5 by the principle of data efficiency. The term dL dD can be interpreted as the efficiency of each unit of data. With the increase of r , the proportion of valid data in each unit of data rises while the proportion of noisy data diminishes, resulting in enhanced efficiency of each unit of data. Given that lower loss signifies improved model performance, dL dD consequently displays a decreasing trend as r increases.

E.3 Details behind D-CPT Law

In this section, we will first derive and demonstrate that D-CPT law satisfies the 4 requirements mentioned in Section 3.1. Subsequently, we will briefly describe the algorithm's setup and some minor improvements.

Adaptability : The newly introduced variable, the mixture ratio r , significantly differs from N and D , in that the range of values for N and D is greater than 0, whereas r is limited to the range [0,1]. This means that r should yield valid results at both 0 and 1, and it is crucial to ensure that values of r near 0 + or 1 -do not cause L to exhibit infinity. The trend of L with respect to r generally exhibits an initially rapid and subsequently slow pattern, a behavior that can be accurately modeled by a power function. However, positioning r in the denominator leads to an asymptotic increase to infinity as r approaches zero from the positive direction. To mitigate this issue, we have introduced a small positive bias ϵ to r , which is a fitting parameter. Typically, the value of ϵ lies near 0.1. This adjustment effectively prevents explosive growth near r = 0 + .

· Explicit trends :

It is important to note that for the third equation, having ∂L ∂r < 0 requires certain constraints on the fitting parameters, specifically:

If these two constraints are satisfied, we have:

In our experimental setup, D has a minimum value, with the minimum value D min being approximately 0.1311B. Therefore, as long as we set C greater than C 0 and η greater than 1, the condition ∂L ∂r < 0 can be satisfied. This effectively imposes constraints on the fitting parameters. In our actual fitting process, we have modified the algorithm to seamlessly incorporate these constraints. Specific details will be mentioned when introducing the algorithm.

· Implicit trends :

Consistency :

which means that if r is a constant r 0 , then D-CPT Law can be transformed into a conventional Chinchilla Scaling Law. This suggests that under specific conditions where r assumes a fixed value, D-CPT Law aligns with the more universally recognized Chinchilla Scaling Law.

Constrained L-BFGS We utilize L-BFGS to fit data points, with the objective being:

where LSE is the log-sum-exp operator. Our improvements to the algorithm primarily focus on the third and the last item. Previously, we mentioned that C must be greater than C 0 to ensure the monotonic decrease of the D-CPT law with respect to r . Without any restrictions and fitting directly, it would sometimes lead to fitting results where C does not satisfy C ≥ C 0 . Therefore, to ensure that the fitted C must be greater than C 0 , we have indirectly imposed certain restrictions on the algorithm. We decomposed the original C into two parts: C 0 and C 1 , and due to the characteristics of the exponential function, the fitted result of C 1 = exp c 1 will be greater than 0. Consequently, C will be greater than C 0 , i.e.,

Following Chinchilla Scaling Law, we find local minima of the objective function, initiating our search on a predefined grid of starting points as follows: a ∈ {-1 ., -0 ., ..., 5 . } , b ∈ {-1 ., 0 ., ..., 5 . } , c ∈ {-1 ., 0 ., ..., 5 . } , e ∈ {-1 ., 0 . 5 , ..., 1 . } , α ∈ {-0 . 5 ., 0 ., 0 . 5 } , β ∈ {-0 . 5 ., 0 ., 0 . 5 } , γ ∈ {-0 . 5 ., 0 ., 0 . 5 } , η 1 ∈ {-0 . 5 ., 0 ., 0 . 5 } , ϵ ∈ { 0 ., 0 . 5 } . Besides, we use δ = 10 -3 for the Huber loss.

E.4 The choices of D-CPT Law Parameterizations

In Section 4.2, we have proposed five possible parameterizations of the D-CPT Law. After analyzing the experimental results, we have selected L 3 as the final parameterization of the D-CPT Law. Could there be other parameterizations better than L 3 ? We acknowledge that there may exist better parameterization than L 3 , but we believe that this is not crucial because our core objective is to find one that satisfies the 4 requirements outlined in Section 3.1, and we assume that when these 4 requirements are met, the parameterization is considered a good option of D-CPT Law. Specifically, regarding the choice of parameterization for the D-CPT Law, our research goal can be understood as finding a parameterization that matches the trend of real (N, D, r) data points and possesses certain mathematical properties. These trends and mathematical properties can be explicitly expressed as the 4 requirements in Section 3.1: Adaptability, Explicit trends, Implicit trends, and Consistency. Besides, in Section 4.2, we provide 5 parameterizations, only L 3 can meet all 4 requirements.

Moreover, as there are also other parameterizations that can meet these 4 requirements, we provide another 2 parameterizations that satisfy these 4 requirements as follows:

The fitting results for L 6 , L 7 , and L 3 on general-corpus and domain-corpus are as listed in Table 11. We observe that when 4 requirements are met, the fitting results do not differ a lot. In conclusion, first, as there are many parameterizations that meet the 4 requirements, it is challenging to find the optimal parameterization. Second, the L 3 mentioned in the paper is relatively simple and meets the 4 requirements with good fitting results.

E.5 Compute resources

Our main experiment requires approximately 150k hours of runtime on a single A100.

F Supplementary Materials of Experiments

F.1 Validation datasets collection

Specifically, for each domain, we first randomly select 5,000 samples from the original dataset, and then we use four open-sourced LLMs (i.e., Qwen-1.5 72B [6], Yi-34B [3], LLaMA2-13B [52], InternLM2-20B [8]) to compute the perplexity (PPL) and sort these samples based on the PPL values. Specifically, a lower PPL value denotes higher fluency of the data indicated by the model. If a sample ranks in the bottom 10% under all four open-source LLMs, we consider this sample to be noisy and exclude it. Subsequently, we randomly sample 1,000 samples from the filtered sample pool to serve as the validation set for each domain. In this way, we can obtain a high-quality validation set for all domains.

F.2 Hyperparameters

The hyperparameters for the experiments are listed in Table 12.

G Mathematical Derivation behind use case

G.1 Usage 1

First, we will standardize the notation: r g denotes the proportion of the general corpus, r d represents the proportion of the domain-corpus, L g signifies the general-corpus validation loss, L d indicates the domain-corpus validation loss, D represents the dataset size, and N denotes the model size. Therefore, we have:

Note that we have the loss L monotonically decreasing with respect to r , therefore we have:

Within the context of D-CPT, we focus on L d , the domain-corpus validation loss. As the proportion of domain-corpus r d increases, L d is expected to decrease, indicating an improvement in domainspecific performance. Conversely, L g , the general-corpus validation loss, is expected to increase with the growing r d , suggesting a decline in general abilities. Therefore, we need to strike a balance between general and domain-specific abilities. To be specific, we will revisit the objective function of Usage 1:

where L 0 g represents the initial general validation loss. Since L g monotonically increases with r d , a maximal r d will certainly be attained under the constraint. Concurrently, as L d monotonically decreases with r d , there must exist a unique r d that minimizes L d .

G.2 Usage 2

For simplicity, we restate the objective function for usage 2:

where D d denotes the domain-corpus dataset size, for L d in format of D-CPT Law, we have:

Based on Appendix E.3, we have η > 1 , therefore we have:

The derivative dL d dr d is continuously differentiable and monotonically increasing. Given that dL d dr d is negative at r d = 0 and if dL d dr d is greater than 0 at r d = 1 , then it follows that Equation 49 attains its minimum within the interval [0 < r d < 1] 3 . Therefore, to ensure the existence of a valid minimum for the objective function 48, the following conditions must be satisfied:

G.3 Usage 3

For convenience, we repeat the objective function of resource allocation as follows:

Following [31], we calculate compute budget C by:

To validate its effectiveness in real-world scenarios, we take the law domain as an example and by fixing the mixture ratio at 1:1, fit D-CPT Law. We fix compute budget C = 5 e 19 . Subsequently, based on the Efficient Frontier of Chinchilla[27], we obtain:

As the closest available model size to the optimal model size indicated by Qwen1.5 is 14B, we conducted our experiments using this 14B model. The experimental results are as shown in Table 13. The experimental results reveal that the model sizes of 0.5B, 1.8B, and 4B suffer from data insufficiency. The optimal model size (14B) indeed exhibits the best performance.

H Details behind Domain-specific Learnable Coefficient

In practice, the data points we obtain are discrete, thus we can only utilize approximate values to express k 2 and k 3 . Specifically, we use the difference between the initial validation loss and the validation loss after 5k-steps 4 continual pre-training,i.e.,

Besides, we define k 3 as the difference in the decline values between the intervals of 0 to 5k steps and 5k to 10k steps,i.e.,

Lastly, we denote k 1 as the validation loss obtained after training for 1k steps.

I Further Analysis

I.1 Fitting Efficiency

As each data point requires computational resources, we also investigate to improve the fitting efficiency with relatively low computational resources. In Table 14, we have compared different sampling methods for data points and introduced a decay sampling method based on the exponential decay function to enhance fitting efficiency. Specifically, we focus on the fitting efficiency across dataset size while maintaining a constant model size.

* For Resource consumption, we focus on evaluation costs and storage costs.

We have experimented with 4 different sampling methods, as follows:

M 1 : Dense sampling, evaluating validation loss every 1,000 steps.

M 2 : Sparse sampling, evaluating validation loss every 5,000 steps.

M 3 : Sectional sampling, evaluating every 4,000 steps in the initial 60% steps, every 8,000 steps in the remaining 40% steps.

M 4 : Sampling-based on an exponential decay function, detailed in Appendix I.2.

Experimental results show that the performance of M 1 is relatively poor. In situations where resource consumption is comparatively high, no significant improvement in fitting performance is observed, thus indicating that the sampling density in our main experiments is excessively high. The overall performance of M 3 and M 4 surpasses that of M 2 because both M 3 and M 4 adopt a strategy of dense sampling in the initial phase and sparser sampling in the later phase. The trend of L with respect to D also shifts from rapid to slow changes, and sampling more points during phases of faster decline can considerably enhance fitting efficiency. However, the sampling setup of M 3 is of fixed paradigm and the sampling function follows a step-wise pattern. Of course, the overall performance of M 4 is slightly better than M 3 , it also offers a richer paradigm. In summary, sampling more points in the early phase of D can improve the overall fitting efficiency. In practical applications, it has the potential to save on evaluation costs and storage costs.

I.2 Decay function

In our main experiments, each experiment trains for 200,000 steps, with evaluations every 1,000 steps, resulting in a total of 200 data points. The decay function is represented as follows:

For M 4 in Section I.1, we set the decay parameter λ to 0.02 which yields 45 data points sampled. Figure 11 illustrates the decay function.

I.3 Analysis of near-zero

Interestingly, we have found from the experiments that the trends between L and D are reversed when r approaches 0, in this section, we will explore it in depth and find that D-CPT Law between L and D has an inflection point r i to change its trend.

We take the Law domain for example, the experimental results show that most of L g decreases strictly with r g when N and D are fixed, which is consistent with D-CPT Law. However, as r g approaches

0

25

50

75

100

125

150

175

200

Steps(10 3 )

0.0

0.2

0.4

0.6

0.8

1.0

Sampling Probability

Exponential Decay Function

0

5

10

15

20

25

Dataset size D

2.7

2.8

2.9

3.0

3.1

3.2

3.3

3.4

General Validation Loss

Lg

General validation loss with respect to dataset size on different mixture ratio

rg

=0.00

rg

=0.05

rg

=0.10

rg

=0.20

rg

=0.33

rg

=0.50

rg

=0.67

rg

=0.80

rg

=0.90

rg

=1.00

0, the trend of L changes. Through analysis of Figure 12, we observe that when r g is greater than 0.1, L g monotonically decreases with D , which aligns with the findings of D-CPT Law and previous works. However, when r g is less than or equal to 0.05, L g monotonically increases with D . This phenomenon is not limited to just one domain, we find that almost all domains exhibit this kind of behavior. We name the mixture ratio which changes the trends of L as inflection point r i . Accurately pinpointing r i is challenging. From an experimental perspective, it requires repeated experiments to approach r i progressively, which requires high experimental costs. Additionally, the exact value of r i changes across different domains, in our experimental setup, we find that r i for 6 domains all fall between 0 and 0.1.

When the mixture ratio is less than the inflection point, L monotonically increases with D , which is inconsistent with the D-CPT Law. Therefore, the D-CPT Law predicts poorly when the mixture ratio is less than r i . Fortunately, predictions when the mixture ratio is less than r i are meaningless in the context of our works for two reasons: (1) In practical situations, we may not be particularly concerned with cases where the mixture ratio is very small, as the inflection point in most domains is less than 0.05. (2) When the mixture ratio is lower than r i , L monotonically increases with D , meaning that as the training cost increases, the performance of the model worsens. This is contrary to our initial objective, as we hope that after D-CPT, the domain-specific ability is enhanced. Thus, predictions when the mixture ratio is less than r i are considered meaningless.

Of course, if we collect data points of small mixture ratios which means that the curves for these data points all show L increasing with D , then we can fit these data points. In that case, the fitting parameter B in D-CPT Law would be a negative value. If we know accurately the value of r i , we can express D-CPT Law in form of a piecewise function or represent it with a unified equation. However, the problem lies in precisely determining the value of r i . In future works, we hope to propose a low-cost method to accurately determine the value of r i . For example, we could conduct experiments with both small and large mixture ratios and fit them separately, then determine the value of r i based on the intersection of two resulting laws.

J Supplementary Tables

K Supplementary Figures

K.1 Effectiveness of D-CPT Law

5

10

15

20

25

Numbers of training tokens D

2.6

2.7

2.8

2.9

3.0

General Validation Loss

Lg

N=0.5B

Real Loss

Predicted loss

5

10

15

20

25

Numbers of training tokens D

N=1.8B

Real Loss

Predicted loss

5

10

15

20

25

Numbers of training tokens D

N=4.0B

Real Loss

Predicted loss

5

10

15

20

25

Numbers of training tokens D

1.65

1.70

1.75

1.80

1.85

1.90

1.95

2.00

Domain-specific Validation Loss

Ld

N=0.5B

Real Loss

Predicted loss

5

10

15

20

25

Numbers of training tokens D

N=1.8B

Real Loss

Predicted loss

5

10

15

20

25

Numbers of training tokens D

N=4.0B

Real Loss

Predicted loss

K.2 Dataset Size Generalizability of the D-CPT Law

5

10

15

20

25

Numbers of training tokens D

2.6

2.7

2.8

2.9

3.0

General Validation Loss

Lg

N=0.5B

Real Loss

Predicted loss

5

10

15

20

25

Numbers of training tokens D

N=1.8B

Real Loss

Predicted loss

5

10

15

20

25

Numbers of training tokens D

N=4.0B

Real Loss

Predicted loss

K.3 Domain Generalizability of the Cross-Domain D-CPT Law

5

10

15

20

25

2.6

2.7

2.8

2.9

3.0

Lg

N=0.5B

Real Loss

Predicted Loss

5

10

15

20

25

Lg

N=1.8B

Real Loss

Predicted Loss

5

10

15

20

25

Lg

N=4.0B

Real Loss

Predicted Loss

D

D

D

5

10

15

20

25

2.6

2.7

2.8

2.9

3.0

Lg

N=0.5B

Real Loss

Predicted Loss

5

10

15

20

25

Lg

N=1.8B

Real Loss

Predicted Loss

5

10

15

20

25

Lg

N=4.0B

Real Loss

Predicted Loss

D

D

D

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated our main contributions in our Introduction and Abstract. The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of our work in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Certainly, we have corresponding detailed derivations for all theoretical results, such as Appendix E.3 and Appendix G.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The main experimental results are reproducible. We have mentioned our experimental setup in Section 4 and Appendix F.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code belongs to the company's intellectual property, but the data can be downloaded from open-source repositories. For example, the Dolma dataset can be downloaded from https://github.com/allenai/dolma . In the main text, we cited all the data sources.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details in Section 4 and Appendix F.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experimental results have been validated through cross-validation and repeated experiments. The effectiveness and significance of the method have been confirmed by multiple domains and multiple k-fold cross-validation experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have discussed our compute resources in Appendix E.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research respects the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts of our works in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have respected all the licenses and terms of use for CPT data and the Qwen model.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our works do not involve crowdsourcing and research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our works do not involve crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This work proposes a domain-specific scaling law to optimize mixture ratio between the general domain training corpus and the specific domain corpus. The authors considered several parameterizations and experimented on six different downstream domains such as code, math, and Medical. The result show that D-CPT Law has a high fitting accuracy across various model sizes, dataset sizes, and mixture ratios. Furthermore, the proposed cross-domain D-CPT Law reduced training costs while maintaining high performance in predicting domain-specific continual pre-training results.

STRENGTHS:
1. The paper proposes a domain-specific scaling that can more efficiently determine thec optimal mixture ratio between general and domain-specific corpora. The experiments demonstrate its effectiveness.

2. The paper presents three applications of D-CPT Law that may be of interest and benefit to the community.

WEAKNESSES:
1. The models are limited to the Qwen series, which are relatively small in size. This makes it unclear whether similar results would be observed with larger models.

QUESTIONS:
1. The author stated, "we test the validation loss every 1,000 steps and the total training steps are 20k." How did you determine the step parameters, such as 1,000 and 20,000?

2. You defined five parameterizations and demonstrated that L3 is the best through experiments. Could there be other parameterizations better than L3? In other words, is it possible to theoretically prove that L3 is the optimal D-CPT Law?

LIMITATIONS:
The authors state several limitations and will consider addressing them in their future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explore the data mixture scaling law ( D-CPT Law) between the general corpus and the downstream domain-corpus for the continual pre-training of large language model.They also extend the fitted standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to  predict the D-CPT law of target domains.  D-CPT Law could be applied on three important scenarios: optimal mixture on the trade-off between general and domain-specific abilities, optimal  mixture for limited domain-specific data, and resource allocation setting, which is useful in practice.\n\nSTRENGTHS:\n1.This work experimented and analyzed on three scenarios including optimal mixture on the trade-off between general and domain-specific abilities, optimal  mixture for limited domain-specific data, and resource allocation setting, which is useful in practice\n2. D-CPT Law considered both the in-domain setting and the cross-domain setting with a new involved parameter K, which provide a reference for the unseen domain data mixing.\n3. The analysis on the selected fitting function is comprehensive and make sense.\n\nWEAKNESSES:\nThis work focus on the data mixing scaling law for the continue pre-training phase of LLM. However, I think this work didn’t specifically have any design for the continue pre-training. The setting for the final scaling Function (Equation 6) have not mentioned any specific for continue pre-training. As the author mentioned , Equation 6  is supposed to transform into the Chinchilla  Scaling Law (which is for a LLM training from scratch) when r is fixed.\n\nQUESTIONS:\n1.Could you explain why this scaling law function is for continue pre-training?\n2.In Equation 7, the newly involved parameter K, why is also in a similar power law pattern? The domain-specific learnable coefficient might be a much different dimension with the total number of model parameters and training data.\n3.How many combination did you choose for the experiments? There are many between the data sizes from 0.1B to 26B. The way the portfolio is chosen has an impact on the on the average results.\n\nLIMITATIONS:\nAs the authors have stated comprehensive in the limitation Section, I have no further comments for this part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
For continual pretraining of domain-specific large language models, an important question is how to choose the optimal mixture ratio between the general corpus and the downstream domain corpus. This paper proposes to fit the scaling law for domain-specific continual pre-training (D-CPT Law), using small-scale training costs to predict the general and downstream performance of arbitrary model sizes, data sizes, and data mixture ratios. They also extend it to Cross-Domain D-CPT Law to reduce training costs for the target domain. The paper shows empirical evidence for the effectiveness and applications of the Laws.

STRENGTHS:
- Discovered that model performance can be predicted given the data mixture ratio between general and domain-specific corpus and a few other factors in domain-specific language model training.
- Proposed essential requirements of the scaling law formula, compared multiple parametrization of the scaling law, and proposed a parametrization that extends the Chinchilla Scaling Law.
- Extended the law to cross-domain scenarios and described its features.
- Empirically verified the effectiveness, model size generalizability, and dataset size generalizability of the proposed methods.

WEAKNESSES:
- The paper lacks a comparison between applying the proposed laws and performing a grid search over hyperparameters. How much computation is saved in various applications?

QUESTIONS:
- Is it possible to fit a set of parameters in the scaling law, such that all independent variables can change their values?
- How many different $(N, D, r)$ are needed to fit the parameters in the D-CPT Law formula?

LIMITATIONS:
The authors addressed the limitations well in Appendix A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14