Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment
Authors: Jinhao Jiang
Junyi Li
Xin Zhao
Yang Song
Tao Zhang
Ji-Rong Wen
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
Continual Pre-training
knowledge distillation
Primary area: foundation or frontier models, including LLMs

Abstract: Adapting large language models (LLMs) to specialized domains typically requires domain-specific corpora for continual pre-training to facilitate knowledge memorization and related instructions for fine-tuning to apply this knowledge.
However, this method may lead to inefficient knowledge memorization due to a lack of awareness of knowledge utilization during the continual pre-training and demands LLMs to simultaneously learn knowledge utilization and format alignment with divergent training objectives during the fine-tuning.
To enhance the domain adaptation of LLMs, we revise this process and propose a new domain adaptation framework including domain knowledge learning and general format alignment, called \emph{Mix-CPT}. Specifically, we first conduct a knowledge mixture continual pre-training that concurrently focuses on knowledge memorization and utilization. To avoid catastrophic forgetting, we further propose a logit swap self-distillation constraint. By leveraging the knowledge and capabilities acquired during continual pre-training, we then efficiently perform instruction tuning and alignment with a few general training samples to achieve format alignment.
Extensive experiments show that our proposed \emph{Mix-CPT} framework can simultaneously improve the task-solving capabilities of LLMs on the target and general domains.

Published as a conference paper at ICLR 2025
MIX-CPT: A DOMAIN ADAPTATION FRAMEWORK VIA DECOUPLING KNOWLEDGE LEARNING AND FORMAT ALIGNMENT
Jinhao Jiang", Junyi Li"", Wayne Xin Zhao Yang Song Tao Zhang' and Ji-Rong Wen "aaoling School of Artificial Intelligence, Renmin University of China. *Department of Computer Science, National University of Singapore. Nanbeige Lab, BOSS Zhipin.
Laangiinhoo@rerr edu. en, batmanfly@gmail.. com
ABSTRACT
Adapting large language models (LLMs) to specialized domains typically requires domain-specific corpora for continual pre-training to facilitate knowledge mem- orization and related instructions for fine-tuning to apply this knowledge. How- ever, this method may lead to inefficient knowledge memorization due to lack of awareness of knowledge utilization during the continual pre-training and de- mands LLMS simultaneously learn knowledge utilization and format alignment with divergent training objectives during the fine-tuning. To enhance the domain adaptation of LLMs, we revise this process and propose new domain adaptation framework including domain knowledge learning and general format alignment, called Mix-CPI Specifically, we first conduct a knowledge mixture continual pre- training that concurrently focuses on knowledge memorization and utilization. To avoid catastrophic forgetting, we further propose logit swap self-distillation con- straint. By leveraging the knowledge and capabilities acquired during continual pre-training, we then efficiently perform instruction tuning and alignment with few general training samples to achieve format alignment. Extensive experi- ments show that our proposed Mix-CPT framework can simultancously improve the task-solving capabilities of LLMs the target and general domains.
INTRODUCTION
Large Language Models (LLMs) Zhao et al. 2023) have revolutionized the field of natural language processing (NLP) (OpenAI 2023), showing exceptional capabilities such as instruction following and complex reasoning (Wei et al. 2022). However, due to their limited exposure to relevant data, such general LLMs still considerably lag behind in specific domains requiring professional knowl- edge. This situation has necessitated the effective adaptation of general purpose LLMs to specific domains (e.g., mathematics and code), called domain adaptation of L.LMs (Guo Yu 2022).
In essence, tailoring general L.LSs to specific domains requires adaptation in two main aspects, namely knowledge learning (acquiring and leveraging the necessary domain knowledge) and format alignment (responding the user in expected output form) (Jiang et al. 2024 Zhou et al. 2023 Hu et al. 2024b). Specially, knowledge learning can be further fulfilled via knowledge memoriza- tion and utilization. In practice, domain adaptation of LLMs typically involves three consecutive stages (Rozière et al. 2023 Azerbayev et al. 2023), ), i.e., pre-training, instruction tuning, and align- ment, where the first stage primarily aimed knowledge memorization and the other two stages are mainly focused on knowledge utilization and format alignment. However, at the pre-training stage, knowledge memorization based on raw domain-specific corpora would be somehow ineffi- cient without eliciting the acquired knowledge according to task goals (Jiang et al.) (024)). Despite that some studies incorporate instruction data for pre-training, they often rely on proprictary models to synthesize high-quality instructions at scale (Cheng et al. 2024a Wang et al. 2024), which may
Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2025
D CPT
DSPT
DDPO
Albert Einstein ...
[EOS]
[ST] You are math tutor [UT] Please this problem [AT]
[ST] You are ...[UT]] Deseribe [AT]
Nobel Prizes are ... [FOS]
Summer is scason of sun, Run..
Jame decides run sprints
Computer is [BOS]
times a week. He runs 60
X Summer is scason of cadiess...
Domain Corpora
Instruction Data
Alignment Data
Knowledge Memorization
Knowledge Utilization & Format Alignment
DMIX
D'SFT
DDPO
Albert Einstein was... ... (EOS) Nobel Prizes [EOS] Computer is [EOS]]
[ST] You (UTT] Plezssc
[ST) Von (UT] Piazee
Please solive this problem. Jame docides run 3 sprints times week (FOSS
These pictures come from crtiel....
You may nct
Deseribe what summer means, Summer is a season of sun. fun, and ... [EOS]
This is not...
Knowledge Memorization & Utilization
Format A Alignment
Figure 1: Comparison of traditional domain adaptation approaches (top) and our proposed Mix-CPT paradigm (bottom). "[EOS]" is a special token representing the end of a document. "[ST]", "[UT]", and "[AT]" denote the system, user, and assistant chat template, respectively.
not be that easy without extensive fine-tuning experiences. Another issue is that learning to master knowledge utilization and format alignment in both instruction tuning and alignment stages might lead to suboptimal performance, since the two goals can be divergent in model optimization (Ren) al. 2024).
Considering the above issues, this paper explores a new domain adaptation approach that only uses raw Nomain--ppeific corpora and general instruction or alignment data. Our hypothesis is that the knowledge utilization capacity can be essentially learned from general instruction or alignment data, which has also been evidenced by prior studies (Ouyang al. 2022b). In this way, we can remove the tedious instruction synthesis step from the training pipeline, since it is much easier to obtain general mixed domain instruction data from open resources. Another important attempt to enhance knowledge learning by jointly acquiring both memorization and utilization of knowledge. Therefore, we schedule all the instruction and aligument data the pre-training stage (with a suitable format), then only reuse minor proportion of instruction and alignment data for fine-tuning to achieve format alignment. We give a comparison Figure
Specially, our approach for domain adaptation of LLMSs consists of two main stages, i.e., domain knowledge learning and general formaal format aligument. the first stage, we conduct knowledge mixture continual pre-training (Mix-CPT) to integrate both knowledge memorization and utilization. In the second stage, based on the knowledge and capabilities that are already acquired during pre-training, we perform instruction tuning and alignment in an efficient manner to achieve format alignment. For continual pre-traine vre-training, we convert raw domain documents, general instructions, and alignment data into a unified format. To avoid catastrophic forgetting, we propose Logit Swap Self-Distillation (LSSD)) which exchanges the predicted top-1 token logit with the logit of the ground-truth token, serving the surrogate target. In this way, LSSD maintains most probabilities of the original distribution of LLMs, thereby preserving original capabilities. In instruction tuning and alignment, we propose novel format alignment score criterion select small number of instructions from the pre-training instruction set. These instructions have already been seen during pre-training, so that the model can mainly focus on pure style or format learning for downstream tasks.
We conduct experiments on domain-specific and general tasks to verify the effectiveness of our Mix- CPT method, including a total of seven distinctive capabilities based on 17 representative bench- marks. For both base L.MSs and chat I.LMs. our approach can effectively improve their domain- specific and general performance compared to traditional methods of first performing continual pre- training. followed by instruction tuning and alignment.
APPROACH
To adapt general LLMs to specific domains (e.g., wiki, code), our core idea is to decouple knowledge learning and format alignment, and propose effective two-stage domain adaptation framework,
2

---

Published as a conference paper at ICLR 2025
teacher
Togit
= ulATo + (1 Loss
gold label
Supervised Fine-tuning
Direct Preference Optimization
Alignod Model
swop
Coupat
Knowledge Mixture
logit
training
Continual Pre-training
top-R
tep-X
logis
1..01
1.10
1.50
Initial
Domain--adapiad Model
Model
format
soct by
aligumeni score
alignment score
Domaln Corpora
MIX
DSFT
Doro
lbbr Hinstein [BOS] Nobel was
[BO]] Computer
Planse solve this problem, Jame decides
(EOS) Summarize the
Q:[ST] You [UT]
Plense ... [AT]
Describe ... [AT]
q: [ST] JUT]
Alignment Data
Describe
menns,
scasun
[EOS] Why
F: rr.Jmme decides to
dan''t know.....
Figure 2: Overall framework of our proposed rescheduled domain adaptation paradigm, Mix-CPT.
"[ST]", "[UT]", and "[AT]" denote the system, user, and assistant prompts, respectively.
i.e, first performing knowledge mixture continual pre-training (Section 2.1 and then performing efficient format alignment (Section 2.2p. We show the overall architecture Fi Figure
2.1 KNOWLEDGE MIXTURE CONTINUAL PRE- TRAINING
Different from previous work that performs continual pre- training solely based domain-specifer corpora (Que et al. 2024 Ke et al. 2022), we propose to mix domain-specific documents, general instructions and alignment data as pre-training data. The QA-based instruction and alignment data is useful to reflect how the knowledge will be accessed and utilized through questions, enhancing the learning of new knowledge domain-related documents. Besides, using general instructions can facilitate LLMSs to transfer the general knowledge utilization capability to specific domains instructions in prior work (Tiang et al. [024] Cheng et al.) (024a). Specifically, we first transform raw domain documents, general instructions, and alignment data into a unified format for continual pre-training. To avoid catastrophic forgetting, we further introduce a logit swap self-distillation (LSSD) approach during the continual pre-training process.
without relying on
2.1.1 UNIFIED KNOWLEDGE FORMAT
Typically, adapting LLMs to specific domain involves three distinct and relatively independent stages, each based on corresponding data in different formats. Specifically, the base model firstly performs continual pre-training (CPT) domain-specific corpora for learning new knowledge, then conducts supervised fine-tuning (SFT) based on instructions for enhancing the instruction following ability, and finally utilizes the preference data for human alignment. In this work, we adopt direct preference optimization (DPO) as the alignment algorithm. Formally, we denote the domain-specific corpus as Dcrr = {d.) nh where d, represents a raw domain document consisting of a sequence of tokens. For the instructions used in SFT, we denote as DSFT { {q: 1==1 where 9; and r: represent the user query and the expected response. repectively. For alignment data used in DPO, we denote by DDPo = {y: r ) where li, ri. and r, represent the user query, positive response, and negative response, respectively.
In this paper, we propose to mix DCPT, DSFT and DDPO into a mixture dataset DMIX with unified text format, building upon which we further perform knowledge mixture continual pre-training on a general LLM. Unlike previous work relying synthesizing high-quality domain instructions (Jiang et al. 2024CCheng et al. 2024a), we empirically find that knowledge utilization is actually general capability that be learned from general instructions, and
further transfer such ability to enhance the learning of domain knowledge. Specially, we remove any templates (e.g., [UT]) from instructions and alignment data to construct the mixture data, which consists of three kinds of samples denoted by DMIX = (Fept- Istt ] Tdpo where Teps d; is the original domain document, Jaft = [g:rr] denotes the concatenation of user query and expected response in the instruction, and Jdpo = 1q:r, r is the concatenation of user query and positive response in the alignment data.
Following existing pre-training methods (Touvron et al. 2023). we concatenate samples within each kind of data (i.e., DePr, DsEr and Dpro) and truncate the sequence when reaching the maximum
3

---

Published as a conference paper at ICLR 2025
imput length of the LLM. Besides, add extra special symbol (i.e., (EOS]) the end of each sample to separate them. We repeat this process until concatenating all samples to obtain our final knowledge mixture pre pre-training data DMIX-
Note that though we use general instruction data to derive the mixture data here, our approach can be generally extended to incorporating domain-specifie instruction data (Cheng al. 2024a), which often relies on specific data synthesis techniques.
2.1.2 LOGIT SWAP SELF- DISEILLLIIIIIOON
After obtaining the unified mixture data DMIX = [xcpes Tsht J. we then perform continual pre- training on the base LLM. For simplicity, we remove the subscript of each training sample in the mixture data, denoted as z. We adopt the pre-training task of next token prediction (NTP). Specif- ically, given input (wi, W2, ..., Wn J. we adopt the standard language modeling objective to minimize the cross-entropy loss as follows:
NIP =
Σ log Pr(wylwes; 2=1
(1)
where wy denotes the j-th token in the input, wey is the previous tokens, and è denotes the model parameters. During continual pre-training, the task of next-token prediction enables the base LLM to learn domain knowledge, and the incorporation of general instructions and the alignment data further transfers the general knowledge utilization capability specific domains.
However, the traditional language modeling objective is prone to suffer from the issue of catas- trophic forgetting for previously learned knowledge of LLMs. Therefore, we propose auxiliary training objective, i.e., Logit Swap Self-Distillation (LSSD), which serves as an extra constraint for the standard language modeling objective. Specifically, we first utilize the original base LLM be- fore continual pre-training (paramerized by θon) to infer the output logits following the standard language modeling objective yet without computing the loss:
hj = -LLM(W<F' Ooa):
(2)
1 h,W.".
(3)
where W. is the token embedding matrix, h, is the hidden state of the last transformer block, and L denotes the output logit at the j-th position. Then, we exchange the logit value of the top-1 predicted token (i.e., wy and the ground-truth token (i.e., wy) if they are not equal:
i, = Exchange(I,: Imy Iuy) if Im + Lwy:
(4)
where Lūg and Iwy are indices of Wy and wy in the vocabulary respectively, the function Exchange(.) will exchange their logit values in Lj. The swapped logit i, will be regarded as the teacher logit in LSSD. In essence, LSSD only calibrates the prediction of ground-truth token for adapting to the current domain knowledge while maintaining most originally learned knowledge of the LLM (i.e., represented by the unchanged logit values in I,). Next, we can compute the teacher model's probability distribution for the j-th token with softmax function:
Pr(wy (wey; a.s) softmax((,)-
Finally, we compute the self-knowledge distillation objective and minimize the reverse Kullback- Leibler divergence loss (Gu et al., (023) between the current model's probability distribution and the teacher model's probability distribution as follows:
LSSS = Σ Pr(wy w Q) log( Pr(wslwesi@ +) Pr(w, |W O.n)
(6)
In the knowledge mixture continual pre-training stage, the final total loss is the combination of next token prediction loss and self-distillation loss, controlled by a co-efficient a as follows:
LCPT a LNTP + (1 a) LISSD-
(7)
Due to the significant distributional differences between domain corpora and original training data, traditional approach of using one-hot label can lead to substantial model updates, which may cause catastrophic forgetting of the original knowledge after adaptation. In contrast, LSSD adopts the prediction from the original model as a surrogate label by only modifying the logit of top- token. This way can effectively maintain most previously learned knowledge but adapt to new domains.
4

---

Published as a conference paper at ICLR 2025
2.2 GENERAL FORMAT ALIGNMENT
In the domain knowledge learning stage, the LLM has simultaneously learned to memorize domain knowledge and understand how utilize the knowledge through our proposed knowledge mixture continual pre-training. After that, during the format alignment stage, the LLM can be more esficiently fine-tuned master the task format with only a small number of alignment samples. Next, we first introduce the selection of training samples and then perform efficient format alignment.
Alignment Sample Selection. Since we would like to decouple knowledge learning and format alignment, aim to select training samples from DEFT and Dro that have been encountered during continual pre-training which can avoid introducing new knowledge in subsequent fine-tuning. The selection criterion is based on the format alignment score (FAS), which estimates the difficulty of given SFT sample (g:, rz) by comparing the conditional loss with and without formatted instruction:
FAS( r.)) LNTP(rila.)
(rI)(r;)
where q: denote the formatted query equipped with chat templates for interaction with humans. For a DPO sample (g:, ..tr,tr r;),, FAS compares the conditional losses of positive response and negative response given the formatted query:
CNTP(r: 1g:)
FAS T. (T))
CNTP (rT (4:)
In essence, high FAS scores infer the difficulty of generating responses given query with an inter- action format and the significant disparity between positive and negative ones. Therefore, we finally select top- K samples with the highest FAS scores conduct supervised fine-tuning and direct pref- erence optimization.
Efficient Format Alignment. After selecting very few training samples, we can utilize them to conduct format alignment in an efficient manner. Firstly, we utilize the selected instruction samples from DET to perform supervised fine-tuning following the standard way (Ouyang et al., 20226), which is to minimize the cross-entropy loss:
7
SSFT
Σ
Pog
(10)
where ry and rzj denote the j-th token and its previous tokens in the response. Secondly, we utilize the selected preference samples from DDPO to conduct direct preference optimization following Rafailov et al. (203bb) as follows:
Cooo log or Blog 0))
(r+ O,rr) Bog <r 1g: +)
(11)
((r lq; 2
where a denotes the sigmoid function, O denotes the parameters of the reference LLM (usually the original LLM itself), and π denotes the product of the probabilities all output tokens.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
In our experiments, we mainly focus on three popular domains for adapting general LLMs, i.e. encyclopedia, mathematics, and code. And select official Wikipedia dump"] Auto (Zhang jet al.] 2024), and StarCoder (Li et al. (023) as the corresponding domain-specific corpus. For general instruction datasets, we choose TULU-V2-mi (lvison et al. (023) and Ultraeedbacck (Cui et al. 2023) for instruction tuning and alignment, respectively. We employ two prominent LLMs as the base model, i.e., QWen2-7B (Yang et al. 2024) and Meta-Llama-3-SB (Dubey et al. 2024). For comparative analysis, we consider official chat LLMs, reimplemented chat LLMs, and continual pre-training augmented LLMs.. We evaluate seven distinctive capabilities of LLMs in a total of 17 representative NLP datasets based on the OpenCompass platform (Contributors 2023). We show the details of our complete experimental setup in Appendix A
5

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results on three specialized domains and two general examinations. The underline and bold fonts denote the best results in the target domain and the average results in each domain adaptation group, respectively.
English Examination
Chinese Examination
Model
Wiki
Math
Code
Average
LLLMA3-8B-Bas
53.81
33.62
42.31
51.02
46.69
45.92
-CPT
54.20
34.07
37.42
52.97
46.07
45.64
Wiki
Mix-CPT (w/v KD)
55.81
38,96
41.62
52.19
47.44
47.68
Mix-CPT
55.47
37.91
42.37
52.55
49.93
47.91
CPT
54.14
34,80
28.95
50.47
49.02
43.62
Math
Mix-CPT (w/o KD)
55.55
37.99
42.43
46.16
48.60
45.90
Mix-CPT
55.11
37.71
45.14
51.54
49.89
48.04
CPT
50.98
34.19
35.64
51.14
47.92
44.29
Code
Mix-CPT (w/o KD)
54.93
39.38
42.35
46.31
47.91
46.02
Mix-CPT
55.50
38.15
42.82
51.55
48.50
47.61
QWen2-7B
46.37
62.33
60.02
46.56
82.90
56.00
CPT
47.20
59.48
29.45
$2.32
81.92
51.12
Wiki
Mix-CPT (w/o KD)
46.65
58.54
58.67
48.03
80.86
55.27
Mix-CPT
46.88
59.77
62.06
48.78
81.84
56.56
CPT
46.22
58.60
48.13
44.42
82.51
52.17
Math + Mix-CPT (w/o KD)
46.19
59.42
59.93
46.12
82.70
55.21
Mix-CPT
46.42
60.58
61.40
45.60
82.25
55.58
CPT
41.37
60.02
41.93
42.05
82.97
49.58
Code
Mix-CPT (w/o KD)
45.29
59.40
58.53
48.36
81,68
55.32
Mix-CPT
45.50
59.20
60.21
48.18
82.94
55.73
3.2
MAIN RESULTS
Table D and Table 2 display the evaluation results for base and chat LLMs respectively, comparing our proposed Mix-CPT method and other basclines.
3.2. RESULTS OF BASE LLMS
We initially assess the effectiveness of our proposed Mix-CPT framework for base LLMs in miti- gating catastrophic forgetting and facilitating the knowledge learning during continual pre- pre-training, especially the logit swap self-distillation constraint. To better observe the impact of knowledge mix- ture continual pre-training on the performance target domains and general capabilities, we select domain-specific tasks and general examination tasks. We show the evaluation results in Table [
First, we can see that traditional continual pre- treiiraining (i.e., + CPT) does not necessarily enhance the performance of base LLMs in the target domain, and may instead impair their performance therein. Besides, this method inevitably leads to a certain degree of catastrophic forgetting, thereby damaging the overall performance of the base LLM. For example, compared the LLaMA3-8B-BB 8B-Base model, typical CPT leads to observed improvements in math (i.e., 33.62 + 34.80) but diminishes overall performance (i.e,, 45.92 - 43.62). The phenomenon has been reported in existing work (Lin al., 2024). We notice that the performance of the QWen2-7B: B-BBaae in math and code domains declines after undergoing either traditional continual pre-training Mix-CPT. We speculate that this may be due to the significant discrepancy between the distribution of the continual pre- traininnnngg data and the data used the final stage of original pre-training (e.g., annealing) YYagg et al.) 2024 Dubey al. 2024). Besides, the model might have already been trained on these two datasets, and further training leads to overfitting. For example, the typical CPT method shows a decline in the math domain (i.e. 62.33 + 58.60) and the overall result (i.e., 56.00 + 52..7). By contrast, Mix-CPT significantly alleviates such a decline and better maintains general performance (i.e., 56.00 - 55.58). This demonstrates that our framework by traiitioaal incremental training processes. This highly beneficial in rorl-world scenarios where the training data proportion and distribution of the model usually unknown.
effectively mitigate the impact of knowledge forgetting caused

---

Published as a conference paper at ICLR 2025
Table Evaluation results on three specialized domains and four general capabilities (i.e., Reading Comprehension, Complex Reasoning, EXamination, and Instruction Following). The "CSD" de-
notes the typical domain adaptation method of conducting CPT. SFT, and DPO. The underline and
bold fonts denote the same meaning as Table [
Specialized Domain
General Domain
Model
Average
Wiki
Math
Code
RC
CR
EX
TF
Official Chat
46.97
50.88
63.29
77.03
78.21
62.02
80.40
64.96
Reimplemented Chat
33.72
30.88
41.96
74.19
77.42
57.62
69.55
55.86
+ +CSD
28.27
30.67
43.47
72.61
73.74
55.82
66.51
53.82
Wiki
LLaMA3
+ MixxCPP (ours)
33.21
30.21
47.22
72.60
71.36
58.65
68.88
55.25
8B-Base
+ CSD
27.88
39.48
50.95
74.89
76.90
60.90
69.89
58.16
Math
+ Mix-CPT (ours)
32.12
39.40
51.21
73.36
77.00
60.68
71.81
58.61
CSD
27.64
39.27
51.75
75.04
77.44
62.34
70.88
58.75
Code
Mix-CPT (ours)
31.98
38.30
52.67
73.33
74.86
60.71
72.11
58.25
Official Chat
37.39
67.05
72.31
85.79
82.08
73.48
82.63
71.74
Reimplemented Chat
28.10
58.12
61.22
84.35
81.86
71.08
75.40
66.80
CSD
30.29
56.75
61.78
82.61
81.66
71.23
75.92
66.79
Wiki
QWen2 7B-Bast
+ Mix-CPT (ours)
34.55
56.74
64.66
83.93
80.61
72.66
74.73
67.94
+ +SD
28.58
56.78
60.85
83.11 1
81.83
70.79
75.87
66.45
Math
+ Mix-CPT (ours)
33.81
57.48
67.38
83.29
80.88
72.44
76.27
68.29
CSD
29.41
56.87
62.74
83,58
81.77
70.94
76.79
66.94
Code
+ Mix-CP CPT (ours)
36.05
55.07
64.84
83.42
80.57
73.65
76.77
68.24
Second, when mixing the domain raw data with the additional instructions and alignment data (i.e., Mix-CPT w/o KD), the domain-specific capability can be further improved, which indicates that the mixed instruction data can benefit the learning of the domain knowledge during continual pre- training. At the same time, it can mitigate the effect of other general capabilities and reduce the degradation of the overall performance. For example, compared the traditional CPT, mixing do- main corpus with additional instruction data (Mix-CPT w/o KD) can almost consistently improve the domain capability and overall average performance (i.e., 41.93 + 58.53 and 49.58 55.32 for the QWen2-7B (ren2-7B-Bass B-Base model in the target Code domain and overall average performance respectively).
Finally, through applying the logit swap self distilllilaooon strategy to the knowledge mixture continual pre-training process (i.e., Mix-CPT), we can further reduce the impact on the pre-leamed knowledge while maintaining the domain capability improvement, thereby mitigating the degradation of the general capabilities of LLMs. Therefore, these results demonstrate that the Mix-CPT framework with the logit swap self-distillation constraint can indeed promote knowledge learning and alleviate the issue of catastrophic forgetting to some extent.
3.2.2 RESULTS OF CHAT LLM
Subsequently, we assess the performance of final chat LLMs after instruction tuning and alignment for achieving format alignment. We show the evaluation results in Table
First, the typical domain adaptation method of conducting CPT, SFT, and DPO (called CSD) faces challenges in simultaneously enhancing domains-specifie capabilities while preserving general ca- pabilities, in contrast to reimplemented chat models that do not utilize domain- specciii raw data, For example, compared to reproduced I.LAMA3-BB chat model, traditional CSD method not only de- creases the factual question answering performance in Wiki (i.e., 33.72 - 28.27) ) but also hurts the average performance (i.e., 55.86 53.82). This phenomenon indicates that conventional domain adaptation methods may cause catastrophic forgetting and merely focus on knowledge memorization without considering how utilize them, which might suffer from memorization trap.
Second, with the same instruction data and alignment data, our method can successfully improve the performance the target domain while maintaining the general capability compared to the reim- plemented models. Compared to the reimplemented chat LLM with large-scale closed-source pre- training data but using the same instruction and alignment data, proposed method achieve

---

Published as a conference paper at ICLR 2025
58
62
65
Code Avenge
Math Average
Math
56
60
39
60
Average
Pass@ak
Average
Accuracy
Average
40-
60
54
38
52
56
37
35
55
50
0.1
0.3
0.5
0.7
54
0.5
0.6
0.7
0.8 54
30
Rand Low High Half 50
Quality Score of Math Data Figure 3: (Left) @ and Average results w.r.t. Proportion of code data. (Right) Accuracy and Average results w.r.. Quality score math data.
Proportion of Code Data
Figure 4: Accuracy math and Average results w.r.t. se- lection strategy.
better performance on the target domain (e.g., 41.96 ++ 52.67 the code domain for LLaMA3-8B- Base), which indicates the effectiveness of our method.
Finally, our proposed Mix-CPP method can simultaneously improve the performance of the target domains and the general capability. The main reasons are two fold. On one hand, based on the logit swap self-knowledge distillation constraint, the LLM can effectively memorize the raw domain data while maintaining its originally learned knowledge. On the other hand, by mixing the domain data with the general instructions and aligmment data (removing any templates), the model can learn the general knowledge utilization capability and transfer to specific domains. In this way, the model can perform efficient format alignment with only a few formatted samples to better utilize both target domain knowledge and other general knowledge.
3.3 DETAILED ANALYSIS
In this section, we conduct detailed analysis of the proposed method with LLaMA3-8B, using the same benchmarks as those presented in Table
Effect of Quantity and Quality of Raw Domain Data. We first utilize the StarCoder dataset to explore the effect of the number of domain documents on domain-specific and general capabilities with consistent quality. Then, we adopt the Tuxx dataset to explore the effect of the quality of raw domain data LLMs' performance by leveraging the annotated quality scores. Specifically, we conduct two group experiments as follows:
Proportion of Code Data: This group aims to compare the variants using different proportions of the StarCoder, including 10%, 30%, 50%, and 70%, while maintaining constancy in other variables.
Quality of Math Data: This group aims to compare variants by employing various AutoMath- Text, each characterized by distinct quality scores with thresholds exceeding 0.5, 0.6, 0.7, and 0.8 respectively, while maintaining constancy in other variables.
We can see that increasing the amount of raw domain documents indeed further enhance the target domain performance under the same quality. However, though using a larger number of raw domain data (e.g., more math texts with 2 0.5 score than those with 2 0.7 score), the low quality of raw data can also decrease the LLMs' performance regardless of the target domain or general domain, which indicates the quality of domain data is a priority over its amount when performing continual pre-traine tra-trainnng The final decline is due to that there are much less math texts with 2 0.8 score (i.e., only 10% of math texts with > 0.7 score).
We show the results in Figure 3
Effeet of Format Alignment Data Selection. We conduct a further ablation study to explore the impact of different sample selection strategies for format alignment data, which consists of the dif- ficulty and amount of selected data. Specifically, we conduct two groups of experiments including:
Difficulty of Samples for SFT and DPO: This group compares four distinct selection strategies: random selection (Rand), easiest samples with the lowest FAS scores (Low), hardest samples with the highest FAS scores (High), and half easiest samples and half hardest samples (Half).
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation results on medicine and general domains with LLaMA3-8B- model when
adapting to medicine domain. The underline and bold fonts denote the same meaning as Table
Medicine Domain
General Domain
Model
RCT
PubMedQA
MQP
Avg
MATH
MBPP
MMLU
CEVAL
Avg
LLaMA3-8B-Baa
73.6
59.8
66.2
60.1
18.0
57.9
65.9
47.8
+ CPT
70.6
56.7
55.4
55.7
5.9
57.0
63.2
48.6
43.7
Mix-CPT
71.0
63.4
79.2
63.4
16,6
58.0
66.2
49.7
47.6
Mix-CPT (Half)
71.2
64.1
83.0
64.7
16.0
57.7
65.5
50.3
47.4
41
61
41
61
50
55
Math Average
40
60
40
60
Accuracy
Average
Accuracy
Average
45
50
39
3
39
40
45
38
58
38
Average
Average
37 10K
20K
40K
80K
375K
10K
20K
40K
LSSD
KL 40
Amount of DPO Data Figure 5: Accuracy on math and Average results w.t.t Amount of SFT data (Left) and Amount of DPO data (Right).
Amount of SFT Data
Figure 6: Accuracy on Math and Average results w.r.. dis- tillation and normalization.
Amout of Samples for SFT and DPO: This involves four variants using different quantities of easiest samples from the TULU-V2-mix dataset (i.e., 10K, 20K, 40K, and 80K) and from the Ultra- Feedback dataset (i.e., 5K, 10K, 20K, and 40K).
We show the results in Figure and Figure Firstly, highest FAS scores can balance the domain and general capability best compared to other selection strategies. Then, enhances both domain and general abilities simultaneously increasing the amount of SFT training samples. Conversely, when increasing the amount of DPÒ training samples, the results remain fluctuating, which might be due the alignment tax.
that, by using the samples with the
certain extent by
Effeet of Knowledge Distillation. To examine the effectiveness of our proposed logit swap self- distillation (LSSD), we compare our method to two distillation and normalization methods:
w/ LSSD: this is our Mix-CPT model using LSSD.
w/ KD: this replaces LSSD with a typical knowledge distillation method without logits swapping.
L2: this replaces LSSD with L2 normalization.
The results are shown Figure We can see that compared directly using L2 regularization to constrain the changes in model parameter weights, KD method can better preserve general capa- bilities when learning domain capabilities. Through logits swapping, our proposed LSSD method refines the inconsistencies with the gold label in KD method, further enhancing the learning of domain-specific capabilities. Finally, LSSD achieves the best balance among the three methods.
Performance Medicine Domain. Here, we further explore the performance of Mix-CPT and traditional CPT in the medical domain. Specifically, following the existing work (Cheng et al. 2024b), we select the PubMed dataset as the domain-specifee corpus and choose RCT (Dernoncourl & Lee 2017), PubMedQA (Jin et al.] 2019), and MQP McCreery et al.) (2020) as the domain- related evaluation collections. For the general domain evaluation benchmarks, we select MATH, MBPP. MMLU, and CEVAL from the aforementioned experimental settings. We show the results Table 3 We observe that compared to traditional CPT methods, our proposed Mix-CPT approach more effectively facilitates the learning of domain-specifir skills and better maintains the general capabilities, indicating the effectiveness and generalizability of our method.
Computational Efficiency of Mix-CPT. We have discussed the additional overhead of our method in Appendix AA.5) Here, we further conduct additional experiments by only randomly employing 50% pre-training data for performing the logit swap self-distillation, i.e. Mix-CPT (Half), which

---

Published as a conference paper at ICLR 2025
further reduce half of the costs introduced by self-distillation. We conduct the experiment on the medical domain with the LLaMA3- 3--8--Base model and evaluate its performance on domain -speciffc and general tasks before. We show the results in Table []] We find that compared to distilling the entire dataset, reducing the data by half further enhances the model's performance on domain- specific tasks, with only slight decrease in general capabilities. This suggests that exploring even lower distillation ratios in the future could achieve greater tradcoff between cost and performance.
4 RELATED WORK
Domain Adaptation of LLMs. Our work is closely related to efforts in adapting general LLMs to
specific domains (Yildiz al. 2024 Ke al. 2022 Scialom al.. 2022). Due to the increasing
scale and complexity of LLMs, training domain-specifie LLMs from scratch involves significantly
high financial and ecological costs (Luccioni et al. 2023 To address this issue, recent work has
been devoted to studying efficient approaches like continual pre-training, which involves incremen-
tally training general LLMs based on new domain corpora Quee et al.) [024] Ke al. 2022), and continual fine-tuning, aiming to fune-tune general LLMs on a series of downstream tasks related to target domains (Razdaibiedina et al., 2023] Scialom et al. [022] [uo et al., 2023a). However, these approaches might result in catastrophic forgetting and performance degradation in general language tasks (Kar et al. 2022 Mehta et al., 2023). Another line of work has explored conducting instruc- tion fine-tuning by synthesizing commin-relatee instructions (Cheng et al. 2024a Jiang al.. (024). Nevertheless, these studies require additional models to synthesize amounts of instructions highly related to specific domains, resulting in high computational costs. It is noted that our method differs from these works in several ways. Firstly, we disentangle domain adaptation into knowledge memo- rization and capability elicitation, focusing on learning domain-specific knowledge and solving do- main tasks with learned knowledge, respectively. Secondly, we employ token swap self-distillation in the mixture pre-training to retain general knowledge and avoid catastrophic forgetting.
Instruction Tuming and Alignment. Instruction tuning (also known supervised fine-tuning) employs human--nnotatee instructions (Sanh et al. 2022 Mishra et al. [0022] Köpf et al., 2023 Sun
ct al. 2023) or synthetic instructions by proprictary models (Taori et al. 2023 Chiang et al. 2023
Wang et al. 2023) to fine-tune LLMs. Besides, alignment with reinforcement learning from human
feedback (RLHF) (Ouyang et al. 2022a) or direct preference optimization (DPO) (Rafailov et al., 2023a) aims to align LLMS with human preference. Both instruction tuning and aligmment are able to elicit knowledge from LLMs and improve their capabilities to solve downstream tasks. Recent work (Zhou et al. (023) has demonstrated that L.MSs mainly learn the style or format for interacting with users through simple instruction tuning and alignment, by leveraging their prior knowledge and capabilities already acquired during the pre-training stage. Furthermore, by comparing the token distribution before and after alignment, recent work (Lin et al. 2023) found that the most significant distribution shifts appear dominantly in stylistic tokens such as transitional phrases and discourse markers instead of contextual words that involve rich knowledge for solving downstream tasks. Inspired by these studies, we propose to expose knowledge memorization and capability elicitation from instruction tuning and alignment. Unlike these studies which typically focused on instruction tuning or alignment, pre- -reitraining, instruction tuning, and alignment) and conduct a knowledge mixture pre-training to mainly focus learning new domain knowledge while maintaining general knowledge.
differ in that we unify the three stages of training LLMs (i.e., continual
5 CONCLUSION
In this work, we proposed a two-stage domain adaptation approach, termed Mix-CPT, which en- compasses both domain knowledge learning and general format alignment. Mix-CPT employed knowledge mixture continual pre-training to learn domain knowledge by integrating domain-specifer raw data with general instructions and alignment data. Besides, we proposed Logit Swap Self- Distillation (LSSD) to relieve catastrophic forgetting. Based the knowledge and capabilities acquired during pre-training, we then selected a small number of easy instructions to make the LLM to learn the style format for interacting with human. Extensive experiments three benchmark datasets showed that our proposed Mix-CPT outperforms the traditional method, obtaining improve- ments both the domain and general capabilities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by National Natural Science Foundation of China under Grant No. 92470205, Beijing Municipal Science and Technology Project under Grant No. Z231 100010323009, and Beijing Natural Science Foundation under Grant No. L233008. Xin Zhao the corresponding author.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732. 2021.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q Jiang, Jia Deng. Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. CoRR, 0bs/22300116311 2023.
Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richard- son, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03315, 2021.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA:
reasoning AAAI Conference on about physical commonsense in natural language. In The Applications of Artificial Intelli- Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative gence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439, AAAI Press, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob MeGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. CoRR, abs/2107.03374, 2021. URL httpss//aaxxi org/ abs 03374
Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. Instruction pre-training: Language models are supervised multitask learners, 2024a.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading com- prehension. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph Gonzalez. lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https: /tts:///viuua.... imsys.
ore 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2 14.164 2021.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. http:///hthbb..oooo//oooooo//oa/
2023.

---

Published as a conference paper at ICLR 2025
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedbckk: Boosting language models with high-quality feedback. CoRR, abs/2310.01377, 2023.
Franck Dernoncourt and Young Lee. Pubmed 200k RCT: a dataset for sequential sentence classifi- cation in medical abstracts. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 December 1, 2017, Volume 2: Short Papers, pp. 303-313. Asian Federation of Natural Language Processing, 2017.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnacve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Ilama 3 herd of models. CoRR, abs/2407.2178.. 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang, Knowledge distillation of large language models. CoRR, Ahs/2306.08543 2023.
Xu Guo and Han Yu. On the domain adaptation and generalization of pretrained language models: A survey. CoRR, abs/221 1.03154, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021
Hanxu Hu, Pinzhen Chen, and Edoardo M. Ponti. Fine-tuning large language models with sequential instructions. CoRR, abs/2403.00794 2024a.
Yiwen Hu, Huatong Song, Jia Deng, Jiapeng Wang, Jie Chen, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Wayne Xin Zhao, and Ji-Rong Wen. Yulan-mini: An open data-efficient language model. CoRR, abs/2412. 2024b.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Hamish Ivison, Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hamnanch Hajishirzi. Camels in changing climate: Enhancing LM adaptation with tulu 2. CoRR, abs/2311. 1,1002, 2023.
12

---

Published as a conference paper at ICLR 2025
Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, and Srinivasan Iyer. Instruction-tune -untedd language models are bet- ter knowledge learners. arXiv preprint arXiv:2402.12844 2024.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 2567-2577. Association for Computational Linguistics, 2019.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of 55th Annual Meet- ing of the Association for Computational Linguistics, ACI. 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguisties. 2017.
Sudipta Kar, Giuseppe Castellucci, Simone Filice, Shervin Malmasi, and Oleg Rokhlenko. Prevent- ing catastrophic forgetting in continual learning of new natural language tasks. KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 18, 2022. pp. 3137-3145, ACM, 2022.
Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. Adapting a language model while preserving its general knowledge. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 10177-10188. Association for Computational Linguistics, 2022.
Andreas Köpf. Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. Openassistant conversations democratizing large language model alignment. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering rescarch. Trans. Assoc. Comput. Linguistics, 7:452-466, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. RACE: large-scale read- ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empir- ical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 785-794. Association for Computational Linguistics, 2017.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo Thomas Wang. Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V. Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang. Nour Moustafa- Fahmay, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas. Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferran- dis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! CORR, abs/2305.06166 2023.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell base llms: Rethinking alignment
13

---

Published as a conference paper at ICLR 2025
via in-context learning. CoRR, abs/2312.015522 2023. doi: 1048550/ARXXV.231201552 URL pettps://doi.ozei http://.....50/.211222222......
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need. CoRR, has/2404.079965. 2024.
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon foot- print of bloom, 176b parameter language model. Mach. Learn, Res., 24:253:1-253:15 2023.
Yun Luo, Zhen Yang, Fandong Meng. Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. CoRR, abs/2308.00 087477, 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. CoRR, hts/2306.08566 2023b.
Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. Ef- fective transfer learning for identifying similar questions: Matching user questions to COVID-19 faqs. Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 3458-3465 3365. ACM, 2020.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investi- gation of the role of pre-training in lifelong learning. J. Mach. Learn. Res., 24:214:1-214:50, 2023.
Yu Meng, Mengzhou Xia. and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. CoRR, ab4/2405.14734, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp.
Association for Computational Linguistics, 2018
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3470-3487 3487. Association for Computational Linguistics, 2022.
OpenAl. Gpt-4 technical report. OpenAl Blog. 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint 22Xiv:2003.02111 2022a.
Long Ouyang, Jeffrey Wu. Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022b.
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, et al. D-cpt law: Domain-speciffe continual pre-training scaling law for large language models. arXiv preprint 2001v:2406.01375. 2024.
14

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint Stti:::2305.820000 2023a.
Rafael Rafailov, Archit Sharma, Bric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Alma- hairi. Progressive prompts: Continual learning for language models. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
OpenReview.net, 2023.
Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, and Le Sun. Learning or self-aligning? rethinking instruction fine-tuning. CoRR, abs/2402. 2024.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnacve. Code llama: Open foundation models for code. CoRR, abs/2308. 2023.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. The Tenth International Conference Leurning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 6107-6122. Association for Computational Linguistics, 2022.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le. Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguisties: ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 13003-13051. Association for Computational I. Linguistics, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensegua A question answering challenge targeting commonsense knowledge. Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguisties: Human Lan- guage Technologies, NAACL-HI.T 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers). pp. 4149-4158. Association for Computational Linguistics, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followii llama model. https: //g: thub. wttp://ttoottottoooto lab/ htt/sstaffod___c l lpaca 2023.
15

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.1 13971, 2023.
Rui Wang, Fei Mi, Yi Chen, Boyang Xue, Hongru Wang, Qi Zhu, Kam-Fai Wong, and Ruifeng Xu. Role prompting guided domain adaptation with general capability preserve for large language models. CoRR, abs/2403.02756, 2024.
Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04755 0,4551,1.. 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28. December 9, 2022, 2022.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li. Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.1061 10671, 2024.
Çagatay Yildiz, Nishaanth Kanna Ravichandran, Prishruit Punia, Matthias Bethge, and Beyza Ermis. Investigating continual pretraining in large language models; Insights and implications. CoRR, abs/2402. 17400, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019.
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao. Automathtext: Autonomous data selection with language models for mathematical texts. CoRR, abs/2402.076255, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303. 18223, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. In Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Kun Zhou, Beichen Zhang, Jiapeng Wang, Zhipeng Chen, Wayne Xin Zhao Jing Sha, Zhichao Sheng, Shijin Wang, and Ji-Rong Wen. Jiuzhang3.0 .: Efficiently improving mathematical reason- ing by training small data synthesis models. CoRR, abs/2405. 14365, 2024.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXPERIMENTAL SETUP
A.1 DOMAIN-SPECIRE CORPUS
In our experiments, we mainly focus on three popular domains for adapting general-purposs LLMs, i.e, encyclopedia, mathematics, and code. For the encyclopedia domain, we select Wikipedia as the primary corpus, which is collaboratively developed by volunteers globally and can be freely accessed online. To enable LLMs to learn knowledge from new documents, we utilize the official 2024/03/01 Wikipedia dump and conduct necessary data cleaning and filtering processes such as deduplication, resulting approximately 4B tokens raw Wikipedia documents. For the domain of mathematics, we opt for Auth (Zhang et al. 2024), a carefully curated corpus derived from various sources including websites, arXiv, and GitHub. Each sample in this corpus has been labeled with a quality score from 0.0 ("the poorest") to 1.0 ("the best"), reflecting its relevance, quality, and educational value in the context mathematical intelligence. Following previous work (Zhou et al., 2024). we specifically select those samples with scores higher than 0.7, containing about 0.7B tokens. For the field of code, we select the StarCoder (Li et al.) (023) corpus, which is widely rec- ognized and employed in several studies (Luo et al. (023b). It contains 86 programming languages, and we select the Python subset with approximately 1B tokens.
2
A.2 GENERAL INSTRUCTION DATASETS
For general instruction datasets, we choose TULU-V2-mix (Ivison et al. 2023) and UltraFeed- back (Cui et al., 2023) for instruction tuning and alignment, respectively. Specifically, each sample in TULU-V2-miX is either manually curated for quality or generated from GPT models to encour- age complexity and diversity. We utilize the entire dataset of TULU-V2-mix (about 326K samples) mixed with then select easy samples with the top-K lowest perplexity score for subsequent instruction tuning- In addition, UltraFeedback is a widely-used diverse human preference alignment dataset, containing approximately 64K preference pairs. Similarly, we employ the whole dataset of UltraFeedback for knowledge mixture continual pre-training and then downsample easy pairs with the top-K; lowest perplexity score for aligmment. It is noted that our TULU-V2-min -ii and UltraFeedback datasets are open-source and widely used in previous work (Meng et al. [024 Hu et al. (022a)), ensuring high level of transparency and facilitating fair experimental comparisons.
corpus for knowledge mixture continual pre-training (Section 2.1 and
A.3 BASELINES
In our experiments, we employ two prominent LLMs as the base model, i.e., QWen2-7B (Yang al. 2024) and Meta-Llama-33SB (Dubey et al. 2024). For comparative analysis, we consider the following three types of baseline methods:
Official Chat LLMs consist of the official Chat LLMs that have undergone both instruction tun- ing and preference alignment using closed-source data. Here, we select two official chat LLMs corresponding to our selected base models, including QWen2-7B-Chat and --B-aa--------stucc...
Reimplemented Chat I.LMs are developed by us following the processes of instruction tuning and alignment. Based on our selected three base LLMs, we conduct supervised fine-tuning (SFT) using TULU-V2-mix, followed by direct preference optimization (DPO) with UltraFeedback.
Continual Pre-training Augmented LLMs include domain knowledge-enhanced Chat LLMs which initially undergo continual pre-training (CPT) with our domain-specific corpus, followed by the same implementation of supervised fine-tuning (SFT) and direct preference optimization (DPO) using TULU-V2-na --iim and UltraFeedback as open-source chat LLMs.
17

---

Published as a conference paper at ICLR 2025
Table 4: The categorization of the evaluation benchmarks
Wiki
NQ, TQ
Specialized
Math
GSM8K, MATH
Domain
Code
MBPP, HvmanEval
Reading
RACE-Hard, OpenBookQA
Comprehension
Complex
HellaSwag, CSQA, PIQA
Reasoning
General
Examination
MMLU, BBH, ARC-Challenge, C-EVAL
Domain
Instruction
MT-Bench
Following
A.4 EVALUATION BENCHMARKS AND METRICS
For comprehensive evaluation, we evaluate seven distinctive capabilities of LLMs based on total of 16 representative NLP datasets:
Factual Question Answering assesses the factual knowledge of LLMs in the Wikipedia domain. We employ NaturalQuestion (NQ) (Kwiatkowski et al.) 2019) and TrivialQA (TQ) (Joshi et al., 2017) datasets and use the Exact Match (EM) metric determine if the prediction is the same as the gold answer.
Math Reasoning tests the LLMs' ability to solve mathematical problems. We use the GSM8K (Cobbe et al. 2021) and MATH (Hendrycks et al. 2021b) datasets, and evaluate predictions using the Accurucy metric.
Code Reasoning tests the LLMs ability to solve programming problems. We use MBPP (Austin al., (02I)) and HumanEval (Chen et al. (20TT) datasets, with the Pass@K metric assessing the likelihood that at least one of the top-K generated code samples for a problem passes the unit tests.
Reading Comprehension measures the LLMs' ability to comprehend a passage and answer related questions. We use the RACE-Hard (Lai et al.] 2017) and OpenBookQA (Mihaylov et al. 2018) datasets, and employ the Exact Match (MM) metric.
Commonsense Reasoning evaluates the ability to answer questions using commonsense knowl- edge. We use the HellaSwag (Zellers et al. 2019), CSQA (Talmor et al. 2019). and PIQA (Bisk
et al. 2020) datasets, and employ the Accuracy metric.
Examination includes comprehensive and challenging benchmarks designed to assess problem- solving ability across various domains. We MMLU (Hendrycks et al. (021a), BBH (Suzgun et al. 2023), and ARC-Challenge (Bhakthavatsalam et al. 2001) for English examinations and C- EVAL. (Huang et al. 2023) for Chinese. Both benchmarks are evaluated using the Accuracy metric.
Instruction Following assesses the LLMs' ability to engage in coherent, informative, and engaging conversations, We use the MT-Bench (Zheng et al., 2023) datasets. For evaluation, we utilize the as the judging model following the OpenCompass settings, assigning score ranging
GPT-4-turbo Fl from to 10 the answer. We multiply this score by ten, resulting in final score of 100.
Specifically, we evaluate the above datasets based on the OpenCompass framework (Contributors) 2023), which a one-stop platform for large model evaluation, aiming provide a fair, open, and reproducible benchmark for large model evaluation. We summarize the datasets in Table
A.5 IMPLEMENTATION DETAILS
For both models, we conduct the mixed continual pre-training with the same Specifically, we set the batch size as 1920, the maximal sequence length 2048, the maximal and minimal learning rate as 2c-5 and 5e-6, and the co-efficient a as 0.8. Then, for LLaMA3-8B model, select the top 10,000 samples and 10,000 samples for SFT and DPO, respectively. For the SFT
18

---

Published as a conference paper at ICLR 2025
stage, perform 2 epochs of training with learning rate of 5e-6 and batch size of 128. For the DPO stage, we perform 2 epochs of training with a learning rate of 5e-7, a batch size of 128, and 3 of 0.01. For the QWen2-7B model, select the top 10,000 samples and 10,000 samples for SFT and DPO, respectively. For the SFT stage, we perform 2 epochs of training with a learning perform epochs of training with
rate of 5e-7 and batch size of 128. For the DPO stage, learning rate of le-7, a batch size of 128, and a 8 of 0.1. When evaluating, we set the temperature 0.3 and top-p 0.9 for MT-Bench according the official guideline and set the temperature as O for other benchmarks to control the randomness. As for the computational cost, the additional overhead introduced by the instruction and alignment data may be negligible, which only introduces approximately 0.09B tokens in continual pre-training and 0.008B tokens for totally selected 20K pairs in the format alignment stage.
19


Reviews:
[{"review_id": "kA6tnjbFXn", "reviewer": "Reviewer_tVX8", "summary": "In this paper, the author addresses improving the efficiency of LLMs in adapting to new knowledge. Here, efficiency refers to adapting the LLM without relying solely on instruction-based tuning for the new domain; instead, the author employs a combination of domain-specific and general data. Additionally, alignment is achieved during the pretraining stage through a unified format and an additional training objective.", "questions": "See weakness. \n\nAnd I do have another question. What’s the fundamental difference between doing all of this during pretraining versus handling each step separately? Does this suggest that there’s no need to separate these steps and that it might be better to combine everything in a single pretraining process for new domain knowledge?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well-organized, and the problem it addresses is substantial. Improving the efficiency of knowledge distillation is indeed a relevant challenge that needs attention. Furthermore, the experimental results appear promising.", "weaknesses": "1. The paper talks about mixing CPT, SFT, and DPO data into a combined dataset for pretraining, but mixing instruction-tuning and pretraining data isn’t really new. Plus, there aren’t many details on how exactly these datasets are blended—like what proportions are used. The approach feels a bit empirical, but it would be great to have more specifics for reproducibility.\n\n2. The author claims that the LLM learns new knowledge efficiently by understanding the task it’s meant to perform on the data, along with alignment during pretraining. I get the alignment part, but I’m a bit concerned that task-specific instruction data might limit the generalization of domain-specific data. I wonder if it’s always beneficial for new knowledge to align closely with existing world knowledge. While task performance in the new domain improves, it’s unclear if the model’s general ability is preserved—this wasn’t really explored in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CerBnREky0", "reviewer": "Reviewer_9g2u", "summary": "The paper proposes a novel two-stage domain adaptation method for language models. The authors discuss the limitations of the existing training pipeline, which includes pre-training, instruction-tuning, and preference alignment. Based on this discussion, they propose a mix of data from each stage in an appropriate format to enable the model to learn both domain knowledge (i.e., pre-training) and knowledge utilization (i.e., instruction-tuning and preference learning). Additionally, the authors enhance the approach by training the model further on a small yet essential subset of data used in pre-training for improved model output alignment. The proposed method was evaluated with QWen2-7B and Llama-3-8B on domain data, including Wiki data, AutoMathText, and StarCoder. This method outperforms the simple CPT model.", "questions": "- Refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is well-motivated\n- The paper is well-written and easy to follow\n- The proposed method outperforms the baselines", "weaknesses": "- Logit swap self-distillation requires the base LLM to perform inferences on the entire training dataset. This process can be quite expensive, especially with large datasets and models.\n- A concern with the experimental setup is that Wikipedia data is used for domain adaptation. However, Wikipedia is more general-purpose than domain-specific, as it covers a wide range of topics. Additionally, the base models have likely already been trained on this widely used open-source dataset, which weakens the observations drawn from the experiments.\n- The authors should have compared their method to a recent, similar CPT method [1]. Method [1] has an advantage as it trains a base model on both raw documents and formatted reading comprehension data. This approach is more straightforward than the authors’ method, as it requires only single-stage training and does not necessitate the separate collection of CPT, SFT, and DPO data.\n- It is unclear why the model requires a second alignment stage, as it has already been trained on formatted DPO data. Is this due to issues with the initial stage of training?\n\n[1] Adapting Large Language Models to Domains via Reading Comprehension, ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jfIL5mZ4gH", "reviewer": "Reviewer_wQCa", "summary": "The paper introduces Mix-CPT, a domain adaptation framework for LLMs that separates knowledge learning from format alignment to enhance efficiency and prevent knowledge loss. In the first stage, Mix-CPT trains on a mixture of domain-specific and general instructional data to balance domain learning and general capabilities, using Logit Swap Self-Distillation to retain prior knowledge. The second stage, format alignment, fine-tunes the model with a few selected samples to align responses to human-preferred formats. Extensive experiments and ablations have been carried out to support the proposed framework.", "questions": "1. The results indicate a dependence on the quality of domain-specific data, with lower-quality data adversely impacting performance (Fig. 3). This reliance could restrict the framework’s applicability in real-world scenarios where high-quality datasets are limited.\n\n2. Why is the logit of the top-1 token specifically swapped with the ground truth in LSSD? Beyond the math and average measurements in Fig. 6, could you provide a more detailed explanation of why this technique is effective?\n\n3. Could you provide an analysis of computational efficiency compared to traditional domain adaptation frameworks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel view of CPT that separates domain knowledge learning from format alignment, potentially enhances both domain adaptation efficiency and task-specific performance.\n\n2. The proposed LSSD method to avoid catastrophic forgetting is clear and straightforward, allowing the model to adapt without sacrificing previously learned information. \n\n3. Integrating domain-specific raw data with general instructional data during the pre-training phase is a strength, as it appears to balance both domain-specific and general knowledge effectively.", "weaknesses": "see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "COASiE8Qdn", "reviewer": "Reviewer_Yvdt", "summary": "This paper proposes the Mix-CPT strategy to mix knowledge memorization and utilization into the continual pre-training stage. Specifically, the authors eliminate the chat template of general instruction-following and preference-alignment data (which is traditionally used in the SFT stage with chat templates), and mix them with domain knowledge corpus for vanilla next-token prediction optimization. A logit swap self-distillation strategy is developed to mitigate catastrophic forgetting. After that, the authors derived a format alignment score to select a small number of instructions from the CPT dataset to execute SFT and DPO, in order to focus mainly on pure style or format learning for downstream tasks.", "questions": "1. What is the FAS score's physical meaning?  Does a higher score mean a better or worse sample? Besides, if the authors aim at selecting the samples that LLM have not currently learned the format, taking Eq.8 for an example, it seems more reasonable to compare the loss on response (r) conditioned on questions with and without chat templates, i.e., $\\frac{L(r|\\hat{q})}{L(r|\\bar{q})}$. And Eq.9 presents the same problem.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The research point is important. Efficient knowledge memorization is crucial to derive both fundamental and domain-specific LLMs.\n2. The idea of decoupling knowledge learning and format alignment has promising efficacy, as they serve different purposes.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. My core concern is the lack of novelty. Taking SFT samples into the CPT stage has been explored by several works[1][2], but this paper neither discusses the methodology difference nor compares the experimental performance. Besides, some influential technical reports[3] have pointed out that moving SFT samples to CPT cannot help the final model performance, since they will eventually go through the SFT and preference alignment stages. I wonder how the authors consider this problem.\n2. I'm not convinced whether general alignment training samples can truly teach LLMs to utilize domain knowledge to answer professional questions. In lines 73-75, the referred two papers (RLHF and DPO) cannot support this hypothesis. \n3. The experiment setting is unreasonable. Math, code, and wiki are hard to be considered as 'domain knowledge', since they have already been exposed to LLM's pre-training corpus and take a considerable proportion.  The 'knowledge injection' should involve specific domains[1][2] like medicine, finance, etc.\n\n[1] Adapting large language models via reading comprehension. 2023.\n\n[2] Instruction pre-training: Language models are supervised multitask learners, 2024.\n\n[3] Deepseek llm: Scaling open-source language models with longtermism. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)