Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment

## Abstract

Abstract Adapting general large language models (LLMs) to specialized domains presents great challenges due to varied data distributions. This adaptation typically requires continual pre-training on massive domain-specific corpora to facilitate knowledge memorization, followed by training to apply this knowledge following human instructions and preferences. However, this method may result in inefficient knowledge memorization due to a lack of awareness of knowledge utilization and imposes substantial demands on LLMs to simultaneously learn knowledge utilization and format alignment with limited training samples.
To facilitate the domain adaptation of LLM, we revise this process and propose a new domain adaptation framework including domain knowledge learning and general format alignment, called Mix-CPT . Specifically, we first conduct knowledge mixture continual pre-training that concurrently focuses on knowledge memorization and utilization, allowing for mutual reinforcement. To avoid catastrophic forgetting during the continual pre-training process, we further incorporate a logit swap self-distillation constraint. Subsequently, leveraging the knowledge and capabilities acquired during continual pre-training, we efficiently perform instruction tuning and alignment with a few general training samples to achieve format alignment.
Extensive experiments demonstrate that our proposed Mix-CPT framework can simultaneously improve the task-solving capabilities of LLMs on the target and general domains compared to the traditional adaptation methods.

## 1 Introduction

Large Language Models (LLMs) (Zhao et al., ) have revolutionized the field of natural language processing (NLP) (Brown et al., ; OpenAI, ), showing exceptional capabilities such as instruction following (Ouyang et al., ; Taori et al., ) and complex reasoning (Wei et al., ; Wang et al., ).
However, due to their limited exposure to relevant data, such general LLMs still considerably lag behind in specific domains requiring professional knowledge. This situation has necessitated the effective adaptation of general-purpose LLMs to specific domains (*e.g.,* mathematics and code), called *domain adaptation* of LLMs (Guo & Yu, ).

In essence, tailoring general LLMs to specific domains requires adaptation in two main aspects, namely *knowledge learning* (acquiring and leveraging the necessary domain knowledge) and *format alignment* (responding to the user in an expected output form) (Jiang et al., ; Zhou et al., ). Specially, knowledge learning can be further fulfilled via knowledge memorization and utilization. In practice, domain adaptation of LLMs typically involves three consecutive stages (Rozière et al., ; Azerbayev et al., ), *i.e.,* pre-training, instruction tuning, and alignment, where the first stage is primarily aimed at knowledge memorization and the other two stages are mainly focused on knowledge utilization and format alignment. However, at the pre-training stage, knowledge memorization based on raw domain-specific corpus would be somehow inefficient without eliciting the acquired knowledge according to task goals (Jiang et al., ).
Despite that some studies incorporate instruction data for pre-training, they often rely on proprietary models to synthesize high-quality instructions at scale (Cheng et al., ; Wang et al., ), which may not be that easy without extensive fine-tuning experiences. Another issue is that learning to master knowledge utilization and format alignment in the instruction tuning and alignment stages might lead to suboptimal performance, due to the fact that the two goals can be divergent in model optimization (Ren et al., ).

Figure: Figure 1: Comparison of traditional domain adaptation approaches (top) and our proposed rescheduled domain adaptation paradigm (bottom). “[EOS]” is the special token representing the end of the document. “[ST]”, “[UT]” and “[AT]” denote the system, user, assistant chat template, repesctively.
Refer to caption: x1.png

Considering the above issues, this paper explores a new domain adaptation approach that only uses raw domain-specific corpus and general instruction or alignment data.
Our hypothesis is that the knowledge utilization capacity can be essentially learned from general instruction or alignment data, which has been also evidenced by prior studies (Ouyang et al., ; Rafailov et al., ).
In this way, we can remove the tedious instruction synthesis step from the training pipeline, since it is much easier to obtain general or mixed-domain instruction data from open resources.
Another important attempt is to enhance knowledge learning by jointly attaining both memorization and utilization of knowledge.
To implement this idea, we schedule all the instruction and alignment data at the pre-training stage (with a suitable format), then only reuse a minor proportion of instruction and alignment data for the instruction-tuning and alignment stages to achieve format alignment. We compare our rescheduled process with the traditional domain adaptation in Figure 1.

Specially, our approach for domain adaptation of LLMs consists of two main stages, including *domain knowledge learning* and *general format alignment*. In the first stage, we conduct knowledge mixture continual pre-training to integrate both knowledge memorization and utilization. The memorization of new knowledge can be facilitated by taking into account how this knowledge will be utilized. In the second stage, based on the knowledge and capabilities that are already acquired during pre-training, we perform instruction tuning and alignment in an efficient manner to achieve format alignment.
For unified training, we convert raw domain documents, instruction tuning data, and alignment data into a unified format for conducting knowledge mixture continual pre-training (*Mix-CPT*).
To avoid catastrophic forgetting in continual pre-training, we propose Logit Swap Self-Distillation (LSSD), which exchanges the predicted top-$1$ token logit with the logit of the ground-truth token, serving as the surrogate target. In this way, LSSD maintains most probabilities of the original distribution of LLMs, avoiding dramatic model update and thereby preserving original capabilities. In instruction tuning and alignment, we select a small number of *easy* instructions from the pre-training instruction set based on the perplexity scores of LLMs as criteria. These instructions have already been seen during pre-training, so that the model can mainly focus on pure style or format learning for downstream tasks.

To verify the effectiveness of our proposed Mix-CPT method, we evaluate it on both domain-specific and general tasks, including a total of seven distinctive capabilities based on 17 representative benchmarks.
For both base LLMs and chat LLMs, our approach can effectively improve their domain-specific and general performance simultaneously compared to traditional methods of first performing continual pre-training, followed by instruction tuning and alignment.

## 2 Approach

### 2.1 Overview

To adapt general-purpose LLMs to specific domains (*e.g.,* Wiki, mathematics, code), our core idea is to decouple knowledge learning and format alignment, and propose an effective two-stage domain adaptation framework for general LLMs, *i.e.,* first performing knowledge mixture continual pre-training (Section 2.2) and then performing efficient format alignment (Section 2.3). We show the overall architecture in Figure 2.

In the first stage, we conduct continual pre-training on the mixed data of raw domain-related documents, general instruction and alignment data via a unified text format. We aim to utilize general instruction data to better elicit the capacities of knowledge memorization and utilization during continual pre-training.
To avoid catastrophic forgetting in pre-training, we design a new learning method, *i.e.,* Logit Swap Self-Distillation (LSSD), that exchanges the top-$1$ token logit with the ground-truth token logit. In the second stage, based on the domain knowledge augmented LLMs, we conduct efficient format alignment with a small number of easy instructions or preference samples that have been seen during pre-training. In this way, LLMs can focus on learning the simple style and formet for interacting with human, without much consideration of how to utilize the attained knowledge. Next, we will describe each part in detail.

Figure: Figure 2: The illustration of our proposed rescheduled domain adaptation paradigm, including first conducting knowledge mixture continual pre-training, then selecting top-$K$ easy training samples with the lowest perplexity for performing supervised fine-tuning and direct preference optimization.
Refer to caption: x2.png

### 2.2 Knowledge Mixture Continual Pre-training

Different from prior work that performs continual pre-training solely based on domain-specific corpus (Que et al., ; Ke et al., ), we propose to mix domain-specific documents, general instructions and alignment data as pre-training data.
The QA-based instruction data is useful to reflect how the knowledge will be accessed and utilized through questions.
In this way, LLMs can improve their capabilities to learn new knowledge from those domain-related documents.
Incorporating general instructions can facilitate LLMs to transfer the general knowledge utilization capability to domain-specific knowledge without relying on highly domain-related instructions in previous work (Jiang et al., ; Cheng et al., ).
Specifically, we first transform the raw domain documents, general instructions and alignment data into a unified knowledge format. Then, we perform continual pre-training on this mixture collection with the objective of next token prediction. To avoid catastrophic forgetting, we further introduce a logit swap self-distillation approach during the continual pre-training process. Next, we introduce these techniques in detail.

#### 2.2.1 Unified Knowledge Format

Typically, adapting a base LLM to a specific domain involves three distinct and relatively independent stages, each based on corresponding data in different formats.
Specifically, the base model firstly performs continual pre-training (CPT) on domain-specific corpus for learning new knowledge, then conducts supervised fine-tuning (SFT) based on instructions for enhancing the instruction following ability, and finally utilizes the human preference data for human alignment.
In this work, we adopt the direct preference optimization (DPO) to as the alignment algorithm.
Formally, we denote the domain-specific corpus as $\mathcal{D}_{\text{CPT}}=\{{d_{i}}\}_{i=1}^{n_{c}}$, where ${d_{i}}$ represents a raw domain document consisting of a sequence of tokens. For the instructions used in SFT, we denote as $\mathcal{D}_{\text{SFT}}=\{\langle q_{i},r_{i}\rangle\}_{i=1}^{n_{s}}$, where ${q_{i}}$ and ${r_{i}}$ represent the user query and the expected response, repectively. For alignment data used in DPO, we denote by $\mathcal{D}_{\text{DPO}}=\{\langle q_{i},r^{+}_{i},r^{-}_{i}\rangle\}_{i=1}^{n
_{d}}$, where ${q_{i}}$, ${r}^{+}_{i}$, and ${r}^{-}_{i}$ represent the user query, positive response, and negative response, respectively.

In this work, we propose to mix $\mathcal{D}_{\text{CPT}}$, $\mathcal{D}_{\text{SFT}}$ and $\mathcal{D}_{\text{DPO}}$ with a unified text format, building upon which we further perform knowledge mixture continual pre-training on a general LLM. Unlike previous work relying on synthesizing high-quality domain instructions (Jiang et al., ; Cheng et al., ), we empirically find that knowledge utilization
is indeed a general capability that can be learned from general instructions, and we can further transfer such capacity to enhance the learning of domain knowledge.
Specially, we remove any templates and markers (*e.g.,* [User]) from the instructions and alignment data to construct the mixture data in a unified format, denoted by $\mathcal{D}_{\text{MIX}}=\{x_{\text{cpt}},x_{\text{sft}},x_{\text{dpo}}\}$, where $x_{\text{cpt}}=d_{i}$ is the original domain document, $x_{\text{sft}}=[{q_{i}};{r_{i}}]$ denotes the concatenation of user query and expected response in instructions, and $x_{\text{dpo}}=[{q_{i}};{r}^{+}_{i}]$ is the concatenation of user query and positive response in the alignment data. We show some examples in Figure 1.

Following existing pre-training methods (Touvron et al., ), during the continual pre-training process, we concatenate each kind of data sample (*i.e.,* $x_{\text{cpt}}$, $x_{\text{sft}}$, or $x_{\text{dpo}}$) and truncate the sequence when reaching the maximum input length of the LLM. Besides, we add an extra special symbol (*i.e.,* [SEP]) at the end of each sample to separate them. We repeat this process until concatenating all samples from each kind of data (*i.e.,* $\mathcal{D}_{\text{CPT}}$, $\mathcal{D}_{\text{SFT}}$ and $\mathcal{D}_{\text{DPO}}$) to obtain our final knowledge mixture pre-training data $\mathcal{D}_{\text{MIX}}$.

Note that though we use general instruction data to derive the mixture data here, our approach can be generally extended to incorporating domain-specific instruction data (Cheng et al., ), which often relies on specific data synthesis techniques.

#### 2.2.2 Logit Swap Self-Distillation

After obtaining the mixture data $\mathcal{D}_{\text{MIX}}=\{x_{\text{cpt}},x_{\text{sft}},x_{\text{dpo}}\}$, we then perform continual pre-training on the base LLM. For simplicity, we remove the subscript of each training sample in the mixture data, denoted as $x$. We adopt the pre-training task of next token prediction (NTP), which aims to predict the next token conditioned on all previous tokens. Specifically, given an input $x=\{w_{1},w_{2},...,w_{n}\}$, we feed it into the decoder-only LLM and use the standard language modeling objective to minimize the cross-entropy loss as follows:

$$ $\displaystyle\mathcal{L}_{\text{NTP}}=-\sum_{j=1}^{n}\log\text{Pr}(w_{j}|w_{<j };\Theta),$ (1) $$

where $w_{j}$ denotes the $j$-th token in the input, $w_{<j}$ is the previous tokens, and $\Theta$ denotes the model parameters. During continual pre-training, the task of next-token prediction enables the base LLM to learn domain knowledge, and the incorporation of general instruction and alignment data further transfers the general knowledge utilization capability to specific domains.

However, the traditional language modeling objective is prone to suffer from the issue of catastrophic forgetting for previously learned knowledge of LLMs. Therefore, we propose an auxiliary training objective, *i.e.,* *Logit Swap Self-Distillation (LSSD)*, which serves as an extra constraint for standard language modeling objective. Specifically, we first utilize the original base LLM before continual pre-training (paramerized by $\Theta_{\text{ori}}$) to infer the output logits following the standard language modeling objective yet without computing the loss:

$$ $\displaystyle\bm{h}_{j}$ $\displaystyle=\text{LLM}(w_{<j};\Theta_{\text{ori}}),$ (2) $\displaystyle\bm{l}_{j}$ $\displaystyle=\bm{h}_{j}\bm{W}_{e}^{T},$ (3) $$

where $\bm{W}_{e}$ is the token embedding matrix, $\bm{h}_{j}$ is the hidden state of the last transformer block, and $l_{j}$ denotes the output logit at the $j$-th position.
Then, we exchange the logit value of the top-$1$ predicted token (*i.e.,* $\widetilde{w}_{j}$) and the ground-truth token (*i.e.,* $w_{j}$) if they are not equal as follows:

$$ $\displaystyle\bm{\tilde{l}}_{j}=\text{Exchange}(\bm{l}_{j},I_{\widetilde{w}_{j }},I_{w_{j}}),\quad\text{if}~{}~{}I_{\widetilde{w}_{j}}\neq I_{w_{j}},$ (4) $$

where $I_{\widetilde{w}_{j}}$ and $I_{w_{j}}$ denote the indices of the top-$1$ predicted token $\widetilde{w}_{j}$ and the ground-truth token $w_{j}$ in the vocabulary, respectively, the function Exchange($\cdot$) will exchange their logit values in $\bm{l}_{j}$, and the output $\bm{\tilde{l}}_{j}$ will be regarded as the teacher logit in LSSD. In essence, LSSD only calibrates the prediction of ground-truth token for adapting to the current domain knowledge while maintaining the most previously learned knowledge of the base LLM (*i.e.,* represented by the unchanged logit values in $\bm{\tilde{l}}_{j}$).
Then, we can compute the teacher model’s probability distribution for the $j$-th token with softmax function:

$$ $\displaystyle\text{Pr}(w_{j}|w_{<j};\Theta_{\text{ori}})=\text{softmax}(\bm{ \tilde{l}}_{j}),$ (5) $$

Finally, we compute the self-knowledge distillation objective and minimize the reverse Kullback-Leibler divergence loss (Gu et al., ) between the current model’s probability distribution and the teacher model’s probbaility distribution as follows:

$$ $\displaystyle\mathcal{L}_{\text{LSSD}}=-\sum_{j=1}^{n}\sum_{w_{j}\in\mathcal{V }}\text{Pr}(w_{j}|w_{<j};\Theta)\log(\frac{\text{Pr}(w_{j}|w_{<j};\Theta)}{ \text{Pr}(w_{j}|w_{<j};\Theta_{\text{ori}})}),$ (6) $$

where $\mathcal{V}$ is the vocabulary and $\Theta$ denotes the parameters of the current LLM. In the knowledge mixture continual pre-training stage, the final total loss is the combination of next token prediction loss and self-distillation loss as follows:

$$ $\displaystyle\mathcal{L}_{\text{CPT}}=\alpha\cdot\mathcal{L}_{\text{NTP}}+(1- \alpha)\cdot\mathcal{L}_{\text{LSSD}},$ (7) $$

where $\alpha$ is a coefficient to control the proportion of two parts.

### 2.3 Efficient Format Alignment

In the domain knowledge learning stage, the LLM has simultaneously learned to both memorize domain knowledge and understand how to utilize the knowledge through our proposed knowledge mixture continual pre-training. After that, during the format alignment stage, the LLM can more efficiently fine-tuned to master the task format with only a small number of alignment samples. Next, we first introduce the selection of training samples and then perform general format alignment.

Since we would like to decouple knowledge learning and format alignment, we mainly focus on training samples from $\mathcal{D}_{\text{SFT}}$ and $\mathcal{D}_{\text{DPO}}$ that are are both *easy* and have been *encountered* during continual pre-training, which avoids introducing new knowledge during supervised fine-tuning.
These easy samples are selected based on the perplexity scores of the LLMs *w.r.t* the ground-truth output. Specifically, given a sample in $\mathcal{D}_{\text{SFT}}$ and $\mathcal{D}_{\text{DPO}}$, we first equip the input instruction and output response with the corresponding chat template, *i.e.,* the format for interaction with humans. Then, we feed the formatted sequence into the LLM and compute the perplexity score for the output response. Finally, we select top-$K$ samples with the lowest perplexity scores to conduct the supervised fine-tuning and direct preference optimization. Note that for samples in $\mathcal{D}_{\text{DPO}}$, we only utilize the positive response for computing its perplexity score.

After selecting the training samples, we next utilize them to conduct efficient format alignment. Firstly, we utilize the selected easy instruction samples from $\mathcal{D}_{\text{SFT}}$ to perform supervised fine-tuning based on the LLM after continual pre-training following the standard way (Ouyang et al., ), which is to minimize the cross-entropy loss:

$$ $\displaystyle\mathcal{L}_{\text{SFT}}=-\sum_{j=1}^{n}\log\text{Pr}(r_{j}|q,r_{ <j};\Theta),$ (8) $$

where $r_{j}$ and $r_{<j}$ denote the $j$-th token and its previous tokens in the response.
Secondly, we further utilize the selected easy preference samples from $\mathcal{D}_{\text{DPO}}$ to conduct direct preference optimization following its original method (Rafailov et al., ) as follows:

$$ $\displaystyle\mathcal{L}_{\text{DPO}}=-\log\sigma\bigg{(}\beta\log\frac{\pi(r^ {+}|q;\Theta)}{\pi(r^{+}|q;\Theta_{\text{ref}})}-\beta\log\frac{\pi(r^{-}|q; \Theta)}{\pi(r^{-}|q;\Theta_{\text{ref}})}\bigg{)},$ (9) $$

where $\sigma$ denotes the sigmoid function, $\Theta$ and $\Theta_{\text{ref}}$ denote the parameters of the updated LLM and reference LLM during the direct preference optimization process, and $\pi$ denotes the product of the probabilities of all output tokens, conditioned on the given input.

## 3 Experiments

### 3.1 Experimental Setup

Domain-specific Corpus.
In our experiments, we mainly focus on three popular domains for adapting general-purpose LLMs, *i.e.,* encyclopedia, mathematics, and code. For the encyclopedia domain, we select Wikipedia as the primary corpus, which is collaboratively developed by volunteers globally and can be freely accessed online. To enable LLMs to learn knowledge from new documents, we utilize the official 2024/03/01 Wikipedia dump(^1^11https://dumps.wikimedia.org/enwiki/20240301/) and conduct necessary data cleaning and filtering processes such as deduplication, resulting in approximately 4B tokens in raw Wikipedia documents. For the domain of mathematics, we opt for AutoMathText (Zhang et al., ), a carefully curated corpus derived from various sources including websites, arXiv, and GitHub. Each sample in this corpus has been labeled with a quality score from 0.0 (“the poorest”) to 1.0 (“the best”), reflecting its relevance, quality, and educational value in the context of mathematical intelligence. Following previous work (Zhou et al., ), we specifically select those samples with scores higher than 0.7, containing about 0.7B tokens. For the field of code, we select the StarCoder (Li et al., ) corpus, which is widely recognized and employed in several studies (Luo et al., ). It contains 86 programming languages, and we select the Python subset with approximately 1B tokens.

General Instruction Datasets.
For general instruction datasets, we choose TULU-V2-mix (Ivison et al., ) and UltraFeedback (Cui et al., ) for instruction tuning and alignment, respectively. Specifically, each sample in TULU-V2-mix is either manually curated for quality or generated from GPT models for encouraging complexity and diversity. We utilize the entire dataset of TULU-V2-mix (about 326K samples) mixed with domain-specific corpus for knowledge mixture continual pre-training (Section 2.2), and then randomly select 10K samples with the lowest perplexity score for subsequent instruction tuning. In addition, UltraFeedback is a widely-used diverse human preference alignment dataset, containing approximately 64K preference pairs. Similarly, we employ the whole dataset of UltraFeedback for knowledge mixture continual pre-training and then downsample 5K pairs with the lowest perplexity score for alignment. It is noted that our TULU-V2-mix and UltraFeedback datasets are open-source and widely used in previous work (Meng et al., ; Hu et al., ), ensuring a high level of transparency and facilitating fair experimental comparisons.

Baselines. In the experiments, we employ the Meta-Llama-3-8B (*LLaMA3-8B*) (^2^22https://llama.meta.com/llama3/) as the base model, which is extensively utilized in the existing work (Cheng et al., ). For comparative analysis, we consider the following three types of baseline methods:

$\bullet$ *Closed-Source Chat LLMs* consist of the official Chat LLMs that have undergone both instruction tuning and preference alignment using closed-source data. Here, we select the Meta-Llama-3-8B-Instruct (*LLaMA3-8B-Chat*).

$\bullet$ *Open-Source Chat LLMs* are developed by us following the processes of instruction tuning and alignment. Based on our selected base LLM, we conduct supervised fine-tuning (SFT) using TULU-V2-mix, followed by direct preference optimization (DPO) with UltraFeedback.

$\bullet$ *Continual Pre-training Augmented LLMs* include domain knowledge-enhanced Chat LLMs which initially undergo continual pre-training (CPT) with domain-specific corpus, followed by the same implementation of supervised fine-tuning (SFT) and direct preference optimization (DPO) using TULU-V2-mix and UltraFeedback as open-source chat LLMs.

Evaluation Benchmarks and Metrics. For a comprehensive evaluation, we evaluate seven distinctive capabilities of LLMs based on a total of 17 representative NLP datasets:

$\bullet$ *Factual Question Answering* assesses the factual knowledge of LLMs in the Wikipedia domain. We employ NaturalQuestion (NQ) (Kwiatkowski et al., ), TrivialQA (TQ) (Joshi et al., ), and our curated WikiQA (WQ) datasets and use the *Exact Match (EM)* metric to determine if the prediction is the same as the gold answer.

$\bullet$ *Math Reasoning* tests the LLMs’ ability to solve mathematical problems. We use the GSM8K (Cobbe et al., ) and MATH (Hendrycks et al., ) datasets, and evaluate predictions using the *Accuracy* metric.

$\bullet$ *Code Reasoning* tests the LLMs’ ability to solve programming problems. We use MBPP (Austin et al., ) and HumanEval (Chen et al., ) datasets, with the *Pass@K* metric assessing the likelihood that at least one of the top-$K$ generated code samples for a problem passes the unit tests.

$\bullet$ *Reading Comprehension* measures the LLMs’ ability to comprehend a passage and answer related questions. We use the RACE-Hard (Lai et al., ) and OpenBookQA (Mihaylov et al., ) datasets, and employ the *Exact Match (EM)* metric.

$\bullet$ *Commonsense Reasoning* evaluates the ability to answer questions using commonsense knowledge. We use the HellaSwag (Zellers et al., ), CSQA (Talmor et al., ), and PIQA (Bisk et al., ) datasets, and employ the *Accuracy* metric.

$\bullet$ *Examination* includes comprehensive and challenging benchmarks designed to assess problem-solving ability across various domains. We use MMLU (Hendrycks et al., ), BBH (Suzgun et al., ), and ARC-Challenge (Bhakthavatsalam et al., ) for English examinations and C-EVAL (Huang et al., ) for Chinese. Both benchmarks are evaluated using the *Accuracy* metric.

$\bullet$ *Instruction Following* assesses the LLMs’ ability to engage in coherent, informative, and engaging conversations. We use the MT-Bench (Zheng et al., ) datasets. For evaluation, we utilize the GPT-4-0613 (^3^33https://platform.openai.com/docs/models) as the judging model, assigning a score ranging from 1 to 10 to the answer. We multiply this score by ten, resulting in a final score of 100.

Specifically, we evaluate the above datasets based on the OpenCompass framework (Contributors, ), which is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation.

### 3.2 Main Results

Table 1 and Table 2 display the evaluation results of our proposed Mix-CPT framework and other baselines using specific evaluation benchmarks. In the next, we give a detailed analysis.

#### 3.2.1 Results of Base LLMs

We initially assess the effectiveness of our proposed Mix-CPT framework, especially the logit swap self-distillation constraint, in mitigating catastrophic forgetting and facilitating the knowledge learning of base LLMs during the continual pre-training stage.
To better observe the impact of knowledge mixture continual pre-training on the performance in target domains and general capabilities, we select domain-specific tasks and comprehensive examination tasks.
Specifically, we evaluate the domain-specific tasks (*e.g.,* factual question answering (Wiki), math reasoning (Math), and code reasoning (Code)) under few-shot settings and the Chinese and English examination tasks (*i.e.,* multi-choice question answering) with perplexity-based zero-shot setting. We show the evaluation results in Table 1.

First, we can see that traditional continual pre-training (*i.e.,* + CPT) on the raw domain data does not necessarily enhance the performance of base LLMs in the target domain, and may instead impair their performance therein. Additionally, this method inevitably leads to a certain degree of catastrophic forgetting, thereby damaging the overall performance of the base LLM.
For example, when compared to the LLaMA3-8B-Base model, performing continual pre-training in the Wiki or Math domains leads to observed improvements in the corresponding target domains (*i.e.,* 50.48 $\rightarrow$ 50.79 in the Wiki domain and 33.17 $\rightarrow$ 37.72 in the Math domain). However, this approach is inefficient in the Code domain, where it results in diminished performance (*i.e.,* 58.75 $\rightarrow$ 55.64 in Code domain).
The phenomenon is also discovered in existing work (Lin et al., ), which suggests that the data used for continual pre-training should be of superior quality relative to the data utilized during the initial pre-training phase; otherwise, it may detrimentally affect performance. Indeed, the publicly available domain data we utilized (*e.g.,* AutoMathText and StarCoder) has likely already been used to train the base model. Consequently, further employing it for continual pre-training may impair the model’s performance.

Second, when mixing the domain raw data with the additional instructions and alignment data (*i.e.,* Mix-CPT w/o KD), the domain-specific capability can be further improved, which indicates that the mixed instruction data can benefit the learning of the domain knowledge during continual pre-training. At the same time, it can mitigate the effect of other general capabilities and reduce the degradation of the overall average performance.
For example, compared to the traditional continual pre-training method (CPT), mixing domain corpus with additional instruction data (Mix-CPT w/o KD) can almost consistently improve the domain capability and overall average performance (*i.e.,* 55.64 $\rightarrow$ 57.98 and 43.94 $\rightarrow$ 45.16 in the target Code domain and overall average performance).

Finally, through applying the logit swap self-distillation strategy to the knowledge mixture continual pre-training process (*i.e.,* Mix-CPT), we can further reduce the impact on the pre-learned knowledge for the base LLM while maintaining the domain capability improvement obtained from simple knowledge mixture (*i.e.,* Mix-CPT w/o KD), thereby mitigating the degradation of the general capabilities of LLMs. Therefore, these results demonstrate that the Mix-CPT framework with the logit swap self-distillation constraint can indeed promote knowledge learning and alleviate the issue of catastrophic forgetting to some extent.

**Table 1: Evaluation results on three specialized domains and two comprehensive examination domains. The underline and bold fonts denote the best results in the target domain and the average performance in each domain adaptation group, respectively.**
| Model | Wiki | Math | Code | English<br>Examination | Chinese<br>Examination | Average |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA3-8B-Base | 50.48 | 33.17 | 58.75 | 50.93 | 47.95 | 45.72 |  |
| Wiki | + CPT | 50.79 | 34.02 | 15.56 | 52.79 | 46.27 | 42.00 |
| + Mix-CPT (*w/o KD*) | 52.47 | 39.14 | 57.20 | 52.25 | 47.64 | 47.59 |  |
| + Mix-CPT | 52.25 | 38.13 | 56.42 | 52.58 | 49.83 | 47.73 |  |
| Math | + CPT | 50.60 | 37.72 | 54.47 | 44.13 | 46.78 | 44.06 |
| + Mix-CPT (*w/o KD*) | 50.60 | 37.73 | 54.47 | 43.47 | 48.77 | 44.78 |  |
| + Mix-CPT | 50.70 | 37.67 | 57.98 | 51.45 | 49.93 | 47.51 |  |
| Code | + CPT | 46.14 | 33.71 | 55.64 | 51.38 | 47.58 | 43.94 |
| + Mix-CPT (*w/o KD*) | 50.09 | 37.85 | 57.98 | 46.82 | 47.91 | 45.16 |  |
| + Mix-CPT | 50.73 | 38.01 | 59.14 | 51.49 | 47.87 | 46.90 |  |

#### 3.2.2 Results of Chat LLM

Subsequently, we assess the performance of final chat LLMs after performing format alignment with selectively chosen training samples for instruction tuning and alignment. We want to examine the instruction following capabilities of these LLMs using comprehensive benchmarks in a zero-shot scenario. Accordingly, we assess both domain-specific tasks (*i.e.,* factual question answering (Wiki), math reasoning (Math), and code reasoning (Code)) and more general tasks (*i.e.,* Reading Comprehension (RC), Commonsense Reasoning (CR), Examination (EX), and Instruction Following (IF)). We show the final results in Table 2.

First, the traditional domain adaptation method (*i.e.,* CSD consisting of CPT, SFT, and DPO), faces challenges in simultaneously enhancing domains-specific capabilities while preserving general capabilities, in contrast to
open-source chat models that do not utilize domain-specific raw data.
For example, compared to open-source LLaMA3-8B chat model in the Wiki domain, traditional methods of conducting an extra continual pre-training based on raw Wikipedia documents not only decreases the factual question answering performance (*i.e.,* 26.10 $\rightarrow$ 22.20 in Wiki domain), but also hurts the average performance (*i.e.,* 52.57 $\rightarrow$ 50.96 in overall average performance).
In the Math and Code domains, despite obtaining improvements in the target domains (*i.e.,* 30.83 $\rightarrow$ 37.35 in Math domain and 42.53 $\rightarrow$ 48.55 in Code domain), the overall capability of LLMs exhibited a decline (*i.e.,* 52.57 $\rightarrow$ 52.10 after adapting to the Math domain).
This phenomenon can also be observed in the other two LLMs, which indicates that conventional domain adaptation methods of performing continual pre-training on raw domain data may cause catastrophic forgetting and merely focus on knowledge memorization without considering how to utilize the learned knowledge, which might suffer from the memorization trap.

Second, our proposed Mix-CPT method can simultaneously improve the performance of the target domains and the general capability. The main reasons are two fold. On one hand, with the constraint of logit swap self-knowledge distillation during continual pre-training, the LLM can effectively memorize the raw domain data while maintaining its originally learned knowledge in the previous pre-training stage. On the other hand, by mixing the raw domain data with the general instruction and alignment data (removing any templates), the model can learn the general knowledge utilization capability and transfer this capability to utilize the raw domain data. In this way, the model can perform efficient format alignment with only a few formatted samples to better utilize both target domain knowledge and other general knowledge.

Finally, with the same instruction data and alignment data, our method can successfully improve the performance on the target domain while maintaining the general capability compared to the traditional continual pre-training augmented method based on the raw domain data. Compared to the official chat LLM with large-scale closed-source instruction tuning and alignment tuning, our proposed method can achieve comparable even better performance on the target domain (*e.g.,* 40.28 vs 44.09 in the Wiki domain), which also indicates the effectiveness of our method.

**Table 2: Evaluation results on three domain capabilities (*i.e.,* Wiki, Math, and Code) and four general capabilities (*i.e.,* Reading Comprehension, Complex Reasoning, EXamination, and Instruction Following) with the average performance. The underline and bold fonts denote the best results in the target domain and the average performance in each domain adaptation group, respectively.**
| Model | Specialized Domain | General Domain | Average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Wiki | Math | Code | RC | CR | EX | IF |  |  |  |
| LLaMA3<br>8B-Base | Closed-Source Chat | 40.28 | 50.52 | 61.54 | 77.23 | 76.59 | 63.60 | 81.04 | 62.62 |
| Open-Source Chat | 26.10 | 30.83 | 42.53 | 73.90 | 75.57 | 56.44 | 68.50 | 52.57 |  |
| Wiki | + CSD | 22.20 | 30.39 | 40.73 | 74.21 | 73.66 | 56.22 | 63.15 | 50.96 |
| + Mix-CPT (ours) | 44.09 | 31.18 | 48.53 | 70.07 | 68.85 | 57.47 | 70.72 | 55.23 |  |
| Math | + CSD | 12.65 | 37.35 | 44.19 | 74.34 | 77.80 | 58.38 | 69.02 | 52.10 |
| + Mix-CPT (ours) | 44.35 | 35.99 | 52.55 | 72.88 | 72.91 | 61.64 | 71.31 | 58.38 |  |
| Code | + CSD | 17.27 | 38.13 | 48.55 | 74.75 | 77.82 | 59.21 | 70.88 | 53.87 |
| + Mix-CPT (ours) | 42.17 | 30.70 | 53.91 | 72.60 | 71.57 | 60.44 | 71.52 | 56.99 |  |

### 3.3 Detailed Analysis

Figure: Figure 3: (Left) Pass@K on code and Average results *w.r.t.* Proportion of code data. (Right) Accuracy on math and Average results *w.r.t.* Quality score of math data.
Refer to caption: x3.png

Figure: Figure 5: Accuracy on math and Average results *w.r.t.* selection strategy.
Refer to caption: x5.png

In this section, we conduct a detailed analysis of the proposed method. Specifically, the investigation focuses on examining the impact of several factors, including the quantity and quality of raw domain data and the selection criteria of samples for format alignment, on the performance of the final chat LLMs. The evaluation employs the same benchmarks as those presented in Table 2.

Effect of Quantity and Quality of Raw Domain Data. We further explore the effect of the quantity and quality of the raw domain documents on the model performance. Considering that the quality of the original StarCoder dataset is approximately consistent, we first utilize it to explore the effect of the amount of domain documents on the domain-specific and general capabilities under similar quality. Moreover, we utilize the AutoMathText dataset to explore the effect of the quality of raw domain data on LLMs’ performance by leveraging the annotated quality score for each sample in the AutoMathText dataset.
Specifically, we conduct two group experiments utilizing the StarCoder and AutoMathText datasets as follows:

$\bullet$ *Proportion of Code Data*: This group aims to compare the variants using different proportions of the original StarCoder corpus, including 10%, 30%, and 50%, while maintaining constancy in other variables.

$\bullet$ *Quality of Math Data*: This group aims to compare variants by employing various AutoMathText corpora, each characterized by distinct quality scores with thresholds exceeding 0.5, 0.6, and 0.7 respectively, while ensuring consistency in all other variables.

We show the results in Figure 4. We can see that increasing the amount of raw domain documents can indeed further enhance the target domain performance under the same quality. However, even though utilizing a larger number of raw domain data (*e.g.,* more math texts with $\geq 0.5$ score than those with $\geq 0.7$ score), the low quality of raw data can also deduce the LLMs’ performance regardless of the target domain or general domain, which indicates the quality of domain raw data is a priority over its amount when performing continual pre-training.

Effect of Format Alignment Data Selection. Our experiments indicate that selecting 10,000 instruction samples (about 3% of the original TULU-V2-mix) and 5,000 alignment samples (3% of original UltraFeedback) with the lowest perplexity scores can effectively perform format alignment during supervised fine-tuning and direct preference optimization. Here, we conduct a further ablation study to explore the impact of different selection strategies of format alignment data on the final chat model, which consists of the difficulty and the amount of selected data. Specifically, we conduct three groups of experiments including:

$\bullet$ *Difficulty of Samples for SFT and DPO*: This group compares four distinct selection strategies: random selection (R), easiest samples with the lowest perplexity (E), hardest samples with the highest perplexity (H), and half easiest samples and half hardest samples (EH).

$\bullet$ *Amout of Samples for SFT and DPO*: This involves four variants using different quantities of easiest samples from the original TULU-V2-mix dataset (*i.e.,* 10K, 20K, 40K, and 80K) and from the original UltraFeedback dataset (*i.e.,* 5K, 10K, 20K, and 40K).

$\bullet$ *Ratios between SFT and DPO Samples*: This experiment utilizes a total of 15,000 samples for format alignment by adopting five different ratios between samples used in supervised fine-tuning and direct preference optimization, ranging from 1:2, 1:1, 2:1, 3:1, to 4:1.

We show the results of each group in Figure 4, Figure 6, and Figure 6. Firstly, using the easiest samples with the lowest perplexity can balance the domain capability and general capability best compared to other selection strategies. Secondly, it enhances both the domain and general capabilities simultaneously to a certain extent by continuously increasing the amount of SFT training samples. Conversely, this phenomenon is not observed when continuously increasing the amount of DPO training samples, rather, both remain in a state of fluctuation. Finally, we can see that the proportion of SFT and DPO data we selected (*i.e.,* 2:1) can optimally balance the general and domain-specific capabilities.

## 4 Related Work

Domain Adaptation of LLMs. Our work is closely related to efforts in adapting general LLMs to specific domains (Yildiz et al., ; Ke et al., ; Scialom et al., ). Due to the increasing scale and complexity of LLMs, training domain-specific LLMs from scratch involves significantly high financial and ecological costs (Luccioni et al., ). To address this issue, recent work has been devoted to studying efficient approaches like continual pre-training, which involves incrementally training general LLMs based on new domain corpora (Que et al., ; Ke et al., ), and continual fine-tuning, aiming to fine-tune general LLMs on a series of downstream tasks related to target domains (Razdaibiedina et al., ; Scialom et al., ; Luo et al., ). Specially, continual pre-training updates LLMs with large and unlabeled domain-specific corpora, which mainly focuses on memorizing and injecting new knowledge into the parameters of LLMs. However, these approaches might result in catastrophic forgetting and performance degradation in general language tasks (Kar et al., ; Mehta et al., ). Another line of work has explored conducting instruction fine-tuning by synthesizing domain-related instructions (Cheng et al., ; Jiang et al., ). Nevertheless, these studies require additional models to synthesize amounts of instructions highly related to specific domains, resulting in high computational costs. It is noted that our method differs from these works in several ways. Firstly, we disentangle domain adaptation into knowledge memorization and capability elicitation, focusing on learning domain-specific knowledge and solving domain tasks with learned knowledge, respectively. Secondly, we employ token swap self-distillation in the knowledge mixture pre-training to retain general knowledge and avoid catastrophic forgetting.

Instruction Tuning and Alignment. Instruction tuning (also known as supervised fine-tuning) employs human-annotated instructions (Sanh et al., ; Mishra et al., ; Köpf et al., ; Sun et al., ) or synthetic instructions by proprietary models (Taori et al., ; Chiang et al., ; Wang et al., ) to fine-tune LLMs. Besides, alignment with reinforcement learning from human feedback (RLHF) (Ouyang et al., ) or direct preference optimization (DPO) (Rafailov et al., ) aims to align LLMs with human preference. Both instruction tuning and alignment are able to elicit knowledge from LLMs and improve their capabilities to solve downstream tasks. Recent work (Zhou et al., ) has demonstrated that LLMs mainly learn the style or format for interacting with users through simple instruction tuning and alignment, by leveraging their prior knowledge and capabilities already acquired during the pre-training stage. Therefore, only employing as few as 1,000 examples in supervised fine-tuning can also achieve satisfactory alignment performance (Zhou et al., ). Furthermore, by comparing the token distribution before and after alignment, recent work (Lin et al., ) found that the most significant distribution shifts appear dominantly in stylistic tokens such as transitional phrases and discourse markers instead of contextual words that involve rich knowledge for solving downstream tasks. Inspired by these studies, we propose to expose knowledge memorization and capability elicitation from instruction tuning and alignment. Unlike these studies which typically focused on instruction tuning or alignment, we differ in that we unify the three stages of training LLMs (*i.e.,* continual pre-training, instruction tuning, and alignment) and conduct a knowledge mixture pre-training to mainly focus on learning new domain knowledge while maintaining general knowledge.

## 5 Conclusion

In this study, we refined the conventional approach to domain adaptation for LLMs by introducing a novel two-stage approach, termed Mix-CPT, which encompasses both domain knowledge learning and general format alignment. Distinct from previous strategies, Mix-CPT employed a knowledge mixture of continual pre-training that learns knowledge memorization and utilization simultaneously through the integration of domain-specific raw data with general instruction tuning and alignment data. Besides, we further incorporated the Logit Swap Self-Distillation (LSSD) constraint into the continual pre-training process to relieve catastrophic forgetting. Subsequently, based on the knowledge and capabilities that are already acquired during pre-training, we strategically selected a small number of easy instructions with lower perplexity scores from the instruction set used during the continual pre-training process to make the LLM focus on learning the pure style and format for interacting with humans. We conducted extensive experiments on three benchmark datasets, and the experiment results show that our proposed Mix-CPT outperforms the traditional method, obtaining improvements on both the domain and general capabilities.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See weakness. 

And I do have another question. What’s the fundamental difference between doing all of this during pretraining versus handling each step separately? Does this suggest that there’s no need to separate these steps and that it might be better to combine everything in a single pretraining process for new domain knowledge?
soundness: 2
strengths: The paper is well-organized, and the problem it addresses is substantial. Improving the efficiency of knowledge distillation is indeed a relevant challenge that needs attention. Furthermore, the experimental results appear promising.
summary: In this paper, the author addresses improving the efficiency of LLMs in adapting to new knowledge. Here, efficiency refers to adapting the LLM without relying solely on instruction-based tuning for the new domain; instead, the author employs a combination of domain-specific and general data. Additionally, alignment is achieved during the pretraining stage through a unified format and an additional training objective.
weaknesses: 1. The paper talks about mixing CPT, SFT, and DPO data into a combined dataset for pretraining, but mixing instruction-tuning and pretraining data isn’t really new. Plus, there aren’t many details on how exactly these datasets are blended—like what proportions are used. The approach feels a bit empirical, but it would be great to have more specifics for reproducibility.

2. The author claims that the LLM learns new knowledge efficiently by understanding the task it’s meant to perform on the data, along with alignment during pretraining. I get the alignment part, but I’m a bit concerned that task-specific instruction data might limit the generalization of domain-specific data. I wonder if it’s always beneficial for new knowledge to align closely with existing world knowledge. While task performance in the new domain improves, it’s unclear if the model’s general ability is preserved—this wasn’t really explored in the experiments.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Refer to the weakness.
soundness: 3
strengths: - The proposed method is well-motivated
- The paper is well-written and easy to follow
- The proposed method outperforms the baselines
summary: The paper proposes a novel two-stage domain adaptation method for language models. The authors discuss the limitations of the existing training pipeline, which includes pre-training, instruction-tuning, and preference alignment. Based on this discussion, they propose a mix of data from each stage in an appropriate format to enable the model to learn both domain knowledge (i.e., pre-training) and knowledge utilization (i.e., instruction-tuning and preference learning). Additionally, the authors enhance the approach by training the model further on a small yet essential subset of data used in pre-training for improved model output alignment. The proposed method was evaluated with QWen2-7B and Llama-3-8B on domain data, including Wiki data, AutoMathText, and StarCoder. This method outperforms the simple CPT model.
weaknesses: - Logit swap self-distillation requires the base LLM to perform inferences on the entire training dataset. This process can be quite expensive, especially with large datasets and models.
- A concern with the experimental setup is that Wikipedia data is used for domain adaptation. However, Wikipedia is more general-purpose than domain-specific, as it covers a wide range of topics. Additionally, the base models have likely already been trained on this widely used open-source dataset, which weakens the observations drawn from the experiments.
- The authors should have compared their method to a recent, similar CPT method [1]. Method [1] has an advantage as it trains a base model on both raw documents and formatted reading comprehension data. This approach is more straightforward than the authors’ method, as it requires only single-stage training and does not necessitate the separate collection of CPT, SFT, and DPO data.
- It is unclear why the model requires a second alignment stage, as it has already been trained on formatted DPO data. Is this due to issues with the initial stage of training?

[1] Adapting Large Language Models to Domains via Reading Comprehension, ICLR 2024

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. The results indicate a dependence on the quality of domain-specific data, with lower-quality data adversely impacting performance (Fig. 3). This reliance could restrict the framework’s applicability in real-world scenarios where high-quality datasets are limited.

2. Why is the logit of the top-1 token specifically swapped with the ground truth in LSSD? Beyond the math and average measurements in Fig. 6, could you provide a more detailed explanation of why this technique is effective?

3. Could you provide an analysis of computational efficiency compared to traditional domain adaptation frameworks?
soundness: 3
strengths: 1. The paper introduces a novel view of CPT that separates domain knowledge learning from format alignment, potentially enhances both domain adaptation efficiency and task-specific performance.

2. The proposed LSSD method to avoid catastrophic forgetting is clear and straightforward, allowing the model to adapt without sacrificing previously learned information. 

3. Integrating domain-specific raw data with general instructional data during the pre-training phase is a strength, as it appears to balance both domain-specific and general knowledge effectively.
summary: The paper introduces Mix-CPT, a domain adaptation framework for LLMs that separates knowledge learning from format alignment to enhance efficiency and prevent knowledge loss. In the first stage, Mix-CPT trains on a mixture of domain-specific and general instructional data to balance domain learning and general capabilities, using Logit Swap Self-Distillation to retain prior knowledge. The second stage, format alignment, fine-tunes the model with a few selected samples to align responses to human-preferred formats. Extensive experiments and ablations have been carried out to support the proposed framework.
weaknesses: see questions.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What is the FAS score's physical meaning?  Does a higher score mean a better or worse sample? Besides, if the authors aim at selecting the samples that LLM have not currently learned the format, taking Eq.8 for an example, it seems more reasonable to compare the loss on response (r) conditioned on questions with and without chat templates, i.e., $\frac{L(r|\hat{q})}{L(r|\bar{q})}$. And Eq.9 presents the same problem.
soundness: 2
strengths: 1. The research point is important. Efficient knowledge memorization is crucial to derive both fundamental and domain-specific LLMs.
2. The idea of decoupling knowledge learning and format alignment has promising efficacy, as they serve different purposes.
3. The paper is well-written and easy to follow.
summary: This paper proposes the Mix-CPT strategy to mix knowledge memorization and utilization into the continual pre-training stage. Specifically, the authors eliminate the chat template of general instruction-following and preference-alignment data (which is traditionally used in the SFT stage with chat templates), and mix them with domain knowledge corpus for vanilla next-token prediction optimization. A logit swap self-distillation strategy is developed to mitigate catastrophic forgetting. After that, the authors derived a format alignment score to select a small number of instructions from the CPT dataset to execute SFT and DPO, in order to focus mainly on pure style or format learning for downstream tasks.
weaknesses: 1. My core concern is the lack of novelty. Taking SFT samples into the CPT stage has been explored by several works[1][2], but this paper neither discusses the methodology difference nor compares the experimental performance. Besides, some influential technical reports[3] have pointed out that moving SFT samples to CPT cannot help the final model performance, since they will eventually go through the SFT and preference alignment stages. I wonder how the authors consider this problem.
2. I'm not convinced whether general alignment training samples can truly teach LLMs to utilize domain knowledge to answer professional questions. In lines 73-75, the referred two papers (RLHF and DPO) cannot support this hypothesis. 
3. The experiment setting is unreasonable. Math, code, and wiki are hard to be considered as 'domain knowledge', since they have already been exposed to LLM's pre-training corpus and take a considerable proportion.  The 'knowledge injection' should involve specific domains[1][2] like medicine, finance, etc.

[1] Adapting large language models via reading comprehension. 2023.

[2] Instruction pre-training: Language models are supervised multitask learners, 2024.

[3] Deepseek llm: Scaling open-source language models with longtermism. 2024.

Meta-review: The paper presents Mix-CPT that conducts a knowledge mixture continual pre-training that concurrently focuses on knowledge memorization and utilization. A logit swap self-distillation constraint is used to avoid catastrophic forgetting, then performs format alignment through instruction tuning and alignment. 

1. The problem it addresses is substantial. [Reviewer tVX8]
2. The logit swap self-distillation constraint is a novel contribution. 
3. Novel view of CPT that separates domain knowledge learning from format alignment, potentially enhances both domain adaptation efficiency and task-specific performance.Integrating domain-specific raw data with general instructional data during the pre-training phase is a strength, as it appears to balance both domain-specific and general knowledge effectively. [Reviewer wQCa]. 
4. The method outperforms all baselines (in most data sets in domain specific settings) [All reviewers].

Weaknesses:
1. Wikipedia is not a domain specific data, and is not a relevant data set for performance study.
2. As indicated by reviewer Yvdt, the domain specific adaptation is not better than baselines in USMLE and PubMedQA data sets.

Decision: Accept (Poster)