DDK: Distilling Domain Knowledge for Efficient Large Language Models
Authors: Jiaheng Liu
Chenchen Zhang
Jinyang Guo
Yuanxing Zhang
Haoran Que
Ken Deng
ZhiqiBai
Jie Liu
Ge Zhang
JiakaiWang
Yanan Wu
Congnan Liu
Jiamang Wang
Lin Qu
Wenbo Su
Bo Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Knowledge Distillation
Large Language Models
Model Acceralation
Primary area: natural_language_processing

Abstract: Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands.  Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, or utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal performance gaps and insufficient attention to domains with large gaps, reducing overall performance. In this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin.

DDK: Distilling Domain Knowledge for Efficient Large Language Models
Jiaheng Liuat.1, Chenchen Zhang*? Jinyang Jinyang Guo Yuanxing Zhang", Haoran Que', Ken Deng'. Zhiqi Bai², Jie Liu", Ge Zhang Jiakai Wang", Yanan Wu², Congnan Liu Jiamang Wang². Lin Qu Wenbo Su Bo Zheng! Taooboo & Tmall Group of Alibaba, AAlibaba Group, The University of Sydney, "The Chinese University of Hong Kong, University of Waterloo [1jh411989) (a1999999lllll. aaal@alllbl Dalibaba-inee com
Abstract
Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal per- formance gaps and insufficient attention to domains with large gaps, reducing overall performance. this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the perfor- mance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by large margin.
Introduction
Recent advancements Large Language Models (LLMs) such as LLaMA [7, 9, 58, 59] have garnered significant attention due to their strong intelligence. However, these models also impose considerable computational and storage demands, partieularly in practical deployments such as instant chat, copilot, and query rewriting. Consequently, the development of lightweight yer efficacious LLMs suitable for real-world applications has become an area of increasing research interest. Several small-scale LLMs, e.g., Phi [39] and MiniCPM [32], have been designed to facilitate rapid inference on devices with limited resources. These models are generally trained from scratch using a large volume of selectively curated high-quality datasets, which could be prohibitive for the broader research community. Meanwhile, there has been a surge in the exploration model compression techniques [40] reduce the resource footprint of LLMs. Apart from these techniques, knowledge distillation (KD) emerges as a prominent method for creating effective neural networks, which transfer knowledge from high-peeforming teacher model to compact student model.
First two authors contributed equally. Corresponding Author: Jiaheng Liu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Perplexity (PPL) on Different Domains
Stadent
CPI
20
KD
DDK
PPL
Teacher
10
CommonCrawl C4
The Stack
Wikipedia
Books
ArKi
StackExchange
Chinese Books
Chinese CC
OpenWebMath
Figure I: The perplexity scores of different methods across different domains for different methods (See Section 4 for more details.). Note that "Chinese CC" denotes "Chinese CommonCrawl".
The primary challenges in enhancing the performance of KD approaches on LLMs stem from two main aspects: i) appropriately utilizing the data [3, 67]; ii) stabilize the distillation process [68]. Recently, it has become increasingly acknowledged that the mixture ratios of various domains within the training dataset substantially affect the performance [20, 67, 69]. Regarding the issue of data composition, the influence of domain-specifer mixtures for KD remains underexplored, As shown in Fig. the performance between Qwen-1.5 1.8B [7] (student) and Qwen-1.5 14B [7] (teacher) reveals that the performance gap varies significantly across domains. For instance, in the "Books" domain, student model significantly underperforms the teacher model, while "The Stack" domain, the difference is minimal, which indicates that the "Books" domain is relatively not optimized well for student model compared to the teacher model, and more data from the "Books" domain should be included. Therefore, aim to design knowledge distillation framework that dynamically adjust the data composition during distillation to reallocate more computation to domains, where the student and teacher models have larger performance gaps.
In this paper, we introduce a novel methodology, termed Distill Domain Knowledge for LLMs (DDK), which effectively optimizes domain-speciff, mixtures to address the performance discrepancy between teacher and student models across different domains. Specifically, DDK begins by quantifying the performance deviations between the teacher and student LLMs using offline-collected validation dataset across various domains. Next, it periodically re-calculates the domain discrepancy factor based on the performance gap between the teacher and student models. Finally, DDK employs a domain soowledge-gudeet sampling strategy to sample data from different domains with varying probabilities based the calculated domain discrepancy factor. Additionally, inspired by the optimization algorithms [36], we propose a factor smooth updating mechanism to augment the stability and robustness of the DDK approach. For the supervision loss, minimize the differences in the output logits between the teacher and student models. As demonstrated in Fig. 1, the performance gap across domains is significantly reduced by DDK.
Our main contributions are summarized as follows:
To the best of our knowledge, are the first study the influence of Nomain-speciffe data mixtures for distilling LLMs, and efficiently transfer the domain knowledge of the teacher network upon the domain weights.
DDK proposes a factor smooth updating strategy to strategically enhance the appropriate focus of the distillation process on targeted domains, which effectively stabilizes the domain knowledge guided sampling process for smoother distillation.
Extensive experiments on multiple benchmark datasets demonstrate the effectiveness and generalization ability our proposed DDK.
2 Related Works
Large Language Models. The emergence of LLMs [62,, 72, 24, 41, 19, 65, 49, 53, 27, 6, 66) marks a significant milestone the domain of natural language processing, with notable examples including GPT3, Lamda, Palm, and several others [2, 4, 10, 44, 57]. For example, Radford and Narasimhan

---

[50] introduced the GPT model, leveraging multiple layers of transformer decoder blocks, while Meta later developed LLaMA [58] employing an enhanced transformer architecture, subsequently evolved into ILLMA2 [59]. Recent advancements have also seen the application of instruction tuning [13, 63] and learning through human feedback [8, 45, 74] to better align L.LMs with human understanding and foster the creation of versatile Al assistants [21, 43]. Despite their potential, LLMs extensive capabilities are often accompanied by vast sizes [35, 64], demanding significant computational resources. In this work, we aim to focus on how to produce small LLMs based on the knowledge distillation approach.
Knowledge Distillation. Knowledge distillation is a pivotal technique in model compression and acceleration [61, 42, 26, 70, 25], primarily employed transfer knowledge from robust, well-trained teacher model to a compact student model [29, 1, 37]. Recently, several approaches to knowledge distillation tailored for L.LMs have been proposed. These approaches can be broadly classified into two categories: White-box KD leverages either the internal parameters or the logits of the teacher LLM during the distillation process [22, 46, 56, 71]. For example, Gu et al. [23] propose that traditional Kullback-Leibler divergence (KLD) objective is inappropriate for open text generation tasks and propose MiniLLM to minimize reverse KLD through policy gradient techniques [54]. Conversely, black-box KD relies solely on the outputs from the teacher model [12, 30, 34, 48, 60]. For example, "Distilling Step-by-Sten strategy [31] employs Chain of Thought (CoT) prompting to provide sophisticated guidance during distillation. These two types of KD approaches mainly focus on aligning the generative behaviors the teacher and student models. DDK delves into the efficacies of domain- specific distillation, aiming to mitigate the discrepancies in performance between the teacher and student model across different domains. Hence, DDK is fundamentally orthogonal these methods.
3 Methodology
3.1
Overview
Figure 2 illustrates the comprehensive architecture of the DDK framework. DDK employs a large- scale teacher LLM and a comparatively smaller student LLM, with the objective of transferring knowledge from the former to the latter to enhance performance utilizing specially curated distilla- tion dataset. Initially, the distillation dataset is constructed by randomly sampling from the training corpus. Throughout the distillation process, of both the teacher and student LLMs, and use domain knowledge guided sampling to dynamically update the data mixture the student's abilities within specific domains. As the domain proficiency of the student LLM evolves during distillation, we introduce a factor smooth updating strategy to ensure the robustness of the domain knowledge-guide sampling approach. Finally, DDK provides of a better student LLM, optimized for enhanced performance across targeted domains.
continuously assess the domain-specific performance
3.2 Domain Knowledge Guided Sampling
The distilled student LLMs are anticipated to exhibit robust competence across various preset domains. Nevertheless, prevailing knowledge distillation techniques tailored for LLMs tend to homogeneously optimize performance across these domains, leading to potential performance degradation. To address this issue, we design the domain knowledge guided sampling strategy to enhance distillation efficacy by prioritizing domain-specift complexities.
Domain discrepancy factor construction. We consider dataset D that has been partitioned into N distinct domains. We denote the pre-trained teacher LLM as My and the student model, which is currently under training, as Ms. To efficiently identify and prioritize data that may yield the most learning benefit, particularly from domains where the student model underperforms, we introduce a domain discrepancy factor denoted as r E N Each component r(i] of this vector quantitatively represents the discrepancy in performance between the teacher and student models within the i-th domain. As we assume a good student should exhibit close approximation to the teacher across all domains, is calibrated to reflect differential performance indices as follows:
i exp(ts(0)/4+10)
Σ VE(......N
(1)
where (s[ <e))(CC(.Ms(V)), Y:)) and [ = (CE( X.))
3

---

Teacher LLM
DDK: Distilling Domain Knowledge
Student LLM
Distiliation dataset
Old distillation dataset
Updated distilla tion dataset
Domain knowledge guided sampling
Code
function guaat) Google ara provites suglet.
sel Galckate seris
Tastion prottrernch
Code
Web
pabliccimm HelloWWerd
Web
Content macksting snskvo.. No penzzr anal at.
Law
Domain Discrepancy
No parsershal expreator..
Law
Teacher domain performance
foctors
Student domain performance
Law
Lon
Code
Foctor smooth updoting
Cade
MIkI
G p( gethig
(1 x//X
Wild
Figure 2: Overview of the distillation process of DDK. First, the training dataset is divided into distinct domains based on predefined criteria. Then, DDK dynamically modulates the distribution of domain-specific data, augmenting the amount allocated to domains where the student model struggles the most. The proportions attributed to each domain are recalculated distillation intervals by employing factor smooth updating approach.
Here, Vi and Y; are the inputs and the ground truth labels of the validation dataset of the ith domain. CH(-) represents the cross-entropy loss. fs E TRN and & E R. are the perplexity scores over the validation sets of all domains for student and teacher respectively, indexed by the domain index i. In this case, higher value of r[i] signifies a pronounced disparity in domain-specific proficiency between the student model and the teacher model. Accordingly, it is imperative allocate more relevant data to enhance the domain expertise.
Domain knowledge guided sampling. We employ a domain knowledge-informed sampling strategy to refine the composition of the distillation dataset, which utilizes probabilistic mechanism defined by vector r to iteratively select samples from the training corpus. The process continues cyclically once domain data has been exhausted. Finally, DDK strategically increases the data allocation towards underperforming domains, thereby mitigating the performance discrepancies between the teacher and student models across all domains.
3.3 Factor Smooth Updating
With the domain knowledge guided sampling strategy, we can dynamically focus on more challenging domains during the distillation process. Nonetheless, we observe that the domain discrepancy factor exhibits significant fluctuations throughout this procedure. Such rapid alterations may precipitate exceedingly unbalanced data sampling, potentially compromising the stability of the distillation.
Factor smooth updating. To enhance the stability of the distillation process, periodically adjust the domain discrepancy factor every K iterations throughout the distillation process, thereby partitioning it into discrete intervals. The parameter K is pivotal it governs the system's 's capacity to address immediate discrepancies and influences the stability of the data mixture. We denote the domain discrepancy factor for the i-th domain at the t-th interval of distillation r' [t]. Similarly, let [i] and [ denote the perplexity scores at the beginning of the 1-th distillation interval. In DDK, the domain discrepancy factor at the (L 1)-th interval is defined as:
+ (1 (1-a)/N,
Σ
+
i
(2)
where 2b
[expp( 1:/4 111/4
Note that constant term is incorporated in r to preclude the occurrence of excessively small values, thereby guaranteeing baseline probability for data sampling across various domains. The parameter a, designated as the smoothing coefficient, is fixed at value of 0.5 in our experimental

---

Algorithm Distillation procedure of the DDK framework.
Input: Distillation dataset D; The steps per distillation interval K;
1: Initialize domain discrepancy factor r based Eq.
2: Randomly sample D D that supports K steps distillation;
Initialize student training iteration
distillation interval
4: for each iteration in the training process do
5: Update student LLM parameters
6: Read a batch of samples and use Eq. 3 to update the parameters of student LLM;
7: c c. c+1 1
8:
if e mod K == O then
9:
Update distillation data mixture
10:
t= (. 1;
Use Eq. to update domain discrepancy factor r'; Sample dataset, D' C D, that supports K steps distillation according to r";
12:
13:
Shuffle D";
14:
if É reaches preset maximal number of intervals then Stop the distillation loop;
15:
Output:
The distilled student LLM:
addition, the inclusion of eb" imparts history mixture information on the modification of the domain discrepancy factor. This mechanism facilitates gradual modification of r' [i]. thereby minimizing fluctuations and ensuring a stable, domain knowledge-driven distillation process for fetching informative data.
setup.
3.4 Overall Optimization
As jointly update the student L.LM parameters and the domain discrepancy factor in the distillation process, the optimization object can be written follows:
min
CB(Ms(VV), Y.) -KL(SSolmmax<>((V) Softmax(2r(V)))
(E(I.....N)
where Os is the parameters of the student model. 2s(.) and 2T( are the output hidden states from student and teacher LLMs, respectively. We leverage KL-divergence to approximate the student model's output to the teacher model's output, over distillation temperature T. is the factor to balance these two terms. Algorithm summarizes the pseudo-code the distillation process is typically concluded either when all available data has been fully utilized or when the domain discrepancy factor approaches threshold indicative of minimal disparity between the teacher and student models.
DDK process. practice,
4 Experiments
In this section, make comprehensive evaluations to answer two research questions: RQ1: To what extent does the DDK process improve the performance small-scale LLM? RQ2: How does the dynamic domain-specifee guidance contribute to the overall improvement?
4.1 Experimental Setup
Model configuration details. We use the Qwen-1.5 [7] and LLaMA2 [59] to demonstrate the effectiveness of DDK. Regarding the Qwen-1.5 series, Qwen-1.5 14B and Qwen-1.5 1.8B as the teacher and student models, respectively. For LLaMA2 series, we use LLaMA2 13B and TinyLLaMA I.I [73] as the teacher and student models, respectively.
Training details. Due to the unavailability of training data for LLaMA2 and Qwen-1.5 models, we mainly utilize RedPajama [16] for distillation, which consists of training data derived from seven distinct domains: CommonCrawl, C4, The Stack, Wikipedia. Books, ArXiv, and StackExchange.
5

---

Table 1: Results of different methods the 1wen-1.5 models. Note that Qwen-1.5 14B and Qwen-1.5 1.5 1.8B as teacher and student models, respectively. "W.G.", "C.QA" and "H.E." denote Winogrande, Commonsense@Q and Humeneval datasets, respectively.
Methods
CEval
MMLU
RACE
C3
W.G.
GSMSK
C.QA
Are-E
Are-C
H.E.
MBPP
Avg.
Teacher (14B)
78.68
64.34
89.95
77.38
68.74
67.63
82.06
87.58
80.59
37,80
44.00
70.80
Student (1.88)
66
69.57
58.27
57.85
70.23
11.87
18.00
49.39
+ CPT
60.13
45.01
69.00
60.30
56.98
42.50
64.78
72.00
51.03
13.12
20.45
50.48
+ KD [29]
61.29
43.63
70.12
63.92
58.01
49.58
66.26
73.41
54.56
15.63
25.15
$2.87
TED [40]
62.04
45.21
69.95
63.18
57.38
49.28
65.27
74.74
55.00
13.75
22.69
52.59
MiniLLM [23]
61.66
45.07
68.92
63.37
57.14
48.90
64.46
74.52
53.92
16.88
23.55
52.58
-DK (Ours)
63.75
46.01
71.56
65,53
59.10
53,54
66.75
75.01
55.03
27.13
26.10
55.41
Table Results of different methods on the LLaMA models. Note that we use LLaMA2 13B and I.IB as teacher and student models, respectively.
Methods
CEval
MMLU
RACE
C3
W.G.
GSM8K
COSE-QA
Arc-E
Are-C
H.E.
MBPP
Avg.
Teacher (13B)
34.32
49.31
62.85
46.03
63.77
24.10
52.17
73.30
49.40
18,30
28.10 10
45.60
Student (B.IB)
23.92
24.89
22.92
35.24
55.49
14.19
19.08
24.18
24.12
5.62
16.58
24.20
38.91
15.03
20.39
28.06
6.88
17.35
+ KD
27.12
26.13
23.84
37.43
53.91
15.92
22.52
29.40
26.27
7,50
17.97
26.18
+ TED
27.49
26.43
24.18
37.61
55.72
14.74
22.93
28.61
25.40
8.13
17.45
26.24
MiniLLM
26.74
26.45
24.32
37.18
54,46
16.30
22.93
29.46
25.84
8.13
18.28
26.37
DDK (Ours)
27.86
28.74
27.76
42.41
57.62
17.44
25.39
36.29
30.15
9.36
19.51
29.32
Moreover, to enhance the model's proficiency in Chinese and Mathematics, we also incorporate three cleaned open-source datasets (i.e., Chinese Books [19], Chinese CommonCrawl [19], and OpenWebMatt [47]). Therefore, there are ten domain datasets for the distillation. In addition, to assess the disparity in performance between teacher and student models across the domains, have constructed domain-specific validation set for each domain, where each domain includes 500 samples. During the distillation phase, the student models are trained approximately 15B tokens. For the training framework, we employ the DeepSpeed-Chat code' as our codebase, and conduct all experiments using 16 NVIDIA A100 GPUs (80G), where FlashAttention V2 [17] is used to accelerate training. For the training schedule, we first apply the warm-up strategy to increase the learning rate from 0 3e in 1,000 steps. Then, we use the cosine learning rate schedule, where the final learning rate is 3e and the whole training step is about 30,000 steps. Empirically, we set the distillation interval K as 1,000 and the temperature T 1.0.
Evaluation details. As we do not conduct instruction tuning on the student models, we mainly report the zero-shot, close-ended results across commonly used datasets including C-Eval [33] (val), MMLU [28] (test), RACE [38] (high, test), C3 [52] (test), WinoGrande [51] (val), GSM8K [15] (test), CommonsenseQA [55] (val), Are-E [14] (test), Arc-C [14] (test) and HumanEval [11] (test). We also report the 3-shot performance on MBPP [5] (test).
Baseline details.
We compare DDK with four baseline methods:
CPT denotes that we continue to pre-train the student model by using the same number of training tokens without considering domains.
KD [29] denotes the standard knowledge distillation by computing the KLD between the teacher and student logits without considering domains.
TED [40] denotes to use task-aware filters to align the hidden representations the student and the teacher each transformer layer.
MiniLLM [23] denotes to replace the forward KI. divergence with reverse KI. divergence, which prevents the student model from overestimating the low-probability regions of the teacher distribution.
http:///ss.ccccicoooooooooocooce DeepSpeed- Chat

---

70
MMLU
70
MMLU
Arc-C
Arc-C
RACE
65
RACE
Accuracy
Accuracy
60
55
50
45
500
1000
1500
2000
5
10
15
20
25
30
Distillation interval
# Tokens (B)
(a)
(b)
Figure (a). Effect of distillation interval. (b). Effect of the number of training tokens.
4.2
Main Results
As shown in Table 1-2, we report the performance results of different baseline methods. The following observations provide a comprehensive response RQ1: (1) We see that the absence of knowledge transfer from the teacher model significantly impedes the student model's capabilities in intricate tasks such as coding ge.g., HumanEval) and Chinese comprehension (e-g. C3). (2) DDK outperforms other baseline methods when using different types of teacher and student models, which demonstrates the effectiveness of DDK for training small student LLMs. (3) The baseline methods KD, TED, and MiniLLM exhibit similar performance. For instance, the average accuracy of these three approaches hovers around 52% when distilling onto the Qwen student model. We hypothesize that the context of LLM distillation, domain data mixture may emerge as a key performance bottleneck, and the existing baseline techniques fail to adequately address this challenge. (4) The performance gains vary across different domains. Notably, when distilling the Qwen model, achieve significant improvements on the reasoning tasks (e.g., Code on Humaneval and MBPP, Math on GSM8K), which indicates that the student model improve lot the reasoning tasks under the guidance of the teacher model. This empirical observation suggests that DDK successful in directing additional attention toward the more challenging problem domains.
4.3 Ablation Study
In this section, perform ablation studies to assess the robustness of the DDK model and its sensitivity to key hyperparameters. We collected data using Qwen 1.5 and reported its performance on the validation sets of MMLU, RACE, and ARC-C, which differ from those discussed in the previous subsection. Initially, we concentrate on addressing RQI through ane-grained analyses.
Effect of data sampling strategies. We pro- pose two variants of data sampling strategies on DDK. For DDK (w/o FS), we just remove the
MMLU
70
RACE
factor smooth updating mechanism and directly
Accuracy
Arc-C
take re as the probability of each domain. For DDK (ES), we sample data from each domain
60
equally. The results are shown in Fig. 4, and we can suppose that both factor smooth updating and domain knowledge guided sampling con- tribute to the distillation owing to the existence of domain-specitfe discrepancy.
50
40
Effeet of distillation interval. Fig. (a) shows the evaluation results on the effect of the distilla-
DDK
DDK (w/o FS) DDK (ES)
Figure 4: Effect of data sampling strategies.
tion interval hyperparameter (i.e., K) in Alg. We observe that increasing K from 100 to 1,000
leads to better performance, indicating that a rapid updating frequency may destabilize the distillation process. However, further increasing K leads to inferior results. We conclude that when the updating

---

frequency is small, the domain weights update quickly and the student LLM weights can not be sufficiently optimized for the current distillation interval. Meanwhile, when the updating frequency is large, there is insufficient alignment between the LLM weights and the optimal domain weights.
Effect of training tokens. As shown in Fig. 3 (b), we investigate the relationship between the results on three representative datasets and the number of training steps for Qwen-1.5 1.8B model when using Qwen-1.5 14B model as teacher. At the first 10B tokens, the results improve quickly, which indicates that the student models can benefit a lot with the supervision of the teacher model. When further increasing the training iterations, we observe that the performance tends to plateau, which indicates fast convergence of distillation by DDK.
4.4 Further Analysis
We provide further investigation to show the applicability of DDK across more scenarios.
Generalization ability of using different teacher student models. To show the generalization ability of DDK on different student models, we use Qwen- 1.5 14B as the teacher model and use Qwen-1.5 4B as the student model. As shown in Table DDK surpasses the baseline methods by a large margin. Additionally, comparative analysis with the enhancements observed when employing Qwen-1.5 1.8B as the student model, as presented in Table I, verifies that a more capable student model tends to yield superior performance improvements. We then apply another teacher model to show the generalization ability of DDK. Specifically, we take Qwen-1.5 7B and Qwen-1.5 1.8B as teacher and student models, respectively. As documented in Table 4, DDK consistently facilitates the most substantial enhancement. These results demonstrate the efficacy and robustness of DDK in leveraging diverse teacher-student model configurations.
Generalization ability on Code LLMs. We implement DDK on LLMs, selecting the Code LLM StarCoder as a case study for empirical evaluation. Within the StarCoder series, we deploy StarCoder 15.5B as the teacher model and StarCoder 3B as the student model. The training corpus primarily derived from four programming language domains-Pyttee from The Stack V2 dataset 2, with each language representing distinct domain. We report the performance on the repository-level code completion dataset (i.e., CrossCodeEval [18]). The results in Table 5 affirm that DDK brings notable enhancements the performance of StarCoder 3B, thereby highlighting DDK's ' efficacy in tackling the vertical distillation task.
Java, TypeScript, and C# sampled
Table 3: Results of different methods on the Qwen-1.5 models. Note that we use Qwen-1.5 14B and Qwen-1.5 4B as teacher and student models, respectively.
Methods
CEval
MMLU
RACE
W.G.
GSM8K
C.QA
ARC-E
ARC-C
H.E.
Avg.
Teacher (14B)
78.68
64.34
89.95
77.38
68.74
67.63
82.06
87.58
80.59
37,80
44.00
70.80
Student (4B)
67.60
53.23
80.17
65.26
64.08
52.24
74.24
79.30
66.20
25 60
29.20
59.74
+CPT
68.05
52.78
79.56
67.72
63.61
54.00
74.32
80.20
66.67
26.30
31.00
60.38
-KD
68.35
52.90
80.13
70.31
63.53
56.00
75.51
82.19
67.18
27.50
32.85
61.50
-MiiiLLM
68.20
51.93
79.22
68.78
62.27
55.72
73.87
83.92
67.37
28.13
33.05
61.13
-DDK (Ours)
68.57
53.17
82.53
70.25
64.85
62.09
75.14
84.10
68.95
30.63
39.12
63.58
Table 4: Results of different methods on the Qwen-1.5 models. Note that we use Qwen-1.5 7B and
1.e 1.8B as teacher and student models, respectively.
Methods
CEval
MMLU
RACE
C3
W.G.
GSM8K
C.QA
ARC-E
ARC-C
H.F.
MBPP
Avg.
Teacher (7B)
74.10
58.39
85.78
76.03
65.59
54.53
79,28
85.78
72.30
35.63
37.40
65.89
Student (1.8B
59.66
44.48
69.57
58.27
57.85
38.4
64.70
70.23
50.31
11.87
18.00
49,39
+CPT
60.13
45.01
69.00
60.30
56.98
42.50
64.78
72.00
51.03
13.12
20.45
50.48
-KD
62.63
45.07
69.86
61.18
$7.08
48.14
65.27
73.74
$2.50
13.75
22.69
51.99
MimiL.LM
62.40
45.20
57.46
47.56
65.11
73.86
52.97
14.38
23.31
52.07
DDK (Ours)
64.41
46.44
70.98
63.37
57.54
54.06
66.83
74.43
55.09
11.88
24.98
53.64
Analysis on the in-context learning abilities. We evaluate in-context learning capabilities utilizing DDK and the other baselines through several few-shot benchmarks in Table As shown in Table we observe that our DDK consistently manifests considerable enhancements in performance, affirming that DDK transcends mere static knowledge transfer to the student model and augments the in-context learning capacity greatly.
aattps://hugiingtann hnttgp///uuggggaooc..///aaaaaat http://hgg.gggoeoooo//////////// stackkk
8

---

Table Results of different methods on the StarCoder models. Note that we use StarCoder 15.5B and StarCoder 3B as teacher and student models, respectively.
Python EM
JAVA
TypeScript EM
C#
Avg.
Methods
ES
EM
ES
ES
EM
ES
EM
ES
Teacher (15.5B)
35.9
66.1
41.5
72.9
38.7
73.7
56.3
79.3
43.1
73.0
Student (3B)
20.8
41.5
25.3
51.4
25.7
56.2
40.5
60.5
28.1
52.4
CPT
24.8
49.3
31.6
61.5
30.5
63.7
47.1
68.4
33.5
60.7
KD
26.5
53.2
32.4
61.1
31.6
64.5
48.0
69.8
34.6
61.2
DDK (Ours)
31.7
62.2
34.6
69.8
33.2
69.3
50.9
76.2
37.6
69.4
Table 6: Few-shot (5-shot) performance results of different methods on the Qwen- models. Note that we use Qwen-1.5 14B and Qwen- 1.5 1.8B as teacher and student models, respectively.
Methods
CEval
MMLJ
GSM8K
Arc-E
Arc-C
Avg.
Qwen- 14B
79.86
66.30
69.14
89.24
82.25
77.36
Student (I.8B)
61.96
45.59
38.4
72.16
52.11
54.04
+ CPT
60.92
45.60
43.36
73.10
52.28
55.05
+ KD
61.66
44.28
50.26
73.87
54.69
56.95
+ DDK (Ours)
65.38
47.59
55.19
76.64
57.01
60.36
(w/o FS)
DDK
1.8
CommonCrawi Wikipedia Baoks
CommonCrawi
Wikipedia
11/111111
1.6
1.6
Books
1.4
1.4
1.2
0
10
12
14
16
10
12
14
16
Takens (B)
Takens (B)
Figure Visualization on the domain discrepancy among three domains.
Visualization. To better show the effectiveness of the factor smooth strategy in DDK, we compare the DDK (w/o FS) with DDK by showing the domain discrepancy in the training process, where DDK (w/o FS) means that we remove the factor smooth updating strategy. Specifically, in Fig. 5, we compute the (ssli//T(l as the ratio to represent the domain discrepancy for i-th domain, where large ratio means large discrepancy. As shown in Fig. 5, we observe that the ratio updates smoothly DDK. Besides, in Table the DDK is better than DDK (w/o FS), which means DDK can benefit lot when using the factor smooth updating strategy.
Moreover, refer readers costs and inference examples.
Appendix B.2 and Appendix C for more details the training
Conclusion
In this study, we introduce DDK, novel framework for knowledge distillation tailored for LLMs. Our initial investigations underscore the criticality of optimizing domain data mixtures in the context of LLM distillation. To address this, we propose a domain knowledge-guidde sampling approach that dynamically modulates the sampling probabilities across various domains. Furthermore, we put forward factor smooth update strategy aimed at enhancing both the stability and the efficacy of the distillation process. Comprehensive evaluations of several benchmark datasets with diverse reacher-student model configurations demonstrate the effectiveness of the DDK framework.

---

References
[1] R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem. On- policy distillation of language models: Learning from self-generate mistakes. In The Twelfth International Conference on Learning Representations, 2024.
[2] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-4OB: an open large language model with state-of-the-ar performance. 2023.
[3] I. Amos, J. Berant, and A. Gupta. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. In The Twelfth International Conference Learning Representations, 2024. URL https: http::///pperevviee www.eee..eeeeeeeeeeeeeeeeeeeeeee net//erruu ordooiimmdddiee PdaPky8MUn
[4] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint 0rXiv:2305.00003 2023. URL https: https://arxivv. org/ abaaabs///00/ 2205. 10403.
[5] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E, Jiang, C. Cai, M. Terry, Q Le, and C. Sutton. Program synthesis with large language models, 2021.
[6] G. Bai, Liu, X. Bu, Y. He, J. Liu, Z. Zhou, Z. Lin, W. Su, Ge, B. Zheng, and W. Ouyang. Mt-bench-100 101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues, Xrxivv 2024.
[7] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, Ren, C. Tan, S. Tan, J. Tu, P. Wang. S. Wang. W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang. J. Yang, Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu. Qwen technical report. arXiv preprint arXiv:2309 16609, 2023.
[8] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022. URL http///axxvvvv///e/ org/ abs/ 2204.05862. 05862.
[9] Baichuan. Baichuan 2:
Open large-scale language models. arXiv preprint arXiv:2309. 10305, 2023. URL https: //arxiv...vvv//////// org/ a23/////..//..////////////////// 10305.
[10] T. Brown, B. Mann, N. Ryder, M. Subbiah, al. Language models are few-shot learners. In Proceedings of NewrIPS, 2020. URL https: //pppe//ppppppeeee nips wwwps.cc/pppeeee pttpp/ppppr///////////// 1457c0d6bf Btw.leescccb.77/11bccceccccee
Abstract html.
[11] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, Tillet, F. P. Such, Cummings, M. Plappert, Chantzis, Barnes, A. Herbert- Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, Achiam, V. Misra, Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, Sutskever, and W. Zaremba. Evaluating large language models trained on code. 2021.
[12] Z. Chen, Q. Gao, A. Bosselut, A. Sabharwal, and K. Richardson. DISCO: Distilling coun- terfactuals with large language models. A. Rogers, J. Boyd-Graber, and N. Okazaki, ed- itors, Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Pupers), pages 5514-5528, Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10.88653//220222cllolooo //aclanthology. org/ 2023 acl 1ong.1on..333 302.
URL https:
[13] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al. Scaling instruction-fineuueed language models. arXiv preprint axXiv:2200I11416 2022. URL https: /ttps://arxiv. org/ hbs//bb//2211 11416.
10

---

[14] P. Clark, I. Cowhey, O, Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 1803. 05457, 2018.
[15] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L., Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[16] Computer. Redpajama: an open dataset for training large language models, October 2023. URL https://github. wwtp:/cccoooooooooooeeoo www.tttgetteeeecomttttee///////e SeeiipuuterrRePPaaa a -ata.
[17] T. Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
[18] Y. Ding, Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, Nallapati, P. Bhatia, D. Roth, and B. Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.
A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[19] X. Du, Z. Yu, S. Gao, D. Pan, Y. Cheng, Z. Ma, R. Yuan, X. Qu, J. Liu, T. Zheng, X. Luo, G. Zhou, B. Yuan, W. Chen, J. Fu, and G. Zhang. Chinese tiny Ilm: Pretraining chinese-centric large language model, 2024.
[20] S. Fan, M. Pagliardini, and M. Jaggi. DOGE: Domain reweighting with generalization es- timation. In Second Agent Learning in Open-Endedness Workshop, 2023. URL https: //openreview. wwwpiiiwwwee//fooreee
[21] Google.
Bard, 2023.
URL https://blog google/technologg///
bard- google sii--search-ppdee
[22] Gou, B. Yu, S. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129: 1789-19988,9, 2021.
[23] Y. Gu, L. Dong, F. Wei, and M. Huang. Knowledge distillation of large language models, 2024.
[24] H. Guo, J. Yang, J. Liu, L. Yang. L. Chai, Bai, Peng, X. Hu, C. Chen, D. Zhang, al. Owl: A large language model for it operations. arXiv preprini arXiv:2309.0020) 2023.
[25] J. Guo, J. Liu, and D. Xu. Jointpruning: Pruning networks along multiple dimensions for efficient point cloud processing. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
[26] J. Guo, J. Liu, and D. Xu. 3d-pruning: A model compression framework for efficient 3d action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 32(12): 8717-8729, 2022.
[27] J. Guo, J. Wu, Z. Wang, J. Liu, G. Yang, Y. Ding, R. Gong, H. Qin, and X. Liu. Compressing large language models by joint sparsification and quantization. In Proceedings of the 4 International Conference on Machine Learning, pages 16945-16957, 2024.
[28] D. Hendryeks, C. Burns, Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, arXiv preprint arXiv:2009.03300 2020.
[29] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint 1rXiv=1503.0253), 2015.
[30] N. Ho, L. Schmid, and S.-Y. Yun. Large language models are reasoning teachers. In A. Rogers, Boyd-Graber, and Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852-14882, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18/33/1/2023, long.830.. URL https: /tatpp://acleeo /accanttoooogyy org/ /22//2223..22 acl- 1o11-1onn.8333 830.

---

[31] C.-Y. Hsich, Li, C.-k. Yeh, H. Nakhost, Y. Fujii. A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister. Distilling step-by-step! outprrorring larger language models with less training data and smaller model sizes. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 8003-8017, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: URL https:
(tt.///////22..mnmnnnoo https////aaaaaahloaaeeee .rrg org.oog//// findings-acl. 507.
[32] Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint 24Xv:24044006355 2024.
[33] Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang. T. Su, Liu, Lv, Y. Zhang, J. Lei, Y. Fu, M. Sun, and J. He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Informaiion Processing Systems, 2023.
[34] Y. Jiang, C. Chan, M. Chen, and W. Wang. Lion: Adversarial distillation of proprietary large language models, 2023.
[35] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:200.008361, 2020. URL https://arxiv.. org/abs/
[36] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[37] J. Ko, S. Kim, T. Chen, and Yun. Distillm: Towards streamlined distillation for large language models. ArXiv, abs/2402.03898, 2024.
[38] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, Race: Large-scale reading comprchension dataset from examinations. arXiv preprint arXiv: 1704, 04683, 2017.
[39] Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and T. Lee. Texthooks are all you need ii: phi-1.5 technical report. arXiv preprint 2.XXiv:2009004663 2023.
[40] C. Liang, S. Zuo, Q Zhang, P. He, W. Chen, and T. Zhao. Less is more: Task-aware layer-wise distillation for language model compression. ICML, 2023.
[41] J. Liu, Z. Bai, Y. Zhang, C. Zhang, Y. Zhang, G. Zhang, J. Wang, H. Que, Y. Chen, W. Su, et al. E2-llm: Efficient and extreme length extension of large language models. arXiv preprint 2.Xiv:2401.0695) 2024.
[42] J. Liu, J. Li, K. Wang, H. Guo, J. Yang, J. Peng, K. Xu, X. Liu, and J. Guo. Lta-pes: Learnable task-agnostic point cloud sampling. In CVPR, 2024.
[43] OpenAl. OpenAl: Introducing ChatGPT, 2022. URL
com
[44] OpenAL. GPT-4 technical report, 2023. URL https::/cdn..opeee/
con/papers/gpt--.
pdf.
[45] I.. Ouyang, J. Wu, X. Jiang, D. Almeida, C. I.. Wainwright, P. Mishkin, Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Proceedings of NeurIPS, 2022. URL /tttps://arxxx wrg./or////aaa//aoa 200.00005.220001555. 02155.
[46] S. Padmanabhan, Y. Onoe, M. J. Zhang, G. Durrett, and Choi. Propagating knowledge updates to Ims through distillation. arXiv preprint 200.:::3306603306 2023.
[47] K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba. Openwebmath: open dataset of high-quality mathematical web text, 2023.
[48] B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. ArXiv, abs/2304.03277, 2023.
[49] H. Que, J. Liu, G. Zhang, C. Zhang, X. Qu, Y. Ma, F. Duan, Z. Bai, J. Wang, Y. Zhang, X. Tan, J. Fu, W. Su, J. Wang, I.. Qu, and B. Zheng. D-ept law: Domain-specific continual pre-training scaling law for large language models. ArXiy, abs/2406.01375, 2024.
12

---

[50] A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018. URL https: / /pi. semanticscholaar org/Corpus 49313245..
[51] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9);99-106, 2021.
[52] K. Sun, D. Yu, D. Yu, and C. Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension. Transactions of the Association for Computational Linguistics, 2020. URL https: / /aaxivaiiv..//aaaaaaaaaaaaaaa//a org/ abaaaba///99 1904. 09679v3.
[53] T. Sun, L. Chai, Yang, Yin, H. Guo, J. Liu. B. Wang. L. Yang, and Z. Li. Uni- Coder: Scaling code large language model via universal code. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Pupers), pages 1812-1824, Bangkok, Thailand, Aug. 2024. Association for Computational Linguisties. doi: 10. 1/2653//1/0224.all-ala 100. URL attps://aclantholl
org wttp:/2224.ccl11100 acl-long....... 100.
[54] R. S. Sutton, D. A. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, 1999.
[55] A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint
2018.
[56] Timiryasov and J.-L.. Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on small dataset with no performance penalty. arXiv preprint S.Niv:2338.000111 2023.
[57] H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.13974 2023. URL https: //axxiv
arxiv.vvvvvv... org.oogo/g....oooooooo. pdff//f//// 2202. 19971.pff.
[58] H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023.
[59] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228 2023.
[60] Wang, Z. Wang, Z. Li, Y. Gao, Yin, and X. Ren. Scott: Self-consistent chain-of of-thought distillation. In Annual Meeting of the Association for Computational Linguistics, 2023.
[61] Z. Wang, J. Guo, Gong, Y. Yong, A. Liu, Y. Huang. J. Liu, and X. Liu. Ptsbench: A comprehensive post-training sparsity benchmark towards algorithms and models. In ACM Multimedia 2024.
[62] Z. M. Wang, Z. Peng. H. Que, J. Liu, W. Zhou, Y. Wu, H. Guo, R. Gan, Z. Ni, M. Zhang, Zhang, W. Ouyang, K. Xu, W. Chen, J. Fu, and J. Peng. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: 2310.00746. 2023.
[63] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In Proceedings of ICLR, 2022. URL https:///opnrevier. www.:eeee/ooommddddd/eeeeeoe
[64] Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. Transuctions on Machine Learning Research, 2022. URL WtRLhtts::// hattps::/oppeee /ttp:////pee/eeieeeee net/ net/ Fid
[65] Y. Wu, J. Liu, X. Bu, J. Liu, Z. Zhou, Y. Zhang, C. Zhang, Z. Bai, H. Chen, T. Ge, et al. Conceptmath: bilingual concept-wise benchmark for measuring mathematical reasoning of 2024.
large language models. arXiv preprint
13

---

[66] M. Xia, T. Gao, Z. Zeng, and D. Chen. Sheared Ilama: Accelerating language model pre-training structured pruning, arXiv preprint arXiv:2310.0669,, 2023.
[67] Xie, H. Pham, Dong, N. Du, Liu, Y. Lu, S. Liang, Q. V. Le, T. Ma, and A. W. Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.
[68] Z. Yang. A. Zeng, Z. Li, T. Zhang, C. Yuan, and Y. Li. From knowledge distillation to seff-knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17185-17194, 2023.
[69] Ye, P. Liu, T. Sun, Y. Zhou, Zhan, and X. Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint WrXiv:2403.16952 2024.
[70] 7. Yuan, J. Guo, D. An, J. Wu, H. Zhu, J. Li, X. Chen, K. Xu, and J. Liu. VRDistill: Vote refinement distillation for efficient indoor 3d object detection. ACM Multimedia 2024, 2024.
[71] C. Zhang, D. Song. Z. Ye, and Y. Gao. Towards the law of capacity gap in distilling language models. arXiv preprint arXiv:2311.00 1.00052,, 2023.
[72] G. Zhang, S. Qu, J. Liu, C. Zhang, C. Lin, C. Yu, D. Pan, E. Cheng, Liu, Q. Lin, al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405.1932 /9327, 2024.
[73] Zhang, G. Zeng. Wang, and W. Lu. Tinyllama: An open-souree small language model, 2024.
[74] Z. Zhou, J. Liu, C. Yang, J. Shao, Y. Liu, X. Yue, W. Ouyang, and Y. Qiao. Be- yond ooe-prlferencc-for-all::: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023.
14

---

A Broader Impacts and Limitations
Broader Impacts. Based on our DDK, we can boost the performance of small LLMs a lot under the guidance of the teacher LLMs. In real-world scenes, we hope our enhanced small LLMs can be deployed on low-resource devices (e.g., PCs Mobiles) and make future life better with better small LLMs.
Limitations. First, have DDK. Second, due to the GPU resource limitations, we only distill the 1.1B, 1.8B and 4B sizes under the guidance of teachers with 7B and 10B+ sizes. In the future, will investigate the performance of knowledge distillation using larger student and larger teacher models.
several hyperparameters to achieve better performance in
B
More Details
B.1
Discussion on the differences between DDK and Sheared LLLMA
First, for the solved tasks. the Sheared LLaMA [66] is proposed for structure pruning. while our DDK proposed for knowledge distillation on the I.I.s. Therefore, our DDK orthogonal the Sheared LLaMA and we can further improve the performance of small models pruned by Sheared LLaMA. Second, for the technical details, the sheared ILaMA needs to first scaling function using three open-sourced LLaMA models (i.e., ALAMA2-7B/13BBBB where the fitting data points are very limited. In Sheared LLaMA, the authors also claim the estimated reference losses for different domains are also biased the limited data points for estimating the scaling law parameters. Besides, in many real-world pruning scenarios, we cannot usually obtain a series of models under the same training setting. Thus, cannot predict the losses across different domains the small-size model when series models are not provided. In contrast, in DDK, we directly use the domain loss predicted by the teacher model the reference loss without fitting the teacher-predicted the student model. Besides, the Sheared LLaMA is proposed enhance the training efficiency of the continued pre-training setting of the pruned model, the guidance of the teacher model is not included. In addition, our DDK also introduces the factor smooth updating strategy, which is not used in Sheared LLaMA and can make the change of the domain mixture more stable. Moreover, in the following table, we also provide the results of replacing our proposed factor smooth updating strategy with the strategy of Sheared LLaMA, and observe that DDK is better a lot, which further shows the effectiveness of our factor smooth updating strategy.
when pruning the LLaMA model series,
scaling function, where the teacher- predicted losses are used as the accurate guidance to improve
B.2 Details on the training costs
As shown in Table 7, we compare the TFLOPs of three representative baseline methods, and observe that the training costs of our DDK are acceptable when compared with the baseline KD method.
Table 7: Training TFLOPs all data of different methods for Qwen-1.5, For KD and DDK, the Qwen-1.5 14B to distill the Qwen- 1.5 1.8B.
Models
CPT
KD
DDK
TFLOPs
1.456e8
5.364e8
5.401e8
C More Sentence Examples
In the following, we provide more examples generated by the original and distilled models. We find that the sentences generated by the distilled model are superior those generated by the original model in terms of fluency, relevance, and informativeness regarding the given topic.
15

---

GSM8K
Prompt:
Question: Janet's ducks lay 16 eggs per day. She three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day the farmers' market?
Answer: 18
Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts total does it take?
Answer: 3
Question: Josh decides to try flipping house. He buys house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?
Answer: 70000
Question: James decides to run sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run week?
Answer:
Assistant (Qwen-1.5 1.8B):
1800
Assistant (Qwen-1.5 1.8B DDK):
540
Prompt:
Gunter is trying to count the jelly beans in jar. He asks his friends how many they think are in the jar. One says 80. Another says 20 more than half the first one. A third says 25% more than the first one. What is their average guess? Answer:
Assistant (Qwen-1.5 1.8B):
First, we need find half of the first number, which 80/ 40. 40.
Then, we need to add 20 more to that number, which is 40 + 20 = 60.
Next, we need to find 25% of the first number, which 80 0.25 20.
Finally, we add that number to the first number, which is 60 + +0 80.
Now, we can find the average by adding up all the guesses and dividing by the number of guesses, which is 80 60 + 20 160.
Finally, we divide that number by the number of guesses, which is 3, find the average, which is 160 / 53.33.
Therefore, the answer is 53,
Assistant (Qwen-1.5 1.8B DDK):
The first friend's guess 80 jelly beans.
The second friend's guess is 20 more than half of 80, which is (80/2) 20 40 20 60 jelly beans.
The third friend's guess is 25% more than 80, which is 80 (0.25 80) 80 + 20 100 jelly beans.
To find the average guess, add up all the guesses and divide by the number of friends, which is 3.
So the average guess is (80 60 100) / 240 / 3 80 jelly beans.
#*## 80
The answer is: 80
16

---

Chinese
Prompt:
[
JR
/ - T #E
E
Assistant (Qwen-1.5 1.8B):
Hi - J TORTE
Assistant (Qwen-1.5 1.88 DDK):
tT:: IJ
/
-
TA WA
Prompt:
?
: (
T J II??
3 T
#
Assistant (Qwen-1.5 1.8B):
Assistant (Qwen-1.5 1.8B DDK):
2 Ap
Prompt:
Assistant (Qwen-151.8B)
X HAS G
n Hit.
Pj A
5 - -0
11  2A113331111111111333333333333333
MH - Mi #F XTN.A//SSTSSSSST.TTTTTTTTTTTTTTT EE
un. 4: 4.0% X . 1.001 N
+ 11
H... 31
X
SO , $
1E3
#X. #
XIl
DDK): *F#) T
70 !
4*
6 1 1
-
HX.
$ EE DETTEEAAALLLTTTTTETTTTTTTTTTSELL S
mfj
#
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have highlighted the main contribution of this paper the abstract and instruction sections DKK focuses on the domain discrepancies to adaptively improve the efficacy of knowledge distillation tailored LLMs.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
As mentioned Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: Equations 2 and 3 concludes the main theory assumptions and proofs of this paper. We also provide sufficient explanations attached to them.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The full details of the experiments are provided in Scction 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All data used this paper is open-sourced. The codes for the baseline methods are also collected from Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Pooeiiisssoooloiioooomooooo ooiicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
This information is provided in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have made sufficient ablation studies to show the confidence of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Mentioned in Section and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttpr//eerup/eeeeee//ee/eeeeeeee httppplicccchcccccideeeeeeee
Answer: [Yes]
Justification: No ethic issues are related to this topies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all relevant papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not yet release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
All evaluations are conducted by rules.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Unrelated.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "IQFYwRUPY2", "reviewer": "Reviewer_foae", "summary": "This work proposed a KD strategy for LLMs. Specifically, with assess to the domain-specific performance of both the teacher and student LLMs, DDK uses domain knowledge guided sampling to dynamically update the data mixture. In addition the paper also conducts a statistical analysis of the domain distribution of the datasets involved. The training process is relatively straightforward and easy to generalize. The experimental results also show that DDK's training method improves the performance average of different data sets.", "questions": "The questions I expect to ask would be similar to the above section.", "limitations": "The method proposed in this paper is very complete and solves the problem well, but there are perhaps two points to note:\n1. The distillation method designed in this paper does not seem to be necessarily limited to LLMs. One of the main difficulties of KD on LLM may be that the distribution difference between the teacher and student models is too large, but the key points that this paper focuses on and solves seem to be orthogonal to this point. So it also leads to a similar question: why previous similar methods that focus on domain sampling cannot be migrated to LLMs, and what are the advantages and novelties of this paper's design.\n2. Following point 1, in the experiment, what are the specific advantages of this method over the predecessors in the domain problem (not just the overall average, of which result shows DDK outperformed others). This may be what I am very curious about after reading the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A complete training algorithm is designed, and the process is explained clearly. The process of the DDK algorithm is easy to extend to the training process of other models.\n2. The authors conducted a comprehensive knowledge distillation experiment on two large model families and a comprehensive ablation study.", "weaknesses": "1. Although the method proposed in this paper is easy to understand and effective, I doubt that the method in this paper is limited to LLMs. In other words, this paper does not mention (or needs to explain) how previous researchers (before LLMs) performed domain-enhanced distillation for domain-biased datasets, and why these previous methods cannot be applied to the distillation of LLMs to achieve similar results. The advantages and novelty of this paper's domain sampling method over previous work that may be transferable to LLMs need further explanation.\n2. In the experimental part, there is a lack of key comparison between DDK and other methods that focus on similar domain sampling. The baseline actually involves the work that focuses on domain in KD (cited as [60], etc.), but the subsequent analysis only compares the total average score of DDK and these works, which seems to lack comparison and analysis of similar works. As far as I know, other baselines are more general KDs, and do not focus on domain information.\nIt is certainly worth noting that DDK performs better than baselines such as MiniLLM, but I think what can better illustrate the effectiveness and novelty of this paper is the comparison with similar domain data sampling, including experimental analysis.\n3. In the experimental section, you can add experiments on the dataset and the scale property of the teacher model. This is a possible suggestion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your nice comments and suggestions.\n\n**Q1: Compare domain-enhanced KD methods.**\n\n **A1:** After investigating KD [R1] and applications on LLMs [R2], we have observed existing domain-enhanced KD methods can be divided into two categories. The first is cross-domain KD for domain adaptation to improve the generalization abilities of the unseen target domains based on seen source domains. In contrast, DDK aims to improve training efficiency and effectiveness given multiple seen domains, and these cross-domain KD methods cannot be applied to LLM pretraining. The second is based on multi-teacher strategy [R3, R4], which first trains different domain expert models and then distills the overall domain capacities into one single model. Thus, we implement two baselines following [R3, R4].\n\n [R1]. Knowledge Distillation: A Survey\n\n [R2]. A Survey on Knowledge Distillation of Large Language Models\n\n [R3]. BAM! Born-Again Multi-Task Networks for Natural Language Understanding\n\n [R4]. Knowledge Fusion of Large Language Models\n\n Specifically, first, we train the domain expert models on 10 different domains as teacher models, where each domain expert is trained on about 5B tokens on the corresponding domain. Then, for the first baseline [R3], we select a corresponding teacher to produce logits for each sample based on the belonged domain. For the second baseline [R4], we ensemble logits from different domain teacher models. The table below lists the PPL on validation datasets for 10 domains. \n\n|         | CC    | C4    | The Stack | Wikipedia | Books | Arxiv | StackExchange | Chinese Books | Chinese CC | OpenWebMath |\n| ------- | ----- | ----- | --------- | --------- | ----- | ----- | ------------- | ------------- | ---------- | ----------- |\n| Student | 10.15 | 18.50 | 4.14      | 13.27     | 25.81 | 12.97 | 10.04         | 23.54         | 21.20      | 9.50        |\n| +[R3]   | 10.00 | 19.14 | 4.14      | 10.12     | 22.73 | 12.90 | 8.37          | 20.59         | 20.19      | 4.75        |\n| +[R4]   | 10.06 | 19.09 | 4.38      | 9.54      | 26.01 | 12.72 | 9.46          | 20.51         | 20.12      | 5.97        |\n| +DDK    | 10.02 | 18.94 | 3.95      | 8.36      | 20.43 | 11.23 | 7.22          | 19.69         | 20.33      | 4.62        |\n\n We also report the downstream results and observe that baseline methods are inferior to DDK. We assume that these methods only consider producing better teacher guidance, and do not consider the effect of the domain mixture for  LLM pretraining.\n\n| Models  | MMLU  | RACE  | W.G.  | GSM8K | Arc-E | Arc-C | H.E.  | Avg.  |\n| ------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Student | 44.48 | 69.57 | 57.85 | 38.4  | 70.23 | 50.31 | 11.87 | 48.96 |\n| +[R3]   | 45.02 | 68.25 | 58.8  | 47.24 | 71.17 | 52.44 | 19.09 | 51.72 |\n| +[R4]   | 45.53 | 68.32 | 58.25 | 39.65 | 74.33 | 53.84 | 20.26 | 51.45 |\n| +DDK    | 46.01 | 71.56 | 59.1  | 53.54 | 75.01 | 55.03 | 27.13 | 55.34 |\n\n**Q2: Experiments on dataset and scale property of teacher.**\n\n**A2:** For experiments on dataset, we have performed experiments using StarCoder on Stack V2 dataset in Table 11. Besides, we perform experiments on The Pile dataset [https://arxiv.org/pdf/2101.00027] with 22 domains, and we use Qwen-1.5 14B and Qwen-1.5 1.8B as teacher and student following Table 1. Note that as The Pile focuses on English, we do not evaluate CEval and C3.\n\n| Models            | MMLU  | RACE  | W.G.  | GSM8K | C.QA  | Arc-E | Arc-C | H.E.  | MBPP  | Avg.  |\n| ----------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| +CPT (main paper) | 45.01 | 69.00 | 56.98 | 42.50 | 64.78 | 72.00 | 51.03 | 13.12 | 20.45 | 48.32 |\n| +DDK (main paper) | 46.01 | 71.56 | 59.10 | 53.54 | 66.75 | 75.01 | 55.03 | 27.13 | 26.10 | 53.36 |\n| +CPT(The Pile)    | 44.74 | 68.86 | 56.43 | 40.56 | 64.29 | 71.13 | 50.34 | 12.20 | 18.48 | 47.45 |\n| +DDK(The Pile)    | 45.72 | 69.98 | 58.80 | 45.41 | 65.44 | 74.03 | 54.52 | 22.56 | 22.59 | 51.01 |\n\nWe observe that DDK  achieves better performance than CPT baseline on The Pile and results on The Pile are relatively lower than results in our main paper. We assume that the data quality used in our main paper (See Line 166-171) is better than The Pile, as we directly use the original Pile dataset in 2021 without additional cleaning strategies.\n\nFor scale property of teacher, we have used different sizes of teacher models (i.e., Qwen1.5-14B and Qwen1.5-7B) to distill Qwen1.5-1.8B in Table 1 and Table 8, and average results distilled by Qwen1.5-14B and Qwen1.5-7B are 55.41 and 53.64, respectively, which means that better results are achieved using a relatively larger teacher.\n\n**Q3: Advantages and novelties of DDK.**\n\n**A3:** Please See **General Response** (**G.Q1**).\n\n**Q4: Advantages over predecessors  (not the overall average).**\n\n**A4:** First, in Fig. 1, DDK can reduce the performance gaps between teacher and student a lot in many domains. For example, for DDK, PPL on the ``Books`` domain is less than other methods a lot, and PPL  on  ``StackExchange`` domain is very close to teacher. \n\nSecond, in Table 1, we observe that a large relative performance gap usually exists in reasoning-related tasks for teacher and student. For example, relative performance gap ratios are 43.22% [(67.63-38.4)/67.63=43.22%] and 68.59%[(37.80-11.87)/37.80=68.59%] for GSM8K and Humeneval, respectively. In contrast, for knowledge-related tasks, the relative performance gap ratios are 24.17% [(78.68-59.66)/78.68=24.17%] and 30.87%[(64.34-44.48)/64.34=30.87%] for CEval and MMLU. After using DDK, the relative performance gaps are 20.84%, 28.23%, 18.98%, 28.49% on GSM8K, Humeneval, CEval and MMLU. In contrast, for ``CPT & DoReMi'' and MiniLLM, the relative performance gaps are [32.18%, 76.85%, 21.91%, 30.15%] and[27.69%, 55.34%, 21.63%, 29.95%], respectively. Thus, we assume that  DDK can greatly improve the weaknesses on reasoning-related tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rhBhVfF0Va", "reviewer": "Reviewer_EYh3", "summary": "The work introduces a novel framework for knowledge distillation (KD) for LLMs. The key innovation of DDK is its dynamic adjustment of the distillation dataset composition based on domain performance differences between the teacher and student models. The paper presents extensive evaluations demonstrating that DDK significantly improves performance in various KD settings, outperforming both continuous training baselines and existing KD methods.", "questions": "1. In the setting of paper, the domains are predefined. How to extend the DDK framework for new domain during the distillation training process? Could give more experiments on continual domain learning settings?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.  The authors provide extensive empirical evidence demonstrating the effectiveness of DDK in improving the performance of student models across various benchmarks.\n2.  As the computational and storage demands of LLMs are significant barriers to their widespread deployment, KD is a promising solution.  The proposed KD method is simple and easy to follow.", "weaknesses": "1. Discuss the difference between DDK and the Dynamic Batch Loading proposed by Sheared LLaMA[1], which is also \n proposed to adjust domain proportions for dynamically training smaller models. They also identify discrepancies in loss between smaller and larger models across various domains, and accordingly, they sample more data from domains where the discrepancy is more pronounced. While they concentrate on structural pruning, it is akin to the DDK. Consequently, I perceive the novelty of DDK as being somewhat limited.\n2. The results of Qwen 1.5 in Table 1 are not significantly convincing. The MMLU/HumanEval of Qwen 1.5 1.8B in the Qwen official blog are  46.8/20.1 while the authors' report is 44.5/11.9. In addition, compared to the official results, we can see that the DDK fails to improve the model of the students on MMLU. The authors need to check this and provide **more robust results of baselines**.\n\n[1] SHEARED LLAMA: ACCELERATING LANGUAGE MODEL PRE-TRAINING VIA STRUCTURED PRUNING. Xia et al., 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.\n\n**Q1: Discuss with Sheared LLaMA. Novelty.**\n\n**A1:** Please See **General Response** (**G.Q1** and **G.Q2**).\n\n**Q2: Qwen1.5 results on MMLU and Humaneval.**\n\n**A2:** For MMLU, the accuracy from Qwen Blog is based on a 5-shot setting, while the result in Table 1 is based on a zero-shot setting (See Line 182). Besides, we provide the 5-shot result of MMLU in Table 12 and observe the result is 45.59, which is close to the provided result (46.8) of the Qwen blog. Besides, we assume the gap between 46.8 and 45.59 comes from the evaluation prompt. Specifically,  the evaluation prompt for DDK is as follows:\n\n```SQL\n# Prompt-1\nfor opt in ['A', 'B', 'C', 'D']:\n    prompt=\"Answer the question.\\nQuestion: {question}\\nChoices:\\nA. {A}\\nB. {B}\\nC. {C}\\nD. {D}\\nAnswer: \" + opt\n```\n\n The ``{question}``, ``{A}``, ``{B}``, ``{C}``, ``{D}`` are the corresponding inputs for each evaluation sample. \n\nWe also have tried the following prompt from OpenCompass [https://github.com/open-compass/opencompass], where the task description is also added to the evaluation prompt. Note that the ``mmlu_all_sets`` contains 57 tasks. Based on this prompt, we obtain 46.88 and 47.05 under the zero-shot and few-shot settings, respectively.\n\n```SQL\n# Prompt-2\nmmlu_all_sets = ['college biology', 'college chemistry', 'college computer science', ..., 'human aging', 'us foreign policy', 'conceptual physics']\nfor task in mmlu_all_sets:\n    for opt in ['A', 'B', 'C', 'D']:\n        prompt=\"The following are multiple choice questions (with answers) about\" + task + '\\n\\n' + \"{question}\\nA. {A}\\nB. {B}\\nC. {C}\\nD. {D}\" + '\\n' + 'Answer: ' + opt\n```\n\n We also provide the following results and DDK achieves consistent improvements in MMLU.\n\n|                | Prompt-1 | Prompt-2 | Prompt-1(few-shot) | Prompt-2(few-shot) |\n| -------------- | -------- | -------- | ------------------ | ------------------ |\n| Student (1.8B) | 44.48    | 46.88    | 45.59              | 47.05              |\n| +DDK           | 46.01    | 47.95    | 47.59              | 48.12              |\n\nFor Humeneval, we observe that the results of Qwen1.5-1.8B  are very sensitive to the prompt, and we use the naive testing prompt (accuracy  of 11.87) as follows:\n\n```SQL\n# Prompt-1\nprompt='{prompt}'\n```\n\nSimilarly, we have tried following two prompts following OpenCompass:\n\n```SQL\n# Prompt-2\nprompt='Complete the following python code:\\n{prompt}'\n# Prompt-3\nprompt='You are an intelligent programming assistant to produce Python algorithmic solutions.\\nCan you complete the following Python function?\\n```python\\n{prompt}\\n```'\n```\n\nAnd we observe the results of ``Prompt-2`` and ``Prompt-3`` are 23.17 and 26.83, respectively, which are higher than the reported result (20.1) on Qwen blog.\n\nMeanwhile, we report the DDK results on these three evaluation prompt settings. In the following table, we observe that DDK still achieves better performance than the baseline. In new version, we will provide a detailed analysis of the evaluation prompt and provide more solid and robust results.\n\n|                | Prompt-1 | Prompt-2 | Prompt-3 |\n| -------------- | -------- | -------- | -------- |\n| Student (1.8B) | 11.87    | 23.17    | 26.83    |\n| +DDK           | 27.13    | 29.59    | 31.08    |\n\n**Q3: Extend DDK for new domains during distillation.**\n\n**A3:** In distillation, when a new domain is introduced, we first need to prepare the validation dataset for this domain following Appendix B.1 (Line 490-510). Then, following Eq. (1),  based on the current model checkpoint, we compute the perplexity scores over the validation sets of all domains for student and teacher respectively, and obtain the newly initialized domain discrepancy factor, which is then used to change the domain mixture following the Algorithm 1  of the main paper. \n\n**Q4: Experiments on continual domain learning setting.**\n\n**Q4:** We provide the results of involving a new source domain data (i.e., Law) using the Pile-of-Law dataset [https://huggingface.co/datasets/pile-of-law/pile-of-law]. Specifically, we first train the original 10 domains for 5B/10B tokens. Then, we append the Law domain into the training dataset. After that,  following Algorithm 1 of the main paper, we continue the distillation training on the student for additional 10B/5B tokens on these 11 domains, which aims to achieve a fair comparison with the DDK baseline results on 10 domains for about 15B tokens.  To show the effect of introducing the law domain, we additionally report the results on the law subset of MMLU-Pro[https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro]. (MMLU-Pro has 14 high-quality subsets with challenging questions.), and observe that by introducing the law domain, better results on the MMLU-Pro(Law) are obtained, and results on other datasets are also preserved well.\n\n| Models                                                       | MMLU  | RACE  | W.G.  | GSM8K | Arc-E | Arc-C | MMLU-Pro(Law) | Avg.  |\n| ------------------------------------------------------------ | ----- | ----- | ----- | ----- | ----- | ----- | ------------- | ----- |\n| Student (Qwen1.5-1.8B)                                       | 44.48 | 69.57 | 57.85 | 38.4  | 70.23 | 50.31 | 14.4          | 49.32 |\n| Student+DDK(train 5B on original 10 domains)                 | 44.12 | 69.79 | 58.17 | 44.35 | 72.39 | 52.75 | 14.8          | 50.91 |\n| Student+DDK(train 5B on original 10 domains+10B with additional new law domain) | 45.98 | 71.43 | 58.61 | 51.12 | 74.78 | 54.95 | 16.29         | 53.31 |\n| Student+DDK(train 10B on original 10 domains)                | 45.85 | 70.96 | 59.19 | 49.66 | 74.07 | 53.18 | 14.68         | 52.51 |\n| Student+DDK(train 10B on original 10 domains+5B with additional new law domain) | 46.12 | 71.02    | 58.8  | 52.51 | 74.36 | 54.27 | 15.75         | 53.26 |\n| Student+DDK(train 15B on original 10 domains, baseline in Table 1) | 46.01 | 71.56 | 59.1  | 53.54 | 75.01 | 55.03 | 14.74         | 53.57 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WugHyZa5Cg", "reviewer": "Reviewer_oZHE", "summary": "The paper introduces a new framework called Dynamic Domain Knowledge Distillation (DDK) to enhance the efficiency of knowledge distillation for large language models (LLMs). Unlike traditional methods that overlook domain performance differences between student and teacher models, DDK dynamically adjusts the distillation dataset composition based on these differences, ensuring a more stable and effective knowledge transfer. This approach addresses the issue of excessive focus on domains with minimal performance gaps and enhances overall model performance. Extensive evaluations demonstrate that DDK significantly outperforms existing knowledge distillation methods and continuously pretrained baselines.", "questions": "- How the methods perform on under other KD losses, such as reversed KLD, JSD, skew-KLD.\n\nOn-policy distillation of language models: Learning from self-generated mistakes.\n\nDistiLLM: Towards Streamlined Distillation for Large Language Models.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed dynamic dataloader for KD is technically sound. \n- Numerical experiments well validate the efficacy of the method.", "weaknesses": "- Dynamic dataloader requires knowing the training data distribution and category beforehand. \n- Missing references. Similar ideas have been explored in pruning LLMs, such as ShearedLLaMA, LoRAShear to recover the knowledge . The paper needs to discuss with them in the related work section due to the closed relation between pruning and KD. \n\nSheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning\n\nLoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and constructive suggestions. We will address your concerns shown below in detail.\n\n**Q1: Require knowing the training data distribution and category beforehand.**\n\n**A1**: Our DDK requires the training data distribution and category to divide the domains and we acknowledge this limitation. However, it should be mentioned that existing pretraining datasets usually provide the source composition. For example, the Repajama[https://www.together.ai/blog/redpajama] includes 7 domains (i.e.,  CommonCrawl, C4, Github, Wikipedia, Books, ArXiv, and StackExchange), The Pile [https://arxiv.org/pdf/2101.00027] includes 22 domains (i.e., Pile-CC, PubMed Central, Books3, OpenWebText2 and etc.) and Dolma[https://arxiv.org/pdf/2402.00159] includes 6 domains (i.e., Common Crawl, GitHub, Reddit, Semantic Scholar and etc.)\n\nMeanwhile, we also tried another option to dive the training data into 10 domains based on the clustering method to address this limitation, where we first extract the feature of each document using [https://huggingface.co/BAAI/bge-m3] and then use  k-means  to divide the training corpus into 10 domains implicitly. Then, we apply DDK on  the divided implicit domains based on the clustering and we call this method  DDK (clustering), where we do not need to know the training data distribution and category. In the following table, we observe that the DDK (clustering) achieves relatively comparable performance with original DDK. We assume that the explicitly divided domains based on the sources (e.g., The stack, OpenWebMath) are distinct. Besides, we will continue to investigate the clustering method, which is more scalable. \n\n| Models                        | CEval | MMLU  | RACE  | C3    | W.G.  | GSM8K | C.QA  | Arc-E | Arc-C | H.E.  | MBPP  | Avg.  |\n| ----------------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Student (1.8B)+DDK            | 63.75 | 46.01 | 71.56 | 65.53 | 59.1  | 53.54 | 66.75 | 75.01 | 55.03 | 27.13 | 26.1  | 55.41 |\n| Student(1.8B)+DDK(clustering) | 63.19 | 46.11 | 72.41 | 67.03 | 59.03 | 51.59 | 65.01 | 74.91 | 55.95 | 25.43 | 26.38 | 55.19 |\n\n**Q2: Cite and discuss shearllama and lora shear.**\n\n**A2:**  Please See **General Response** (**G.Q2**) on the discussion between Sheared LLaMA and DDK.\n\nThe discussion between LoRAShear and DDK is as follows. \n\nThe LoraShear is proposed for structured pruning on LLMs, which first creates the dependency graphs over LoRA module and then proceeds progressive structured pruning on LoRA adaptors.  To recover the lost knowledge during pruning, LoRAShear proposes a Dynamic Knowledge Recovery scheme on both the pre-training and instructed fine-tuning datasets to effectively narrow down the performance gap to the full models.\nIn contrast to  LoraShear, DDK aims to efficiently transfer the domain knowledge of the teacher network, and propose a factor smooth updating strategy to stabilize the domain knowledge guided sampling process.\n\nWe will cite these works in our new version.\n\n**Q3: How the methods perform under other KD losses (reversed KLD, JSD, skew-KLD)?**\n\n**A3:** To address the mode-averaging problem of the student model, where the student model learns an overly smooth distribution in an attempt to cover the teacher’s entire support set, the reversed KLD in MiniLLM and generalized GSD in  [R1]  are proposed. However, these on-policy approaches (i.e.,  the reversed KLD and the generalized JSD) show lower efficiency, the skew-KLD in [R2] introduces the adaptive off-policy approach to enhance the efficiency. In contrast to these works, our DDK focuses on efficiently transferring the domain knowledge of the teacher network based on standard forward KD loss, where the motivation and technical details are different from these works a lot. Besides, our DDK is also orthogonal to these KD losses, where we can directly replace the KL divergence in Eq. 3 with the above losses in MiniLLM, [R1] and [R2] easily. In the following table, we report the results of our DDL using different distillation losses, and we observe that these losses have not provided more performance gains for our DDK based on KLD. For this phenomenon, we assume that our DDK focuses on the pre-training setting with a relatively large training data set, where the data quality and data mixture are very important for pretraining (See **G.Q1**). Meanwhile, in MiniLLM and [R1, R2], these losses mainly focus on the SFT phase to improve learning efficiency by addressing the mode-averaging problem. Therefore, we assume that the main contribution on the improvements comes from changing the domain mixture in our DDK. In the future, we will also continue to investigate the optimal distillation loss formulation for our DDK following [R1, R2].\n\n| Models                         | CEval | MMLU  | RACE  | C3    | W.G.  | GSM8K | C.QA  | Arc-E | Arc-C | H.E.  | MBPP  | Avg   |\n| ------------------------------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Student (Qwen1.5-1.8B)         | 59.66 | 44.48 | 69.57 | 58.27 | 57.85 | 38.4  | 64.7  | 70.23 | 50.31 | 11.87 | 18    | 49.39 |\n| +DDK (Default KLD in Eq.3)     | 63.75 | 46.01 | 71.56 | 65.53 | 59.1  | 53.54 | 66.75 | 75.01 | 55.03 | 27.13 | 26.1  | 55.41 |\n| +DDK (Reversed KLD in MiniLLM) | 62.19 | 45.85 | 70.66 | 65.03 | 54.67 | 50.6  | 65.11 | 74.51 | 52.38 | 23.44 | 24.67 | 53.56 |\n| +DDK(JSD in [R1])              | 62.63 | 45.72 | 71.7  | 65.58 | 57.14 | 53.07 | 67.08 | 75.34 | 54.95 | 22.61 | 25.15 | 54.63 |\n| +DDK(skew-KLD in [R2])         | 62.96 | 45.42 | 71.84 | 65.71 | 58.59 | 53.45 | 67.16 | 75.46 | 55.03 | 26.05 | 25.98 | 55.24 |\n\n[R1]: On-policy distillation of language models: Learning from self-generated mistakes, https://arxiv.org/pdf/2306.13649\n \n[R2]: DistiLLM: Towards Streamlined Distillation for Large Language Models, https://arxiv.org/pdf/2402.03898", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s8Bq5I2zjj", "reviewer": "Reviewer_6kDi", "summary": "This paper proposes DDK, a knowledge distillation (KD) framework that distills large language models (LMs) into small LMs. Unlike previous KD methods, DDK dynamically adjusts the domain weights during distillation. Experiments show that DDK outperforms other KD baselines across various tasks.", "questions": "1. What are the training data for the baselines (CPT, TED, KD, and MiniLLM)? Is the data for DDK the same as that for the baseline methods?\n2. In lines 178-179, is the learning rate 3e-5 ($3\\times 10^{-5}$) rather than $3e^{-5}$?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and the method is easy to follow.\n2. The experiments show that DDK outperforms other KD baselines on various tasks.", "weaknesses": "The extra computation introduced by KKD should be considered. It seems KKD requires the inference of a large LM during the training of the small LM. When the teacher model is much larger than the student model (QWen-1.5 14B v.s. QWen-1.5 1.8B), the inference cost of the teacher model would be even larger than training the student model. Therefore, it is more reasonable to compare the performance of the distilled model and the baselines given the same FLOPs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and constructive suggestions. We will address your concerns shown below in detail.\n\n**Q1: Extra computation introduced by DDK should be considered. Compare the performance of the distilled model and the baselines given the same FLOPs.**\n\n**A1**: Thanks for your insightful suggestions. In Table 13 of Appendix B.2, we also report the training costs (i.e., TFLOPs) of different baseline methods, and we observe that the additional computation costs are acceptable when compared with the baseline KD method. Besides, we observe the  ratio between the TFLOPs of DDK and  TFLOPs of CPT is (5.401e8/1.456e8)=3.709. Then, we also perform the experiments by continuing  pre-training the student  model with about 56B tokens, where the TFLOPs is about 5.4e8. The results are shown in the following table. In the table, we observe that when we continue pretraining the student model for more tokens without using the teacher guidance, little gain is obtained. We assume that the Qwen1.5 model has been trained on >3T tokens, and the convergence status is stable when using the standard pretraining method. In contrast, when using the teacher guidance of the DDK, we observe consistent  performance improvements in Table 1, Table 2, Table 7, and Table 8, which further demonstrates the effectiveness of our DDK.\n\n| Models                | TFLOPs  | CEval | MMLU  | RACE  | C3    | W.G.  | GSM8K | C.QA  | Arc-E | Arc-C | H.E.  | MBPP  | Avg.  |\n| --------------------- | ------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Teacher (Qwen1.5-14B) | -       | 78.68 | 64.34 | 89.95 | 77.38 | 68.74 | 67.63 | 82.06 | 87.58 | 80.59 | 37.80 | 44.00 | 70.80 |\n| Student (Qwen1.5-1.8B)        | -       | 59.66 | 44.48 | 69.57 | 58.27 | 57.85 | 38.4  | 64.70 | 70.23 | 50.31 | 11.87 | 18.00 | 49.39 |\n| +CPT (15B tokens)     | 1.456e8 | 60.13 | 45.01 | 69.00 | 60.30 | 56.98 | 42.50 | 64.78 | 72.00 | 51.03 | 13.12 | 20.45 | 50.48 |\n| +DDK (15B toekns)            | 5.401e8 | 63.75 | 46.01 | 71.56 | 65.53 | 59.10 | 53.54 | 66.75 | 75.01 | 55.03 | 27.13 | 26.10 | 55.41 |\n| +CPT (56B tokens)     | 5.436e8 | 60.15 | 45.00 | 69.16 | 60.43 | 57.06 | 42.91 | 64.86 | 72.14 | 51.19 | 13.41 | 20.74 | 50.64 |\n\nWe will revise the above discussion in our new version.\n\n**Q2: Is the data for DDK the same as that for the baseline methods?**\n\n**A2**: The training data is the same and we adopt the same seed for shuffling the dataset, which can ensure the training data  same for different methods. We will clarify this detail in our new version.\n\n**Q3: Learning rate issue.**\n\n**A3**: Thanks for your kind suggestions. We will revise this typo in the new version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **General Response**

Thanks a lot for handling/reviewing our submitted manuscript. We would like to thank the reviewers for their thoughtful and constructive comments and suggestions. By addressing each of the issues raised by the reviewers, we believe that the quality and clarity of our DDK can be improved a lot. The general responses are summarized as follows:

**G.Q1: The advantages and novelties of this paper's design.**

**A1:** The pretraining datasets for  LLMs are typically sampled from a mixture of many domains. These data from different domains interplay with each other, showing complex interchangeable, unrelated, or contradictory relationships. Besides, many works have claimed that the data mixture of the pretraining data greatly affects the effectiveness of LLMs[R1,R2,R3,R4,R5,R6] and selecting the optimal domain mixture is a challenging problem. Based on knowledge distillation, our DDK aims to produce effective lightweight LLMs by transferring the domain knowledge of the teacher network. Specifically, in Fig. 1 of the main paper, we observe that the performance gap between student (Qwen-1.5 1.8B) and teacher (Qwen-1.5 14B) varies significantly across domains, which means that we need to adjust the domain mixture to better improve the weaker domains by reallocating more data sampling weights. Besides, in Table 1, Table 2, Table 7 and Table8, our DDK is better than the baseline KD a lot, which shows the effectiveness of changing the domain mixture in distillation. Therefore, we are the first to investigate the effectiveness of domain mixture for distilling LLMs and introduce the DDK to enhance the student LLMs. Meanwhile, we observe that naively changing the domain weight compromises the stability of the distillation (See Fig.3) and propose the factor smooth updating strategy to stabilize the distillation. 

Overall, our DDK is motivated by the effectiveness of the domain mixture on the lightweight LLMs and we design  corresponding strategies to better  transfer the domain knowledge of the teacher network.

**G.Q2: Discuss with Sheared LLaMA.**

**A2:** Datasets for  training LLMs are  sampled from many domains and the data mixture of the pretraining data greatly affects the effectiveness of LLMs[R1,R2,R3,R4,R5,R6]. We should emphasize that the motivation of adjusting domain mixtures is similar for Sheared LLaMA and our DDK, but the solved tasks and  technical details are different a lot.

First, for the solved tasks. the Sheared LLaMA is proposed for structure pruning, while our DDK is proposed for knowledge distillation on the LLMs. Therefore, our DDK is orthogonal to the  Sheared LLaMA and we can further improve the performance of small models pruned by Sheared LLaMA. In the following table, following the setting of Table 2, we propose to use the LLaMA2-13B to distill the Sheared-LLaMA-1.3B model, which is pruned by Sheared LLaMA based on LLaMA2-7B.

| Models                       | MMLU  | RACE  | W.G.  | Arc-E | Arc-C | Avg.  |
| ---------------------------- | ----- | ----- | ----- | ----- | ----- | ----- |
| Student (Sheared-LLaMA-1.3B) | 25.71 | 23.62 | 56.81 | 41.53 | 28.14 | 35.16 |
| +DDK                         | 28.25 | 27.43 | 58.12 | 44.92 | 31.85 | 38.14 |

Second, for the technical details, the sheared LLaMA needs to first fit a scaling function using three open-sourced LLaMA models (i.e., LLAMA2-7B/13B/70B) when pruning the LLaMA model series, where the fitting data points are very limited. In Sheared LLaMA, the authors also claim the estimated reference losses for different domains are also biased as the limited data points for estimating the scaling law parameters. Besides, in many real-world pruning scenarios,  we cannot usually obtain a series of models under the same training setting. Thus, we cannot predict the losses across different domains of the small size model when a series of models are not provided. In contrast, in DDK, we directly use the domain loss predicted by the teacher model as the reference loss without fitting the scaling function, where the teacher predicted losses are used as the accurate guidance to improve the student model. Besides, the Sheared LLaMA is proposed to enhance the training efficiency of the continued pre-training setting of the pruned model, the guidance of the teacher model is not included. In addition, our DDK also introduces the factor smooth updating strategy, which is not used in Sheared LLaMA and can make the change of the domain mixture more stable. Moreover, in the following table, we also provide the results of replacing our proposed factor smooth updating strategy with the strategy of Sheared LLaMA, and observe that DDK is better a lot, which further shows the effectiveness of our factor smooth updating strategy.

| Models                                                       | MMLU  | RACE  | W.G.  | Arc-E | Arc-C | Avg.  |
| ------------------------------------------------------------ | ----- | ----- | ----- | ----- | ----- | ----- |
| Student+DDK(Qwen-1.5-1.8B)                                   | 46.01 | 71.56 | 59.1  | 75.01 | 55.03 | 61.34 |
| Student+DDK using Sheared LLaMA updating strategy (Qwen-1.5-1.8B) | 45.03 | 70.16 | 57.09 | 73.98 | 53.18 | 59.88 |

[R1]. An empirical analysis of compute-optimal large language model training, https://arxiv.org/abs/2203.15556

[R2]. Data selection for language models via importance resampling, https://arxiv.org/abs/2302.03169

[R3]. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning, https://arxiv.org/abs/2310.06694

[R4]. RegMix: Data Mixture as Regression for Language Model Pre-training, https://arxiv.org/abs/2407.01492

[R5]. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance, https://arxiv.org/abs/2403.16952

[R6]. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, https://arxiv.org/abs/2305.10429

Meta-review: The paper works on language model distillation. The paper is a borderline submisison. All reviewers agree to accept the submission, although with borderline accept or weak accept. The distillation of LLMs is a hot and useful topic of LLM research. Although the novelity of the paper is limited, it provides a lot of empirical studies, which would be valuable for other researchers. Besides, the reviewers' comments can be included in the final version in order to improve the overall papre quality.
Decision: Accept (poster)