DDK: Distilling Domain Knowledge for Efficient Large Language Models

Authors: jiaheng  liu
chenchen  zhang
jinyang  guo
yuanxing  zhang
haoran  que
ken  deng
zhiqi  bai
jie  liu
ge  zhang
jiakai  wang
yanan  wu
congnan  liu
jiamang  wang
lin  qu
wenbo  su
bo  zheng
taobao & tmall group of alibaba
alibaba  group
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:41:37

Abstract: Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands.  Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, or utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal performance gaps and insufficient attention to domains with large gaps, reducing overall performance. In this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin.

Main paper: 
 introduction: 
 Recent advancements in Large Language Models (LLMs) such as LLaMA [7, 9, 58, 59 ] have garnered significant attention due to their strong intelligence. However, these models also impose considerable computational and storage demands, particularly in practical deployments such as instant chat, copilot, and query rewriting. Consequently, the development of lightweight yet efficacious LLMs suitable for real-world applications has become an area of increasing research interest. Several small-scale LLMs, e.g., Phi [39] and MiniCPM [32] , have been designed to facilitate rapid inference on devices with limited resources. These models are generally trained from scratch using a large volume of selectively curated high-quality datasets, which could be prohibitive for the broader research community. Meanwhile, there has been a surge in the exploration of model compression techniques [40] to reduce the resource footprint of LLMs. Apart from these techniques, knowledge distillation (KD) emerges as a prominent method for creating effective neural networks, which transfer knowledge from a high-performing teacher model to a compact student model. ). Note that "Chinese CC" denotes "Chinese CommonCrawl".The primary challenges in enhancing the performance of KD approaches on LLMs stem from two main aspects: i) appropriately utilizing the data [3, 67] ; ii) stabilize the distillation process [68] .Recently, it has become increasingly acknowledged that the mixture ratios of various domains within the training dataset substantially affect the performance [20, 67, 69] . Regarding the issue of data composition, the influence of domain-specific mixtures for KD remains underexplored. As shown in Fig. 1 , the performance between Qwen-1.5 1.8B [7] (student) and Qwen-1.5 14B [7] (teacher) reveals that the performance gap varies significantly across domains. For instance, in the "Books" domain, the student model significantly underperforms the teacher model, while in "The Stack" domain, the difference is minimal, which indicates that the "Books" domain is relatively not optimized well for the student model compared to the teacher model, and more data from the "Books" domain should be included. Therefore, we aim to design a knowledge distillation framework that can dynamically adjust the data composition during distillation to reallocate more computation to domains, where the student and teacher models have larger performance gaps.In this paper, we introduce a novel methodology, termed Distill Domain Knowledge for LLMs (DDK), which effectively optimizes domain-specific mixtures to address the performance discrepancy between teacher and student models across different domains. Specifically, DDK begins by quantifying the performance deviations between the teacher and student LLMs using an offline-collected validation dataset across various domains. Next, it periodically re-calculates the domain discrepancy factor based on the performance gap between the teacher and student models. Finally, DDK employs a domain knowledge-guided sampling strategy to sample data from different domains with varying probabilities based on the calculated domain discrepancy factor. Additionally, inspired by the optimization algorithms [36] , we propose a factor smooth updating mechanism to augment the stability and robustness of the DDK approach. For the supervision loss, we minimize the differences in the output logits between the teacher and student models. As demonstrated in Fig. 1 , the performance gap across domains is significantly reduced by DDK.Our main contributions are summarized as follows:• To the best of our knowledge, we are the first to study the influence of domain-specific data mixtures for distilling LLMs, and efficiently transfer the domain knowledge of the teacher network upon the domain weights. • DDK proposes a factor smooth updating strategy to strategically enhance the appropriate focus of the distillation process on targeted domains, which effectively stabilizes the domain knowledge guided sampling process for smoother distillation. • Extensive experiments on multiple benchmark datasets demonstrate the effectiveness and generalization ability of our proposed DDK. 
related works: 
 Large Language Models. The emergence of LLMs [62, 72, 24, 41, 19, 65, 49, 53, 27, 6, 66] marks a significant milestone in the domain of natural language processing, with notable examples including GPT3, Lamda, Palm, and several others [2, 4, 10, 44, 57] . For example, Radford and Narasimhan [50] introduced the GPT model, leveraging multiple layers of transformer decoder blocks, while Meta later developed LLaMA [58] employing an enhanced transformer architecture, subsequently evolved into LLaMA2 [59] . Recent advancements have also seen the application of instruction tuning [13, 63] and learning through human feedback [8, 45, 74] to better align LLMs with human understanding and foster the creation of versatile AI assistants [21, 43] . Despite their potential, LLMs' extensive capabilities are often accompanied by vast sizes [35, 64] , demanding significant computational resources. In this work, we aim to focus on how to produce small LLMs based on the knowledge distillation approach.Knowledge Distillation. Knowledge distillation is a pivotal technique in model compression and acceleration [61, 42, 26, 70, 25] , primarily employed to transfer knowledge from a robust, well-trained teacher model to a compact student model [29, 1, 37] . Recently, several approaches to knowledge distillation tailored for LLMs have been proposed. These approaches can be broadly classified into two categories: White-box KD leverages either the internal parameters or the logits of the teacher LLM during the distillation process [22, 46, 56, 71] . For example, Gu et al. [23] propose that traditional Kullback-Leibler divergence (KLD) objective is inappropriate for open text generation tasks and propose MiniLLM to minimize reverse KLD through policy gradient techniques [54] .Conversely, black-box KD relies solely on the outputs from the teacher model [12, 30, 34, 48, 60] . For example, "Distilling Step-by-Step" strategy [31] employs Chain of Thought (CoT) prompting to provide sophisticated guidance during distillation. These two types of KD approaches mainly focus on aligning the generative behaviors of the teacher and student models. DDK delves into the efficacies of domain-specific distillation, aiming to mitigate the discrepancies in performance between the teacher and student model across different domains. Hence, DDK is fundamentally orthogonal to these methods.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: As mentioned in Appendix A.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 Figure 2 illustrates the comprehensive architecture of the DDK framework. DDK employs a largescale teacher LLM and a comparatively smaller student LLM, with the objective of transferring knowledge from the former to the latter to enhance performance utilizing a specially curated distillation dataset. Initially, the distillation dataset is constructed by randomly sampling from the training corpus. Throughout the distillation process, we continuously assess the domain-specific performance of both the teacher and student LLMs, and use domain knowledge guided sampling to dynamically update the data mixture on the student's abilities within specific domains. As the domain proficiency of the student LLM evolves during distillation, we introduce a factor smooth updating strategy to ensure the robustness of the domain knowledge-guided sampling approach. Finally, DDK provides of a better student LLM, optimized for enhanced performance across targeted domains.The distilled student LLMs are anticipated to exhibit robust competence across various preset domains. Nevertheless, prevailing knowledge distillation techniques tailored for LLMs tend to homogeneously optimize performance across these domains, leading to potential performance degradation. To address this issue, we design the domain knowledge guided sampling strategy to enhance distillation efficacy by prioritizing domain-specific complexities.Domain discrepancy factor construction. We consider a dataset D that has been partitioned into N distinct domains. We denote the pre-trained teacher LLM as M T and the student model, which is currently under training, as M S . To efficiently identify and prioritize data that may yield the most learning benefit, particularly from domains where the student model underperforms, we introduce a domain discrepancy factor denoted as r ∈ R N . Each component r[i] of this vector quantitatively represents the discrepancy in performance between the teacher and student models within the i-th domain. As we assume a good student should exhibit close approximation to the teacher across all domains, r is calibrated to reflect differential performance indices as follows: Here, V i and Y i are the inputs and the ground-truth labels of the validation dataset of the ith domain. CE(•) represents the cross-entropy loss. ℓ S ∈ R N and ℓ T ∈ R N are the perplexity scores over the validation sets of all domains for student and teacher respectively, indexed by the domain index i. In this case, a higher value of r[i] signifies a pronounced disparity in domain-specific proficiency between the student model and the teacher model. Accordingly, it is imperative to allocate more relevant data to enhance the domain expertise.EQUATIONDomain knowledge guided sampling. We employ a domain knowledge-informed sampling strategy to refine the composition of the distillation dataset, which utilizes a probabilistic mechanism defined by vector r to iteratively select samples from the training corpus. The process continues cyclically once a domain data has been exhausted. Finally, DDK strategically increases the data allocation towards underperforming domains, thereby mitigating the performance discrepancies between the teacher and student models across all domains.With the domain knowledge guided sampling strategy, we can dynamically focus on more challenging domains during the distillation process. Nonetheless, we observe that the domain discrepancy factor exhibits significant fluctuations throughout this procedure. Such rapid alterations may precipitate exceedingly unbalanced data sampling, potentially compromising the stability of the distillation.Factor smooth updating. To enhance the stability of the distillation process, we periodically adjust the domain discrepancy factor every K iterations throughout the distillation process, thereby partitioning it into discrete intervals. The parameter K is pivotal as it governs the system's capacity to address immediate discrepancies and influences the stability of the data mixture. We denote the domain discrepancy factor for the i-th domain at the t-th interval of distillation as r t [i]. Similarly, let ℓ t S [i] and ℓ t T [i] denote the perplexity scores at the beginning of the t-th distillation interval. In DDK, the domain discrepancy factor at the (t + 1)-th interval is defined as:EQUATIONNote that a constant term is incorporated in r t [i] to preclude the occurrence of excessively small values, thereby guaranteeing a baseline probability for data sampling across various domains. The parameter α, designated as the smoothing coefficient, is fixed at a value of 0.5 in our experimental Read a batch of samples and use Eq. 3 to update the parameters of student LLM;7:c = c + 1 8: if c mod K == 0 then 9:// Update distillation data mixture Stop the distillation loop; Output: The distilled student LLM; setup. In addition, the inclusion of ψ t imparts a history mixture information on the modification of the domain discrepancy factor. This mechanism facilitates a gradual modification of r t [i], thereby minimizing fluctuations and ensuring a stable, domain knowledge-driven distillation process for fetching informative data.As we jointly update the student LLM parameters and the domain discrepancy factor in the distillation process, the optimization object can be written as follows:EQUATIONwhere θ S is the parameters of the student model. z S (•) and z T (•) are the output hidden states from student and teacher LLMs, respectively. We leverage KL-divergence to approximate the student model's output to the teacher model's output, over a distillation temperature T . γ is the factor to balance these two terms. Algorithm 1 summarizes the pseudo-code of the DDK process. In practice, the distillation process is typically concluded either when all available data has been fully utilized or when the domain discrepancy factor approaches a threshold indicative of minimal disparity between the teacher and student models.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: All data used in this paper is open-sourced. The codes for the baseline methods are also collected from Github.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we make comprehensive evaluations to answer two research questions: RQ1: To what extent does the DDK process improve the performance of a small-scale LLM? RQ2: How does the dynamic domain-specific guidance contribute to the overall improvement?Model configuration details. We use the Qwen-1.5 [7] and LLaMA2 [59] to demonstrate the effectiveness of DDK. Regarding the Qwen-1.5 series, we use Qwen-1.5 14B and Qwen-1.5 1.8B as the teacher and student models, respectively. For LLaMA2 series, we use LLaMA2 13B and TinyLLaMA 1.1B [73] as the teacher and student models, respectively.Training details. Due to the unavailability of training data for LLaMA2 and Qwen-1.5 models, we mainly utilize RedPajama [16] for distillation, which consists of training data derived from seven distinct domains: CommonCrawl, C4, The Stack, Wikipedia, Books, ArXiv, and StackExchange. Moreover, to enhance the model's proficiency in Chinese and Mathematics, we also incorporate three cleaned open-source datasets (i.e., Chinese Books [19] , Chinese CommonCrawl [19] , and OpenWebMath [47] ). Therefore, there are ten domain datasets for the distillation. In addition, to assess the disparity in performance between teacher and student models across the ten domains, we have constructed a domain-specific validation set for each domain, where each domain includes 500 samples. During the distillation phase, the student models are trained on approximately 15B tokens. For the training framework, we employ the DeepSpeed-Chat code1 as our codebase, and conduct all experiments using 16 NVIDIA A100 GPUs (80G), where FlashAttention V2 [17] is used to accelerate training. For the training schedule, we first apply the warm-up strategy to increase the learning rate from 0 to 3e -5 in 1,000 steps. Then, we use the cosine learning rate schedule, where the final learning rate is 3e -6 and the whole training step is about 30,000 steps. Empirically, we set the distillation interval K as 1,000 and the temperature T as 1.0.Evaluation details. As we do not conduct instruction tuning on the student models, we mainly report the zero-shot, close-ended results across commonly used datasets including C-Eval [33] (val), MMLU [28] (test), RACE [38] (high, test), C3 [52] (test), WinoGrande [51] (val), GSM8K [15] (test), CommonsenseQA [55] (val), Arc-E [14] (test), Arc-C [14] (test) and HumanEval [11] (test). We also report the 3-shot performance on MBPP [5] (test).Baseline details. We compare DDK with four baseline methods:• CPT denotes that we continue to pre-train the student model by using the same number of training tokens without considering domains.• KD [29] denotes the standard knowledge distillation by computing the KLD between the teacher and student logits without considering domains.• TED [40] denotes to use task-aware filters to align the hidden representations of the student and the teacher at each transformer layer.• MiniLLM [23] denotes to replace the forward KL divergence with reverse KL divergence, which prevents the student model from overestimating the low-probability regions of the teacher distribution. As shown in Table 1 -2, we report the performance results of different baseline methods. The following observations provide a comprehensive response to RQ1: (1) The integration of KD and domain knowledge guided sampling plays a pivotal role. By comparing the results of the "CPT&DoReMi" configuration against the "CPT" alone, we see that the absence of knowledge transfer from the teacher model significantly impedes the student model's capabilities in intricate tasks such as coding (e.g., HumanEval) and Chinese comprehension (e.g., C3). ( 2) DDK outperforms other baseline methods when using different types of teacher and student models, which demonstrates the effectiveness of DDK for training small student LLMs. (3) The baseline methods KD, TED, and MiniLLM exhibit similar performance. For instance, the average accuracy of these three approaches hovers around 52% when distilling onto the Qwen student model. We hypothesize that in the context of LLM distillation, domain data mixture may emerge as a key performance bottleneck, and the existing baseline techniques fail to adequately address this challenge. (4) The performance gains vary across different domains. Notably, when distilling the Qwen model, we achieve significant improvements on the reasoning tasks (e.g., Code on Humaneval and MBPP, Math on GSM8K), which indicates that the student model can improve a lot on the reasoning tasks under the guidance of the teacher model. This empirical observation suggests that DDK is successful in directing additional attention toward the more challenging problem domains.In this section, we perform ablation studies to assess the robustness of the DDK model and its sensitivity to key hyperparameters. We collected data using Qwen 1.5 and reported its performance on the validation sets of MMLU, RACE, and ARC-C, which differ from those discussed in the previous subsection. Initially, we concentrate on addressing RQ1 through fine-grained analyses.DDK DDK (w/o FS) DDK (ES) 40 Effect of data sampling strategies. We propose two variants of data sampling strategies on DDK. For DDK (w/o FS), we just remove the factor smooth updating mechanism and directly take r t as the probability of each domain. For DDK (ES), we sample data from each domain equally. The results are shown in Fig. 4 , and we can suppose that both factor smooth updating and domain knowledge guided sampling contribute to the distillation owing to the existence of domain-specific discrepancy.Effect of distillation interval. Fig. 3 (a) shows the evaluation results on the effect of the distillation interval hyperparameter (i.e., K) in Alg. 1. We observe that increasing K from 100 to 1,000 leads to better performance, indicating that a rapid updating frequency may destabilize the distillation process. However, further increasing K leads to inferior results. We conclude that when the updating frequency is small, the domain weights update quickly and the student LLM weights can not be sufficiently optimized for the current distillation interval. Meanwhile, when the updating frequency is large, there is insufficient alignment between the LLM weights and the optimal domain weights.Effect of training tokens. As shown in Fig. 3 (b) , we investigate the relationship between the results on three representative datasets and the number of training steps for Qwen-1.5 1.8B model when using Qwen-1.5 14B model as teacher. At the first 10B tokens, the results improve quickly, which indicates that the student models can benefit a lot with the supervision of the teacher model. When further increasing the training iterations, we observe that the performance tends to plateau, which indicates a fast convergence of distillation by DDK.We provide further investigation to show the applicability of DDK across more scenarios.Generalization ability of using different teacher / student models. To show the generalization ability of DDK on different student models, we use Qwen-1.5 14B as the teacher model and use Qwen-1.5 4B as the student model. As shown in Table 3 , DDK surpasses the baseline methods by a large margin. Additionally, comparative analysis with the enhancements observed when employing Qwen-1.5 1.8B as the student model, as presented in Table 1 , verifies that a more capable student model tends to yield superior performance improvements. We then apply another teacher model to show the generalization ability of DDK. Specifically, we take Qwen-1.5 7B and Qwen-1.5 1.8B as teacher and student models, respectively. As documented in Table 4 , DDK consistently facilitates the most substantial enhancement. These results demonstrate the efficacy and robustness of DDK in leveraging diverse teacher-student model configurations.Generalization ability on Code LLMs. We implement DDK on LLMs, selecting the Code LLM StarCoder as a case study for empirical evaluation. Within the StarCoder series, we deploy StarCoder 15.5B as the teacher model and StarCoder 3B as the student model. The training corpus is primarily derived from four programming language domains-Python, Java, TypeScript, and C#-sampled from The Stack V2 dataset2 , with each language representing a distinct domain. We report the performance on the repository-level code completion dataset (i.e., CrossCodeEval [18] ). The results in Table 5 affirm that DDK brings notable enhancements in the performance of StarCoder 3B, thereby highlighting DDK's efficacy in tackling the vertical distillation task.Table 3 : Results of different methods on the Qwen-1.5 models. Note that we use Qwen-1.5 14B and Qwen-1.5 4B as teacher and student models, respectively. Analysis on the in-context learning abilities. We evaluate in-context learning capabilities utilizing DDK and the other baselines through several few-shot benchmarks in Table 6 . As shown in Table 6 , we observe that our DDK consistently manifests considerable enhancements in performance, affirming Visualization. To better show the effectiveness of the factor smooth strategy in DDK, we compare the DDK (w/o FS) with our DDK by showing the domain discrepancy in the training process, where DDK (w/o FS) means that we remove the factor smooth updating strategy. Specifically, in Fig. 5 , we compute the (ℓS [i]/ℓ T [i]) as the ratio to represent the domain discrepancy for i-th domain, where a large ratio means a large discrepancy. As shown in Fig. 5 , we observe that the ratio updates smoothly in DDK. Besides, in Table 4 , the DDK is better than DDK (w/o FS), which means DDK can benefit a lot when using the factor smooth updating strategy.  
conclusion: 
 In this study, we introduce DDK, a novel framework for knowledge distillation tailored for LLMs. Our initial investigations underscore the criticality of optimizing domain data mixtures in the context of LLM distillation. To address this, we propose a domain knowledge-guided sampling approach that dynamically modulates the sampling probabilities across various domains. Furthermore, we put forward a factor smooth update strategy aimed at enhancing both the stability and the efficacy of the distillation process. Comprehensive evaluations of several benchmark datasets with diverse teacher-student model configurations demonstrate the effectiveness of the DDK framework. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This information is provided in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have made sufficient ablation studies to show the confidence of the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all relevant papers.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Broader Impacts. Based on our DDK, we can boost the performance of small LLMs a lot under the guidance of the teacher LLMs. In real-world scenes, we hope our enhanced small LLMs can be deployed on low-resource devices (e.g., PCs or Mobiles) and make future life better with better small LLMs.Limitations. First, we have to tune several hyperparameters to achieve better performance in our DDK. Second, due to the GPU resource limitations, we only distill the 1.1B, 1.8B and 4B sizes under the guidance of teachers with 7B and 10B+ sizes. In the future, we will investigate the performance of knowledge distillation using larger student and larger teacher models.B.1 Discussion on the differences between DDK and Sheared LLaMA First, for the solved tasks. the Sheared LLaMA [66] is proposed for structure pruning, while our DDK is proposed for knowledge distillation on the LLMs. Therefore, our DDK is orthogonal to the Sheared LLaMA and we can further improve the performance of small models pruned by Sheared LLaMA. Second, for the technical details, the sheared LLaMA needs to first fit a scaling function using three open-sourced LLaMA models (i.e., LLAMA2-7B/13B/70B) when pruning the LLaMA model series, where the fitting data points are very limited. In Sheared LLaMA, the authors also claim the estimated reference losses for different domains are also biased as the limited data points for estimating the scaling law parameters. Besides, in many real-world pruning scenarios, we cannot usually obtain a series of models under the same training setting. Thus, we cannot predict the losses across different domains of the small-size model when a series of models are not provided. In contrast, in DDK, we directly use the domain loss predicted by the teacher model as the reference loss without fitting the scaling function, where the teacher-predicted losses are used as the accurate guidance to improve the student model. Besides, the Sheared LLaMA is proposed to enhance the training efficiency of the continued pre-training setting of the pruned model, the guidance of the teacher model is not included. In addition, our DDK also introduces the factor smooth updating strategy, which is not used in Sheared LLaMA and can make the change of the domain mixture more stable. Moreover, in the following table, we also provide the results of replacing our proposed factor smooth updating strategy with the strategy of Sheared LLaMA, and observe that DDK is better a lot, which further shows the effectiveness of our factor smooth updating strategy.As shown in Table 7 , we compare the TFLOPs of three representative baseline methods, and observe that the training costs of our DDK are acceptable when compared with the baseline KD method.Table 7 : Training TFLOPs on all data of different methods for Qwen-1.5. For KD and DDK, we use the Qwen-1.5 14B to distill the Qwen-1.5 1.8B.TFLOPs 1.456e8 5.364e8 5.401e8In the following, we provide more examples generated by the original and distilled models. We find that the sentences generated by the distilled model are superior to those generated by the original model in terms of fluency, relevance, and informativeness regarding the given topic.Prompt:问题：冬瓜、黄瓜、西瓜、南瓜都能吃，什么瓜不能吃？ 答案： Assistant (Qwen-1.5 1.8B): 冬瓜、黄瓜、西瓜、南瓜都能吃，南瓜不能吃。 Assistant (Qwen-1.5 1.8B DDK): 傻瓜 解析：冬瓜、黄瓜、西瓜、南瓜都能吃，傻瓜不能吃。 Prompt: 问题：什么东西天气越热，它爬的越高？ 答案：温度计 问题：什么动物，你打死了它却流了你的血？ 答案：蚊子 问题：什么布剪不断？ 答案： Assistant (Qwen-1.5 1.8B): 蚊帐 Assistant (Qwen-1.5 1.8B DDK): 瀑布 Prompt: 如何写一篇关于环保的激励人心的演讲稿？： Assistant (Qwen-1.5 1.8B): 环保是当今社会面临的一个重要问题，它不仅关系到我们的生存环境，也关系到我 们的未来。因此，我们需要采取行动来保护我们的地球。以下是一篇关于环保的激 励人心的演讲稿： 尊敬的各位领导、各位来宾、亲爱的朋友们： 大家好！今天，我非常荣幸能够在这里发表关于环保的演讲。环保是我们每个人都 应该关注的问题，它不仅关系到我们的生存环境，也关系到我们的未来。 首先，让我们来看看环保的重要性。我们的地球是我们的家园，它是我们生存的基 础。然而，由于人类的过度开发和污染，我们的地球正面临着严重的环境问题。例 如，全球变暖、气候变化、水资源短缺、土地退化等问题，这些问题不仅影响着我 们的健康和生活质量，也威胁着我们的生存环境。 因此，我们必须采取行动来保护我们的地球。我们可以从以下几个方面入手： 首先，我们应该减少对环境的污染。我们应该尽可能地减少使用一次性塑料制品， 如塑料袋、塑料瓶等。我们应该尽量使用可重复使用的物品，如购物袋、水瓶等。 此外，我们还可以选择使用环保产品，如环保清洁剂、环保洗衣液等。 其次，我们应该节约能源。我们应该尽可能地减少能源的消耗，如 Assistant (Qwen-1.5 1.8B DDK): 好的，以下是一篇关于环保的激励人心的演讲稿：尊敬的各位来宾，大家好！ 今天，我想和大家分享一些关于环保的话题。我们都知道，环保是我们每个人都应 该关注的问题。我们的地球是我们共同的家园，我们应该尽我们所能来保护它。 首先，我们需要认识到环保的重要性。我们的地球正面临着许多环境问题，如气候 变化、空气污染、水污染等等。这些问题不仅影响着我们的健康和生活质量，也影 响着我们的未来。因此，我们必须采取行动来保护我们的地球。 其次，我们需要采取行动来保护我们的环境。我们可以从小事做起，如节约用水、 节约用电、减少使用塑料袋等等。我们还可以参加环保活动，如植树造林、垃圾分 类等等。这些行动虽然看似微不足道，但它们可以积少成多，对环境产生积极的影 响。 最后，我们需要鼓励更多的人加入到环保行动中来。我们可以向身边的人宣传环保 知识，鼓励他们采取环保行动。我们还可以加入环保组织，参与环保活动，为环保 事业贡献自己的力量。 尊敬的各位来宾，让我们一起行动起来，保护我们的地球，让我们的家园更加美 好！谢谢大家！ NeurIPS Paper Checklist 1.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We have highlighted the main contribution of this paper in the abstract and instruction sections -DKK focuses on the domain discrepancies to adaptively improve the efficacy of knowledge distillation tailored to LLMs.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/ DeepSpeed-Chathttps://huggingface.co/datasets/bigcode/the-stack-v2 


Decision: The paper works on language model distillation. The paper is a borderline submisison. All reviewers agree to accept the submission, although with borderline accept or weak accept. The distillation of LLMs is a hot and useful topic of LLM research. Although the novelity of the paper is limited, it provides a lot of empirical studies, which would be valuable for other researchers. Besides, the reviewers' comments can be included in the final version in order to improve the overall papre quality.

Reviews: review: paper_summary: paper_summary: This work proposed a KD strategy for LLMs. Specifically, with assess to the domain-specific performance of both the teacher and student LLMs, DDK uses domain knowledge guided sampling to dynamically update the data mixture. In addition the paper also conducts a statistical analysis of the domain distribution of the datasets involved. The training process is relatively straightforward and easy to generalize. The experimental results also show that DDK's training method improves the performance average of different data sets.
strength_weakness: strength_weakness: 1. Although the method proposed in this paper is easy to understand and effective, I doubt that the method in this paper is limited to LLMs. In other words, this paper does not mention (or needs to explain) how previous researchers (before LLMs) performed domain-enhanced distillation for domain-biased datasets, and why these previous methods cannot be applied to the distillation of LLMs to achieve similar results. The advantages and novelty of this paper's domain sampling method over previous work that may be transferable to LLMs need further explanation.
2. In the experimental part, there is a lack of key comparison between DDK and other methods that focus on similar domain sampling. The baseline actually involves the work that focuses on domain in KD (cited as [60], etc.), but the subsequent analysis only compares the total average score of DDK and these works, which seems to lack comparison and analysis of similar works. As far as I know, other baselines are more general KDs, and do not focus on domain information.
It is certainly worth noting that DDK performs better than baselines such as MiniLLM, but I think what can better illustrate the effectiveness and novelty of this paper is the comparison with similar domain data sampling, including experimental analysis.
3. In the experimental section, you can add experiments on the dataset and the scale property of the teacher model. This is a possible suggestion.
questions: questions: The questions I expect to ask would be similar to the above section.
limitations: limitations: The method proposed in this paper is very complete and solves the problem well, but there are perhaps two points to note:
1. The distillation method designed in this paper does not seem to be necessarily limited to LLMs. One of the main difficulties of KD on LLM may be that the distribution difference between the teacher and student models is too large, but the key points that this paper focuses on and solves seem to be orthogonal to this point. So it also leads to a similar question: why previous similar methods that focus on domain sampling cannot be migrated to LLMs, and what are the advantages and novelties of this paper's design.
2. Following point 1, in the experiment, what are the specific advantages of this method over the predecessors in the domain problem (not just the overall average, of which result shows DDK outperformed others). This may be what I am very curious about after reading the paper.
paper_summary: paper_summary: The work introduces a novel framework for knowledge distillation (KD) for LLMs. The key innovation of DDK is its dynamic adjustment of the distillation dataset composition based on domain performance differences between the teacher and student models. The paper presents extensive evaluations demonstrating that DDK significantly improves performance in various KD settings, outperforming both continuous training baselines and existing KD methods.
strength_weakness: strength_weakness: 1. Discuss the difference between DDK and the Dynamic Batch Loading proposed by Sheared LLaMA[1], which is also 
 proposed to adjust domain proportions for dynamically training smaller models. They also identify discrepancies in loss between smaller and larger models across various domains, and accordingly, they sample more data from domains where the discrepancy is more pronounced. While they concentrate on structural pruning, it is akin to the DDK. Consequently, I perceive the novelty of DDK as being somewhat limited.
2. The results of Qwen 1.5 in Table 1 are not significantly convincing. The MMLU/HumanEval of Qwen 1.5 1.8B in the Qwen official blog are  46.8/20.1 while the authors' report is 44.5/11.9. In addition, compared to the official results, we can see that the DDK fails to improve the model of the students on MMLU. The authors need to check this and provide **more robust results of baselines**.

[1] SHEARED LLAMA: ACCELERATING LANGUAGE MODEL PRE-TRAINING VIA STRUCTURED PRUNING. Xia et al., 2023
questions: questions: 1. In the setting of paper, the domains are predefined. How to extend the DDK framework for new domain during the distillation training process? Could give more experiments on continual domain learning settings?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces a new framework called Dynamic Domain Knowledge Distillation (DDK) to enhance the efficiency of knowledge distillation for large language models (LLMs). Unlike traditional methods that overlook domain performance differences between student and teacher models, DDK dynamically adjusts the distillation dataset composition based on these differences, ensuring a more stable and effective knowledge transfer. This approach addresses the issue of excessive focus on domains with minimal performance gaps and enhances overall model performance. Extensive evaluations demonstrate that DDK significantly outperforms existing knowledge distillation methods and continuously pretrained baselines.
strength_weakness: strength_weakness: - Dynamic dataloader requires knowing the training data distribution and category beforehand. 
- Missing references. Similar ideas have been explored in pruning LLMs, such as ShearedLLaMA, LoRAShear to recover the knowledge . The paper needs to discuss with them in the related work section due to the closed relation between pruning and KD. 

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery
questions: questions: - How the methods perform on under other KD losses, such as reversed KLD, JSD, skew-KLD.

On-policy distillation of language models: Learning from self-generated mistakes.

DistiLLM: Towards Streamlined Distillation for Large Language Models.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes DDK, a knowledge distillation (KD) framework that distills large language models (LMs) into small LMs. Unlike previous KD methods, DDK dynamically adjusts the domain weights during distillation. Experiments show that DDK outperforms other KD baselines across various tasks.
strength_weakness: strength_weakness: The extra computation introduced by KKD should be considered. It seems KKD requires the inference of a large LM during the training of the small LM. When the teacher model is much larger than the student model (QWen-1.5 14B v.s. QWen-1.5 1.8B), the inference cost of the teacher model would be even larger than training the student model. Therefore, it is more reasonable to compare the performance of the distilled model and the baselines given the same FLOPs.
questions: questions: 1. What are the training data for the baselines (CPT, TED, KD, and MiniLLM)? Is the data for DDK the same as that for the baseline methods?
2. In lines 178-179, is the learning rate 3e-5 ($3\times 10^{-5}$) rather than $3e^{-5}$?
limitations: limitations: N/A

Comments: title: Combine MiniLLM with the domain sampling method of DDK.
Looking forward to Feedback as Discussion Deadline Approaches
Looking forward to Feedback as Discussion Deadline Approaches
Looking forward to feedback on the Responses
Summarization on the Responses
Official Comment by Reviewer 6kDi
comment: Dear **Reviewer foae**,

Hello! We appreciate your support for the detailed suggestions on our paper. We would like to kindly ask you to take a look at our responses and reevaluate our work given our clarifications. Again, we would like to sincerely thank you very much for these constructive comments and evaluation of our manuscript.
The comparison with MiniLLM in a similar domain sampling strategy is also discussed in **Q3** from **Reviewer oZHE**, where we can directly replace the KL divergence loss in Eq. 3 with the reversed KLD loss in MiniLLM. In **A3** for **Q3** from **Reviewer oZHE**, we call this alternative method as **DDK (Reversed KLD in MiniLLM)**, and we observe that replacing the distillation loss does not bring more gains when compared with the DDK with default KL divergence loss. For this phenomenon, just discussed in **A3** , we assume that our DDK focuses on the pre-training setting with a relatively large training data set, where the data quality and data mixture are very important for pretraining (See **G.Q1**). Meanwhile, the MiniLLM focuses on the SFT phase to address the mode-averaging problem in distillation. Therefore, we assume that the main contribution on the improvements comes from changing the domain mixture in our DDK. 

We will include this discussion in our new version.
Thanks for your feedback. We will carefully address your concerns and questions in our new version.
My main concerns and questions are set out in the rebuttal. I suggest that the author add a comparison or discussion of **DYNAMIC BATCH LOADING** in sheared llama in the next version, as I still think they are similar techniques.
Dear reviewers: as you are aware, the reviewer-author discussions phase ends on Aug 13. We request you to kindly make use of the remaining time to contribute productively to these discussions. If you have not read and/or responded to author rebuttal, please do it asap so that the authors get a chance to respond to you. If you have more questions to ask or want further clarification from the authors, please feel free to do it.
Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. As the discussion phase will be closed on Aug 13 11:59 pm AoE, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.
Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. As the discussion phase will be closed on Aug 13 11:59 pm AoE,  we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns. 

Thank you again for dedicating your time to reviewing our paper.
Thanks for your feedback. We will carefully address your concerns in our new version.
I am satisfied with the additional discussions and results, upon which I decide to increase my rate.
Thanks for your feedback. We will carefully address your concerns in our new version.
Dear Reviewers:

Hello! Since the discussion period has started, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address them. We appreciate your time and consideration!
Thanks for handling/reviewing our submitted manuscript: "DDK: Distilling Domain Knowledge for Efficient Large Language Models". We would like to thank the reviewers for their insightful and constructive comments and suggestions. By addressing each of the issues raised by the reviewers, we believe that the quality and clarity of our DDK can be improved a lot. The major responses are summarized as follows:

(1). We have carefully discussed the advantages and novelties of DDK (See Reviewer foae.Q3, Reviewer EYh3.Q1)

(2). We have additionally compared the domain-enhanced KD methods and discussed the advantages over previous methods in detail (See Reviewer foae.Q1&Q4).

(3). We have provided experiments on another training dataset and discussed the scale property of teacher (See Reviewer foae.Q2).

(4). We have provided more clarification and discussion on the evaluation prompts (MMLU and Humeneval) and continual domain learning (See Reviewer EYh3.Q2&Q3&Q4)

(5). We have discussed the limitations of requiring data distribution and provided experiments based on the clustering method to address this limitation (See Reviewer oZHE.Q1).

(6). We have discussed more related works (e.g., Sheared LLaMA, LoRAShear, Skew-KLD) and conducted a detailed analysis of these works (See Reviewer EYh3.Q1, Reviewer oZHE.Q2&Q3).

(7). We have provided a more detailed analysis and clarification on the training costs, training data, and learning rate (See Reviewer 6kDi.Q1&Q2&Q3).

Again, we would like to sincerely thank you very much for these constructive comments and evaluation for our manuscript.
I thank the authors for their response. After reading the response, I think my current score is appropriate.